mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-09 03:18:44 +00:00
Litellm oss staging 04 02 2026 p1 (#25055)
* fix(vertex_ai): support pluggable (executable) credential_source for WIF auth (#24700) The WIF credential dispatch in load_auth() only handled identity_pool and aws credential types. When credential_source.executable was present (used for Azure Managed Identity via Workload Identity Federation), it fell through to identity_pool.Credentials which rejected it with MalformedError. Add dispatch to google.auth.pluggable.Credentials for executable-type credential sources, following the same pattern as the existing identity_pool and aws helpers. Fixes authentication for Azure Container Apps → GCP Vertex AI via WIF with executable credential sources. * feat(logging): add component and logger fields to JSON logs for 3rd p… (#24447) * feat(logging): add component and logger fields to JSON logs for 3rd party filtering * Let user-supplied extra fields win over auto-generated component/logger, tighten test assertions * Feat - Add organization into the metrics metadata for org_id & org_alias (#24440) * Add org_id and org_alias label names to Prometheus metric definitions * Add user_api_key_org_alias to StandardLoggingUserAPIKeyMetadata * Populate user_api_key_org_alias in pre-call metadata * Pass org_id and org_alias into per-request Prometheus metric labels * Add test for org labels on per-request Prometheus metrics * chore: resolve test mockdata * Address review: populate org_alias from DB view, add feature flag, use .get() for org metadata * Add org labels to failure path and verify flag behavior in test * Fix test: build flag-off enum_values without org fields * Gate org labels behind feature flag in get_labels() instead of static metric lists * Scope org label injection to metrics that carry team context, remove orphaned budget label defs, add test teardown * Use explicit metric allowlist for org label injection instead of team heuristic * Fix duplicate org label guard, move _org_label_metrics to class constant * Reset custom_prometheus_metadata_labels after duplicate label assertion * fix: emit org labels by default, remove flag, fix missing org_alias in all metadata paths * fix: emit org labels by default, no opt-in flag required * fix: write org_alias to metadata unconditionally in proxy_server.py * fix: 429s from batch creation being converted to 500 (#24703) * add us gov models (#24660) * add us gov models * added max tokens * Litellm dev 04 02 2026 p1 (#25052) * fix: replace hardcoded url * fix: Anthropic web search cost not tracked for Chat Completions The ModelResponse branch in response_object_includes_web_search_call() only checked url_citation annotations and prompt_tokens_details, missing Anthropic's server_tool_use.web_search_requests field. This caused _handle_web_search_cost() to never fire for Anthropic Claude models. Also routes vertex_ai/claude-* models to the Anthropic cost calculator instead of the Gemini one, since Claude on Vertex uses the same server_tool_use billing structure as the direct Anthropic API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(anthropic): pass logging_obj to client.post for litellm_overhead_time_ms (#24071) When LITELLM_DETAILED_TIMING=true, litellm_overhead_time_ms was null for Anthropic because the handler did not pass logging_obj to client.post(), so track_llm_api_timing could not set llm_api_duration_ms. Pass logging_obj=logging_obj at all four post() call sites (make_call, make_sync_call, acompletion, completion). Add test to ensure make_call passes logging_obj to client.post. Made-with: Cursor * sap - add additional parameters for grounding - additional parameter for grounding added for the sap provider * sap - fix models * (sap) add filtering, masking, translation SAP GEN AI Hub modules * (sap) add tests and docs for new SAP modules * (sap) add support of multiple modules config * (sap) code refactoring * (sap) rename file * test(): add safeguard tests * (sap) update tests * (sap) update docs, solve merge conflict in transformation.py * (sap) linter fix * (sap) Align embedding request transformation with current API * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) mock commit * (sap) run black formater * (sap) add literals to models, add negative tests, fix test for tool transformation * (sap) fix formating * (sap) fix models * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) commit for rerun bot review * (sap) minor improve * (sap) fix after bot review * (sap) lint fix * docs(sap): update documentation * fix(sap): change creds priority * fix(sap): change creds priority * fix(sap): fix sap creds unit test * fix(sap): linter fix * fix(sap): linter fix * linter fix * (sap) update logic of fetching creds, add additional tests * (sap) clean up code * (sap) fix after review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) add a possibility to put the service key by both variants * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) update test * (sap) update service key resolve function * (sap) run black formater * (sap) fix validate credentials, add negative tests for credential fetching * (sap) fix validate credentials, add negative tests for credential fetching * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) lint fix * (sap) lint fix * feat: support service_tier in gemini * chore: add a service_tier field mapping from openai to gemini * fix: use x-gemini-service-tier header in response * docs: add service_tier to gemini docs * chore: add defaut/standard mapping, and some tests * chore: tidying up some case insensitivity * chore: remove unnecessary guard * fix: remove redundant test file * fix: handle 'auto' case-insensitively * fix: return service_tier on final steamed chunk * chore: black * feat: enable supports_service_tier to gemini models * Fix get_standard_logging_metadata tests * Fix test_get_model_info_bedrock_models * Fix test_get_model_info_bedrock_models * Fix remaining tests * Fix mypy issues * Fix tests * Fix merge conflicts * Fix code qa * Fix code qa * Fix code qa * Fix greptile review --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Josh <36064836+J-Byron@users.noreply.github.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Alperen Kömürcü <alperen.koemuercue@sap.com> Co-authored-by: Vasilisa Parshikova <vasilisa.parshikova@sap.com> Co-authored-by: Lin Xu <lin.xu03@sap.com> Co-authored-by: Mark McDonald <macd@google.com> Co-authored-by: Sameer Kankute <sameer@berri.ai>
This commit is contained in:
Krrish Dholakia
GitHub
parent
6a0e0ce061
commit
f42ffed2bd
41 changed files with 3041 additions and 389 deletions
|
|
@ -65,14 +65,13 @@ response = completion(
|
|||
- modalities
|
||||
- reasoning_content
|
||||
- audio (for TTS models only)
|
||||
- service_tier
|
||||
|
||||
**Anthropic Params**
|
||||
- thinking (used to set max budget tokens across anthropic/gemini models)
|
||||
|
||||
[**See Updated List**](https://github.com/BerriAI/litellm/blob/main/litellm/llms/gemini/chat/transformation.py#L70)
|
||||
|
||||
|
||||
|
||||
## Usage - Thinking / `reasoning_content`
|
||||
|
||||
LiteLLM translates OpenAI's `reasoning_effort` to Gemini's `thinking` parameter. [Code](https://github.com/BerriAI/litellm/blob/620664921902d7a9bfb29897a7b27c1a7ef4ddfb/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py#L362)
|
||||
|
|
@ -298,6 +297,19 @@ curl http://0.0.0.0:4000/v1/chat/completions \
|
|||
|
||||
|
||||
|
||||
## Usage - `service_tier`
|
||||
|
||||
LiteLLM propagates OpenAI's `service_tier` parameter to Gemini, and also extracts it from the response headers (`x-gemini-service-tier`) into `model_response.service_tier`.
|
||||
|
||||
| OpenAI `service_tier` | Gemini `service_tier` | Notes |
|
||||
| --------------------- | --------------------- | ----- |
|
||||
| `"auto"` | `"priority"` | LiteLLM maps OpenAI's `"auto"` to Gemini's `"priority"` tier, as `priority` will fall back on Gemini. |
|
||||
| `"flex"` | `"flex"` | Direct mapping. |
|
||||
| `"priority"` | `"priority"` | Direct mapping. |
|
||||
| `"default"` | `"standard"` | LiteLLM maps `"default"` to `"standard"`. |
|
||||
| Any other value | Passed as-is (lowercased) | Values are case-insensitive and normalized to lowercase. |
|
||||
|
||||
On the response, LiteLLM maps `"standard"` back to `"default"` for the Gemini API.
|
||||
|
||||
|
||||
## Text-to-Speech (TTS) Audio Output
|
||||
|
|
|
|||
|
|
@ -55,24 +55,33 @@ pip install litellm
|
|||
```
|
||||
|
||||
### Step 2: Set Your Credentials
|
||||
|
||||
Choose **one** of these authentication methods:
|
||||
|
||||
> **Breaking change**: credential resolution is "first-source-wins"
|
||||
>
|
||||
> Credential resolution no longer merges individual fields across sources.
|
||||
>
|
||||
> Resolution order is:
|
||||
`kwargs` → `service key` → `env (AICORE_*)` → `config` → `VCAP service`
|
||||
>
|
||||
> **Important behavior:** once LiteLLM finds *any* credential value in a source, it takes **all** credentials from that source exclusively (except `resource_group`, which may still be resolved separately).
|
||||
|
||||
Choose **one** of these authentication methods:
|
||||
<Tabs>
|
||||
<TabItem value="service-key" label="Service Key JSON (Recommended)">
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="service-key" label="Service Key JSON (Recommended)">
|
||||
The simplest approach - paste your entire service key as a single environment variable.
|
||||
|
||||
The simplest approach - paste your entire service key as a single environment variable. The service key must be wrapped in a `credentials` object:
|
||||
> **Note:** the service key no more needs to be wrapped in a "credentials" key.
|
||||
|
||||
```bash
|
||||
export AICORE_SERVICE_KEY='{
|
||||
"credentials": {
|
||||
"clientid": "your-client-id",
|
||||
"clientsecret": "your-client-secret",
|
||||
"url": "https://<your-instance>.authentication.sap.hana.ondemand.com",
|
||||
"serviceurls": {
|
||||
"AI_API_URL": "https://api.ai.<your-region>.aws.ml.hana.ondemand.com"
|
||||
}
|
||||
}
|
||||
}'
|
||||
export AICORE_RESOURCE_GROUP="default"
|
||||
```
|
||||
|
|
@ -220,6 +229,17 @@ model="sap/gemini-2.5-pro"
|
|||
# Incorrect - missing prefix
|
||||
model="gpt-4o" # ❌ Won't work
|
||||
```
|
||||
3. **Environment variables** - Set the following list of credentials in .env file
|
||||
<pre>
|
||||
AICORE_AUTH_URL = "https://* * * .authentication.sap.hana.ondemand.com/oauth/token",
|
||||
AICORE_CLIENT_ID = " *** ",
|
||||
AICORE_CLIENT_SECRET = " *** ",
|
||||
AICORE_RESOURCE_GROUP = " *** ",
|
||||
AICORE_BASE_URL = "https://api.ai.***.cfapps.sap.hana.ondemand.com/v2"
|
||||
</pre>
|
||||
|
||||
Other credential configuration options are also available. For more information, see the [SAP AI Core Documentation](https://help.sap.com/doc/generative-ai-hub-sdk/CLOUD/en-US/_reference/README_sphynx.html#configuration).
|
||||
## Usage - LiteLLM Python SDK
|
||||
|
||||
### Proxy Usage
|
||||
|
||||
|
|
@ -506,6 +526,241 @@ response = embedding(
|
|||
print(response.data[0]["embedding"]) # Vector representation
|
||||
```
|
||||
|
||||
### Additional Modules
|
||||
The SAP Gen AI Hub includes additional modules for advanced use cases:
|
||||
- [Grounding](https://help.sap.com/docs/sap-ai-core/generative-ai/grounding-035c455a5a424697b60f4a24b6d791fe?locale=en-US)
|
||||
- [Translation](https://help.sap.com/docs/sap-ai-core/generative-ai/translation?locale=en-US)
|
||||
- [Data Masking](https://help.sap.com/docs/sap-ai-core/generative-ai/data-masking-d9a54d9ca54b40beacbd24e1663ec3b4?locale=en-US)
|
||||
- [Content Filtering](https://help.sap.com/docs/sap-ai-core/generative-ai/content-filtering?locale=en-US)
|
||||
|
||||
#### Grounding
|
||||
Grounding is a service designed to handle data-related tasks, such as grounding and retrieval, using vector databases. It provides specialized data retrieval through these databases, grounding the retrieval process with your own external and context-relevant data. Grounding combines generative AI capabilities with the ability to use real-time, precise data to improve decision-making and business operations for specific AI-driven business solutions.
|
||||
##### Prerequisites
|
||||
To use the Grounding module in the orchestration pipeline, you need to prepare the knowledge base in advance.
|
||||
|
||||
Generative AI hub offers multiple options for users to provide data (prepare a knowledge base):
|
||||
- For Option 1: Upload the documents to a supported data repository and run the data pipeline to vectorize the documents.
|
||||
- For Option 2: Provide the chunks of document via Vector API directly.
|
||||
|
||||
To use grounding, choose from one of the following options.
|
||||
|
||||
Usage example:
|
||||
```python showLineNumbers title="Grounding Example"
|
||||
from litellm import completion
|
||||
|
||||
grounding_config = {
|
||||
'type': 'document_grounding_service',
|
||||
'config': {
|
||||
'filters': [
|
||||
{'id': 's3-docs',
|
||||
'data_repository_type': 'vector',
|
||||
'search_config': {'max_chunk_count': 2},
|
||||
'data_repositories': ['012345-6789-0123-4567-890123456789']
|
||||
}
|
||||
],
|
||||
'placeholders': {'input': ['user_query'], 'output': 'grounding_response'},
|
||||
'metadata_params': ['source', 'webUrl', 'title', 'mimeType', 'fileSuffix']
|
||||
}
|
||||
}
|
||||
|
||||
response = completion(model="sap/gpt-4o",
|
||||
messages=[
|
||||
{"content":"""Facility Solutions Company provides services to luxury residential complexes,
|
||||
apartments, individual homes, and commercial properties such as office buildings, retail
|
||||
spaces, industrial facilities, and educational institutions. Customers are encouraged to
|
||||
reach out with maintenance requests, service deficiencies, follow-ups, or any issues they
|
||||
need by email.""", "role": "system"},
|
||||
{"content":"""You are a helpful assistant for any queries for answering questions.
|
||||
Answer the request by providing relevant answers that fit to the request.
|
||||
Request: {{ ?user_query }}
|
||||
Context:{{ ?grounding_response }}""", "role": "user"}
|
||||
],
|
||||
placeholder_values={"user_query": "Is there a complaint?"},
|
||||
grounding=grounding_config
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
For more information about all available grounding configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/using-grounding-module-e1c4dd100dfb42ab890e1d95f3516187?locale=en-US).
|
||||
|
||||
#### Translation
|
||||
The translation module allows you to translate LLM text prompts into a chosen target language.
|
||||
|
||||
```python showLineNumbers title="Translation Example"
|
||||
from litellm import completion
|
||||
|
||||
translation_config = {
|
||||
'input':
|
||||
{'type': 'sap_document_translation',
|
||||
'config':
|
||||
{'source_language': 'en-US',
|
||||
'target_language': 'de-DE'}
|
||||
},
|
||||
'output':
|
||||
{'type': 'sap_document_translation',
|
||||
'config':
|
||||
{'source_language': 'de-DE',
|
||||
'target_language': 'fr-FR'}
|
||||
}
|
||||
}
|
||||
|
||||
response = completion(model="sap/gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello world!"}],
|
||||
translation=translation_config)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
For more information about all available translation configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/translation?locale=en-US)
|
||||
|
||||
#### Data Masking
|
||||
The data masking module serves to anonymize or pseudonymize personally identifiable information from the input for selected entities.
|
||||
|
||||
```python showLineNumbers title="Data Masking Example"
|
||||
from litellm import completion, embedding
|
||||
masking_config = {
|
||||
'providers':
|
||||
[
|
||||
{
|
||||
'type': 'sap_data_privacy_integration',
|
||||
'method': 'anonymization',
|
||||
'entities': [
|
||||
{'type': 'profile-address'},
|
||||
{'type': 'profile-email'},
|
||||
{'type': 'profile-phone'},
|
||||
{'type': 'profile-person'},
|
||||
{'type': 'profile-location'}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_cv = "some text with personal information"
|
||||
|
||||
response = completion(model="sap/gpt-4o",
|
||||
messages=[{"role": "user", "content": "Give a one sentence summary of the CV. CV: {{?cv}}?"}],
|
||||
placeholder_values={"cv": mock_cv},
|
||||
masking=masking_config)
|
||||
print(response.choices[0].message.content)
|
||||
|
||||
# Data masking module also available for embedding
|
||||
response = embedding(model="sap/text-embedding-3-small",
|
||||
input=mock_cv,
|
||||
masking=masking_config)
|
||||
print(response.data[0])
|
||||
```
|
||||
For more information about all available data masking configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/enhancing-model-consumption-with-data-masking-66ad6f469afc4c2cbaa91a27a33f7b21?locale=en-US)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
#### Content Filtering
|
||||
The content filtering module allows you to filter input and output based on content safety criteria.
|
||||
|
||||
The module supports two services:
|
||||
* Azure Content Safety
|
||||
* Llama Guard 3
|
||||
|
||||
```python showLineNumbers title="Content Filtering Example"
|
||||
from litellm import completion
|
||||
|
||||
filtering_config_azure = {
|
||||
'input':
|
||||
{
|
||||
'filters':
|
||||
[
|
||||
{'type': 'azure_content_safety',
|
||||
'config':
|
||||
{'hate': 0,
|
||||
'sexual': 0,
|
||||
'violence': 0,
|
||||
'self_harm': 0
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
'output':
|
||||
{
|
||||
'filters':
|
||||
[
|
||||
{'type': 'azure_content_safety',
|
||||
'config': {'hate': 0,
|
||||
'sexual': 0,
|
||||
'violence': 0,
|
||||
'self_harm': 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
response = completion(model="sap/gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello world!"}],
|
||||
filtering=filtering_config_azure)
|
||||
print(response.choices[0].message.content)
|
||||
# The model responds normally because the content does not violate any safety rules.
|
||||
|
||||
try:
|
||||
response = completion(model="sap/gpt-4o",
|
||||
messages=[{"role": "user", "content": "I hate you"}],
|
||||
filtering=filtering_config_azure)
|
||||
except Exception as e:
|
||||
print(e)
|
||||
# The service raises an error:
|
||||
# "Input Filter: Content filtered due to safety violations. Please modify the prompt and try again."
|
||||
```
|
||||
For more information about all available content filtering configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/content-filtering?locale=en-US)
|
||||
|
||||
#### List of modules configuration for fallback
|
||||
SAP GEN AI Hub supports a fallback mechanism for handling errors. This mechanism allows you to specify a list of fallback modules to use in case of errors. The fallback modules should contain all parameters that are required for configuring the request.
|
||||
|
||||
Required parameters:
|
||||
- `model`
|
||||
- `messages`
|
||||
|
||||
Optional parameters:
|
||||
- `filtering`
|
||||
- `grounding`
|
||||
- `translation`
|
||||
- `masking`
|
||||
- `tools`
|
||||
|
||||
- and any of model's specific parameters.
|
||||
|
||||
|
||||
```python showLineNumbers title="Fallback Example"
|
||||
from litellm import completion
|
||||
|
||||
translation_config = {
|
||||
'input':
|
||||
{'type': 'sap_document_translation',
|
||||
'config':
|
||||
{'source_language': 'en-US',
|
||||
'target_language': 'de-DE'}
|
||||
},
|
||||
'output':
|
||||
{'type': 'sap_document_translation',
|
||||
'config':
|
||||
{'source_language': 'de-DE',
|
||||
'target_language': 'fr-FR'}
|
||||
}
|
||||
}
|
||||
|
||||
response = completion(model="sap/gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello world!"}],
|
||||
translation=translation_config,
|
||||
fallback_sap_modules=[{
|
||||
"model":"sap/gemini-2.5-flash",
|
||||
"messages":[{"role": "user", "content": "Hello world!"}],
|
||||
"translation":translation_config
|
||||
}])
|
||||
|
||||
# In case of error with the first configuration (model gpt-4o), the fallback module is used.
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
|
||||
```
|
||||
|
||||
|
||||
## Reference
|
||||
|
||||
### Supported Parameters
|
||||
|
|
|
|||
|
|
@ -114,6 +114,7 @@ class PagerDutyAlerting(SlackAlerting):
|
|||
user_api_key_max_budget=_meta.get("user_api_key_max_budget"),
|
||||
user_api_key_budget_reset_at=_meta.get("user_api_key_budget_reset_at"),
|
||||
user_api_key_org_id=_meta.get("user_api_key_org_id"),
|
||||
user_api_key_org_alias=_meta.get("user_api_key_org_alias"),
|
||||
user_api_key_team_id=_meta.get("user_api_key_team_id"),
|
||||
user_api_key_project_id=_meta.get("user_api_key_project_id"),
|
||||
user_api_key_project_alias=_meta.get("user_api_key_project_alias"),
|
||||
|
|
@ -196,6 +197,7 @@ class PagerDutyAlerting(SlackAlerting):
|
|||
else None
|
||||
),
|
||||
user_api_key_org_id=user_api_key_dict.org_id,
|
||||
user_api_key_org_alias=user_api_key_dict.organization_alias,
|
||||
user_api_key_team_id=user_api_key_dict.team_id,
|
||||
user_api_key_project_id=user_api_key_dict.project_id,
|
||||
user_api_key_project_alias=user_api_key_dict.project_alias,
|
||||
|
|
|
|||
|
|
@ -243,6 +243,12 @@ class JsonFormatter(Formatter):
|
|||
if key not in _STANDARD_RECORD_ATTRS and key not in json_record:
|
||||
json_record[key] = value
|
||||
|
||||
# Set component/logger only if not already supplied via extra={...}
|
||||
if "component" not in json_record:
|
||||
json_record["component"] = record.name
|
||||
if "logger" not in json_record:
|
||||
json_record["logger"] = f"{record.filename}:{record.lineno}"
|
||||
|
||||
if record.exc_info:
|
||||
json_record["stacktrace"] = record.exc_text or self.formatException(
|
||||
record.exc_info
|
||||
|
|
|
|||
|
|
@ -21,11 +21,11 @@ class PydanticAIProviderConfig(BaseA2AProviderConfig):
|
|||
request_id: str,
|
||||
params: Dict[str, Any],
|
||||
api_base: Optional[str] = None,
|
||||
**kwargs,
|
||||
**kwargs: Any,
|
||||
) -> Dict[str, Any]:
|
||||
"""Handle non-streaming request to Pydantic AI agent."""
|
||||
if not api_base:
|
||||
raise ValueError("api_base is required for Pydantic AI agents")
|
||||
if api_base is None:
|
||||
raise ValueError("api_base is required for PydanticAIProviderConfig")
|
||||
return await PydanticAIHandler.handle_non_streaming(
|
||||
request_id=request_id,
|
||||
params=params,
|
||||
|
|
|
|||
|
|
@ -1031,6 +1031,9 @@ class PrometheusLogger(CustomLogger):
|
|||
user_api_key_org_id = standard_logging_payload["metadata"].get(
|
||||
"user_api_key_org_id"
|
||||
)
|
||||
user_api_key_org_alias = standard_logging_payload["metadata"].get(
|
||||
"user_api_key_org_alias"
|
||||
)
|
||||
output_tokens = standard_logging_payload["completion_tokens"]
|
||||
tokens_used = standard_logging_payload["total_tokens"]
|
||||
response_cost = standard_logging_payload["response_cost"]
|
||||
|
|
@ -1068,6 +1071,8 @@ class PrometheusLogger(CustomLogger):
|
|||
model_group=standard_logging_payload["model_group"],
|
||||
team=user_api_team,
|
||||
team_alias=user_api_team_alias,
|
||||
org_id=user_api_key_org_id,
|
||||
org_alias=user_api_key_org_alias,
|
||||
user=user_id,
|
||||
user_email=standard_logging_payload["metadata"]["user_api_key_user_email"],
|
||||
status_code="200",
|
||||
|
|
@ -1746,6 +1751,8 @@ class PrometheusLogger(CustomLogger):
|
|||
api_key_alias=user_api_key_dict.key_alias,
|
||||
team=user_api_key_dict.team_id,
|
||||
team_alias=user_api_key_dict.team_alias,
|
||||
org_id=user_api_key_dict.org_id,
|
||||
org_alias=user_api_key_dict.organization_alias,
|
||||
requested_model=request_data.get("model", ""),
|
||||
status_code=str(status_code),
|
||||
exception_status=str(status_code),
|
||||
|
|
|
|||
|
|
@ -4754,6 +4754,7 @@ class StandardLoggingPayloadSetup:
|
|||
user_api_key_budget_reset_at=None,
|
||||
user_api_key_team_id=None,
|
||||
user_api_key_org_id=None,
|
||||
user_api_key_org_alias=None,
|
||||
user_api_key_project_id=None,
|
||||
user_api_key_project_alias=None,
|
||||
user_api_key_user_id=None,
|
||||
|
|
@ -5586,6 +5587,7 @@ def get_standard_logging_metadata(
|
|||
user_api_key_budget_reset_at=None,
|
||||
user_api_key_team_id=None,
|
||||
user_api_key_org_id=None,
|
||||
user_api_key_org_alias=None,
|
||||
user_api_key_project_id=None,
|
||||
user_api_key_project_alias=None,
|
||||
user_api_key_user_id=None,
|
||||
|
|
|
|||
|
|
@ -322,9 +322,8 @@ class StandardBuiltInToolCostTracking:
|
|||
)
|
||||
if has_url_citations:
|
||||
return True
|
||||
# Fallback: Check usage object for providers that use usage instead of annotations
|
||||
# (e.g., Vertex AI Gemini uses usage.prompt_tokens_details.web_search_requests)
|
||||
if usage is not None:
|
||||
# Vertex AI Gemini uses usage.prompt_tokens_details.web_search_requests
|
||||
if (
|
||||
hasattr(usage, "prompt_tokens_details")
|
||||
and usage.prompt_tokens_details is not None
|
||||
|
|
@ -335,6 +334,15 @@ class StandardBuiltInToolCostTracking:
|
|||
and usage.prompt_tokens_details.web_search_requests is not None
|
||||
):
|
||||
return True
|
||||
# Anthropic Claude (direct API and Vertex AI) uses server_tool_use.web_search_requests.
|
||||
# Without this check, Claude ModelResponse always falls through to return False
|
||||
# and _handle_web_search_cost() is never called.
|
||||
if (
|
||||
hasattr(usage, "server_tool_use")
|
||||
and usage.server_tool_use is not None
|
||||
and usage.server_tool_use.web_search_requests is not None
|
||||
):
|
||||
return True
|
||||
return False
|
||||
elif isinstance(response_object, ResponsesAPIResponse):
|
||||
# response api explicitly includes web_search_call in the output
|
||||
|
|
|
|||
|
|
@ -34,6 +34,18 @@ def get_cost_for_web_search_request(
|
|||
|
||||
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
|
||||
elif custom_llm_provider.startswith("vertex_ai"):
|
||||
# Anthropic Claude models on Vertex AI populate server_tool_use.web_search_requests
|
||||
# (same as the direct Anthropic API), not prompt_tokens_details.web_search_requests
|
||||
# (which is the Gemini field). Route claude-* models to the Anthropic calculator.
|
||||
model_key: str = model_info.get("key", "") if model_info else ""
|
||||
if "claude" in model_key.lower():
|
||||
from .anthropic.cost_calculation import get_cost_for_anthropic_web_search
|
||||
|
||||
verbose_logger.debug(
|
||||
"vertex_ai/claude model detected — routing web search cost to Anthropic calculator"
|
||||
)
|
||||
return get_cost_for_anthropic_web_search(model_info=model_info, usage=usage)
|
||||
|
||||
from .vertex_ai.gemini.cost_calculator import (
|
||||
cost_per_web_search_request as cost_per_web_search_request_vertex_ai,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -89,7 +89,12 @@ async def make_call(
|
|||
|
||||
try:
|
||||
response = await client.post(
|
||||
api_base, headers=headers, data=data, stream=True, timeout=timeout
|
||||
api_base,
|
||||
headers=headers,
|
||||
data=data,
|
||||
stream=True,
|
||||
timeout=timeout,
|
||||
logging_obj=logging_obj,
|
||||
)
|
||||
except httpx.HTTPStatusError as e:
|
||||
error_headers = getattr(e, "headers", None)
|
||||
|
|
@ -142,7 +147,12 @@ def make_sync_call(
|
|||
|
||||
try:
|
||||
response = client.post(
|
||||
api_base, headers=headers, data=data, stream=True, timeout=timeout
|
||||
api_base,
|
||||
headers=headers,
|
||||
data=data,
|
||||
stream=True,
|
||||
timeout=timeout,
|
||||
logging_obj=logging_obj,
|
||||
)
|
||||
except httpx.HTTPStatusError as e:
|
||||
error_headers = getattr(e, "headers", None)
|
||||
|
|
@ -266,7 +276,11 @@ class AnthropicChatCompletion(BaseLLM):
|
|||
|
||||
try:
|
||||
response = await async_handler.post(
|
||||
api_base, headers=headers, json=data, timeout=timeout
|
||||
api_base,
|
||||
headers=headers,
|
||||
json=data,
|
||||
timeout=timeout,
|
||||
logging_obj=logging_obj,
|
||||
)
|
||||
except Exception as e:
|
||||
## LOGGING
|
||||
|
|
@ -469,6 +483,7 @@ class AnthropicChatCompletion(BaseLLM):
|
|||
headers=headers,
|
||||
data=json.dumps(data),
|
||||
timeout=timeout,
|
||||
logging_obj=logging_obj,
|
||||
)
|
||||
except Exception as e:
|
||||
status_code = getattr(e, "status_code", 500)
|
||||
|
|
|
|||
|
|
@ -91,6 +91,7 @@ class GoogleAIStudioGeminiConfig(VertexGeminiConfig):
|
|||
"modalities",
|
||||
"parallel_tool_calls",
|
||||
"web_search_options",
|
||||
"service_tier",
|
||||
]
|
||||
if supports_reasoning(model, custom_llm_provider="gemini"):
|
||||
supported_params.append("reasoning_effort")
|
||||
|
|
|
|||
0
litellm/llms/sap/__init__.py
Normal file
0
litellm/llms/sap/__init__.py
Normal file
|
|
@ -1,6 +1,8 @@
|
|||
from typing import Union, Literal
|
||||
from typing import Union, Literal, Optional
|
||||
from enum import Enum
|
||||
import warnings
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from pydantic import BaseModel, Field, field_validator, model_validator
|
||||
|
||||
|
||||
def validate_different_content(v: Union[str, dict, list]) -> str:
|
||||
|
|
@ -20,7 +22,7 @@ def validate_different_content(v: Union[str, dict, list]) -> str:
|
|||
elif isinstance(v, str):
|
||||
return v
|
||||
raise ValueError("Content must be a string")
|
||||
return v
|
||||
|
||||
|
||||
|
||||
class TextContent(BaseModel):
|
||||
|
|
@ -49,6 +51,10 @@ class FunctionTool(BaseModel):
|
|||
parameters: dict = {"type": "object", "properties": {}}
|
||||
strict: bool = False
|
||||
|
||||
def model_dump(self, **kwargs) -> dict:
|
||||
kwargs["exclude_unset"] = False
|
||||
return super().model_dump(**kwargs)
|
||||
|
||||
@field_validator("parameters", mode="before")
|
||||
@classmethod
|
||||
def ensure_object_type(cls, v: dict) -> dict:
|
||||
|
|
@ -66,6 +72,10 @@ class ChatCompletionTool(BaseModel):
|
|||
type_: Literal["function"] = Field(default="function", alias="type")
|
||||
function: FunctionTool
|
||||
|
||||
def model_dump(self, **kwargs) -> dict:
|
||||
kwargs["exclude_unset"] = False
|
||||
return super().model_dump(**kwargs)
|
||||
|
||||
|
||||
class MessageToolCall(BaseModel):
|
||||
id: str
|
||||
|
|
@ -114,6 +124,9 @@ class SAPToolChatMessage(BaseModel):
|
|||
)
|
||||
|
||||
|
||||
ChatMessage = Union[SAPMessage, SAPUserMessage, SAPAssistantMessage, SAPToolChatMessage]
|
||||
|
||||
|
||||
class ResponseFormat(BaseModel):
|
||||
type_: Literal["text", "json_object"] = Field(default="text", alias="type")
|
||||
|
||||
|
|
@ -128,3 +141,607 @@ class JSONResponseSchema(BaseModel):
|
|||
class ResponseFormatJSONSchema(BaseModel):
|
||||
type_: Literal["json_schema"] = Field(default="json_schema", alias="type")
|
||||
json_schema: JSONResponseSchema
|
||||
|
||||
|
||||
class KeyValueListPair(BaseModel):
|
||||
key: str
|
||||
value: list[str]
|
||||
|
||||
|
||||
class DocumentMetadataKeyValueListPairs(KeyValueListPair):
|
||||
select_mode: Optional[list[Literal["ignoreIfKeyAbsent"]]] = None
|
||||
|
||||
|
||||
class GroundingSearchConfig(BaseModel):
|
||||
max_chunk_count: Optional[int] = Field(default=None, ge=0)
|
||||
max_document_count: Optional[int] = Field(default=None, ge=0)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_max_chunk_count_and_max_document_count(self):
|
||||
if self.max_chunk_count is not None and self.max_document_count is not None:
|
||||
raise ValueError("Cannot specify both maxChunkCount and maxDocumentCount.")
|
||||
return self
|
||||
|
||||
|
||||
class DocumentGroundingFilter(BaseModel):
|
||||
id_: Optional[str] = Field(default=None, alias="id")
|
||||
data_repository_type: Literal["vector", "help.sap.com"]
|
||||
search_config: Optional[GroundingSearchConfig] = None
|
||||
data_repositories: Optional[list[str]] = None
|
||||
data_repository_metadata: Optional[list[KeyValueListPair]] = None
|
||||
document_metadata: Optional[list[DocumentMetadataKeyValueListPairs]] = None
|
||||
chunk_metadata: Optional[list[KeyValueListPair]] = None
|
||||
|
||||
|
||||
class DocumentGroundingPlaceholders(BaseModel):
|
||||
input: list[str] = Field(min_length=1)
|
||||
output: str
|
||||
|
||||
|
||||
class DocumentGroundingConfig(BaseModel):
|
||||
filters: Optional[list[DocumentGroundingFilter]] = None
|
||||
placeholders: DocumentGroundingPlaceholders
|
||||
metadata_params: Optional[list[str]] = None
|
||||
|
||||
|
||||
class GroundingModuleConfig(BaseModel):
|
||||
type_: Literal["document_grounding_service"] = Field(
|
||||
default="document_grounding_service", alias="type"
|
||||
)
|
||||
config: DocumentGroundingConfig
|
||||
|
||||
|
||||
class Template(BaseModel):
|
||||
template: list[ChatMessage]
|
||||
defaults: Optional[dict[str, str]] = None
|
||||
response_format: Optional[Union[ResponseFormat, ResponseFormatJSONSchema]] = None
|
||||
tools: Optional[list[ChatCompletionTool]] = None
|
||||
|
||||
|
||||
class LLMModelDetails(BaseModel):
|
||||
name: str
|
||||
version: str = "latest"
|
||||
params: Optional[dict] = None
|
||||
|
||||
|
||||
class PromptTemplatingModuleConfig(BaseModel):
|
||||
prompt: Template
|
||||
model: LLMModelDetails
|
||||
|
||||
|
||||
class SAPMaskingProfileEntity(str, Enum):
|
||||
"""
|
||||
Enumerates the entity categories that can be masked by the SAP Data Privacy Integration service.
|
||||
|
||||
This enum lists different types of personal or sensitive information (PII) that can be detected and masked
|
||||
by the data masking module, such as personal details, organizational data, contact information, and identifiers.
|
||||
|
||||
Values:
|
||||
PERSON: Represents personal names.
|
||||
|
||||
ORG: Represents organizational names.
|
||||
|
||||
UNIVERSITY: Represents educational institutions.
|
||||
|
||||
LOCATION: Represents geographical locations.
|
||||
|
||||
EMAIL: Represents email addresses.
|
||||
|
||||
PHONE: Represents phone numbers.
|
||||
|
||||
ADDRESS: Represents physical addresses.
|
||||
|
||||
SAP_IDS_INTERNAL: Represents internal SAP identifiers.
|
||||
|
||||
SAP_IDS_PUBLIC: Represents public SAP identifiers.
|
||||
|
||||
URL: Represents URLs.
|
||||
|
||||
USERNAME_PASSWORD: Represents usernames and passwords.
|
||||
|
||||
NATIONAL_ID: Represents national identification numbers.
|
||||
|
||||
IBAN: Represents International Bank Account Numbers.
|
||||
|
||||
SSN: Represents Social Security Numbers.
|
||||
|
||||
CREDIT_CARD_NUMBER: Represents credit card numbers.
|
||||
|
||||
PASSPORT: Represents passport numbers.
|
||||
|
||||
DRIVING_LICENSE: Represents driving license numbers.
|
||||
|
||||
NATIONALITY: Represents nationality information.
|
||||
|
||||
RELIGIOUS_GROUP: Represents religious group affiliation.
|
||||
|
||||
POLITICAL_GROUP: Represents political group affiliation.
|
||||
|
||||
PRONOUNS_GENDER: Represents pronouns and gender identity.
|
||||
|
||||
GENDER: Represents gender information.
|
||||
|
||||
SEXUAL_ORIENTATION: Represents sexual orientation.
|
||||
|
||||
TRADE_UNION: Represents trade union membership.
|
||||
|
||||
SENSITIVE_DATA: Represents any other sensitive information.
|
||||
"""
|
||||
|
||||
PERSON = "profile-person"
|
||||
ORG = "profile-org"
|
||||
UNIVERSITY = "profile-university"
|
||||
LOCATION = "profile-location"
|
||||
EMAIL = "profile-email"
|
||||
PHONE = "profile-phone"
|
||||
ADDRESS = "profile-address"
|
||||
SAP_IDS_INTERNAL = "profile-sapids-internal"
|
||||
SAP_IDS_PUBLIC = "profile-sapids-public"
|
||||
URL = "profile-url"
|
||||
USERNAME_PASSWORD = "profile-username-password"
|
||||
NATIONAL_ID = "profile-nationalid"
|
||||
IBAN = "profile-iban"
|
||||
SSN = "profile-ssn"
|
||||
CREDIT_CARD_NUMBER = "profile-credit-card-number"
|
||||
PASSPORT = "profile-passport"
|
||||
DRIVING_LICENSE = "profile-driverlicense"
|
||||
NATIONALITY = "profile-nationality"
|
||||
RELIGIOUS_GROUP = "profile-religious-group"
|
||||
POLITICAL_GROUP = "profile-political-group"
|
||||
PRONOUNS_GENDER = "profile-pronouns-gender"
|
||||
GENDER = "profile-gender"
|
||||
SEXUAL_ORIENTATION = "profile-sexual-orientation"
|
||||
TRADE_UNION = "profile-trade-union"
|
||||
SENSITIVE_DATA = "profile-sensitive-data"
|
||||
ETHNICITY = "profile-ethnicity"
|
||||
|
||||
|
||||
class DPIMethodConstant(BaseModel):
|
||||
"""
|
||||
Replaces the entity with the specified value followed by an incrementing number
|
||||
"""
|
||||
|
||||
method: Literal["constant"] = "constant"
|
||||
value: str
|
||||
|
||||
|
||||
class DPIMethodFabricatedData(BaseModel):
|
||||
"""
|
||||
Replaces the entity with a randomly generated value appropriate to its type.
|
||||
"""
|
||||
|
||||
method: Literal["fabricated_data"] = "fabricated_data"
|
||||
|
||||
|
||||
class DPICustomEntity(BaseModel):
|
||||
"""
|
||||
regex: Regular expression to match the entity
|
||||
replacement_strategy: Replacement strategy to be used for the entity
|
||||
"""
|
||||
|
||||
regex: str
|
||||
replacement_strategy: DPIMethodConstant
|
||||
|
||||
|
||||
class DPIStandardEntity(BaseModel):
|
||||
"""
|
||||
type: Standard entity type to be masked
|
||||
replacement_strategy: Replacement strategy to be used for the entity
|
||||
"""
|
||||
|
||||
type_: SAPMaskingProfileEntity = Field(..., alias="type")
|
||||
replacement_strategy: Optional[
|
||||
Union[DPIMethodConstant, DPIMethodFabricatedData]
|
||||
] = None
|
||||
|
||||
|
||||
class MaskGroundingInput(BaseModel):
|
||||
"""
|
||||
Controls whether the input to the grounding module will be masked with the configuration
|
||||
supplied in the masking module
|
||||
"""
|
||||
|
||||
enabled: bool = False
|
||||
|
||||
|
||||
class MaskingProviderConfig(BaseModel):
|
||||
"""
|
||||
SAP Data Privacy Integration provider for data masking.
|
||||
|
||||
This class implements the SAP Data Privacy Integration service, which can anonymize or pseudonymize
|
||||
specified entity categories in the input data. It supports masking sensitive information like personal names,
|
||||
contact details, and identifiers.
|
||||
|
||||
Args:
|
||||
method: The method of masking to apply (anonymization or pseudonymization).
|
||||
|
||||
entities: A list of entity categories to be masked, such as names, locations, or emails.
|
||||
|
||||
allowlist: A list of strings that should not be masked.
|
||||
|
||||
mask_grounding_input: A flag indicating whether to mask input to the grounding module.
|
||||
"""
|
||||
|
||||
type_: Literal["sap_data_privacy_integration"] = Field(
|
||||
default="sap_data_privacy_integration", alias="type"
|
||||
)
|
||||
method: Literal["anonymization", "pseudonymization"]
|
||||
entities: list[Union[DPIStandardEntity, DPICustomEntity]]
|
||||
allowlist: Optional[list[str]] = None
|
||||
mask_grounding_input: Optional[MaskGroundingInput] = None
|
||||
|
||||
|
||||
class MaskingModuleConfig(BaseModel):
|
||||
"""
|
||||
Configuration for the data masking module.
|
||||
|
||||
Args:
|
||||
providers: list of masking service provider configurations
|
||||
masking_providers: list of masking provider configurations
|
||||
IMPORTANT: use exactly one of the parameters to set the list of masking provider configurations.
|
||||
DEPRECATED: parameter 'masking_providers' will be removed Sept 15, 2026. Use 'providers' instead.
|
||||
"""
|
||||
|
||||
providers: Optional[list[MaskingProviderConfig]] = Field(min_length=1, default=None)
|
||||
masking_providers: Optional[list[MaskingProviderConfig]] = Field(
|
||||
min_length=1, default=None
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def enforce_exactly_one_provider_list(self):
|
||||
has_providers = self.providers is not None
|
||||
has_masking_providers = self.masking_providers is not None
|
||||
|
||||
if not has_providers and not has_masking_providers:
|
||||
raise ValueError(
|
||||
"For SAP Masking Module Config you must provide 'providers'."
|
||||
)
|
||||
if has_providers and has_masking_providers:
|
||||
raise ValueError(
|
||||
"For SAP Masking Module Config you must set exactly one of: 'providers' or 'masking_providers', not both."
|
||||
)
|
||||
|
||||
if has_masking_providers:
|
||||
warnings.warn(
|
||||
"The 'masking_providers' parameter is deprecated and will be removed on Sept 15, 2026. "
|
||||
"Use 'providers' instead.",
|
||||
DeprecationWarning,
|
||||
stacklevel=5,
|
||||
)
|
||||
|
||||
return self
|
||||
|
||||
|
||||
class AzureThreshold(int, Enum):
|
||||
"""
|
||||
Enumerates the threshold levels for the Azure Content Safety service.
|
||||
|
||||
This enum defines the various threshold levels that can be used to filter
|
||||
content based on its safety score. Each threshold value represents a specific
|
||||
level of content moderation.
|
||||
|
||||
Values:
|
||||
ALLOW_SAFE: Allows only Safe content.
|
||||
|
||||
ALLOW_SAFE_LOW: Allows Safe and Low content.
|
||||
|
||||
ALLOW_SAFE_LOW_MEDIUM: Allows Safe, Low, and Medium content.
|
||||
|
||||
ALLOW_ALL: Allows all content (Safe, Low, Medium, and High).
|
||||
"""
|
||||
|
||||
ALLOW_SAFE = 0
|
||||
ALLOW_SAFE_LOW = 2
|
||||
ALLOW_SAFE_LOW_MEDIUM = 4
|
||||
ALLOW_ALL = 6
|
||||
|
||||
|
||||
class AzureContentFilter(BaseModel):
|
||||
"""
|
||||
Specific filter configuration for Azure Content Safety.
|
||||
|
||||
This class configures content filtering based on Azure's categories and
|
||||
severity levels. It allows setting thresholds for hate speech, sexual content,
|
||||
violence, and self-harm content.
|
||||
|
||||
Values:
|
||||
hate: Threshold for hate speech content.
|
||||
|
||||
sexual: Threshold for sexual content.
|
||||
|
||||
violence: Threshold for violent content.
|
||||
|
||||
self_harm: Threshold for self-harm content.
|
||||
"""
|
||||
|
||||
hate: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None
|
||||
sexual: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None
|
||||
violence: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None
|
||||
self_harm: Optional[Union[AzureThreshold, Literal[0, 2, 4, 6]]] = None
|
||||
|
||||
|
||||
class AzureContentSafetyInput(AzureContentFilter):
|
||||
"""
|
||||
Filter configuration for Azure Content Safety Input
|
||||
|
||||
Values:
|
||||
hate: Threshold for hate speech content.
|
||||
|
||||
sexual: Threshold for sexual content.
|
||||
|
||||
violence: Threshold for violent content.
|
||||
|
||||
self_harm: Threshold for self-harm content.
|
||||
|
||||
prompt_shield: A flag to use prompt shield
|
||||
"""
|
||||
|
||||
prompt_shield: Optional[bool] = False
|
||||
|
||||
|
||||
class AzureContentSafetyOutput(AzureContentFilter):
|
||||
"""
|
||||
Filter configuration for Azure Content Safety Output
|
||||
|
||||
Values:
|
||||
hate: Threshold for hate speech content.
|
||||
|
||||
sexual: Threshold for sexual content.
|
||||
|
||||
violence: Threshold for violent content.
|
||||
|
||||
self_harm: Threshold for self-harm content.
|
||||
|
||||
protected_material_code: Detect protected code content from known GitHub repositories.
|
||||
The scan includes software libraries, source code, algorithms,
|
||||
and other proprietary programming content.
|
||||
"""
|
||||
|
||||
protected_material_code: Optional[bool] = False
|
||||
|
||||
|
||||
class LlamaGuard38bFilter(BaseModel):
|
||||
"""
|
||||
Specific implementation of ContentFilter for Llama Guard 3. Llama Guard 3 is a
|
||||
Llama-3.1-8B pretrained model, fine-tuned for content safety classification.
|
||||
|
||||
Args:
|
||||
violent_crimes: Responses that enable, encourage, or endorse the commission of violent crimes.
|
||||
|
||||
non_violent_crimes: Responses that enable, encourage, or endorse the commission of non-violent crimes.
|
||||
|
||||
sex_crimes: Responses that enable, encourage, or endorse the commission of sex-related crimes.
|
||||
|
||||
child_exploitation: Responses that contain, describe, enable or endorse the sexual abuse of children.
|
||||
|
||||
defamation: Responses that are both verifiably false and likely to injure a living person's reputation.
|
||||
|
||||
specialized_advice: Responses that contain specialized financial, medical or legal advice.
|
||||
|
||||
privacy: Responses that contain sensitive or nonpublic personal information.
|
||||
|
||||
intellectual_property: Responses that may violate the intellectual property rights of any third party.
|
||||
|
||||
indiscriminate_weapons: Responses that enable, encourage, or endorse the creation of indiscriminate weapons.
|
||||
|
||||
hate: Responses that demean or dehumanize people on the basis of their sensitive, personal characteristics.
|
||||
|
||||
self_harm: Responses that enable, encourage, or endorse acts of intentional self-harm.
|
||||
|
||||
sexual_content: Responses that contain erotica.
|
||||
|
||||
elections: Responses that contain factually incorrect information about electoral systems and processes.
|
||||
|
||||
code_interpreter_abuse: Responses that seek to abuse code interpreters.
|
||||
"""
|
||||
|
||||
violent_crimes: bool = Field(default=False)
|
||||
non_violent_crimes: bool = Field(default=False)
|
||||
sex_crimes: bool = Field(default=False)
|
||||
child_exploitation: bool = Field(default=False)
|
||||
defamation: bool = Field(default=False)
|
||||
specialized_advice: bool = Field(default=False)
|
||||
privacy: bool = Field(default=False)
|
||||
intellectual_property: bool = Field(default=False)
|
||||
indiscriminate_weapons: bool = Field(default=False)
|
||||
hate: bool = Field(default=False)
|
||||
self_harm: bool = Field(default=False)
|
||||
sexual_content: bool = Field(default=False)
|
||||
elections: bool = Field(default=False)
|
||||
code_interpreter_abuse: bool = Field(default=False)
|
||||
|
||||
|
||||
class LlamaGuard38bFilterConfig(BaseModel):
|
||||
type_: Literal["llama_guard_3_8b"] = Field(default="llama_guard_3_8b", alias="type")
|
||||
config: LlamaGuard38bFilter
|
||||
|
||||
|
||||
class AzureContentSafetyInputFilterConfig(BaseModel):
|
||||
type_: Literal["azure_content_safety"] = Field(
|
||||
default="azure_content_safety", alias="type"
|
||||
)
|
||||
config: Optional[AzureContentSafetyInput] = None
|
||||
|
||||
|
||||
class AzureContentSafetyOutputFilterConfig(BaseModel):
|
||||
type_: Literal["azure_content_safety"] = Field(
|
||||
default="azure_content_safety", alias="type"
|
||||
)
|
||||
config: Optional[AzureContentSafetyOutput] = None
|
||||
|
||||
|
||||
class FilteringStreamOptions(BaseModel):
|
||||
"""
|
||||
overlap: Number of characters that should be additionally sent to content filtering services
|
||||
from previous chunks as additional context.
|
||||
"""
|
||||
|
||||
overlap: Optional[int] = Field(default=0, ge=0, le=10000)
|
||||
|
||||
|
||||
class InputFiltering(BaseModel):
|
||||
"""Module for managing and applying input content filters.
|
||||
|
||||
Args:
|
||||
filters: List of ContentFilter objects to be applied to input content.
|
||||
"""
|
||||
|
||||
filters: list[
|
||||
Union[AzureContentSafetyInputFilterConfig, LlamaGuard38bFilterConfig]
|
||||
] = Field(min_length=1)
|
||||
|
||||
|
||||
class OutputFiltering(BaseModel):
|
||||
"""Module for managing and applying output content filters.
|
||||
|
||||
Args:
|
||||
filters: List of ContentFilter objects to be applied to output content.
|
||||
|
||||
stream_options: Module-specific streaming options.
|
||||
"""
|
||||
|
||||
filters: list[
|
||||
Union[AzureContentSafetyOutputFilterConfig, LlamaGuard38bFilterConfig]
|
||||
] = Field(min_length=1)
|
||||
stream_options: Optional[FilteringStreamOptions] = None
|
||||
|
||||
|
||||
class FilteringModuleConfig(BaseModel):
|
||||
"""Module for managing and applying content filters.
|
||||
|
||||
Args:
|
||||
input: Module for filtering and validating input content before processing.
|
||||
|
||||
output: Module for filtering and validating output content after generation.
|
||||
"""
|
||||
|
||||
input: Optional[InputFiltering] = None
|
||||
output: Optional[OutputFiltering] = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def enforce_min_properties(self) -> "FilteringModuleConfig":
|
||||
"""
|
||||
Ensure at least one of input or output filtering is provided.
|
||||
"""
|
||||
if self.input is None and self.output is None:
|
||||
raise ValueError(
|
||||
"For using SAP Filtering Module you must provide at least one property: input or output filters."
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class SAPDocumentTranslationApplyToSelector(BaseModel):
|
||||
"""
|
||||
This selector allows you to define the scope of translation, such as specific placeholders or
|
||||
messages with specific roles.
|
||||
For example, {"category": "placeholders",
|
||||
"items": ["user_input"],
|
||||
"source_language": "de-DE"}
|
||||
targets the value of "user_input" in placeholder_values specified in the request payload;
|
||||
and considers the value to be in German.
|
||||
"""
|
||||
|
||||
category: Literal["placeholders", "template_roles"]
|
||||
items: list[str]
|
||||
source_language: str
|
||||
|
||||
|
||||
class InputTranslationConfig(BaseModel):
|
||||
"""
|
||||
Configuration for input translation.
|
||||
|
||||
Args:
|
||||
source_language: Language of the text to be translated. Example: de-DE
|
||||
target_language: Language to which the text should be translated. Example: en-US
|
||||
apply_to: List of selectors that define the scope of translation.
|
||||
"""
|
||||
|
||||
source_language: Optional[str] = None
|
||||
target_language: str
|
||||
apply_to: Optional[list[SAPDocumentTranslationApplyToSelector]] = None
|
||||
|
||||
|
||||
class OutputTranslationConfig(BaseModel):
|
||||
source_language: Optional[str] = None
|
||||
target_language: Union[str, SAPDocumentTranslationApplyToSelector]
|
||||
|
||||
|
||||
class SAPDocumentTranslationInput(BaseModel):
|
||||
"""
|
||||
Configuration for input translation
|
||||
|
||||
Args:
|
||||
type: The type of translation module (e.g., 'sap_document_translation').
|
||||
|
||||
translate_messages_history: If true, the messages history will be translated as well.
|
||||
|
||||
config: Configuration object for the translation module.
|
||||
"""
|
||||
|
||||
type_: Literal["sap_document_translation"] = Field(
|
||||
default="sap_document_translation", alias="type"
|
||||
)
|
||||
translate_messages_history: Optional[bool] = None
|
||||
config: InputTranslationConfig
|
||||
|
||||
|
||||
class SAPDocumentTranslationOutput(BaseModel):
|
||||
"""
|
||||
Configuration for output translation
|
||||
|
||||
Args:
|
||||
type: The type of translation module (e.g., 'sap_document_translation').
|
||||
|
||||
config: Configuration object for the translation module.
|
||||
"""
|
||||
|
||||
type_: Literal["sap_document_translation"] = Field(
|
||||
default="sap_document_translation", alias="type"
|
||||
)
|
||||
config: OutputTranslationConfig
|
||||
|
||||
|
||||
class TranslationModuleConfig(BaseModel):
|
||||
"""
|
||||
Configuration for translation module
|
||||
|
||||
Args:
|
||||
input: Configuration for input translation
|
||||
|
||||
output: Configuration for output translation
|
||||
"""
|
||||
|
||||
input: Optional[SAPDocumentTranslationInput] = None
|
||||
output: Optional[SAPDocumentTranslationOutput] = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def enforce_min_properties(self) -> "TranslationModuleConfig":
|
||||
if self.input is None and self.output is None:
|
||||
raise ValueError(
|
||||
"TranslationModuleConfig requires at least one of 'input' or 'output'."
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class ModuleConfig(BaseModel):
|
||||
prompt_templating: PromptTemplatingModuleConfig
|
||||
filtering: Optional[FilteringModuleConfig] = None
|
||||
masking: Optional[MaskingModuleConfig] = None
|
||||
grounding: Optional[GroundingModuleConfig] = None
|
||||
translation: Optional[TranslationModuleConfig] = None
|
||||
|
||||
|
||||
class GlobalStreamOptions(BaseModel):
|
||||
enabled: bool = False
|
||||
chunk_size: Optional[int] = Field(default=None, ge=1)
|
||||
delimiters: Optional[list[str]] = None
|
||||
|
||||
|
||||
class OrchestrationConfig(BaseModel):
|
||||
modules: Union[ModuleConfig, list[ModuleConfig]]
|
||||
stream: Optional[GlobalStreamOptions] = None
|
||||
|
||||
|
||||
class OrchestrationRequest(BaseModel):
|
||||
config: OrchestrationConfig
|
||||
placeholder_values: Optional[dict[str, str]] = None
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ from typing import (
|
|||
TYPE_CHECKING,
|
||||
Iterator,
|
||||
AsyncIterator,
|
||||
FrozenSet,
|
||||
)
|
||||
from functools import cached_property
|
||||
import litellm
|
||||
|
|
@ -31,12 +32,13 @@ else:
|
|||
|
||||
from ..credentials import get_token_creator
|
||||
from .models import (
|
||||
SAPMessage,
|
||||
SAPAssistantMessage,
|
||||
SAPToolChatMessage,
|
||||
ChatCompletionTool,
|
||||
ResponseFormatJSONSchema,
|
||||
OrchestrationRequest,
|
||||
ResponseFormat,
|
||||
ResponseFormatJSONSchema,
|
||||
SAPAssistantMessage,
|
||||
SAPMessage,
|
||||
SAPToolChatMessage,
|
||||
SAPUserMessage,
|
||||
)
|
||||
from .handler import (
|
||||
|
|
@ -45,9 +47,65 @@ from .handler import (
|
|||
SAPStreamIterator,
|
||||
)
|
||||
|
||||
# Keys routed outside SAP orchestration `model.params` (prompt, stream, fallbacks, etc.)
|
||||
_SAP_MODEL_PARAMS_EXCLUDED_KEYS: FrozenSet[str] = frozenset(
|
||||
{
|
||||
"tools",
|
||||
"tool_choice",
|
||||
"stream_options",
|
||||
"fallback_sap_modules",
|
||||
"placeholder_values",
|
||||
"model_version",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def validate_dict(data: dict, model) -> dict:
|
||||
return model(**data).model_dump(by_alias=True)
|
||||
return model(**data).model_dump(by_alias=True, exclude_unset=True)
|
||||
|
||||
|
||||
def _messages_to_sap_template(messages: List[Dict[str, str]]) -> list: # type: ignore[type-arg]
|
||||
template = []
|
||||
for message in messages:
|
||||
if message["role"] == "user":
|
||||
template.append(validate_dict(message, SAPUserMessage))
|
||||
elif message["role"] == "assistant":
|
||||
template.append(validate_dict(message, SAPAssistantMessage))
|
||||
elif message["role"] == "tool":
|
||||
template.append(validate_dict(message, SAPToolChatMessage))
|
||||
else:
|
||||
template.append(validate_dict(message, SAPMessage))
|
||||
return template
|
||||
|
||||
|
||||
def _tools_response_format_and_stream(
|
||||
optional_params: dict, model_params: dict
|
||||
) -> Tuple[dict, dict, dict]:
|
||||
tools_ = optional_params.pop("tools", [])
|
||||
tools_ = [validate_dict(tool, ChatCompletionTool) for tool in tools_]
|
||||
tools: dict = {"tools": tools_} if tools_ else {}
|
||||
|
||||
response_format = model_params.pop("response_format", {})
|
||||
resp_type = response_format.get("type", None)
|
||||
if resp_type:
|
||||
if resp_type == "json_schema":
|
||||
response_format = validate_dict(
|
||||
response_format, ResponseFormatJSONSchema
|
||||
)
|
||||
else:
|
||||
response_format = validate_dict(response_format, ResponseFormat)
|
||||
response_format = {"response_format": response_format}
|
||||
|
||||
model_params.pop("stream", False)
|
||||
stream_config: dict = {}
|
||||
if "stream_options" in optional_params:
|
||||
stream_options = optional_params.pop("stream_options", {})
|
||||
if "chunk_size" in stream_options:
|
||||
stream_config["chunk_size"] = stream_options.get("chunk_size")
|
||||
if "delimiters" in stream_options:
|
||||
stream_config["delimiters"] = stream_options.get("delimiters")
|
||||
|
||||
return tools, response_format, stream_config
|
||||
|
||||
|
||||
class GenAIHubOrchestrationConfig(OpenAIGPTConfig):
|
||||
|
|
@ -208,48 +266,25 @@ class GenAIHubOrchestrationConfig(OpenAIGPTConfig):
|
|||
api_base_ = f"{self.deployment_url}/v2/completion"
|
||||
return api_base_
|
||||
|
||||
def transform_request(
|
||||
def _build_prompt_module(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[Dict[str, str]], # type: ignore
|
||||
optional_params: dict,
|
||||
litellm_params: dict,
|
||||
headers: dict,
|
||||
model_name: str,
|
||||
template_messages: List[Dict[str, str]],
|
||||
params: dict,
|
||||
) -> dict:
|
||||
# Filter out parameters that are not valid model params for SAP Orchestration API
|
||||
# - tools, model_version, deployment_url: handled separately
|
||||
excluded_params = {"tools", "model_version", "deployment_url"}
|
||||
|
||||
# Filter strict for GPT models only - SAP AI Core doesn't accept it as a model param
|
||||
# LangChain agents pass strict=true at top level, which fails for GPT models
|
||||
# Anthropic models accept strict, so preserve it for them
|
||||
if model.startswith("gpt"):
|
||||
excluded_params.add("strict")
|
||||
if model_name.startswith("gpt") and "strict" in params:
|
||||
params.pop("strict")
|
||||
|
||||
model_params = {
|
||||
k: v for k, v in optional_params.items() if k not in excluded_params
|
||||
}
|
||||
model_version = params.pop("model_version", "latest")
|
||||
|
||||
model_version = optional_params.pop("model_version", "latest")
|
||||
template = []
|
||||
for message in messages:
|
||||
if message["role"] == "user":
|
||||
template.append(validate_dict(message, SAPUserMessage))
|
||||
elif message["role"] == "assistant":
|
||||
template.append(validate_dict(message, SAPAssistantMessage))
|
||||
elif message["role"] == "tool":
|
||||
template.append(validate_dict(message, SAPToolChatMessage))
|
||||
else:
|
||||
template.append(validate_dict(message, SAPMessage))
|
||||
|
||||
tools_ = optional_params.pop("tools", [])
|
||||
tools_ = params.pop("tools", [])
|
||||
tools_ = [validate_dict(tool, ChatCompletionTool) for tool in tools_]
|
||||
if tools_ != []:
|
||||
tools = {"tools": tools_}
|
||||
else:
|
||||
tools = {}
|
||||
tools = {"tools": tools_} if tools_ else {}
|
||||
|
||||
response_format = model_params.pop("response_format", {})
|
||||
response_format = params.pop("response_format", {})
|
||||
resp_type = response_format.get("type", None)
|
||||
if resp_type:
|
||||
if resp_type == "json_schema":
|
||||
|
|
@ -259,33 +294,104 @@ class GenAIHubOrchestrationConfig(OpenAIGPTConfig):
|
|||
else:
|
||||
response_format = validate_dict(response_format, ResponseFormat)
|
||||
response_format = {"response_format": response_format}
|
||||
model_params.pop("stream", False)
|
||||
stream_config = {}
|
||||
if "stream_options" in model_params:
|
||||
# stream_config["enabled"] = True
|
||||
stream_options = model_params.pop("stream_options", {})
|
||||
stream_config["chunk_size"] = stream_options.get("chunk_size", 100)
|
||||
if "delimiters" in stream_options:
|
||||
stream_config["delimiters"] = stream_options.get("delimiters")
|
||||
# else:
|
||||
# stream_config["enabled"] = False
|
||||
config = {
|
||||
"config": {
|
||||
"modules": {
|
||||
"prompt_templating": {
|
||||
"prompt": {"template": template, **tools, **response_format},
|
||||
"model": {
|
||||
"name": model,
|
||||
"params": model_params,
|
||||
"version": model_version,
|
||||
},
|
||||
},
|
||||
else:
|
||||
response_format = {}
|
||||
|
||||
placeholder_defaults = params.pop("placeholder_defaults", {})
|
||||
placeholder_defaults = (
|
||||
{"defaults": placeholder_defaults} if placeholder_defaults else {}
|
||||
)
|
||||
|
||||
optional_modules = {}
|
||||
optional_modules_lst = ["grounding", "masking", "filtering", "translation"]
|
||||
for module in optional_modules_lst:
|
||||
if params.get(module, None) is not None:
|
||||
optional_modules[module] = params.pop(module)
|
||||
|
||||
return {
|
||||
"prompt_templating": {
|
||||
"prompt": {
|
||||
"template": template_messages,
|
||||
**placeholder_defaults,
|
||||
**tools,
|
||||
**response_format,
|
||||
},
|
||||
"stream": stream_config,
|
||||
}
|
||||
"model": {
|
||||
"name": model_name,
|
||||
"params": params,
|
||||
"version": model_version,
|
||||
},
|
||||
},
|
||||
**optional_modules,
|
||||
}
|
||||
|
||||
return config
|
||||
def transform_request(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[Dict[str, str]], # type: ignore
|
||||
optional_params: dict,
|
||||
litellm_params: dict,
|
||||
headers: dict,
|
||||
) -> dict:
|
||||
optional_params = dict(optional_params)
|
||||
optional_params.pop("deployment_url", None)
|
||||
|
||||
template = _messages_to_sap_template(messages)
|
||||
|
||||
placeholder_values = optional_params.pop("placeholder_values", None)
|
||||
fallback_modules = optional_params.pop("fallback_sap_modules", [])
|
||||
|
||||
optional_params.pop("stream", None)
|
||||
stream_config: dict = {}
|
||||
if "stream_options" in optional_params:
|
||||
stream_options = optional_params.pop("stream_options", {})
|
||||
if "chunk_size" in stream_options:
|
||||
stream_config["chunk_size"] = stream_options["chunk_size"]
|
||||
if "delimiters" in stream_options:
|
||||
stream_config["delimiters"] = stream_options["delimiters"]
|
||||
|
||||
optional_params.pop("tool_choice", None)
|
||||
|
||||
modules = [
|
||||
self._build_prompt_module(
|
||||
model_name=model,
|
||||
template_messages=template,
|
||||
params=dict(optional_params),
|
||||
)
|
||||
]
|
||||
|
||||
for modules_dict in fallback_modules:
|
||||
modules_dict = dict(modules_dict)
|
||||
fallback_model = modules_dict.pop("model", None)
|
||||
if fallback_model is None:
|
||||
raise ValueError(
|
||||
"Each entry in `fallback_sap_modules` must include a 'model' key."
|
||||
)
|
||||
if fallback_model.startswith("sap/"):
|
||||
fallback_model = fallback_model[4:]
|
||||
fallback_template = modules_dict.pop("messages", [])
|
||||
|
||||
modules.append(
|
||||
self._build_prompt_module(
|
||||
model_name=fallback_model,
|
||||
template_messages=fallback_template,
|
||||
params=modules_dict,
|
||||
)
|
||||
)
|
||||
|
||||
config_payload: Dict[str, Any] = {
|
||||
"modules": modules if len(modules) > 1 else modules[0],
|
||||
}
|
||||
if stream_config:
|
||||
config_payload["stream"] = stream_config
|
||||
|
||||
request_body: Dict[str, Any] = {"config": config_payload}
|
||||
if placeholder_values is not None:
|
||||
request_body["placeholder_values"] = placeholder_values
|
||||
|
||||
body = validate_dict(request_body, OrchestrationRequest)
|
||||
|
||||
return body
|
||||
|
||||
def transform_response(
|
||||
self,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
from __future__ import annotations
|
||||
from typing import Any, Callable, Dict, Final, List, Optional, Sequence, Tuple
|
||||
from typing import Any, Callable, Dict, Final, List, Optional, Sequence, Tuple, Union
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from threading import Lock
|
||||
from pathlib import Path
|
||||
|
|
@ -7,9 +7,11 @@ from dataclasses import dataclass
|
|||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import httpx
|
||||
|
||||
from litellm import sap_service_key
|
||||
from litellm.llms.custom_httpx.http_handler import _get_httpx_client
|
||||
from litellm.llms.custom_httpx.http_handler import _get_httpx_client, HTTPHandler
|
||||
from litellm._logging import verbose_logger
|
||||
import litellm
|
||||
|
||||
AUTH_ENDPOINT_SUFFIX = "/oauth/token"
|
||||
|
||||
|
|
@ -28,11 +30,25 @@ def _get_home() -> str:
|
|||
return os.getenv(HOME_PATH_ENV_VAR, DEFAULT_HOME_PATH)
|
||||
|
||||
|
||||
def _get_nested(d: Dict[str, Any], path: Sequence[str]) -> Any:
|
||||
def _get_nested(d: Union[Dict[str, Any], str], path: Sequence[str]) -> Any:
|
||||
cur: Any = d
|
||||
if isinstance(cur, str):
|
||||
# This shouldn't happen if service keys are pre-parsed correctly
|
||||
try:
|
||||
cur = json.loads(cur)
|
||||
except json.JSONDecodeError:
|
||||
verbose_logger.warning(
|
||||
"SAP service key or VCAP service is a string but not valid JSON."
|
||||
)
|
||||
return None
|
||||
for k in path:
|
||||
if not isinstance(cur, dict) or k not in cur:
|
||||
raise KeyError(".".join(path))
|
||||
if not isinstance(cur, dict):
|
||||
verbose_logger.warning(
|
||||
f"SAP service key or VCAP service traversal hit non-dict type '{type(cur).__name__}' at key '{k}'."
|
||||
)
|
||||
return None
|
||||
if k not in cur:
|
||||
return None
|
||||
cur = cur[k]
|
||||
return cur
|
||||
|
||||
|
|
@ -47,6 +63,13 @@ def _load_json_env(var_name: str) -> Optional[Dict[str, Any]]:
|
|||
return None
|
||||
|
||||
|
||||
def _str_or_none(value) -> Optional[str]:
|
||||
try:
|
||||
return str(value) if value is not None else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _load_vcap() -> Dict[str, Any]:
|
||||
return _load_json_env(VCAP_SERVICES_ENV_VAR) or {}
|
||||
|
||||
|
|
@ -59,6 +82,12 @@ def _get_vcap_service(label: str) -> Optional[Dict[str, Any]]:
|
|||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Source:
|
||||
name: str
|
||||
get: Callable[[CredentialsValue], Optional[str]]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CredentialsValue:
|
||||
name: str
|
||||
|
|
@ -82,7 +111,6 @@ CREDENTIAL_VALUES: Final[List[CredentialsValue]] = [
|
|||
transform_fn=lambda url: url.rstrip("/")
|
||||
+ ("" if url.endswith("/v2") else "/v2"),
|
||||
),
|
||||
CredentialsValue("resource_group", default="default"),
|
||||
CredentialsValue(
|
||||
"cert_url",
|
||||
("certurl",),
|
||||
|
|
@ -145,81 +173,239 @@ def _env_name(name: str) -> str:
|
|||
return f"AICORE_{name.upper()}"
|
||||
|
||||
|
||||
def _resolve_value(
|
||||
cred: CredentialsValue,
|
||||
*,
|
||||
kwargs: Dict[str, Any],
|
||||
env: Dict[str, str],
|
||||
config: Dict[str, Any],
|
||||
service_like: Optional[Dict[str, Any]],
|
||||
) -> Optional[str]:
|
||||
# 1) explicit kwargs
|
||||
if cred.name in kwargs and kwargs[cred.name] is not None:
|
||||
return kwargs[cred.name]
|
||||
def extract_credentials(source: Source) -> Dict[str, str]:
|
||||
"""Extract all credentials from a source."""
|
||||
credentials = {}
|
||||
for cv in CREDENTIAL_VALUES:
|
||||
value = source.get(cv)
|
||||
if value is not None:
|
||||
credentials[cv.name] = cv.transform_fn(value) if cv.transform_fn else value
|
||||
return credentials
|
||||
|
||||
# 2) environment variables (primary name)
|
||||
env_key = _env_name(cred.name)
|
||||
if env_key in env and env[env_key] is not None:
|
||||
return env[env_key]
|
||||
|
||||
# 3) config file (accept both prefixed and plain keys)
|
||||
for key in (env_key, cred.name):
|
||||
if key in config and config[key] is not None:
|
||||
return config[key]
|
||||
def resolve_credentials(sources: List[Source]) -> Dict[str, str]:
|
||||
"""Extract credentials from the first source that has any defined."""
|
||||
for source in sources:
|
||||
credentials = extract_credentials(source)
|
||||
if credentials:
|
||||
verbose_logger.debug(f"Resolved SAP credentials from source {source.name}")
|
||||
return credentials
|
||||
raise ValueError("No credentials found in any source")
|
||||
|
||||
# 4) service-like source (AICORE_SERVICE_KEY first, else VCAP)
|
||||
if service_like and cred.vcap_key:
|
||||
|
||||
def resolve_resource_group(sources: List[Source]) -> Optional[str]:
|
||||
"""Find resource_group from the first source that defines it."""
|
||||
rg_cred = CredentialsValue("resource_group", default="default")
|
||||
for source in sources:
|
||||
value = source.get(rg_cred)
|
||||
if value is not None:
|
||||
verbose_logger.debug(
|
||||
f"Resolved GEN AI Hub resource_group from source {source.name}"
|
||||
)
|
||||
return value
|
||||
return rg_cred.default
|
||||
|
||||
|
||||
def _parse_service_key_once(
|
||||
service_key: Optional[Union[str, dict]]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Pre-parse service_key if it's a string to avoid repeated JSON parsing.
|
||||
|
||||
Returns None if parsing fails (other credential sources may still work).
|
||||
"""
|
||||
if service_key is None:
|
||||
return None
|
||||
if isinstance(service_key, dict):
|
||||
return service_key
|
||||
if isinstance(service_key, str):
|
||||
try:
|
||||
val = _get_nested(service_like, ("credentials",) + cred.vcap_key)
|
||||
if val is not None:
|
||||
return val
|
||||
except KeyError:
|
||||
pass
|
||||
return json.loads(service_key)
|
||||
except json.JSONDecodeError:
|
||||
verbose_logger.warning(
|
||||
"SAP service key is a string but not valid JSON. Skipping this source."
|
||||
)
|
||||
return None
|
||||
verbose_logger.warning(
|
||||
f"SAP service key has unexpected type '{type(service_key).__name__}'. Expected str or dict. Ignoring."
|
||||
)
|
||||
return None
|
||||
|
||||
# 5) default
|
||||
return cred.default
|
||||
|
||||
def _resolve_credential_from_service_key(
|
||||
service_key: Optional[Union[str, dict]], cv: CredentialsValue
|
||||
) -> Optional[str]:
|
||||
if service_key is None:
|
||||
return None
|
||||
val = _str_or_none(
|
||||
_get_nested(
|
||||
service_key, (("credentials",) + cv.vcap_key) if cv.vcap_key else (cv.name,)
|
||||
)
|
||||
)
|
||||
if val is None:
|
||||
return _str_or_none(
|
||||
_get_nested(service_key, cv.vcap_key if cv.vcap_key else (cv.name,))
|
||||
)
|
||||
return val
|
||||
|
||||
|
||||
def fetch_credentials(
|
||||
service_key: Optional[str] = None, profile: Optional[str] = None, **kwargs
|
||||
service_key: Optional[Union[str, dict]] = None,
|
||||
profile: Optional[str] = None,
|
||||
**kwargs,
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Resolution order per key:
|
||||
Resolution order (first-source-wins):
|
||||
|
||||
Sources are checked in this order:
|
||||
kwargs
|
||||
> service key
|
||||
> env (AICORE_<NAME>)
|
||||
> config (AICORE_<NAME> or plain <name>)
|
||||
> service-like source from JSON in $AICORE_SERVICE_KEY (same structure as a VCAP service object)
|
||||
falling back to service entry in $VCAP_SERVICES with label 'aicore'
|
||||
> vcap service key
|
||||
> default
|
||||
|
||||
Important:
|
||||
- Credentials are extracted from the FIRST source that provides any credential value.
|
||||
- Values are NOT merged per key across sources. Except resource_group, which is merged.
|
||||
|
||||
Warning:
|
||||
- This function does NOT validate the returned credentials just parsed it from the sources.
|
||||
- Callers MUST explicitly call validate_credentials() on the returned dict
|
||||
"""
|
||||
config = init_conf(profile)
|
||||
env = os.environ # snapshot for testability
|
||||
service_like = None
|
||||
|
||||
if not config:
|
||||
# Prefer AICORE_SERVICE_KEY if present; otherwise fall back to the VCAP service.
|
||||
service_like = (
|
||||
service_key
|
||||
or sap_service_key
|
||||
or _load_json_env(SERVICE_KEY_ENV_VAR)
|
||||
or _get_vcap_service(VCAP_AICORE_SERVICE_NAME)
|
||||
service_key = _parse_service_key_once(
|
||||
service_key or litellm.sap_service_key or os.environ.get(SERVICE_KEY_ENV_VAR)
|
||||
)
|
||||
vcap_service = _get_vcap_service(VCAP_AICORE_SERVICE_NAME)
|
||||
|
||||
sources = [
|
||||
Source("kwargs", lambda cv: _str_or_none(kwargs.get(cv.name))),
|
||||
Source(
|
||||
"service key",
|
||||
lambda cv: _resolve_credential_from_service_key(service_key, cv),
|
||||
),
|
||||
Source(
|
||||
"environment variables",
|
||||
lambda cv: _str_or_none(os.environ.get(f"AICORE_{cv.name.upper()}")),
|
||||
),
|
||||
Source(
|
||||
"config file",
|
||||
lambda cv: _str_or_none(
|
||||
config.get(f"AICORE_{cv.name.upper()}")
|
||||
if config.get(f"AICORE_{cv.name.upper()}") is not None
|
||||
else config.get(cv.name)
|
||||
),
|
||||
),
|
||||
Source(
|
||||
"VCAP service",
|
||||
lambda cv: (
|
||||
_str_or_none(
|
||||
_get_nested(
|
||||
vcap_service,
|
||||
(("credentials",) + cv.vcap_key) if cv.vcap_key else (cv.name,),
|
||||
)
|
||||
)
|
||||
if vcap_service
|
||||
else None
|
||||
),
|
||||
), # type: ignore[arg-type]
|
||||
]
|
||||
|
||||
credentials = resolve_credentials(sources)
|
||||
|
||||
resource_group = resolve_resource_group(sources)
|
||||
if resource_group is not None:
|
||||
credentials["resource_group"] = resource_group
|
||||
|
||||
if "cert_url" in credentials:
|
||||
credentials["auth_url"] = credentials.pop("cert_url")
|
||||
return credentials
|
||||
|
||||
|
||||
def validate_credentials(
|
||||
auth_url: Optional[str] = None,
|
||||
base_url: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None,
|
||||
cert_str: Optional[str] = None,
|
||||
key_str: Optional[str] = None,
|
||||
cert_file_path: Optional[str] = None,
|
||||
key_file_path: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Validate SAP AI Core credentials for completeness and consistency.
|
||||
|
||||
Args:
|
||||
auth_url: OAuth2 token endpoint URL (required)
|
||||
base_url: SAP AI Core API base URL (required)
|
||||
client_id: OAuth2 client ID (required)
|
||||
client_secret: OAuth2 client secret (for secret-based auth)
|
||||
cert_str: PEM-encoded certificate string (for cert-based auth)
|
||||
key_str: PEM-encoded private key string (for cert-based auth)
|
||||
cert_file_path: Path to certificate file (for file-based cert auth)
|
||||
key_file_path: Path to private key file (for file-based cert auth)
|
||||
|
||||
Raises:
|
||||
ValueError: If required fields are missing or authentication mode is ambiguous.
|
||||
|
||||
Note:
|
||||
- This function does NOT validate resource_group (resolved separately).
|
||||
- Exactly one authentication method must be provided:
|
||||
* client_secret, OR
|
||||
* (cert_str AND key_str), OR
|
||||
* (cert_file_path AND key_file_path)
|
||||
"""
|
||||
if not auth_url or not client_id or not base_url:
|
||||
raise ValueError(
|
||||
"SAP AI Core credentials not found. "
|
||||
"Please provide credentials by setting appropriate environment variables "
|
||||
"(e.g. AICORE_CLIENT_ID, AICORE_CLIENT_SECRET, etc.)"
|
||||
)
|
||||
|
||||
out: Dict[str, str] = {}
|
||||
for cred in CREDENTIAL_VALUES:
|
||||
value = _resolve_value(cred, kwargs=kwargs, env=env, config=config, service_like=service_like) # type: ignore
|
||||
if value is None:
|
||||
continue
|
||||
if cred.transform_fn:
|
||||
value = cred.transform_fn(value)
|
||||
out[cred.name] = value
|
||||
if "cert_url" in out.keys():
|
||||
out["auth_url"] = out.pop("cert_url")
|
||||
return out
|
||||
modes = [
|
||||
bool(client_secret),
|
||||
bool(cert_str) and bool(key_str),
|
||||
bool(cert_file_path) and bool(key_file_path),
|
||||
]
|
||||
if sum(bool(m) for m in modes) != 1:
|
||||
raise ValueError(
|
||||
"SAP AI Core credentials are incomplete. "
|
||||
"Invalid credentials: provide exactly one of client_secret, "
|
||||
"(cert_str & key_str), or (cert_file_path & key_file_path)."
|
||||
)
|
||||
|
||||
|
||||
def _request_token(
|
||||
client_id: str, auth_url: str, timeout: float, cert_pair=None, client_secret=None
|
||||
) -> tuple[str, datetime]:
|
||||
data = {"grant_type": "client_credentials", "client_id": client_id}
|
||||
if client_secret:
|
||||
data["client_secret"] = client_secret
|
||||
|
||||
resp: Optional[httpx.Response] = None
|
||||
try:
|
||||
if cert_pair:
|
||||
with httpx.Client(cert=cert_pair) as raw_client:
|
||||
handler = HTTPHandler(client=raw_client)
|
||||
resp = handler.post(auth_url, data=data, timeout=timeout) # type: ignore[arg-type]
|
||||
payload = resp.json()
|
||||
else:
|
||||
handler = _get_httpx_client()
|
||||
resp = handler.post(auth_url, data=data, timeout=timeout) # type: ignore[arg-type]
|
||||
payload = resp.json()
|
||||
access_token = payload["access_token"]
|
||||
expires_in = int(payload.get("expires_in", 3600))
|
||||
expiry_date = datetime.now(timezone.utc) + timedelta(seconds=expires_in)
|
||||
return f"Bearer {access_token}", expiry_date
|
||||
except Exception as e:
|
||||
msg = resp.text if resp is not None else getattr(e, "text", str(e))
|
||||
raise RuntimeError(f"Token request failed: {msg}") from e
|
||||
|
||||
|
||||
def get_token_creator(
|
||||
service_key: Optional[str] = None,
|
||||
service_key: Optional[Union[str, dict]] = None,
|
||||
profile: Optional[str] = None,
|
||||
*,
|
||||
timeout: float = 30.0,
|
||||
|
|
@ -237,7 +423,7 @@ def get_token_creator(
|
|||
|
||||
Args:
|
||||
profile: Optional AICore profile name
|
||||
timeout: HTTP request timeout in seconds (default 30s)
|
||||
timeout: Timeout for HTTP requests
|
||||
expiry_buffer_minutes: Refresh the token this many minutes before expiry
|
||||
overrides: Any explicit credential overrides (client_id, client_secret, etc.)
|
||||
|
||||
|
|
@ -251,6 +437,7 @@ def get_token_creator(
|
|||
)
|
||||
|
||||
auth_url = credentials.get("auth_url")
|
||||
base_url = credentials.get("base_url")
|
||||
client_id = credentials.get("client_id")
|
||||
client_secret = credentials.get("client_secret")
|
||||
cert_str = credentials.get("cert_str")
|
||||
|
|
@ -259,49 +446,30 @@ def get_token_creator(
|
|||
key_file_path = credentials.get("key_file_path")
|
||||
|
||||
# Sanity check
|
||||
if not auth_url or not client_id:
|
||||
raise ValueError(
|
||||
"fetch_credentials did not return valid 'auth_url' or 'client_id'"
|
||||
)
|
||||
|
||||
modes = [
|
||||
client_secret is not None,
|
||||
(cert_str is not None and key_str is not None),
|
||||
(cert_file_path is not None and key_file_path is not None),
|
||||
]
|
||||
if sum(bool(m) for m in modes) != 1:
|
||||
raise ValueError(
|
||||
"Invalid credentials: provide exactly one of client_secret, "
|
||||
"(cert_str & key_str), or (cert_file_path & key_file_path)."
|
||||
)
|
||||
validate_credentials(
|
||||
auth_url,
|
||||
base_url,
|
||||
client_id,
|
||||
client_secret,
|
||||
cert_str,
|
||||
key_str,
|
||||
cert_file_path,
|
||||
key_file_path,
|
||||
)
|
||||
|
||||
lock = Lock()
|
||||
token: Optional[str] = None
|
||||
token_expiry: Optional[datetime] = None
|
||||
|
||||
def _request_token(cert_pair=None) -> tuple[str, datetime]:
|
||||
data = {"grant_type": "client_credentials", "client_id": client_id}
|
||||
if client_secret:
|
||||
data["client_secret"] = client_secret
|
||||
|
||||
client = _get_httpx_client()
|
||||
# with httpx.Client(cert=cert_pair, timeout=timeout) as client:
|
||||
resp = client.post(auth_url, data=data)
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
payload = resp.json()
|
||||
access_token = payload["access_token"]
|
||||
expires_in = int(payload.get("expires_in", 3600))
|
||||
expiry_date = datetime.now(timezone.utc) + timedelta(seconds=expires_in)
|
||||
return f"Bearer {access_token}", expiry_date
|
||||
except Exception as e:
|
||||
msg = getattr(resp, "text", str(e))
|
||||
raise RuntimeError(f"Token request failed: {msg}") from e
|
||||
|
||||
def _fetch_token() -> tuple[str, datetime]:
|
||||
# Case 1: secret-based auth
|
||||
if client_secret:
|
||||
return _request_token()
|
||||
return _request_token(
|
||||
auth_url=auth_url, # type: ignore[arg-type]
|
||||
client_id=client_id, # type: ignore[arg-type]
|
||||
timeout=timeout,
|
||||
client_secret=client_secret,
|
||||
)
|
||||
# Case 2: cert/key strings
|
||||
if cert_str and key_str:
|
||||
cert_str_fixed = cert_str.replace("\\n", "\n")
|
||||
|
|
@ -313,9 +481,24 @@ def get_token_creator(
|
|||
f.write(cert_str_fixed)
|
||||
with open(key_path, "w") as f:
|
||||
f.write(key_str_fixed)
|
||||
return _request_token(cert_pair=(cert_path, key_path))
|
||||
return _request_token(
|
||||
auth_url=auth_url, # type: ignore[arg-type]
|
||||
client_id=client_id, # type: ignore[arg-type]
|
||||
timeout=timeout,
|
||||
cert_pair=(cert_path, key_path),
|
||||
)
|
||||
# Case 3: file-based cert/key
|
||||
return _request_token(cert_pair=(cert_file_path, key_file_path))
|
||||
if cert_file_path is not None and key_file_path is not None:
|
||||
return _request_token(
|
||||
auth_url=auth_url, # type: ignore[arg-type]
|
||||
client_id=client_id, # type: ignore[arg-type]
|
||||
timeout=timeout,
|
||||
cert_pair=(cert_file_path, key_file_path),
|
||||
)
|
||||
# Defensive guard: should never reach here due to validate_credentials()
|
||||
raise ValueError(
|
||||
"Invalid authentication configuration: no valid credentials found. "
|
||||
)
|
||||
|
||||
def get_token() -> str:
|
||||
nonlocal token, token_expiry
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ Translates from OpenAI's `/v1/embeddings` to IBM's `/text/embeddings` route.
|
|||
from typing import Optional, List, Dict, Literal, Union
|
||||
from pydantic import BaseModel, Field
|
||||
from functools import cached_property
|
||||
from litellm.llms.sap.chat.models import MaskingModuleConfig
|
||||
|
||||
import httpx
|
||||
|
||||
|
|
@ -47,25 +48,36 @@ class EmbeddingsResponse(BaseModel):
|
|||
class EmbeddingModel(BaseModel):
|
||||
name: str
|
||||
version: str = "latest"
|
||||
params: dict = Field(default_factory=dict, validation_alias="parameters")
|
||||
params: dict = Field(default_factory=dict)
|
||||
timeout: Optional[int] = Field(default=None, ge=1, le=600)
|
||||
max_retries: Optional[int] = Field(default=None, ge=0, le=5)
|
||||
|
||||
|
||||
class EmbeddingsModelConfig(BaseModel):
|
||||
model: EmbeddingModel
|
||||
|
||||
|
||||
class EmbeddingsModules(BaseModel):
|
||||
embeddings: EmbeddingModel
|
||||
embeddings: EmbeddingsModelConfig
|
||||
masking: Optional[MaskingModuleConfig] = None
|
||||
|
||||
|
||||
class EmbeddingInput(BaseModel):
|
||||
text: Union[str, List[str]]
|
||||
type: Literal["text", "document", "query"] = "text"
|
||||
type: Optional[Literal["text", "document", "query"]] = None
|
||||
|
||||
|
||||
class EmbeddingConfig(BaseModel):
|
||||
modules: EmbeddingsModules
|
||||
|
||||
|
||||
class EmbeddingRequest(BaseModel):
|
||||
config: EmbeddingsModules
|
||||
config: EmbeddingConfig
|
||||
input: EmbeddingInput
|
||||
|
||||
|
||||
def validate_dict(data: dict, model) -> dict:
|
||||
return model(**data).model_dump()
|
||||
return model(**data).model_dump(exclude_unset=True, by_alias=True)
|
||||
|
||||
|
||||
class GenAIHubEmbeddingConfig(BaseEmbeddingConfig):
|
||||
|
|
@ -152,15 +164,23 @@ class GenAIHubEmbeddingConfig(BaseEmbeddingConfig):
|
|||
model_dict["name"] = model
|
||||
model_dict["version"] = optional_params.get("version", "latest")
|
||||
model_dict["params"] = optional_params.get("parameters", {})
|
||||
timeout = optional_params.get("timeout", None)
|
||||
if timeout is not None:
|
||||
model_dict["timeout"] = timeout
|
||||
max_retries = optional_params.get("max_retries", None)
|
||||
if max_retries is not None:
|
||||
model_dict["max_retries"] = max_retries
|
||||
input_dict = {"text": input}
|
||||
input_type = optional_params.get("type")
|
||||
if input_type is not None:
|
||||
input_dict["type"] = input_type
|
||||
masking = optional_params.get("masking")
|
||||
masking = {"masking": masking} if masking is not None else {}
|
||||
body = {
|
||||
"config": {
|
||||
"modules": {
|
||||
"embeddings": {"model": validate_dict(model_dict, EmbeddingModel)}
|
||||
}
|
||||
},
|
||||
"input": validate_dict(input_dict, EmbeddingInput),
|
||||
"config": {"modules": {"embeddings": {"model": model_dict}, **masking}},
|
||||
"input": input_dict,
|
||||
}
|
||||
body = validate_dict(body, EmbeddingRequest)
|
||||
return body
|
||||
|
||||
def transform_embedding_response(
|
||||
|
|
|
|||
|
|
@ -763,6 +763,16 @@ def _transform_request_body( # noqa: PLR0915
|
|||
data["generationConfig"] = generation_config
|
||||
if cached_content is not None:
|
||||
data["cachedContent"] = cached_content
|
||||
|
||||
if service_tier := optional_params.pop("service_tier", None):
|
||||
if isinstance(service_tier, str):
|
||||
if service_tier.lower() == "default":
|
||||
data["serviceTier"] = "standard"
|
||||
else:
|
||||
data["serviceTier"] = service_tier.lower()
|
||||
else:
|
||||
data["serviceTier"] = service_tier
|
||||
|
||||
# Only add labels for Vertex AI endpoints (not Google GenAI/AI Studio) and only if non-empty
|
||||
if labels and custom_llm_provider != LlmProviders.GEMINI:
|
||||
data["labels"] = labels
|
||||
|
|
|
|||
|
|
@ -318,6 +318,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
|||
"parallel_tool_calls",
|
||||
"web_search_options",
|
||||
"include_server_side_tool_invocations",
|
||||
"service_tier",
|
||||
]
|
||||
|
||||
# Add penalty parameters only for non-preview models
|
||||
|
|
@ -362,6 +363,17 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
|||
"""
|
||||
return Tools(googleSearch={})
|
||||
|
||||
def _map_service_tier_param(self, value: str, optional_params: dict) -> None:
|
||||
"""
|
||||
Map OpenAI service_tier (string) to Gemini serviceTier.
|
||||
'auto' maps to 'priority'.
|
||||
Other values are passed lowercased.
|
||||
"""
|
||||
if value.lower() == "auto":
|
||||
optional_params["service_tier"] = "priority"
|
||||
else:
|
||||
optional_params["service_tier"] = value.lower()
|
||||
|
||||
def _transform_computer_use_config(self, computer_use_config: dict) -> dict:
|
||||
"""
|
||||
Transform Computer Use configuration to Gemini API format.
|
||||
|
|
@ -1121,6 +1133,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
|||
optional_params = self._add_tools_to_optional_params(
|
||||
optional_params, [_tools]
|
||||
)
|
||||
elif param == "service_tier" and isinstance(value, str):
|
||||
self._map_service_tier_param(value, optional_params)
|
||||
elif param == "include_server_side_tool_invocations" and value is True:
|
||||
optional_params["include_server_side_tool_invocations"] = True
|
||||
if litellm.vertex_ai_safety_settings is not None:
|
||||
|
|
@ -2415,6 +2429,14 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
|||
"provider_specific_fields", {}
|
||||
)["traffic_type"] = traffic_type
|
||||
|
||||
## ADD SERVICE TIER ##
|
||||
if getattr(raw_response, "headers", None):
|
||||
if service_tier := raw_response.headers.get("x-gemini-service-tier"):
|
||||
if service_tier.lower() == "standard":
|
||||
setattr(model_response, "service_tier", "default")
|
||||
else:
|
||||
setattr(model_response, "service_tier", service_tier.lower())
|
||||
|
||||
except Exception as e:
|
||||
raise VertexAIError(
|
||||
message="Received={}, Error converting to valid response block={}. File an issue if litellm error - https://github.com/BerriAI/litellm/issues".format(
|
||||
|
|
@ -2513,6 +2535,7 @@ async def make_call(
|
|||
streaming_response=response.aiter_lines(),
|
||||
sync_stream=False,
|
||||
logging_obj=logging_obj,
|
||||
response_headers=response.headers,
|
||||
)
|
||||
# LOGGING
|
||||
logging_obj.post_call(
|
||||
|
|
@ -2555,6 +2578,7 @@ def make_sync_call(
|
|||
streaming_response=response.iter_lines(),
|
||||
sync_stream=True,
|
||||
logging_obj=logging_obj,
|
||||
response_headers=response.headers,
|
||||
)
|
||||
|
||||
# LOGGING
|
||||
|
|
@ -3011,7 +3035,11 @@ class VertexLLM(VertexBase):
|
|||
|
||||
class ModelResponseIterator:
|
||||
def __init__(
|
||||
self, streaming_response, sync_stream: bool, logging_obj: LoggingClass
|
||||
self,
|
||||
streaming_response,
|
||||
sync_stream: bool,
|
||||
logging_obj: LoggingClass,
|
||||
response_headers: Optional[Dict[str, str]] = None,
|
||||
):
|
||||
from litellm.litellm_core_utils.prompt_templates.common_utils import (
|
||||
check_is_function_call,
|
||||
|
|
@ -3022,10 +3050,120 @@ class ModelResponseIterator:
|
|||
self.accumulated_json = ""
|
||||
self.sent_first_chunk = False
|
||||
self.logging_obj = logging_obj
|
||||
self.response_headers = response_headers or {}
|
||||
self.is_function_call = check_is_function_call(logging_obj)
|
||||
self.cumulative_tool_call_index: int = 0
|
||||
self.has_seen_tool_calls: bool = False
|
||||
|
||||
def _apply_stream_candidates(
|
||||
self,
|
||||
_candidates: List[Candidates],
|
||||
model_response: Any,
|
||||
) -> Tuple[List[dict], List[dict], List[dict], List[dict]]:
|
||||
(
|
||||
grounding_metadata,
|
||||
url_context_metadata,
|
||||
safety_ratings,
|
||||
citation_metadata,
|
||||
self.cumulative_tool_call_index,
|
||||
) = VertexGeminiConfig._process_candidates(
|
||||
_candidates,
|
||||
model_response,
|
||||
self.logging_obj.optional_params,
|
||||
cumulative_tool_call_index=self.cumulative_tool_call_index,
|
||||
)
|
||||
|
||||
# Track whether tool_calls have been seen across streaming chunks.
|
||||
# Gemini sends tool_calls and finishReason in separate chunks,
|
||||
# so we need to remember if earlier chunks contained tool_calls
|
||||
# to correctly set finish_reason="tool_calls" per the OpenAI spec.
|
||||
if not self.has_seen_tool_calls:
|
||||
for choice in model_response.choices:
|
||||
if (
|
||||
hasattr(choice, "delta")
|
||||
and choice.delta
|
||||
and choice.delta.tool_calls
|
||||
):
|
||||
self.has_seen_tool_calls = True
|
||||
break
|
||||
|
||||
# Handle final chunk with finishReason but no content.
|
||||
# _process_candidates skips candidates without "content",
|
||||
# so the finish_reason from the final chunk is lost.
|
||||
if not model_response.choices and _candidates:
|
||||
from litellm.types.utils import Delta, StreamingChoices
|
||||
|
||||
for candidate in _candidates:
|
||||
finish_reason_str = candidate.get("finishReason")
|
||||
if finish_reason_str is not None:
|
||||
if self.has_seen_tool_calls:
|
||||
mapped_finish_reason = "tool_calls"
|
||||
else:
|
||||
mapped_finish_reason = VertexGeminiConfig._check_finish_reason(
|
||||
None, finish_reason_str
|
||||
)
|
||||
choice = StreamingChoices(
|
||||
finish_reason=mapped_finish_reason,
|
||||
index=candidate.get("index", 0),
|
||||
delta=Delta(content=None, role=None),
|
||||
logprobs=None,
|
||||
enhancements=None,
|
||||
)
|
||||
model_response.choices.append(choice)
|
||||
|
||||
# Also handle the case where the final chunk has empty
|
||||
# content (e.g. text:"") WITH finishReason. In this case
|
||||
# _process_candidates DOES create a choice, but maps
|
||||
# finishReason="STOP" to "stop" because the current chunk
|
||||
# has no tool_calls. Override if we saw tool_calls earlier.
|
||||
if self.has_seen_tool_calls:
|
||||
for choice in model_response.choices:
|
||||
if choice.finish_reason == "stop":
|
||||
choice.finish_reason = "tool_calls"
|
||||
|
||||
setattr(model_response, "vertex_ai_grounding_metadata", grounding_metadata) # type: ignore
|
||||
setattr(model_response, "vertex_ai_url_context_metadata", url_context_metadata) # type: ignore
|
||||
setattr(model_response, "vertex_ai_safety_ratings", safety_ratings) # type: ignore
|
||||
setattr(model_response, "vertex_ai_citation_metadata", citation_metadata) # type: ignore
|
||||
|
||||
return grounding_metadata, url_context_metadata, safety_ratings, citation_metadata
|
||||
|
||||
def _apply_stream_usage_metadata(
|
||||
self,
|
||||
processed_chunk: Any,
|
||||
model_response: Any,
|
||||
grounding_metadata: List[dict],
|
||||
) -> Optional[Usage]:
|
||||
if "usageMetadata" not in processed_chunk:
|
||||
return None
|
||||
|
||||
usage = VertexGeminiConfig._calculate_usage(
|
||||
completion_response=processed_chunk,
|
||||
)
|
||||
|
||||
web_search_requests = VertexGeminiConfig._calculate_web_search_requests(
|
||||
grounding_metadata
|
||||
)
|
||||
if web_search_requests is not None:
|
||||
cast(
|
||||
PromptTokensDetailsWrapper, usage.prompt_tokens_details
|
||||
).web_search_requests = web_search_requests
|
||||
|
||||
traffic_type = processed_chunk.get("usageMetadata", {}).get("trafficType")
|
||||
if traffic_type:
|
||||
model_response._hidden_params.setdefault(
|
||||
"provider_specific_fields", {}
|
||||
)["traffic_type"] = traffic_type
|
||||
|
||||
service_tier = self.response_headers.get("x-gemini-service-tier")
|
||||
if service_tier:
|
||||
if service_tier.lower() == "standard":
|
||||
setattr(model_response, "service_tier", "default")
|
||||
else:
|
||||
setattr(model_response, "service_tier", service_tier.lower())
|
||||
|
||||
return usage
|
||||
|
||||
def chunk_parser(self, chunk: dict) -> Optional["ModelResponseStream"]:
|
||||
try:
|
||||
verbose_logger.debug(f"RAW GEMINI CHUNK: {chunk}")
|
||||
|
|
@ -3043,101 +3181,23 @@ class ModelResponseIterator:
|
|||
if blocked_response is not None:
|
||||
model_response = blocked_response
|
||||
|
||||
usage: Optional[Usage] = None
|
||||
_candidates: Optional[List[Candidates]] = processed_chunk.get("candidates")
|
||||
grounding_metadata: List[dict] = []
|
||||
url_context_metadata: List[dict] = []
|
||||
safety_ratings: List[dict] = []
|
||||
citation_metadata: List[dict] = []
|
||||
|
||||
_candidates: Optional[List[Candidates]] = processed_chunk.get("candidates")
|
||||
if _candidates:
|
||||
(
|
||||
grounding_metadata,
|
||||
url_context_metadata,
|
||||
safety_ratings,
|
||||
citation_metadata,
|
||||
self.cumulative_tool_call_index,
|
||||
) = VertexGeminiConfig._process_candidates(
|
||||
_candidates,
|
||||
model_response,
|
||||
self.logging_obj.optional_params,
|
||||
cumulative_tool_call_index=self.cumulative_tool_call_index,
|
||||
)
|
||||
) = self._apply_stream_candidates(_candidates, model_response)
|
||||
|
||||
# Track whether tool_calls have been seen across streaming chunks.
|
||||
# Gemini sends tool_calls and finishReason in separate chunks,
|
||||
# so we need to remember if earlier chunks contained tool_calls
|
||||
# to correctly set finish_reason="tool_calls" per the OpenAI spec.
|
||||
if not self.has_seen_tool_calls:
|
||||
for choice in model_response.choices:
|
||||
if (
|
||||
hasattr(choice, "delta")
|
||||
and choice.delta
|
||||
and choice.delta.tool_calls
|
||||
):
|
||||
self.has_seen_tool_calls = True
|
||||
break
|
||||
|
||||
# Handle final chunk with finishReason but no content.
|
||||
# _process_candidates skips candidates without "content",
|
||||
# so the finish_reason from the final chunk is lost.
|
||||
if not model_response.choices and _candidates:
|
||||
from litellm.types.utils import Delta, StreamingChoices
|
||||
|
||||
for candidate in _candidates:
|
||||
finish_reason_str = candidate.get("finishReason")
|
||||
if finish_reason_str is not None:
|
||||
if self.has_seen_tool_calls:
|
||||
mapped_finish_reason = "tool_calls"
|
||||
else:
|
||||
mapped_finish_reason = (
|
||||
VertexGeminiConfig._check_finish_reason(
|
||||
None, finish_reason_str
|
||||
)
|
||||
)
|
||||
choice = StreamingChoices(
|
||||
finish_reason=mapped_finish_reason,
|
||||
index=candidate.get("index", 0),
|
||||
delta=Delta(content=None, role=None),
|
||||
logprobs=None,
|
||||
enhancements=None,
|
||||
)
|
||||
model_response.choices.append(choice)
|
||||
|
||||
# Also handle the case where the final chunk has empty
|
||||
# content (e.g. text:"") WITH finishReason. In this case
|
||||
# _process_candidates DOES create a choice, but maps
|
||||
# finishReason="STOP" to "stop" because the current chunk
|
||||
# has no tool_calls. Override if we saw tool_calls earlier.
|
||||
if self.has_seen_tool_calls:
|
||||
for choice in model_response.choices:
|
||||
if choice.finish_reason == "stop":
|
||||
choice.finish_reason = "tool_calls"
|
||||
|
||||
setattr(model_response, "vertex_ai_grounding_metadata", grounding_metadata) # type: ignore
|
||||
setattr(model_response, "vertex_ai_url_context_metadata", url_context_metadata) # type: ignore
|
||||
setattr(model_response, "vertex_ai_safety_ratings", safety_ratings) # type: ignore
|
||||
setattr(model_response, "vertex_ai_citation_metadata", citation_metadata) # type: ignore
|
||||
|
||||
if "usageMetadata" in processed_chunk:
|
||||
usage = VertexGeminiConfig._calculate_usage(
|
||||
completion_response=processed_chunk,
|
||||
)
|
||||
|
||||
web_search_requests = VertexGeminiConfig._calculate_web_search_requests(
|
||||
grounding_metadata
|
||||
)
|
||||
if web_search_requests is not None:
|
||||
cast(
|
||||
PromptTokensDetailsWrapper, usage.prompt_tokens_details
|
||||
).web_search_requests = web_search_requests
|
||||
|
||||
traffic_type = processed_chunk.get("usageMetadata", {}).get(
|
||||
"trafficType"
|
||||
)
|
||||
if traffic_type:
|
||||
model_response._hidden_params.setdefault(
|
||||
"provider_specific_fields", {}
|
||||
)["traffic_type"] = traffic_type
|
||||
usage = self._apply_stream_usage_metadata(
|
||||
processed_chunk, model_response, grounding_metadata
|
||||
)
|
||||
|
||||
setattr(model_response, "usage", usage) # type: ignore
|
||||
|
||||
|
|
|
|||
|
|
@ -136,6 +136,11 @@ class VertexBase:
|
|||
json_obj,
|
||||
scopes=["https://www.googleapis.com/auth/cloud-platform"],
|
||||
)
|
||||
elif isinstance(credential_source, dict) and "executable" in credential_source:
|
||||
creds = self._credentials_from_pluggable(
|
||||
json_obj,
|
||||
scopes=["https://www.googleapis.com/auth/cloud-platform"],
|
||||
)
|
||||
else:
|
||||
creds = self._credentials_from_identity_pool(
|
||||
json_obj,
|
||||
|
|
@ -190,6 +195,17 @@ class VertexBase:
|
|||
creds = creds.with_scopes(scopes)
|
||||
return creds
|
||||
|
||||
def _credentials_from_pluggable(self, json_obj, scopes):
|
||||
try:
|
||||
from google.auth import pluggable
|
||||
except ImportError:
|
||||
raise ImportError(GOOGLE_IMPORT_ERROR_MESSAGE)
|
||||
|
||||
creds = pluggable.Credentials.from_info(json_obj)
|
||||
if scopes and hasattr(creds, "requires_scopes") and creds.requires_scopes:
|
||||
creds = creds.with_scopes(scopes)
|
||||
return creds
|
||||
|
||||
def _credentials_from_identity_pool_with_aws(self, json_obj, scopes):
|
||||
try:
|
||||
from google.auth import aws
|
||||
|
|
|
|||
|
|
@ -7818,26 +7818,6 @@
|
|||
"cache_read_input_token_cost": 3e-08,
|
||||
"cache_creation_input_token_cost": 3.75e-07
|
||||
},
|
||||
"bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": {
|
||||
"input_cost_per_token": 3.6e-06,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.8e-05,
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"cache_read_input_token_cost": 3.6e-07,
|
||||
"cache_creation_input_token_cost": 4.5e-06
|
||||
},
|
||||
"bedrock/us-gov-east-1/anthropic.claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
|
|
@ -7856,7 +7836,29 @@
|
|||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_native_structured_output": true
|
||||
},
|
||||
"bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
"input_cost_per_token": 3.3e-06,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.65e-05,
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_native_structured_output": true
|
||||
},
|
||||
"bedrock/us-gov-east-1/meta.llama3-70b-instruct-v1:0": {
|
||||
"input_cost_per_token": 2.65e-06,
|
||||
|
|
@ -7989,26 +7991,6 @@
|
|||
"cache_read_input_token_cost": 3e-08,
|
||||
"cache_creation_input_token_cost": 3.75e-07
|
||||
},
|
||||
"bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": {
|
||||
"input_cost_per_token": 3.6e-06,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.8e-05,
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"cache_read_input_token_cost": 3.6e-07,
|
||||
"cache_creation_input_token_cost": 4.5e-06
|
||||
},
|
||||
"bedrock/us-gov-west-1/anthropic.claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
|
|
@ -8027,7 +8009,29 @@
|
|||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_native_structured_output": true
|
||||
},
|
||||
"bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
"input_cost_per_token": 3.3e-06,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.65e-05,
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_native_structured_output": true
|
||||
},
|
||||
"bedrock/us-gov-west-1/meta.llama3-70b-instruct-v1:0": {
|
||||
"input_cost_per_token": 2.65e-06,
|
||||
|
|
@ -13735,7 +13739,8 @@
|
|||
"supports_tool_choice": true,
|
||||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-2.5-flash-image": {
|
||||
"cache_read_input_token_cost": 3e-08,
|
||||
|
|
@ -13784,7 +13789,8 @@
|
|||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": false,
|
||||
"tpm": 8000000
|
||||
"tpm": 8000000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3-pro-image-preview": {
|
||||
"input_cost_per_image": 0.0011,
|
||||
|
|
@ -13818,7 +13824,8 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3.1-flash-image-preview": {
|
||||
"input_cost_per_image": 0.00056,
|
||||
|
|
@ -13901,7 +13908,8 @@
|
|||
"supports_video_input": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"deep-research-pro-preview-12-2025": {
|
||||
"input_cost_per_image": 0.0011,
|
||||
|
|
@ -13980,7 +13988,8 @@
|
|||
"supports_tool_choice": true,
|
||||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-2.5-flash-lite-preview-09-2025": {
|
||||
"cache_read_input_token_cost": 1e-08,
|
||||
|
|
@ -14251,7 +14260,8 @@
|
|||
"supports_tool_choice": true,
|
||||
"supports_video_input": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3-pro-preview": {
|
||||
"deprecation_date": "2026-03-26",
|
||||
|
|
@ -15033,7 +15043,8 @@
|
|||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"tpm": 8000000
|
||||
"tpm": 8000000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-2.5-flash-image": {
|
||||
"cache_read_input_token_cost": 3e-08,
|
||||
|
|
@ -15083,7 +15094,8 @@
|
|||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"tpm": 8000000
|
||||
"tpm": 8000000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3-pro-image-preview": {
|
||||
"input_cost_per_image": 0.0011,
|
||||
|
|
@ -15119,7 +15131,8 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3.1-flash-image-preview": {
|
||||
"input_cost_per_token": 2.5e-07,
|
||||
|
|
@ -15238,7 +15251,8 @@
|
|||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"tpm": 250000
|
||||
"tpm": 250000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-2.5-flash-lite-preview-09-2025": {
|
||||
"cache_read_input_token_cost": 1e-08,
|
||||
|
|
@ -15678,7 +15692,8 @@
|
|||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true,
|
||||
"tpm": 250000
|
||||
"tpm": 250000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3-flash-preview": {
|
||||
"cache_read_input_token_cost": 5e-08,
|
||||
|
|
@ -38123,4 +38138,4 @@
|
|||
"supports_native_structured_output": true,
|
||||
"supports_pdf_input": true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -2435,6 +2435,7 @@ class LiteLLM_VerificationTokenView(LiteLLM_VerificationToken):
|
|||
end_user_model_max_budget: Optional[dict] = None
|
||||
|
||||
# Organization Params
|
||||
organization_alias: Optional[str] = None
|
||||
organization_max_budget: Optional[float] = None
|
||||
organization_tpm_limit: Optional[int] = None
|
||||
organization_rpm_limit: Optional[int] = None
|
||||
|
|
|
|||
|
|
@ -684,6 +684,7 @@ class LiteLLMProxyRequestSetup:
|
|||
user_api_key_project_alias=user_api_key_dict.project_alias,
|
||||
user_api_key_user_id=user_api_key_dict.user_id,
|
||||
user_api_key_org_id=user_api_key_dict.org_id,
|
||||
user_api_key_org_alias=user_api_key_dict.organization_alias,
|
||||
user_api_key_team_alias=user_api_key_dict.team_alias,
|
||||
user_api_key_end_user_id=user_api_key_dict.end_user_id,
|
||||
user_api_key_user_email=user_api_key_dict.user_email,
|
||||
|
|
|
|||
|
|
@ -7128,6 +7128,13 @@ async def chat_completion( # noqa: PLR0915
|
|||
and user_api_key_dict.org_id is not None
|
||||
):
|
||||
data["metadata"]["user_api_key_org_id"] = user_api_key_dict.org_id
|
||||
if (
|
||||
hasattr(user_api_key_dict, "organization_alias")
|
||||
and user_api_key_dict.organization_alias is not None
|
||||
):
|
||||
data["metadata"]["user_api_key_org_alias"] = (
|
||||
user_api_key_dict.organization_alias
|
||||
)
|
||||
if (
|
||||
hasattr(user_api_key_dict, "agent_id")
|
||||
and user_api_key_dict.agent_id is not None
|
||||
|
|
@ -7302,6 +7309,13 @@ async def completion( # noqa: PLR0915
|
|||
and user_api_key_dict.org_id is not None
|
||||
):
|
||||
data["metadata"]["user_api_key_org_id"] = user_api_key_dict.org_id
|
||||
if (
|
||||
hasattr(user_api_key_dict, "organization_alias")
|
||||
and user_api_key_dict.organization_alias is not None
|
||||
):
|
||||
data["metadata"]["user_api_key_org_alias"] = (
|
||||
user_api_key_dict.organization_alias
|
||||
)
|
||||
if (
|
||||
hasattr(user_api_key_dict, "agent_id")
|
||||
and user_api_key_dict.agent_id is not None
|
||||
|
|
@ -7544,6 +7558,13 @@ async def embeddings( # noqa: PLR0915
|
|||
and user_api_key_dict.org_id is not None
|
||||
):
|
||||
data["metadata"]["user_api_key_org_id"] = user_api_key_dict.org_id
|
||||
if (
|
||||
hasattr(user_api_key_dict, "organization_alias")
|
||||
and user_api_key_dict.organization_alias is not None
|
||||
):
|
||||
data["metadata"]["user_api_key_org_alias"] = (
|
||||
user_api_key_dict.organization_alias
|
||||
)
|
||||
if (
|
||||
hasattr(user_api_key_dict, "agent_id")
|
||||
and user_api_key_dict.agent_id is not None
|
||||
|
|
|
|||
|
|
@ -3004,6 +3004,7 @@ class PrismaClient:
|
|||
b.model_max_budget as litellm_budget_table_model_max_budget,
|
||||
b.soft_budget as litellm_budget_table_soft_budget,
|
||||
o.metadata as organization_metadata,
|
||||
o.organization_alias as organization_alias,
|
||||
b2.max_budget as organization_max_budget,
|
||||
b2.tpm_limit as organization_tpm_limit,
|
||||
b2.rpm_limit as organization_rpm_limit
|
||||
|
|
@ -5293,11 +5294,12 @@ def handle_exception_on_proxy(e: Exception) -> ProxyException:
|
|||
)
|
||||
elif isinstance(e, ProxyException):
|
||||
return e
|
||||
_status_code = getattr(e, "status_code", status.HTTP_500_INTERNAL_SERVER_ERROR)
|
||||
return ProxyException(
|
||||
message="Internal Server Error, " + str(e),
|
||||
message=str(e),
|
||||
type=ProxyErrorTypes.internal_server_error,
|
||||
param=getattr(e, "param", "None"),
|
||||
code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
code=_status_code,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import re
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Any, Dict, List, Literal, Optional, Tuple
|
||||
from typing import Any, ClassVar, Dict, List, Literal, Optional, Tuple
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from typing_extensions import Annotated
|
||||
|
|
@ -665,6 +665,24 @@ class PrometheusMetricLabels:
|
|||
litellm_cache_misses_metric = _cache_metric_labels
|
||||
litellm_cached_tokens_metric = _cache_metric_labels
|
||||
|
||||
# Metrics whose emission paths supply org context (used by get_labels)
|
||||
_org_label_metrics: ClassVar[frozenset] = frozenset(
|
||||
{
|
||||
"litellm_llm_api_latency_metric",
|
||||
"litellm_llm_api_time_to_first_token_metric",
|
||||
"litellm_request_total_latency_metric",
|
||||
"litellm_request_queue_time_seconds",
|
||||
"litellm_proxy_total_requests_metric",
|
||||
"litellm_proxy_failed_requests_metric",
|
||||
"litellm_deployment_latency_per_output_token",
|
||||
"litellm_requests_metric",
|
||||
"litellm_spend_metric",
|
||||
"litellm_input_tokens_metric",
|
||||
"litellm_total_tokens_metric",
|
||||
"litellm_output_tokens_metric",
|
||||
}
|
||||
)
|
||||
|
||||
# Managed batch metrics
|
||||
_batch_user_labels = [
|
||||
UserAPIKeyLabelNames.v1_LITELLM_MODEL_NAME.value,
|
||||
|
|
@ -731,6 +749,14 @@ class PrometheusMetricLabels:
|
|||
):
|
||||
custom_labels.append(UserAPIKeyLabelNames.STREAM.value)
|
||||
|
||||
if label_name in PrometheusMetricLabels._org_label_metrics:
|
||||
for label in [
|
||||
UserAPIKeyLabelNames.ORG_ID.value,
|
||||
UserAPIKeyLabelNames.ORG_ALIAS.value,
|
||||
]:
|
||||
if label not in default_labels and label not in custom_labels:
|
||||
custom_labels.append(label)
|
||||
|
||||
return default_labels + custom_labels
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -325,6 +325,7 @@ class RequestBody(TypedDict, total=False):
|
|||
generationConfig: GenerationConfig
|
||||
cachedContent: str
|
||||
labels: Dict[str, str]
|
||||
serviceTier: str
|
||||
|
||||
|
||||
class CachedContentRequestBody(TypedDict, total=False):
|
||||
|
|
|
|||
|
|
@ -2507,6 +2507,7 @@ class StandardLoggingUserAPIKeyMetadata(TypedDict):
|
|||
user_api_key_max_budget: Optional[float]
|
||||
user_api_key_budget_reset_at: Optional[str]
|
||||
user_api_key_org_id: Optional[str]
|
||||
user_api_key_org_alias: Optional[str]
|
||||
user_api_key_team_id: Optional[str]
|
||||
user_api_key_project_id: Optional[str]
|
||||
user_api_key_project_alias: Optional[str]
|
||||
|
|
|
|||
|
|
@ -7818,26 +7818,6 @@
|
|||
"cache_read_input_token_cost": 3e-08,
|
||||
"cache_creation_input_token_cost": 3.75e-07
|
||||
},
|
||||
"bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": {
|
||||
"input_cost_per_token": 3.6e-06,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.8e-05,
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"cache_read_input_token_cost": 3.6e-07,
|
||||
"cache_creation_input_token_cost": 4.5e-06
|
||||
},
|
||||
"bedrock/us-gov-east-1/anthropic.claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
|
|
@ -7856,7 +7836,29 @@
|
|||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_native_structured_output": true
|
||||
},
|
||||
"bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
"input_cost_per_token": 3.3e-06,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.65e-05,
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_native_structured_output": true
|
||||
},
|
||||
"bedrock/us-gov-east-1/meta.llama3-70b-instruct-v1:0": {
|
||||
"input_cost_per_token": 2.65e-06,
|
||||
|
|
@ -7989,26 +7991,6 @@
|
|||
"cache_read_input_token_cost": 3e-08,
|
||||
"cache_creation_input_token_cost": 3.75e-07
|
||||
},
|
||||
"bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": {
|
||||
"input_cost_per_token": 3.6e-06,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.8e-05,
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"cache_read_input_token_cost": 3.6e-07,
|
||||
"cache_creation_input_token_cost": 4.5e-06
|
||||
},
|
||||
"bedrock/us-gov-west-1/anthropic.claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
|
|
@ -8027,7 +8009,29 @@
|
|||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_native_structured_output": true
|
||||
},
|
||||
"bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
"input_cost_per_token": 3.3e-06,
|
||||
"litellm_provider": "bedrock",
|
||||
"max_input_tokens": 200000,
|
||||
"max_output_tokens": 8192,
|
||||
"max_tokens": 8192,
|
||||
"mode": "chat",
|
||||
"output_cost_per_token": 1.65e-05,
|
||||
"supports_assistant_prefill": true,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_native_structured_output": true
|
||||
},
|
||||
"bedrock/us-gov-west-1/meta.llama3-70b-instruct-v1:0": {
|
||||
"input_cost_per_token": 2.65e-06,
|
||||
|
|
@ -13735,7 +13739,8 @@
|
|||
"supports_tool_choice": true,
|
||||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-2.5-flash-image": {
|
||||
"cache_read_input_token_cost": 3e-08,
|
||||
|
|
@ -13784,7 +13789,8 @@
|
|||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": false,
|
||||
"tpm": 8000000
|
||||
"tpm": 8000000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3-pro-image-preview": {
|
||||
"input_cost_per_image": 0.0011,
|
||||
|
|
@ -13818,7 +13824,8 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3.1-flash-image-preview": {
|
||||
"input_cost_per_image": 0.00056,
|
||||
|
|
@ -13901,7 +13908,8 @@
|
|||
"supports_video_input": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true
|
||||
"supports_native_streaming": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"deep-research-pro-preview-12-2025": {
|
||||
"input_cost_per_image": 0.0011,
|
||||
|
|
@ -13980,7 +13988,8 @@
|
|||
"supports_tool_choice": true,
|
||||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-2.5-flash-lite-preview-09-2025": {
|
||||
"cache_read_input_token_cost": 1e-08,
|
||||
|
|
@ -14251,7 +14260,8 @@
|
|||
"supports_tool_choice": true,
|
||||
"supports_video_input": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini-3-pro-preview": {
|
||||
"deprecation_date": "2026-03-26",
|
||||
|
|
@ -15033,7 +15043,8 @@
|
|||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"tpm": 8000000
|
||||
"tpm": 8000000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-2.5-flash-image": {
|
||||
"cache_read_input_token_cost": 3e-08,
|
||||
|
|
@ -15083,7 +15094,8 @@
|
|||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"tpm": 8000000
|
||||
"tpm": 8000000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3-pro-image-preview": {
|
||||
"input_cost_per_image": 0.0011,
|
||||
|
|
@ -15119,7 +15131,8 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3.1-flash-image-preview": {
|
||||
"input_cost_per_token": 2.5e-07,
|
||||
|
|
@ -15238,7 +15251,8 @@
|
|||
"supports_url_context": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"tpm": 250000
|
||||
"tpm": 250000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-2.5-flash-lite-preview-09-2025": {
|
||||
"cache_read_input_token_cost": 1e-08,
|
||||
|
|
@ -15678,7 +15692,8 @@
|
|||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_native_streaming": true,
|
||||
"tpm": 250000
|
||||
"tpm": 250000,
|
||||
"supports_service_tier": true
|
||||
},
|
||||
"gemini/gemini-3-flash-preview": {
|
||||
"cache_read_input_token_cost": 5e-08,
|
||||
|
|
@ -38108,4 +38123,4 @@
|
|||
"supports_native_structured_output": true,
|
||||
"supports_pdf_input": true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -223,6 +223,8 @@ def test_increment_token_metrics(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
requested_model=None,
|
||||
model="gpt-3.5-turbo",
|
||||
model_id="model-123",
|
||||
|
|
@ -237,6 +239,8 @@ def test_increment_token_metrics(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
requested_model=None,
|
||||
model="gpt-3.5-turbo",
|
||||
model_id="model-123",
|
||||
|
|
@ -253,6 +257,8 @@ def test_increment_token_metrics(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
requested_model=None,
|
||||
model="gpt-3.5-turbo",
|
||||
model_id="model-123",
|
||||
|
|
@ -414,6 +420,8 @@ def test_set_latency_metrics(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
requested_model="openai-gpt",
|
||||
model="gpt-3.5-turbo",
|
||||
model_id="model-123",
|
||||
|
|
@ -430,6 +438,8 @@ def test_set_latency_metrics(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
requested_model="openai-gpt",
|
||||
model="gpt-3.5-turbo",
|
||||
model_id="model-123",
|
||||
|
|
@ -446,6 +456,8 @@ def test_set_latency_metrics(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
requested_model="openai-gpt",
|
||||
model="gpt-3.5-turbo",
|
||||
model_id="model-123",
|
||||
|
|
@ -589,6 +601,8 @@ def test_increment_top_level_request_and_spend_metrics(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
model="gpt-3.5-turbo",
|
||||
model_id="model-123",
|
||||
client_ip=None,
|
||||
|
|
@ -605,6 +619,8 @@ def test_increment_top_level_request_and_spend_metrics(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
model="gpt-3.5-turbo",
|
||||
model_id="model-123",
|
||||
client_ip=None,
|
||||
|
|
@ -758,6 +774,8 @@ async def test_async_post_call_failure_hook(prometheus_logger):
|
|||
api_key_alias="test_alias",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
requested_model="gpt-3.5-turbo",
|
||||
exception_status="429",
|
||||
exception_class="Openai.RateLimitError",
|
||||
|
|
@ -776,6 +794,8 @@ async def test_async_post_call_failure_hook(prometheus_logger):
|
|||
requested_model="gpt-3.5-turbo",
|
||||
team="test_team",
|
||||
team_alias="test_team_alias",
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
user="test_user",
|
||||
status_code="429",
|
||||
user_email=None,
|
||||
|
|
@ -955,6 +975,8 @@ def test_set_llm_deployment_success_metrics(prometheus_logger):
|
|||
api_key_alias=standard_logging_payload["metadata"]["user_api_key_alias"],
|
||||
team=standard_logging_payload["metadata"]["user_api_key_team_id"],
|
||||
team_alias=standard_logging_payload["metadata"]["user_api_key_team_alias"],
|
||||
org_id=None,
|
||||
org_alias=None,
|
||||
)
|
||||
prometheus_logger.litellm_overhead_latency_metric.labels.assert_called_once_with(
|
||||
api_base="https://api.openai.com",
|
||||
|
|
|
|||
|
|
@ -2637,3 +2637,50 @@ async def test_handle_logging_proxy_only_error_skips_handlers_for_pass_through()
|
|||
mock_async.assert_not_called()
|
||||
mock_sync.assert_not_called()
|
||||
assert logging_obj.call_type == CallTypes.pass_through.value
|
||||
|
||||
|
||||
def test_handle_exception_on_proxy_preserves_status_code():
|
||||
"""
|
||||
OpenAI batch creation returns 429 for rate limits. LiteLLM wraps this as a
|
||||
RateLimitError with status_code=429. handle_exception_on_proxy must pass
|
||||
that status code through instead of hardcoding 500.
|
||||
"""
|
||||
from litellm.proxy.utils import handle_exception_on_proxy
|
||||
|
||||
rate_limit_error = litellm.RateLimitError(
|
||||
message="Rate limit exceeded: batch creation limit of 2000/hour hit",
|
||||
llm_provider="openai",
|
||||
model="gpt-4o",
|
||||
)
|
||||
|
||||
result = handle_exception_on_proxy(rate_limit_error)
|
||||
|
||||
assert int(result.code) == 429, f"Expected 429, got {result.code}"
|
||||
|
||||
|
||||
def test_handle_exception_on_proxy_defaults_to_500_for_unknown_exceptions():
|
||||
"""
|
||||
Generic exceptions with no status_code should still return 500.
|
||||
"""
|
||||
from litellm.proxy.utils import handle_exception_on_proxy
|
||||
|
||||
result = handle_exception_on_proxy(Exception("something went wrong"))
|
||||
|
||||
assert int(result.code) == 500, f"Expected 500, got {result.code}"
|
||||
|
||||
|
||||
def test_handle_exception_on_proxy_preserves_auth_error_status_code():
|
||||
"""
|
||||
AuthenticationError (401) should also pass through correctly.
|
||||
"""
|
||||
from litellm.proxy.utils import handle_exception_on_proxy
|
||||
|
||||
auth_error = litellm.AuthenticationError(
|
||||
message="Invalid API key",
|
||||
llm_provider="openai",
|
||||
model="gpt-4o",
|
||||
)
|
||||
|
||||
result = handle_exception_on_proxy(auth_error)
|
||||
|
||||
assert int(result.code) == 401, f"Expected 401, got {result.code}"
|
||||
|
|
|
|||
|
|
@ -118,6 +118,8 @@ async def test_async_post_call_success_hook_includes_client_ip_user_agent():
|
|||
"user_api_key_alias": "alias_1",
|
||||
"user_api_key_team_id": "team_1",
|
||||
"user_api_key_team_alias": "team_alias_1",
|
||||
"user_api_key_org_id": None,
|
||||
"user_api_key_org_alias": None,
|
||||
"user_api_key_user_email": "test@example.com",
|
||||
"user_api_key_request_route": "/chat/completions",
|
||||
"requester_ip_address": "192.168.1.1",
|
||||
|
|
|
|||
|
|
@ -525,6 +525,61 @@ async def test_set_user_budget_metrics_after_api_request_inf_when_genuinely_no_b
|
|||
)
|
||||
|
||||
|
||||
def test_per_request_metrics_emit_all_identity_labels(prometheus_logger):
|
||||
"""Verify org labels appear when flag is on and are absent when flag is off."""
|
||||
import litellm
|
||||
from litellm.types.integrations.prometheus import UserAPIKeyLabelValues
|
||||
|
||||
prometheus_logger.litellm_requests_metric = MagicMock()
|
||||
prometheus_logger.litellm_spend_metric = MagicMock()
|
||||
|
||||
enum_values = UserAPIKeyLabelValues(
|
||||
hashed_api_key="hashed-key",
|
||||
api_key_alias="my-key",
|
||||
model="gpt-4",
|
||||
team="team-abc",
|
||||
team_alias="my-team",
|
||||
org_id="org-abc",
|
||||
org_alias="my-org",
|
||||
user="user-1",
|
||||
)
|
||||
|
||||
common_kwargs = dict(
|
||||
end_user_id=None,
|
||||
user_api_key="hashed-key",
|
||||
user_api_key_alias="my-key",
|
||||
model="gpt-4",
|
||||
user_api_team="team-abc",
|
||||
user_api_team_alias="my-team",
|
||||
user_id="user-1",
|
||||
response_cost=0.001,
|
||||
enum_values=enum_values,
|
||||
)
|
||||
|
||||
try:
|
||||
# org labels are always included in per-request metrics
|
||||
prometheus_logger._increment_top_level_request_and_spend_metrics(**common_kwargs)
|
||||
label_kwargs = prometheus_logger.litellm_requests_metric.labels.call_args.kwargs
|
||||
assert label_kwargs["org_id"] == "org-abc"
|
||||
assert label_kwargs["org_alias"] == "my-org"
|
||||
assert label_kwargs["team"] == "team-abc"
|
||||
assert label_kwargs["user"] == "user-1"
|
||||
|
||||
# Metrics not in the org-emission list must NOT get org labels
|
||||
from litellm.types.integrations.prometheus import PrometheusMetricLabels
|
||||
for metric in ("litellm_remaining_api_key_budget_metric", "litellm_remaining_team_budget_metric"):
|
||||
labels = PrometheusMetricLabels.get_labels(metric)
|
||||
assert "org_id" not in labels, f"{metric} should not have org_id"
|
||||
assert "org_alias" not in labels, f"{metric} should not have org_alias"
|
||||
|
||||
# org_id in custom_prometheus_metadata_labels must not produce duplicate labels
|
||||
litellm.custom_prometheus_metadata_labels = ["org_id"]
|
||||
labels = PrometheusMetricLabels.get_labels("litellm_requests_metric")
|
||||
assert labels.count("org_id") == 1
|
||||
finally:
|
||||
litellm.custom_prometheus_metadata_labels = []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Org budget metric tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
from unittest.mock import MagicMock
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from litellm.constants import RESPONSE_FORMAT_TOOL_NAME
|
||||
from litellm.llms.anthropic.chat.handler import ModelResponseIterator
|
||||
from litellm.llms.anthropic.chat.handler import ModelResponseIterator, make_call
|
||||
from litellm.types.llms.openai import (
|
||||
ChatCompletionToolCallChunk,
|
||||
ChatCompletionToolCallFunctionChunk,
|
||||
|
|
@ -9,6 +11,33 @@ from litellm.types.llms.openai import (
|
|||
from litellm.types.responses.main import OutputCodeInterpreterCall
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_make_call_passes_logging_obj_to_client_post():
|
||||
"""make_call must pass logging_obj to client.post so track_llm_api_timing can set llm_api_duration_ms for litellm_overhead_time_ms."""
|
||||
mock_client = AsyncMock()
|
||||
mock_response = MagicMock()
|
||||
mock_response.aiter_lines = MagicMock(return_value=iter([b'data: {"type":"message_start"}\n', b'data: {"type":"message_delta"}\n']))
|
||||
mock_client.post.return_value = mock_response
|
||||
|
||||
logging_obj = MagicMock()
|
||||
|
||||
await make_call(
|
||||
client=mock_client,
|
||||
api_base="https://api.anthropic.com/v1/messages",
|
||||
headers={},
|
||||
data="{}",
|
||||
model="claude-3-5-haiku",
|
||||
messages=[{"role": "user", "content": "Hi"}],
|
||||
logging_obj=logging_obj,
|
||||
timeout=60.0,
|
||||
json_mode=False,
|
||||
)
|
||||
|
||||
mock_client.post.assert_called_once()
|
||||
call_kwargs = mock_client.post.call_args[1]
|
||||
assert call_kwargs.get("logging_obj") is logging_obj
|
||||
|
||||
|
||||
def test_redacted_thinking_content_block_delta():
|
||||
chunk = {
|
||||
"type": "content_block_start",
|
||||
|
|
|
|||
|
|
@ -127,11 +127,12 @@ class TestToolTransformationIntegration:
|
|||
}
|
||||
|
||||
validated_tool = validate_dict(openai_tool, ChatCompletionTool)
|
||||
|
||||
|
||||
# After validation, parameters should have type='object'
|
||||
assert validated_tool["function"]["parameters"]["type"] == "object"
|
||||
assert "properties" in validated_tool["function"]["parameters"]
|
||||
|
||||
|
||||
def test_should_transform_tool_with_existing_parameters(self):
|
||||
"""Tool with parameters should preserve them while ensuring type='object'."""
|
||||
from litellm.llms.sap.chat.transformation import validate_dict
|
||||
|
|
|
|||
564
tests/test_litellm/llms/sap/chat/test_sap_transformation.py
Normal file
564
tests/test_litellm/llms/sap/chat/test_sap_transformation.py
Normal file
|
|
@ -0,0 +1,564 @@
|
|||
import warnings
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
class TestSAPTransformationIntegration:
|
||||
"""Integration tests for SAP transformation."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_config(self):
|
||||
from litellm.llms.sap.chat.transformation import GenAIHubOrchestrationConfig
|
||||
|
||||
config = GenAIHubOrchestrationConfig()
|
||||
config.token_creator = lambda: "Bearer TEST_TOKEN"
|
||||
config._base_url = "https://api.test-sap.com"
|
||||
config._resource_group = "test-group"
|
||||
|
||||
return config
|
||||
|
||||
def test_parameter_classification_in_transform_request(self, mock_config):
|
||||
"""Test parameter classification within the actual transform_request method."""
|
||||
|
||||
model = "gpt-4o"
|
||||
messages = [{"role": "user", "content": "Hello"}]
|
||||
|
||||
optional_params = {
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 100,
|
||||
"deployment_url": "https://custom.sap.com/deployment/123",
|
||||
"model_version": "v1.5",
|
||||
"tools": [{"type": "function", "function": {"name": "calculator"}}],
|
||||
"frequency_penalty": 0.1
|
||||
}
|
||||
|
||||
result = mock_config.transform_request(
|
||||
model, messages, optional_params, {}, {}
|
||||
)
|
||||
|
||||
model_params = result["config"]["modules"]["prompt_templating"]["model"]["params"]
|
||||
|
||||
assert "temperature" in model_params
|
||||
assert "frequency_penalty" in model_params
|
||||
assert "deployment_url" not in model_params
|
||||
assert "model_version" not in model_params
|
||||
assert "tools" not in model_params
|
||||
|
||||
model_version = result["config"]["modules"]["prompt_templating"]["model"]["version"]
|
||||
assert model_version == "v1.5"
|
||||
|
||||
prompt = result["config"]["modules"]["prompt_templating"]["prompt"]
|
||||
if "tools" in prompt:
|
||||
assert isinstance(prompt["tools"], list)
|
||||
for tool in prompt["tools"]:
|
||||
assert tool["function"]["parameters"]["type"] == "object", (
|
||||
"SAP API requires parameters.type == 'object'"
|
||||
)
|
||||
assert "properties" in tool["function"]["parameters"]
|
||||
|
||||
def test_transform_request_parameter_handling_robustness(self, mock_config):
|
||||
"""Test transform_request method handles various parameter combinations correctly."""
|
||||
|
||||
model = "gpt-4o"
|
||||
messages = [{"role": "user", "content": "Hello"}]
|
||||
|
||||
test_cases = [
|
||||
# Case 1: Basic parameters only
|
||||
{
|
||||
"params": {"temperature": 0.7, "max_tokens": 100},
|
||||
"expected_in_model": {"temperature", "max_tokens"},
|
||||
"expected_excluded": set()
|
||||
},
|
||||
# Case 2: Parameters with auth/infrastructure components
|
||||
{
|
||||
"params": {
|
||||
"temperature": 0.8,
|
||||
"deployment_url": "https://api.sap.com/deployments/test",
|
||||
"max_tokens": 150
|
||||
},
|
||||
"expected_in_model": {"temperature", "max_tokens"},
|
||||
"expected_excluded": {"deployment_url"}
|
||||
},
|
||||
# Case 3: Parameters with framework components
|
||||
{
|
||||
"params": {
|
||||
"temperature": 0.6,
|
||||
"model_version": "v2.0",
|
||||
"tools": [{"function": {"name": "test"}}],
|
||||
"frequency_penalty": 0.1
|
||||
},
|
||||
"expected_in_model": {"temperature", "frequency_penalty"},
|
||||
"expected_excluded": {"model_version", "tools"}
|
||||
}
|
||||
]
|
||||
|
||||
for i, test_case in enumerate(test_cases):
|
||||
filtered_params = {
|
||||
k: v for k, v in test_case["params"].items()
|
||||
if k not in {"tools", "model_version", "deployment_url"}
|
||||
}
|
||||
|
||||
for expected_param in test_case["expected_in_model"]:
|
||||
assert expected_param in filtered_params, f"Case {i + 1}: {expected_param} should be in model params"
|
||||
|
||||
for excluded_param in test_case["expected_excluded"]:
|
||||
assert excluded_param not in filtered_params, f"Case {i + 1}: {excluded_param} should be excluded from model params"
|
||||
|
||||
result = mock_config.transform_request(
|
||||
model, messages, test_case["params"], {}, {}
|
||||
)
|
||||
if result and "config" in result:
|
||||
model_params = result["config"]["modules"]["prompt_templating"]["model"]["params"]
|
||||
|
||||
for excluded_param in test_case["expected_excluded"]:
|
||||
assert excluded_param not in model_params, (
|
||||
f"Case {i + 1}: {excluded_param} should not be in actual model params"
|
||||
)
|
||||
|
||||
def test_config_transform_with_response_format_json_object(self, mock_config):
|
||||
expected_dict = {'config':
|
||||
{'modules':
|
||||
{'prompt_templating':
|
||||
{'prompt':
|
||||
{'template':
|
||||
[{'role': 'user', 'content': 'First man on the moon, answer in json'}],
|
||||
'response_format': {'type': 'json_object'}},
|
||||
'model': {'name': 'gpt-4o', 'params': {}, 'version': 'latest'}
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{'role': 'user', 'content': 'First man on the moon, answer in json'}],
|
||||
optional_params={'response_format': {'type': 'json_object'},
|
||||
'deployment_url': "shouldn't be in results"},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
assert config == expected_dict
|
||||
|
||||
def test_config_transform_with_response_format_json_schema(self, mock_config):
|
||||
|
||||
expected_response_format = {
|
||||
'type': 'json_schema',
|
||||
'json_schema': {
|
||||
'description': 'Schema for person information',
|
||||
'name': 'person_info',
|
||||
'schema': {
|
||||
'type': 'object',
|
||||
'properties': {
|
||||
'name': {
|
||||
'type': 'string',
|
||||
'description': "The person's full name"
|
||||
},
|
||||
'age': {
|
||||
'type': 'integer',
|
||||
'description': "The person's age in years"
|
||||
},
|
||||
'occupation': {
|
||||
'type': 'string',
|
||||
'description': "The person's job title"
|
||||
}
|
||||
},
|
||||
'required': ['name', 'age', 'occupation'],
|
||||
'additionalProperties': False
|
||||
},
|
||||
'strict': True
|
||||
}
|
||||
}
|
||||
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{'role': 'user', 'content': 'First man on the moon, answer in json'}],
|
||||
optional_params={'response_format': expected_response_format,
|
||||
'deployment_url': "shouldn't be in results"},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
assert config["config"]["modules"]["prompt_templating"]["prompt"]["response_format"] == expected_response_format
|
||||
assert len(config["config"]["modules"]["prompt_templating"]["model"]["params"]) == 0
|
||||
|
||||
def test_config_transform_with_stream(self, mock_config):
|
||||
expected_dict = {
|
||||
'config': {
|
||||
'modules': {
|
||||
'prompt_templating': {
|
||||
'prompt': {
|
||||
'template': [{'role': 'user', 'content': 'Hello, how are you?'}]
|
||||
},
|
||||
'model': {
|
||||
'name': 'anthropic--claude-4-sonnet',
|
||||
'params': {},
|
||||
'version': 'latest'
|
||||
}
|
||||
}
|
||||
},
|
||||
'stream': {'chunk_size': 10}
|
||||
}
|
||||
}
|
||||
config = mock_config.transform_request(
|
||||
model="anthropic--claude-4-sonnet",
|
||||
messages=[{'content': 'Hello, how are you?', 'role': 'user'}],
|
||||
optional_params={'stream': True,
|
||||
'stream_options': {'chunk_size': 10},
|
||||
'model_version': 'latest',
|
||||
'deployment_url': "shouldn't be in results"},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
|
||||
assert config == expected_dict
|
||||
|
||||
def test_sap_placeholder_defaults(self, mock_config):
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[
|
||||
{"role": "user", "content": "Hello. Answer {{ ?user_query }}"}
|
||||
],
|
||||
optional_params={'deployment_url': "shouldn't be in results",
|
||||
"placeholder_defaults": {"user_query": "default value"}},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
|
||||
assert config["config"]["modules"]["prompt_templating"]["prompt"]["defaults"] == {
|
||||
"user_query": "default value"}
|
||||
assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {}
|
||||
|
||||
def test_sap_placeholder_values(self, mock_config):
|
||||
placeholder_values = {"user_query": "Some text"}
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[
|
||||
{"role": "user", "content": "Hello. Answer {{ ?user_query }}"}
|
||||
],
|
||||
optional_params={'deployment_url': "shouldn't be in results",
|
||||
"placeholder_values": placeholder_values},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
|
||||
assert config["placeholder_values"] == placeholder_values
|
||||
assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {}
|
||||
|
||||
def test_sap_grounding(self, mock_config):
|
||||
grounding_config = {
|
||||
'type': 'document_grounding_service',
|
||||
'config': {
|
||||
'filters': [
|
||||
{'id': 's3-docs',
|
||||
'data_repository_type': 'vector',
|
||||
'search_config': {'max_chunk_count': 2},
|
||||
'data_repositories': ['123456890-test']
|
||||
}
|
||||
],
|
||||
'placeholders': {'input': ['user_query'], 'output': 'grounding_response'},
|
||||
'metadata_params': ['source', 'webUrl', 'title', 'mimeType', 'fileSuffix']
|
||||
}
|
||||
}
|
||||
placeholder_values = {"user_query": "Some text"}
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[
|
||||
{"role": "user", "content": "Hello. Answer {{ ?user_query }} using context: {{ ?grounding_response }}"}
|
||||
],
|
||||
optional_params={'deployment_url': "shouldn't be in results",
|
||||
"grounding": grounding_config,
|
||||
"placeholder_values": placeholder_values},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
assert config["placeholder_values"] == placeholder_values
|
||||
modules = config["config"]["modules"]
|
||||
assert modules["grounding"]["type"] == "document_grounding_service"
|
||||
assert modules["grounding"]["config"]["placeholders"]["output"] == "grounding_response"
|
||||
assert modules["grounding"]["config"]["filters"][0]["data_repository_type"] == "vector"
|
||||
assert modules["prompt_templating"]["model"]["params"] == {}
|
||||
|
||||
def test_grounding_search_config_rejects_both_count_fields(self, mock_config):
|
||||
with pytest.raises(ValidationError):
|
||||
mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hi"}],
|
||||
optional_params={
|
||||
"grounding": {
|
||||
"type": "document_grounding_service",
|
||||
"config": {
|
||||
"filters": [{"data_repository_type": "vector",
|
||||
"search_config": {"max_chunk_count": 2,
|
||||
"max_document_count": 5}}],
|
||||
"placeholders": {"input": ["q"], "output": "r"},
|
||||
}
|
||||
}
|
||||
},
|
||||
litellm_params={}, headers={}
|
||||
)
|
||||
|
||||
def test_sap_filtering(self, mock_config):
|
||||
filtering_config_azure = {
|
||||
'input':
|
||||
{
|
||||
'filters':
|
||||
[
|
||||
{'type': 'azure_content_safety',
|
||||
'config':
|
||||
{'hate': 0,
|
||||
'sexual': 0,
|
||||
'violence': 0,
|
||||
'self_harm': 0
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
'output':
|
||||
{
|
||||
'filters':
|
||||
[
|
||||
{'type': 'azure_content_safety',
|
||||
'config': {'hate': 0,
|
||||
'sexual': 0,
|
||||
'violence': 0,
|
||||
'self_harm': 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
filtering_config_llama = {
|
||||
'input':
|
||||
{
|
||||
'filters':
|
||||
[
|
||||
{
|
||||
'type': 'llama_guard_3_8b',
|
||||
'config': {'hate': True,
|
||||
"elections": True}
|
||||
}
|
||||
]
|
||||
},
|
||||
'output':
|
||||
{
|
||||
'filters':
|
||||
[
|
||||
{
|
||||
'type': 'llama_guard_3_8b',
|
||||
'config': {'hate': True, "elections": True}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello."}],
|
||||
optional_params={'deployment_url': "shouldn't be in results",
|
||||
"filtering": filtering_config_azure},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
assert config["config"]["modules"]["filtering"] == filtering_config_azure
|
||||
assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {}
|
||||
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello."}],
|
||||
optional_params={'deployment_url': "shouldn't be in results",
|
||||
"filtering": filtering_config_llama},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
assert config["config"]["modules"]["filtering"] == filtering_config_llama
|
||||
assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {}
|
||||
|
||||
def test_filtering_config_requires_at_least_one_property(self, mock_config):
|
||||
with pytest.raises(ValidationError) as exc_info:
|
||||
mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello"}],
|
||||
optional_params={
|
||||
"filtering": {}
|
||||
},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
|
||||
assert "For using SAP Filtering Module you must provide at least one property" in str(exc_info.value)
|
||||
|
||||
|
||||
def test_sap_masking(self, mock_config):
|
||||
masking_config = {
|
||||
'providers':
|
||||
[
|
||||
{
|
||||
'type': 'sap_data_privacy_integration',
|
||||
'method': 'anonymization',
|
||||
'entities': [
|
||||
{'type': 'profile-address'},
|
||||
{'type': 'profile-email'},
|
||||
{'type': 'profile-phone'},
|
||||
{'type': 'profile-person'},
|
||||
{'type': 'profile-location'}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello."}],
|
||||
optional_params={'deployment_url': "shouldn't be in results",
|
||||
"masking": masking_config},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
assert config["config"]["modules"]["masking"] == masking_config
|
||||
assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {}
|
||||
|
||||
def test_masking_config_requires_exactly_one_provider_list(self, mock_config):
|
||||
masking_config = {
|
||||
'providers':
|
||||
[
|
||||
{
|
||||
'type': 'sap_data_privacy_integration',
|
||||
'method': 'anonymization',
|
||||
'entities': [
|
||||
{'type': 'profile-address'},
|
||||
{'type': 'profile-email'},
|
||||
{'type': 'profile-phone'},
|
||||
{'type': 'profile-person'},
|
||||
{'type': 'profile-location'}
|
||||
]
|
||||
}
|
||||
],
|
||||
'masking_providers':
|
||||
[
|
||||
{
|
||||
'type': 'sap_data_privacy_integration',
|
||||
'method': 'anonymization',
|
||||
'entities': [
|
||||
{'type': 'profile-address'}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValidationError) as exc_info:
|
||||
mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello"}],
|
||||
optional_params={
|
||||
"masking": masking_config
|
||||
},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
|
||||
assert "must set exactly one of: 'providers' or 'masking_providers'" in str(exc_info.value)
|
||||
|
||||
def test_masking_providers_deprecated_emits_warning(self, mock_config):
|
||||
masking_config = {
|
||||
'masking_providers':
|
||||
[
|
||||
{
|
||||
'type': 'sap_data_privacy_integration',
|
||||
'method': 'anonymization',
|
||||
'entities': [
|
||||
{'type': 'profile-address'}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
with warnings.catch_warnings(record=True) as w:
|
||||
warnings.simplefilter("always")
|
||||
mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hi"}],
|
||||
optional_params={"masking": masking_config},
|
||||
litellm_params={},
|
||||
headers={},
|
||||
)
|
||||
assert any(
|
||||
issubclass(warning.category, DeprecationWarning)
|
||||
and "masking_providers" in str(warning.message)
|
||||
for warning in w
|
||||
), "Expected DeprecationWarning for 'masking_providers'"
|
||||
|
||||
def test_sap_translation(self, mock_config):
|
||||
translation_config = {
|
||||
'input':
|
||||
{'type': 'sap_document_translation',
|
||||
'config':
|
||||
{'source_language': 'en-US',
|
||||
'target_language': 'de-DE'}
|
||||
},
|
||||
'output':
|
||||
{'type': 'sap_document_translation',
|
||||
'config':
|
||||
{'source_language': 'de-DE',
|
||||
'target_language': 'fr-FR'}
|
||||
}
|
||||
}
|
||||
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello."}],
|
||||
optional_params={'deployment_url': "shouldn't be in results",
|
||||
"translation": translation_config},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
assert config["config"]["modules"]["translation"] == translation_config
|
||||
assert config["config"]["modules"]["prompt_templating"]["model"]["params"] == {}
|
||||
|
||||
def test_translation_config_requires_at_least_one_property(self, mock_config):
|
||||
with pytest.raises(ValidationError) as exc_info:
|
||||
mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello"}],
|
||||
optional_params={
|
||||
"translation": {}
|
||||
},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
|
||||
assert "TranslationModuleConfig requires at least one of 'input' or 'output'" in str(exc_info.value)
|
||||
|
||||
def test_sap_multiple_modules(self, mock_config):
|
||||
translation_config = {
|
||||
'input':
|
||||
{'type': 'sap_document_translation',
|
||||
'config':
|
||||
{'source_language': 'en-US',
|
||||
'target_language': 'de-DE'}
|
||||
},
|
||||
'output':
|
||||
{'type': 'sap_document_translation',
|
||||
'config':
|
||||
{'source_language': 'de-DE',
|
||||
'target_language': 'fr-FR'}
|
||||
}
|
||||
}
|
||||
for model in ["sap/gpt-5", "gpt-5"]:
|
||||
config = mock_config.transform_request(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "Hello."}],
|
||||
optional_params={'deployment_url': "shouldn't be in results",
|
||||
"fallback_sap_modules": [{"model": model,
|
||||
"messages": [{"role": "user", "content": "Hello world!"}],
|
||||
"translation": translation_config
|
||||
}]
|
||||
,
|
||||
},
|
||||
litellm_params={},
|
||||
headers={}
|
||||
)
|
||||
assert "translation" not in config["config"]["modules"][0]
|
||||
translation = config["config"]["modules"][1]["translation"]
|
||||
assert translation["input"]["config"]["source_language"] == "en-US"
|
||||
assert translation["input"]["config"]["target_language"] == "de-DE"
|
||||
assert translation["output"]["config"]["target_language"] == "fr-FR"
|
||||
assert config["config"]["modules"][1]["prompt_templating"]["model"]["name"] == "gpt-5"
|
||||
assert config["config"]["modules"][0]["prompt_templating"]["model"]["name"] == "gpt-4o"
|
||||
assert config["config"]["modules"][0]["prompt_templating"]["model"]["params"] == {}
|
||||
assert config["config"]["modules"][1]["prompt_templating"]["prompt"]["template"][0]["content"] == "Hello world!"
|
||||
assert config["config"]["modules"][0]["prompt_templating"]["prompt"]["template"][0]["content"] == "Hello."
|
||||
assert config["config"]["modules"][1]["translation"]["input"]["type"] == "sap_document_translation"
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
from unittest.mock import patch, PropertyMock
|
||||
|
||||
import pytest
|
||||
|
||||
from litellm.llms.sap.embed.transformation import GenAIHubEmbeddingConfig
|
||||
|
||||
@pytest.fixture
|
||||
def fake_token_creator():
|
||||
return (lambda: "Bearer FAKE_TOKEN", "https://api.ai.moke-sap.com", "fake-group")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_deployment_url():
|
||||
return "https://api.ai.moke-sap.com/v2/inference/deployments/mokeid"
|
||||
|
||||
def test_basic_config_transform(fake_token_creator, fake_deployment_url):
|
||||
expected_dict = {
|
||||
'config': {
|
||||
'modules': {
|
||||
'embeddings': {
|
||||
'model': {
|
||||
'name': 'text-embedding-3-small',
|
||||
'version': 'latest',
|
||||
'params': {}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
'input': {
|
||||
'text': 'Hi'
|
||||
}
|
||||
}
|
||||
with patch(
|
||||
"litellm.llms.sap.embed.transformation.GenAIHubEmbeddingConfig.deployment_url",
|
||||
new_callable=PropertyMock,
|
||||
return_value=fake_deployment_url,
|
||||
), patch(
|
||||
"litellm.llms.sap.embed.transformation.get_token_creator",
|
||||
return_value=fake_token_creator,
|
||||
):
|
||||
body = GenAIHubEmbeddingConfig().transform_embedding_request(
|
||||
model="text-embedding-3-small",
|
||||
input="Hi",
|
||||
optional_params={},
|
||||
headers={}
|
||||
)
|
||||
assert body == expected_dict
|
||||
|
||||
def test_model_params(fake_token_creator, fake_deployment_url):
|
||||
with patch(
|
||||
"litellm.llms.sap.embed.transformation.GenAIHubEmbeddingConfig.deployment_url",
|
||||
new_callable=PropertyMock,
|
||||
return_value=fake_deployment_url,
|
||||
), patch(
|
||||
"litellm.llms.sap.embed.transformation.get_token_creator",
|
||||
return_value=fake_token_creator,
|
||||
):
|
||||
body = GenAIHubEmbeddingConfig().transform_embedding_request(
|
||||
model="text-embedding-3-small",
|
||||
input="Hi",
|
||||
optional_params={"parameters": {"truncate": "END"}},
|
||||
headers={}
|
||||
)
|
||||
assert body["config"]["modules"]["embeddings"]["model"]["params"] == {"truncate": "END"}
|
||||
|
||||
def test_embed_with_masking(fake_token_creator, fake_deployment_url):
|
||||
masking_config = {
|
||||
'providers':
|
||||
[
|
||||
{
|
||||
'type': 'sap_data_privacy_integration',
|
||||
'method': 'anonymization',
|
||||
'entities': [
|
||||
{'type': 'profile-address'},
|
||||
{'type': 'profile-phone'},
|
||||
{'type': 'profile-person'},
|
||||
{'type': 'profile-location'}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
with patch(
|
||||
"litellm.llms.sap.embed.transformation.GenAIHubEmbeddingConfig.deployment_url",
|
||||
new_callable=PropertyMock,
|
||||
return_value=fake_deployment_url,
|
||||
), patch(
|
||||
"litellm.llms.sap.embed.transformation.get_token_creator",
|
||||
return_value=fake_token_creator,
|
||||
):
|
||||
body = GenAIHubEmbeddingConfig().transform_embedding_request(
|
||||
model="text-embedding-3-small",
|
||||
input="Hi",
|
||||
optional_params={"parameters": {"truncate": "END"},
|
||||
"masking": masking_config},
|
||||
headers={}
|
||||
)
|
||||
assert body["config"]["modules"]["masking"] == masking_config
|
||||
142
tests/test_litellm/llms/sap/test_sap_fetch_creds.py
Normal file
142
tests/test_litellm/llms/sap/test_sap_fetch_creds.py
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
import json
|
||||
import pytest
|
||||
import litellm.llms.sap.credentials as sap_credentials
|
||||
|
||||
mock_sap_service_key_dict = {
|
||||
"serviceurls":
|
||||
{
|
||||
"AI_API_URL":"https://testurl.hana.ondemand.com/"
|
||||
},
|
||||
"clientid":"mockclientid",
|
||||
"clientsecret":"mockclientsecret",
|
||||
"url":"https://test.sap.hana.ondemand.com/"
|
||||
}
|
||||
|
||||
mock_wrapped_sap_service_key_dict = {
|
||||
"credentials": {
|
||||
"serviceurls":
|
||||
{
|
||||
"AI_API_URL":"https://testurl.hana.ondemand.com/"
|
||||
},
|
||||
"clientid":"mockclientid",
|
||||
"clientsecret":"mockclientsecret",
|
||||
"url":"https://test.sap.hana.ondemand.com/"
|
||||
}
|
||||
}
|
||||
|
||||
expected_creds = {'client_id': "mockclientid",
|
||||
'client_secret': "mockclientsecret",
|
||||
'auth_url': 'https://test.sap.hana.ondemand.com/oauth/token',
|
||||
'base_url': 'https://testurl.hana.ondemand.com/v2',
|
||||
'resource_group': 'default'}
|
||||
|
||||
mock_sap_vcap_service_key_dict = {
|
||||
'aicore': [{
|
||||
'label': 'aicore',
|
||||
'name': 'aicore-instance',
|
||||
'instance_guid': '53ad5b47-a49a-4fec-9f0b-cd921c00b828',
|
||||
'credentials': {
|
||||
'serviceurls': {
|
||||
'AI_API_URL': 'vcap-api-url'
|
||||
},
|
||||
'url': 'vcap-auth-url',
|
||||
'clientid': 'vcap-clientid',
|
||||
'clientsecret': 'vcap-clientsecret'
|
||||
}
|
||||
}]
|
||||
}
|
||||
def _prep_env(monkeypatch):
|
||||
for var in ("AICORE_CLIENT_ID", "AICORE_CLIENT_SECRET", "AICORE_AUTH_URL", "AICORE_RESOURCE_GROUP",
|
||||
"AICORE_BASE_URL", "AICORE_CERT_URL", "AICORE_SERVICE_KEY", "VCAP_SERVICES"):
|
||||
monkeypatch.delenv(var, raising=False)
|
||||
monkeypatch.setenv("AICORE_HOME", 'notexist')
|
||||
monkeypatch.setattr('litellm.sap_service_key', None)
|
||||
|
||||
def test_sap_fetch_creds_from_env_service_key(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
monkeypatch.setenv("AICORE_SERVICE_KEY", json.dumps(mock_sap_service_key_dict))
|
||||
creds = sap_credentials.fetch_credentials()
|
||||
assert creds == expected_creds
|
||||
|
||||
def test_sap_fetch_creds_from_env_wrapped_service_key(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
monkeypatch.setenv("AICORE_SERVICE_KEY", json.dumps(mock_wrapped_sap_service_key_dict))
|
||||
creds = sap_credentials.fetch_credentials()
|
||||
assert creds == expected_creds
|
||||
|
||||
def test_sap_fetch_creds_from_arg_service_key(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
creds = sap_credentials.fetch_credentials(service_key=json.dumps(mock_sap_service_key_dict))
|
||||
assert creds == expected_creds
|
||||
|
||||
def test_fetch_creds_from_env_vcap_service(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
monkeypatch.setenv("VCAP_SERVICES", json.dumps(mock_sap_vcap_service_key_dict))
|
||||
creds = sap_credentials.fetch_credentials()
|
||||
assert creds['client_id'] == "vcap-clientid"
|
||||
assert creds['client_secret'] == "vcap-clientsecret"
|
||||
assert creds['auth_url'] == "vcap-auth-url/oauth/token"
|
||||
assert creds['base_url'] == "vcap-api-url/v2"
|
||||
assert creds['resource_group'] == "default"
|
||||
|
||||
def test_fetch_creds_from_env(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
monkeypatch.setenv("AICORE_CLIENT_ID", "env-client-id")
|
||||
monkeypatch.setenv("AICORE_CLIENT_SECRET", "env-client-secret")
|
||||
monkeypatch.setenv("AICORE_AUTH_URL", "env-auth-url")
|
||||
monkeypatch.setenv("AICORE_BASE_URL", "env-base-url")
|
||||
monkeypatch.setenv("AICORE_RESOURCE_GROUP", "env-resource-group")
|
||||
|
||||
creds = sap_credentials.fetch_credentials()
|
||||
|
||||
assert creds['client_id'] == "env-client-id"
|
||||
assert creds['client_secret'] == "env-client-secret"
|
||||
assert creds['auth_url'] == "env-auth-url/oauth/token"
|
||||
assert creds['base_url'] == "env-base-url/v2"
|
||||
assert creds['resource_group'] == "env-resource-group"
|
||||
|
||||
def test_creds_priority_order(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
monkeypatch.setenv("AICORE_CLIENT_ID", "env-client-id")
|
||||
monkeypatch.setenv("AICORE_CLIENT_SECRET", "env-client-secret")
|
||||
monkeypatch.setenv("AICORE_AUTH_URL", "env-auth-url")
|
||||
monkeypatch.setenv("AICORE_BASE_URL", "env-base-url")
|
||||
monkeypatch.setenv("AICORE_RESOURCE_GROUP", "env-resource-group")
|
||||
creds = sap_credentials.fetch_credentials(service_key=json.dumps(mock_sap_service_key_dict))
|
||||
assert creds['client_id'] == "mockclientid"
|
||||
assert creds['resource_group'] == "env-resource-group"
|
||||
|
||||
def test_no_credentials_configured(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
with pytest.raises(ValueError, match="No credentials found in any source"):
|
||||
sap_credentials.fetch_credentials()
|
||||
|
||||
|
||||
def test_partial_credentials_missing_auth_url(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
|
||||
# Set only client_id and base_url, missing auth_url
|
||||
monkeypatch.setenv("AICORE_CLIENT_ID", "test-client-id")
|
||||
monkeypatch.setenv("AICORE_BASE_URL", "test-base-url")
|
||||
|
||||
# fetch_credentials should succeed (it returns whatever it finds)
|
||||
creds = sap_credentials.fetch_credentials()
|
||||
creds.pop('resource_group')
|
||||
|
||||
with pytest.raises(ValueError, match="SAP AI Core credentials not found"):
|
||||
sap_credentials.validate_credentials(**creds)
|
||||
|
||||
def test_credentials_without_authentication_mode(monkeypatch):
|
||||
_prep_env(monkeypatch)
|
||||
|
||||
# Set all required fields but no authentication mode (no client_secret, no certs)
|
||||
monkeypatch.setenv("AICORE_CLIENT_ID", "test-client-id")
|
||||
monkeypatch.setenv("AICORE_AUTH_URL", "test-auth-url")
|
||||
monkeypatch.setenv("AICORE_BASE_URL", "test-base-url")
|
||||
|
||||
creds = sap_credentials.fetch_credentials()
|
||||
creds.pop('resource_group')
|
||||
|
||||
# validate_credentials should raise because no authentication mode is provided
|
||||
with pytest.raises(ValueError, match="SAP AI Core credentials are incomplete"):
|
||||
sap_credentials.validate_credentials(**creds)
|
||||
|
|
@ -127,6 +127,24 @@ def test_vertex_ai_includes_labels():
|
|||
assert result["labels"] == {"project": "test", "team": "ai"}
|
||||
|
||||
|
||||
def test_service_tier_forwarded_to_vertex_ai():
|
||||
"""Test that service_tier in optional_params is mapped to serviceTier in request body."""
|
||||
messages = [{"role": "user", "content": "test"}]
|
||||
optional_params = {"service_tier": "flex"}
|
||||
litellm_params = {}
|
||||
|
||||
result = _transform_request_body(
|
||||
messages=messages,
|
||||
model="gemini-2.5-pro",
|
||||
optional_params=optional_params,
|
||||
custom_llm_provider="vertex_ai",
|
||||
litellm_params=litellm_params,
|
||||
cached_content=None,
|
||||
)
|
||||
|
||||
assert "serviceTier" in result
|
||||
assert result["serviceTier"] == "flex"
|
||||
|
||||
|
||||
def test_extra_body_cache_not_forwarded_to_vertex_ai():
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -3504,6 +3504,73 @@ def test_vertex_ai_traffic_type_preserved_in_hidden_params_non_streaming():
|
|||
assert result._hidden_params["provider_specific_fields"]["traffic_type"] == "PROVISIONED_THROUGHPUT"
|
||||
|
||||
|
||||
def test_vertex_ai_service_tier_streaming():
|
||||
"""Test service_tier is preserved in model_response from headers for streaming."""
|
||||
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
|
||||
ModelResponseIterator,
|
||||
)
|
||||
|
||||
chunk = {
|
||||
"candidates": [{"content": {"parts": [{"text": "Hello"}]}}],
|
||||
}
|
||||
|
||||
iterator = ModelResponseIterator(
|
||||
streaming_response=[],
|
||||
sync_stream=True,
|
||||
logging_obj=MagicMock(),
|
||||
response_headers={"x-gemini-service-tier": "FLEX"},
|
||||
)
|
||||
# Undefined when usageMetadata is missing
|
||||
result = iterator.chunk_parser(chunk)
|
||||
|
||||
# But definitely set when usageMetadata is present
|
||||
chunk_with_usage = {
|
||||
"candidates": [{"content": {"parts": [{"text": "hi"}]}}],
|
||||
"usageMetadata": {"promptTokenCount": 1, "candidatesTokenCount": 1, "totalTokenCount": 2}
|
||||
}
|
||||
result_with_usage = iterator.chunk_parser(chunk_with_usage)
|
||||
assert result_with_usage.service_tier == "flex"
|
||||
|
||||
|
||||
def test_vertex_ai_service_tier_non_streaming():
|
||||
"""Test service_tier is preserved in model_response from headers for non-streaming."""
|
||||
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
|
||||
VertexGeminiConfig,
|
||||
)
|
||||
|
||||
completion_response = {
|
||||
"candidates": [
|
||||
{
|
||||
"content": {"parts": [{"text": "Hello"}], "role": "model"},
|
||||
"finishReason": "STOP",
|
||||
}
|
||||
],
|
||||
"usageMetadata": {
|
||||
"promptTokenCount": 50,
|
||||
"candidatesTokenCount": 100,
|
||||
"totalTokenCount": 150,
|
||||
},
|
||||
}
|
||||
|
||||
raw_response = MagicMock()
|
||||
raw_response.json.return_value = completion_response
|
||||
raw_response.headers = {"x-gemini-service-tier": "FLEX"}
|
||||
|
||||
result = VertexGeminiConfig().transform_response(
|
||||
model="gemini-pro",
|
||||
raw_response=raw_response,
|
||||
model_response=ModelResponse(),
|
||||
logging_obj=MagicMock(),
|
||||
request_data={},
|
||||
messages=[],
|
||||
optional_params={},
|
||||
litellm_params={},
|
||||
encoding=None,
|
||||
)
|
||||
|
||||
assert result.service_tier == "flex"
|
||||
|
||||
|
||||
def test_vertex_ai_traffic_type_surfaced_in_responses_api():
|
||||
"""Test trafficType is surfaced as provider_specific_fields in ResponsesAPIResponse."""
|
||||
from litellm.responses.litellm_completion_transformation.transformation import (
|
||||
|
|
@ -3609,6 +3676,54 @@ def test_vertex_ai_web_search_options_in_map_openai_params():
|
|||
assert "web_search_options" not in optional_params, "web_search_options should be removed after transformation"
|
||||
|
||||
|
||||
def test_vertex_ai_service_tier_in_map_openai_params():
|
||||
"""Test that service_tier is correctly mapped to optional_params."""
|
||||
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
|
||||
VertexGeminiConfig,
|
||||
)
|
||||
|
||||
v = VertexGeminiConfig()
|
||||
|
||||
# Test pass-through
|
||||
optional_params = {}
|
||||
non_default_params = {"service_tier": "FLEX"}
|
||||
|
||||
result = v.map_openai_params(
|
||||
non_default_params=non_default_params,
|
||||
optional_params=optional_params,
|
||||
model="gemini-3-pro-preview",
|
||||
drop_params=True,
|
||||
)
|
||||
|
||||
assert result["service_tier"] == "flex"
|
||||
|
||||
# Test auto -> priority
|
||||
optional_params_auto = {}
|
||||
non_default_params_auto = {"service_tier": "auto"}
|
||||
|
||||
result_auto = v.map_openai_params(
|
||||
non_default_params=non_default_params_auto,
|
||||
optional_params=optional_params_auto,
|
||||
model="gemini-3-pro-preview",
|
||||
drop_params=True,
|
||||
)
|
||||
|
||||
assert result_auto["service_tier"] == "priority"
|
||||
|
||||
# Test AUTO (uppercase) -> priority
|
||||
optional_params_auto_upper = {}
|
||||
non_default_params_auto_upper = {"service_tier": "AUTO"}
|
||||
|
||||
result_auto_upper = v.map_openai_params(
|
||||
non_default_params=non_default_params_auto_upper,
|
||||
optional_params=optional_params_auto_upper,
|
||||
model="gemini-3-pro-preview",
|
||||
drop_params=True,
|
||||
)
|
||||
|
||||
assert result_auto_upper["service_tier"] == "priority"
|
||||
|
||||
|
||||
def test_vertex_ai_usage_metadata_with_video_tokens_in_prompt():
|
||||
"""Test promptTokensDetails with VIDEO modality for video inputs.
|
||||
|
||||
|
|
|
|||
|
|
@ -1050,6 +1050,85 @@ class TestVertexBase:
|
|||
mock_creds.with_scopes.assert_called_once_with(scopes)
|
||||
assert result == "scoped_creds"
|
||||
|
||||
def test_credentials_from_pluggable_implementation(self):
|
||||
"""Test _credentials_from_pluggable dispatches to pluggable.Credentials"""
|
||||
vertex_base = VertexBase()
|
||||
json_obj = {
|
||||
"type": "external_account",
|
||||
"credential_source": {
|
||||
"executable": {"command": "/path/to/executable", "timeout_millis": 5000}
|
||||
},
|
||||
}
|
||||
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
|
||||
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.requires_scopes = True
|
||||
mock_creds.with_scopes.return_value = "scoped_creds"
|
||||
|
||||
with patch("google.auth.pluggable.Credentials") as MockCredentials:
|
||||
MockCredentials.from_info.return_value = mock_creds
|
||||
|
||||
result = vertex_base._credentials_from_pluggable(json_obj, scopes)
|
||||
|
||||
MockCredentials.from_info.assert_called_once_with(json_obj)
|
||||
mock_creds.with_scopes.assert_called_once_with(scopes)
|
||||
assert result == "scoped_creds"
|
||||
|
||||
def test_credentials_from_pluggable_no_scopes_needed(self):
|
||||
"""Test _credentials_from_pluggable when scopes are not needed"""
|
||||
vertex_base = VertexBase()
|
||||
json_obj = {
|
||||
"type": "external_account",
|
||||
"credential_source": {
|
||||
"executable": {"command": "/path/to/executable"}
|
||||
},
|
||||
}
|
||||
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
|
||||
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.requires_scopes = False
|
||||
|
||||
with patch("google.auth.pluggable.Credentials") as MockCredentials:
|
||||
MockCredentials.from_info.return_value = mock_creds
|
||||
|
||||
result = vertex_base._credentials_from_pluggable(json_obj, scopes)
|
||||
|
||||
MockCredentials.from_info.assert_called_once_with(json_obj)
|
||||
mock_creds.with_scopes.assert_not_called()
|
||||
assert result == mock_creds
|
||||
|
||||
def test_load_auth_dispatches_to_pluggable_for_executable(self):
|
||||
"""Test that load_auth routes executable credential_source to _credentials_from_pluggable"""
|
||||
vertex_base = VertexBase()
|
||||
json_obj = {
|
||||
"type": "external_account",
|
||||
"credential_source": {
|
||||
"executable": {"command": "/path/to/executable", "timeout_millis": 5000}
|
||||
},
|
||||
}
|
||||
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.project_id = "test-project"
|
||||
|
||||
with patch.object(
|
||||
vertex_base, "_credentials_from_pluggable", return_value=mock_creds
|
||||
) as mock_pluggable, patch.object(
|
||||
vertex_base, "_credentials_from_identity_pool"
|
||||
) as mock_identity_pool, patch.object(
|
||||
vertex_base, "refresh_auth"
|
||||
):
|
||||
creds, project_id = vertex_base.load_auth(
|
||||
credentials=json.dumps(json_obj), project_id=None
|
||||
)
|
||||
|
||||
mock_pluggable.assert_called_once_with(
|
||||
json_obj,
|
||||
scopes=["https://www.googleapis.com/auth/cloud-platform"],
|
||||
)
|
||||
mock_identity_pool.assert_not_called()
|
||||
assert creds == mock_creds
|
||||
assert project_id == "test-project"
|
||||
|
||||
def test_extract_aws_params(self):
|
||||
"""Test _extract_aws_params: extraction, empty case, and unrecognized keys."""
|
||||
# Case 1: Extracts recognized aws_* keys, ignores GCP-standard fields
|
||||
|
|
|
|||
|
|
@ -177,6 +177,72 @@ def test_json_formatter_parses_embedded_python_dict_repr():
|
|||
assert obj["model_info"]["db_model"] is False
|
||||
|
||||
|
||||
def test_json_formatter_includes_component_field():
|
||||
"""
|
||||
Test that JsonFormatter always emits a 'component' field equal to the logger name.
|
||||
This allows filtering by component (e.g. "LiteLLM Proxy") in Datadog / third-party log services.
|
||||
"""
|
||||
formatter = JsonFormatter()
|
||||
for logger_name in ("LiteLLM Proxy", "LiteLLM Router", "LiteLLM"):
|
||||
record = logging.LogRecord(
|
||||
name=logger_name,
|
||||
level=logging.ERROR,
|
||||
pathname="proxy_server.py",
|
||||
lineno=42,
|
||||
msg="something went wrong",
|
||||
args=(),
|
||||
exc_info=None,
|
||||
)
|
||||
output = formatter.format(record)
|
||||
obj = json.loads(output)
|
||||
assert obj["component"] == logger_name, (
|
||||
f"Expected component={logger_name!r}, got {obj.get('component')!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_json_formatter_includes_logger_field():
|
||||
"""
|
||||
Test that JsonFormatter always emits a 'logger' field with filename:lineno.
|
||||
This allows pinpointing the exact source of a log line in third-party services.
|
||||
"""
|
||||
formatter = JsonFormatter()
|
||||
record = logging.LogRecord(
|
||||
name="LiteLLM Proxy",
|
||||
level=logging.INFO,
|
||||
pathname="/app/litellm/proxy/proxy_server.py",
|
||||
lineno=123,
|
||||
msg="request received",
|
||||
args=(),
|
||||
exc_info=None,
|
||||
)
|
||||
output = formatter.format(record)
|
||||
obj = json.loads(output)
|
||||
assert obj["logger"] == "proxy_server.py:123", (
|
||||
f"Expected logger='proxy_server.py:123', got {obj['logger']!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_json_formatter_extra_component_not_overwritten():
|
||||
"""
|
||||
User-supplied extra={"component": "..."} must not be silently dropped.
|
||||
"""
|
||||
formatter = JsonFormatter()
|
||||
record = logging.LogRecord(
|
||||
name="LiteLLM Proxy",
|
||||
level=logging.INFO,
|
||||
pathname="proxy_server.py",
|
||||
lineno=1,
|
||||
msg="event",
|
||||
args=(),
|
||||
exc_info=None,
|
||||
)
|
||||
record.component = "auth-service"
|
||||
obj = json.loads(formatter.format(record))
|
||||
assert obj["component"] == "auth-service", (
|
||||
f"User-supplied component was overwritten, got {obj['component']!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_initialize_loggers_with_handler_sets_propagate_false():
|
||||
"""
|
||||
Test that the initialize_loggers_with_handler function sets propagate to False for all loggers
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue