scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-06 08:16:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 79

Merge branch 'main' into litellm_allow_setting_route_controls

Browse source
This commit is contained in:
Ishaan Jaff 2024-09-03 17:00:22 -07:00 • committed by GitHub
commit 115de195f1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
42 changed files with 983 additions and 449 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

99
docs/my-website/docs/batches.md
View file

@ -222,4 +222,101 @@ curl http://0.0.0.0:4000/v1/batches?limit=2 \
-H "Content-Type: application/json"
```
### [👉 Health Check Azure Batch models](./proxy/health.md#batch-models-azure-only)
### [👉 Health Check Azure Batch models](./proxy/health.md#batch-models-azure-only)
### [BETA] Loadbalance Multiple Azure Deployments
In your config.yaml, set `enable_loadbalancing_on_batch_endpoints: true`
```yaml
model_list:
- model_name: "batch-gpt-4o-mini"
litellm_params:
model: "azure/gpt-4o-mini"
api_key: os.environ/AZURE_API_KEY
api_base: os.environ/AZURE_API_BASE
model_info:
mode: batch
litellm_settings:
enable_loadbalancing_on_batch_endpoints: true # 👈 KEY CHANGE
```
Note: This works on `{PROXY_BASE_URL}/v1/files` and `{PROXY_BASE_URL}/v1/batches`.
Note: Response is in the OpenAI-format.
1. Upload a file
Just set `model: batch-gpt-4o-mini` in your .jsonl.
```bash
curl http://localhost:4000/v1/files \
-H "Authorization: Bearer sk-1234" \
-F purpose="batch" \
-F file="@mydata.jsonl"
```
**Example File**
Note: `model` should be your azure deployment name.
```json
{"custom_id": "task-0", "method": "POST", "url": "/chat/completions", "body": {"model": "batch-gpt-4o-mini", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "When was Microsoft founded?"}]}}
{"custom_id": "task-1", "method": "POST", "url": "/chat/completions", "body": {"model": "batch-gpt-4o-mini", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "When was the first XBOX released?"}]}}
{"custom_id": "task-2", "method": "POST", "url": "/chat/completions", "body": {"model": "batch-gpt-4o-mini", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "What is Altair Basic?"}]}}
```
Expected Response (OpenAI-compatible)
```bash
{"id":"file-f0be81f654454113a922da60acb0eea6",...}
```
2. Create a batch
```bash
curl http://0.0.0.0:4000/v1/batches \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file-f0be81f654454113a922da60acb0eea6",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"model: "batch-gpt-4o-mini"
}'
```
Expected Response:
```bash
{"id":"batch_94e43f0a-d805-477d-adf9-bbb9c50910ed",...}
```
3. Retrieve a batch
```bash
curl http://0.0.0.0:4000/v1/batches/batch_94e43f0a-d805-477d-adf9-bbb9c50910ed \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "Content-Type: application/json" \
```
Expected Response:
```
{"id":"batch_94e43f0a-d805-477d-adf9-bbb9c50910ed",...}
```
4. List batch
```bash
curl http://0.0.0.0:4000/v1/batches?limit=2 \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "Content-Type: application/json"
```
Expected Response:
```bash
{"data":[{"id":"batch_R3V...}
```

2
docs/my-website/docs/enterprise.md
View file

@ -23,7 +23,7 @@ This covers:
- ✅ [Audit Logs with retention policy](./proxy/enterprise#audit-logs)
- ✅ [JWT-Auth](../docs/proxy/token_auth.md)
- ✅ [Control available public, private routes (Restrict certain endpoints on proxy)](./proxy/enterprise#control-available-public-private-routes)
- ✅ [[BETA] AWS Key Manager v2 - Key Decryption](./proxy/enterprise#beta-aws-key-manager---key-decryption)
- ✅ [**Secret Managers** AWS Key Manager, Google Secret Manager, Azure Key](./secret)
- ✅ IP address‑based access control lists
- ✅ Track Request IP Address
- ✅ [Use LiteLLM keys/authentication on Pass Through Endpoints](./proxy/pass_through#✨-enterprise---use-litellm-keysauthentication-on-pass-through-endpoints)

37
docs/my-website/docs/providers/azure.md
View file

@ -533,3 +533,40 @@ response = litellm.completion(
)
```
### Azure AD Token Refresh
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
token_provider = get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default")
response = completion(
model = "azure/<your deployment name>", # model = azure/<your deployment name>
api_base = "", # azure api base
api_version = "", # azure api version
azure_ad_token_provider=token_provider
messages = [{"role": "user", "content": "good morning"}],
)
```
</TabItem>
<TabItem value="proxy" label="PROXY config.yaml">
```yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: azure/your-deployment-name
api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
litellm_settings:
enable_azure_ad_token_refresh: true # 👈 KEY CHANGE
```
</TabItem>
</Tabs>

17
docs/my-website/docs/providers/bedrock.md
View file

@ -894,12 +894,17 @@ print(response)
## Supported AWS Bedrock Embedding Models
| Model Name | Function Call |
|----------------------|---------------------------------------------|
| Titan Embeddings V2 | `embedding(model="bedrock/amazon.titan-embed-text-v2:0", input=input)` |
| Titan Embeddings - V1 | `embedding(model="bedrock/amazon.titan-embed-text-v1", input=input)` |
| Cohere Embeddings - English | `embedding(model="bedrock/cohere.embed-english-v3", input=input)` |
| Cohere Embeddings - Multilingual | `embedding(model="bedrock/cohere.embed-multilingual-v3", input=input)` |
| Model Name | Usage | Supported Additional OpenAI params |
|----------------------|---------------------------------------------|-----|
| Titan Embeddings V2 | `embedding(model="bedrock/amazon.titan-embed-text-v2:0", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/amazon_titan_v2_transformation.py#L59) |
| Titan Embeddings - V1 | `embedding(model="bedrock/amazon.titan-embed-text-v1", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/amazon_titan_g1_transformation.py#L53)
| Titan Multimodal Embeddings | `embedding(model="bedrock/amazon.titan-embed-image-v1", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.py#L28) |
| Cohere Embeddings - English | `embedding(model="bedrock/cohere.embed-english-v3", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/cohere_transformation.py#L18)
| Cohere Embeddings - Multilingual | `embedding(model="bedrock/cohere.embed-multilingual-v3", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/cohere_transformation.py#L18)
### Advanced - [Drop Unsupported Params](https://docs.litellm.ai/docs/completion/drop_params#openai-proxy-usage)
### Advanced - [Pass model/provider-specific Params](https://docs.litellm.ai/docs/completion/provider_specific_params#proxy-usage)
## Image Generation
Use this for stable diffusion on bedrock

1
docs/my-website/docs/proxy/enterprise.md
View file

@ -17,6 +17,7 @@ Features:
- ✅ [Audit Logs with retention policy](#audit-logs)
- ✅ [JWT-Auth](../docs/proxy/token_auth.md)
- ✅ [Control available public, private routes (Restrict certain endpoints on proxy)](#control-available-public-private-routes)
- ✅ [Control available public, private routes](#control-available-public-private-routes)
- ✅ [[BETA] AWS Key Manager v2 - Key Decryption](#beta-aws-key-manager---key-decryption)
- ✅ IP address‑based access control lists
- ✅ Track Request IP Address

6
docs/my-website/docs/proxy/reliability.md
View file

@ -31,7 +31,9 @@ model_list:
api_base: https://openai-france-1234.openai.azure.com/
api_key: <your-azure-api-key>
rpm: 1440
routing_strategy: simple-shuffle # Literal["simple-shuffle", "least-busy", "usage-based-routing","latency-based-routing"], default="simple-shuffle"
router_settings:
routing_strategy: simple-shuffle # Literal["simple-shuffle", "least-busy", "usage-based-routing","latency-based-routing"], default="simple-shuffle"
model_group_alias: {"gpt-4": "gpt-3.5-turbo"} # all requests with `gpt-4` will be routed to models with `gpt-3.5-turbo`
num_retries: 2
timeout: 30 # 30 seconds
@ -85,8 +87,6 @@ print(response)
<TabItem value="Curl" label="Curl Request">
Pass `metadata` as part of the request body
```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \

56
docs/my-website/docs/routing.md
View file

@ -25,6 +25,11 @@ If you want a server to load balance across different LLM APIs, use our [LiteLLM
### Quick Start
Loadbalance across multiple [azure](./providers/azure.md)/[bedrock](./providers/bedrock.md)/[provider](./providers/) deployments. LiteLLM will handle retrying in different regions if a call fails.
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import Router
@ -84,6 +89,57 @@ response = await router.acompletion(model="gpt-4",
print(response)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
:::info
See detailed proxy loadbalancing/fallback docs [here](./proxy/reliability.md)
:::
1. Setup model_list with multiple deployments
```yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: azure/<your-deployment-name>
api_base: <your-azure-endpoint>
api_key: <your-azure-api-key>
- model_name: gpt-3.5-turbo
litellm_params:
model: azure/gpt-turbo-small-ca
api_base: https://my-endpoint-canada-berri992.openai.azure.com/
api_key: <your-azure-api-key>
- model_name: gpt-3.5-turbo
litellm_params:
model: azure/gpt-turbo-large
api_base: https://openai-france-1234.openai.azure.com/
api_key: <your-azure-api-key>
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "Hi there!"}
],
"mock_testing_rate_limit_error": true
}'
```
</TabItem>
</Tabs>
### Available Endpoints
- `router.completion()` - chat completions endpoint to call 100+ LLMs

56
docs/my-website/docs/secret.md
View file

@ -1,9 +1,22 @@
# Secret Manager
LiteLLM supports reading secrets from Azure Key Vault and Infisical
LiteLLM supports reading secrets from Azure Key Vault, Google Secret Manager
- AWS Key Managemenet Service
:::info
✨ **This is an Enterprise Feature**
[Enterprise Pricing](https://www.litellm.ai/#pricing)
[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat)
:::
## Supported Secret Managers
- AWS Key Management Service
- AWS Secret Manager
- [Azure Key Vault](#azure-key-vault)
- [Google Secret Manager](#google-secret-manager)
- Google Key Management Service
- [Infisical Secret Manager](#infisical-secret-manager)
- [.env Files](#env-files)
@ -125,6 +138,45 @@ litellm --config /path/to/config.yaml
[Quick Test Proxy](./proxy/quick_start#using-litellm-proxy---curl-request-openai-package-langchain-langchain-js)
## Google Secret Manager
Support for [Google Secret Manager](https://cloud.google.com/security/products/secret-manager)
1. Save Google Secret Manager details in your environment
```shell
GOOGLE_SECRET_MANAGER_PROJECT_ID="your-project-id-on-gcp" # example: adroit-crow-413218
```
Optional Params
```shell
export GOOGLE_SECRET_MANAGER_REFRESH_INTERVAL = "" # (int) defaults to 86400
export GOOGLE_SECRET_MANAGER_ALWAYS_READ_SECRET_MANAGER = "" # (str) set to "true" if you want to always read from google secret manager without using in memory caching. NOT RECOMMENDED in PROD
```
2. Add to proxy config.yaml
```yaml
model_list:
- model_name: fake-openai-endpoint
litellm_params:
model: openai/fake
api_base: https://exampleopenaiendpoint-production.up.railway.app/
api_key: os.environ/OPENAI_API_KEY # this will be read from Google Secret Manager
general_settings:
key_management_system: "google_secret_manager"
```
You can now test this by starting your proxy:
```bash
litellm --config /path/to/config.yaml
```
[Quick Test Proxy](./proxy/quick_start#using-litellm-proxy---curl-request-openai-package-langchain-langchain-js)
## Google Key Management Service
Use encrypted keys from Google KMS on the proxy

21
docs/my-website/docusaurus.config.js
View file

@ -26,26 +26,33 @@ const config = {
defaultLocale: 'en',
locales: ['en'],
},
plugins: [
themes: [
[
require.resolve("@getcanary/docusaurus-pagefind"),
require.resolve("@getcanary/docusaurus-theme-search-pagefind"),
{
indexOnly: true,
styles: {
"--canary-color-primary-c": 0.1,
"--canary-color-primary-h": 270,
},
pagefind: {
ranking: {
pageLength: 0.9,
// https://pagefind.app/docs/ranking
pageLength: 0.0,
termFrequency: 1.0,
termSimilarity: 1.0,
termSaturation: 1.5,
termSaturation: 2.0,
}
}
},
tabs: [
{ name: "All", pattern: "**/*" },
{ name: "Providers", pattern: "/docs/providers/**" },
{ name: "Proxy", pattern: "/docs/proxy/**" }
],
indexOnly: true,
},
],
],
plugins: [
[
'@docusaurus/plugin-ideal-image',
{

195
docs/my-website/package-lock.json generated
View file

@ -12,8 +12,8 @@
"@docusaurus/plugin-google-gtag": "^2.4.1",
"@docusaurus/plugin-ideal-image": "^2.4.1",
"@docusaurus/preset-classic": "2.4.1",
"@getcanary/docusaurus-pagefind": "^0.0.12",
"@getcanary/web": "^0.0.55",
"@getcanary/docusaurus-theme-search-pagefind": "^0.0.13",
"@getcanary/web": "^0.0.101",
"@mdx-js/react": "^1.6.22",
"clsx": "^1.2.1",
"docusaurus": "^1.14.7",
@ -3621,31 +3621,31 @@
}
},
"node_modules/@floating-ui/core": {
"version": "1.6.5",
"resolved": "https://registry.npmjs.org/@floating-ui/core/-/core-1.6.5.tgz",
"integrity": "sha512-8GrTWmoFhm5BsMZOTHeGD2/0FLKLQQHvO/ZmQga4tKempYRLz8aqJGqXVuQgisnMObq2YZ2SgkwctN1LOOxcqA==",
"version": "1.6.7",
"resolved": "https://registry.npmjs.org/@floating-ui/core/-/core-1.6.7.tgz",
"integrity": "sha512-yDzVT/Lm101nQ5TCVeK65LtdN7Tj4Qpr9RTXJ2vPFLqtLxwOrpoxAHAJI8J3yYWUc40J0BDBheaitK5SJmno2g==",
"dependencies": {
"@floating-ui/utils": "^0.2.5"
"@floating-ui/utils": "^0.2.7"
}
},
"node_modules/@floating-ui/dom": {
"version": "1.6.8",
"resolved": "https://registry.npmjs.org/@floating-ui/dom/-/dom-1.6.8.tgz",
"integrity": "sha512-kx62rP19VZ767Q653wsP1XZCGIirkE09E0QUGNYTM/ttbbQHqcGPdSfWFxUyyNLc/W6aoJRBajOSXhP6GXjC0Q==",
"version": "1.6.10",
"resolved": "https://registry.npmjs.org/@floating-ui/dom/-/dom-1.6.10.tgz",
"integrity": "sha512-fskgCFv8J8OamCmyun8MfjB1Olfn+uZKjOKZ0vhYF3gRmEUXcGOjxWL8bBr7i4kIuPZ2KD2S3EUIOxnjC8kl2A==",
"dependencies": {
"@floating-ui/core": "^1.6.0",
"@floating-ui/utils": "^0.2.5"
"@floating-ui/utils": "^0.2.7"
}
},
"node_modules/@floating-ui/utils": {
"version": "0.2.5",
"resolved": "https://registry.npmjs.org/@floating-ui/utils/-/utils-0.2.5.tgz",
"integrity": "sha512-sTcG+QZ6fdEUObICavU+aB3Mp8HY4n14wYHdxK4fXjPmv3PXZZeY5RaguJmGyeH/CJQhX3fqKUtS4qc1LoHwhQ=="
"version": "0.2.7",
"resolved": "https://registry.npmjs.org/@floating-ui/utils/-/utils-0.2.7.tgz",
"integrity": "sha512-X8R8Oj771YRl/w+c1HqAC1szL8zWQRwFvgDwT129k9ACdBoud/+/rX9V0qiMl6LWUdP9voC2nDVZYPMQQsb6eA=="
},
"node_modules/@getcanary/docusaurus-pagefind": {
"version": "0.0.12",
"resolved": "https://registry.npmjs.org/@getcanary/docusaurus-pagefind/-/docusaurus-pagefind-0.0.12.tgz",
"integrity": "sha512-F0OQ0Lb/GltewDEr0w+BgPbNyYpzAQZ/TtuG5rbtC3PnrOL+9pDMe/Gs0kE8AuY1uEd/YQOKr61rbY/k7kkFig==",
"node_modules/@getcanary/docusaurus-theme-search-pagefind": {
"version": "0.0.13",
"resolved": "https://registry.npmjs.org/@getcanary/docusaurus-theme-search-pagefind/-/docusaurus-theme-search-pagefind-0.0.13.tgz",
"integrity": "sha512-w/Jz4q98cJQQtTAq8KCDI1jAW1HnHpsYpFoXaIjplLt+RoBiUKkCXocxDEI0yfKzRtmkH77dvf6LyJIdqMDWZg==",
"dependencies": {
"cli-progress": "^3.12.0",
"micromatch": "^4.0.7",
@ -3659,17 +3659,61 @@
}
},
"node_modules/@getcanary/web": {
"version": "0.0.55",
"resolved": "https://registry.npmjs.org/@getcanary/web/-/web-0.0.55.tgz",
"integrity": "sha512-DjIhTMeuLZaHT+/h+O6Keg9Gb58frPURpM4lkKrN/wmRMoCnOuly3oXIH2X37YhAoHXi4udDRJ60mtD0UZy0uw==",
"version": "0.0.101",
"resolved": "https://registry.npmjs.org/@getcanary/web/-/web-0.0.101.tgz",
"integrity": "sha512-/GeoSpVKq0w2HmjeF855zOeKE8a6cFcdtAsLHC0/ATJTkrAjAuAaTn5We0WX2w0UGBdzHuc94bYXJCklbTEoCg==",
"dependencies": {
"@floating-ui/dom": "^1.6.8",
"@lit-labs/observers": "^2.0.2",
"@lit/context": "^1.1.2",
"@lit/task": "^1.0.1",
"highlight.js": "^11.10.0",
"@xstate/store": "^2.1.0",
"lit": "^3.1.4",
"marked": "^13.0.2"
"marked": "^14.0.0",
"marked-footnote": "^1.2.2",
"picomatch": "^4.0.2",
"prismjs": "^1.29.0"
}
},
"node_modules/@getcanary/web/node_modules/@xstate/store": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/@xstate/store/-/store-2.3.0.tgz",
"integrity": "sha512-vdIlbI4TxtadYNrGrKezrR0Y0Hid2xYBm5ohk403ZsaU7p6IjCCaacF8t8tp9qUZnYuLYzldBMhiGwBuNzmCoA==",
"peerDependencies": {
"react": "^18.2.0",
"solid-js": "^1.7.6"
},
"peerDependenciesMeta": {
"react": {
"optional": true
},
"solid-js": {
"optional": true
}
}
},
"node_modules/@getcanary/web/node_modules/picomatch": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz",
"integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/@getcanary/web/node_modules/react": {
"version": "18.3.1",
"resolved": "https://registry.npmjs.org/react/-/react-18.3.1.tgz",
"integrity": "sha512-wS+hAgJShR0KhEvPJArfuPVN1+Hz1t0Y6n5jLrGQbkb4urgPE/0Rve+1kMB1v/oWgHgm4WIcV+i7F2pTVj+2iQ==",
"optional": true,
"peer": true,
"dependencies": {
"loose-envify": "^1.1.0"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/@hapi/hoek": {
@ -3770,17 +3814,18 @@
"integrity": "sha512-Vo+PSpZG2/fmgmiNzYK9qWRh8h/CHrwD0mo1h1DzL4yzHNSfWYujGTYsWGreD000gcgmZ7K4Ys6Tx9TxtsKdDw=="
},
"node_modules/@lit-labs/observers": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/@lit-labs/observers/-/observers-2.0.2.tgz",
"integrity": "sha512-eZb5+W9Cb0e/Y5m1DNxBSGTvGB2TAVTGMnTxL/IzFhPQEcZIAHewW1eVBhN8W07A5tirRaAmmF6fGL1V20p3gQ==",
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/@lit-labs/observers/-/observers-2.0.3.tgz",
"integrity": "sha512-CeftEJ2TId9iohDJHLjUXiSBVndqjIBaALjeTt8OmgWLh2dnIzwlj4WtPCiJw15uR1s6D6wyCsw0AoJC5/9QXw==",
"dependencies": {
"@lit/reactive-element": "^1.0.0 || ^2.0.0"
"@lit/reactive-element": "^1.0.0 || ^2.0.0",
"lit-html": "^3.2.0"
}
},
"node_modules/@lit-labs/ssr-dom-shim": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/@lit-labs/ssr-dom-shim/-/ssr-dom-shim-1.2.0.tgz",
"integrity": "sha512-yWJKmpGE6lUURKAaIltoPIE/wrbY3TEkqQt+X0m+7fQNnAv0keydnYvbiJFP1PnMhizmIWRWOG5KLhYyc/xl+g=="
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/@lit-labs/ssr-dom-shim/-/ssr-dom-shim-1.2.1.tgz",
"integrity": "sha512-wx4aBmgeGvFmOKucFKY+8VFJSYZxs9poN3SDNQFF6lT6NrQUnHiPB2PWz2sc4ieEcAaYYzN+1uWahEeTq2aRIQ=="
},
"node_modules/@lit/context": {
"version": "1.1.2",
@ -3994,9 +4039,9 @@
}
},
"node_modules/@pagefind/darwin-arm64": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@pagefind/darwin-arm64/-/darwin-arm64-1.1.0.tgz",
"integrity": "sha512-SLsXNLtSilGZjvqis8sX42fBWsWAVkcDh1oerxwqbac84HbiwxpxOC2jm8hRwcR0Z55HPZPWO77XeRix/8GwTg==",
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/@pagefind/darwin-arm64/-/darwin-arm64-1.1.1.tgz",
"integrity": "sha512-tZ9tysUmQpFs2EqWG2+E1gc+opDAhSyZSsgKmFzhnWfkK02YHZhvL5XJXEZDqYy3s1FAKhwjTg8XDxneuBlDZQ==",
"cpu": [
"arm64"
],
@ -4006,9 +4051,9 @@
]
},
"node_modules/@pagefind/darwin-x64": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@pagefind/darwin-x64/-/darwin-x64-1.1.0.tgz",
"integrity": "sha512-QjQSE/L5oS1C8N8GdljGaWtjCBMgMtfrPAoiCmINTu9Y9dp0ggAyXvF8K7Qg3VyIMYJ6v8vg2PN7Z3b+AaAqUA==",
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/@pagefind/darwin-x64/-/darwin-x64-1.1.1.tgz",
"integrity": "sha512-ChohLQ39dLwaxQv0jIQB/SavP3TM5K5ENfDTqIdzLkmfs3+JlzSDyQKcJFjTHYcCzQOZVeieeGq8PdqvLJxJxQ==",
"cpu": [
"x64"
],
@ -4018,9 +4063,9 @@
]
},
"node_modules/@pagefind/linux-arm64": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@pagefind/linux-arm64/-/linux-arm64-1.1.0.tgz",
"integrity": "sha512-8zjYCa2BtNEL7KnXtysPtBELCyv5DSQ4yHeK/nsEq6w4ToAMTBl0K06khqxdSGgjMSwwrxvLzq3so0LC5Q14dA==",
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/@pagefind/linux-arm64/-/linux-arm64-1.1.1.tgz",
"integrity": "sha512-H5P6wDoCoAbdsWp0Zx0DxnLUrwTGWGLu/VI1rcN2CyFdY2EGSvPQsbGBMrseKRNuIrJDFtxHHHyjZ7UbzaM9EA==",
"cpu": [
"arm64"
],
@ -4030,9 +4075,9 @@
]
},
"node_modules/@pagefind/linux-x64": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@pagefind/linux-x64/-/linux-x64-1.1.0.tgz",
"integrity": "sha512-4lsg6VB7A6PWTwaP8oSmXV4O9H0IHX7AlwTDcfyT+YJo/sPXOVjqycD5cdBgqNLfUk8B9bkWcTDCRmJbHrKeCw==",
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/@pagefind/linux-x64/-/linux-x64-1.1.1.tgz",
"integrity": "sha512-yJs7tTYbL2MI3HT+ngs9E1BfUbY9M4/YzA0yEM5xBo4Xl8Yu8Qg2xZTOQ1/F6gwvMrjCUFo8EoACs6LRDhtMrQ==",
"cpu": [
"x64"
],
@ -4042,9 +4087,9 @@
]
},
"node_modules/@pagefind/windows-x64": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@pagefind/windows-x64/-/windows-x64-1.1.0.tgz",
"integrity": "sha512-OboCM76BcMKT9IoSfZuFhiqMRgTde8x4qDDvKulFmycgiJrlL5WnIqBHJLQxZq+o2KyZpoHF97iwsGAm8c32sQ==",
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/@pagefind/windows-x64/-/windows-x64-1.1.1.tgz",
"integrity": "sha512-b7/qPqgIl+lMzkQ8fJt51SfguB396xbIIR+VZ3YrL2tLuyifDJ1wL5mEm+ddmHxJ2Fki340paPcDan9en5OmAw==",
"cpu": [
"x64"
],
@ -12261,14 +12306,6 @@
"resolved": "https://registry.npmjs.org/hex-color-regex/-/hex-color-regex-1.1.0.tgz",
"integrity": "sha512-l9sfDFsuqtOqKDsQdqrMRk0U85RZc0RtOR9yPI7mRVOa4FsR/BVnZ0shmQRM96Ji99kYZP/7hn1cedc1+ApsTQ=="
},
"node_modules/highlight.js": {
"version": "11.10.0",
"resolved": "https://registry.npmjs.org/highlight.js/-/highlight.js-11.10.0.tgz",
"integrity": "sha512-SYVnVFswQER+zu1laSya563s+F8VDGt7o35d4utbamowvUNLLMovFqwCLSocpZTz3MgaSRA1IbqRWZv97dtErQ==",
"engines": {
"node": ">=12.0.0"
}
},
"node_modules/history": {
"version": "4.10.1",
"resolved": "https://registry.npmjs.org/history/-/history-4.10.1.tgz",
@ -14266,29 +14303,29 @@
"integrity": "sha512-3mk/Zag0+IJxeDrxSgaDPy4zZ3w05PRZeJNnlWhzFz5OkX49J4krc+A8X2d2M69vGMBEX0uyl8M+W+8gH+kBqQ=="
},
"node_modules/lit": {
"version": "3.1.4",
"resolved": "https://registry.npmjs.org/lit/-/lit-3.1.4.tgz",
"integrity": "sha512-q6qKnKXHy2g1kjBaNfcoLlgbI3+aSOZ9Q4tiGa9bGYXq5RBXxkVTqTIVmP2VWMp29L4GyvCFm8ZQ2o56eUAMyA==",
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/lit/-/lit-3.2.0.tgz",
"integrity": "sha512-s6tI33Lf6VpDu7u4YqsSX78D28bYQulM+VAzsGch4fx2H0eLZnJsUBsPWmGYSGoKDNbjtRv02rio1o+UdPVwvw==",
"dependencies": {
"@lit/reactive-element": "^2.0.4",
"lit-element": "^4.0.4",
"lit-html": "^3.1.2"
"lit-element": "^4.1.0",
"lit-html": "^3.2.0"
}
},
"node_modules/lit-element": {
"version": "4.0.6",
"resolved": "https://registry.npmjs.org/lit-element/-/lit-element-4.0.6.tgz",
"integrity": "sha512-U4sdJ3CSQip7sLGZ/uJskO5hGiqtlpxndsLr6mt3IQIjheg93UKYeGQjWMRql1s/cXNOaRrCzC2FQwjIwSUqkg==",
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/lit-element/-/lit-element-4.1.0.tgz",
"integrity": "sha512-gSejRUQJuMQjV2Z59KAS/D4iElUhwKpIyJvZ9w+DIagIQjfJnhR20h2Q5ddpzXGS+fF0tMZ/xEYGMnKmaI/iww==",
"dependencies": {
"@lit-labs/ssr-dom-shim": "^1.2.0",
"@lit/reactive-element": "^2.0.4",
"lit-html": "^3.1.2"
"lit-html": "^3.2.0"
}
},
"node_modules/lit-html": {
"version": "3.1.4",
"resolved": "https://registry.npmjs.org/lit-html/-/lit-html-3.1.4.tgz",
"integrity": "sha512-yKKO2uVv7zYFHlWMfZmqc+4hkmSbFp8jgjdZY9vvR9jr4J8fH6FUMXhr+ljfELgmjpvlF7Z1SJ5n5/Jeqtc9YA==",
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/lit-html/-/lit-html-3.2.0.tgz",
"integrity": "sha512-pwT/HwoxqI9FggTrYVarkBKFN9MlTUpLrDHubTmW4SrkL3kkqW5gxwbxMMUnbbRHBC0WTZnYHcjDSCM559VyfA==",
"dependencies": {
"@types/trusted-types": "^2.0.2"
}
@ -14691,9 +14728,9 @@
}
},
"node_modules/marked": {
"version": "13.0.3",
"resolved": "https://registry.npmjs.org/marked/-/marked-13.0.3.tgz",
"integrity": "sha512-rqRix3/TWzE9rIoFGIn8JmsVfhiuC8VIQ8IdX5TfzmeBucdY05/0UlzKaw0eVtpcN/OdVFpBk7CjKGo9iHJ/zA==",
"version": "14.1.0",
"resolved": "https://registry.npmjs.org/marked/-/marked-14.1.0.tgz",
"integrity": "sha512-P93GikH/Pde0hM5TAXEd8I4JAYi8IB03n8qzW8Bh1BIEFpEyBoYxi/XWZA53LSpTeLBiMQOoSMj0u5E/tiVYTA==",
"bin": {
"marked": "bin/marked.js"
},
@ -14701,6 +14738,14 @@
"node": ">= 18"
}
},
"node_modules/marked-footnote": {
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/marked-footnote/-/marked-footnote-1.2.3.tgz",
"integrity": "sha512-ATIAPjuTlaNx3T139KNNy5X16BaYauxFEAGB0DtjonxIwRQ2DExdP6tAvUuG7UTkMGPOf6OjgIKhwXpSv7MoAg==",
"peerDependencies": {
"marked": ">=7.0.0"
}
},
"node_modules/math-random": {
"version": "1.0.4",
"resolved": "https://registry.npmjs.org/math-random/-/math-random-1.0.4.tgz",
@ -15917,18 +15962,18 @@
}
},
"node_modules/pagefind": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/pagefind/-/pagefind-1.1.0.tgz",
"integrity": "sha512-1nmj0/vfYcMxNEQj0YDRp6bTVv9hI7HLdPhK/vBBYlrnwjATndQvHyicj5Y7pUHrpCFZpFnLVQXIF829tpFmaw==",
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/pagefind/-/pagefind-1.1.1.tgz",
"integrity": "sha512-U2YR0dQN5B2fbIXrLtt/UXNS0yWSSYfePaad1KcBPTi0p+zRtsVjwmoPaMQgTks5DnHNbmDxyJUL5TGaLljK3A==",
"bin": {
"pagefind": "lib/runner/bin.cjs"
},
"optionalDependencies": {
"@pagefind/darwin-arm64": "1.1.0",
"@pagefind/darwin-x64": "1.1.0",
"@pagefind/linux-arm64": "1.1.0",
"@pagefind/linux-x64": "1.1.0",
"@pagefind/windows-x64": "1.1.0"
"@pagefind/darwin-arm64": "1.1.1",
"@pagefind/darwin-x64": "1.1.1",
"@pagefind/linux-arm64": "1.1.1",
"@pagefind/linux-x64": "1.1.1",
"@pagefind/windows-x64": "1.1.1"
}
},
"node_modules/param-case": {

4
docs/my-website/package.json
View file

@ -18,8 +18,8 @@
"@docusaurus/plugin-google-gtag": "^2.4.1",
"@docusaurus/plugin-ideal-image": "^2.4.1",
"@docusaurus/preset-classic": "2.4.1",
"@getcanary/docusaurus-pagefind": "^0.0.12",
"@getcanary/web": "^0.0.55",
"@getcanary/docusaurus-theme-search-pagefind": "^0.0.13",
"@getcanary/web": "^0.0.101",
"@mdx-js/react": "^1.6.22",
"clsx": "^1.2.1",
"docusaurus": "^1.14.7",

2
docs/my-website/sidebars.js
View file

@ -44,7 +44,7 @@ const sidebars = {
"proxy/reliability",
{
type: "category",
label: "Use with Vertex, Bedrock, Cohere SDK",
label: "Use with Provider SDKs",
items: [
"pass_through/vertex_ai",
"pass_through/google_ai_studio",

37
docs/my-website/src/theme/SearchBar.js
View file

@ -6,7 +6,7 @@ import { usePluginData } from "@docusaurus/useGlobalData";
export default function SearchBarWrapper(props) {
const { siteConfig } = useDocusaurusContext();
const { options } = usePluginData("docusaurus-plugin-pagefind-canary");
const { options } = usePluginData("docusaurus-theme-search-pagefind");
const [path, setPath] = React.useState("");
const [loaded, setLoaded] = React.useState(false);
@ -24,10 +24,7 @@ export default function SearchBarWrapper(props) {
import("@getcanary/web/components/canary-content"),
import("@getcanary/web/components/canary-search"),
import("@getcanary/web/components/canary-search-input"),
import("@getcanary/web/components/canary-search-results-group"),
import("@getcanary/web/components/canary-footer"),
import("@getcanary/web/components/canary-callout-calendly"),
import("@getcanary/web/components/canary-callout-discord"),
import("@getcanary/web/components/canary-search-results-tabs"),
])
.then(() => setLoaded(true))
.catch(console.error);
@ -57,32 +54,18 @@ export default function SearchBarWrapper(props) {
</button>
) : (
<canary-root framework="docusaurus">
<canary-provider-pagefind
options={JSON.stringify({ ...options, path })}
>
<canary-provider-pagefind options={JSON.stringify(options)}>
<canary-modal>
<canary-trigger-logo slot="trigger"></canary-trigger-logo>
<canary-content slot="content">
<canary-search slot="search">
<canary-search slot="mode">
<canary-search-input slot="input"></canary-search-input>
<canary-search-results-group
slot="results"
groups="SDK:*;Proxy:/docs/(simple_proxy|proxy/.*)"
></canary-search-results-group>
<canary-callout-discord
slot="callout"
message="👋 Looking for help?"
url="https://discord.com/invite/wuPM9dRgDw"
keywords="discord,help,support,community"
></canary-callout-discord>
<canary-callout-calendly
slot="callout"
message="🚅 Interested in enterprise features?"
keywords="sso,enterprise,security,audit"
url="https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat"
></canary-callout-calendly>
<canary-search-results-tabs
slot="body"
tabs={JSON.stringify(options.tabs)}
group
></canary-search-results-tabs>
</canary-search>
<canary-footer slot="footer"></canary-footer>
</canary-content>
</canary-modal>
</canary-provider-pagefind>
@ -92,4 +75,4 @@ export default function SearchBarWrapper(props) {
<SearchBar {...props} />
</div>
);
}
}

1
litellm/__init__.py
View file

@ -834,7 +834,6 @@ from .utils import (
decode,
_calculate_retry_after,
_should_retry,
get_secret,
get_supported_openai_params,
get_api_base,
get_first_chars_messages,

3
litellm/batches/main.py
View file

@ -22,6 +22,7 @@ import litellm
from litellm import client
from litellm.llms.azure import AzureBatchesAPI
from litellm.llms.openai import OpenAIBatchesAPI
from litellm.secret_managers.main import get_secret
from litellm.types.llms.openai import (
Batch,
CancelBatchRequest,
@ -34,7 +35,7 @@ from litellm.types.llms.openai import (
RetrieveBatchRequest,
)
from litellm.types.router import GenericLiteLLMParams
from litellm.utils import get_secret, supports_httpx_timeout
from litellm.utils import supports_httpx_timeout
####### ENVIRONMENT VARIABLES ###################
openai_batches_instance = OpenAIBatchesAPI()

2
litellm/fine_tuning/main.py
View file

@ -17,7 +17,6 @@ from typing import Any, Coroutine, Dict, Literal, Optional, Union
import httpx
import litellm
from litellm import get_secret
from litellm._logging import verbose_logger
from litellm.llms.fine_tuning_apis.azure import AzureOpenAIFineTuningAPI
from litellm.llms.fine_tuning_apis.openai import (
@ -26,6 +25,7 @@ from litellm.llms.fine_tuning_apis.openai import (
OpenAIFineTuningAPI,
)
from litellm.llms.fine_tuning_apis.vertex_ai import VertexFineTuningAPI
from litellm.secret_managers.main import get_secret
from litellm.types.llms.openai import Hyperparameters
from litellm.types.router import *
from litellm.utils import supports_httpx_timeout

21
litellm/integrations/gcs_bucket_base.py
View file

@ -52,6 +52,27 @@ class GCSBucketBase(CustomLogger):
return headers
def sync_construct_request_headers(self) -> Dict[str, str]:
from litellm import vertex_chat_completion
auth_header, _ = vertex_chat_completion._get_token_and_url(
model="gcs-bucket",
vertex_credentials=self.path_service_account_json,
vertex_project=None,
vertex_location=None,
gemini_api_key=None,
stream=None,
custom_llm_provider="vertex_ai",
api_base=None,
)
verbose_logger.debug("constructed auth_header %s", auth_header)
headers = {
"Authorization": f"Bearer {auth_header}", # auth_header
"Content-Type": "application/json",
}
return headers
async def download_gcs_object(self, object_name):
"""
Download an object from GCS.

2
litellm/llms/base_aws_llm.py
View file

@ -5,7 +5,7 @@ import httpx
from litellm._logging import verbose_logger
from litellm.caching import DualCache, InMemoryCache
from litellm.utils import get_secret
from litellm.secret_managers.main import get_secret
from .base import BaseLLM

2
litellm/llms/bedrock/common_utils.py
View file

@ -10,7 +10,7 @@ from typing import List, Optional, Union
import httpx
import litellm
from litellm import get_secret
from litellm.secret_managers.main import get_secret
class BedrockError(Exception):

2
litellm/llms/bedrock/embed/embedding.py
View file

@ -11,7 +11,6 @@ from typing import Any, Callable, List, Literal, Optional, Tuple, Union
import httpx
import litellm
from litellm import get_secret
from litellm.llms.cohere.embed import embedding as cohere_embedding
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
@ -19,6 +18,7 @@ from litellm.llms.custom_httpx.http_handler import (
_get_async_httpx_client,
_get_httpx_client,
)
from litellm.secret_managers.main import get_secret
from litellm.types.llms.bedrock import AmazonEmbeddingRequest, CohereEmbeddingRequest
from litellm.types.utils import Embedding, EmbeddingResponse, Usage

2
litellm/proxy/_types.py
View file

@ -1063,6 +1063,7 @@ class KeyManagementSystem(enum.Enum):
GOOGLE_KMS = "google_kms"
AZURE_KEY_VAULT = "azure_key_vault"
AWS_SECRET_MANAGER = "aws_secret_manager"
GOOGLE_SECRET_MANAGER = "google_secret_manager"
LOCAL = "local"
AWS_KMS = "aws_kms"
@ -1638,6 +1639,7 @@ class AllCallbacks(LiteLLMBase):
"LAGO_API_BASE",
"LAGO_API_KEY",
"LAGO_API_EVENT_CODE",
"LAGO_API_CHARGE_BY",
],
ui_callback_name="Lago Billing",
)

2
litellm/proxy/auth/rds_iam_token.py
View file

@ -14,7 +14,7 @@ def init_rds_client(
aws_web_identity_token: Optional[str] = None,
timeout: Optional[Union[float, httpx.Timeout]] = None,
):
from litellm import get_secret
from litellm.secret_managers.main import get_secret
# check for custom AWS_REGION_NAME and use it if not passed to init_bedrock_client
litellm_aws_region_name = get_secret("AWS_REGION_NAME", None)

83
litellm/proxy/db/dynamo_db.py
View file

@ -1,16 +1,17 @@
from litellm.proxy.db.base_client import CustomDB
from litellm.proxy._types import (
DynamoDBArgs,
LiteLLM_VerificationToken,
LiteLLM_Config,
LiteLLM_UserTable,
)
from litellm.proxy.utils import hash_token
from litellm import get_secret
from typing import Any, List, Literal, Optional, Union
import json
from datetime import datetime
from typing import Any, List, Literal, Optional, Union
from litellm._logging import verbose_proxy_logger
from litellm.proxy._types import (
DynamoDBArgs,
LiteLLM_Config,
LiteLLM_UserTable,
LiteLLM_VerificationToken,
)
from litellm.proxy.db.base_client import CustomDB
from litellm.proxy.utils import hash_token
from litellm.secret_managers.main import get_secret
class DynamoDBWrapper(CustomDB):
@ -21,19 +22,19 @@ class DynamoDBWrapper(CustomDB):
def __init__(self, database_arguments: DynamoDBArgs):
from aiodynamo.client import Client
from aiodynamo.credentials import Credentials, StaticCredentials
from aiodynamo.expressions import F, UpdateExpression, Value
from aiodynamo.http.aiohttp import AIOHTTP
from aiodynamo.http.httpx import HTTPX
from aiodynamo.models import (
Throughput,
KeySchema,
KeySpec,
KeyType,
PayPerRequest,
ReturnValues,
Throughput,
)
from yarl import URL
from aiodynamo.expressions import UpdateExpression, F, Value
from aiodynamo.models import ReturnValues
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession
from yarl import URL
self.throughput_type = None
if database_arguments.billing_mode == "PAY_PER_REQUEST":
@ -59,7 +60,9 @@ class DynamoDBWrapper(CustomDB):
verbose_proxy_logger.debug(
f"DynamoDB: setting env vars based on arn={self.database_arguments.aws_role_name}"
)
import boto3, os
import os
import boto3
sts_client = boto3.client("sts")
@ -92,22 +95,22 @@ class DynamoDBWrapper(CustomDB):
"""
Connect to DB, and creating / updating any tables
"""
import aiohttp
from aiodynamo.client import Client
from aiodynamo.credentials import Credentials, StaticCredentials
from aiodynamo.expressions import F, UpdateExpression, Value
from aiodynamo.http.aiohttp import AIOHTTP
from aiodynamo.http.httpx import HTTPX
from aiodynamo.models import (
Throughput,
KeySchema,
KeySpec,
KeyType,
PayPerRequest,
ReturnValues,
Throughput,
)
from yarl import URL
from aiodynamo.expressions import UpdateExpression, F, Value
from aiodynamo.models import ReturnValues
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession
import aiohttp
from yarl import URL
verbose_proxy_logger.debug("DynamoDB Wrapper - Attempting to connect")
self.set_env_vars_based_on_arn()
@ -192,22 +195,22 @@ class DynamoDBWrapper(CustomDB):
async def insert_data(
self, value: Any, table_name: Literal["user", "key", "config", "spend"]
):
import aiohttp
from aiodynamo.client import Client
from aiodynamo.credentials import Credentials, StaticCredentials
from aiodynamo.expressions import F, UpdateExpression, Value
from aiodynamo.http.aiohttp import AIOHTTP
from aiodynamo.http.httpx import HTTPX
from aiodynamo.models import (
Throughput,
KeySchema,
KeySpec,
KeyType,
PayPerRequest,
ReturnValues,
Throughput,
)
from yarl import URL
from aiodynamo.expressions import UpdateExpression, F, Value
from aiodynamo.models import ReturnValues
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession
import aiohttp
from yarl import URL
self.set_env_vars_based_on_arn()
@ -237,22 +240,22 @@ class DynamoDBWrapper(CustomDB):
return await table.put_item(item=value, return_values=ReturnValues.all_old)
async def get_data(self, key: str, table_name: Literal["user", "key", "config"]):
import aiohttp
from aiodynamo.client import Client
from aiodynamo.credentials import Credentials, StaticCredentials
from aiodynamo.expressions import F, UpdateExpression, Value
from aiodynamo.http.aiohttp import AIOHTTP
from aiodynamo.http.httpx import HTTPX
from aiodynamo.models import (
Throughput,
KeySchema,
KeySpec,
KeyType,
PayPerRequest,
ReturnValues,
Throughput,
)
from yarl import URL
from aiodynamo.expressions import UpdateExpression, F, Value
from aiodynamo.models import ReturnValues
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession
import aiohttp
from yarl import URL
self.set_env_vars_based_on_arn()
@ -311,22 +314,22 @@ class DynamoDBWrapper(CustomDB):
self, key: str, value: dict, table_name: Literal["user", "key", "config"]
):
self.set_env_vars_based_on_arn()
import aiohttp
from aiodynamo.client import Client
from aiodynamo.credentials import Credentials, StaticCredentials
from aiodynamo.expressions import F, UpdateExpression, Value
from aiodynamo.http.aiohttp import AIOHTTP
from aiodynamo.http.httpx import HTTPX
from aiodynamo.models import (
Throughput,
KeySchema,
KeySpec,
KeyType,
PayPerRequest,
ReturnValues,
Throughput,
)
from yarl import URL
from aiodynamo.expressions import UpdateExpression, F, Value
from aiodynamo.models import ReturnValues
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession
import aiohttp
from yarl import URL
if self.database_arguments.ssl_verify == False:
client_session = ClientSession(connector=aiohttp.TCPConnector(ssl=False))

2
litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py
View file

@ -24,7 +24,6 @@ import httpx
from fastapi import HTTPException
import litellm
from litellm import get_secret
from litellm._logging import verbose_proxy_logger
from litellm.caching import DualCache
from litellm.integrations.custom_guardrail import CustomGuardrail
@ -38,6 +37,7 @@ from litellm.llms.custom_httpx.http_handler import (
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.guardrails.guardrail_helpers import should_proceed_based_on_metadata
from litellm.secret_managers.main import get_secret
from litellm.types.guardrails import (
BedrockContentItem,
BedrockRequest,

2
litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py
View file

@ -19,12 +19,12 @@ import httpx
from fastapi import HTTPException
import litellm
from litellm import get_secret
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.guardrails.guardrail_helpers import should_proceed_based_on_metadata
from litellm.secret_managers.main import get_secret
from litellm.types.guardrails import (
GuardrailItem,
LakeraCategoryThresholds,

2
litellm/proxy/prisma_migration.py
View file

@ -9,7 +9,7 @@ import time
sys.path.insert(
0, os.path.abspath("./")
) # Adds the parent directory to the system path
from litellm.proxy.secret_managers.aws_secret_manager import decrypt_env_var
from litellm.secret_managers.aws_secret_manager import decrypt_env_var
if os.getenv("USE_AWS_KMS", None) is not None and os.getenv("USE_AWS_KMS") == "True":
## V2 IMPLEMENTATION OF AWS KMS - USER WANTS TO DECRYPT MULTIPLE KEYS IN THEIR ENV

13
litellm/proxy/proxy_cli.py
View file

@ -475,7 +475,7 @@ def run_server(
### DECRYPT ENV VAR ###
from litellm.proxy.secret_managers.aws_secret_manager import decrypt_env_var
from litellm.secret_managers.aws_secret_manager import decrypt_env_var
if (
os.getenv("USE_AWS_KMS", None) is not None
@ -544,6 +544,15 @@ def run_server(
load_aws_secret_manager(use_aws_secret_manager=True)
elif key_management_system == KeyManagementSystem.AWS_KMS.value:
load_aws_kms(use_aws_kms=True)
elif (
key_management_system
== KeyManagementSystem.GOOGLE_SECRET_MANAGER.value
):
from litellm.secret_managers.google_secret_manager import (
GoogleSecretManager,
)
GoogleSecretManager()
else:
raise ValueError("Invalid Key Management System selected")
key_management_settings = general_settings.get(
@ -598,7 +607,7 @@ def run_server(
or os.getenv("DIRECT_URL", None) is not None
):
try:
from litellm import get_secret
from litellm.secret_managers.main import get_secret
if os.getenv("DATABASE_URL", None) is not None:
### add connection pool + pool timeout args

21
litellm/proxy/proxy_config.yaml
View file

@ -1,10 +1,15 @@
model_list:
- model_name: fake-openai-endpoint
litellm_params:
model: openai/fake
api_key: fake-key
model_info:
id: "team-a-model" # used for identifying model in response headers
- model_name: fake-openai-endpoint
litellm_params:
model: openai/fake
api_base: https://exampleopenaiendpoint-production.up.railway.app/
api_key: os.environ/OPENAI_API_KEY
- model_name: gpt-3.5-turbo-end-user-test
litellm_params:
model: azure/chatgpt-v-2
api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
api_version: "2023-05-15"
api_key: os.environ/AZURE_API_KEY
litellm_settings:
success_callback: ["prometheus"]
@ -13,7 +18,3 @@ litellm_settings:
general_settings:
master_key: sk-1234
allowed_routes: ["/chat/completions"]
public_routes: ["LiteLLMRoutes.public_routes"]
admin_only_routes: ["/key/generate"]

19
litellm/proxy/proxy_server.py
View file

@ -212,11 +212,6 @@ from litellm.proxy.pass_through_endpoints.pass_through_endpoints import (
)
from litellm.proxy.rerank_endpoints.endpoints import router as rerank_router
from litellm.proxy.route_llm_request import route_request
from litellm.proxy.secret_managers.aws_secret_manager import (
load_aws_kms,
load_aws_secret_manager,
)
from litellm.proxy.secret_managers.google_kms import load_google_kms
from litellm.proxy.spend_tracking.spend_management_endpoints import (
router as spend_management_router,
)
@ -257,6 +252,11 @@ from litellm.router import (
from litellm.router import ModelInfo as RouterModelInfo
from litellm.router import updateDeployment
from litellm.scheduler import DefaultPriorities, FlowItem, Scheduler
from litellm.secret_managers.aws_secret_manager import (
load_aws_kms,
load_aws_secret_manager,
)
from litellm.secret_managers.google_kms import load_google_kms
from litellm.types.llms.anthropic import (
AnthropicMessagesRequest,
AnthropicResponse,
@ -1765,6 +1765,15 @@ class ProxyConfig:
load_aws_secret_manager(use_aws_secret_manager=True)
elif key_management_system == KeyManagementSystem.AWS_KMS.value:
load_aws_kms(use_aws_kms=True)
elif (
key_management_system
== KeyManagementSystem.GOOGLE_SECRET_MANAGER.value
):
from litellm.secret_managers.google_secret_manager import (
GoogleSecretManager,
)
GoogleSecretManager()
else:
raise ValueError("Invalid Key Management System selected")
key_management_settings = general_settings.get(

2
litellm/rerank_api/main.py
View file

@ -4,10 +4,10 @@ from functools import partial
from typing import Any, Coroutine, Dict, List, Literal, Optional, Union
import litellm
from litellm import get_secret
from litellm._logging import verbose_logger
from litellm.llms.cohere.rerank import CohereRerank
from litellm.llms.togetherai.rerank import TogetherAIRerank
from litellm.secret_managers.main import get_secret
from litellm.types.router import *
from litellm.utils import supports_httpx_timeout

2
litellm/router_utils/client_initalization_utils.py
View file

@ -9,7 +9,7 @@ import openai
import litellm
from litellm._logging import verbose_router_logger
from litellm.llms.azure import get_azure_ad_token_from_oidc
from litellm.proxy.secret_managers.get_azure_ad_token_provider import (
from litellm.secret_managers.get_azure_ad_token_provider import (
get_azure_ad_token_provider,
)
from litellm.utils import calculate_max_parallel_requests

3
litellm/secret_managers/Readme.md Normal file
View file

@ -0,0 +1,3 @@
## Supported Secret Managers to read credentials from
Example read OPENAI_API_KEY, AZURE_API_KEY from a secret manager

0
litellm/proxy/secret_managers/aws_secret_manager.py → litellm/secret_managers/aws_secret_manager.py
View file

7
litellm/proxy/secret_managers/get_azure_ad_token_provider.py → litellm/secret_managers/get_azure_ad_token_provider.py
View file

@ -14,8 +14,7 @@ def get_azure_ad_token_provider() -> Callable[[], str]:
Returns:
Callable that returns a temporary authentication token.
"""
from azure.identity import ClientSecretCredential
from azure.identity import get_bearer_token_provider
from azure.identity import ClientSecretCredential, get_bearer_token_provider
try:
credential = ClientSecretCredential(
@ -24,7 +23,9 @@ def get_azure_ad_token_provider() -> Callable[[], str]:
tenant_id=os.environ["AZURE_TENANT_ID"],
)
except KeyError as e:
raise ValueError("Missing environment variable required by Azure AD workflow.") from e
raise ValueError(
"Missing environment variable required by Azure AD workflow."
) from e
return get_bearer_token_provider(
credential,

5
litellm/proxy/secret_managers/google_kms.py → litellm/secret_managers/google_kms.py
View file

@ -7,8 +7,11 @@ Requires:
* `os.environ["GOOGLE_APPLICATION_CREDENTIALS"], os.environ["GOOGLE_KMS_RESOURCE_NAME"]`
* `pip install google-cloud-kms`
"""
import litellm, os
import os
from typing import Optional
import litellm
from litellm.proxy._types import KeyManagementSystem

116
litellm/secret_managers/google_secret_manager.py Normal file
View file

@ -0,0 +1,116 @@
import base64
import os
from typing import Optional
import litellm
from litellm._logging import verbose_logger
from litellm.caching import InMemoryCache
from litellm.integrations.gcs_bucket_base import GCSBucketBase
from litellm.llms.custom_httpx.http_handler import _get_httpx_client
from litellm.proxy._types import CommonProxyErrors, KeyManagementSystem
class GoogleSecretManager(GCSBucketBase):
def __init__(
self,
refresh_interval: Optional[int] = 86400,
always_read_secret_manager: Optional[bool] = False,
) -> None:
"""
Args:
refresh_interval (int, optional): The refresh interval in seconds. Defaults to 86400. (24 hours)
always_read_secret_manager (bool, optional): Whether to always read from the secret manager. Defaults to False. Since we do want to cache values
"""
from litellm.proxy.proxy_server import premium_user
if premium_user is not True:
raise ValueError(
f"Google Secret Manager requires an Enterprise License {CommonProxyErrors.not_premium_user.value}"
)
super().__init__()
self.PROJECT_ID = os.environ.get("GOOGLE_SECRET_MANAGER_PROJECT_ID", None)
if self.PROJECT_ID is None:
raise ValueError(
"Google Secret Manager requires a project ID, please set 'GOOGLE_SECRET_MANAGER_PROJECT_ID' in your .env"
)
self.sync_httpx_client = _get_httpx_client()
litellm.secret_manager_client = self
litellm._key_management_system = KeyManagementSystem.GOOGLE_SECRET_MANAGER
_refresh_interval = os.environ.get(
"GOOGLE_SECRET_MANAGER_REFRESH_INTERVAL", refresh_interval
)
_refresh_interval = (
int(_refresh_interval) if _refresh_interval else refresh_interval
)
self.cache = InMemoryCache(
default_ttl=_refresh_interval
) # store in memory for 1 day
_always_read_secret_manager = os.environ.get(
"GOOGLE_SECRET_MANAGER_ALWAYS_READ_SECRET_MANAGER",
)
if (
_always_read_secret_manager
and _always_read_secret_manager.lower() == "true"
):
self.always_read_secret_manager = True
else:
# by default this should be False, we want to use in memory caching for this. It's a bad idea to fetch from secret manager for all requests
self.always_read_secret_manager = always_read_secret_manager or False
def get_secret_from_google_secret_manager(self, secret_name: str) -> Optional[str]:
"""
Retrieve a secret from Google Secret Manager or cache.
Args:
secret_name (str): The name of the secret.
Returns:
str: The secret value if successful, None otherwise.
"""
if self.always_read_secret_manager is not True:
cached_secret = self.cache.get_cache(secret_name)
if cached_secret is not None:
return cached_secret
if secret_name in self.cache.cache_dict:
return cached_secret
_secret_name = (
f"projects/{self.PROJECT_ID}/secrets/{secret_name}/versions/latest"
)
headers = self.sync_construct_request_headers()
url = f"https://secretmanager.googleapis.com/v1/{_secret_name}:access"
# Send the GET request to retrieve the secret
response = self.sync_httpx_client.get(url=url, headers=headers)
if response.status_code != 200:
verbose_logger.error(
"Google Secret Manager retrieval error: %s", str(response.text)
)
self.cache.set_cache(
secret_name, None
) # Cache that the secret was not found
raise ValueError(
f"secret {secret_name} not found in Google Secret Manager. Error: {response.text}"
)
verbose_logger.debug(
"Google Secret Manager retrieval response status code: %s",
response.status_code,
)
# Parse the JSON response and return the secret value
secret_data = response.json()
_base64_encoded_value = secret_data.get("payload", {}).get("data")
# decode the base64 encoded value
if _base64_encoded_value is not None:
_decoded_value = base64.b64decode(_base64_encoded_value).decode("utf-8")
self.cache.set_cache(
secret_name, _decoded_value
) # Cache the retrieved secret
return _decoded_value
self.cache.set_cache(secret_name, None) # Cache that the secret was not found
raise ValueError(f"secret {secret_name} not found in Google Secret Manager")

276
litellm/secret_managers/main.py Normal file
View file

@ -0,0 +1,276 @@
import ast
import base64
import binascii
import json
import os
import sys
import traceback
from typing import Any, Optional, Union
import httpx
from dotenv import load_dotenv
import litellm
from litellm._logging import print_verbose, verbose_logger
from litellm.caching import DualCache
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.proxy._types import KeyManagementSystem
oidc_cache = DualCache()
######### Secret Manager ############################
# checks if user has passed in a secret manager client
# if passed in then checks the secret there
def _is_base64(s):
try:
return base64.b64encode(base64.b64decode(s)).decode() == s
except binascii.Error:
return False
def get_secret(
secret_name: str,
default_value: Optional[Union[str, bool]] = None,
):
key_management_system = litellm._key_management_system
key_management_settings = litellm._key_management_settings
if secret_name.startswith("os.environ/"):
secret_name = secret_name.replace("os.environ/", "")
# Example: oidc/google/https://bedrock-runtime.us-east-1.amazonaws.com/model/stability.stable-diffusion-xl-v1/invoke
if secret_name.startswith("oidc/"):
secret_name_split = secret_name.replace("oidc/", "")
oidc_provider, oidc_aud = secret_name_split.split("/", 1)
# TODO: Add caching for HTTP requests
if oidc_provider == "google":
oidc_token = oidc_cache.get_cache(key=secret_name)
if oidc_token is not None:
return oidc_token
oidc_client = HTTPHandler(timeout=httpx.Timeout(timeout=600.0, connect=5.0))
# https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
response = oidc_client.get(
"http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity",
params={"audience": oidc_aud},
headers={"Metadata-Flavor": "Google"},
)
if response.status_code == 200:
oidc_token = response.text
oidc_cache.set_cache(key=secret_name, value=oidc_token, ttl=3600 - 60)
return oidc_token
else:
raise ValueError("Google OIDC provider failed")
elif oidc_provider == "circleci":
# https://circleci.com/docs/openid-connect-tokens/
env_secret = os.getenv("CIRCLE_OIDC_TOKEN")
if env_secret is None:
raise ValueError("CIRCLE_OIDC_TOKEN not found in environment")
return env_secret
elif oidc_provider == "circleci_v2":
# https://circleci.com/docs/openid-connect-tokens/
env_secret = os.getenv("CIRCLE_OIDC_TOKEN_V2")
if env_secret is None:
raise ValueError("CIRCLE_OIDC_TOKEN_V2 not found in environment")
return env_secret
elif oidc_provider == "github":
# https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-cloud-providers#using-custom-actions
actions_id_token_request_url = os.getenv("ACTIONS_ID_TOKEN_REQUEST_URL")
actions_id_token_request_token = os.getenv("ACTIONS_ID_TOKEN_REQUEST_TOKEN")
if (
actions_id_token_request_url is None
or actions_id_token_request_token is None
):
raise ValueError(
"ACTIONS_ID_TOKEN_REQUEST_URL or ACTIONS_ID_TOKEN_REQUEST_TOKEN not found in environment"
)
oidc_token = oidc_cache.get_cache(key=secret_name)
if oidc_token is not None:
return oidc_token
oidc_client = HTTPHandler(timeout=httpx.Timeout(timeout=600.0, connect=5.0))
response = oidc_client.get(
actions_id_token_request_url,
params={"audience": oidc_aud},
headers={
"Authorization": f"Bearer {actions_id_token_request_token}",
"Accept": "application/json; api-version=2.0",
},
)
if response.status_code == 200:
oidc_token = response.text["value"]
oidc_cache.set_cache(key=secret_name, value=oidc_token, ttl=300 - 5)
return oidc_token
else:
raise ValueError("Github OIDC provider failed")
elif oidc_provider == "azure":
# https://azure.github.io/azure-workload-identity/docs/quick-start.html
azure_federated_token_file = os.getenv("AZURE_FEDERATED_TOKEN_FILE")
if azure_federated_token_file is None:
raise ValueError("AZURE_FEDERATED_TOKEN_FILE not found in environment")
with open(azure_federated_token_file, "r") as f:
oidc_token = f.read()
return oidc_token
elif oidc_provider == "file":
# Load token from a file
with open(oidc_aud, "r") as f:
oidc_token = f.read()
return oidc_token
elif oidc_provider == "env":
# Load token directly from an environment variable
oidc_token = os.getenv(oidc_aud)
if oidc_token is None:
raise ValueError(f"Environment variable {oidc_aud} not found")
return oidc_token
elif oidc_provider == "env_path":
# Load token from a file path specified in an environment variable
token_file_path = os.getenv(oidc_aud)
if token_file_path is None:
raise ValueError(f"Environment variable {oidc_aud} not found")
with open(token_file_path, "r") as f:
oidc_token = f.read()
return oidc_token
else:
raise ValueError("Unsupported OIDC provider")
try:
if litellm.secret_manager_client is not None:
try:
client = litellm.secret_manager_client
key_manager = "local"
if key_management_system is not None:
key_manager = key_management_system.value
if key_management_settings is not None:
if (
secret_name not in key_management_settings.hosted_keys
): # allow user to specify which keys to check in hosted key manager
key_manager = "local"
if (
key_manager == KeyManagementSystem.AZURE_KEY_VAULT.value
or type(client).__module__ + "." + type(client).__name__
== "azure.keyvault.secrets._client.SecretClient"
): # support Azure Secret Client - from azure.keyvault.secrets import SecretClient
secret = client.get_secret(secret_name).value
elif (
key_manager == KeyManagementSystem.GOOGLE_KMS.value
or client.__class__.__name__ == "KeyManagementServiceClient"
):
encrypted_secret: Any = os.getenv(secret_name)
if encrypted_secret is None:
raise ValueError(
f"Google KMS requires the encrypted secret to be in the environment!"
)
b64_flag = _is_base64(encrypted_secret)
if b64_flag == True: # if passed in as encoded b64 string
encrypted_secret = base64.b64decode(encrypted_secret)
ciphertext = encrypted_secret
else:
raise ValueError(
f"Google KMS requires the encrypted secret to be encoded in base64"
) # fix for this vulnerability https://huntr.com/bounties/ae623c2f-b64b-4245-9ed4-f13a0a5824ce
response = client.decrypt(
request={
"name": litellm._google_kms_resource_name,
"ciphertext": ciphertext,
}
)
secret = response.plaintext.decode(
"utf-8"
) # assumes the original value was encoded with utf-8
elif key_manager == KeyManagementSystem.AWS_KMS.value:
"""
Only check the tokens which start with 'aws_kms/'. This prevents latency impact caused by checking all keys.
"""
encrypted_value = os.getenv(secret_name, None)
if encrypted_value is None:
raise Exception(
"AWS KMS - Encrypted Value of Key={} is None".format(
secret_name
)
)
# Decode the base64 encoded ciphertext
ciphertext_blob = base64.b64decode(encrypted_value)
# Set up the parameters for the decrypt call
params = {"CiphertextBlob": ciphertext_blob}
# Perform the decryption
response = client.decrypt(**params)
# Extract and decode the plaintext
plaintext = response["Plaintext"]
secret = plaintext.decode("utf-8")
if isinstance(secret, str):
secret = secret.strip()
elif key_manager == KeyManagementSystem.AWS_SECRET_MANAGER.value:
try:
get_secret_value_response = client.get_secret_value(
SecretId=secret_name
)
print_verbose(
f"get_secret_value_response: {get_secret_value_response}"
)
except Exception as e:
print_verbose(f"An error occurred - {str(e)}")
# For a list of exceptions thrown, see
# https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
raise e
# assume there is 1 secret per secret_name
secret_dict = json.loads(get_secret_value_response["SecretString"])
print_verbose(f"secret_dict: {secret_dict}")
for k, v in secret_dict.items():
secret = v
print_verbose(f"secret: {secret}")
if key_manager == KeyManagementSystem.GOOGLE_SECRET_MANAGER.value:
try:
secret = client.get_secret_from_google_secret_manager(
secret_name
)
print_verbose(f"secret from google secret manager: {secret}")
if secret is None:
raise ValueError(
f"No secret found in Google Secret Manager for {secret_name}"
)
except Exception as e:
print_verbose(f"An error occurred - {str(e)}")
raise e
elif key_manager == "local":
secret = os.getenv(secret_name)
else: # assume the default is infisicial client
secret = client.get_secret(secret_name).secret_value
except Exception as e: # check if it's in os.environ
verbose_logger.error(
f"Defaulting to os.environ value for key={secret_name}. An exception occurred - {str(e)}.\n\n{traceback.format_exc()}"
)
secret = os.getenv(secret_name)
try:
secret_value_as_bool = ast.literal_eval(secret)
if isinstance(secret_value_as_bool, bool):
return secret_value_as_bool
else:
return secret
except:
return secret
else:
secret = os.environ.get(secret_name)
try:
secret_value_as_bool = (
ast.literal_eval(secret) if secret is not None else None
)
if isinstance(secret_value_as_bool, bool):
return secret_value_as_bool
else:
return secret
except Exception:
if default_value is not None:
return default_value
return secret
except Exception as e:
if default_value is not None:
return default_value
else:
raise e

4
litellm/tests/test_router_client_init.py
View file

@ -83,7 +83,7 @@ async def test_router_init():
)
@patch("litellm.proxy.secret_managers.get_azure_ad_token_provider.os")
@patch("litellm.secret_managers.get_azure_ad_token_provider.os")
def test_router_init_with_neither_api_key_nor_azure_service_principal_with_secret(
mocked_os_lib: MagicMock,
) -> None:
@ -128,7 +128,7 @@ def test_router_init_with_neither_api_key_nor_azure_service_principal_with_secre
@patch("azure.identity.get_bearer_token_provider")
@patch("azure.identity.ClientSecretCredential")
@patch("litellm.proxy.secret_managers.get_azure_ad_token_provider.os")
@patch("litellm.secret_managers.get_azure_ad_token_provider.os")
def test_router_init_azure_service_principal_with_secret_with_environment_variables(
mocked_os_lib: MagicMock,
mocked_credential: MagicMock,

55
litellm/tests/test_secret_manager.py
View file

@ -16,10 +16,10 @@ sys.path.insert(
) # Adds the parent directory to the system path
import pytest
from litellm import get_secret
from litellm.llms.azure import get_azure_ad_token_from_oidc
from litellm.llms.bedrock.chat import BedrockConverseLLM, BedrockLLM
from litellm.proxy.secret_managers.aws_secret_manager import load_aws_secret_manager
from litellm.secret_managers.aws_secret_manager import load_aws_secret_manager
from litellm.secret_managers.main import get_secret
@pytest.mark.skip(reason="AWS Suspended Account")
@ -189,3 +189,54 @@ def test_oidc_env_path():
assert secret_val == secret_value
del os.environ[env_var_name]
def test_google_secret_manager():
"""
Test that we can get a secret from Google Secret Manager
"""
os.environ["GOOGLE_SECRET_MANAGER_PROJECT_ID"] = "adroit-crow-413218"
from test_amazing_vertex_completion import load_vertex_ai_credentials
from litellm.secret_managers.google_secret_manager import GoogleSecretManager
# load_vertex_ai_credentials()
secret_manager = GoogleSecretManager()
secret_val = secret_manager.get_secret_from_google_secret_manager(
secret_name="OPENAI_API_KEY"
)
print("secret_val: {}".format(secret_val))
assert (
secret_val == "anything"
), "did not get expected secret value. expect 'anything', got '{}'".format(
secret_val
)
def test_google_secret_manager_read_in_memory():
"""
Test that Google Secret manager returs in memory value when it exists
"""
from test_amazing_vertex_completion import load_vertex_ai_credentials
from litellm.secret_managers.google_secret_manager import GoogleSecretManager
# load_vertex_ai_credentials()
os.environ["GOOGLE_SECRET_MANAGER_PROJECT_ID"] = "adroit-crow-413218"
secret_manager = GoogleSecretManager()
secret_manager.cache.cache_dict["UNIQUE_KEY"] = None
secret_manager.cache.cache_dict["UNIQUE_KEY_2"] = "lite-llm"
secret_val = secret_manager.get_secret_from_google_secret_manager(
secret_name="UNIQUE_KEY"
)
print("secret_val: {}".format(secret_val))
assert secret_val == None
secret_val = secret_manager.get_secret_from_google_secret_manager(
secret_name="UNIQUE_KEY_2"
)
print("secret_val: {}".format(secret_val))
assert secret_val == "lite-llm"

1
litellm/tests/test_streaming.py
View file

@ -3712,6 +3712,7 @@ def test_unit_test_custom_stream_wrapper_function_call():
"vertex_ai/claude-3-5-sonnet@20240620",
],
)
@pytest.mark.flaky(retries=3, delay=1)
def test_streaming_tool_calls_valid_json_str(model):
if "vertex_ai" in model:
from litellm.tests.test_amazing_vertex_completion import (

247
litellm/utils.py
View file

@ -68,6 +68,7 @@ from litellm.litellm_core_utils.redact_messages import (
)
from litellm.litellm_core_utils.token_counter import get_modified_max_tokens
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.secret_managers.main import get_secret
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionNamedToolChoiceParam,
@ -93,8 +94,6 @@ from litellm.types.utils import (
Usage,
)
oidc_cache = DualCache()
try:
# New and recommended way to access resources
from importlib import resources
@ -8662,250 +8661,6 @@ def exception_type(
raise raised_exc
######### Secret Manager ############################
# checks if user has passed in a secret manager client
# if passed in then checks the secret there
def _is_base64(s):
try:
return base64.b64encode(base64.b64decode(s)).decode() == s
except binascii.Error:
return False
def get_secret(
secret_name: str,
default_value: Optional[Union[str, bool]] = None,
):
key_management_system = litellm._key_management_system
key_management_settings = litellm._key_management_settings
if secret_name.startswith("os.environ/"):
secret_name = secret_name.replace("os.environ/", "")
# Example: oidc/google/https://bedrock-runtime.us-east-1.amazonaws.com/model/stability.stable-diffusion-xl-v1/invoke
if secret_name.startswith("oidc/"):
secret_name_split = secret_name.replace("oidc/", "")
oidc_provider, oidc_aud = secret_name_split.split("/", 1)
# TODO: Add caching for HTTP requests
if oidc_provider == "google":
oidc_token = oidc_cache.get_cache(key=secret_name)
if oidc_token is not None:
return oidc_token
oidc_client = HTTPHandler(timeout=httpx.Timeout(timeout=600.0, connect=5.0))
# https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
response = oidc_client.get(
"http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity",
params={"audience": oidc_aud},
headers={"Metadata-Flavor": "Google"},
)
if response.status_code == 200:
oidc_token = response.text
oidc_cache.set_cache(key=secret_name, value=oidc_token, ttl=3600 - 60)
return oidc_token
else:
raise ValueError("Google OIDC provider failed")
elif oidc_provider == "circleci":
# https://circleci.com/docs/openid-connect-tokens/
env_secret = os.getenv("CIRCLE_OIDC_TOKEN")
if env_secret is None:
raise ValueError("CIRCLE_OIDC_TOKEN not found in environment")
return env_secret
elif oidc_provider == "circleci_v2":
# https://circleci.com/docs/openid-connect-tokens/
env_secret = os.getenv("CIRCLE_OIDC_TOKEN_V2")
if env_secret is None:
raise ValueError("CIRCLE_OIDC_TOKEN_V2 not found in environment")
return env_secret
elif oidc_provider == "github":
# https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-cloud-providers#using-custom-actions
actions_id_token_request_url = os.getenv("ACTIONS_ID_TOKEN_REQUEST_URL")
actions_id_token_request_token = os.getenv("ACTIONS_ID_TOKEN_REQUEST_TOKEN")
if (
actions_id_token_request_url is None
or actions_id_token_request_token is None
):
raise ValueError(
"ACTIONS_ID_TOKEN_REQUEST_URL or ACTIONS_ID_TOKEN_REQUEST_TOKEN not found in environment"
)
oidc_token = oidc_cache.get_cache(key=secret_name)
if oidc_token is not None:
return oidc_token
oidc_client = HTTPHandler(timeout=httpx.Timeout(timeout=600.0, connect=5.0))
response = oidc_client.get(
actions_id_token_request_url,
params={"audience": oidc_aud},
headers={
"Authorization": f"Bearer {actions_id_token_request_token}",
"Accept": "application/json; api-version=2.0",
},
)
if response.status_code == 200:
oidc_token = response.text["value"]
oidc_cache.set_cache(key=secret_name, value=oidc_token, ttl=300 - 5)
return oidc_token
else:
raise ValueError("Github OIDC provider failed")
elif oidc_provider == "azure":
# https://azure.github.io/azure-workload-identity/docs/quick-start.html
azure_federated_token_file = os.getenv("AZURE_FEDERATED_TOKEN_FILE")
if azure_federated_token_file is None:
raise ValueError("AZURE_FEDERATED_TOKEN_FILE not found in environment")
with open(azure_federated_token_file, "r") as f:
oidc_token = f.read()
return oidc_token
elif oidc_provider == "file":
# Load token from a file
with open(oidc_aud, "r") as f:
oidc_token = f.read()
return oidc_token
elif oidc_provider == "env":
# Load token directly from an environment variable
oidc_token = os.getenv(oidc_aud)
if oidc_token is None:
raise ValueError(f"Environment variable {oidc_aud} not found")
return oidc_token
elif oidc_provider == "env_path":
# Load token from a file path specified in an environment variable
token_file_path = os.getenv(oidc_aud)
if token_file_path is None:
raise ValueError(f"Environment variable {oidc_aud} not found")
with open(token_file_path, "r") as f:
oidc_token = f.read()
return oidc_token
else:
raise ValueError("Unsupported OIDC provider")
try:
if litellm.secret_manager_client is not None:
try:
client = litellm.secret_manager_client
key_manager = "local"
if key_management_system is not None:
key_manager = key_management_system.value
if key_management_settings is not None:
if (
secret_name not in key_management_settings.hosted_keys
): # allow user to specify which keys to check in hosted key manager
key_manager = "local"
if (
key_manager == KeyManagementSystem.AZURE_KEY_VAULT.value
or type(client).__module__ + "." + type(client).__name__
== "azure.keyvault.secrets._client.SecretClient"
): # support Azure Secret Client - from azure.keyvault.secrets import SecretClient
secret = client.get_secret(secret_name).value
elif (
key_manager == KeyManagementSystem.GOOGLE_KMS.value
or client.__class__.__name__ == "KeyManagementServiceClient"
):
encrypted_secret: Any = os.getenv(secret_name)
if encrypted_secret is None:
raise ValueError(
f"Google KMS requires the encrypted secret to be in the environment!"
)
b64_flag = _is_base64(encrypted_secret)
if b64_flag == True: # if passed in as encoded b64 string
encrypted_secret = base64.b64decode(encrypted_secret)
ciphertext = encrypted_secret
else:
raise ValueError(
f"Google KMS requires the encrypted secret to be encoded in base64"
) # fix for this vulnerability https://huntr.com/bounties/ae623c2f-b64b-4245-9ed4-f13a0a5824ce
response = client.decrypt(
request={
"name": litellm._google_kms_resource_name,
"ciphertext": ciphertext,
}
)
secret = response.plaintext.decode(
"utf-8"
) # assumes the original value was encoded with utf-8
elif key_manager == KeyManagementSystem.AWS_KMS.value:
"""
Only check the tokens which start with 'aws_kms/'. This prevents latency impact caused by checking all keys.
"""
encrypted_value = os.getenv(secret_name, None)
if encrypted_value is None:
raise Exception(
"AWS KMS - Encrypted Value of Key={} is None".format(
secret_name
)
)
# Decode the base64 encoded ciphertext
ciphertext_blob = base64.b64decode(encrypted_value)
# Set up the parameters for the decrypt call
params = {"CiphertextBlob": ciphertext_blob}
# Perform the decryption
response = client.decrypt(**params)
# Extract and decode the plaintext
plaintext = response["Plaintext"]
secret = plaintext.decode("utf-8")
if isinstance(secret, str):
secret = secret.strip()
elif key_manager == KeyManagementSystem.AWS_SECRET_MANAGER.value:
try:
get_secret_value_response = client.get_secret_value(
SecretId=secret_name
)
print_verbose(
f"get_secret_value_response: {get_secret_value_response}"
)
except Exception as e:
print_verbose(f"An error occurred - {str(e)}")
# For a list of exceptions thrown, see
# https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
raise e
# assume there is 1 secret per secret_name
secret_dict = json.loads(get_secret_value_response["SecretString"])
print_verbose(f"secret_dict: {secret_dict}")
for k, v in secret_dict.items():
secret = v
print_verbose(f"secret: {secret}")
elif key_manager == "local":
secret = os.getenv(secret_name)
else: # assume the default is infisicial client
secret = client.get_secret(secret_name).secret_value
except Exception as e: # check if it's in os.environ
verbose_logger.error(
f"Defaulting to os.environ value for key={secret_name}. An exception occurred - {str(e)}.\n\n{traceback.format_exc()}"
)
secret = os.getenv(secret_name)
try:
secret_value_as_bool = ast.literal_eval(secret)
if isinstance(secret_value_as_bool, bool):
return secret_value_as_bool
else:
return secret
except:
return secret
else:
secret = os.environ.get(secret_name)
try:
secret_value_as_bool = (
ast.literal_eval(secret) if secret is not None else None
)
if isinstance(secret_value_as_bool, bool):
return secret_value_as_bool
else:
return secret
except Exception:
if default_value is not None:
return default_value
return secret
except Exception as e:
if default_value is not None:
return default_value
else:
raise e
######## Streaming Class ############################
# wraps the completion stream to return the correct format for the model
# replicate/anthropic/cohere

2
tests/test_entrypoint.py
View file

@ -17,7 +17,7 @@ def test_decrypt_and_reset_env():
os.environ["DATABASE_URL"] = (
"aws_kms/AQICAHgwddjZ9xjVaZ9CNCG8smFU6FiQvfdrjL12DIqi9vUAQwHwF6U7caMgHQa6tK+TzaoMAAAAzjCBywYJKoZIhvcNAQcGoIG9MIG6AgEAMIG0BgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCmu+DVeKTm5tFZu6AIBEICBhnOFQYviL8JsciGk0bZsn9pfzeYWtNkVXEsl01AdgHBqT9UOZOI4ZC+T3wO/fXA7wdNF4o8ASPDbVZ34ZFdBs8xt4LKp9niufL30WYBkuuzz89ztly0jvE9pZ8L6BMw0ATTaMgIweVtVSDCeCzEb5PUPyxt4QayrlYHBGrNH5Aq/axFTe0La"
)
from litellm.proxy.secret_managers.aws_secret_manager import (
from litellm.secret_managers.aws_secret_manager import (
decrypt_and_reset_env_var,
)