diff --git a/docs/my-website/docs/batches.md b/docs/my-website/docs/batches.md index 898738f632a..144873928dc 100644 --- a/docs/my-website/docs/batches.md +++ b/docs/my-website/docs/batches.md @@ -222,4 +222,101 @@ curl http://0.0.0.0:4000/v1/batches?limit=2 \ -H "Content-Type: application/json" ``` -### [πŸ‘‰ Health Check Azure Batch models](./proxy/health.md#batch-models-azure-only) \ No newline at end of file +### [πŸ‘‰ Health Check Azure Batch models](./proxy/health.md#batch-models-azure-only) + + +### [BETA] Loadbalance Multiple Azure Deployments +In your config.yaml, set `enable_loadbalancing_on_batch_endpoints: true` + +```yaml +model_list: + - model_name: "batch-gpt-4o-mini" + litellm_params: + model: "azure/gpt-4o-mini" + api_key: os.environ/AZURE_API_KEY + api_base: os.environ/AZURE_API_BASE + model_info: + mode: batch + +litellm_settings: + enable_loadbalancing_on_batch_endpoints: true # πŸ‘ˆ KEY CHANGE +``` + +Note: This works on `{PROXY_BASE_URL}/v1/files` and `{PROXY_BASE_URL}/v1/batches`. +Note: Response is in the OpenAI-format. + +1. Upload a file + +Just set `model: batch-gpt-4o-mini` in your .jsonl. + +```bash +curl http://localhost:4000/v1/files \ + -H "Authorization: Bearer sk-1234" \ + -F purpose="batch" \ + -F file="@mydata.jsonl" +``` + +**Example File** + +Note: `model` should be your azure deployment name. + +```json +{"custom_id": "task-0", "method": "POST", "url": "/chat/completions", "body": {"model": "batch-gpt-4o-mini", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "When was Microsoft founded?"}]}} +{"custom_id": "task-1", "method": "POST", "url": "/chat/completions", "body": {"model": "batch-gpt-4o-mini", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "When was the first XBOX released?"}]}} +{"custom_id": "task-2", "method": "POST", "url": "/chat/completions", "body": {"model": "batch-gpt-4o-mini", "messages": [{"role": "system", "content": "You are an AI assistant that helps people find information."}, {"role": "user", "content": "What is Altair Basic?"}]}} +``` + +Expected Response (OpenAI-compatible) + +```bash +{"id":"file-f0be81f654454113a922da60acb0eea6",...} +``` + +2. Create a batch + +```bash +curl http://0.0.0.0:4000/v1/batches \ + -H "Authorization: Bearer $LITELLM_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "input_file_id": "file-f0be81f654454113a922da60acb0eea6", + "endpoint": "/v1/chat/completions", + "completion_window": "24h", + "model: "batch-gpt-4o-mini" + }' +``` + +Expected Response: + +```bash +{"id":"batch_94e43f0a-d805-477d-adf9-bbb9c50910ed",...} +``` + +3. Retrieve a batch + +```bash +curl http://0.0.0.0:4000/v1/batches/batch_94e43f0a-d805-477d-adf9-bbb9c50910ed \ + -H "Authorization: Bearer $LITELLM_API_KEY" \ + -H "Content-Type: application/json" \ +``` + + +Expected Response: + +``` +{"id":"batch_94e43f0a-d805-477d-adf9-bbb9c50910ed",...} +``` + +4. List batch + +```bash +curl http://0.0.0.0:4000/v1/batches?limit=2 \ + -H "Authorization: Bearer $LITELLM_API_KEY" \ + -H "Content-Type: application/json" +``` + +Expected Response: + +```bash +{"data":[{"id":"batch_R3V...} +``` \ No newline at end of file diff --git a/docs/my-website/docs/enterprise.md b/docs/my-website/docs/enterprise.md index cd83365ac3e..acc1331f90d 100644 --- a/docs/my-website/docs/enterprise.md +++ b/docs/my-website/docs/enterprise.md @@ -23,7 +23,7 @@ This covers: - βœ… [Audit Logs with retention policy](./proxy/enterprise#audit-logs) - βœ… [JWT-Auth](../docs/proxy/token_auth.md) - βœ… [Control available public, private routes (Restrict certain endpoints on proxy)](./proxy/enterprise#control-available-public-private-routes) - - βœ… [[BETA] AWS Key Manager v2 - Key Decryption](./proxy/enterprise#beta-aws-key-manager---key-decryption) + - βœ… [**Secret Managers** AWS Key Manager, Google Secret Manager, Azure Key](./secret) - βœ… IP address‑based access control lists - βœ… Track Request IP Address - βœ… [Use LiteLLM keys/authentication on Pass Through Endpoints](./proxy/pass_through#✨-enterprise---use-litellm-keysauthentication-on-pass-through-endpoints) diff --git a/docs/my-website/docs/providers/azure.md b/docs/my-website/docs/providers/azure.md index 381ecfed54f..ee5d9c3dbb9 100644 --- a/docs/my-website/docs/providers/azure.md +++ b/docs/my-website/docs/providers/azure.md @@ -533,3 +533,40 @@ response = litellm.completion( ) ``` +### Azure AD Token Refresh + + + + +```python +from litellm import completion +from azure.identity import DefaultAzureCredential, get_bearer_token_provider + +token_provider = get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default") + + +response = completion( + model = "azure/", # model = azure/ + api_base = "", # azure api base + api_version = "", # azure api version + azure_ad_token_provider=token_provider + messages = [{"role": "user", "content": "good morning"}], +) +``` + + + + +```yaml +model_list: + - model_name: gpt-3.5-turbo + litellm_params: + model: azure/your-deployment-name + api_base: https://openai-gpt-4-test-v-1.openai.azure.com/ + +litellm_settings: + enable_azure_ad_token_refresh: true # πŸ‘ˆ KEY CHANGE +``` + + + \ No newline at end of file diff --git a/docs/my-website/docs/providers/bedrock.md b/docs/my-website/docs/providers/bedrock.md index d8a10fd7585..06892659ed1 100644 --- a/docs/my-website/docs/providers/bedrock.md +++ b/docs/my-website/docs/providers/bedrock.md @@ -894,12 +894,17 @@ print(response) ## Supported AWS Bedrock Embedding Models -| Model Name | Function Call | -|----------------------|---------------------------------------------| -| Titan Embeddings V2 | `embedding(model="bedrock/amazon.titan-embed-text-v2:0", input=input)` | -| Titan Embeddings - V1 | `embedding(model="bedrock/amazon.titan-embed-text-v1", input=input)` | -| Cohere Embeddings - English | `embedding(model="bedrock/cohere.embed-english-v3", input=input)` | -| Cohere Embeddings - Multilingual | `embedding(model="bedrock/cohere.embed-multilingual-v3", input=input)` | +| Model Name | Usage | Supported Additional OpenAI params | +|----------------------|---------------------------------------------|-----| +| Titan Embeddings V2 | `embedding(model="bedrock/amazon.titan-embed-text-v2:0", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/amazon_titan_v2_transformation.py#L59) | +| Titan Embeddings - V1 | `embedding(model="bedrock/amazon.titan-embed-text-v1", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/amazon_titan_g1_transformation.py#L53) +| Titan Multimodal Embeddings | `embedding(model="bedrock/amazon.titan-embed-image-v1", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/amazon_titan_multimodal_transformation.py#L28) | +| Cohere Embeddings - English | `embedding(model="bedrock/cohere.embed-english-v3", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/cohere_transformation.py#L18) +| Cohere Embeddings - Multilingual | `embedding(model="bedrock/cohere.embed-multilingual-v3", input=input)` | [here](https://github.com/BerriAI/litellm/blob/f5905e100068e7a4d61441d7453d7cf5609c2121/litellm/llms/bedrock/embed/cohere_transformation.py#L18) + +### Advanced - [Drop Unsupported Params](https://docs.litellm.ai/docs/completion/drop_params#openai-proxy-usage) + +### Advanced - [Pass model/provider-specific Params](https://docs.litellm.ai/docs/completion/provider_specific_params#proxy-usage) ## Image Generation Use this for stable diffusion on bedrock diff --git a/docs/my-website/docs/proxy/enterprise.md b/docs/my-website/docs/proxy/enterprise.md index ac26b9e4a8e..9b117c7e2f1 100644 --- a/docs/my-website/docs/proxy/enterprise.md +++ b/docs/my-website/docs/proxy/enterprise.md @@ -17,6 +17,7 @@ Features: - βœ… [Audit Logs with retention policy](#audit-logs) - βœ… [JWT-Auth](../docs/proxy/token_auth.md) - βœ… [Control available public, private routes (Restrict certain endpoints on proxy)](#control-available-public-private-routes) + - βœ… [Control available public, private routes](#control-available-public-private-routes) - βœ… [[BETA] AWS Key Manager v2 - Key Decryption](#beta-aws-key-manager---key-decryption) - βœ… IP address‑based access control lists - βœ… Track Request IP Address diff --git a/docs/my-website/docs/proxy/reliability.md b/docs/my-website/docs/proxy/reliability.md index 5045ba36a59..4aa03228d39 100644 --- a/docs/my-website/docs/proxy/reliability.md +++ b/docs/my-website/docs/proxy/reliability.md @@ -31,7 +31,9 @@ model_list: api_base: https://openai-france-1234.openai.azure.com/ api_key: rpm: 1440 -routing_strategy: simple-shuffle # Literal["simple-shuffle", "least-busy", "usage-based-routing","latency-based-routing"], default="simple-shuffle" + +router_settings: + routing_strategy: simple-shuffle # Literal["simple-shuffle", "least-busy", "usage-based-routing","latency-based-routing"], default="simple-shuffle" model_group_alias: {"gpt-4": "gpt-3.5-turbo"} # all requests with `gpt-4` will be routed to models with `gpt-3.5-turbo` num_retries: 2 timeout: 30 # 30 seconds @@ -85,8 +87,6 @@ print(response) -Pass `metadata` as part of the request body - ```shell curl --location 'http://0.0.0.0:4000/chat/completions' \ --header 'Content-Type: application/json' \ diff --git a/docs/my-website/docs/routing.md b/docs/my-website/docs/routing.md index 651c0739567..c7c6c3c9704 100644 --- a/docs/my-website/docs/routing.md +++ b/docs/my-website/docs/routing.md @@ -25,6 +25,11 @@ If you want a server to load balance across different LLM APIs, use our [LiteLLM ### Quick Start +Loadbalance across multiple [azure](./providers/azure.md)/[bedrock](./providers/bedrock.md)/[provider](./providers/) deployments. LiteLLM will handle retrying in different regions if a call fails. + + + + ```python from litellm import Router @@ -84,6 +89,57 @@ response = await router.acompletion(model="gpt-4", print(response) ``` + + + +:::info + +See detailed proxy loadbalancing/fallback docs [here](./proxy/reliability.md) + +::: + +1. Setup model_list with multiple deployments +```yaml +model_list: + - model_name: gpt-3.5-turbo + litellm_params: + model: azure/ + api_base: + api_key: + - model_name: gpt-3.5-turbo + litellm_params: + model: azure/gpt-turbo-small-ca + api_base: https://my-endpoint-canada-berri992.openai.azure.com/ + api_key: + - model_name: gpt-3.5-turbo + litellm_params: + model: azure/gpt-turbo-large + api_base: https://openai-france-1234.openai.azure.com/ + api_key: +``` + +2. Start proxy + +```bash +litellm --config /path/to/config.yaml +``` + +3. Test it! + +```bash +curl -X POST 'http://0.0.0.0:4000/chat/completions' \ +-H 'Content-Type: application/json' \ +-H 'Authorization: Bearer sk-1234' \ +-d '{ + "model": "gpt-3.5-turbo", + "messages": [ + {"role": "user", "content": "Hi there!"} + ], + "mock_testing_rate_limit_error": true +}' +``` + + ### Available Endpoints - `router.completion()` - chat completions endpoint to call 100+ LLMs diff --git a/docs/my-website/docs/secret.md b/docs/my-website/docs/secret.md index c2b6774c0b1..db5ec6910c6 100644 --- a/docs/my-website/docs/secret.md +++ b/docs/my-website/docs/secret.md @@ -1,9 +1,22 @@ # Secret Manager -LiteLLM supports reading secrets from Azure Key Vault and Infisical +LiteLLM supports reading secrets from Azure Key Vault, Google Secret Manager -- AWS Key Managemenet Service +:::info + +✨ **This is an Enterprise Feature** + +[Enterprise Pricing](https://www.litellm.ai/#pricing) + +[Contact us here to get a free trial](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat) + +::: + +## Supported Secret Managers + +- AWS Key Management Service - AWS Secret Manager - [Azure Key Vault](#azure-key-vault) +- [Google Secret Manager](#google-secret-manager) - Google Key Management Service - [Infisical Secret Manager](#infisical-secret-manager) - [.env Files](#env-files) @@ -125,6 +138,45 @@ litellm --config /path/to/config.yaml [Quick Test Proxy](./proxy/quick_start#using-litellm-proxy---curl-request-openai-package-langchain-langchain-js) +## Google Secret Manager + +Support for [Google Secret Manager](https://cloud.google.com/security/products/secret-manager) + + +1. Save Google Secret Manager details in your environment + +```shell +GOOGLE_SECRET_MANAGER_PROJECT_ID="your-project-id-on-gcp" # example: adroit-crow-413218 +``` + +Optional Params + +```shell +export GOOGLE_SECRET_MANAGER_REFRESH_INTERVAL = "" # (int) defaults to 86400 +export GOOGLE_SECRET_MANAGER_ALWAYS_READ_SECRET_MANAGER = "" # (str) set to "true" if you want to always read from google secret manager without using in memory caching. NOT RECOMMENDED in PROD +``` + +2. Add to proxy config.yaml +```yaml +model_list: + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + api_key: os.environ/OPENAI_API_KEY # this will be read from Google Secret Manager + +general_settings: + key_management_system: "google_secret_manager" +``` + +You can now test this by starting your proxy: +```bash +litellm --config /path/to/config.yaml +``` + +[Quick Test Proxy](./proxy/quick_start#using-litellm-proxy---curl-request-openai-package-langchain-langchain-js) + + ## Google Key Management Service Use encrypted keys from Google KMS on the proxy diff --git a/docs/my-website/docusaurus.config.js b/docs/my-website/docusaurus.config.js index 33ac8cd8463..8ede990b176 100644 --- a/docs/my-website/docusaurus.config.js +++ b/docs/my-website/docusaurus.config.js @@ -26,26 +26,33 @@ const config = { defaultLocale: 'en', locales: ['en'], }, - - plugins: [ + themes: [ [ - require.resolve("@getcanary/docusaurus-pagefind"), + require.resolve("@getcanary/docusaurus-theme-search-pagefind"), { - indexOnly: true, styles: { "--canary-color-primary-c": 0.1, "--canary-color-primary-h": 270, }, pagefind: { ranking: { - pageLength: 0.9, + // https://pagefind.app/docs/ranking + pageLength: 0.0, termFrequency: 1.0, termSimilarity: 1.0, - termSaturation: 1.5, + termSaturation: 2.0, } - } + }, + tabs: [ + { name: "All", pattern: "**/*" }, + { name: "Providers", pattern: "/docs/providers/**" }, + { name: "Proxy", pattern: "/docs/proxy/**" } + ], + indexOnly: true, }, ], + ], + plugins: [ [ '@docusaurus/plugin-ideal-image', { diff --git a/docs/my-website/package-lock.json b/docs/my-website/package-lock.json index d5f92af5c2e..a79c1f85925 100644 --- a/docs/my-website/package-lock.json +++ b/docs/my-website/package-lock.json @@ -12,8 +12,8 @@ "@docusaurus/plugin-google-gtag": "^2.4.1", "@docusaurus/plugin-ideal-image": "^2.4.1", "@docusaurus/preset-classic": "2.4.1", - "@getcanary/docusaurus-pagefind": "^0.0.12", - "@getcanary/web": "^0.0.55", + "@getcanary/docusaurus-theme-search-pagefind": "^0.0.13", + "@getcanary/web": "^0.0.101", "@mdx-js/react": "^1.6.22", "clsx": "^1.2.1", "docusaurus": "^1.14.7", @@ -3621,31 +3621,31 @@ } }, "node_modules/@floating-ui/core": { - "version": "1.6.5", - "resolved": "https://registry.npmjs.org/@floating-ui/core/-/core-1.6.5.tgz", - "integrity": "sha512-8GrTWmoFhm5BsMZOTHeGD2/0FLKLQQHvO/ZmQga4tKempYRLz8aqJGqXVuQgisnMObq2YZ2SgkwctN1LOOxcqA==", + "version": "1.6.7", + "resolved": "https://registry.npmjs.org/@floating-ui/core/-/core-1.6.7.tgz", + "integrity": "sha512-yDzVT/Lm101nQ5TCVeK65LtdN7Tj4Qpr9RTXJ2vPFLqtLxwOrpoxAHAJI8J3yYWUc40J0BDBheaitK5SJmno2g==", "dependencies": { - "@floating-ui/utils": "^0.2.5" + "@floating-ui/utils": "^0.2.7" } }, "node_modules/@floating-ui/dom": { - "version": "1.6.8", - "resolved": "https://registry.npmjs.org/@floating-ui/dom/-/dom-1.6.8.tgz", - "integrity": "sha512-kx62rP19VZ767Q653wsP1XZCGIirkE09E0QUGNYTM/ttbbQHqcGPdSfWFxUyyNLc/W6aoJRBajOSXhP6GXjC0Q==", + "version": "1.6.10", + "resolved": "https://registry.npmjs.org/@floating-ui/dom/-/dom-1.6.10.tgz", + "integrity": "sha512-fskgCFv8J8OamCmyun8MfjB1Olfn+uZKjOKZ0vhYF3gRmEUXcGOjxWL8bBr7i4kIuPZ2KD2S3EUIOxnjC8kl2A==", "dependencies": { "@floating-ui/core": "^1.6.0", - "@floating-ui/utils": "^0.2.5" + "@floating-ui/utils": "^0.2.7" } }, "node_modules/@floating-ui/utils": { - "version": "0.2.5", - "resolved": "https://registry.npmjs.org/@floating-ui/utils/-/utils-0.2.5.tgz", - "integrity": "sha512-sTcG+QZ6fdEUObICavU+aB3Mp8HY4n14wYHdxK4fXjPmv3PXZZeY5RaguJmGyeH/CJQhX3fqKUtS4qc1LoHwhQ==" + "version": "0.2.7", + "resolved": "https://registry.npmjs.org/@floating-ui/utils/-/utils-0.2.7.tgz", + "integrity": "sha512-X8R8Oj771YRl/w+c1HqAC1szL8zWQRwFvgDwT129k9ACdBoud/+/rX9V0qiMl6LWUdP9voC2nDVZYPMQQsb6eA==" }, - "node_modules/@getcanary/docusaurus-pagefind": { - "version": "0.0.12", - "resolved": "https://registry.npmjs.org/@getcanary/docusaurus-pagefind/-/docusaurus-pagefind-0.0.12.tgz", - "integrity": "sha512-F0OQ0Lb/GltewDEr0w+BgPbNyYpzAQZ/TtuG5rbtC3PnrOL+9pDMe/Gs0kE8AuY1uEd/YQOKr61rbY/k7kkFig==", + "node_modules/@getcanary/docusaurus-theme-search-pagefind": { + "version": "0.0.13", + "resolved": "https://registry.npmjs.org/@getcanary/docusaurus-theme-search-pagefind/-/docusaurus-theme-search-pagefind-0.0.13.tgz", + "integrity": "sha512-w/Jz4q98cJQQtTAq8KCDI1jAW1HnHpsYpFoXaIjplLt+RoBiUKkCXocxDEI0yfKzRtmkH77dvf6LyJIdqMDWZg==", "dependencies": { "cli-progress": "^3.12.0", "micromatch": "^4.0.7", @@ -3659,17 +3659,61 @@ } }, "node_modules/@getcanary/web": { - "version": "0.0.55", - "resolved": "https://registry.npmjs.org/@getcanary/web/-/web-0.0.55.tgz", - "integrity": "sha512-DjIhTMeuLZaHT+/h+O6Keg9Gb58frPURpM4lkKrN/wmRMoCnOuly3oXIH2X37YhAoHXi4udDRJ60mtD0UZy0uw==", + "version": "0.0.101", + "resolved": "https://registry.npmjs.org/@getcanary/web/-/web-0.0.101.tgz", + "integrity": "sha512-/GeoSpVKq0w2HmjeF855zOeKE8a6cFcdtAsLHC0/ATJTkrAjAuAaTn5We0WX2w0UGBdzHuc94bYXJCklbTEoCg==", "dependencies": { "@floating-ui/dom": "^1.6.8", "@lit-labs/observers": "^2.0.2", "@lit/context": "^1.1.2", "@lit/task": "^1.0.1", - "highlight.js": "^11.10.0", + "@xstate/store": "^2.1.0", "lit": "^3.1.4", - "marked": "^13.0.2" + "marked": "^14.0.0", + "marked-footnote": "^1.2.2", + "picomatch": "^4.0.2", + "prismjs": "^1.29.0" + } + }, + "node_modules/@getcanary/web/node_modules/@xstate/store": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/@xstate/store/-/store-2.3.0.tgz", + "integrity": "sha512-vdIlbI4TxtadYNrGrKezrR0Y0Hid2xYBm5ohk403ZsaU7p6IjCCaacF8t8tp9qUZnYuLYzldBMhiGwBuNzmCoA==", + "peerDependencies": { + "react": "^18.2.0", + "solid-js": "^1.7.6" + }, + "peerDependenciesMeta": { + "react": { + "optional": true + }, + "solid-js": { + "optional": true + } + } + }, + "node_modules/@getcanary/web/node_modules/picomatch": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz", + "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/@getcanary/web/node_modules/react": { + "version": "18.3.1", + "resolved": "https://registry.npmjs.org/react/-/react-18.3.1.tgz", + "integrity": "sha512-wS+hAgJShR0KhEvPJArfuPVN1+Hz1t0Y6n5jLrGQbkb4urgPE/0Rve+1kMB1v/oWgHgm4WIcV+i7F2pTVj+2iQ==", + "optional": true, + "peer": true, + "dependencies": { + "loose-envify": "^1.1.0" + }, + "engines": { + "node": ">=0.10.0" } }, "node_modules/@hapi/hoek": { @@ -3770,17 +3814,18 @@ "integrity": "sha512-Vo+PSpZG2/fmgmiNzYK9qWRh8h/CHrwD0mo1h1DzL4yzHNSfWYujGTYsWGreD000gcgmZ7K4Ys6Tx9TxtsKdDw==" }, "node_modules/@lit-labs/observers": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/@lit-labs/observers/-/observers-2.0.2.tgz", - "integrity": "sha512-eZb5+W9Cb0e/Y5m1DNxBSGTvGB2TAVTGMnTxL/IzFhPQEcZIAHewW1eVBhN8W07A5tirRaAmmF6fGL1V20p3gQ==", + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/@lit-labs/observers/-/observers-2.0.3.tgz", + "integrity": "sha512-CeftEJ2TId9iohDJHLjUXiSBVndqjIBaALjeTt8OmgWLh2dnIzwlj4WtPCiJw15uR1s6D6wyCsw0AoJC5/9QXw==", "dependencies": { - "@lit/reactive-element": "^1.0.0 || ^2.0.0" + "@lit/reactive-element": "^1.0.0 || ^2.0.0", + "lit-html": "^3.2.0" } }, "node_modules/@lit-labs/ssr-dom-shim": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/@lit-labs/ssr-dom-shim/-/ssr-dom-shim-1.2.0.tgz", - "integrity": "sha512-yWJKmpGE6lUURKAaIltoPIE/wrbY3TEkqQt+X0m+7fQNnAv0keydnYvbiJFP1PnMhizmIWRWOG5KLhYyc/xl+g==" + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/@lit-labs/ssr-dom-shim/-/ssr-dom-shim-1.2.1.tgz", + "integrity": "sha512-wx4aBmgeGvFmOKucFKY+8VFJSYZxs9poN3SDNQFF6lT6NrQUnHiPB2PWz2sc4ieEcAaYYzN+1uWahEeTq2aRIQ==" }, "node_modules/@lit/context": { "version": "1.1.2", @@ -3994,9 +4039,9 @@ } }, "node_modules/@pagefind/darwin-arm64": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@pagefind/darwin-arm64/-/darwin-arm64-1.1.0.tgz", - "integrity": "sha512-SLsXNLtSilGZjvqis8sX42fBWsWAVkcDh1oerxwqbac84HbiwxpxOC2jm8hRwcR0Z55HPZPWO77XeRix/8GwTg==", + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@pagefind/darwin-arm64/-/darwin-arm64-1.1.1.tgz", + "integrity": "sha512-tZ9tysUmQpFs2EqWG2+E1gc+opDAhSyZSsgKmFzhnWfkK02YHZhvL5XJXEZDqYy3s1FAKhwjTg8XDxneuBlDZQ==", "cpu": [ "arm64" ], @@ -4006,9 +4051,9 @@ ] }, "node_modules/@pagefind/darwin-x64": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@pagefind/darwin-x64/-/darwin-x64-1.1.0.tgz", - "integrity": "sha512-QjQSE/L5oS1C8N8GdljGaWtjCBMgMtfrPAoiCmINTu9Y9dp0ggAyXvF8K7Qg3VyIMYJ6v8vg2PN7Z3b+AaAqUA==", + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@pagefind/darwin-x64/-/darwin-x64-1.1.1.tgz", + "integrity": "sha512-ChohLQ39dLwaxQv0jIQB/SavP3TM5K5ENfDTqIdzLkmfs3+JlzSDyQKcJFjTHYcCzQOZVeieeGq8PdqvLJxJxQ==", "cpu": [ "x64" ], @@ -4018,9 +4063,9 @@ ] }, "node_modules/@pagefind/linux-arm64": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@pagefind/linux-arm64/-/linux-arm64-1.1.0.tgz", - "integrity": "sha512-8zjYCa2BtNEL7KnXtysPtBELCyv5DSQ4yHeK/nsEq6w4ToAMTBl0K06khqxdSGgjMSwwrxvLzq3so0LC5Q14dA==", + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@pagefind/linux-arm64/-/linux-arm64-1.1.1.tgz", + "integrity": "sha512-H5P6wDoCoAbdsWp0Zx0DxnLUrwTGWGLu/VI1rcN2CyFdY2EGSvPQsbGBMrseKRNuIrJDFtxHHHyjZ7UbzaM9EA==", "cpu": [ "arm64" ], @@ -4030,9 +4075,9 @@ ] }, "node_modules/@pagefind/linux-x64": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@pagefind/linux-x64/-/linux-x64-1.1.0.tgz", - "integrity": "sha512-4lsg6VB7A6PWTwaP8oSmXV4O9H0IHX7AlwTDcfyT+YJo/sPXOVjqycD5cdBgqNLfUk8B9bkWcTDCRmJbHrKeCw==", + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@pagefind/linux-x64/-/linux-x64-1.1.1.tgz", + "integrity": "sha512-yJs7tTYbL2MI3HT+ngs9E1BfUbY9M4/YzA0yEM5xBo4Xl8Yu8Qg2xZTOQ1/F6gwvMrjCUFo8EoACs6LRDhtMrQ==", "cpu": [ "x64" ], @@ -4042,9 +4087,9 @@ ] }, "node_modules/@pagefind/windows-x64": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@pagefind/windows-x64/-/windows-x64-1.1.0.tgz", - "integrity": "sha512-OboCM76BcMKT9IoSfZuFhiqMRgTde8x4qDDvKulFmycgiJrlL5WnIqBHJLQxZq+o2KyZpoHF97iwsGAm8c32sQ==", + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@pagefind/windows-x64/-/windows-x64-1.1.1.tgz", + "integrity": "sha512-b7/qPqgIl+lMzkQ8fJt51SfguB396xbIIR+VZ3YrL2tLuyifDJ1wL5mEm+ddmHxJ2Fki340paPcDan9en5OmAw==", "cpu": [ "x64" ], @@ -12261,14 +12306,6 @@ "resolved": "https://registry.npmjs.org/hex-color-regex/-/hex-color-regex-1.1.0.tgz", "integrity": "sha512-l9sfDFsuqtOqKDsQdqrMRk0U85RZc0RtOR9yPI7mRVOa4FsR/BVnZ0shmQRM96Ji99kYZP/7hn1cedc1+ApsTQ==" }, - "node_modules/highlight.js": { - "version": "11.10.0", - "resolved": "https://registry.npmjs.org/highlight.js/-/highlight.js-11.10.0.tgz", - "integrity": "sha512-SYVnVFswQER+zu1laSya563s+F8VDGt7o35d4utbamowvUNLLMovFqwCLSocpZTz3MgaSRA1IbqRWZv97dtErQ==", - "engines": { - "node": ">=12.0.0" - } - }, "node_modules/history": { "version": "4.10.1", "resolved": "https://registry.npmjs.org/history/-/history-4.10.1.tgz", @@ -14266,29 +14303,29 @@ "integrity": "sha512-3mk/Zag0+IJxeDrxSgaDPy4zZ3w05PRZeJNnlWhzFz5OkX49J4krc+A8X2d2M69vGMBEX0uyl8M+W+8gH+kBqQ==" }, "node_modules/lit": { - "version": "3.1.4", - "resolved": "https://registry.npmjs.org/lit/-/lit-3.1.4.tgz", - "integrity": "sha512-q6qKnKXHy2g1kjBaNfcoLlgbI3+aSOZ9Q4tiGa9bGYXq5RBXxkVTqTIVmP2VWMp29L4GyvCFm8ZQ2o56eUAMyA==", + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/lit/-/lit-3.2.0.tgz", + "integrity": "sha512-s6tI33Lf6VpDu7u4YqsSX78D28bYQulM+VAzsGch4fx2H0eLZnJsUBsPWmGYSGoKDNbjtRv02rio1o+UdPVwvw==", "dependencies": { "@lit/reactive-element": "^2.0.4", - "lit-element": "^4.0.4", - "lit-html": "^3.1.2" + "lit-element": "^4.1.0", + "lit-html": "^3.2.0" } }, "node_modules/lit-element": { - "version": "4.0.6", - "resolved": "https://registry.npmjs.org/lit-element/-/lit-element-4.0.6.tgz", - "integrity": "sha512-U4sdJ3CSQip7sLGZ/uJskO5hGiqtlpxndsLr6mt3IQIjheg93UKYeGQjWMRql1s/cXNOaRrCzC2FQwjIwSUqkg==", + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/lit-element/-/lit-element-4.1.0.tgz", + "integrity": "sha512-gSejRUQJuMQjV2Z59KAS/D4iElUhwKpIyJvZ9w+DIagIQjfJnhR20h2Q5ddpzXGS+fF0tMZ/xEYGMnKmaI/iww==", "dependencies": { "@lit-labs/ssr-dom-shim": "^1.2.0", "@lit/reactive-element": "^2.0.4", - "lit-html": "^3.1.2" + "lit-html": "^3.2.0" } }, "node_modules/lit-html": { - "version": "3.1.4", - "resolved": "https://registry.npmjs.org/lit-html/-/lit-html-3.1.4.tgz", - "integrity": "sha512-yKKO2uVv7zYFHlWMfZmqc+4hkmSbFp8jgjdZY9vvR9jr4J8fH6FUMXhr+ljfELgmjpvlF7Z1SJ5n5/Jeqtc9YA==", + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/lit-html/-/lit-html-3.2.0.tgz", + "integrity": "sha512-pwT/HwoxqI9FggTrYVarkBKFN9MlTUpLrDHubTmW4SrkL3kkqW5gxwbxMMUnbbRHBC0WTZnYHcjDSCM559VyfA==", "dependencies": { "@types/trusted-types": "^2.0.2" } @@ -14691,9 +14728,9 @@ } }, "node_modules/marked": { - "version": "13.0.3", - "resolved": "https://registry.npmjs.org/marked/-/marked-13.0.3.tgz", - "integrity": "sha512-rqRix3/TWzE9rIoFGIn8JmsVfhiuC8VIQ8IdX5TfzmeBucdY05/0UlzKaw0eVtpcN/OdVFpBk7CjKGo9iHJ/zA==", + "version": "14.1.0", + "resolved": "https://registry.npmjs.org/marked/-/marked-14.1.0.tgz", + "integrity": "sha512-P93GikH/Pde0hM5TAXEd8I4JAYi8IB03n8qzW8Bh1BIEFpEyBoYxi/XWZA53LSpTeLBiMQOoSMj0u5E/tiVYTA==", "bin": { "marked": "bin/marked.js" }, @@ -14701,6 +14738,14 @@ "node": ">= 18" } }, + "node_modules/marked-footnote": { + "version": "1.2.3", + "resolved": "https://registry.npmjs.org/marked-footnote/-/marked-footnote-1.2.3.tgz", + "integrity": "sha512-ATIAPjuTlaNx3T139KNNy5X16BaYauxFEAGB0DtjonxIwRQ2DExdP6tAvUuG7UTkMGPOf6OjgIKhwXpSv7MoAg==", + "peerDependencies": { + "marked": ">=7.0.0" + } + }, "node_modules/math-random": { "version": "1.0.4", "resolved": "https://registry.npmjs.org/math-random/-/math-random-1.0.4.tgz", @@ -15917,18 +15962,18 @@ } }, "node_modules/pagefind": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/pagefind/-/pagefind-1.1.0.tgz", - "integrity": "sha512-1nmj0/vfYcMxNEQj0YDRp6bTVv9hI7HLdPhK/vBBYlrnwjATndQvHyicj5Y7pUHrpCFZpFnLVQXIF829tpFmaw==", + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/pagefind/-/pagefind-1.1.1.tgz", + "integrity": "sha512-U2YR0dQN5B2fbIXrLtt/UXNS0yWSSYfePaad1KcBPTi0p+zRtsVjwmoPaMQgTks5DnHNbmDxyJUL5TGaLljK3A==", "bin": { "pagefind": "lib/runner/bin.cjs" }, "optionalDependencies": { - "@pagefind/darwin-arm64": "1.1.0", - "@pagefind/darwin-x64": "1.1.0", - "@pagefind/linux-arm64": "1.1.0", - "@pagefind/linux-x64": "1.1.0", - "@pagefind/windows-x64": "1.1.0" + "@pagefind/darwin-arm64": "1.1.1", + "@pagefind/darwin-x64": "1.1.1", + "@pagefind/linux-arm64": "1.1.1", + "@pagefind/linux-x64": "1.1.1", + "@pagefind/windows-x64": "1.1.1" } }, "node_modules/param-case": { diff --git a/docs/my-website/package.json b/docs/my-website/package.json index ca010f0cf69..b0c824dab6f 100644 --- a/docs/my-website/package.json +++ b/docs/my-website/package.json @@ -18,8 +18,8 @@ "@docusaurus/plugin-google-gtag": "^2.4.1", "@docusaurus/plugin-ideal-image": "^2.4.1", "@docusaurus/preset-classic": "2.4.1", - "@getcanary/docusaurus-pagefind": "^0.0.12", - "@getcanary/web": "^0.0.55", + "@getcanary/docusaurus-theme-search-pagefind": "^0.0.13", + "@getcanary/web": "^0.0.101", "@mdx-js/react": "^1.6.22", "clsx": "^1.2.1", "docusaurus": "^1.14.7", diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 048b04171c5..1687872ca4b 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -44,7 +44,7 @@ const sidebars = { "proxy/reliability", { type: "category", - label: "Use with Vertex, Bedrock, Cohere SDK", + label: "Use with Provider SDKs", items: [ "pass_through/vertex_ai", "pass_through/google_ai_studio", diff --git a/docs/my-website/src/theme/SearchBar.js b/docs/my-website/src/theme/SearchBar.js index 66c8c2b5c3d..0d6f1abf214 100644 --- a/docs/my-website/src/theme/SearchBar.js +++ b/docs/my-website/src/theme/SearchBar.js @@ -6,7 +6,7 @@ import { usePluginData } from "@docusaurus/useGlobalData"; export default function SearchBarWrapper(props) { const { siteConfig } = useDocusaurusContext(); - const { options } = usePluginData("docusaurus-plugin-pagefind-canary"); + const { options } = usePluginData("docusaurus-theme-search-pagefind"); const [path, setPath] = React.useState(""); const [loaded, setLoaded] = React.useState(false); @@ -24,10 +24,7 @@ export default function SearchBarWrapper(props) { import("@getcanary/web/components/canary-content"), import("@getcanary/web/components/canary-search"), import("@getcanary/web/components/canary-search-input"), - import("@getcanary/web/components/canary-search-results-group"), - import("@getcanary/web/components/canary-footer"), - import("@getcanary/web/components/canary-callout-calendly"), - import("@getcanary/web/components/canary-callout-discord"), + import("@getcanary/web/components/canary-search-results-tabs"), ]) .then(() => setLoaded(true)) .catch(console.error); @@ -57,32 +54,18 @@ export default function SearchBarWrapper(props) { ) : ( - + - + - - - + - @@ -92,4 +75,4 @@ export default function SearchBarWrapper(props) { ); -} +} \ No newline at end of file diff --git a/litellm/__init__.py b/litellm/__init__.py index 2c68fa9af3f..5af751c3fee 100644 --- a/litellm/__init__.py +++ b/litellm/__init__.py @@ -834,7 +834,6 @@ from .utils import ( decode, _calculate_retry_after, _should_retry, - get_secret, get_supported_openai_params, get_api_base, get_first_chars_messages, diff --git a/litellm/batches/main.py b/litellm/batches/main.py index 9489e09cbe8..e927a18b666 100644 --- a/litellm/batches/main.py +++ b/litellm/batches/main.py @@ -22,6 +22,7 @@ import litellm from litellm import client from litellm.llms.azure import AzureBatchesAPI from litellm.llms.openai import OpenAIBatchesAPI +from litellm.secret_managers.main import get_secret from litellm.types.llms.openai import ( Batch, CancelBatchRequest, @@ -34,7 +35,7 @@ from litellm.types.llms.openai import ( RetrieveBatchRequest, ) from litellm.types.router import GenericLiteLLMParams -from litellm.utils import get_secret, supports_httpx_timeout +from litellm.utils import supports_httpx_timeout ####### ENVIRONMENT VARIABLES ################### openai_batches_instance = OpenAIBatchesAPI() diff --git a/litellm/fine_tuning/main.py b/litellm/fine_tuning/main.py index abf28285782..81b075f5176 100644 --- a/litellm/fine_tuning/main.py +++ b/litellm/fine_tuning/main.py @@ -17,7 +17,6 @@ from typing import Any, Coroutine, Dict, Literal, Optional, Union import httpx import litellm -from litellm import get_secret from litellm._logging import verbose_logger from litellm.llms.fine_tuning_apis.azure import AzureOpenAIFineTuningAPI from litellm.llms.fine_tuning_apis.openai import ( @@ -26,6 +25,7 @@ from litellm.llms.fine_tuning_apis.openai import ( OpenAIFineTuningAPI, ) from litellm.llms.fine_tuning_apis.vertex_ai import VertexFineTuningAPI +from litellm.secret_managers.main import get_secret from litellm.types.llms.openai import Hyperparameters from litellm.types.router import * from litellm.utils import supports_httpx_timeout diff --git a/litellm/integrations/gcs_bucket_base.py b/litellm/integrations/gcs_bucket_base.py index 2f34205ce31..073f0f265c9 100644 --- a/litellm/integrations/gcs_bucket_base.py +++ b/litellm/integrations/gcs_bucket_base.py @@ -52,6 +52,27 @@ class GCSBucketBase(CustomLogger): return headers + def sync_construct_request_headers(self) -> Dict[str, str]: + from litellm import vertex_chat_completion + + auth_header, _ = vertex_chat_completion._get_token_and_url( + model="gcs-bucket", + vertex_credentials=self.path_service_account_json, + vertex_project=None, + vertex_location=None, + gemini_api_key=None, + stream=None, + custom_llm_provider="vertex_ai", + api_base=None, + ) + verbose_logger.debug("constructed auth_header %s", auth_header) + headers = { + "Authorization": f"Bearer {auth_header}", # auth_header + "Content-Type": "application/json", + } + + return headers + async def download_gcs_object(self, object_name): """ Download an object from GCS. diff --git a/litellm/llms/base_aws_llm.py b/litellm/llms/base_aws_llm.py index 8de42eda739..7449dc2d7e8 100644 --- a/litellm/llms/base_aws_llm.py +++ b/litellm/llms/base_aws_llm.py @@ -5,7 +5,7 @@ import httpx from litellm._logging import verbose_logger from litellm.caching import DualCache, InMemoryCache -from litellm.utils import get_secret +from litellm.secret_managers.main import get_secret from .base import BaseLLM diff --git a/litellm/llms/bedrock/common_utils.py b/litellm/llms/bedrock/common_utils.py index 19a4f098601..f2032d110b3 100644 --- a/litellm/llms/bedrock/common_utils.py +++ b/litellm/llms/bedrock/common_utils.py @@ -10,7 +10,7 @@ from typing import List, Optional, Union import httpx import litellm -from litellm import get_secret +from litellm.secret_managers.main import get_secret class BedrockError(Exception): diff --git a/litellm/llms/bedrock/embed/embedding.py b/litellm/llms/bedrock/embed/embedding.py index 6585ec4f2ce..6398c2c3412 100644 --- a/litellm/llms/bedrock/embed/embedding.py +++ b/litellm/llms/bedrock/embed/embedding.py @@ -11,7 +11,6 @@ from typing import Any, Callable, List, Literal, Optional, Tuple, Union import httpx import litellm -from litellm import get_secret from litellm.llms.cohere.embed import embedding as cohere_embedding from litellm.llms.custom_httpx.http_handler import ( AsyncHTTPHandler, @@ -19,6 +18,7 @@ from litellm.llms.custom_httpx.http_handler import ( _get_async_httpx_client, _get_httpx_client, ) +from litellm.secret_managers.main import get_secret from litellm.types.llms.bedrock import AmazonEmbeddingRequest, CohereEmbeddingRequest from litellm.types.utils import Embedding, EmbeddingResponse, Usage diff --git a/litellm/proxy/_types.py b/litellm/proxy/_types.py index 39f65ac2dfa..dbaea990f61 100644 --- a/litellm/proxy/_types.py +++ b/litellm/proxy/_types.py @@ -1063,6 +1063,7 @@ class KeyManagementSystem(enum.Enum): GOOGLE_KMS = "google_kms" AZURE_KEY_VAULT = "azure_key_vault" AWS_SECRET_MANAGER = "aws_secret_manager" + GOOGLE_SECRET_MANAGER = "google_secret_manager" LOCAL = "local" AWS_KMS = "aws_kms" @@ -1638,6 +1639,7 @@ class AllCallbacks(LiteLLMBase): "LAGO_API_BASE", "LAGO_API_KEY", "LAGO_API_EVENT_CODE", + "LAGO_API_CHARGE_BY", ], ui_callback_name="Lago Billing", ) diff --git a/litellm/proxy/auth/rds_iam_token.py b/litellm/proxy/auth/rds_iam_token.py index ec3a424b9ff..f8362158457 100644 --- a/litellm/proxy/auth/rds_iam_token.py +++ b/litellm/proxy/auth/rds_iam_token.py @@ -14,7 +14,7 @@ def init_rds_client( aws_web_identity_token: Optional[str] = None, timeout: Optional[Union[float, httpx.Timeout]] = None, ): - from litellm import get_secret + from litellm.secret_managers.main import get_secret # check for custom AWS_REGION_NAME and use it if not passed to init_bedrock_client litellm_aws_region_name = get_secret("AWS_REGION_NAME", None) diff --git a/litellm/proxy/db/dynamo_db.py b/litellm/proxy/db/dynamo_db.py index 6056a61e25e..09e14b46057 100644 --- a/litellm/proxy/db/dynamo_db.py +++ b/litellm/proxy/db/dynamo_db.py @@ -1,16 +1,17 @@ -from litellm.proxy.db.base_client import CustomDB -from litellm.proxy._types import ( - DynamoDBArgs, - LiteLLM_VerificationToken, - LiteLLM_Config, - LiteLLM_UserTable, -) -from litellm.proxy.utils import hash_token -from litellm import get_secret -from typing import Any, List, Literal, Optional, Union import json from datetime import datetime +from typing import Any, List, Literal, Optional, Union + from litellm._logging import verbose_proxy_logger +from litellm.proxy._types import ( + DynamoDBArgs, + LiteLLM_Config, + LiteLLM_UserTable, + LiteLLM_VerificationToken, +) +from litellm.proxy.db.base_client import CustomDB +from litellm.proxy.utils import hash_token +from litellm.secret_managers.main import get_secret class DynamoDBWrapper(CustomDB): @@ -21,19 +22,19 @@ class DynamoDBWrapper(CustomDB): def __init__(self, database_arguments: DynamoDBArgs): from aiodynamo.client import Client from aiodynamo.credentials import Credentials, StaticCredentials + from aiodynamo.expressions import F, UpdateExpression, Value + from aiodynamo.http.aiohttp import AIOHTTP from aiodynamo.http.httpx import HTTPX from aiodynamo.models import ( - Throughput, KeySchema, KeySpec, KeyType, PayPerRequest, + ReturnValues, + Throughput, ) - from yarl import URL - from aiodynamo.expressions import UpdateExpression, F, Value - from aiodynamo.models import ReturnValues - from aiodynamo.http.aiohttp import AIOHTTP from aiohttp import ClientSession + from yarl import URL self.throughput_type = None if database_arguments.billing_mode == "PAY_PER_REQUEST": @@ -59,7 +60,9 @@ class DynamoDBWrapper(CustomDB): verbose_proxy_logger.debug( f"DynamoDB: setting env vars based on arn={self.database_arguments.aws_role_name}" ) - import boto3, os + import os + + import boto3 sts_client = boto3.client("sts") @@ -92,22 +95,22 @@ class DynamoDBWrapper(CustomDB): """ Connect to DB, and creating / updating any tables """ + import aiohttp from aiodynamo.client import Client from aiodynamo.credentials import Credentials, StaticCredentials + from aiodynamo.expressions import F, UpdateExpression, Value + from aiodynamo.http.aiohttp import AIOHTTP from aiodynamo.http.httpx import HTTPX from aiodynamo.models import ( - Throughput, KeySchema, KeySpec, KeyType, PayPerRequest, + ReturnValues, + Throughput, ) - from yarl import URL - from aiodynamo.expressions import UpdateExpression, F, Value - from aiodynamo.models import ReturnValues - from aiodynamo.http.aiohttp import AIOHTTP from aiohttp import ClientSession - import aiohttp + from yarl import URL verbose_proxy_logger.debug("DynamoDB Wrapper - Attempting to connect") self.set_env_vars_based_on_arn() @@ -192,22 +195,22 @@ class DynamoDBWrapper(CustomDB): async def insert_data( self, value: Any, table_name: Literal["user", "key", "config", "spend"] ): + import aiohttp from aiodynamo.client import Client from aiodynamo.credentials import Credentials, StaticCredentials + from aiodynamo.expressions import F, UpdateExpression, Value + from aiodynamo.http.aiohttp import AIOHTTP from aiodynamo.http.httpx import HTTPX from aiodynamo.models import ( - Throughput, KeySchema, KeySpec, KeyType, PayPerRequest, + ReturnValues, + Throughput, ) - from yarl import URL - from aiodynamo.expressions import UpdateExpression, F, Value - from aiodynamo.models import ReturnValues - from aiodynamo.http.aiohttp import AIOHTTP from aiohttp import ClientSession - import aiohttp + from yarl import URL self.set_env_vars_based_on_arn() @@ -237,22 +240,22 @@ class DynamoDBWrapper(CustomDB): return await table.put_item(item=value, return_values=ReturnValues.all_old) async def get_data(self, key: str, table_name: Literal["user", "key", "config"]): + import aiohttp from aiodynamo.client import Client from aiodynamo.credentials import Credentials, StaticCredentials + from aiodynamo.expressions import F, UpdateExpression, Value + from aiodynamo.http.aiohttp import AIOHTTP from aiodynamo.http.httpx import HTTPX from aiodynamo.models import ( - Throughput, KeySchema, KeySpec, KeyType, PayPerRequest, + ReturnValues, + Throughput, ) - from yarl import URL - from aiodynamo.expressions import UpdateExpression, F, Value - from aiodynamo.models import ReturnValues - from aiodynamo.http.aiohttp import AIOHTTP from aiohttp import ClientSession - import aiohttp + from yarl import URL self.set_env_vars_based_on_arn() @@ -311,22 +314,22 @@ class DynamoDBWrapper(CustomDB): self, key: str, value: dict, table_name: Literal["user", "key", "config"] ): self.set_env_vars_based_on_arn() + import aiohttp from aiodynamo.client import Client from aiodynamo.credentials import Credentials, StaticCredentials + from aiodynamo.expressions import F, UpdateExpression, Value + from aiodynamo.http.aiohttp import AIOHTTP from aiodynamo.http.httpx import HTTPX from aiodynamo.models import ( - Throughput, KeySchema, KeySpec, KeyType, PayPerRequest, + ReturnValues, + Throughput, ) - from yarl import URL - from aiodynamo.expressions import UpdateExpression, F, Value - from aiodynamo.models import ReturnValues - from aiodynamo.http.aiohttp import AIOHTTP from aiohttp import ClientSession - import aiohttp + from yarl import URL if self.database_arguments.ssl_verify == False: client_session = ClientSession(connector=aiohttp.TCPConnector(ssl=False)) diff --git a/litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py b/litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py index 01433b55595..eee26bd4283 100644 --- a/litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py +++ b/litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py @@ -24,7 +24,6 @@ import httpx from fastapi import HTTPException import litellm -from litellm import get_secret from litellm._logging import verbose_proxy_logger from litellm.caching import DualCache from litellm.integrations.custom_guardrail import CustomGuardrail @@ -38,6 +37,7 @@ from litellm.llms.custom_httpx.http_handler import ( ) from litellm.proxy._types import UserAPIKeyAuth from litellm.proxy.guardrails.guardrail_helpers import should_proceed_based_on_metadata +from litellm.secret_managers.main import get_secret from litellm.types.guardrails import ( BedrockContentItem, BedrockRequest, diff --git a/litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py b/litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py index 364bcb22274..5aebbcf3e7d 100644 --- a/litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py +++ b/litellm/proxy/guardrails/guardrail_hooks/lakera_ai.py @@ -19,12 +19,12 @@ import httpx from fastapi import HTTPException import litellm -from litellm import get_secret from litellm._logging import verbose_proxy_logger from litellm.integrations.custom_guardrail import CustomGuardrail from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler from litellm.proxy._types import UserAPIKeyAuth from litellm.proxy.guardrails.guardrail_helpers import should_proceed_based_on_metadata +from litellm.secret_managers.main import get_secret from litellm.types.guardrails import ( GuardrailItem, LakeraCategoryThresholds, diff --git a/litellm/proxy/prisma_migration.py b/litellm/proxy/prisma_migration.py index c79649f6f22..51dab42fa04 100644 --- a/litellm/proxy/prisma_migration.py +++ b/litellm/proxy/prisma_migration.py @@ -9,7 +9,7 @@ import time sys.path.insert( 0, os.path.abspath("./") ) # Adds the parent directory to the system path -from litellm.proxy.secret_managers.aws_secret_manager import decrypt_env_var +from litellm.secret_managers.aws_secret_manager import decrypt_env_var if os.getenv("USE_AWS_KMS", None) is not None and os.getenv("USE_AWS_KMS") == "True": ## V2 IMPLEMENTATION OF AWS KMS - USER WANTS TO DECRYPT MULTIPLE KEYS IN THEIR ENV diff --git a/litellm/proxy/proxy_cli.py b/litellm/proxy/proxy_cli.py index cf2638f3c8b..b9eddc25726 100644 --- a/litellm/proxy/proxy_cli.py +++ b/litellm/proxy/proxy_cli.py @@ -475,7 +475,7 @@ def run_server( ### DECRYPT ENV VAR ### - from litellm.proxy.secret_managers.aws_secret_manager import decrypt_env_var + from litellm.secret_managers.aws_secret_manager import decrypt_env_var if ( os.getenv("USE_AWS_KMS", None) is not None @@ -544,6 +544,15 @@ def run_server( load_aws_secret_manager(use_aws_secret_manager=True) elif key_management_system == KeyManagementSystem.AWS_KMS.value: load_aws_kms(use_aws_kms=True) + elif ( + key_management_system + == KeyManagementSystem.GOOGLE_SECRET_MANAGER.value + ): + from litellm.secret_managers.google_secret_manager import ( + GoogleSecretManager, + ) + + GoogleSecretManager() else: raise ValueError("Invalid Key Management System selected") key_management_settings = general_settings.get( @@ -598,7 +607,7 @@ def run_server( or os.getenv("DIRECT_URL", None) is not None ): try: - from litellm import get_secret + from litellm.secret_managers.main import get_secret if os.getenv("DATABASE_URL", None) is not None: ### add connection pool + pool timeout args diff --git a/litellm/proxy/proxy_config.yaml b/litellm/proxy/proxy_config.yaml index 12fa2558b61..7566f348afc 100644 --- a/litellm/proxy/proxy_config.yaml +++ b/litellm/proxy/proxy_config.yaml @@ -1,10 +1,15 @@ model_list: - - model_name: fake-openai-endpoint - litellm_params: - model: openai/fake - api_key: fake-key - model_info: - id: "team-a-model" # used for identifying model in response headers + - model_name: fake-openai-endpoint + litellm_params: + model: openai/fake + api_base: https://exampleopenaiendpoint-production.up.railway.app/ + api_key: os.environ/OPENAI_API_KEY + - model_name: gpt-3.5-turbo-end-user-test + litellm_params: + model: azure/chatgpt-v-2 + api_base: https://openai-gpt-4-test-v-1.openai.azure.com/ + api_version: "2023-05-15" + api_key: os.environ/AZURE_API_KEY litellm_settings: success_callback: ["prometheus"] @@ -13,7 +18,3 @@ litellm_settings: general_settings: master_key: sk-1234 - allowed_routes: ["/chat/completions"] - public_routes: ["LiteLLMRoutes.public_routes"] - admin_only_routes: ["/key/generate"] - \ No newline at end of file diff --git a/litellm/proxy/proxy_server.py b/litellm/proxy/proxy_server.py index dd6869c6618..d7160a09fc1 100644 --- a/litellm/proxy/proxy_server.py +++ b/litellm/proxy/proxy_server.py @@ -212,11 +212,6 @@ from litellm.proxy.pass_through_endpoints.pass_through_endpoints import ( ) from litellm.proxy.rerank_endpoints.endpoints import router as rerank_router from litellm.proxy.route_llm_request import route_request -from litellm.proxy.secret_managers.aws_secret_manager import ( - load_aws_kms, - load_aws_secret_manager, -) -from litellm.proxy.secret_managers.google_kms import load_google_kms from litellm.proxy.spend_tracking.spend_management_endpoints import ( router as spend_management_router, ) @@ -257,6 +252,11 @@ from litellm.router import ( from litellm.router import ModelInfo as RouterModelInfo from litellm.router import updateDeployment from litellm.scheduler import DefaultPriorities, FlowItem, Scheduler +from litellm.secret_managers.aws_secret_manager import ( + load_aws_kms, + load_aws_secret_manager, +) +from litellm.secret_managers.google_kms import load_google_kms from litellm.types.llms.anthropic import ( AnthropicMessagesRequest, AnthropicResponse, @@ -1765,6 +1765,15 @@ class ProxyConfig: load_aws_secret_manager(use_aws_secret_manager=True) elif key_management_system == KeyManagementSystem.AWS_KMS.value: load_aws_kms(use_aws_kms=True) + elif ( + key_management_system + == KeyManagementSystem.GOOGLE_SECRET_MANAGER.value + ): + from litellm.secret_managers.google_secret_manager import ( + GoogleSecretManager, + ) + + GoogleSecretManager() else: raise ValueError("Invalid Key Management System selected") key_management_settings = general_settings.get( diff --git a/litellm/rerank_api/main.py b/litellm/rerank_api/main.py index 2e69f281808..41de82ab660 100644 --- a/litellm/rerank_api/main.py +++ b/litellm/rerank_api/main.py @@ -4,10 +4,10 @@ from functools import partial from typing import Any, Coroutine, Dict, List, Literal, Optional, Union import litellm -from litellm import get_secret from litellm._logging import verbose_logger from litellm.llms.cohere.rerank import CohereRerank from litellm.llms.togetherai.rerank import TogetherAIRerank +from litellm.secret_managers.main import get_secret from litellm.types.router import * from litellm.utils import supports_httpx_timeout diff --git a/litellm/router_utils/client_initalization_utils.py b/litellm/router_utils/client_initalization_utils.py index bd5337b331e..9d68891c4b9 100644 --- a/litellm/router_utils/client_initalization_utils.py +++ b/litellm/router_utils/client_initalization_utils.py @@ -9,7 +9,7 @@ import openai import litellm from litellm._logging import verbose_router_logger from litellm.llms.azure import get_azure_ad_token_from_oidc -from litellm.proxy.secret_managers.get_azure_ad_token_provider import ( +from litellm.secret_managers.get_azure_ad_token_provider import ( get_azure_ad_token_provider, ) from litellm.utils import calculate_max_parallel_requests diff --git a/litellm/secret_managers/Readme.md b/litellm/secret_managers/Readme.md new file mode 100644 index 00000000000..9b22689059d --- /dev/null +++ b/litellm/secret_managers/Readme.md @@ -0,0 +1,3 @@ +## Supported Secret Managers to read credentials from + +Example read OPENAI_API_KEY, AZURE_API_KEY from a secret manager \ No newline at end of file diff --git a/litellm/proxy/secret_managers/aws_secret_manager.py b/litellm/secret_managers/aws_secret_manager.py similarity index 100% rename from litellm/proxy/secret_managers/aws_secret_manager.py rename to litellm/secret_managers/aws_secret_manager.py diff --git a/litellm/proxy/secret_managers/get_azure_ad_token_provider.py b/litellm/secret_managers/get_azure_ad_token_provider.py similarity index 83% rename from litellm/proxy/secret_managers/get_azure_ad_token_provider.py rename to litellm/secret_managers/get_azure_ad_token_provider.py index 0ecdae514a4..82e725ee871 100644 --- a/litellm/proxy/secret_managers/get_azure_ad_token_provider.py +++ b/litellm/secret_managers/get_azure_ad_token_provider.py @@ -14,8 +14,7 @@ def get_azure_ad_token_provider() -> Callable[[], str]: Returns: Callable that returns a temporary authentication token. """ - from azure.identity import ClientSecretCredential - from azure.identity import get_bearer_token_provider + from azure.identity import ClientSecretCredential, get_bearer_token_provider try: credential = ClientSecretCredential( @@ -24,7 +23,9 @@ def get_azure_ad_token_provider() -> Callable[[], str]: tenant_id=os.environ["AZURE_TENANT_ID"], ) except KeyError as e: - raise ValueError("Missing environment variable required by Azure AD workflow.") from e + raise ValueError( + "Missing environment variable required by Azure AD workflow." + ) from e return get_bearer_token_provider( credential, diff --git a/litellm/proxy/secret_managers/google_kms.py b/litellm/secret_managers/google_kms.py similarity index 97% rename from litellm/proxy/secret_managers/google_kms.py rename to litellm/secret_managers/google_kms.py index 5e83d54c7f4..d4d321b22a3 100644 --- a/litellm/proxy/secret_managers/google_kms.py +++ b/litellm/secret_managers/google_kms.py @@ -7,8 +7,11 @@ Requires: * `os.environ["GOOGLE_APPLICATION_CREDENTIALS"], os.environ["GOOGLE_KMS_RESOURCE_NAME"]` * `pip install google-cloud-kms` """ -import litellm, os + +import os from typing import Optional + +import litellm from litellm.proxy._types import KeyManagementSystem diff --git a/litellm/secret_managers/google_secret_manager.py b/litellm/secret_managers/google_secret_manager.py new file mode 100644 index 00000000000..0d0c3100538 --- /dev/null +++ b/litellm/secret_managers/google_secret_manager.py @@ -0,0 +1,116 @@ +import base64 +import os +from typing import Optional + +import litellm +from litellm._logging import verbose_logger +from litellm.caching import InMemoryCache +from litellm.integrations.gcs_bucket_base import GCSBucketBase +from litellm.llms.custom_httpx.http_handler import _get_httpx_client +from litellm.proxy._types import CommonProxyErrors, KeyManagementSystem + + +class GoogleSecretManager(GCSBucketBase): + def __init__( + self, + refresh_interval: Optional[int] = 86400, + always_read_secret_manager: Optional[bool] = False, + ) -> None: + """ + Args: + refresh_interval (int, optional): The refresh interval in seconds. Defaults to 86400. (24 hours) + always_read_secret_manager (bool, optional): Whether to always read from the secret manager. Defaults to False. Since we do want to cache values + """ + from litellm.proxy.proxy_server import premium_user + + if premium_user is not True: + raise ValueError( + f"Google Secret Manager requires an Enterprise License {CommonProxyErrors.not_premium_user.value}" + ) + super().__init__() + self.PROJECT_ID = os.environ.get("GOOGLE_SECRET_MANAGER_PROJECT_ID", None) + if self.PROJECT_ID is None: + raise ValueError( + "Google Secret Manager requires a project ID, please set 'GOOGLE_SECRET_MANAGER_PROJECT_ID' in your .env" + ) + self.sync_httpx_client = _get_httpx_client() + litellm.secret_manager_client = self + litellm._key_management_system = KeyManagementSystem.GOOGLE_SECRET_MANAGER + _refresh_interval = os.environ.get( + "GOOGLE_SECRET_MANAGER_REFRESH_INTERVAL", refresh_interval + ) + _refresh_interval = ( + int(_refresh_interval) if _refresh_interval else refresh_interval + ) + self.cache = InMemoryCache( + default_ttl=_refresh_interval + ) # store in memory for 1 day + + _always_read_secret_manager = os.environ.get( + "GOOGLE_SECRET_MANAGER_ALWAYS_READ_SECRET_MANAGER", + ) + if ( + _always_read_secret_manager + and _always_read_secret_manager.lower() == "true" + ): + self.always_read_secret_manager = True + else: + # by default this should be False, we want to use in memory caching for this. It's a bad idea to fetch from secret manager for all requests + self.always_read_secret_manager = always_read_secret_manager or False + + def get_secret_from_google_secret_manager(self, secret_name: str) -> Optional[str]: + """ + Retrieve a secret from Google Secret Manager or cache. + + Args: + secret_name (str): The name of the secret. + + Returns: + str: The secret value if successful, None otherwise. + """ + if self.always_read_secret_manager is not True: + cached_secret = self.cache.get_cache(secret_name) + if cached_secret is not None: + return cached_secret + if secret_name in self.cache.cache_dict: + return cached_secret + + _secret_name = ( + f"projects/{self.PROJECT_ID}/secrets/{secret_name}/versions/latest" + ) + headers = self.sync_construct_request_headers() + url = f"https://secretmanager.googleapis.com/v1/{_secret_name}:access" + + # Send the GET request to retrieve the secret + response = self.sync_httpx_client.get(url=url, headers=headers) + + if response.status_code != 200: + verbose_logger.error( + "Google Secret Manager retrieval error: %s", str(response.text) + ) + self.cache.set_cache( + secret_name, None + ) # Cache that the secret was not found + raise ValueError( + f"secret {secret_name} not found in Google Secret Manager. Error: {response.text}" + ) + + verbose_logger.debug( + "Google Secret Manager retrieval response status code: %s", + response.status_code, + ) + + # Parse the JSON response and return the secret value + secret_data = response.json() + _base64_encoded_value = secret_data.get("payload", {}).get("data") + + # decode the base64 encoded value + if _base64_encoded_value is not None: + _decoded_value = base64.b64decode(_base64_encoded_value).decode("utf-8") + self.cache.set_cache( + secret_name, _decoded_value + ) # Cache the retrieved secret + return _decoded_value + + self.cache.set_cache(secret_name, None) # Cache that the secret was not found + raise ValueError(f"secret {secret_name} not found in Google Secret Manager") diff --git a/litellm/secret_managers/main.py b/litellm/secret_managers/main.py new file mode 100644 index 00000000000..e136654c1c4 --- /dev/null +++ b/litellm/secret_managers/main.py @@ -0,0 +1,276 @@ +import ast +import base64 +import binascii +import json +import os +import sys +import traceback +from typing import Any, Optional, Union + +import httpx +from dotenv import load_dotenv + +import litellm +from litellm._logging import print_verbose, verbose_logger +from litellm.caching import DualCache +from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler +from litellm.proxy._types import KeyManagementSystem + +oidc_cache = DualCache() + + +######### Secret Manager ############################ +# checks if user has passed in a secret manager client +# if passed in then checks the secret there +def _is_base64(s): + try: + return base64.b64encode(base64.b64decode(s)).decode() == s + except binascii.Error: + return False + + +def get_secret( + secret_name: str, + default_value: Optional[Union[str, bool]] = None, +): + key_management_system = litellm._key_management_system + key_management_settings = litellm._key_management_settings + + if secret_name.startswith("os.environ/"): + secret_name = secret_name.replace("os.environ/", "") + + # Example: oidc/google/https://bedrock-runtime.us-east-1.amazonaws.com/model/stability.stable-diffusion-xl-v1/invoke + if secret_name.startswith("oidc/"): + secret_name_split = secret_name.replace("oidc/", "") + oidc_provider, oidc_aud = secret_name_split.split("/", 1) + # TODO: Add caching for HTTP requests + if oidc_provider == "google": + oidc_token = oidc_cache.get_cache(key=secret_name) + if oidc_token is not None: + return oidc_token + + oidc_client = HTTPHandler(timeout=httpx.Timeout(timeout=600.0, connect=5.0)) + # https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature + response = oidc_client.get( + "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity", + params={"audience": oidc_aud}, + headers={"Metadata-Flavor": "Google"}, + ) + if response.status_code == 200: + oidc_token = response.text + oidc_cache.set_cache(key=secret_name, value=oidc_token, ttl=3600 - 60) + return oidc_token + else: + raise ValueError("Google OIDC provider failed") + elif oidc_provider == "circleci": + # https://circleci.com/docs/openid-connect-tokens/ + env_secret = os.getenv("CIRCLE_OIDC_TOKEN") + if env_secret is None: + raise ValueError("CIRCLE_OIDC_TOKEN not found in environment") + return env_secret + elif oidc_provider == "circleci_v2": + # https://circleci.com/docs/openid-connect-tokens/ + env_secret = os.getenv("CIRCLE_OIDC_TOKEN_V2") + if env_secret is None: + raise ValueError("CIRCLE_OIDC_TOKEN_V2 not found in environment") + return env_secret + elif oidc_provider == "github": + # https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-cloud-providers#using-custom-actions + actions_id_token_request_url = os.getenv("ACTIONS_ID_TOKEN_REQUEST_URL") + actions_id_token_request_token = os.getenv("ACTIONS_ID_TOKEN_REQUEST_TOKEN") + if ( + actions_id_token_request_url is None + or actions_id_token_request_token is None + ): + raise ValueError( + "ACTIONS_ID_TOKEN_REQUEST_URL or ACTIONS_ID_TOKEN_REQUEST_TOKEN not found in environment" + ) + + oidc_token = oidc_cache.get_cache(key=secret_name) + if oidc_token is not None: + return oidc_token + + oidc_client = HTTPHandler(timeout=httpx.Timeout(timeout=600.0, connect=5.0)) + response = oidc_client.get( + actions_id_token_request_url, + params={"audience": oidc_aud}, + headers={ + "Authorization": f"Bearer {actions_id_token_request_token}", + "Accept": "application/json; api-version=2.0", + }, + ) + if response.status_code == 200: + oidc_token = response.text["value"] + oidc_cache.set_cache(key=secret_name, value=oidc_token, ttl=300 - 5) + return oidc_token + else: + raise ValueError("Github OIDC provider failed") + elif oidc_provider == "azure": + # https://azure.github.io/azure-workload-identity/docs/quick-start.html + azure_federated_token_file = os.getenv("AZURE_FEDERATED_TOKEN_FILE") + if azure_federated_token_file is None: + raise ValueError("AZURE_FEDERATED_TOKEN_FILE not found in environment") + with open(azure_federated_token_file, "r") as f: + oidc_token = f.read() + return oidc_token + elif oidc_provider == "file": + # Load token from a file + with open(oidc_aud, "r") as f: + oidc_token = f.read() + return oidc_token + elif oidc_provider == "env": + # Load token directly from an environment variable + oidc_token = os.getenv(oidc_aud) + if oidc_token is None: + raise ValueError(f"Environment variable {oidc_aud} not found") + return oidc_token + elif oidc_provider == "env_path": + # Load token from a file path specified in an environment variable + token_file_path = os.getenv(oidc_aud) + if token_file_path is None: + raise ValueError(f"Environment variable {oidc_aud} not found") + with open(token_file_path, "r") as f: + oidc_token = f.read() + return oidc_token + else: + raise ValueError("Unsupported OIDC provider") + + try: + if litellm.secret_manager_client is not None: + try: + client = litellm.secret_manager_client + key_manager = "local" + if key_management_system is not None: + key_manager = key_management_system.value + + if key_management_settings is not None: + if ( + secret_name not in key_management_settings.hosted_keys + ): # allow user to specify which keys to check in hosted key manager + key_manager = "local" + + if ( + key_manager == KeyManagementSystem.AZURE_KEY_VAULT.value + or type(client).__module__ + "." + type(client).__name__ + == "azure.keyvault.secrets._client.SecretClient" + ): # support Azure Secret Client - from azure.keyvault.secrets import SecretClient + secret = client.get_secret(secret_name).value + elif ( + key_manager == KeyManagementSystem.GOOGLE_KMS.value + or client.__class__.__name__ == "KeyManagementServiceClient" + ): + encrypted_secret: Any = os.getenv(secret_name) + if encrypted_secret is None: + raise ValueError( + f"Google KMS requires the encrypted secret to be in the environment!" + ) + b64_flag = _is_base64(encrypted_secret) + if b64_flag == True: # if passed in as encoded b64 string + encrypted_secret = base64.b64decode(encrypted_secret) + ciphertext = encrypted_secret + else: + raise ValueError( + f"Google KMS requires the encrypted secret to be encoded in base64" + ) # fix for this vulnerability https://huntr.com/bounties/ae623c2f-b64b-4245-9ed4-f13a0a5824ce + response = client.decrypt( + request={ + "name": litellm._google_kms_resource_name, + "ciphertext": ciphertext, + } + ) + secret = response.plaintext.decode( + "utf-8" + ) # assumes the original value was encoded with utf-8 + elif key_manager == KeyManagementSystem.AWS_KMS.value: + """ + Only check the tokens which start with 'aws_kms/'. This prevents latency impact caused by checking all keys. + """ + encrypted_value = os.getenv(secret_name, None) + if encrypted_value is None: + raise Exception( + "AWS KMS - Encrypted Value of Key={} is None".format( + secret_name + ) + ) + # Decode the base64 encoded ciphertext + ciphertext_blob = base64.b64decode(encrypted_value) + + # Set up the parameters for the decrypt call + params = {"CiphertextBlob": ciphertext_blob} + # Perform the decryption + response = client.decrypt(**params) + + # Extract and decode the plaintext + plaintext = response["Plaintext"] + secret = plaintext.decode("utf-8") + if isinstance(secret, str): + secret = secret.strip() + elif key_manager == KeyManagementSystem.AWS_SECRET_MANAGER.value: + try: + get_secret_value_response = client.get_secret_value( + SecretId=secret_name + ) + print_verbose( + f"get_secret_value_response: {get_secret_value_response}" + ) + except Exception as e: + print_verbose(f"An error occurred - {str(e)}") + # For a list of exceptions thrown, see + # https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html + raise e + + # assume there is 1 secret per secret_name + secret_dict = json.loads(get_secret_value_response["SecretString"]) + print_verbose(f"secret_dict: {secret_dict}") + for k, v in secret_dict.items(): + secret = v + print_verbose(f"secret: {secret}") + if key_manager == KeyManagementSystem.GOOGLE_SECRET_MANAGER.value: + try: + secret = client.get_secret_from_google_secret_manager( + secret_name + ) + print_verbose(f"secret from google secret manager: {secret}") + if secret is None: + raise ValueError( + f"No secret found in Google Secret Manager for {secret_name}" + ) + except Exception as e: + print_verbose(f"An error occurred - {str(e)}") + raise e + elif key_manager == "local": + secret = os.getenv(secret_name) + else: # assume the default is infisicial client + secret = client.get_secret(secret_name).secret_value + except Exception as e: # check if it's in os.environ + verbose_logger.error( + f"Defaulting to os.environ value for key={secret_name}. An exception occurred - {str(e)}.\n\n{traceback.format_exc()}" + ) + secret = os.getenv(secret_name) + try: + secret_value_as_bool = ast.literal_eval(secret) + if isinstance(secret_value_as_bool, bool): + return secret_value_as_bool + else: + return secret + except: + return secret + else: + secret = os.environ.get(secret_name) + try: + secret_value_as_bool = ( + ast.literal_eval(secret) if secret is not None else None + ) + if isinstance(secret_value_as_bool, bool): + return secret_value_as_bool + else: + return secret + except Exception: + if default_value is not None: + return default_value + return secret + except Exception as e: + if default_value is not None: + return default_value + else: + raise e diff --git a/litellm/tests/test_router_client_init.py b/litellm/tests/test_router_client_init.py index 0984e406d32..1b064e3f5f9 100644 --- a/litellm/tests/test_router_client_init.py +++ b/litellm/tests/test_router_client_init.py @@ -83,7 +83,7 @@ async def test_router_init(): ) -@patch("litellm.proxy.secret_managers.get_azure_ad_token_provider.os") +@patch("litellm.secret_managers.get_azure_ad_token_provider.os") def test_router_init_with_neither_api_key_nor_azure_service_principal_with_secret( mocked_os_lib: MagicMock, ) -> None: @@ -128,7 +128,7 @@ def test_router_init_with_neither_api_key_nor_azure_service_principal_with_secre @patch("azure.identity.get_bearer_token_provider") @patch("azure.identity.ClientSecretCredential") -@patch("litellm.proxy.secret_managers.get_azure_ad_token_provider.os") +@patch("litellm.secret_managers.get_azure_ad_token_provider.os") def test_router_init_azure_service_principal_with_secret_with_environment_variables( mocked_os_lib: MagicMock, mocked_credential: MagicMock, diff --git a/litellm/tests/test_secret_manager.py b/litellm/tests/test_secret_manager.py index 1cf374148bb..a380e6287a4 100644 --- a/litellm/tests/test_secret_manager.py +++ b/litellm/tests/test_secret_manager.py @@ -16,10 +16,10 @@ sys.path.insert( ) # Adds the parent directory to the system path import pytest -from litellm import get_secret from litellm.llms.azure import get_azure_ad_token_from_oidc from litellm.llms.bedrock.chat import BedrockConverseLLM, BedrockLLM -from litellm.proxy.secret_managers.aws_secret_manager import load_aws_secret_manager +from litellm.secret_managers.aws_secret_manager import load_aws_secret_manager +from litellm.secret_managers.main import get_secret @pytest.mark.skip(reason="AWS Suspended Account") @@ -189,3 +189,54 @@ def test_oidc_env_path(): assert secret_val == secret_value del os.environ[env_var_name] + + +def test_google_secret_manager(): + """ + Test that we can get a secret from Google Secret Manager + """ + os.environ["GOOGLE_SECRET_MANAGER_PROJECT_ID"] = "adroit-crow-413218" + from test_amazing_vertex_completion import load_vertex_ai_credentials + + from litellm.secret_managers.google_secret_manager import GoogleSecretManager + + # load_vertex_ai_credentials() + secret_manager = GoogleSecretManager() + + secret_val = secret_manager.get_secret_from_google_secret_manager( + secret_name="OPENAI_API_KEY" + ) + print("secret_val: {}".format(secret_val)) + + assert ( + secret_val == "anything" + ), "did not get expected secret value. expect 'anything', got '{}'".format( + secret_val + ) + + +def test_google_secret_manager_read_in_memory(): + """ + Test that Google Secret manager returs in memory value when it exists + """ + from test_amazing_vertex_completion import load_vertex_ai_credentials + + from litellm.secret_managers.google_secret_manager import GoogleSecretManager + + # load_vertex_ai_credentials() + os.environ["GOOGLE_SECRET_MANAGER_PROJECT_ID"] = "adroit-crow-413218" + secret_manager = GoogleSecretManager() + secret_manager.cache.cache_dict["UNIQUE_KEY"] = None + secret_manager.cache.cache_dict["UNIQUE_KEY_2"] = "lite-llm" + + secret_val = secret_manager.get_secret_from_google_secret_manager( + secret_name="UNIQUE_KEY" + ) + print("secret_val: {}".format(secret_val)) + assert secret_val == None + + secret_val = secret_manager.get_secret_from_google_secret_manager( + secret_name="UNIQUE_KEY_2" + ) + print("secret_val: {}".format(secret_val)) + assert secret_val == "lite-llm" diff --git a/litellm/tests/test_streaming.py b/litellm/tests/test_streaming.py index 1b155bea850..4cc259b41f5 100644 --- a/litellm/tests/test_streaming.py +++ b/litellm/tests/test_streaming.py @@ -3712,6 +3712,7 @@ def test_unit_test_custom_stream_wrapper_function_call(): "vertex_ai/claude-3-5-sonnet@20240620", ], ) +@pytest.mark.flaky(retries=3, delay=1) def test_streaming_tool_calls_valid_json_str(model): if "vertex_ai" in model: from litellm.tests.test_amazing_vertex_completion import ( diff --git a/litellm/utils.py b/litellm/utils.py index 26bf993aada..d5b9fde2dd4 100644 --- a/litellm/utils.py +++ b/litellm/utils.py @@ -68,6 +68,7 @@ from litellm.litellm_core_utils.redact_messages import ( ) from litellm.litellm_core_utils.token_counter import get_modified_max_tokens from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler +from litellm.secret_managers.main import get_secret from litellm.types.llms.openai import ( AllMessageValues, ChatCompletionNamedToolChoiceParam, @@ -93,8 +94,6 @@ from litellm.types.utils import ( Usage, ) -oidc_cache = DualCache() - try: # New and recommended way to access resources from importlib import resources @@ -8662,250 +8661,6 @@ def exception_type( raise raised_exc -######### Secret Manager ############################ -# checks if user has passed in a secret manager client -# if passed in then checks the secret there -def _is_base64(s): - try: - return base64.b64encode(base64.b64decode(s)).decode() == s - except binascii.Error: - return False - - -def get_secret( - secret_name: str, - default_value: Optional[Union[str, bool]] = None, -): - key_management_system = litellm._key_management_system - key_management_settings = litellm._key_management_settings - - if secret_name.startswith("os.environ/"): - secret_name = secret_name.replace("os.environ/", "") - - # Example: oidc/google/https://bedrock-runtime.us-east-1.amazonaws.com/model/stability.stable-diffusion-xl-v1/invoke - if secret_name.startswith("oidc/"): - secret_name_split = secret_name.replace("oidc/", "") - oidc_provider, oidc_aud = secret_name_split.split("/", 1) - # TODO: Add caching for HTTP requests - if oidc_provider == "google": - oidc_token = oidc_cache.get_cache(key=secret_name) - if oidc_token is not None: - return oidc_token - - oidc_client = HTTPHandler(timeout=httpx.Timeout(timeout=600.0, connect=5.0)) - # https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature - response = oidc_client.get( - "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity", - params={"audience": oidc_aud}, - headers={"Metadata-Flavor": "Google"}, - ) - if response.status_code == 200: - oidc_token = response.text - oidc_cache.set_cache(key=secret_name, value=oidc_token, ttl=3600 - 60) - return oidc_token - else: - raise ValueError("Google OIDC provider failed") - elif oidc_provider == "circleci": - # https://circleci.com/docs/openid-connect-tokens/ - env_secret = os.getenv("CIRCLE_OIDC_TOKEN") - if env_secret is None: - raise ValueError("CIRCLE_OIDC_TOKEN not found in environment") - return env_secret - elif oidc_provider == "circleci_v2": - # https://circleci.com/docs/openid-connect-tokens/ - env_secret = os.getenv("CIRCLE_OIDC_TOKEN_V2") - if env_secret is None: - raise ValueError("CIRCLE_OIDC_TOKEN_V2 not found in environment") - return env_secret - elif oidc_provider == "github": - # https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-cloud-providers#using-custom-actions - actions_id_token_request_url = os.getenv("ACTIONS_ID_TOKEN_REQUEST_URL") - actions_id_token_request_token = os.getenv("ACTIONS_ID_TOKEN_REQUEST_TOKEN") - if ( - actions_id_token_request_url is None - or actions_id_token_request_token is None - ): - raise ValueError( - "ACTIONS_ID_TOKEN_REQUEST_URL or ACTIONS_ID_TOKEN_REQUEST_TOKEN not found in environment" - ) - - oidc_token = oidc_cache.get_cache(key=secret_name) - if oidc_token is not None: - return oidc_token - - oidc_client = HTTPHandler(timeout=httpx.Timeout(timeout=600.0, connect=5.0)) - response = oidc_client.get( - actions_id_token_request_url, - params={"audience": oidc_aud}, - headers={ - "Authorization": f"Bearer {actions_id_token_request_token}", - "Accept": "application/json; api-version=2.0", - }, - ) - if response.status_code == 200: - oidc_token = response.text["value"] - oidc_cache.set_cache(key=secret_name, value=oidc_token, ttl=300 - 5) - return oidc_token - else: - raise ValueError("Github OIDC provider failed") - elif oidc_provider == "azure": - # https://azure.github.io/azure-workload-identity/docs/quick-start.html - azure_federated_token_file = os.getenv("AZURE_FEDERATED_TOKEN_FILE") - if azure_federated_token_file is None: - raise ValueError("AZURE_FEDERATED_TOKEN_FILE not found in environment") - with open(azure_federated_token_file, "r") as f: - oidc_token = f.read() - return oidc_token - elif oidc_provider == "file": - # Load token from a file - with open(oidc_aud, "r") as f: - oidc_token = f.read() - return oidc_token - elif oidc_provider == "env": - # Load token directly from an environment variable - oidc_token = os.getenv(oidc_aud) - if oidc_token is None: - raise ValueError(f"Environment variable {oidc_aud} not found") - return oidc_token - elif oidc_provider == "env_path": - # Load token from a file path specified in an environment variable - token_file_path = os.getenv(oidc_aud) - if token_file_path is None: - raise ValueError(f"Environment variable {oidc_aud} not found") - with open(token_file_path, "r") as f: - oidc_token = f.read() - return oidc_token - else: - raise ValueError("Unsupported OIDC provider") - - try: - if litellm.secret_manager_client is not None: - try: - client = litellm.secret_manager_client - key_manager = "local" - if key_management_system is not None: - key_manager = key_management_system.value - - if key_management_settings is not None: - if ( - secret_name not in key_management_settings.hosted_keys - ): # allow user to specify which keys to check in hosted key manager - key_manager = "local" - - if ( - key_manager == KeyManagementSystem.AZURE_KEY_VAULT.value - or type(client).__module__ + "." + type(client).__name__ - == "azure.keyvault.secrets._client.SecretClient" - ): # support Azure Secret Client - from azure.keyvault.secrets import SecretClient - secret = client.get_secret(secret_name).value - elif ( - key_manager == KeyManagementSystem.GOOGLE_KMS.value - or client.__class__.__name__ == "KeyManagementServiceClient" - ): - encrypted_secret: Any = os.getenv(secret_name) - if encrypted_secret is None: - raise ValueError( - f"Google KMS requires the encrypted secret to be in the environment!" - ) - b64_flag = _is_base64(encrypted_secret) - if b64_flag == True: # if passed in as encoded b64 string - encrypted_secret = base64.b64decode(encrypted_secret) - ciphertext = encrypted_secret - else: - raise ValueError( - f"Google KMS requires the encrypted secret to be encoded in base64" - ) # fix for this vulnerability https://huntr.com/bounties/ae623c2f-b64b-4245-9ed4-f13a0a5824ce - response = client.decrypt( - request={ - "name": litellm._google_kms_resource_name, - "ciphertext": ciphertext, - } - ) - secret = response.plaintext.decode( - "utf-8" - ) # assumes the original value was encoded with utf-8 - elif key_manager == KeyManagementSystem.AWS_KMS.value: - """ - Only check the tokens which start with 'aws_kms/'. This prevents latency impact caused by checking all keys. - """ - encrypted_value = os.getenv(secret_name, None) - if encrypted_value is None: - raise Exception( - "AWS KMS - Encrypted Value of Key={} is None".format( - secret_name - ) - ) - # Decode the base64 encoded ciphertext - ciphertext_blob = base64.b64decode(encrypted_value) - - # Set up the parameters for the decrypt call - params = {"CiphertextBlob": ciphertext_blob} - # Perform the decryption - response = client.decrypt(**params) - - # Extract and decode the plaintext - plaintext = response["Plaintext"] - secret = plaintext.decode("utf-8") - if isinstance(secret, str): - secret = secret.strip() - elif key_manager == KeyManagementSystem.AWS_SECRET_MANAGER.value: - try: - get_secret_value_response = client.get_secret_value( - SecretId=secret_name - ) - print_verbose( - f"get_secret_value_response: {get_secret_value_response}" - ) - except Exception as e: - print_verbose(f"An error occurred - {str(e)}") - # For a list of exceptions thrown, see - # https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html - raise e - - # assume there is 1 secret per secret_name - secret_dict = json.loads(get_secret_value_response["SecretString"]) - print_verbose(f"secret_dict: {secret_dict}") - for k, v in secret_dict.items(): - secret = v - print_verbose(f"secret: {secret}") - elif key_manager == "local": - secret = os.getenv(secret_name) - else: # assume the default is infisicial client - secret = client.get_secret(secret_name).secret_value - except Exception as e: # check if it's in os.environ - verbose_logger.error( - f"Defaulting to os.environ value for key={secret_name}. An exception occurred - {str(e)}.\n\n{traceback.format_exc()}" - ) - secret = os.getenv(secret_name) - try: - secret_value_as_bool = ast.literal_eval(secret) - if isinstance(secret_value_as_bool, bool): - return secret_value_as_bool - else: - return secret - except: - return secret - else: - secret = os.environ.get(secret_name) - try: - secret_value_as_bool = ( - ast.literal_eval(secret) if secret is not None else None - ) - if isinstance(secret_value_as_bool, bool): - return secret_value_as_bool - else: - return secret - except Exception: - if default_value is not None: - return default_value - return secret - except Exception as e: - if default_value is not None: - return default_value - else: - raise e - - ######## Streaming Class ############################ # wraps the completion stream to return the correct format for the model # replicate/anthropic/cohere diff --git a/tests/test_entrypoint.py b/tests/test_entrypoint.py index 803135e35d7..8a6708545ec 100644 --- a/tests/test_entrypoint.py +++ b/tests/test_entrypoint.py @@ -17,7 +17,7 @@ def test_decrypt_and_reset_env(): os.environ["DATABASE_URL"] = ( "aws_kms/AQICAHgwddjZ9xjVaZ9CNCG8smFU6FiQvfdrjL12DIqi9vUAQwHwF6U7caMgHQa6tK+TzaoMAAAAzjCBywYJKoZIhvcNAQcGoIG9MIG6AgEAMIG0BgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCmu+DVeKTm5tFZu6AIBEICBhnOFQYviL8JsciGk0bZsn9pfzeYWtNkVXEsl01AdgHBqT9UOZOI4ZC+T3wO/fXA7wdNF4o8ASPDbVZ34ZFdBs8xt4LKp9niufL30WYBkuuzz89ztly0jvE9pZ8L6BMw0ATTaMgIweVtVSDCeCzEb5PUPyxt4QayrlYHBGrNH5Aq/axFTe0La" ) - from litellm.proxy.secret_managers.aws_secret_manager import ( + from litellm.secret_managers.aws_secret_manager import ( decrypt_and_reset_env_var, )