"
- ```
-
-#### --model or -m
- - **Default:** `None`
- - The model name to pass to Litellm.
- - **Usage:**
- ```shell
- litellm --model gpt-3.5-turbo
- ```
-
-#### --test
- - **Type:** `bool` (Flag)
- - Proxy chat completions URL to make a test request.
- - **Usage:**
- ```shell
- litellm --test
- ```
-
-#### --health
- - **Type:** `bool` (Flag)
- - Runs a health check on all models in config.yaml
- - **Usage:**
- ```shell
- litellm --health
- ```
-
-#### --alias
- - **Default:** `None`
- - An alias for the model, for user-friendly reference.
- - **Usage:**
- ```shell
- litellm --alias my-gpt-model
- ```
-
-#### --debug
- - **Default:** `False`
- - **Type:** `bool` (Flag)
- - Enable debugging mode for the input.
- - **Usage:**
- ```shell
- litellm --debug
- ```
-#### --detailed_debug
- - **Default:** `False`
- - **Type:** `bool` (Flag)
- - Enable debugging mode for the input.
- - **Usage:**
- ```shell
- litellm --detailed_debug
- ```
-
-#### --temperature
- - **Default:** `None`
- - **Type:** `float`
- - Set the temperature for the model.
- - **Usage:**
- ```shell
- litellm --temperature 0.7
- ```
-
-#### --max_tokens
- - **Default:** `None`
- - **Type:** `int`
- - Set the maximum number of tokens for the model output.
- - **Usage:**
- ```shell
- litellm --max_tokens 50
- ```
-
-#### --request_timeout
- - **Default:** `6000`
- - **Type:** `int`
- - Set the timeout in seconds for completion calls.
- - **Usage:**
- ```shell
- litellm --request_timeout 300
- ```
-
-#### --drop_params
- - **Type:** `bool` (Flag)
- - Drop any unmapped params.
- - **Usage:**
- ```shell
- litellm --drop_params
- ```
-
-#### --add_function_to_prompt
- - **Type:** `bool` (Flag)
- - If a function passed but unsupported, pass it as a part of the prompt.
- - **Usage:**
- ```shell
- litellm --add_function_to_prompt
- ```
-
-#### --config
- - Configure Litellm by providing a configuration file path.
- - **Usage:**
- ```shell
- litellm --config path/to/config.yaml
- ```
-
-#### --telemetry
- - **Default:** `True`
- - **Type:** `bool`
- - Help track usage of this feature.
- - **Usage:**
- ```shell
- litellm --telemetry False
- ```
diff --git a/docs/my-website/docs/troubleshoot.md b/docs/my-website/docs/troubleshoot.md
index b6a9c6a6b92..9d2b3757ee2 100644
--- a/docs/my-website/docs/troubleshoot.md
+++ b/docs/my-website/docs/troubleshoot.md
@@ -2,7 +2,7 @@
[Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
[Community Discord 💭](https://discord.gg/wuPM9dRgDw)
-[Community Slack 💭](https://join.slack.com/share/enQtOTE0ODczMzk2Nzk4NC01YjUxNjY2YjBlYTFmNDRiZTM3NDFiYTM3MzVkODFiMDVjOGRjMmNmZTZkZTMzOWQzZGQyZWIwYjQ0MWExYmE3)
+[Community Slack 💭](https://litellmossslack.slack.com/)
Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
diff --git a/docs/my-website/docs/tutorials/claude_responses_api.md b/docs/my-website/docs/tutorials/claude_responses_api.md
index a06be87409b..87d019f11b9 100644
--- a/docs/my-website/docs/tutorials/claude_responses_api.md
+++ b/docs/my-website/docs/tutorials/claude_responses_api.md
@@ -4,14 +4,20 @@ import TabItem from '@theme/TabItem';
# Claude Code
-This tutorial shows how to call the Responses API models like `codex-mini` and `o3-pro` from the Claude Code endpoint on LiteLLM.
+This tutorial shows how to call Claude models through LiteLLM proxy from Claude Code.
:::info
-This tutorial is based on [Anthropic's official LiteLLM configuration documentation](https://docs.anthropic.com/en/docs/claude-code/llm-gateway#litellm-configuration). This integration allows you to use any LiteLLM supported model through Claude Code.
+This tutorial is based on [Anthropic's official LiteLLM configuration documentation](https://docs.anthropic.com/en/docs/claude-code/llm-gateway#litellm-configuration). This integration allows you to use any LiteLLM supported model through Claude Code with centralized authentication, usage tracking, and cost controls.
:::
+
+
+### Video Walkthrough
+
+
+
## Prerequisites
- [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) installed
@@ -31,19 +37,18 @@ Create a secure configuration using environment variables:
```yaml
model_list:
- # Responses API models
- - model_name: codex-mini
+ # Claude models
+ - model_name: claude-3-5-sonnet-20241022
litellm_params:
- model: openai/codex-mini
- api_key: os.environ/OPENAI_API_KEY
- api_base: https://api.openai.com/v1
+ model: anthropic/claude-3-5-sonnet-20241022
+ api_key: os.environ/ANTHROPIC_API_KEY
- - model_name: o3-pro
+ - model_name: claude-3-5-haiku-20241022
litellm_params:
- model: openai/o3-pro
- api_key: os.environ/OPENAI_API_KEY
- api_base: https://api.openai.com/v1
+ model: anthropic/claude-3-5-haiku-20241022
+ api_key: os.environ/ANTHROPIC_API_KEY
+
litellm_settings:
master_key: os.environ/LITELLM_MASTER_KEY
```
@@ -51,7 +56,7 @@ litellm_settings:
Set your environment variables:
```bash
-export OPENAI_API_KEY="your-openai-api-key"
+export ANTHROPIC_API_KEY="your-anthropic-api-key"
export LITELLM_MASTER_KEY="sk-1234567890" # Generate a secure key
```
@@ -72,14 +77,32 @@ curl -X POST http://0.0.0.0:4000/v1/messages \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{
- "model": "codex-mini",
+ "model": "claude-3-5-sonnet-20241022",
+ "max_tokens": 1000,
"messages": [{"role": "user", "content": "What is the capital of France?"}]
}'
```
### 4. Configure Claude Code
-Setup Claude Code to use your LiteLLM proxy:
+#### Method 1: Unified Endpoint (Recommended)
+
+Configure Claude Code to use LiteLLM's unified endpoint:
+
+Either a virtual key / master key can be used here
+
+```bash
+export ANTHROPIC_BASE_URL="http://0.0.0.0:4000"
+export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
+```
+
+:::tip
+LITELLM_MASTER_KEY gives claude access to all proxy models, whereas a virtual key would be limited to the models set in UI
+:::
+
+#### Method 2: Provider-specific Pass-through Endpoint
+
+Alternatively, use the Anthropic pass-through endpoint:
```bash
export ANTHROPIC_BASE_URL="http://0.0.0.0:4000"
@@ -88,15 +111,15 @@ export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
### 5. Use Claude Code
-Start Claude Code with any configured model:
+Start Claude Code and it will automatically use your configured models:
```bash
-# Use Responses API models
-claude --model codex-mini
-claude --model o3-pro
+# Claude Code will use the models configured in your LiteLLM proxy
+claude
-# Or use the latest model alias
-claude --model codex-mini-latest
+# Or specify a model if you have multiple configured
+claude --model claude-3-5-sonnet-20241022
+claude --model claude-3-5-haiku-20241022
```
Example conversation:
@@ -112,7 +135,8 @@ Common issues and solutions:
**Authentication errors:**
- Verify your environment variables are set: `echo $LITELLM_MASTER_KEY`
-- Check that your OpenAI API key is valid and has sufficient credits
+- Check that your API keys are valid and have sufficient credits
+- Ensure the `ANTHROPIC_AUTH_TOKEN` matches your LiteLLM master key
**Model not found:**
- Ensure the model name in Claude Code matches exactly with your `config.yaml`
@@ -123,33 +147,47 @@ Common issues and solutions:
Expand your configuration to support multiple providers and models:
-
+
```yaml
model_list:
- # Responses API models
+ # OpenAI models
- model_name: codex-mini
- litellm_params:
+ litellm_params:
model: openai/codex-mini
api_key: os.environ/OPENAI_API_KEY
api_base: https://api.openai.com/v1
-
+
- model_name: o3-pro
litellm_params:
model: openai/o3-pro
api_key: os.environ/OPENAI_API_KEY
api_base: https://api.openai.com/v1
- # Standard models
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
+ api_base: https://api.openai.com/v1
- - model_name: claude-3-5-sonnet
+ # Anthropic models
+ - model_name: claude-3-5-sonnet-20241022
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
+
+ - model_name: claude-3-5-haiku-20241022
+ litellm_params:
+ model: anthropic/claude-3-5-haiku-20241022
+ api_key: os.environ/ANTHROPIC_API_KEY
+
+ # AWS Bedrock
+ - model_name: claude-bedrock
+ litellm_params:
+ model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
+ aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
+ aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
+ aws_region_name: us-east-1
litellm_settings:
master_key: os.environ/LITELLM_MASTER_KEY
@@ -158,16 +196,94 @@ litellm_settings:
Switch between models seamlessly:
```bash
-# Use Responses API models for advanced reasoning
-claude --model o3-pro
-claude --model codex-mini
+# Use Claude for complex reasoning
+claude --model claude-3-5-sonnet-20241022
-# Use standard models for general tasks
-claude --model gpt-4o
-claude --model claude-3-5-sonnet
+# Use Haiku for fast responses
+claude --model claude-3-5-haiku-20241022
+
+# Use Bedrock deployment
+claude --model claude-bedrock
```
-
\ No newline at end of file
+
+
+
+## Connecting MCP Servers
+
+You can also connect MCP servers to Claude Code via LiteLLM Proxy.
+
+:::note
+
+Limitations:
+
+- Currently, only HTTP MCP servers are supported
+- Does not work in Cursor IDE yet.
+
+:::
+
+1. Add the MCP server to your `config.yaml`
+
+In this example, we'll add the Github MCP server to our `config.yaml`
+
+```yaml title="config.yaml" showLineNumbers
+mcp_servers:
+ github_mcp:
+ url: "https://api.githubcopilot.com/mcp"
+ auth_type: oauth2
+ authorization_url: https://github.com/login/oauth/authorize
+ token_url: https://github.com/login/oauth/access_token
+ client_id: os.environ/GITHUB_OAUTH_CLIENT_ID
+ client_secret: os.environ/GITHUB_OAUTH_CLIENT_SECRET
+ scopes: ["public_repo", "user:email"]
+```
+
+2. Start LiteLLM Proxy
+
+```bash
+litellm --config /path/to/config.yaml
+
+# RUNNING on http://0.0.0.0:4000
+```
+
+3. Use the MCP server in Claude Code
+
+```bash
+claude mcp add --transport http litellm_proxy http://0.0.0.0:4000/github_mcp/mcp --header "Authorization: Bearer sk-LITELLM_VIRTUAL_KEY"
+```
+
+4. Authenticate via Claude Code
+
+a. Start Claude Code
+
+```bash
+claude
+```
+
+b. Authenticate via Claude Code
+
+```bash
+/mcp
+```
+
+c. Select the MCP server
+
+```bash
+> litellm_proxy
+```
+
+d. Start Oauth flow via Claude Code
+
+```bash
+> 1. Authenticate
+ 2. Reconnect
+ 3. Disable
+```
+
+e. Once completed, you should see this success message:
+
+
+
diff --git a/docs/my-website/docs/tutorials/cost_tracking_coding.md b/docs/my-website/docs/tutorials/cost_tracking_coding.md
new file mode 100644
index 00000000000..ffad2d45c80
--- /dev/null
+++ b/docs/my-website/docs/tutorials/cost_tracking_coding.md
@@ -0,0 +1,91 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import Image from '@theme/IdealImage';
+
+# Track Usage for Coding Tools
+
+Track usage and costs for AI-powered coding tools like Claude Code, Roo Code, Gemini CLI, and OpenAI Codex through LiteLLM.
+
+Monitor requests, costs, and user engagement metrics for each coding tool using User-Agent headers.
+
+
+
+
+## Who This Is For
+
+Central AI Platform teams providing developers access to coding tools through LiteLLM. Monitor tool engagement and track individual user usage patterns.
+
+## What You Can Track
+
+### Summary Metrics
+- Cost per coding tool
+- Successful requests and token usage per tool
+
+### User Engagement Metrics
+- Daily, weekly, and monthly active users for each User-Agent
+
+## Quick Start
+
+### 1. Connect Your Coding Tool to LiteLLM
+
+Configure your coding tool to send requests through the LiteLLM proxy with appropriate User-Agent headers.
+
+**Setup guides:**
+- [Use LiteLLM with Claude Code](../../docs/tutorials/claude_responses_api)
+- [Use LiteLLM with Gemini CLI](../../docs/tutorials/litellm_gemini_cli)
+- [Use LiteLLM with OpenAI Codex](../../docs/tutorials/openai_codex)
+
+### 2. Send Requests with User-Agent Headers
+
+Ensure your coding tool includes identifying User-Agent headers in API requests.
+
+### 3. Verify Tracking in LiteLLM Logs
+
+Confirm LiteLLM is properly tracking requests by checking logs for the expected User-Agent values.
+
+
+
+### 4. View Usage Dashboard
+
+Access the LiteLLM dashboard to view aggregated usage metrics and user engagement data.
+
+#### Summary Metrics
+
+View total cost and successful requests for each coding tool.
+
+
+
+#### Daily, Weekly, and Monthly Active Users
+
+View active user metrics for each coding tool.
+
+
+
+## How LiteLLM Identifies Coding Tools
+
+LiteLLM tracks coding tools by monitoring the `User-Agent` header in incoming API requests (`/chat/completions`, `/responses`, etc.). Each unique User-Agent is tracked separately for usage analytics.
+
+### Example Request
+
+Example using `claude-cli` as the User-Agent:
+
+```shell
+curl -X POST \
+ -H "Content-Type: application/json" \
+ -H "Authorization: Bearer sk-1234" \
+ -H "User-Agent: claude-cli/1.0" \
+ -d '{"model": "claude-3-5-sonnet-latest", "messages": [{"role": "user", "content": "Hello, how are you?"}]}' \
+ http://localhost:4000/chat/completions
+```
diff --git a/docs/my-website/docs/tutorials/msft_sso.md b/docs/my-website/docs/tutorials/msft_sso.md
index f7ad6440f2e..2936f27297f 100644
--- a/docs/my-website/docs/tutorials/msft_sso.md
+++ b/docs/my-website/docs/tutorials/msft_sso.md
@@ -140,6 +140,54 @@ litellm_settings:
+## 4. Using Entra ID App Roles for User Permissions
+
+You can assign user roles directly from Entra ID using App Roles. LiteLLM will automatically read the app roles from the JWT token during SSO sign-in and assign the corresponding role to the user.
+
+### 4.1 Supported Roles
+
+LiteLLM supports the following app roles (case-insensitive):
+
+- `proxy_admin` - Admin over the entire LiteLLM platform
+- `proxy_admin_viewer` - Read-only admin access (can view all keys and spend)
+- `org_admin` - Admin over a specific organization (can create teams and users within their org)
+- `internal_user` - Standard user (can create/view/delete their own keys and view their own spend)
+
+### 4.2 Create App Roles in Entra ID
+
+1. Navigate to your App Registration on https://portal.azure.com/
+2. Go to **App roles** > **Create app role**
+
+3. Configure the app role:
+ - **Display name**: Proxy Admin (or your preferred display name)
+ - **Value**: `proxy_admin` (use one of the supported role values above)
+ - **Description**: Administrator access to LiteLLM proxy
+ - **Allowed member types**: Users/Groups
+
+
+4. Click **Apply** to save the role
+
+### 4.3 Assign Users to App Roles
+
+1. Navigate to **Enterprise Applications** on https://portal.azure.com/
+2. Select your LiteLLM application
+3. Go to **Users and groups** > **Add user/group**
+4. Select the user and assign them to one of the app roles you created
+
+
+### 4.4 Test the Role Assignment
+
+1. Sign in to LiteLLM UI via SSO as a user with an assigned app role
+2. LiteLLM will automatically extract the app role from the JWT token
+3. The user will be assigned the corresponding LiteLLM role in the database
+4. The user's permissions will reflect their assigned role
+
+**How it works:**
+- When a user signs in via Microsoft SSO, LiteLLM extracts the `roles` claim from the JWT `id_token`
+- If any of the roles match a valid LiteLLM role (case-insensitive), that role is assigned to the user
+- If multiple roles are present, LiteLLM uses the first valid role it finds
+- This role assignment persists in the LiteLLM database and determines the user's access level
+
## Video Walkthrough
This walks through setting up sso auto-add for **Microsoft Entra ID**
diff --git a/docs/my-website/docs/tutorials/openweb_ui.md b/docs/my-website/docs/tutorials/openweb_ui.md
index ecf1e289da3..38f1ec38260 100644
--- a/docs/my-website/docs/tutorials/openweb_ui.md
+++ b/docs/my-website/docs/tutorials/openweb_ui.md
@@ -89,16 +89,20 @@ To track spend and usage for each Open WebUI user, configure both Open WebUI and
2. **Configure LiteLLM to Parse User Headers**
- Add the following to your LiteLLM `config.yaml` to specify a header to use for user tracking:
+ Add the following to your LiteLLM `config.yaml` to specify the request header mapping for user tracking:
```yaml
general_settings:
- user_header_name: X-OpenWebUI-User-Id
+ user_header_mappings:
+ - header_name: X-OpenWebUI-User-Id
+ litellm_user_role: internal_user
+ - header_name: X-OpenWebUI-User-Email
+ litellm_user_role: customer
```
ⓘ Available tracking options
- You can use any of the following headers for `user_header_name`:
+ You can use any of the following headers in `header_name` in `user_header_mappings` :
- `X-OpenWebUI-User-Id`
- `X-OpenWebUI-User-Email`
- `X-OpenWebUI-User-Name`
@@ -109,6 +113,12 @@ To track spend and usage for each Open WebUI user, configure both Open WebUI and
- Users can modify their own usernames
- Administrators can modify both usernames and emails of any account
+This video walks through on how we can map the openweb ui headers to LiteLLM user roles
+
+
+
+
+
## Render `thinking` content on Open WebUI
diff --git a/docs/my-website/docs/tutorials/scim_litellm.md b/docs/my-website/docs/tutorials/scim_litellm.md
index 851379610b0..f7168531f80 100644
--- a/docs/my-website/docs/tutorials/scim_litellm.md
+++ b/docs/my-website/docs/tutorials/scim_litellm.md
@@ -72,6 +72,7 @@ On the LiteLLM UI, Navigate to `Teams`, You should see the new team `Production
+> **Note:** When a user is removed from your organization via SCIM, all API keys and access tokens associated with that user will be automatically deleted from LiteLLM. This ensures that removed users lose all access immediately and securely.
diff --git a/docs/my-website/docusaurus.config.js b/docs/my-website/docusaurus.config.js
index cab1669824c..cec0479f673 100644
--- a/docs/my-website/docusaurus.config.js
+++ b/docs/my-website/docusaurus.config.js
@@ -136,6 +136,11 @@ const config = {
],
],
+ themes: ['@docusaurus/theme-mermaid'],
+ markdown: {
+ mermaid: true,
+ },
+
scripts: [
{
async: true,
diff --git a/docs/my-website/img/admin_settings_ui_theme.png b/docs/my-website/img/admin_settings_ui_theme.png
new file mode 100644
index 00000000000..81e6d761e17
Binary files /dev/null and b/docs/my-website/img/admin_settings_ui_theme.png differ
diff --git a/docs/my-website/img/admin_settings_ui_theme_logo.png b/docs/my-website/img/admin_settings_ui_theme_logo.png
new file mode 100644
index 00000000000..38f36e61602
Binary files /dev/null and b/docs/my-website/img/admin_settings_ui_theme_logo.png differ
diff --git a/docs/my-website/img/agent_1.png b/docs/my-website/img/agent_1.png
new file mode 100644
index 00000000000..42ef6ebdd98
Binary files /dev/null and b/docs/my-website/img/agent_1.png differ
diff --git a/docs/my-website/img/agent_2.png b/docs/my-website/img/agent_2.png
new file mode 100644
index 00000000000..13819a8c711
Binary files /dev/null and b/docs/my-website/img/agent_2.png differ
diff --git a/docs/my-website/img/agent_3.png b/docs/my-website/img/agent_3.png
new file mode 100644
index 00000000000..81cf96070cf
Binary files /dev/null and b/docs/my-website/img/agent_3.png differ
diff --git a/docs/my-website/img/agent_4.png b/docs/my-website/img/agent_4.png
new file mode 100644
index 00000000000..2239e70cd86
Binary files /dev/null and b/docs/my-website/img/agent_4.png differ
diff --git a/docs/my-website/img/create_team_member_rate_limits.png b/docs/my-website/img/create_team_member_rate_limits.png
new file mode 100644
index 00000000000..0c5eba04461
Binary files /dev/null and b/docs/my-website/img/create_team_member_rate_limits.png differ
diff --git a/docs/my-website/img/dd_llm_obs.png b/docs/my-website/img/dd_llm_obs.png
new file mode 100644
index 00000000000..be7c7c77178
Binary files /dev/null and b/docs/my-website/img/dd_llm_obs.png differ
diff --git a/docs/my-website/img/default_user_settings_admin_ui.png b/docs/my-website/img/default_user_settings_admin_ui.png
new file mode 100644
index 00000000000..5910154cd51
Binary files /dev/null and b/docs/my-website/img/default_user_settings_admin_ui.png differ
diff --git a/docs/my-website/img/key_r.png b/docs/my-website/img/key_r.png
new file mode 100644
index 00000000000..0e31d41fa60
Binary files /dev/null and b/docs/my-website/img/key_r.png differ
diff --git a/docs/my-website/img/key_u.png b/docs/my-website/img/key_u.png
new file mode 100644
index 00000000000..39f085dc343
Binary files /dev/null and b/docs/my-website/img/key_u.png differ
diff --git a/docs/my-website/img/mcp_tools.png b/docs/my-website/img/mcp_tools.png
new file mode 100644
index 00000000000..825dbf6ed8c
Binary files /dev/null and b/docs/my-website/img/mcp_tools.png differ
diff --git a/docs/my-website/img/mcp_updates.jpg b/docs/my-website/img/mcp_updates.jpg
new file mode 100644
index 00000000000..c53c735116c
Binary files /dev/null and b/docs/my-website/img/mcp_updates.jpg differ
diff --git a/docs/my-website/img/oauth_2_success.png b/docs/my-website/img/oauth_2_success.png
new file mode 100644
index 00000000000..4011b55d35c
Binary files /dev/null and b/docs/my-website/img/oauth_2_success.png differ
diff --git a/docs/my-website/img/release_notes/1_78_0_perf.png b/docs/my-website/img/release_notes/1_78_0_perf.png
new file mode 100644
index 00000000000..ed84c3a420a
Binary files /dev/null and b/docs/my-website/img/release_notes/1_78_0_perf.png differ
diff --git a/docs/my-website/img/release_notes/auto_router.png b/docs/my-website/img/release_notes/auto_router.png
new file mode 100644
index 00000000000..238d2dc22cd
Binary files /dev/null and b/docs/my-website/img/release_notes/auto_router.png differ
diff --git a/docs/my-website/img/release_notes/faster_caching_calls.png b/docs/my-website/img/release_notes/faster_caching_calls.png
new file mode 100644
index 00000000000..fb7409aec28
Binary files /dev/null and b/docs/my-website/img/release_notes/faster_caching_calls.png differ
diff --git a/docs/my-website/img/release_notes/mcp_header_propogation.png b/docs/my-website/img/release_notes/mcp_header_propogation.png
new file mode 100644
index 00000000000..e37d2255d11
Binary files /dev/null and b/docs/my-website/img/release_notes/mcp_header_propogation.png differ
diff --git a/docs/my-website/img/release_notes/model_level_guardrails.jpg b/docs/my-website/img/release_notes/model_level_guardrails.jpg
new file mode 100644
index 00000000000..a432bd9e296
Binary files /dev/null and b/docs/my-website/img/release_notes/model_level_guardrails.jpg differ
diff --git a/docs/my-website/img/release_notes/perf_77_5.png b/docs/my-website/img/release_notes/perf_77_5.png
new file mode 100644
index 00000000000..3aaebaf6164
Binary files /dev/null and b/docs/my-website/img/release_notes/perf_77_5.png differ
diff --git a/docs/my-website/img/release_notes/perf_77_7.png b/docs/my-website/img/release_notes/perf_77_7.png
new file mode 100644
index 00000000000..bcf6a9afd54
Binary files /dev/null and b/docs/my-website/img/release_notes/perf_77_7.png differ
diff --git a/docs/my-website/img/release_notes/perf_imp.png b/docs/my-website/img/release_notes/perf_imp.png
new file mode 100644
index 00000000000..9fef6a6b2d7
Binary files /dev/null and b/docs/my-website/img/release_notes/perf_imp.png differ
diff --git a/docs/my-website/img/release_notes/quota.png b/docs/my-website/img/release_notes/quota.png
new file mode 100644
index 00000000000..f8d15747f81
Binary files /dev/null and b/docs/my-website/img/release_notes/quota.png differ
diff --git a/docs/my-website/img/release_notes/responses_api_session_mgt_images.jpg b/docs/my-website/img/release_notes/responses_api_session_mgt_images.jpg
new file mode 100644
index 00000000000..852d2fdd6d0
Binary files /dev/null and b/docs/my-website/img/release_notes/responses_api_session_mgt_images.jpg differ
diff --git a/docs/my-website/img/release_notes/schedule_key_rotations.png b/docs/my-website/img/release_notes/schedule_key_rotations.png
new file mode 100644
index 00000000000..6ea7d8527d3
Binary files /dev/null and b/docs/my-website/img/release_notes/schedule_key_rotations.png differ
diff --git a/docs/my-website/img/release_notes/team_member_rate_limits.png b/docs/my-website/img/release_notes/team_member_rate_limits.png
new file mode 100644
index 00000000000..ec0affb1271
Binary files /dev/null and b/docs/my-website/img/release_notes/team_member_rate_limits.png differ
diff --git a/docs/my-website/img/release_notes/tool_control.png b/docs/my-website/img/release_notes/tool_control.png
new file mode 100644
index 00000000000..3d7fc42e6ad
Binary files /dev/null and b/docs/my-website/img/release_notes/tool_control.png differ
diff --git a/docs/my-website/img/tag_budget1.png b/docs/my-website/img/tag_budget1.png
new file mode 100644
index 00000000000..061e406f490
Binary files /dev/null and b/docs/my-website/img/tag_budget1.png differ
diff --git a/docs/my-website/img/tag_budget2.png b/docs/my-website/img/tag_budget2.png
new file mode 100644
index 00000000000..f44fd79dd32
Binary files /dev/null and b/docs/my-website/img/tag_budget2.png differ
diff --git a/docs/my-website/package-lock.json b/docs/my-website/package-lock.json
index da4687e0e40..b71a15cc8e6 100644
--- a/docs/my-website/package-lock.json
+++ b/docs/my-website/package-lock.json
@@ -12,6 +12,7 @@
"@docusaurus/plugin-google-gtag": "3.8.1",
"@docusaurus/plugin-ideal-image": "3.8.1",
"@docusaurus/preset-classic": "3.8.1",
+ "@docusaurus/theme-mermaid": "^3.8.1",
"@inkeep/cxkit-docusaurus": "^0.5.89",
"@mdx-js/react": "^3.0.0",
"clsx": "^1.2.1",
@@ -254,6 +255,26 @@
"node": ">=6.0.0"
}
},
+ "node_modules/@antfu/install-pkg": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@antfu/install-pkg/-/install-pkg-1.1.0.tgz",
+ "integrity": "sha512-MGQsmw10ZyI+EJo45CdSER4zEb+p31LpDAFp2Z3gkSd1yqVZGi0Ebx++YTEMonJy4oChEMLsxZ64j8FH6sSqtQ==",
+ "dependencies": {
+ "package-manager-detector": "^1.3.0",
+ "tinyexec": "^1.0.1"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/antfu"
+ }
+ },
+ "node_modules/@antfu/utils": {
+ "version": "8.1.1",
+ "resolved": "https://registry.npmjs.org/@antfu/utils/-/utils-8.1.1.tgz",
+ "integrity": "sha512-Mex9nXf9vR6AhcXmMrlz/HVgYYZpVGJ6YlPgwl7UnaFpnshXs6EK/oa5Gpf3CzENMjkvEx2tQtntGnb7UtSTOQ==",
+ "funding": {
+ "url": "https://github.com/sponsors/antfu"
+ }
+ },
"node_modules/@babel/code-frame": {
"version": "7.27.1",
"resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.27.1.tgz",
@@ -1819,6 +1840,45 @@
"node": ">=6.9.0"
}
},
+ "node_modules/@braintree/sanitize-url": {
+ "version": "7.1.1",
+ "resolved": "https://registry.npmjs.org/@braintree/sanitize-url/-/sanitize-url-7.1.1.tgz",
+ "integrity": "sha512-i1L7noDNxtFyL5DmZafWy1wRVhGehQmzZaz1HiN5e7iylJMSZR7ekOV7NsIqa5qBldlLrsKv4HbgFUVlQrz8Mw=="
+ },
+ "node_modules/@chevrotain/cst-dts-gen": {
+ "version": "11.0.3",
+ "resolved": "https://registry.npmjs.org/@chevrotain/cst-dts-gen/-/cst-dts-gen-11.0.3.tgz",
+ "integrity": "sha512-BvIKpRLeS/8UbfxXxgC33xOumsacaeCKAjAeLyOn7Pcp95HiRbrpl14S+9vaZLolnbssPIUuiUd8IvgkRyt6NQ==",
+ "dependencies": {
+ "@chevrotain/gast": "11.0.3",
+ "@chevrotain/types": "11.0.3",
+ "lodash-es": "4.17.21"
+ }
+ },
+ "node_modules/@chevrotain/gast": {
+ "version": "11.0.3",
+ "resolved": "https://registry.npmjs.org/@chevrotain/gast/-/gast-11.0.3.tgz",
+ "integrity": "sha512-+qNfcoNk70PyS/uxmj3li5NiECO+2YKZZQMbmjTqRI3Qchu8Hig/Q9vgkHpI3alNjr7M+a2St5pw5w5F6NL5/Q==",
+ "dependencies": {
+ "@chevrotain/types": "11.0.3",
+ "lodash-es": "4.17.21"
+ }
+ },
+ "node_modules/@chevrotain/regexp-to-ast": {
+ "version": "11.0.3",
+ "resolved": "https://registry.npmjs.org/@chevrotain/regexp-to-ast/-/regexp-to-ast-11.0.3.tgz",
+ "integrity": "sha512-1fMHaBZxLFvWI067AVbGJav1eRY7N8DDvYCTwGBiE/ytKBgP8azTdgyrKyWZ9Mfh09eHWb5PgTSO8wi7U824RA=="
+ },
+ "node_modules/@chevrotain/types": {
+ "version": "11.0.3",
+ "resolved": "https://registry.npmjs.org/@chevrotain/types/-/types-11.0.3.tgz",
+ "integrity": "sha512-gsiM3G8b58kZC2HaWR50gu6Y1440cHiJ+i3JUvcp/35JchYejb2+5MVeJK0iKThYpAa/P2PYFV4hoi44HD+aHQ=="
+ },
+ "node_modules/@chevrotain/utils": {
+ "version": "11.0.3",
+ "resolved": "https://registry.npmjs.org/@chevrotain/utils/-/utils-11.0.3.tgz",
+ "integrity": "sha512-YslZMgtJUyuMbZ+aKvfF3x1f5liK4mWNxghFRv7jqRR9C3R3fAOGTTKvxXDa2Y1s9zSbcpuO0cAxDYsc9SrXoQ=="
+ },
"node_modules/@colors/colors": {
"version": "1.5.0",
"resolved": "https://registry.npmjs.org/@colors/colors/-/colors-1.5.0.tgz",
@@ -3590,6 +3650,27 @@
"react": ">=16.0.0"
}
},
+ "node_modules/@docusaurus/theme-mermaid": {
+ "version": "3.8.1",
+ "resolved": "https://registry.npmjs.org/@docusaurus/theme-mermaid/-/theme-mermaid-3.8.1.tgz",
+ "integrity": "sha512-IWYqjyTPjkNnHsFFu9+4YkeXS7PD1xI3Bn2shOhBq+f95mgDfWInkpfBN4aYvx4fTT67Am6cPtohRdwh4Tidtg==",
+ "dependencies": {
+ "@docusaurus/core": "3.8.1",
+ "@docusaurus/module-type-aliases": "3.8.1",
+ "@docusaurus/theme-common": "3.8.1",
+ "@docusaurus/types": "3.8.1",
+ "@docusaurus/utils-validation": "3.8.1",
+ "mermaid": ">=11.6.0",
+ "tslib": "^2.6.0"
+ },
+ "engines": {
+ "node": ">=18.0"
+ },
+ "peerDependencies": {
+ "react": "^18.0.0 || ^19.0.0",
+ "react-dom": "^18.0.0 || ^19.0.0"
+ }
+ },
"node_modules/@docusaurus/theme-search-algolia": {
"version": "3.8.1",
"resolved": "https://registry.npmjs.org/@docusaurus/theme-search-algolia/-/theme-search-algolia-3.8.1.tgz",
@@ -3781,6 +3862,37 @@
"@hapi/hoek": "^9.0.0"
}
},
+ "node_modules/@iconify/types": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/@iconify/types/-/types-2.0.0.tgz",
+ "integrity": "sha512-+wluvCrRhXrhyOmRDJ3q8mux9JkKy5SJ/v8ol2tu4FVjyYvtEzkc/3pK15ET6RKg4b4w4BmTk1+gsCUhf21Ykg=="
+ },
+ "node_modules/@iconify/utils": {
+ "version": "2.3.0",
+ "resolved": "https://registry.npmjs.org/@iconify/utils/-/utils-2.3.0.tgz",
+ "integrity": "sha512-GmQ78prtwYW6EtzXRU1rY+KwOKfz32PD7iJh6Iyqw68GiKuoZ2A6pRtzWONz5VQJbp50mEjXh/7NkumtrAgRKA==",
+ "dependencies": {
+ "@antfu/install-pkg": "^1.0.0",
+ "@antfu/utils": "^8.1.0",
+ "@iconify/types": "^2.0.0",
+ "debug": "^4.4.0",
+ "globals": "^15.14.0",
+ "kolorist": "^1.8.0",
+ "local-pkg": "^1.0.0",
+ "mlly": "^1.7.4"
+ }
+ },
+ "node_modules/@iconify/utils/node_modules/globals": {
+ "version": "15.15.0",
+ "resolved": "https://registry.npmjs.org/globals/-/globals-15.15.0.tgz",
+ "integrity": "sha512-7ACyT3wmyp3I61S4fG682L0VA2RGD9otkqGJIwNUMF1SWUombIIk+af1unuDYgMm082aHYwD+mzJvv9Iu8dsgg==",
+ "engines": {
+ "node": ">=18"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/sindresorhus"
+ }
+ },
"node_modules/@inkeep/cxkit-color-mode": {
"version": "0.5.91",
"resolved": "https://registry.npmjs.org/@inkeep/cxkit-color-mode/-/cxkit-color-mode-0.5.91.tgz",
@@ -4106,6 +4218,14 @@
"react": ">=16"
}
},
+ "node_modules/@mermaid-js/parser": {
+ "version": "0.6.2",
+ "resolved": "https://registry.npmjs.org/@mermaid-js/parser/-/parser-0.6.2.tgz",
+ "integrity": "sha512-+PO02uGF6L6Cs0Bw8RpGhikVvMWEysfAyl27qTlroUB8jSWr1lL0Sf6zi78ZxlSnmgSY2AMMKVgghnN9jTtwkQ==",
+ "dependencies": {
+ "langium": "3.3.1"
+ }
+ },
"node_modules/@nodelib/fs.scandir": {
"version": "2.1.5",
"resolved": "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz",
@@ -6385,6 +6505,228 @@
"@types/node": "*"
}
},
+ "node_modules/@types/d3": {
+ "version": "7.4.3",
+ "resolved": "https://registry.npmjs.org/@types/d3/-/d3-7.4.3.tgz",
+ "integrity": "sha512-lZXZ9ckh5R8uiFVt8ogUNf+pIrK4EsWrx2Np75WvF/eTpJ0FMHNhjXk8CKEx/+gpHbNQyJWehbFaTvqmHWB3ww==",
+ "dependencies": {
+ "@types/d3-array": "*",
+ "@types/d3-axis": "*",
+ "@types/d3-brush": "*",
+ "@types/d3-chord": "*",
+ "@types/d3-color": "*",
+ "@types/d3-contour": "*",
+ "@types/d3-delaunay": "*",
+ "@types/d3-dispatch": "*",
+ "@types/d3-drag": "*",
+ "@types/d3-dsv": "*",
+ "@types/d3-ease": "*",
+ "@types/d3-fetch": "*",
+ "@types/d3-force": "*",
+ "@types/d3-format": "*",
+ "@types/d3-geo": "*",
+ "@types/d3-hierarchy": "*",
+ "@types/d3-interpolate": "*",
+ "@types/d3-path": "*",
+ "@types/d3-polygon": "*",
+ "@types/d3-quadtree": "*",
+ "@types/d3-random": "*",
+ "@types/d3-scale": "*",
+ "@types/d3-scale-chromatic": "*",
+ "@types/d3-selection": "*",
+ "@types/d3-shape": "*",
+ "@types/d3-time": "*",
+ "@types/d3-time-format": "*",
+ "@types/d3-timer": "*",
+ "@types/d3-transition": "*",
+ "@types/d3-zoom": "*"
+ }
+ },
+ "node_modules/@types/d3-array": {
+ "version": "3.2.1",
+ "resolved": "https://registry.npmjs.org/@types/d3-array/-/d3-array-3.2.1.tgz",
+ "integrity": "sha512-Y2Jn2idRrLzUfAKV2LyRImR+y4oa2AntrgID95SHJxuMUrkNXmanDSed71sRNZysveJVt1hLLemQZIady0FpEg=="
+ },
+ "node_modules/@types/d3-axis": {
+ "version": "3.0.6",
+ "resolved": "https://registry.npmjs.org/@types/d3-axis/-/d3-axis-3.0.6.tgz",
+ "integrity": "sha512-pYeijfZuBd87T0hGn0FO1vQ/cgLk6E1ALJjfkC0oJ8cbwkZl3TpgS8bVBLZN+2jjGgg38epgxb2zmoGtSfvgMw==",
+ "dependencies": {
+ "@types/d3-selection": "*"
+ }
+ },
+ "node_modules/@types/d3-brush": {
+ "version": "3.0.6",
+ "resolved": "https://registry.npmjs.org/@types/d3-brush/-/d3-brush-3.0.6.tgz",
+ "integrity": "sha512-nH60IZNNxEcrh6L1ZSMNA28rj27ut/2ZmI3r96Zd+1jrZD++zD3LsMIjWlvg4AYrHn/Pqz4CF3veCxGjtbqt7A==",
+ "dependencies": {
+ "@types/d3-selection": "*"
+ }
+ },
+ "node_modules/@types/d3-chord": {
+ "version": "3.0.6",
+ "resolved": "https://registry.npmjs.org/@types/d3-chord/-/d3-chord-3.0.6.tgz",
+ "integrity": "sha512-LFYWWd8nwfwEmTZG9PfQxd17HbNPksHBiJHaKuY1XeqscXacsS2tyoo6OdRsjf+NQYeB6XrNL3a25E3gH69lcg=="
+ },
+ "node_modules/@types/d3-color": {
+ "version": "3.1.3",
+ "resolved": "https://registry.npmjs.org/@types/d3-color/-/d3-color-3.1.3.tgz",
+ "integrity": "sha512-iO90scth9WAbmgv7ogoq57O9YpKmFBbmoEoCHDB2xMBY0+/KVrqAaCDyCE16dUspeOvIxFFRI+0sEtqDqy2b4A=="
+ },
+ "node_modules/@types/d3-contour": {
+ "version": "3.0.6",
+ "resolved": "https://registry.npmjs.org/@types/d3-contour/-/d3-contour-3.0.6.tgz",
+ "integrity": "sha512-BjzLgXGnCWjUSYGfH1cpdo41/hgdWETu4YxpezoztawmqsvCeep+8QGfiY6YbDvfgHz/DkjeIkkZVJavB4a3rg==",
+ "dependencies": {
+ "@types/d3-array": "*",
+ "@types/geojson": "*"
+ }
+ },
+ "node_modules/@types/d3-delaunay": {
+ "version": "6.0.4",
+ "resolved": "https://registry.npmjs.org/@types/d3-delaunay/-/d3-delaunay-6.0.4.tgz",
+ "integrity": "sha512-ZMaSKu4THYCU6sV64Lhg6qjf1orxBthaC161plr5KuPHo3CNm8DTHiLw/5Eq2b6TsNP0W0iJrUOFscY6Q450Hw=="
+ },
+ "node_modules/@types/d3-dispatch": {
+ "version": "3.0.7",
+ "resolved": "https://registry.npmjs.org/@types/d3-dispatch/-/d3-dispatch-3.0.7.tgz",
+ "integrity": "sha512-5o9OIAdKkhN1QItV2oqaE5KMIiXAvDWBDPrD85e58Qlz1c1kI/J0NcqbEG88CoTwJrYe7ntUCVfeUl2UJKbWgA=="
+ },
+ "node_modules/@types/d3-drag": {
+ "version": "3.0.7",
+ "resolved": "https://registry.npmjs.org/@types/d3-drag/-/d3-drag-3.0.7.tgz",
+ "integrity": "sha512-HE3jVKlzU9AaMazNufooRJ5ZpWmLIoc90A37WU2JMmeq28w1FQqCZswHZ3xR+SuxYftzHq6WU6KJHvqxKzTxxQ==",
+ "dependencies": {
+ "@types/d3-selection": "*"
+ }
+ },
+ "node_modules/@types/d3-dsv": {
+ "version": "3.0.7",
+ "resolved": "https://registry.npmjs.org/@types/d3-dsv/-/d3-dsv-3.0.7.tgz",
+ "integrity": "sha512-n6QBF9/+XASqcKK6waudgL0pf/S5XHPPI8APyMLLUHd8NqouBGLsU8MgtO7NINGtPBtk9Kko/W4ea0oAspwh9g=="
+ },
+ "node_modules/@types/d3-ease": {
+ "version": "3.0.2",
+ "resolved": "https://registry.npmjs.org/@types/d3-ease/-/d3-ease-3.0.2.tgz",
+ "integrity": "sha512-NcV1JjO5oDzoK26oMzbILE6HW7uVXOHLQvHshBUW4UMdZGfiY6v5BeQwh9a9tCzv+CeefZQHJt5SRgK154RtiA=="
+ },
+ "node_modules/@types/d3-fetch": {
+ "version": "3.0.7",
+ "resolved": "https://registry.npmjs.org/@types/d3-fetch/-/d3-fetch-3.0.7.tgz",
+ "integrity": "sha512-fTAfNmxSb9SOWNB9IoG5c8Hg6R+AzUHDRlsXsDZsNp6sxAEOP0tkP3gKkNSO/qmHPoBFTxNrjDprVHDQDvo5aA==",
+ "dependencies": {
+ "@types/d3-dsv": "*"
+ }
+ },
+ "node_modules/@types/d3-force": {
+ "version": "3.0.10",
+ "resolved": "https://registry.npmjs.org/@types/d3-force/-/d3-force-3.0.10.tgz",
+ "integrity": "sha512-ZYeSaCF3p73RdOKcjj+swRlZfnYpK1EbaDiYICEEp5Q6sUiqFaFQ9qgoshp5CzIyyb/yD09kD9o2zEltCexlgw=="
+ },
+ "node_modules/@types/d3-format": {
+ "version": "3.0.4",
+ "resolved": "https://registry.npmjs.org/@types/d3-format/-/d3-format-3.0.4.tgz",
+ "integrity": "sha512-fALi2aI6shfg7vM5KiR1wNJnZ7r6UuggVqtDA+xiEdPZQwy/trcQaHnwShLuLdta2rTymCNpxYTiMZX/e09F4g=="
+ },
+ "node_modules/@types/d3-geo": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/@types/d3-geo/-/d3-geo-3.1.0.tgz",
+ "integrity": "sha512-856sckF0oP/diXtS4jNsiQw/UuK5fQG8l/a9VVLeSouf1/PPbBE1i1W852zVwKwYCBkFJJB7nCFTbk6UMEXBOQ==",
+ "dependencies": {
+ "@types/geojson": "*"
+ }
+ },
+ "node_modules/@types/d3-hierarchy": {
+ "version": "3.1.7",
+ "resolved": "https://registry.npmjs.org/@types/d3-hierarchy/-/d3-hierarchy-3.1.7.tgz",
+ "integrity": "sha512-tJFtNoYBtRtkNysX1Xq4sxtjK8YgoWUNpIiUee0/jHGRwqvzYxkq0hGVbbOGSz+JgFxxRu4K8nb3YpG3CMARtg=="
+ },
+ "node_modules/@types/d3-interpolate": {
+ "version": "3.0.4",
+ "resolved": "https://registry.npmjs.org/@types/d3-interpolate/-/d3-interpolate-3.0.4.tgz",
+ "integrity": "sha512-mgLPETlrpVV1YRJIglr4Ez47g7Yxjl1lj7YKsiMCb27VJH9W8NVM6Bb9d8kkpG/uAQS5AmbA48q2IAolKKo1MA==",
+ "dependencies": {
+ "@types/d3-color": "*"
+ }
+ },
+ "node_modules/@types/d3-path": {
+ "version": "3.1.1",
+ "resolved": "https://registry.npmjs.org/@types/d3-path/-/d3-path-3.1.1.tgz",
+ "integrity": "sha512-VMZBYyQvbGmWyWVea0EHs/BwLgxc+MKi1zLDCONksozI4YJMcTt8ZEuIR4Sb1MMTE8MMW49v0IwI5+b7RmfWlg=="
+ },
+ "node_modules/@types/d3-polygon": {
+ "version": "3.0.2",
+ "resolved": "https://registry.npmjs.org/@types/d3-polygon/-/d3-polygon-3.0.2.tgz",
+ "integrity": "sha512-ZuWOtMaHCkN9xoeEMr1ubW2nGWsp4nIql+OPQRstu4ypeZ+zk3YKqQT0CXVe/PYqrKpZAi+J9mTs05TKwjXSRA=="
+ },
+ "node_modules/@types/d3-quadtree": {
+ "version": "3.0.6",
+ "resolved": "https://registry.npmjs.org/@types/d3-quadtree/-/d3-quadtree-3.0.6.tgz",
+ "integrity": "sha512-oUzyO1/Zm6rsxKRHA1vH0NEDG58HrT5icx/azi9MF1TWdtttWl0UIUsjEQBBh+SIkrpd21ZjEv7ptxWys1ncsg=="
+ },
+ "node_modules/@types/d3-random": {
+ "version": "3.0.3",
+ "resolved": "https://registry.npmjs.org/@types/d3-random/-/d3-random-3.0.3.tgz",
+ "integrity": "sha512-Imagg1vJ3y76Y2ea0871wpabqp613+8/r0mCLEBfdtqC7xMSfj9idOnmBYyMoULfHePJyxMAw3nWhJxzc+LFwQ=="
+ },
+ "node_modules/@types/d3-scale": {
+ "version": "4.0.9",
+ "resolved": "https://registry.npmjs.org/@types/d3-scale/-/d3-scale-4.0.9.tgz",
+ "integrity": "sha512-dLmtwB8zkAeO/juAMfnV+sItKjlsw2lKdZVVy6LRr0cBmegxSABiLEpGVmSJJ8O08i4+sGR6qQtb6WtuwJdvVw==",
+ "dependencies": {
+ "@types/d3-time": "*"
+ }
+ },
+ "node_modules/@types/d3-scale-chromatic": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/@types/d3-scale-chromatic/-/d3-scale-chromatic-3.1.0.tgz",
+ "integrity": "sha512-iWMJgwkK7yTRmWqRB5plb1kadXyQ5Sj8V/zYlFGMUBbIPKQScw+Dku9cAAMgJG+z5GYDoMjWGLVOvjghDEFnKQ=="
+ },
+ "node_modules/@types/d3-selection": {
+ "version": "3.0.11",
+ "resolved": "https://registry.npmjs.org/@types/d3-selection/-/d3-selection-3.0.11.tgz",
+ "integrity": "sha512-bhAXu23DJWsrI45xafYpkQ4NtcKMwWnAC/vKrd2l+nxMFuvOT3XMYTIj2opv8vq8AO5Yh7Qac/nSeP/3zjTK0w=="
+ },
+ "node_modules/@types/d3-shape": {
+ "version": "3.1.7",
+ "resolved": "https://registry.npmjs.org/@types/d3-shape/-/d3-shape-3.1.7.tgz",
+ "integrity": "sha512-VLvUQ33C+3J+8p+Daf+nYSOsjB4GXp19/S/aGo60m9h1v6XaxjiT82lKVWJCfzhtuZ3yD7i/TPeC/fuKLLOSmg==",
+ "dependencies": {
+ "@types/d3-path": "*"
+ }
+ },
+ "node_modules/@types/d3-time": {
+ "version": "3.0.4",
+ "resolved": "https://registry.npmjs.org/@types/d3-time/-/d3-time-3.0.4.tgz",
+ "integrity": "sha512-yuzZug1nkAAaBlBBikKZTgzCeA+k1uy4ZFwWANOfKw5z5LRhV0gNA7gNkKm7HoK+HRN0wX3EkxGk0fpbWhmB7g=="
+ },
+ "node_modules/@types/d3-time-format": {
+ "version": "4.0.3",
+ "resolved": "https://registry.npmjs.org/@types/d3-time-format/-/d3-time-format-4.0.3.tgz",
+ "integrity": "sha512-5xg9rC+wWL8kdDj153qZcsJ0FWiFt0J5RB6LYUNZjwSnesfblqrI/bJ1wBdJ8OQfncgbJG5+2F+qfqnqyzYxyg=="
+ },
+ "node_modules/@types/d3-timer": {
+ "version": "3.0.2",
+ "resolved": "https://registry.npmjs.org/@types/d3-timer/-/d3-timer-3.0.2.tgz",
+ "integrity": "sha512-Ps3T8E8dZDam6fUyNiMkekK3XUsaUEik+idO9/YjPtfj2qruF8tFBXS7XhtE4iIXBLxhmLjP3SXpLhVf21I9Lw=="
+ },
+ "node_modules/@types/d3-transition": {
+ "version": "3.0.9",
+ "resolved": "https://registry.npmjs.org/@types/d3-transition/-/d3-transition-3.0.9.tgz",
+ "integrity": "sha512-uZS5shfxzO3rGlu0cC3bjmMFKsXv+SmZZcgp0KD22ts4uGXp5EVYGzu/0YdwZeKmddhcAccYtREJKkPfXkZuCg==",
+ "dependencies": {
+ "@types/d3-selection": "*"
+ }
+ },
+ "node_modules/@types/d3-zoom": {
+ "version": "3.0.8",
+ "resolved": "https://registry.npmjs.org/@types/d3-zoom/-/d3-zoom-3.0.8.tgz",
+ "integrity": "sha512-iqMC4/YlFCSlO8+2Ii1GGGliCAY4XdeG748w5vQUbevlbDu0zSjH/+jojorQVBK/se0j6DUFNPBGSqD3YWYnDw==",
+ "dependencies": {
+ "@types/d3-interpolate": "*",
+ "@types/d3-selection": "*"
+ }
+ },
"node_modules/@types/debug": {
"version": "4.1.12",
"resolved": "https://registry.npmjs.org/@types/debug/-/debug-4.1.12.tgz",
@@ -6457,6 +6799,11 @@
"@types/send": "*"
}
},
+ "node_modules/@types/geojson": {
+ "version": "7946.0.16",
+ "resolved": "https://registry.npmjs.org/@types/geojson/-/geojson-7946.0.16.tgz",
+ "integrity": "sha512-6C8nqWur3j98U6+lXDfTUWIfgvZU+EumvpHKcYjujKH7woYyLj2sUmff0tRhrqM7BohUw7Pz3ZB1jj2gW9Fvmg=="
+ },
"node_modules/@types/gtag.js": {
"version": "0.0.12",
"resolved": "https://registry.npmjs.org/@types/gtag.js/-/gtag.js-0.0.12.tgz",
@@ -6677,6 +7024,12 @@
"@types/node": "*"
}
},
+ "node_modules/@types/trusted-types": {
+ "version": "2.0.7",
+ "resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz",
+ "integrity": "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw==",
+ "optional": true
+ },
"node_modules/@types/unist": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/@types/unist/-/unist-3.0.3.tgz",
@@ -7875,6 +8228,30 @@
"url": "https://github.com/sponsors/fb55"
}
},
+ "node_modules/chevrotain": {
+ "version": "11.0.3",
+ "resolved": "https://registry.npmjs.org/chevrotain/-/chevrotain-11.0.3.tgz",
+ "integrity": "sha512-ci2iJH6LeIkvP9eJW6gpueU8cnZhv85ELY8w8WiFtNjMHA5ad6pQLaJo9mEly/9qUyCpvqX8/POVUTf18/HFdw==",
+ "dependencies": {
+ "@chevrotain/cst-dts-gen": "11.0.3",
+ "@chevrotain/gast": "11.0.3",
+ "@chevrotain/regexp-to-ast": "11.0.3",
+ "@chevrotain/types": "11.0.3",
+ "@chevrotain/utils": "11.0.3",
+ "lodash-es": "4.17.21"
+ }
+ },
+ "node_modules/chevrotain-allstar": {
+ "version": "0.3.1",
+ "resolved": "https://registry.npmjs.org/chevrotain-allstar/-/chevrotain-allstar-0.3.1.tgz",
+ "integrity": "sha512-b7g+y9A0v4mxCW1qUhf3BSVPg+/NvGErk/dOkrDaHA0nQIQGAtrOjlX//9OQtRlSCy+x9rfB5N8yC71lH1nvMw==",
+ "dependencies": {
+ "lodash-es": "^4.17.21"
+ },
+ "peerDependencies": {
+ "chevrotain": "^11.0.0"
+ }
+ },
"node_modules/chokidar": {
"version": "3.6.0",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.6.0.tgz",
@@ -8203,6 +8580,11 @@
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
"integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg=="
},
+ "node_modules/confbox": {
+ "version": "0.2.2",
+ "resolved": "https://registry.npmjs.org/confbox/-/confbox-0.2.2.tgz",
+ "integrity": "sha512-1NB+BKqhtNipMsov4xI/NnhCKp9XG9NamYp5PVm9klAT0fsrNPjaFICsCFhNhwZJKNh7zB/3q8qXz0E9oaMNtQ=="
+ },
"node_modules/config-chain": {
"version": "1.1.13",
"resolved": "https://registry.npmjs.org/config-chain/-/config-chain-1.1.13.tgz",
@@ -8394,6 +8776,14 @@
"resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.3.tgz",
"integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ=="
},
+ "node_modules/cose-base": {
+ "version": "1.0.3",
+ "resolved": "https://registry.npmjs.org/cose-base/-/cose-base-1.0.3.tgz",
+ "integrity": "sha512-s9whTXInMSgAp/NVXVNuVxVKzGH2qck3aQlVHxDCdAEPgtMKwc4Wq6/QKhgdEdgbLSi9rBTAcPoRa6JpiG4ksg==",
+ "dependencies": {
+ "layout-base": "^1.0.0"
+ }
+ },
"node_modules/cosmiconfig": {
"version": "8.3.6",
"resolved": "https://registry.npmjs.org/cosmiconfig/-/cosmiconfig-8.3.6.tgz",
@@ -8853,6 +9243,487 @@
"resolved": "https://registry.npmjs.org/csstype/-/csstype-3.1.3.tgz",
"integrity": "sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw=="
},
+ "node_modules/cytoscape": {
+ "version": "3.33.0",
+ "resolved": "https://registry.npmjs.org/cytoscape/-/cytoscape-3.33.0.tgz",
+ "integrity": "sha512-2d2EwwhaxLWC8ahkH1PpQwCyu6EY3xDRdcEJXrLTb4fOUtVc+YWQalHU67rFS1a6ngj1fgv9dQLtJxP/KAFZEw==",
+ "engines": {
+ "node": ">=0.10"
+ }
+ },
+ "node_modules/cytoscape-cose-bilkent": {
+ "version": "4.1.0",
+ "resolved": "https://registry.npmjs.org/cytoscape-cose-bilkent/-/cytoscape-cose-bilkent-4.1.0.tgz",
+ "integrity": "sha512-wgQlVIUJF13Quxiv5e1gstZ08rnZj2XaLHGoFMYXz7SkNfCDOOteKBE6SYRfA9WxxI/iBc3ajfDoc6hb/MRAHQ==",
+ "dependencies": {
+ "cose-base": "^1.0.0"
+ },
+ "peerDependencies": {
+ "cytoscape": "^3.2.0"
+ }
+ },
+ "node_modules/cytoscape-fcose": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/cytoscape-fcose/-/cytoscape-fcose-2.2.0.tgz",
+ "integrity": "sha512-ki1/VuRIHFCzxWNrsshHYPs6L7TvLu3DL+TyIGEsRcvVERmxokbf5Gdk7mFxZnTdiGtnA4cfSmjZJMviqSuZrQ==",
+ "dependencies": {
+ "cose-base": "^2.2.0"
+ },
+ "peerDependencies": {
+ "cytoscape": "^3.2.0"
+ }
+ },
+ "node_modules/cytoscape-fcose/node_modules/cose-base": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/cose-base/-/cose-base-2.2.0.tgz",
+ "integrity": "sha512-AzlgcsCbUMymkADOJtQm3wO9S3ltPfYOFD5033keQn9NJzIbtnZj+UdBJe7DYml/8TdbtHJW3j58SOnKhWY/5g==",
+ "dependencies": {
+ "layout-base": "^2.0.0"
+ }
+ },
+ "node_modules/cytoscape-fcose/node_modules/layout-base": {
+ "version": "2.0.1",
+ "resolved": "https://registry.npmjs.org/layout-base/-/layout-base-2.0.1.tgz",
+ "integrity": "sha512-dp3s92+uNI1hWIpPGH3jK2kxE2lMjdXdr+DH8ynZHpd6PUlH6x6cbuXnoMmiNumznqaNO31xu9e79F0uuZ0JFg=="
+ },
+ "node_modules/d3": {
+ "version": "7.9.0",
+ "resolved": "https://registry.npmjs.org/d3/-/d3-7.9.0.tgz",
+ "integrity": "sha512-e1U46jVP+w7Iut8Jt8ri1YsPOvFpg46k+K8TpCb0P+zjCkjkPnV7WzfDJzMHy1LnA+wj5pLT1wjO901gLXeEhA==",
+ "dependencies": {
+ "d3-array": "3",
+ "d3-axis": "3",
+ "d3-brush": "3",
+ "d3-chord": "3",
+ "d3-color": "3",
+ "d3-contour": "4",
+ "d3-delaunay": "6",
+ "d3-dispatch": "3",
+ "d3-drag": "3",
+ "d3-dsv": "3",
+ "d3-ease": "3",
+ "d3-fetch": "3",
+ "d3-force": "3",
+ "d3-format": "3",
+ "d3-geo": "3",
+ "d3-hierarchy": "3",
+ "d3-interpolate": "3",
+ "d3-path": "3",
+ "d3-polygon": "3",
+ "d3-quadtree": "3",
+ "d3-random": "3",
+ "d3-scale": "4",
+ "d3-scale-chromatic": "3",
+ "d3-selection": "3",
+ "d3-shape": "3",
+ "d3-time": "3",
+ "d3-time-format": "4",
+ "d3-timer": "3",
+ "d3-transition": "3",
+ "d3-zoom": "3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-array": {
+ "version": "3.2.4",
+ "resolved": "https://registry.npmjs.org/d3-array/-/d3-array-3.2.4.tgz",
+ "integrity": "sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==",
+ "dependencies": {
+ "internmap": "1 - 2"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-axis": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/d3-axis/-/d3-axis-3.0.0.tgz",
+ "integrity": "sha512-IH5tgjV4jE/GhHkRV0HiVYPDtvfjHQlQfJHs0usq7M30XcSBvOotpmH1IgkcXsO/5gEQZD43B//fc7SRT5S+xw==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-brush": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/d3-brush/-/d3-brush-3.0.0.tgz",
+ "integrity": "sha512-ALnjWlVYkXsVIGlOsuWH1+3udkYFI48Ljihfnh8FZPF2QS9o+PzGLBslO0PjzVoHLZ2KCVgAM8NVkXPJB2aNnQ==",
+ "dependencies": {
+ "d3-dispatch": "1 - 3",
+ "d3-drag": "2 - 3",
+ "d3-interpolate": "1 - 3",
+ "d3-selection": "3",
+ "d3-transition": "3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-chord": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-chord/-/d3-chord-3.0.1.tgz",
+ "integrity": "sha512-VE5S6TNa+j8msksl7HwjxMHDM2yNK3XCkusIlpX5kwauBfXuyLAtNg9jCp/iHH61tgI4sb6R/EIMWCqEIdjT/g==",
+ "dependencies": {
+ "d3-path": "1 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-color": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/d3-color/-/d3-color-3.1.0.tgz",
+ "integrity": "sha512-zg/chbXyeBtMQ1LbD/WSoW2DpC3I0mpmPdW+ynRTj/x2DAWYrIY7qeZIHidozwV24m4iavr15lNwIwLxRmOxhA==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-contour": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/d3-contour/-/d3-contour-4.0.2.tgz",
+ "integrity": "sha512-4EzFTRIikzs47RGmdxbeUvLWtGedDUNkTcmzoeyg4sP/dvCexO47AaQL7VKy/gul85TOxw+IBgA8US2xwbToNA==",
+ "dependencies": {
+ "d3-array": "^3.2.0"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-delaunay": {
+ "version": "6.0.4",
+ "resolved": "https://registry.npmjs.org/d3-delaunay/-/d3-delaunay-6.0.4.tgz",
+ "integrity": "sha512-mdjtIZ1XLAM8bm/hx3WwjfHt6Sggek7qH043O8KEjDXN40xi3vx/6pYSVTwLjEgiXQTbvaouWKynLBiUZ6SK6A==",
+ "dependencies": {
+ "delaunator": "5"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-dispatch": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-dispatch/-/d3-dispatch-3.0.1.tgz",
+ "integrity": "sha512-rzUyPU/S7rwUflMyLc1ETDeBj0NRuHKKAcvukozwhshr6g6c5d8zh4c2gQjY2bZ0dXeGLWc1PF174P2tVvKhfg==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-drag": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/d3-drag/-/d3-drag-3.0.0.tgz",
+ "integrity": "sha512-pWbUJLdETVA8lQNJecMxoXfH6x+mO2UQo8rSmZ+QqxcbyA3hfeprFgIT//HW2nlHChWeIIMwS2Fq+gEARkhTkg==",
+ "dependencies": {
+ "d3-dispatch": "1 - 3",
+ "d3-selection": "3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-dsv": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-dsv/-/d3-dsv-3.0.1.tgz",
+ "integrity": "sha512-UG6OvdI5afDIFP9w4G0mNq50dSOsXHJaRE8arAS5o9ApWnIElp8GZw1Dun8vP8OyHOZ/QJUKUJwxiiCCnUwm+Q==",
+ "dependencies": {
+ "commander": "7",
+ "iconv-lite": "0.6",
+ "rw": "1"
+ },
+ "bin": {
+ "csv2json": "bin/dsv2json.js",
+ "csv2tsv": "bin/dsv2dsv.js",
+ "dsv2dsv": "bin/dsv2dsv.js",
+ "dsv2json": "bin/dsv2json.js",
+ "json2csv": "bin/json2dsv.js",
+ "json2dsv": "bin/json2dsv.js",
+ "json2tsv": "bin/json2dsv.js",
+ "tsv2csv": "bin/dsv2dsv.js",
+ "tsv2json": "bin/dsv2json.js"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-dsv/node_modules/commander": {
+ "version": "7.2.0",
+ "resolved": "https://registry.npmjs.org/commander/-/commander-7.2.0.tgz",
+ "integrity": "sha512-QrWXB+ZQSVPmIWIhtEO9H+gwHaMGYiF5ChvoJ+K9ZGHG/sVsa6yiesAD1GC/x46sET00Xlwo1u49RVVVzvcSkw==",
+ "engines": {
+ "node": ">= 10"
+ }
+ },
+ "node_modules/d3-dsv/node_modules/iconv-lite": {
+ "version": "0.6.3",
+ "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
+ "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==",
+ "dependencies": {
+ "safer-buffer": ">= 2.1.2 < 3.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/d3-ease": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-ease/-/d3-ease-3.0.1.tgz",
+ "integrity": "sha512-wR/XK3D3XcLIZwpbvQwQ5fK+8Ykds1ip7A2Txe0yxncXSdq1L9skcG7blcedkOX+ZcgxGAmLX1FrRGbADwzi0w==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-fetch": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-fetch/-/d3-fetch-3.0.1.tgz",
+ "integrity": "sha512-kpkQIM20n3oLVBKGg6oHrUchHM3xODkTzjMoj7aWQFq5QEM+R6E4WkzT5+tojDY7yjez8KgCBRoj4aEr99Fdqw==",
+ "dependencies": {
+ "d3-dsv": "1 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-force": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/d3-force/-/d3-force-3.0.0.tgz",
+ "integrity": "sha512-zxV/SsA+U4yte8051P4ECydjD/S+qeYtnaIyAs9tgHCqfguma/aAQDjo85A9Z6EKhBirHRJHXIgJUlffT4wdLg==",
+ "dependencies": {
+ "d3-dispatch": "1 - 3",
+ "d3-quadtree": "1 - 3",
+ "d3-timer": "1 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-format": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/d3-format/-/d3-format-3.1.0.tgz",
+ "integrity": "sha512-YyUI6AEuY/Wpt8KWLgZHsIU86atmikuoOmCfommt0LYHiQSPjvX2AcFc38PX0CBpr2RCyZhjex+NS/LPOv6YqA==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-geo": {
+ "version": "3.1.1",
+ "resolved": "https://registry.npmjs.org/d3-geo/-/d3-geo-3.1.1.tgz",
+ "integrity": "sha512-637ln3gXKXOwhalDzinUgY83KzNWZRKbYubaG+fGVuc/dxO64RRljtCTnf5ecMyE1RIdtqpkVcq0IbtU2S8j2Q==",
+ "dependencies": {
+ "d3-array": "2.5.0 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-hierarchy": {
+ "version": "3.1.2",
+ "resolved": "https://registry.npmjs.org/d3-hierarchy/-/d3-hierarchy-3.1.2.tgz",
+ "integrity": "sha512-FX/9frcub54beBdugHjDCdikxThEqjnR93Qt7PvQTOHxyiNCAlvMrHhclk3cD5VeAaq9fxmfRp+CnWw9rEMBuA==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-interpolate": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-interpolate/-/d3-interpolate-3.0.1.tgz",
+ "integrity": "sha512-3bYs1rOD33uo8aqJfKP3JWPAibgw8Zm2+L9vBKEHJ2Rg+viTR7o5Mmv5mZcieN+FRYaAOWX5SJATX6k1PWz72g==",
+ "dependencies": {
+ "d3-color": "1 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-path": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/d3-path/-/d3-path-3.1.0.tgz",
+ "integrity": "sha512-p3KP5HCf/bvjBSSKuXid6Zqijx7wIfNW+J/maPs+iwR35at5JCbLUT0LzF1cnjbCHWhqzQTIN2Jpe8pRebIEFQ==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-polygon": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-polygon/-/d3-polygon-3.0.1.tgz",
+ "integrity": "sha512-3vbA7vXYwfe1SYhED++fPUQlWSYTTGmFmQiany/gdbiWgU/iEyQzyymwL9SkJjFFuCS4902BSzewVGsHHmHtXg==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-quadtree": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-quadtree/-/d3-quadtree-3.0.1.tgz",
+ "integrity": "sha512-04xDrxQTDTCFwP5H6hRhsRcb9xxv2RzkcsygFzmkSIOJy3PeRJP7sNk3VRIbKXcog561P9oU0/rVH6vDROAgUw==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-random": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-random/-/d3-random-3.0.1.tgz",
+ "integrity": "sha512-FXMe9GfxTxqd5D6jFsQ+DJ8BJS4E/fT5mqqdjovykEB2oFbTMDVdg1MGFxfQW+FBOGoB++k8swBrgwSHT1cUXQ==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-sankey": {
+ "version": "0.12.3",
+ "resolved": "https://registry.npmjs.org/d3-sankey/-/d3-sankey-0.12.3.tgz",
+ "integrity": "sha512-nQhsBRmM19Ax5xEIPLMY9ZmJ/cDvd1BG3UVvt5h3WRxKg5zGRbvnteTyWAbzeSvlh3tW7ZEmq4VwR5mB3tutmQ==",
+ "dependencies": {
+ "d3-array": "1 - 2",
+ "d3-shape": "^1.2.0"
+ }
+ },
+ "node_modules/d3-sankey/node_modules/d3-array": {
+ "version": "2.12.1",
+ "resolved": "https://registry.npmjs.org/d3-array/-/d3-array-2.12.1.tgz",
+ "integrity": "sha512-B0ErZK/66mHtEsR1TkPEEkwdy+WDesimkM5gpZr5Dsg54BiTA5RXtYW5qTLIAcekaS9xfZrzBLF/OAkB3Qn1YQ==",
+ "dependencies": {
+ "internmap": "^1.0.0"
+ }
+ },
+ "node_modules/d3-sankey/node_modules/d3-path": {
+ "version": "1.0.9",
+ "resolved": "https://registry.npmjs.org/d3-path/-/d3-path-1.0.9.tgz",
+ "integrity": "sha512-VLaYcn81dtHVTjEHd8B+pbe9yHWpXKZUC87PzoFmsFrJqgFwDe/qxfp5MlfsfM1V5E/iVt0MmEbWQ7FVIXh/bg=="
+ },
+ "node_modules/d3-sankey/node_modules/d3-shape": {
+ "version": "1.3.7",
+ "resolved": "https://registry.npmjs.org/d3-shape/-/d3-shape-1.3.7.tgz",
+ "integrity": "sha512-EUkvKjqPFUAZyOlhY5gzCxCeI0Aep04LwIRpsZ/mLFelJiUfnK56jo5JMDSE7yyP2kLSb6LtF+S5chMk7uqPqw==",
+ "dependencies": {
+ "d3-path": "1"
+ }
+ },
+ "node_modules/d3-sankey/node_modules/internmap": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/internmap/-/internmap-1.0.1.tgz",
+ "integrity": "sha512-lDB5YccMydFBtasVtxnZ3MRBHuaoE8GKsppq+EchKL2U4nK/DmEpPHNH8MZe5HkMtpSiTSOZwfN0tzYjO/lJEw=="
+ },
+ "node_modules/d3-scale": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/d3-scale/-/d3-scale-4.0.2.tgz",
+ "integrity": "sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==",
+ "dependencies": {
+ "d3-array": "2.10.0 - 3",
+ "d3-format": "1 - 3",
+ "d3-interpolate": "1.2.0 - 3",
+ "d3-time": "2.1.1 - 3",
+ "d3-time-format": "2 - 4"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-scale-chromatic": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/d3-scale-chromatic/-/d3-scale-chromatic-3.1.0.tgz",
+ "integrity": "sha512-A3s5PWiZ9YCXFye1o246KoscMWqf8BsD9eRiJ3He7C9OBaxKhAd5TFCdEx/7VbKtxxTsu//1mMJFrEt572cEyQ==",
+ "dependencies": {
+ "d3-color": "1 - 3",
+ "d3-interpolate": "1 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-selection": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz",
+ "integrity": "sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-shape": {
+ "version": "3.2.0",
+ "resolved": "https://registry.npmjs.org/d3-shape/-/d3-shape-3.2.0.tgz",
+ "integrity": "sha512-SaLBuwGm3MOViRq2ABk3eLoxwZELpH6zhl3FbAoJ7Vm1gofKx6El1Ib5z23NUEhF9AsGl7y+dzLe5Cw2AArGTA==",
+ "dependencies": {
+ "d3-path": "^3.1.0"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-time": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/d3-time/-/d3-time-3.1.0.tgz",
+ "integrity": "sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==",
+ "dependencies": {
+ "d3-array": "2 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-time-format": {
+ "version": "4.1.0",
+ "resolved": "https://registry.npmjs.org/d3-time-format/-/d3-time-format-4.1.0.tgz",
+ "integrity": "sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==",
+ "dependencies": {
+ "d3-time": "1 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-timer": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-timer/-/d3-timer-3.0.1.tgz",
+ "integrity": "sha512-ndfJ/JxxMd3nw31uyKoY2naivF+r29V+Lc0svZxe1JvvIRmi8hUsrMvdOwgS1o6uBHmiz91geQ0ylPP0aj1VUA==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/d3-transition": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/d3-transition/-/d3-transition-3.0.1.tgz",
+ "integrity": "sha512-ApKvfjsSR6tg06xrL434C0WydLr7JewBB3V+/39RMHsaXTOG0zmt/OAXeng5M5LBm0ojmxJrpomQVZ1aPvBL4w==",
+ "dependencies": {
+ "d3-color": "1 - 3",
+ "d3-dispatch": "1 - 3",
+ "d3-ease": "1 - 3",
+ "d3-interpolate": "1 - 3",
+ "d3-timer": "1 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ },
+ "peerDependencies": {
+ "d3-selection": "2 - 3"
+ }
+ },
+ "node_modules/d3-zoom": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/d3-zoom/-/d3-zoom-3.0.0.tgz",
+ "integrity": "sha512-b8AmV3kfQaqWAuacbPuNbL6vahnOJflOhexLzMMNLga62+/nh0JzvJ0aO/5a5MVgUFGS7Hu1P9P03o3fJkDCyw==",
+ "dependencies": {
+ "d3-dispatch": "1 - 3",
+ "d3-drag": "2 - 3",
+ "d3-interpolate": "1 - 3",
+ "d3-selection": "2 - 3",
+ "d3-transition": "2 - 3"
+ },
+ "engines": {
+ "node": ">=12"
+ }
+ },
+ "node_modules/dagre-d3-es": {
+ "version": "7.0.11",
+ "resolved": "https://registry.npmjs.org/dagre-d3-es/-/dagre-d3-es-7.0.11.tgz",
+ "integrity": "sha512-tvlJLyQf834SylNKax8Wkzco/1ias1OPw8DcUMDE7oUIoSEW25riQVuiu/0OWEFqT0cxHT3Pa9/D82Jr47IONw==",
+ "dependencies": {
+ "d3": "^7.9.0",
+ "lodash-es": "^4.17.21"
+ }
+ },
+ "node_modules/dayjs": {
+ "version": "1.11.13",
+ "resolved": "https://registry.npmjs.org/dayjs/-/dayjs-1.11.13.tgz",
+ "integrity": "sha512-oaMBel6gjolK862uaPQOVTA7q3TZhuSvuMQAAglQDOWYO9A91IrAOUJEyKVlqJlHE0vq5p5UXxzdPfMH/x6xNg=="
+ },
"node_modules/debounce": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/debounce/-/debounce-1.2.1.tgz",
@@ -8976,6 +9847,14 @@
"url": "https://github.com/sponsors/ljharb"
}
},
+ "node_modules/delaunator": {
+ "version": "5.0.1",
+ "resolved": "https://registry.npmjs.org/delaunator/-/delaunator-5.0.1.tgz",
+ "integrity": "sha512-8nvh+XBe96aCESrGOqMp/84b13H9cdKbG5P2ejQCh4d4sK9RL4371qou9drQjMhvnPmhWl5hnmqbEE0fXr9Xnw==",
+ "dependencies": {
+ "robust-predicates": "^3.0.2"
+ }
+ },
"node_modules/delayed-stream": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
@@ -9123,6 +10002,14 @@
"url": "https://github.com/fb55/domhandler?sponsor=1"
}
},
+ "node_modules/dompurify": {
+ "version": "3.2.6",
+ "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.2.6.tgz",
+ "integrity": "sha512-/2GogDQlohXPZe6D6NOgQvXLPSYBqIWMnZ8zzOhn09REE4eyAzb+Hed3jhoM9OkuaJ8P6ZGTTVWQKAi8ieIzfQ==",
+ "optionalDependencies": {
+ "@types/trusted-types": "^2.0.7"
+ }
+ },
"node_modules/domutils": {
"version": "3.2.2",
"resolved": "https://registry.npmjs.org/domutils/-/domutils-3.2.2.tgz",
@@ -9691,6 +10578,11 @@
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
},
+ "node_modules/exsolve": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/exsolve/-/exsolve-1.0.7.tgz",
+ "integrity": "sha512-VO5fQUzZtI6C+vx4w/4BWJpg3s/5l+6pRQEHzFRM8WFi4XffSP1Z+4qi7GbjWbvRQEbdIco5mIMq+zX4rPuLrw=="
+ },
"node_modules/extend": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz",
@@ -10220,6 +11112,11 @@
"js-yaml": "bin/js-yaml.js"
}
},
+ "node_modules/hachure-fill": {
+ "version": "0.5.2",
+ "resolved": "https://registry.npmjs.org/hachure-fill/-/hachure-fill-0.5.2.tgz",
+ "integrity": "sha512-3GKBOn+m2LX9iq+JC1064cSFprJY4jL1jCXTcpnfER5HYE2l/4EfWSGzkPa/ZDBmYI0ZOEj5VHV/eKnPGkHuOg=="
+ },
"node_modules/handle-thing": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/handle-thing/-/handle-thing-2.0.1.tgz",
@@ -10860,6 +11757,14 @@
"resolved": "https://registry.npmjs.org/inline-style-parser/-/inline-style-parser-0.2.4.tgz",
"integrity": "sha512-0aO8FkhNZlj/ZIbNi7Lxxr12obT7cL1moPfE4tg1LkX7LlLfC6DeX4l2ZEud1ukP9jNQyNnfzQVqwbwmAATY4Q=="
},
+ "node_modules/internmap": {
+ "version": "2.0.3",
+ "resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz",
+ "integrity": "sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==",
+ "engines": {
+ "node": ">=12"
+ }
+ },
"node_modules/invariant": {
"version": "2.2.4",
"resolved": "https://registry.npmjs.org/invariant/-/invariant-2.2.4.tgz",
@@ -11296,6 +12201,34 @@
"graceful-fs": "^4.1.6"
}
},
+ "node_modules/katex": {
+ "version": "0.16.22",
+ "resolved": "https://registry.npmjs.org/katex/-/katex-0.16.22.tgz",
+ "integrity": "sha512-XCHRdUw4lf3SKBaJe4EvgqIuWwkPSo9XoeO8GjQW94Bp7TWv9hNhzZjZ+OH9yf1UmLygb7DIT5GSFQiyt16zYg==",
+ "funding": [
+ "https://opencollective.com/katex",
+ "https://github.com/sponsors/katex"
+ ],
+ "dependencies": {
+ "commander": "^8.3.0"
+ },
+ "bin": {
+ "katex": "cli.js"
+ }
+ },
+ "node_modules/katex/node_modules/commander": {
+ "version": "8.3.0",
+ "resolved": "https://registry.npmjs.org/commander/-/commander-8.3.0.tgz",
+ "integrity": "sha512-OkTL9umf+He2DZkUq8f8J9of7yL6RJKI24dVITBmNfZBmri9zYZQrKkuXiKhyfPSu8tUhnVBB1iKXevvnlR4Ww==",
+ "engines": {
+ "node": ">= 12"
+ }
+ },
+ "node_modules/khroma": {
+ "version": "2.1.0",
+ "resolved": "https://registry.npmjs.org/khroma/-/khroma-2.1.0.tgz",
+ "integrity": "sha512-Ls993zuzfayK269Svk9hzpeGUKob/sIgZzyHYdjQoAdQetRKpOLj+k/QQQ/6Qi0Yz65mlROrfd+Ev+1+7dz9Kw=="
+ },
"node_modules/kind-of": {
"version": "6.0.3",
"resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz",
@@ -11312,6 +12245,26 @@
"node": ">=6"
}
},
+ "node_modules/kolorist": {
+ "version": "1.8.0",
+ "resolved": "https://registry.npmjs.org/kolorist/-/kolorist-1.8.0.tgz",
+ "integrity": "sha512-Y+60/zizpJ3HRH8DCss+q95yr6145JXZo46OTpFvDZWLfRCE4qChOyk1b26nMaNpfHHgxagk9dXT5OP0Tfe+dQ=="
+ },
+ "node_modules/langium": {
+ "version": "3.3.1",
+ "resolved": "https://registry.npmjs.org/langium/-/langium-3.3.1.tgz",
+ "integrity": "sha512-QJv/h939gDpvT+9SiLVlY7tZC3xB2qK57v0J04Sh9wpMb6MP1q8gB21L3WIo8T5P1MSMg3Ep14L7KkDCFG3y4w==",
+ "dependencies": {
+ "chevrotain": "~11.0.3",
+ "chevrotain-allstar": "~0.3.0",
+ "vscode-languageserver": "~9.0.1",
+ "vscode-languageserver-textdocument": "~1.0.11",
+ "vscode-uri": "~3.0.8"
+ },
+ "engines": {
+ "node": ">=16.0.0"
+ }
+ },
"node_modules/latest-version": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/latest-version/-/latest-version-7.0.0.tgz",
@@ -11346,6 +12299,11 @@
"url": "https://github.com/sponsors/ljharb"
}
},
+ "node_modules/layout-base": {
+ "version": "1.0.2",
+ "resolved": "https://registry.npmjs.org/layout-base/-/layout-base-1.0.2.tgz",
+ "integrity": "sha512-8h2oVEZNktL4BH2JCOI90iD1yXwL6iNW7KcCKT2QZgQJR2vbqDsldCTPRU9NifTCqHZci57XvQQ15YTu+sTYPg=="
+ },
"node_modules/leven": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/leven/-/leven-3.1.0.tgz",
@@ -11391,6 +12349,22 @@
"node": ">=8.9.0"
}
},
+ "node_modules/local-pkg": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/local-pkg/-/local-pkg-1.1.1.tgz",
+ "integrity": "sha512-WunYko2W1NcdfAFpuLUoucsgULmgDBRkdxHxWQ7mK0cQqwPiy8E1enjuRBrhLtZkB5iScJ1XIPdhVEFK8aOLSg==",
+ "dependencies": {
+ "mlly": "^1.7.4",
+ "pkg-types": "^2.0.1",
+ "quansync": "^0.2.8"
+ },
+ "engines": {
+ "node": ">=14"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/antfu"
+ }
+ },
"node_modules/locate-path": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/locate-path/-/locate-path-7.2.0.tgz",
@@ -11410,6 +12384,11 @@
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz",
"integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg=="
},
+ "node_modules/lodash-es": {
+ "version": "4.17.21",
+ "resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.17.21.tgz",
+ "integrity": "sha512-mKnC+QJ9pWVzv+C4/U3rRsHapFfHvQFoFB92e52xeyGMcX6/OlIl78je1u8vePzYZSkkogMPJ2yjxxsb89cxyw=="
+ },
"node_modules/lodash.debounce": {
"version": "4.0.8",
"resolved": "https://registry.npmjs.org/lodash.debounce/-/lodash.debounce-4.0.8.tgz",
@@ -11953,6 +12932,56 @@
"node": ">= 8"
}
},
+ "node_modules/mermaid": {
+ "version": "11.10.0",
+ "resolved": "https://registry.npmjs.org/mermaid/-/mermaid-11.10.0.tgz",
+ "integrity": "sha512-oQsFzPBy9xlpnGxUqLbVY8pvknLlsNIJ0NWwi8SUJjhbP1IT0E0o1lfhU4iYV3ubpy+xkzkaOyDUQMn06vQElQ==",
+ "dependencies": {
+ "@braintree/sanitize-url": "^7.0.4",
+ "@iconify/utils": "^2.1.33",
+ "@mermaid-js/parser": "^0.6.2",
+ "@types/d3": "^7.4.3",
+ "cytoscape": "^3.29.3",
+ "cytoscape-cose-bilkent": "^4.1.0",
+ "cytoscape-fcose": "^2.2.0",
+ "d3": "^7.9.0",
+ "d3-sankey": "^0.12.3",
+ "dagre-d3-es": "7.0.11",
+ "dayjs": "^1.11.13",
+ "dompurify": "^3.2.5",
+ "katex": "^0.16.22",
+ "khroma": "^2.1.0",
+ "lodash-es": "^4.17.21",
+ "marked": "^16.0.0",
+ "roughjs": "^4.6.6",
+ "stylis": "^4.3.6",
+ "ts-dedent": "^2.2.0",
+ "uuid": "^11.1.0"
+ }
+ },
+ "node_modules/mermaid/node_modules/marked": {
+ "version": "16.1.2",
+ "resolved": "https://registry.npmjs.org/marked/-/marked-16.1.2.tgz",
+ "integrity": "sha512-rNQt5EvRinalby7zJZu/mB+BvaAY2oz3wCuCjt1RDrWNpS1Pdf9xqMOeC9Hm5adBdcV/3XZPJpG58eT+WBc0XQ==",
+ "bin": {
+ "marked": "bin/marked.js"
+ },
+ "engines": {
+ "node": ">= 20"
+ }
+ },
+ "node_modules/mermaid/node_modules/uuid": {
+ "version": "11.1.0",
+ "resolved": "https://registry.npmjs.org/uuid/-/uuid-11.1.0.tgz",
+ "integrity": "sha512-0/A9rDy9P7cJ+8w1c9WD9V//9Wj15Ce2MPz8Ri6032usz+NfePxx5AcN3bN+r6ZL6jEo066/yNYB3tn4pQEx+A==",
+ "funding": [
+ "https://github.com/sponsors/broofa",
+ "https://github.com/sponsors/ctavan"
+ ],
+ "bin": {
+ "uuid": "dist/esm/bin/uuid"
+ }
+ },
"node_modules/methods": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz",
@@ -13757,6 +14786,32 @@
"resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
"integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A=="
},
+ "node_modules/mlly": {
+ "version": "1.7.4",
+ "resolved": "https://registry.npmjs.org/mlly/-/mlly-1.7.4.tgz",
+ "integrity": "sha512-qmdSIPC4bDJXgZTCR7XosJiNKySV7O215tsPtDN9iEO/7q/76b/ijtgRu/+epFXSJhijtTCCGp3DWS549P3xKw==",
+ "dependencies": {
+ "acorn": "^8.14.0",
+ "pathe": "^2.0.1",
+ "pkg-types": "^1.3.0",
+ "ufo": "^1.5.4"
+ }
+ },
+ "node_modules/mlly/node_modules/confbox": {
+ "version": "0.1.8",
+ "resolved": "https://registry.npmjs.org/confbox/-/confbox-0.1.8.tgz",
+ "integrity": "sha512-RMtmw0iFkeR4YV+fUOSucriAQNb9g8zFR52MWCtl+cCZOFRNL6zeB395vPzFhEjjn4fMxXudmELnl/KF/WrK6w=="
+ },
+ "node_modules/mlly/node_modules/pkg-types": {
+ "version": "1.3.1",
+ "resolved": "https://registry.npmjs.org/pkg-types/-/pkg-types-1.3.1.tgz",
+ "integrity": "sha512-/Jm5M4RvtBFVkKWRu2BLUTNP8/M2a+UwuAX+ae4770q1qVGtfjG+WTCupoZixokjmHiry8uI+dlY8KXYV5HVVQ==",
+ "dependencies": {
+ "confbox": "^0.1.8",
+ "mlly": "^1.7.4",
+ "pathe": "^2.0.1"
+ }
+ },
"node_modules/mrmime": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/mrmime/-/mrmime-2.0.1.tgz",
@@ -14405,6 +15460,11 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
+ "node_modules/package-manager-detector": {
+ "version": "1.3.0",
+ "resolved": "https://registry.npmjs.org/package-manager-detector/-/package-manager-detector-1.3.0.tgz",
+ "integrity": "sha512-ZsEbbZORsyHuO00lY1kV3/t72yp6Ysay6Pd17ZAlNGuGwmWDLCJxFpRs0IzfXfj1o4icJOkUEioexFHzyPurSQ=="
+ },
"node_modules/param-case": {
"version": "3.0.4",
"resolved": "https://registry.npmjs.org/param-case/-/param-case-3.0.4.tgz",
@@ -14530,6 +15590,11 @@
"util": "^0.10.3"
}
},
+ "node_modules/path-data-parser": {
+ "version": "0.1.0",
+ "resolved": "https://registry.npmjs.org/path-data-parser/-/path-data-parser-0.1.0.tgz",
+ "integrity": "sha512-NOnmBpt5Y2RWbuv0LMzsayp3lVylAHLPUTut412ZA3l+C4uw4ZVkQbjShYCQ8TCpUMdPapr4YjUqLYD6v68j+w=="
+ },
"node_modules/path-exists": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/path-exists/-/path-exists-5.0.0.tgz",
@@ -14569,6 +15634,11 @@
"node": ">=8"
}
},
+ "node_modules/pathe": {
+ "version": "2.0.3",
+ "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz",
+ "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="
+ },
"node_modules/picocolors": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -14599,6 +15669,30 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
+ "node_modules/pkg-types": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/pkg-types/-/pkg-types-2.2.0.tgz",
+ "integrity": "sha512-2SM/GZGAEkPp3KWORxQZns4M+WSeXbC2HEvmOIJe3Cmiv6ieAJvdVhDldtHqM5J1Y7MrR1XhkBT/rMlhh9FdqQ==",
+ "dependencies": {
+ "confbox": "^0.2.2",
+ "exsolve": "^1.0.7",
+ "pathe": "^2.0.3"
+ }
+ },
+ "node_modules/points-on-curve": {
+ "version": "0.2.0",
+ "resolved": "https://registry.npmjs.org/points-on-curve/-/points-on-curve-0.2.0.tgz",
+ "integrity": "sha512-0mYKnYYe9ZcqMCWhUjItv/oHjvgEsfKvnUTg8sAtnHr3GVy7rGkXCb6d5cSyqrWqL4k81b9CPg3urd+T7aop3A=="
+ },
+ "node_modules/points-on-path": {
+ "version": "0.2.1",
+ "resolved": "https://registry.npmjs.org/points-on-path/-/points-on-path-0.2.1.tgz",
+ "integrity": "sha512-25ClnWWuw7JbWZcgqY/gJ4FQWadKxGWk+3kR/7kD0tCaDtPPMj7oHu2ToLaVhfpnHrZzYby2w6tUA0eOIuUg8g==",
+ "dependencies": {
+ "path-data-parser": "0.1.0",
+ "points-on-curve": "0.2.0"
+ }
+ },
"node_modules/postcss": {
"version": "8.5.6",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.6.tgz",
@@ -16026,9 +17120,10 @@
}
},
"node_modules/prebuild-install/node_modules/tar-fs": {
- "version": "2.1.3",
- "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.3.tgz",
- "integrity": "sha512-090nwYJDmlhwFwEW3QQl+vaNnxsO2yVsd45eTKRBzSzu+hlb1w2K9inVq5b0ngXuLVqQ4ApvsUHHnu/zQNkWAg==",
+ "version": "2.1.4",
+ "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.4.tgz",
+ "integrity": "sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==",
+ "license": "MIT",
"dependencies": {
"chownr": "^1.1.1",
"mkdirp-classic": "^0.5.2",
@@ -16195,6 +17290,21 @@
"url": "https://github.com/sponsors/ljharb"
}
},
+ "node_modules/quansync": {
+ "version": "0.2.10",
+ "resolved": "https://registry.npmjs.org/quansync/-/quansync-0.2.10.tgz",
+ "integrity": "sha512-t41VRkMYbkHyCYmOvx/6URnN80H7k4X0lLdBMGsz+maAwrJQYB1djpV6vHrQIBE0WBSGqhtEHrK9U3DWWH8v7A==",
+ "funding": [
+ {
+ "type": "individual",
+ "url": "https://github.com/sponsors/antfu"
+ },
+ {
+ "type": "individual",
+ "url": "https://github.com/sponsors/sxzz"
+ }
+ ]
+ },
"node_modules/queue-microtask": {
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz",
@@ -17087,6 +18197,22 @@
"node": ">=0.10.0"
}
},
+ "node_modules/robust-predicates": {
+ "version": "3.0.2",
+ "resolved": "https://registry.npmjs.org/robust-predicates/-/robust-predicates-3.0.2.tgz",
+ "integrity": "sha512-IXgzBWvWQwE6PrDI05OvmXUIruQTcoMDzRsOd5CDvHCVLcLHMTSYvOK5Cm46kWqlV3yAbuSpBZdJ5oP5OUoStg=="
+ },
+ "node_modules/roughjs": {
+ "version": "4.6.6",
+ "resolved": "https://registry.npmjs.org/roughjs/-/roughjs-4.6.6.tgz",
+ "integrity": "sha512-ZUz/69+SYpFN/g/lUlo2FXcIjRkSu3nDarreVdGGndHEBJ6cXPdKguS8JGxwj5HA5xIbVKSmLgr5b3AWxtRfvQ==",
+ "dependencies": {
+ "hachure-fill": "^0.5.2",
+ "path-data-parser": "^0.1.0",
+ "points-on-curve": "^0.2.0",
+ "points-on-path": "^0.2.1"
+ }
+ },
"node_modules/rtlcss": {
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/rtlcss/-/rtlcss-4.3.0.tgz",
@@ -17137,6 +18263,11 @@
"queue-microtask": "^1.2.2"
}
},
+ "node_modules/rw": {
+ "version": "1.3.3",
+ "resolved": "https://registry.npmjs.org/rw/-/rw-1.3.3.tgz",
+ "integrity": "sha512-PdhdWy89SiZogBLaw42zdeqtRJ//zFd2PgQavcICDUgJT5oW10QCRKbJ6bg4r0/UY2M6BWd5tkxuGFRvCkgfHQ=="
+ },
"node_modules/safe-buffer": {
"version": "5.2.1",
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
@@ -18083,6 +19214,11 @@
"postcss": "^8.4.31"
}
},
+ "node_modules/stylis": {
+ "version": "4.3.6",
+ "resolved": "https://registry.npmjs.org/stylis/-/stylis-4.3.6.tgz",
+ "integrity": "sha512-yQ3rwFWRfwNUY7H5vpU0wfdkNSnvnJinhF9830Swlaxl03zsOjCfmX0ugac+3LtK0lYSgwL/KXc8oYL3mG4YFQ=="
+ },
"node_modules/supports-color": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz",
@@ -18160,9 +19296,10 @@
}
},
"node_modules/tar-fs": {
- "version": "3.0.10",
- "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.0.10.tgz",
- "integrity": "sha512-C1SwlQGNLe/jPNqapK8epDsXME7CAJR5RL3GcE6KWx1d9OUByzoHVcbu1VPI8tevg9H8Alae0AApHHFGzrD5zA==",
+ "version": "3.1.1",
+ "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.1.1.tgz",
+ "integrity": "sha512-LZA0oaPOc2fVo82Txf3gw+AkEd38szODlptMYejQUhndHMLQ9M059uXR+AfS7DNo0NpINvSqDsvyaCrBVkptWg==",
+ "license": "MIT",
"dependencies": {
"pump": "^3.0.0",
"tar-stream": "^3.1.5"
@@ -18298,6 +19435,11 @@
"resolved": "https://registry.npmjs.org/tiny-warning/-/tiny-warning-1.0.3.tgz",
"integrity": "sha512-lBN9zLN/oAf68o3zNXYrdCt1kP8WsiGW8Oo2ka41b2IM5JL/S1CTyX1rW0mb/zSuJun0ZUrDxx4sqvYS2FWzPA=="
},
+ "node_modules/tinyexec": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.0.1.tgz",
+ "integrity": "sha512-5uC6DDlmeqiOwCPmK9jMSdOuZTh8bU39Ys6yidB+UTt5hfZUPGAypSgFRiEp+jbi9qH40BLDvy85jIU88wKSqw=="
+ },
"node_modules/tinypool": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz",
@@ -18379,6 +19521,14 @@
"url": "https://github.com/sponsors/wooorm"
}
},
+ "node_modules/ts-dedent": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/ts-dedent/-/ts-dedent-2.2.0.tgz",
+ "integrity": "sha512-q5W7tVM71e2xjHZTlgfTDoPF/SmqKG5hddq9SzR49CH2hayqRKJtQ4mtRlSxKaJlR/+9rEM+mnBHf7I2/BQcpQ==",
+ "engines": {
+ "node": ">=6.10"
+ }
+ },
"node_modules/tslib": {
"version": "2.8.1",
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
@@ -18426,6 +19576,11 @@
"is-typedarray": "^1.0.0"
}
},
+ "node_modules/ufo": {
+ "version": "1.6.1",
+ "resolved": "https://registry.npmjs.org/ufo/-/ufo-1.6.1.tgz",
+ "integrity": "sha512-9a4/uxlTWJ4+a5i0ooc1rU7C7YOw3wT+UGqdeNNHWnOF9qcMBgLRS+4IYUqbczewFx4mLEig6gawh7X6mFlEkA=="
+ },
"node_modules/undici-types": {
"version": "7.8.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.8.0.tgz",
@@ -18961,6 +20116,49 @@
"url": "https://opencollective.com/unified"
}
},
+ "node_modules/vscode-jsonrpc": {
+ "version": "8.2.0",
+ "resolved": "https://registry.npmjs.org/vscode-jsonrpc/-/vscode-jsonrpc-8.2.0.tgz",
+ "integrity": "sha512-C+r0eKJUIfiDIfwJhria30+TYWPtuHJXHtI7J0YlOmKAo7ogxP20T0zxB7HZQIFhIyvoBPwWskjxrvAtfjyZfA==",
+ "engines": {
+ "node": ">=14.0.0"
+ }
+ },
+ "node_modules/vscode-languageserver": {
+ "version": "9.0.1",
+ "resolved": "https://registry.npmjs.org/vscode-languageserver/-/vscode-languageserver-9.0.1.tgz",
+ "integrity": "sha512-woByF3PDpkHFUreUa7Hos7+pUWdeWMXRd26+ZX2A8cFx6v/JPTtd4/uN0/jB6XQHYaOlHbio03NTHCqrgG5n7g==",
+ "dependencies": {
+ "vscode-languageserver-protocol": "3.17.5"
+ },
+ "bin": {
+ "installServerIntoExtension": "bin/installServerIntoExtension"
+ }
+ },
+ "node_modules/vscode-languageserver-protocol": {
+ "version": "3.17.5",
+ "resolved": "https://registry.npmjs.org/vscode-languageserver-protocol/-/vscode-languageserver-protocol-3.17.5.tgz",
+ "integrity": "sha512-mb1bvRJN8SVznADSGWM9u/b07H7Ecg0I3OgXDuLdn307rl/J3A9YD6/eYOssqhecL27hK1IPZAsaqh00i/Jljg==",
+ "dependencies": {
+ "vscode-jsonrpc": "8.2.0",
+ "vscode-languageserver-types": "3.17.5"
+ }
+ },
+ "node_modules/vscode-languageserver-textdocument": {
+ "version": "1.0.12",
+ "resolved": "https://registry.npmjs.org/vscode-languageserver-textdocument/-/vscode-languageserver-textdocument-1.0.12.tgz",
+ "integrity": "sha512-cxWNPesCnQCcMPeenjKKsOCKQZ/L6Tv19DTRIGuLWe32lyzWhihGVJ/rcckZXJxfdKCFvRLS3fpBIsV/ZGX4zA=="
+ },
+ "node_modules/vscode-languageserver-types": {
+ "version": "3.17.5",
+ "resolved": "https://registry.npmjs.org/vscode-languageserver-types/-/vscode-languageserver-types-3.17.5.tgz",
+ "integrity": "sha512-Ld1VelNuX9pdF39h2Hgaeb5hEZM2Z3jUrrMgWQAu82jMtZp7p3vJT3BzToKtZI7NgQssZje5o0zryOrhQvzQAg=="
+ },
+ "node_modules/vscode-uri": {
+ "version": "3.0.8",
+ "resolved": "https://registry.npmjs.org/vscode-uri/-/vscode-uri-3.0.8.tgz",
+ "integrity": "sha512-AyFQ0EVmsOZOlAnxoFOGOq1SQDWAB7C6aqMGS23svWAllfOaxbuFvcT8D1i8z3Gyn8fraVeZNNmN6e9bxxXkKw=="
+ },
"node_modules/watchpack": {
"version": "2.4.4",
"resolved": "https://registry.npmjs.org/watchpack/-/watchpack-2.4.4.tgz",
diff --git a/docs/my-website/package.json b/docs/my-website/package.json
index 24d212ea2c6..955e63c2d84 100644
--- a/docs/my-website/package.json
+++ b/docs/my-website/package.json
@@ -18,6 +18,7 @@
"@docusaurus/plugin-google-gtag": "3.8.1",
"@docusaurus/plugin-ideal-image": "3.8.1",
"@docusaurus/preset-classic": "3.8.1",
+ "@docusaurus/theme-mermaid": "^3.8.1",
"@inkeep/cxkit-docusaurus": "^0.5.89",
"@mdx-js/react": "^3.0.0",
"clsx": "^1.2.1",
@@ -48,6 +49,7 @@
},
"overrides": {
"webpack-dev-server": ">=5.2.1",
- "form-data": ">=4.0.4"
+ "form-data": ">=4.0.4",
+ "mermaid": ">=11.10.0"
}
}
diff --git a/docs/my-website/release_notes/v1.67.4-stable/index.md b/docs/my-website/release_notes/v1.67.4-stable/index.md
index 6750ced47c7..93a27155d2b 100644
--- a/docs/my-website/release_notes/v1.67.4-stable/index.md
+++ b/docs/my-website/release_notes/v1.67.4-stable/index.md
@@ -106,7 +106,7 @@ This release allow you to group requests to LiteLLM proxy into a session. If you
1. Added support for max_completion_tokens parameter [Get Started](https://docs.litellm.ai/docs/providers/sagemaker), [PR](https://github.com/BerriAI/litellm/pull/10300)
- **Responses API**
1. Added support for GET and DELETE operations - `/v1/responses/{response_id}` [Get Started](../../docs/response_api)
- 2. Added session management support for non-OpenAI models [PR](https://github.com/BerriAI/litellm/pull/10321)
+ 2. Added session management support for all supported models [PR](https://github.com/BerriAI/litellm/pull/10321)
3. Added routing affinity to maintain model consistency within sessions [Get Started](https://docs.litellm.ai/docs/response_api#load-balancing-with-routing-affinity), [PR](https://github.com/BerriAI/litellm/pull/10193)
diff --git a/docs/my-website/release_notes/v1.74.15-stable/index.md b/docs/my-website/release_notes/v1.74.15-stable/index.md
new file mode 100644
index 00000000000..9807a00b7e7
--- /dev/null
+++ b/docs/my-website/release_notes/v1.74.15-stable/index.md
@@ -0,0 +1,291 @@
+---
+title: "v1.74.15-stable"
+slug: "v1-74-15"
+date: 2025-08-02T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.74.15-stable
+```
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.74.15.post2
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **User Agent Activity Tracking** - Track how much usage each coding tool gets.
+- **Prompt Management** - Use Git-Ops style prompt management with prompt templates.
+- **MCP Gateway: Guardrails** - Support for using Guardrails with MCP servers.
+- **Google AI Studio Imagen4** - Support for using Imagen4 models on Google AI Studio.
+
+---
+
+## User Agent Activity Tracking
+
+
+
+
+
+This release brings support for tracking usage and costs for AI-powered coding tools like Claude Code, Roo Code, Gemini CLI through LiteLLM. You can now track LLM cost, total tokens used, and DAU/WAU/MAU for each coding tool.
+
+This is great to central AI Platform teams looking to track how they are helping developer productivity.
+
+[Read More](https://docs.litellm.ai/docs/tutorials/cost_tracking_coding)
+
+---
+
+## Prompt Management
+
+
+
+
+
+[Read More](../../docs/proxy/prompt_management)
+
+---
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Cost per Image |
+| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- | -------------- |
+| OpenRouter | `openrouter/x-ai/grok-4` | 256k | $3 | $15 | N/A |
+| Google AI Studio | `gemini/imagen-4.0-generate-001` | N/A | N/A | N/A | $0.04 |
+| Google AI Studio | `gemini/imagen-4.0-ultra-generate-001` | N/A | N/A | N/A | $0.06 |
+| Google AI Studio | `gemini/imagen-4.0-fast-generate-001` | N/A | N/A | N/A | $0.02 |
+| Google AI Studio | `gemini/imagen-3.0-generate-002` | N/A | N/A | N/A | $0.04 |
+| Google AI Studio | `gemini/imagen-3.0-generate-001` | N/A | N/A | N/A | $0.04 |
+| Google AI Studio | `gemini/imagen-3.0-fast-generate-001` | N/A | N/A | N/A | $0.02 |
+
+#### Features
+
+- **[Google AI Studio](../../docs/providers/gemini)**
+ - Added Google AI Studio Imagen4 model family support - [PR #13065](https://github.com/BerriAI/litellm/pull/13065), [Get Started](../../docs/providers/google_ai_studio/image_gen)
+- **[Azure OpenAI](../../docs/providers/azure/azure)**
+ - Azure `api_version="preview"` support - [PR #13072](https://github.com/BerriAI/litellm/pull/13072), [Get Started](../../docs/providers/azure/azure#setting-api-version)
+ - Password protected certificate files support - [PR #12995](https://github.com/BerriAI/litellm/pull/12995), [Get Started](../../docs/providers/azure/azure#authentication)
+- **[AWS Bedrock](../../docs/providers/bedrock)**
+ - Cost tracking via Anthropic `/v1/messages` - [PR #13072](https://github.com/BerriAI/litellm/pull/13072)
+ - Computer use support - [PR #13150](https://github.com/BerriAI/litellm/pull/13150)
+- **[OpenRouter](../../docs/providers/openrouter)**
+ - Added Grok4 model support - [PR #13018](https://github.com/BerriAI/litellm/pull/13018)
+- **[Anthropic](../../docs/providers/anthropic)**
+ - Auto Cache Control Injection - Improved cache_control_injection_points with negative index support - [PR #13187](https://github.com/BerriAI/litellm/pull/13187), [Get Started](../../docs/tutorials/prompt_caching)
+ - Working mid-stream fallbacks with token usage tracking - [PR #13149](https://github.com/BerriAI/litellm/pull/13149), [PR #13170](https://github.com/BerriAI/litellm/pull/13170)
+- **[Perplexity](../../docs/providers/perplexity)**
+ - Citation annotations support - [PR #13225](https://github.com/BerriAI/litellm/pull/13225)
+
+#### Bugs
+
+- **[Gemini](../../docs/providers/gemini)**
+ - Fix merge_reasoning_content_in_choices parameter issue - [PR #13066](https://github.com/BerriAI/litellm/pull/13066), [Get Started](../../docs/tutorials/openweb_ui#render-thinking-content-on-open-webui)
+ - Added support for using `GOOGLE_API_KEY` environment variable for Google AI Studio - [PR #12507](https://github.com/BerriAI/litellm/pull/12507)
+- **[vLLM/OpenAI-like](../../docs/providers/vllm)**
+ - Fix missing extra_headers support for embeddings - [PR #13198](https://github.com/BerriAI/litellm/pull/13198)
+
+---
+
+## LLM API Endpoints
+
+#### Bugs
+
+- **[/generateContent](../../docs/generateContent)**
+ - Support for query_params in generateContent routes for API Key setting - [PR #13100](https://github.com/BerriAI/litellm/pull/13100)
+ - Ensure "x-goog-api-key" is used for auth to google ai studio when using /generateContent on LiteLLM - [PR #13098](https://github.com/BerriAI/litellm/pull/13098)
+ - Ensure tool calling works as expected on generateContent - [PR #13189](https://github.com/BerriAI/litellm/pull/13189)
+- **[/vertex_ai (Passthrough)](../../docs/pass_through/vertex_ai)**
+ - Ensure multimodal embedding responses are logged properly - [PR #13050](https://github.com/BerriAI/litellm/pull/13050)
+
+---
+
+## [MCP Gateway](../../docs/mcp)
+
+#### Features
+
+- **Health Check Improvements**
+ - Add health check endpoints for MCP servers - [PR #13106](https://github.com/BerriAI/litellm/pull/13106)
+- **Guardrails Integration**
+ - Add pre and during call hooks initialization - [PR #13067](https://github.com/BerriAI/litellm/pull/13067)
+ - Move pre and during hooks to ProxyLogging - [PR #13109](https://github.com/BerriAI/litellm/pull/13109)
+ - MCP pre and during guardrails implementation - [PR #13188](https://github.com/BerriAI/litellm/pull/13188)
+- **Protocol & Header Support**
+ - Add protocol headers support - [PR #13062](https://github.com/BerriAI/litellm/pull/13062)
+- **URL & Namespacing**
+ - Improve MCP server URL validation for internal/Kubernetes URLs - [PR #13099](https://github.com/BerriAI/litellm/pull/13099)
+
+
+#### Bugs
+
+- **UI**
+ - Fix scrolling issue with MCP tools - [PR #13015](https://github.com/BerriAI/litellm/pull/13015)
+ - Fix MCP client list failure - [PR #13114](https://github.com/BerriAI/litellm/pull/13114)
+
+
+[Read More](../../docs/mcp)
+
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **Usage Analytics**
+ - New tab for user agent activity tracking - [PR #13146](https://github.com/BerriAI/litellm/pull/13146)
+ - Daily usage per user analytics - [PR #13147](https://github.com/BerriAI/litellm/pull/13147)
+ - Default usage chart date range set to last 7 days - [PR #12917](https://github.com/BerriAI/litellm/pull/12917)
+ - New advanced date range picker component - [PR #13141](https://github.com/BerriAI/litellm/pull/13141), [PR #13221](https://github.com/BerriAI/litellm/pull/13221)
+ - Show loader on usage cost charts after date selection - [PR #13113](https://github.com/BerriAI/litellm/pull/13113)
+- **Models**
+ - Added Voyage, Jinai, Deepinfra and VolcEngine providers on UI - [PR #13131](https://github.com/BerriAI/litellm/pull/13131)
+ - Added Sagemaker on UI - [PR #13117](https://github.com/BerriAI/litellm/pull/13117)
+ - Preserve model order in `/v1/models` and `/model_group/info` endpoints - [PR #13178](https://github.com/BerriAI/litellm/pull/13178)
+
+- **Key Management**
+ - Properly parse JSON options for key generation in UI - [PR #12989](https://github.com/BerriAI/litellm/pull/12989)
+- **Authentication**
+ - **JWT Fields**
+ - Add dot notation support for all JWT fields - [PR #13013](https://github.com/BerriAI/litellm/pull/13013)
+
+#### Bugs
+
+- **Permissions**
+ - Fix object permission for organizations - [PR #13142](https://github.com/BerriAI/litellm/pull/13142)
+ - Fix list team v2 security check - [PR #13094](https://github.com/BerriAI/litellm/pull/13094)
+- **Models**
+ - Fix model reload on model update - [PR #13216](https://github.com/BerriAI/litellm/pull/13216)
+- **Router Settings**
+ - Fix displaying models for fallbacks in UI - [PR #13191](https://github.com/BerriAI/litellm/pull/13191)
+ - Fix wildcard model name handling with custom values - [PR #13116](https://github.com/BerriAI/litellm/pull/13116)
+ - Fix fallback delete functionality - [PR #12606](https://github.com/BerriAI/litellm/pull/12606)
+
+---
+
+## Logging / Guardrail Integrations
+
+#### Features
+
+- **[MLFlow](../../docs/proxy/logging#mlflow)**
+ - Allow adding tags for MLFlow logging requests - [PR #13108](https://github.com/BerriAI/litellm/pull/13108)
+- **[Langfuse OTEL](../../docs/proxy/logging#langfuse)**
+ - Add comprehensive metadata support to Langfuse OpenTelemetry integration - [PR #12956](https://github.com/BerriAI/litellm/pull/12956)
+- **[Datadog LLM Observability](../../docs/proxy/logging#datadog)**
+ - Allow redacting message/response content for specific logging integrations - [PR #13158](https://github.com/BerriAI/litellm/pull/13158)
+
+#### Bugs
+
+- **API Key Logging**
+ - Fix API Key being logged inappropriately - [PR #12978](https://github.com/BerriAI/litellm/pull/12978)
+- **MCP Spend Tracking**
+ - Set default value for MCP namespace tool name in spend table - [PR #12894](https://github.com/BerriAI/litellm/pull/12894)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+#### Features
+
+- **Background Health Checks**
+ - Allow disabling background health checks for specific deployments - [PR #13186](https://github.com/BerriAI/litellm/pull/13186)
+- **Database Connection Management**
+ - Ensure stale Prisma clients disconnect DB connections properly - [PR #13140](https://github.com/BerriAI/litellm/pull/13140)
+- **Jitter Improvements**
+ - Fix jitter calculation (should be added not multiplied) - [PR #12901](https://github.com/BerriAI/litellm/pull/12901)
+
+#### Bugs
+
+- **Anthropic Streaming**
+ - Always use choice index=0 for Anthropic streaming responses - [PR #12666](https://github.com/BerriAI/litellm/pull/12666)
+- **Custom Auth**
+ - Bubble up custom exceptions properly - [PR #13093](https://github.com/BerriAI/litellm/pull/13093)
+- **OTEL with Managed Files**
+ - Fix using managed files with OTEL integration - [PR #13171](https://github.com/BerriAI/litellm/pull/13171)
+
+---
+
+## General Proxy Improvements
+
+#### Features
+
+- **Database Migration**
+ - Move to use_prisma_migrate by default - [PR #13117](https://github.com/BerriAI/litellm/pull/13117)
+ - Resolve team-only models on auth checks - [PR #13117](https://github.com/BerriAI/litellm/pull/13117)
+- **Infrastructure**
+ - Loosened MCP Python version restrictions - [PR #13102](https://github.com/BerriAI/litellm/pull/13102)
+ - Migrate build_and_test to CI/CD Postgres DB - [PR #13166](https://github.com/BerriAI/litellm/pull/13166)
+- **Helm Charts**
+ - Allow Helm hooks for migration jobs - [PR #13174](https://github.com/BerriAI/litellm/pull/13174)
+ - Fix Helm migration job schema updates - [PR #12809](https://github.com/BerriAI/litellm/pull/12809)
+
+#### Bugs
+
+- **Docker**
+ - Remove obsolete `version` attribute in docker-compose - [PR #13172](https://github.com/BerriAI/litellm/pull/13172)
+ - Add openssl in runtime stage for non-root Dockerfile - [PR #13168](https://github.com/BerriAI/litellm/pull/13168)
+- **Database Configuration**
+ - Fix DB config through environment variables - [PR #13111](https://github.com/BerriAI/litellm/pull/13111)
+- **Logging**
+ - Suppress httpx logging - [PR #13217](https://github.com/BerriAI/litellm/pull/13217)
+- **Token Counting**
+ - Ignore unsupported keys like prefix in token counter - [PR #11954](https://github.com/BerriAI/litellm/pull/11954)
+---
+
+## New Contributors
+* @5731la made their first contribution in https://github.com/BerriAI/litellm/pull/12989
+* @restato made their first contribution in https://github.com/BerriAI/litellm/pull/12980
+* @strickvl made their first contribution in https://github.com/BerriAI/litellm/pull/12956
+* @Ne0-1 made their first contribution in https://github.com/BerriAI/litellm/pull/12995
+* @maxrabin made their first contribution in https://github.com/BerriAI/litellm/pull/13079
+* @lvuna made their first contribution in https://github.com/BerriAI/litellm/pull/12894
+* @Maximgitman made their first contribution in https://github.com/BerriAI/litellm/pull/12666
+* @pathikrit made their first contribution in https://github.com/BerriAI/litellm/pull/12901
+* @huetterma made their first contribution in https://github.com/BerriAI/litellm/pull/12809
+* @betterthanbreakfast made their first contribution in https://github.com/BerriAI/litellm/pull/13029
+* @phosae made their first contribution in https://github.com/BerriAI/litellm/pull/12606
+* @sahusiddharth made their first contribution in https://github.com/BerriAI/litellm/pull/12507
+* @Amit-kr26 made their first contribution in https://github.com/BerriAI/litellm/pull/11954
+* @kowyo made their first contribution in https://github.com/BerriAI/litellm/pull/13172
+* @AnandKhinvasara made their first contribution in https://github.com/BerriAI/litellm/pull/13187
+* @unique-jakub made their first contribution in https://github.com/BerriAI/litellm/pull/13174
+* @tyumentsev4 made their first contribution in https://github.com/BerriAI/litellm/pull/13134
+* @aayush-malviya-acquia made their first contribution in https://github.com/BerriAI/litellm/pull/12978
+* @kankute-sameer made their first contribution in https://github.com/BerriAI/litellm/pull/13225
+* @AlexanderYastrebov made their first contribution in https://github.com/BerriAI/litellm/pull/13178
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.74.9-stable...v1.74.15.rc)**
\ No newline at end of file
diff --git a/docs/my-website/release_notes/v1.74.7/index.md b/docs/my-website/release_notes/v1.74.7/index.md
index e3a2ac0aa00..7d7a568e13f 100644
--- a/docs/my-website/release_notes/v1.74.7/index.md
+++ b/docs/my-website/release_notes/v1.74.7/index.md
@@ -148,7 +148,6 @@ Starting with this release, you can run health endpoints on an isolated process
- New provider integration for v0.dev - [PR #12751](https://github.com/BerriAI/litellm/pull/12751), [Get Started](../../docs/providers/v0)
- **[OpenAI](../../docs/providers/openai)**
- Use OpenAI DeepResearch models with `litellm.completion` (`/chat/completions`) - [PR #12627](https://github.com/BerriAI/litellm/pull/12627) **DOC NEEDED**
- - Add `input_fidelity` parameter for OpenAI image generation - [PR #12662](https://github.com/BerriAI/litellm/pull/12662), [Get Started](../../docs/image_generation)
- **[Azure OpenAI](../../docs/providers/azure_openai)**
- Use Azure OpenAI DeepResearch models with `litellm.completion` (`/chat/completions`) - [PR #12627](https://github.com/BerriAI/litellm/pull/12627) **DOC NEEDED**
- Added `response_format` support for openai gpt-4.1 models - [PR #12745](https://github.com/BerriAI/litellm/pull/12745)
diff --git a/docs/my-website/release_notes/v1.74.9-stable/index.md b/docs/my-website/release_notes/v1.74.9-stable/index.md
index a72ffe1a2cf..3f100745dfe 100644
--- a/docs/my-website/release_notes/v1.74.9-stable/index.md
+++ b/docs/my-website/release_notes/v1.74.9-stable/index.md
@@ -1,5 +1,5 @@
---
-title: "[PRE-RELEASE] v1.74.9-stable"
+title: "v1.74.9-stable - Auto-Router"
slug: "v1-74-9"
date: 2025-07-27T10:00:00
authors:
@@ -21,21 +21,104 @@ import TabItem from '@theme/TabItem';
## Deploy this version
-:::info
+
+
-This release is not live yet.
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.74.9-stable.patch.1
+```
+
-:::
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.74.9.post2
+```
+
+
+
---
+## Key Highlights
+
+- **Auto-Router** - Automatically route requests to specific models based on request content.
+- **Model-level Guardrails** - Only run guardrails when specific models are used.
+- **MCP Header Propagation** - Propagate headers from client to backend MCP.
+- **New LLM Providers** - Added Bedrock inpainting support and Recraft API image generation / image edits support.
+
+---
+
+## Auto-Router
+
+
+
+
+
+This release introduces auto-routing to models based on request content. This means **Proxy Admins** can define a set of keywords that always routes to specific models when **users** opt in to using the auto-router.
+
+This is great for internal use cases where you don't want **users** to think about which model to use - for example, use Claude models for coding vs GPT models for generating ad copy.
+
+
+[Read More](../../docs/proxy/auto_routing)
+
+---
+
+## Model-level Guardrails
+
+
+
+
+
+This release brings model-level guardrails support to your config.yaml + UI. This is great for cases when you have an on-prem and hosted model, and just want to run prevent sending PII to the hosted model.
+
+```yaml
+model_list:
+ - model_name: claude-sonnet-4
+ litellm_params:
+ model: anthropic/claude-sonnet-4-20250514
+ api_key: os.environ/ANTHROPIC_API_KEY
+ api_base: https://api.anthropic.com/v1
+ guardrails: ["azure-text-moderation"] # 👈 KEY CHANGE
+
+guardrails:
+ - guardrail_name: azure-text-moderation
+ litellm_params:
+ guardrail: azure/text_moderations
+ mode: "post_call"
+ api_key: os.environ/AZURE_GUARDRAIL_API_KEY
+ api_base: os.environ/AZURE_GUARDRAIL_API_BASE
+```
+
+
+[Read More](../../docs/proxy/guardrails/quick_start#model-level-guardrails)
+
+---
+## MCP Header Propagation
+
+
+
+
+
+v1.74.9-stable allows you to propagate MCP server specific authentication headers via LiteLLM
+
+- Allowing users to specify which `header_name` is to be propagated to which `mcp_server` via headers
+- Allows adding of different deployments of same MCP server type to use different authentication headers
+
+
+[Read More](https://docs.litellm.ai/docs/mcp#new-server-specific-auth-headers-recommended)
+
+---
## New Models / Updated Models
#### Pricing / Context Window Updates
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) |
| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- |
-| Fireworks AI | `fireworks/models/kimi-k2-instruct | 131k | $0.6 | $2.5 |
+| Fireworks AI | `fireworks/models/kimi-k2-instruct` | 131k | $0.6 | $2.5 |
| OpenRouter | `openrouter/qwen/qwen-vl-plus` | 8192 | $0.21 | $0.63 |
| OpenRouter | `openrouter/qwen/qwen3-coder` | 8192 | $1 | $5 |
| OpenRouter | `openrouter/bytedance/ui-tars-1.5-7b` | 128k | $0.10 | $0.20 |
diff --git a/docs/my-website/release_notes/v1.75.5-stable/index.md b/docs/my-website/release_notes/v1.75.5-stable/index.md
new file mode 100644
index 00000000000..7035d285057
--- /dev/null
+++ b/docs/my-website/release_notes/v1.75.5-stable/index.md
@@ -0,0 +1,300 @@
+---
+title: "v1.75.5-stable - Redis latency improvements"
+slug: "v1-75-5"
+date: 2025-08-10T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.75.5-stable
+```
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.75.5.post2
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **Redis - Latency Improvements** - Reduces P99 latency by 50% with Redis enabled.
+- **Responses API Session Management** - Support for managing responses API sessions with images.
+- **Oracle Cloud Infrastructure** - New LLM provider for calling models on Oracle Cloud Infrastructure.
+- **Digital Ocean's Gradient AI** - New LLM provider for calling models on Digital Ocean's Gradient AI platform.
+
+---
+
+### Risk of Upgrade
+
+If you build the proxy from the pip package, you should hold off on upgrading. This version makes `prisma migrate deploy` our default for managing the DB. This is safer, as it doesn't reset the DB, but it requires a manual `prisma generate` step.
+
+Users of our Docker image, are **not** affected by this change.
+
+---
+
+## Redis Latency Improvements
+
+
+
+
+
+This release adds in-memory caching for Redis requests, enabling faster response times in high-traffic. Now, LiteLLM instances will check their in-memory cache for a cache hit, before checking Redis. This reduces caching-related latency from 100ms for LLM API calls to sub-1ms, on cache hits.
+
+---
+
+## Responses API Session Management w/ Images
+
+
+
+
+
+LiteLLM now supports session management for Responses API requests with images. This is great for use-cases like chatbots, that are using the Responses API to track the state of a conversation. LiteLLM session management works across **ALL** LLM API's (including Anthropic, Bedrock, OpenAI, etc). LiteLLM session management works by storing the request and response content in an s3 bucket, you can specify.
+
+---
+
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) |
+| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- |
+| Bedrock | `bedrock/us.anthropic.claude-opus-4-1-20250805-v1:0` | 200k | $15 | $75 |
+| Bedrock | `bedrock/openai.gpt-oss-20b-1:0` | 200k | 0.07 | 0.3 |
+| Bedrock | `bedrock/openai.gpt-oss-120b-1:0` | 200k | 0.15 | 0.6 |
+| Fireworks AI | `fireworks_ai/accounts/fireworks/models/glm-4p5` | 128k | 0.55 | 2.19 |
+| Fireworks AI | `fireworks_ai/accounts/fireworks/models/glm-4p5-air` | 128k | 0.22 | 0.88 |
+| Fireworks AI | `fireworks_ai/accounts/fireworks/models/gpt-oss-120b` | 131072 | 0.15 | 0.6 |
+| Fireworks AI | `fireworks_ai/accounts/fireworks/models/gpt-oss-20b` | 131072 | 0.05 | 0.2 |
+| Groq | `groq/openai/gpt-oss-20b` | 131072 | 0.1 | 0.5 |
+| Groq | `groq/openai/gpt-oss-120b` | 131072 | 0.15 | 0.75 |
+| OpenAI | `openai/gpt-5` | 400k | 1.25 | 10 |
+| OpenAI | `openai/gpt-5-2025-08-07` | 400k | 1.25 | 10 |
+| OpenAI | `openai/gpt-5-mini` | 400k | 0.25 | 2 |
+| OpenAI | `openai/gpt-5-mini-2025-08-07` | 400k | 0.25 | 2 |
+| OpenAI | `openai/gpt-5-nano` | 400k | 0.05 | 0.4 |
+| OpenAI | `openai/gpt-5-nano-2025-08-07` | 400k | 0.05 | 0.4 |
+| OpenAI | `openai/gpt-5-chat` | 400k | 1.25 | 10 |
+| OpenAI | `openai/gpt-5-chat-latest` | 400k | 1.25 | 10 |
+| Azure | `azure/gpt-5` | 400k | 1.25 | 10 |
+| Azure | `azure/gpt-5-2025-08-07` | 400k | 1.25 | 10 |
+| Azure | `azure/gpt-5-mini` | 400k | 0.25 | 2 |
+| Azure | `azure/gpt-5-mini-2025-08-07` | 400k | 0.25 | 2 |
+| Azure | `azure/gpt-5-nano-2025-08-07` | 400k | 0.05 | 0.4 |
+| Azure | `azure/gpt-5-nano` | 400k | 0.05 | 0.4 |
+| Azure | `azure/gpt-5-chat` | 400k | 1.25 | 10 |
+| Azure | `azure/gpt-5-chat-latest` | 400k | 1.25 | 10 |
+
+#### Features
+
+- **[OCI](../../docs/providers/oci)**
+ - New LLM provider - [PR #13206](https://github.com/BerriAI/litellm/pull/13206)
+- **[JinaAI](../../docs/providers/jina_ai)**
+ - support multimodal embedding models - [PR #13181](https://github.com/BerriAI/litellm/pull/13181)
+- **GPT-5 ([OpenAI](../../docs/providers/openai)/[Azure](../../docs/providers/azure))**
+ - Support drop_params for temperature - [PR #13390](https://github.com/BerriAI/litellm/pull/13390)
+ - Map max_tokens to max_completion_tokens - [PR #13390](https://github.com/BerriAI/litellm/pull/13390)
+- **[Anthropic](../../docs/providers/anthropic)**
+ - Add claude-opus-4-1 on model cost map - [PR #13384](https://github.com/BerriAI/litellm/pull/13384)
+- **[OpenRouter](../../docs/providers/openrouter)**
+ - Add gpt-oss to model cost map - [PR #13442](https://github.com/BerriAI/litellm/pull/13442)
+- **[Cerebras](../../docs/providers/cerebras)**
+ - Add gpt-oss to model cost map - [PR #13442](https://github.com/BerriAI/litellm/pull/13442)
+- **[Azure](../../docs/providers/azure)**
+ - Support drop params for ‘temperature’ on o-series models - [PR #13353](https://github.com/BerriAI/litellm/pull/13353)
+- **[GradientAI](../../docs/providers/gradient_ai)**
+ - New LLM Provider - [PR #12169](https://github.com/BerriAI/litellm/pull/12169)
+
+#### Bugs
+
+- **[OpenAI](../../docs/providers/openai)**
+ - Add ‘service_tier’ and ‘safety_identifier’ as supported responses api params - [PR #13258](https://github.com/BerriAI/litellm/pull/13258)
+ - Correct pricing for web search on 4o-mini - [PR #13269](https://github.com/BerriAI/litellm/pull/13269)
+- **[Mistral](../../docs/providers/mistral)**
+ - Handle $id and $schema fields when calling mistral - [PR #13389](https://github.com/BerriAI/litellm/pull/13389)
+---
+
+## LLM API Endpoints
+
+#### Features
+
+- `/responses`
+ - Responses API Session Handling w/ support for images - [PR #13347](https://github.com/BerriAI/litellm/pull/13347)
+ - failed if input containing ResponseReasoningItem - [PR #13465](https://github.com/BerriAI/litellm/pull/13465)
+ - Support custom tools - [PR #13418](https://github.com/BerriAI/litellm/pull/13418)
+
+#### Bugs
+
+- `/chat/completions`
+ - Fix completion_token_details usage object missing ‘text’ tokens - [PR #13234](https://github.com/BerriAI/litellm/pull/13234)
+ - (SDK) handle tool being a pydantic object - [PR #13274](https://github.com/BerriAI/litellm/pull/13274)
+ - include cost in streaming usage object - [PR #13418](https://github.com/BerriAI/litellm/pull/13418)
+ - Exclude none fields on /chat/completion - allows usage with n8n - [PR #13320](https://github.com/BerriAI/litellm/pull/13320)
+- `/responses`
+ - Transform function call in response for non-openai models (gemini/anthropic) - [PR #13260](https://github.com/BerriAI/litellm/pull/13260)
+ - Fix unsupported operand error with model groups - [PR #13293](https://github.com/BerriAI/litellm/pull/13293)
+ - Responses api session management for streaming responses - [PR #13396](https://github.com/BerriAI/litellm/pull/13396)
+- `/v1/messages`
+ - Added litellm claude code count tokens - [PR #13261](https://github.com/BerriAI/litellm/pull/13261)
+- `/vector_stores`
+ - Fix create/search vector store errors - [PR #13285](https://github.com/BerriAI/litellm/pull/13285)
+---
+
+## [MCP Gateway](../../docs/mcp)
+
+#### Features
+
+- Add route check for internal users - [PR #13350](https://github.com/BerriAI/litellm/pull/13350)
+- MCP Guardrails - docs - [PR #13392](https://github.com/BerriAI/litellm/pull/13392)
+
+
+#### Bugs
+
+- Fix auth on UI for bearer token servers - [PR #13312](https://github.com/BerriAI/litellm/pull/13312)
+- allow access group on mcp tool retrieval - [PR #13425](https://github.com/BerriAI/litellm/pull/13425)
+
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **Teams**
+ - Add team deletion check for teams with keys - [PR #12953](https://github.com/BerriAI/litellm/pull/12953)
+- **Models**
+ - Add ability to set model alias per key/team - [PR #13276](https://github.com/BerriAI/litellm/pull/13276)
+ - New button to reload model pricing from model cost map - [PR #13464](https://github.com/BerriAI/litellm/pull/13464), [PR #13470](https://github.com/BerriAI/litellm/pull/13470)
+- **Keys**
+ - Make ‘team’ field required when creating service account keys - [PR #13302](https://github.com/BerriAI/litellm/pull/13302)
+ - Gray out key-based logging settings for non-enterprise users - prevents confusion on if ‘logging’ all up is supported - [PR #13431](https://github.com/BerriAI/litellm/pull/13431)
+- **Navbar**
+ - Add logo customization for LiteLLM admin UI - [PR #12958](https://github.com/BerriAI/litellm/pull/12958)
+- **Logs**
+ - Add token breakdowns on logs + session page - [PR #13357](https://github.com/BerriAI/litellm/pull/13357)
+- **Usage**
+ - Ensure Usage Page loads after the DB has large entries - [PR #13400](https://github.com/BerriAI/litellm/pull/13400)
+- **Test Key Page**
+ - allow uploading images for /chat/completions and /responses - [PR #13445](https://github.com/BerriAI/litellm/pull/13445)
+- **MCP**
+ - Add auth tokens to local storage auth - [PR #13473](https://github.com/BerriAI/litellm/pull/13473)
+
+#### Bugs
+
+- **Custom Root Path**
+ - Fix login route when SSO is enabled - [PR #13267](https://github.com/BerriAI/litellm/pull/13267)
+- **Customers/End-users**
+ - Allow calling /v1/models when end user over budget - allows model listing to work on OpenWebUI when customer over budget - [PR #13320](https://github.com/BerriAI/litellm/pull/13320)
+- **Teams**
+ - Remove user - team membership, when user removed from team - [PR #13433](https://github.com/BerriAI/litellm/pull/13433)
+- **Errors**
+ - Bubble up network errors to user for Logging and Alerts page - [PR #13427](https://github.com/BerriAI/litellm/pull/13427)
+- **Model Hub**
+ - Show pricing for azure models, when base model is set - [PR #13418](https://github.com/BerriAI/litellm/pull/13418)
+---
+
+## Logging / Guardrail Integrations
+
+#### Features
+
+- **Bedrock Guardrails**
+ - Redacted sensitive information in bedrock guardrails error message - [PR #13356](https://github.com/BerriAI/litellm/pull/13356)
+- **Standard Logging Payload**
+ - Fix ‘can’t register atextexit’ bug - [PR #13436](https://github.com/BerriAI/litellm/pull/13436)
+
+#### Bugs
+
+- **Braintrust**
+ - Allow setting of braintrust callback base url - [PR #13368](https://github.com/BerriAI/litellm/pull/13368)
+- **OTEL**
+ - Track pre_call hook latency - [PR #13362](https://github.com/BerriAI/litellm/pull/13362)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+#### Features
+
+- **Team-BYOK models**
+ - Add wildcard model support - [PR #13278](https://github.com/BerriAI/litellm/pull/13278)
+- **Caching**
+ - GCP IAM auth support for caching - [PR #13275](https://github.com/BerriAI/litellm/pull/13275)
+- **Latency**
+ - reduce p99 latency w/ redis enabled by 50% - only updates model usage if tpm/rpm limits set - [PR #13362](https://github.com/BerriAI/litellm/pull/13362)
+
+---
+
+## General Proxy Improvements
+
+#### Features
+
+- **Models**
+ - Support /v1/models/\{model_id\} retrieval - [PR #13268](https://github.com/BerriAI/litellm/pull/13268)
+- **Multi-instance**
+ - Ensure disable_llm_api_endpoints works - [PR #13278](https://github.com/BerriAI/litellm/pull/13278)
+- **Logs**
+ - Add apscheduler log suppress - [PR #13299](https://github.com/BerriAI/litellm/pull/13299)
+- **Helm**
+ - Add labels to migrations job template - [PR #13343](https://github.com/BerriAI/litellm/pull/13343) s/o [@unique-jakub](https://github.com/unique-jakub)
+
+#### Bugs
+
+- **Non-root image**
+ - Fix non-root image for migration - [PR #13379](https://github.com/BerriAI/litellm/pull/13379)
+- **Get Routes**
+ - Load get routes when using fastapi-offline - [PR #13466](https://github.com/BerriAI/litellm/pull/13466)
+- **Health checks**
+ - Generate unique trace IDs for Langfuse health checks - [PR #13468](https://github.com/BerriAI/litellm/pull/13468)
+- **Swagger**
+ - Allow using Swagger for /chat/completions - [PR #13469](https://github.com/BerriAI/litellm/pull/13469)
+- **Auth**
+ - Fix JWTs access not working with model access groups - [PR #13474](https://github.com/BerriAI/litellm/pull/13474)
+
+---
+
+## New Contributors
+
+* @bbartels made their first contribution in https://github.com/BerriAI/litellm/pull/13244
+* @breno-aumo made their first contribution in https://github.com/BerriAI/litellm/pull/13206
+* @pascalwhoop made their first contribution in https://github.com/BerriAI/litellm/pull/13122
+* @ZPerling made their first contribution in https://github.com/BerriAI/litellm/pull/13045
+* @zjx20 made their first contribution in https://github.com/BerriAI/litellm/pull/13181
+* @edwarddamato made their first contribution in https://github.com/BerriAI/litellm/pull/13368
+* @msannan2 made their first contribution in https://github.com/BerriAI/litellm/pull/12169
+
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.74.15-stable...v1.75.5-stable.rc-draft)**
\ No newline at end of file
diff --git a/docs/my-website/release_notes/v1.75.8/index.md b/docs/my-website/release_notes/v1.75.8/index.md
new file mode 100644
index 00000000000..d7d4f37c4ee
--- /dev/null
+++ b/docs/my-website/release_notes/v1.75.8/index.md
@@ -0,0 +1,247 @@
+---
+title: "v1.75.8-stable - Team Member Rate Limits"
+slug: "v1-75-8"
+date: 2025-08-16T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.75.8-stable
+```
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.75.8
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **Team Member Rate Limits** - Individual rate limiting for team members with JWT authentication support.
+- **Performance Improvements** - New experimental HTTP handler flag for 100+ RPS improvement on OpenAI calls.
+- **GPT-5 Model Family Support** - Full support for OpenAI's GPT-5 models with `reasoning_effort` parameter and Azure OpenAI integration.
+- **Azure AI Flux Image Generation** - Support for Azure AI's Flux image generation models.
+
+---
+
+## Team Member Rate Limits
+
+
+
+ LiteLLM MCP Architecture: Use MCP tools with all LiteLLM supported models
+
+
+
+This release adds support for setting rate limits on individual members (including machine users) within a team. Teams can now give each agent its own rate limits—so that heavy-traffic agents don’t impact other agents or human users.
+
+Agents can authenticate with LiteLLM using JWT and the same team role as human users, while still enforcing per-agent rate limits.
+
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
+| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- | -------- |
+| Azure AI | `azure_ai/FLUX-1.1-pro` | - | - | $40/image | Image generation |
+| Azure AI | `azure_ai/FLUX.1-Kontext-pro` | - | - | $40/image | Image generation |
+| Vertex AI | `vertex_ai/deepseek-ai/deepseek-r1-0528-maas` | 65k | $1.35 | $5.4 | Chat completions + reasoning |
+| OpenRouter | `openrouter/deepseek/deepseek-chat-v3-0324` | 65k | $0.14 | $0.28 | Chat completions |
+
+
+#### Features
+
+- **[OpenAI](../../docs/providers/openai)**
+ - Added `reasoning_effort` parameter support for GPT-5 model family - [PR #13475](https://github.com/BerriAI/litellm/pull/13475), [Get Started](../../docs/providers/openai#openai-chat-completion-models)
+ - Support for `reasoning` parameter in Responses API - [PR #13475](https://github.com/BerriAI/litellm/pull/13475), [Get Started](../../docs/response_api)
+- **[Azure OpenAI](../../docs/providers/azure/azure)**
+ - GPT-5 support with max_tokens and `reasoning` parameter - [PR #13510](https://github.com/BerriAI/litellm/pull/13510), [Get Started](../../docs/providers/azure/azure#gpt-5-models)
+- **[AWS Bedrock](../../docs/providers/bedrock)**
+ - Streaming support for bedrock gpt-oss model family - [PR #13346](https://github.com/BerriAI/litellm/pull/13346), [Get Started](../../docs/providers/bedrock#openai-gpt-oss)
+ - `/messages` endpoint compatibility with `bedrock/converse/` - [PR #13627](https://github.com/BerriAI/litellm/pull/13627)
+ - Cache point support for assistant and tool messages - [PR #13640](https://github.com/BerriAI/litellm/pull/13640)
+- **[Azure AI](../../docs/providers/azure)**
+ - New Azure AI Flux Image Generation provider - [PR #13592](https://github.com/BerriAI/litellm/pull/13592), [Get Started](../../docs/providers/azure_ai_img)
+ - Fixed Content-Type header for image generation - [PR #13584](https://github.com/BerriAI/litellm/pull/13584)
+- **[CometAPI](../../docs/providers/comet)**
+ - New provider support with chat completions and streaming - [PR #13458](https://github.com/BerriAI/litellm/pull/13458)
+- **[SambaNova](../../docs/providers/sambanova)**
+ - Added embedding model support - [PR #13308](https://github.com/BerriAI/litellm/pull/13308), [Get Started](../../docs/providers/sambanova#sambanova---embeddings)
+- **[Vertex AI](../../docs/providers/vertex)**
+ - Added `/countTokens` endpoint support for Gemini CLI integration - [PR #13545](https://github.com/BerriAI/litellm/pull/13545)
+ - Token counter support for VertexAI models - [PR #13558](https://github.com/BerriAI/litellm/pull/13558)
+- **[hosted_vllm](../../docs/providers/vllm)**
+ - Added `reasoning_effort` parameter support - [PR #13620](https://github.com/BerriAI/litellm/pull/13620), [Get Started](../../docs/providers/vllm#reasoning-effort)
+
+#### Bugs
+
+- **[OCI](../../docs/providers/oci)**
+ - Fixed streaming issues - [PR #13437](https://github.com/BerriAI/litellm/pull/13437)
+- **[Ollama](../../docs/providers/ollama)**
+ - Fixed GPT-OSS streaming with 'thinking' field - [PR #13375](https://github.com/BerriAI/litellm/pull/13375)
+- **[VolcEngine](../../docs/providers/volcengine)**
+ - Fixed thinking disabled parameter handling - [PR #13598](https://github.com/BerriAI/litellm/pull/13598)
+- **[Streaming](../../docs/completion/stream)**
+ - Consistent 'finish_reason' chunk indexing - [PR #13560](https://github.com/BerriAI/litellm/pull/13560)
+---
+
+## LLM API Endpoints
+
+#### Features
+
+- **[/messages](../../docs/anthropic/messages)**
+ - Tool use arguments properly returned for non-anthropic models - [PR #13638](https://github.com/BerriAI/litellm/pull/13638)
+
+#### Bugs
+
+- **[Real-time API](../../docs/realtime)**
+ - Fixed endpoint for no intent scenarios - [PR #13476](https://github.com/BerriAI/litellm/pull/13476)
+- **[Responses API](../../docs/response_api)**
+ - Fixed `stream=True` + `background=True` with Responses API - [PR #13654](https://github.com/BerriAI/litellm/pull/13654)
+
+---
+
+## [MCP Gateway](../../docs/mcp)
+
+#### Features
+
+- **Access Control & Configuration**
+ - Enhanced MCPServerManager with access groups and description support - [PR #13549](https://github.com/BerriAI/litellm/pull/13549)
+
+#### Bugs
+
+- **Authentication**
+ - Fixed MCP gateway key authentication - [PR #13630](https://github.com/BerriAI/litellm/pull/13630)
+
+[Read More](../../docs/mcp)
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **Team Management**
+ - Team Member Rate Limits implementation - [PR #13601](https://github.com/BerriAI/litellm/pull/13601)
+ - JWT authentication support for team member rate limits - [PR #13601](https://github.com/BerriAI/litellm/pull/13601)
+ - Show team member TPM/RPM limits in UI - [PR #13662](https://github.com/BerriAI/litellm/pull/13662)
+ - Allow editing team member RPM/TPM limits - [PR #13669](https://github.com/BerriAI/litellm/pull/13669)
+ - Allow unsetting TPM and RPM in Teams Settings - [PR #13430](https://github.com/BerriAI/litellm/pull/13430)
+ - Team Member Permissions Page access column changes - [PR #13145](https://github.com/BerriAI/litellm/pull/13145)
+- **Key Management**
+ - Display errors from backend on the UI Keys page - [PR #13435](https://github.com/BerriAI/litellm/pull/13435)
+ - Added confirmation modal before deleting keys - [PR #13655](https://github.com/BerriAI/litellm/pull/13655)
+ - Support for `user` parameter in LiteLLM SDK to Proxy communication - [PR #13555](https://github.com/BerriAI/litellm/pull/13555)
+- **UI Improvements**
+ - Fixed internal users table overflow - [PR #12736](https://github.com/BerriAI/litellm/pull/12736)
+ - Enhanced chart readability with short-form notation for large numbers - [PR #12370](https://github.com/BerriAI/litellm/pull/12370)
+ - Fixed image overflow in LiteLLM model display - [PR #13639](https://github.com/BerriAI/litellm/pull/13639)
+ - Removed ambiguous network response errors - [PR #13582](https://github.com/BerriAI/litellm/pull/13582)
+- **Credentials**
+ - Added CredentialDeleteModal component and integration with CredentialsPanel - [PR #13550](https://github.com/BerriAI/litellm/pull/13550)
+- **Admin & Permissions**
+ - Allow routes for admin viewer - [PR #13588](https://github.com/BerriAI/litellm/pull/13588)
+
+#### Bugs
+
+- **SCIM Integration**
+ - Fixed SCIM Team Memberships metadata handling - [PR #13553](https://github.com/BerriAI/litellm/pull/13553)
+- **Authentication**
+ - Fixed incorrect key info endpoint - [PR #13633](https://github.com/BerriAI/litellm/pull/13633)
+
+---
+
+## Logging / Guardrail Integrations
+
+#### Features
+
+- **[Langfuse OTEL](../../docs/proxy/logging#langfuse)**
+ - Added key/team logging for Langfuse OTEL Logger - [PR #13512](https://github.com/BerriAI/litellm/pull/13512)
+ - Fixed LangfuseOtelSpanAttributes constants to match expected values - [PR #13659](https://github.com/BerriAI/litellm/pull/13659)
+- **[MLflow](../../docs/proxy/logging#mlflow)**
+ - Updated MLflow logger usage span attributes - [PR #13561](https://github.com/BerriAI/litellm/pull/13561)
+
+#### Bugs
+
+- **Security**
+ - Hide sensitive data in `/model/info` - azure entra client_secret - [PR #13577](https://github.com/BerriAI/litellm/pull/13577)
+ - Fixed trivy/secrets false positives - [PR #13631](https://github.com/BerriAI/litellm/pull/13631)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+#### Features
+
+- **HTTP Performance**
+ - New 'EXPERIMENTAL_OPENAI_BASE_LLM_HTTP_HANDLER' flag for +100 RPS improvement on OpenAI calls - [PR #13625](https://github.com/BerriAI/litellm/pull/13625)
+- **Database Monitoring**
+ - Added DB metrics to Prometheus - [PR #13626](https://github.com/BerriAI/litellm/pull/13626)
+- **Error Handling**
+ - Added safe divide by 0 protection to prevent crashes - [PR #13624](https://github.com/BerriAI/litellm/pull/13624)
+
+#### Bugs
+
+- **Dependencies**
+ - Updated boto3 to 1.36.0 and aioboto3 to 13.4.0 - [PR #13665](https://github.com/BerriAI/litellm/pull/13665)
+
+---
+
+## General Proxy Improvements
+
+#### Features
+
+- **Database**
+ - Removed redundant `use_prisma_migrate` flag - now default - [PR #13555](https://github.com/BerriAI/litellm/pull/13555)
+- **LLM Translation**
+ - Added model ID check - [PR #13507](https://github.com/BerriAI/litellm/pull/13507)
+ - Refactored Anthropic configurations and added support for `anthropic_beta` headers - [PR #13590](https://github.com/BerriAI/litellm/pull/13590)
+
+
+---
+
+## New Contributors
+* @TensorNull made their first contribution in [PR #13458](https://github.com/BerriAI/litellm/pull/13458)
+* @MajorD00m made their first contribution in [PR #13577](https://github.com/BerriAI/litellm/pull/13577)
+* @VerunicaM made their first contribution in [PR #13584](https://github.com/BerriAI/litellm/pull/13584)
+* @huangyafei made their first contribution in [PR #13607](https://github.com/BerriAI/litellm/pull/13607)
+* @TomeHirata made their first contribution in [PR #13561](https://github.com/BerriAI/litellm/pull/13561)
+* @willfinnigan made their first contribution in [PR #13659](https://github.com/BerriAI/litellm/pull/13659)
+* @dcbark01 made their first contribution in [PR #13633](https://github.com/BerriAI/litellm/pull/13633)
+* @javacruft made their first contribution in [PR #13631](https://github.com/BerriAI/litellm/pull/13631)
+
+---
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.75.5-stable.rc-draft...v1.75.8-nightly)**
+
diff --git a/docs/my-website/release_notes/v1.76.0-stable/index.md b/docs/my-website/release_notes/v1.76.0-stable/index.md
new file mode 100644
index 00000000000..d93568d49dc
--- /dev/null
+++ b/docs/my-website/release_notes/v1.76.0-stable/index.md
@@ -0,0 +1,189 @@
+---
+title: "v1.76.0-stable - RPS Improvements"
+slug: "v1-76-0"
+date: 2025-08-23T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+:::info
+
+LiteLLM is hiring a **Founding Backend Engineer**, in San Francisco.
+
+[Apply here](https://www.ycombinator.com/companies/litellm/jobs/6uvoBp3-founding-backend-engineer) if you're interested!
+:::
+
+
+
+
+
+## Deploy this version
+
+:::info
+
+This release is not live yet.
+:::
+
+
+---
+
+## New Models / Updated Models
+
+#### Bugs
+- **[OpenAI](../../docs/providers/openai)**
+ - Gpt-5 chat: clarify does not support function calling [PR #13612](https://github.com/BerriAI/litellm/pull/13612), s/o @[superpoussin22](https://github.com/superpoussin22)
+- **[VertexAI](../../docs/providers/vertex)**
+ - fix vertexai batch file format by @[thiagosalvatore](https://github.com/thiagosalvatore) in [PR #13576](https://github.com/BerriAI/litellm/pull/13576)
+- **[LiteLLM Proxy](../../docs/providers/litellm_proxy)**
+ - Add support for calling image_edits + image_generations via SDK to Proxy - [PR #13735](https://github.com/BerriAI/litellm/pull/13735)
+- **[OpenRouter](../../docs/providers/openrouter)**
+ - Fix max_output_tokens value for anthropic Claude 4 - [PR #13526](https://github.com/BerriAI/litellm/pull/13526)
+- **[Gemini](../../docs/providers/gemini)**
+ - Fix prompt caching cost calculation - [PR #13742](https://github.com/BerriAI/litellm/pull/13742)
+- **[Azure](../../docs/providers/azure)**
+ - Support `../openai/v1/respones` api base - [PR #13526](https://github.com/BerriAI/litellm/pull/13526)
+ - Fix azure/gpt-5-chat max_input_tokens - [PR #13660](https://github.com/BerriAI/litellm/pull/13660)
+- **[Groq](../../docs/providers/groq)**
+ - streaming ASCII encoding issue - [PR #13675](https://github.com/BerriAI/litellm/pull/13675)
+- **[Baseten](../../docs/providers/baseten)**
+ - Refactored integration to use new openai-compatible endpoints - [PR #13783](https://github.com/BerriAI/litellm/pull/13783)
+- **[Bedrock](../../docs/providers/bedrock)**
+ - fix application inference profile for pass-through endpoints for bedrock - [PR #13881](https://github.com/BerriAI/litellm/pull/13881)
+- **[DataRobot](../../docs/providers/datarobot)**
+ - Updated URL handling for DataRobot provider URL - [PR #13880](https://github.com/BerriAI/litellm/pull/13880)
+
+#### Features
+- **[Together AI](../../docs/providers/together)**
+ - Added Qwen3, Deepseek R1 0528 Throughput, GLM 4.5 and GPT-OSS models cost tracking - [PR #13637](https://github.com/BerriAI/litellm/pull/13637), s/o @[Tasmay-Tibrewal](https://github.com/Tasmay-Tibrewal)
+- **[Fireworks AI](../../docs/providers/fireworks_ai)**
+ - add fireworks_ai/accounts/fireworks/models/deepseek-v3-0324 - [PR #13821](https://github.com/BerriAI/litellm/pull/13821)
+- **[VertexAI](../../docs/providers/vertex)**
+ - Add VertexAI qwen API Service - [PR #13828](https://github.com/BerriAI/litellm/pull/13828)
+ - Add new VertexAI image models vertex_ai/imagen-4.0-generate-001, vertex_ai/imagen-4.0-ultra-generate-001, vertex_ai/imagen-4.0-fast-generate-001 - [PR #13874](https://github.com/BerriAI/litellm/pull/13874)
+- **[Anthropic](../../docs/providers/anthropic)**
+ - Add long context support w/ cost tracking - [PR #13759](https://github.com/BerriAI/litellm/pull/13759)
+- **[DeepInfra](../../docs/providers/deepinfra)**
+ - Add rerank endpoint support for deepinfra - [PR #13820](https://github.com/BerriAI/litellm/pull/13820)
+ - Add new models for cost tracking - [PR #13883](https://github.com/BerriAI/litellm/pull/13883), s/o @[Toy-97](https://github.com/Toy-97)
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Add tool prompt caching on async calls - [PR #13803](https://github.com/BerriAI/litellm/pull/13803), s/o @[UlookEE](https://github.com/UlookEE)
+ - role chaining and session name with webauthentication for aws bedrock - [PR #13753](https://github.com/BerriAI/litellm/pull/13753), s/o @[RichardoC](https://github.com/RichardoC)
+- **[Ollama](../../docs/providers/ollama)**
+ - Handle Ollama null response when using tool calling with non-tool trained models - [PR #13902](https://github.com/BerriAI/litellm/pull/13902)
+- **[OpenRouter](../../docs/providers/openrouter)**
+ - Add deepseek/deepseek-chat-v3.1 support - [PR #13897](https://github.com/BerriAI/litellm/pull/13897)
+- **[Mistral](../../docs/providers/mistral)**
+ - Add support for calling mistral files via chat completions - [PR #13866](https://github.com/BerriAI/litellm/pull/13866), s/o @[jinskjoy](https://github.com/jinskjoy)
+ - Handle empty assistant content - [PR #13671](https://github.com/BerriAI/litellm/pull/13671)
+ - Support new ‘thinking’ response block - [PR #13671](https://github.com/BerriAI/litellm/pull/13671)
+- **[Databricks](../../docs/providers/databricks)**
+ - remove deprecated dbrx models (dbrx-instruct, llama 3.1) - [PR #13843](https://github.com/BerriAI/litellm/pull/13843)
+- **[AI/ML API](../../docs/providers/ai_ml_api)**
+ - Image gen api support - [PR #13893](https://github.com/BerriAI/litellm/pull/13893)
+
+
+## LLM API Endpoints
+#### Bugs
+- **[Responses API](../../docs/response_api)**
+ - add default api version for openai responses api calls - [PR #13526](https://github.com/BerriAI/litellm/pull/13526)
+ - support allowed_openai_params - [PR #13671](https://github.com/BerriAI/litellm/pull/13671)
+
+
+## MCP Gateway
+#### Bugs
+- fix StreamableHTTPSessionManager .run() error - [PR #13666](https://github.com/BerriAI/litellm/pull/13666)
+
+## Vector Stores
+#### Bugs
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Using LiteLLM Managed Credentials for Query - [PR #13787](https://github.com/BerriAI/litellm/pull/13787)
+
+## Management Endpoints / UI
+#### Bugs
+- **[Passthrough](../../docs/pass_through/intro)**
+ - Fix query passthrough deletion - [PR #13622](https://github.com/BerriAI/litellm/pull/13622)
+
+#### Features
+- **Models**
+ - Add Search Functionality for Public Model Names in Model Dashboard - [PR #13687](https://github.com/BerriAI/litellm/pull/13687)
+ - Auto-Add `azure/` to deployment Name in UI - [PR #13685](https://github.com/BerriAI/litellm/pull/13685)
+ - Models page row UI restructure - [PR #13771](https://github.com/BerriAI/litellm/pull/13771)
+- **Notifications**
+ - Add new notifications toast UI everywhere - [PR #13813](https://github.com/BerriAI/litellm/pull/13813)
+- **Keys**
+ - Fix key edit settings after regenerating a key - [PR #13815](https://github.com/BerriAI/litellm/pull/13815)
+ - Require team_id when creating service account keys - [PR #13873](https://github.com/BerriAI/litellm/pull/13873)
+ - Filter - show all options on filter option click - [PR #13858](https://github.com/BerriAI/litellm/pull/13858)
+- **Usage**
+ - Fix ‘Cannot read properties of undefined’ exception on user agent activity tab - [PR #13892](https://github.com/BerriAI/litellm/pull/13892)
+- **SSO**
+ - Free SSO usage for up to 5 users - [PR #13843](https://github.com/BerriAI/litellm/pull/13843)
+
+## Logging / Guardrail Integrations
+#### Bugs
+- **[Bedrock Guardrails](../../docs/proxy/guardrails/bedrock)**
+ - Add bedrock api key support - [PR #13835](https://github.com/BerriAI/litellm/pull/13835)
+#### Features
+- **[Datadog LLM Observability](../../docs/integrations/datadog)**
+ - Add support for Failure Logging [PR #13726](https://github.com/BerriAI/litellm/pull/13726)
+ - Add time to first token, litellm overhead, guardrail overhead latency metrics - [PR #13734](https://github.com/BerriAI/litellm/pull/13734)
+ - Add support for tracing guardrail input/output - [PR #13767](https://github.com/BerriAI/litellm/pull/13767)
+- **[Langfuse OTEL](../../docs/integrations/langfuse)**
+ - Allow using Key/Team Based Logging - [PR #13791](https://github.com/BerriAI/litellm/pull/13791)
+- **[AIM](../../docs/integrations/aim)**
+ - Migrate to new firewall API - [PR #13748](https://github.com/BerriAI/litellm/pull/13748)
+- **[OTEL](../../docs/observability/opentelemetry_integration)**
+ - Add OTEL tracing for actual LLM API call - [PR #13836](https://github.com/BerriAI/litellm/pull/13836)
+- **[MLFlow](../../docs/observability/mlflow_integration)**
+ - Include predicted output in MLflow tracing - [PR #13795](https://github.com/BerriAI/litellm/pull/13795), s/o @TomeHirata
+
+
+## Performance / Loadbalancing / Reliability improvements
+#### Bugs
+- **[Cooldowns](../../docs/routing#how-cooldowns-work)**
+ - don't return raw Azure Exceptions to client (can contain prompt leakage) - [PR #13529](https://github.com/BerriAI/litellm/pull/13529)
+- **[Auto-router](../../docs/proxy/auto_routing)**
+ - Ensures the relevant dependencies for auto router existing on LiteLLM Docker - [PR #13788](https://github.com/BerriAI/litellm/pull/13788)
+- **Model Alias**
+ - Fix calling key with access to model alias - [PR #13830](https://github.com/BerriAI/litellm/pull/13830)
+
+#### Features
+- **[S3 Caching](../../docs/proxy/caching)**
+ - Use namespace as prefix for s3 cache - [PR #13704](https://github.com/BerriAI/litellm/pull/13704)
+ - Async S3 Caching support (4x RPS improvement) - [PR #13852](https://github.com/BerriAI/litellm/pull/13852), s/o @[michal-otmianowski](https://github.com/michal-otmianowski)
+- **Model Group header forwarding**
+ - reuse same logic as global header forwarding - [PR #13741](https://github.com/BerriAI/litellm/pull/13741)
+ - add support for hosted_vllm on UI - [PR #13885](https://github.com/BerriAI/litellm/pull/13885)
+- **Performance**
+ - Improve LiteLLM Python SDK RPS by +200 RPS (braintrust import + aiohttp transport fixes) - [PR #13839](https://github.com/BerriAI/litellm/pull/13839)
+ - Use O(1) Set lookups for model routing - [PR #13879](https://github.com/BerriAI/litellm/pull/13879)
+ - Reduce Significant CPU overhead from litellm_logging.py - [PR #13895](https://github.com/BerriAI/litellm/pull/13895)
+ - Improvements for Async Success Handler (Logging Callbacks) - Approx +130 RPS - [PR #13905](https://github.com/BerriAI/litellm/pull/13905)
+
+
+## General Proxy Improvements
+#### Bugs
+
+- **SDK**
+ - Fix litellm compatibility with newest release of openAI (>v1.100.0) - [PR #13728](https://github.com/BerriAI/litellm/pull/13728)
+- **Helm**
+ - Add possibility to configure resources for migrations-job - [PR #13617](https://github.com/BerriAI/litellm/pull/13617)
+ - Ensure Helm chart auto generated master keys follow sk-xxxx format - [PR #13871](https://github.com/BerriAI/litellm/pull/13871)
+ - Enhance database configuration: add support for optional endpointKey - [PR #13763](https://github.com/BerriAI/litellm/pull/13763)
+- **Rate Limits**
+ - fixing descriptor/response size mismatch on parallel_request_limiter_v3 - [PR #13863](https://github.com/BerriAI/litellm/pull/13863), s/o @[luizrennocosta](https://github.com/luizrennocosta)
+- **Non-root**
+ - fix permission access on prisma migrate in non-root image - [PR #13848](https://github.com/BerriAI/litellm/pull/13848), s/o @[Ithanil](https://github.com/Ithanil)
\ No newline at end of file
diff --git a/docs/my-website/release_notes/v1.76.1-stable/index.md b/docs/my-website/release_notes/v1.76.1-stable/index.md
new file mode 100644
index 00000000000..4437b7f5799
--- /dev/null
+++ b/docs/my-website/release_notes/v1.76.1-stable/index.md
@@ -0,0 +1,269 @@
+---
+title: "v1.76.1-stable - Gemini 2.5 Flash Image"
+slug: "v1-76-1"
+date: 2025-08-30T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.76.1
+```
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.76.1
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **Major Performance Improvements** - 6.5x faster LiteLLM Python SDK completion with fastuuid integration.
+- **New Model Support** - Gemini 2.5 Flash Image Preview, Grok Code Fast, and GPT Realtime models
+- **Enhanced Provider Support** - DeepSeek-v3.1 pricing on Fireworks AI, Vercel AI Gateway, and improved Anthropic/GitHub Copilot integration
+- **MCP Improvements** - Better connection testing and SSE MCP tools bug fixes
+
+## Major Changes
+- Added support for using Gemini 2.5 Flash Image Preview with /chat/completions. **🚨 Warning** If you were using `gemini-2.0-flash-exp-image-generation` please follow this migration guide.
+ [Gemini Image Generation Migration Guide](../../docs/extras/gemini_img_migration)
+---
+
+## Performance Improvements
+
+This release includes significant performance optimizations:
+
+- **6.5x faster LiteLLM Python SDK Completion** - Major performance boost for completion operations - [PR #13990](https://github.com/BerriAI/litellm/pull/13990)
+- **fastuuid Integration** - 2.1x faster UUID generation with +80 RPS improvement for /chat/completions and other LLM endpoints - [PR #13992](https://github.com/BerriAI/litellm/pull/13992), [PR #14016](https://github.com/BerriAI/litellm/pull/14016)
+- **Optimized Request Logging** - Don't print request params by default for +50 RPS improvement - [PR #14015](https://github.com/BerriAI/litellm/pull/14015)
+- **Cache Performance** - 21% speedup in InMemoryCache.evict_cache and 45% speedup in `_is_debugging_on` function - [PR #14012](https://github.com/BerriAI/litellm/pull/14012), [PR #13988](https://github.com/BerriAI/litellm/pull/13988)
+
+---
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
+| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- | -------- |
+| Google | `gemini-2.5-flash-image-preview` | 1M | $0.30 | $2.50 | Chat completions + image generation ($0.039/image) |
+| X.AI | `xai/grok-code-fast` | 256K | $0.20 | $1.50 | Code generation |
+| OpenAI | `gpt-realtime` | 32K | $4.00 | $16.00 | Real-time conversation + audio |
+| Vercel AI Gateway | `vercel_ai_gateway/openai/o3` | 200K | $2.00 | $8.00 | Advanced reasoning |
+| Vercel AI Gateway | `vercel_ai_gateway/openai/o3-mini` | 200K | $1.10 | $4.40 | Efficient reasoning |
+| Vercel AI Gateway | `vercel_ai_gateway/openai/o4-mini` | 200K | $1.10 | $4.40 | Latest mini model |
+| DeepInfra | `deepinfra/zai-org/GLM-4.5` | 131K | $0.55 | $2.00 | Chat completions |
+| Perplexity | `perplexity/codellama-34b-instruct` | 16K | $0.35 | $1.40 | Code generation |
+| Fireworks AI | `fireworks_ai/accounts/fireworks/models/deepseek-v3p1` | 128K | $0.56 | $1.68 | Chat completions |
+
+**Additional Models Added:** Various other Vercel AI Gateway models were added too. See [models.litellm.ai](https://models.litellm.ai) for the full list.
+
+#### Features
+
+- **[Google Gemini](../../docs/providers/gemini)**
+ - Added support for `gemini-2.5-flash-image-preview` with image return capability - [PR #13979](https://github.com/BerriAI/litellm/pull/13979), [PR #13983](https://github.com/BerriAI/litellm/pull/13983)
+ - Support for requests with only system prompt - [PR #14010](https://github.com/BerriAI/litellm/pull/14010)
+ - Fixed invalid model name error for Gemini Imagen models - [PR #13991](https://github.com/BerriAI/litellm/pull/13991)
+- **[X.AI](../../docs/providers/xai)**
+ - Added `xai/grok-code-fast` model family support - [PR #14054](https://github.com/BerriAI/litellm/pull/14054)
+ - Fixed frequency_penalty parameter for grok-4 models - [PR #14078](https://github.com/BerriAI/litellm/pull/14078)
+- **[OpenAI](../../docs/providers/openai)**
+ - Added support for gpt-realtime models - [PR #14082](https://github.com/BerriAI/litellm/pull/14082)
+ - Support for reasoning and reasoning_effort parameters by default - [PR #12865](https://github.com/BerriAI/litellm/pull/12865)
+- **[Fireworks AI](../../docs/providers/fireworks_ai)**
+ - Added DeepSeek-v3.1 pricing - [PR #13958](https://github.com/BerriAI/litellm/pull/13958)
+- **[DeepInfra](../../docs/providers/deepinfra)**
+ - Fixed reasoning_effort setting for DeepSeek-V3.1 - [PR #14053](https://github.com/BerriAI/litellm/pull/14053)
+- **[GitHub Copilot](../../docs/providers/github_copilot)**
+ - Added support for thinking and reasoning_effort parameters - [PR #13691](https://github.com/BerriAI/litellm/pull/13691)
+ - Added image headers support - [PR #13955](https://github.com/BerriAI/litellm/pull/13955)
+- **[Anthropic](../../docs/providers/anthropic)**
+ - Support for custom Anthropic-compatible API endpoints - [PR #13945](https://github.com/BerriAI/litellm/pull/13945)
+ - Fixed /messages fallback from Anthropic API to Bedrock API - [PR #13946](https://github.com/BerriAI/litellm/pull/13946)
+- **[Nebius](../../docs/providers/nebius)**
+ - Expanded provider models and normalized model IDs - [PR #13965](https://github.com/BerriAI/litellm/pull/13965)
+- **[Vertex AI](../../docs/providers/vertex)**
+ - Fixed Vertex Mistral streaming issues - [PR #13952](https://github.com/BerriAI/litellm/pull/13952)
+ - Fixed anyOf corner cases for Gemini tool calls - [PR #12797](https://github.com/BerriAI/litellm/pull/12797)
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Fixed structure output issues - [PR #14005](https://github.com/BerriAI/litellm/pull/14005)
+- **[OpenRouter](../../docs/providers/openrouter)**
+ - Added GPT-5 family models pricing - [PR #13536](https://github.com/BerriAI/litellm/pull/13536)
+
+#### New Provider Support
+
+- **[Vercel AI Gateway](../../docs/providers/vercel_ai_gateway)**
+ - New provider support added - [PR #13144](https://github.com/BerriAI/litellm/pull/13144)
+- **[DataRobot](../../docs/providers/datarobot)**
+ - Added provider documentation - [PR #14038](https://github.com/BerriAI/litellm/pull/14038), [PR #14074](https://github.com/BerriAI/litellm/pull/14074)
+
+---
+
+## LLM API Endpoints
+
+#### Features
+
+- **[Images API](../../docs/image_generation)**
+ - Support for multiple images in OpenAI images/edits endpoint - [PR #13916](https://github.com/BerriAI/litellm/pull/13916)
+ - Allow using dynamic `api_key` for image generation requests - [PR #14007](https://github.com/BerriAI/litellm/pull/14007)
+- **[Responses API](../../docs/response_api)**
+ - Fixed `/responses` endpoint ignoring extra_headers in GitHub Copilot - [PR #13775](https://github.com/BerriAI/litellm/pull/13775)
+ - Added support for new web_search tool - [PR #14083](https://github.com/BerriAI/litellm/pull/14083)
+- **[Azure Passthrough](../../docs/providers/azure/azure)**
+ - Fixed Azure Passthrough request with streaming - [PR #13831](https://github.com/BerriAI/litellm/pull/13831)
+
+#### Bugs
+
+- **General**
+ - Fixed handling of None metadata in batch requests - [PR #13996](https://github.com/BerriAI/litellm/pull/13996)
+ - Fixed token_counter with special token input - [PR #13374](https://github.com/BerriAI/litellm/pull/13374)
+ - Removed incorrect web search support for azure/gpt-4.1 family - [PR #13566](https://github.com/BerriAI/litellm/pull/13566)
+
+---
+
+## [MCP Gateway](../../docs/mcp)
+
+#### Features
+
+- **SSE MCP Tools**
+ - Bug fix for adding SSE MCP tools - improved connection testing when adding MCPs - [PR #14048](https://github.com/BerriAI/litellm/pull/14048)
+
+[Read More](../../docs/mcp)
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **Team Management**
+ - Allow setting Team Member RPM/TPM limits when creating a team - [PR #13943](https://github.com/BerriAI/litellm/pull/13943)
+- **UI Improvements**
+ - Fixed Next.js Security Vulnerabilities in UI Dashboard - [PR #14084](https://github.com/BerriAI/litellm/pull/14084)
+ - Fixed collapsible navbar design - [PR #14075](https://github.com/BerriAI/litellm/pull/14075)
+
+#### Bugs
+
+- **Authentication**
+ - Fixed Virtual keys with llm_api type causing Internal Server Error for /anthropic/* and other LLM passthrough routes - [PR #14046](https://github.com/BerriAI/litellm/pull/14046)
+
+---
+
+## Logging / Guardrail Integrations
+
+#### Features
+
+- **[Langfuse OTEL](../../docs/proxy/logging#langfuse)**
+ - Allow using LANGFUSE_OTEL_HOST for configuring host - [PR #14013](https://github.com/BerriAI/litellm/pull/14013)
+- **[Braintrust](../../docs/proxy/logging#braintrust)**
+ - Added span name metadata feature - [PR #13573](https://github.com/BerriAI/litellm/pull/13573)
+ - Fixed tests to reference moved attributes in `braintrust_logging` module - [PR #13978](https://github.com/BerriAI/litellm/pull/13978)
+- **[OpenMeter](../../docs/proxy/logging#openmeter)**
+ - Set user from token user_id for OpenMeter integration - [PR #13152](https://github.com/BerriAI/litellm/pull/13152)
+
+#### New Guardrail Support
+
+- **[Noma Security](../../docs/proxy/guardrails)**
+ - Added Noma Security guardrail support - [PR #13572](https://github.com/BerriAI/litellm/pull/13572)
+- **[Pangea](../../docs/proxy/guardrails)**
+ - Updated Pangea Guardrail to support new AIDR endpoint - [PR #13160](https://github.com/BerriAI/litellm/pull/13160)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+#### Features
+
+- **Caching**
+ - Verify if cache entry has expired prior to serving it to client - [PR #13933](https://github.com/BerriAI/litellm/pull/13933)
+ - Fixed error saving latency as timedelta on Redis - [PR #14040](https://github.com/BerriAI/litellm/pull/14040)
+- **Router**
+ - Refactored router to choose weights by 'weight', 'rpm', 'tpm' in one loop for simple_shuffle - [PR #13562](https://github.com/BerriAI/litellm/pull/13562)
+- **Logging**
+ - Fixed LoggingWorker graceful shutdown to prevent CancelledError warnings - [PR #14050](https://github.com/BerriAI/litellm/pull/14050)
+ - Enhanced logging for containers to log on files both with usual format and json format - [PR #13394](https://github.com/BerriAI/litellm/pull/13394)
+
+#### Bugs
+
+- **Dependencies**
+ - Bumped `orjson` version to "3.11.2" - [PR #13969](https://github.com/BerriAI/litellm/pull/13969)
+
+---
+
+## General Proxy Improvements
+
+#### Features
+
+- **AWS**
+ - Add support for AWS assume_role with a session token - [PR #13919](https://github.com/BerriAI/litellm/pull/13919)
+- **OCI Provider**
+ - Added oci_key_file as an optional_parameter - [PR #14036](https://github.com/BerriAI/litellm/pull/14036)
+- **Configuration**
+ - Allow configuration to set threshold before request entry in spend log gets truncated - [PR #14042](https://github.com/BerriAI/litellm/pull/14042)
+ - Enhanced proxy_config configuration: add support for existing configmap in Helm charts - [PR #14041](https://github.com/BerriAI/litellm/pull/14041)
+- **Docker**
+ - Added back supervisor to non-root image - [PR #13922](https://github.com/BerriAI/litellm/pull/13922)
+
+
+---
+
+## New Contributors
+* @ArthurRenault made their first contribution in [PR #13922](https://github.com/BerriAI/litellm/pull/13922)
+* @stevenmanton made their first contribution in [PR #13919](https://github.com/BerriAI/litellm/pull/13919)
+* @uc4w6c made their first contribution in [PR #13914](https://github.com/BerriAI/litellm/pull/13914)
+* @nielsbosma made their first contribution in [PR #13573](https://github.com/BerriAI/litellm/pull/13573)
+* @Yuki-Imajuku made their first contribution in [PR #13567](https://github.com/BerriAI/litellm/pull/13567)
+* @codeflash-ai[bot] made their first contribution in [PR #13988](https://github.com/BerriAI/litellm/pull/13988)
+* @ColeFrench made their first contribution in [PR #13978](https://github.com/BerriAI/litellm/pull/13978)
+* @dttran-glo made their first contribution in [PR #13969](https://github.com/BerriAI/litellm/pull/13969)
+* @manascb1344 made their first contribution in [PR #13965](https://github.com/BerriAI/litellm/pull/13965)
+* @DorZion made their first contribution in [PR #13572](https://github.com/BerriAI/litellm/pull/13572)
+* @edwardsamuel made their first contribution in [PR #13536](https://github.com/BerriAI/litellm/pull/13536)
+* @blahgeek made their first contribution in [PR #13374](https://github.com/BerriAI/litellm/pull/13374)
+* @Deviad made their first contribution in [PR #13394](https://github.com/BerriAI/litellm/pull/13394)
+* @XSAM made their first contribution in [PR #13775](https://github.com/BerriAI/litellm/pull/13775)
+* @KRRT7 made their first contribution in [PR #14012](https://github.com/BerriAI/litellm/pull/14012)
+* @ikaadil made their first contribution in [PR #13991](https://github.com/BerriAI/litellm/pull/13991)
+* @timelfrink made their first contribution in [PR #13691](https://github.com/BerriAI/litellm/pull/13691)
+* @qidu made their first contribution in [PR #13562](https://github.com/BerriAI/litellm/pull/13562)
+* @nagyv made their first contribution in [PR #13243](https://github.com/BerriAI/litellm/pull/13243)
+* @xywei made their first contribution in [PR #12885](https://github.com/BerriAI/litellm/pull/12885)
+* @ericgtkb made their first contribution in [PR #12797](https://github.com/BerriAI/litellm/pull/12797)
+* @NoWall57 made their first contribution in [PR #13945](https://github.com/BerriAI/litellm/pull/13945)
+* @lmwang9527 made their first contribution in [PR #14050](https://github.com/BerriAI/litellm/pull/14050)
+* @WilsonSunBritten made their first contribution in [PR #14042](https://github.com/BerriAI/litellm/pull/14042)
+* @Const-antine made their first contribution in [PR #14041](https://github.com/BerriAI/litellm/pull/14041)
+* @dmvieira made their first contribution in [PR #14040](https://github.com/BerriAI/litellm/pull/14040)
+* @gotsysdba made their first contribution in [PR #14036](https://github.com/BerriAI/litellm/pull/14036)
+* @moshemorad made their first contribution in [PR #14005](https://github.com/BerriAI/litellm/pull/14005)
+* @joshualipman123 made their first contribution in [PR #13144](https://github.com/BerriAI/litellm/pull/13144)
+
+---
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.76.0-nightly...v1.76.1)**
diff --git a/docs/my-website/release_notes/v1.76.3-stable/index.md b/docs/my-website/release_notes/v1.76.3-stable/index.md
new file mode 100644
index 00000000000..6b40e4f5b35
--- /dev/null
+++ b/docs/my-website/release_notes/v1.76.3-stable/index.md
@@ -0,0 +1,289 @@
+---
+title: "v1.76.3-stable - Performance, Video Generation & CloudZero Integration"
+slug: "v1-76-3"
+date: 2025-09-06T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+:::warning
+
+This release has a known issue where startup is leading to Out of Memory errors when deploying on Kubernetes. We recommend waiting before upgrading to this version.
+
+:::
+
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.76.3
+```
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.76.3
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **Major Performance Improvements** +400 RPS when using correct amount of workers + CPU cores combination
+- **Video Generation Support** - Added Google AI Studio and Vertex AI Veo Video Generation through LiteLLM Pass through routes
+- **CloudZero Integration** - New cost tracking integration for exporting LiteLLM Usage and Spend data to CloudZero.
+
+## Major Changes
+- **Performance Optimization**: LiteLLM Proxy now achieves +400 RPS when using correct amount of CPU cores - [PR #14153](https://github.com/BerriAI/litellm/pull/14153), [PR #14242](https://github.com/BerriAI/litellm/pull/14242)
+
+ By default, LiteLLM will now use `num_workers = os.cpu_count()` to achieve optimal performance.
+
+ **Override Options:**
+
+ Set environment variable:
+ ```bash
+ DEFAULT_NUM_WORKERS_LITELLM_PROXY=1
+ ```
+
+ Or start LiteLLM Proxy with:
+ ```bash
+ litellm --num_workers 1
+ ```
+
+- **Security Fix**: Fixed memory_usage_in_mem_cache cache endpoint vulnerability - [PR #14229](https://github.com/BerriAI/litellm/pull/14229)
+
+---
+
+## Performance Improvements
+
+This release includes significant performance optimizations. On our internal benchmarks we saw 1 instance get +400 RPS when using correct amount of workers + CPU cores combination.
+
+- **+400 RPS Performance Boost** - LiteLLM Proxy now uses correct amount of CPU cores for optimal performance - [PR #14153](https://github.com/BerriAI/litellm/pull/14153)
+- **Default CPU Workers** - Changed DEFAULT_NUM_WORKERS_LITELLM_PROXY default to number of CPUs - [PR #14242](https://github.com/BerriAI/litellm/pull/14242)
+
+
+---
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
+| ----------- | -------------------------------------- | -------------- | ------------------- | -------------------- | -------- |
+| OpenRouter | `openrouter/openai/gpt-4.1` | 1M | $2.00 | $8.00 | Chat completions with vision |
+| OpenRouter | `openrouter/openai/gpt-4.1-mini` | 1M | $0.40 | $1.60 | Efficient chat completions |
+| OpenRouter | `openrouter/openai/gpt-4.1-nano` | 1M | $0.10 | $0.40 | Ultra-efficient chat |
+| Vertex AI | `vertex_ai/openai/gpt-oss-20b-maas` | 131K | $0.075 | $0.30 | Reasoning support |
+| Vertex AI | `vertex_ai/openai/gpt-oss-120b-maas` | 131K | $0.15 | $0.60 | Advanced reasoning |
+| Gemini | `gemini/veo-3.0-generate-preview` | 1K | - | $0.75/sec | Video generation |
+| Gemini | `gemini/veo-3.0-fast-generate-preview` | 1K | - | $0.40/sec | Fast video generation |
+| Gemini | `gemini/veo-2.0-generate-001` | 1K | - | $0.35/sec | Video generation |
+| Volcengine | `doubao-embedding-large` | 4K | Free | Free | 2048-dim embeddings |
+| Together AI | `together_ai/deepseek-ai/DeepSeek-V3.1` | 128K | $0.60 | $1.70 | Reasoning support |
+
+#### Features
+
+- **[Google Gemini](../../docs/providers/gemini)**
+ - Added 'thoughtSignature' support via 'thinking_blocks' - [PR #14122](https://github.com/BerriAI/litellm/pull/14122)
+ - Added support for reasoning_effort='minimal' for Gemini models - [PR #14262](https://github.com/BerriAI/litellm/pull/14262)
+- **[OpenRouter](../../docs/providers/openrouter)**
+ - Added GPT-4.1 model family - [PR #14101](https://github.com/BerriAI/litellm/pull/14101)
+- **[Groq](../../docs/providers/groq)**
+ - Added support for reasoning_effort parameter - [PR #14207](https://github.com/BerriAI/litellm/pull/14207)
+- **[X.AI](../../docs/providers/xai)**
+ - Fixed XAI cost calculation - [PR #14127](https://github.com/BerriAI/litellm/pull/14127)
+- **[Vertex AI](../../docs/providers/vertex)**
+ - Added support for GPT-OSS models on Vertex AI - [PR #14184](https://github.com/BerriAI/litellm/pull/14184)
+ - Added additionalProperties to Vertex AI Schema definition - [PR #14252](https://github.com/BerriAI/litellm/pull/14252)
+- **[VLLM](../../docs/providers/vllm)**
+ - Handle output parsing responses API output - [PR #14121](https://github.com/BerriAI/litellm/pull/14121)
+- **[Ollama](../../docs/providers/ollama)**
+ - Added unified 'thinking' param support via `reasoning_content` - [PR #14121](https://github.com/BerriAI/litellm/pull/14121)
+- **[Anthropic](../../docs/providers/anthropic)**
+ - Added supported text field to anthropic citation response - [PR #14126](https://github.com/BerriAI/litellm/pull/14126)
+- **[OCI Provider](../../docs/providers/oci)**
+ - Handle assistant messages with both content and tool_calls - [PR #14171](https://github.com/BerriAI/litellm/pull/14171)
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Fixed structure output - [PR #14130](https://github.com/BerriAI/litellm/pull/14130)
+ - Added initial support for Bedrock Batches API - [PR #14190](https://github.com/BerriAI/litellm/pull/14190)
+- **[Databricks](../../docs/providers/databricks)**
+ - Added support for anthropic citation API in Databricks - [PR #14077](https://github.com/BerriAI/litellm/pull/14077)
+
+### Bug Fixes
+- **[Google Gemini (Google AI Studio + Vertex AI)](../../docs/providers/gemini)**
+ - Fixed Gemini 2.5 Pro schema validation with OpenAI-style type arrays in tools - [PR #14154](https://github.com/BerriAI/litellm/pull/14154)
+ - Fixed Gemini Tool Calling empty enum property - [PR #14155](https://github.com/BerriAI/litellm/pull/14155)
+
+#### New Provider Support
+
+- **[Volcengine](../../docs/providers/volcengine)**
+ - Added Volcengine embedding module with handler and transformation logic - [PR #14028](https://github.com/BerriAI/litellm/pull/14028)
+
+---
+
+## LLM API Endpoints
+
+#### Features
+
+- **[Images API](../../docs/image_generation)**
+ - Added pass through image generation and image editing on OpenAI - [PR #14292](https://github.com/BerriAI/litellm/pull/14292)
+ - Support extra_body parameter for image generation - [PR #14211](https://github.com/BerriAI/litellm/pull/14211)
+- **[Responses API](../../docs/response_api)**
+ - Fixed response API for reasoning item in input for litellm proxy - [PR #14200](https://github.com/BerriAI/litellm/pull/14200)
+ - Added structured output for SDK - [PR #14206](https://github.com/BerriAI/litellm/pull/14206)
+- **[Bedrock Passthrough](../../docs/pass_through/bedrock)**
+ - Support AWS_BEDROCK_RUNTIME_ENDPOINT on bedrock passthrough - [PR #14156](https://github.com/BerriAI/litellm/pull/14156)
+- **[Google AI Studio Passthrough](../../docs/pass_through/google_ai_studio)**
+ - Allow using Veo Video Generation through LiteLLM Pass through routes - [PR #14228](https://github.com/BerriAI/litellm/pull/14228)
+- **General**
+ - Added support for safety_identifier parameter in chat.completions.create - [PR #14174](https://github.com/BerriAI/litellm/pull/14174)
+ - Fixed misclassified 500 error on invalid image_url in /chat/completions request - [PR #14149](https://github.com/BerriAI/litellm/pull/14149)
+ - Fixed token count error for Gemini CLI - [PR #14133](https://github.com/BerriAI/litellm/pull/14133)
+
+#### Bugs
+
+- **General**
+ - Remove "/" or ":" from model name when being used as h11 header name - [PR #14191](https://github.com/BerriAI/litellm/pull/14191)
+ - Bug fix for openai.gpt-oss when using reasoning_effort parameter - [PR #14300](https://github.com/BerriAI/litellm/pull/14300)
+
+---
+
+## Spend Tracking, Budgets and Rate Limiting
+
+### Features
+ - Added header support for spend_logs_metadata - [PR #14186](https://github.com/BerriAI/litellm/pull/14186)
+ - Litellm passthrough cost tracking for chat completion - [PR #14256](https://github.com/BerriAI/litellm/pull/14256)
+
+### Bug Fixes
+ - Fixed TPM Rate Limit Bug - [PR #14237](https://github.com/BerriAI/litellm/pull/14237)
+ - Fixed Key Budget not resets at expectable times - [PR #14241](https://github.com/BerriAI/litellm/pull/14241)
+
+
+
+## Management Endpoints / UI
+
+#### Features
+
+- **UI Improvements**
+ - Logs page screen size fixed - [PR #14135](https://github.com/BerriAI/litellm/pull/14135)
+ - Create Organization Tooltip added on Success - [PR #14132](https://github.com/BerriAI/litellm/pull/14132)
+ - Back to Keys should say Back to Logs - [PR #14134](https://github.com/BerriAI/litellm/pull/14134)
+ - Add client side pagination on All Models table - [PR #14136](https://github.com/BerriAI/litellm/pull/14136)
+ - Model Filters UI improvement - [PR #14131](https://github.com/BerriAI/litellm/pull/14131)
+ - Remove table filter on user info page - [PR #14169](https://github.com/BerriAI/litellm/pull/14169)
+ - Team name badge added on the User Details - [PR #14003](https://github.com/BerriAI/litellm/pull/14003)
+ - Fix: Log page parameter passing error - [PR #14193](https://github.com/BerriAI/litellm/pull/14193)
+- **Authentication & Authorization**
+ - Support for ES256/ES384/ES512 and EdDSA JWT verification - [PR #14118](https://github.com/BerriAI/litellm/pull/14118)
+ - Ensure `team_id` is a required field for generating service account keys - [PR #14270](https://github.com/BerriAI/litellm/pull/14270)
+
+#### Bugs
+
+- **General**
+ - Validate store model in db setting - [PR #14269](https://github.com/BerriAI/litellm/pull/14269)
+
+---
+
+## Logging / Guardrail Integrations
+
+#### Features
+
+- **[Datadog](../../docs/proxy/logging#datadog)**
+ - Ensure `apm_id` is set on DD LLM Observability traces - [PR #14272](https://github.com/BerriAI/litellm/pull/14272)
+- **[Braintrust](../../docs/proxy/logging#braintrust)**
+ - Fix logging when OTEL is enabled - [PR #14122](https://github.com/BerriAI/litellm/pull/14122)
+- **[OTEL](../../docs/proxy/logging#otel)**
+ - Optional Metrics and Logs following semantic conventions - [PR #14179](https://github.com/BerriAI/litellm/pull/14179)
+- **[Slack Alerting](../../docs/proxy/alerting)**
+ - Added alert type to alert message to slack for easier handling - [PR #14176](https://github.com/BerriAI/litellm/pull/14176)
+
+#### Guardrails
+ - Added guardrail to the Anthropic API endpoint - [PR #14107](https://github.com/BerriAI/litellm/pull/14107)
+
+#### New Integration
+
+- **[CloudZero](../../docs/proxy/cost_tracking)**
+ - LiteLLM x CloudZero Integration for Cost Tracking - [PR #14296](https://github.com/BerriAI/litellm/pull/14296)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+#### Features
+
+- **Performance**
+ - LiteLLM Proxy: +400 RPS when using correct amount of CPU cores - [PR #14153](https://github.com/BerriAI/litellm/pull/14153)
+ - Allow using `x-litellm-stream-timeout` header for stream timeout in requests - [PR #14147](https://github.com/BerriAI/litellm/pull/14147)
+ - Change DEFAULT_NUM_WORKERS_LITELLM_PROXY default to number CPUs - [PR #14242](https://github.com/BerriAI/litellm/pull/14242)
+- **Monitoring**
+ - Added Prometheus missing metrics - [PR #14139](https://github.com/BerriAI/litellm/pull/14139)
+- **Timeout**
+ - **Stream Timeout Control** - Allow using `x-litellm-stream-timeout` header for stream timeout in requests - [PR #14147](https://github.com/BerriAI/litellm/pull/14147)
+- **Routing**
+ - Fixed x-litellm-tags not routing with Responses API - [PR #14289](https://github.com/BerriAI/litellm/pull/14289)
+
+#### Bugs
+
+- **Security**
+ - Fixed memory_usage_in_mem_cache cache endpoint vulnerability - [PR #14229](https://github.com/BerriAI/litellm/pull/14229)
+
+---
+
+## General Proxy Improvements
+
+#### Features
+
+- **SCIM Support**
+ - Added better SCIM debugging - [PR #14221](https://github.com/BerriAI/litellm/pull/14221)
+ - Bug fixes for handling SCIM Group Memberships - [PR #14226](https://github.com/BerriAI/litellm/pull/14226)
+- **Kubernetes**
+ - Added optional PodDisruptionBudget for litellm proxy - [PR #14093](https://github.com/BerriAI/litellm/pull/14093)
+- **Error Handling**
+ - Add model to azure error message - [PR #14294](https://github.com/BerriAI/litellm/pull/14294)
+
+---
+
+## New Contributors
+* @iabhi4 made their first contribution in [PR #14093](https://github.com/BerriAI/litellm/pull/14093)
+* @zainhas made their first contribution in [PR #14087](https://github.com/BerriAI/litellm/pull/14087)
+* @LifeDJIK made their first contribution in [PR #14146](https://github.com/BerriAI/litellm/pull/14146)
+* @retanoj made their first contribution in [PR #14133](https://github.com/BerriAI/litellm/pull/14133)
+* @zhxlp made their first contribution in [PR #14193](https://github.com/BerriAI/litellm/pull/14193)
+* @kayoch1n made their first contribution in [PR #14191](https://github.com/BerriAI/litellm/pull/14191)
+* @kutsushitaneko made their first contribution in [PR #14171](https://github.com/BerriAI/litellm/pull/14171)
+* @mjmendo made their first contribution in [PR #14176](https://github.com/BerriAI/litellm/pull/14176)
+* @HarshavardhanK made their first contribution in [PR #14213](https://github.com/BerriAI/litellm/pull/14213)
+* @eycjur made their first contribution in [PR #14207](https://github.com/BerriAI/litellm/pull/14207)
+* @22mSqRi made their first contribution in [PR #14241](https://github.com/BerriAI/litellm/pull/14241)
+* @onlylhf made their first contribution in [PR #14028](https://github.com/BerriAI/litellm/pull/14028)
+* @btpemercier made their first contribution in [PR #11319](https://github.com/BerriAI/litellm/pull/11319)
+* @tremlin made their first contribution in [PR #14287](https://github.com/BerriAI/litellm/pull/14287)
+* @TobiMayr made their first contribution in [PR #14262](https://github.com/BerriAI/litellm/pull/14262)
+* @Eitan1112 made their first contribution in [PR #14252](https://github.com/BerriAI/litellm/pull/14252)
+
+---
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.76.1-nightly...v1.76.3-nightly)**
diff --git a/docs/my-website/release_notes/v1.77.2-stable/index.md b/docs/my-website/release_notes/v1.77.2-stable/index.md
new file mode 100644
index 00000000000..fdd80693d05
--- /dev/null
+++ b/docs/my-website/release_notes/v1.77.2-stable/index.md
@@ -0,0 +1,156 @@
+---
+title: "v1.77.2-stable - Bedrock Batches API"
+slug: "v1-77-2"
+date: 2025-09-13T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaffer
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:main-v1.77.2-stable
+```
+
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.77.2.post1
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **Bedrock Batches API** - Support for creating Batch Inference Jobs on Bedrock using LiteLLM's unified batch API (OpenAI compatible)
+- **Qwen API Tiered Pricing** - Cost tracking support for Dashscope (Qwen) models with multiple pricing tiers
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Pricing ($/1M tokens) | Features |
+| ----------- | ------------------------------- | -------------- | --------------------- | -------- |
+| DeepInfra | `deepinfra/deepseek-ai/DeepSeek-R1` | 164K | **Input:** $0.70
**Output:** $2.40 | Chat completions, tool calling |
+| Heroku | `heroku/claude-4-sonnet` | 8K | Contact provider for pricing | Function calling, tool choice |
+| Heroku | `heroku/claude-3-7-sonnet` | 8K | Contact provider for pricing | Function calling, tool choice |
+| Heroku | `heroku/claude-3-5-sonnet-latest` | 8K | Contact provider for pricing | Function calling, tool choice |
+| Heroku | `heroku/claude-3-5-haiku` | 4K | Contact provider for pricing | Function calling, tool choice |
+| Dashscope | `dashscope/qwen-plus-latest` | 1M | **Tiered Pricing:**
• 0-256K tokens: $0.40 / $1.20
• 256K-1M tokens: $1.20 / $3.60 | Function calling, reasoning |
+| Dashscope | `dashscope/qwen3-max-preview` | 262K | **Tiered Pricing:**
• 0-32K tokens: $1.20 / $6.00
• 32K-128K tokens: $2.40 / $12.00
• 128K-252K tokens: $3.00 / $15.00 | Function calling, reasoning |
+| Dashscope | `dashscope/qwen-flash` | 1M | **Tiered Pricing:**
• 0-256K tokens: $0.05 / $0.40
• 256K-1M tokens: $0.25 / $2.00 | Function calling, reasoning |
+| Dashscope | `dashscope/qwen3-coder-plus` | 1M | **Tiered Pricing:**
• 0-32K tokens: $1.00 / $5.00
• 32K-128K tokens: $1.80 / $9.00
• 128K-256K tokens: $3.00 / $15.00
• 256K-1M tokens: $6.00 / $60.00 | Function calling, reasoning, caching |
+| Dashscope | `dashscope/qwen3-coder-flash` | 1M | **Tiered Pricing:**
• 0-32K tokens: $0.30 / $1.50
• 32K-128K tokens: $0.50 / $2.50
• 128K-256K tokens: $0.80 / $4.00
• 256K-1M tokens: $1.60 / $9.60 | Function calling, reasoning, caching |
+
+---
+
+#### Features
+
+- **[Bedrock](../../docs/providers/bedrock_batches)**
+ - Bedrock Batches API - batch processing support with file upload and request transformation - [PR #14518](https://github.com/BerriAI/litellm/pull/14518), [PR #14522](https://github.com/BerriAI/litellm/pull/14522)
+- **[VLLM](../../docs/providers/vllm)**
+ - Added transcription endpoint support - [PR #14523](https://github.com/BerriAI/litellm/pull/14523)
+- **[Ollama](../../docs/providers/ollama)**
+ - `ollama_chat/` - images, thinking, and content as list handling - [PR #14523](https://github.com/BerriAI/litellm/pull/14523)
+- **General**
+ - New debug flag for detailed request/response logging [PR #14482](https://github.com/BerriAI/litellm/pull/14482)
+
+#### Bug Fixes
+
+- **[Azure OpenAI](../../docs/providers/azure)**
+ - Fixed extra_body injection causing payload rejection in image generation - [PR #14475](https://github.com/BerriAI/litellm/pull/14475)
+- **[LM Studio](../../docs/providers/lm-studio)**
+ - Resolved illegal Bearer header value issue - [PR #14512](https://github.com/BerriAI/litellm/pull/14512)
+
+---
+
+## LLM API Endpoints
+
+#### Bug Fixes
+
+- **[/messages](../../docs/anthropic_unified)**
+ - Don't send content block after message w/ finish reason + usage block - [PR #14477](https://github.com/BerriAI/litellm/pull/14477)
+- **[/generateContent](../../docs/generateContent)**
+ - Gemini CLI Integration - Fixed token count errors - [PR #14451](https://github.com/BerriAI/litellm/pull/14451), [PR #14417](https://github.com/BerriAI/litellm/pull/14417)
+
+---
+
+## Spend Tracking, Budgets and Rate Limiting
+
+#### Features
+
+- **[Qwen API Tiered Pricing](../../docs/providers/dashscope)** - Added comprehensive tiered cost tracking for Dashscope/Qwen models - [PR #14471](https://github.com/BerriAI/litellm/pull/14471), [PR #14479](https://github.com/BerriAI/litellm/pull/14479)
+
+#### Bug Fixes
+
+- **Provider Budgets** - Fixed provider budget calculations - [PR #14459](https://github.com/BerriAI/litellm/pull/14459)
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **User Headers Mapping** - New X-LiteLLM Users mapping feature for enhanced user tracking - [PR #14485](https://github.com/BerriAI/litellm/pull/14485)
+- **Key Unblocking** - Support for hashed tokens in `/key/unblock` endpoint - [PR #14477](https://github.com/BerriAI/litellm/pull/14477)
+- **Model Group Header Forwarding** - Enhanced wildcard model support with documentation - [PR #14528](https://github.com/BerriAI/litellm/pull/14528)
+
+#### Bug Fixes
+
+- **Log Tab Key Alias** - Fixed filtering inaccuracies for failed logs - [PR #14469](https://github.com/BerriAI/litellm/pull/14469), [PR #14529](https://github.com/BerriAI/litellm/pull/14529)
+
+---
+
+## Logging / Guardrail Integrations
+
+#### Features
+
+- **Noma Integration** - Added non-blocking monitor mode with anonymize input support - [PR #14401](https://github.com/BerriAI/litellm/pull/14401)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+#### Performance
+- Removed dynamic creation of static values - [PR #14538](https://github.com/BerriAI/litellm/pull/14538)
+- Using `_PROXY_MaxParallelRequestsHandler_v3` by default for optimal throughput - [PR #14450](https://github.com/BerriAI/litellm/pull/14450)
+- Improved execution context propagation into logging tasks - [PR #14455](https://github.com/BerriAI/litellm/pull/14455)
+
+---
+
+
+
+## New Contributors
+* @Sameerlite made their first contribution in [PR #14460](https://github.com/BerriAI/litellm/pull/14460)
+* @holzman made their first contribution in [PR #14459](https://github.com/BerriAI/litellm/pull/14459)
+* @sashank5644 made their first contribution in [PR #14469](https://github.com/BerriAI/litellm/pull/14469)
+* @TomAlon made their first contribution in [PR #14401](https://github.com/BerriAI/litellm/pull/14401)
+* @AlexsanderHamir made their first contribution in [PR #14538](https://github.com/BerriAI/litellm/pull/14538)
+
+---
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.77.1.dev.2...v1.77.2.dev)**
diff --git a/docs/my-website/release_notes/v1.77.3-stable/index.md b/docs/my-website/release_notes/v1.77.3-stable/index.md
new file mode 100644
index 00000000000..c7c17e5baee
--- /dev/null
+++ b/docs/my-website/release_notes/v1.77.3-stable/index.md
@@ -0,0 +1,274 @@
+---
+title: "v1.77.3-stable - Priority Based Rate Limiting"
+slug: "v1-77-3"
+date: 2025-09-21T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaff
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.77.3-stable
+```
+
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.77.3
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **+550 RPS Performance Improvements** - Optimizations in request handling and object initialization.
+- **Priority Quota Reservation** - Proxy admins can now reserve TPM/RPM capacity for specific keys.
+
+## Priority Quota Reservation
+
+This release adds support for priority quota reservation. This allows Proxy Admins to reserve specific percentages of model capacity for different use cases.
+
+This is great for use cases where you want to ensure your realtime use cases must always get priority responses and background development jobs can take longer.
+
+
+
+
+
+This release adds support for priority quota reservation. This allows **Proxy Admins** to reserve TPM/RPM capacity for keys based on metadata priority levels, ensuring critical production workloads get guaranteed access regardless of development traffic volume.
+
+Get started [here](../../docs/proxy/dynamic_rate_limit#priority-quota-reservation)
+
+## +550 RPS Performance Improvements
+
+
+
+
+
+This release delivers significant RPS improvements through targeted optimizations.
+
+We've achieved a +500 RPS boost by fixing cache type inconsistencies that were causing frequent cache misses, plus an additional +50 RPS by removing unnecessary coroutine checks from the hot path.
+
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
+| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
+| SambaNova | `sambanova/deepseek-v3.1` | 128K | $0.90 | $0.90 | Chat completions |
+| SambaNova | `sambanova/gpt-oss-120b` | 128K | $0.72 | $0.72 | Chat completions |
+| OVHCloud | Various models | Varies | Contact provider | Contact provider | Chat completions |
+| CompactifAI | Various models | Varies | Contact provider | Contact provider | Chat completions |
+| TwelveLabs | `twelvelabs/marengo-embed-2.7` | 32K | $0.12 | $0.00 | Embeddings |
+
+#### Features
+
+- **[OVHCloud AI Endpoints](../../docs/providers/ovhcloud)**
+ - New provider support with comprehensive model catalog - [PR #14494](https://github.com/BerriAI/litellm/pull/14494)
+- **[CompactifAI](../../docs/providers/compactifai)**
+ - New provider integration - [PR #14532](https://github.com/BerriAI/litellm/pull/14532)
+- **[SambaNova](../../docs/providers/sambanova)**
+ - Added DeepSeek v3.1 and GPT-OSS-120B models - [PR #14500](https://github.com/BerriAI/litellm/pull/14500)
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Cross-region inference profile cost calculation - [PR #14566](https://github.com/BerriAI/litellm/pull/14566)
+ - AWS external ID parameter support for authentication - [PR #14582](https://github.com/BerriAI/litellm/pull/14582)
+ - CountTokens API implementation - [PR #14557](https://github.com/BerriAI/litellm/pull/14557)
+ - Titan V2 encoding_format parameter support - [PR #14687](https://github.com/BerriAI/litellm/pull/14687)
+ - Nova Canvas image generation inference profiles - [PR #14578](https://github.com/BerriAI/litellm/pull/14578)
+ - Bedrock Batches API - batch processing support with file upload and request transformation - [PR #14618](https://github.com/BerriAI/litellm/pull/14618)
+ - Bedrock Twelve Labs embedding provider support - [PR #14697](https://github.com/BerriAI/litellm/pull/14697)
+- **[Vertex AI](../../docs/providers/vertex)**
+ - Gemini labels field provider-aware filtering - [PR #14563](https://github.com/BerriAI/litellm/pull/14563)
+ - Gemini Batch API support - [PR #14733](https://github.com/BerriAI/litellm/pull/14733)
+- **[Volcengine](../../docs/providers/volcengine)**
+ - Fixed thinking parameters when disabled - [PR #14569](https://github.com/BerriAI/litellm/pull/14569)
+- **[Cohere](../../docs/providers/cohere)**
+ - Handle Generate API deprecation, default to chat endpoints - [PR #14676](https://github.com/BerriAI/litellm/pull/14676)
+- **[TwelveLabs](../../docs/providers/twelvelabs)**
+ - Added Marengo Embed 2.7 embedding support - [PR #14674](https://github.com/BerriAI/litellm/pull/14674)
+
+### Bug Fixes
+
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Empty arguments handling in tool call invocation - [PR #14583](https://github.com/BerriAI/litellm/pull/14583)
+- **[Vertex AI](../../docs/providers/vertex)**
+ - Avoid deepcopy crash with non-pickleables in Gemini/Vertex - [PR #14418](https://github.com/BerriAI/litellm/pull/14418)
+- **[XAI](../../docs/providers/xai)**
+ - Fix unsupported stop parameter for grok-code models - [PR #14565](https://github.com/BerriAI/litellm/pull/14565)
+- **[Gemini](../../docs/providers/gemini)**
+ - Updated error message for Gemini API - [PR #14589](https://github.com/BerriAI/litellm/pull/14589)
+ - Fixed 2.5 Flash Image Preview model routing - [PR #14715](https://github.com/BerriAI/litellm/pull/14715)
+ - API key passing for token counting endpoints - [PR #14744](https://github.com/BerriAI/litellm/pull/14744)
+
+#### New Provider Support
+
+- **[OVHCloud AI Endpoints](../../docs/providers/ovhcloud)**
+ - Complete provider integration with model catalog and authentication - [PR #14494](https://github.com/BerriAI/litellm/pull/14494)
+- **[CompactifAI](../../docs/providers/compactifai)**
+ - New provider support with documentation - [PR #14532](https://github.com/BerriAI/litellm/pull/14532)
+
+---
+
+## LLM API Endpoints
+
+#### Features
+
+- **[/responses](../../docs/response_api)**
+ - Added cancel endpoint support for non-admin users - [PR #14594](https://github.com/BerriAI/litellm/pull/14594)
+ - Improved response session handling and cold storage configuration with s3 - [PR #14534](https://github.com/BerriAI/litellm/pull/14534)
+ - Added OpenAI & Azure /responses/cancel endpoint support - [PR #14561](https://github.com/BerriAI/litellm/pull/14561)
+- **General**
+ - Enhanced rate limit error messages with details - [PR #14736](https://github.com/BerriAI/litellm/pull/14736)
+ - Middle-truncation for spend log payloads - [PR #14637](https://github.com/BerriAI/litellm/pull/14637)
+
+#### Bugs
+
+- **[/chat/completions](../../docs/completion/input)**
+ - Fixed completion chat ID handling - [PR #14548](https://github.com/BerriAI/litellm/pull/14548)
+ - Prevent AttributeError for _get_tags_from_request_kwargs - [PR #14735](https://github.com/BerriAI/litellm/pull/14735)
+- **[/responses](../../docs/response_api)**
+ - Fixed cost calculation - [PR #14675](https://github.com/BerriAI/litellm/pull/14675)
+- **General**
+ - Rate limiter AttributeError fix - [PR #14609](https://github.com/BerriAI/litellm/pull/14609)
+
+---
+
+## Spend Tracking, Budgets and Rate Limiting
+
+- **Responses API Cost Calculation** fix - [PR #14675](https://github.com/BerriAI/litellm/pull/14675)
+- **Anthropic Cache Token Pricing** - Separate 1-hour vs 5-minute cache creation costs - [PR #14620](https://github.com/BerriAI/litellm/pull/14620), [PR #14652](https://github.com/BerriAI/litellm/pull/14652)
+- **Indochina Time Timezone** support for budget resets - [PR #14666](https://github.com/BerriAI/litellm/pull/14666)
+- **Soft Budget Alert Cache Issues** - Resolved soft budget alert cache issues - [PR #14491](https://github.com/BerriAI/litellm/pull/14491)
+- **Dynamic Rate Limiter v3** - Priority routing improvements - [PR #14734](https://github.com/BerriAI/litellm/pull/14734)
+- **Enhanced Rate Limit Errors** - More detailed error messages - [PR #14736](https://github.com/BerriAI/litellm/pull/14736)
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **Team Member Service Account Keys** - Allow team members to view keys they create - [PR #14619](https://github.com/BerriAI/litellm/pull/14619)
+- **Default Budget for JWT Teams** - Auto-assign budgets to generated teams - [PR #14514](https://github.com/BerriAI/litellm/pull/14514)
+- **SSO Access Control Groups** - Enhanced token info endpoint integration - [PR #14738](https://github.com/BerriAI/litellm/pull/14738)
+- **Health Test Connect Protection** - Restrict access based on model creation permissions - [PR #14650](https://github.com/BerriAI/litellm/pull/14650)
+- **Amazon Bedrock Guardrail Info View** - Enhanced logging visualization - [PR #14696](https://github.com/BerriAI/litellm/pull/14696)
+
+#### Bug Fixes
+
+- **SCIM v2** - Fix group PUSH and PUT operations for non-existent members - [PR #14581](https://github.com/BerriAI/litellm/pull/14581)
+- **Guardrail View/Edit/Delete** behavior fixes - [PR #14622](https://github.com/BerriAI/litellm/pull/14622)
+- **In-Memory Guardrail** update failures - [PR #14653](https://github.com/BerriAI/litellm/pull/14653)
+
+---
+
+## Logging / Guardrail Integrations
+
+#### Features
+
+- **[DataDog](../../docs/proxy/logging#datadog)**
+ - Enhanced spend tracking metrics - [PR #14555](https://github.com/BerriAI/litellm/pull/14555)
+ - Stream support with is_streamed_request parameter - [PR #14673](https://github.com/BerriAI/litellm/pull/14673)
+ - Fixed tool calls metadata passing - [PR #14531](https://github.com/BerriAI/litellm/pull/14531)
+- **[Langfuse](../../docs/proxy/logging#langfuse)**
+ - Added logging support for Responses API - [PR #14597](https://github.com/BerriAI/litellm/pull/14597)
+- **[Langsmith](../../docs/proxy/logging#langsmith)**
+ - Langsmith Sampling Rate - Key/Team-level tracing configuration - [PR #14740](https://github.com/BerriAI/litellm/pull/14740)
+- **[Prometheus](../../docs/proxy/logging#prometheus)**
+ - Multi-worker support improvements - [PR #14530](https://github.com/BerriAI/litellm/pull/14530)
+ - User email labels in monitoring - [PR #14520](https://github.com/BerriAI/litellm/pull/14520)
+- **[Opik](../../docs/proxy/logging#opik)**
+ - Fixed timezone issue - [PR #14708](https://github.com/BerriAI/litellm/pull/14708)
+
+### Bug Fixes
+
+- **[S3](../../docs/proxy/logging#s3-buckets)**
+ - Fixed 404 error when using s3_endpoint_url - [PR #14559](https://github.com/BerriAI/litellm/pull/14559)
+
+#### Guardrails
+
+- **Tool Permission Guardrail** - Fine-grained tool access control - [PR #14519](https://github.com/BerriAI/litellm/pull/14519)
+- **Bedrock Guardrails** - Selective guarding support with runtime endpoint configuration - [PR #14575](https://github.com/BerriAI/litellm/pull/14575), [PR #14650](https://github.com/BerriAI/litellm/pull/14650)
+- **Default Last Message** in guardrails - [PR #14640](https://github.com/BerriAI/litellm/pull/14640)
+- **AWS exceptions handling despite 200 response** - [PR #14658](https://github.com/BerriAI/litellm/pull/14658)
+#### New Integration
+
+- **[PostHog](../../docs/observability/posthog)** - Complete observability integration for LiteLLM usage tracking and analytics - [PR #14610](https://github.com/BerriAI/litellm/pull/14610)
+
+---
+
+
+## MCP Gateway
+
+- **MCP Server Alias Parsing** - Multi-part URL path support - [PR #14558](https://github.com/BerriAI/litellm/pull/14558)
+- **MCP Filter Recomputation** - After server deletion - [PR #14542](https://github.com/BerriAI/litellm/pull/14542)
+- **MCP Gateway Tools List** improvements - [PR #14695](https://github.com/BerriAI/litellm/pull/14695)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+- **+500 RPS Performance Boost** when sending the `user` field - [PR #14616](https://github.com/BerriAI/litellm/pull/14616)
+- **+50 RPS** by removing iscoroutine from hot path - [PR #14649](https://github.com/BerriAI/litellm/pull/14649)
+- **7% reduction** in __init__ overhead - [PR #14689](https://github.com/BerriAI/litellm/pull/14689)
+- **Generic Object Pool** implementation for better resource management - [PR #14702](https://github.com/BerriAI/litellm/pull/14702)
+
+---
+
+## General Proxy Improvements
+
+- **Middle-Truncation** for spend log payloads - [PR #14637](https://github.com/BerriAI/litellm/pull/14637)
+
+#### Security
+
+- **Security Update** - Bump aiohttp==3.12.14, fix CVE-2025-53643 - [PR #14638](https://github.com/BerriAI/litellm/pull/14638)
+
+---
+
+## New Contributors
+
+* @luisfucros made their first contribution in [PR #14500](https://github.com/BerriAI/litellm/pull/14500)
+* @hanakannzashi made their first contribution in [PR #14548](https://github.com/BerriAI/litellm/pull/14548)
+* @eliasto made their first contribution in [PR #14494](https://github.com/BerriAI/litellm/pull/14494)
+* @Rasmusafj made their first contribution in [PR #14491](https://github.com/BerriAI/litellm/pull/14491)
+* @LingXuanYin made their first contribution in [PR #14569](https://github.com/BerriAI/litellm/pull/14569)
+* @ronaldpereira made their first contribution in [PR #14613](https://github.com/BerriAI/litellm/pull/14613)
+* @hula-la made their first contribution in [PR #14534](https://github.com/BerriAI/litellm/pull/14534)
+* @carlos-marchal-ph made their first contribution in [PR #14610](https://github.com/BerriAI/litellm/pull/14610)
+* @akraines made their first contribution in [PR #14637](https://github.com/BerriAI/litellm/pull/14637)
+* @mrFranklin made their first contribution in [PR #14708](https://github.com/BerriAI/litellm/pull/14708)
+* @tcx4c70 made their first contribution in [PR #14675](https://github.com/BerriAI/litellm/pull/14675)
+* @michaeltansg made their first contribution in [PR #14666](https://github.com/BerriAI/litellm/pull/14666)
+* @tosi29 made their first contribution in [PR #14725](https://github.com/BerriAI/litellm/pull/14725)
+* @gmdfalk made their first contribution in [PR #14735](https://github.com/BerriAI/litellm/pull/14735)
+* @FelipeRodriguesGare made their first contribution in [PR #14733](https://github.com/BerriAI/litellm/pull/14733)
+* @mritunjaysharma394 made their first contribution in [PR #14678](https://github.com/BerriAI/litellm/pull/14678)
+
+---
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.77.2.rc.1...v1.77.3.rc.1)**
diff --git a/docs/my-website/release_notes/v1.77.5-stable/index.md b/docs/my-website/release_notes/v1.77.5-stable/index.md
new file mode 100644
index 00000000000..1b06018d8a8
--- /dev/null
+++ b/docs/my-website/release_notes/v1.77.5-stable/index.md
@@ -0,0 +1,328 @@
+---
+title: "v1.77.5-stable - MCP OAuth 2.0 Support"
+slug: "v1-77-5"
+date: 2025-09-29T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaff
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+ - name: Alexsander Hamir
+ title: Backend Performance Engineer
+ url: https://www.linkedin.com/in/alexsander-baptista/
+ image_url: https://media.licdn.com/dms/image/v2/D5603AQGXnziu4kqNCQ/profile-displayphoto-crop_800_800/B56ZkxEcuOKEAI-/0/1757464874550?e=1762387200&v=beta&t=9SNXLsWhx8OnYPAMQ9fqAr02oevDYEAL2vMYg2f9ieg
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.77.5-stable
+```
+
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.77.5
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **MCP OAuth 2.0 Support** - Enhanced authentication for Model Context Protocol integrations
+- **Scheduled Key Rotations** - Automated key rotation capabilities for enhanced security
+- **New Gemini 2.5 Flash & Flash-lite Models** - Latest September 2025 preview models with improved pricing and features
+- **Performance Improvements** - 54% RPS improvement
+
+---
+
+### Performance Improvements - 54% RPS Improvement
+
+
+
+
+
+This release brings a 54% RPS improvement (1,040 → 1,602 RPS, aggregated) per instance.
+
+The improvement comes from fixing O(n²) inefficiencies in the LiteLLM Router, primarily caused by repeated use of `in` statements inside loops over large arrays.
+
+Tests were run with a database-only setup (no cache hits).
+
+#### Test Setup
+
+All benchmarks were executed using Locust with 1,000 concurrent users and a ramp-up of 500. The environment was configured to stress the routing layer and eliminate caching as a variable.
+
+**System Specs**
+
+- **CPU:** 8 vCPUs
+- **Memory:** 32 GB RAM
+
+**Configuration (config.yaml)**
+
+View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4)
+
+**Load Script (no_cache_hits.py)**
+
+View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42)
+
+---
+
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
+| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
+| Gemini | `gemini-2.5-flash-preview-09-2025` | 1M | $0.30 | $2.50 | Chat, reasoning, vision, audio |
+| Gemini | `gemini-2.5-flash-lite-preview-09-2025` | 1M | $0.10 | $0.40 | Chat, reasoning, vision, audio |
+| Gemini | `gemini-flash-latest` | 1M | $0.30 | $2.50 | Chat, reasoning, vision, audio |
+| Gemini | `gemini-flash-lite-latest` | 1M | $0.10 | $0.40 | Chat, reasoning, vision, audio |
+| DeepSeek | `deepseek-chat` | 131K | $0.60 | $1.70 | Chat, function calling, caching |
+| DeepSeek | `deepseek-reasoner` | 131K | $0.60 | $1.70 | Chat, reasoning |
+| Bedrock | `deepseek.v3-v1:0` | 164K | $0.58 | $1.68 | Chat, reasoning, function calling |
+| Azure | `azure/gpt-5-codex` | 272K | $1.25 | $10.00 | Responses API, reasoning, vision |
+| OpenAI | `gpt-5-codex` | 272K | $1.25 | $10.00 | Responses API, reasoning, vision |
+| SambaNova | `sambanova/DeepSeek-V3.1` | 33K | $3.00 | $4.50 | Chat, reasoning, function calling |
+| SambaNova | `sambanova/gpt-oss-120b` | 131K | $3.00 | $4.50 | Chat, reasoning, function calling |
+| Bedrock | `qwen.qwen3-coder-480b-a35b-v1:0` | 262K | $0.22 | $1.80 | Chat, reasoning, function calling |
+| Bedrock | `qwen.qwen3-235b-a22b-2507-v1:0` | 262K | $0.22 | $0.88 | Chat, reasoning, function calling |
+| Bedrock | `qwen.qwen3-coder-30b-a3b-v1:0` | 262K | $0.15 | $0.60 | Chat, reasoning, function calling |
+| Bedrock | `qwen.qwen3-32b-v1:0` | 131K | $0.15 | $0.60 | Chat, reasoning, function calling |
+| Vertex AI | `vertex_ai/qwen/qwen3-next-80b-a3b-instruct-maas` | 262K | $0.15 | $1.20 | Chat, function calling |
+| Vertex AI | `vertex_ai/qwen/qwen3-next-80b-a3b-thinking-maas` | 262K | $0.15 | $1.20 | Chat, function calling |
+| Vertex AI | `vertex_ai/deepseek-ai/deepseek-v3.1-maas` | 164K | $1.35 | $5.40 | Chat, reasoning, function calling |
+| OpenRouter | `openrouter/x-ai/grok-4-fast:free` | 2M | $0.00 | $0.00 | Chat, reasoning, function calling |
+| XAI | `xai/grok-4-fast-reasoning` | 2M | $0.20 | $0.50 | Chat, reasoning, function calling |
+| XAI | `xai/grok-4-fast-non-reasoning` | 2M | $0.20 | $0.50 | Chat, function calling |
+
+#### Features
+
+- **[Gemini](../../docs/providers/gemini)**
+ - Added Gemini 2.5 Flash and Flash-lite preview models (September 2025 release) with improved pricing - [PR #14948](https://github.com/BerriAI/litellm/pull/14948)
+ - Added new Anthropic web fetch tool support - [PR #14951](https://github.com/BerriAI/litellm/pull/14951)
+- **[XAI](../../docs/providers/xai)**
+ - Add xai/grok-4-fast models - [PR #14833](https://github.com/BerriAI/litellm/pull/14833)
+- **[Anthropic](../../docs/providers/anthropic)**
+ - Updated Claude Sonnet 4 configs to reflect million-token context window pricing - [PR #14639](https://github.com/BerriAI/litellm/pull/14639)
+ - Added supported text field to anthropic citation response - [PR #14164](https://github.com/BerriAI/litellm/pull/14164)
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Added support for Qwen models family & Deepseek 3.1 to Amazon Bedrock - [PR #14845](https://github.com/BerriAI/litellm/pull/14845)
+ - Support requestMetadata in Bedrock Converse API - [PR #14570](https://github.com/BerriAI/litellm/pull/14570)
+- **[Vertex AI](../../docs/providers/vertex)**
+ - Added vertex_ai/qwen models and azure/gpt-5-codex - [PR #14844](https://github.com/BerriAI/litellm/pull/14844)
+ - Update vertex ai qwen model pricing - [PR #14828](https://github.com/BerriAI/litellm/pull/14828)
+ - Vertex AI Context Caching: use Vertex ai API v1 instead of v1beta1 and accept 'cachedContent' param - [PR #14831](https://github.com/BerriAI/litellm/pull/14831)
+- **[SambaNova](../../docs/providers/sambanova)**
+ - Add sambanova deepseek v3.1 and gpt-oss-120b - [PR #14866](https://github.com/BerriAI/litellm/pull/14866)
+- **[OpenAI](../../docs/providers/openai)**
+ - Fix inconsistent token configs for gpt-5 models - [PR #14942](https://github.com/BerriAI/litellm/pull/14942)
+ - GPT-3.5-Turbo price updated - [PR #14858](https://github.com/BerriAI/litellm/pull/14858)
+- **[OpenRouter](../../docs/providers/openrouter)**
+ - Add gpt-5 and gpt-5-codex to OpenRouter cost map - [PR #14879](https://github.com/BerriAI/litellm/pull/14879)
+- **[VLLM](../../docs/providers/vllm)**
+ - Fix vllm passthrough - [PR #14778](https://github.com/BerriAI/litellm/pull/14778)
+- **[Flux](../../docs/image_generation)**
+ - Support flux image edit - [PR #14790](https://github.com/BerriAI/litellm/pull/14790)
+
+### Bug Fixes
+
+- **[Anthropic](../../docs/providers/anthropic)**
+ - Fix: Support claude code auth via subscription (anthropic) - [PR #14821](https://github.com/BerriAI/litellm/pull/14821)
+ - Fix Anthropic streaming IDs - [PR #14965](https://github.com/BerriAI/litellm/pull/14965)
+ - Revert incorrect changes to sonnet-4 max output tokens - [PR #14933](https://github.com/BerriAI/litellm/pull/14933)
+- **[OpenAI](../../docs/providers/openai)**
+ - Fix a bug where openai image edit silently ignores multiple images - [PR #14893](https://github.com/BerriAI/litellm/pull/14893)
+- **[VLLM](../../docs/providers/vllm)**
+ - Fix: vLLM provider's rerank endpoint from /v1/rerank to /rerank - [PR #14938](https://github.com/BerriAI/litellm/pull/14938)
+
+#### New Provider Support
+
+- **[W&B Inference](../../docs/providers/wandb)**
+ - Add W&B Inference to LiteLLM - [PR #14416](https://github.com/BerriAI/litellm/pull/14416)
+
+---
+
+## LLM API Endpoints
+
+#### Features
+
+- **General**
+ - Add SDK support for additional headers - [PR #14761](https://github.com/BerriAI/litellm/pull/14761)
+ - Add shared_session parameter for aiohttp ClientSession reuse - [PR #14721](https://github.com/BerriAI/litellm/pull/14721)
+
+#### Bugs
+
+- **General**
+ - Fix: Streaming tool call index assignment for multiple tool calls - [PR #14587](https://github.com/BerriAI/litellm/pull/14587)
+ - Fix load credentials in token counter proxy - [PR #14808](https://github.com/BerriAI/litellm/pull/14808)
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **Proxy CLI Auth**
+ - Allow re-using cli auth token - [PR #14780](https://github.com/BerriAI/litellm/pull/14780)
+ - Create a python method to login using litellm proxy - [PR #14782](https://github.com/BerriAI/litellm/pull/14782)
+ - Fixes for LiteLLM Proxy CLI to Auth to Gateway - [PR #14836](https://github.com/BerriAI/litellm/pull/14836)
+
+**Virtual Keys**
+ - Initial support for scheduled key rotations - [PR #14877](https://github.com/BerriAI/litellm/pull/14877)
+ - Allow scheduling key rotations when creating virtual keys - [PR #14960](https://github.com/BerriAI/litellm/pull/14960)
+
+**Models + Endpoints**
+ - Fix: added Oracle to provider's list - [PR #14835](https://github.com/BerriAI/litellm/pull/14835)
+
+
+#### Bugs
+
+- **SSO** - Fix: SSO "Clear" button writes empty values instead of removing SSO config - [PR #14826](https://github.com/BerriAI/litellm/pull/14826)
+- **Admin Settings** - Remove useful links from admin settings - [PR #14918](https://github.com/BerriAI/litellm/pull/14918)
+- **Management Routes** - Add /user/list to management routes - [PR #14868](https://github.com/BerriAI/litellm/pull/14868)
+---
+
+## Logging / Guardrail / Prompt Management Integrations
+
+#### Features
+
+- **[DataDog](../../docs/proxy/logging#datadog)**
+ - Logging - `datadog` callback Log message content w/o sending to datadog - [PR #14909](https://github.com/BerriAI/litellm/pull/14909)
+- **[Langfuse](../../docs/proxy/logging#langfuse)**
+ - Adding langfuse usage details for cached tokens - [PR #10955](https://github.com/BerriAI/litellm/pull/10955)
+- **[Opik](../../docs/proxy/logging#opik)**
+ - Improve opik integration code - [PR #14888](https://github.com/BerriAI/litellm/pull/14888)
+- **[SQS](../../docs/proxy/logging#sqs)**
+ - Error logging support for SQS Logger - [PR #14974](https://github.com/BerriAI/litellm/pull/14974)
+
+#### Guardrails
+
+- **LakeraAI v2 Guardrail** - Ensure exception is raised correctly - [PR #14867](https://github.com/BerriAI/litellm/pull/14867)
+- **Presidio Guardrail** - Support custom entity types in Presidio guardrail with Union[PiiEntityType, str] - [PR #14899](https://github.com/BerriAI/litellm/pull/14899)
+- **Noma Guardrail** - Add noma guardrail provider to ui - [PR #14415](https://github.com/BerriAI/litellm/pull/14415)
+
+#### Prompt Management
+
+- **BitBucket Integration** - Add BitBucket Integration for Prompt Management - [PR #14882](https://github.com/BerriAI/litellm/pull/14882)
+
+---
+
+## Spend Tracking, Budgets and Rate Limiting
+
+- **Service Tier Pricing** - Add service_tier based pricing support for openai (BOTH Service & Priority Support) - [PR #14796](https://github.com/BerriAI/litellm/pull/14796)
+- **Cost Tracking** - Show input, output, tool call cost breakdown in StandardLoggingPayload - [PR #14921](https://github.com/BerriAI/litellm/pull/14921)
+- **Parallel Request Limiter v3**
+ - Ensure Lua scripts can execute on redis cluster - [PR #14968](https://github.com/BerriAI/litellm/pull/14968)
+ - Fix: get metadata info from both metadata and litellm_metadata fields - [PR #14783](https://github.com/BerriAI/litellm/pull/14783)
+- **Priority Reservation** - Fix: Priority Reservation: keys without priority metadata receive higher priority than keys with explicit priority configurations - [PR #14832](https://github.com/BerriAI/litellm/pull/14832)
+
+---
+
+## MCP Gateway
+
+- **MCP Configuration** - Enable custom fields in mcp_info configuration - [PR #14794](https://github.com/BerriAI/litellm/pull/14794)
+- **MCP Tools** - Remove server_name prefix from list_tools - [PR #14720](https://github.com/BerriAI/litellm/pull/14720)
+- **OAuth Flow** - Initial commit for v2 oauth flow - [PR #14964](https://github.com/BerriAI/litellm/pull/14964)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+- **Memory Leak Fix** - Fix InMemoryCache unbounded growth when TTLs are set - [PR #14869](https://github.com/BerriAI/litellm/pull/14869)
+- **Cache Performance** - Fix: cache root cause - [PR #14827](https://github.com/BerriAI/litellm/pull/14827)
+- **Concurrency Fix** - Fix concurrency/scaling when many Python threads do streaming using *sync* completions - [PR #14816](https://github.com/BerriAI/litellm/pull/14816)
+- **Performance Optimization** - Fix: reduce get_deployment cost to O(1) - [PR #14967](https://github.com/BerriAI/litellm/pull/14967)
+- **Performance Optimization** - Fix: remove slow string operation - [PR #14955](https://github.com/BerriAI/litellm/pull/14955)
+- **DB Connection Management** - Fix: DB connection state retries - [PR #14925](https://github.com/BerriAI/litellm/pull/14925)
+
+
+
+---
+
+## Documentation Updates
+
+- **Provider Documentation** - Fix docs for provider_specific_params.md - [PR #14787](https://github.com/BerriAI/litellm/pull/14787)
+- **Model References** - Update model references from gemini-pro to gemini-2.5-pro - [PR #14775](https://github.com/BerriAI/litellm/pull/14775)
+- **Letta Guide** - Add Letta Guide documentation - [PR #14798](https://github.com/BerriAI/litellm/pull/14798)
+- **README** - Make the README document clearer - [PR #14860](https://github.com/BerriAI/litellm/pull/14860)
+- **Session Management** - Update docs for session management availability - [PR #14914](https://github.com/BerriAI/litellm/pull/14914)
+- **Cost Documentation** - Add documentation for additional cost-related keys in custom pricing - [PR #14949](https://github.com/BerriAI/litellm/pull/14949)
+- **Azure Passthrough** - Add azure passthrough documentation - [PR #14958](https://github.com/BerriAI/litellm/pull/14958)
+- **General Documentation** - Doc updates sept 2025 - [PR #14769](https://github.com/BerriAI/litellm/pull/14769)
+ - Clarified bridging between endpoints and mode in docs.
+ - Added Vertex AI Gemini API configuration as an alternative in relevant guides.
+ Linked AWS authentication info in the Bedrock guardrails documentation.
+ - Added Cancel Response API usage with code snippets
+ - Clarified that SSO (Single Sign-On) is free for up to 5 users:
+ - Alphabetized sidebar, leaving quick start / intros at top of categories
+ - Documented max_connections under cache_params.
+ - Clarified IAM AssumeRole Policy requirements.
+ - Added transform utilities example to Getting Started (showing request transformation).
+ - Added references to models.litellm.ai as the full models list in various docs.
+ - Added a code snippet for async_post_call_success_hook.
+ - Removed broken links to callbacks management guide. - Reformatted and linked cookbooks + other relevant docs
+- **Documentation Corrections** - Corrected docs updates sept 2025 - [PR #14916](https://github.com/BerriAI/litellm/pull/14916)
+
+---
+
+## New Contributors
+
+* @uzaxirr made their first contribution in [PR #14761](https://github.com/BerriAI/litellm/pull/14761)
+* @xprilion made their first contribution in [PR #14416](https://github.com/BerriAI/litellm/pull/14416)
+* @CH-GAGANRAJ made their first contribution in [PR #14779](https://github.com/BerriAI/litellm/pull/14779)
+* @otaviofbrito made their first contribution in [PR #14778](https://github.com/BerriAI/litellm/pull/14778)
+* @danielmklein made their first contribution in [PR #14639](https://github.com/BerriAI/litellm/pull/14639)
+* @Jetemple made their first contribution in [PR #14826](https://github.com/BerriAI/litellm/pull/14826)
+* @akshoop made their first contribution in [PR #14818](https://github.com/BerriAI/litellm/pull/14818)
+* @hazyone made their first contribution in [PR #14821](https://github.com/BerriAI/litellm/pull/14821)
+* @leventov made their first contribution in [PR #14816](https://github.com/BerriAI/litellm/pull/14816)
+* @fabriciojoc made their first contribution in [PR #10955](https://github.com/BerriAI/litellm/pull/10955)
+* @onlylonly made their first contribution in [PR #14845](https://github.com/BerriAI/litellm/pull/14845)
+* @Copilot made their first contribution in [PR #14869](https://github.com/BerriAI/litellm/pull/14869)
+* @arsh72 made their first contribution in [PR #14899](https://github.com/BerriAI/litellm/pull/14899)
+* @berri-teddy made their first contribution in [PR #14914](https://github.com/BerriAI/litellm/pull/14914)
+* @vpbill made their first contribution in [PR #14415](https://github.com/BerriAI/litellm/pull/14415)
+* @kgritesh made their first contribution in [PR #14893](https://github.com/BerriAI/litellm/pull/14893)
+* @oytunkutrup1 made their first contribution in [PR #14858](https://github.com/BerriAI/litellm/pull/14858)
+* @nherment made their first contribution in [PR #14933](https://github.com/BerriAI/litellm/pull/14933)
+* @deepanshululla made their first contribution in [PR #14974](https://github.com/BerriAI/litellm/pull/14974)
+* @TeddyAmkie made their first contribution in [PR #14758](https://github.com/BerriAI/litellm/pull/14758)
+* @SmartManoj made their first contribution in [PR #14775](https://github.com/BerriAI/litellm/pull/14775)
+* @uc4w6c made their first contribution in [PR #14720](https://github.com/BerriAI/litellm/pull/14720)
+* @luizrennocosta made their first contribution in [PR #14783](https://github.com/BerriAI/litellm/pull/14783)
+* @AlexsanderHamir made their first contribution in [PR #14827](https://github.com/BerriAI/litellm/pull/14827)
+* @dharamendrak made their first contribution in [PR #14721](https://github.com/BerriAI/litellm/pull/14721)
+* @TomeHirata made their first contribution in [PR #14164](https://github.com/BerriAI/litellm/pull/14164)
+* @mrFranklin made their first contribution in [PR #14860](https://github.com/BerriAI/litellm/pull/14860)
+* @luisfucros made their first contribution in [PR #14866](https://github.com/BerriAI/litellm/pull/14866)
+* @huangyafei made their first contribution in [PR #14879](https://github.com/BerriAI/litellm/pull/14879)
+* @thiswillbeyourgithub made their first contribution in [PR #14949](https://github.com/BerriAI/litellm/pull/14949)
+* @Maximgitman made their first contribution in [PR #14965](https://github.com/BerriAI/litellm/pull/14965)
+* @subnet-dev made their first contribution in [PR #14938](https://github.com/BerriAI/litellm/pull/14938)
+* @22mSqRi made their first contribution in [PR #14972](https://github.com/BerriAI/litellm/pull/14972)
+
+---
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.77.3.rc.1...v1.77.5.rc.1)**
diff --git a/docs/my-website/release_notes/v1.77.7-stable/index.md b/docs/my-website/release_notes/v1.77.7-stable/index.md
new file mode 100644
index 00000000000..03456297f23
--- /dev/null
+++ b/docs/my-website/release_notes/v1.77.7-stable/index.md
@@ -0,0 +1,389 @@
+---
+title: "v1.77.7-stable - 2.9x Lower Median Latency"
+slug: "v1-77-7"
+date: 2025-10-04T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaff
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+ - name: Alexsander Hamir
+ title: Backend Performance Engineer
+ url: https://www.linkedin.com/in/alexsander-baptista/
+ image_url: https://media.licdn.com/dms/image/v2/D5603AQGXnziu4kqNCQ/profile-displayphoto-crop_800_800/B56ZkxEcuOKEAI-/0/1757464874550?e=1762387200&v=beta&t=9SNXLsWhx8OnYPAMQ9fqAr02oevDYEAL2vMYg2f9ieg
+ - name: Achintya Rajan
+ title: Fullstack Engineer
+ url: https://www.linkedin.com/in/achintya-rajan/
+ image_url: https://media.licdn.com/dms/image/v2/D5603AQGdkEeyJTdljw/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1716271140869?e=1762387200&v=beta&t=9gOoLPeqR2E5z3KSX61EUj3HVZXmgo87vhVuSHeffjc
+ - name: Sameer Kankute
+ title: Backend Engineer (LLM Translation)
+ url: https://www.linkedin.com/in/sameer-kankute/
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQHB_loQYd5gjg/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1719137160975?e=1762387200&v=beta&t=0jbuX-f4eSnDxBY3olI6meuYr-LMbObhFmFbRcKF5mY
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.77.7.rc.1
+```
+
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.77.7.rc.1
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **Dynamic Rate Limiter v3** - Automatically maximizes throughput when capacity is available (< 80% saturation) by allowing lower-priority requests to use unused capacity, then switches to fair priority-based allocation under high load (≥ 80%) to prevent blocking
+- **Major Performance Improvements** - 2.9x lower median latency at 1,000 concurrent users.
+- **Claude Sonnet 4.5** - Support for Anthropic's new Claude Sonnet 4.5 model family with 200K+ context and tiered pricing
+- **MCP Gateway Enhancements** - Fine-grained tool control, server permissions, and forwardable headers
+- **AMD Lemonade & Nvidia NIM** - New provider support for AMD Lemonade and Nvidia NIM Rerank
+- **GitLab Prompt Management** - GitLab-based prompt management integration
+
+### Performance - 2.9x Lower Median Latency
+
+
+
+
+
+This update removes LiteLLM router inefficiencies, reducing complexity from O(M×N) to O(1). Previously, it built a new array and ran repeated checks like data["model"] in llm_router.get_model_ids(). Now, a direct ID-to-deployment map eliminates redundant allocations and scans.
+
+As a result, performance improved across all latency percentiles:
+
+- **Median latency:** 320 ms → **110 ms** (−65.6%)
+- **p95 latency:** 850 ms → **440 ms** (−48.2%)
+- **p99 latency:** 1,400 ms → **810 ms** (−42.1%)
+- **Average latency:** 864 ms → **310 ms** (−64%)
+
+
+#### Test Setup
+
+**Locust**
+
+- **Concurrent users:** 1,000
+- **Ramp-up:** 500
+
+**System Specs**
+
+- **CPU:** 4 vCPUs
+- **Memory:** 8 GB RAM
+- **LiteLLM Workers:** 4
+- **Instances**: 4
+
+**Configuration (config.yaml)**
+
+View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4)
+
+**Load Script (no_cache_hits.py)**
+
+View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42)
+
+### MCP OAuth 2.0 Support
+
+
+
+
+
+This release adds support for OAuth 2.0 Client Credentials for MCP servers. This is great for **Internal Dev Tools** use-cases, as it enables your users to call MCP servers, with their own credentials. E.g. Allowing your developers to call the Github MCP, with their own credentials.
+
+[Set it up today on Claude Code](../../docs/tutorials/claude_responses_api#connecting-mcp-servers)
+
+### Scheduled Key Rotations
+
+
+
+
+
+This release brings support for scheduling virtual key rotations on LiteLLM AI Gateway.
+
+From this release you can enforce Virtual Keys to rotate on a schedule of your choice e.g every 15 days/30 days/60 days etc.
+
+This is great for Proxy Admins who need to enforce security policies for production workloads.
+
+[Get Started](../../docs/proxy/virtual_keys#scheduled-key-rotations)
+
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
+| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
+| Anthropic | `claude-sonnet-4-5` | 200K | $3.00 | $15.00 | Chat, reasoning, vision, function calling, prompt caching |
+| Anthropic | `claude-sonnet-4-5-20250929` | 200K | $3.00 | $15.00 | Chat, reasoning, vision, function calling, prompt caching |
+| Bedrock | `eu.anthropic.claude-sonnet-4-5-20250929-v1:0` | 200K | $3.00 | $15.00 | Chat, reasoning, vision, function calling, prompt caching |
+| Azure AI | `azure_ai/grok-4` | 131K | $5.50 | $27.50 | Chat, reasoning, function calling, web search |
+| Azure AI | `azure_ai/grok-4-fast-reasoning` | 131K | $0.43 | $1.73 | Chat, reasoning, function calling, web search |
+| Azure AI | `azure_ai/grok-4-fast-non-reasoning` | 131K | $0.43 | $1.73 | Chat, function calling, web search |
+| Azure AI | `azure_ai/grok-code-fast-1` | 131K | $3.50 | $17.50 | Chat, function calling, web search |
+| Groq | `groq/moonshotai/kimi-k2-instruct-0905` | Context varies | Pricing varies | Pricing varies | Chat, function calling |
+| Ollama | Ollama Cloud models | Varies | Free | Free | Self-hosted models via Ollama Cloud |
+
+#### Features
+
+- **[Anthropic](../../docs/providers/anthropic)**
+ - Add new claude-sonnet-4-5 model family with tiered pricing above 200K tokens - [PR #15041](https://github.com/BerriAI/litellm/pull/15041)
+ - Add anthropic/claude-sonnet-4-5 to model price json with prompt caching support - [PR #15049](https://github.com/BerriAI/litellm/pull/15049)
+ - Add 200K prices for Sonnet 4.5 - [PR #15140](https://github.com/BerriAI/litellm/pull/15140)
+ - Add cost tracking for /v1/messages in streaming response - [PR #15102](https://github.com/BerriAI/litellm/pull/15102)
+ - Add /v1/messages/count_tokens to Anthropic routes for non-admin user access - [PR #15034](https://github.com/BerriAI/litellm/pull/15034)
+- **[Gemini](../../docs/providers/gemini)**
+ - Ignore type param for gemini tools - [PR #15022](https://github.com/BerriAI/litellm/pull/15022)
+- **[Vertex AI](../../docs/providers/vertex)**
+ - Add LiteLLM Overhead metric for VertexAI - [PR #15040](https://github.com/BerriAI/litellm/pull/15040)
+ - Support googlemap grounding in vertex ai - [PR #15179](https://github.com/BerriAI/litellm/pull/15179)
+- **[Azure](../../docs/providers/azure)**
+ - Add azure_ai grok-4 model family - [PR #15137](https://github.com/BerriAI/litellm/pull/15137)
+ - Use the `extra_query` parameter for GET requests in Azure Batch - [PR #14997](https://github.com/BerriAI/litellm/pull/14997)
+ - Use extra_query for download results (Batch API) - [PR #15025](https://github.com/BerriAI/litellm/pull/15025)
+ - Add support for Azure AD token-based authorization - [PR #14813](https://github.com/BerriAI/litellm/pull/14813)
+- **[Ollama](../../docs/providers/ollama)**
+ - Add ollama cloud models - [PR #15008](https://github.com/BerriAI/litellm/pull/15008)
+- **[Groq](../../docs/providers/groq)**
+ - Add groq/moonshotai/kimi-k2-instruct-0905 - [PR #15079](https://github.com/BerriAI/litellm/pull/15079)
+- **[OpenAI](../../docs/providers/openai)**
+ - Add support for GPT 5 codex models - [PR #14841](https://github.com/BerriAI/litellm/pull/14841)
+- **[DeepInfra](../../docs/providers/deepinfra)**
+ - Update DeepInfra model data refresh with latest pricing - [PR #14939](https://github.com/BerriAI/litellm/pull/14939)
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Add JP Cross-Region Inference - [PR #15188](https://github.com/BerriAI/litellm/pull/15188)
+ - Add "eu.anthropic.claude-sonnet-4-5-20250929-v1:0" - [PR #15181](https://github.com/BerriAI/litellm/pull/15181)
+ - Add twelvelabs bedrock Async Invoke Support - [PR #14871](https://github.com/BerriAI/litellm/pull/14871)
+- **[Nvidia NIM](../../docs/providers/nvidia_nim)**
+ - Add Nvidia NIM Rerank Support - [PR #15152](https://github.com/BerriAI/litellm/pull/15152)
+
+### Bug Fixes
+
+- **[VLLM](../../docs/providers/vllm)**
+ - Fix response_format bug in hosted vllm audio_transcription - [PR #15010](https://github.com/BerriAI/litellm/pull/15010)
+ - Fix passthrough of atranscription into kwargs going to upstream provider - [PR #15005](https://github.com/BerriAI/litellm/pull/15005)
+- **[OCI](../../docs/providers/oci)**
+ - Fix OCI Generative AI Integration when using Proxy - [PR #15072](https://github.com/BerriAI/litellm/pull/15072)
+- **General**
+ - Fix: Authorization header to use correct "Bearer" capitalization - [PR #14764](https://github.com/BerriAI/litellm/pull/14764)
+ - Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - [PR #15116](https://github.com/BerriAI/litellm/pull/15116)
+ - Update request handling for original exceptions - [PR #15013](https://github.com/BerriAI/litellm/pull/15013)
+
+#### New Provider Support
+
+- **[AMD Lemonade](../../docs/providers/lemonade)**
+ - Add AMD Lemonade provider support - [PR #14840](https://github.com/BerriAI/litellm/pull/14840)
+
+---
+
+## LLM API Endpoints
+
+#### Features
+
+- **[Responses API](../../docs/response_api)**
+ - Return Cost for Responses API Streaming requests - [PR #15053](https://github.com/BerriAI/litellm/pull/15053)
+
+- **[/generateContent](../../docs/providers/gemini)**
+ - Add full support for native Gemini API translation - [PR #15029](https://github.com/BerriAI/litellm/pull/15029)
+
+- **Passthrough Gemini Routes**
+ - Add Gemini generateContent passthrough cost tracking - [PR #15014](https://github.com/BerriAI/litellm/pull/15014)
+ - Add streamGenerateContent cost tracking in passthrough - [PR #15199](https://github.com/BerriAI/litellm/pull/15199)
+
+- **Passthrough Vertex AI Routes**
+ - Add cost tracking for Vertex AI Passthrough `/predict` endpoint - [PR #15019](https://github.com/BerriAI/litellm/pull/15019)
+ - Add cost tracking for Vertex AI Live API WebSocket Passthrough - [PR #14956](https://github.com/BerriAI/litellm/pull/14956)
+
+- **General**
+ - Preserve Whitespace Characters in Model Response Streams - [PR #15160](https://github.com/BerriAI/litellm/pull/15160)
+ - Add provider name to payload specification - [PR #15130](https://github.com/BerriAI/litellm/pull/15130)
+ - Ensure query params are forwarded from origin url to downstream request - [PR #15087](https://github.com/BerriAI/litellm/pull/15087)
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **Virtual Keys**
+ - Ensure LLM_API_KEYs can access pass through routes - [PR #15115](https://github.com/BerriAI/litellm/pull/15115)
+ - Support 'guaranteed_throughput' when setting limits on keys belonging to a team - [PR #15120](https://github.com/BerriAI/litellm/pull/15120)
+
+- **Models + Endpoints**
+ - Ensure OCI secret fields not shared on /models and /v1/models endpoints - [PR #15085](https://github.com/BerriAI/litellm/pull/15085)
+ - Add snowflake on UI - [PR #15083](https://github.com/BerriAI/litellm/pull/15083)
+ - Make UI theme settings publicly accessible for custom branding - [PR #15074](https://github.com/BerriAI/litellm/pull/15074)
+
+- **Admin Settings**
+ - Ensure OTEL settings are saved in DB after set on UI - [PR #15118](https://github.com/BerriAI/litellm/pull/15118)
+ - Top api key tags - [PR #15151](https://github.com/BerriAI/litellm/pull/15151), [PR #15156](https://github.com/BerriAI/litellm/pull/15156)
+
+- **MCP**
+ - show health status of MCP servers - [PR #15185](https://github.com/BerriAI/litellm/pull/15185)
+ - allow setting extra headers on the UI - [PR #15185](https://github.com/BerriAI/litellm/pull/15185)
+ - allow editing allowed tools on the UI - [PR #15185](https://github.com/BerriAI/litellm/pull/15185)
+
+### Bug Fixes
+
+- **Virtual Keys**
+ - (security) prevent user key from updating other user keys - [PR #15201](https://github.com/BerriAI/litellm/pull/15201)
+ - (security) don't return all keys with blank key alias on /v2/key/info - [PR #15201](https://github.com/BerriAI/litellm/pull/15201)
+ - Fix Session Token Cookie Infinite Logout Loop - [PR #15146](https://github.com/BerriAI/litellm/pull/15146)
+
+- **Models + Endpoints**
+ - Make UI theme settings publicly accessible for custom branding - [PR #15074](https://github.com/BerriAI/litellm/pull/15074)
+
+- **Teams**
+ - fix failed copy to clipboard for http ui - [PR #15195](https://github.com/BerriAI/litellm/pull/15195)
+
+- **Logs**
+ - fix logs page render logs on filter lookup - [PR #15195](https://github.com/BerriAI/litellm/pull/15195)
+ - fix lookup list of end users (migrate to more efficient /customers/list lookup) - [PR #15195](https://github.com/BerriAI/litellm/pull/15195)
+
+- **Test key**
+ - update selected model on key change - [PR #15197](https://github.com/BerriAI/litellm/pull/15197)
+
+- **Dashboard**
+ - Fix LiteLLM model name fallback in dashboard overview - [PR #14998](https://github.com/BerriAI/litellm/pull/14998)
+
+
+---
+
+## Logging / Guardrail / Prompt Management Integrations
+
+#### Features
+
+- **[OpenTelemetry](../../docs/observability/otel)**
+ - Use generation_name for span naming in logging method - [PR #14799](https://github.com/BerriAI/litellm/pull/14799)
+- **[Langfuse](../../docs/proxy/logging#langfuse)**
+ - Handle non-serializable objects in Langfuse logging - [PR #15148](https://github.com/BerriAI/litellm/pull/15148)
+ - Set usage_details.total in langfuse integration - [PR #15015](https://github.com/BerriAI/litellm/pull/15015)
+- **[Prometheus](../../docs/proxy/prometheus)**
+ - support custom metadata labels on key/team - [PR #15094](https://github.com/BerriAI/litellm/pull/15094)
+
+
+#### Guardrails
+
+- **[Javelin](../../docs/proxy/guardrails)**
+ - Add Javelin standalone guardrails integration for LiteLLM Proxy - [PR #14983](https://github.com/BerriAI/litellm/pull/14983)
+ - Add logging for important status fields in guardrails - [PR #15090](https://github.com/BerriAI/litellm/pull/15090)
+ - Don't run post_call guardrail if no text returned from Bedrock - [PR #15106](https://github.com/BerriAI/litellm/pull/15106)
+
+#### Prompt Management
+
+- **[GitLab](../../docs/proxy/prompt_management)**
+ - GitLab based Prompt manager - [PR #14988](https://github.com/BerriAI/litellm/pull/14988)
+
+---
+
+## Spend Tracking, Budgets and Rate Limiting
+
+- **Cost Tracking**
+ - Proxy: end user cost tracking in the responses API - [PR #15124](https://github.com/BerriAI/litellm/pull/15124)
+- **Parallel Request Limiter v3**
+ - Use well known redis cluster hashing algorithm - [PR #15052](https://github.com/BerriAI/litellm/pull/15052)
+ - Fixes to dynamic rate limiter v3 - add saturation detection - [PR #15119](https://github.com/BerriAI/litellm/pull/15119)
+ - Dynamic Rate Limiter v3 - fixes for detecting saturation + fixes for post saturation behavior - [PR #15192](https://github.com/BerriAI/litellm/pull/15192)
+- **Teams**
+ - Add model specific tpm/rpm limits to teams on LiteLLM - [PR #15044](https://github.com/BerriAI/litellm/pull/15044)
+
+---
+
+## MCP Gateway
+
+- **Server Configuration**
+ - Specify forwardable headers, specify allowed/disallowed tools for MCP servers - [PR #15002](https://github.com/BerriAI/litellm/pull/15002)
+ - Enforce server permissions on call tools - [PR #15044](https://github.com/BerriAI/litellm/pull/15044)
+ - MCP Gateway Fine-grained Tools Addition - [PR #15153](https://github.com/BerriAI/litellm/pull/15153)
+- **Bug Fixes**
+ - Remove servername prefix mcp tools tests - [PR #14986](https://github.com/BerriAI/litellm/pull/14986)
+ - Resolve regression with duplicate Mcp-Protocol-Version header - [PR #15050](https://github.com/BerriAI/litellm/pull/15050)
+ - Fix test_mcp_server.py - [PR #15183](https://github.com/BerriAI/litellm/pull/15183)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+- **Router Optimizations**
+ - **+62.5% P99 Latency Improvement** - Remove router inefficiencies (from O(M*N) to O(1)) - [PR #15046](https://github.com/BerriAI/litellm/pull/15046)
+ - Remove hasattr checks in Router - [PR #15082](https://github.com/BerriAI/litellm/pull/15082)
+ - Remove Double Lookups - [PR #15084](https://github.com/BerriAI/litellm/pull/15084)
+ - Optimize _filter_cooldown_deployments from O(n×m + k×n) to O(n) - [PR #15091](https://github.com/BerriAI/litellm/pull/15091)
+ - Optimize unhealthy deployment filtering in retry path (O(n*m) → O(n+m)) - [PR #15110](https://github.com/BerriAI/litellm/pull/15110)
+- **Cache Optimizations**
+ - Reduce complexity of InMemoryCache.evict_cache from O(n*log(n)) to O(log(n)) - [PR #15000](https://github.com/BerriAI/litellm/pull/15000)
+ - Avoiding expensive operations when cache isn't available - [PR #15182](https://github.com/BerriAI/litellm/pull/15182)
+- **Worker Management**
+ - Add proxy CLI option to recycle workers after N requests - [PR #15007](https://github.com/BerriAI/litellm/pull/15007)
+- **Metrics & Monitoring**
+ - LiteLLM Overhead metric tracking - Add support for tracking litellm overhead on cache hits - [PR #15045](https://github.com/BerriAI/litellm/pull/15045)
+
+---
+
+## Documentation Updates
+
+- **Provider Documentation**
+ - Update litellm docs from latest release - [PR #15004](https://github.com/BerriAI/litellm/pull/15004)
+ - Add missing api_key parameter - [PR #15058](https://github.com/BerriAI/litellm/pull/15058)
+- **General Documentation**
+ - Use docker compose instead of docker-compose - [PR #15024](https://github.com/BerriAI/litellm/pull/15024)
+ - Add railtracks to projects that are using litellm - [PR #15144](https://github.com/BerriAI/litellm/pull/15144)
+ - Perf: Last week improvement - [PR #15193](https://github.com/BerriAI/litellm/pull/15193)
+ - Sync models GitHub documentation with Loom video and cross-reference - [PR #15191](https://github.com/BerriAI/litellm/pull/15191)
+
+---
+
+## Security Fixes
+
+- **JWT Token Security** - Don't log JWT SSO token on .info() log - [PR #15145](https://github.com/BerriAI/litellm/pull/15145)
+
+---
+
+## New Contributors
+
+* @herve-ves made their first contribution in [PR #14998](https://github.com/BerriAI/litellm/pull/14998)
+* @wenxi-onyx made their first contribution in [PR #15008](https://github.com/BerriAI/litellm/pull/15008)
+* @jpetrucciani made their first contribution in [PR #15005](https://github.com/BerriAI/litellm/pull/15005)
+* @abhijitjavelin made their first contribution in [PR #14983](https://github.com/BerriAI/litellm/pull/14983)
+* @ZeroClover made their first contribution in [PR #15039](https://github.com/BerriAI/litellm/pull/15039)
+* @cedarm made their first contribution in [PR #15043](https://github.com/BerriAI/litellm/pull/15043)
+* @Isydmr made their first contribution in [PR #15025](https://github.com/BerriAI/litellm/pull/15025)
+* @serializer made their first contribution in [PR #15013](https://github.com/BerriAI/litellm/pull/15013)
+* @eddierichter-amd made their first contribution in [PR #14840](https://github.com/BerriAI/litellm/pull/14840)
+* @malags made their first contribution in [PR #15000](https://github.com/BerriAI/litellm/pull/15000)
+* @henryhwang made their first contribution in [PR #15029](https://github.com/BerriAI/litellm/pull/15029)
+* @plafleur made their first contribution in [PR #15111](https://github.com/BerriAI/litellm/pull/15111)
+* @tyler-liner made their first contribution in [PR #14799](https://github.com/BerriAI/litellm/pull/14799)
+* @Amir-R25 made their first contribution in [PR #15144](https://github.com/BerriAI/litellm/pull/15144)
+* @georg-wolflein made their first contribution in [PR #15124](https://github.com/BerriAI/litellm/pull/15124)
+* @niharm made their first contribution in [PR #15140](https://github.com/BerriAI/litellm/pull/15140)
+* @anthony-liner made their first contribution in [PR #15015](https://github.com/BerriAI/litellm/pull/15015)
+* @rishiganesh2002 made their first contribution in [PR #15153](https://github.com/BerriAI/litellm/pull/15153)
+* @danielaskdd made their first contribution in [PR #15160](https://github.com/BerriAI/litellm/pull/15160)
+* @JVenberg made their first contribution in [PR #15146](https://github.com/BerriAI/litellm/pull/15146)
+* @speglich made their first contribution in [PR #15072](https://github.com/BerriAI/litellm/pull/15072)
+* @daily-kim made their first contribution in [PR #14764](https://github.com/BerriAI/litellm/pull/14764)
+
+---
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.77.5.rc.4...v1.77.7.rc.1)**
diff --git a/docs/my-website/release_notes/v1.78.0-stable/index.md b/docs/my-website/release_notes/v1.78.0-stable/index.md
new file mode 100644
index 00000000000..63d5eaca0b0
--- /dev/null
+++ b/docs/my-website/release_notes/v1.78.0-stable/index.md
@@ -0,0 +1,394 @@
+---
+title: "[Preview] v1.78.0-stable - MCP Gateway: Control Tool Access by Team, Key"
+slug: "v1-78-0"
+date: 2025-10-11T10:00:00
+authors:
+ - name: Krrish Dholakia
+ title: CEO, LiteLLM
+ url: https://www.linkedin.com/in/krish-d/
+ image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
+ - name: Ishaan Jaff
+ title: CTO, LiteLLM
+ url: https://www.linkedin.com/in/reffajnaahsi/
+ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
+ - name: Alexsander Hamir
+ title: Backend Performance Engineer
+ url: https://www.linkedin.com/in/alexsander-baptista/
+ image_url: https://media.licdn.com/dms/image/v2/D5603AQGXnziu4kqNCQ/profile-displayphoto-crop_800_800/B56ZkxEcuOKEAI-/0/1757464874550?e=1762387200&v=beta&t=9SNXLsWhx8OnYPAMQ9fqAr02oevDYEAL2vMYg2f9ieg
+ - name: Achintya Rajan
+ title: Fullstack Engineer
+ url: https://www.linkedin.com/in/achintya-rajan/
+ image_url: https://media.licdn.com/dms/image/v2/D5603AQGdkEeyJTdljw/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1716271140869?e=1762387200&v=beta&t=9gOoLPeqR2E5z3KSX61EUj3HVZXmgo87vhVuSHeffjc
+ - name: Sameer Kankute
+ title: Backend Engineer (LLM Translation)
+ url: https://www.linkedin.com/in/sameer-kankute/
+ image_url: https://media.licdn.com/dms/image/v2/D4D03AQHB_loQYd5gjg/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1719137160975?e=1762387200&v=beta&t=0jbuX-f4eSnDxBY3olI6meuYr-LMbObhFmFbRcKF5mY
+
+hide_table_of_contents: false
+---
+
+import Image from '@theme/IdealImage';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Deploy this version
+
+
+
+
+``` showLineNumbers title="docker run litellm"
+docker run \
+-e STORE_MODEL_IN_DB=True \
+-p 4000:4000 \
+ghcr.io/berriai/litellm:v1.78.0.rc.2
+```
+
+
+
+
+
+``` showLineNumbers title="pip install litellm"
+pip install litellm==1.78.0.rc.2
+```
+
+
+
+
+---
+
+## Key Highlights
+
+- **MCP Gateway - Control Tool Access by Team, Key** - Control MCP tool access by team/key.
+- **Performance Improvements** - 70% Lower p99 Latency
+- **GPT-5 Pro & GPT-Image-1-Mini** - Day 0 support for OpenAI's GPT-5 Pro (400K context) and gpt-image-1-mini image generation
+- **EnkryptAI Guardrails** - New guardrail integration for content moderation
+- **Tag-Based Budgets** - Support for setting budgets based on request tags
+
+---
+
+### MCP Gateway - Control Tool Access by Team, Key
+
+
+
+
+
+Proxy admins can now control MCP tool access by team or key. This makes it easy to grant different teams selective access to tools from the same MCP server.
+
+For example, you can now give your Engineering team access to `list_repositories`, `create_issue`, and `search_code` tools, while Sales only gets `search_code` and `close_issue` tools.
+
+This makes it easier for Proxy Admins to govern MCP Tool Access.
+
+[Get Started](../../docs/mcp_control#set-allowed-tools-for-a-key-team-or-organization)
+
+---
+
+## Performance - 70% Lower p99 Latency
+
+
+
+
+
+This release cuts p99 latency by 70% on LiteLLM AI Gateway, making it even better for low-latency use cases.
+
+These gains come from two key enhancements:
+
+**Reliable Sessions**
+
+Added support for shared sessions with aiohttp. The shared_session parameter is now consistently used across all calls, enabling connection pooling.
+
+**Faster Routing**
+
+A new `model_name_to_deployment_indices` hash map replaces O(n) list scans in `_get_all_deployments()` with O(1) hash lookups, boosting routing performance and scalability.
+
+As a result, performance improved across all latency percentiles:
+
+- **Median latency:** 110 ms → **100 ms** (−9.1%)
+- **p95 latency:** 440 ms → **150 ms** (−65.9%)
+- **p99 latency:** 810 ms → **240 ms** (−70.4%)
+- **Average latency:** 310 ms → **111.73 ms** (−64.0%)
+
+### **Test Setup**
+
+**Locust**
+
+- **Concurrent users:** 1,000
+- **Ramp-up:** 500
+
+**System Specs**
+
+- **Database was used**
+- **CPU:** 4 vCPUs
+- **Memory:** 8 GB RAM
+- **LiteLLM Workers:** 4
+- **Instances**: 4
+
+**Configuration (config.yaml)**
+
+View the complete configuration: [gist.github.com/AlexsanderHamir/config.yaml](https://gist.github.com/AlexsanderHamir/53f7d554a5d2afcf2c4edb5b6be68ff4)
+
+**Load Script (no_cache_hits.py)**
+
+View the complete load testing script: [gist.github.com/AlexsanderHamir/no_cache_hits.py](https://gist.github.com/AlexsanderHamir/42c33d7a4dc7a57f56a78b560dee3a42)
+
+---
+
+## New Models / Updated Models
+
+#### New Model Support
+
+| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
+| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
+| OpenAI | `gpt-5-pro` | 400K | $15.00 | $120.00 | Responses API, reasoning, vision, function calling, prompt caching, web search |
+| OpenAI | `gpt-5-pro-2025-10-06` | 400K | $15.00 | $120.00 | Responses API, reasoning, vision, function calling, prompt caching, web search |
+| OpenAI | `gpt-image-1-mini` | - | $2.00/img | - | Image generation and editing |
+| OpenAI | `gpt-realtime-mini` | 128K | $0.60 | $2.40 | Realtime audio, function calling |
+| Azure AI | `azure_ai/Phi-4-mini-reasoning` | 131K | $0.08 | $0.32 | Function calling |
+| Azure AI | `azure_ai/Phi-4-reasoning` | 32K | $0.125 | $0.50 | Function calling, reasoning |
+| Azure AI | `azure_ai/MAI-DS-R1` | 128K | $1.35 | $5.40 | Reasoning, function calling |
+| Bedrock | `au.anthropic.claude-sonnet-4-5-20250929-v1:0` | 200K | $3.30 | $16.50 | Chat, reasoning, vision, function calling, prompt caching |
+| Bedrock | `global.anthropic.claude-sonnet-4-5-20250929-v1:0` | 200K | $3.00 | $15.00 | Chat, reasoning, vision, function calling, prompt caching |
+| Bedrock | `global.anthropic.claude-sonnet-4-20250514-v1:0` | 1M | $3.00 | $15.00 | Chat, reasoning, vision, function calling, prompt caching |
+| Bedrock | `cohere.embed-v4:0` | 128K | $0.12 | - | Embeddings, image input support |
+| OCI | `oci/cohere.command-latest` | 128K | $1.56 | $1.56 | Function calling |
+| OCI | `oci/cohere.command-a-03-2025` | 256K | $1.56 | $1.56 | Function calling |
+| OCI | `oci/cohere.command-plus-latest` | 128K | $1.56 | $1.56 | Function calling |
+| Together AI | `together_ai/moonshotai/Kimi-K2-Instruct-0905` | 262K | $1.00 | $3.00 | Function calling |
+| Together AI | `together_ai/Qwen/Qwen3-Next-80B-A3B-Instruct` | 262K | $0.15 | $1.50 | Function calling |
+| Together AI | `together_ai/Qwen/Qwen3-Next-80B-A3B-Thinking` | 262K | $0.15 | $1.50 | Function calling |
+| Vertex AI | MedGemma models | Varies | Varies | Varies | Medical-focused Gemma models on custom endpoints |
+| Watson X | 27 new foundation models | Varies | Varies | Varies | Granite, Llama, Mistral families |
+
+#### Features
+
+- **[OpenAI](../../docs/providers/openai)**
+ - Add GPT-5 Pro model configuration and documentation - [PR #15258](https://github.com/BerriAI/litellm/pull/15258)
+ - Add stop parameter to non-supported params for GPT-5 - [PR #15244](https://github.com/BerriAI/litellm/pull/15244)
+ - Day 0 Support, Add gpt-image-1-mini - [PR #15259](https://github.com/BerriAI/litellm/pull/15259)
+ - Add gpt-realtime-mini support - [PR #15283](https://github.com/BerriAI/litellm/pull/15283)
+ - Add gpt-5-pro-2025-10-06 to model costs - [PR #15344](https://github.com/BerriAI/litellm/pull/15344)
+ - Minimal fix: gpt5 models should not go on cooldown when called with temperature!=1 - [PR #15330](https://github.com/BerriAI/litellm/pull/15330)
+
+- **[Snowflake Cortex](../../docs/providers/snowflake)**
+ - Add function calling support for Snowflake Cortex REST API - [PR #15221](https://github.com/BerriAI/litellm/pull/15221)
+
+- **[Gemini](../../docs/providers/gemini)**
+ - Fix header forwarding for Gemini/Vertex AI providers in proxy mode - [PR #15231](https://github.com/BerriAI/litellm/pull/15231)
+
+- **[Azure](../../docs/providers/azure)**
+ - Removed stop param from unsupported azure models - [PR #15229](https://github.com/BerriAI/litellm/pull/15229)
+ - Fix(azure/responses): remove invalid status param from azure call - [PR #15253](https://github.com/BerriAI/litellm/pull/15253)
+ - Add new Azure AI models with pricing details - [PR #15387](https://github.com/BerriAI/litellm/pull/15387)
+ - AzureAD Default credentials - select credential type based on environment - [PR #14470](https://github.com/BerriAI/litellm/pull/14470)
+
+- **[Bedrock](../../docs/providers/bedrock)**
+ - Add Global Cross-Region Inference - [PR #15210](https://github.com/BerriAI/litellm/pull/15210)
+ - Add Cohere Embed v4 support for AWS Bedrock - [PR #15298](https://github.com/BerriAI/litellm/pull/15298)
+ - Fix(bedrock): include cacheWriteInputTokens in prompt_tokens calculation - [PR #15292](https://github.com/BerriAI/litellm/pull/15292)
+ - Add Bedrock AU Cross-Region Inference for Claude Sonnet 4.5 - [PR #15402](https://github.com/BerriAI/litellm/pull/15402)
+ - Converse → /v1/messages streaming doesn't handle parallel tool calls with Claude models - [PR #15315](https://github.com/BerriAI/litellm/pull/15315)
+
+- **[Vertex AI](../../docs/providers/vertex)**
+ - Implement Context Caching for Vertex AI provider - [PR #15226](https://github.com/BerriAI/litellm/pull/15226)
+ - Support for Vertex AI Gemma Models on Custom Endpoints - [PR #15397](https://github.com/BerriAI/litellm/pull/15397)
+ - VertexAI - gemma model family support (custom endpoints) - [PR #15419](https://github.com/BerriAI/litellm/pull/15419)
+ - VertexAI Gemma model family streaming support + Added MedGemma - [PR #15427](https://github.com/BerriAI/litellm/pull/15427)
+
+- **[OCI](../../docs/providers/oci)**
+ - Add OCI Cohere support with tool calling and streaming capabilities - [PR #15365](https://github.com/BerriAI/litellm/pull/15365)
+
+- **[Watson X](../../docs/providers/watsonx)**
+ - Add Watson X foundation model definitions to model_prices_and_context_window.json - [PR #15219](https://github.com/BerriAI/litellm/pull/15219)
+ - Watsonx - Apply correct prompt templates for openai/gpt-oss model family - [PR #15341](https://github.com/BerriAI/litellm/pull/15341)
+
+- **[OpenRouter](../../docs/providers/openrouter)**
+ - Fix - (openrouter): move cache_control to content blocks for claude/gemini - [PR #15345](https://github.com/BerriAI/litellm/pull/15345)
+ - Fix - OpenRouter cache_control to only apply to last content block - [PR #15395](https://github.com/BerriAI/litellm/pull/15395)
+
+- **[Together AI](../../docs/providers/togetherai)**
+ - Add new together models - [PR #15383](https://github.com/BerriAI/litellm/pull/15383)
+
+### Bug Fixes
+
+- **General**
+ - Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - [PR #15116](https://github.com/BerriAI/litellm/pull/15116)
+ - Fix reasoning response ID - [PR #15265](https://github.com/BerriAI/litellm/pull/15265)
+ - Fix issue with parsing assistant messages - [PR #15320](https://github.com/BerriAI/litellm/pull/15320)
+ - Fix litellm_param based costing - [PR #15336](https://github.com/BerriAI/litellm/pull/15336)
+ - Fix lint errors - [PR #15406](https://github.com/BerriAI/litellm/pull/15406)
+
+---
+
+## LLM API Endpoints
+
+#### Features
+
+- **[Responses API](../../docs/response_api)**
+ - Added streaming support for response api streaming image generation - [PR #15269](https://github.com/BerriAI/litellm/pull/15269)
+ - Add native Responses API support for litellm_proxy provider - [PR #15347](https://github.com/BerriAI/litellm/pull/15347)
+ - Temporarily relax ResponsesAPIResponse parsing to support custom backends (e.g., vLLM) - [PR #15362](https://github.com/BerriAI/litellm/pull/15362)
+
+- **[Files API](../../docs/files_api)**
+ - Feat(files): add @client decorator to file operations - [PR #15339](https://github.com/BerriAI/litellm/pull/15339)
+
+- **[/generateContent](../../docs/providers/gemini)**
+ - Fix gemini cli by actually streaming the response - [PR #15264](https://github.com/BerriAI/litellm/pull/15264)
+
+- **[Azure Passthrough](../../docs/pass_through/azure)**
+ - Azure - passthrough support with router models - [PR #15240](https://github.com/BerriAI/litellm/pull/15240)
+
+#### Bugs
+
+- **General**
+ - Fix x-litellm-cache-key header not being returned on cache hit - [PR #15348](https://github.com/BerriAI/litellm/pull/15348)
+
+---
+
+## Management Endpoints / UI
+
+#### Features
+
+- **Proxy CLI Auth**
+ - Proxy CLI - dont store existing key in the URL, store it in the state param - [PR #15290](https://github.com/BerriAI/litellm/pull/15290)
+
+- **Models + Endpoints**
+ - Make PATCH `/model/{model_id}/update` handle `team_id` consistently with POST `/model/new` - [PR #15297](https://github.com/BerriAI/litellm/pull/15297)
+ - Feature: adds Infinity as a provider in the UI - [PR #15285](https://github.com/BerriAI/litellm/pull/15285)
+ - Fix: model + endpoints page crash when config file contains router_settings.model_group_alias - [PR #15308](https://github.com/BerriAI/litellm/pull/15308)
+ - Models & Endpoints Initial Refactor - [PR #15435](https://github.com/BerriAI/litellm/pull/15435)
+ - Litellm UI API Reference page updates - [PR #15438](https://github.com/BerriAI/litellm/pull/15438)
+
+- **Teams**
+ - Teams page: new column "Your Role" on the teams table - [PR #15384](https://github.com/BerriAI/litellm/pull/15384)
+ - LiteLLM Dashboard Teams UI refactor - [PR #15418](https://github.com/BerriAI/litellm/pull/15418)
+
+- **UI Infrastructure**
+ - Added prettier to autoformat frontend - [PR #15215](https://github.com/BerriAI/litellm/pull/15215)
+ - Adds turbopack to the npm run dev command in UI to build faster during development - [PR #15250](https://github.com/BerriAI/litellm/pull/15250)
+ - (perf) fix: Replaces bloated key list calls with lean key aliases endpoint - [PR #15252](https://github.com/BerriAI/litellm/pull/15252)
+ - Potentially fixes a UI spasm issue with an expired cookie - [PR #15309](https://github.com/BerriAI/litellm/pull/15309)
+ - LiteLLM UI Refactor Infrastructure - [PR #15236](https://github.com/BerriAI/litellm/pull/15236)
+ - Enforces removal of unused imports from UI - [PR #15416](https://github.com/BerriAI/litellm/pull/15416)
+ - Fix: usage page >> Model Activity >> spend per day graph: y-axis clipping on large spend values - [PR #15389](https://github.com/BerriAI/litellm/pull/15389)
+ - Updates guardrail provider logos - [PR #15421](https://github.com/BerriAI/litellm/pull/15421)
+
+- **Admin Settings**
+ - Fix: Router settings do not update despite success message - [PR #15249](https://github.com/BerriAI/litellm/pull/15249)
+ - Fix: Prevents DB from accidentally overriding config file values if they are empty in DB - [PR #15340](https://github.com/BerriAI/litellm/pull/15340)
+
+- **SSO**
+ - SSO - support EntraID app roles - [PR #15351](https://github.com/BerriAI/litellm/pull/15351)
+
+---
+
+## Logging / Guardrail / Prompt Management Integrations
+
+#### Features
+
+- **[PostHog](../../docs/observability/posthog)**
+ - Feat: posthog per request api key - [PR #15379](https://github.com/BerriAI/litellm/pull/15379)
+
+#### Guardrails
+
+- **[EnkryptAI](../../docs/proxy/guardrails)**
+ - Add EnkryptAI Guardrails on LiteLLM - [PR #15390](https://github.com/BerriAI/litellm/pull/15390)
+
+---
+
+## Spend Tracking, Budgets and Rate Limiting
+
+- **Tag Management**
+ - Tag Management - Add support for setting tag based budgets - [PR #15433](https://github.com/BerriAI/litellm/pull/15433)
+
+- **Dynamic Rate Limiter v3**
+ - QA/Fixes - Dynamic Rate Limiter v3 - final QA - [PR #15311](https://github.com/BerriAI/litellm/pull/15311)
+ - Fix dynamic Rate limiter v3 - inserting litellm_model_saturation - [PR #15394](https://github.com/BerriAI/litellm/pull/15394)
+
+- **Shared Health Check**
+ - Implement Shared Health Check State Across Pods - [PR #15380](https://github.com/BerriAI/litellm/pull/15380)
+
+---
+
+## MCP Gateway
+
+- **Tool Control**
+ - MCP Gateway - UI - Select allowed tools for Key, Teams - [PR #15241](https://github.com/BerriAI/litellm/pull/15241)
+ - MCP Gateway - Backend - Allow storing allowed tools by team/key - [PR #15243](https://github.com/BerriAI/litellm/pull/15243)
+ - MCP Gateway - Fine-grained Database Object Storage Control - [PR #15255](https://github.com/BerriAI/litellm/pull/15255)
+ - MCP Gateway - Litellm mcp fixes team control - [PR #15304](https://github.com/BerriAI/litellm/pull/15304)
+ - MCP Gateway - QA/Fixes - Ensure Team/Key level enforcement works for MCPs - [PR #15305](https://github.com/BerriAI/litellm/pull/15305)
+ - Feature: Include server_name in /v1/mcp/server/health endpoint response - [PR #15431](https://github.com/BerriAI/litellm/pull/15431)
+
+- **OpenAPI Integration**
+ - MCP - support converting OpenAPI specs to MCP servers - [PR #15343](https://github.com/BerriAI/litellm/pull/15343)
+ - MCP - specify allowed params per tool - [PR #15346](https://github.com/BerriAI/litellm/pull/15346)
+
+- **Configuration**
+ - MCP - support setting CA_BUNDLE_PATH - [PR #15253](https://github.com/BerriAI/litellm/pull/15253)
+ - Fix: Ensure MCP client stays open during tool call - [PR #15391](https://github.com/BerriAI/litellm/pull/15391)
+ - Remove hardcoded "public" schema in migration.sql - [PR #15363](https://github.com/BerriAI/litellm/pull/15363)
+
+---
+
+## Performance / Loadbalancing / Reliability improvements
+
+- **Router Optimizations**
+ - Fix - Router: add model_name index for O(1) deployment lookups - [PR #15113](https://github.com/BerriAI/litellm/pull/15113)
+ - Refactor Utils: extract inner function from client - [PR #15234](https://github.com/BerriAI/litellm/pull/15234)
+ - Fix Networking: remove limitations - [PR #15302](https://github.com/BerriAI/litellm/pull/15302)
+
+- **Session Management**
+ - Fix - Sessions not being shared - [PR #15388](https://github.com/BerriAI/litellm/pull/15388)
+ - Fix: remove panic from hot path - [PR #15396](https://github.com/BerriAI/litellm/pull/15396)
+ - Fix - shared session parsing and usage issue - [PR #15440](https://github.com/BerriAI/litellm/pull/15440)
+ - Fix: handle closed aiohttp sessions - [PR #15442](https://github.com/BerriAI/litellm/pull/15442)
+ - Fix: prevent session leaks when recreating aiohttp sessions - [PR #15443](https://github.com/BerriAI/litellm/pull/15443)
+
+- **SSL/TLS Performance**
+ - Perf: optimize SSL/TLS handshake performance with prioritized cipher - [PR #15398](https://github.com/BerriAI/litellm/pull/15398)
+
+- **Dependencies**
+ - Upgrades tenacity version to 8.5.0 - [PR #15303](https://github.com/BerriAI/litellm/pull/15303)
+
+- **Data Masking**
+ - Fix - SensitiveDataMasker converts lists to string - [PR #15420](https://github.com/BerriAI/litellm/pull/15420)
+
+---
+
+
+## General AI Gateway Improvements
+
+#### Security
+
+- **General**
+ - Fix: redact AWS credentials when redact_user_api_key_info enabled - [PR #15321](https://github.com/BerriAI/litellm/pull/15321)
+
+---
+
+## Documentation Updates
+
+- **Provider Documentation**
+ - Update doc: perf update - [PR #15211](https://github.com/BerriAI/litellm/pull/15211)
+ - Add W&B Inference documentation - [PR #15278](https://github.com/BerriAI/litellm/pull/15278)
+
+- **Deployment**
+ - Deletion of docker-compose buggy comment that cause `config.yaml` based startup fail - [PR #15425](https://github.com/BerriAI/litellm/pull/15425)
+
+---
+
+## New Contributors
+
+* @Gal-bloch made their first contribution in [PR #15219](https://github.com/BerriAI/litellm/pull/15219)
+* @lcfyi made their first contribution in [PR #15315](https://github.com/BerriAI/litellm/pull/15315)
+* @ashengstd made their first contribution in [PR #15362](https://github.com/BerriAI/litellm/pull/15362)
+* @vkolehmainen made their first contribution in [PR #15363](https://github.com/BerriAI/litellm/pull/15363)
+* @jlan-nl made their first contribution in [PR #15330](https://github.com/BerriAI/litellm/pull/15330)
+* @BCook98 made their first contribution in [PR #15402](https://github.com/BerriAI/litellm/pull/15402)
+* @PabloGmz96 made their first contribution in [PR #15425](https://github.com/BerriAI/litellm/pull/15425)
+
+---
+
+## **[Full Changelog](https://github.com/BerriAI/litellm/compare/v1.77.7.rc.1...v1.78.0.rc.1)**
+
diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js
index 47785a7fde0..53577029e88 100644
--- a/docs/my-website/sidebars.js
+++ b/docs/my-website/sidebars.js
@@ -36,10 +36,12 @@ const sidebars = {
"proxy/guardrails/aporia_api",
"proxy/guardrails/azure_content_guardrail",
"proxy/guardrails/bedrock",
+ "proxy/guardrails/enkryptai",
"proxy/guardrails/lasso_security",
"proxy/guardrails/guardrails_ai",
"proxy/guardrails/lakera_ai",
"proxy/guardrails/model_armor",
+ "proxy/guardrails/noma_security",
"proxy/guardrails/openai_moderation",
"proxy/guardrails/pangea",
"proxy/guardrails/pillar_security",
@@ -48,6 +50,8 @@ const sidebars = {
"proxy/guardrails/secret_detection",
"proxy/guardrails/custom_guardrail",
"proxy/guardrails/prompt_injection",
+ "proxy/guardrails/tool_permission",
+ "proxy/guardrails/javelin",
].sort(),
],
},
@@ -55,43 +59,45 @@ const sidebars = {
type: "category",
label: "Alerting & Monitoring",
items: [
- "proxy/prometheus",
"proxy/alerting",
- "proxy/pagerduty"
- ].sort()
+ "proxy/pagerduty",
+ "proxy/prometheus"
+ ]
},
{
type: "category",
label: "[Beta] Prompt Management",
items: [
- "proxy/prompt_management",
- "proxy/custom_prompt_management"
- ].sort()
+ "proxy/custom_prompt_management",
+ "proxy/native_litellm_prompt",
+ "proxy/prompt_management"
+ ]
},
{
type: "category",
label: "AI Tools (OpenWebUI, Claude Code, etc.)",
items: [
- "tutorials/openweb_ui",
- "tutorials/openai_codex",
+ "tutorials/claude_responses_api",
+ "tutorials/cost_tracking_coding",
+ "tutorials/github_copilot_integration",
"tutorials/litellm_gemini_cli",
"tutorials/litellm_qwen_code_cli",
- "tutorials/github_copilot_integration",
- "tutorials/claude_responses_api",
+ "tutorials/openai_codex",
+ "tutorials/openweb_ui"
]
},
-
+
],
// But you can create a sidebar manually
tutorialSidebar: [
{ type: "doc", id: "index" }, // NEW
-
+
{
type: "category",
- label: "LiteLLM Proxy Server",
+ label: "LiteLLM AI Gateway",
link: {
type: "generated-index",
- title: "LiteLLM Proxy Server (LLM Gateway)",
+ title: "LiteLLM AI Gateway (LLM Proxy)",
description: `OpenAI Proxy Server (LLM Gateway) to call 100+ LLMs in a unified interface & track spend, set budgets per virtual key/user`,
slug: "/simple_proxy",
},
@@ -106,40 +112,64 @@ const sidebars = {
type: "category",
label: "Setup & Deployment",
items: [
- "proxy/deploy",
- "proxy/prod",
+ "proxy/quick_start",
"proxy/cli",
- "proxy/release_cycle",
- "proxy/model_management",
- "proxy/health",
"proxy/debugging",
+ "proxy/deploy",
+ "proxy/health",
"proxy/master_key_rotations",
+ "proxy/model_management",
+ "proxy/prod",
+ "proxy/release_cycle",
],
},
"proxy/demo",
+ {
+ type: "category",
+ label: "Admin UI",
+ items: [
+ "proxy/admin_ui_sso",
+ "proxy/custom_root_ui",
+ "proxy/custom_sso",
+ "proxy/model_hub",
+ "proxy/public_teams",
+ "proxy/self_serve",
+ "proxy/ui",
+ "proxy/ui/bulk_edit_users",
+ "proxy/ui_credentials",
+ "tutorials/scim_litellm",
+ {
+ type: "category",
+ label: "UI Logs",
+ items: [
+ "proxy/ui_logs",
+ "proxy/ui_logs_sessions"
+ ]
+ }
+ ],
+ },
{
type: "category",
label: "Architecture",
- items: ["proxy/architecture", "proxy/control_plane_and_data_plane", "proxy/db_info", "proxy/db_deadlocks", "router_architecture", "proxy/user_management_heirarchy", "proxy/jwt_auth_arch", "proxy/image_handling", "proxy/spend_logs_deletion"],
+ items: [
+ "proxy/architecture",
+ "proxy/control_plane_and_data_plane",
+ "proxy/db_deadlocks",
+ "proxy/db_info",
+ "proxy/image_handling",
+ "proxy/jwt_auth_arch",
+ "proxy/spend_logs_deletion",
+ "proxy/user_management_heirarchy",
+ "router_architecture"
+ ],
},
{
type: "link",
label: "All Endpoints (Swagger)",
href: "https://litellm-api.up.railway.app/",
},
- "proxy/enterprise",
- "proxy/management_cli",
- {
- type: "category",
- label: "Making LLM Requests",
- items: [
- "proxy/user_keys",
- "proxy/clientside_auth",
- "proxy/request_headers",
- "proxy/response_headers",
- "proxy/model_discovery",
- ],
- },
+ "proxy/enterprise",
+ "proxy/management_cli",
{
type: "category",
label: "Authentication",
@@ -157,45 +187,26 @@ const sidebars = {
},
{
type: "category",
- label: "Model Access",
+ label: "Budgets + Rate Limits",
items: [
- "proxy/model_access",
- "proxy/team_model_add"
- ]
- },
- {
- type: "category",
- label: "Admin UI",
- items: [
- "proxy/ui",
- "proxy/admin_ui_sso",
- "proxy/custom_root_ui",
- "proxy/model_hub",
- "proxy/self_serve",
- "proxy/public_teams",
- "tutorials/scim_litellm",
- "proxy/custom_sso",
- "proxy/ui_credentials",
- "proxy/ui/bulk_edit_users",
- {
- type: "category",
- label: "UI Logs",
- items: [
- "proxy/ui_logs",
- "proxy/ui_logs_sessions"
- ]
- }
+ "proxy/users",
+ "proxy/team_budgets",
+ "proxy/tag_budgets",
+ "proxy/customers",
+ "proxy/dynamic_rate_limit",
+ "proxy/rate_limit_tiers",
+ "proxy/temporary_budget_increase",
],
},
+ "proxy/caching",
{
type: "category",
- label: "Spend Tracking",
- items: ["proxy/cost_tracking", "proxy/custom_pricing", "proxy/billing",],
- },
- {
- type: "category",
- label: "Budgets + Rate Limits",
- items: ["proxy/users", "proxy/temporary_budget_increase", "proxy/rate_limit_tiers", "proxy/team_budgets", "proxy/customers"],
+ label: "Create Custom Plugins",
+ description: "Modify requests, responses, and more",
+ items: [
+ "proxy/call_hooks",
+ "proxy/rules",
+ ]
},
{
type: "link",
@@ -206,13 +217,32 @@ const sidebars = {
type: "category",
label: "Logging, Alerting, Metrics",
items: [
+ "proxy/dynamic_logging",
"proxy/logging",
"proxy/logging_spec",
- "proxy/team_logging",
- "proxy/dynamic_logging"
+ "proxy/team_logging"
],
},
-
+ {
+ type: "category",
+ label: "Making LLM Requests",
+ items: [
+ "proxy/user_keys",
+ "proxy/clientside_auth",
+ "proxy/request_headers",
+ "proxy/response_headers",
+ "proxy/forward_client_headers",
+ "proxy/model_discovery",
+ ],
+ },
+ {
+ type: "category",
+ label: "Model Access",
+ items: [
+ "proxy/model_access",
+ "proxy/team_model_add"
+ ]
+ },
{
type: "category",
label: "Secret Managers",
@@ -223,14 +253,13 @@ const sidebars = {
},
{
type: "category",
- label: "Create Custom Plugins",
- description: "Modify requests, responses, and more",
+ label: "Spend Tracking",
items: [
- "proxy/call_hooks",
- "proxy/rules",
- ]
+ "proxy/billing",
+ "proxy/cost_tracking",
+ "proxy/custom_pricing"
+ ],
},
- "proxy/caching",
]
},
{
@@ -244,6 +273,23 @@ const sidebars = {
slug: "/supported_endpoints",
},
items: [
+ "assistants",
+ {
+ type: "category",
+ label: "/audio",
+ items: [
+ "audio_transcription",
+ "text_to_speech",
+ ]
+ },
+ {
+ type: "category",
+ label: "/batches",
+ items: [
+ "batches",
+ "proxy/managed_batches",
+ ]
+ },
{
type: "category",
label: "/chat/completions",
@@ -257,59 +303,11 @@ const sidebars = {
"completion/input",
"completion/output",
"completion/usage",
+ "completion/http_handler_config",
],
},
- "response_api",
"text_completion",
"embedding/supported_embedding",
- "anthropic_unified",
- "mcp",
- "generateContent",
- {
- type: "category",
- label: "/images",
- items: [
- "image_generation",
- "image_edits",
- "image_variations",
- ]
- },
- {
- type: "category",
- label: "/audio",
- "items": [
- "audio_transcription",
- "text_to_speech",
- ]
- },
- {
- type: "category",
- label: "/vector_stores",
- items: [
- "vector_stores/search",
- ]
- },
- {
- type: "category",
- label: "Pass-through Endpoints (Anthropic SDK, etc.)",
- items: [
- "pass_through/intro",
- "pass_through/vertex_ai",
- "pass_through/google_ai_studio",
- "pass_through/cohere",
- "pass_through/vllm",
- "pass_through/mistral",
- "pass_through/openai_passthrough",
- "pass_through/anthropic_completion",
- "pass_through/bedrock",
- "pass_through/assembly_ai",
- "pass_through/langfuse",
- "proxy/pass_through",
- ],
- },
- "rerank",
- "assistants",
-
{
type: "category",
label: "/files",
@@ -318,15 +316,6 @@ const sidebars = {
"proxy/litellm_managed_files",
],
},
- {
- type: "category",
- label: "/batches",
- items: [
- "batches",
- "proxy/managed_batches",
- ]
- },
- "realtime",
{
type: "category",
label: "/fine_tuning",
@@ -334,9 +323,60 @@ const sidebars = {
"fine_tuning",
"proxy/managed_finetuning",
]
+ },
+ "generateContent",
+ "apply_guardrail",
+ {
+ type: "category",
+ label: "/images",
+ items: [
+ "image_edits",
+ "image_generation",
+ "image_variations",
+ ]
+ },
+ {
+ type: "category",
+ label: "/mcp - Model Context Protocol",
+ items: [
+ "mcp",
+ "mcp_usage",
+ "mcp_control",
+ "mcp_cost",
+ "mcp_guardrail",
+ ]
},
"moderation",
- "apply_guardrail",
+ {
+ type: "category",
+ label: "Pass-through Endpoints (Anthropic SDK, etc.)",
+ items: [
+ "pass_through/intro",
+ "pass_through/anthropic_completion",
+ "pass_through/assembly_ai",
+ "pass_through/bedrock",
+ "pass_through/azure_passthrough",
+ "pass_through/cohere",
+ "pass_through/google_ai_studio",
+ "pass_through/langfuse",
+ "pass_through/mistral",
+ "pass_through/openai_passthrough",
+ "pass_through/vertex_ai",
+ "pass_through/vllm",
+ "proxy/pass_through"
+ ]
+ },
+ "realtime",
+ "rerank",
+ "response_api",
+ "anthropic_unified",
+ {
+ type: "category",
+ label: "/vector_stores",
+ items: [
+ "vector_stores/search",
+ ]
+ },
],
},
{
@@ -370,14 +410,23 @@ const sidebars = {
"providers/azure/azure_embedding",
]
},
- "providers/azure_ai",
+ {
+ type: "category",
+ label: "Azure AI",
+ items: [
+ "providers/azure_ai",
+ "providers/azure_ai_img",
+ ]
+ },
{
type: "category",
label: "Vertex AI",
items: [
"providers/vertex",
"providers/vertex_partner",
+ "providers/vertex_self_deployed",
"providers/vertex_image",
+ "providers/vertex_batch",
]
},
{
@@ -397,7 +446,9 @@ const sidebars = {
label: "Bedrock",
items: [
"providers/bedrock",
+ "providers/bedrock_embedding",
"providers/bedrock_agents",
+ "providers/bedrock_batches",
"providers/bedrock_vector_store",
]
},
@@ -420,7 +471,14 @@ const sidebars = {
"providers/deepgram",
"providers/watsonx",
"providers/predibase",
- "providers/nvidia_nim",
+ {
+ type: "category",
+ label: "Nvidia NIM",
+ items: [
+ "providers/nvidia_nim",
+ "providers/nvidia_nim_rerank",
+ ]
+ },
{ type: "doc", id: "providers/nscale", label: "Nscale (EU Sovereign)" },
"providers/xai",
"providers/moonshot",
@@ -438,6 +496,8 @@ const sidebars = {
"providers/elevenlabs",
"providers/fireworks_ai",
"providers/clarifai",
+ "providers/compactifai",
+ "providers/lemonade",
"providers/vllm",
"providers/llamafile",
"providers/infinity",
@@ -453,6 +513,7 @@ const sidebars = {
"providers/replicate",
"providers/togetherai",
"providers/v0",
+ "providers/vercel_ai_gateway",
"providers/morph",
"providers/lambda_ai",
"providers/novita",
@@ -465,43 +526,52 @@ const sidebars = {
"providers/custom_llm_server",
"providers/petals",
"providers/snowflake",
+ "providers/gradient_ai",
"providers/featherless_ai",
"providers/nebius",
"providers/dashscope",
- "providers/bytez"
+ "providers/bytez",
+ "providers/heroku",
+ "providers/oci",
+ "providers/datarobot",
+ "providers/ovhcloud",
+ "providers/wandb_inference",
],
},
{
type: "category",
label: "Guides",
items: [
- "exception_mapping",
+ "completion/computer_use",
+ "completion/web_search",
+ "completion/web_fetch",
+ "completion/function_call",
+ "completion/audio",
+ "completion/document_understanding",
+ "completion/drop_params",
+ "completion/image_generation_chat",
+ "completion/json_mode",
+ "completion/knowledgebase",
+ "completion/message_trimming",
+ "completion/model_alias",
+ "completion/mock_requests",
+ "completion/predict_outputs",
+ "completion/prefix",
+ "completion/prompt_caching",
+ "completion/prompt_formatting",
+ "completion/reliable_completions",
+ "completion/stream",
"completion/provider_specific_params",
+ "completion/vision",
+ "exception_mapping",
+ "completion/batching",
"guides/finetuned_models",
"guides/security_settings",
- "completion/audio",
- "completion/web_search",
- "completion/document_understanding",
- "completion/vision",
- "completion/json_mode",
- "reasoning_content",
- "completion/prompt_caching",
- "completion/predict_outputs",
- "completion/knowledgebase",
- "completion/prefix",
- "completion/drop_params",
- "completion/prompt_formatting",
- "completion/stream",
- "completion/message_trimming",
- "completion/function_call",
- "completion/model_alias",
- "completion/batching",
- "completion/mock_requests",
- "completion/reliable_completions",
-
+ "proxy/veo_video_generation",
+ "reasoning_content"
]
},
-
+
{
type: "category",
label: "Routing, Loadbalancing & Fallbacks",
@@ -511,28 +581,39 @@ const sidebars = {
description: "Learn how to load balance, route, and set fallbacks for your LLM requests",
slug: "/routing-load-balancing",
},
- items: ["routing", "scheduler", "proxy/load_balancing", "proxy/reliability", "proxy/timeout", "proxy/auto_routing", "proxy/tag_routing", "proxy/provider_budget_routing", "wildcard_routing"],
+ items: [
+ "routing",
+ "scheduler",
+ "proxy/auto_routing",
+ "proxy/load_balancing",
+ "proxy/provider_budget_routing",
+ "proxy/reliability",
+ "proxy/tag_routing",
+ "proxy/timeout",
+ "wildcard_routing"
+ ],
},
{
type: "category",
label: "LiteLLM Python SDK",
items: [
"set_keys",
+ "budget_manager",
+ "caching/all_caches",
"completion/token_usage",
"sdk_custom_pricing",
"embedding/async_embedding",
"embedding/moderation",
- "budget_manager",
- "caching/all_caches",
"migration",
+ "sdk_custom_pricing",
{
type: "category",
label: "LangChain, LlamaIndex, Instructor Integration",
items: ["langchain/langchain", "tutorials/instructor"],
- },
+ }
],
},
-
+
{
type: "category",
label: "Load Testing",
@@ -601,6 +682,7 @@ const sidebars = {
items: [
"data_security",
"data_retention",
+ "proxy/security_encryption_faq",
"migration_policy",
{
type: "category",
@@ -633,7 +715,8 @@ const sidebars = {
"projects/llm_cord",
"projects/pgai",
"projects/GPTLocalhost",
- "projects/HolmesGPT"
+ "projects/HolmesGPT",
+ "projects/Railtracks",
],
},
"extras/code_quality",
diff --git a/docs/my-website/src/pages/completion/supported.md b/docs/my-website/src/pages/completion/supported.md
index 097af2bb4cb..e146e6efc97 100644
--- a/docs/my-website/src/pages/completion/supported.md
+++ b/docs/my-website/src/pages/completion/supported.md
@@ -8,6 +8,7 @@
| gpt-3.5-turbo-16k | `completion('gpt-3.5-turbo-16k', messages)` | `os.environ['OPENAI_API_KEY']` |
| gpt-3.5-turbo-16k-0613 | `completion('gpt-3.5-turbo-16k-0613', messages)` | `os.environ['OPENAI_API_KEY']` |
| gpt-4 | `completion('gpt-4', messages)` | `os.environ['OPENAI_API_KEY']` |
+| gpt-5-pro | `completion('gpt-5-pro', messages)` | `os.environ['OPENAI_API_KEY']` |
## Azure OpenAI Chat Completion Models
For Azure calls add the `azure/` prefix to `model`. If your azure deployment name is `gpt-v-2` set `model` = `azure/gpt-v-2`
diff --git a/docs/my-website/static/llms-full.txt b/docs/my-website/static/llms-full.txt
index c64d4170968..203dfd12bab 100644
--- a/docs/my-website/static/llms-full.txt
+++ b/docs/my-website/static/llms-full.txt
@@ -1699,7 +1699,7 @@ This release allow you to group requests to LiteLLM proxy into a session. If you
1. Added support for max\_completion\_tokens parameter [Get Started](https://docs.litellm.ai/docs/providers/sagemaker), [PR](https://github.com/BerriAI/litellm/pull/10300)
- **Responses API**
1. Added support for GET and DELETE operations - `/v1/responses/{response_id}` [Get Started](https://docs.litellm.ai/docs/response_api)
-2. Added session management support for non-OpenAI models [PR](https://github.com/BerriAI/litellm/pull/10321)
+2. Added session management support for all supported models [PR](https://github.com/BerriAI/litellm/pull/10321)
3. Added routing affinity to maintain model consistency within sessions [Get Started](https://docs.litellm.ai/docs/response_api#load-balancing-with-routing-affinity), [PR](https://github.com/BerriAI/litellm/pull/10193)
## Spend Tracking Improvements [](https://docs.litellm.ai/release_notes\#spend-tracking-improvements "Direct link to Spend Tracking Improvements")
@@ -7736,7 +7736,7 @@ This release allow you to group requests to LiteLLM proxy into a session. If you
1. Added support for max\_completion\_tokens parameter [Get Started](https://docs.litellm.ai/docs/providers/sagemaker), [PR](https://github.com/BerriAI/litellm/pull/10300)
- **Responses API**
1. Added support for GET and DELETE operations - `/v1/responses/{response_id}` [Get Started](https://docs.litellm.ai/docs/response_api)
-2. Added session management support for non-OpenAI models [PR](https://github.com/BerriAI/litellm/pull/10321)
+2. Added session management support for all supported models [PR](https://github.com/BerriAI/litellm/pull/10321)
3. Added routing affinity to maintain model consistency within sessions [Get Started](https://docs.litellm.ai/docs/response_api#load-balancing-with-routing-affinity), [PR](https://github.com/BerriAI/litellm/pull/10193)
## Spend Tracking Improvements [](https://docs.litellm.ai/release_notes/tags/responses-api\#spend-tracking-improvements "Direct link to Spend Tracking Improvements")
@@ -8295,7 +8295,7 @@ This release allow you to group requests to LiteLLM proxy into a session. If you
1. Added support for max\_completion\_tokens parameter [Get Started](https://docs.litellm.ai/docs/providers/sagemaker), [PR](https://github.com/BerriAI/litellm/pull/10300)
- **Responses API**
1. Added support for GET and DELETE operations - `/v1/responses/{response_id}` [Get Started](https://docs.litellm.ai/docs/response_api)
-2. Added session management support for non-OpenAI models [PR](https://github.com/BerriAI/litellm/pull/10321)
+2. Added session management support for all supported models [PR](https://github.com/BerriAI/litellm/pull/10321)
3. Added routing affinity to maintain model consistency within sessions [Get Started](https://docs.litellm.ai/docs/response_api#load-balancing-with-routing-affinity), [PR](https://github.com/BerriAI/litellm/pull/10193)
## Spend Tracking Improvements [](https://docs.litellm.ai/release_notes/tags/security\#spend-tracking-improvements "Direct link to Spend Tracking Improvements")
@@ -8821,7 +8821,7 @@ This release allow you to group requests to LiteLLM proxy into a session. If you
1. Added support for max\_completion\_tokens parameter [Get Started](https://docs.litellm.ai/docs/providers/sagemaker), [PR](https://github.com/BerriAI/litellm/pull/10300)
- **Responses API**
1. Added support for GET and DELETE operations - `/v1/responses/{response_id}` [Get Started](https://docs.litellm.ai/docs/response_api)
-2. Added session management support for non-OpenAI models [PR](https://github.com/BerriAI/litellm/pull/10321)
+2. Added session management support for all supported models [PR](https://github.com/BerriAI/litellm/pull/10321)
3. Added routing affinity to maintain model consistency within sessions [Get Started](https://docs.litellm.ai/docs/response_api#load-balancing-with-routing-affinity), [PR](https://github.com/BerriAI/litellm/pull/10193)
## Spend Tracking Improvements [](https://docs.litellm.ai/release_notes/tags/session-management\#spend-tracking-improvements "Direct link to Spend Tracking Improvements")
diff --git a/enterprise/dist/litellm_enterprise-0.1.17-py3-none-any.whl b/enterprise/dist/litellm_enterprise-0.1.17-py3-none-any.whl
new file mode 100644
index 00000000000..9c2856b4652
Binary files /dev/null and b/enterprise/dist/litellm_enterprise-0.1.17-py3-none-any.whl differ
diff --git a/enterprise/dist/litellm_enterprise-0.1.17.tar.gz b/enterprise/dist/litellm_enterprise-0.1.17.tar.gz
new file mode 100644
index 00000000000..92d4a6ee92f
Binary files /dev/null and b/enterprise/dist/litellm_enterprise-0.1.17.tar.gz differ
diff --git a/enterprise/dist/litellm_enterprise-0.1.19-py3-none-any.whl b/enterprise/dist/litellm_enterprise-0.1.19-py3-none-any.whl
new file mode 100644
index 00000000000..5b48b65e4d2
Binary files /dev/null and b/enterprise/dist/litellm_enterprise-0.1.19-py3-none-any.whl differ
diff --git a/enterprise/dist/litellm_enterprise-0.1.19.tar.gz b/enterprise/dist/litellm_enterprise-0.1.19.tar.gz
new file mode 100644
index 00000000000..2f99960bdeb
Binary files /dev/null and b/enterprise/dist/litellm_enterprise-0.1.19.tar.gz differ
diff --git a/enterprise/enterprise_hooks/aporia_ai.py b/enterprise/enterprise_hooks/aporia_ai.py
index d2184e92f2f..de741aa6ca7 100644
--- a/enterprise/enterprise_hooks/aporia_ai.py
+++ b/enterprise/enterprise_hooks/aporia_ai.py
@@ -173,6 +173,7 @@ class AporiaGuardrail(CustomGuardrail):
"moderation",
"audio_transcription",
"responses",
+ "mcp_call",
],
):
from litellm.proxy.common_utils.callback_utils import (
diff --git a/enterprise/enterprise_hooks/google_text_moderation.py b/enterprise/enterprise_hooks/google_text_moderation.py
index fe26a03207f..61987af7532 100644
--- a/enterprise/enterprise_hooks/google_text_moderation.py
+++ b/enterprise/enterprise_hooks/google_text_moderation.py
@@ -95,6 +95,7 @@ class _ENTERPRISE_GoogleTextModeration(CustomLogger):
"moderation",
"audio_transcription",
"responses",
+ "mcp_call",
],
):
"""
diff --git a/enterprise/enterprise_hooks/openai_moderation.py b/enterprise/enterprise_hooks/openai_moderation.py
index ee8ac495099..0b6f34018b4 100644
--- a/enterprise/enterprise_hooks/openai_moderation.py
+++ b/enterprise/enterprise_hooks/openai_moderation.py
@@ -42,6 +42,7 @@ class _ENTERPRISE_OpenAI_Moderation(CustomLogger):
"moderation",
"audio_transcription",
"responses",
+ "mcp_call",
],
):
text = ""
diff --git a/enterprise/litellm_enterprise/enterprise_callbacks/generic_api_callback.py b/enterprise/litellm_enterprise/enterprise_callbacks/generic_api_callback.py
index d239be41257..7e259d4e19d 100644
--- a/enterprise/litellm_enterprise/enterprise_callbacks/generic_api_callback.py
+++ b/enterprise/litellm_enterprise/enterprise_callbacks/generic_api_callback.py
@@ -9,7 +9,7 @@ Callback to log events to a Generic API Endpoint
import asyncio
import os
import traceback
-import uuid
+from litellm._uuid import uuid
from typing import Dict, List, Optional, Union
import litellm
diff --git a/enterprise/litellm_enterprise/enterprise_callbacks/llama_guard.py b/enterprise/litellm_enterprise/enterprise_callbacks/llama_guard.py
index a44af55d4b1..ea428b51b8e 100644
--- a/enterprise/litellm_enterprise/enterprise_callbacks/llama_guard.py
+++ b/enterprise/litellm_enterprise/enterprise_callbacks/llama_guard.py
@@ -105,6 +105,7 @@ class _ENTERPRISE_LlamaGuard(CustomLogger):
"moderation",
"audio_transcription",
"responses",
+ "mcp_call",
],
):
"""
diff --git a/enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py b/enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py
index 1475a94303e..e290013248d 100644
--- a/enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py
+++ b/enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py
@@ -63,7 +63,7 @@ class _ENTERPRISE_LLMGuard(CustomLogger):
analyze_url, json=analyze_payload
) as response:
redacted_text = await response.json()
- verbose_proxy_logger.info(
+ verbose_proxy_logger.debug(
f"LLM Guard: Received response - {redacted_text}"
)
if redacted_text is not None:
@@ -127,6 +127,7 @@ class _ENTERPRISE_LLMGuard(CustomLogger):
"moderation",
"audio_transcription",
"responses",
+ "mcp_call",
],
):
"""
diff --git a/enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py b/enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py
index 00230937b32..8db0fcf752c 100644
--- a/enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py
+++ b/enterprise/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.py
@@ -109,6 +109,9 @@ class PagerDutyAlerting(SlackAlerting):
error_llm_provider=error_info.get("llm_provider"),
user_api_key_hash=_meta.get("user_api_key_hash"),
user_api_key_alias=_meta.get("user_api_key_alias"),
+ user_api_key_spend=_meta.get("user_api_key_spend"),
+ user_api_key_max_budget=_meta.get("user_api_key_max_budget"),
+ user_api_key_budget_reset_at=_meta.get("user_api_key_budget_reset_at"),
user_api_key_org_id=_meta.get("user_api_key_org_id"),
user_api_key_team_id=_meta.get("user_api_key_team_id"),
user_api_key_user_id=_meta.get("user_api_key_user_id"),
@@ -116,6 +119,7 @@ class PagerDutyAlerting(SlackAlerting):
user_api_key_end_user_id=_meta.get("user_api_key_end_user_id"),
user_api_key_user_email=_meta.get("user_api_key_user_email"),
user_api_key_request_route=_meta.get("user_api_key_request_route"),
+ user_api_key_auth_metadata=_meta.get("user_api_key_auth_metadata"),
)
)
@@ -147,6 +151,7 @@ class PagerDutyAlerting(SlackAlerting):
"audio_transcription",
"pass_through_endpoint",
"rerank",
+ "mcp_call",
],
) -> Optional[Union[Exception, str, dict]]:
"""
@@ -190,6 +195,13 @@ class PagerDutyAlerting(SlackAlerting):
error_llm_provider="HangingRequest",
user_api_key_hash=user_api_key_dict.api_key,
user_api_key_alias=user_api_key_dict.key_alias,
+ user_api_key_spend=user_api_key_dict.spend,
+ user_api_key_max_budget=user_api_key_dict.max_budget,
+ user_api_key_budget_reset_at=(
+ user_api_key_dict.budget_reset_at.isoformat()
+ if user_api_key_dict.budget_reset_at
+ else None
+ ),
user_api_key_org_id=user_api_key_dict.org_id,
user_api_key_team_id=user_api_key_dict.team_id,
user_api_key_user_id=user_api_key_dict.user_id,
@@ -197,6 +209,7 @@ class PagerDutyAlerting(SlackAlerting):
user_api_key_end_user_id=user_api_key_dict.end_user_id,
user_api_key_user_email=user_api_key_dict.user_email,
user_api_key_request_route=user_api_key_dict.request_route,
+ user_api_key_auth_metadata=user_api_key_dict.metadata,
)
)
diff --git a/enterprise/litellm_enterprise/enterprise_callbacks/session_handler.py b/enterprise/litellm_enterprise/enterprise_callbacks/session_handler.py
deleted file mode 100644
index 1a08a8f9101..00000000000
--- a/enterprise/litellm_enterprise/enterprise_callbacks/session_handler.py
+++ /dev/null
@@ -1,160 +0,0 @@
-import json
-from typing import TYPE_CHECKING, Any, List, Optional, Union, cast
-
-from litellm._logging import verbose_proxy_logger
-from litellm.proxy._types import SpendLogsPayload
-from litellm.responses.utils import ResponsesAPIRequestUtils
-from litellm.types.llms.openai import (
- AllMessageValues,
- ChatCompletionResponseMessage,
- GenericChatCompletionMessage,
- ResponseInputParam,
-)
-from litellm.types.utils import ChatCompletionMessageToolCall, Message, ModelResponse
-
-if TYPE_CHECKING:
- from litellm.responses.litellm_completion_transformation.transformation import (
- ChatCompletionSession,
- )
-else:
- ChatCompletionSession = Any
-
-
-class _ENTERPRISE_ResponsesSessionHandler:
- @staticmethod
- async def get_chat_completion_message_history_for_previous_response_id(
- previous_response_id: str,
- ) -> ChatCompletionSession:
- """
- Return the chat completion message history for a previous response id
- """
- from litellm.responses.litellm_completion_transformation.transformation import (
- ChatCompletionSession,
- LiteLLMCompletionResponsesConfig,
- )
-
- verbose_proxy_logger.debug(
- "inside get_chat_completion_message_history_for_previous_response_id"
- )
- all_spend_logs: List[
- SpendLogsPayload
- ] = await _ENTERPRISE_ResponsesSessionHandler.get_all_spend_logs_for_previous_response_id(
- previous_response_id
- )
- verbose_proxy_logger.debug(
- "found %s spend logs for this response id", len(all_spend_logs)
- )
-
- litellm_session_id: Optional[str] = None
- if len(all_spend_logs) > 0:
- litellm_session_id = all_spend_logs[0].get("session_id")
-
- chat_completion_message_history: List[
- Union[
- AllMessageValues,
- GenericChatCompletionMessage,
- ChatCompletionMessageToolCall,
- ChatCompletionResponseMessage,
- Message,
- ]
- ] = []
- for spend_log in all_spend_logs:
- proxy_server_request: Union[str, dict] = (
- spend_log.get("proxy_server_request") or "{}"
- )
- proxy_server_request_dict: Optional[dict] = None
- response_input_param: Optional[Union[str, ResponseInputParam]] = None
- if isinstance(proxy_server_request, dict):
- proxy_server_request_dict = proxy_server_request
- else:
- proxy_server_request_dict = json.loads(proxy_server_request)
-
- ############################################################
- # Add Input messages for this Spend Log
- ############################################################
- if proxy_server_request_dict:
- _response_input_param = proxy_server_request_dict.get("input", None)
- if isinstance(_response_input_param, str):
- response_input_param = _response_input_param
- elif isinstance(_response_input_param, dict):
- response_input_param = cast(
- ResponseInputParam, _response_input_param
- )
-
- if response_input_param:
- chat_completion_messages = LiteLLMCompletionResponsesConfig.transform_responses_api_input_to_messages(
- input=response_input_param,
- responses_api_request=proxy_server_request_dict or {},
- )
- chat_completion_message_history.extend(chat_completion_messages)
-
- ############################################################
- # Add Output messages for this Spend Log
- ############################################################
- _response_output = spend_log.get("response", "{}")
- if isinstance(_response_output, dict):
- # transform `ChatCompletion Response` to `ResponsesAPIResponse`
- model_response = ModelResponse(**_response_output)
- for choice in model_response.choices:
- if hasattr(choice, "message"):
- chat_completion_message_history.append(
- getattr(choice, "message")
- )
-
- verbose_proxy_logger.debug(
- "chat_completion_message_history %s",
- json.dumps(chat_completion_message_history, indent=4, default=str),
- )
- return ChatCompletionSession(
- messages=chat_completion_message_history,
- litellm_session_id=litellm_session_id,
- )
-
- @staticmethod
- async def get_all_spend_logs_for_previous_response_id(
- previous_response_id: str,
- ) -> List[SpendLogsPayload]:
- """
- Get all spend logs for a previous response id
-
-
- SQL query
-
- SELECT session_id FROM spend_logs WHERE response_id = previous_response_id, SELECT * FROM spend_logs WHERE session_id = session_id
- """
- from litellm.proxy.proxy_server import prisma_client
-
- verbose_proxy_logger.debug("decoding response id=%s", previous_response_id)
-
- decoded_response_id = (
- ResponsesAPIRequestUtils._decode_responses_api_response_id(
- previous_response_id
- )
- )
- previous_response_id = decoded_response_id.get(
- "response_id", previous_response_id
- )
- if prisma_client is None:
- return []
-
- query = """
- WITH matching_session AS (
- SELECT session_id
- FROM "LiteLLM_SpendLogs"
- WHERE request_id = $1
- )
- SELECT *
- FROM "LiteLLM_SpendLogs"
- WHERE session_id IN (SELECT session_id FROM matching_session)
- ORDER BY "endTime" ASC;
- """
-
- spend_logs = await prisma_client.db.query_raw(query, previous_response_id)
-
- verbose_proxy_logger.debug(
- "Found the following spend logs for previous response id %s: %s",
- previous_response_id,
- json.dumps(spend_logs, indent=4, default=str),
- )
-
- return spend_logs
diff --git a/enterprise/litellm_enterprise/integrations/prometheus.py b/enterprise/litellm_enterprise/integrations/prometheus.py
index ddbcf948c85..3b37e14b896 100644
--- a/enterprise/litellm_enterprise/integrations/prometheus.py
+++ b/enterprise/litellm_enterprise/integrations/prometheus.py
@@ -21,6 +21,7 @@ from litellm._logging import print_verbose, verbose_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import LiteLLM_TeamTable, UserAPIKeyAuth
from litellm.types.integrations.prometheus import *
+from litellm.types.integrations.prometheus import _sanitize_prometheus_label_name
from litellm.types.utils import StandardLoggingPayload
from litellm.utils import get_end_user_id_for_cost_tracking
@@ -95,13 +96,16 @@ class PrometheusLogger(CustomLogger):
self.litellm_llm_api_time_to_first_token_metric = self._histogram_factory(
"litellm_llm_api_time_to_first_token_metric",
"Time to first token for a models LLM API call",
- labelnames=[
- "model",
- "hashed_api_key",
- "api_key_alias",
- "team",
- "team_alias",
- ],
+ # labelnames=[
+ # "model",
+ # "hashed_api_key",
+ # "api_key_alias",
+ # "team",
+ # "team_alias",
+ # ],
+ labelnames=self.get_labels_for_metric(
+ "litellm_llm_api_time_to_first_token_metric"
+ ),
buckets=LATENCY_BUCKETS,
)
@@ -109,32 +113,24 @@ class PrometheusLogger(CustomLogger):
self.litellm_spend_metric = self._counter_factory(
"litellm_spend_metric",
"Total spend on LLM requests",
- labelnames=[
- "end_user",
- "hashed_api_key",
- "api_key_alias",
- "model",
- "team",
- "team_alias",
- "user",
- ],
+ labelnames=self.get_labels_for_metric("litellm_spend_metric"),
)
# Counter for total_output_tokens
self.litellm_tokens_metric = self._counter_factory(
- "litellm_total_tokens",
+ "litellm_total_tokens_metric",
"Total number of input + output tokens from LLM requests",
labelnames=self.get_labels_for_metric("litellm_total_tokens_metric"),
)
self.litellm_input_tokens_metric = self._counter_factory(
- "litellm_input_tokens",
+ "litellm_input_tokens_metric",
"Total number of input tokens from LLM requests",
labelnames=self.get_labels_for_metric("litellm_input_tokens_metric"),
)
self.litellm_output_tokens_metric = self._counter_factory(
- "litellm_output_tokens",
+ "litellm_output_tokens_metric",
"Total number of output tokens from LLM requests",
labelnames=self.get_labels_for_metric("litellm_output_tokens_metric"),
)
@@ -243,25 +239,18 @@ class PrometheusLogger(CustomLogger):
labelnames=["api_provider"],
)
- # Get all keys
- _logged_llm_labels = [
- UserAPIKeyLabelNames.v2_LITELLM_MODEL_NAME.value,
- UserAPIKeyLabelNames.MODEL_ID.value,
- UserAPIKeyLabelNames.API_BASE.value,
- UserAPIKeyLabelNames.API_PROVIDER.value,
- ]
-
# Metric for deployment state
self.litellm_deployment_state = self._gauge_factory(
"litellm_deployment_state",
"LLM Deployment Analytics - The state of the deployment: 0 = healthy, 1 = partial outage, 2 = complete outage",
- labelnames=_logged_llm_labels,
+ labelnames=self.get_labels_for_metric("litellm_deployment_state"),
)
self.litellm_deployment_cooled_down = self._counter_factory(
"litellm_deployment_cooled_down",
"LLM Deployment Analytics - Number of times a deployment has been cooled down by LiteLLM load balancing logic. exception_status is the status of the exception that caused the deployment to be cooled down",
- labelnames=_logged_llm_labels + [EXCEPTION_STATUS],
+ # labelnames=_logged_llm_labels + [EXCEPTION_STATUS],
+ labelnames=self.get_labels_for_metric("litellm_deployment_cooled_down"),
)
self.litellm_deployment_success_responses = self._counter_factory(
@@ -327,6 +316,7 @@ class PrometheusLogger(CustomLogger):
documentation="deprecated - use litellm_proxy_total_requests_metric. Total number of LLM calls to litellm - track total per API Key, team, user",
labelnames=self.get_labels_for_metric("litellm_requests_metric"),
)
+
except Exception as e:
print_verbose(f"Got exception on init prometheus client {str(e)}")
raise e
@@ -805,9 +795,16 @@ class PrometheusLogger(CustomLogger):
output_tokens = standard_logging_payload["completion_tokens"]
tokens_used = standard_logging_payload["total_tokens"]
response_cost = standard_logging_payload["response_cost"]
- _requester_metadata = standard_logging_payload["metadata"].get(
+ _requester_metadata: Optional[dict] = standard_logging_payload["metadata"].get(
"requester_metadata"
)
+ user_api_key_auth_metadata: Optional[dict] = standard_logging_payload[
+ "metadata"
+ ].get("user_api_key_auth_metadata")
+ combined_metadata: Dict[str, Any] = {
+ **(_requester_metadata if _requester_metadata else {}),
+ **(user_api_key_auth_metadata if user_api_key_auth_metadata else {}),
+ }
if standard_logging_payload is not None and isinstance(
standard_logging_payload, dict
):
@@ -839,8 +836,7 @@ class PrometheusLogger(CustomLogger):
exception_status=None,
exception_class=None,
custom_metadata_labels=get_custom_labels_from_metadata(
- metadata=standard_logging_payload["metadata"].get("requester_metadata")
- or {}
+ metadata=combined_metadata
),
route=standard_logging_payload["metadata"].get(
"user_api_key_request_route"
@@ -1052,20 +1048,12 @@ class PrometheusLogger(CustomLogger):
_labels = prometheus_label_factory(
supported_enum_labels=self.get_labels_for_metric(
- metric_name="litellm_proxy_total_requests_metric"
+ metric_name="litellm_spend_metric"
),
enum_values=enum_values,
)
- self.litellm_spend_metric.labels(
- end_user_id,
- user_api_key,
- user_api_key_alias,
- model,
- user_api_team,
- user_api_team_alias,
- user_id,
- ).inc(response_cost)
+ self.litellm_spend_metric.labels(**_labels).inc(response_cost)
def _set_virtual_key_rate_limit_metrics(
self,
@@ -1668,9 +1656,22 @@ class PrometheusLogger(CustomLogger):
api_base: Optional[str],
api_provider: str,
):
- self.litellm_deployment_state.labels(
- litellm_model_name, model_id, api_base, api_provider
- ).set(state)
+ """
+ Set the deployment state.
+ """
+ ### get labels
+ _labels = prometheus_label_factory(
+ supported_enum_labels=self.get_labels_for_metric(
+ metric_name="litellm_deployment_state"
+ ),
+ enum_values=UserAPIKeyLabelValues(
+ litellm_model_name=litellm_model_name,
+ model_id=model_id,
+ api_base=api_base,
+ api_provider=api_provider,
+ ),
+ )
+ self.litellm_deployment_state.labels(**_labels).set(state)
def set_deployment_healthy(
self,
@@ -2247,8 +2248,10 @@ def prometheus_label_factory(
if enum_values.custom_metadata_labels is not None:
for key, value in enum_values.custom_metadata_labels.items():
- if key in supported_enum_labels:
- filtered_labels[key] = value
+ # check sanitized key
+ sanitized_key = _sanitize_prometheus_label_name(key)
+ if sanitized_key in supported_enum_labels:
+ filtered_labels[sanitized_key] = value
# Add custom tags if configured
if enum_values.tags is not None:
@@ -2281,9 +2284,12 @@ def get_custom_labels_from_metadata(metadata: dict) -> Dict[str, str]:
keys_parts = key.split(".")
# Traverse through the dictionary using the parts
- value = metadata
+ value: Any = metadata
for part in keys_parts:
- value = value.get(part, None) # Get the value, return None if not found
+ if isinstance(value, dict):
+ value = value.get(part, None) # Get the value, return None if not found
+ else:
+ value = None
if value is None:
break
@@ -2293,10 +2299,62 @@ def get_custom_labels_from_metadata(metadata: dict) -> Dict[str, str]:
return result
+def _tag_matches_wildcard_configured_pattern(
+ tags: List[str], configured_tag: str
+) -> bool:
+ """
+ Check if any of the request tags matches a wildcard configured pattern
+
+ Args:
+ tags: List[str] - The request tags
+ configured_tag: str - The configured tag
+
+ Returns:
+ bool - True if any of the request tags matches the configured tag, False otherwise
+
+ e.g.
+ tags = ["User-Agent: curl/7.68.0", "User-Agent: python-requests/2.28.1", "prod"]
+ configured_tag = "User-Agent: curl/*"
+ _tag_matches_wildcard_configured_pattern(tags=tags, configured_tag=configured_tag) # True
+
+ configured_tag = "User-Agent: python-requests/*"
+ _tag_matches_wildcard_configured_pattern(tags=tags, configured_tag=configured_tag) # True
+
+ configured_tag = "gm"
+ _tag_matches_wildcard_configured_pattern(tags=tags, configured_tag=configured_tag) # False
+ """
+ import re
+
+ from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
+
+ pattern_router = PatternMatchRouter()
+ regex_pattern = pattern_router._pattern_to_regex(configured_tag)
+ return any(re.match(pattern=regex_pattern, string=tag) for tag in tags)
+
+
def get_custom_labels_from_tags(tags: List[str]) -> Dict[str, str]:
"""
- Get custom labels from tags based on admin configuration
+ Get custom labels from tags based on admin configuration.
+
+ Supports both exact matches and wildcard patterns:
+ - Exact match: "prod" matches "prod" exactly
+ - Wildcard pattern: "User-Agent: curl/*" matches "User-Agent: curl/7.68.0"
+
+ Reuses PatternMatchRouter for wildcard pattern matching.
+
+ Returns dict of label_name: "true" if the tag matches the configured tag, "false" otherwise
+
+ {
+ "tag_User-Agent_curl": "true",
+ "tag_User-Agent_python_requests": "false",
+ "tag_Environment_prod": "true",
+ "tag_Environment_dev": "false",
+ "tag_Service_api_gateway_v2": "true",
+ "tag_Service_web_app_v1": "false",
+ }
"""
+
+ from litellm.router_utils.pattern_match_deployments import PatternMatchRouter
from litellm.types.integrations.prometheus import _sanitize_prometheus_label_name
configured_tags = litellm.custom_prometheus_tags
@@ -2304,16 +2362,24 @@ def get_custom_labels_from_tags(tags: List[str]) -> Dict[str, str]:
return {}
result: Dict[str, str] = {}
+ pattern_router = PatternMatchRouter()
- # Map each configured tag to its presence in the request tags
for configured_tag in configured_tags:
- # Create a safe prometheus label name
label_name = _sanitize_prometheus_label_name(f"tag_{configured_tag}")
- # Check if this tag is present in the request tags
+ # Check for exact match first (backwards compatibility)
if configured_tag in tags:
result[label_name] = "true"
- else:
- result[label_name] = "false"
+ continue
+
+ # Use PatternMatchRouter for wildcard pattern matching
+ if "*" in configured_tag and _tag_matches_wildcard_configured_pattern(
+ tags=tags, configured_tag=configured_tag
+ ):
+ result[label_name] = "true"
+ continue
+
+ # No match found
+ result[label_name] = "false"
return result
diff --git a/enterprise/litellm_enterprise/proxy/auth/route_checks.py b/enterprise/litellm_enterprise/proxy/auth/route_checks.py
index 1d4bfc664d5..6cce781faf3 100644
--- a/enterprise/litellm_enterprise/proxy/auth/route_checks.py
+++ b/enterprise/litellm_enterprise/proxy/auth/route_checks.py
@@ -20,7 +20,6 @@ class EnterpriseRouteChecks:
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"🚨🚨🚨 DISABLING LLM API ENDPOINTS is an Enterprise feature\n🚨 {CommonProxyErrors.not_premium_user.value}",
)
- return False
return get_secret_bool("DISABLE_LLM_API_ENDPOINTS") is True
diff --git a/enterprise/litellm_enterprise/proxy/auth/user_api_key_auth.py b/enterprise/litellm_enterprise/proxy/auth/user_api_key_auth.py
index 35b4c2a1f3b..dc9fdeb78e2 100644
--- a/enterprise/litellm_enterprise/proxy/auth/user_api_key_auth.py
+++ b/enterprise/litellm_enterprise/proxy/auth/user_api_key_auth.py
@@ -3,7 +3,7 @@ from typing import Any, Optional
from fastapi import Request
from litellm._logging import verbose_proxy_logger
-from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy._types import ProxyException, UserAPIKeyAuth
async def enterprise_custom_auth(
@@ -24,6 +24,8 @@ async def enterprise_custom_auth(
elif custom_auth_settings["mode"] == "auto":
try:
return await user_custom_auth(request, api_key)
+ except ProxyException as e:
+ raise e
except Exception as e:
verbose_proxy_logger.debug(
f"Error in custom auth, checking litellm auth: {e}"
diff --git a/enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py b/enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
index 6edd198cd8e..4b1bb024ac6 100644
--- a/enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
+++ b/enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py
@@ -2,7 +2,7 @@
Polls LiteLLM_ManagedObjectTable to check if the batch job is complete, and if the cost has been tracked.
"""
-import uuid
+from litellm._uuid import uuid
from datetime import datetime
from typing import TYPE_CHECKING, Optional, cast
diff --git a/enterprise/litellm_enterprise/proxy/guardrails/endpoints.py b/enterprise/litellm_enterprise/proxy/guardrails/endpoints.py
index cdf86dcea67..8b42b2549cd 100644
--- a/enterprise/litellm_enterprise/proxy/guardrails/endpoints.py
+++ b/enterprise/litellm_enterprise/proxy/guardrails/endpoints.py
@@ -36,6 +36,8 @@ async def apply_guardrail(
if active_guardrail is None:
raise Exception(f"Guardrail {request.guardrail_name} not found")
- return await active_guardrail.apply_guardrail(
+ response_text = await active_guardrail.apply_guardrail(
text=request.text, language=request.language, entities=request.entities
)
+
+ return ApplyGuardrailResponse(response_text=response_text)
diff --git a/enterprise/litellm_enterprise/proxy/hooks/managed_files.py b/enterprise/litellm_enterprise/proxy/hooks/managed_files.py
index a2c788a8f21..e2963f8fb87 100644
--- a/enterprise/litellm_enterprise/proxy/hooks/managed_files.py
+++ b/enterprise/litellm_enterprise/proxy/hooks/managed_files.py
@@ -4,7 +4,7 @@
import asyncio
import base64
import json
-import uuid
+from litellm._uuid import uuid
from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Union, cast
from fastapi import HTTPException
@@ -290,6 +290,7 @@ class _PROXY_LiteLLMManagedFiles(CustomLogger, BaseFileEndpoints):
"aretrieve_fine_tuning_job",
"alist_fine_tuning_jobs",
"acancel_fine_tuning_job",
+ "mcp_call",
],
) -> Union[Exception, str, Dict, None]:
"""
diff --git a/enterprise/litellm_enterprise/proxy/management_endpoints/internal_user_endpoints.py b/enterprise/litellm_enterprise/proxy/management_endpoints/internal_user_endpoints.py
index d17946171bb..2f53f9e9281 100644
--- a/enterprise/litellm_enterprise/proxy/management_endpoints/internal_user_endpoints.py
+++ b/enterprise/litellm_enterprise/proxy/management_endpoints/internal_user_endpoints.py
@@ -2,6 +2,7 @@
Enterprise internal user management endpoints
"""
+
from fastapi import APIRouter, Depends, HTTPException
from litellm.proxy._types import UserAPIKeyAuth
@@ -21,7 +22,7 @@ async def available_enterprise_users(
"""
For keys with `max_users` set, return the list of users that are allowed to use the key.
"""
- from litellm.proxy._types import CommonProxyErrors
+ from litellm.proxy._types import CommonProxyErrors, EnterpriseLicenseData
from litellm.proxy.proxy_server import (
premium_user,
premium_user_data,
@@ -34,10 +35,14 @@ async def available_enterprise_users(
detail={"error": CommonProxyErrors.db_not_connected_error.value},
)
- if premium_user is None:
- raise HTTPException(
- status_code=500, detail={"error": CommonProxyErrors.not_premium_user.value}
- )
+ if not premium_user:
+ # check if SSO is enabled - show 5 user limit
+ from litellm.proxy.auth.auth_utils import _has_user_setup_sso
+
+ if _has_user_setup_sso():
+ premium_user_data = EnterpriseLicenseData(
+ max_users=5,
+ )
# Count number of rows in LiteLLM_UserTable
user_count = await prisma_client.db.litellm_usertable.count()
diff --git a/enterprise/litellm_enterprise/proxy/vector_stores/endpoints.py b/enterprise/litellm_enterprise/proxy/vector_stores/endpoints.py
index 43bdfa3844f..bb4b546b8d3 100644
--- a/enterprise/litellm_enterprise/proxy/vector_stores/endpoints.py
+++ b/enterprise/litellm_enterprise/proxy/vector_stores/endpoints.py
@@ -11,7 +11,7 @@ All /vector_store management endpoints
import copy
from typing import List, Optional
-from fastapi import APIRouter, Depends, HTTPException, Request, Response
+from fastapi import APIRouter, Depends, HTTPException
import litellm
from litellm._logging import verbose_proxy_logger
diff --git a/enterprise/litellm_enterprise/types/proxy/proxy_server.py b/enterprise/litellm_enterprise/types/proxy/proxy_server.py
index 497be59c4b9..f1a1f2639ed 100644
--- a/enterprise/litellm_enterprise/types/proxy/proxy_server.py
+++ b/enterprise/litellm_enterprise/types/proxy/proxy_server.py
@@ -1,4 +1,6 @@
-from typing import Literal, TypedDict
+from typing import Literal
+
+from typing_extensions import TypedDict
class CustomAuthSettings(TypedDict):
diff --git a/enterprise/pyproject.toml b/enterprise/pyproject.toml
index 182001e3d50..1d1fa64549c 100644
--- a/enterprise/pyproject.toml
+++ b/enterprise/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-enterprise"
-version = "0.1.16"
+version = "0.1.20"
description = "Package for LiteLLM Enterprise features"
authors = ["BerriAI"]
readme = "README.md"
@@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
-version = "0.1.16"
+version = "0.1.20"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-enterprise==",
diff --git a/git_model_armor.py b/git_model_armor.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/litellm-js/spend-logs/package-lock.json b/litellm-js/spend-logs/package-lock.json
index 2f9e2248351..95f8acdec3a 100644
--- a/litellm-js/spend-logs/package-lock.json
+++ b/litellm-js/spend-logs/package-lock.json
@@ -6,7 +6,7 @@
"": {
"dependencies": {
"@hono/node-server": "^1.10.1",
- "hono": "^4.6.5"
+ "hono": "^4.9.7"
},
"devDependencies": {
"@types/node": "^20.11.17",
@@ -463,9 +463,10 @@
}
},
"node_modules/hono": {
- "version": "4.6.5",
- "resolved": "https://registry.npmjs.org/hono/-/hono-4.6.5.tgz",
- "integrity": "sha512-qsmN3V5fgtwdKARGLgwwHvcdLKursMd+YOt69eGpl1dUCJb8mCd7hZfyZnBYjxCegBG7qkJRQRUy2oO25yHcyQ==",
+ "version": "4.9.7",
+ "resolved": "https://registry.npmjs.org/hono/-/hono-4.9.7.tgz",
+ "integrity": "sha512-t4Te6ERzIaC48W3x4hJmBwgNlLhmiEdEE5ViYb02ffw4ignHNHa5IBtPjmbKstmtKa8X6C35iWwK4HaqvrzG9w==",
+ "license": "MIT",
"engines": {
"node": ">=16.9.0"
}
diff --git a/litellm-js/spend-logs/package.json b/litellm-js/spend-logs/package.json
index 9e51f1018a6..5370f7a0eca 100644
--- a/litellm-js/spend-logs/package.json
+++ b/litellm-js/spend-logs/package.json
@@ -4,7 +4,7 @@
},
"dependencies": {
"@hono/node-server": "^1.10.1",
- "hono": "^4.6.5"
+ "hono": "^4.9.7"
},
"devDependencies": {
"@types/node": "^20.11.17",
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.14-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.14-py3-none-any.whl
new file mode 100644
index 00000000000..fc160319c07
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.14-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.14.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.14.tar.gz
new file mode 100644
index 00000000000..b5d3f317b96
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.14.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.16-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.16-py3-none-any.whl
new file mode 100644
index 00000000000..ce275d59451
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.16-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.16.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.16.tar.gz
new file mode 100644
index 00000000000..16e8acf09ae
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.16.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.17-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.17-py3-none-any.whl
new file mode 100644
index 00000000000..71160d51a7e
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.17-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.17.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.17.tar.gz
new file mode 100644
index 00000000000..7bab2b9c8b6
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.17.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.18-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.18-py3-none-any.whl
new file mode 100644
index 00000000000..fca66b532ff
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.18-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.18.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.18.tar.gz
new file mode 100644
index 00000000000..ddd00e8439e
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.18.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.19-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.19-py3-none-any.whl
new file mode 100644
index 00000000000..c035bb44215
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.19-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.19.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.19.tar.gz
new file mode 100644
index 00000000000..85069c622b0
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.19.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.20-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.20-py3-none-any.whl
new file mode 100644
index 00000000000..0a94ef6ff62
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.20-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.20.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.20.tar.gz
new file mode 100644
index 00000000000..1562aacc22b
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.20.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.21-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.21-py3-none-any.whl
new file mode 100644
index 00000000000..75baeb5c575
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.21-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.21.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.21.tar.gz
new file mode 100644
index 00000000000..bc934024b3a
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.21.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.22-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.22-py3-none-any.whl
new file mode 100644
index 00000000000..0194c9148aa
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.22-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.22.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.22.tar.gz
new file mode 100644
index 00000000000..17cb242663c
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.22.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.23-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.23-py3-none-any.whl
new file mode 100644
index 00000000000..4220fad36c4
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.23-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.23.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.23.tar.gz
new file mode 100644
index 00000000000..ceccaacda43
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.23.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.25-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.25-py3-none-any.whl
new file mode 100644
index 00000000000..8e0f50c2121
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.25-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.25.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.25.tar.gz
new file mode 100644
index 00000000000..2565f68a2b8
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.25.tar.gz differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.26-py3-none-any.whl b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.26-py3-none-any.whl
new file mode 100644
index 00000000000..47b31557f88
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.26-py3-none-any.whl differ
diff --git a/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.26.tar.gz b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.26.tar.gz
new file mode 100644
index 00000000000..62fd4733428
Binary files /dev/null and b/litellm-proxy-extras/dist/litellm_proxy_extras-0.2.26.tar.gz differ
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20250507161526_add_mcp_table_to_db/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250507161526_add_mcp_table_to_db/migration.sql
index fb0cb661a75..6b8adc6e7e8 100644
--- a/litellm-proxy-extras/litellm_proxy_extras/migrations/20250507161526_add_mcp_table_to_db/migration.sql
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250507161526_add_mcp_table_to_db/migration.sql
@@ -15,13 +15,3 @@ CREATE TABLE "LiteLLM_MCPServerTable" (
CONSTRAINT "LiteLLM_MCPServerTable_pkey" PRIMARY KEY ("server_id")
);
--- Migration for existing tables: rename alias to server_name if upgrading
-DO $$
-BEGIN
- IF EXISTS (SELECT 1 FROM information_schema.columns WHERE table_name = 'LiteLLM_MCPServerTable' AND column_name = 'alias') THEN
- ALTER TABLE "LiteLLM_MCPServerTable" RENAME COLUMN "alias" TO "server_name";
- END IF;
-END $$;
--- Migration for existing tables: add alias column if upgrading
-ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN IF NOT EXISTS "alias" TEXT;
-
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20250507161527_add_health_check_fields_to_mcp_servers/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250507161527_add_health_check_fields_to_mcp_servers/migration.sql
new file mode 100644
index 00000000000..d5c206d1929
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250507161527_add_health_check_fields_to_mcp_servers/migration.sql
@@ -0,0 +1,4 @@
+-- Add health check fields to MCP server table
+ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "status" TEXT DEFAULT 'unknown';
+ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "last_health_check" TIMESTAMP(3);
+ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "health_check_error" TEXT;
\ No newline at end of file
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20250802162330_prompt_table/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250802162330_prompt_table/migration.sql
new file mode 100644
index 00000000000..e5c00ef4adb
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250802162330_prompt_table/migration.sql
@@ -0,0 +1,15 @@
+-- CreateTable
+CREATE TABLE "LiteLLM_PromptTable" (
+ "id" TEXT NOT NULL,
+ "prompt_id" TEXT NOT NULL,
+ "litellm_params" JSONB NOT NULL,
+ "prompt_info" JSONB,
+ "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ "updated_at" TIMESTAMP(3) NOT NULL,
+
+ CONSTRAINT "LiteLLM_PromptTable_pkey" PRIMARY KEY ("id")
+);
+
+-- CreateIndex
+CREATE UNIQUE INDEX "LiteLLM_PromptTable_prompt_id_key" ON "LiteLLM_PromptTable"("prompt_id");
+
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20250806095134_rename_alias_to_server_name_mcp_table/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250806095134_rename_alias_to_server_name_mcp_table/migration.sql
new file mode 100644
index 00000000000..11463d44b0e
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250806095134_rename_alias_to_server_name_mcp_table/migration.sql
@@ -0,0 +1,10 @@
+-- Migration for existing tables: rename alias to server_name if upgrading
+DO $$
+BEGIN
+ IF EXISTS (SELECT 1 FROM information_schema.columns WHERE table_name = 'LiteLLM_MCPServerTable' AND column_name = 'alias') THEN
+ ALTER TABLE "LiteLLM_MCPServerTable" RENAME COLUMN "alias" TO "server_name";
+ END IF;
+END $$;
+
+-- Migration for existing tables: add alias column if upgrading
+ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN IF NOT EXISTS "alias" TEXT;
\ No newline at end of file
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20250918083359_drop_spec_version_column_from_mcp_table/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250918083359_drop_spec_version_column_from_mcp_table/migration.sql
new file mode 100644
index 00000000000..472e2ea1e0c
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250918083359_drop_spec_version_column_from_mcp_table/migration.sql
@@ -0,0 +1,8 @@
+/*
+ Warnings:
+
+ - You are about to drop the column `spec_version` on the `LiteLLM_MCPServerTable` table. All the data in the column will be lost.
+
+*/
+-- AlterTable
+ALTER TABLE "LiteLLM_MCPServerTable" DROP COLUMN "spec_version";
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20250926194702_unnamed_migration/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250926194702_unnamed_migration/migration.sql
new file mode 100644
index 00000000000..ea28db19662
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20250926194702_unnamed_migration/migration.sql
@@ -0,0 +1,7 @@
+-- AlterTable
+ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN "auto_rotate" BOOLEAN DEFAULT false,
+ADD COLUMN "key_rotation_at" TIMESTAMP(3),
+ADD COLUMN "last_rotation_at" TIMESTAMP(3),
+ADD COLUMN "rotation_count" INTEGER DEFAULT 0,
+ADD COLUMN "rotation_interval" TEXT;
+
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20251003165142_add_allowed_tools_to_mcp/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251003165142_add_allowed_tools_to_mcp/migration.sql
new file mode 100644
index 00000000000..bdac1e42bc2
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251003165142_add_allowed_tools_to_mcp/migration.sql
@@ -0,0 +1,3 @@
+-- AlterTable
+ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "allowed_tools" TEXT[] DEFAULT ARRAY[]::TEXT[];
+
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20251003190954_extra_headers_to_mcp_table/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251003190954_extra_headers_to_mcp_table/migration.sql
new file mode 100644
index 00000000000..1cfcf062eb1
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251003190954_extra_headers_to_mcp_table/migration.sql
@@ -0,0 +1,3 @@
+-- AlterTable
+ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN "extra_headers" TEXT[] DEFAULT ARRAY[]::TEXT[];
+
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20251006143948_add_mcp_tool_permissions/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251006143948_add_mcp_tool_permissions/migration.sql
new file mode 100644
index 00000000000..51f3be87582
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251006143948_add_mcp_tool_permissions/migration.sql
@@ -0,0 +1,3 @@
+-- AlterTable
+ALTER TABLE "LiteLLM_ObjectPermissionTable" ADD COLUMN "mcp_tool_permissions" JSONB;
+
diff --git a/litellm-proxy-extras/litellm_proxy_extras/migrations/20251011084309_add_tag_table/migration.sql b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251011084309_add_tag_table/migration.sql
new file mode 100644
index 00000000000..541c70c7e48
--- /dev/null
+++ b/litellm-proxy-extras/litellm_proxy_extras/migrations/20251011084309_add_tag_table/migration.sql
@@ -0,0 +1,18 @@
+-- CreateTable
+CREATE TABLE "LiteLLM_TagTable" (
+ "tag_name" TEXT NOT NULL,
+ "description" TEXT,
+ "models" TEXT[],
+ "model_info" JSONB,
+ "spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0,
+ "budget_id" TEXT,
+ "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ "created_by" TEXT,
+ "updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+
+ CONSTRAINT "LiteLLM_TagTable_pkey" PRIMARY KEY ("tag_name")
+);
+
+-- AddForeignKey
+ALTER TABLE "LiteLLM_TagTable" ADD CONSTRAINT "LiteLLM_TagTable_budget_id_fkey" FOREIGN KEY ("budget_id") REFERENCES "LiteLLM_BudgetTable"("budget_id") ON DELETE SET NULL ON UPDATE CASCADE;
+
diff --git a/litellm-proxy-extras/litellm_proxy_extras/schema.prisma b/litellm-proxy-extras/litellm_proxy_extras/schema.prisma
index ef29d7c9bfa..a13af1afc5f 100644
--- a/litellm-proxy-extras/litellm_proxy_extras/schema.prisma
+++ b/litellm-proxy-extras/litellm_proxy_extras/schema.prisma
@@ -25,6 +25,7 @@ model LiteLLM_BudgetTable {
organization LiteLLM_OrganizationTable[] // multiple orgs can have the same budget
keys LiteLLM_VerificationToken[] // multiple keys can have the same budget
end_users LiteLLM_EndUserTable[] // multiple end-users can have the same budget
+ tags LiteLLM_TagTable[] // multiple tags can have the same budget
team_membership LiteLLM_TeamMembership[] // budgets of Users within a Team
organization_membership LiteLLM_OrganizationMembership[] // budgets of Users within a Organization
}
@@ -156,6 +157,7 @@ model LiteLLM_ObjectPermissionTable {
object_permission_id String @id @default(uuid())
mcp_servers String[] @default([])
mcp_access_groups String[] @default([])
+ mcp_tool_permissions Json? // Tool-level permissions for MCP servers. Format: {"server_id": ["tool_name_1", "tool_name_2"]}
vector_stores String[] @default([])
teams LiteLLM_TeamTable[]
verification_tokens LiteLLM_VerificationToken[]
@@ -171,7 +173,6 @@ model LiteLLM_MCPServerTable {
description String?
url String?
transport String @default("sse")
- spec_version String @default("2025-03-26")
auth_type String?
created_at DateTime? @default(now()) @map("created_at")
created_by String?
@@ -179,6 +180,12 @@ model LiteLLM_MCPServerTable {
updated_by String?
mcp_info Json? @default("{}")
mcp_access_groups String[]
+ allowed_tools String[] @default([])
+ extra_headers String[] @default([])
+ // Health check status
+ status String? @default("unknown")
+ last_health_check DateTime?
+ health_check_error String?
// Stdio-specific fields
command String?
args String[] @default([])
@@ -218,6 +225,11 @@ model LiteLLM_VerificationToken {
created_by String?
updated_at DateTime? @default(now()) @updatedAt @map("updated_at")
updated_by String?
+ rotation_count Int? @default(0) // Number of times key has been rotated
+ auto_rotate Boolean? @default(false) // Whether this key should be auto-rotated
+ rotation_interval String? // How often to rotate (e.g., "30d", "90d")
+ last_rotation_at DateTime? // When this key was last rotated
+ key_rotation_at DateTime? // When this key should next be rotated
litellm_budget_table LiteLLM_BudgetTable? @relation(fields: [budget_id], references: [budget_id])
litellm_organization_table LiteLLM_OrganizationTable? @relation(fields: [organization_id], references: [organization_id])
object_permission LiteLLM_ObjectPermissionTable? @relation(fields: [object_permission_id], references: [object_permission_id])
@@ -234,6 +246,20 @@ model LiteLLM_EndUserTable {
blocked Boolean @default(false)
}
+// Track tags with budgets and spend
+model LiteLLM_TagTable {
+ tag_name String @id
+ description String?
+ models String[]
+ model_info Json? // maps model_id to model_name
+ spend Float @default(0.0)
+ budget_id String?
+ litellm_budget_table LiteLLM_BudgetTable? @relation(fields: [budget_id], references: [budget_id])
+ created_at DateTime @default(now()) @map("created_at")
+ created_by String?
+ updated_at DateTime @default(now()) @updatedAt @map("updated_at")
+}
+
// store proxy config.yaml
model LiteLLM_Config {
param_name String @id
@@ -516,6 +542,16 @@ model LiteLLM_GuardrailsTable {
updated_at DateTime @updatedAt
}
+// Prompt table for storing prompt configurations
+model LiteLLM_PromptTable {
+ id String @id @default(uuid())
+ prompt_id String @unique
+ litellm_params Json
+ prompt_info Json?
+ created_at DateTime @default(now())
+ updated_at DateTime @updatedAt
+}
+
model LiteLLM_HealthCheckTable {
health_check_id String @id @default(uuid())
model_name String
diff --git a/litellm-proxy-extras/litellm_proxy_extras/utils.py b/litellm-proxy-extras/litellm_proxy_extras/utils.py
index 21c9131887b..73065b050b7 100644
--- a/litellm-proxy-extras/litellm_proxy_extras/utils.py
+++ b/litellm-proxy-extras/litellm_proxy_extras/utils.py
@@ -131,7 +131,9 @@ class ProxyExtrasDBManager:
)
@staticmethod
- def _resolve_all_migrations(migrations_dir: str, schema_path: str):
+ def _resolve_all_migrations(
+ migrations_dir: str, schema_path: str, mark_all_applied: bool = True
+ ):
"""
1. Compare the current database state to schema.prisma and generate a migration for the diff.
2. Run prisma migrate deploy to apply any pending migrations.
@@ -210,6 +212,8 @@ class ProxyExtrasDBManager:
logger.warning("Migration diff application timed out.")
# 3. Mark all migrations as applied
+ if not mark_all_applied:
+ return
migration_names = ProxyExtrasDBManager._get_migration_names(migrations_dir)
logger.info(f"Resolving {len(migration_names)} migrations")
for migration_name in migration_names:
@@ -243,7 +247,6 @@ class ProxyExtrasDBManager:
bool: True if setup was successful, False otherwise
"""
schema_path = ProxyExtrasDBManager._get_prisma_dir() + "/schema.prisma"
- use_migrate = str_to_bool(os.getenv("USE_PRISMA_MIGRATE")) or use_migrate
for attempt in range(4):
original_dir = os.getcwd()
migrations_dir = ProxyExtrasDBManager._get_prisma_dir()
@@ -264,6 +267,13 @@ class ProxyExtrasDBManager:
logger.info(f"prisma migrate deploy stdout: {result.stdout}")
logger.info("prisma migrate deploy completed")
+
+ # Run sanity check to ensure DB matches schema
+ logger.info("Running post-migration sanity check...")
+ ProxyExtrasDBManager._resolve_all_migrations(
+ migrations_dir, schema_path, mark_all_applied=False
+ )
+ logger.info("✅ Post-migration sanity check completed")
return True
except subprocess.CalledProcessError as e:
logger.info(f"prisma db error: {e.stderr}, e: {e.stdout}")
@@ -299,7 +309,7 @@ class ProxyExtrasDBManager:
and "database schema is not empty" in e.stderr
):
logger.info(
- "Database schema is not empty, creating baseline migration"
+ "Database schema is not empty, creating baseline migration. In read-only file system, please set an environment variable `LITELLM_MIGRATION_DIR` to a writable directory to enable migrations. Learn more - https://docs.litellm.ai/docs/proxy/prod#read-only-file-system"
)
ProxyExtrasDBManager._create_baseline_migration(schema_path)
logger.info(
diff --git a/litellm-proxy-extras/migration_runbook.md b/litellm-proxy-extras/migration_runbook.md
new file mode 100644
index 00000000000..93948f24b13
--- /dev/null
+++ b/litellm-proxy-extras/migration_runbook.md
@@ -0,0 +1,50 @@
+# Database Migration Runbook
+
+This is a runbook for creating and running database migrations for the LiteLLM proxy. For use for litellm engineers only.
+
+## Quick Start
+
+```bash
+# Install deps (one time)
+pip install testing.postgresql
+brew install postgresql@14 # macOS
+
+# Add to PATH
+export PATH="/opt/homebrew/opt/postgresql@14/bin:$PATH"
+
+# Run migration
+python ci_cd/run_migration.py "your_migration_name"
+```
+
+## What It Does
+
+1. Creates temp PostgreSQL DB
+2. Applies existing migrations
+3. Compares with `schema.prisma`
+4. Generates new migration if changes found
+
+## Common Fixes
+
+**Missing testing module:**
+```bash
+pip install testing.postgresql
+```
+
+**initdb not found:**
+```bash
+brew install postgresql@14
+export PATH="/opt/homebrew/opt/postgresql@14/bin:$PATH"
+```
+
+**Empty migration directory error:**
+```bash
+rm -rf litellm-proxy-extras/litellm_proxy_extras/migrations/[empty_dir]
+```
+
+## Rules
+
+- Update `schema.prisma` first
+- Review generated SQL before committing
+- Use descriptive migration names
+- Never edit existing migration files
+- Commit schema + migration together
diff --git a/litellm-proxy-extras/pyproject.toml b/litellm-proxy-extras/pyproject.toml
index db24e4f42a1..8af7c212f52 100644
--- a/litellm-proxy-extras/pyproject.toml
+++ b/litellm-proxy-extras/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "litellm-proxy-extras"
-version = "0.2.12"
+version = "0.2.27"
description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
authors = ["BerriAI"]
readme = "README.md"
@@ -22,7 +22,7 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.commitizen]
-version = "0.2.12"
+version = "0.2.27"
version_files = [
"pyproject.toml:version",
"../requirements.txt:litellm-proxy-extras==",
diff --git a/litellm/__init__.py b/litellm/__init__.py
index 412f552e9c2..e461c88efd6 100644
--- a/litellm/__init__.py
+++ b/litellm/__init__.py
@@ -5,7 +5,19 @@ warnings.filterwarnings("ignore", message=".*conflict with protected namespace.*
### INIT VARIABLES ####################
import threading
import os
-from typing import Callable, List, Optional, Dict, Union, Any, Literal, get_args
+from typing import (
+ Callable,
+ List,
+ Optional,
+ Dict,
+ Union,
+ Any,
+ Literal,
+ get_args,
+ TYPE_CHECKING,
+)
+from litellm.types.integrations.datadog_llm_obs import DatadogLLMObsInitParams
+from litellm.types.integrations.datadog import DatadogInitParams
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.caching.caching import Cache, DualCache, RedisCache, InMemoryCache
from litellm.caching.llm_caching_handler import LLMClientCache
@@ -49,6 +61,7 @@ from litellm.constants import (
empower_models,
together_ai_models,
baseten_models,
+ WANDB_MODELS,
REPEATED_STREAMING_CHUNK_LIMIT,
request_timeout,
open_ai_embedding_models,
@@ -56,10 +69,17 @@ from litellm.constants import (
bedrock_embedding_models,
known_tokenizer_config,
BEDROCK_INVOKE_PROVIDERS_LITERAL,
+ BEDROCK_EMBEDDING_PROVIDERS_LITERAL,
+ BEDROCK_CONVERSE_MODELS,
DEFAULT_MAX_TOKENS,
DEFAULT_SOFT_BUDGET,
DEFAULT_ALLOWED_FAILS,
)
+from litellm.integrations.dotprompt import (
+ global_prompt_manager,
+ global_prompt_directory,
+ set_global_prompt_directory,
+)
from litellm.types.guardrails import GuardrailItem
from litellm.types.secret_managers.main import (
KeyManagementSystem,
@@ -70,6 +90,7 @@ from litellm.types.proxy.management_endpoints.ui_sso import (
LiteLLM_UpperboundKeyGenerateParams,
)
from litellm.types.utils import StandardKeyGenerationConfig, LlmProviders
+from litellm.types.utils import PriorityReservationSettings
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.logging_callback_manager import LoggingCallbackManager
import httpx
@@ -82,7 +103,6 @@ if litellm_mode == "DEV":
# Register async client cleanup to prevent resource leaks
register_async_client_cleanup()
-
####################################################
if set_verbose == True:
_turn_on_debug()
@@ -100,6 +120,7 @@ _custom_logger_compatible_callbacks_literal = Literal[
"logfire",
"literalai",
"dynamic_rate_limiter",
+ "dynamic_rate_limiter_v3",
"langsmith",
"prometheus",
"otel",
@@ -129,7 +150,15 @@ _custom_logger_compatible_callbacks_literal = Literal[
"s3_v2",
"aws_sqs",
"vector_store_pre_call_hook",
+ "dotprompt",
+ "bitbucket",
+ "gitlab",
+ "cloudzero",
+ "posthog",
]
+configured_cold_storage_logger: Optional[
+ _custom_logger_compatible_callbacks_literal
+] = None
logged_real_time_event_types: Optional[Union[List[str], Literal["*"]]] = None
_known_custom_logger_compatible_callbacks: List = list(
get_args(_custom_logger_compatible_callbacks_literal)
@@ -209,13 +238,20 @@ vertex_location: Optional[str] = None
predibase_tenant_id: Optional[str] = None
togetherai_api_key: Optional[str] = None
cloudflare_api_key: Optional[str] = None
+vercel_ai_gateway_key: Optional[str] = None
baseten_key: Optional[str] = None
llama_api_key: Optional[str] = None
aleph_alpha_key: Optional[str] = None
nlp_cloud_key: Optional[str] = None
novita_api_key: Optional[str] = None
snowflake_key: Optional[str] = None
+gradient_ai_api_key: Optional[str] = None
nebius_key: Optional[str] = None
+wandb_key: Optional[str] = None
+heroku_key: Optional[str] = None
+cometapi_key: Optional[str] = None
+ovhcloud_key: Optional[str] = None
+lemonade_key: Optional[str] = None
common_cloud_provider_auth_params: dict = {
"params": ["project", "region_name", "token"],
"providers": ["vertex_ai", "bedrock", "watsonx", "azure", "vertex_ai_beta"],
@@ -253,6 +289,12 @@ blocked_user_list: Optional[Union[str, List]] = None
banned_keywords_list: Optional[Union[str, List]] = None
llm_guard_mode: Literal["all", "key-specific", "request-specific"] = "all"
guardrail_name_config_map: Dict[str, GuardrailItem] = {}
+include_cost_in_streaming_usage: bool = False
+### PROMPTS ####
+from litellm.types.prompts.init_prompts import PromptSpec
+
+prompt_name_config_map: Dict[str, PromptSpec] = {}
+
##################
### PREVIEW FEATURES ###
enable_preview_features: bool = False
@@ -275,7 +317,6 @@ default_in_memory_ttl: Optional[float] = None
default_redis_ttl: Optional[float] = None
default_redis_batch_cache_expiry: Optional[float] = None
model_alias_map: Dict[str, str] = {}
-model_group_alias_map: Dict[str, str] = {}
model_group_settings: Optional["ModelGroupSettings"] = None
max_budget: float = 0.0 # set the max budget across all providers
budget_duration: Optional[
@@ -297,6 +338,8 @@ model_cost_map_url: str = "https://raw.githubusercontent.com/BerriAI/litellm/mai
suppress_debug_info = False
dynamodb_table_name: Optional[str] = None
s3_callback_params: Optional[Dict] = None
+datadog_llm_observability_params: Optional[Union[DatadogLLMObsInitParams, Dict]] = None
+datadog_params: Optional[Union[DatadogInitParams, Dict]] = None
aws_sqs_callback_params: Optional[Dict] = None
generic_logger_headers: Optional[Dict] = None
default_key_generate_params: Optional[Dict] = None
@@ -324,8 +367,11 @@ disable_add_prefix_to_prompt: bool = (
disable_copilot_system_to_assistant: bool = False # If false (default), converts all 'system' role messages to 'assistant' for GitHub Copilot compatibility. Set to true to disable this behavior.
public_model_groups: Optional[List[str]] = None
public_model_groups_links: Dict[str, str] = {}
-#### REQUEST PRIORITIZATION #####
+#### REQUEST PRIORITIZATION #######
priority_reservation: Optional[Dict[str, float]] = None
+priority_reservation_settings: "PriorityReservationSettings" = (
+ PriorityReservationSettings()
+)
######## Networking Settings ########
@@ -387,112 +433,100 @@ def identify(event_details):
####### ADDITIONAL PARAMS ################### configurable params if you use proxy models like Helicone, map spend to org id, etc.
api_base: Optional[str] = None
headers = None
-api_version = None
+api_version: Optional[str] = None
organization = None
project = None
config_path = None
vertex_ai_safety_settings: Optional[dict] = None
-BEDROCK_CONVERSE_MODELS = [
- "anthropic.claude-opus-4-20250514-v1:0",
- "anthropic.claude-sonnet-4-20250514-v1:0",
- "anthropic.claude-3-7-sonnet-20250219-v1:0",
- "anthropic.claude-3-5-haiku-20241022-v1:0",
- "anthropic.claude-3-5-sonnet-20241022-v2:0",
- "anthropic.claude-3-5-sonnet-20240620-v1:0",
- "anthropic.claude-3-opus-20240229-v1:0",
- "anthropic.claude-3-sonnet-20240229-v1:0",
- "anthropic.claude-3-haiku-20240307-v1:0",
- "anthropic.claude-v2",
- "anthropic.claude-v2:1",
- "anthropic.claude-v1",
- "anthropic.claude-instant-v1",
- "ai21.jamba-instruct-v1:0",
- "ai21.jamba-1-5-mini-v1:0",
- "ai21.jamba-1-5-large-v1:0",
- "meta.llama3-70b-instruct-v1:0",
- "meta.llama3-8b-instruct-v1:0",
- "meta.llama3-1-8b-instruct-v1:0",
- "meta.llama3-1-70b-instruct-v1:0",
- "meta.llama3-1-405b-instruct-v1:0",
- "meta.llama3-70b-instruct-v1:0",
- "mistral.mistral-large-2407-v1:0",
- "mistral.mistral-large-2402-v1:0",
- "mistral.mistral-small-2402-v1:0",
- "meta.llama3-2-1b-instruct-v1:0",
- "meta.llama3-2-3b-instruct-v1:0",
- "meta.llama3-2-11b-instruct-v1:0",
- "meta.llama3-2-90b-instruct-v1:0",
-]
####### COMPLETION MODELS ###################
-open_ai_chat_completion_models: List = []
-open_ai_text_completion_models: List = []
-cohere_models: List = []
-cohere_chat_models: List = []
-mistral_chat_models: List = []
-text_completion_codestral_models: List = []
-anthropic_models: List = []
-openrouter_models: List = []
-datarobot_models: List = []
-vertex_language_models: List = []
-vertex_vision_models: List = []
-vertex_chat_models: List = []
-vertex_code_chat_models: List = []
-vertex_ai_image_models: List = []
-vertex_text_models: List = []
-vertex_code_text_models: List = []
-vertex_embedding_models: List = []
-vertex_anthropic_models: List = []
-vertex_llama3_models: List = []
-vertex_ai_ai21_models: List = []
-vertex_mistral_models: List = []
-ai21_models: List = []
-ai21_chat_models: List = []
-nlp_cloud_models: List = []
-aleph_alpha_models: List = []
-bedrock_models: List = []
-bedrock_converse_models: List = BEDROCK_CONVERSE_MODELS
-fireworks_ai_models: List = []
-fireworks_ai_embedding_models: List = []
-deepinfra_models: List = []
-perplexity_models: List = []
-watsonx_models: List = []
-gemini_models: List = []
-xai_models: List = []
-deepseek_models: List = []
-azure_ai_models: List = []
-jina_ai_models: List = []
-voyage_models: List = []
-infinity_models: List = []
-databricks_models: List = []
-cloudflare_models: List = []
-codestral_models: List = []
-friendliai_models: List = []
-featherless_ai_models: List = []
-palm_models: List = []
-groq_models: List = []
-azure_models: List = []
-azure_text_models: List = []
-anyscale_models: List = []
-cerebras_models: List = []
-galadriel_models: List = []
-sambanova_models: List = []
-novita_models: List = []
-assemblyai_models: List = []
-snowflake_models: List = []
-llama_models: List = []
-nscale_models: List = []
-nebius_models: List = []
-nebius_embedding_models: List = []
-deepgram_models: List = []
-elevenlabs_models: List = []
-dashscope_models: List = []
-moonshot_models: List = []
-v0_models: List = []
-morph_models: List = []
-lambda_ai_models: List = []
-hyperbolic_models: List = []
-recraft_models: List = []
+from typing import Set
+
+open_ai_chat_completion_models: Set = set()
+open_ai_text_completion_models: Set = set()
+cohere_models: Set = set()
+cohere_chat_models: Set = set()
+mistral_chat_models: Set = set()
+text_completion_codestral_models: Set = set()
+anthropic_models: Set = set()
+openrouter_models: Set = set()
+datarobot_models: Set = set()
+vertex_language_models: Set = set()
+vertex_vision_models: Set = set()
+vertex_chat_models: Set = set()
+vertex_code_chat_models: Set = set()
+vertex_ai_image_models: Set = set()
+vertex_ai_video_models: Set = set()
+vertex_text_models: Set = set()
+vertex_code_text_models: Set = set()
+vertex_embedding_models: Set = set()
+vertex_anthropic_models: Set = set()
+vertex_llama3_models: Set = set()
+vertex_deepseek_models: Set = set()
+vertex_ai_ai21_models: Set = set()
+vertex_mistral_models: Set = set()
+vertex_openai_models: Set = set()
+ai21_models: Set = set()
+ai21_chat_models: Set = set()
+nlp_cloud_models: Set = set()
+aleph_alpha_models: Set = set()
+bedrock_models: Set = set()
+bedrock_converse_models: Set = set(BEDROCK_CONVERSE_MODELS)
+fireworks_ai_models: Set = set()
+fireworks_ai_embedding_models: Set = set()
+deepinfra_models: Set = set()
+perplexity_models: Set = set()
+watsonx_models: Set = set()
+gemini_models: Set = set()
+xai_models: Set = set()
+deepseek_models: Set = set()
+azure_ai_models: Set = set()
+jina_ai_models: Set = set()
+voyage_models: Set = set()
+infinity_models: Set = set()
+heroku_models: Set = set()
+databricks_models: Set = set()
+cloudflare_models: Set = set()
+codestral_models: Set = set()
+friendliai_models: Set = set()
+featherless_ai_models: Set = set()
+palm_models: Set = set()
+groq_models: Set = set()
+azure_models: Set = set()
+azure_text_models: Set = set()
+anyscale_models: Set = set()
+cerebras_models: Set = set()
+galadriel_models: Set = set()
+nvidia_nim_models: Set = set()
+sambanova_models: Set = set()
+sambanova_embedding_models: Set = set()
+novita_models: Set = set()
+assemblyai_models: Set = set()
+snowflake_models: Set = set()
+gradient_ai_models: Set = set()
+llama_models: Set = set()
+nscale_models: Set = set()
+nebius_models: Set = set()
+nebius_embedding_models: Set = set()
+aiml_models: Set = set()
+deepgram_models: Set = set()
+elevenlabs_models: Set = set()
+dashscope_models: Set = set()
+moonshot_models: Set = set()
+v0_models: Set = set()
+morph_models: Set = set()
+lambda_ai_models: Set = set()
+hyperbolic_models: Set = set()
+recraft_models: Set = set()
+cometapi_models: Set = set()
+oci_models: Set = set()
+vercel_ai_gateway_models: Set = set()
+volcengine_models: Set = set()
+wandb_models: Set = set(WANDB_MODELS)
+ovhcloud_models: Set = set()
+ovhcloud_embedding_models: Set = set()
+lemonade_models: Set = set()
+
def is_bedrock_pricing_only_model(key: str) -> bool:
"""
@@ -532,155 +566,190 @@ def add_known_models():
if value.get("litellm_provider") == "openai" and not is_openai_finetune_model(
key
):
- open_ai_chat_completion_models.append(key)
+ open_ai_chat_completion_models.add(key)
elif value.get("litellm_provider") == "text-completion-openai":
- open_ai_text_completion_models.append(key)
+ open_ai_text_completion_models.add(key)
elif value.get("litellm_provider") == "azure_text":
- azure_text_models.append(key)
+ azure_text_models.add(key)
elif value.get("litellm_provider") == "cohere":
- cohere_models.append(key)
+ cohere_models.add(key)
elif value.get("litellm_provider") == "cohere_chat":
- cohere_chat_models.append(key)
+ cohere_chat_models.add(key)
elif value.get("litellm_provider") == "mistral":
- mistral_chat_models.append(key)
+ mistral_chat_models.add(key)
elif value.get("litellm_provider") == "anthropic":
- anthropic_models.append(key)
+ anthropic_models.add(key)
elif value.get("litellm_provider") == "empower":
- empower_models.append(key)
+ empower_models.add(key)
elif value.get("litellm_provider") == "openrouter":
- openrouter_models.append(key)
+ openrouter_models.add(key)
+ elif value.get("litellm_provider") == "vercel_ai_gateway":
+ vercel_ai_gateway_models.add(key)
elif value.get("litellm_provider") == "datarobot":
- datarobot_models.append(key)
+ datarobot_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-text-models":
- vertex_text_models.append(key)
+ vertex_text_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-code-text-models":
- vertex_code_text_models.append(key)
+ vertex_code_text_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-language-models":
- vertex_language_models.append(key)
+ vertex_language_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-vision-models":
- vertex_vision_models.append(key)
+ vertex_vision_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-chat-models":
- vertex_chat_models.append(key)
+ vertex_chat_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-code-chat-models":
- vertex_code_chat_models.append(key)
+ vertex_code_chat_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-embedding-models":
- vertex_embedding_models.append(key)
+ vertex_embedding_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-anthropic_models":
key = key.replace("vertex_ai/", "")
- vertex_anthropic_models.append(key)
+ vertex_anthropic_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-llama_models":
key = key.replace("vertex_ai/", "")
- vertex_llama3_models.append(key)
+ vertex_llama3_models.add(key)
+ elif value.get("litellm_provider") == "vertex_ai-deepseek_models":
+ key = key.replace("vertex_ai/", "")
+ vertex_deepseek_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-mistral_models":
key = key.replace("vertex_ai/", "")
- vertex_mistral_models.append(key)
+ vertex_mistral_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-ai21_models":
key = key.replace("vertex_ai/", "")
- vertex_ai_ai21_models.append(key)
+ vertex_ai_ai21_models.add(key)
elif value.get("litellm_provider") == "vertex_ai-image-models":
key = key.replace("vertex_ai/", "")
- vertex_ai_image_models.append(key)
+ vertex_ai_image_models.add(key)
+ elif value.get("litellm_provider") == "vertex_ai-video-models":
+ key = key.replace("vertex_ai/", "")
+ vertex_ai_video_models.add(key)
+ elif value.get("litellm_provider") == "vertex_ai-openai_models":
+ key = key.replace("vertex_ai/", "")
+ vertex_openai_models.add(key)
elif value.get("litellm_provider") == "ai21":
if value.get("mode") == "chat":
- ai21_chat_models.append(key)
+ ai21_chat_models.add(key)
else:
- ai21_models.append(key)
+ ai21_models.add(key)
elif value.get("litellm_provider") == "nlp_cloud":
- nlp_cloud_models.append(key)
+ nlp_cloud_models.add(key)
elif value.get("litellm_provider") == "aleph_alpha":
- aleph_alpha_models.append(key)
+ aleph_alpha_models.add(key)
elif value.get(
"litellm_provider"
) == "bedrock" and not is_bedrock_pricing_only_model(key):
- bedrock_models.append(key)
+ bedrock_models.add(key)
elif value.get("litellm_provider") == "bedrock_converse":
- bedrock_converse_models.append(key)
+ bedrock_converse_models.add(key)
elif value.get("litellm_provider") == "deepinfra":
- deepinfra_models.append(key)
+ deepinfra_models.add(key)
elif value.get("litellm_provider") == "perplexity":
- perplexity_models.append(key)
+ perplexity_models.add(key)
elif value.get("litellm_provider") == "watsonx":
- watsonx_models.append(key)
+ watsonx_models.add(key)
elif value.get("litellm_provider") == "gemini":
- gemini_models.append(key)
+ gemini_models.add(key)
elif value.get("litellm_provider") == "fireworks_ai":
# ignore the 'up-to', '-to-' model names -> not real models. just for cost tracking based on model params.
if "-to-" not in key and "fireworks-ai-default" not in key:
- fireworks_ai_models.append(key)
+ fireworks_ai_models.add(key)
elif value.get("litellm_provider") == "fireworks_ai-embedding-models":
# ignore the 'up-to', '-to-' model names -> not real models. just for cost tracking based on model params.
if "-to-" not in key:
- fireworks_ai_embedding_models.append(key)
+ fireworks_ai_embedding_models.add(key)
elif value.get("litellm_provider") == "text-completion-codestral":
- text_completion_codestral_models.append(key)
+ text_completion_codestral_models.add(key)
elif value.get("litellm_provider") == "xai":
- xai_models.append(key)
+ xai_models.add(key)
elif value.get("litellm_provider") == "deepseek":
- deepseek_models.append(key)
+ deepseek_models.add(key)
elif value.get("litellm_provider") == "meta_llama":
- llama_models.append(key)
+ llama_models.add(key)
elif value.get("litellm_provider") == "nscale":
- nscale_models.append(key)
+ nscale_models.add(key)
elif value.get("litellm_provider") == "azure_ai":
- azure_ai_models.append(key)
+ azure_ai_models.add(key)
elif value.get("litellm_provider") == "voyage":
- voyage_models.append(key)
+ voyage_models.add(key)
elif value.get("litellm_provider") == "infinity":
- infinity_models.append(key)
+ infinity_models.add(key)
elif value.get("litellm_provider") == "databricks":
- databricks_models.append(key)
+ databricks_models.add(key)
elif value.get("litellm_provider") == "cloudflare":
- cloudflare_models.append(key)
+ cloudflare_models.add(key)
elif value.get("litellm_provider") == "codestral":
- codestral_models.append(key)
+ codestral_models.add(key)
elif value.get("litellm_provider") == "friendliai":
- friendliai_models.append(key)
+ friendliai_models.add(key)
elif value.get("litellm_provider") == "palm":
- palm_models.append(key)
+ palm_models.add(key)
elif value.get("litellm_provider") == "groq":
- groq_models.append(key)
+ groq_models.add(key)
elif value.get("litellm_provider") == "azure":
- azure_models.append(key)
+ azure_models.add(key)
elif value.get("litellm_provider") == "anyscale":
- anyscale_models.append(key)
+ anyscale_models.add(key)
elif value.get("litellm_provider") == "cerebras":
- cerebras_models.append(key)
+ cerebras_models.add(key)
elif value.get("litellm_provider") == "galadriel":
- galadriel_models.append(key)
+ galadriel_models.add(key)
+ elif value.get("litellm_provider") == "nvidia_nim":
+ nvidia_nim_models.add(key)
elif value.get("litellm_provider") == "sambanova":
- sambanova_models.append(key)
+ sambanova_models.add(key)
+ elif value.get("litellm_provider") == "sambanova-embedding-models":
+ sambanova_embedding_models.add(key)
elif value.get("litellm_provider") == "novita":
- novita_models.append(key)
+ novita_models.add(key)
elif value.get("litellm_provider") == "nebius-chat-models":
- nebius_models.append(key)
+ nebius_models.add(key)
elif value.get("litellm_provider") == "nebius-embedding-models":
- nebius_embedding_models.append(key)
+ nebius_embedding_models.add(key)
+ elif value.get("litellm_provider") == "aiml":
+ aiml_models.add(key)
elif value.get("litellm_provider") == "assemblyai":
- assemblyai_models.append(key)
+ assemblyai_models.add(key)
elif value.get("litellm_provider") == "jina_ai":
- jina_ai_models.append(key)
+ jina_ai_models.add(key)
elif value.get("litellm_provider") == "snowflake":
- snowflake_models.append(key)
+ snowflake_models.add(key)
+ elif value.get("litellm_provider") == "gradient_ai":
+ gradient_ai_models.add(key)
elif value.get("litellm_provider") == "featherless_ai":
- featherless_ai_models.append(key)
+ featherless_ai_models.add(key)
elif value.get("litellm_provider") == "deepgram":
- deepgram_models.append(key)
+ deepgram_models.add(key)
elif value.get("litellm_provider") == "elevenlabs":
- elevenlabs_models.append(key)
+ elevenlabs_models.add(key)
+ elif value.get("litellm_provider") == "heroku":
+ heroku_models.add(key)
elif value.get("litellm_provider") == "dashscope":
- dashscope_models.append(key)
+ dashscope_models.add(key)
elif value.get("litellm_provider") == "moonshot":
- moonshot_models.append(key)
+ moonshot_models.add(key)
elif value.get("litellm_provider") == "v0":
- v0_models.append(key)
+ v0_models.add(key)
elif value.get("litellm_provider") == "morph":
- morph_models.append(key)
+ morph_models.add(key)
elif value.get("litellm_provider") == "lambda_ai":
- lambda_ai_models.append(key)
+ lambda_ai_models.add(key)
elif value.get("litellm_provider") == "hyperbolic":
- hyperbolic_models.append(key)
+ hyperbolic_models.add(key)
elif value.get("litellm_provider") == "recraft":
- recraft_models.append(key)
+ recraft_models.add(key)
+ elif value.get("litellm_provider") == "cometapi":
+ cometapi_models.add(key)
+ elif value.get("litellm_provider") == "oci":
+ oci_models.add(key)
+ elif value.get("litellm_provider") == "volcengine":
+ volcengine_models.add(key)
+ elif value.get("litellm_provider") == "wandb":
+ wandb_models.add(key)
+ elif value.get("litellm_provider") == "ovhcloud":
+ ovhcloud_models.add(key)
+ elif value.get("litellm_provider") == "ovhcloud-embedding-models":
+ ovhcloud_embedding_models.add(key)
+ elif value.get("litellm_provider") == "lemonade":
+ lemonade_models.add(key)
add_known_models()
@@ -710,65 +779,75 @@ ollama_models = ["llama2"]
maritalk_models = ["maritalk"]
-model_list = (
+model_list = list(
open_ai_chat_completion_models
- + open_ai_text_completion_models
- + cohere_models
- + cohere_chat_models
- + anthropic_models
- + replicate_models
- + openrouter_models
- + datarobot_models
- + huggingface_models
- + vertex_chat_models
- + vertex_text_models
- + ai21_models
- + ai21_chat_models
- + together_ai_models
- + baseten_models
- + aleph_alpha_models
- + nlp_cloud_models
- + ollama_models
- + bedrock_models
- + deepinfra_models
- + perplexity_models
- + maritalk_models
- + vertex_language_models
- + watsonx_models
- + gemini_models
- + text_completion_codestral_models
- + xai_models
- + deepseek_models
- + azure_ai_models
- + voyage_models
- + infinity_models
- + databricks_models
- + cloudflare_models
- + codestral_models
- + friendliai_models
- + palm_models
- + groq_models
- + azure_models
- + anyscale_models
- + cerebras_models
- + galadriel_models
- + sambanova_models
- + azure_text_models
- + novita_models
- + assemblyai_models
- + jina_ai_models
- + snowflake_models
- + llama_models
- + featherless_ai_models
- + nscale_models
- + deepgram_models
- + elevenlabs_models
- + dashscope_models
- + moonshot_models
- + v0_models
- + morph_models
- + lambda_ai_models
- + recraft_models
+ | open_ai_text_completion_models
+ | cohere_models
+ | cohere_chat_models
+ | anthropic_models
+ | set(replicate_models)
+ | openrouter_models
+ | datarobot_models
+ | set(huggingface_models)
+ | vertex_chat_models
+ | vertex_text_models
+ | ai21_models
+ | ai21_chat_models
+ | set(together_ai_models)
+ | set(baseten_models)
+ | aleph_alpha_models
+ | nlp_cloud_models
+ | set(ollama_models)
+ | bedrock_models
+ | deepinfra_models
+ | perplexity_models
+ | set(maritalk_models)
+ | vertex_language_models
+ | watsonx_models
+ | gemini_models
+ | text_completion_codestral_models
+ | xai_models
+ | deepseek_models
+ | azure_ai_models
+ | voyage_models
+ | infinity_models
+ | databricks_models
+ | cloudflare_models
+ | codestral_models
+ | friendliai_models
+ | palm_models
+ | groq_models
+ | azure_models
+ | anyscale_models
+ | cerebras_models
+ | galadriel_models
+ | nvidia_nim_models
+ | sambanova_models
+ | azure_text_models
+ | novita_models
+ | assemblyai_models
+ | jina_ai_models
+ | snowflake_models
+ | gradient_ai_models
+ | llama_models
+ | featherless_ai_models
+ | nscale_models
+ | deepgram_models
+ | elevenlabs_models
+ | dashscope_models
+ | moonshot_models
+ | v0_models
+ | morph_models
+ | lambda_ai_models
+ | recraft_models
+ | cometapi_models
+ | oci_models
+ | heroku_models
+ | vercel_ai_gateway_models
+ | volcengine_models
+ | wandb_models
+ | ovhcloud_models
+ | lemonade_models
)
model_list_set = set(model_list)
@@ -777,9 +856,9 @@ provider_list: List[Union[LlmProviders, str]] = list(LlmProviders)
models_by_provider: dict = {
- "openai": open_ai_chat_completion_models + open_ai_text_completion_models,
+ "openai": open_ai_chat_completion_models | open_ai_text_completion_models,
"text-completion-openai": open_ai_text_completion_models,
- "cohere": cohere_models + cohere_chat_models,
+ "cohere": cohere_models | cohere_chat_models,
"cohere_chat": cohere_chat_models,
"anthropic": anthropic_models,
"replicate": replicate_models,
@@ -787,14 +866,16 @@ models_by_provider: dict = {
"together_ai": together_ai_models,
"baseten": baseten_models,
"openrouter": openrouter_models,
+ "vercel_ai_gateway": vercel_ai_gateway_models,
"datarobot": datarobot_models,
"vertex_ai": vertex_chat_models
- + vertex_text_models
- + vertex_anthropic_models
- + vertex_vision_models
- + vertex_language_models,
+ | vertex_text_models
+ | vertex_anthropic_models
+ | vertex_vision_models
+ | vertex_language_models
+ | vertex_deepseek_models,
"ai21": ai21_models,
- "bedrock": bedrock_models + bedrock_converse_models,
+ "bedrock": bedrock_models | bedrock_converse_models,
"petals": petals_models,
"ollama": ollama_models,
"ollama_chat": ollama_models,
@@ -803,7 +884,7 @@ models_by_provider: dict = {
"maritalk": maritalk_models,
"watsonx": watsonx_models,
"gemini": gemini_models,
- "fireworks_ai": fireworks_ai_models + fireworks_ai_embedding_models,
+ "fireworks_ai": fireworks_ai_models | fireworks_ai_embedding_models,
"aleph_alpha": aleph_alpha_models,
"text-completion-codestral": text_completion_codestral_models,
"xai": xai_models,
@@ -819,22 +900,26 @@ models_by_provider: dict = {
"friendliai": friendliai_models,
"palm": palm_models,
"groq": groq_models,
- "azure": azure_models + azure_text_models,
+ "azure": azure_models | azure_text_models,
"azure_text": azure_text_models,
"anyscale": anyscale_models,
"cerebras": cerebras_models,
"galadriel": galadriel_models,
- "sambanova": sambanova_models,
+ "nvidia_nim": nvidia_nim_models,
+ "sambanova": sambanova_models | sambanova_embedding_models,
"novita": novita_models,
- "nebius": nebius_models + nebius_embedding_models,
+ "nebius": nebius_models | nebius_embedding_models,
+ "aiml": aiml_models,
"assemblyai": assemblyai_models,
"jina_ai": jina_ai_models,
"snowflake": snowflake_models,
+ "gradient_ai": gradient_ai_models,
"meta_llama": llama_models,
"nscale": nscale_models,
"featherless_ai": featherless_ai_models,
"deepgram": deepgram_models,
"elevenlabs": elevenlabs_models,
+ "heroku": heroku_models,
"dashscope": dashscope_models,
"moonshot": moonshot_models,
"v0": v0_models,
@@ -842,6 +927,12 @@ models_by_provider: dict = {
"lambda_ai": lambda_ai_models,
"hyperbolic": hyperbolic_models,
"recraft": recraft_models,
+ "cometapi": cometapi_models,
+ "oci": oci_models,
+ "volcengine": volcengine_models,
+ "wandb": wandb_models,
+ "ovhcloud": ovhcloud_models | ovhcloud_embedding_models,
+ "lemonade": lemonade_models,
}
# mapping for those models which have larger equivalents
@@ -870,11 +961,13 @@ longer_context_model_fallback_dict: dict = {
all_embedding_models = (
open_ai_embedding_models
- + cohere_embedding_models
- + bedrock_embedding_models
- + vertex_embedding_models
- + fireworks_ai_embedding_models
- + nebius_embedding_models
+ | set(cohere_embedding_models)
+ | set(bedrock_embedding_models)
+ | vertex_embedding_models
+ | fireworks_ai_embedding_models
+ | nebius_embedding_models
+ | sambanova_embedding_models
+ | ovhcloud_embedding_models
)
####### IMAGE GENERATION MODELS ###################
@@ -945,6 +1038,7 @@ from .llms.openai_like.chat.handler import OpenAILikeChatConfig
from .llms.aiohttp_openai.chat.transformation import AiohttpOpenAIChatConfig
from .llms.galadriel.chat.transformation import GaladrielChatConfig
from .llms.github.chat.transformation import GithubChatConfig
+from .llms.compactifai.chat.transformation import CompactifAIChatConfig
from .llms.empower.chat.transformation import EmpowerChatConfig
from .llms.huggingface.chat.transformation import HuggingFaceChatConfig
from .llms.huggingface.embedding.transformation import HuggingFaceEmbeddingConfig
@@ -965,13 +1059,14 @@ from .llms.databricks.chat.transformation import DatabricksConfig
from .llms.databricks.embed.transformation import DatabricksEmbeddingConfig
from .llms.predibase.chat.transformation import PredibaseConfig
from .llms.replicate.chat.transformation import ReplicateConfig
-from .llms.cohere.completion.transformation import CohereTextConfig as CohereConfig
from .llms.snowflake.chat.transformation import SnowflakeConfig
from .llms.cohere.rerank.transformation import CohereRerankConfig
from .llms.cohere.rerank_v2.transformation import CohereRerankV2Config
from .llms.azure_ai.rerank.transformation import AzureAIRerankConfig
from .llms.infinity.rerank.transformation import InfinityRerankConfig
from .llms.jina_ai.rerank.transformation import JinaAIRerankConfig
+from .llms.deepinfra.rerank.transformation import DeepinfraRerankConfig
+from .llms.nvidia_nim.rerank.transformation import NvidiaNimRerankConfig
from .llms.clarifai.chat.transformation import ClarifaiConfig
from .llms.ai21.chat.transformation import AI21ChatConfig, AI21ChatConfig as AI21Config
from .llms.meta_llama.chat.transformation import LlamaAPIConfig
@@ -979,7 +1074,7 @@ from .llms.anthropic.experimental_pass_through.messages.transformation import (
AnthropicMessagesConfig,
)
from .llms.bedrock.messages.invoke_transformations.anthropic_claude3_transformation import (
- AmazonAnthropicClaude3MessagesConfig,
+ AmazonAnthropicClaudeMessagesConfig,
)
from .llms.together_ai.chat import TogetherAIConfig
from .llms.together_ai.completion.transformation import TogetherAITextCompletionConfig
@@ -1039,7 +1134,7 @@ from .llms.bedrock.chat.invoke_transformations.anthropic_claude2_transformation
AmazonAnthropicConfig,
)
from .llms.bedrock.chat.invoke_transformations.anthropic_claude3_transformation import (
- AmazonAnthropicClaude3Config,
+ AmazonAnthropicClaudeConfig,
)
from .llms.bedrock.chat.invoke_transformations.amazon_cohere_transformation import (
AmazonCohereConfig,
@@ -1072,6 +1167,7 @@ from .llms.bedrock.embed.amazon_titan_v2_transformation import (
)
from .llms.cohere.chat.transformation import CohereChatConfig
from .llms.bedrock.embed.cohere_transformation import BedrockCohereEmbeddingConfig
+from .llms.bedrock.embed.twelvelabs_marengo_transformation import TwelveLabsMarengoEmbeddingConfig
from .llms.openai.openai import OpenAIConfig, MistralEmbeddingConfig
from .llms.openai.image_variations.transformation import OpenAIImageVariationConfig
from .llms.deepinfra.chat.transformation import DeepInfraConfig
@@ -1083,22 +1179,35 @@ from .llms.topaz.image_variations.transformation import TopazImageVariationConfi
from litellm.llms.openai.completion.transformation import OpenAITextCompletionConfig
from .llms.groq.chat.transformation import GroqChatConfig
from .llms.voyage.embedding.transformation import VoyageEmbeddingConfig
+from .llms.voyage.embedding.transformation_contextual import (
+ VoyageContextualEmbeddingConfig,
+)
from .llms.infinity.embedding.transformation import InfinityEmbeddingConfig
from .llms.azure_ai.chat.transformation import AzureAIStudioConfig
from .llms.mistral.chat.transformation import MistralConfig
from .llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from .llms.azure.responses.transformation import AzureOpenAIResponsesAPIConfig
+from .llms.azure.responses.o_series_transformation import (
+ AzureOpenAIOSeriesResponsesAPIConfig,
+)
+from .llms.litellm_proxy.responses.transformation import (
+ LiteLLMProxyResponsesAPIConfig,
+)
from .llms.openai.chat.o_series_transformation import (
OpenAIOSeriesConfig as OpenAIO1Config, # maintain backwards compatibility
OpenAIOSeriesConfig,
)
from .llms.snowflake.chat.transformation import SnowflakeConfig
+from .llms.gradient_ai.chat.transformation import GradientAIConfig
openaiOSeriesConfig = OpenAIOSeriesConfig()
from .llms.openai.chat.gpt_transformation import (
OpenAIGPTConfig,
)
+from .llms.openai.chat.gpt_5_transformation import (
+ OpenAIGPT5Config,
+)
from .llms.openai.transcriptions.whisper_transformation import (
OpenAIWhisperAudioTranscriptionConfig,
)
@@ -1112,6 +1221,7 @@ from .llms.openai.chat.gpt_audio_transformation import (
)
openAIGPTAudioConfig = OpenAIGPTAudioConfig()
+openAIGPT5Config = OpenAIGPT5Config()
from .llms.nvidia_nim.chat.transformation import NvidiaNimConfig
from .llms.nvidia_nim.embed import NvidiaNimEmbeddingConfig
@@ -1121,7 +1231,9 @@ nvidiaNimEmbeddingConfig = NvidiaNimEmbeddingConfig()
from .llms.featherless_ai.chat.transformation import FeatherlessAIConfig
from .llms.cerebras.chat import CerebrasConfig
+from .llms.baseten.chat import BasetenConfig
from .llms.sambanova.chat import SambanovaConfig
+from .llms.sambanova.embedding.transformation import SambaNovaEmbeddingConfig
from .llms.ai21.chat.transformation import AI21ChatConfig
from .llms.fireworks_ai.chat.transformation import FireworksAIConfig
from .llms.fireworks_ai.completion.transformation import FireworksAITextCompletionConfig
@@ -1135,14 +1247,19 @@ from .llms.friendliai.chat.transformation import FriendliaiChatConfig
from .llms.jina_ai.embedding.transformation import JinaAIEmbeddingConfig
from .llms.xai.chat.transformation import XAIChatConfig
from .llms.xai.common_utils import XAIModelInfo
-from .llms.volcengine import VolcEngineConfig
+from .llms.aiml.chat.transformation import AIMLChatConfig
+from .llms.volcengine.chat.transformation import (
+ VolcEngineChatConfig as VolcEngineConfig,
+)
from .llms.codestral.completion.transformation import CodestralTextCompletionConfig
from .llms.azure.azure import (
AzureOpenAIError,
AzureOpenAIAssistantsAPIConfig,
)
-
+from .llms.heroku.chat.transformation import HerokuChatConfig
+from .llms.cometapi.chat.transformation import CometAPIConfig
from .llms.azure.chat.gpt_transformation import AzureOpenAIConfig
+from .llms.azure.chat.gpt_5_transformation import AzureOpenAIGPT5Config
from .llms.azure.completion.transformation import AzureOpenAITextConfig
from .llms.hosted_vllm.chat.transformation import HostedVLLMChatConfig
from .llms.llamafile.chat.transformation import LlamafileChatConfig
@@ -1159,12 +1276,18 @@ from .llms.watsonx.chat.transformation import IBMWatsonXChatConfig
from .llms.watsonx.embed.transformation import IBMWatsonXEmbeddingConfig
from .llms.github_copilot.chat.transformation import GithubCopilotConfig
from .llms.nebius.chat.transformation import NebiusConfig
+from .llms.wandb.chat.transformation import WandbConfig
from .llms.dashscope.chat.transformation import DashScopeChatConfig
from .llms.moonshot.chat.transformation import MoonshotChatConfig
from .llms.v0.chat.transformation import V0ChatConfig
+from .llms.oci.chat.transformation import OCIChatConfig
from .llms.morph.chat.transformation import MorphChatConfig
from .llms.lambda_ai.chat.transformation import LambdaAIChatConfig
from .llms.hyperbolic.chat.transformation import HyperbolicChatConfig
+from .llms.vercel_ai_gateway.chat.transformation import VercelAIGatewayConfig
+from .llms.ovhcloud.chat.transformation import OVHCloudChatConfig
+from .llms.ovhcloud.embedding.transformation import OVHCloudEmbeddingConfig
+from .llms.lemonade.chat.transformation import LemonadeChatConfig
from .main import * # type: ignore
from .integrations import *
from .llms.custom_httpx.async_client_cleanup import close_litellm_async_clients
@@ -1172,6 +1295,7 @@ from .exceptions import (
AuthenticationError,
InvalidRequestError,
BadRequestError,
+ ImageFetchError,
NotFoundError,
RateLimitError,
ServiceUnavailableError,
@@ -1196,7 +1320,6 @@ from .router import Router
from .assistants.main import *
from .batches.main import *
from .images.main import *
-from .vector_stores import *
from .batch_completion.main import * # type: ignore
from .rerank_api.main import *
from .llms.anthropic.experimental_pass_through.messages.handler import *
@@ -1231,5 +1354,26 @@ disable_hf_tokenizer_download: Optional[
] = None # disable huggingface tokenizer download. Defaults to openai clk100
global_disable_no_log_param: bool = False
+### CLI UTILITIES ###
+from litellm.litellm_core_utils.cli_token_utils import get_litellm_gateway_api_key
+
### PASSTHROUGH ###
from .passthrough import allm_passthrough_route, llm_passthrough_route
+from .google_genai import agenerate_content
+
+### GLOBAL CONFIG ###
+global_bitbucket_config: Optional[Dict[str, Any]] = None
+
+
+def set_global_bitbucket_config(config: Dict[str, Any]) -> None:
+ """Set global BitBucket configuration for prompt management."""
+ global global_bitbucket_config
+ global_bitbucket_config = config
+
+### GLOBAL CONFIG ###
+global_gitlab_config: Optional[Dict[str, Any]] = None
+
+def set_global_gitlab_config(config: Dict[str, Any]) -> None:
+ """Set global BitBucket configuration for prompt management."""
+ global global_gitlab_config
+ global_gitlab_config = config
diff --git a/litellm/_logging.py b/litellm/_logging.py
index 356bb3dcaf7..73902d2fc5a 100644
--- a/litellm/_logging.py
+++ b/litellm/_logging.py
@@ -108,6 +108,23 @@ verbose_router_logger.addHandler(handler)
verbose_proxy_logger.addHandler(handler)
verbose_logger.addHandler(handler)
+
+def _suppress_loggers():
+ """Suppress noisy loggers at INFO level"""
+ # Suppress httpx request logging at INFO level
+ httpx_logger = logging.getLogger("httpx")
+ httpx_logger.setLevel(logging.WARNING)
+
+ # Suppress APScheduler logging at INFO level
+ apscheduler_executors_logger = logging.getLogger("apscheduler.executors.default")
+ apscheduler_executors_logger.setLevel(logging.WARNING)
+ apscheduler_scheduler_logger = logging.getLogger("apscheduler.scheduler")
+ apscheduler_scheduler_logger.setLevel(logging.WARNING)
+
+
+# Call the suppression function
+_suppress_loggers()
+
ALL_LOGGERS = [
logging.getLogger(),
verbose_logger,
@@ -172,6 +189,4 @@ def _is_debugging_on() -> bool:
"""
Returns True if debugging is on
"""
- if verbose_logger.isEnabledFor(logging.DEBUG) or set_verbose is True:
- return True
- return False
+ return verbose_logger.isEnabledFor(logging.DEBUG) or set_verbose is True
diff --git a/litellm/_redis.py b/litellm/_redis.py
index cb01064f413..e6ac323ff5a 100644
--- a/litellm/_redis.py
+++ b/litellm/_redis.py
@@ -12,7 +12,7 @@ import json
# s/o [@Frank Colson](https://www.linkedin.com/in/frank-colson-422b9b183/) for this redis implementation
import os
-from typing import List, Optional, Union
+from typing import Callable, List, Optional, Union
import redis # type: ignore
import redis.asyncio as async_redis # type: ignore
@@ -34,7 +34,7 @@ def _get_redis_kwargs():
"retry",
}
- include_args = ["url"]
+ include_args = ["url", "redis_connect_func", "gcp_service_account", "gcp_ssl_ca_certs"]
available_args = [x for x in arg_spec.args if x not in exclude_args] + include_args
@@ -72,6 +72,12 @@ def _get_redis_cluster_kwargs(client=None):
available_args.append("password")
available_args.append("username")
available_args.append("ssl")
+ available_args.append("ssl_cert_reqs")
+ available_args.append("ssl_check_hostname")
+ available_args.append("ssl_ca_certs")
+ available_args.append("redis_connect_func") # Needed for sync clusters and IAM detection
+ available_args.append("gcp_service_account")
+ available_args.append("gcp_ssl_ca_certs")
return available_args
@@ -93,6 +99,76 @@ def _redis_kwargs_from_environment():
return return_dict
+def _generate_gcp_iam_access_token(service_account: str) -> str:
+ """
+ Generate GCP IAM access token for Redis authentication.
+
+ Args:
+ service_account: GCP service account in format 'projects/-/serviceAccounts/name@project.iam.gserviceaccount.com'
+
+ Returns:
+ Access token string for GCP IAM authentication
+ """
+ try:
+ from google.cloud import iam_credentials_v1
+ except ImportError:
+ raise ImportError(
+ "google-cloud-iam is required for GCP IAM Redis authentication. "
+ "Install it with: pip install google-cloud-iam"
+ )
+
+ client = iam_credentials_v1.IAMCredentialsClient()
+ request = iam_credentials_v1.GenerateAccessTokenRequest(
+ name=service_account,
+ scope=['https://www.googleapis.com/auth/cloud-platform'],
+ )
+ response = client.generate_access_token(request=request)
+ return str(response.access_token)
+
+
+def create_gcp_iam_redis_connect_func(
+ service_account: str,
+ ssl_ca_certs: Optional[str] = None,
+) -> Callable:
+ """
+ Creates a custom Redis connection function for GCP IAM authentication.
+
+ Args:
+ service_account: GCP service account in format 'projects/-/serviceAccounts/name@project.iam.gserviceaccount.com'
+ ssl_ca_certs: Path to SSL CA certificate file for secure connections
+
+ Returns:
+ A connection function that can be used with Redis clients
+ """
+ def iam_connect(self):
+ """Initialize the connection and authenticate using GCP IAM"""
+ from redis.exceptions import (
+ AuthenticationError,
+ AuthenticationWrongNumberOfArgsError,
+ )
+ from redis.utils import str_if_bytes
+
+ self._parser.on_connect(self)
+
+ auth_args = (_generate_gcp_iam_access_token(service_account),)
+ self.send_command("AUTH", *auth_args, check_health=False)
+
+ try:
+ auth_response = self.read_response()
+ except AuthenticationWrongNumberOfArgsError:
+ # Fallback to password auth if IAM fails
+ if hasattr(self, 'password') and self.password:
+ self.send_command("AUTH", self.password, check_health=False)
+ auth_response = self.read_response()
+ else:
+ raise
+
+ if str_if_bytes(auth_response) != "OK":
+ raise AuthenticationError("GCP IAM authentication failed")
+
+ return iam_connect
+
+
def get_redis_url_from_environment():
if "REDIS_URL" in os.environ:
return os.environ["REDIS_URL"]
@@ -101,14 +177,21 @@ def get_redis_url_from_environment():
raise ValueError(
"Either 'REDIS_URL' or both 'REDIS_HOST' and 'REDIS_PORT' must be specified for Redis."
)
-
- if "REDIS_PASSWORD" in os.environ:
- redis_password = f":{os.environ['REDIS_PASSWORD']}@"
+
+ if "REDIS_SSL" in os.environ and os.environ["REDIS_SSL"].lower() == "true":
+ redis_protocol = "rediss"
else:
- redis_password = ""
-
+ redis_protocol = "redis"
+
+ # Build authentication part of URL
+ auth_part = ""
+ if "REDIS_USERNAME" in os.environ and "REDIS_PASSWORD" in os.environ:
+ auth_part = f"{os.environ['REDIS_USERNAME']}:{os.environ['REDIS_PASSWORD']}@"
+ elif "REDIS_PASSWORD" in os.environ:
+ auth_part = f"{os.environ['REDIS_PASSWORD']}@"
+
return (
- f"redis://{redis_password}{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}"
+ f"{redis_protocol}://{auth_part}{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}"
)
@@ -156,6 +239,27 @@ def _get_redis_client_logic(**env_overrides):
if _service_name is not None:
redis_kwargs["service_name"] = _service_name
+ # Handle GCP IAM authentication
+ _gcp_service_account = redis_kwargs.get("gcp_service_account") or get_secret_str("REDIS_GCP_SERVICE_ACCOUNT")
+ _gcp_ssl_ca_certs = redis_kwargs.get("gcp_ssl_ca_certs") or get_secret_str("REDIS_GCP_SSL_CA_CERTS")
+
+ if _gcp_service_account is not None:
+ verbose_logger.debug("Setting up GCP IAM authentication for Redis with service account.")
+ redis_kwargs["redis_connect_func"] = create_gcp_iam_redis_connect_func(
+ service_account=_gcp_service_account,
+ ssl_ca_certs=_gcp_ssl_ca_certs
+ )
+ # Store GCP service account in redis_connect_func for async cluster access
+ redis_kwargs["redis_connect_func"]._gcp_service_account = _gcp_service_account
+
+ # Remove GCP-specific kwargs that shouldn't be passed to Redis client
+ redis_kwargs.pop("gcp_service_account", None)
+ redis_kwargs.pop("gcp_ssl_ca_certs", None)
+
+ # Only enable SSL if explicitly requested AND SSL CA certs are provided
+ if _gcp_ssl_ca_certs and redis_kwargs.get("ssl", False):
+ redis_kwargs["ssl_ca_certs"] = _gcp_ssl_ca_certs
+
if "url" in redis_kwargs and redis_kwargs["url"] is not None:
redis_kwargs.pop("host", None)
redis_kwargs.pop("port", None)
@@ -198,7 +302,7 @@ def init_redis_cluster(redis_kwargs) -> redis.RedisCluster:
for item in redis_kwargs["startup_nodes"]:
new_startup_nodes.append(ClusterNode(**item))
- redis_kwargs.pop("startup_nodes")
+ cluster_kwargs.pop("startup_nodes", None)
return redis.RedisCluster(startup_nodes=new_startup_nodes, **cluster_kwargs) # type: ignore
@@ -273,7 +377,7 @@ def get_redis_client(**env_overrides):
def get_redis_async_client(
**env_overrides,
-) -> async_redis.Redis:
+) -> Union[async_redis.Redis, async_redis.RedisCluster]:
redis_kwargs = _get_redis_client_logic(**env_overrides)
if "url" in redis_kwargs and redis_kwargs["url"] is not None:
args = _get_redis_url_kwargs(client=async_redis.Redis.from_url)
@@ -298,14 +402,46 @@ def get_redis_async_client(
if arg in args:
cluster_kwargs[arg] = redis_kwargs[arg]
+ # Handle GCP IAM authentication for async clusters
+ redis_connect_func = cluster_kwargs.pop("redis_connect_func", None)
+ from litellm import get_secret_str
+
+ # Get GCP service account - first try from redis_connect_func, then from environment
+ gcp_service_account = None
+ if redis_connect_func and hasattr(redis_connect_func, '_gcp_service_account'):
+ gcp_service_account = redis_connect_func._gcp_service_account
+ else:
+ gcp_service_account = redis_kwargs.get("gcp_service_account") or get_secret_str("REDIS_GCP_SERVICE_ACCOUNT")
+
+ verbose_logger.debug(f"DEBUG: Redis cluster kwargs: redis_connect_func={redis_connect_func is not None}, gcp_service_account_provided={gcp_service_account is not None}")
+
+ # If GCP IAM is configured (indicated by redis_connect_func), generate access token and use as password
+ if redis_connect_func and gcp_service_account:
+ verbose_logger.debug("DEBUG: Generating IAM token for service account (value not logged for security reasons)")
+ try:
+ # Generate IAM access token using the helper function
+ access_token = _generate_gcp_iam_access_token(gcp_service_account)
+ cluster_kwargs["password"] = access_token
+ verbose_logger.debug("DEBUG: Successfully generated GCP IAM access token for async Redis cluster")
+ except Exception as e:
+ verbose_logger.error(f"Failed to generate GCP IAM access token: {e}")
+ from redis.exceptions import AuthenticationError
+ raise AuthenticationError("Failed to generate GCP IAM access token")
+ else:
+ verbose_logger.debug(f"DEBUG: Not using GCP IAM auth - redis_connect_func={redis_connect_func is not None}, gcp_service_account_provided={gcp_service_account is not None}")
+
new_startup_nodes: List[ClusterNode] = []
for item in redis_kwargs["startup_nodes"]:
new_startup_nodes.append(ClusterNode(**item))
- redis_kwargs.pop("startup_nodes")
- return async_redis.RedisCluster(
+ cluster_kwargs.pop("startup_nodes", None)
+
+ # Create async RedisCluster with IAM token as password if available
+ cluster_client = async_redis.RedisCluster(
startup_nodes=new_startup_nodes, **cluster_kwargs # type: ignore
)
+
+ return cluster_client
# Check for Redis Sentinel
if "sentinel_nodes" in redis_kwargs and "service_name" in redis_kwargs:
diff --git a/litellm/_uuid.py b/litellm/_uuid.py
new file mode 100644
index 00000000000..52acf647dd8
--- /dev/null
+++ b/litellm/_uuid.py
@@ -0,0 +1,16 @@
+"""
+Internal unified UUID helper.
+
+Always uses fastuuid for performance.
+"""
+
+import fastuuid as _uuid # type: ignore
+
+
+# Expose a module-like alias so callers can use: uuid.uuid4()
+uuid = _uuid
+
+
+def uuid4():
+ """Return a UUID4 using the selected backend."""
+ return uuid.uuid4()
diff --git a/litellm/batches/main.py b/litellm/batches/main.py
index 3ea0f95157f..48521e5fba0 100644
--- a/litellm/batches/main.py
+++ b/litellm/batches/main.py
@@ -14,13 +14,16 @@ import asyncio
import contextvars
import os
from functools import partial
-from typing import Any, Coroutine, Dict, Literal, Optional, Union
+from typing import Any, Coroutine, Dict, Literal, Optional, Union, cast
import httpx
import litellm
+from litellm._logging import verbose_logger
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.azure.batches.handler import AzureBatchesAPI
+from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
+from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.llms.openai.openai import OpenAIBatchesAPI
from litellm.llms.vertex_ai.batches.handler import VertexAIBatchPrediction
from litellm.secret_managers.main import get_secret_str
@@ -31,22 +34,72 @@ from litellm.types.llms.openai import (
RetrieveBatchRequest,
)
from litellm.types.router import GenericLiteLLMParams
-from litellm.types.utils import LiteLLMBatch
-from litellm.utils import client, get_litellm_params, supports_httpx_timeout
+from litellm.types.utils import LiteLLMBatch, LlmProviders
+from litellm.utils import (
+ ProviderConfigManager,
+ client,
+ get_litellm_params,
+ get_llm_provider,
+ supports_httpx_timeout,
+)
####### ENVIRONMENT VARIABLES ###################
openai_batches_instance = OpenAIBatchesAPI()
azure_batches_instance = AzureBatchesAPI()
vertex_ai_batches_instance = VertexAIBatchPrediction(gcs_bucket_name="")
+base_llm_http_handler = BaseLLMHTTPHandler()
#################################################
+def _resolve_timeout(
+ optional_params: GenericLiteLLMParams,
+ kwargs: Dict[str, Any],
+ custom_llm_provider: str,
+ default_timeout: float = 600.0,
+) -> float:
+ """
+ Resolve timeout value from various sources and handle httpx.Timeout objects.
+
+ Args:
+ optional_params: GenericLiteLLMParams object containing timeout
+ kwargs: Additional kwargs that may contain request_timeout
+ custom_llm_provider: Provider name for httpx timeout support check
+ default_timeout: Default timeout value to use
+
+ Returns:
+ Resolved timeout as float
+ """
+ timeout = (
+ optional_params.timeout
+ or kwargs.get("request_timeout", default_timeout)
+ or default_timeout
+ )
+
+ # Handle httpx.Timeout objects
+ if isinstance(timeout, httpx.Timeout):
+ if supports_httpx_timeout(custom_llm_provider) is False:
+ # Extract read timeout for providers that don't support httpx.Timeout
+ read_timeout = timeout.read or default_timeout
+ return float(read_timeout)
+ else:
+ # For providers that support httpx.Timeout, we still need to return a float
+ # This case might need to be handled differently based on the actual use case
+ return float(timeout.read or default_timeout)
+
+ # Handle None case
+ if timeout is None:
+ return float(default_timeout)
+
+ # Handle numeric values (int, float, string representations)
+ return float(timeout)
+
+
@client
async def acreate_batch(
completion_window: Literal["24h"],
endpoint: Literal["/v1/chat/completions", "/v1/embeddings", "/v1/completions"],
input_file_id: str,
- custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
+ custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@@ -94,7 +147,7 @@ def create_batch(
completion_window: Literal["24h"],
endpoint: Literal["/v1/chat/completions", "/v1/embeddings", "/v1/completions"],
input_file_id: str,
- custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
+ custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@@ -110,13 +163,27 @@ def create_batch(
litellm_call_id = kwargs.get("litellm_call_id", None)
proxy_server_request = kwargs.get("proxy_server_request", None)
model_info = kwargs.get("model_info", None)
+ model: Optional[str] = kwargs.get("model", None)
+ try:
+ if model is not None:
+ model, _, _, _ = get_llm_provider(
+ model=model,
+ custom_llm_provider=None,
+ )
+ except Exception as e:
+ verbose_logger.exception(
+ f"litellm.batches.main.py::create_batch() - Error inferring custom_llm_provider - {str(e)}"
+ )
+
_is_async = kwargs.pop("acreate_batch", False) is True
- litellm_params = get_litellm_params(**kwargs)
- litellm_logging_obj: LiteLLMLoggingObj = kwargs.get("litellm_logging_obj", None)
+ litellm_params = dict(GenericLiteLLMParams(**kwargs))
+ litellm_logging_obj: LiteLLMLoggingObj = cast(
+ LiteLLMLoggingObj, kwargs.get("litellm_logging_obj", None)
+ )
### TIMEOUT LOGIC ###
- timeout = optional_params.timeout or kwargs.get("request_timeout", 600) or 600
+ timeout = _resolve_timeout(optional_params, kwargs, custom_llm_provider)
litellm_logging_obj.update_environment_variables(
- model=None,
+ model=model,
user=None,
optional_params=optional_params.model_dump(),
litellm_params={
@@ -131,18 +198,6 @@ def create_batch(
custom_llm_provider=custom_llm_provider,
)
- if (
- timeout is not None
- and isinstance(timeout, httpx.Timeout)
- and supports_httpx_timeout(custom_llm_provider) is False
- ):
- read_timeout = timeout.read or 600
- timeout = read_timeout # default 10 min timeout
- elif timeout is not None and not isinstance(timeout, httpx.Timeout):
- timeout = float(timeout) # type: ignore
- elif timeout is None:
- timeout = 600.0
-
_create_batch_request = CreateBatchRequest(
completion_window=completion_window,
endpoint=endpoint,
@@ -151,6 +206,31 @@ def create_batch(
extra_headers=extra_headers,
extra_body=extra_body,
)
+ if model is not None:
+ provider_config = ProviderConfigManager.get_provider_batches_config(
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ )
+ else:
+ provider_config = None
+ if provider_config is not None:
+ response = base_llm_http_handler.create_batch(
+ provider_config=provider_config,
+ litellm_params=litellm_params,
+ create_batch_data=_create_batch_request,
+ headers=extra_headers or {},
+ api_base=optional_params.api_base,
+ api_key=optional_params.api_key,
+ logging_obj=litellm_logging_obj,
+ _is_async=_is_async,
+ client=client
+ if client is not None
+ and isinstance(client, (HTTPHandler, AsyncHTTPHandler))
+ else None,
+ timeout=timeout,
+ model=model,
+ )
+ return response
api_base: Optional[str] = None
if custom_llm_provider == "openai":
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
@@ -267,7 +347,7 @@ def create_batch(
@client
async def aretrieve_batch(
batch_id: str,
- custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
+ custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@@ -306,10 +386,130 @@ async def aretrieve_batch(
raise e
+def _handle_retrieve_batch_providers_without_provider_config(
+ batch_id: str,
+ optional_params: GenericLiteLLMParams,
+ timeout: Union[float, httpx.Timeout],
+ litellm_params: dict,
+ _retrieve_batch_request: RetrieveBatchRequest,
+ _is_async: bool,
+ custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
+):
+ api_base: Optional[str] = None
+ if custom_llm_provider == "openai":
+ # for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
+ api_base = (
+ optional_params.api_base
+ or litellm.api_base
+ or os.getenv("OPENAI_BASE_URL")
+ or os.getenv("OPENAI_API_BASE")
+ or "https://api.openai.com/v1"
+ )
+ organization = (
+ optional_params.organization
+ or litellm.organization
+ or os.getenv("OPENAI_ORGANIZATION", None)
+ or None # default - https://github.com/openai/openai-python/blob/284c1799070c723c6a553337134148a7ab088dd8/openai/util.py#L105
+ )
+ # set API KEY
+ api_key = (
+ optional_params.api_key
+ or litellm.api_key # for deepinfra/perplexity/anyscale we check in get_llm_provider and pass in the api key from there
+ or litellm.openai_key
+ or os.getenv("OPENAI_API_KEY")
+ )
+
+ response = openai_batches_instance.retrieve_batch(
+ _is_async=_is_async,
+ retrieve_batch_data=_retrieve_batch_request,
+ api_base=api_base,
+ api_key=api_key,
+ organization=organization,
+ timeout=timeout,
+ max_retries=optional_params.max_retries,
+ )
+ elif custom_llm_provider == "azure":
+ api_base = (
+ optional_params.api_base
+ or litellm.api_base
+ or get_secret_str("AZURE_API_BASE")
+ )
+ api_version = (
+ optional_params.api_version
+ or litellm.api_version
+ or get_secret_str("AZURE_API_VERSION")
+ )
+
+ api_key = (
+ optional_params.api_key
+ or litellm.api_key
+ or litellm.azure_key
+ or get_secret_str("AZURE_OPENAI_API_KEY")
+ or get_secret_str("AZURE_API_KEY")
+ )
+
+ extra_body = optional_params.get("extra_body", {})
+ if extra_body is not None:
+ extra_body.pop("azure_ad_token", None)
+ else:
+ get_secret_str("AZURE_AD_TOKEN") # type: ignore
+
+ response = azure_batches_instance.retrieve_batch(
+ _is_async=_is_async,
+ api_base=api_base,
+ api_key=api_key,
+ api_version=api_version,
+ timeout=timeout,
+ max_retries=optional_params.max_retries,
+ retrieve_batch_data=_retrieve_batch_request,
+ litellm_params=litellm_params,
+ )
+ elif custom_llm_provider == "vertex_ai":
+ api_base = optional_params.api_base or ""
+ vertex_ai_project = (
+ optional_params.vertex_project
+ or litellm.vertex_project
+ or get_secret_str("VERTEXAI_PROJECT")
+ )
+ vertex_ai_location = (
+ optional_params.vertex_location
+ or litellm.vertex_location
+ or get_secret_str("VERTEXAI_LOCATION")
+ )
+ vertex_credentials = optional_params.vertex_credentials or get_secret_str(
+ "VERTEXAI_CREDENTIALS"
+ )
+
+ response = vertex_ai_batches_instance.retrieve_batch(
+ _is_async=_is_async,
+ batch_id=batch_id,
+ api_base=api_base,
+ vertex_project=vertex_ai_project,
+ vertex_location=vertex_ai_location,
+ vertex_credentials=vertex_credentials,
+ timeout=timeout,
+ max_retries=optional_params.max_retries,
+ )
+ else:
+ raise litellm.exceptions.BadRequestError(
+ message="LiteLLM doesn't support {} for 'create_batch'. Only 'openai' is supported.".format(
+ custom_llm_provider
+ ),
+ model="n/a",
+ llm_provider=custom_llm_provider,
+ response=httpx.Response(
+ status_code=400,
+ content="Unsupported provider",
+ request=httpx.Request(method="create_thread", url="https://github.com/BerriAI/litellm"), # type: ignore
+ ),
+ )
+ return response
+
+
@client
def retrieve_batch(
batch_id: str,
- custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
+ custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@@ -322,20 +522,23 @@ def retrieve_batch(
"""
try:
optional_params = GenericLiteLLMParams(**kwargs)
- litellm_logging_obj: LiteLLMLoggingObj = kwargs.get("litellm_logging_obj", None)
+ litellm_logging_obj: Optional[LiteLLMLoggingObj] = kwargs.get(
+ "litellm_logging_obj", None
+ )
### TIMEOUT LOGIC ###
timeout = optional_params.timeout or kwargs.get("request_timeout", 600) or 600
litellm_params = get_litellm_params(
custom_llm_provider=custom_llm_provider,
**kwargs,
)
- litellm_logging_obj.update_environment_variables(
- model=None,
- user=None,
- optional_params=optional_params.model_dump(),
- litellm_params=litellm_params,
- custom_llm_provider=custom_llm_provider,
- )
+ if litellm_logging_obj is not None:
+ litellm_logging_obj.update_environment_variables(
+ model=None,
+ user=None,
+ optional_params=optional_params.model_dump(),
+ litellm_params=litellm_params,
+ custom_llm_provider=custom_llm_provider,
+ )
if (
timeout is not None
@@ -356,115 +559,78 @@ def retrieve_batch(
)
_is_async = kwargs.pop("aretrieve_batch", False) is True
- api_base: Optional[str] = None
- if custom_llm_provider == "openai":
- # for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
- api_base = (
- optional_params.api_base
- or litellm.api_base
- or os.getenv("OPENAI_BASE_URL")
- or os.getenv("OPENAI_API_BASE")
- or "https://api.openai.com/v1"
- )
- organization = (
- optional_params.organization
- or litellm.organization
- or os.getenv("OPENAI_ORGANIZATION", None)
- or None # default - https://github.com/openai/openai-python/blob/284c1799070c723c6a553337134148a7ab088dd8/openai/util.py#L105
- )
- # set API KEY
- api_key = (
- optional_params.api_key
- or litellm.api_key # for deepinfra/perplexity/anyscale we check in get_llm_provider and pass in the api key from there
- or litellm.openai_key
- or os.getenv("OPENAI_API_KEY")
- )
+ client = kwargs.get("client", None)
- response = openai_batches_instance.retrieve_batch(
- _is_async=_is_async,
- retrieve_batch_data=_retrieve_batch_request,
- api_base=api_base,
- api_key=api_key,
- organization=organization,
- timeout=timeout,
- max_retries=optional_params.max_retries,
- )
- elif custom_llm_provider == "azure":
- api_base = (
- optional_params.api_base
- or litellm.api_base
- or get_secret_str("AZURE_API_BASE")
- )
- api_version = (
- optional_params.api_version
- or litellm.api_version
- or get_secret_str("AZURE_API_VERSION")
- )
+ # Check if this is an async invoke ARN (different from regular batch ARN)
+ # Async invoke ARNs have format: arn:aws(-[^:]+)?:bedrock:[a-z0-9-]{1,20}:[0-9]{12}:async-invoke/[a-z0-9]{12}
+ if (
+ batch_id.startswith("arn:aws")
+ and ":bedrock:" in batch_id
+ and ":async-invoke/" in batch_id
+ ):
+ # Handle async invoke status check
+ # Remove aws_region_name from kwargs to avoid duplicate parameter
+ async_kwargs = kwargs.copy()
+ async_kwargs.pop("aws_region_name", None)
- api_key = (
- optional_params.api_key
- or litellm.api_key
- or litellm.azure_key
- or get_secret_str("AZURE_OPENAI_API_KEY")
- or get_secret_str("AZURE_API_KEY")
- )
-
- extra_body = optional_params.get("extra_body", {})
- if extra_body is not None:
- extra_body.pop("azure_ad_token", None)
- else:
- get_secret_str("AZURE_AD_TOKEN") # type: ignore
-
- response = azure_batches_instance.retrieve_batch(
- _is_async=_is_async,
- api_base=api_base,
- api_key=api_key,
- api_version=api_version,
- timeout=timeout,
- max_retries=optional_params.max_retries,
- retrieve_batch_data=_retrieve_batch_request,
- litellm_params=litellm_params,
- )
- elif custom_llm_provider == "vertex_ai":
- api_base = optional_params.api_base or ""
- vertex_ai_project = (
- optional_params.vertex_project
- or litellm.vertex_project
- or get_secret_str("VERTEXAI_PROJECT")
- )
- vertex_ai_location = (
- optional_params.vertex_location
- or litellm.vertex_location
- or get_secret_str("VERTEXAI_LOCATION")
- )
- vertex_credentials = optional_params.vertex_credentials or get_secret_str(
- "VERTEXAI_CREDENTIALS"
- )
-
- response = vertex_ai_batches_instance.retrieve_batch(
- _is_async=_is_async,
+ return _handle_async_invoke_status(
batch_id=batch_id,
- api_base=api_base,
- vertex_project=vertex_ai_project,
- vertex_location=vertex_ai_location,
- vertex_credentials=vertex_credentials,
- timeout=timeout,
- max_retries=optional_params.max_retries,
+ aws_region_name=kwargs.get("aws_region_name", "us-east-1"),
+ logging_obj=litellm_logging_obj,
+ **async_kwargs,
+ )
+
+ # Try to use provider config first (for providers like bedrock)
+ model: Optional[str] = kwargs.get("model", None)
+ if model is not None:
+ provider_config = ProviderConfigManager.get_provider_batches_config(
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
)
else:
- raise litellm.exceptions.BadRequestError(
- message="LiteLLM doesn't support {} for 'create_batch'. Only 'openai' is supported.".format(
- custom_llm_provider
- ),
- model="n/a",
- llm_provider=custom_llm_provider,
- response=httpx.Response(
- status_code=400,
- content="Unsupported provider",
- request=httpx.Request(method="create_thread", url="https://github.com/BerriAI/litellm"), # type: ignore
+ provider_config = None
+
+ if provider_config is not None:
+ response = base_llm_http_handler.retrieve_batch(
+ batch_id=batch_id,
+ provider_config=provider_config,
+ litellm_params=litellm_params,
+ headers=extra_headers or {},
+ api_base=optional_params.api_base,
+ api_key=optional_params.api_key,
+ logging_obj=litellm_logging_obj
+ or LiteLLMLoggingObj(
+ model=model or "bedrock/unknown",
+ messages=[],
+ stream=False,
+ call_type="batch_retrieve",
+ start_time=None,
+ litellm_call_id="batch_retrieve_" + batch_id,
+ function_id="batch_retrieve",
),
+ _is_async=_is_async,
+ client=client
+ if client is not None
+ and isinstance(client, (HTTPHandler, AsyncHTTPHandler))
+ else None,
+ timeout=timeout,
+ model=model,
)
- return response
+ return response
+
+ #########################################################
+ # Handle providers without provider config
+ #########################################################
+ return _handle_retrieve_batch_providers_without_provider_config(
+ batch_id=batch_id,
+ custom_llm_provider=custom_llm_provider,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ _retrieve_batch_request=_retrieve_batch_request,
+ _is_async=_is_async,
+ timeout=timeout,
+ )
+
except Exception as e:
raise e
@@ -797,3 +963,79 @@ def cancel_batch(
return response
except Exception as e:
raise e
+
+
+def _handle_async_invoke_status(
+ batch_id: str, aws_region_name: str, logging_obj=None, **kwargs
+) -> "LiteLLMBatch":
+ """
+ Handle async invoke status check for AWS Bedrock.
+
+ Args:
+ batch_id: The async invoke ARN
+ aws_region_name: AWS region name
+ **kwargs: Additional parameters
+
+ Returns:
+ dict: Status information including status, output_file_id (S3 URL), etc.
+ """
+ import asyncio
+
+ from litellm.llms.bedrock.embed.embedding import BedrockEmbedding
+
+ async def _async_get_status():
+ # Create embedding handler instance
+ embedding_handler = BedrockEmbedding()
+
+ # Get the status of the async invoke job
+ status_response = await embedding_handler._get_async_invoke_status(
+ invocation_arn=batch_id,
+ aws_region_name=aws_region_name,
+ logging_obj=logging_obj,
+ **kwargs,
+ )
+
+ # Transform response to a LiteLLMBatch object
+ from litellm.types.utils import LiteLLMBatch
+
+ result = LiteLLMBatch(
+ id=status_response["invocationArn"],
+ object="batch",
+ status=status_response["status"],
+ created_at=status_response["submitTime"],
+ in_progress_at=status_response["lastModifiedTime"],
+ completed_at=status_response.get("endTime"),
+ failed_at=status_response.get("endTime")
+ if status_response["status"] == "failed"
+ else None,
+ request_counts={
+ "total": 1,
+ "completed": 1 if status_response["status"] == "completed" else 0,
+ "failed": 1 if status_response["status"] == "failed" else 0,
+ },
+ metadata={
+ "output_file_id": status_response["outputDataConfig"][
+ "s3OutputDataConfig"
+ ]["s3Uri"],
+ "failure_message": status_response.get("failureMessage"),
+ "model_arn": status_response["modelArn"],
+ },
+ )
+
+ return result
+
+ # Since this function is called from within an async context via run_in_executor,
+ # we need to create a new event loop in a thread to avoid conflicts
+ import concurrent.futures
+
+ def run_in_thread():
+ new_loop = asyncio.new_event_loop()
+ asyncio.set_event_loop(new_loop)
+ try:
+ return new_loop.run_until_complete(_async_get_status())
+ finally:
+ new_loop.close()
+
+ with concurrent.futures.ThreadPoolExecutor() as executor:
+ future = executor.submit(run_in_thread)
+ return future.result()
diff --git a/litellm/caching/__init__.py b/litellm/caching/__init__.py
index badc462e09b..bbe90b04121 100644
--- a/litellm/caching/__init__.py
+++ b/litellm/caching/__init__.py
@@ -7,4 +7,5 @@ from .qdrant_semantic_cache import QdrantSemanticCache
from .redis_cache import RedisCache
from .redis_cluster_cache import RedisClusterCache
from .redis_semantic_cache import RedisSemanticCache
-from .s3_cache import S3Cache
\ No newline at end of file
+from .s3_cache import S3Cache
+from .gcs_cache import GCSCache
diff --git a/litellm/caching/caching.py b/litellm/caching/caching.py
index 6959467cddd..82fc37e0cb4 100644
--- a/litellm/caching/caching.py
+++ b/litellm/caching/caching.py
@@ -28,6 +28,7 @@ from .azure_blob_cache import AzureBlobCache
from .base_cache import BaseCache
from .disk_cache import DiskCache
from .dual_cache import DualCache # noqa
+from .gcs_cache import GCSCache
from .in_memory_cache import InMemoryCache
from .qdrant_semantic_cache import QdrantSemanticCache
from .redis_cache import RedisCache
@@ -92,6 +93,9 @@ class Cache:
s3_aws_session_token: Optional[str] = None,
s3_config: Optional[Any] = None,
s3_path: Optional[str] = None,
+ gcs_bucket_name: Optional[str] = None,
+ gcs_path_service_account: Optional[str] = None,
+ gcs_path: Optional[str] = None,
redis_semantic_cache_embedding_model: str = "text-embedding-ada-002",
redis_semantic_cache_index_name: Optional[str] = None,
redis_flush_size: Optional[int] = None,
@@ -102,6 +106,9 @@ class Cache:
qdrant_collection_name: Optional[str] = None,
qdrant_quantization_config: Optional[str] = None,
qdrant_semantic_cache_embedding_model: str = "text-embedding-ada-002",
+ # GCP IAM authentication parameters
+ gcp_service_account: Optional[str] = None,
+ gcp_ssl_ca_certs: Optional[str] = None,
**kwargs,
):
"""
@@ -140,6 +147,11 @@ class Cache:
s3_aws_session_token (str, optional): The aws session token for the s3 cache. Defaults to None.
s3_config (dict, optional): The config for the s3 cache. Defaults to None.
+ # GCS Cache Args
+ gcs_bucket_name (str, optional): The bucket name for the gcs cache. Defaults to None.
+ gcs_path_service_account (str, optional): Path to the service account json.
+ gcs_path (str, optional): Folder path inside the bucket to store cache files.
+
# Common Cache Args
supported_call_types (list, optional): List of call types to cache for. Defaults to cache == on for all call types.
**kwargs: Additional keyword arguments for redis.Redis() cache
@@ -152,14 +164,21 @@ class Cache:
"""
if type == LiteLLMCacheType.REDIS:
if redis_startup_nodes:
- self.cache: BaseCache = RedisClusterCache(
- host=host,
- port=port,
- password=password,
- redis_flush_size=redis_flush_size,
- startup_nodes=redis_startup_nodes,
+ # Only pass GCP parameters if they are provided
+ cluster_kwargs = {
+ "host": host,
+ "port": port,
+ "password": password,
+ "redis_flush_size": redis_flush_size,
+ "startup_nodes": redis_startup_nodes,
**kwargs,
- )
+ }
+ if gcp_service_account is not None:
+ cluster_kwargs["gcp_service_account"] = gcp_service_account
+ if gcp_ssl_ca_certs is not None:
+ cluster_kwargs["gcp_ssl_ca_certs"] = gcp_ssl_ca_certs
+
+ self.cache: BaseCache = RedisClusterCache(**cluster_kwargs)
else:
self.cache = RedisCache(
host=host,
@@ -204,6 +223,12 @@ class Cache:
s3_path=s3_path,
**kwargs,
)
+ elif type == LiteLLMCacheType.GCS:
+ self.cache = GCSCache(
+ bucket_name=gcs_bucket_name,
+ path_service_account=gcs_path_service_account,
+ gcs_path=gcs_path,
+ )
elif type == LiteLLMCacheType.AZURE_BLOB:
self.cache = AzureBlobCache(
account_url=azure_account_url,
@@ -456,7 +481,7 @@ class Cache:
return cached_response
return cached_result
- def get_cache(self, **kwargs):
+ def get_cache(self, dynamic_cache_object: Optional[BaseCache] = None, **kwargs):
"""
Retrieves the cached result for the given arguments.
@@ -482,8 +507,12 @@ class Cache:
or cache_control_args.get("s-max-age")
or float("inf")
)
- cached_result = self.cache.get_cache(cache_key, messages=messages)
- cached_result = self.cache.get_cache(cache_key, messages=messages)
+ if dynamic_cache_object is not None:
+ cached_result = dynamic_cache_object.get_cache(
+ cache_key, messages=messages
+ )
+ else:
+ cached_result = self.cache.get_cache(cache_key, messages=messages)
return self._get_cache_logic(
cached_result=cached_result, max_age=max_age
)
@@ -491,7 +520,9 @@ class Cache:
print_verbose(f"An exception occurred: {traceback.format_exc()}")
return None
- async def async_get_cache(self, **kwargs):
+ async def async_get_cache(
+ self, dynamic_cache_object: Optional[BaseCache] = None, **kwargs
+ ):
"""
Async get cache implementation.
@@ -512,7 +543,14 @@ class Cache:
max_age = cache_control_args.get(
"s-max-age", cache_control_args.get("s-maxage", float("inf"))
)
- cached_result = await self.cache.async_get_cache(cache_key, **kwargs)
+ if dynamic_cache_object is not None:
+ cached_result = await dynamic_cache_object.async_get_cache(
+ cache_key, **kwargs
+ )
+ else:
+ cached_result = await self.cache.async_get_cache(
+ cache_key, **kwargs
+ )
return self._get_cache_logic(
cached_result=cached_result, max_age=max_age
)
@@ -571,7 +609,9 @@ class Cache:
except Exception as e:
verbose_logger.exception(f"LiteLLM Cache: Excepton add_cache: {str(e)}")
- async def async_add_cache(self, result, **kwargs):
+ async def async_add_cache(
+ self, result, dynamic_cache_object: Optional[BaseCache] = None, **kwargs
+ ):
"""
Async implementation of add_cache
"""
@@ -585,12 +625,18 @@ class Cache:
cache_key, cached_data, kwargs = self._add_cache_logic(
result=result, **kwargs
)
-
- await self.cache.async_set_cache(cache_key, cached_data, **kwargs)
+ if dynamic_cache_object is not None:
+ await dynamic_cache_object.async_set_cache(
+ cache_key, cached_data, **kwargs
+ )
+ else:
+ await self.cache.async_set_cache(cache_key, cached_data, **kwargs)
except Exception as e:
verbose_logger.exception(f"LiteLLM Cache: Excepton add_cache: {str(e)}")
- def _convert_to_cached_embedding(self, embedding_response: Any, model: Optional[str]) -> CachedEmbedding:
+ def _convert_to_cached_embedding(
+ self, embedding_response: Any, model: Optional[str]
+ ) -> CachedEmbedding:
"""
Convert any embedding response into the standardized CachedEmbedding TypedDict format.
"""
@@ -602,7 +648,7 @@ class Cache:
"object": embedding_response.get("object"),
"model": model,
}
- elif hasattr(embedding_response, 'model_dump'):
+ elif hasattr(embedding_response, "model_dump"):
data = embedding_response.model_dump()
return {
"embedding": data.get("embedding"),
@@ -621,7 +667,6 @@ class Cache:
except KeyError as e:
raise ValueError(f"Missing expected key in embedding response: {e}")
-
def add_embedding_response_to_cache(
self,
result: EmbeddingResponse,
@@ -632,18 +677,22 @@ class Cache:
preset_cache_key = self.get_cache_key(**{**kwargs, "input": input})
kwargs["cache_key"] = preset_cache_key
embedding_response = result.data[idx_in_result_data]
-
+
# Always convert to properly typed CachedEmbedding
model_name = result.model
- embedding_dict: CachedEmbedding = self._convert_to_cached_embedding(embedding_response, model_name)
-
+ embedding_dict: CachedEmbedding = self._convert_to_cached_embedding(
+ embedding_response, model_name
+ )
+
cache_key, cached_data, kwargs = self._add_cache_logic(
result=embedding_dict,
**kwargs,
)
return cache_key, cached_data, kwargs
- async def async_add_cache_pipeline(self, result, **kwargs):
+ async def async_add_cache_pipeline(
+ self, result, dynamic_cache_object: Optional[BaseCache] = None, **kwargs
+ ):
"""
Async implementation of add_cache for Embedding calls
@@ -672,14 +721,14 @@ class Cache:
)
cache_list.append((cache_key, cached_data))
- await self.cache.async_set_cache_pipeline(cache_list=cache_list, **kwargs)
- # if async_set_cache_pipeline:
- # await async_set_cache_pipeline(cache_list=cache_list, **kwargs)
- # else:
- # tasks = []
- # for val in cache_list:
- # tasks.append(self.cache.async_set_cache(val[0], val[1], **kwargs))
- # await asyncio.gather(*tasks)
+ if dynamic_cache_object is not None:
+ await dynamic_cache_object.async_set_cache_pipeline(
+ cache_list=cache_list, **kwargs
+ )
+ else:
+ await self.cache.async_set_cache_pipeline(
+ cache_list=cache_list, **kwargs
+ )
except Exception as e:
verbose_logger.exception(f"LiteLLM Cache: Excepton add_cache: {str(e)}")
@@ -725,11 +774,9 @@ class Cache:
"""
Internal method to check if the cache type supports async get/set operations
- Only S3 Cache Does NOT support async operations
+ All cache types now support async operations
"""
- if self.type and self.type == LiteLLMCacheType.S3:
- return False
return True
diff --git a/litellm/caching/caching_handler.py b/litellm/caching/caching_handler.py
index dcc59b20714..6bbc3231224 100644
--- a/litellm/caching/caching_handler.py
+++ b/litellm/caching/caching_handler.py
@@ -1,5 +1,5 @@
"""
-This contains LLMCachingHandler
+This contains LLMCachingHandler
This exposes two methods:
- async_get_cache
@@ -17,7 +17,7 @@ In each method it will call the appropriate method from caching.py
import asyncio
import datetime
import inspect
-import threading
+import time
from typing import (
TYPE_CHECKING,
Any,
@@ -35,13 +35,18 @@ from pydantic import BaseModel
import litellm
from litellm._logging import print_verbose, verbose_logger
+from litellm.caching import InMemoryCache
from litellm.caching.caching import S3Cache
-from litellm.types.caching import CachedEmbedding
+from litellm.litellm_core_utils.llm_response_utils.response_metadata import (
+ update_response_metadata,
+)
from litellm.litellm_core_utils.logging_utils import (
_assemble_complete_response_from_streaming_chunks,
)
+from litellm.types.caching import CachedEmbedding
from litellm.types.rerank import RerankResponse
from litellm.types.utils import (
+ CachingDetails,
CallTypes,
Embedding,
EmbeddingResponse,
@@ -53,10 +58,14 @@ from litellm.types.utils import (
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
- from litellm.utils import CustomStreamWrapper
else:
LiteLLMLoggingObj = Any
- CustomStreamWrapper = Any
+
+
+from litellm.litellm_core_utils.core_helpers import (
+ _get_parent_otel_span_from_kwargs,
+)
+from litellm.litellm_core_utils.streaming_handler import CustomStreamWrapper
class CachingHandlerResponse(BaseModel):
@@ -68,7 +77,12 @@ class CachingHandlerResponse(BaseModel):
cached_result: Optional[Any] = None
final_embedding_cached_response: Optional[EmbeddingResponse] = None
- embedding_all_elements_cache_hit: bool = False # this is set to True when all elements in the list have a cache hit in the embedding cache, if true return the final_embedding_cached_response no need to make an API call
+ embedding_all_elements_cache_hit: bool = (
+ False # this is set to True when all elements in the list have a cache hit in the embedding cache, if true return the final_embedding_cached_response no need to make an API call
+ )
+
+
+in_memory_cache_obj = InMemoryCache()
class LLMCachingHandler:
@@ -78,11 +92,20 @@ class LLMCachingHandler:
request_kwargs: Dict[str, Any],
start_time: datetime.datetime,
):
+ from litellm.caching import DualCache, RedisCache
+
self.async_streaming_chunks: List[ModelResponse] = []
self.sync_streaming_chunks: List[ModelResponse] = []
self.request_kwargs = request_kwargs
self.original_function = original_function
self.start_time = start_time
+ if litellm.cache is not None and isinstance(litellm.cache.cache, RedisCache):
+ self.dual_cache: Optional[DualCache] = DualCache(
+ redis_cache=litellm.cache.cache,
+ in_memory_cache=in_memory_cache_obj,
+ )
+ else:
+ self.dual_cache = None
pass
async def _async_get_cache(
@@ -94,7 +117,7 @@ class LLMCachingHandler:
call_type: str,
kwargs: Dict[str, Any],
args: Optional[Tuple[Any, ...]] = None,
- ) -> CachingHandlerResponse:
+ ) -> Optional[CachingHandlerResponse]:
"""
Internal method to get from the cache.
Handles different call types (embeddings, chat/completions, text_completion, transcription)
@@ -115,19 +138,27 @@ class LLMCachingHandler:
Raises:
None
"""
- from litellm.utils import CustomStreamWrapper
-
- args = args or ()
-
- final_embedding_cached_response: Optional[EmbeddingResponse] = None
- embedding_all_elements_cache_hit: bool = False
- cached_result: Optional[Any] = None
+ # Check if caching should be performed BEFORE doing expensive operations
if (
(kwargs.get("caching", None) is None and litellm.cache is not None)
or kwargs.get("caching", False) is True
) and (
kwargs.get("cache", {}).get("no-cache", False) is not True
): # allow users to control returning cached responses from the completion function
+ args = args or ()
+ final_embedding_cached_response: Optional[EmbeddingResponse] = None
+ embedding_all_elements_cache_hit: bool = False
+ cached_result: Optional[Any] = None
+ kwargs = kwargs.copy()
+ #########################################################
+ # Init cache timing metrics
+ #########################################################
+ cache_check_start_time = time.perf_counter()
+ cache_check_end_time: Optional[float] = None
+ #########################################################
+ parent_otel_span = _get_parent_otel_span_from_kwargs(kwargs)
+ kwargs["parent_otel_span"] = parent_otel_span
+
if litellm.cache is not None and self._is_call_type_supported_by_cache(
original_function=original_function
):
@@ -137,6 +168,7 @@ class LLMCachingHandler:
kwargs=kwargs,
args=args,
)
+ cache_check_end_time = time.perf_counter()
if cached_result is not None and not isinstance(cached_result, list):
verbose_logger.debug("Cache Hit!")
@@ -148,6 +180,7 @@ class LLMCachingHandler:
api_base=kwargs.get("api_base", None),
api_key=kwargs.get("api_key", None),
)
+ cache_duration_ms = (cache_check_end_time - cache_check_start_time) * 1000
self._update_litellm_logging_obj_environment(
logging_obj=logging_obj,
model=model,
@@ -155,10 +188,12 @@ class LLMCachingHandler:
cached_result=cached_result,
is_async=True,
custom_llm_provider=custom_llm_provider,
+ cache_duration_ms=cache_duration_ms,
)
call_type = original_function.__name__
+
cached_result = self._convert_cached_result_to_model_response(
cached_result=cached_result,
call_type=call_type,
@@ -177,9 +212,7 @@ class LLMCachingHandler:
end_time=end_time,
cache_hit=cache_hit,
)
- cache_key = litellm.cache._get_preset_cache_key_from_kwargs(
- **kwargs
- )
+ cache_key = litellm.cache.get_cache_key(**kwargs)
if (
isinstance(cached_result, BaseModel)
or isinstance(cached_result, CustomStreamWrapper)
@@ -210,11 +243,14 @@ class LLMCachingHandler:
final_embedding_cached_response=final_embedding_cached_response,
embedding_all_elements_cache_hit=embedding_all_elements_cache_hit,
)
- verbose_logger.debug(f"CACHE RESULT: {cached_result}")
- return CachingHandlerResponse(
- cached_result=cached_result,
- final_embedding_cached_response=final_embedding_cached_response,
- )
+
+ verbose_logger.debug(f"CACHE RESULT: {cached_result}")
+ return CachingHandlerResponse(
+ cached_result=cached_result,
+ final_embedding_cached_response=final_embedding_cached_response,
+ )
+ # Caching disabled - return None to indicate no caching attempted
+ return None
def _sync_get_cache(
self,
@@ -228,18 +264,22 @@ class LLMCachingHandler:
) -> CachingHandlerResponse:
from litellm.utils import CustomStreamWrapper
- args = args or ()
- new_kwargs = kwargs.copy()
- new_kwargs.update(
- convert_args_to_kwargs(
- self.original_function,
- args,
- )
- )
+
cached_result: Optional[Any] = None
+
+ # Check if caching should be performed BEFORE doing expensive kwargs copy
if litellm.cache is not None and self._is_call_type_supported_by_cache(
original_function=original_function
):
+ args = args or ()
+ # Now that we confirmed caching will happen, prepare kwargs
+ new_kwargs = kwargs.copy()
+ new_kwargs.update(
+ convert_args_to_kwargs(
+ self.original_function,
+ args,
+ )
+ )
print_verbose("Checking Sync Cache")
cached_result = litellm.cache.get_cache(**new_kwargs)
if cached_result is not None:
@@ -280,13 +320,13 @@ class LLMCachingHandler:
is_async=False,
)
- threading.Thread(
- target=logging_obj.success_handler,
- args=(cached_result, start_time, end_time, cache_hit),
- ).start()
- cache_key = litellm.cache._get_preset_cache_key_from_kwargs(
- **kwargs
+ logging_obj.handle_sync_success_callbacks_for_async_calls(
+ result=cached_result,
+ start_time=start_time,
+ end_time=end_time,
+ cache_hit=cache_hit
)
+ cache_key = litellm.cache.get_cache_key(**kwargs)
if (
isinstance(cached_result, BaseModel)
or isinstance(cached_result, CustomStreamWrapper)
@@ -306,13 +346,15 @@ class LLMCachingHandler:
else:
raise ValueError("input must be a string or a list")
- def _extract_model_from_cached_results(self, non_null_list: List[Tuple[int, CachedEmbedding]]) -> Optional[str]:
+ def _extract_model_from_cached_results(
+ self, non_null_list: List[Tuple[int, CachedEmbedding]]
+ ) -> Optional[str]:
"""
Helper method to extract the model name from cached results.
-
+
Args:
non_null_list: List of (idx, cr) tuples where cr is the cached result dict
-
+
Returns:
Optional[str]: The model name if found, None otherwise
"""
@@ -507,15 +549,17 @@ class LLMCachingHandler:
end_time (datetime): The end time of the operation.
cache_hit (bool): Whether it was a cache hit.
"""
- asyncio.create_task(
- logging_obj.async_success_handler(
- cached_result, start_time, end_time, cache_hit
+ from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
+
+ GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue(
+ async_coroutine=logging_obj.async_success_handler(
+ result=cached_result, start_time=start_time, end_time=end_time, cache_hit=cache_hit
)
)
- threading.Thread(
- target=logging_obj.success_handler,
- args=(cached_result, start_time, end_time, cache_hit),
- ).start()
+
+ logging_obj.handle_sync_success_callbacks_for_async_calls(
+ result=cached_result, start_time=start_time, end_time=end_time, cache_hit=cache_hit
+ )
async def _retrieve_from_cache(
self, call_type: str, kwargs: Dict[str, Any], args: Tuple[Any, ...]
@@ -558,7 +602,12 @@ class LLMCachingHandler:
preset_cache_key = litellm.cache.get_cache_key(
**{**new_kwargs, "input": i}
)
- tasks.append(litellm.cache.async_get_cache(cache_key=preset_cache_key))
+ tasks.append(
+ litellm.cache.async_get_cache(
+ cache_key=preset_cache_key,
+ dynamic_cache_object=self.dual_cache,
+ )
+ )
cached_result = await asyncio.gather(*tasks)
## check if cached result is None ##
if cached_result is not None and isinstance(cached_result, list):
@@ -567,9 +616,14 @@ class LLMCachingHandler:
cached_result = None
else:
if litellm.cache._supports_async() is True:
- cached_result = await litellm.cache.async_get_cache(**new_kwargs)
- else: # for s3 caching. [NOT RECOMMENDED IN PROD - this will slow down responses since boto3 is sync]
- cached_result = litellm.cache.get_cache(**new_kwargs)
+ ## check if dual cache is supported ##
+ cached_result = await litellm.cache.async_get_cache(
+ dynamic_cache_object=self.dual_cache, **new_kwargs
+ )
+ else: # fallback for caches that don't support async
+ cached_result = litellm.cache.get_cache(
+ dynamic_cache_object=self.dual_cache, **new_kwargs
+ )
return cached_result
def _convert_cached_result_to_model_response(
@@ -680,6 +734,18 @@ class LLMCachingHandler:
and isinstance(cached_result._hidden_params, dict)
):
cached_result._hidden_params["cache_hit"] = True
+
+ #########################################################
+ # Add final timing metrics to the cached result
+ #########################################################
+ update_response_metadata(
+ result=cached_result,
+ logging_obj=logging_obj,
+ model=model,
+ kwargs=kwargs,
+ start_time=self.start_time,
+ end_time=datetime.datetime.now(),
+ )
return cached_result
def _convert_cached_stream_response(
@@ -735,6 +801,9 @@ class LLMCachingHandler:
Raises:
None
"""
+ from litellm.litellm_core_utils.core_helpers import (
+ _get_parent_otel_span_from_kwargs,
+ )
if litellm.cache is None:
return
@@ -746,6 +815,8 @@ class LLMCachingHandler:
args,
)
)
+ parent_otel_span = _get_parent_otel_span_from_kwargs(new_kwargs)
+ new_kwargs["parent_otel_span"] = parent_otel_span
# [OPTIONAL] ADD TO CACHE
if self._should_store_result_in_cache(
original_function=original_function, kwargs=new_kwargs
@@ -764,18 +835,16 @@ class LLMCachingHandler:
) # s3 doesn't support bulk writing. Exclude.
):
asyncio.create_task(
- litellm.cache.async_add_cache_pipeline(result, **new_kwargs)
+ litellm.cache.async_add_cache_pipeline(
+ result, dynamic_cache_object=self.dual_cache, **new_kwargs
+ )
)
- elif isinstance(litellm.cache.cache, S3Cache):
- threading.Thread(
- target=litellm.cache.add_cache,
- args=(result,),
- kwargs=new_kwargs,
- ).start()
else:
asyncio.create_task(
litellm.cache.async_add_cache(
- result.model_dump_json(), **new_kwargs
+ result.model_dump_json(),
+ dynamic_cache_object=self.dual_cache,
+ **new_kwargs,
)
)
else:
@@ -905,6 +974,7 @@ class LLMCachingHandler:
is_async: bool,
is_embedding: bool = False,
custom_llm_provider: Optional[str] = None,
+ cache_duration_ms: Optional[float] = None,
):
"""
Helper function to update the LiteLLMLoggingObj environment variables.
@@ -933,9 +1003,9 @@ class LLMCachingHandler:
}
if litellm.cache is not None:
- litellm_params[
- "preset_cache_key"
- ] = litellm.cache._get_preset_cache_key_from_kwargs(**kwargs)
+ litellm_params["preset_cache_key"] = (
+ litellm.cache._get_preset_cache_key_from_kwargs(**kwargs)
+ )
else:
litellm_params["preset_cache_key"] = None
@@ -956,6 +1026,11 @@ class LLMCachingHandler:
custom_llm_provider=custom_llm_provider,
)
+ logging_obj.caching_details = CachingDetails(
+ cache_hit=True,
+ cache_duration_ms=cache_duration_ms,
+ )
+
def convert_args_to_kwargs(
original_function: Callable,
diff --git a/litellm/caching/gcs_cache.py b/litellm/caching/gcs_cache.py
new file mode 100644
index 00000000000..88857ba0e70
--- /dev/null
+++ b/litellm/caching/gcs_cache.py
@@ -0,0 +1,97 @@
+"""GCS Cache implementation
+Supports syncing responses to Google Cloud Storage Buckets using HTTP requests.
+"""
+import json
+import asyncio
+from typing import Optional
+
+from litellm._logging import print_verbose, verbose_logger
+from litellm.integrations.gcs_bucket.gcs_bucket_base import GCSBucketBase
+from litellm.llms.custom_httpx.http_handler import (
+ get_async_httpx_client,
+ _get_httpx_client,
+ httpxSpecialProvider,
+)
+from .base_cache import BaseCache
+
+
+class GCSCache(BaseCache):
+ def __init__(self, bucket_name: Optional[str] = None, path_service_account: Optional[str] = None, gcs_path: Optional[str] = None) -> None:
+ super().__init__()
+ self.bucket_name = bucket_name or GCSBucketBase(bucket_name=None).BUCKET_NAME
+ self.path_service_account = path_service_account or GCSBucketBase(bucket_name=None).path_service_account_json
+ self.key_prefix = gcs_path.rstrip("/") + "/" if gcs_path else ""
+ # create httpx clients
+ self.async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.LoggingCallback)
+ self.sync_client = _get_httpx_client()
+
+ def _construct_headers(self) -> dict:
+ base = GCSBucketBase(bucket_name=self.bucket_name)
+ base.path_service_account_json = self.path_service_account
+ base.BUCKET_NAME = self.bucket_name
+ return base.sync_construct_request_headers()
+
+ def set_cache(self, key, value, **kwargs):
+ try:
+ print_verbose(f"LiteLLM SET Cache - GCS. Key={key}. Value={value}")
+ headers = self._construct_headers()
+ object_name = self.key_prefix + key
+ bucket_name = self.bucket_name
+ url = f"https://storage.googleapis.com/upload/storage/v1/b/{bucket_name}/o?uploadType=media&name={object_name}"
+ data = json.dumps(value)
+ self.sync_client.post(url=url, data=data, headers=headers)
+ except Exception as e:
+ print_verbose(f"GCS Caching: set_cache() - Got exception from GCS: {e}")
+
+ async def async_set_cache(self, key, value, **kwargs):
+ try:
+ headers = self._construct_headers()
+ object_name = self.key_prefix + key
+ bucket_name = self.bucket_name
+ url = f"https://storage.googleapis.com/upload/storage/v1/b/{bucket_name}/o?uploadType=media&name={object_name}"
+ data = json.dumps(value)
+ await self.async_client.post(url=url, data=data, headers=headers)
+ except Exception as e:
+ print_verbose(f"GCS Caching: async_set_cache() - Got exception from GCS: {e}")
+
+ def get_cache(self, key, **kwargs):
+ try:
+ headers = self._construct_headers()
+ object_name = self.key_prefix + key
+ bucket_name = self.bucket_name
+ url = f"https://storage.googleapis.com/storage/v1/b/{bucket_name}/o/{object_name}?alt=media"
+ response = self.sync_client.get(url=url, headers=headers)
+ if response.status_code == 200:
+ cached_response = json.loads(response.text)
+ verbose_logger.debug(
+ f"Got GCS Cache: key: {key}, cached_response {cached_response}. Type Response {type(cached_response)}"
+ )
+ return cached_response
+ return None
+ except Exception as e:
+ verbose_logger.error(f"GCS Caching: get_cache() - Got exception from GCS: {e}")
+
+ async def async_get_cache(self, key, **kwargs):
+ try:
+ headers = self._construct_headers()
+ object_name = self.key_prefix + key
+ bucket_name = self.bucket_name
+ url = f"https://storage.googleapis.com/storage/v1/b/{bucket_name}/o/{object_name}?alt=media"
+ response = await self.async_client.get(url=url, headers=headers)
+ if response.status_code == 200:
+ return json.loads(response.text)
+ return None
+ except Exception as e:
+ verbose_logger.error(f"GCS Caching: async_get_cache() - Got exception from GCS: {e}")
+
+ def flush_cache(self):
+ pass
+
+ async def disconnect(self):
+ pass
+
+ async def async_set_cache_pipeline(self, cache_list, **kwargs):
+ tasks = []
+ for val in cache_list:
+ tasks.append(self.async_set_cache(val[0], val[1], **kwargs))
+ await asyncio.gather(*tasks)
diff --git a/litellm/caching/in_memory_cache.py b/litellm/caching/in_memory_cache.py
index 47f911894a3..5239fa1f4b0 100644
--- a/litellm/caching/in_memory_cache.py
+++ b/litellm/caching/in_memory_cache.py
@@ -11,6 +11,7 @@ Has 4 methods:
import json
import sys
import time
+import heapq
from typing import TYPE_CHECKING, Any, List, Optional
if TYPE_CHECKING:
@@ -36,7 +37,7 @@ class InMemoryCache(BaseCache):
max_size_in_memory [int]: Maximum number of items in cache. done to prevent memory leaks. Use 200 items as a default
"""
self.max_size_in_memory = (
- max_size_in_memory or 200
+ max_size_in_memory if max_size_in_memory is not None else 200
) # set an upper bound of 200 items in-memory
self.default_ttl = default_ttl or 600
self.max_size_per_item = (
@@ -46,6 +47,7 @@ class InMemoryCache(BaseCache):
# in-memory cache
self.cache_dict: dict = {}
self.ttl_dict: dict = {}
+ self.expiration_heap: list[tuple[float, str]] = []
def check_value_size(self, value: Any):
"""
@@ -103,23 +105,44 @@ class InMemoryCache(BaseCache):
def evict_cache(self):
"""
Eviction policy:
- - check if any items in ttl_dict are expired -> remove them from ttl_dict and cache_dict
+ 1. First, remove expired items from ttl_dict and cache_dict
+ 2. If cache is still at or above max_size_in_memory, evict items with earliest expiration times
This guarantees the following:
- - 1. When item ttl not set: At minimumm each item will remain in memory for 5 minutes
- - 2. When ttl is set: the item will remain in memory for at least that amount of time
+ - 1. When item ttl not set: At minimum each item will remain in memory for the default ttl
+ - 2. When ttl is set: the item will remain in memory for at least that amount of time, unless cache size requires eviction
- 3. the size of in-memory cache is bounded
"""
- for key in list(self.ttl_dict.keys()):
- if self._is_key_expired(key):
+ current_time = time.time()
+
+ # Step 1: Remove expired or outdated items
+ while self.expiration_heap:
+ expiration_time, key = self.expiration_heap[0]
+
+ # Case 1: Heap entry is outdated
+ if expiration_time != self.ttl_dict.get(key):
+ heapq.heappop(self.expiration_heap)
+ # Case 2: Entry is valid but expired
+ elif expiration_time <= current_time:
+ heapq.heappop(self.expiration_heap)
+ self._remove_key(key)
+ else:
+ # Case 3: Entry is valid and not expired
+ break
+
+ # Step 2: Evict if cache is still full
+ while len(self.cache_dict) >= self.max_size_in_memory:
+ expiration_time, key = heapq.heappop(self.expiration_heap)
+ # Skip if key was removed or updated
+ if self.ttl_dict.get(key) == expiration_time:
self._remove_key(key)
- # de-reference the removed item
- # https://www.geeksforgeeks.org/diagnosing-and-fixing-memory-leaks-in-python/
- # One of the most common causes of memory leaks in Python is the retention of objects that are no longer being used.
- # This can occur when an object is referenced by another object, but the reference is never removed.
+ # de-reference the removed item
+ # https://www.geeksforgeeks.org/diagnosing-and-fixing-memory-leaks-in-python/
+ # One of the most common causes of memory leaks in Python is the retention of objects that are no longer being used.
+ # This can occur when an object is referenced by another object, but the reference is never removed.
def allow_ttl_override(self, key: str) -> bool:
"""
@@ -134,6 +157,10 @@ class InMemoryCache(BaseCache):
return False
def set_cache(self, key, value, **kwargs):
+ # Handle the edge case where max_size_in_memory is 0
+ if self.max_size_in_memory == 0:
+ return # Don't cache anything if max size is 0
+
if len(self.cache_dict) >= self.max_size_in_memory:
# only evict when cache is full
self.evict_cache()
@@ -144,8 +171,10 @@ class InMemoryCache(BaseCache):
if self.allow_ttl_override(key): # if ttl is not set, set it to default ttl
if "ttl" in kwargs and kwargs["ttl"] is not None:
self.ttl_dict[key] = time.time() + float(kwargs["ttl"])
+ heapq.heappush(self.expiration_heap, (self.ttl_dict[key], key))
else:
self.ttl_dict[key] = time.time() + self.default_ttl
+ heapq.heappush(self.expiration_heap, (self.ttl_dict[key], key))
async def async_set_cache(self, key, value, **kwargs):
self.set_cache(key=key, value=value, **kwargs)
@@ -236,6 +265,7 @@ class InMemoryCache(BaseCache):
def flush_cache(self):
self.cache_dict.clear()
self.ttl_dict.clear()
+ self.expiration_heap.clear()
async def disconnect(self):
pass
diff --git a/litellm/caching/qdrant_semantic_cache.py b/litellm/caching/qdrant_semantic_cache.py
index 32d4d8b0fdc..0e77b5a6c21 100644
--- a/litellm/caching/qdrant_semantic_cache.py
+++ b/litellm/caching/qdrant_semantic_cache.py
@@ -168,7 +168,7 @@ class QdrantSemanticCache(BaseCache):
def set_cache(self, key, value, **kwargs):
print_verbose(f"qdrant semantic-cache set_cache, kwargs: {kwargs}")
- import uuid
+ from litellm._uuid import uuid
# get the prompt
messages = kwargs["messages"]
@@ -279,7 +279,7 @@ class QdrantSemanticCache(BaseCache):
pass
async def async_set_cache(self, key, value, **kwargs):
- import uuid
+ from litellm._uuid import uuid
from litellm.proxy.proxy_server import llm_model_list, llm_router
diff --git a/litellm/caching/redis_cache.py b/litellm/caching/redis_cache.py
index b8091187bfa..af7468ba14c 100644
--- a/litellm/caching/redis_cache.py
+++ b/litellm/caching/redis_cache.py
@@ -19,6 +19,7 @@ from typing import TYPE_CHECKING, Any, List, Optional, Tuple, Union, cast
import litellm
from litellm._logging import print_verbose, verbose_logger
from litellm.litellm_core_utils.core_helpers import _get_parent_otel_span_from_kwargs
+from litellm.litellm_core_utils.coroutine_checker import coroutine_checker
from litellm.types.caching import RedisPipelineIncrementOperation
from litellm.types.services import ServiceTypes
@@ -43,6 +44,45 @@ else:
Span = Any
+def _get_call_stack_info(num_frames: int = 2) -> str:
+ """
+ Get the function names from the previous 1-2 functions in the call stack.
+
+ Args:
+ num_frames: Number of previous frames to include (default: 2)
+
+ Returns:
+ A string with format "current_function <- caller_function [<- grandparent_function]"
+ """
+ try:
+ current_frame = inspect.currentframe()
+ if current_frame is None:
+ return "unknown"
+
+ # Skip this function and the immediate caller (which sets call_type)
+ f_back = current_frame.f_back
+ if f_back is None:
+ return "unknown"
+ frame = f_back.f_back
+ if frame is None:
+ return "unknown"
+ function_names = []
+
+ for _ in range(num_frames):
+ if frame is None:
+ break
+ func_name = frame.f_code.co_name
+ function_names.append(func_name)
+ frame = frame.f_back
+
+ if not function_names:
+ return "unknown"
+
+ return " <- ".join(function_names)
+ except Exception:
+ return "unknown"
+
+
class RedisCache(BaseCache):
# if users don't provider one, use the default litellm cache
@@ -99,7 +139,7 @@ class RedisCache(BaseCache):
self.redis_flush_size = redis_flush_size
self.redis_version = "Unknown"
try:
- if not inspect.iscoroutinefunction(self.redis_client):
+ if not coroutine_checker.is_async_callable(self.redis_client):
self.redis_version = self.redis_client.info()["redis_version"] # type: ignore
except Exception:
pass
@@ -181,7 +221,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="set_cache",
+ call_type=f"set_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -205,7 +245,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="increment_cache",
+ call_type=f"increment_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -219,7 +259,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="increment_cache_ttl",
+ call_type=f"increment_cache_ttl <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -232,7 +272,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="increment_cache_expire",
+ call_type=f"increment_cache_expire <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -271,7 +311,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_scan_iter",
+ call_type=f"async_scan_iter <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -287,7 +327,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_scan_iter",
+ call_type=f"async_scan_iter <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -341,7 +381,7 @@ class RedisCache(BaseCache):
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
- call_type="async_set_cache",
+ call_type=f"async_set_cache <- {_get_call_stack_info()}",
)
)
verbose_logger.error(
@@ -374,7 +414,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_set_cache",
+ call_type=f"async_set_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -390,7 +430,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_set_cache",
+ call_type=f"async_set_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -463,7 +503,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_set_cache_pipeline",
+ call_type=f"async_set_cache_pipeline <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -479,7 +519,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_set_cache_pipeline",
+ call_type=f"async_set_cache_pipeline <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -528,7 +568,7 @@ class RedisCache(BaseCache):
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
- call_type="async_set_cache_sadd",
+ call_type=f"async_set_cache_sadd <- {_get_call_stack_info()}",
)
)
# NON blocking - notify users Redis is throwing an exception
@@ -554,7 +594,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_set_cache_sadd",
+ call_type=f"async_set_cache_sadd <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -568,7 +608,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_set_cache_sadd",
+ call_type=f"async_set_cache_sadd <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -620,7 +660,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_increment",
+ call_type=f"async_increment <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -636,7 +676,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_increment",
+ call_type=f"async_increment <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -683,7 +723,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="get_cache",
+ call_type=f"get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -745,7 +785,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="batch_get_cache",
+ call_type=f"batch_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -790,7 +830,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_get_cache",
+ call_type=f"async_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -806,7 +846,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_get_cache",
+ call_type=f"async_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -851,7 +891,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_batch_get_cache",
+ call_type=f"async_batch_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -879,7 +919,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_batch_get_cache",
+ call_type=f"async_batch_get_cache <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=parent_otel_span,
@@ -903,7 +943,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="sync_ping",
+ call_type=f"sync_ping <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
)
@@ -917,7 +957,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="sync_ping",
+ call_type=f"sync_ping <- {_get_call_stack_info()}",
)
verbose_logger.error(
f"LiteLLM Redis Cache PING: - Got exception from REDIS : {str(e)}"
@@ -938,7 +978,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_ping",
+ call_type=f"async_ping <- {_get_call_stack_info()}",
)
)
return response
@@ -952,7 +992,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_ping",
+ call_type=f"async_ping <- {_get_call_stack_info()}",
)
)
verbose_logger.error(
@@ -1051,7 +1091,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_increment_pipeline",
+ call_type=f"async_increment_pipeline <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -1067,7 +1107,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_increment_pipeline",
+ call_type=f"async_increment_pipeline <- {_get_call_stack_info()}",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
@@ -1131,7 +1171,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_rpush",
+ call_type=f"async_rpush <- {_get_call_stack_info()}",
)
)
return response
@@ -1145,7 +1185,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_rpush",
+ call_type=f"async_rpush <- {_get_call_stack_info()}",
)
)
verbose_logger.error(
@@ -1202,7 +1242,7 @@ class RedisCache(BaseCache):
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.REDIS,
duration=_duration,
- call_type="async_lpop",
+ call_type=f"async_lpop <- {_get_call_stack_info()}",
)
)
@@ -1230,7 +1270,7 @@ class RedisCache(BaseCache):
service=ServiceTypes.REDIS,
duration=_duration,
error=e,
- call_type="async_lpop",
+ call_type=f"async_lpop <- {_get_call_stack_info()}",
)
)
verbose_logger.error(
diff --git a/litellm/caching/s3_cache.py b/litellm/caching/s3_cache.py
index c02e1091369..180964605f6 100644
--- a/litellm/caching/s3_cache.py
+++ b/litellm/caching/s3_cache.py
@@ -1,18 +1,19 @@
"""
S3 Cache implementation
-WARNING: DO NOT USE THIS IN PRODUCTION - This is not ASYNC
Has 4 methods:
- set_cache
- get_cache
- - async_set_cache
- - async_get_cache
+ - async_set_cache (uses run_in_executor)
+ - async_get_cache (uses run_in_executor)
"""
import ast
import asyncio
import json
+from functools import partial
from typing import Optional
+from datetime import datetime, timezone, timedelta
from litellm._logging import print_verbose, verbose_logger
@@ -55,21 +56,23 @@ class S3Cache(BaseCache):
**kwargs,
)
+ def _to_s3_key(self, key: str) -> str:
+ """Convert cache key to S3 key"""
+ return self.key_prefix + key.replace(":", "/")
+
def set_cache(self, key, value, **kwargs):
try:
print_verbose(f"LiteLLM SET Cache - S3. Key={key}. Value={value}")
ttl = kwargs.get("ttl", None)
# Convert value to JSON before storing in S3
serialized_value = json.dumps(value)
- key = self.key_prefix + key
+ key = self._to_s3_key(key)
if ttl is not None:
cache_control = f"immutable, max-age={ttl}, s-maxage={ttl}"
- import datetime
# Calculate expiration time
- expiration_time = datetime.datetime.now() + ttl
-
+ expiration_time = datetime.now(timezone.utc) + timedelta(seconds=ttl)
# Upload the data to S3 with the calculated expiration time
self.s3_client.put_object(
Bucket=self.bucket_name,
@@ -94,17 +97,26 @@ class S3Cache(BaseCache):
ContentDisposition=f'inline; filename="{key}.json"',
)
except Exception as e:
- # NON blocking - notify users S3 is throwing an exception
print_verbose(f"S3 Caching: set_cache() - Got exception from S3: {e}")
async def async_set_cache(self, key, value, **kwargs):
- self.set_cache(key=key, value=value, **kwargs)
+ """
+ Asynchronously set cache using run_in_executor to avoid blocking the event loop.
+ Compatible with Python 3.8+.
+ """
+ try:
+ verbose_logger.debug(f"Set ASYNC S3 Cache: Key={key}. Value={value}")
+ loop = asyncio.get_event_loop()
+ func = partial(self.set_cache, key, value, **kwargs)
+ await loop.run_in_executor(None, func)
+ except Exception as e:
+ verbose_logger.error(f"S3 Caching: async_set_cache() - Got exception from S3: {e}")
def get_cache(self, key, **kwargs):
import botocore
try:
- key = self.key_prefix + key
+ key = self._to_s3_key(key)
print_verbose(f"Get S3 Cache: key: {key}")
# Download the data from S3
@@ -113,6 +125,13 @@ class S3Cache(BaseCache):
)
if cached_response is not None:
+ if "Expires" in cached_response:
+ expires_time = cached_response['Expires']
+ current_time = datetime.now(expires_time.tzinfo)
+
+ if current_time > expires_time:
+ return None
+
# cached_response is in `b{} convert it to ModelResponse
cached_response = (
cached_response["Body"].read().decode("utf-8")
@@ -138,13 +157,26 @@ class S3Cache(BaseCache):
return None
except Exception as e:
- # NON blocking - notify users S3 is throwing an exception
verbose_logger.error(
f"S3 Caching: get_cache() - Got exception from S3: {e}"
)
async def async_get_cache(self, key, **kwargs):
- return self.get_cache(key=key, **kwargs)
+ """
+ Asynchronously get cache using run_in_executor to avoid blocking the event loop.
+ Compatible with Python 3.8+.
+ """
+ try:
+ verbose_logger.debug(f"Get ASYNC S3 Cache: key: {key}")
+ loop = asyncio.get_event_loop()
+ func = partial(self.get_cache, key, **kwargs)
+ result = await loop.run_in_executor(None, func)
+ return result
+ except Exception as e:
+ verbose_logger.error(
+ f"S3 Caching: async_get_cache() - Got exception from S3: {e}"
+ )
+ return None
def flush_cache(self):
pass
diff --git a/litellm/completion_extras/litellm_responses_transformation/handler.py b/litellm/completion_extras/litellm_responses_transformation/handler.py
index f2eeaf04554..6ec49ce0620 100644
--- a/litellm/completion_extras/litellm_responses_transformation/handler.py
+++ b/litellm/completion_extras/litellm_responses_transformation/handler.py
@@ -2,7 +2,9 @@
Handler for transforming /chat/completions api requests to litellm.responses requests
"""
-from typing import TYPE_CHECKING, Any, Coroutine, TypedDict, Union
+from typing import TYPE_CHECKING, Any, Coroutine, Union
+
+from typing_extensions import TypedDict
if TYPE_CHECKING:
from litellm import CustomStreamWrapper, LiteLLMLoggingObj, ModelResponse
diff --git a/litellm/completion_extras/litellm_responses_transformation/transformation.py b/litellm/completion_extras/litellm_responses_transformation/transformation.py
index f35510e41ba..b060f22d355 100644
--- a/litellm/completion_extras/litellm_responses_transformation/transformation.py
+++ b/litellm/completion_extras/litellm_responses_transformation/transformation.py
@@ -18,13 +18,15 @@ from typing import (
cast,
)
+from openai.types.responses.tool_param import FunctionToolParam
+
from litellm import ModelResponse
from litellm._logging import verbose_logger
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
from litellm.llms.base_llm.bridges.completion_transformation import (
CompletionTransformationBridge,
)
-from litellm.types.llms.openai import Reasoning
+from litellm.types.llms.openai import ChatCompletionToolParamFunctionChunk, Reasoning
if TYPE_CHECKING:
from openai.types.responses import ResponseInputImageParam
@@ -50,6 +52,45 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
def __init__(self):
pass
+ def _handle_raw_dict_response_item(
+ self, item: Dict[str, Any], index: int
+ ) -> Tuple[Optional[Any], int]:
+ """
+ Handle raw dict response items from Responses API (e.g., GPT-5 Codex format).
+
+ Args:
+ item: Raw dict response item with 'type' field
+ index: Current choice index
+
+ Returns:
+ Tuple of (Choice object or None, updated index)
+ """
+ from litellm.types.utils import Choices, Message
+
+ item_type = item.get("type")
+
+ # Ignore reasoning items for now
+ if item_type == "reasoning":
+ return None, index
+
+ # Handle message items with output_text content
+ if item_type == "message":
+ content_list = item.get("content", [])
+ for content_item in content_list:
+ if isinstance(content_item, dict):
+ content_type = content_item.get("type")
+ if content_type == "output_text":
+ response_text = content_item.get("text", "")
+ msg = Message(
+ role=item.get("role", "assistant"),
+ content=response_text if response_text else "",
+ )
+ choice = Choices(message=msg, finish_reason="stop", index=index)
+ return choice, index + 1
+
+ # Unknown or unsupported type
+ return None, index
+
def convert_chat_completion_messages_to_responses_api(
self, messages: List["AllMessageValues"]
) -> Tuple[List[Any], Optional[str]]:
@@ -201,6 +242,11 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
if value is not None:
if key == "instructions" and instructions:
request_data["instructions"] = instructions
+ elif key == "stream_options" and isinstance(value, dict):
+ request_data["stream_options"] = value.get("include_obfuscation")
+ elif key == "user": # string can't be longer than 64 characters
+ if isinstance(value, str) and len(value) <= 64:
+ request_data["user"] = value
else:
request_data[key] = value
@@ -221,7 +267,6 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
json_mode: Optional[bool] = None,
) -> "ModelResponse":
"""Transform Responses API response to chat completion response"""
-
from openai.types.responses import (
ResponseFunctionToolCall,
ResponseOutputMessage,
@@ -240,19 +285,35 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
choices: List[Choices] = []
index = 0
+
+ reasoning_content: Optional[str] = None
+
for item in raw_response.output:
+
if isinstance(item, ResponseReasoningItem):
- pass # ignore for now.
+
+ for summary_item in item.summary:
+ response_text = getattr(summary_item, "text", "")
+ reasoning_content = response_text if response_text else ""
+
elif isinstance(item, ResponseOutputMessage):
for content in item.content:
response_text = getattr(content, "text", "")
msg = Message(
- role=item.role, content=response_text if response_text else ""
+ role=item.role,
+ content=response_text if response_text else "",
+ reasoning_content=reasoning_content,
)
choices.append(
- Choices(message=msg, finish_reason="stop", index=index)
+ Choices(
+ message=msg,
+ finish_reason="stop",
+ index=index,
+ )
)
+
+ reasoning_content = None # flush reasoning content
index += 1
elif isinstance(item, ResponseFunctionToolCall):
msg = Message(
@@ -267,12 +328,21 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
"type": "function",
}
],
+ reasoning_content=reasoning_content,
)
choices.append(
Choices(message=msg, finish_reason="tool_calls", index=index)
)
+ reasoning_content = None # flush reasoning content
index += 1
+ elif isinstance(item, dict):
+ # Handle raw dict responses (e.g., from GPT-5 Codex)
+ choice, index = self._handle_raw_dict_response_item(
+ item=item, index=index
+ )
+ if choice is not None:
+ choices.append(choice)
else:
pass # don't fail request if item in list is not supported
@@ -447,9 +517,25 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
self, tools: List[Dict[str, Any]]
) -> List["ALL_RESPONSES_API_TOOL_PARAMS"]:
"""Convert chat completion tools to responses API tools format"""
- responses_tools = []
+ responses_tools: List["ALL_RESPONSES_API_TOOL_PARAMS"] = []
for tool in tools:
- responses_tools.append(tool)
+ # convert function tool from chat completion to responses API format
+ if tool.get("type") == "function":
+ function_tool = cast(
+ ChatCompletionToolParamFunctionChunk, tool.get("function")
+ )
+ responses_tools.append(
+ FunctionToolParam(
+ name=function_tool["name"],
+ parameters=function_tool.get("parameters"),
+ strict=function_tool.get("strict"),
+ type="function",
+ description=function_tool.get("description"),
+ )
+ )
+ else:
+ responses_tools.append(tool) # type: ignore
+
return cast(List["ALL_RESPONSES_API_TOOL_PARAMS"], responses_tools)
def _map_reasoning_effort(self, reasoning_effort: str) -> Optional[Reasoning]:
@@ -460,6 +546,8 @@ class LiteLLMResponsesTransformationHandler(CompletionTransformationBridge):
return Reasoning(effort="medium", summary="auto")
elif reasoning_effort == "low":
return Reasoning(effort="low", summary="auto")
+ elif reasoning_effort == "minimal":
+ return Reasoning(effort="minimal", summary="auto")
return None
def _map_responses_status_to_finish_reason(self, status: Optional[str]) -> str:
diff --git a/litellm/constants.py b/litellm/constants.py
index b25e2ff0fb9..54ac3e6d6b8 100644
--- a/litellm/constants.py
+++ b/litellm/constants.py
@@ -1,6 +1,9 @@
import os
from typing import List, Literal
+AZURE_DEFAULT_RESPONSES_API_VERSION = str(
+ os.getenv("AZURE_DEFAULT_RESPONSES_API_VERSION", "preview")
+)
ROUTER_MAX_FALLBACKS = int(os.getenv("ROUTER_MAX_FALLBACKS", 5))
DEFAULT_BATCH_SIZE = int(os.getenv("DEFAULT_BATCH_SIZE", 512))
DEFAULT_FLUSH_INTERVAL_SECONDS = int(os.getenv("DEFAULT_FLUSH_INTERVAL_SECONDS", 5))
@@ -11,6 +14,9 @@ DEFAULT_S3_BATCH_SIZE = int(os.getenv("DEFAULT_S3_BATCH_SIZE", 512))
DEFAULT_SQS_FLUSH_INTERVAL_SECONDS = int(
os.getenv("DEFAULT_SQS_FLUSH_INTERVAL_SECONDS", 10)
)
+DEFAULT_NUM_WORKERS_LITELLM_PROXY = int(
+ os.getenv("DEFAULT_NUM_WORKERS_LITELLM_PROXY", 1)
+)
DEFAULT_SQS_BATCH_SIZE = int(os.getenv("DEFAULT_SQS_BATCH_SIZE", 512))
SQS_SEND_MESSAGE_ACTION = "SendMessage"
SQS_API_VERSION = "2012-11-05"
@@ -45,6 +51,25 @@ SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD = int(
DEFAULT_REASONING_EFFORT_DISABLE_THINKING_BUDGET = int(
os.getenv("DEFAULT_REASONING_EFFORT_DISABLE_THINKING_BUDGET", 0)
)
+
+# Gemini model-specific minimal thinking budget constants
+DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH = int(
+ os.getenv("DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH", 1)
+)
+DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_PRO = int(
+ os.getenv("DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_PRO", 128)
+)
+DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH_LITE = int(
+ os.getenv(
+ "DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH_LITE", 512
+ )
+)
+
+# Generic fallback for unknown models
+DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET = int(
+ os.getenv("DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET", 128)
+)
+
DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET = int(
os.getenv("DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET", 1024)
)
@@ -62,6 +87,35 @@ MAX_TOKEN_TRIMMING_ATTEMPTS = int(
########## Networking constants ##############################################################
_DEFAULT_TTL_FOR_HTTPX_CLIENTS = 3600 # 1 hour, re-use the same httpx client for 1 hour
+# Aiohttp connection pooling constants
+AIOHTTP_CONNECTOR_LIMIT = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT", 0))
+AIOHTTP_KEEPALIVE_TIMEOUT = int(os.getenv("AIOHTTP_KEEPALIVE_TIMEOUT", 120))
+AIOHTTP_TTL_DNS_CACHE = int(os.getenv("AIOHTTP_TTL_DNS_CACHE", 300))
+
+# SSL/TLS cipher configuration for faster handshakes
+# Strategy: Strongly prefer fast modern ciphers, but allow fallback to commonly supported ones
+# This balances performance with broad compatibility
+DEFAULT_SSL_CIPHERS = os.getenv(
+ "LITELLM_SSL_CIPHERS",
+ # Priority 1: TLS 1.3 ciphers (fastest, ~50ms handshake)
+ "TLS_AES_256_GCM_SHA384:" # Fastest observed in testing
+ "TLS_AES_128_GCM_SHA256:" # Slightly faster than 256-bit
+ "TLS_CHACHA20_POLY1305_SHA256:" # Fast on ARM/mobile
+ # Priority 2: TLS 1.2 ECDHE+GCM (fast, ~100ms handshake, widely supported)
+ "ECDHE-RSA-AES256-GCM-SHA384:"
+ "ECDHE-RSA-AES128-GCM-SHA256:"
+ "ECDHE-ECDSA-AES256-GCM-SHA384:"
+ "ECDHE-ECDSA-AES128-GCM-SHA256:"
+ # Priority 3: Additional modern ciphers (good balance)
+ "ECDHE-RSA-CHACHA20-POLY1305:"
+ "ECDHE-ECDSA-CHACHA20-POLY1305:"
+ # Priority 4: Widely compatible fallbacks (slower but universally supported)
+ "ECDHE-RSA-AES256-SHA384:" # Common fallback
+ "ECDHE-RSA-AES128-SHA256:" # Very widely supported
+ "AES256-GCM-SHA384:" # Non-PFS fallback (compatibility)
+ "AES128-GCM-SHA256", # Last resort (maximum compatibility)
+)
+
########### v2 Architecture constants for managing writing updates to the database ###########
REDIS_UPDATE_BUFFER_KEY = "litellm_spend_update_buffer"
REDIS_DAILY_SPEND_UPDATE_BUFFER_KEY = "litellm_daily_spend_update_buffer"
@@ -154,6 +208,7 @@ NON_LLM_CONNECTION_TIMEOUT = int(
os.getenv("NON_LLM_CONNECTION_TIMEOUT", 15)
) # timeout for adjacent services (e.g. jwt auth)
MAX_EXCEPTION_MESSAGE_LENGTH = int(os.getenv("MAX_EXCEPTION_MESSAGE_LENGTH", 2000))
+MAX_STRING_LENGTH_PROMPT_IN_DB = int(os.getenv("MAX_STRING_LENGTH_PROMPT_IN_DB", 2048))
BEDROCK_MAX_POLICY_SIZE = int(os.getenv("BEDROCK_MAX_POLICY_SIZE", 75))
REPLICATE_POLLING_DELAY_SECONDS = float(
os.getenv("REPLICATE_POLLING_DELAY_SECONDS", 0.5)
@@ -224,6 +279,7 @@ LITELLM_CHAT_PROVIDERS = [
"together_ai",
"datarobot",
"openrouter",
+ "cometapi",
"vertex_ai",
"vertex_ai_beta",
"gemini",
@@ -247,6 +303,7 @@ LITELLM_CHAT_PROVIDERS = [
"groq",
"nvidia_nim",
"cerebras",
+ "baseten",
"ai21_chat",
"volcengine",
"codestral",
@@ -270,6 +327,7 @@ LITELLM_CHAT_PROVIDERS = [
"llamafile",
"lm_studio",
"galadriel",
+ "gradient_ai",
"github_copilot", # GitHub Copilot Chat API
"novita",
"meta_llama",
@@ -279,8 +337,14 @@ LITELLM_CHAT_PROVIDERS = [
"dashscope",
"moonshot",
"v0",
+ "heroku",
+ "oci",
"morph",
"lambda_ai",
+ "vercel_ai_gateway",
+ "wandb",
+ "ovhcloud",
+ "lemonade"
]
LITELLM_EMBEDDING_PROVIDERS_SUPPORTING_INPUT_ARRAY_OF_TOKENS = [
@@ -385,6 +449,7 @@ DEFAULT_CHAT_COMPLETION_PARAM_VALUES = {
"reasoning_effort": None,
"thinking": None,
"web_search_options": None,
+ "safety_identifier": None,
}
openai_compatible_endpoints: List = [
@@ -413,6 +478,8 @@ openai_compatible_endpoints: List = [
"https://api.morphllm.com/v1",
"https://api.lambda.ai/v1",
"https://api.hyperbolic.xyz/v1",
+ "https://ai-gateway.vercel.sh/v1",
+ "https://api.inference.wandb.ai/v1",
]
@@ -421,6 +488,7 @@ openai_compatible_providers: List = [
"groq",
"nvidia_nim",
"cerebras",
+ "baseten",
"sambanova",
"ai21_chat",
"ai21",
@@ -454,6 +522,9 @@ openai_compatible_providers: List = [
"morph",
"lambda_ai",
"hyperbolic",
+ "vercel_ai_gateway",
+ "aiml",
+ "wandb",
]
openai_text_completion_compatible_providers: List = (
[ # providers that support `/v1/completions`
@@ -469,6 +540,7 @@ openai_text_completion_compatible_providers: List = (
"v0",
"lambda_ai",
"hyperbolic",
+ "wandb",
]
)
_openai_like_providers: List = [
@@ -477,189 +549,279 @@ _openai_like_providers: List = [
"watsonx",
] # private helper. similar to openai but require some custom auth / endpoint handling, so can't use the openai sdk
# well supported replicate llms
-replicate_models: List = [
- # llama replicate supported LLMs
- "replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf",
- "a16z-infra/llama-2-13b-chat:2a7f981751ec7fdf87b5b91ad4db53683a98082e9ff7bfd12c8cd5ea85980a52",
- "meta/codellama-13b:1c914d844307b0588599b8393480a3ba917b660c7e9dfae681542b5325f228db",
- # Vicuna
- "replicate/vicuna-13b:6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b",
- "joehoover/instructblip-vicuna13b:c4c54e3c8c97cd50c2d2fec9be3b6065563ccf7d43787fb99f84151b867178fe",
- # Flan T-5
- "daanelson/flan-t5-large:ce962b3f6792a57074a601d3979db5839697add2e4e02696b3ced4c022d4767f",
- # Others
- "replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5",
- "replit/replit-code-v1-3b:b84f4c074b807211cd75e3e8b1589b6399052125b4c27106e43d47189e8415ad",
-]
+replicate_models: set = set(
+ [
+ # llama replicate supported LLMs
+ "replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf",
+ "a16z-infra/llama-2-13b-chat:2a7f981751ec7fdf87b5b91ad4db53683a98082e9ff7bfd12c8cd5ea85980a52",
+ "meta/codellama-13b:1c914d844307b0588599b8393480a3ba917b660c7e9dfae681542b5325f228db",
+ # Vicuna
+ "replicate/vicuna-13b:6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b",
+ "joehoover/instructblip-vicuna13b:c4c54e3c8c97cd50c2d2fec9be3b6065563ccf7d43787fb99f84151b867178fe",
+ # Flan T-5
+ "daanelson/flan-t5-large:ce962b3f6792a57074a601d3979db5839697add2e4e02696b3ced4c022d4767f",
+ # Others
+ "replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5",
+ "replit/replit-code-v1-3b:b84f4c074b807211cd75e3e8b1589b6399052125b4c27106e43d47189e8415ad",
+ ]
+)
-clarifai_models: List = [
- "clarifai/meta.Llama-3.Llama-3-8B-Instruct",
- "clarifai/gcp.generate.gemma-1_1-7b-it",
- "clarifai/mistralai.completion.mixtral-8x22B",
- "clarifai/cohere.generate.command-r-plus",
- "clarifai/databricks.drbx.dbrx-instruct",
- "clarifai/mistralai.completion.mistral-large",
- "clarifai/mistralai.completion.mistral-medium",
- "clarifai/mistralai.completion.mistral-small",
- "clarifai/mistralai.completion.mixtral-8x7B-Instruct-v0_1",
- "clarifai/gcp.generate.gemma-2b-it",
- "clarifai/gcp.generate.gemma-7b-it",
- "clarifai/deci.decilm.deciLM-7B-instruct",
- "clarifai/mistralai.completion.mistral-7B-Instruct",
- "clarifai/gcp.generate.gemini-pro",
- "clarifai/anthropic.completion.claude-v1",
- "clarifai/anthropic.completion.claude-instant-1_2",
- "clarifai/anthropic.completion.claude-instant",
- "clarifai/anthropic.completion.claude-v2",
- "clarifai/anthropic.completion.claude-2_1",
- "clarifai/meta.Llama-2.codeLlama-70b-Python",
- "clarifai/meta.Llama-2.codeLlama-70b-Instruct",
- "clarifai/openai.completion.gpt-3_5-turbo-instruct",
- "clarifai/meta.Llama-2.llama2-7b-chat",
- "clarifai/meta.Llama-2.llama2-13b-chat",
- "clarifai/meta.Llama-2.llama2-70b-chat",
- "clarifai/openai.chat-completion.gpt-4-turbo",
- "clarifai/microsoft.text-generation.phi-2",
- "clarifai/meta.Llama-2.llama2-7b-chat-vllm",
- "clarifai/upstage.solar.solar-10_7b-instruct",
- "clarifai/openchat.openchat.openchat-3_5-1210",
- "clarifai/togethercomputer.stripedHyena.stripedHyena-Nous-7B",
- "clarifai/gcp.generate.text-bison",
- "clarifai/meta.Llama-2.llamaGuard-7b",
- "clarifai/fblgit.una-cybertron.una-cybertron-7b-v2",
- "clarifai/openai.chat-completion.GPT-4",
- "clarifai/openai.chat-completion.GPT-3_5-turbo",
- "clarifai/ai21.complete.Jurassic2-Grande",
- "clarifai/ai21.complete.Jurassic2-Grande-Instruct",
- "clarifai/ai21.complete.Jurassic2-Jumbo-Instruct",
- "clarifai/ai21.complete.Jurassic2-Jumbo",
- "clarifai/ai21.complete.Jurassic2-Large",
- "clarifai/cohere.generate.cohere-generate-command",
- "clarifai/wizardlm.generate.wizardCoder-Python-34B",
- "clarifai/wizardlm.generate.wizardLM-70B",
- "clarifai/tiiuae.falcon.falcon-40b-instruct",
- "clarifai/togethercomputer.RedPajama.RedPajama-INCITE-7B-Chat",
- "clarifai/gcp.generate.code-gecko",
- "clarifai/gcp.generate.code-bison",
- "clarifai/mistralai.completion.mistral-7B-OpenOrca",
- "clarifai/mistralai.completion.openHermes-2-mistral-7B",
- "clarifai/wizardlm.generate.wizardLM-13B",
- "clarifai/huggingface-research.zephyr.zephyr-7B-alpha",
- "clarifai/wizardlm.generate.wizardCoder-15B",
- "clarifai/microsoft.text-generation.phi-1_5",
- "clarifai/databricks.Dolly-v2.dolly-v2-12b",
- "clarifai/bigcode.code.StarCoder",
- "clarifai/salesforce.xgen.xgen-7b-8k-instruct",
- "clarifai/mosaicml.mpt.mpt-7b-instruct",
- "clarifai/anthropic.completion.claude-3-opus",
- "clarifai/anthropic.completion.claude-3-sonnet",
- "clarifai/gcp.generate.gemini-1_5-pro",
- "clarifai/gcp.generate.imagen-2",
- "clarifai/salesforce.blip.general-english-image-caption-blip-2",
-]
+clarifai_models: set = set(
+ [
+ "clarifai/meta.Llama-3.Llama-3-8B-Instruct",
+ "clarifai/gcp.generate.gemma-1_1-7b-it",
+ "clarifai/mistralai.completion.mixtral-8x22B",
+ "clarifai/cohere.generate.command-r-plus",
+ "clarifai/databricks.drbx.dbrx-instruct",
+ "clarifai/mistralai.completion.mistral-large",
+ "clarifai/mistralai.completion.mistral-medium",
+ "clarifai/mistralai.completion.mistral-small",
+ "clarifai/mistralai.completion.mixtral-8x7B-Instruct-v0_1",
+ "clarifai/gcp.generate.gemma-2b-it",
+ "clarifai/gcp.generate.gemma-7b-it",
+ "clarifai/deci.decilm.deciLM-7B-instruct",
+ "clarifai/mistralai.completion.mistral-7B-Instruct",
+ "clarifai/gcp.generate.gemini-pro",
+ "clarifai/anthropic.completion.claude-v1",
+ "clarifai/anthropic.completion.claude-instant-1_2",
+ "clarifai/anthropic.completion.claude-instant",
+ "clarifai/anthropic.completion.claude-v2",
+ "clarifai/anthropic.completion.claude-2_1",
+ "clarifai/meta.Llama-2.codeLlama-70b-Python",
+ "clarifai/meta.Llama-2.codeLlama-70b-Instruct",
+ "clarifai/openai.completion.gpt-3_5-turbo-instruct",
+ "clarifai/meta.Llama-2.llama2-7b-chat",
+ "clarifai/meta.Llama-2.llama2-13b-chat",
+ "clarifai/meta.Llama-2.llama2-70b-chat",
+ "clarifai/openai.chat-completion.gpt-4-turbo",
+ "clarifai/microsoft.text-generation.phi-2",
+ "clarifai/meta.Llama-2.llama2-7b-chat-vllm",
+ "clarifai/upstage.solar.solar-10_7b-instruct",
+ "clarifai/openchat.openchat.openchat-3_5-1210",
+ "clarifai/togethercomputer.stripedHyena.stripedHyena-Nous-7B",
+ "clarifai/gcp.generate.text-bison",
+ "clarifai/meta.Llama-2.llamaGuard-7b",
+ "clarifai/fblgit.una-cybertron.una-cybertron-7b-v2",
+ "clarifai/openai.chat-completion.GPT-4",
+ "clarifai/openai.chat-completion.GPT-3_5-turbo",
+ "clarifai/ai21.complete.Jurassic2-Grande",
+ "clarifai/ai21.complete.Jurassic2-Grande-Instruct",
+ "clarifai/ai21.complete.Jurassic2-Jumbo-Instruct",
+ "clarifai/ai21.complete.Jurassic2-Jumbo",
+ "clarifai/ai21.complete.Jurassic2-Large",
+ "clarifai/cohere.generate.cohere-generate-command",
+ "clarifai/wizardlm.generate.wizardCoder-Python-34B",
+ "clarifai/wizardlm.generate.wizardLM-70B",
+ "clarifai/tiiuae.falcon.falcon-40b-instruct",
+ "clarifai/togethercomputer.RedPajama.RedPajama-INCITE-7B-Chat",
+ "clarifai/gcp.generate.code-gecko",
+ "clarifai/gcp.generate.code-bison",
+ "clarifai/mistralai.completion.mistral-7B-OpenOrca",
+ "clarifai/mistralai.completion.openHermes-2-mistral-7B",
+ "clarifai/wizardlm.generate.wizardLM-13B",
+ "clarifai/huggingface-research.zephyr.zephyr-7B-alpha",
+ "clarifai/wizardlm.generate.wizardCoder-15B",
+ "clarifai/microsoft.text-generation.phi-1_5",
+ "clarifai/databricks.Dolly-v2.dolly-v2-12b",
+ "clarifai/bigcode.code.StarCoder",
+ "clarifai/salesforce.xgen.xgen-7b-8k-instruct",
+ "clarifai/mosaicml.mpt.mpt-7b-instruct",
+ "clarifai/anthropic.completion.claude-3-opus",
+ "clarifai/anthropic.completion.claude-3-sonnet",
+ "clarifai/gcp.generate.gemini-1_5-pro",
+ "clarifai/gcp.generate.imagen-2",
+ "clarifai/salesforce.blip.general-english-image-caption-blip-2",
+ ]
+)
-huggingface_models: List = [
- "meta-llama/Llama-2-7b-hf",
- "meta-llama/Llama-2-7b-chat-hf",
- "meta-llama/Llama-2-13b-hf",
- "meta-llama/Llama-2-13b-chat-hf",
- "meta-llama/Llama-2-70b-hf",
- "meta-llama/Llama-2-70b-chat-hf",
- "meta-llama/Llama-2-7b",
- "meta-llama/Llama-2-7b-chat",
- "meta-llama/Llama-2-13b",
- "meta-llama/Llama-2-13b-chat",
- "meta-llama/Llama-2-70b",
- "meta-llama/Llama-2-70b-chat",
-] # these have been tested on extensively. But by default all text2text-generation and text-generation models are supported by liteLLM. - https://docs.litellm.ai/docs/providers
-empower_models = [
- "empower/empower-functions",
- "empower/empower-functions-small",
-]
+huggingface_models: set = set(
+ [
+ "meta-llama/Llama-2-7b-hf",
+ "meta-llama/Llama-2-7b-chat-hf",
+ "meta-llama/Llama-2-13b-hf",
+ "meta-llama/Llama-2-13b-chat-hf",
+ "meta-llama/Llama-2-70b-hf",
+ "meta-llama/Llama-2-70b-chat-hf",
+ "meta-llama/Llama-2-7b",
+ "meta-llama/Llama-2-7b-chat",
+ "meta-llama/Llama-2-13b",
+ "meta-llama/Llama-2-13b-chat",
+ "meta-llama/Llama-2-70b",
+ "meta-llama/Llama-2-70b-chat",
+ ]
+) # these have been tested on extensively. But by default all text2text-generation and text-generation models are supported by liteLLM. - https://docs.litellm.ai/docs/providers
+empower_models = set(
+ [
+ "empower/empower-functions",
+ "empower/empower-functions-small",
+ ]
+)
-together_ai_models: List = [
- # llama llms - chat
- "togethercomputer/llama-2-70b-chat",
- # llama llms - language / instruct
- "togethercomputer/llama-2-70b",
- "togethercomputer/LLaMA-2-7B-32K",
- "togethercomputer/Llama-2-7B-32K-Instruct",
- "togethercomputer/llama-2-7b",
- # falcon llms
- "togethercomputer/falcon-40b-instruct",
- "togethercomputer/falcon-7b-instruct",
- # alpaca
- "togethercomputer/alpaca-7b",
- # chat llms
- "HuggingFaceH4/starchat-alpha",
- # code llms
- "togethercomputer/CodeLlama-34b",
- "togethercomputer/CodeLlama-34b-Instruct",
- "togethercomputer/CodeLlama-34b-Python",
- "defog/sqlcoder",
- "NumbersStation/nsql-llama-2-7B",
- "WizardLM/WizardCoder-15B-V1.0",
- "WizardLM/WizardCoder-Python-34B-V1.0",
- # language llms
- "NousResearch/Nous-Hermes-Llama2-13b",
- "Austism/chronos-hermes-13b",
- "upstage/SOLAR-0-70b-16bit",
- "WizardLM/WizardLM-70B-V1.0",
-] # supports all together ai models, just pass in the model id e.g. completion(model="together_computer/replit_code_3b",...)
+together_ai_models: set = set(
+ [
+ # llama llms - chat
+ "togethercomputer/llama-2-70b-chat",
+ # llama llms - language / instruct
+ "togethercomputer/llama-2-70b",
+ "togethercomputer/LLaMA-2-7B-32K",
+ "togethercomputer/Llama-2-7B-32K-Instruct",
+ "togethercomputer/llama-2-7b",
+ # falcon llms
+ "togethercomputer/falcon-40b-instruct",
+ "togethercomputer/falcon-7b-instruct",
+ # alpaca
+ "togethercomputer/alpaca-7b",
+ # chat llms
+ "HuggingFaceH4/starchat-alpha",
+ # code llms
+ "togethercomputer/CodeLlama-34b",
+ "togethercomputer/CodeLlama-34b-Instruct",
+ "togethercomputer/CodeLlama-34b-Python",
+ "defog/sqlcoder",
+ "NumbersStation/nsql-llama-2-7B",
+ "WizardLM/WizardCoder-15B-V1.0",
+ "WizardLM/WizardCoder-Python-34B-V1.0",
+ # language llms
+ "NousResearch/Nous-Hermes-Llama2-13b",
+ "Austism/chronos-hermes-13b",
+ "upstage/SOLAR-0-70b-16bit",
+ "WizardLM/WizardLM-70B-V1.0",
+ ]
+)
+# supports all together ai models, just pass in the model id e.g. completion(model="together_computer/replit_code_3b",...)
-baseten_models: List = [
- "qvv0xeq",
- "q841o8w",
- "31dxrj3",
-] # FALCON 7B # WizardLM # Mosaic ML
+baseten_models: set = set(
+ [
+ "qvv0xeq",
+ "q841o8w",
+ "31dxrj3",
+ ]
+) # FALCON 7B # WizardLM # Mosaic ML
-featherless_ai_models: List = [
- "featherless-ai/Qwerky-72B",
- "featherless-ai/Qwerky-QwQ-32B",
- "Qwen/Qwen2.5-72B-Instruct",
- "all-hands/openhands-lm-32b-v0.1",
- "Qwen/Qwen2.5-Coder-32B-Instruct",
- "deepseek-ai/DeepSeek-V3-0324",
- "mistralai/Mistral-Small-24B-Instruct-2501",
- "mistralai/Mistral-Nemo-Instruct-2407",
- "ProdeusUnity/Stellar-Odyssey-12b-v0.0",
-]
+featherless_ai_models: set = set(
+ [
+ "featherless-ai/Qwerky-72B",
+ "featherless-ai/Qwerky-QwQ-32B",
+ "Qwen/Qwen2.5-72B-Instruct",
+ "all-hands/openhands-lm-32b-v0.1",
+ "Qwen/Qwen2.5-Coder-32B-Instruct",
+ "deepseek-ai/DeepSeek-V3-0324",
+ "mistralai/Mistral-Small-24B-Instruct-2501",
+ "mistralai/Mistral-Nemo-Instruct-2407",
+ "ProdeusUnity/Stellar-Odyssey-12b-v0.0",
+ ]
+)
-nebius_models: List = [
- "Qwen/Qwen3-235B-A22B",
- "Qwen/Qwen3-30B-A3B-fast",
- "Qwen/Qwen3-32B",
- "Qwen/Qwen3-14B",
- "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1",
- "deepseek-ai/DeepSeek-V3-0324",
- "deepseek-ai/DeepSeek-V3-0324-fast",
- "deepseek-ai/DeepSeek-R1",
- "deepseek-ai/DeepSeek-R1-fast",
- "meta-llama/Llama-3.3-70B-Instruct-fast",
- "Qwen/Qwen2.5-32B-Instruct-fast",
- "Qwen/Qwen2.5-Coder-32B-Instruct-fast",
-]
+nebius_models: set = set(
+ [
+ # deepseek models
+ "deepseek-ai/DeepSeek-R1-0528",
+ "deepseek-ai/DeepSeek-V3-0324",
+ "deepseek-ai/DeepSeek-V3",
+ "deepseek-ai/DeepSeek-R1",
+ "deepseek-ai/DeepSeek-R1-Distill-Llama-70B",
+ # google models
+ "google/gemma-2-2b-it",
+ "google/gemma-2-9b-it-fast",
+ # llama models
+ "meta-llama/Llama-3.3-70B-Instruct",
+ "meta-llama/Meta-Llama-3.1-70B-Instruct",
+ "meta-llama/Meta-Llama-3.1-8B-Instruct",
+ "meta-llama/Meta-Llama-3.1-405B-Instruct",
+ "NousResearch/Hermes-3-Llama-405B",
+ # microsoft models
+ "microsoft/phi-4",
+ # mistral models
+ "mistralai/Mistral-Nemo-Instruct-2407",
+ "mistralai/Devstral-Small-2505",
+ # moonshot models
+ "moonshotai/Kimi-K2-Instruct",
+ # nvidia models
+ "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1",
+ "nvidia/Llama-3_3-Nemotron-Super-49B-v1",
+ # openai models
+ "openai/gpt-oss-120b",
+ "openai/gpt-oss-20b",
+ # qwen models
+ "Qwen/Qwen3-Coder-480B-A35B-Instruct",
+ "Qwen/Qwen3-235B-A22B-Instruct-2507",
+ "Qwen/Qwen3-235B-A22B",
+ "Qwen/Qwen3-30B-A3B",
+ "Qwen/Qwen3-32B",
+ "Qwen/Qwen3-14B",
+ "Qwen/Qwen3-4B-fast",
+ "Qwen/Qwen2.5-Coder-7B",
+ "Qwen/Qwen2.5-Coder-32B-Instruct",
+ "Qwen/Qwen2.5-72B-Instruct",
+ "Qwen/QwQ-32B",
+ "Qwen/Qwen3-30B-A3B-Thinking-2507",
+ "Qwen/Qwen3-30B-A3B-Instruct-2507",
+ # zai models
+ "zai-org/GLM-4.5",
+ "zai-org/GLM-4.5-Air",
+ # other models
+ "aaditya/Llama3-OpenBioLLM-70B",
+ "ProdeusUnity/Stellar-Odyssey-12b-v0.0",
+ "all-hands/openhands-lm-32b-v0.1",
+ ]
+)
-dashscope_models: List = [
- "qwen-turbo",
- "qwen-plus",
- "qwen-max",
- "qwen-turbo-latest",
- "qwen-plus-latest",
- "qwen-max-latest",
- "qwq-32b",
- "qwen3-235b-a22b",
- "qwen3-32b",
- "qwen3-30b-a3b",
-]
+dashscope_models: set = set(
+ [
+ "qwen-turbo",
+ "qwen-plus",
+ "qwen-max",
+ "qwen-turbo-latest",
+ "qwen-plus-latest",
+ "qwen-max-latest",
+ "qwq-32b",
+ "qwen3-235b-a22b",
+ "qwen3-32b",
+ "qwen3-30b-a3b",
+ ]
+)
-nebius_embedding_models: List = [
- "BAAI/bge-en-icl",
- "BAAI/bge-multilingual-gemma2",
- "intfloat/e5-mistral-7b-instruct",
-]
+nebius_embedding_models: set = set(
+ [
+ "BAAI/bge-en-icl",
+ "BAAI/bge-multilingual-gemma2",
+ "intfloat/e5-mistral-7b-instruct",
+ ]
+)
+
+WANDB_MODELS: set = set(
+ [
+ # openai models
+ "openai/gpt-oss-120b",
+ "openai/gpt-oss-20b",
+
+ # zai-org models
+ "zai-org/GLM-4.5",
+
+ # Qwen models
+ "Qwen/Qwen3-235B-A22B-Instruct-2507",
+ "Qwen/Qwen3-Coder-480B-A35B-Instruct",
+ "Qwen/Qwen3-235B-A22B-Thinking-2507",
+
+ # moonshotai
+ "moonshotai/Kimi-K2-Instruct",
+
+ # meta models
+ "meta-llama/Llama-3.1-8B-Instruct",
+ "meta-llama/Llama-3.3-70B-Instruct",
+ "meta-llama/Llama-4-Scout-17B-16E-Instruct",
+
+ # deepseek-ai
+ "deepseek-ai/DeepSeek-V3.1",
+ "deepseek-ai/DeepSeek-R1-0528",
+ "deepseek-ai/DeepSeek-V3-0324",
+
+ # microsoft
+ "microsoft/Phi-4-mini-instruct",
+ ]
+)
BEDROCK_INVOKE_PROVIDERS_LITERAL = Literal[
"cohere",
@@ -673,22 +835,76 @@ BEDROCK_INVOKE_PROVIDERS_LITERAL = Literal[
"deepseek_r1",
]
-open_ai_embedding_models: List = ["text-embedding-ada-002"]
-cohere_embedding_models: List = [
- "embed-v4.0",
- "embed-english-v3.0",
- "embed-english-light-v3.0",
- "embed-multilingual-v3.0",
- "embed-english-v2.0",
- "embed-english-light-v2.0",
- "embed-multilingual-v2.0",
+BEDROCK_EMBEDDING_PROVIDERS_LITERAL = Literal[
+ "cohere",
+ "amazon",
+ "twelvelabs",
]
-bedrock_embedding_models: List = [
- "amazon.titan-embed-text-v1",
- "cohere.embed-english-v3",
- "cohere.embed-multilingual-v3",
+
+BEDROCK_CONVERSE_MODELS = [
+ "qwen.qwen3-coder-480b-a35b-v1:0",
+ "qwen.qwen3-235b-a22b-2507-v1:0",
+ "qwen.qwen3-coder-30b-a3b-v1:0",
+ "qwen.qwen3-32b-v1:0",
+ "deepseek.v3-v1:0",
+ "openai.gpt-oss-20b-1:0",
+ "openai.gpt-oss-120b-1:0",
+ "anthropic.claude-sonnet-4-5-20250929-v1:0",
+ "anthropic.claude-opus-4-1-20250805-v1:0",
+ "anthropic.claude-opus-4-20250514-v1:0",
+ "anthropic.claude-sonnet-4-20250514-v1:0",
+ "anthropic.claude-3-7-sonnet-20250219-v1:0",
+ "anthropic.claude-3-5-haiku-20241022-v1:0",
+ "anthropic.claude-3-5-sonnet-20241022-v2:0",
+ "anthropic.claude-3-5-sonnet-20240620-v1:0",
+ "anthropic.claude-3-opus-20240229-v1:0",
+ "anthropic.claude-3-sonnet-20240229-v1:0",
+ "anthropic.claude-3-haiku-20240307-v1:0",
+ "anthropic.claude-v2",
+ "anthropic.claude-v2:1",
+ "anthropic.claude-v1",
+ "anthropic.claude-instant-v1",
+ "ai21.jamba-instruct-v1:0",
+ "ai21.jamba-1-5-mini-v1:0",
+ "ai21.jamba-1-5-large-v1:0",
+ "meta.llama3-70b-instruct-v1:0",
+ "meta.llama3-8b-instruct-v1:0",
+ "meta.llama3-1-8b-instruct-v1:0",
+ "meta.llama3-1-70b-instruct-v1:0",
+ "meta.llama3-1-405b-instruct-v1:0",
+ "meta.llama3-70b-instruct-v1:0",
+ "mistral.mistral-large-2407-v1:0",
+ "mistral.mistral-large-2402-v1:0",
+ "mistral.mistral-small-2402-v1:0",
+ "meta.llama3-2-1b-instruct-v1:0",
+ "meta.llama3-2-3b-instruct-v1:0",
+ "meta.llama3-2-11b-instruct-v1:0",
+ "meta.llama3-2-90b-instruct-v1:0",
]
+
+open_ai_embedding_models: set = set(["text-embedding-ada-002"])
+cohere_embedding_models: set = set(
+ [
+ "embed-v4.0",
+ "embed-english-v3.0",
+ "embed-english-light-v3.0",
+ "embed-multilingual-v3.0",
+ "embed-english-v2.0",
+ "embed-english-light-v2.0",
+ "embed-multilingual-v2.0",
+ ]
+)
+bedrock_embedding_models: set = set(
+ [
+ "amazon.titan-embed-text-v1",
+ "cohere.embed-english-v3",
+ "cohere.embed-multilingual-v3",
+ "cohere.embed-v4:0",
+ "twelvelabs.marengo-embed-2-7-v1:0",
+ ]
+)
+
known_tokenizer_config = {
"mistralai/Mistral-7B-Instruct-v0.1": {
"tokenizer": {
@@ -758,6 +974,9 @@ AZURE_STORAGE_MSFT_VERSION = "2019-07-07"
PROMETHEUS_BUDGET_METRICS_REFRESH_INTERVAL_MINUTES = int(
os.getenv("PROMETHEUS_BUDGET_METRICS_REFRESH_INTERVAL_MINUTES", 5)
)
+CLOUDZERO_EXPORT_INTERVAL_MINUTES = int(
+ os.getenv("CLOUDZERO_EXPORT_INTERVAL_MINUTES", 60)
+)
MCP_TOOL_NAME_PREFIX = "mcp_tool"
MAXIMUM_TRACEBACK_LINES_TO_LOG = int(os.getenv("MAXIMUM_TRACEBACK_LINES_TO_LOG", 100))
@@ -765,6 +984,7 @@ MAXIMUM_TRACEBACK_LINES_TO_LOG = int(os.getenv("MAXIMUM_TRACEBACK_LINES_TO_LOG",
X_LITELLM_DISABLE_CALLBACKS = "x-litellm-disable-callbacks"
LITELLM_METADATA_FIELD = "litellm_metadata"
OLD_LITELLM_METADATA_FIELD = "metadata"
+LITELLM_TRUNCATED_PAYLOAD_FIELD = "litellm_truncated"
########################### LiteLLM Proxy Specific Constants ###########################
########################################################################################
@@ -800,7 +1020,12 @@ HEALTH_CHECK_TIMEOUT_SECONDS = int(
os.getenv("HEALTH_CHECK_TIMEOUT_SECONDS", 60)
) # 60 seconds
LITTELM_INTERNAL_HEALTH_SERVICE_ACCOUNT_NAME = "litellm-internal-health-check"
+LITTELM_CLI_SERVICE_ACCOUNT_NAME = "litellm-cli"
+LITELLM_INTERNAL_JOBS_SERVICE_ACCOUNT_NAME = "litellm_internal_jobs"
+# Key Rotation Constants
+LITELLM_KEY_ROTATION_ENABLED = os.getenv("LITELLM_KEY_ROTATION_ENABLED", "false")
+LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDS = int(os.getenv("LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDS", 86400)) # 24 hours default
UI_SESSION_TOKEN_TEAM_ID = "litellm-dashboard"
LITELLM_PROXY_ADMIN_NAME = "default_user_id"
@@ -811,6 +1036,10 @@ LITELLM_CLI_SESSION_TOKEN_PREFIX = "litellm-session-token"
########################### DB CRON JOB NAMES ###########################
DB_SPEND_UPDATE_JOB_NAME = "db_spend_update_job"
PROMETHEUS_EMIT_BUDGET_METRICS_JOB_NAME = "prometheus_emit_budget_metrics"
+CLOUDZERO_EXPORT_USAGE_DATA_JOB_NAME = "cloudzero_export_usage_data"
+CLOUDZERO_MAX_FETCHED_DATA_RECORDS = int(
+ os.getenv("CLOUDZERO_MAX_FETCHED_DATA_RECORDS", 50000)
+)
SPEND_LOG_CLEANUP_JOB_NAME = "spend_log_cleanup"
SPEND_LOG_RUN_LOOPS = int(os.getenv("SPEND_LOG_RUN_LOOPS", 500))
SPEND_LOG_CLEANUP_BATCH_SIZE = int(os.getenv("SPEND_LOG_CLEANUP_BATCH_SIZE", 1000))
@@ -828,6 +1057,12 @@ PROXY_BATCH_WRITE_AT = int(os.getenv("PROXY_BATCH_WRITE_AT", 10)) # in seconds
DEFAULT_HEALTH_CHECK_INTERVAL = int(
os.getenv("DEFAULT_HEALTH_CHECK_INTERVAL", 300)
) # 5 minutes
+DEFAULT_SHARED_HEALTH_CHECK_TTL = int(
+ os.getenv("DEFAULT_SHARED_HEALTH_CHECK_TTL", 300)
+) # 5 minutes - TTL for cached health check results
+DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTL = int(
+ os.getenv("DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTL", 60)
+) # 1 minute - TTL for health check lock
PROMETHEUS_FALLBACK_STATS_SEND_TIME_HOURS = int(
os.getenv("PROMETHEUS_FALLBACK_STATS_SEND_TIME_HOURS", 9)
)
@@ -875,10 +1110,12 @@ SENTRY_DENYLIST = [
"CLOUDFLARE_API_KEY",
"BASETEN_KEY",
"OPENROUTER_KEY",
+ "COMETAPI_KEY",
"DATAROBOT_API_TOKEN",
"FIREWORKS_API_KEY",
"FIREWORKS_AI_API_KEY",
"FIREWORKSAI_API_KEY",
+ "OVHCLOUD_API_KEY",
# Database and Connection Strings
"database_url",
"redis_url",
@@ -917,3 +1154,8 @@ SENTRY_PII_DENYLIST = [
"SMTP_SENDER_EMAIL",
"TEST_EMAIL_ADDRESS",
]
+
+# CoroutineChecker cache configuration
+COROUTINE_CHECKER_MAX_SIZE_IN_MEMORY = int(
+ os.getenv("COROUTINE_CHECKER_MAX_SIZE_IN_MEMORY", 1000)
+)
diff --git a/litellm/cost_calculator.py b/litellm/cost_calculator.py
index c8892cd26a5..4bb14eb8391 100644
--- a/litellm/cost_calculator.py
+++ b/litellm/cost_calculator.py
@@ -32,9 +32,6 @@ from litellm.llms.azure.cost_calculation import (
from litellm.llms.bedrock.cost_calculation import (
cost_per_token as bedrock_cost_per_token,
)
-from litellm.llms.bedrock.image.cost_calculator import (
- cost_calculator as bedrock_image_cost_calculator,
-)
from litellm.llms.databricks.cost_calculator import (
cost_per_token as databricks_cost_per_token,
)
@@ -60,8 +57,9 @@ from litellm.llms.vertex_ai.cost_calculator import (
cost_per_token as google_cost_per_token,
)
from litellm.llms.vertex_ai.cost_calculator import cost_router as google_cost_router
-from litellm.llms.vertex_ai.image_generation.cost_calculator import (
- cost_calculator as vertex_ai_image_cost_calculator,
+from litellm.llms.xai.cost_calculator import cost_per_token as xai_cost_per_token
+from litellm.llms.lemonade.cost_calculator import (
+ cost_per_token as lemonade_cost_per_token,
)
from litellm.responses.utils import ResponseAPILoggingUtils
from litellm.types.llms.openai import (
@@ -153,6 +151,8 @@ def cost_per_token( # noqa: PLR0915
### CALL TYPE ###
call_type: CallTypesLiteral = "completion",
audio_transcription_file_duration: float = 0.0, # for audio transcription calls - the file time in seconds
+ ### SERVICE TIER ###
+ service_tier: Optional[str] = None, # for OpenAI service tier pricing
) -> Tuple[float, float]: # type: ignore
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@@ -283,6 +283,7 @@ def cost_per_token( # noqa: PLR0915
model=model_without_prefix,
usage=usage_block,
custom_llm_provider=custom_llm_provider,
+ service_tier=service_tier,
)
return prompt_cost, completion_cost
@@ -332,7 +333,7 @@ def cost_per_token( # noqa: PLR0915
elif custom_llm_provider == "bedrock":
return bedrock_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "openai":
- return openai_cost_per_token(model=model, usage=usage_block)
+ return openai_cost_per_token(model=model, usage=usage_block, service_tier=service_tier)
elif custom_llm_provider == "databricks":
return databricks_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "fireworks_ai":
@@ -347,6 +348,15 @@ def cost_per_token( # noqa: PLR0915
return deepseek_cost_per_token(model=model, usage=usage_block)
elif custom_llm_provider == "perplexity":
return perplexity_cost_per_token(model=model, usage=usage_block)
+ elif custom_llm_provider == "xai":
+ return xai_cost_per_token(model=model, usage=usage_block)
+ elif custom_llm_provider == "lemonade":
+ return lemonade_cost_per_token(model=model, usage=usage_block)
+ elif custom_llm_provider == "dashscope":
+ from litellm.llms.dashscope.cost_calculator import (
+ cost_per_token as dashscope_cost_per_token,
+ )
+ return dashscope_cost_per_token(model=model, usage=usage_block)
else:
model_info = _cached_get_model_info_helper(
model=model, custom_llm_provider=custom_llm_provider
@@ -579,6 +589,42 @@ def _infer_call_type(
return call_type
+def _store_cost_breakdown_in_logging_obj(
+ litellm_logging_obj: Optional[LitellmLoggingObject],
+ prompt_tokens_cost_usd_dollar: float,
+ completion_tokens_cost_usd_dollar: float,
+ cost_for_built_in_tools_cost_usd_dollar: float,
+ total_cost_usd_dollar: float,
+) -> None:
+ """
+ Helper function to store cost breakdown in the logging object.
+
+ Args:
+ litellm_logging_obj: The logging object to store breakdown in
+ call_type: Type of call (completion, etc.)
+ prompt_tokens_cost_usd_dollar: Cost of input tokens
+ completion_tokens_cost_usd_dollar: Cost of completion tokens (includes reasoning if applicable)
+ cost_for_built_in_tools_cost_usd_dollar: Cost of built-in tools
+ total_cost_usd_dollar: Total cost of request
+ """
+ if (litellm_logging_obj is None):
+ return
+
+ try:
+ # Store the cost breakdown - reasoning cost is 0 since it's already included in completion cost
+ litellm_logging_obj.set_cost_breakdown(
+ input_cost=prompt_tokens_cost_usd_dollar,
+ output_cost=completion_tokens_cost_usd_dollar,
+ total_cost=total_cost_usd_dollar,
+ cost_for_built_in_tools_cost_usd_dollar=cost_for_built_in_tools_cost_usd_dollar
+ )
+
+ except Exception as breakdown_error:
+ verbose_logger.debug(f"Error storing cost breakdown: {str(breakdown_error)}")
+ # Don't fail the main cost calculation if breakdown storage fails
+ pass
+
+
def completion_cost( # noqa: PLR0915
completion_response=None,
model: Optional[str] = None,
@@ -604,6 +650,8 @@ def completion_cost( # noqa: PLR0915
litellm_model_name: Optional[str] = None,
router_model_id: Optional[str] = None,
litellm_logging_obj: Optional[LitellmLoggingObject] = None,
+ ### SERVICE TIER ###
+ service_tier: Optional[str] = None, # for OpenAI service tier pricing
) -> float:
"""
Calculate the cost of a given completion call fot GPT-3.5-turbo, llama2, any litellm supported llm.
@@ -656,6 +704,10 @@ def completion_cost( # noqa: PLR0915
completion_response=completion_response
)
rerank_billed_units: Optional[RerankBilledUnits] = None
+
+ # Extract service_tier from optional_params if not provided directly
+ if service_tier is None and optional_params is not None:
+ service_tier = optional_params.get("service_tier")
selected_model = _select_model_name_for_cost_calc(
model=model,
@@ -681,9 +733,9 @@ def completion_cost( # noqa: PLR0915
or isinstance(completion_response, dict)
): # tts returns a custom class
if isinstance(completion_response, dict):
- usage_obj: Optional[Union[dict, Usage]] = (
- completion_response.get("usage", {})
- )
+ usage_obj: Optional[
+ Union[dict, Usage]
+ ] = completion_response.get("usage", {})
else:
usage_obj = getattr(completion_response, "usage", {})
if isinstance(usage_obj, BaseModel) and not _is_known_usage_objects(
@@ -768,48 +820,15 @@ def completion_cost( # noqa: PLR0915
)
if CostCalculatorUtils._call_type_has_image_response(call_type):
### IMAGE GENERATION COST CALCULATION ###
- if custom_llm_provider == "vertex_ai":
- if isinstance(completion_response, ImageResponse):
- return vertex_ai_image_cost_calculator(
- model=model,
- image_response=completion_response,
- )
- elif custom_llm_provider == "bedrock":
- if isinstance(completion_response, ImageResponse):
- return bedrock_image_cost_calculator(
- model=model,
- size=size,
- image_response=completion_response,
- optional_params=optional_params,
- )
- raise TypeError(
- "completion_response must be of type ImageResponse for bedrock image cost calculation"
- )
- elif custom_llm_provider == litellm.LlmProviders.RECRAFT.value:
- from litellm.llms.recraft.cost_calculator import (
- cost_calculator as recraft_image_cost_calculator,
- )
- return recraft_image_cost_calculator(
- model=model,
- image_response=completion_response,
- )
- elif custom_llm_provider == litellm.LlmProviders.GEMINI.value:
- from litellm.llms.gemini.image_generation.cost_calculator import (
- cost_calculator as gemini_image_cost_calculator,
- )
- return gemini_image_cost_calculator(
- model=model,
- image_response=completion_response,
- )
- else:
- return default_image_cost_calculator(
- model=model,
- quality=quality,
- custom_llm_provider=custom_llm_provider,
- n=n,
- size=size,
- optional_params=optional_params,
- )
+ return CostCalculatorUtils.route_image_generation_cost_calculator(
+ model=model,
+ custom_llm_provider=custom_llm_provider,
+ completion_response=completion_response,
+ quality=quality,
+ n=n,
+ size=size,
+ optional_params=optional_params,
+ )
elif (
call_type == CallTypes.speech.value
or call_type == CallTypes.aspeech.value
@@ -867,7 +886,10 @@ def completion_cost( # noqa: PLR0915
from litellm.proxy._experimental.mcp_server.cost_calculator import (
MCPCostCalculator,
)
- return MCPCostCalculator.calculate_mcp_tool_call_cost(litellm_logging_obj=litellm_logging_obj)
+
+ return MCPCostCalculator.calculate_mcp_tool_call_cost(
+ litellm_logging_obj=litellm_logging_obj
+ )
# Calculate cost based on prompt_tokens, completion_tokens
if (
"togethercomputer" in model
@@ -937,11 +959,12 @@ def completion_cost( # noqa: PLR0915
call_type=cast(CallTypesLiteral, call_type),
audio_transcription_file_duration=audio_transcription_file_duration,
rerank_billed_units=rerank_billed_units,
+ service_tier=service_tier,
)
_final_cost = (
prompt_tokens_cost_usd_dollar + completion_tokens_cost_usd_dollar
)
- _final_cost += (
+ cost_for_built_in_tools = (
StandardBuiltInToolCostTracking.get_cost_for_built_in_tools(
model=model,
response_object=completion_response,
@@ -950,6 +973,17 @@ def completion_cost( # noqa: PLR0915
custom_llm_provider=custom_llm_provider,
)
)
+ _final_cost += cost_for_built_in_tools
+
+ # Store cost breakdown in logging object if available
+ _store_cost_breakdown_in_logging_obj(
+ litellm_logging_obj=litellm_logging_obj,
+ prompt_tokens_cost_usd_dollar=prompt_tokens_cost_usd_dollar,
+ completion_tokens_cost_usd_dollar=completion_tokens_cost_usd_dollar,
+ cost_for_built_in_tools_cost_usd_dollar=cost_for_built_in_tools,
+ total_cost_usd_dollar=_final_cost
+ )
+
return _final_cost
except Exception as e:
verbose_logger.debug(
@@ -1031,6 +1065,8 @@ def response_cost_calculator(
litellm_model_name: Optional[str] = None,
router_model_id: Optional[str] = None,
litellm_logging_obj: Optional[LitellmLoggingObject] = None,
+ ### SERVICE TIER ###
+ service_tier: Optional[str] = None, # for OpenAI service tier pricing
) -> float:
"""
Returns
@@ -1064,6 +1100,7 @@ def response_cost_calculator(
litellm_model_name=litellm_model_name,
router_model_id=router_model_id,
litellm_logging_obj=litellm_logging_obj,
+ service_tier=service_tier,
)
return response_cost
except Exception as e:
@@ -1260,7 +1297,7 @@ class BaseTokenUsageProcessor:
Combine multiple Usage objects into a single Usage object, checking model keys for nested values.
"""
from litellm.types.utils import (
- CompletionTokensDetails,
+ CompletionTokensDetailsWrapper,
PromptTokensDetailsWrapper,
Usage,
)
@@ -1315,10 +1352,12 @@ class BaseTokenUsageProcessor:
not hasattr(combined, "completion_tokens_details")
or not combined.completion_tokens_details
):
- combined.completion_tokens_details = CompletionTokensDetails()
+ combined.completion_tokens_details = (
+ CompletionTokensDetailsWrapper()
+ )
# Check what keys exist in the model's completion_tokens_details
- for attr in dir(usage.completion_tokens_details):
+ for attr in usage.completion_tokens_details.model_fields:
if not attr.startswith("_") and not callable(
getattr(usage.completion_tokens_details, attr)
):
@@ -1326,7 +1365,8 @@ class BaseTokenUsageProcessor:
combined.completion_tokens_details, attr, 0
)
new_val = getattr(usage.completion_tokens_details, attr, 0)
- if new_val is not None:
+
+ if new_val is not None and current_val is not None:
setattr(
combined.completion_tokens_details,
attr,
diff --git a/litellm/endpoints/speech/speech_to_completion_bridge/handler.py b/litellm/endpoints/speech/speech_to_completion_bridge/handler.py
index 3035c5065c5..13af0a30fe0 100644
--- a/litellm/endpoints/speech/speech_to_completion_bridge/handler.py
+++ b/litellm/endpoints/speech/speech_to_completion_bridge/handler.py
@@ -2,7 +2,9 @@
Handler for transforming /chat/completions api requests to litellm.responses requests
"""
-from typing import TYPE_CHECKING, Optional, TypedDict, Union
+from typing import TYPE_CHECKING, Optional, Union
+
+from typing_extensions import TypedDict
if TYPE_CHECKING:
from litellm import LiteLLMLoggingObj
diff --git a/litellm/exceptions.py b/litellm/exceptions.py
index 9f3411143a6..77fb9c1faef 100644
--- a/litellm/exceptions.py
+++ b/litellm/exceptions.py
@@ -153,6 +153,29 @@ class BadRequestError(openai.BadRequestError): # type: ignore
_message += f", LiteLLM Max Retries: {self.max_retries}"
return _message
+class ImageFetchError(BadRequestError):
+ def __init__(
+ self,
+ message,
+ model=None,
+ llm_provider=None,
+ response: Optional[httpx.Response] = None,
+ litellm_debug_info: Optional[str] = None,
+ max_retries: Optional[int] = None,
+ num_retries: Optional[int] = None,
+ body: Optional[dict] = None,
+ ):
+ super().__init__(
+ message=message,
+ model=model,
+ llm_provider=llm_provider,
+ response=response,
+ litellm_debug_info=litellm_debug_info,
+ max_retries=max_retries,
+ num_retries=num_retries,
+ body=body,
+ )
+
class UnprocessableEntityError(openai.UnprocessableEntityError): # type: ignore
def __init__(
@@ -829,3 +852,65 @@ class BlockedPiiEntityError(Exception):
self.guardrail_name = guardrail_name
self.message = f"Blocked entity detected: {entity_type} by Guardrail: {guardrail_name}. This entity is not allowed to be used in this request."
super().__init__(self.message)
+
+
+class MidStreamFallbackError(ServiceUnavailableError): # type: ignore
+ def __init__(
+ self,
+ message: str,
+ model: str,
+ llm_provider: str,
+ original_exception: Optional[Exception] = None,
+ response: Optional[httpx.Response] = None,
+ litellm_debug_info: Optional[str] = None,
+ max_retries: Optional[int] = None,
+ num_retries: Optional[int] = None,
+ generated_content: str = "",
+ is_pre_first_chunk: bool = False,
+ ):
+ self.status_code = 503 # Service Unavailable
+ self.message = f"litellm.MidStreamFallbackError: {message}"
+ self.model = model
+ self.llm_provider = llm_provider
+ self.original_exception = original_exception
+ self.litellm_debug_info = litellm_debug_info
+ self.max_retries = max_retries
+ self.num_retries = num_retries
+ self.generated_content = generated_content
+ self.is_pre_first_chunk = is_pre_first_chunk
+
+ # Create a response if one wasn't provided
+ if response is None:
+ self.response = httpx.Response(
+ status_code=self.status_code,
+ request=httpx.Request(
+ method="POST",
+ url=f"https://{llm_provider}.com/v1/",
+ ),
+ )
+ else:
+ self.response = response
+
+ # Call the parent constructor
+ super().__init__(
+ message=self.message,
+ llm_provider=llm_provider,
+ model=model,
+ response=self.response,
+ litellm_debug_info=self.litellm_debug_info,
+ max_retries=self.max_retries,
+ num_retries=self.num_retries,
+ )
+
+ def __str__(self):
+ _message = self.message
+ if self.num_retries:
+ _message += f" LiteLLM Retried: {self.num_retries} times"
+ if self.max_retries:
+ _message += f", LiteLLM Max Retries: {self.max_retries}"
+ if self.original_exception:
+ _message += f" Original exception: {type(self.original_exception).__name__}: {str(self.original_exception)}"
+ return _message
+
+ def __repr__(self):
+ return self.__str__()
diff --git a/litellm/experimental_mcp_client/client.py b/litellm/experimental_mcp_client/client.py
index 843a9fb0e20..6aa671a5011 100644
--- a/litellm/experimental_mcp_client/client.py
+++ b/litellm/experimental_mcp_client/client.py
@@ -1,19 +1,25 @@
"""
LiteLLM Proxy uses this MCP Client to connnect to other MCP servers.
"""
+
import asyncio
import base64
from datetime import timedelta
-from typing import List, Optional
+from typing import Callable, Dict, List, Optional, Union
+import httpx
from mcp import ClientSession, StdioServerParameters
from mcp.client.sse import sse_client
from mcp.client.stdio import stdio_client
from mcp.client.streamable_http import streamablehttp_client
from mcp.types import CallToolRequestParams as MCPCallToolRequestParams
from mcp.types import CallToolResult as MCPCallToolResult
+from mcp.types import TextContent
from mcp.types import Tool as MCPTool
+from litellm._logging import verbose_logger
+from litellm.llms.custom_httpx.http_handler import get_ssl_configuration
+from litellm.types.llms.custom_http import VerifyTypes
from litellm.types.mcp import (
MCPAuth,
MCPAuthType,
@@ -41,15 +47,17 @@ class MCPClient:
server_url: str = "",
transport_type: MCPTransportType = MCPTransport.http,
auth_type: MCPAuthType = None,
- auth_value: Optional[str] = None,
+ auth_value: Optional[Union[str, Dict[str, str]]] = None,
timeout: float = 60.0,
stdio_config: Optional[MCPStdioConfig] = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ ssl_verify: Optional[VerifyTypes] = None,
):
self.server_url: str = server_url
self.transport_type: MCPTransport = transport_type
self.auth_type: MCPAuthType = auth_type
self.timeout: float = timeout
- self._mcp_auth_value: Optional[str] = None
+ self._mcp_auth_value: Optional[Union[str, Dict[str, str]]] = None
self._session: Optional[ClientSession] = None
self._context = None
self._transport_ctx = None
@@ -57,7 +65,8 @@ class MCPClient:
self._session_ctx = None
self._task: Optional[asyncio.Task] = None
self.stdio_config: Optional[MCPStdioConfig] = stdio_config
-
+ self.extra_headers: Optional[Dict[str, str]] = extra_headers
+ self.ssl_verify: Optional[VerifyTypes] = ssl_verify
# handle the basic auth value if provided
if auth_value:
self.update_auth_value(auth_value)
@@ -77,50 +86,88 @@ class MCPClient:
async def connect(self):
"""Initialize the transport and session."""
if self._session:
+ verbose_logger.debug(
+ f"MCP client already connected to {self.server_url or 'stdio'}"
+ )
return # Already connected
-
+
+ verbose_logger.info(
+ f"MCP client connecting to {self.server_url or 'stdio'} via {self.transport_type}"
+ )
+
try:
if self.transport_type == MCPTransport.stdio:
# For stdio transport, use stdio_client with command-line parameters
if not self.stdio_config:
raise ValueError("stdio_config is required for stdio transport")
-
+
server_params = StdioServerParameters(
command=self.stdio_config.get("command", ""),
args=self.stdio_config.get("args", []),
- env=self.stdio_config.get("env", {})
+ env=self.stdio_config.get("env", {}),
)
-
+
self._transport_ctx = stdio_client(server_params)
self._transport = await self._transport_ctx.__aenter__()
- self._session_ctx = ClientSession(self._transport[0], self._transport[1])
+ self._session_ctx = ClientSession(
+ self._transport[0], self._transport[1]
+ )
self._session = await self._session_ctx.__aenter__()
await self._session.initialize()
+ verbose_logger.info(
+ f"MCP client successfully connected via stdio: {self.stdio_config.get('command', '')}"
+ )
elif self.transport_type == MCPTransport.sse:
headers = self._get_auth_headers()
+ httpx_client_factory = self._create_httpx_client_factory()
self._transport_ctx = sse_client(
url=self.server_url,
timeout=self.timeout,
headers=headers,
+ httpx_client_factory=httpx_client_factory,
)
self._transport = await self._transport_ctx.__aenter__()
- self._session_ctx = ClientSession(self._transport[0], self._transport[1])
+ self._session_ctx = ClientSession(
+ self._transport[0], self._transport[1]
+ )
self._session = await self._session_ctx.__aenter__()
await self._session.initialize()
+ verbose_logger.info(
+ f"MCP client successfully connected via SSE to {self.server_url}"
+ )
else: # http
headers = self._get_auth_headers()
+ httpx_client_factory = self._create_httpx_client_factory()
+ verbose_logger.debug(
+ "litellm headers for streamablehttp_client: %s", headers
+ )
self._transport_ctx = streamablehttp_client(
url=self.server_url,
timeout=timedelta(seconds=self.timeout),
headers=headers,
+ httpx_client_factory=httpx_client_factory,
)
self._transport = await self._transport_ctx.__aenter__()
- self._session_ctx = ClientSession(self._transport[0], self._transport[1])
+ self._session_ctx = ClientSession(
+ self._transport[0], self._transport[1]
+ )
self._session = await self._session_ctx.__aenter__()
await self._session.initialize()
- except Exception:
+ verbose_logger.info(
+ f"MCP client successfully connected via HTTP to {self.server_url}"
+ )
+ except ValueError as e:
+ # Re-raise ValueError exceptions (like missing stdio_config)
+ verbose_logger.warning(f"MCP client connection failed: {str(e)}")
await self.disconnect()
raise
+ except Exception as e:
+ verbose_logger.warning(f"MCP client connection failed: {str(e)}")
+ await self.disconnect()
+ # Don't raise other exceptions, let the calling code handle it gracefully
+ # This allows the server manager to continue with other servers
+ # Instead of raising, we'll let the calling code handle the failure
+ pass
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""Cleanup when exiting context manager."""
@@ -128,7 +175,12 @@ class MCPClient:
async def disconnect(self):
"""Clean up session and connections."""
+ verbose_logger.info(
+ f"MCP client disconnecting from {self.server_url or 'stdio'}"
+ )
+
if self._task and not self._task.done():
+ verbose_logger.debug("MCP client cancelling background task")
self._task.cancel()
try:
await self._task
@@ -137,16 +189,24 @@ class MCPClient:
if self._session:
try:
+ verbose_logger.debug("MCP client closing session")
await self._session_ctx.__aexit__(None, None, None) # type: ignore
- except Exception:
+ except Exception as e:
+ verbose_logger.debug(
+ f"Error closing MCP session: {type(e).__name__}: {str(e)}"
+ )
pass
self._session = None
self._session_ctx = None
if self._transport_ctx:
try:
+ verbose_logger.debug("MCP client closing transport")
await self._transport_ctx.__aexit__(None, None, None)
- except Exception:
+ except Exception as e:
+ verbose_logger.debug(
+ f"Error closing MCP transport: {type(e).__name__}: {str(e)}"
+ )
pass
self._transport_ctx = None
self._transport = None
@@ -158,44 +218,130 @@ class MCPClient:
pass
self._context = None
- def update_auth_value(self, mcp_auth_value: str):
+ def update_auth_value(self, mcp_auth_value: Union[str, Dict[str, str]]):
"""
Set the authentication header for the MCP client.
"""
- if self.auth_type == MCPAuth.basic:
- # Assuming mcp_auth_value is in format "username:password", convert it when updating
- mcp_auth_value = to_basic_auth(mcp_auth_value)
- self._mcp_auth_value = mcp_auth_value
+ if isinstance(mcp_auth_value, dict):
+ self._mcp_auth_value = mcp_auth_value
+ else:
+ if self.auth_type == MCPAuth.basic:
+ # Assuming mcp_auth_value is in format "username:password", convert it when updating
+ mcp_auth_value = to_basic_auth(mcp_auth_value)
+ self._mcp_auth_value = mcp_auth_value
def _get_auth_headers(self) -> dict:
"""Generate authentication headers based on auth type."""
- if not self._mcp_auth_value:
- return {}
+ headers = {}
- if self.auth_type == MCPAuth.bearer_token:
- return {"Authorization": f"Bearer {self._mcp_auth_value}"}
- elif self.auth_type == MCPAuth.basic:
- return {"Authorization": f"Basic {self._mcp_auth_value}"}
- elif self.auth_type == MCPAuth.api_key:
- return {"X-API-Key": self._mcp_auth_value}
- return {}
+ if self._mcp_auth_value:
+ if isinstance(self._mcp_auth_value, str):
+ if self.auth_type == MCPAuth.bearer_token:
+ headers["Authorization"] = f"Bearer {self._mcp_auth_value}"
+ elif self.auth_type == MCPAuth.basic:
+ headers["Authorization"] = f"Basic {self._mcp_auth_value}"
+ elif self.auth_type == MCPAuth.api_key:
+ headers["X-API-Key"] = self._mcp_auth_value
+ elif self.auth_type == MCPAuth.authorization:
+ headers["Authorization"] = self._mcp_auth_value
+ elif isinstance(self._mcp_auth_value, dict):
+ headers.update(self._mcp_auth_value)
+
+ # update the headers with the extra headers
+ if self.extra_headers:
+ headers.update(self.extra_headers)
+
+ return headers
+
+ def _create_httpx_client_factory(self) -> Callable[..., httpx.AsyncClient]:
+ """
+ Create a custom httpx client factory that uses LiteLLM's SSL configuration.
+
+ This factory follows the same CA bundle path logic as http_handler.py:
+ 1. Check ssl_verify parameter (can be SSLContext, bool, or path to CA bundle)
+ 2. Check SSL_VERIFY environment variable
+ 3. Check SSL_CERT_FILE environment variable
+ 4. Fall back to certifi CA bundle
+ """
+
+ def factory(
+ *,
+ headers: Optional[Dict[str, str]] = None,
+ timeout: Optional[httpx.Timeout] = None,
+ auth: Optional[httpx.Auth] = None,
+ ) -> httpx.AsyncClient:
+ """Create an httpx.AsyncClient with LiteLLM's SSL configuration."""
+ # Get unified SSL configuration using the same logic as http_handler.py
+ ssl_config = get_ssl_configuration(self.ssl_verify)
+
+ verbose_logger.debug(
+ f"MCP client using SSL configuration: {type(ssl_config).__name__}"
+ )
+
+ return httpx.AsyncClient(
+ headers=headers,
+ timeout=timeout,
+ auth=auth,
+ verify=ssl_config,
+ follow_redirects=True,
+ )
+
+ return factory
async def list_tools(self) -> List[MCPTool]:
"""List available tools from the server."""
+ verbose_logger.debug(
+ f"MCP client listing tools from {self.server_url or 'stdio'}"
+ )
+
if not self._session:
- await self.connect()
+ verbose_logger.debug("MCP client session not found, attempting to connect")
+ try:
+ await self.connect()
+ except Exception as e:
+ verbose_logger.error(
+ f"MCP client connection failed during list_tools: {type(e).__name__}: {str(e)}"
+ )
+ return []
+
if self._session is None:
- raise ValueError("Session is not initialized")
+ verbose_logger.error(
+ "MCP client session is not initialized after connection attempt"
+ )
+ return []
try:
result = await self._session.list_tools()
+ tool_count = len(result.tools)
+ tool_names = [tool.name for tool in result.tools]
+ verbose_logger.info(
+ f"MCP client listed {tool_count} tools from {self.server_url or 'stdio'}: {tool_names}"
+ )
return result.tools
except asyncio.CancelledError:
+ verbose_logger.warning("MCP client list_tools was cancelled")
await self.disconnect()
raise
- except Exception:
+ except Exception as e:
+ error_type = type(e).__name__
+ verbose_logger.error(
+ f"MCP client list_tools failed - "
+ f"Error Type: {error_type}, "
+ f"Error: {str(e)}, "
+ f"Server: {self.server_url or 'stdio'}, "
+ f"Transport: {self.transport_type}"
+ )
+
+ # Check if it's a stream/connection error
+ if "BrokenResourceError" in error_type or "Broken" in error_type:
+ verbose_logger.error(
+ "MCP client detected broken connection/stream during list_tools - "
+ "the MCP server may have crashed, disconnected, or timed out"
+ )
+
await self.disconnect()
- raise
+ # Return empty list instead of raising to allow graceful degradation
+ return []
async def call_tool(
self, call_tool_request_params: MCPCallToolRequestParams
@@ -203,23 +349,90 @@ class MCPClient:
"""
Call an MCP Tool.
"""
+ verbose_logger.info(
+ f"MCP client calling tool '{call_tool_request_params.name}' with arguments: {call_tool_request_params.arguments}"
+ )
+
if not self._session:
- await self.connect()
+ verbose_logger.warning(
+ "MCP client session not found, attempting to connect"
+ )
+ try:
+ await self.connect()
+ except Exception as e:
+ verbose_logger.error(
+ f"MCP client connection failed before tool call: {type(e).__name__}: {str(e)}"
+ )
+ return MCPCallToolResult(
+ content=[TextContent(type="text", text=f"{str(e)}")], isError=True
+ )
if self._session is None:
- raise ValueError("Session is not initialized")
-
+ verbose_logger.error(
+ "MCP client session is not initialized after connection attempt"
+ )
+ return MCPCallToolResult(
+ content=[
+ TextContent(
+ type="text", text="MCP client session is not initialized"
+ )
+ ],
+ isError=True,
+ )
+
+ # Check session and transport state before calling tool
+ verbose_logger.debug(
+ f"MCP client state before tool call - "
+ f"session: {'active' if self._session else 'none'}, "
+ f"transport: {'active' if self._transport else 'none'}, "
+ f"session_ctx: {'active' if self._session_ctx else 'none'}, "
+ f"transport_ctx: {'active' if self._transport_ctx else 'none'}"
+ )
+
try:
+ verbose_logger.debug("MCP client sending tool call to session")
tool_result = await self._session.call_tool(
name=call_tool_request_params.name,
arguments=call_tool_request_params.arguments,
)
+ verbose_logger.info(
+ f"MCP client tool call '{call_tool_request_params.name}' completed successfully"
+ )
return tool_result
except asyncio.CancelledError:
+ verbose_logger.warning("MCP client tool call was cancelled")
await self.disconnect()
raise
- except Exception:
- await self.disconnect()
- raise
-
+ except Exception as e:
+ import traceback
+ error_trace = traceback.format_exc()
+ verbose_logger.debug(f"MCP client tool call traceback:\n{error_trace}")
+
+ # Log detailed error information
+ error_type = type(e).__name__
+ verbose_logger.error(
+ f"MCP client call_tool failed - "
+ f"Error Type: {error_type}, "
+ f"Error: {str(e)}, "
+ f"Tool: {call_tool_request_params.name}, "
+ f"Server: {self.server_url or 'stdio'}, "
+ f"Transport: {self.transport_type}"
+ )
+
+ # Check if it's a stream/connection error
+ if "BrokenResourceError" in error_type or "Broken" in error_type:
+ verbose_logger.error(
+ "MCP client detected broken connection/stream - "
+ "the MCP server may have crashed, disconnected, or timed out. "
+ "Session and transport will be disconnected."
+ )
+
+ await self.disconnect()
+ # Return a default error result instead of raising
+ return MCPCallToolResult(
+ content=[
+ TextContent(type="text", text=f"{error_type}: {str(e)}")
+ ], # Empty content for error case
+ isError=True,
+ )
diff --git a/litellm/experimental_mcp_client/tools.py b/litellm/experimental_mcp_client/tools.py
index bfbd3f96a5c..b716e3171e7 100644
--- a/litellm/experimental_mcp_client/tools.py
+++ b/litellm/experimental_mcp_client/tools.py
@@ -17,22 +17,60 @@ from litellm.types.utils import ChatCompletionMessageToolCall
########################################################
def transform_mcp_tool_to_openai_tool(mcp_tool: MCPTool) -> ChatCompletionToolParam:
"""Convert an MCP tool to an OpenAI tool."""
+ normalized_parameters = _normalize_mcp_input_schema(mcp_tool.inputSchema)
+
return ChatCompletionToolParam(
type="function",
function=FunctionDefinition(
name=mcp_tool.name,
description=mcp_tool.description or "",
- parameters=mcp_tool.inputSchema,
+ parameters=normalized_parameters,
strict=False,
),
)
+def _normalize_mcp_input_schema(input_schema: dict) -> dict:
+ """
+ Normalize MCP input schema to ensure it's valid for OpenAI function calling.
+
+ OpenAI requires that function parameters have:
+ - type: 'object'
+ - properties: dict (can be empty)
+ - additionalProperties: false (recommended)
+ """
+ if not input_schema:
+ return {
+ "type": "object",
+ "properties": {},
+ "additionalProperties": False
+ }
+
+ # Make a copy to avoid modifying the original
+ normalized_schema = dict(input_schema)
+
+ # Ensure type is 'object'
+ if "type" not in normalized_schema:
+ normalized_schema["type"] = "object"
+
+ # Ensure properties exists (can be empty)
+ if "properties" not in normalized_schema:
+ normalized_schema["properties"] = {}
+
+ # Add additionalProperties if not present (recommended by OpenAI)
+ if "additionalProperties" not in normalized_schema:
+ normalized_schema["additionalProperties"] = False
+
+ return normalized_schema
+
+
def transform_mcp_tool_to_openai_responses_api_tool(mcp_tool: MCPTool) -> FunctionToolParam:
"""Convert an MCP tool to an OpenAI Responses API tool."""
+ normalized_parameters = _normalize_mcp_input_schema(mcp_tool.inputSchema)
+
return FunctionToolParam(
name=mcp_tool.name,
- parameters=mcp_tool.inputSchema,
+ parameters=normalized_parameters,
strict=False,
type="function",
description=mcp_tool.description or "",
diff --git a/litellm/files/main.py b/litellm/files/main.py
index 5d0dc05771a..7bc2c136726 100644
--- a/litellm/files/main.py
+++ b/litellm/files/main.py
@@ -18,6 +18,7 @@ from litellm import get_secret_str
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.azure.files.handler import AzureOpenAIFilesAPI
+from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.llms.openai.openai import FileDeleted, FileObject, OpenAIFilesAPI
from litellm.llms.vertex_ai.files.handler import VertexAIFilesHandler
@@ -50,7 +51,7 @@ vertex_ai_files_instance = VertexAIFilesHandler()
async def acreate_file(
file: FileTypes,
purpose: Literal["assistants", "batch", "fine-tune"],
- custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
+ custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@@ -94,7 +95,7 @@ async def acreate_file(
def create_file(
file: FileTypes,
purpose: Literal["assistants", "batch", "fine-tune"],
- custom_llm_provider: Optional[Literal["openai", "azure", "vertex_ai"]] = None,
+ custom_llm_provider: Optional[Literal["openai", "azure", "vertex_ai", "bedrock"]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@@ -109,7 +110,7 @@ def create_file(
try:
_is_async = kwargs.pop("acreate_file", False) is True
optional_params = GenericLiteLLMParams(**kwargs)
- litellm_params_dict = get_litellm_params(**kwargs)
+ litellm_params_dict = dict(**kwargs)
logging_obj = cast(
Optional[LiteLLMLoggingObj], kwargs.get("litellm_logging_obj")
)
@@ -268,6 +269,7 @@ def create_file(
raise e
+@client
async def afile_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@@ -308,6 +310,7 @@ async def afile_retrieve(
raise e
+@client
def file_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@@ -422,6 +425,7 @@ def file_retrieve(
# Delete file
+@client
async def afile_delete(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@@ -462,6 +466,7 @@ async def afile_delete(
raise e
+@client
def file_delete(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
@@ -577,6 +582,7 @@ def file_delete(
# List files
+@client
async def afile_list(
custom_llm_provider: Literal["openai", "azure"] = "openai",
purpose: Optional[str] = None,
@@ -617,6 +623,7 @@ async def afile_list(
raise e
+@client
def file_list(
custom_llm_provider: Literal["openai", "azure"] = "openai",
purpose: Optional[str] = None,
@@ -729,9 +736,10 @@ def file_list(
raise e
+@client
async def afile_content(
file_id: str,
- custom_llm_provider: Literal["openai", "azure"] = "openai",
+ custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@@ -771,6 +779,7 @@ async def afile_content(
raise e
+@client
def file_content(
file_id: str,
model: Optional[str] = None,
@@ -887,6 +896,32 @@ def file_content(
client=client,
litellm_params=litellm_params_dict,
)
+ elif custom_llm_provider == "vertex_ai":
+ api_base = optional_params.api_base or ""
+ vertex_ai_project = (
+ optional_params.vertex_project
+ or litellm.vertex_project
+ or get_secret_str("VERTEXAI_PROJECT")
+ )
+ vertex_ai_location = (
+ optional_params.vertex_location
+ or litellm.vertex_location
+ or get_secret_str("VERTEXAI_LOCATION")
+ )
+ vertex_credentials = optional_params.vertex_credentials or get_secret_str(
+ "VERTEXAI_CREDENTIALS"
+ )
+
+ response = vertex_ai_files_instance.file_content(
+ _is_async=_is_async,
+ file_content_request=_file_content_request,
+ api_base=api_base,
+ vertex_credentials=vertex_credentials,
+ vertex_project=vertex_ai_project,
+ vertex_location=vertex_ai_location,
+ timeout=timeout,
+ max_retries=optional_params.max_retries,
+ )
else:
raise litellm.exceptions.BadRequestError(
message="LiteLLM doesn't support {} for 'custom_llm_provider'. Supported providers are 'openai', 'azure', 'vertex_ai'.".format(
diff --git a/litellm/files/utils.py b/litellm/files/utils.py
new file mode 100644
index 00000000000..a56a29467d9
--- /dev/null
+++ b/litellm/files/utils.py
@@ -0,0 +1,27 @@
+from typing import Optional
+
+from litellm.types.llms.openai import CreateFileRequest
+from litellm.types.utils import ExtractedFileData
+
+
+class FilesAPIUtils:
+ """
+ Utils for files API interface on litellm
+ """
+ @staticmethod
+ def is_batch_jsonl_file(create_file_data: CreateFileRequest, extracted_file_data: ExtractedFileData) -> bool:
+ """
+ Check if the file is a batch jsonl file
+ """
+ return (
+ create_file_data.get("purpose") == "batch"
+ and FilesAPIUtils.valid_content_type(extracted_file_data.get("content_type"))
+ and extracted_file_data.get("content") is not None
+ )
+
+ @staticmethod
+ def valid_content_type(content_type: Optional[str]) -> bool:
+ """
+ Check if the content type is valid
+ """
+ return content_type in set(["application/jsonl", "application/octet-stream"])
diff --git a/litellm/google_genai/adapters/handler.py b/litellm/google_genai/adapters/handler.py
index 1f575f27591..575c36b946a 100644
--- a/litellm/google_genai/adapters/handler.py
+++ b/litellm/google_genai/adapters/handler.py
@@ -37,6 +37,10 @@ class GenerateContentToCompletionHandler:
completion_kwargs: Dict[str, Any] = dict(completion_request)
+ # feed metadata for custom callback
+ if extra_kwargs is not None and "metadata" in extra_kwargs:
+ completion_kwargs["metadata"] = extra_kwargs["metadata"]
+
if stream:
completion_kwargs["stream"] = stream
@@ -68,15 +72,24 @@ class GenerateContentToCompletionHandler:
completion_response = await litellm.acompletion(**completion_kwargs)
if stream:
- # Transform streaming completion response to generate_content format
- transformed_stream = (
- GOOGLE_GENAI_ADAPTER.translate_completion_output_params_streaming(
+ # Check if completion_response is actually a stream or a ModelResponse
+ # This can happen in error cases or when stream is not properly supported
+ if not hasattr(completion_response, "__aiter__"):
+ # If it's not a stream, treat it as a regular response
+ generate_content_response = (
+ GOOGLE_GENAI_ADAPTER.translate_completion_to_generate_content(
+ cast(ModelResponse, completion_response)
+ )
+ )
+ return generate_content_response
+ else:
+ # Transform streaming completion response to generate_content format
+ transformed_stream = GOOGLE_GENAI_ADAPTER.translate_completion_output_params_streaming(
completion_response
)
- )
- if transformed_stream is not None:
- return transformed_stream
- raise ValueError("Failed to transform streaming response")
+ if transformed_stream is not None:
+ return transformed_stream
+ raise ValueError("Failed to transform streaming response")
else:
# Transform completion response back to generate_content format
generate_content_response = (
@@ -132,15 +145,24 @@ class GenerateContentToCompletionHandler:
completion_response = litellm.completion(**completion_kwargs)
if stream:
- # Transform streaming completion response to generate_content format
- transformed_stream = (
- GOOGLE_GENAI_ADAPTER.translate_completion_output_params_streaming(
+ # Check if completion_response is actually a stream or a ModelResponse
+ # This can happen in error cases or when stream is not properly supported
+ if not hasattr(completion_response, "__iter__"):
+ # If it's not a stream, treat it as a regular response
+ generate_content_response = (
+ GOOGLE_GENAI_ADAPTER.translate_completion_to_generate_content(
+ cast(ModelResponse, completion_response)
+ )
+ )
+ return generate_content_response
+ else:
+ # Transform streaming completion response to generate_content format
+ transformed_stream = GOOGLE_GENAI_ADAPTER.translate_completion_output_params_streaming(
completion_response
)
- )
- if transformed_stream is not None:
- return transformed_stream
- raise ValueError("Failed to transform streaming response")
+ if transformed_stream is not None:
+ return transformed_stream
+ raise ValueError("Failed to transform streaming response")
else:
# Transform completion response back to generate_content format
generate_content_response = (
diff --git a/litellm/google_genai/adapters/transformation.py b/litellm/google_genai/adapters/transformation.py
index 7617312302e..9d3f990b1aa 100644
--- a/litellm/google_genai/adapters/transformation.py
+++ b/litellm/google_genai/adapters/transformation.py
@@ -1,12 +1,15 @@
import json
from typing import Any, AsyncIterator, Dict, Iterator, List, Optional, Union, cast
+from litellm import verbose_logger
+
from litellm.litellm_core_utils.json_validation_rule import normalize_tool_schema
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionAssistantMessage,
ChatCompletionAssistantToolCall,
ChatCompletionRequest,
+ ChatCompletionSystemMessage,
ChatCompletionToolCallFunctionChunk,
ChatCompletionToolChoiceValues,
ChatCompletionToolMessage,
@@ -36,43 +39,103 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper):
def __init__(self, completion_stream: Any):
self.sent_first_chunk = False
self.accumulated_tool_calls = {}
+ self._returned_response = False
super().__init__(completion_stream)
def __next__(self):
try:
+ if not hasattr(self.completion_stream, "__iter__"):
+ if self._returned_response:
+ raise StopIteration
+ self._returned_response = True
+ return GoogleGenAIAdapter().translate_completion_to_generate_content(
+ self.completion_stream
+ )
+
for chunk in self.completion_stream:
if chunk == "None" or chunk is None:
continue
- # Transform OpenAI streaming chunk to Google GenAI format
transformed_chunk = GoogleGenAIAdapter().translate_streaming_completion_to_generate_content(
chunk, self
)
- if transformed_chunk: # Only return non-empty chunks
+ if transformed_chunk:
return transformed_chunk
raise StopIteration
except StopIteration:
- raise StopIteration
+ raise
except Exception:
raise StopIteration
async def __anext__(self):
try:
+ if not hasattr(self.completion_stream, "__aiter__"):
+ if self._returned_response:
+ raise StopAsyncIteration
+ self._returned_response = True
+ return GoogleGenAIAdapter().translate_completion_to_generate_content(
+ self.completion_stream
+ )
+
async for chunk in self.completion_stream:
if chunk == "None" or chunk is None:
continue
- # Transform OpenAI streaming chunk to Google GenAI format
transformed_chunk = GoogleGenAIAdapter().translate_streaming_completion_to_generate_content(
chunk, self
)
- if transformed_chunk: # Only return non-empty chunks
+ if transformed_chunk:
return transformed_chunk
+ # After the stream is exhausted, check for any remaining accumulated tool calls
+ if self.accumulated_tool_calls:
+ try:
+ parts = []
+ for (
+ tool_call_index,
+ tool_call_data,
+ ) in self.accumulated_tool_calls.items():
+ try:
+ # For tool calls with no arguments, accumulated_args will be "", which is not valid JSON.
+ # We default to an empty JSON object in this case.
+ parsed_args = json.loads(
+ tool_call_data["arguments"] or "{}"
+ )
+ function_call_part = {
+ "functionCall": {
+ "name": tool_call_data["name"]
+ or "undefined_tool_name",
+ "args": parsed_args,
+ }
+ }
+ parts.append(function_call_part)
+ except json.JSONDecodeError:
+ # This can happen if the stream is abruptly cut off mid-argument string.
+ verbose_logger.warning(
+ f"Could not parse tool call arguments at end of stream for index {tool_call_index}. "
+ f"Name: {tool_call_data['name']}. "
+ f"Partial args: {tool_call_data['arguments']}"
+ )
+ pass
+ if parts:
+ final_chunk = {
+ "candidates": [
+ {
+ "content": {"parts": parts, "role": "model"},
+ "finishReason": "STOP",
+ "index": 0,
+ "safetyRatings": [],
+ }
+ ]
+ }
+ return final_chunk
+ finally:
+ # Ensure the accumulator is always cleared to prevent memory leaks
+ self.accumulated_tool_calls.clear()
raise StopAsyncIteration
except StopAsyncIteration:
- raise StopAsyncIteration
+ raise
except Exception:
raise StopAsyncIteration
@@ -107,9 +170,14 @@ class GoogleGenAIStreamWrapper(AdapterCompletionStreamWrapper):
payload = f"data: {json.dumps(transformed_chunk)}\n\n"
yield payload.encode()
else:
- raise ValueError(f"Invalid chunk 1: {chunk}")
+ # For empty chunks, continue to next iteration
+ continue
else:
- raise ValueError(f"Invalid chunk 2: {chunk}")
+ # For other chunk types, yield them directly
+ if hasattr(chunk, "encode"):
+ yield chunk.encode()
+ else:
+ yield str(chunk).encode()
class GoogleGenAIAdapter:
@@ -133,12 +201,19 @@ class GoogleGenAIAdapter:
model: The model name
contents: Generate content contents (can be list or single dict)
config: Optional config parameters
- **kwargs: Additional parameters
+ **kwargs: Additional parameters from the original request
Returns:
Dict in OpenAI format
"""
+ # Extract top-level fields from kwargs
+ system_instruction = kwargs.get("systemInstruction") or kwargs.get(
+ "system_instruction"
+ )
+ tools = kwargs.get("tools")
+ tool_config = kwargs.get("toolConfig") or kwargs.get("tool_config")
+
# Normalize contents to list format
if isinstance(contents, dict):
contents_list = [contents]
@@ -146,7 +221,9 @@ class GoogleGenAIAdapter:
contents_list = contents
# Transform contents to OpenAI messages format
- messages = self._transform_contents_to_messages(contents_list)
+ messages = self._transform_contents_to_messages(
+ contents_list, system_instruction=system_instruction
+ )
# Create base request as dict (which is compatible with ChatCompletionRequest)
completion_request: ChatCompletionRequest = {
@@ -182,20 +259,19 @@ class GoogleGenAIAdapter:
completion_request["stop"] = config["stopSequences"]
# Handle tools transformation
- if "tools" in kwargs:
- tools = kwargs["tools"]
-
+ if tools:
# Check if tools are already in OpenAI format or Google GenAI format
if isinstance(tools, list) and len(tools) > 0:
# Tools are in Google GenAI format, transform them
openai_tools = self._transform_google_genai_tools_to_openai(tools)
+
if openai_tools:
completion_request["tools"] = openai_tools
# Handle tool_config (tool choice)
- if "tool_config" in kwargs:
+ if tool_config:
tool_choice = self._transform_google_genai_tool_config_to_openai(
- kwargs["tool_config"]
+ tool_config
)
if tool_choice:
completion_request["tool_choice"] = tool_choice
@@ -235,7 +311,8 @@ class GoogleGenAIAdapter:
return completion_request_dict
def translate_completion_output_params_streaming(
- self, completion_stream: Any
+ self,
+ completion_stream: Any,
) -> Union[AsyncIterator[bytes], None]:
"""Transform streaming completion output to Google GenAI format"""
google_genai_wrapper = GoogleGenAIStreamWrapper(
@@ -245,7 +322,8 @@ class GoogleGenAIAdapter:
return google_genai_wrapper.async_google_genai_sse_wrapper()
def _transform_google_genai_tools_to_openai(
- self, tools: List[Dict[str, Any]]
+ self,
+ tools: List[Dict[str, Any]],
) -> List[ChatCompletionToolParam]:
"""Transform Google GenAI tools to OpenAI tools format"""
openai_tools: List[Dict[str, Any]] = []
@@ -259,8 +337,8 @@ class GoogleGenAIAdapter:
if "description" in func_decl:
function_chunk["description"] = func_decl["description"]
- if "parameters" in func_decl:
- function_chunk["parameters"] = func_decl["parameters"]
+ if "parametersJsonSchema" in func_decl:
+ function_chunk["parameters"] = func_decl["parametersJsonSchema"]
openai_tool = {"type": "function", "function": function_chunk}
openai_tools.append(openai_tool)
@@ -271,7 +349,8 @@ class GoogleGenAIAdapter:
return cast(List[ChatCompletionToolParam], normalized_tools)
def _transform_google_genai_tool_config_to_openai(
- self, tool_config: Dict[str, Any]
+ self,
+ tool_config: Dict[str, Any],
) -> Optional[ChatCompletionToolChoiceValues]:
"""Transform Google GenAI tool_config to OpenAI tool_choice"""
function_calling_config = tool_config.get("functionCallingConfig", {})
@@ -283,11 +362,23 @@ class GoogleGenAIAdapter:
return cast(ChatCompletionToolChoiceValues, tool_choice)
def _transform_contents_to_messages(
- self, contents: List[Dict[str, Any]]
+ self,
+ contents: List[Dict[str, Any]],
+ system_instruction: Optional[Dict[str, Any]] = None,
) -> List[AllMessageValues]:
"""Transform Google GenAI contents to OpenAI messages format"""
messages: List[AllMessageValues] = []
+ # Handle system instruction
+ if system_instruction:
+ system_parts = system_instruction.get("parts", [])
+ if system_parts and "text" in system_parts[0]:
+ messages.append(
+ ChatCompletionSystemMessage(
+ role="system", content=system_parts[0]["text"]
+ )
+ )
+
for content in contents:
role = content.get("role", "user")
parts = content.get("parts", [])
@@ -364,7 +455,8 @@ class GoogleGenAIAdapter:
return messages
def translate_completion_to_generate_content(
- self, response: ModelResponse
+ self,
+ response: ModelResponse,
) -> Dict[str, Any]:
"""
Transform litellm completion response to Google GenAI generate_content format
@@ -376,6 +468,7 @@ class GoogleGenAIAdapter:
Dict in Google GenAI generate_content response format
"""
+
# Extract the main response content
choice = response.choices[0] if response.choices else None
if not choice:
@@ -388,12 +481,6 @@ class GoogleGenAIAdapter:
"Invalid completion response: no message found in choice"
)
parts = self._transform_openai_message_to_google_genai_parts(choice.message)
- elif isinstance(choice, StreamingChoices):
- if not choice.delta:
- raise ValueError(
- "Invalid completion response: no delta found in streaming choice"
- )
- parts = self._transform_openai_delta_to_google_genai_parts(choice.delta)
else:
# Fallback for generic choice objects
message_content = getattr(choice, "message", {}).get(
@@ -438,7 +525,7 @@ class GoogleGenAIAdapter:
self,
response: Union[ModelResponse, ModelResponseStream],
wrapper: GoogleGenAIStreamWrapper,
- ) -> Dict[str, Any]:
+ ) -> Optional[Dict[str, Any]]:
"""
Transform streaming litellm completion chunk to Google GenAI generate_content format
@@ -454,7 +541,7 @@ class GoogleGenAIAdapter:
choice = response.choices[0] if response.choices else None
if not choice:
# Return empty chunk if no choices
- return {}
+ return None
# Handle streaming choice
if isinstance(choice, StreamingChoices):
@@ -473,7 +560,7 @@ class GoogleGenAIAdapter:
# Only create response chunk if we have parts or it's the final chunk
if not parts and not finish_reason:
- return {}
+ return None
# Create Google GenAI streaming format response
streaming_chunk: Dict[str, Any] = {
@@ -515,7 +602,8 @@ class GoogleGenAIAdapter:
return streaming_chunk
def _transform_openai_message_to_google_genai_parts(
- self, message: Any
+ self,
+ message: Any,
) -> List[Dict[str, Any]]:
"""Transform OpenAI message to Google GenAI parts format"""
parts: List[Dict[str, Any]] = []
@@ -537,112 +625,94 @@ class GoogleGenAIAdapter:
except json.JSONDecodeError:
args = {}
- function_call_part = {
- "functionCall": {"name": tool_call.function.name, "args": args}
- }
- parts.append(function_call_part)
-
- return parts if parts else [{"text": ""}]
-
- def _transform_openai_delta_to_google_genai_parts(
- self, delta: Any
- ) -> List[Dict[str, Any]]:
- """Transform OpenAI delta to Google GenAI parts format for streaming"""
- parts: List[Dict[str, Any]] = []
-
- # Add text content if present
- if hasattr(delta, "content") and delta.content:
- parts.append({"text": delta.content})
-
- # Add tool calls if present (for streaming tool calls)
- if hasattr(delta, "tool_calls") and delta.tool_calls:
- for tool_call in delta.tool_calls:
- if hasattr(tool_call, "function") and tool_call.function:
- # For streaming, we might get partial function arguments
- args_str = getattr(tool_call.function, "arguments", "") or ""
- try:
- args = json.loads(args_str) if args_str else {}
- except json.JSONDecodeError:
- # For partial JSON in streaming, return as text for now
- args = {"partial": args_str}
-
function_call_part = {
"functionCall": {
- "name": getattr(tool_call.function, "name", "") or "",
+ "name": tool_call.function.name or "undefined_tool_name",
"args": args,
}
}
parts.append(function_call_part)
- return parts
+ return parts if parts else [{"text": ""}]
def _transform_openai_delta_to_google_genai_parts_with_accumulation(
self, delta: Any, wrapper: GoogleGenAIStreamWrapper
) -> List[Dict[str, Any]]:
- """Transform OpenAI delta to Google GenAI parts format with tool call accumulation"""
+ """Transforms OpenAI delta to Google GenAI parts, accumulating streaming tool calls."""
+
+ # 1. Initialize wrapper state if it doesn't exist
+ if not hasattr(wrapper, "accumulated_tool_calls"):
+ wrapper.accumulated_tool_calls = {}
+
parts: List[Dict[str, Any]] = []
- # Add text content if present
if hasattr(delta, "content") and delta.content:
parts.append({"text": delta.content})
- # Handle tool calls with accumulation for streaming
- if hasattr(delta, "tool_calls") and delta.tool_calls:
- for tool_call in delta.tool_calls:
- if hasattr(tool_call, "function") and tool_call.function:
- tool_call_id = getattr(tool_call, "id", "") or "call_unknown"
- function_name = getattr(tool_call.function, "name", "") or ""
- args_str = getattr(tool_call.function, "arguments", "") or ""
+ # 2. Ensure tool_calls is iterable
+ tool_calls = delta.tool_calls or []
- # Initialize accumulation for this tool call if not exists
- if tool_call_id not in wrapper.accumulated_tool_calls:
- wrapper.accumulated_tool_calls[tool_call_id] = {
- "name": "",
- "arguments": "",
- "complete": False,
- }
+ for tool_call in tool_calls:
+ if not hasattr(tool_call, "function"):
+ continue
- # Accumulate function name if provided
- if function_name:
- wrapper.accumulated_tool_calls[tool_call_id][
- "name"
- ] = function_name
+ # 3. Use `index` as the primary key for accumulation
+ tool_call_index = getattr(tool_call, "index", None)
+ if tool_call_index is None:
+ continue # Index is essential for tracking streaming tool calls
- # Accumulate arguments if provided
- if args_str:
- wrapper.accumulated_tool_calls[tool_call_id][
- "arguments"
- ] += args_str
+ # Initialize accumulator for this index if it's new
+ if tool_call_index not in wrapper.accumulated_tool_calls:
+ wrapper.accumulated_tool_calls[tool_call_index] = {
+ "name": "",
+ "arguments": "",
+ }
- # Try to parse the accumulated arguments as JSON
- accumulated_args = wrapper.accumulated_tool_calls[tool_call_id][
- "arguments"
- ]
- try:
- if accumulated_args:
- parsed_args = json.loads(accumulated_args)
- # JSON is valid, mark as complete and create function call part
- wrapper.accumulated_tool_calls[tool_call_id][
- "complete"
- ] = True
+ # Accumulate name and arguments
+ function_name = getattr(tool_call.function, "name", None)
+ args_chunk = getattr(tool_call.function, "arguments", None)
- function_call_part = {
- "functionCall": {
- "name": wrapper.accumulated_tool_calls[
- tool_call_id
- ]["name"],
- "args": parsed_args,
- }
- }
- parts.append(function_call_part)
+ # Optimization: Skip chunks that have no new data
+ if not function_name and not args_chunk:
+ verbose_logger.debug(
+ f"Skipping empty tool call chunk for index: {tool_call_index}"
+ )
+ continue
- # Clean up completed tool call
- del wrapper.accumulated_tool_calls[tool_call_id]
+ if function_name:
+ wrapper.accumulated_tool_calls[tool_call_index]["name"] = function_name
- except json.JSONDecodeError:
- # JSON is still incomplete, continue accumulating
- # Don't add to parts yet
- pass
+ if args_chunk:
+ wrapper.accumulated_tool_calls[tool_call_index][
+ "arguments"
+ ] += args_chunk
+
+ # Attempt to parse and emit a complete tool call
+ accumulated_data = wrapper.accumulated_tool_calls[tool_call_index]
+ accumulated_name = accumulated_data["name"]
+ accumulated_args = accumulated_data["arguments"]
+
+ # 5. Attempt to parse arguments even if name hasn't arrived.
+ try:
+ # Attempt to parse the accumulated arguments string
+ parsed_args = json.loads(accumulated_args)
+
+ # If parsing succeeds, but we don't have a name yet, wait.
+ # The part will be created by a later chunk that brings the name.
+ if accumulated_name:
+ # If successful, create the part and clean up
+ function_call_part = {
+ "functionCall": {"name": accumulated_name, "args": parsed_args}
+ }
+ parts.append(function_call_part)
+
+ # Remove the completed tool call from the accumulator
+ del wrapper.accumulated_tool_calls[tool_call_index]
+
+ except json.JSONDecodeError:
+ # The JSON for arguments is still incomplete.
+ # We will continue to accumulate and wait for more chunks.
+ pass
return parts
diff --git a/litellm/google_genai/main.py b/litellm/google_genai/main.py
index c34b1663e6f..8a9cb809404 100644
--- a/litellm/google_genai/main.py
+++ b/litellm/google_genai/main.py
@@ -24,11 +24,14 @@ if TYPE_CHECKING:
GenerateContentConfigDict,
GenerateContentContentListUnionDict,
GenerateContentResponse,
+ ToolConfigDict,
)
else:
GenerateContentConfigDict = Any
GenerateContentContentListUnionDict = Any
GenerateContentResponse = Any
+ ToolConfigDict = Any
+
####### ENVIRONMENT VARIABLES ###################
# Initialize any necessary instances or variables here
@@ -82,7 +85,7 @@ class GenerateContentHelper:
contents: GenerateContentContentListUnionDict,
config: Optional[GenerateContentConfigDict] = None,
custom_llm_provider: Optional[str] = None,
- stream: bool = False,
+ tools: Optional[ToolConfigDict] = None,
**kwargs,
) -> GenerateContentSetupResult:
"""
@@ -93,8 +96,7 @@ class GenerateContentHelper:
contents: The content to generate from
config: Optional configuration
custom_llm_provider: Optional custom LLM provider
- stream: Whether this is a streaming call
- local_vars: Local variables from the calling function
+ tools: Optional tools
**kwargs: Additional keyword arguments
Returns:
@@ -110,7 +112,7 @@ class GenerateContentHelper:
## MOCK RESPONSE LOGIC (only for non-streaming)
if (
- not stream
+ not kwargs.get("stream", False)
and litellm_params.mock_response
and isinstance(litellm_params.mock_response, str)
):
@@ -166,6 +168,7 @@ class GenerateContentHelper:
generate_content_provider_config.transform_generate_content_request(
model=model,
contents=contents,
+ tools=tools,
generate_content_config_dict=generate_content_config_dict,
)
)
@@ -200,6 +203,7 @@ async def agenerate_content(
model: str,
contents: GenerateContentContentListUnionDict,
config: Optional[GenerateContentConfigDict] = None,
+ tools: Optional[ToolConfigDict] = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@@ -218,6 +222,9 @@ async def agenerate_content(
loop = asyncio.get_event_loop()
kwargs["agenerate_content"] = True
+ # Handle generationConfig parameter from kwargs for backward compatibility
+ if "generationConfig" in kwargs and config is None:
+ config = kwargs.pop("generationConfig")
# get custom llm provider so we can use this for mapping exceptions
if custom_llm_provider is None:
_, custom_llm_provider, _, _ = litellm.get_llm_provider(
@@ -235,6 +242,7 @@ async def agenerate_content(
extra_body=extra_body,
timeout=timeout,
custom_llm_provider=custom_llm_provider,
+ tools=tools,
**kwargs,
)
@@ -263,6 +271,7 @@ def generate_content(
model: str,
contents: GenerateContentContentListUnionDict,
config: Optional[GenerateContentConfigDict] = None,
+ tools: Optional[ToolConfigDict] = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@@ -278,8 +287,11 @@ def generate_content(
"""
local_vars = locals()
try:
- _is_async = kwargs.pop("agenerate_content", False) is True
+ _is_async = kwargs.pop("agenerate_content", False)
+ # Handle generationConfig parameter from kwargs for backward compatibility
+ if "generationConfig" in kwargs and config is None:
+ config = kwargs.pop("generationConfig")
# Check for mock response first
litellm_params = GenericLiteLLMParams(**kwargs)
if litellm_params.mock_response and isinstance(
@@ -295,7 +307,7 @@ def generate_content(
contents=contents,
config=config,
custom_llm_provider=custom_llm_provider,
- stream=False,
+ tools=tools,
**kwargs,
)
@@ -306,7 +318,7 @@ def generate_content(
model=model,
contents=contents, # type: ignore
config=setup_result.generate_content_config_dict,
- stream=False,
+ tools=tools,
_is_async=_is_async,
litellm_params=setup_result.litellm_params,
**kwargs,
@@ -316,6 +328,7 @@ def generate_content(
response = base_llm_http_handler.generate_content_handler(
model=setup_result.model,
contents=contents,
+ tools=tools,
generate_content_provider_config=setup_result.generate_content_provider_config,
generate_content_config_dict=setup_result.generate_content_config_dict,
custom_llm_provider=setup_result.custom_llm_provider,
@@ -326,7 +339,6 @@ def generate_content(
timeout=timeout or request_timeout,
_is_async=_is_async,
client=kwargs.get("client"),
- stream=False,
litellm_metadata=kwargs.get("litellm_metadata", {}),
)
@@ -346,6 +358,7 @@ async def agenerate_content_stream(
model: str,
contents: GenerateContentContentListUnionDict,
config: Optional[GenerateContentConfigDict] = None,
+ tools: Optional[ToolConfigDict] = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@@ -363,6 +376,9 @@ async def agenerate_content_stream(
try:
kwargs["agenerate_content_stream"] = True
+ # Handle generationConfig parameter from kwargs for backward compatibility
+ if "generationConfig" in kwargs and config is None:
+ config = kwargs.pop("generationConfig")
# get custom llm provider so we can use this for mapping exceptions
if custom_llm_provider is None:
_, custom_llm_provider, _, _ = litellm.get_llm_provider(
@@ -371,14 +387,12 @@ async def agenerate_content_stream(
# Setup the call
setup_result = GenerateContentHelper.setup_generate_content_call(
- **{
- "model": model,
- "contents": contents,
- "config": config,
- "custom_llm_provider": custom_llm_provider,
- "stream": True,
- **kwargs,
- }
+ model=model,
+ contents=contents,
+ config=config,
+ custom_llm_provider=custom_llm_provider,
+ tools=tools,
+ **kwargs,
)
# Check if we should use the adapter (when provider config is None)
@@ -390,6 +404,7 @@ async def agenerate_content_stream(
contents=contents, # type: ignore
config=setup_result.generate_content_config_dict,
litellm_params=setup_result.litellm_params,
+ tools=tools,
stream=True,
**kwargs,
)
@@ -402,6 +417,7 @@ async def agenerate_content_stream(
contents=contents,
generate_content_provider_config=setup_result.generate_content_provider_config,
generate_content_config_dict=setup_result.generate_content_config_dict,
+ tools=tools,
custom_llm_provider=setup_result.custom_llm_provider,
litellm_params=setup_result.litellm_params,
logging_obj=setup_result.litellm_logging_obj,
@@ -429,6 +445,7 @@ def generate_content_stream(
model: str,
contents: GenerateContentContentListUnionDict,
config: Optional[GenerateContentConfigDict] = None,
+ tools: Optional[ToolConfigDict] = None,
# Use the following arguments if you need to pass additional parameters to the API that aren't available via kwargs.
# The extra values given here take precedence over values defined on the client or passed to this method.
extra_headers: Optional[Dict[str, Any]] = None,
@@ -447,13 +464,16 @@ def generate_content_stream(
# Remove any async-related flags since this is the sync function
_is_async = kwargs.pop("agenerate_content_stream", False)
+ # Handle generationConfig parameter from kwargs for backward compatibility
+ if "generationConfig" in kwargs and config is None:
+ config = kwargs.pop("generationConfig")
# Setup the call
setup_result = GenerateContentHelper.setup_generate_content_call(
model=model,
contents=contents,
config=config,
custom_llm_provider=custom_llm_provider,
- stream=True,
+ tools=tools,
**kwargs,
)
@@ -464,9 +484,9 @@ def generate_content_stream(
model=model,
contents=contents, # type: ignore
config=setup_result.generate_content_config_dict,
- stream=True,
_is_async=_is_async,
litellm_params=setup_result.litellm_params,
+ stream=True,
**kwargs,
)
@@ -476,6 +496,7 @@ def generate_content_stream(
contents=contents,
generate_content_provider_config=setup_result.generate_content_provider_config,
generate_content_config_dict=setup_result.generate_content_config_dict,
+ tools=tools,
custom_llm_provider=setup_result.custom_llm_provider,
litellm_params=setup_result.litellm_params,
logging_obj=setup_result.litellm_logging_obj,
diff --git a/litellm/images/main.py b/litellm/images/main.py
index b808388d83e..2a8b62bce24 100644
--- a/litellm/images/main.py
+++ b/litellm/images/main.py
@@ -1,7 +1,7 @@
import asyncio
import contextvars
from functools import partial
-from typing import Any, Coroutine, Dict, Literal, Optional, Union, cast, overload
+from typing import Any, Coroutine, Dict, List, Literal, Optional, Union, cast, overload
import httpx
@@ -90,12 +90,12 @@ async def aimage_generation(*args, **kwargs) -> ImageResponse:
response = init_response
elif asyncio.iscoroutine(init_response):
response = await init_response # type: ignore
-
+
if response is None:
raise ValueError(
"Unable to get Image Response. Please pass a valid llm_provider."
)
-
+
return response
except Exception as e:
custom_llm_provider = custom_llm_provider or "openai"
@@ -108,6 +108,8 @@ async def aimage_generation(*args, **kwargs) -> ImageResponse:
)
+# fmt: off
+
# Overload for when aimg_generation=True (returns Coroutine)
@overload
def image_generation(
@@ -119,7 +121,6 @@ def image_generation(
size: Optional[str] = None,
style: Optional[str] = None,
user: Optional[str] = None,
- input_fidelity: Optional[str] = None,
timeout=600, # default to 10 minutes
api_key: Optional[str] = None,
api_base: Optional[str] = None,
@@ -128,10 +129,11 @@ def image_generation(
*,
aimg_generation: Literal[True],
**kwargs,
-) -> Coroutine[Any, Any, ImageResponse]:
+) -> Coroutine[Any, Any, ImageResponse]:
...
+
# Overload for when aimg_generation=False or not specified (returns ImageResponse)
@overload
def image_generation(
@@ -143,7 +145,6 @@ def image_generation(
size: Optional[str] = None,
style: Optional[str] = None,
user: Optional[str] = None,
- input_fidelity: Optional[str] = None,
timeout=600, # default to 10 minutes
api_key: Optional[str] = None,
api_base: Optional[str] = None,
@@ -152,9 +153,11 @@ def image_generation(
*,
aimg_generation: Literal[False] = False,
**kwargs,
-) -> ImageResponse:
+) -> ImageResponse:
...
+# fmt: on
+
@client
def image_generation( # noqa: PLR0915
@@ -166,7 +169,6 @@ def image_generation( # noqa: PLR0915
size: Optional[str] = None,
style: Optional[str] = None,
user: Optional[str] = None,
- input_fidelity: Optional[str] = None,
timeout=600, # default to 10 minutes
api_key: Optional[str] = None,
api_base: Optional[str] = None,
@@ -174,9 +176,9 @@ def image_generation( # noqa: PLR0915
custom_llm_provider=None,
**kwargs,
) -> Union[
- ImageResponse,
- Coroutine[Any, Any, ImageResponse],
- ]:
+ ImageResponse,
+ Coroutine[Any, Any, ImageResponse],
+]:
"""
Maps the https://api.openai.com/v1/images/generations endpoint.
@@ -227,7 +229,6 @@ def image_generation( # noqa: PLR0915
"quality",
"size",
"style",
- "input_fidelity",
]
litellm_params = all_litellm_params
default_params = openai_params + litellm_params
@@ -255,7 +256,6 @@ def image_generation( # noqa: PLR0915
size=size,
style=style,
user=user,
- input_fidelity=input_fidelity,
custom_llm_provider=custom_llm_provider,
provider_config=image_generation_config,
**non_default_params,
@@ -311,7 +311,7 @@ def image_generation( # noqa: PLR0915
) or get_secret_str("AZURE_AD_TOKEN")
default_headers = {
- "Content-Type": "application/json;",
+ "Content-Type": "application/json",
"api-key": api_key,
}
for k, v in default_headers.items():
@@ -335,8 +335,67 @@ def image_generation( # noqa: PLR0915
headers=headers,
litellm_params=litellm_params_dict,
)
+ #########################################################
+ # Providers using llm_http_handler
+ #########################################################
+ elif custom_llm_provider in (
+ litellm.LlmProviders.RECRAFT,
+ litellm.LlmProviders.AIML,
+ litellm.LlmProviders.GEMINI,
+ ):
+ if image_generation_config is None:
+ raise ValueError(
+ f"image generation config is not supported for {custom_llm_provider}"
+ )
+
+ return llm_http_handler.image_generation_handler(
+ api_key=api_key,
+ model=model,
+ prompt=prompt,
+ image_generation_provider_config=image_generation_config,
+ image_generation_optional_request_params=optional_params,
+ custom_llm_provider=custom_llm_provider,
+ litellm_params=litellm_params_dict,
+ logging_obj=litellm_logging_obj,
+ timeout=timeout,
+ client=client,
+ )
+ elif custom_llm_provider == "azure_ai":
+ from litellm.llms.azure_ai.common_utils import AzureFoundryModelInfo
+
+ api_base = AzureFoundryModelInfo.get_api_base(api_base)
+ api_key = AzureFoundryModelInfo.get_api_key(api_key)
+ if extra_headers is not None:
+ optional_params["extra_headers"] = extra_headers
+
+ default_headers = {
+ "Content-Type": "application/json",
+ "api-key": api_key,
+ }
+ for k, v in default_headers.items():
+ if k not in headers:
+ headers[k] = v
+
+ model_response = azure_chat_completions.image_generation(
+ model=model,
+ prompt=prompt,
+ timeout=timeout,
+ api_key=api_key,
+ api_base=api_base,
+ azure_ad_token=None,
+ azure_ad_token_provider=azure_ad_token_provider,
+ logging_obj=litellm_logging_obj,
+ optional_params=optional_params,
+ model_response=model_response,
+ api_version=api_version,
+ aimg_generation=aimg_generation,
+ client=client,
+ headers=headers,
+ litellm_params=litellm_params_dict,
+ )
elif (
custom_llm_provider == "openai"
+ or custom_llm_provider == LlmProviders.LITELLM_PROXY.value
or custom_llm_provider in litellm.openai_compatible_providers
):
model_response = openai_chat_completions.image_generation(
@@ -364,7 +423,7 @@ def image_generation( # noqa: PLR0915
aimg_generation=aimg_generation,
client=client,
api_base=api_base,
- api_key=api_key
+ api_key=api_key,
)
elif custom_llm_provider == "vertex_ai":
vertex_ai_project = (
@@ -406,28 +465,6 @@ def image_generation( # noqa: PLR0915
api_base=api_base,
client=client,
)
- #########################################################
- # Providers using llm_http_handler
- #########################################################
- elif custom_llm_provider in (
- litellm.LlmProviders.RECRAFT,
- litellm.LlmProviders.GEMINI,
-
- ):
- if image_generation_config is None:
- raise ValueError(f"image generation config is not supported for {custom_llm_provider}")
-
- return llm_http_handler.image_generation_handler(
- model=model,
- prompt=prompt,
- image_generation_provider_config=image_generation_config,
- image_generation_optional_request_params=optional_params,
- custom_llm_provider=custom_llm_provider,
- litellm_params=litellm_params_dict,
- logging_obj=litellm_logging_obj,
- timeout=timeout,
- client=client,
- )
elif (
custom_llm_provider in litellm._custom_providers
): # Assume custom LLM provider
@@ -643,7 +680,7 @@ def image_variation(
@client
def image_edit(
- image: FileTypes,
+ image: Union[FileTypes, List[FileTypes]],
prompt: str,
model: Optional[str] = None,
mask: Optional[str] = None,
@@ -671,6 +708,9 @@ def image_edit(
litellm_call_id: Optional[str] = kwargs.get("litellm_call_id", None)
_is_async = kwargs.pop("async_call", False) is True
+ # add images / or return a single image
+ images = image if isinstance(image, list) else [image]
+
# get llm provider logic
litellm_params = GenericLiteLLMParams(**kwargs)
model, custom_llm_provider, _, _ = get_llm_provider(
@@ -679,11 +719,11 @@ def image_edit(
)
# get provider config
- image_edit_provider_config: Optional[
- BaseImageEditConfig
- ] = ProviderConfigManager.get_provider_image_edit_config(
- model=model,
- provider=litellm.LlmProviders(custom_llm_provider),
+ image_edit_provider_config: Optional[BaseImageEditConfig] = (
+ ProviderConfigManager.get_provider_image_edit_config(
+ model=model,
+ provider=litellm.LlmProviders(custom_llm_provider),
+ )
)
if image_edit_provider_config is None:
@@ -719,7 +759,7 @@ def image_edit(
# Call the handler with _is_async flag instead of directly calling the async handler
return base_llm_http_handler.image_edit_handler(
model=model,
- image=image,
+ image=images,
prompt=prompt,
image_edit_provider_config=image_edit_provider_config,
image_edit_optional_request_params=image_edit_request_params,
@@ -745,7 +785,7 @@ def image_edit(
@client
async def aimage_edit(
- image: FileTypes,
+ image: Union[FileTypes, List[FileTypes]],
model: str,
prompt: str,
mask: Optional[str] = None,
@@ -785,9 +825,11 @@ async def aimage_edit(
model=model, api_base=local_vars.get("base_url", None)
)
+ images = image if isinstance(image, list) else [image]
+
func = partial(
image_edit,
- image=image,
+ image=images,
prompt=prompt,
mask=mask,
model=model,
diff --git a/litellm/integrations/SlackAlerting/budget_alert_types.py b/litellm/integrations/SlackAlerting/budget_alert_types.py
index beebee8b6bf..1e9ad286e37 100644
--- a/litellm/integrations/SlackAlerting/budget_alert_types.py
+++ b/litellm/integrations/SlackAlerting/budget_alert_types.py
@@ -31,7 +31,7 @@ class SoftBudgetAlert(BaseBudgetAlertType):
return "Soft Budget Crossed: "
def get_id(self, user_info: CallInfo) -> str:
- return "default_id"
+ return user_info.token or "default_id"
class UserBudgetAlert(BaseBudgetAlertType):
diff --git a/litellm/integrations/SlackAlerting/slack_alerting.py b/litellm/integrations/SlackAlerting/slack_alerting.py
index 41db4a551bd..7da38e193b6 100644
--- a/litellm/integrations/SlackAlerting/slack_alerting.py
+++ b/litellm/integrations/SlackAlerting/slack_alerting.py
@@ -805,9 +805,9 @@ class SlackAlerting(CustomBatchLogger):
### UNIQUE CACHE KEY ###
cache_key = provider + region_name
- outage_value: Optional[ProviderRegionOutageModel] = (
- await self.internal_usage_cache.async_get_cache(key=cache_key)
- )
+ outage_value: Optional[
+ ProviderRegionOutageModel
+ ] = await self.internal_usage_cache.async_get_cache(key=cache_key)
if (
getattr(exception, "status_code", None) is None
@@ -1367,12 +1367,13 @@ Model Info:
# Get the current timestamp
current_time = datetime.now().strftime("%H:%M:%S")
_proxy_base_url = os.getenv("PROXY_BASE_URL", None)
+ # Use .name if it's an enum, otherwise use as is
+ alert_type_name = getattr(alert_type, 'name', alert_type)
+ alert_type_formatted = f"Alert type: `{alert_type_name}`"
if alert_type == "daily_reports" or alert_type == "new_model_added":
- formatted_message = message
+ formatted_message = alert_type_formatted + message
else:
- formatted_message = (
- f"Level: `{level}`\nTimestamp: `{current_time}`\n\nMessage: {message}"
- )
+ formatted_message = f"{alert_type_formatted}\nLevel: `{level}`\nTimestamp: `{current_time}`\n\nMessage: {message}"
if kwargs:
for key, value in kwargs.items():
@@ -1388,9 +1389,9 @@ Model Info:
self.alert_to_webhook_url is not None
and alert_type in self.alert_to_webhook_url
):
- slack_webhook_url: Optional[Union[str, List[str]]] = (
- self.alert_to_webhook_url[alert_type]
- )
+ slack_webhook_url: Optional[
+ Union[str, List[str]]
+ ] = self.alert_to_webhook_url[alert_type]
elif self.default_webhook_url is not None:
slack_webhook_url = self.default_webhook_url
else:
diff --git a/litellm/integrations/anthropic_cache_control_hook.py b/litellm/integrations/anthropic_cache_control_hook.py
index 29d9920da43..c1fb45b3042 100644
--- a/litellm/integrations/anthropic_cache_control_hook.py
+++ b/litellm/integrations/anthropic_cache_control_hook.py
@@ -9,6 +9,7 @@ Users can define
import copy
from typing import Dict, List, Optional, Tuple, Union, cast
+from litellm._logging import verbose_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.custom_prompt_management import CustomPromptManagement
from litellm.types.integrations.anthropic_cache_control_hook import (
@@ -80,12 +81,22 @@ class AnthropicCacheControlHook(CustomPromptManagement):
# Case 1: Target by specific index
if targetted_index is not None:
+ original_index = targetted_index
+ # Handle negative indices (convert to positive)
+ if targetted_index < 0:
+ targetted_index += len(messages)
+
if 0 <= targetted_index < len(messages):
messages[targetted_index] = (
AnthropicCacheControlHook._safe_insert_cache_control_in_message(
messages[targetted_index], control
)
)
+ else:
+ verbose_logger.warning(
+ f"AnthropicCacheControlHook: Provided index {original_index} is out of bounds for message list of length {len(messages)}. "
+ f"Targeted index was {targetted_index}. Skipping cache control injection for this point."
+ )
# Case 2: Target by role
elif targetted_role is not None:
for msg in messages:
diff --git a/litellm/integrations/azure_storage/azure_storage.py b/litellm/integrations/azure_storage/azure_storage.py
index 6ffb1e542fc..b4362665a4c 100644
--- a/litellm/integrations/azure_storage/azure_storage.py
+++ b/litellm/integrations/azure_storage/azure_storage.py
@@ -2,7 +2,7 @@ import asyncio
import json
import os
import time
-import uuid
+from litellm._uuid import uuid
from datetime import datetime, timedelta
from typing import List, Optional
diff --git a/litellm/integrations/bitbucket/README.md b/litellm/integrations/bitbucket/README.md
new file mode 100644
index 00000000000..473beeea9e0
--- /dev/null
+++ b/litellm/integrations/bitbucket/README.md
@@ -0,0 +1,317 @@
+# LiteLLM BitBucket Prompt Management
+
+A powerful prompt management system for LiteLLM that fetches `.prompt` files from BitBucket repositories. This enables team-based prompt management with BitBucket's built-in access control and version control capabilities.
+
+## Features
+
+- **🏢 Team-based access control**: Leverage BitBucket's workspace and repository permissions
+- **📁 Repository-based prompt storage**: Store prompts in BitBucket repositories
+- **🔐 Multiple authentication methods**: Support for access tokens and basic auth
+- **🎯 YAML frontmatter**: Define model, parameters, and schemas in file headers
+- **🔧 Handlebars templating**: Use `{{variable}}` syntax with Jinja2 backend
+- **✅ Input validation**: Automatic validation against defined schemas
+- **🔗 LiteLLM integration**: Works seamlessly with `litellm.completion()`
+- **💬 Smart message parsing**: Converts prompts to proper chat messages
+- **⚙️ Parameter extraction**: Automatically applies model settings from prompts
+
+## Quick Start
+
+### 1. Set up BitBucket Repository
+
+Create a repository in your BitBucket workspace and add `.prompt` files:
+
+```
+your-repo/
+├── prompts/
+│ ├── chat_assistant.prompt
+│ ├── code_reviewer.prompt
+│ └── data_analyst.prompt
+```
+
+### 2. Create a `.prompt` file
+
+Create a file called `prompts/chat_assistant.prompt`:
+
+```yaml
+---
+model: gpt-4
+temperature: 0.7
+max_tokens: 150
+input:
+ schema:
+ user_message: string
+ system_context?: string
+---
+
+{% if system_context %}System: {{system_context}}
+
+{% endif %}User: {{user_message}}
+```
+
+### 3. Configure BitBucket Access
+
+#### Option A: Access Token (Recommended)
+
+```python
+import litellm
+
+# Configure BitBucket access
+bitbucket_config = {
+ "workspace": "your-workspace",
+ "repository": "your-repo",
+ "access_token": "your-access-token",
+ "branch": "main" # optional, defaults to main
+}
+
+# Set global BitBucket configuration
+litellm.set_global_bitbucket_config(bitbucket_config)
+```
+
+#### Option B: Basic Authentication
+
+```python
+import litellm
+
+# Configure BitBucket access with basic auth
+bitbucket_config = {
+ "workspace": "your-workspace",
+ "repository": "your-repo",
+ "username": "your-username",
+ "access_token": "your-app-password", # Use app password for basic auth
+ "auth_method": "basic",
+ "branch": "main"
+}
+
+litellm.set_global_bitbucket_config(bitbucket_config)
+```
+
+### 4. Use with LiteLLM
+
+```python
+# Use with completion - the model prefix 'bitbucket/' tells LiteLLM to use BitBucket prompt management
+response = litellm.completion(
+ model="bitbucket/gpt-4", # The actual model comes from the .prompt file
+ prompt_id="prompts/chat_assistant", # Location of the prompt file
+ prompt_variables={
+ "user_message": "What is machine learning?",
+ "system_context": "You are a helpful AI tutor."
+ },
+ # Any additional messages will be appended after the prompt
+ messages=[{"role": "user", "content": "Please explain it simply."}]
+)
+
+print(response.choices[0].message.content)
+```
+
+## Proxy Server Configuration
+
+### 1. Create a `.prompt` file
+
+Create `prompts/hello.prompt`:
+
+```yaml
+---
+model: gpt-4
+temperature: 0.7
+---
+System: You are a helpful assistant.
+
+User: {{user_message}}
+```
+
+### 2. Setup config.yaml
+
+```yaml
+model_list:
+ - model_name: my-bitbucket-model
+ litellm_params:
+ model: bitbucket/gpt-4
+ prompt_id: "prompts/hello"
+ api_key: os.environ/OPENAI_API_KEY
+
+litellm_settings:
+ global_bitbucket_config:
+ workspace: "your-workspace"
+ repository: "your-repo"
+ access_token: "your-access-token"
+ branch: "main"
+```
+
+### 3. Start the proxy
+
+```bash
+litellm --config config.yaml --detailed_debug
+```
+
+### 4. Test it!
+
+```bash
+curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
+-H 'Content-Type: application/json' \
+-H 'Authorization: Bearer sk-1234' \
+-d '{
+ "model": "my-bitbucket-model",
+ "messages": [{"role": "user", "content": "IGNORED"}],
+ "prompt_variables": {
+ "user_message": "What is the capital of France?"
+ }
+}'
+```
+
+## Prompt File Format
+
+### Basic Structure
+
+```yaml
+---
+# Model configuration
+model: gpt-4
+temperature: 0.7
+max_tokens: 500
+
+# Input schema (optional)
+input:
+ schema:
+ user_message: string
+ system_context?: string
+---
+
+System: You are a helpful {{role}} assistant.
+
+User: {{user_message}}
+```
+
+### Advanced Features
+
+**Multi-role conversations:**
+
+```yaml
+---
+model: gpt-4
+temperature: 0.3
+---
+System: You are a helpful coding assistant.
+
+User: {{user_question}}
+```
+
+**Dynamic model selection:**
+
+```yaml
+---
+model: "{{preferred_model}}" # Model can be a variable
+temperature: 0.7
+---
+System: You are a helpful assistant specialized in {{domain}}.
+
+User: {{user_message}}
+```
+
+## Team-Based Access Control
+
+BitBucket's built-in permission system provides team-based access control:
+
+1. **Workspace-level permissions**: Control access to entire workspaces
+2. **Repository-level permissions**: Control access to specific repositories
+3. **Branch-level permissions**: Control access to specific branches
+4. **User and group management**: Manage team members and their access levels
+
+### Setting up Team Access
+
+1. **Create workspaces for each team**:
+ ```
+ team-a-prompts/
+ team-b-prompts/
+ team-c-prompts/
+ ```
+
+2. **Configure repository permissions**:
+ - Grant read access to team members
+ - Grant write access to prompt maintainers
+ - Use branch protection rules for production prompts
+
+3. **Use different access tokens**:
+ - Each team can have their own access token
+ - Tokens can be scoped to specific repositories
+ - Use app passwords for additional security
+
+## API Reference
+
+### BitBucket Configuration
+
+```python
+bitbucket_config = {
+ "workspace": str, # Required: BitBucket workspace name
+ "repository": str, # Required: Repository name
+ "access_token": str, # Required: BitBucket access token or app password
+ "branch": str, # Optional: Branch to fetch from (default: "main")
+ "base_url": str, # Optional: Custom BitBucket API URL
+ "auth_method": str, # Optional: "token" or "basic" (default: "token")
+ "username": str, # Optional: Username for basic auth
+ "base_url" : str # Optional: Incase where the base url is not https://api.bitbucket.org/2.0
+}
+```
+
+### LiteLLM Integration
+
+```python
+response = litellm.completion(
+ model="bitbucket/", # required (e.g., bitbucket/gpt-4)
+ prompt_id=str, # required - the .prompt filename without extension
+ prompt_variables=dict, # optional - variables for template rendering
+ bitbucket_config=dict, # optional - BitBucket configuration (if not set globally)
+ messages=list, # optional - additional messages
+)
+```
+
+## Error Handling
+
+The BitBucket integration provides detailed error messages for common issues:
+
+- **Authentication errors**: Invalid access tokens or credentials
+- **Permission errors**: Insufficient access to workspace/repository
+- **File not found**: Missing .prompt files
+- **Network errors**: Connection issues with BitBucket API
+
+## Security Considerations
+
+1. **Access Token Security**: Store access tokens securely using environment variables or secret management systems
+2. **Repository Permissions**: Use BitBucket's permission system to control access
+3. **Branch Protection**: Protect main branches from unauthorized changes
+4. **Audit Logging**: BitBucket provides audit logs for all repository access
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"Access denied" errors**: Check your BitBucket permissions for the workspace and repository
+2. **"Authentication failed" errors**: Verify your access token or credentials
+3. **"File not found" errors**: Ensure the .prompt file exists in the specified branch
+4. **Template rendering errors**: Check your Handlebars syntax in the .prompt file
+
+### Debug Mode
+
+Enable debug logging to troubleshoot issues:
+
+```python
+import litellm
+litellm.set_verbose = True
+
+# Your BitBucket prompt calls will now show detailed logs
+response = litellm.completion(
+ model="bitbucket/gpt-4",
+ prompt_id="your_prompt",
+ prompt_variables={"key": "value"}
+)
+```
+
+## Migration from File-Based Prompts
+
+If you're currently using file-based prompts with the dotprompt integration, you can easily migrate to BitBucket:
+
+1. **Upload your .prompt files** to a BitBucket repository
+2. **Update your configuration** to use BitBucket instead of local files
+3. **Set up team access** using BitBucket's permission system
+4. **Update your code** to use `bitbucket/` model prefix instead of `dotprompt/`
+
+This provides better collaboration, version control, and team-based access control for your prompts.
diff --git a/litellm/integrations/bitbucket/__init__.py b/litellm/integrations/bitbucket/__init__.py
new file mode 100644
index 00000000000..111d38f78a4
--- /dev/null
+++ b/litellm/integrations/bitbucket/__init__.py
@@ -0,0 +1,66 @@
+from typing import TYPE_CHECKING, Optional
+
+if TYPE_CHECKING:
+ from .bitbucket_prompt_manager import BitBucketPromptManager
+ from litellm.types.prompts.init_prompts import PromptLiteLLMParams, PromptSpec
+ from litellm.integrations.custom_prompt_management import CustomPromptManagement
+
+from litellm.types.prompts.init_prompts import SupportedPromptIntegrations
+
+from .bitbucket_prompt_manager import BitBucketPromptManager
+
+# Global instances
+global_bitbucket_config: Optional[dict] = None
+
+
+def set_global_bitbucket_config(config: dict) -> None:
+ """
+ Set the global BitBucket configuration for prompt management.
+
+ Args:
+ config: Dictionary containing BitBucket configuration
+ - workspace: BitBucket workspace name
+ - repository: Repository name
+ - access_token: BitBucket access token
+ - branch: Branch to fetch prompts from (default: main)
+ """
+ import litellm
+
+ litellm.global_bitbucket_config = config # type: ignore
+
+
+def prompt_initializer(
+ litellm_params: "PromptLiteLLMParams", prompt_spec: "PromptSpec"
+) -> "CustomPromptManagement":
+ """
+ Initialize a prompt from a BitBucket repository.
+ """
+ bitbucket_config = getattr(litellm_params, "bitbucket_config", None)
+ prompt_id = getattr(litellm_params, "prompt_id", None)
+
+ if not bitbucket_config:
+ raise ValueError(
+ "bitbucket_config is required for BitBucket prompt integration"
+ )
+
+ try:
+ bitbucket_prompt_manager = BitBucketPromptManager(
+ bitbucket_config=bitbucket_config,
+ prompt_id=prompt_id,
+ )
+
+ return bitbucket_prompt_manager
+ except Exception as e:
+ raise e
+
+
+prompt_initializer_registry = {
+ SupportedPromptIntegrations.BITBUCKET.value: prompt_initializer,
+}
+
+# Export public API
+__all__ = [
+ "BitBucketPromptManager",
+ "set_global_bitbucket_config",
+ "global_bitbucket_config",
+]
diff --git a/litellm/integrations/bitbucket/bitbucket_client.py b/litellm/integrations/bitbucket/bitbucket_client.py
new file mode 100644
index 00000000000..0502422cf8b
--- /dev/null
+++ b/litellm/integrations/bitbucket/bitbucket_client.py
@@ -0,0 +1,241 @@
+"""
+BitBucket API client for fetching .prompt files from BitBucket repositories.
+"""
+
+import base64
+from typing import Any, Dict, List, Optional
+
+from litellm.llms.custom_httpx.http_handler import HTTPHandler
+
+
+class BitBucketClient:
+ """
+ Client for interacting with BitBucket API to fetch .prompt files.
+
+ Supports:
+ - Authentication with access tokens
+ - Fetching file contents from repositories
+ - Team-based access control through BitBucket permissions
+ - Branch-specific file fetching
+ """
+
+ def __init__(self, config: Dict[str, Any]):
+ """
+ Initialize the BitBucket client.
+
+ Args:
+ config: Dictionary containing:
+ - workspace: BitBucket workspace name
+ - repository: Repository name
+ - access_token: BitBucket access token (or app password)
+ - branch: Branch to fetch from (default: main)
+ - base_url: Custom BitBucket API base URL (optional)
+ - auth_method: Authentication method ('token' or 'basic', default: 'token')
+ - username: Username for basic auth (optional)
+ """
+ self.workspace = config.get("workspace")
+ self.repository = config.get("repository")
+ self.access_token = config.get("access_token")
+ self.branch = config.get("branch", "main")
+ self.base_url = config.get("", "https://api.bitbucket.org/2.0")
+ self.auth_method = config.get("auth_method", "token")
+ self.username = config.get("username")
+
+ if not all([self.workspace, self.repository, self.access_token]):
+ raise ValueError("workspace, repository, and access_token are required")
+
+ # Set up authentication headers
+ self.headers = {
+ "Accept": "application/json",
+ "Content-Type": "application/json",
+ }
+
+ if self.auth_method == "basic" and self.username:
+ # Use basic auth with username and app password
+ credentials = f"{self.username}:{self.access_token}"
+ encoded_credentials = base64.b64encode(credentials.encode()).decode()
+ self.headers["Authorization"] = f"Basic {encoded_credentials}"
+ else:
+ # Use token-based authentication (default)
+ self.headers["Authorization"] = f"Bearer {self.access_token}"
+
+ # Initialize HTTPHandler
+ self.http_handler = HTTPHandler()
+
+ def get_file_content(self, file_path: str) -> Optional[str]:
+ """
+ Fetch the content of a file from the BitBucket repository.
+
+ Args:
+ file_path: Path to the file in the repository
+
+ Returns:
+ File content as string, or None if file not found
+ """
+ url = f"{self.base_url}/repositories/{self.workspace}/{self.repository}/src/{self.branch}/{file_path}"
+
+ try:
+ response = self.http_handler.get(url, headers=self.headers)
+ response.raise_for_status()
+
+ # BitBucket returns file content as base64 encoded
+ if response.headers.get("content-type", "").startswith("text/"):
+ return response.text
+ else:
+ # For binary files or when content-type is not text, try to decode as base64
+ try:
+ return base64.b64decode(response.content).decode("utf-8")
+ except Exception:
+ return response.text
+
+ except Exception as e:
+ # Check if it's an HTTP error
+ if hasattr(e, "response") and hasattr(e.response, "status_code"):
+ if e.response.status_code == 404:
+ return None
+ elif e.response.status_code == 403:
+ raise Exception(
+ f"Access denied to file '{file_path}'. Check your BitBucket permissions for workspace '{self.workspace}' and repository '{self.repository}'."
+ )
+ elif e.response.status_code == 401:
+ raise Exception(
+ "Authentication failed. Check your BitBucket access token and permissions."
+ )
+ else:
+ raise Exception(f"Failed to fetch file '{file_path}': {e}")
+ else:
+ raise Exception(f"Error fetching file '{file_path}': {e}")
+
+ def list_files(
+ self, directory_path: str = "", file_extension: str = ".prompt"
+ ) -> List[str]:
+ """
+ List files in a directory with a specific extension.
+
+ Args:
+ directory_path: Directory path in the repository (empty for root)
+ file_extension: File extension to filter by (default: .prompt)
+
+ Returns:
+ List of file paths
+ """
+ url = f"{self.base_url}/repositories/{self.workspace}/{self.repository}/src/{self.branch}/{directory_path}"
+
+ try:
+ response = self.http_handler.get(url, headers=self.headers)
+ response.raise_for_status()
+
+ data = response.json()
+ files = []
+
+ for item in data.get("values", []):
+ if item.get("type") == "commit_file":
+ file_path = item.get("path", "")
+ if file_path.endswith(file_extension):
+ files.append(file_path)
+
+ return files
+
+ except Exception as e:
+ # Check if it's an HTTP error
+ if hasattr(e, "response") and hasattr(e.response, "status_code"):
+ if e.response.status_code == 404:
+ return []
+ elif e.response.status_code == 403:
+ raise Exception(
+ f"Access denied to directory '{directory_path}'. Check your BitBucket permissions for workspace '{self.workspace}' and repository '{self.repository}'."
+ )
+ elif e.response.status_code == 401:
+ raise Exception(
+ "Authentication failed. Check your BitBucket access token and permissions."
+ )
+ else:
+ raise Exception(f"Failed to list files in '{directory_path}': {e}")
+ else:
+ raise Exception(f"Error listing files in '{directory_path}': {e}")
+
+ def get_repository_info(self) -> Dict[str, Any]:
+ """
+ Get information about the repository.
+
+ Returns:
+ Dictionary containing repository information
+ """
+ url = f"{self.base_url}/repositories/{self.workspace}/{self.repository}"
+
+ try:
+ response = self.http_handler.get(url, headers=self.headers)
+ response.raise_for_status()
+ return response.json()
+ except Exception as e:
+ raise Exception(f"Failed to get repository info: {e}")
+
+ def test_connection(self) -> bool:
+ """
+ Test the connection to the BitBucket repository.
+
+ Returns:
+ True if connection is successful, False otherwise
+ """
+ try:
+ self.get_repository_info()
+ return True
+ except Exception:
+ return False
+
+ def get_branches(self) -> List[Dict[str, Any]]:
+ """
+ Get list of branches in the repository.
+
+ Returns:
+ List of branch information dictionaries
+ """
+ url = f"{self.base_url}/repositories/{self.workspace}/{self.repository}/refs/branches"
+
+ try:
+ response = self.http_handler.get(url, headers=self.headers)
+ response.raise_for_status()
+
+ data = response.json()
+ return data.get("values", [])
+ except Exception as e:
+ raise Exception(f"Failed to get branches: {e}")
+
+ def get_file_metadata(self, file_path: str) -> Optional[Dict[str, Any]]:
+ """
+ Get metadata about a file (size, last modified, etc.).
+
+ Args:
+ file_path: Path to the file in the repository
+
+ Returns:
+ Dictionary containing file metadata, or None if file not found
+ """
+ url = f"{self.base_url}/repositories/{self.workspace}/{self.repository}/src/{self.branch}/{file_path}"
+
+ try:
+ # Use GET with Range header to get just the headers (HEAD equivalent)
+ headers = self.headers.copy()
+ headers["Range"] = "bytes=0-0" # Request only first byte to get headers
+
+ response = self.http_handler.get(url, headers=headers)
+ response.raise_for_status()
+
+ return {
+ "content_type": response.headers.get("content-type"),
+ "content_length": response.headers.get("content-length"),
+ "last_modified": response.headers.get("last-modified"),
+ }
+ except Exception as e:
+ # Check if it's an HTTP error
+ if hasattr(e, "response") and hasattr(e.response, "status_code"):
+ if e.response.status_code == 404:
+ return None
+ raise Exception(f"Failed to get file metadata for '{file_path}': {e}")
+ else:
+ raise Exception(f"Error getting file metadata for '{file_path}': {e}")
+
+ def close(self):
+ """Close the HTTP handler to free resources."""
+ if hasattr(self, "http_handler"):
+ self.http_handler.close()
diff --git a/litellm/integrations/bitbucket/bitbucket_prompt_manager.py b/litellm/integrations/bitbucket/bitbucket_prompt_manager.py
new file mode 100644
index 00000000000..d683fa3a0d4
--- /dev/null
+++ b/litellm/integrations/bitbucket/bitbucket_prompt_manager.py
@@ -0,0 +1,508 @@
+"""
+BitBucket prompt manager that integrates with LiteLLM's prompt management system.
+Fetches .prompt files from BitBucket repositories and provides team-based access control.
+"""
+
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+from jinja2 import DictLoader, Environment, select_autoescape
+
+from litellm.integrations.custom_prompt_management import CustomPromptManagement
+from litellm.integrations.prompt_management_base import (
+ PromptManagementBase,
+ PromptManagementClient,
+)
+from litellm.types.llms.openai import AllMessageValues
+from litellm.types.utils import StandardCallbackDynamicParams
+
+from .bitbucket_client import BitBucketClient
+
+
+class BitBucketPromptTemplate:
+ """
+ Represents a prompt template loaded from BitBucket.
+ """
+
+ def __init__(
+ self,
+ template_id: str,
+ content: str,
+ metadata: Dict[str, Any],
+ model: Optional[str] = None,
+ ):
+ self.template_id = template_id
+ self.content = content
+ self.metadata = metadata
+ self.model = model or metadata.get("model")
+ self.temperature = metadata.get("temperature")
+ self.max_tokens = metadata.get("max_tokens")
+ self.input_schema = metadata.get("input", {}).get("schema", {})
+ self.optional_params = {
+ k: v for k, v in metadata.items() if k not in ["model", "input", "content"]
+ }
+
+ def __repr__(self):
+ return f"BitBucketPromptTemplate(id='{self.template_id}', model='{self.model}')"
+
+
+class BitBucketTemplateManager:
+ """
+ Manager for loading and rendering .prompt files from BitBucket repositories.
+
+ Supports:
+ - Fetching .prompt files from BitBucket repositories
+ - Team-based access control through BitBucket permissions
+ - YAML frontmatter for metadata
+ - Handlebars-style templating (using Jinja2)
+ - Input/output schema validation
+ - Model configuration
+ """
+
+ def __init__(
+ self,
+ bitbucket_config: Dict[str, Any],
+ prompt_id: Optional[str] = None,
+ ):
+ self.bitbucket_config = bitbucket_config
+ self.prompt_id = prompt_id
+ self.prompts: Dict[str, BitBucketPromptTemplate] = {}
+ self.bitbucket_client = BitBucketClient(bitbucket_config)
+
+ self.jinja_env = Environment(
+ loader=DictLoader({}),
+ autoescape=select_autoescape(["html", "xml"]),
+ # Use Handlebars-style delimiters to match Dotprompt spec
+ variable_start_string="{{",
+ variable_end_string="}}",
+ block_start_string="{%",
+ block_end_string="%}",
+ comment_start_string="{#",
+ comment_end_string="#}",
+ )
+
+ # Load prompts from BitBucket if prompt_id is provided
+ if self.prompt_id:
+ self._load_prompt_from_bitbucket(self.prompt_id)
+
+ def _load_prompt_from_bitbucket(self, prompt_id: str) -> None:
+ """Load a specific .prompt file from BitBucket."""
+ try:
+ # Fetch the .prompt file from BitBucket
+ prompt_content = self.bitbucket_client.get_file_content(
+ f"{prompt_id}.prompt"
+ )
+
+ if prompt_content:
+ template = self._parse_prompt_file(prompt_content, prompt_id)
+ self.prompts[prompt_id] = template
+ except Exception as e:
+ raise Exception(f"Failed to load prompt '{prompt_id}' from BitBucket: {e}")
+
+ def _parse_prompt_file(
+ self, content: str, prompt_id: str
+ ) -> BitBucketPromptTemplate:
+ """Parse a .prompt file content and extract metadata and template."""
+ # Split frontmatter and content
+ if content.startswith("---"):
+ parts = content.split("---", 2)
+ if len(parts) >= 3:
+ frontmatter_str = parts[1].strip()
+ template_content = parts[2].strip()
+ else:
+ frontmatter_str = ""
+ template_content = content
+ else:
+ frontmatter_str = ""
+ template_content = content
+
+ # Parse YAML frontmatter
+ metadata: Dict[str, Any] = {}
+ if frontmatter_str:
+ try:
+ import yaml
+
+ metadata = yaml.safe_load(frontmatter_str) or {}
+ except ImportError:
+ # Fallback to basic parsing if PyYAML is not available
+ metadata = self._parse_yaml_basic(frontmatter_str)
+ except Exception:
+ metadata = {}
+
+ return BitBucketPromptTemplate(
+ template_id=prompt_id,
+ content=template_content,
+ metadata=metadata,
+ )
+
+ def _parse_yaml_basic(self, yaml_str: str) -> Dict[str, Any]:
+ """Basic YAML parser for simple cases when PyYAML is not available."""
+ result: Dict[str, Any] = {}
+ for line in yaml_str.split("\n"):
+ line = line.strip()
+ if ":" in line and not line.startswith("#"):
+ key, value = line.split(":", 1)
+ key = key.strip()
+ value = value.strip()
+
+ # Try to parse value as appropriate type
+ if value.lower() in ["true", "false"]:
+ result[key] = value.lower() == "true"
+ elif value.isdigit():
+ result[key] = int(value)
+ elif value.replace(".", "").isdigit():
+ result[key] = float(value)
+ else:
+ result[key] = value.strip("\"'")
+ return result
+
+ def render_template(
+ self, template_id: str, variables: Optional[Dict[str, Any]] = None
+ ) -> str:
+ """Render a template with the given variables."""
+ if template_id not in self.prompts:
+ raise ValueError(f"Template '{template_id}' not found")
+
+ template = self.prompts[template_id]
+ jinja_template = self.jinja_env.from_string(template.content)
+
+ return jinja_template.render(**(variables or {}))
+
+ def get_template(self, template_id: str) -> Optional[BitBucketPromptTemplate]:
+ """Get a template by ID."""
+ return self.prompts.get(template_id)
+
+ def list_templates(self) -> List[str]:
+ """List all available template IDs."""
+ return list(self.prompts.keys())
+
+
+class BitBucketPromptManager(CustomPromptManagement):
+ """
+ BitBucket prompt manager that integrates with LiteLLM's prompt management system.
+
+ This class enables using .prompt files from BitBucket repositories with the
+ litellm completion() function by implementing the PromptManagementBase interface.
+
+ Usage:
+ # Configure BitBucket access
+ bitbucket_config = {
+ "workspace": "your-workspace",
+ "repository": "your-repo",
+ "access_token": "your-token",
+ "branch": "main" # optional, defaults to main
+ }
+
+ # Use with completion
+ response = litellm.completion(
+ model="bitbucket/gpt-4",
+ prompt_id="my_prompt",
+ prompt_variables={"variable": "value"},
+ bitbucket_config=bitbucket_config,
+ messages=[{"role": "user", "content": "This will be combined with the prompt"}]
+ )
+ """
+
+ def __init__(
+ self,
+ bitbucket_config: Dict[str, Any],
+ prompt_id: Optional[str] = None,
+ ):
+ self.bitbucket_config = bitbucket_config
+ self.prompt_id = prompt_id
+ self._prompt_manager: Optional[BitBucketTemplateManager] = None
+
+ @property
+ def integration_name(self) -> str:
+ """Integration name used in model names like 'bitbucket/gpt-4'."""
+ return "bitbucket"
+
+ @property
+ def prompt_manager(self) -> BitBucketTemplateManager:
+ """Get or create the prompt manager instance."""
+ if self._prompt_manager is None:
+ self._prompt_manager = BitBucketTemplateManager(
+ bitbucket_config=self.bitbucket_config,
+ prompt_id=self.prompt_id,
+ )
+ return self._prompt_manager
+
+ def get_prompt_template(
+ self,
+ prompt_id: str,
+ prompt_variables: Optional[Dict[str, Any]] = None,
+ ) -> Tuple[str, Dict[str, Any]]:
+ """
+ Get a prompt template and render it with variables.
+
+ Args:
+ prompt_id: The ID of the prompt template
+ prompt_variables: Variables to substitute in the template
+
+ Returns:
+ Tuple of (rendered_prompt, metadata)
+ """
+ template = self.prompt_manager.get_template(prompt_id)
+ if not template:
+ raise ValueError(f"Prompt template '{prompt_id}' not found")
+
+ # Render the template
+ rendered_prompt = self.prompt_manager.render_template(
+ prompt_id, prompt_variables or {}
+ )
+
+ # Extract metadata
+ metadata = {
+ "model": template.model,
+ "temperature": template.temperature,
+ "max_tokens": template.max_tokens,
+ **template.optional_params,
+ }
+
+ return rendered_prompt, metadata
+
+ def pre_call_hook(
+ self,
+ user_id: Optional[str],
+ messages: List[AllMessageValues],
+ function_call: Optional[Union[Dict[str, Any], str]] = None,
+ litellm_params: Optional[Dict[str, Any]] = None,
+ prompt_id: Optional[str] = None,
+ prompt_variables: Optional[Dict[str, Any]] = None,
+ **kwargs,
+ ) -> Tuple[List[AllMessageValues], Optional[Dict[str, Any]]]:
+ """
+ Pre-call hook that processes the prompt template before making the LLM call.
+ """
+ if not prompt_id:
+ return messages, litellm_params
+
+ try:
+ # Get the rendered prompt and metadata
+ rendered_prompt, prompt_metadata = self.get_prompt_template(
+ prompt_id, prompt_variables
+ )
+
+ # Parse the rendered prompt into messages
+ parsed_messages = self._parse_prompt_to_messages(rendered_prompt)
+
+ # Merge with existing messages
+ if parsed_messages:
+ # If we have parsed messages, use them instead of the original messages
+ final_messages: List[AllMessageValues] = parsed_messages
+ else:
+ # If no messages were parsed, prepend the prompt to existing messages
+ final_messages = [
+ {"role": "user", "content": rendered_prompt} # type: ignore
+ ] + messages
+
+ # Update litellm_params with prompt metadata
+ if litellm_params is None:
+ litellm_params = {}
+
+ # Apply model and parameters from prompt metadata
+ if prompt_metadata.get("model"):
+ litellm_params["model"] = prompt_metadata["model"]
+
+ for param in [
+ "temperature",
+ "max_tokens",
+ "top_p",
+ "frequency_penalty",
+ "presence_penalty",
+ ]:
+ if param in prompt_metadata:
+ litellm_params[param] = prompt_metadata[param]
+
+ return final_messages, litellm_params
+
+ except Exception as e:
+ # Log error but don't fail the call
+ import litellm
+
+ litellm._logging.verbose_proxy_logger.error(
+ f"Error in BitBucket prompt pre_call_hook: {e}"
+ )
+ return messages, litellm_params
+
+ def _parse_prompt_to_messages(self, prompt_content: str) -> List[AllMessageValues]:
+ """
+ Parse prompt content into a list of messages.
+ Handles both simple prompts and multi-role conversations.
+ """
+ messages = []
+ lines = prompt_content.strip().split("\n")
+ current_role = None
+ current_content = []
+
+ for line in lines:
+ line = line.strip()
+ if not line:
+ continue
+
+ # Check for role indicators
+ if line.lower().startswith("system:"):
+ if current_role and current_content:
+ messages.append(
+ {
+ "role": current_role,
+ "content": "\n".join(current_content).strip(),
+ } # type: ignore
+ )
+ current_role = "system"
+ current_content = [line[7:].strip()] # Remove "System:" prefix
+ elif line.lower().startswith("user:"):
+ if current_role and current_content:
+ messages.append(
+ {
+ "role": current_role,
+ "content": "\n".join(current_content).strip(),
+ } # type: ignore
+ )
+ current_role = "user"
+ current_content = [line[5:].strip()] # Remove "User:" prefix
+ elif line.lower().startswith("assistant:"):
+ if current_role and current_content:
+ messages.append(
+ {
+ "role": current_role,
+ "content": "\n".join(current_content).strip(),
+ } # type: ignore
+ )
+ current_role = "assistant"
+ current_content = [line[10:].strip()] # Remove "Assistant:" prefix
+ else:
+ # Continue building current message
+ current_content.append(line)
+
+ # Add the last message
+ if current_role and current_content:
+ messages.append(
+ {"role": current_role, "content": "\n".join(current_content).strip()}
+ )
+
+ # If no role indicators found, treat as a single user message
+ if not messages and prompt_content.strip():
+ messages = [{"role": "user", "content": prompt_content.strip()}] # type: ignore
+
+ return messages # type: ignore
+
+ def post_call_hook(
+ self,
+ user_id: Optional[str],
+ response: Any,
+ input_messages: List[AllMessageValues],
+ function_call: Optional[Union[Dict[str, Any], str]] = None,
+ litellm_params: Optional[Dict[str, Any]] = None,
+ prompt_id: Optional[str] = None,
+ prompt_variables: Optional[Dict[str, Any]] = None,
+ **kwargs,
+ ) -> Any:
+ """
+ Post-call hook for any post-processing after the LLM call.
+ """
+ return response
+
+ def get_available_prompts(self) -> List[str]:
+ """Get list of available prompt IDs."""
+ return self.prompt_manager.list_templates()
+
+ def reload_prompts(self) -> None:
+ """Reload prompts from BitBucket."""
+ if self.prompt_id:
+ self._prompt_manager = None # Reset to force reload
+ self.prompt_manager # This will trigger reload
+
+ def should_run_prompt_management(
+ self,
+ prompt_id: str,
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ ) -> bool:
+ """
+ Determine if prompt management should run based on the prompt_id.
+
+ For BitBucket, we always return True and handle the prompt loading
+ in the _compile_prompt_helper method.
+ """
+ return True
+
+ def _compile_prompt_helper(
+ self,
+ prompt_id: str,
+ prompt_variables: Optional[dict],
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ prompt_label: Optional[str] = None,
+ prompt_version: Optional[int] = None,
+ ) -> PromptManagementClient:
+ """
+ Compile a BitBucket prompt template into a PromptManagementClient structure.
+
+ This method:
+ 1. Loads the prompt template from BitBucket
+ 2. Renders it with the provided variables
+ 3. Converts the rendered text into chat messages
+ 4. Extracts model and optional parameters from metadata
+ """
+ try:
+ # Load the prompt from BitBucket if not already loaded
+ if prompt_id not in self.prompt_manager.prompts:
+ self.prompt_manager._load_prompt_from_bitbucket(prompt_id)
+
+ # Get the rendered prompt and metadata
+ rendered_prompt, prompt_metadata = self.get_prompt_template(
+ prompt_id, prompt_variables
+ )
+
+ # Convert rendered content to chat messages
+ messages = self._parse_prompt_to_messages(rendered_prompt)
+
+ # Extract model from metadata (if specified)
+ template_model = prompt_metadata.get("model")
+
+ # Extract optional parameters from metadata
+ optional_params = {}
+ for param in [
+ "temperature",
+ "max_tokens",
+ "top_p",
+ "frequency_penalty",
+ "presence_penalty",
+ ]:
+ if param in prompt_metadata:
+ optional_params[param] = prompt_metadata[param]
+
+ return PromptManagementClient(
+ prompt_id=prompt_id,
+ prompt_template=messages,
+ prompt_template_model=template_model,
+ prompt_template_optional_params=optional_params,
+ completed_messages=None,
+ )
+
+ except Exception as e:
+ raise ValueError(f"Error compiling prompt '{prompt_id}': {e}")
+
+ def get_chat_completion_prompt(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ non_default_params: dict,
+ prompt_id: Optional[str],
+ prompt_variables: Optional[dict],
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ prompt_label: Optional[str] = None,
+ prompt_version: Optional[int] = None,
+ ) -> Tuple[str, List[AllMessageValues], dict]:
+ """
+ Get chat completion prompt from BitBucket and return processed model, messages, and parameters.
+ """
+ return PromptManagementBase.get_chat_completion_prompt(
+ self,
+ model,
+ messages,
+ non_default_params,
+ prompt_id,
+ prompt_variables,
+ dynamic_callback_params,
+ prompt_label,
+ prompt_version,
+ )
diff --git a/litellm/integrations/braintrust_logging.py b/litellm/integrations/braintrust_logging.py
index c68674f77ba..5bc6afb6dbc 100644
--- a/litellm/integrations/braintrust_logging.py
+++ b/litellm/integrations/braintrust_logging.py
@@ -1,13 +1,11 @@
# What is this?
## Log success + failure events to Braintrust
-import copy
import os
from datetime import datetime
from typing import Dict, Optional
import httpx
-from pydantic import BaseModel
import litellm
from litellm import verbose_logger
@@ -19,16 +17,11 @@ from litellm.llms.custom_httpx.http_handler import (
)
from litellm.utils import print_verbose
-global_braintrust_http_handler = get_async_httpx_client(
- llm_provider=httpxSpecialProvider.LoggingCallback
-)
-global_braintrust_sync_http_handler = HTTPHandler()
API_BASE = "https://api.braintrustdata.com/v1"
def get_utc_datetime():
import datetime as dt
- from datetime import datetime
if hasattr(dt, "UTC"):
return datetime.now(dt.UTC) # type: ignore
@@ -42,16 +35,20 @@ class BraintrustLogger(CustomLogger):
) -> None:
super().__init__()
self.validate_environment(api_key=api_key)
- self.api_base = api_base or API_BASE
+ self.api_base = api_base or os.getenv("BRAINTRUST_API_BASE") or API_BASE
self.default_project_id = None
self.api_key: str = api_key or os.getenv("BRAINTRUST_API_KEY") # type: ignore
self.headers = {
"Authorization": "Bearer " + self.api_key,
"Content-Type": "application/json",
}
- self._project_id_cache: Dict[
- str, str
- ] = {} # Cache mapping project names to IDs
+ self._project_id_cache: Dict[str, str] = (
+ {}
+ ) # Cache mapping project names to IDs
+ self.global_braintrust_http_handler = get_async_httpx_client(
+ llm_provider=httpxSpecialProvider.LoggingCallback
+ )
+ self.global_braintrust_sync_http_handler = HTTPHandler()
def validate_environment(self, api_key: Optional[str]):
"""
@@ -76,7 +73,7 @@ class BraintrustLogger(CustomLogger):
return self._project_id_cache[project_name]
try:
- response = global_braintrust_sync_http_handler.post(
+ response = self.global_braintrust_sync_http_handler.post(
f"{self.api_base}/project",
headers=self.headers,
json={"name": project_name},
@@ -96,7 +93,7 @@ class BraintrustLogger(CustomLogger):
return self._project_id_cache[project_name]
try:
- response = await global_braintrust_http_handler.post(
+ response = await self.global_braintrust_http_handler.post(
f"{self.api_base}/project/register",
headers=self.headers,
json={"name": project_name},
@@ -108,45 +105,8 @@ class BraintrustLogger(CustomLogger):
except httpx.HTTPStatusError as e:
raise Exception(f"Failed to register project: {e.response.text}")
- @staticmethod
- def add_metadata_from_header(litellm_params: dict, metadata: dict) -> dict:
- """
- Adds metadata from proxy request headers to Braintrust logging if keys start with "braintrust_"
- and overwrites litellm_params.metadata if already included.
-
- For example if you want to append your trace to an existing `trace_id` via header, send
- `headers: { ..., langfuse_existing_trace_id: your-existing-trace-id }` via proxy request.
- """
- if litellm_params is None:
- return metadata
-
- if litellm_params.get("proxy_server_request") is None:
- return metadata
-
- if metadata is None:
- metadata = {}
-
- proxy_headers = (
- litellm_params.get("proxy_server_request", {}).get("headers", {}) or {}
- )
-
- for metadata_param_key in proxy_headers:
- if metadata_param_key.startswith("braintrust"):
- trace_param_key = metadata_param_key.replace("braintrust", "", 1)
- if trace_param_key in metadata:
- verbose_logger.warning(
- f"Overwriting Braintrust `{trace_param_key}` from request header"
- )
- else:
- verbose_logger.debug(
- f"Found Braintrust `{trace_param_key}` in request header"
- )
- metadata[trace_param_key] = proxy_headers.get(metadata_param_key)
-
- return metadata
-
async def create_default_project_and_experiment(self):
- project = await global_braintrust_http_handler.post(
+ project = await self.global_braintrust_http_handler.post(
f"{self.api_base}/project", headers=self.headers, json={"name": "litellm"}
)
@@ -155,7 +115,7 @@ class BraintrustLogger(CustomLogger):
self.default_project_id = project_dict["id"]
def create_sync_default_project_and_experiment(self):
- project = global_braintrust_sync_http_handler.post(
+ project = self.global_braintrust_sync_http_handler.post(
f"{self.api_base}/project", headers=self.headers, json={"name": "litellm"}
)
@@ -169,7 +129,9 @@ class BraintrustLogger(CustomLogger):
verbose_logger.debug("REACHES BRAINTRUST SUCCESS")
try:
litellm_call_id = kwargs.get("litellm_call_id")
+ standard_logging_object = kwargs.get("standard_logging_object", {})
prompt = {"messages": kwargs.get("messages")}
+
output = None
choices = []
if response_obj is not None and (
@@ -192,33 +154,13 @@ class BraintrustLogger(CustomLogger):
):
output = response_obj["data"]
- litellm_params = kwargs.get("litellm_params", {})
- metadata = (
- litellm_params.get("metadata", {}) or {}
- ) # if litellm_params['metadata'] == None
- metadata = self.add_metadata_from_header(litellm_params, metadata)
- clean_metadata = {}
- try:
- metadata = copy.deepcopy(
- metadata
- ) # Avoid modifying the original metadata
- except Exception:
- new_metadata = {}
- for key, value in metadata.items():
- if (
- isinstance(value, list)
- or isinstance(value, dict)
- or isinstance(value, str)
- or isinstance(value, int)
- or isinstance(value, float)
- ):
- new_metadata[key] = copy.deepcopy(value)
- metadata = new_metadata
+ litellm_params = kwargs.get("litellm_params", {}) or {}
+ dynamic_metadata = litellm_params.get("metadata", {}) or {}
# Get project_id from metadata or create default if needed
- project_id = metadata.get("project_id")
+ project_id = dynamic_metadata.get("project_id")
if project_id is None:
- project_name = metadata.get("project_name")
+ project_name = dynamic_metadata.get("project_name")
project_id = (
self.get_project_id_sync(project_name) if project_name else None
)
@@ -229,8 +171,9 @@ class BraintrustLogger(CustomLogger):
project_id = self.default_project_id
tags = []
- if isinstance(metadata, dict):
- for key, value in metadata.items():
+
+ if isinstance(dynamic_metadata, dict):
+ for key, value in dynamic_metadata.items():
# generate langfuse tags - Default Tags sent to Langfuse from LiteLLM Proxy
if (
litellm.langfuse_default_tags is not None
@@ -239,25 +182,12 @@ class BraintrustLogger(CustomLogger):
):
tags.append(f"{key}:{value}")
- # clean litellm metadata before logging
- if key in [
- "headers",
- "endpoint",
- "caching_groups",
- "previous_models",
- ]:
- continue
- else:
- clean_metadata[key] = value
+ if (
+ isinstance(value, str) and key not in standard_logging_object
+ ): # support logging dynamic metadata to braintrust
+ standard_logging_object[key] = value
cost = kwargs.get("response_cost", None)
- if cost is not None:
- clean_metadata["litellm_response_cost"] = cost
-
- # metadata.model is required for braintrust to calculate the "Estimated cost" metric
- litellm_model = kwargs.get("model", None)
- if litellm_model is not None:
- clean_metadata["model"] = litellm_model
metrics: Optional[dict] = None
usage_obj = getattr(response_obj, "usage", None)
@@ -274,12 +204,15 @@ class BraintrustLogger(CustomLogger):
"end": end_time.timestamp(),
}
+ # Allow metadata override for span name
+ span_name = dynamic_metadata.get("span_name", "Chat Completion")
+
request_data = {
"id": litellm_call_id,
"input": prompt["messages"],
- "metadata": clean_metadata,
+ "metadata": standard_logging_object,
"tags": tags,
- "span_attributes": {"name": "Chat Completion", "type": "llm"},
+ "span_attributes": {"name": span_name, "type": "llm"},
}
if choices is not None:
request_data["output"] = [choice.dict() for choice in choices]
@@ -291,9 +224,9 @@ class BraintrustLogger(CustomLogger):
try:
print_verbose(
- f"global_braintrust_sync_http_handler.post: {global_braintrust_sync_http_handler.post}"
+ f"self.global_braintrust_sync_http_handler.post: {self.global_braintrust_sync_http_handler.post}"
)
- global_braintrust_sync_http_handler.post(
+ self.global_braintrust_sync_http_handler.post(
url=f"{self.api_base}/project_logs/{project_id}/insert",
json={"events": [request_data]},
headers=self.headers,
@@ -309,6 +242,7 @@ class BraintrustLogger(CustomLogger):
verbose_logger.debug("REACHES BRAINTRUST SUCCESS")
try:
litellm_call_id = kwargs.get("litellm_call_id")
+ standard_logging_object = kwargs.get("standard_logging_object", {})
prompt = {"messages": kwargs.get("messages")}
output = None
choices = []
@@ -333,32 +267,12 @@ class BraintrustLogger(CustomLogger):
output = response_obj["data"]
litellm_params = kwargs.get("litellm_params", {})
- metadata = (
- litellm_params.get("metadata", {}) or {}
- ) # if litellm_params['metadata'] == None
- metadata = self.add_metadata_from_header(litellm_params, metadata)
- clean_metadata = {}
- new_metadata = {}
- for key, value in metadata.items():
- if (
- isinstance(value, list)
- or isinstance(value, str)
- or isinstance(value, int)
- or isinstance(value, float)
- ):
- new_metadata[key] = value
- elif isinstance(value, BaseModel):
- new_metadata[key] = value.model_dump_json()
- elif isinstance(value, dict):
- for k, v in value.items():
- if isinstance(v, datetime):
- value[k] = v.isoformat()
- new_metadata[key] = value
+ dynamic_metadata = litellm_params.get("metadata", {}) or {}
# Get project_id from metadata or create default if needed
- project_id = metadata.get("project_id")
+ project_id = dynamic_metadata.get("project_id")
if project_id is None:
- project_name = metadata.get("project_name")
+ project_name = dynamic_metadata.get("project_name")
project_id = (
await self.get_project_id_async(project_name)
if project_name
@@ -371,8 +285,9 @@ class BraintrustLogger(CustomLogger):
project_id = self.default_project_id
tags = []
- if isinstance(metadata, dict):
- for key, value in metadata.items():
+
+ if isinstance(dynamic_metadata, dict):
+ for key, value in dynamic_metadata.items():
# generate langfuse tags - Default Tags sent to Langfuse from LiteLLM Proxy
if (
litellm.langfuse_default_tags is not None
@@ -381,25 +296,12 @@ class BraintrustLogger(CustomLogger):
):
tags.append(f"{key}:{value}")
- # clean litellm metadata before logging
- if key in [
- "headers",
- "endpoint",
- "caching_groups",
- "previous_models",
- ]:
- continue
- else:
- clean_metadata[key] = value
+ if (
+ isinstance(value, str) and key not in standard_logging_object
+ ): # support logging dynamic metadata to braintrust
+ standard_logging_object[key] = value
cost = kwargs.get("response_cost", None)
- if cost is not None:
- clean_metadata["litellm_response_cost"] = cost
-
- # metadata.model is required for braintrust to calculate the "Estimated cost" metric
- litellm_model = kwargs.get("model", None)
- if litellm_model is not None:
- clean_metadata["model"] = litellm_model
metrics: Optional[dict] = None
usage_obj = getattr(response_obj, "usage", None)
@@ -426,13 +328,16 @@ class BraintrustLogger(CustomLogger):
- api_call_start_time.timestamp()
)
+ # Allow metadata override for span name
+ span_name = dynamic_metadata.get("span_name", "Chat Completion")
+
request_data = {
"id": litellm_call_id,
"input": prompt["messages"],
"output": output,
- "metadata": clean_metadata,
+ "metadata": standard_logging_object,
"tags": tags,
- "span_attributes": {"name": "Chat Completion", "type": "llm"},
+ "span_attributes": {"name": span_name, "type": "llm"},
}
if choices is not None:
request_data["output"] = [choice.dict() for choice in choices]
@@ -446,7 +351,7 @@ class BraintrustLogger(CustomLogger):
request_data["metrics"] = metrics
try:
- await global_braintrust_http_handler.post(
+ await self.global_braintrust_http_handler.post(
url=f"{self.api_base}/project_logs/{project_id}/insert",
json={"events": [request_data]},
headers=self.headers,
diff --git a/litellm/integrations/cloudzero/cloudzero.py b/litellm/integrations/cloudzero/cloudzero.py
index 85aa1679732..ca15962b72a 100644
--- a/litellm/integrations/cloudzero/cloudzero.py
+++ b/litellm/integrations/cloudzero/cloudzero.py
@@ -1,14 +1,15 @@
-import asyncio
import os
-from datetime import datetime, timedelta
-from typing import Optional
+from datetime import datetime
+from typing import TYPE_CHECKING, Any, List, Optional, cast
+import litellm
from litellm._logging import verbose_logger
from litellm.integrations.custom_logger import CustomLogger
-from .cz_stream_api import CloudZeroStreamer
-from .database import LiteLLMDatabase
-from .transform import CBFTransformer
+if TYPE_CHECKING:
+ from apscheduler.schedulers.asyncio import AsyncIOScheduler
+else:
+ AsyncIOScheduler = Any
class CloudZeroLogger(CustomLogger):
@@ -29,20 +30,80 @@ class CloudZeroLogger(CustomLogger):
self.api_key = api_key or os.getenv("CLOUDZERO_API_KEY")
self.connection_id = connection_id or os.getenv("CLOUDZERO_CONNECTION_ID")
self.timezone = timezone or os.getenv("CLOUDZERO_TIMEZONE", "UTC")
+ verbose_logger.debug(f"CloudZero Logger initialized with connection ID: {self.connection_id}, timezone: {self.timezone}")
- async def export_usage_data(self, target_hour: datetime, limit: Optional[int] = 1000, operation: str = "replace_hourly"):
+ async def initialize_cloudzero_export_job(self):
"""
- Exports the usage data for a specific hour to CloudZero.
+ Handler for initializing CloudZero export job.
- - Reads spend logs from the DB for the specified hour
+ Runs when CloudZero logger starts up.
+
+ - If redis cache is available, we use the pod lock manager to acquire a lock and export the data.
+ - Ensures only one pod exports the data at a time.
+ - If redis cache is not available, we export the data directly.
+ """
+ from litellm.constants import (
+ CLOUDZERO_EXPORT_USAGE_DATA_JOB_NAME,
+ )
+ from litellm.proxy.proxy_server import proxy_logging_obj
+ pod_lock_manager = proxy_logging_obj.db_spend_update_writer.pod_lock_manager
+
+ # if using redis, ensure only one pod exports the data at a time
+ if pod_lock_manager and pod_lock_manager.redis_cache:
+ if await pod_lock_manager.acquire_lock(
+ cronjob_id=CLOUDZERO_EXPORT_USAGE_DATA_JOB_NAME
+ ):
+ try:
+ await self._hourly_usage_data_export()
+ finally:
+ await pod_lock_manager.release_lock(
+ cronjob_id=CLOUDZERO_EXPORT_USAGE_DATA_JOB_NAME
+ )
+ else:
+ # if not using redis, export the data directly
+ await self._hourly_usage_data_export()
+
+ async def _hourly_usage_data_export(self):
+ """
+ Exports the hourly usage data to CloudZero.
+
+ Start time: 1 hour ago
+ End time: current time
+ """
+ from datetime import timedelta, timezone
+
+ from litellm.constants import CLOUDZERO_MAX_FETCHED_DATA_RECORDS
+ current_time_utc = datetime.now(timezone.utc)
+ one_hour_ago_utc = current_time_utc - timedelta(hours=1)
+ await self.export_usage_data(
+ limit=CLOUDZERO_MAX_FETCHED_DATA_RECORDS,
+ operation="replace_hourly",
+ start_time_utc=one_hour_ago_utc,
+ end_time_utc=current_time_utc
+ )
+
+
+ async def export_usage_data(
+ self,
+ limit: Optional[int] = None,
+ operation: str = "replace_hourly",
+ start_time_utc: Optional[datetime] = None,
+ end_time_utc: Optional[datetime] = None
+ ):
+ """
+ Exports the usage data to CloudZero.
+
+ - Reads data from the DB
- Transforms the data to the CloudZero format
- Sends the data to CloudZero
Args:
- target_hour: The specific hour to export data for
- limit: Optional limit on number of records to export (default: 1000)
+ limit: Optional limit on number of records to export
operation: CloudZero operation type ("replace_hourly" or "sum")
"""
+ from litellm.integrations.cloudzero.cz_stream_api import CloudZeroStreamer
+ from litellm.integrations.cloudzero.database import LiteLLMDatabase
+ from litellm.integrations.cloudzero.transform import CBFTransformer
try:
verbose_logger.debug("CloudZero Logger: Starting usage data export")
@@ -52,11 +113,27 @@ class CloudZeroLogger(CustomLogger):
"CloudZero configuration missing. Please set CLOUDZERO_API_KEY and CLOUDZERO_CONNECTION_ID environment variables."
)
- # Fetch and transform data using helper
- cbf_data = await self._fetch_cbf_data_for_hour(target_hour, limit)
+ # Initialize database connection and load data
+ database = LiteLLMDatabase()
+ verbose_logger.debug("CloudZero Logger: Loading usage data from database")
+ data = await database.get_usage_data(
+ limit=limit,
+ start_time_utc=start_time_utc,
+ end_time_utc=end_time_utc
+ )
+
+ if data.is_empty():
+ verbose_logger.debug("CloudZero Logger: No usage data found to export")
+ return
+
+ verbose_logger.debug(f"CloudZero Logger: Processing {len(data)} records")
+
+ # Transform data to CloudZero CBF format
+ transformer = CBFTransformer()
+ cbf_data = transformer.transform(data)
if cbf_data.is_empty():
- verbose_logger.info("CloudZero Logger: No usage data found to export")
+ verbose_logger.warning("CloudZero Logger: No valid data after transformation")
return
# Send data to CloudZero
@@ -69,65 +146,91 @@ class CloudZeroLogger(CustomLogger):
verbose_logger.debug(f"CloudZero Logger: Transmitting {len(cbf_data)} records to CloudZero")
streamer.send_batched(cbf_data, operation=operation)
- verbose_logger.info(f"CloudZero Logger: Successfully exported {len(cbf_data)} records to CloudZero")
+ verbose_logger.debug(f"CloudZero Logger: Successfully exported {len(cbf_data)} records to CloudZero")
except Exception as e:
verbose_logger.error(f"CloudZero Logger: Error exporting usage data: {str(e)}")
raise
- async def _fetch_cbf_data_for_hour(self, target_hour: datetime, limit: Optional[int] = 1000):
+ async def dry_run_export_usage_data(self, limit: Optional[int] = 10000):
"""
- Helper method to fetch usage data for a specific hour and transform it to CloudZero CBF format.
+ Returns the data that would be exported to CloudZero without actually sending it.
Args:
- target_hour: The specific hour to fetch data for
- limit: Optional limit on number of records to fetch (default: 1000)
+ limit: Limit number of records to display (default: 10000)
Returns:
- CBF formatted data ready for CloudZero ingestion
- """
- # Initialize database connection and load data
- database = LiteLLMDatabase()
- verbose_logger.debug(f"CloudZero Logger: Loading spend logs for hour {target_hour}")
- data = await database.get_usage_data_for_hour(target_hour=target_hour, limit=limit)
-
- if data.is_empty():
- verbose_logger.info("CloudZero Logger: No usage data found for the specified hour")
- return data # Return empty data
-
- verbose_logger.debug(f"CloudZero Logger: Processing {len(data)} records")
-
- # Transform data to CloudZero CBF format
- transformer = CBFTransformer()
- cbf_data = transformer.transform(data)
-
- if cbf_data.is_empty():
- verbose_logger.warning("CloudZero Logger: No valid data after transformation")
-
- return cbf_data
-
- async def dry_run_export_usage_data(self, target_hour: datetime, limit: Optional[int] = 1000):
- """
- Only prints the spend logs data for a specific hour that would be exported to CloudZero.
-
- Args:
- target_hour: The specific hour to export data for
- limit: Limit number of records to display (default: 1000)
+ dict: Contains usage_data, cbf_data, and summary statistics
"""
+ from litellm.integrations.cloudzero.database import LiteLLMDatabase
+ from litellm.integrations.cloudzero.transform import CBFTransformer
try:
verbose_logger.debug("CloudZero Logger: Starting dry run export")
- # Fetch and transform data using helper
- cbf_data = await self._fetch_cbf_data_for_hour(target_hour, limit)
+ # Initialize database connection and load data
+ database = LiteLLMDatabase()
+ verbose_logger.debug("CloudZero Logger: Loading usage data for dry run")
+ data = await database.get_usage_data(limit=limit)
+
+ if data.is_empty():
+ verbose_logger.warning("CloudZero Dry Run: No usage data found")
+ return {
+ "usage_data": [],
+ "cbf_data": [],
+ "summary": {
+ "total_records": 0,
+ "total_cost": 0,
+ "total_tokens": 0,
+ "unique_accounts": 0,
+ "unique_services": 0
+ }
+ }
+
+ verbose_logger.debug(f"CloudZero Dry Run: Processing {len(data)} records...")
+
+ # Convert usage data to dict format for response
+ usage_data_sample = data.head(50).to_dicts() # Return first 50 rows
+
+ # Transform data to CloudZero CBF format
+ transformer = CBFTransformer()
+ cbf_data = transformer.transform(data)
if cbf_data.is_empty():
- verbose_logger.warning("CloudZero Dry Run: No usage data found")
- return
+ verbose_logger.warning("CloudZero Dry Run: No valid data after transformation")
+ return {
+ "usage_data": usage_data_sample,
+ "cbf_data": [],
+ "summary": {
+ "total_records": len(usage_data_sample),
+ "total_cost": sum(row.get('spend', 0) for row in usage_data_sample),
+ "total_tokens": sum(row.get('prompt_tokens', 0) + row.get('completion_tokens', 0) for row in usage_data_sample),
+ "unique_accounts": 0,
+ "unique_services": 0
+ }
+ }
- # Display the transformed data on screen
- self._display_cbf_data_on_screen(cbf_data)
+ # Convert CBF data to dict format for response
+ cbf_data_dict = cbf_data.to_dicts()
- verbose_logger.info(f"CloudZero Logger: Dry run completed for {len(cbf_data)} records")
+ # Calculate summary statistics
+ total_cost = sum(record.get('cost/cost', 0) for record in cbf_data_dict)
+ unique_accounts = len(set(record.get('resource/account', '') for record in cbf_data_dict if record.get('resource/account')))
+ unique_services = len(set(record.get('resource/service', '') for record in cbf_data_dict if record.get('resource/service')))
+ total_tokens = sum(record.get('usage/amount', 0) for record in cbf_data_dict)
+
+ verbose_logger.debug(f"CloudZero Logger: Dry run completed for {len(cbf_data)} records")
+
+ return {
+ "usage_data": usage_data_sample,
+ "cbf_data": cbf_data_dict,
+ "summary": {
+ "total_records": len(cbf_data_dict),
+ "total_cost": total_cost,
+ "total_tokens": total_tokens,
+ "unique_accounts": unique_accounts,
+ "unique_services": unique_services
+ }
+ }
except Exception as e:
verbose_logger.error(f"CloudZero Logger: Error in dry run export: {str(e)}")
@@ -155,6 +258,11 @@ class CloudZeroLogger(CustomLogger):
cbf_table = Table(show_header=True, header_style="bold cyan", box=SIMPLE, padding=(0, 1))
cbf_table.add_column("time/usage_start", style="blue", no_wrap=False)
cbf_table.add_column("cost/cost", style="green", justify="right", no_wrap=False)
+ cbf_table.add_column("entity_type", style="magenta", justify="right", no_wrap=False)
+ cbf_table.add_column("entity_id", style="magenta", justify="right", no_wrap=False)
+ cbf_table.add_column("team_id", style="cyan", no_wrap=False)
+ cbf_table.add_column("team_alias", style="cyan", no_wrap=False)
+ cbf_table.add_column("api_key_alias", style="yellow", no_wrap=False)
cbf_table.add_column("usage/amount", style="yellow", justify="right", no_wrap=False)
cbf_table.add_column("resource/id", style="magenta", no_wrap=False)
cbf_table.add_column("resource/service", style="cyan", no_wrap=False)
@@ -170,10 +278,20 @@ class CloudZeroLogger(CustomLogger):
resource_service = str(record.get('resource/service', 'N/A'))
resource_account = str(record.get('resource/account', 'N/A'))
resource_region = str(record.get('resource/region', 'N/A'))
+ entity_type = str(record.get('entity_type', 'N/A'))
+ entity_id = str(record.get('entity_id', 'N/A'))
+ team_id = str(record.get('resource/tag:team_id', 'N/A'))
+ team_alias = str(record.get('resource/tag:team_alias', 'N/A'))
+ api_key_alias = str(record.get('resource/tag:api_key_alias', 'N/A'))
cbf_table.add_row(
time_usage_start,
cost_cost,
+ entity_type,
+ entity_id,
+ team_id,
+ team_alias,
+ api_key_alias,
usage_amount,
resource_id,
resource_service,
@@ -199,55 +317,33 @@ class CloudZeroLogger(CustomLogger):
console.print(f" Unique Services: {unique_services}")
console.print("\n[dim]💡 This is the CloudZero CBF format ready for AnyCost ingestion[/dim]")
+
+ @staticmethod
+ async def init_cloudzero_background_job(scheduler: AsyncIOScheduler):
+ """
+ Initialize the CloudZero background job.
- async def init_background_job(self, redis_cache=None):
+ Starts the background job that exports the usage data to CloudZero every hour.
"""
- Initialize a background job that exports usage data every hour.
- Uses PodLockManager to ensure only one instance runs the export at a time.
+ from litellm.constants import CLOUDZERO_EXPORT_INTERVAL_MINUTES
+ from litellm.integrations.custom_logger import CustomLogger
- Args:
- redis_cache: Redis cache instance for pod locking
- """
- from litellm.proxy.db.db_transaction_queue.pod_lock_manager import (
- PodLockManager,
+
+ prometheus_loggers: List[CustomLogger] = (
+ litellm.logging_callback_manager.get_custom_loggers_for_type(
+ callback_type=CloudZeroLogger
+ )
)
-
- lock_manager = PodLockManager(redis_cache=redis_cache)
- cronjob_id = "cloudzero_hourly_export"
-
- async def hourly_export_task():
- while True:
- try:
- # Calculate the previous completed hour
- now = datetime.utcnow()
- target_hour = now.replace(minute=0, second=0, microsecond=0)
- # Export data for the previous hour to ensure all data is available
- target_hour = target_hour - timedelta(hours=1)
-
- # Try to acquire lock
- lock_acquired = await lock_manager.acquire_lock(cronjob_id)
-
- if lock_acquired:
- try:
- verbose_logger.info(f"CloudZero Background Job: Starting export for hour {target_hour}")
- await self.export_usage_data(target_hour)
- verbose_logger.info(f"CloudZero Background Job: Completed export for hour {target_hour}")
- finally:
- # Always release the lock
- await lock_manager.release_lock(cronjob_id)
- else:
- verbose_logger.debug("CloudZero Background Job: Another instance is already running the export")
-
- # Wait until the next hour
- next_hour = (datetime.utcnow() + timedelta(hours=1)).replace(minute=0, second=0, microsecond=0)
- sleep_seconds = (next_hour - datetime.utcnow()).total_seconds()
- await asyncio.sleep(sleep_seconds)
-
- except Exception as e:
- verbose_logger.error(f"CloudZero Background Job: Error in hourly export task: {str(e)}")
- # Sleep for 5 minutes before retrying on error
- await asyncio.sleep(300)
-
- # Start the background task
- asyncio.create_task(hourly_export_task())
- verbose_logger.debug("CloudZero Background Job: Initialized hourly export task")
\ No newline at end of file
+ # we need to get the initialized prometheus logger instance(s) and call logger.initialize_remaining_budget_metrics() on them
+ verbose_logger.debug("found %s cloudzero loggers", len(prometheus_loggers))
+ if len(prometheus_loggers) > 0:
+ cloudzero_logger = cast(CloudZeroLogger, prometheus_loggers[0])
+ verbose_logger.debug(
+ "Initializing remaining budget metrics as a cron job executing every %s minutes"
+ % CLOUDZERO_EXPORT_INTERVAL_MINUTES
+ )
+ scheduler.add_job(
+ cloudzero_logger.initialize_cloudzero_export_job,
+ "interval",
+ minutes=CLOUDZERO_EXPORT_INTERVAL_MINUTES
+ )
\ No newline at end of file
diff --git a/litellm/integrations/cloudzero/cz_resource_names.py b/litellm/integrations/cloudzero/cz_resource_names.py
index 44147f9c210..f1098d20381 100644
--- a/litellm/integrations/cloudzero/cz_resource_names.py
+++ b/litellm/integrations/cloudzero/cz_resource_names.py
@@ -17,11 +17,16 @@
"""CloudZero Resource Names (CZRN) generation and validation for LiteLLM resources."""
import re
+from enum import Enum
from typing import Any, cast
import litellm
+class CZEntityType(str, Enum):
+ TEAM = "team"
+
+
class CZRNGenerator:
"""Generate CloudZero Resource Names (CZRNs) for LiteLLM resources."""
@@ -49,8 +54,8 @@ class CZRNGenerator:
region = 'cross-region'
# Use the actual entity_id (team_id or user_id) as the owner account
- entity_id = row.get('entity_id', 'unknown')
- owner_account_id = self._normalize_component(entity_id)
+ team_id = row.get('team_id', 'unknown')
+ owner_account_id = self._normalize_component(team_id)
resource_type = 'llm-usage'
diff --git a/litellm/integrations/cloudzero/database.py b/litellm/integrations/cloudzero/database.py
index 6d12c5cfbd9..71b4125ed75 100644
--- a/litellm/integrations/cloudzero/database.py
+++ b/litellm/integrations/cloudzero/database.py
@@ -12,14 +12,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-# CHANGELOG: 2025-07-23 - Added support for using LiteLLM_SpendLogs table for CBF mapping (ishaan-jaff)
# CHANGELOG: 2025-01-19 - Refactored to use daily spend tables for proper CBF mapping (erik.peterson)
# CHANGELOG: 2025-01-19 - Migrated from pandas to polars for database operations (erik.peterson)
# CHANGELOG: 2025-01-19 - Initial database module for LiteLLM data extraction (erik.peterson)
"""Database connection and data extraction for LiteLLM."""
-from datetime import datetime, timedelta
+from datetime import datetime
from typing import Any, Dict, Optional
import polars as pl
@@ -37,61 +36,88 @@ class LiteLLMDatabase:
)
return prisma_client
- async def get_usage_data_for_hour(self, target_hour: datetime, limit: Optional[int] = 1000) -> pl.DataFrame:
- """Retrieve spend logs for a specific hour from LiteLLM_SpendLogs table with batching."""
+ async def get_usage_data(
+ self,
+ limit: Optional[int] = None,
+ start_time_utc: Optional[datetime] = None,
+ end_time_utc: Optional[datetime] = None
+ ) -> pl.DataFrame:
+ """Retrieve usage data from LiteLLM daily user spend table."""
client = self._ensure_prisma_client()
- # Calculate hour range
- hour_start = target_hour.replace(minute=0, second=0, microsecond=0)
- hour_end = hour_start + timedelta(hours=1)
+ # Build WHERE clause for time filtering
+ where_conditions = []
+ if start_time_utc:
+ where_conditions.append(f"dus.created_at >= '{start_time_utc.isoformat()}'")
+ if end_time_utc:
+ where_conditions.append(f"dus.created_at <= '{end_time_utc.isoformat()}'")
- # Convert datetime objects to ISO format strings for PostgreSQL compatibility
- hour_start_str = hour_start.isoformat()
- hour_end_str = hour_end.isoformat()
+ where_clause = ""
+ if where_conditions:
+ where_clause = "WHERE " + " AND ".join(where_conditions)
- # Query to get spend logs for the specific hour
- query = """
- SELECT *
- FROM "LiteLLM_SpendLogs"
- WHERE "startTime" >= $1::timestamp
- AND "startTime" < $2::timestamp
- ORDER BY "startTime" ASC
+ # Query to get user spend data with team information
+ query = f"""
+ SELECT
+ dus.id,
+ dus.date,
+ dus.user_id,
+ dus.api_key,
+ dus.model,
+ dus.model_group,
+ dus.custom_llm_provider,
+ dus.prompt_tokens,
+ dus.completion_tokens,
+ dus.spend,
+ dus.api_requests,
+ dus.successful_requests,
+ dus.failed_requests,
+ dus.cache_creation_input_tokens,
+ dus.cache_read_input_tokens,
+ dus.created_at,
+ dus.updated_at,
+ vt.team_id,
+ vt.key_alias as api_key_alias,
+ tt.team_alias
+ FROM "LiteLLM_DailyUserSpend" dus
+ LEFT JOIN "LiteLLM_VerificationToken" vt ON dus.api_key = vt.token
+ LEFT JOIN "LiteLLM_TeamTable" tt ON vt.team_id = tt.team_id
+ {where_clause}
+ ORDER BY dus.date DESC, dus.created_at DESC
"""
if limit:
query += f" LIMIT {limit}"
try:
- db_response = await client.db.query_raw(query, hour_start_str, hour_end_str)
- # Convert the response to polars DataFrame
- return pl.DataFrame(db_response) if db_response else pl.DataFrame()
+ db_response = await client.db.query_raw(query)
+ # Convert the response to polars DataFrame with full schema inference
+ # This prevents schema mismatch errors when data types vary across rows
+ return pl.DataFrame(db_response, infer_schema_length=None)
except Exception as e:
- raise Exception(f"Error retrieving spend logs for hour {target_hour}: {str(e)}")
-
+ raise Exception(f"Error retrieving usage data: {str(e)}")
async def get_table_info(self) -> Dict[str, Any]:
- """Get information about the LiteLLM_SpendLogs table."""
+ """Get information about the daily user spend table."""
client = self._ensure_prisma_client()
try:
- # Get row count from SpendLogs table
- spend_logs_count = await self._get_table_row_count('LiteLLM_SpendLogs')
+ # Get row count from user spend table
+ user_count = await self._get_table_row_count('LiteLLM_DailyUserSpend')
- # Get column structure from spend logs table
+ # Get column structure from user spend table
query = """
SELECT column_name, data_type, is_nullable
FROM information_schema.columns
- WHERE table_name = 'LiteLLM_SpendLogs'
+ WHERE table_name = 'LiteLLM_DailyUserSpend'
ORDER BY ordinal_position;
"""
columns_response = await client.db.query_raw(query)
return {
'columns': columns_response,
- 'row_count': spend_logs_count,
- 'table_breakdown': {
- 'spend_logs': spend_logs_count
- }
+ 'row_count': user_count,
+ 'table_name': 'LiteLLM_DailyUserSpend'
}
except Exception as e:
raise Exception(f"Error getting table info: {str(e)}")
diff --git a/litellm/integrations/cloudzero/transform.py b/litellm/integrations/cloudzero/transform.py
index 7091ea26b95..e0263295388 100644
--- a/litellm/integrations/cloudzero/transform.py
+++ b/litellm/integrations/cloudzero/transform.py
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-# CHANGELOG: 2025-01-19 - Updated CBF transformation for LiteLLM_SpendLogs with hourly aggregation and team_id focus (ishaan-jaff)
+# CHANGELOG: 2025-01-19 - Updated CBF transformation for daily spend tables and proper CloudZero mapping (erik.peterson)
# CHANGELOG: 2025-01-19 - Migrated from pandas to polars for data transformation (erik.peterson)
# CHANGELOG: 2025-01-19 - Initial CBF transformation module (erik.peterson)
@@ -24,7 +24,7 @@ from typing import Any, Optional
import polars as pl
from ...types.integrations.cloudzero import CBFRecord
-from .cz_resource_names import CZRNGenerator
+from .cz_resource_names import CZEntityType, CZRNGenerator
class CBFTransformer:
@@ -35,160 +35,99 @@ class CBFTransformer:
self.czrn_generator = CZRNGenerator()
def transform(self, data: pl.DataFrame) -> pl.DataFrame:
- """Transform LiteLLM SpendLogs data to hourly aggregated CBF format."""
+ """Transform LiteLLM data to CBF format, dropping records with zero successful_requests or invalid CZRNs."""
if data.is_empty():
return pl.DataFrame()
- # Filter out records with zero spend or invalid team_id
+ # Filter out records with zero successful_requests first
original_count = len(data)
- filtered_data = data.filter(
- (pl.col('spend') > 0) &
- (pl.col('team_id').is_not_null()) &
- (pl.col('team_id') != "")
- )
- filtered_count = len(filtered_data)
- zero_spend_dropped = original_count - filtered_count
+ if 'successful_requests' in data.columns:
+ filtered_data = data.filter(pl.col('successful_requests') > 0)
+ zero_requests_dropped = original_count - len(filtered_data)
+ else:
+ filtered_data = data
+ zero_requests_dropped = 0
- if filtered_data.is_empty():
- from rich.console import Console
- console = Console()
- console.print(f"[yellow]⚠️ Dropped all {original_count:,} records due to zero spend or missing team_id[/yellow]")
- return pl.DataFrame()
-
- # Aggregate data to hourly level
- hourly_aggregated = self._aggregate_to_hourly(filtered_data)
-
- # Transform aggregated data to CBF format
cbf_data = []
czrn_dropped_count = 0
-
- for row in hourly_aggregated.iter_rows(named=True):
+ filtered_count = len(filtered_data)
+
+ for row in filtered_data.iter_rows(named=True):
try:
cbf_record = self._create_cbf_record(row)
+ # Only include the record if CZRN generation was successful
cbf_data.append(cbf_record)
except Exception:
# Skip records that fail CZRN generation
czrn_dropped_count += 1
continue
- # Print summary of transformations
+ # Print summary of dropped records if any
from rich.console import Console
console = Console()
- if zero_spend_dropped > 0:
- console.print(f"[yellow]⚠️ Dropped {zero_spend_dropped:,} of {original_count:,} records with zero spend or missing team_id[/yellow]")
+ if zero_requests_dropped > 0:
+ console.print(f"[yellow]⚠️ Dropped {zero_requests_dropped:,} of {original_count:,} records with zero successful_requests[/yellow]")
if czrn_dropped_count > 0:
- console.print(f"[yellow]⚠️ Dropped {czrn_dropped_count:,} of {len(hourly_aggregated):,} aggregated records due to invalid CZRNs[/yellow]")
+ console.print(f"[yellow]⚠️ Dropped {czrn_dropped_count:,} of {filtered_count:,} filtered records due to invalid CZRNs[/yellow]")
if len(cbf_data) > 0:
- console.print(f"[green]✓ Successfully transformed {len(cbf_data):,} hourly aggregated records[/green]")
+ console.print(f"[green]✓ Successfully transformed {len(cbf_data):,} records[/green]")
return pl.DataFrame(cbf_data)
- def _aggregate_to_hourly(self, data: pl.DataFrame) -> pl.DataFrame:
- """Aggregate spend logs to hourly level by team_id, key_name, model, and tags."""
-
- # Extract hour from startTime, skip tags and metadata for now
- data_with_hour = data.with_columns([
- pl.col('startTime').str.to_datetime().dt.truncate('1h').alias('usage_hour'),
- pl.lit([]).cast(pl.List(pl.String)).alias('parsed_tags'), # Empty tags list for now
- pl.lit("").alias('key_name') # Empty key name for now
- ])
-
- # Skip tag explosion for now - just add a null tag column
- all_data = data_with_hour.with_columns([
- pl.lit(None, dtype=pl.String).alias('tag')
- ])
-
- # Group by hour, team_id, key_name, model, provider, and tag
- aggregated = all_data.group_by([
- 'usage_hour',
- 'team_id',
- 'key_name',
- 'model',
- 'model_group',
- 'custom_llm_provider',
- 'tag'
- ]).agg([
- pl.col('spend').sum().alias('total_spend'),
- pl.col('total_tokens').sum().alias('total_tokens'),
- pl.col('prompt_tokens').sum().alias('total_prompt_tokens'),
- pl.col('completion_tokens').sum().alias('total_completion_tokens'),
- pl.col('request_id').count().alias('request_count'),
- pl.col('api_key').first().alias('api_key_sample'), # Keep one for reference
- pl.col('status').filter(pl.col('status') == 'success').count().alias('successful_requests'),
- pl.col('status').filter(pl.col('status') != 'success').count().alias('failed_requests')
- ])
- return aggregated
-
-
def _create_cbf_record(self, row: dict[str, Any]) -> CBFRecord:
- """Create a single CBF record from aggregated hourly spend data."""
+ """Create a single CBF record from LiteLLM daily spend row."""
- # Helper function to extract scalar values from polars data
- def extract_scalar(value):
- if hasattr(value, 'item') and not isinstance(value, (str, int, float, bool)):
- return value.item() if value is not None else None
- return value
+ # Parse date (daily spend tables use date strings like '2025-04-19')
+ usage_date = self._parse_date(row.get('date'))
- # Use the aggregated hour as usage time
- usage_time = self._parse_datetime(extract_scalar(row.get('usage_hour')))
-
- # Use team_id as the primary entity_id
- entity_id = str(extract_scalar(row.get('team_id', '')))
- key_name = str(extract_scalar(row.get('key_name', '')))
- model = str(extract_scalar(row.get('model', '')))
- model_group = str(extract_scalar(row.get('model_group', '')))
- provider = str(extract_scalar(row.get('custom_llm_provider', '')))
- tag = extract_scalar(row.get('tag'))
-
- # Calculate aggregated metrics
- total_spend = float(extract_scalar(row.get('total_spend', 0.0)) or 0.0)
- total_tokens = int(extract_scalar(row.get('total_tokens', 0)) or 0)
- total_prompt_tokens = int(extract_scalar(row.get('total_prompt_tokens', 0)) or 0)
- total_completion_tokens = int(extract_scalar(row.get('total_completion_tokens', 0)) or 0)
- request_count = int(extract_scalar(row.get('request_count', 0)) or 0)
- successful_requests = int(extract_scalar(row.get('successful_requests', 0)) or 0)
- failed_requests = int(extract_scalar(row.get('failed_requests', 0)) or 0)
+ # Calculate total tokens
+ prompt_tokens = int(row.get('prompt_tokens', 0))
+ completion_tokens = int(row.get('completion_tokens', 0))
+ total_tokens = prompt_tokens + completion_tokens
# Create CloudZero Resource Name (CZRN) as resource_id
- # Create a mock row for CZRN generation with team_id as entity_id
- czrn_row = {
- 'entity_id': entity_id,
- 'entity_type': 'team',
- 'model': model,
- 'custom_llm_provider': provider,
- 'api_key': str(extract_scalar(row.get('api_key_sample', '')))
- }
- resource_id = self.czrn_generator.create_from_litellm_data(czrn_row)
+ resource_id = self.czrn_generator.create_from_litellm_data(row)
- # Build dimensions for CloudZero tracking
- dimensions = {
- 'entity_type': 'team',
- 'entity_id': entity_id,
- 'key_name': key_name,
- 'model': model,
- 'model_group': model_group,
- 'provider': provider,
- 'request_count': str(request_count),
- 'successful_requests': str(successful_requests),
- 'failed_requests': str(failed_requests),
- }
+ # Build dimensions for CloudZero
+ model = str(row.get('model', ''))
+ api_key_hash = str(row.get('api_key', ''))[:8] # First 8 chars for identification
- # Add tag if present
- if tag is not None and str(tag) not in ['', 'null', 'None']:
- dimensions['tag'] = str(tag)
+ # Handle team information with fallbacks
+ team_id = row.get('team_id')
+ team_alias = row.get('team_alias')
+
+ # Use team_alias if available, otherwise team_id, otherwise fallback to 'unknown'
+ entity_id = str(team_alias) if team_alias else (str(team_id) if team_id else 'unknown')
+
+ dimensions = {
+ 'entity_type': CZEntityType.TEAM.value,
+ 'entity_id': entity_id,
+ 'team_id': str(team_id) if team_id else 'unknown',
+ 'team_alias': str(team_alias) if team_alias else 'unknown',
+ 'model': model,
+ 'model_group': str(row.get('model_group', '')),
+ 'provider': str(row.get('custom_llm_provider', '')),
+ 'api_key_prefix': api_key_hash,
+ 'api_key_alias': str(row.get('api_key_alias', '')),
+ 'api_requests': str(row.get('api_requests', 0)),
+ 'successful_requests': str(row.get('successful_requests', 0)),
+ 'failed_requests': str(row.get('failed_requests', 0)),
+ 'cache_creation_tokens': str(row.get('cache_creation_input_tokens', 0)),
+ 'cache_read_tokens': str(row.get('cache_read_input_tokens', 0)),
+ }
# Extract CZRN components to populate corresponding CBF columns
czrn_components = self.czrn_generator.extract_components(resource_id)
- service_type, provider_czrn, region, owner_account_id, resource_type, cloud_local_id = czrn_components
+ service_type, provider, region, owner_account_id, resource_type, cloud_local_id = czrn_components
# CloudZero CBF format with proper column names
cbf_record = {
# Required CBF fields
- 'time/usage_start': usage_time.isoformat() if usage_time else None, # Required: ISO-formatted UTC datetime
- 'cost/cost': total_spend, # Required: billed cost
+ 'time/usage_start': usage_date.isoformat() if usage_date else None, # Required: ISO-formatted UTC datetime
+ 'cost/cost': float(row.get('spend', 0.0)), # Required: billed cost
'resource/id': resource_id, # Required when resource tags are present
# Usage metrics for token consumption
@@ -206,41 +145,42 @@ class CBFTransformer:
}
# Add CZRN components that don't have direct CBF column mappings as resource tags
- cbf_record['resource/tag:provider'] = provider_czrn # CZRN provider component
+ cbf_record['resource/tag:provider'] = provider # CZRN provider component
cbf_record['resource/tag:model'] = cloud_local_id # CZRN cloud-local-id component (model)
-
+
# Add resource tags for all dimensions (using resource/tag: format)
for key, value in dimensions.items():
- # Ensure value is a scalar and not empty
- if hasattr(value, 'item') and not isinstance(value, str):
- value = value.item() if value is not None else None
- if value is not None and str(value) not in ['', 'N/A', 'None', 'null']: # Only add non-empty tags
+ if value and value != 'N/A' and value != 'unknown': # Only add meaningful tags
cbf_record[f'resource/tag:{key}'] = str(value)
# Add token breakdown as resource tags for analysis
- if total_prompt_tokens > 0:
- cbf_record['resource/tag:prompt_tokens'] = str(total_prompt_tokens)
- if total_completion_tokens > 0:
- cbf_record['resource/tag:completion_tokens'] = str(total_completion_tokens)
+ if prompt_tokens > 0:
+ cbf_record['resource/tag:prompt_tokens'] = str(prompt_tokens)
+ if completion_tokens > 0:
+ cbf_record['resource/tag:completion_tokens'] = str(completion_tokens)
if total_tokens > 0:
cbf_record['resource/tag:total_tokens'] = str(total_tokens)
return CBFRecord(cbf_record)
- def _parse_datetime(self, datetime_obj) -> Optional[datetime]:
- """Parse datetime object to ensure proper format."""
- if datetime_obj is None:
+ def _parse_date(self, date_str) -> Optional[datetime]:
+ """Parse date string from daily spend tables (e.g., '2025-04-19')."""
+ if date_str is None:
return None
- if isinstance(datetime_obj, datetime):
- return datetime_obj
+ if isinstance(date_str, datetime):
+ return date_str
- if isinstance(datetime_obj, str):
+ if isinstance(date_str, str):
try:
- # Try to parse ISO format
- return pl.Series([datetime_obj]).str.to_datetime().item()
+ # Parse date string and set to midnight UTC for daily aggregation
+ return pl.Series([date_str]).str.to_datetime("%Y-%m-%d").item()
except Exception:
- return None
+ try:
+ # Fallback: try ISO format parsing
+ return pl.Series([date_str]).str.to_datetime().item()
+ except Exception:
+ return None
return None
diff --git a/litellm/integrations/custom_guardrail.py b/litellm/integrations/custom_guardrail.py
index b6792354334..22e652e1d7b 100644
--- a/litellm/integrations/custom_guardrail.py
+++ b/litellm/integrations/custom_guardrail.py
@@ -1,5 +1,5 @@
from datetime import datetime
-from typing import Any, Dict, List, Literal, Optional, Type, Union, get_args
+from typing import Any, Dict, List, Optional, Type, Union, get_args
from litellm._logging import verbose_logger
from litellm.caching import DualCache
@@ -14,6 +14,7 @@ from litellm.types.guardrails import (
from litellm.types.proxy.guardrails.guardrail_hooks.base import GuardrailConfigModel
from litellm.types.utils import (
CallTypes,
+ GuardrailStatus,
LLMResponseTypes,
StandardLoggingGuardrailInformation,
)
@@ -119,11 +120,8 @@ class CustomGuardrail(CustomLogger):
"""
if "guardrails" in data:
return data["guardrails"]
- metadata = data.get("metadata") or {}
- requested_guardrails = metadata.get("guardrails") or []
- if requested_guardrails:
- return requested_guardrails
- return requested_guardrails
+ metadata = data.get("litellm_metadata") or data.get("metadata", {})
+ return metadata.get("guardrails") or []
def _guardrail_is_in_requested_guardrails(
self,
@@ -234,7 +232,6 @@ class CustomGuardrail(CustomLogger):
Returns True if the guardrail should be run on the event_type
"""
requested_guardrails = self.get_guardrail_from_metadata(data)
-
verbose_logger.debug(
"inside should_run_guardrail for guardrail=%s event_type= %s guardrail_supported_event_hooks= %s requested_guardrails= %s self.default_on= %s",
self.guardrail_name,
@@ -243,7 +240,6 @@ class CustomGuardrail(CustomLogger):
requested_guardrails,
self.default_on,
)
-
if self.default_on is True:
if self._event_hook_is_event_type(event_type):
if isinstance(self.event_hook, Mode):
@@ -287,7 +283,6 @@ class CustomGuardrail(CustomLogger):
)
if result is not None:
return result
-
return True
def _event_hook_is_event_type(self, event_type: GuardrailEventHooks) -> bool:
@@ -358,11 +353,12 @@ class CustomGuardrail(CustomLogger):
self,
guardrail_json_response: Union[Exception, str, dict, List[dict]],
request_data: dict,
- guardrail_status: Literal["success", "failure"],
+ guardrail_status: GuardrailStatus,
start_time: Optional[float] = None,
end_time: Optional[float] = None,
duration: Optional[float] = None,
masked_entity_count: Optional[Dict[str, int]] = None,
+ guardrail_provider: Optional[str] = None,
) -> None:
"""
Builds `StandardLoggingGuardrailInformation` and adds it to the request metadata so it can be used for logging to DataDog, Langfuse, etc.
@@ -373,6 +369,7 @@ class CustomGuardrail(CustomLogger):
slg = StandardLoggingGuardrailInformation(
guardrail_name=self.guardrail_name,
+ guardrail_provider=guardrail_provider,
guardrail_mode=(
GuardrailMode(**self.event_hook.model_dump()) # type: ignore
if isinstance(self.event_hook, Mode)
@@ -464,7 +461,7 @@ class CustomGuardrail(CustomLogger):
self.add_standard_logging_guardrail_information_to_request_data(
guardrail_json_response=e,
request_data=request_data,
- guardrail_status="failure",
+ guardrail_status="guardrail_failed_to_respond",
duration=duration,
start_time=start_time,
end_time=end_time,
@@ -493,7 +490,8 @@ class CustomGuardrail(CustomLogger):
"""
Update the guardrails litellm params in memory
"""
- pass
+ for key, value in vars(litellm_params).items():
+ setattr(self, key, value)
def log_guardrail_information(func):
diff --git a/litellm/integrations/custom_logger.py b/litellm/integrations/custom_logger.py
index 08ff197c208..ee7e771faa6 100644
--- a/litellm/integrations/custom_logger.py
+++ b/litellm/integrations/custom_logger.py
@@ -33,7 +33,11 @@ if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._types import UserAPIKeyAuth
- from litellm.types.mcp import MCPPostCallResponseObject
+ from litellm.types.mcp import (
+ MCPPostCallResponseObject,
+ MCPPreCallRequestObject,
+ MCPPreCallResponseObject,
+ )
from litellm.types.router import PreRoutingHookResponse
Span = Union[_Span, Any]
@@ -42,13 +46,30 @@ else:
LiteLLMLoggingObj = Any
UserAPIKeyAuth = Any
MCPPostCallResponseObject = Any
+ MCPPreCallRequestObject = Any
+ MCPPreCallResponseObject = Any
+ MCPDuringCallRequestObject = Any
+ MCPDuringCallResponseObject = Any
PreRoutingHookResponse = Any
class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callback#callback-class
# Class variables or attributes
- def __init__(self, message_logging: bool = True, **kwargs) -> None:
+ def __init__(
+ self,
+ turn_off_message_logging: bool = False,
+
+ # deprecated param, use `turn_off_message_logging` instead
+ message_logging: bool = True,
+ **kwargs
+ ) -> None:
+ """
+ Args:
+ turn_off_message_logging: bool - if True, the message logging will be turned off. Message and response will be redacted from StandardLoggingPayload.
+ message_logging: bool - deprecated param, use `turn_off_message_logging` instead
+ """
self.message_logging = message_logging
+ self.turn_off_message_logging = turn_off_message_logging
pass
def log_pre_api_call(self, model, messages, kwargs):
@@ -258,6 +279,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
"audio_transcription",
"pass_through_endpoint",
"rerank",
+ "mcp_call",
],
) -> Optional[
Union[Exception, str, dict]
@@ -304,6 +326,7 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
"moderation",
"audio_transcription",
"responses",
+ "mcp_call",
],
) -> Any:
pass
@@ -387,6 +410,8 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
#########################################################
# MCP TOOL CALL HOOKS
#########################################################
+
+
async def async_post_mcp_tool_call_hook(
self, kwargs, response_obj: MCPPostCallResponseObject, start_time, end_time
) -> Optional[MCPPostCallResponseObject]:
@@ -470,3 +495,60 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
if LITELLM_METADATA_FIELD in request_kwargs:
return LITELLM_METADATA_FIELD
return OLD_LITELLM_METADATA_FIELD
+
+ def redact_standard_logging_payload_from_model_call_details(
+ self, model_call_details: Dict
+ ) -> Dict:
+ """
+ Only redacts messages and responses when self.turn_off_message_logging is True
+
+
+ By default, self.turn_off_message_logging is False and this does nothing.
+
+ Return a redacted deepcopy of the provided logging payload.
+
+ This is useful for logging payloads that contain sensitive information.
+ """
+ from copy import copy
+
+ from litellm import Choices, Message, ModelResponse
+ from litellm.types.utils import LiteLLMCommonStrings
+ turn_off_message_logging: bool = getattr(self, "turn_off_message_logging", False)
+
+ if turn_off_message_logging is False:
+ return model_call_details
+
+ # Only make a shallow copy of the top-level dict to avoid deepcopy issues
+ # with complex objects like AuthenticationError that may be present
+ model_call_details_copy = copy(model_call_details)
+ redacted_str = LiteLLMCommonStrings.redacted_by_litellm.value
+ standard_logging_object = model_call_details.get("standard_logging_object")
+ if standard_logging_object is None:
+ return model_call_details_copy
+
+ # Make a copy of just the standard_logging_object to avoid modifying the original
+ standard_logging_object_copy = copy(standard_logging_object)
+
+ if standard_logging_object_copy.get("messages") is not None:
+ standard_logging_object_copy["messages"] = [Message(content=redacted_str).model_dump()]
+
+ if standard_logging_object_copy.get("response") is not None:
+ model_response = ModelResponse(
+ choices=[Choices(message=Message(content=redacted_str))]
+ )
+ model_response_dict = model_response.model_dump()
+ standard_logging_object_copy["response"] = model_response_dict
+
+ model_call_details_copy["standard_logging_object"] = standard_logging_object_copy
+ return model_call_details_copy
+
+
+
+ async def get_proxy_server_request_from_cold_storage_with_object_key(
+ self,
+ object_key: str,
+ ) -> Optional[dict]:
+ """
+ Get the proxy server request from cold storage using the object key directly.
+ """
+ pass
diff --git a/litellm/integrations/datadog/datadog.py b/litellm/integrations/datadog/datadog.py
index 1fa651ec71c..0c62667f749 100644
--- a/litellm/integrations/datadog/datadog.py
+++ b/litellm/integrations/datadog/datadog.py
@@ -17,9 +17,9 @@ import asyncio
import datetime
import os
import traceback
-import uuid
+from litellm._uuid import uuid
from datetime import datetime as datetimeObj
-from typing import Any, List, Optional, Union
+from typing import Any, Dict, List, Optional, Union
import httpx
from httpx import Response
@@ -71,6 +71,13 @@ class DataDogLogger(
raise Exception("DD_API_KEY is not set, set 'DD_API_KEY=<>")
if os.getenv("DD_SITE", None) is None:
raise Exception("DD_SITE is not set in .env, set 'DD_SITE=<>")
+
+ #########################################################
+ # Handle datadog_params set as litellm.datadog_params
+ #########################################################
+ dict_datadog_params = self._get_datadog_params()
+ kwargs.update(dict_datadog_params)
+
self.async_client = get_async_httpx_client(
llm_provider=httpxSpecialProvider.LoggingCallback
)
@@ -101,6 +108,21 @@ class DataDogLogger(
)
raise e
+ def _get_datadog_params(self) -> Dict:
+ """
+ Get the datadog_params from litellm.datadog_params
+
+ These are params specific to initializing the DataDogLogger e.g. turn_off_message_logging
+ """
+ dict_datadog_params: Dict = {}
+ if litellm.datadog_params is not None:
+ if isinstance(litellm.datadog_params, DatadogInitParams):
+ dict_datadog_params = litellm.datadog_params.model_dump()
+ elif isinstance(litellm.datadog_params, Dict):
+ # only allow params that are of DatadogInitParams
+ dict_datadog_params = DatadogInitParams(**litellm.datadog_params).model_dump()
+ return dict_datadog_params
+
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
"""
Async Log success events to Datadog
@@ -458,6 +480,7 @@ class DataDogLogger(
else:
clean_metadata[key] = value
+
# Build the initial payload
payload = {
"id": id,
diff --git a/litellm/integrations/datadog/datadog_llm_obs.py b/litellm/integrations/datadog/datadog_llm_obs.py
index 8cee33968b3..fc3cf4b9ff2 100644
--- a/litellm/integrations/datadog/datadog_llm_obs.py
+++ b/litellm/integrations/datadog/datadog_llm_obs.py
@@ -9,7 +9,7 @@ API Reference: https://docs.datadoghq.com/llm_observability/setup/api/?tab=examp
import asyncio
import json
import os
-import uuid
+from litellm._uuid import uuid
from datetime import datetime
from typing import Any, Dict, List, Literal, Optional, Union
@@ -19,6 +19,7 @@ import litellm
from litellm._logging import verbose_logger
from litellm.integrations.custom_batch_logger import CustomBatchLogger
from litellm.integrations.datadog.datadog import DataDogLogger
+from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.prompt_templates.common_utils import (
handle_any_messages_to_chat_completion_str_messages_conversion,
)
@@ -27,7 +28,12 @@ from litellm.llms.custom_httpx.http_handler import (
httpxSpecialProvider,
)
from litellm.types.integrations.datadog_llm_obs import *
-from litellm.types.utils import CallTypes, StandardLoggingPayload
+from litellm.types.utils import (
+ CallTypes,
+ StandardLoggingGuardrailInformation,
+ StandardLoggingPayload,
+ StandardLoggingPayloadErrorInformation,
+)
class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
@@ -58,19 +64,44 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
asyncio.create_task(self.periodic_flush())
self.flush_lock = asyncio.Lock()
self.log_queue: List[LLMObsPayload] = []
+
+ #########################################################
+ # Handle datadog_llm_observability_params set as litellm.datadog_llm_observability_params
+ #########################################################
+ dict_datadog_llm_obs_params = self._get_datadog_llm_obs_params()
+ kwargs.update(dict_datadog_llm_obs_params)
CustomBatchLogger.__init__(self, **kwargs, flush_lock=self.flush_lock)
except Exception as e:
verbose_logger.exception(f"DataDogLLMObs: Error initializing - {str(e)}")
raise e
+ def _get_datadog_llm_obs_params(self) -> Dict:
+ """
+ Get the datadog_llm_observability_params from litellm.datadog_llm_observability_params
+
+ These are params specific to initializing the DataDogLLMObsLogger e.g. turn_off_message_logging
+ """
+ dict_datadog_llm_obs_params: Dict = {}
+ if litellm.datadog_llm_observability_params is not None:
+ if isinstance(
+ litellm.datadog_llm_observability_params, DatadogLLMObsInitParams
+ ):
+ dict_datadog_llm_obs_params = (
+ litellm.datadog_llm_observability_params.model_dump()
+ )
+ elif isinstance(litellm.datadog_llm_observability_params, Dict):
+ # only allow params that are of DatadogLLMObsInitParams
+ dict_datadog_llm_obs_params = DatadogLLMObsInitParams(
+ **litellm.datadog_llm_observability_params
+ ).model_dump()
+ return dict_datadog_llm_obs_params
+
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
try:
verbose_logger.debug(
f"DataDogLLMObs: Logging success event for model {kwargs.get('model', 'unknown')}"
)
- payload = self.create_llm_obs_payload(
- kwargs, response_obj, start_time, end_time
- )
+ payload = self.create_llm_obs_payload(kwargs, start_time, end_time)
verbose_logger.debug(f"DataDogLLMObs: Payload: {payload}")
self.log_queue.append(payload)
@@ -81,6 +112,22 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
f"DataDogLLMObs: Error logging success event - {str(e)}"
)
+ async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
+ try:
+ verbose_logger.debug(
+ f"DataDogLLMObs: Logging failure event for model {kwargs.get('model', 'unknown')}"
+ )
+ payload = self.create_llm_obs_payload(kwargs, start_time, end_time)
+ verbose_logger.debug(f"DataDogLLMObs: Payload: {payload}")
+ self.log_queue.append(payload)
+
+ if len(self.log_queue) >= self.batch_size:
+ await self.async_send_batch()
+ except Exception as e:
+ verbose_logger.exception(
+ f"DataDogLLMObs: Error logging failure event - {str(e)}"
+ )
+
async def async_send_batch(self):
try:
if not self.log_queue:
@@ -101,10 +148,22 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
),
),
}
- verbose_logger.debug("payload %s", json.dumps(payload, indent=4))
+
+ # serialize datetime objects - for budget reset time in spend metrics
+ from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
+
+ try:
+ verbose_logger.debug("payload %s", safe_dumps(payload))
+ except Exception as debug_error:
+ verbose_logger.debug(
+ "payload serialization failed: %s", str(debug_error)
+ )
+
+ json_payload = safe_dumps(payload)
+
response = await self.async_client.post(
url=self.intake_url,
- json=payload,
+ content=json_payload,
headers={
"DD-API-KEY": self.DD_API_KEY,
"Content-Type": "application/json",
@@ -128,7 +187,7 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
verbose_logger.exception(f"DataDogLLMObs: Error sending batch - {str(e)}")
def create_llm_obs_payload(
- self, kwargs: Dict, response_obj: Any, start_time: datetime, end_time: datetime
+ self, kwargs: Dict, start_time: datetime, end_time: datetime
) -> LLMObsPayload:
standard_logging_payload: Optional[StandardLoggingPayload] = kwargs.get(
"standard_logging_object"
@@ -146,13 +205,21 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
messages
)
)
- output_meta = OutputMeta(messages=self._get_response_messages(response_obj))
+ output_meta = OutputMeta(
+ messages=self._get_response_messages(
+ standard_logging_payload=standard_logging_payload,
+ call_type=standard_logging_payload.get("call_type"),
+ )
+ )
+
+ error_info = self._assemble_error_info(standard_logging_payload)
meta = Meta(
kind=self._get_datadog_span_kind(standard_logging_payload.get("call_type")),
input=input_meta,
output=output_meta,
metadata=self._get_dd_llm_obs_payload_metadata(standard_logging_payload),
+ error=error_info,
)
# Calculate metrics (you may need to adjust these based on available data)
@@ -161,10 +228,12 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
output_tokens=float(standard_logging_payload.get("completion_tokens", 0)),
total_tokens=float(standard_logging_payload.get("total_tokens", 0)),
total_cost=float(standard_logging_payload.get("response_cost", 0)),
- time_to_first_token=self._get_time_to_first_token_seconds(standard_logging_payload),
+ time_to_first_token=self._get_time_to_first_token_seconds(
+ standard_logging_payload
+ ),
)
- return LLMObsPayload(
+ payload: LLMObsPayload = LLMObsPayload(
parent_id=metadata.get("parent_id", "undefined"),
trace_id=standard_logging_payload.get("trace_id", str(uuid.uuid4())),
span_id=metadata.get("span_id", str(uuid.uuid4())),
@@ -173,12 +242,60 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
start_ns=int(start_time.timestamp() * 1e9),
duration=int((end_time - start_time).total_seconds() * 1e9),
metrics=metrics,
+ status="error" if error_info else "ok",
tags=[
self._get_datadog_tags(standard_logging_object=standard_logging_payload)
],
)
-
- def _get_time_to_first_token_seconds(self, standard_logging_payload: StandardLoggingPayload) -> float:
+
+ apm_trace_id = self._get_apm_trace_id()
+ if apm_trace_id is not None:
+ payload["apm_id"] = apm_trace_id
+
+ return payload
+
+ def _get_apm_trace_id(self) -> Optional[str]:
+ """Retrieve the current APM trace ID if available."""
+ try:
+ current_span_fn = getattr(tracer, "current_span", None)
+ if callable(current_span_fn):
+ current_span = current_span_fn()
+ if current_span is not None:
+ trace_id = getattr(current_span, "trace_id", None)
+ if trace_id is not None:
+ return str(trace_id)
+ except Exception:
+ pass
+ return None
+
+ def _assemble_error_info(
+ self, standard_logging_payload: StandardLoggingPayload
+ ) -> Optional[DDLLMObsError]:
+ """
+ Assemble error information for failure cases according to DD LLM Obs API spec
+ """
+ # Handle error information for failure cases according to DD LLM Obs API spec
+ error_info: Optional[DDLLMObsError] = None
+
+ if standard_logging_payload.get("status") == "failure":
+ # Try to get structured error information first
+ error_information: Optional[
+ StandardLoggingPayloadErrorInformation
+ ] = standard_logging_payload.get("error_information")
+
+ if error_information:
+ error_info = DDLLMObsError(
+ message=error_information.get("error_message")
+ or standard_logging_payload.get("error_str")
+ or "Unknown error",
+ type=error_information.get("error_class"),
+ stack=error_information.get("traceback"),
+ )
+ return error_info
+
+ def _get_time_to_first_token_seconds(
+ self, standard_logging_payload: StandardLoggingPayload
+ ) -> float:
"""
Get the time to first token in seconds
@@ -187,7 +304,9 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
For non streaming calls, CompletionStartTime is time we get the response back
"""
start_time: Optional[float] = standard_logging_payload.get("startTime")
- completion_start_time: Optional[float] = standard_logging_payload.get("completionStartTime")
+ completion_start_time: Optional[float] = standard_logging_payload.get(
+ "completionStartTime"
+ )
end_time: Optional[float] = standard_logging_payload.get("endTime")
if completion_start_time is not None and start_time is not None:
@@ -197,113 +316,153 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
else:
return 0.0
-
- def _get_response_messages(self, response_obj: Any) -> List[Any]:
+ def _get_response_messages(
+ self, standard_logging_payload: StandardLoggingPayload, call_type: Optional[str]
+ ) -> List[Any]:
"""
Get the messages from the response object
for now this handles logging /chat/completions responses
"""
- if isinstance(response_obj, litellm.ModelResponse):
- return [response_obj["choices"][0]["message"].json()]
+
+ response_obj = standard_logging_payload.get("response")
+ if response_obj is None:
+ return []
+
+ # edge case: handle response_obj is a string representation of a dict
+ if isinstance(response_obj, str):
+ try:
+ import ast
+
+ response_obj = ast.literal_eval(response_obj)
+ except (ValueError, SyntaxError):
+ try:
+ # fallback to json parsing
+ response_obj = json.loads(str(response_obj))
+ except json.JSONDecodeError:
+ return []
+
+ if call_type in [
+ CallTypes.completion.value,
+ CallTypes.acompletion.value,
+ CallTypes.text_completion.value,
+ CallTypes.atext_completion.value,
+ CallTypes.generate_content.value,
+ CallTypes.agenerate_content.value,
+ CallTypes.generate_content_stream.value,
+ CallTypes.agenerate_content_stream.value,
+ CallTypes.anthropic_messages.value,
+ ]:
+ try:
+ # Safely extract message from response_obj, handle failure cases
+ if isinstance(response_obj, dict) and "choices" in response_obj:
+ choices = response_obj["choices"]
+ if choices and len(choices) > 0 and "message" in choices[0]:
+ return [choices[0]["message"]]
+ return []
+ except (KeyError, IndexError, TypeError):
+ # In case of any error accessing the response structure, return empty list
+ return []
return []
- def _get_datadog_span_kind(self, call_type: Optional[str]) -> Literal["llm", "tool", "task", "embedding", "retrieval"]:
+ def _get_datadog_span_kind(
+ self, call_type: Optional[str]
+ ) -> Literal["llm", "tool", "task", "embedding", "retrieval"]:
"""
Map liteLLM call_type to appropriate DataDog LLM Observability span kind.
-
+
Available DataDog span kinds: "llm", "tool", "task", "embedding", "retrieval"
"""
if call_type is None:
return "llm"
-
+
# Embedding operations
if call_type in [CallTypes.embedding.value, CallTypes.aembedding.value]:
return "embedding"
-
- # LLM completion operations
+
+ # LLM completion operations
if call_type in [
- CallTypes.completion.value,
+ CallTypes.completion.value,
CallTypes.acompletion.value,
- CallTypes.text_completion.value,
+ CallTypes.text_completion.value,
CallTypes.atext_completion.value,
- CallTypes.generate_content.value,
+ CallTypes.generate_content.value,
CallTypes.agenerate_content.value,
- CallTypes.generate_content_stream.value,
+ CallTypes.generate_content_stream.value,
CallTypes.agenerate_content_stream.value,
- CallTypes.anthropic_messages.value
+ CallTypes.anthropic_messages.value,
]:
return "llm"
-
+
# Tool operations
if call_type in [CallTypes.call_mcp_tool.value]:
return "tool"
-
+
# Retrieval operations
if call_type in [
- CallTypes.get_assistants.value,
+ CallTypes.get_assistants.value,
CallTypes.aget_assistants.value,
- CallTypes.get_thread.value,
+ CallTypes.get_thread.value,
CallTypes.aget_thread.value,
- CallTypes.get_messages.value,
+ CallTypes.get_messages.value,
CallTypes.aget_messages.value,
- CallTypes.afile_retrieve.value,
+ CallTypes.afile_retrieve.value,
CallTypes.file_retrieve.value,
- CallTypes.afile_list.value,
+ CallTypes.afile_list.value,
CallTypes.file_list.value,
- CallTypes.afile_content.value,
+ CallTypes.afile_content.value,
CallTypes.file_content.value,
- CallTypes.retrieve_batch.value,
+ CallTypes.retrieve_batch.value,
CallTypes.aretrieve_batch.value,
- CallTypes.retrieve_fine_tuning_job.value,
+ CallTypes.retrieve_fine_tuning_job.value,
CallTypes.aretrieve_fine_tuning_job.value,
- CallTypes.responses.value,
+ CallTypes.responses.value,
CallTypes.aresponses.value,
- CallTypes.alist_input_items.value
+ CallTypes.alist_input_items.value,
]:
return "retrieval"
-
+
# Task operations (batch, fine-tuning, file operations, etc.)
if call_type in [
- CallTypes.create_batch.value,
+ CallTypes.create_batch.value,
CallTypes.acreate_batch.value,
- CallTypes.create_fine_tuning_job.value,
+ CallTypes.create_fine_tuning_job.value,
CallTypes.acreate_fine_tuning_job.value,
- CallTypes.cancel_fine_tuning_job.value,
+ CallTypes.cancel_fine_tuning_job.value,
CallTypes.acancel_fine_tuning_job.value,
- CallTypes.list_fine_tuning_jobs.value,
+ CallTypes.list_fine_tuning_jobs.value,
CallTypes.alist_fine_tuning_jobs.value,
- CallTypes.create_assistants.value,
+ CallTypes.create_assistants.value,
CallTypes.acreate_assistants.value,
- CallTypes.delete_assistant.value,
+ CallTypes.delete_assistant.value,
CallTypes.adelete_assistant.value,
- CallTypes.create_thread.value,
+ CallTypes.create_thread.value,
CallTypes.acreate_thread.value,
- CallTypes.add_message.value,
+ CallTypes.add_message.value,
CallTypes.a_add_message.value,
- CallTypes.run_thread.value,
+ CallTypes.run_thread.value,
CallTypes.arun_thread.value,
- CallTypes.run_thread_stream.value,
+ CallTypes.run_thread_stream.value,
CallTypes.arun_thread_stream.value,
- CallTypes.file_delete.value,
+ CallTypes.file_delete.value,
CallTypes.afile_delete.value,
- CallTypes.create_file.value,
+ CallTypes.create_file.value,
CallTypes.acreate_file.value,
- CallTypes.image_generation.value,
+ CallTypes.image_generation.value,
CallTypes.aimage_generation.value,
- CallTypes.image_edit.value,
+ CallTypes.image_edit.value,
CallTypes.aimage_edit.value,
- CallTypes.moderation.value,
+ CallTypes.moderation.value,
CallTypes.amoderation.value,
- CallTypes.transcription.value,
+ CallTypes.transcription.value,
CallTypes.atranscription.value,
- CallTypes.speech.value,
+ CallTypes.speech.value,
CallTypes.aspeech.value,
- CallTypes.rerank.value,
- CallTypes.arerank.value
+ CallTypes.rerank.value,
+ CallTypes.arerank.value,
]:
return "task"
-
+
# Default fallback for unknown or passthrough operations
return "llm"
@@ -322,11 +481,11 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
def _get_dd_llm_obs_payload_metadata(
self, standard_logging_payload: StandardLoggingPayload
- ) -> Dict:
+ ) -> Dict[str, Any]:
"""
Fields to track in DD LLM Observability metadata from litellm standard logging payload
"""
- _metadata = {
+ _metadata: Dict[str, Any] = {
"model_name": standard_logging_payload.get("model", "unknown"),
"model_provider": standard_logging_payload.get(
"custom_llm_provider", "unknown"
@@ -336,9 +495,285 @@ class DataDogLLMObsLogger(DataDogLogger, CustomBatchLogger):
"cache_hit": standard_logging_payload.get("cache_hit", "unknown"),
"cache_key": standard_logging_payload.get("cache_key", "unknown"),
"saved_cache_cost": standard_logging_payload.get("saved_cache_cost", 0),
+ "guardrail_information": standard_logging_payload.get(
+ "guardrail_information", None
+ ),
+ "is_streamed_request": self._get_stream_value_from_payload(standard_logging_payload),
}
+
+ #########################################################
+ # Add latency metrics to metadata
+ #########################################################
+ latency_metrics = self._get_latency_metrics(standard_logging_payload)
+ _metadata.update({"latency_metrics": dict(latency_metrics)})
+
+ #########################################################
+ # Add spend metrics to metadata
+ #########################################################
+ spend_metrics = self._get_spend_metrics(standard_logging_payload)
+ _metadata.update({"spend_metrics": dict(spend_metrics)})
+
+ ## extract tool calls and add to metadata
+ tool_call_metadata = self._extract_tool_call_metadata(standard_logging_payload)
+ _metadata.update(tool_call_metadata)
+
_standard_logging_metadata: dict = (
dict(standard_logging_payload.get("metadata", {})) or {}
)
_metadata.update(_standard_logging_metadata)
return _metadata
+
+ def _get_latency_metrics(
+ self, standard_logging_payload: StandardLoggingPayload
+ ) -> DDLLMObsLatencyMetrics:
+ """
+ Get the latency metrics from the standard logging payload
+ """
+ latency_metrics: DDLLMObsLatencyMetrics = DDLLMObsLatencyMetrics()
+ # Add latency metrics to metadata
+ # Time to first token (convert from seconds to milliseconds for consistency)
+ time_to_first_token_seconds = self._get_time_to_first_token_seconds(
+ standard_logging_payload
+ )
+ if time_to_first_token_seconds > 0:
+ latency_metrics["time_to_first_token_ms"] = (
+ time_to_first_token_seconds * 1000
+ )
+
+ # LiteLLM overhead time
+ hidden_params = standard_logging_payload.get("hidden_params", {})
+ litellm_overhead_ms = hidden_params.get("litellm_overhead_time_ms")
+ if litellm_overhead_ms is not None:
+ latency_metrics["litellm_overhead_time_ms"] = litellm_overhead_ms
+
+ # Guardrail overhead latency
+ guardrail_info: Optional[
+ StandardLoggingGuardrailInformation
+ ] = standard_logging_payload.get("guardrail_information")
+ if guardrail_info is not None:
+ _guardrail_duration_seconds: Optional[float] = guardrail_info.get(
+ "duration"
+ )
+ if _guardrail_duration_seconds is not None:
+ # Convert from seconds to milliseconds for consistency
+ latency_metrics["guardrail_overhead_time_ms"] = (
+ _guardrail_duration_seconds * 1000
+ )
+
+ return latency_metrics
+
+ def _get_stream_value_from_payload(self, standard_logging_payload: StandardLoggingPayload) -> bool:
+ """
+ Extract the stream value from standard logging payload.
+
+ The stream field in StandardLoggingPayload is only set to True for completed streaming responses.
+ For non-streaming requests, it's None. The original stream parameter is in model_parameters.
+
+ Returns:
+ bool: True if this was a streaming request, False otherwise
+ """
+ # Check top-level stream field first (only True for completed streaming)
+ stream_value = standard_logging_payload.get("stream")
+ if stream_value is True:
+ return True
+
+ # Fallback to model_parameters.stream for original request parameters
+ model_params = standard_logging_payload.get("model_parameters", {})
+ if isinstance(model_params, dict):
+ stream_value = model_params.get("stream")
+ if stream_value is True:
+ return True
+
+ # Default to False for non-streaming requests
+ return False
+
+ def _get_spend_metrics(
+ self, standard_logging_payload: StandardLoggingPayload
+ ) -> DDLLMObsSpendMetrics:
+ """
+ Get the spend metrics from the standard logging payload
+ """
+ spend_metrics: DDLLMObsSpendMetrics = DDLLMObsSpendMetrics()
+
+ # send response cost
+ spend_metrics["response_cost"] = standard_logging_payload.get(
+ "response_cost", 0.0
+ )
+
+ # Get budget information from metadata
+ metadata = standard_logging_payload.get("metadata", {})
+
+ # API key max budget
+ user_api_key_max_budget = metadata.get("user_api_key_max_budget")
+ if user_api_key_max_budget is not None:
+ spend_metrics["user_api_key_max_budget"] = float(user_api_key_max_budget)
+
+ # API key spend
+ user_api_key_spend = metadata.get("user_api_key_spend")
+ if user_api_key_spend is not None:
+ try:
+ spend_metrics["user_api_key_spend"] = float(user_api_key_spend)
+ except (ValueError, TypeError):
+ verbose_logger.debug(
+ f"Invalid user_api_key_spend value: {user_api_key_spend}"
+ )
+
+ # API key budget reset datetime
+ user_api_key_budget_reset_at = metadata.get("user_api_key_budget_reset_at")
+ if user_api_key_budget_reset_at is not None:
+ try:
+ from datetime import datetime, timezone
+
+ budget_reset_at = None
+ if isinstance(user_api_key_budget_reset_at, str):
+ # Handle ISO format strings that might have 'Z' suffix
+ iso_string = user_api_key_budget_reset_at.replace("Z", "+00:00")
+ budget_reset_at = datetime.fromisoformat(iso_string)
+ elif isinstance(user_api_key_budget_reset_at, datetime):
+ budget_reset_at = user_api_key_budget_reset_at
+
+ if budget_reset_at is not None:
+ # Preserve timezone info if already present
+ if budget_reset_at.tzinfo is None:
+ budget_reset_at = budget_reset_at.replace(tzinfo=timezone.utc)
+
+ # Convert to ISO string format for JSON serialization
+ # This prevents circular reference issues and ensures proper timezone representation
+ iso_string = budget_reset_at.isoformat()
+ spend_metrics["user_api_key_budget_reset_at"] = iso_string
+
+ # Debug logging to verify the conversion
+ verbose_logger.debug(
+ f"Converted budget_reset_at to ISO format: {iso_string}"
+ )
+ except Exception as e:
+ verbose_logger.debug(f"Error processing budget reset datetime: {e}")
+ verbose_logger.debug(f"Original value: {user_api_key_budget_reset_at}")
+
+ return spend_metrics
+
+ def _process_input_messages_preserving_tool_calls(
+ self, messages: List[Any]
+ ) -> List[Dict[str, Any]]:
+ """
+ Process input messages while preserving tool_calls and tool message types.
+
+ This bypasses the lossy string conversion when tool calls are present,
+ allowing complex nested tool_calls objects to be preserved for Datadog.
+ """
+ processed = []
+ for msg in messages:
+ if isinstance(msg, dict):
+ # Preserve messages with tool_calls or tool role as-is
+ if "tool_calls" in msg or msg.get("role") == "tool":
+ processed.append(msg)
+ else:
+ # For regular messages, still apply string conversion
+ converted = (
+ handle_any_messages_to_chat_completion_str_messages_conversion(
+ [msg]
+ )
+ )
+ processed.extend(converted)
+ else:
+ # For non-dict messages, apply string conversion
+ converted = (
+ handle_any_messages_to_chat_completion_str_messages_conversion(
+ [msg]
+ )
+ )
+ processed.extend(converted)
+ return processed
+
+ @staticmethod
+ def _tool_calls_kv_pair(tool_calls: List[Dict[str, Any]]) -> Dict[str, Any]:
+ """
+ Extract tool call information into key-value pairs for Datadog metadata.
+
+ Similar to OpenTelemetry's implementation but adapted for Datadog's format.
+ """
+ kv_pairs: Dict[str, Any] = {}
+ for idx, tool_call in enumerate(tool_calls):
+ try:
+ # Extract tool call ID
+ tool_id = tool_call.get("id")
+ if tool_id:
+ kv_pairs[f"tool_calls.{idx}.id"] = tool_id
+
+ # Extract tool call type
+ tool_type = tool_call.get("type")
+ if tool_type:
+ kv_pairs[f"tool_calls.{idx}.type"] = tool_type
+
+ # Extract function information
+ function = tool_call.get("function")
+ if function:
+ function_name = function.get("name")
+ if function_name:
+ kv_pairs[f"tool_calls.{idx}.function.name"] = function_name
+
+ function_arguments = function.get("arguments")
+ if function_arguments:
+ # Store arguments as JSON string for Datadog
+ if isinstance(function_arguments, str):
+ kv_pairs[
+ f"tool_calls.{idx}.function.arguments"
+ ] = function_arguments
+ else:
+ import json
+
+ kv_pairs[
+ f"tool_calls.{idx}.function.arguments"
+ ] = json.dumps(function_arguments)
+ except (KeyError, TypeError, ValueError) as e:
+ verbose_logger.debug(
+ f"DataDogLLMObs: Error processing tool call {idx}: {str(e)}"
+ )
+ continue
+
+ return kv_pairs
+
+ def _extract_tool_call_metadata(
+ self, standard_logging_payload: StandardLoggingPayload
+ ) -> Dict[str, Any]:
+ """
+ Extract tool call information from both input messages and response for Datadog metadata.
+ """
+ tool_call_metadata: Dict[str, Any] = {}
+
+ try:
+ # Extract tool calls from input messages
+ messages = standard_logging_payload.get("messages", [])
+ if messages and isinstance(messages, list):
+ for message in messages:
+ if isinstance(message, dict) and "tool_calls" in message:
+ tool_calls = message.get("tool_calls")
+ if tool_calls:
+ input_tool_calls_kv = self._tool_calls_kv_pair(tool_calls)
+ # Prefix with "input_" to distinguish from response tool calls
+ for key, value in input_tool_calls_kv.items():
+ tool_call_metadata[f"input_{key}"] = value
+
+ # Extract tool calls from response
+ response_obj = standard_logging_payload.get("response")
+ if response_obj and isinstance(response_obj, dict):
+ choices = response_obj.get("choices", [])
+ for choice in choices:
+ if isinstance(choice, dict):
+ message = choice.get("message")
+ if message and isinstance(message, dict):
+ tool_calls = message.get("tool_calls")
+ if tool_calls:
+ response_tool_calls_kv = self._tool_calls_kv_pair(
+ tool_calls
+ )
+ # Prefix with "output_" to distinguish from input tool calls
+ for key, value in response_tool_calls_kv.items():
+ tool_call_metadata[f"output_{key}"] = value
+
+ except Exception as e:
+ verbose_logger.debug(
+ f"DataDogLLMObs: Error extracting tool call metadata: {str(e)}"
+ )
+
+ return tool_call_metadata
diff --git a/litellm/integrations/deepeval/deepeval.py b/litellm/integrations/deepeval/deepeval.py
index f548ff50d73..972843e120a 100644
--- a/litellm/integrations/deepeval/deepeval.py
+++ b/litellm/integrations/deepeval/deepeval.py
@@ -1,5 +1,5 @@
import os
-import uuid
+from litellm._uuid import uuid
from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.deepeval.api import Api, Endpoints, HttpMethods
from litellm.integrations.deepeval.types import (
diff --git a/litellm/integrations/dotprompt/README.md b/litellm/integrations/dotprompt/README.md
new file mode 100644
index 00000000000..c69c96824be
--- /dev/null
+++ b/litellm/integrations/dotprompt/README.md
@@ -0,0 +1,316 @@
+# LiteLLM Dotprompt Manager
+
+A powerful prompt management system for LiteLLM that supports [Google's Dotprompt specification](https://google.github.io/dotprompt/getting-started/). This allows you to manage your AI prompts in organized `.prompt` files with YAML frontmatter, Handlebars templating, and full integration with LiteLLM's completion API.
+
+## Features
+
+- **📁 File-based prompt management**: Organize prompts in `.prompt` files
+- **🎯 YAML frontmatter**: Define model, parameters, and schemas in file headers
+- **🔧 Handlebars templating**: Use `{{variable}}` syntax with Jinja2 backend
+- **✅ Input validation**: Automatic validation against defined schemas
+- **🔗 LiteLLM integration**: Works seamlessly with `litellm.completion()`
+- **💬 Smart message parsing**: Converts prompts to proper chat messages
+- **⚙️ Parameter extraction**: Automatically applies model settings from prompts
+
+## Quick Start
+
+### 1. Create a `.prompt` file
+
+Create a file called `chat_assistant.prompt`:
+
+```yaml
+---
+model: gpt-4
+temperature: 0.7
+max_tokens: 150
+input:
+ schema:
+ user_message: string
+ system_context?: string
+---
+
+{% if system_context %}System: {{system_context}}
+
+{% endif %}User: {{user_message}}
+```
+
+### 2. Use with LiteLLM
+
+```python
+import litellm
+
+litellm.set_global_prompt_directory("path/to/your/prompts")
+
+# Use with completion - the model prefix 'dotprompt/' tells LiteLLM to use prompt management
+response = litellm.completion(
+ model="dotprompt/gpt-4", # The actual model comes from the .prompt file
+ prompt_id="chat_assistant",
+ prompt_variables={
+ "user_message": "What is machine learning?",
+ "system_context": "You are a helpful AI tutor."
+ },
+ # Any additional messages will be appended after the prompt
+ messages=[{"role": "user", "content": "Please explain it simply."}]
+)
+
+print(response.choices[0].message.content)
+```
+
+## Prompt File Format
+
+### Basic Structure
+
+```yaml
+---
+# Model configuration
+model: gpt-4
+temperature: 0.7
+max_tokens: 500
+
+# Input schema (optional)
+input:
+ schema:
+ name: string
+ age: integer
+ preferences?: array
+---
+
+# Template content using Handlebars syntax
+Hello {{name}}!
+
+{% if age >= 18 %}
+You're an adult, so here are some mature recommendations:
+{% else %}
+Here are some age-appropriate suggestions:
+{% endif %}
+
+{% for pref in preferences %}
+- Based on your interest in {{pref}}, I recommend...
+{% endfor %}
+```
+
+### Supported Frontmatter Fields
+
+- **`model`**: The LLM model to use (e.g., `gpt-4`, `claude-3-sonnet`)
+- **`input.schema`**: Define expected input variables and their types
+- **`output.format`**: Expected output format (`json`, `text`, etc.)
+- **`output.schema`**: Structure of expected output
+
+### Additional Parameters
+
+- **`temperature`**: Model temperature (0.0 to 1.0)
+- **`max_tokens`**: Maximum tokens to generate
+- **`top_p`**: Nucleus sampling parameter (0.0 to 1.0)
+- **`frequency_penalty`**: Frequency penalty (0.0 to 1.0)
+- **`presence_penalty`**: Presence penalty (0.0 to 1.0)
+- any other parameters that are not model or schema-related will be treated as optional parameters to the model.
+
+### Input Schema Types
+
+- `string` or `str`: Text values
+- `integer` or `int`: Whole numbers
+- `float`: Decimal numbers
+- `boolean` or `bool`: True/false values
+- `array` or `list`: Lists of values
+- `object` or `dict`: Key-value objects
+
+Use `?` suffix for optional fields: `name?: string`
+
+## Message Format Conversion
+
+The dotprompt manager intelligently converts your rendered prompts into proper chat messages:
+
+### Simple Text → User Message
+```yaml
+---
+model: gpt-4
+---
+Tell me about {{topic}}.
+```
+Becomes: `[{"role": "user", "content": "Tell me about AI."}]`
+
+### Role-Based Format → Multiple Messages
+```yaml
+---
+model: gpt-4
+---
+System: You are a {{role}}.
+
+User: {{question}}
+```
+
+Becomes:
+```python
+[
+ {"role": "system", "content": "You are a helpful assistant."},
+ {"role": "user", "content": "What is AI?"}
+]
+```
+
+
+## Example Prompts
+
+### Data Extraction
+```yaml
+# extract_info.prompt
+---
+model: gemini/gemini-1.5-pro
+input:
+ schema:
+ text: string
+output:
+ format: json
+ schema:
+ title?: string
+ summary: string
+ tags: array
+---
+
+Extract the requested information from the given text. Return JSON format.
+
+Text: {{text}}
+```
+
+### Code Assistant
+```yaml
+# code_helper.prompt
+---
+model: claude-3-5-sonnet-20241022
+temperature: 0.2
+max_tokens: 2000
+input:
+ schema:
+ language: string
+ task: string
+ code?: string
+---
+
+You are an expert {{language}} programmer.
+
+Task: {{task}}
+
+{% if code %}
+Current code:
+```{{language}}
+{{code}}
+```
+{% endif %}
+
+Please provide a complete, well-documented solution.
+```
+
+### Multi-turn Conversation
+```yaml
+# conversation.prompt
+---
+model: gpt-4
+temperature: 0.8
+input:
+ schema:
+ personality: string
+ context: string
+---
+
+System: You are a {{personality}}. {{context}}
+
+User: Let's start our conversation.
+```
+
+## API Reference
+
+### PromptManager
+
+The core class for managing `.prompt` files.
+
+#### Methods
+
+- **`__init__(prompt_directory: str)`**: Initialize with directory path
+- **`render(prompt_id: str, variables: dict) -> str`**: Render prompt with variables
+- **`list_prompts() -> List[str]`**: Get all available prompt IDs
+- **`get_prompt(prompt_id: str) -> PromptTemplate`**: Get prompt template object
+- **`get_prompt_metadata(prompt_id: str) -> dict`**: Get prompt metadata
+- **`reload_prompts() -> None`**: Reload all prompts from directory
+- **`add_prompt(prompt_id: str, content: str, metadata: dict)`**: Add prompt programmatically
+
+### DotpromptManager
+
+LiteLLM integration class extending `PromptManagementBase`.
+
+#### Methods
+
+- **`__init__(prompt_directory: str)`**: Initialize with directory path
+- **`should_run_prompt_management(prompt_id: str, params: dict) -> bool`**: Check if prompt exists
+- **`set_prompt_directory(directory: str)`**: Change prompt directory
+- **`reload_prompts()`**: Reload prompts from directory
+
+### PromptTemplate
+
+Represents a single prompt with metadata.
+
+#### Properties
+
+- **`content: str`**: The prompt template content
+- **`metadata: dict`**: Full metadata from frontmatter
+- **`model: str`**: Specified model name
+- **`temperature: float`**: Model temperature
+- **`max_tokens: int`**: Token limit
+- **`input_schema: dict`**: Input validation schema
+- **`output_format: str`**: Expected output format
+- **`output_schema: dict`**: Output structure schema
+
+## Best Practices
+
+1. **Organize by purpose**: Group related prompts in subdirectories
+2. **Use descriptive names**: `extract_user_info.prompt` vs `prompt1.prompt`
+3. **Define schemas**: Always specify input schemas for validation
+4. **Version control**: Store `.prompt` files in git for change tracking
+5. **Test prompts**: Use the test framework to validate prompt behavior
+6. **Keep templates focused**: One prompt should do one thing well
+7. **Use includes**: Break complex prompts into reusable components
+
+## Troubleshooting
+
+### Common Issues
+
+**Prompt not found**: Ensure the `.prompt` file exists and has correct extension
+```python
+# Check available prompts
+from litellm.integrations.dotprompt import get_dotprompt_manager
+manager = get_dotprompt_manager()
+print(manager.prompt_manager.list_prompts())
+```
+
+**Template errors**: Verify Handlebars syntax and variable names
+```python
+# Test rendering directly
+manager.prompt_manager.render("my_prompt", {"test": "value"})
+```
+
+**Model not working**: Check that model name in frontmatter is correct
+```python
+# Check prompt metadata
+metadata = manager.prompt_manager.get_prompt_metadata("my_prompt")
+print(metadata)
+```
+
+### Validation Errors
+
+Input validation failures show helpful error messages:
+```
+ValueError: Invalid type for field 'age': expected int, got str
+```
+
+Make sure your variables match the defined schema types.
+
+## Contributing
+
+The LiteLLM Dotprompt manager follows the [Dotprompt specification](https://google.github.io/dotprompt/) for maximum compatibility. When contributing:
+
+1. Ensure compatibility with existing `.prompt` files
+2. Add tests for new features
+3. Update documentation
+4. Follow the existing code style
+
+## License
+
+This prompt management system is part of LiteLLM and follows the same license terms.
\ No newline at end of file
diff --git a/litellm/integrations/dotprompt/__init__.py b/litellm/integrations/dotprompt/__init__.py
new file mode 100644
index 00000000000..3af7fbf6dd3
--- /dev/null
+++ b/litellm/integrations/dotprompt/__init__.py
@@ -0,0 +1,71 @@
+from typing import TYPE_CHECKING, Optional
+
+if TYPE_CHECKING:
+ from .prompt_manager import PromptManager, PromptTemplate
+ from litellm.types.prompts.init_prompts import PromptLiteLLMParams, PromptSpec
+ from litellm.integrations.custom_prompt_management import CustomPromptManagement
+
+from litellm.types.prompts.init_prompts import SupportedPromptIntegrations
+
+from .dotprompt_manager import DotpromptManager
+
+# Global instances
+global_prompt_directory: Optional[str] = None
+global_prompt_manager: Optional["PromptManager"] = None
+
+
+def set_global_prompt_directory(directory: str) -> None:
+ """
+ Set the global prompt directory for dotprompt files.
+
+ Args:
+ directory: Path to directory containing .prompt files
+ """
+ import litellm
+
+ litellm.global_prompt_directory = directory # type: ignore
+
+
+def prompt_initializer(
+ litellm_params: "PromptLiteLLMParams", prompt_spec: "PromptSpec"
+) -> "CustomPromptManagement":
+ """
+ Initialize a prompt from a .prompt file.
+ """
+ prompt_directory = getattr(litellm_params, "prompt_directory", None)
+ prompt_data = getattr(litellm_params, "prompt_data", None)
+ prompt_id = getattr(litellm_params, "prompt_id", None)
+ if prompt_directory:
+ raise ValueError(
+ "Cannot set prompt_directory when working with prompt_initializer. Needs to be a specific dotprompt file"
+ )
+
+ prompt_file = getattr(litellm_params, "prompt_file", None)
+
+ try:
+ dot_prompt_manager = DotpromptManager(
+ prompt_directory=prompt_directory,
+ prompt_data=prompt_data,
+ prompt_file=prompt_file,
+ prompt_id=prompt_id,
+ )
+
+ return dot_prompt_manager
+ except Exception as e:
+
+ raise e
+
+
+prompt_initializer_registry = {
+ SupportedPromptIntegrations.DOT_PROMPT.value: prompt_initializer,
+}
+
+# Export public API
+__all__ = [
+ "PromptManager",
+ "DotpromptManager",
+ "PromptTemplate",
+ "set_global_prompt_directory",
+ "global_prompt_directory",
+ "global_prompt_manager",
+]
diff --git a/litellm/integrations/dotprompt/dotprompt_manager.py b/litellm/integrations/dotprompt/dotprompt_manager.py
new file mode 100644
index 00000000000..0f0d7b938f3
--- /dev/null
+++ b/litellm/integrations/dotprompt/dotprompt_manager.py
@@ -0,0 +1,291 @@
+"""
+Dotprompt manager that integrates with LiteLLM's prompt management system.
+Builds on top of PromptManagementBase to provide .prompt file support.
+"""
+
+import json
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+from litellm.integrations.custom_prompt_management import CustomPromptManagement
+from litellm.integrations.prompt_management_base import PromptManagementClient
+from litellm.types.llms.openai import AllMessageValues
+from litellm.types.utils import StandardCallbackDynamicParams
+
+from .prompt_manager import PromptManager, PromptTemplate
+
+
+class DotpromptManager(CustomPromptManagement):
+ """
+ Dotprompt manager that integrates with LiteLLM's prompt management system.
+
+ This class enables using .prompt files with the litellm completion() function
+ by implementing the PromptManagementBase interface.
+
+ Usage:
+ # Set global prompt directory
+ litellm.prompt_directory = "path/to/prompts"
+
+ # Use with completion
+ response = litellm.completion(
+ model="dotprompt/gpt-4",
+ prompt_id="my_prompt",
+ prompt_variables={"variable": "value"},
+ messages=[{"role": "user", "content": "This will be combined with the prompt"}]
+ )
+ """
+
+ def __init__(
+ self,
+ prompt_directory: Optional[str] = None,
+ prompt_file: Optional[str] = None,
+ prompt_data: Optional[Union[dict, str]] = None,
+ prompt_id: Optional[str] = None,
+ ):
+ import litellm
+
+ self.prompt_directory = prompt_directory or litellm.global_prompt_directory
+ # Support for JSON-based prompts stored in memory/database
+ if isinstance(prompt_data, str):
+ self.prompt_data = json.loads(prompt_data)
+ else:
+ self.prompt_data = prompt_data or {}
+
+ self._prompt_manager: Optional[PromptManager] = None
+ self.prompt_file = prompt_file
+ self.prompt_id = prompt_id
+
+ @property
+ def integration_name(self) -> str:
+ """Integration name used in model names like 'dotprompt/gpt-4'."""
+ return "dotprompt"
+
+ @property
+ def prompt_manager(self) -> PromptManager:
+ """Lazy-load the prompt manager."""
+ if self._prompt_manager is None:
+ if (
+ self.prompt_directory is None
+ and not self.prompt_data
+ and not self.prompt_file
+ ):
+ raise ValueError(
+ "Either prompt_directory or prompt_data must be set before using dotprompt manager. "
+ "Set litellm.global_prompt_directory, initialize with prompt_directory parameter, or provide prompt_data."
+ )
+ self._prompt_manager = PromptManager(
+ prompt_directory=self.prompt_directory,
+ prompt_data=self.prompt_data,
+ prompt_file=self.prompt_file,
+ prompt_id=self.prompt_id,
+ )
+ return self._prompt_manager
+
+ def should_run_prompt_management(
+ self,
+ prompt_id: str,
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ ) -> bool:
+ """
+ Determine if prompt management should run based on the prompt_id.
+
+ Returns True if the prompt_id exists in our prompt manager.
+ """
+ try:
+ return prompt_id in self.prompt_manager.list_prompts()
+ except Exception:
+ # If there's any error accessing prompts, don't run prompt management
+ return False
+
+ def _compile_prompt_helper(
+ self,
+ prompt_id: str,
+ prompt_variables: Optional[dict],
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ prompt_label: Optional[str] = None,
+ prompt_version: Optional[int] = None,
+ ) -> PromptManagementClient:
+ """
+ Compile a .prompt file into a PromptManagementClient structure.
+
+ This method:
+ 1. Loads the prompt template from the .prompt file
+ 2. Renders it with the provided variables
+ 3. Converts the rendered text into chat messages
+ 4. Extracts model and optional parameters from metadata
+ """
+
+ try:
+
+ # Get the prompt template
+ template = self.prompt_manager.get_prompt(prompt_id)
+ if template is None:
+ raise ValueError(f"Prompt '{prompt_id}' not found in prompt directory")
+
+ # Render the template with variables
+ rendered_content = self.prompt_manager.render(prompt_id, prompt_variables)
+
+ # Convert rendered content to chat messages
+ messages = self._convert_to_messages(rendered_content)
+
+ # Extract model from metadata (if specified)
+ template_model = template.model
+
+ # Extract optional parameters from metadata
+ optional_params = self._extract_optional_params(template)
+
+ return PromptManagementClient(
+ prompt_id=prompt_id,
+ prompt_template=messages,
+ prompt_template_model=template_model,
+ prompt_template_optional_params=optional_params,
+ completed_messages=None,
+ )
+
+ except Exception as e:
+ raise ValueError(f"Error compiling prompt '{prompt_id}': {e}")
+
+ def get_chat_completion_prompt(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ non_default_params: dict,
+ prompt_id: Optional[str],
+ prompt_variables: Optional[dict],
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ prompt_label: Optional[str] = None,
+ prompt_version: Optional[int] = None,
+ ) -> Tuple[str, List[AllMessageValues], dict]:
+
+ from litellm.integrations.prompt_management_base import PromptManagementBase
+
+ return PromptManagementBase.get_chat_completion_prompt(
+ self,
+ model,
+ messages,
+ non_default_params,
+ prompt_id,
+ prompt_variables,
+ dynamic_callback_params,
+ prompt_label,
+ prompt_version,
+ )
+
+ def _convert_to_messages(self, rendered_content: str) -> List[AllMessageValues]:
+ """
+ Convert rendered prompt content to chat messages.
+
+ This method supports multiple formats:
+ 1. Simple text -> converted to user message
+ 2. Text with role prefixes (System:, User:, Assistant:) -> parsed into separate messages
+ 3. Already formatted as a single message
+ """
+ # Clean up the content
+ content = rendered_content.strip()
+
+ # Try to parse role-based format (System: ..., User: ..., etc.)
+ messages = []
+ current_role = None
+ current_content = []
+
+ lines = content.split("\n")
+
+ for line in lines:
+ line = line.strip()
+
+ # Check for role prefixes
+ if line.startswith("System:"):
+ if current_role and current_content:
+ messages.append(
+ self._create_message(
+ current_role, "\n".join(current_content).strip()
+ )
+ )
+ current_role = "system"
+ current_content = [line[7:].strip()] # Remove "System:" prefix
+ elif line.startswith("User:"):
+ if current_role and current_content:
+ messages.append(
+ self._create_message(
+ current_role, "\n".join(current_content).strip()
+ )
+ )
+ current_role = "user"
+ current_content = [line[5:].strip()] # Remove "User:" prefix
+ elif line.startswith("Assistant:"):
+ if current_role and current_content:
+ messages.append(
+ self._create_message(
+ current_role, "\n".join(current_content).strip()
+ )
+ )
+ current_role = "assistant"
+ current_content = [line[10:].strip()] # Remove "Assistant:" prefix
+ else:
+ # Continue current message content
+ if current_role:
+ current_content.append(line)
+ else:
+ # No role prefix found, treat as user message
+ current_role = "user"
+ current_content = [line]
+
+ # Add the last message
+ if current_role and current_content:
+ content_text = "\n".join(current_content).strip()
+ if content_text: # Only add if there's actual content
+ messages.append(self._create_message(current_role, content_text))
+
+ # If no messages were created, treat the entire content as a user message
+ if not messages and content:
+ messages.append(self._create_message("user", content))
+
+ return messages
+
+ def _create_message(self, role: str, content: str) -> AllMessageValues:
+ """Create a message with the specified role and content."""
+ return {
+ "role": role, # type: ignore
+ "content": content,
+ }
+
+ def _extract_optional_params(self, template: PromptTemplate) -> dict:
+ """
+ Extract optional parameters from the prompt template metadata.
+
+ Includes parameters like temperature, max_tokens, etc.
+ """
+ optional_params = {}
+
+ # Extract common parameters from metadata
+ if template.optional_params is not None:
+ optional_params.update(template.optional_params)
+
+ return optional_params
+
+ def set_prompt_directory(self, prompt_directory: str) -> None:
+ """Set the prompt directory and reload prompts."""
+ self.prompt_directory = prompt_directory
+ self._prompt_manager = None # Reset to force reload
+
+ def reload_prompts(self) -> None:
+ """Reload all prompts from the directory."""
+ if self._prompt_manager:
+ self._prompt_manager.reload_prompts()
+
+ def add_prompt_from_json(self, prompt_id: str, json_data: Dict[str, Any]) -> None:
+ """Add a prompt from JSON data."""
+ content = json_data.get("content", "")
+ metadata = json_data.get("metadata", {})
+ self.prompt_manager.add_prompt(prompt_id, content, metadata)
+
+ def load_prompts_from_json(self, prompts_data: Dict[str, Dict[str, Any]]) -> None:
+ """Load multiple prompts from JSON data."""
+ self.prompt_manager.load_prompts_from_json_data(prompts_data)
+
+ def get_prompts_as_json(self) -> Dict[str, Dict[str, Any]]:
+ """Get all prompts in JSON format."""
+ return self.prompt_manager.get_all_prompts_as_json()
+
+ def convert_prompt_file_to_json(self, file_path: str) -> Dict[str, Any]:
+ """Convert a .prompt file to JSON format."""
+ return self.prompt_manager.prompt_file_to_json(file_path)
diff --git a/litellm/integrations/dotprompt/prompt_manager.py b/litellm/integrations/dotprompt/prompt_manager.py
new file mode 100644
index 00000000000..9623ddab5fb
--- /dev/null
+++ b/litellm/integrations/dotprompt/prompt_manager.py
@@ -0,0 +1,343 @@
+"""
+Based on Google's GenAI Kit dotprompt implementation: https://google.github.io/dotprompt/reference/frontmatter/
+"""
+
+import re
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+import yaml
+from jinja2 import DictLoader, Environment, select_autoescape
+
+
+class PromptTemplate:
+ """Represents a single prompt template with metadata and content."""
+
+ def __init__(
+ self,
+ content: str,
+ metadata: Optional[Dict[str, Any]] = None,
+ template_id: Optional[str] = None,
+ ):
+ self.content = content
+ self.metadata = metadata or {}
+ self.template_id = template_id
+
+ # Extract common metadata fields
+ restricted_keys = ["model", "input", "output"]
+ self.model = self.metadata.get("model")
+ self.input_schema = self.metadata.get("input", {}).get("schema", {})
+ self.output_format = self.metadata.get("output", {}).get("format")
+ self.output_schema = self.metadata.get("output", {}).get("schema", {})
+ self.optional_params = {}
+ for key in self.metadata.keys():
+ if key not in restricted_keys:
+ self.optional_params[key] = self.metadata[key]
+
+ def __repr__(self):
+ return f"PromptTemplate(id='{self.template_id}', model='{self.model}')"
+
+
+class PromptManager:
+ """
+ Manager for loading and rendering .prompt files following the Dotprompt specification.
+
+ Supports:
+ - YAML frontmatter for metadata
+ - Handlebars-style templating (using Jinja2)
+ - Input/output schema validation
+ - Model configuration
+ """
+
+ def __init__(
+ self,
+ prompt_id: Optional[str] = None,
+ prompt_directory: Optional[str] = None,
+ prompt_data: Optional[Dict[str, Dict[str, Any]]] = None,
+ prompt_file: Optional[str] = None,
+ ):
+ self.prompt_directory = Path(prompt_directory) if prompt_directory else None
+ self.prompts: Dict[str, PromptTemplate] = {}
+ self.prompt_file = prompt_file
+ self.jinja_env = Environment(
+ loader=DictLoader({}),
+ autoescape=select_autoescape(["html", "xml"]),
+ # Use Handlebars-style delimiters to match Dotprompt spec
+ variable_start_string="{{",
+ variable_end_string="}}",
+ block_start_string="{%",
+ block_end_string="%}",
+ comment_start_string="{#",
+ comment_end_string="#}",
+ )
+
+ # Load prompts from directory if provided
+ if self.prompt_directory:
+ self._load_prompts()
+
+ if self.prompt_file:
+ if not prompt_id:
+ raise ValueError("prompt_id is required when prompt_file is provided")
+
+ template = self._load_prompt_file(self.prompt_file, prompt_id)
+ self.prompts[prompt_id] = template
+
+ # Load prompts from JSON data if provided
+ if prompt_data:
+ self._load_prompts_from_json(prompt_data, prompt_id)
+
+ def _load_prompts(self) -> None:
+ """Load all .prompt files from the prompt directory."""
+ if not self.prompt_directory or not self.prompt_directory.exists():
+ raise ValueError(
+ f"Prompt directory does not exist: {self.prompt_directory}"
+ )
+
+ prompt_files = list(self.prompt_directory.glob("*.prompt"))
+
+ for prompt_file in prompt_files:
+ try:
+ prompt_id = prompt_file.stem # filename without extension
+ template = self._load_prompt_file(prompt_file, prompt_id)
+ self.prompts[prompt_id] = template
+ # Optional: print(f"Loaded prompt: {prompt_id}")
+ except Exception:
+ # Optional: print(f"Error loading prompt file {prompt_file}")
+ pass
+
+ def _load_prompts_from_json(
+ self, prompt_data: Dict[str, Dict[str, Any]], prompt_id: Optional[str] = None
+ ) -> None:
+ """Load prompts from JSON data structure.
+
+ Expected format:
+ {
+ "prompt_id": {
+ "content": "template content",
+ "metadata": {"model": "gpt-4", "temperature": 0.7, ...}
+ }
+ }
+
+ or
+
+ {
+ "content": "template content",
+ "metadata": {"model": "gpt-4", "temperature": 0.7, ...}
+ } + prompt_id
+ """
+ if prompt_id:
+ prompt_data = {prompt_id: prompt_data}
+
+ for prompt_id, prompt_info in prompt_data.items():
+ try:
+ content = prompt_info.get("content", "")
+ metadata = prompt_info.get("metadata", {})
+
+ template = PromptTemplate(
+ content=content,
+ metadata=metadata,
+ template_id=prompt_id,
+ )
+ self.prompts[prompt_id] = template
+ except Exception:
+ # Optional: print(f"Error loading prompt from JSON: {prompt_id}")
+ pass
+
+ def _load_prompt_file(
+ self, file_path: Union[str, Path], prompt_id: str
+ ) -> PromptTemplate:
+ """Load and parse a single .prompt file."""
+ if isinstance(file_path, str):
+ file_path = Path(file_path)
+
+ content = file_path.read_text(encoding="utf-8")
+
+ # Split frontmatter and content
+ frontmatter, template_content = self._parse_frontmatter(content)
+
+ return PromptTemplate(
+ content=template_content.strip(),
+ metadata=frontmatter,
+ template_id=prompt_id,
+ )
+
+ def _parse_frontmatter(self, content: str) -> Tuple[Dict[str, Any], str]:
+ """Parse YAML frontmatter from prompt content."""
+ # Match YAML frontmatter between --- delimiters
+ frontmatter_pattern = r"^---\s*\n(.*?)\n---\s*\n(.*)$"
+ match = re.match(frontmatter_pattern, content, re.DOTALL)
+
+ if match:
+ frontmatter_yaml = match.group(1)
+ template_content = match.group(2)
+
+ try:
+ frontmatter = yaml.safe_load(frontmatter_yaml) or {}
+ except yaml.YAMLError as e:
+ raise ValueError(f"Invalid YAML frontmatter: {e}")
+ else:
+ # No frontmatter found, treat entire content as template
+ frontmatter = {}
+ template_content = content
+
+ return frontmatter, template_content
+
+ def render(
+ self, prompt_id: str, prompt_variables: Optional[Dict[str, Any]] = None
+ ) -> str:
+ """
+ Render a prompt template with the given variables.
+
+ Args:
+ prompt_id: The ID of the prompt template to render
+ prompt_variables: Variables to substitute in the template
+
+ Returns:
+ The rendered prompt string
+
+ Raises:
+ KeyError: If prompt_id is not found
+ ValueError: If template rendering fails
+ """
+ if prompt_id not in self.prompts:
+ available_prompts = list(self.prompts.keys())
+ raise KeyError(
+ f"Prompt '{prompt_id}' not found. Available prompts: {available_prompts}"
+ )
+
+ template = self.prompts[prompt_id]
+ variables = prompt_variables or {}
+
+ # Validate input variables against schema if defined
+ if template.input_schema:
+ self._validate_input(variables, template.input_schema)
+
+ try:
+ # Create Jinja2 template and render
+ jinja_template = self.jinja_env.from_string(template.content)
+ rendered = jinja_template.render(**variables)
+ return rendered
+ except Exception as e:
+ raise ValueError(f"Error rendering template '{prompt_id}': {e}")
+
+ def _validate_input(
+ self, variables: Dict[str, Any], schema: Dict[str, Any]
+ ) -> None:
+ """Basic validation of input variables against schema."""
+ for field_name, field_type in schema.items():
+ if field_name in variables:
+ value = variables[field_name]
+ expected_type = self._get_python_type(field_type)
+
+ if not isinstance(value, expected_type):
+ raise ValueError(
+ f"Invalid type for field '{field_name}': "
+ f"expected {getattr(expected_type, '__name__', str(expected_type))}, got {type(value).__name__}"
+ )
+
+ def _get_python_type(self, schema_type: str) -> Union[type, tuple]:
+ """Convert schema type string to Python type."""
+ type_mapping: Dict[str, Union[type, tuple]] = {
+ "string": str,
+ "str": str,
+ "number": (int, float),
+ "integer": int,
+ "int": int,
+ "float": float,
+ "boolean": bool,
+ "bool": bool,
+ "array": list,
+ "list": list,
+ "object": dict,
+ "dict": dict,
+ }
+
+ return type_mapping.get(schema_type.lower(), str) # type: ignore
+
+ def get_prompt(self, prompt_id: str) -> Optional[PromptTemplate]:
+ """Get a prompt template by ID."""
+ return self.prompts.get(prompt_id)
+
+ def list_prompts(self) -> List[str]:
+ """Get a list of all available prompt IDs."""
+ return list(self.prompts.keys())
+
+ def get_prompt_metadata(self, prompt_id: str) -> Optional[Dict[str, Any]]:
+ """Get metadata for a specific prompt."""
+ template = self.prompts.get(prompt_id)
+ return template.metadata if template else None
+
+ def reload_prompts(self) -> None:
+ """Reload all prompts from the directory (if directory was provided)."""
+ self.prompts.clear()
+ if self.prompt_directory:
+ self._load_prompts()
+
+ def add_prompt(
+ self, prompt_id: str, content: str, metadata: Optional[Dict[str, Any]] = None
+ ) -> None:
+ """Add a prompt template programmatically."""
+ template = PromptTemplate(
+ content=content, metadata=metadata or {}, template_id=prompt_id
+ )
+ self.prompts[prompt_id] = template
+
+ def prompt_file_to_json(self, file_path: Union[str, Path]) -> Dict[str, Any]:
+ """Convert a .prompt file to JSON format.
+
+ Args:
+ file_path: Path to the .prompt file
+
+ Returns:
+ Dictionary with 'content' and 'metadata' keys
+ """
+ file_path = Path(file_path)
+ content = file_path.read_text(encoding="utf-8")
+
+ # Parse frontmatter and content
+ frontmatter, template_content = self._parse_frontmatter(content)
+
+ return {"content": template_content.strip(), "metadata": frontmatter}
+
+ def json_to_prompt_file(self, prompt_data: Dict[str, Any]) -> str:
+ """Convert JSON prompt data to .prompt file format.
+
+ Args:
+ prompt_data: Dictionary with 'content' and 'metadata' keys
+
+ Returns:
+ String content in .prompt file format
+ """
+ content = prompt_data.get("content", "")
+ metadata = prompt_data.get("metadata", {})
+
+ if not metadata:
+ # No metadata, return just the content
+ return content
+
+ # Convert metadata to YAML frontmatter
+ import yaml
+
+ frontmatter_yaml = yaml.dump(metadata, default_flow_style=False)
+
+ return f"---\n{frontmatter_yaml}---\n{content}"
+
+ def get_all_prompts_as_json(self) -> Dict[str, Dict[str, Any]]:
+ """Get all loaded prompts in JSON format.
+
+ Returns:
+ Dictionary mapping prompt_id to prompt data
+ """
+ result = {}
+ for prompt_id, template in self.prompts.items():
+ result[prompt_id] = {
+ "content": template.content,
+ "metadata": template.metadata,
+ }
+ return result
+
+ def load_prompts_from_json_data(
+ self, prompt_data: Dict[str, Dict[str, Any]]
+ ) -> None:
+ """Load additional prompts from JSON data (merges with existing prompts)."""
+ self._load_prompts_from_json(prompt_data)
diff --git a/litellm/integrations/dynamodb.py b/litellm/integrations/dynamodb.py
index 2c527ea8aa9..dfc05ae1f32 100644
--- a/litellm/integrations/dynamodb.py
+++ b/litellm/integrations/dynamodb.py
@@ -3,7 +3,7 @@
import os
import traceback
-import uuid
+from litellm._uuid import uuid
from typing import Any
import litellm
diff --git a/litellm/integrations/gcs_bucket/gcs_bucket.py b/litellm/integrations/gcs_bucket/gcs_bucket.py
index 972a0236666..9190f921d50 100644
--- a/litellm/integrations/gcs_bucket/gcs_bucket.py
+++ b/litellm/integrations/gcs_bucket/gcs_bucket.py
@@ -1,7 +1,7 @@
import asyncio
import json
import os
-import uuid
+from litellm._uuid import uuid
from datetime import datetime, timedelta, timezone
from typing import TYPE_CHECKING, Any, Dict, List, Optional
from urllib.parse import quote
diff --git a/litellm/integrations/gitlab/README.md b/litellm/integrations/gitlab/README.md
new file mode 100644
index 00000000000..14fb62905c8
--- /dev/null
+++ b/litellm/integrations/gitlab/README.md
@@ -0,0 +1,317 @@
+# LiteLLM gitlab Prompt Management
+
+A powerful prompt management system for LiteLLM that fetches `.prompt` files from gitlab repositories. This enables team-based prompt management with gitlab's built-in access control and version control capabilities.
+
+## Features
+
+- **🏢 Team-based access control**: Leverage gitlab's workspace and repository permissions
+- **📁 Repository-based prompt storage**: Store prompts in gitlab repositories
+- **🔐 Multiple authentication methods**: Support for access tokens and basic auth
+- **🎯 YAML frontmatter**: Define model, parameters, and schemas in file headers
+- **🔧 Handlebars templating**: Use `{{variable}}` syntax with Jinja2 backend
+- **✅ Input validation**: Automatic validation against defined schemas
+- **🔗 LiteLLM integration**: Works seamlessly with `litellm.completion()`
+- **💬 Smart message parsing**: Converts prompts to proper chat messages
+- **⚙️ Parameter extraction**: Automatically applies model settings from prompts
+
+## Quick Start
+
+### 1. Set up gitlab Repository
+
+Create a repository in your gitlab workspace and add `.prompt` files:
+
+```
+your-repo/
+├── prompts/
+│ ├── chat_assistant.prompt
+│ ├── code_reviewer.prompt
+│ └── data_analyst.prompt
+```
+
+### 2. Create a `.prompt` file
+
+Create a file called `prompts/chat_assistant.prompt`:
+
+```yaml
+---
+model: gpt-4
+temperature: 0.7
+max_tokens: 150
+input:
+ schema:
+ user_message: string
+ system_context?: string
+---
+
+{% if system_context %}System: {{system_context}}
+
+{% endif %}User: {{user_message}}
+```
+
+### 3. Configure gitlab Access
+
+#### Option A: Access Token (Recommended)
+
+```python
+import litellm
+
+# Configure gitlab access
+gitlab_config = {
+ "project": "a/b/",
+ "access_token": "your-access-token",
+ "base_url": "gitlab url",
+ "prompts_path": "src/prompts", # folder to point to, defaults to root
+ "branch":"main" # optional, defaults to main
+}
+
+# Set global gitlab configuration
+litellm.set_global_gitlab_config(gitlab_config)
+```
+
+#### Option B: Basic Authentication
+
+```python
+import litellm
+
+# Configure gitlab access with basic auth
+gitlab_config = {
+ "project": "a/b/",
+ "base_url": "base url",
+ "access_token": "your-app-password", # Use app password for basic auth
+ "branch": "main",
+ "prompts_path": "src/prompts", # folder to point to, defaults to root
+}
+
+litellm.set_global_gitlab_config(gitlab_config)
+```
+
+### 4. Use with LiteLLM
+
+```python
+# Use with completion - the model prefix 'gitlab/' tells LiteLLM to use gitlab prompt management
+response = litellm.completion(
+ model="gitlab/gpt-4", # The actual model comes from the .prompt file
+ prompt_id="prompts/chat_assistant", # Location of the prompt file
+ prompt_variables={
+ "user_message": "What is machine learning?",
+ "system_context": "You are a helpful AI tutor."
+ },
+ # Any additional messages will be appended after the prompt
+ messages=[{"role": "user", "content": "Please explain it simply."}]
+)
+
+print(response.choices[0].message.content)
+```
+
+## Proxy Server Configuration
+
+### 1. Create a `.prompt` file
+
+Create `prompts/hello.prompt`:
+
+```yaml
+---
+model: gpt-4
+temperature: 0.7
+---
+System: You are a helpful assistant.
+
+User: {{user_message}}
+```
+
+### 2. Setup config.yaml
+
+```yaml
+model_list:
+ - model_name: my-gitlab-model
+ litellm_params:
+ model: gitlab/gpt-4
+ prompt_id: "prompts/hello"
+ api_key: os.environ/OPENAI_API_KEY
+
+litellm_settings:
+ global_gitlab_config:
+ workspace: "your-workspace"
+ repository: "your-repo"
+ access_token: "your-access-token"
+ branch: "main"
+```
+
+### 3. Start the proxy
+
+```bash
+litellm --config config.yaml --detailed_debug
+```
+
+### 4. Test it!
+
+```bash
+curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
+-H 'Content-Type: application/json' \
+-H 'Authorization: Bearer sk-1234' \
+-d '{
+ "model": "my-gitlab-model",
+ "messages": [{"role": "user", "content": "IGNORED"}],
+ "prompt_variables": {
+ "user_message": "What is the capital of France?"
+ }
+}'
+```
+
+## Prompt File Format
+
+### Basic Structure
+
+```yaml
+---
+# Model configuration
+model: gpt-4
+temperature: 0.7
+max_tokens: 500
+
+# Input schema (optional)
+input:
+ schema:
+ user_message: string
+ system_context?: string
+---
+
+System: You are a helpful {{role}} assistant.
+
+User: {{user_message}}
+```
+
+### Advanced Features
+
+**Multi-role conversations:**
+
+```yaml
+---
+model: gpt-4
+temperature: 0.3
+---
+System: You are a helpful coding assistant.
+
+User: {{user_question}}
+```
+
+**Dynamic model selection:**
+
+```yaml
+---
+model: "{{preferred_model}}" # Model can be a variable
+temperature: 0.7
+---
+System: You are a helpful assistant specialized in {{domain}}.
+
+User: {{user_message}}
+```
+
+## Team-Based Access Control
+
+gitlab's built-in permission system provides team-based access control:
+
+1. **Workspace-level permissions**: Control access to entire workspaces
+2. **Repository-level permissions**: Control access to specific repositories
+3. **Branch-level permissions**: Control access to specific branches
+4. **User and group management**: Manage team members and their access levels
+
+### Setting up Team Access
+
+1. **Create workspaces for each team**:
+ ```
+ team-a-prompts/
+ team-b-prompts/
+ team-c-prompts/
+ ```
+
+2. **Configure repository permissions**:
+ - Grant read access to team members
+ - Grant write access to prompt maintainers
+ - Use branch protection rules for production prompts
+
+3. **Use different access tokens**:
+ - Each team can have their own access token
+ - Tokens can be scoped to specific repositories
+ - Use app passwords for additional security
+
+## API Reference
+
+### gitlab Configuration
+
+```python
+gitlab_config = {
+ "workspace": str, # Required: gitlab workspace name
+ "repository": str, # Required: Repository name
+ "access_token": str, # Required: gitlab access token or app password
+ "branch": str, # Optional: Branch to fetch from (default: "main")
+ "base_url": str, # Optional: Custom gitlab API URL
+ "auth_method": str, # Optional: "token" or "basic" (default: "token")
+ "username": str, # Optional: Username for basic auth
+ "base_url" : str # Optional: Incase where the base url is not https://api.gitlab.org/2.0
+}
+```
+
+### LiteLLM Integration
+
+```python
+response = litellm.completion(
+ model="gitlab/", # required (e.g., gitlab/gpt-4)
+ prompt_id=str, # required - the .prompt filename without extension
+ prompt_variables=dict, # optional - variables for template rendering
+ gitlab_config=dict, # optional - gitlab configuration (if not set globally)
+ messages=list, # optional - additional messages
+)
+```
+
+## Error Handling
+
+The gitlab integration provides detailed error messages for common issues:
+
+- **Authentication errors**: Invalid access tokens or credentials
+- **Permission errors**: Insufficient access to workspace/repository
+- **File not found**: Missing .prompt files
+- **Network errors**: Connection issues with gitlab API
+
+## Security Considerations
+
+1. **Access Token Security**: Store access tokens securely using environment variables or secret management systems
+2. **Repository Permissions**: Use gitlab's permission system to control access
+3. **Branch Protection**: Protect main branches from unauthorized changes
+4. **Audit Logging**: gitlab provides audit logs for all repository access
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"Access denied" errors**: Check your gitlab permissions for the workspace and repository
+2. **"Authentication failed" errors**: Verify your access token or credentials
+3. **"File not found" errors**: Ensure the .prompt file exists in the specified branch
+4. **Template rendering errors**: Check your Handlebars syntax in the .prompt file
+
+### Debug Mode
+
+Enable debug logging to troubleshoot issues:
+
+```python
+import litellm
+litellm.set_verbose = True
+
+# Your gitlab prompt calls will now show detailed logs
+response = litellm.completion(
+ model="gitlab/gpt-4",
+ prompt_id="your_prompt",
+ prompt_variables={"key": "value"}
+)
+```
+
+## Migration from File-Based Prompts
+
+If you're currently using file-based prompts with the dotprompt integration, you can easily migrate to gitlab:
+
+1. **Upload your .prompt files** to a gitlab repository
+2. **Update your configuration** to use gitlab instead of local files
+3. **Set up team access** using gitlab's permission system
+4. **Update your code** to use `gitlab/` model prefix instead of `dotprompt/`
+
+This provides better collaboration, version control, and team-based access control for your prompts.
diff --git a/litellm/integrations/gitlab/__init__.py b/litellm/integrations/gitlab/__init__.py
new file mode 100644
index 00000000000..cd22afc2ba0
--- /dev/null
+++ b/litellm/integrations/gitlab/__init__.py
@@ -0,0 +1,95 @@
+from typing import TYPE_CHECKING, Optional, Dict, Any
+
+if TYPE_CHECKING:
+ from .gitlab_prompt_manager import GitLabPromptManager
+ from litellm.types.prompts.init_prompts import PromptLiteLLMParams, PromptSpec
+ from litellm.integrations.custom_prompt_management import CustomPromptManagement
+
+from litellm.types.prompts.init_prompts import SupportedPromptIntegrations
+from litellm.integrations.custom_prompt_management import CustomPromptManagement
+from litellm.types.prompts.init_prompts import PromptSpec, PromptLiteLLMParams
+from .gitlab_prompt_manager import GitLabPromptManager
+
+# Global instances
+global_gitlab_config: Optional[dict] = None
+
+
+def set_global_gitlab_config(config: dict) -> None:
+ """
+ Set the global BitBucket configuration for prompt management.
+
+ Args:
+ config: Dictionary containing BitBucket configuration
+ - workspace: BitBucket workspace name
+ - repository: Repository name
+ - access_token: BitBucket access token
+ - branch: Branch to fetch prompts from (default: main)
+ """
+ import litellm
+
+ litellm.global_gitlab_config = config # type: ignore
+
+
+def prompt_initializer(
+ litellm_params: "PromptLiteLLMParams", prompt_spec: "PromptSpec"
+) -> "CustomPromptManagement":
+ """
+ Initialize a prompt from a BitBucket repository.
+ """
+ gitlab_config = getattr(litellm_params, "gitlab_config", None)
+ prompt_id = getattr(litellm_params, "prompt_id", None)
+
+
+ if not gitlab_config:
+ raise ValueError(
+ "bitbucket_config is required for BitBucket prompt integration"
+ )
+
+ try:
+ bitbucket_prompt_manager = GitLabPromptManager(
+ gitlab_config=gitlab_config,
+ prompt_id=prompt_id,
+ )
+
+ return bitbucket_prompt_manager
+ except Exception as e:
+ raise e
+
+def _gitlab_prompt_initializer(
+ litellm_params: PromptLiteLLMParams,
+ prompt: PromptSpec,
+) -> CustomPromptManagement:
+ """
+ Build a GitLab-backed prompt manager for this prompt.
+ Expected fields on litellm_params:
+ - prompt_integration="gitlab" (handled by the caller)
+ - gitlab_config: Dict[str, Any] (project/access_token/branch/prompts_path/etc.)
+ - git_ref (optional): per-prompt tag/branch/SHA override
+ """
+ # You can store arbitrary integration-specific config on PromptLiteLLMParams.
+ # If your dataclass doesn't have these attributes, add them or put inside
+ # `litellm_params.extra` and pull them from there.
+ gitlab_config: Dict[str, Any] = getattr(litellm_params, "gitlab_config", None) or {}
+ git_ref: Optional[str] = getattr(litellm_params, "git_ref", None)
+
+ if not gitlab_config:
+ raise ValueError("gitlab_config is required for gitlab prompt integration")
+
+ # prompt.prompt_id can map to a file path under prompts_path (e.g. "chat/greet/hi")
+ return GitLabPromptManager(
+ gitlab_config=gitlab_config,
+ prompt_id=prompt.prompt_id,
+ ref=git_ref,
+ )
+
+
+prompt_initializer_registry = {
+ SupportedPromptIntegrations.GITLAB.value: _gitlab_prompt_initializer,
+}
+
+# Export public API
+__all__ = [
+ "GitLabPromptManager",
+ "set_global_gitlab_config",
+ "global_gitlab_config",
+]
diff --git a/litellm/integrations/gitlab/gitlab_client.py b/litellm/integrations/gitlab/gitlab_client.py
new file mode 100644
index 00000000000..ce03a35d48e
--- /dev/null
+++ b/litellm/integrations/gitlab/gitlab_client.py
@@ -0,0 +1,285 @@
+"""
+GitLab API client for fetching files from GitLab repositories.
+Now supports selecting a tag via `config["tag"]`; falls back to branch ("main").
+"""
+
+import base64
+from typing import Any, Dict, List, Optional
+from urllib.parse import quote
+
+from litellm.llms.custom_httpx.http_handler import HTTPHandler
+
+
+class GitLabClient:
+ """
+ Client for interacting with the GitLab API to fetch files.
+
+ Supports:
+ - Authentication with personal/access tokens or OAuth bearer tokens
+ - Fetching file contents from repositories (raw endpoint with JSON fallback)
+ - Namespace/project path or numeric project ID addressing
+ - Ref selection via tag (preferred) or branch (default "main")
+ - Directory listing via the repository tree API
+ """
+
+ def __init__(self, config: Dict[str, Any]):
+ """
+ Initialize the GitLab client.
+
+ Args:
+ config: Dictionary containing:
+ - project: Project path ("group/subgroup/repo") or numeric project ID (str|int) [required]
+ - access_token: GitLab personal/access token or OAuth token [required] (str)
+ - auth_method: 'token' (default; sends Private-Token) or 'oauth' (Authorization: Bearer)
+ - tag: Tag name to fetch from (takes precedence over branch if provided)
+ - branch: Branch to fetch from (default: "main")
+ - base_url: Base GitLab API URL (default: "https://gitlab.com/api/v4")
+ """
+ project = config.get("project")
+ access_token = config.get("access_token")
+ if project is None or access_token is None:
+ raise ValueError("project and access_token are required")
+
+ self.project: str | int = project
+ self.access_token: str = str(access_token)
+ self.auth_method = config.get("auth_method", "token") # 'token' or 'oauth'
+ self.branch = config.get("branch", None)
+ if not self.branch:
+ self.branch = 'main'
+ self.tag = config.get("tag")
+ self.base_url = config.get("base_url", "https://gitlab.com/api/v4")
+
+ if not all([self.project, self.access_token]):
+ raise ValueError("project and access_token are required")
+
+ # Effective ref: prefer tag if provided, else branch ("main")
+ self.ref = str(self.tag or self.branch)
+
+ # Build headers
+ self.headers = {
+ "Accept": "application/json",
+ "Content-Type": "application/json",
+ }
+ if self.auth_method == "oauth":
+ self.headers["Authorization"] = f"Bearer {self.access_token}"
+ else:
+ # Default GitLab token header
+ self.headers["Private-Token"] = self.access_token
+
+ # Project identifier must be URL-encoded (slashes become %2F)
+ self._project_enc = quote(str(self.project), safe="")
+
+ # HTTP handler
+ self.http_handler = HTTPHandler()
+
+ # ------------------------
+ # Core helpers
+ # ------------------------
+
+ def _file_raw_url(self, file_path: str, *, ref: Optional[str] = None) -> str:
+ file_enc = quote(file_path, safe="")
+ ref_q = quote(ref or self.ref, safe="")
+ return f"{self.base_url}/projects/{self._project_enc}/repository/files/{file_enc}/raw?ref={ref_q}"
+
+ def _file_json_url(self, file_path: str, *, ref: Optional[str] = None) -> str:
+ file_enc = quote(file_path, safe="")
+ ref_q = quote(ref or self.ref, safe="")
+ return f"{self.base_url}/projects/{self._project_enc}/repository/files/{file_enc}?ref={ref_q}"
+
+ def _tree_url(self, directory_path: str = "", recursive: bool = False, *, ref: Optional[str] = None) -> str:
+ path_q = f"&path={quote(directory_path, safe='')}" if directory_path else ""
+ rec_q = "&recursive=true" if recursive else ""
+ ref_q = quote(ref or self.ref, safe="")
+ return f"{self.base_url}/projects/{self._project_enc}/repository/tree?ref={ref_q}{path_q}{rec_q}"
+
+ # ------------------------
+ # Public API
+ # ------------------------
+
+ def set_ref(self, ref: str) -> None:
+ """Override the default ref (tag/branch) for subsequent calls."""
+ if not ref:
+ raise ValueError("ref must be a non-empty string")
+ self.ref = ref
+
+ def get_file_content(self, file_path: str, *, ref: Optional[str] = None) -> Optional[str]:
+ """
+ Fetch the content of a file from the GitLab repository at the given ref
+ (tag, branch, or commit SHA). If `ref` is None, uses self.ref.
+
+ Strategy:
+ 1) Try the RAW endpoint (returns bytes of the file)
+ 2) Fallback to the JSON endpoint (returns base64-encoded content)
+
+ Returns:
+ File content as UTF-8 string, or None if file not found.
+ """
+ raw_url = self._file_raw_url(file_path, ref=ref)
+
+ try:
+ resp = self.http_handler.get(raw_url, headers=self.headers)
+ if resp.status_code == 404:
+ # Fallback to JSON endpoint
+ return self._get_file_content_via_json(file_path, ref=ref)
+ resp.raise_for_status()
+
+ ctype = (resp.headers.get("content-type") or "").lower()
+ if ctype.startswith("text/") or "charset=" in ctype or ctype.startswith("application/json"):
+ return resp.text
+ try:
+ return resp.content.decode("utf-8")
+ except Exception:
+ return resp.content.decode("utf-8", errors="replace")
+
+ except Exception as e:
+ status = getattr(getattr(e, "response", None), "status_code", None)
+ if status == 404:
+ return None
+ if status == 403:
+ raise Exception(
+ f"Access denied to file '{file_path}'. Check your GitLab permissions for project '{self.project}'."
+ )
+ if status == 401:
+ raise Exception("Authentication failed. Check your GitLab token and auth_method.")
+ raise Exception(f"Failed to fetch file '{file_path}': {e}")
+
+ def _get_file_content_via_json(self, file_path: str, *, ref: Optional[str] = None) -> Optional[str]:
+ """
+ Fallback for get_file_content(): use the JSON file API which returns base64 content.
+ """
+ json_url = self._file_json_url(file_path, ref=ref)
+ try:
+ resp = self.http_handler.get(json_url, headers=self.headers)
+ if resp.status_code == 404:
+ return None
+ resp.raise_for_status()
+ data = resp.json()
+ content = data.get("content")
+ encoding = data.get("encoding", "")
+ if content and encoding == "base64":
+ try:
+ return base64.b64decode(content).decode("utf-8")
+ except Exception:
+ return base64.b64decode(content).decode("utf-8", errors="replace")
+ return content
+ except Exception as e:
+ status = getattr(getattr(e, "response", None), "status_code", None)
+ if status == 404:
+ return None
+ if status == 403:
+ raise Exception(
+ f"Access denied to file '{file_path}'. Check your GitLab permissions for project '{self.project}'."
+ )
+ if status == 401:
+ raise Exception("Authentication failed. Check your GitLab token and auth_method.")
+ raise Exception(f"Failed to fetch file '{file_path}' via JSON endpoint: {e}")
+
+ def list_files(
+ self,
+ directory_path: str = "",
+ file_extension: str = ".prompt",
+ recursive: bool = False,
+ *,
+ ref: Optional[str] = None,
+ ) -> List[str]:
+ """
+ List files in a directory with a specific extension using the repository tree API.
+
+ Args:
+ directory_path: Directory path in the repository (empty for repo root)
+ file_extension: File extension to filter by (default: .prompt)
+ recursive: If True, traverses subdirectories
+ ref: Optional override (tag/branch/SHA). Defaults to self.ref.
+
+ Returns:
+ List of file paths (relative to repo root)
+ """
+ url = self._tree_url(directory_path, recursive=recursive, ref=ref)
+
+ try:
+ resp = self.http_handler.get(url, headers=self.headers)
+ if resp.status_code == 404:
+ return []
+ resp.raise_for_status()
+
+ data = resp.json() or []
+ files: List[str] = []
+ for item in data:
+ if item.get("type") == "blob":
+ file_path = item.get("path", "")
+ if not file_extension or file_path.endswith(file_extension):
+ files.append(file_path)
+ return files
+
+ except Exception as e:
+ status = getattr(getattr(e, "response", None), "status_code", None)
+ if status == 404:
+ return []
+ if status == 403:
+ raise Exception(
+ f"Access denied to directory '{directory_path}'. Check your GitLab permissions for project '{self.project}'."
+ )
+ if status == 401:
+ raise Exception("Authentication failed. Check your GitLab token and auth_method.")
+ raise Exception(f"Failed to list files in '{directory_path}': {e}")
+
+ def get_repository_info(self) -> Dict[str, Any]:
+ """Get information about the project/repository."""
+ url = f"{self.base_url}/projects/{self._project_enc}"
+ try:
+ resp = self.http_handler.get(url, headers=self.headers)
+ resp.raise_for_status()
+ return resp.json()
+ except Exception as e:
+ raise Exception(f"Failed to get repository info: {e}")
+
+ def test_connection(self) -> bool:
+ """Test the connection to the GitLab project."""
+ try:
+ self.get_repository_info()
+ return True
+ except Exception:
+ return False
+
+ def get_branches(self) -> List[Dict[str, Any]]:
+ """Get list of branches in the repository."""
+ url = f"{self.base_url}/projects/{self._project_enc}/repository/branches"
+ try:
+ resp = self.http_handler.get(url, headers=self.headers)
+ resp.raise_for_status()
+ data = resp.json()
+ return data if isinstance(data, list) else []
+ except Exception as e:
+ raise Exception(f"Failed to get branches: {e}")
+
+ def get_file_metadata(self, file_path: str, *, ref: Optional[str] = None) -> Optional[Dict[str, Any]]:
+ """
+ Get minimal metadata about a file via RAW endpoint headers at a given ref.
+
+ Args:
+ file_path: Path to the file in the repository.
+ ref: Optional override (tag/branch/SHA). Defaults to self.ref.
+ """
+ url = self._file_raw_url(file_path, ref=ref)
+ try:
+ headers = dict(self.headers)
+ headers["Range"] = "bytes=0-0"
+ resp = self.http_handler.get(url, headers=headers)
+ if resp.status_code == 404:
+ return None
+ resp.raise_for_status()
+ return {
+ "content_type": resp.headers.get("content-type"),
+ "content_length": resp.headers.get("content-length"),
+ "last_modified": resp.headers.get("last-modified"),
+ }
+ except Exception as e:
+ status = getattr(getattr(e, "response", None), "status_code", None)
+ if status == 404:
+ return None
+ raise Exception(f"Failed to get file metadata for '{file_path}': {e}")
+
+ def close(self):
+ """Close the HTTP handler to free resources."""
+ if hasattr(self, "http_handler"):
+ self.http_handler.close()
diff --git a/litellm/integrations/gitlab/gitlab_prompt_manager.py b/litellm/integrations/gitlab/gitlab_prompt_manager.py
new file mode 100644
index 00000000000..b782f10ccc5
--- /dev/null
+++ b/litellm/integrations/gitlab/gitlab_prompt_manager.py
@@ -0,0 +1,488 @@
+"""
+GitLab prompt manager with configurable prompts folder.
+"""
+
+from typing import Any, Dict, List, Optional, Tuple, Union
+from jinja2 import DictLoader, Environment, select_autoescape
+
+from litellm.integrations.custom_prompt_management import CustomPromptManagement
+from litellm.integrations.prompt_management_base import (
+ PromptManagementBase,
+ PromptManagementClient,
+)
+from litellm.types.llms.openai import AllMessageValues
+from litellm.types.utils import StandardCallbackDynamicParams
+
+from litellm.integrations.gitlab.gitlab_client import GitLabClient
+
+
+class GitLabPromptTemplate:
+ def __init__(
+ self,
+ template_id: str,
+ content: str,
+ metadata: Dict[str, Any],
+ model: Optional[str] = None,
+ ):
+ self.template_id = template_id
+ self.content = content
+ self.metadata = metadata
+ self.model = model or metadata.get("model")
+ self.temperature = metadata.get("temperature")
+ self.max_tokens = metadata.get("max_tokens")
+ self.input_schema = metadata.get("input", {}).get("schema", {})
+ self.optional_params = {
+ k: v for k, v in metadata.items() if k not in ["model", "input", "content"]
+ }
+
+ def __repr__(self):
+ return f"GitLabPromptTemplate(id='{self.template_id}', model='{self.model}')"
+
+
+class GitLabTemplateManager:
+ """
+ Manager for loading and rendering .prompt files from GitLab repositories.
+
+ New: supports `prompts_path` (or `folder`) in gitlab_config to scope where prompts live.
+ """
+
+
+ def __init__(
+ self,
+ gitlab_config: Dict[str, Any],
+ prompt_id: Optional[str] = None,
+ ref: Optional[str] = None,
+ gitlab_client: Optional[GitLabClient] = None
+ ):
+ self.gitlab_config = dict(gitlab_config)
+ self.prompt_id = prompt_id
+ self.prompts: Dict[str, GitLabPromptTemplate] = {}
+ self.gitlab_client = gitlab_client or GitLabClient(self.gitlab_config)
+
+ if ref:
+ self.gitlab_client.set_ref(ref)
+
+ # Folder inside repo to look for prompts (e.g., "prompts" or "prompts/chat")
+ self.prompts_path: str = (
+ self.gitlab_config.get("prompts_path")
+ or self.gitlab_config.get("folder")
+ or ""
+ ).strip("/")
+
+ self.jinja_env = Environment(
+ loader=DictLoader({}),
+ autoescape=select_autoescape(["html", "xml"]),
+ variable_start_string="{{",
+ variable_end_string="}}",
+ block_start_string="{%",
+ block_end_string="%}",
+ comment_start_string="{#",
+ comment_end_string="#}",
+ )
+
+ if self.prompt_id:
+ self._load_prompt_from_gitlab(self.prompt_id)
+
+ # ---------- path helpers ----------
+
+ def _id_to_repo_path(self, prompt_id: str) -> str:
+ """Map a prompt_id to a repo path (respects prompts_path and adds .prompt)."""
+ if self.prompts_path:
+ return f"{self.prompts_path}/{prompt_id}.prompt"
+ return f"{prompt_id}.prompt"
+
+ def _repo_path_to_id(self, repo_path: str) -> str:
+ """
+ Map a repo path like 'prompts/chat/greeting.prompt' to an ID relative
+ to prompts_path without the extension (e.g., 'chat/greeting').
+ """
+ path = repo_path.strip("/")
+ if self.prompts_path and path.startswith(self.prompts_path.strip("/") + "/"):
+ path = path[len(self.prompts_path.strip("/")) + 1 :]
+ if path.endswith(".prompt"):
+ path = path[: -len(".prompt")]
+ return path
+
+ # ---------- loading ----------
+
+ def _load_prompt_from_gitlab(self, prompt_id: str, *, ref: Optional[str] = None) -> None:
+ """Load a specific .prompt file from GitLab (scoped under prompts_path if set)."""
+ try:
+ file_path = self._id_to_repo_path(prompt_id)
+ prompt_content = self.gitlab_client.get_file_content(file_path, ref=ref)
+ if prompt_content:
+ template = self._parse_prompt_file(prompt_content, prompt_id)
+ self.prompts[prompt_id] = template
+ except Exception as e:
+ raise Exception(f"Failed to load prompt '{prompt_id}' from GitLab: {e}")
+
+ def load_all_prompts(self, *, recursive: bool = True) -> List[str]:
+ """
+ Eagerly load all .prompt files from prompts_path. Returns loaded IDs.
+ """
+ files = self.list_templates(recursive=recursive) # reuse logic
+ loaded: List[str] = []
+ for pid in files:
+ if pid not in self.prompts:
+ self._load_prompt_from_gitlab(pid)
+ loaded.append(pid)
+ return loaded
+
+ # ---------- parsing & rendering ----------
+
+ def _parse_prompt_file(
+ self, content: str, prompt_id: str
+ ) -> GitLabPromptTemplate:
+ if content.startswith("---"):
+ parts = content.split("---", 2)
+ if len(parts) >= 3:
+ frontmatter_str = parts[1].strip()
+ template_content = parts[2].strip()
+ else:
+ frontmatter_str = ""
+ template_content = content
+ else:
+ frontmatter_str = ""
+ template_content = content
+
+ metadata: Dict[str, Any] = {}
+ if frontmatter_str:
+ try:
+ import yaml
+ metadata = yaml.safe_load(frontmatter_str) or {}
+ except ImportError:
+ metadata = self._parse_yaml_basic(frontmatter_str)
+ except Exception:
+ metadata = {}
+
+ return GitLabPromptTemplate(
+ template_id=prompt_id,
+ content=template_content,
+ metadata=metadata,
+ )
+
+ def _parse_yaml_basic(self, yaml_str: str) -> Dict[str, Any]:
+ result: Dict[str, Any] = {}
+ for line in yaml_str.split("\n"):
+ line = line.strip()
+ if ":" in line and not line.startswith("#"):
+ key, value = line.split(":", 1)
+ key = key.strip()
+ value = value.strip()
+ if value.lower() in ["true", "false"]:
+ result[key] = value.lower() == "true"
+ elif value.isdigit():
+ result[key] = int(value)
+ elif value.replace(".", "").isdigit():
+ try:
+ result[key] = float(value)
+ except Exception:
+ result[key] = value
+ else:
+ result[key] = value.strip("\"'")
+ return result
+
+ def render_template(
+ self, template_id: str, variables: Optional[Dict[str, Any]] = None
+ ) -> str:
+ if template_id not in self.prompts:
+ raise ValueError(f"Template '{template_id}' not found")
+ template = self.prompts[template_id]
+ jinja_template = self.jinja_env.from_string(template.content)
+ return jinja_template.render(**(variables or {}))
+
+ def get_template(self, template_id: str) -> Optional[GitLabPromptTemplate]:
+ return self.prompts.get(template_id)
+
+ def list_templates(self, *, recursive: bool = True) -> List[str]:
+ """
+ List available prompt IDs discovered under prompts_path (no extension, relative to prompts_path).
+ """
+ """
+ List available prompt IDs under prompts_path (no extension).
+ Compatible with both list_files signatures:
+ - list_files(directory_path=..., file_extension=..., recursive=...)
+ - list_files(path=..., ref=None, recursive=...)
+ """
+ # First try the "new" signature (directory_path/file_extension)
+ try:
+ files = self.gitlab_client.list_files(
+ directory_path=self.prompts_path,
+ file_extension=".prompt",
+ recursive=recursive,
+ )
+ base = self.prompts_path.strip("/")
+ out: List[str] = []
+ for p in files or []:
+ path = str(p).strip("/")
+ if base and not path.startswith(base + "/"):
+ # if the client returns extra files outside the folder, skip them
+ continue
+ if not path.endswith(".prompt"):
+ continue
+ out.append(self._repo_path_to_id(path))
+ return out
+ except TypeError:
+ # Fallback to the "classic" signature
+ raw = self.gitlab_client.list_files(
+ directory_path=self.prompts_path or "",
+ ref=None,
+ recursive=recursive,
+ )
+ # Classic returns GitLab tree entries; filter *.prompt blobs
+ files = []
+ for f in (raw or []):
+ if isinstance(f, dict) and f.get("type") == "blob" and str(f.get("path", "")).endswith(".prompt") and 'path' in f:
+ files.append(f['path'])
+
+ return [self._repo_path_to_id(p) for p in files]
+
+
+class GitLabPromptManager(CustomPromptManagement):
+ """
+ GitLab prompt manager with folder support.
+
+ Example config:
+ gitlab_config = {
+ "project": "group/subgroup/repo",
+ "access_token": "glpat_***",
+ "tag": "v1.2.3", # optional; takes precedence
+ "branch": "main", # default fallback
+ "prompts_path": "prompts/chat" # <--- NEW
+ }
+ """
+
+ def __init__(
+ self,
+ gitlab_config: Dict[str, Any],
+ prompt_id: Optional[str] = None,
+ ref: Optional[str] = None, # tag/branch/SHA override
+ gitlab_client: Optional[GitLabClient] = None
+ ):
+ self.gitlab_config = gitlab_config
+ self.prompt_id = prompt_id
+ self._prompt_manager: Optional[GitLabTemplateManager] = None
+ self._ref_override = ref
+ self._injected_gitlab_client = gitlab_client
+ if self.prompt_id:
+ self._prompt_manager = GitLabTemplateManager(
+ gitlab_config=self.gitlab_config,
+ prompt_id=self.prompt_id,
+ ref=self._ref_override,
+ )
+
+ @property
+ def integration_name(self) -> str:
+ return "gitlab"
+
+ @property
+ def prompt_manager(self) -> GitLabTemplateManager:
+ if self._prompt_manager is None:
+ self._prompt_manager = GitLabTemplateManager(
+ gitlab_config=self.gitlab_config,
+ prompt_id=self.prompt_id,
+ ref=self._ref_override,
+ gitlab_client=self._injected_gitlab_client
+ )
+ return self._prompt_manager
+
+ def get_prompt_template(
+ self,
+ prompt_id: str,
+ prompt_variables: Optional[Dict[str, Any]] = None,
+ *,
+ ref: Optional[str] = None,
+ ) -> Tuple[str, Dict[str, Any]]:
+ if prompt_id not in self.prompt_manager.prompts:
+ self.prompt_manager._load_prompt_from_gitlab(prompt_id, ref=ref)
+
+ template = self.prompt_manager.get_template(prompt_id)
+ if not template:
+ raise ValueError(f"Prompt template '{prompt_id}' not found")
+
+ rendered_prompt = self.prompt_manager.render_template(
+ prompt_id, prompt_variables or {}
+ )
+
+ metadata = {
+ "model": template.model,
+ "temperature": template.temperature,
+ "max_tokens": template.max_tokens,
+ **template.optional_params,
+ }
+ return rendered_prompt, metadata
+
+ def pre_call_hook(
+ self,
+ user_id: Optional[str],
+ messages: List[AllMessageValues],
+ function_call: Optional[Union[Dict[str, Any], str]] = None,
+ litellm_params: Optional[Dict[str, Any]] = None,
+ prompt_id: Optional[str] = None,
+ prompt_variables: Optional[Dict[str, Any]] = None,
+ prompt_version: Optional[str] = None,
+ **kwargs,
+ ) -> Tuple[List[AllMessageValues], Optional[Dict[str, Any]]]:
+ if not prompt_id:
+ return messages, litellm_params
+ try:
+ # Precedence: explicit prompt_version → per-call git_ref kwarg → manager override → config default
+ git_ref = prompt_version or kwargs.get("git_ref") or self._ref_override
+
+ rendered_prompt, prompt_metadata = self.get_prompt_template(
+ prompt_id, prompt_variables, ref=git_ref
+ )
+ parsed_messages = self._parse_prompt_to_messages(rendered_prompt)
+
+ if parsed_messages:
+ final_messages: List[AllMessageValues] = parsed_messages
+ else:
+ final_messages = [{"role": "user", "content": rendered_prompt}] + messages # type: ignore
+
+ if litellm_params is None:
+ litellm_params = {}
+
+ if prompt_metadata.get("model"):
+ litellm_params["model"] = prompt_metadata["model"]
+
+ for param in ["temperature", "max_tokens", "top_p", "frequency_penalty", "presence_penalty"]:
+ if param in prompt_metadata:
+ litellm_params[param] = prompt_metadata[param]
+
+ return final_messages, litellm_params
+ except Exception as e:
+ import litellm
+ litellm._logging.verbose_proxy_logger.error(f"Error in GitLab prompt pre_call_hook: {e}")
+ return messages, litellm_params
+
+
+ def _parse_prompt_to_messages(self, prompt_content: str) -> List[AllMessageValues]:
+ messages: List[AllMessageValues] = []
+ lines = prompt_content.strip().split("\n")
+ current_role: Optional[str] = None
+ current_content: List[str] = []
+
+ for raw in lines:
+ line = raw.strip()
+ if not line:
+ continue
+ low = line.lower()
+ if low.startswith("system:"):
+ if current_role and current_content:
+ messages.append({"role": current_role, "content": "\n".join(current_content).strip()}) # type: ignore
+ current_role = "system"
+ current_content = [line[7:].strip()]
+ elif low.startswith("user:"):
+ if current_role and current_content:
+ messages.append({"role": current_role, "content": "\n".join(current_content).strip()}) # type: ignore
+ current_role = "user"
+ current_content = [line[5:].strip()]
+ elif low.startswith("assistant:"):
+ if current_role and current_content:
+ messages.append({"role": current_role, "content": "\n".join(current_content).strip()}) # type: ignore
+ current_role = "assistant"
+ current_content = [line[10:].strip()]
+ else:
+ current_content.append(line)
+
+ if current_role and current_content:
+ messages.append({"role": current_role, "content": "\n".join(current_content).strip()}) # type: ignore
+ if not messages and prompt_content.strip():
+ messages = [{"role": "user", "content": prompt_content.strip()}] # type: ignore
+ return messages
+
+ def post_call_hook(
+ self,
+ user_id: Optional[str],
+ response: Any,
+ input_messages: List[AllMessageValues],
+ function_call: Optional[Union[Dict[str, Any], str]] = None,
+ litellm_params: Optional[Dict[str, Any]] = None,
+ prompt_id: Optional[str] = None,
+ prompt_variables: Optional[Dict[str, Any]] = None,
+ **kwargs,
+ ) -> Any:
+ return response
+
+ def get_available_prompts(self) -> List[str]:
+ """
+ Return prompt IDs. Prefer already-loaded templates in memory to avoid
+ unnecessary network calls (and to make tests deterministic).
+ """
+ ids = set(self.prompt_manager.prompts.keys())
+ try:
+ ids.update(self.prompt_manager.list_templates())
+ except Exception:
+ # If GitLab list fails (auth, network), still return what we've loaded.
+ pass
+ return sorted(ids)
+
+ def reload_prompts(self) -> None:
+ if self.prompt_id:
+ self._prompt_manager = None
+ _ = self.prompt_manager # trigger re-init/load
+
+ def should_run_prompt_management(
+ self,
+ prompt_id: str,
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ ) -> bool:
+ return True
+
+ def _compile_prompt_helper(
+ self,
+ prompt_id: str,
+ prompt_variables: Optional[dict],
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ prompt_label: Optional[str] = None,
+ prompt_version: Optional[int] = None,
+ ) -> PromptManagementClient:
+ try:
+ if prompt_id not in self.prompt_manager.prompts:
+ git_ref = getattr(dynamic_callback_params, "extra", {}).get("git_ref") if hasattr(dynamic_callback_params, "extra") else None
+ self.prompt_manager._load_prompt_from_gitlab(prompt_id, ref=git_ref)
+
+ rendered_prompt, prompt_metadata = self.get_prompt_template(
+ prompt_id, prompt_variables
+ )
+
+ messages = self._parse_prompt_to_messages(rendered_prompt)
+ template_model = prompt_metadata.get("model")
+
+ optional_params: Dict[str, Any] = {}
+ for param in ["temperature", "max_tokens", "top_p", "frequency_penalty", "presence_penalty"]:
+ if param in prompt_metadata:
+ optional_params[param] = prompt_metadata[param]
+
+ return PromptManagementClient(
+ prompt_id=prompt_id,
+ prompt_template=messages,
+ prompt_template_model=template_model,
+ prompt_template_optional_params=optional_params,
+ completed_messages=None,
+ )
+ except Exception as e:
+ raise ValueError(f"Error compiling prompt '{prompt_id}': {e}")
+
+ def get_chat_completion_prompt(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ non_default_params: dict,
+ prompt_id: Optional[str],
+ prompt_variables: Optional[dict],
+ dynamic_callback_params: StandardCallbackDynamicParams,
+ prompt_label: Optional[str] = None,
+ prompt_version: Optional[int] = None,
+ ) -> Tuple[str, List[AllMessageValues], dict]:
+ return PromptManagementBase.get_chat_completion_prompt(
+ self,
+ model,
+ messages,
+ non_default_params,
+ prompt_id,
+ prompt_variables,
+ dynamic_callback_params,
+ prompt_label,
+ prompt_version,
+ )
diff --git a/litellm/integrations/humanloop.py b/litellm/integrations/humanloop.py
index 9f43d806266..8e60d3736e0 100644
--- a/litellm/integrations/humanloop.py
+++ b/litellm/integrations/humanloop.py
@@ -4,9 +4,10 @@ Humanloop integration
https://humanloop.com/
"""
-from typing import Any, Dict, List, Optional, Tuple, TypedDict, Union, cast
+from typing import Any, Dict, List, Optional, Tuple, Union, cast
import httpx
+from typing_extensions import TypedDict
import litellm
from litellm.caching import DualCache
diff --git a/litellm/integrations/lago.py b/litellm/integrations/lago.py
index 5dfb1ce097d..b881193e869 100644
--- a/litellm/integrations/lago.py
+++ b/litellm/integrations/lago.py
@@ -3,7 +3,7 @@
import json
import os
-import uuid
+from litellm._uuid import uuid
from typing import Literal, Optional
import httpx
diff --git a/litellm/integrations/langfuse/langfuse.py b/litellm/integrations/langfuse/langfuse.py
index 9c3f07fa1a5..7f807bb8b0c 100644
--- a/litellm/integrations/langfuse/langfuse.py
+++ b/litellm/integrations/langfuse/langfuse.py
@@ -1,6 +1,5 @@
#### What this does ####
# On success, logs events to Langfuse
-import copy
import os
import traceback
from datetime import datetime
@@ -11,11 +10,12 @@ from packaging.version import Version
import litellm
from litellm._logging import verbose_logger
from litellm.constants import MAX_LANGFUSE_INITIALIZED_CLIENTS
+from litellm.litellm_core_utils.core_helpers import safe_deep_copy
from litellm.litellm_core_utils.redact_messages import redact_user_api_key_info
from litellm.llms.custom_httpx.http_handler import _get_httpx_client
from litellm.secret_managers.main import str_to_bool
from litellm.types.integrations.langfuse import *
-from litellm.types.llms.openai import HttpxBinaryResponseContent
+from litellm.types.llms.openai import HttpxBinaryResponseContent, ResponsesAPIResponse
from litellm.types.utils import (
EmbeddingResponse,
ImageResponse,
@@ -196,6 +196,7 @@ class LangFuseLogger:
TranscriptionResponse,
RerankResponse,
HttpxBinaryResponseContent,
+ ResponsesAPIResponse,
],
start_time: Optional[datetime] = None,
end_time: Optional[datetime] = None,
@@ -221,7 +222,7 @@ class LangFuseLogger:
litellm_params.get("metadata", {}) or {}
) # if litellm_params['metadata'] == None
metadata = self.add_metadata_from_header(litellm_params, metadata)
- optional_params = copy.deepcopy(kwargs.get("optional_params", {}))
+ optional_params = safe_deep_copy(kwargs.get("optional_params", {}))
prompt = {"messages": kwargs.get("messages")}
@@ -305,6 +306,7 @@ class LangFuseLogger:
TranscriptionResponse,
RerankResponse,
HttpxBinaryResponseContent,
+ ResponsesAPIResponse,
],
prompt: dict,
level: str,
@@ -369,6 +371,11 @@ class LangFuseLogger:
):
input = prompt
output = response_obj.results
+ elif response_obj is not None and isinstance(
+ response_obj, litellm.ResponsesAPIResponse
+ ):
+ input = prompt
+ output = self._get_responses_api_content_for_langfuse(response_obj)
elif (
kwargs.get("call_type") is not None
and kwargs.get("call_type") == "_arealtime"
@@ -664,6 +671,7 @@ class LangFuseLogger:
generation_id = None
usage = None
+ usage_details = None
if response_obj is not None:
if (
hasattr(response_obj, "id")
@@ -680,6 +688,12 @@ class LangFuseLogger:
"completion_tokens": _usage_obj.completion_tokens,
"total_cost": cost if self._supports_costs() else None,
}
+ usage_details = LangfuseUsageDetails(input=_usage_obj.prompt_tokens,
+ output=_usage_obj.completion_tokens,
+ total=_usage_obj.total_tokens,
+ cache_creation_input_tokens=_usage_obj.get('cache_creation_input_tokens', 0),
+ cache_read_input_tokens=_usage_obj.get('cache_read_input_tokens', 0))
+
generation_name = clean_metadata.pop("generation_name", None)
if generation_name is None:
# if `generation_name` is None, use sensible default values
@@ -712,6 +726,7 @@ class LangFuseLogger:
"input": input if not mask_input else "redacted-by-litellm",
"output": output if not mask_output else "redacted-by-litellm",
"usage": usage,
+ "usage_details": usage_details,
"metadata": log_requester_metadata(clean_metadata),
"level": level,
"version": clean_metadata.pop("version", None),
@@ -768,6 +783,19 @@ class LangFuseLogger:
else:
return None
+ @staticmethod
+ def _get_responses_api_content_for_langfuse(
+ response_obj: ResponsesAPIResponse,
+ ):
+ """
+ Get the responses API content for Langfuse logging
+ """
+ if hasattr(response_obj, 'output') and response_obj.output:
+ # ResponsesAPIResponse.output is a list of strings
+ return response_obj.output
+ else:
+ return None
+
@staticmethod
def _get_langfuse_tags(
standard_logging_object: Optional[StandardLoggingPayload],
diff --git a/litellm/integrations/langfuse/langfuse_otel.py b/litellm/integrations/langfuse/langfuse_otel.py
index 4072be2a256..fbe480be95f 100644
--- a/litellm/integrations/langfuse/langfuse_otel.py
+++ b/litellm/integrations/langfuse/langfuse_otel.py
@@ -1,15 +1,16 @@
import base64
-import os
import json # <--- NEW
-from typing import TYPE_CHECKING, Any, Union
-from urllib.parse import quote
+import os
+from typing import TYPE_CHECKING, Any, Optional, Union
from litellm._logging import verbose_logger
from litellm.integrations.arize import _utils
+from litellm.integrations.opentelemetry import OpenTelemetry
from litellm.types.integrations.langfuse_otel import (
LangfuseOtelConfig,
LangfuseSpanAttributes,
)
+from litellm.types.utils import StandardCallbackDynamicParams
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@@ -33,7 +34,11 @@ LANGFUSE_CLOUD_US_ENDPOINT = "https://us.cloud.langfuse.com/api/public/otel"
-class LangfuseOtelLogger:
+class LangfuseOtelLogger(OpenTelemetry):
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+
+
@staticmethod
def set_langfuse_otel_attributes(span: Span, kwargs, response_obj):
"""
@@ -136,6 +141,17 @@ class LangfuseOtelLogger:
value = str(value)
safe_set_attribute(span, enum_attr.value, value)
+ @staticmethod
+ def _get_langfuse_otel_host() -> Optional[str]:
+ """
+ Returns the Langfuse OTEL host based on environment variables.
+
+ Returned in the following order of precedence:
+ 1. LANGFUSE_OTEL_HOST
+ 2. LANGFUSE_HOST
+ """
+ return os.environ.get("LANGFUSE_OTEL_HOST") or os.environ.get("LANGFUSE_HOST")
+
@staticmethod
def get_langfuse_otel_config() -> LangfuseOtelConfig:
"""
@@ -161,7 +177,7 @@ class LangfuseOtelLogger:
)
# Determine endpoint - default to US cloud
- langfuse_host = os.environ.get("LANGFUSE_HOST", None)
+ langfuse_host = LangfuseOtelLogger._get_langfuse_otel_host()
if langfuse_host:
# If LANGFUSE_HOST is provided, construct OTEL endpoint from it
@@ -174,11 +190,11 @@ class LangfuseOtelLogger:
endpoint = LANGFUSE_CLOUD_US_ENDPOINT
verbose_logger.debug(f"Using Langfuse US cloud endpoint: {endpoint}")
- # Create Basic Auth header
- auth_string = f"{public_key}:{secret_key}"
- auth_header = base64.b64encode(auth_string.encode()).decode()
- # URL encode the entire header value as required by OpenTelemetry specification
- otlp_auth_headers = f"Authorization={quote(f'Basic {auth_header}')}"
+ auth_header = LangfuseOtelLogger._get_langfuse_authorization_header(
+ public_key=public_key,
+ secret_key=secret_key
+ )
+ otlp_auth_headers = f"Authorization={auth_header}"
# Set standard OTEL environment variables
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = endpoint
@@ -187,3 +203,37 @@ class LangfuseOtelLogger:
return LangfuseOtelConfig(
otlp_auth_headers=otlp_auth_headers, protocol="otlp_http"
)
+
+ @staticmethod
+ def _get_langfuse_authorization_header(public_key: str, secret_key: str) -> str:
+ """
+ Get the authorization header for Langfuse OpenTelemetry.
+ """
+ auth_string = f"{public_key}:{secret_key}"
+ auth_header = base64.b64encode(auth_string.encode()).decode()
+ return f'Basic {auth_header}'
+
+ def construct_dynamic_otel_headers(
+ self,
+ standard_callback_dynamic_params: StandardCallbackDynamicParams
+ ) -> Optional[dict]:
+ """
+ Construct dynamic Langfuse headers from standard callback dynamic params
+
+ This is used for team/key based logging.
+
+ Returns:
+ dict: A dictionary of dynamic Langfuse headers
+ """
+ dynamic_headers = {}
+
+ dynamic_langfuse_public_key = standard_callback_dynamic_params.get("langfuse_public_key")
+ dynamic_langfuse_secret_key = standard_callback_dynamic_params.get("langfuse_secret_key")
+ if dynamic_langfuse_public_key and dynamic_langfuse_secret_key:
+ auth_header = LangfuseOtelLogger._get_langfuse_authorization_header(
+ public_key=dynamic_langfuse_public_key,
+ secret_key=dynamic_langfuse_secret_key
+ )
+ dynamic_headers["Authorization"] = auth_header
+
+ return dynamic_headers
diff --git a/litellm/integrations/langsmith.py b/litellm/integrations/langsmith.py
index 7035aa3a819..cc9b361b69d 100644
--- a/litellm/integrations/langsmith.py
+++ b/litellm/integrations/langsmith.py
@@ -5,7 +5,7 @@ import os
import random
import traceback
import types
-import uuid
+from litellm._uuid import uuid
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
@@ -39,6 +39,7 @@ class LangsmithLogger(CustomBatchLogger):
langsmith_api_key: Optional[str] = None,
langsmith_project: Optional[str] = None,
langsmith_base_url: Optional[str] = None,
+ langsmith_sampling_rate: Optional[float] = None,
**kwargs,
):
self.flush_lock = asyncio.Lock()
@@ -49,7 +50,8 @@ class LangsmithLogger(CustomBatchLogger):
langsmith_base_url=langsmith_base_url,
)
self.sampling_rate: float = (
- float(os.getenv("LANGSMITH_SAMPLING_RATE")) # type: ignore
+ langsmith_sampling_rate
+ or float(os.getenv("LANGSMITH_SAMPLING_RATE")) # type: ignore
if os.getenv("LANGSMITH_SAMPLING_RATE") is not None
and os.getenv("LANGSMITH_SAMPLING_RATE").strip().isdigit() # type: ignore
else 1.0
@@ -76,26 +78,14 @@ class LangsmithLogger(CustomBatchLogger):
langsmith_base_url: Optional[str] = None,
) -> LangsmithCredentialsObject:
_credentials_api_key = langsmith_api_key or os.getenv("LANGSMITH_API_KEY")
- if _credentials_api_key is None:
- raise Exception(
- "Invalid Langsmith API Key given. _credentials_api_key=None."
- )
_credentials_project = (
langsmith_project or os.getenv("LANGSMITH_PROJECT") or "litellm-completion"
)
- if _credentials_project is None:
- raise Exception(
- "Invalid Langsmith API Key given. _credentials_project=None."
- )
_credentials_base_url = (
langsmith_base_url
or os.getenv("LANGSMITH_BASE_URL")
or "https://api.smith.langchain.com"
)
- if _credentials_base_url is None:
- raise Exception(
- "Invalid Langsmith API Key given. _credentials_base_url=None."
- )
return LangsmithCredentialsObject(
LANGSMITH_API_KEY=_credentials_api_key,
@@ -200,12 +190,7 @@ class LangsmithLogger(CustomBatchLogger):
def log_success_event(self, kwargs, response_obj, start_time, end_time):
try:
- sampling_rate = (
- float(os.getenv("LANGSMITH_SAMPLING_RATE")) # type: ignore
- if os.getenv("LANGSMITH_SAMPLING_RATE") is not None
- and os.getenv("LANGSMITH_SAMPLING_RATE").strip().isdigit() # type: ignore
- else 1.0
- )
+ sampling_rate = self._get_sampling_rate_to_use_for_request(kwargs=kwargs)
random_sample = random.random()
if random_sample > sampling_rate:
verbose_logger.info(
@@ -219,6 +204,7 @@ class LangsmithLogger(CustomBatchLogger):
kwargs,
response_obj,
)
+
credentials = self._get_credentials_to_use_for_request(kwargs=kwargs)
data = self._prepare_log_data(
kwargs=kwargs,
@@ -245,7 +231,7 @@ class LangsmithLogger(CustomBatchLogger):
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
try:
- sampling_rate = self.sampling_rate
+ sampling_rate = self._get_sampling_rate_to_use_for_request(kwargs=kwargs)
random_sample = random.random()
if random_sample > sampling_rate:
verbose_logger.info(
@@ -286,7 +272,7 @@ class LangsmithLogger(CustomBatchLogger):
)
async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
- sampling_rate = self.sampling_rate
+ sampling_rate = self._get_sampling_rate_to_use_for_request(kwargs=kwargs)
random_sample = random.random()
if random_sample > sampling_rate:
verbose_logger.info(
@@ -417,6 +403,17 @@ class LangsmithLogger(CustomBatchLogger):
for queue_object in self.log_queue:
credentials = queue_object["credentials"]
+ # if credential missing, skip - log warning
+ if (
+ credentials["LANGSMITH_API_KEY"] is None
+ or credentials["LANGSMITH_PROJECT"] is None
+ ):
+ verbose_logger.warning(
+ "Langsmith Logging - credentials missing - api_key: %s, project: %s",
+ credentials["LANGSMITH_API_KEY"],
+ credentials["LANGSMITH_PROJECT"],
+ )
+ continue
key = CredentialsKey(
api_key=credentials["LANGSMITH_API_KEY"],
project=credentials["LANGSMITH_PROJECT"],
@@ -432,6 +429,19 @@ class LangsmithLogger(CustomBatchLogger):
return log_queue_by_credentials
+ def _get_sampling_rate_to_use_for_request(self, kwargs: Dict[str, Any]) -> float:
+ standard_callback_dynamic_params: Optional[StandardCallbackDynamicParams] = (
+ kwargs.get("standard_callback_dynamic_params", None)
+ )
+ sampling_rate: float = self.sampling_rate
+ if standard_callback_dynamic_params is not None:
+ _sampling_rate = standard_callback_dynamic_params.get(
+ "langsmith_sampling_rate"
+ )
+ if _sampling_rate is not None:
+ sampling_rate = float(_sampling_rate)
+ return sampling_rate
+
def _get_credentials_to_use_for_request(
self, kwargs: Dict[str, Any]
) -> LangsmithCredentialsObject:
@@ -442,9 +452,9 @@ class LangsmithLogger(CustomBatchLogger):
Otherwise, use the default credentials.
"""
- standard_callback_dynamic_params: Optional[
- StandardCallbackDynamicParams
- ] = kwargs.get("standard_callback_dynamic_params", None)
+ standard_callback_dynamic_params: Optional[StandardCallbackDynamicParams] = (
+ kwargs.get("standard_callback_dynamic_params", None)
+ )
if standard_callback_dynamic_params is not None:
credentials = self.get_credentials_from_env(
langsmith_api_key=standard_callback_dynamic_params.get(
diff --git a/litellm/integrations/literal_ai.py b/litellm/integrations/literal_ai.py
index 5bf9afd7eb4..042779ba844 100644
--- a/litellm/integrations/literal_ai.py
+++ b/litellm/integrations/literal_ai.py
@@ -2,7 +2,7 @@
# This file contains the LiteralAILogger class which is used to log steps to the LiteralAI observability platform.
import asyncio
import os
-import uuid
+from litellm._uuid import uuid
from typing import List, Optional
import httpx
diff --git a/litellm/integrations/logfire_logger.py b/litellm/integrations/logfire_logger.py
index 516bd4a8e28..2345dc869c6 100644
--- a/litellm/integrations/logfire_logger.py
+++ b/litellm/integrations/logfire_logger.py
@@ -3,7 +3,7 @@
import os
import traceback
-import uuid
+from litellm._uuid import uuid
from enum import Enum
from typing import Any, Dict, NamedTuple
diff --git a/litellm/integrations/mlflow.py b/litellm/integrations/mlflow.py
index e7a458accf9..86af800d732 100644
--- a/litellm/integrations/mlflow.py
+++ b/litellm/integrations/mlflow.py
@@ -1,10 +1,15 @@
import json
import threading
-from typing import Optional
+from typing import TYPE_CHECKING, Any, Optional
from litellm._logging import verbose_logger
from litellm.integrations.custom_logger import CustomLogger
+if TYPE_CHECKING:
+ from litellm.types.utils import StandardLoggingPayload
+else:
+ StandardLoggingPayload = Any
+
class MlflowLogger(CustomLogger):
def __init__(self):
@@ -55,10 +60,7 @@ class MlflowLogger(CustomLogger):
inputs = self._construct_input(kwargs)
input_messages = inputs.get("messages", [])
- output_messages = [
- c.message.model_dump(exclude_none=True)
- for c in getattr(response_obj, "choices", [])
- ]
+ output_messages = [c.message.model_dump(exclude_none=True) for c in getattr(response_obj, "choices", [])]
if messages := [*input_messages, *output_messages]:
set_span_chat_messages(span, messages)
if tools := inputs.get("tools"):
@@ -163,6 +165,10 @@ class MlflowLogger(CustomLogger):
for key in ["functions", "tools", "stream", "tool_choice", "user"]:
if value := kwargs.get("optional_params", {}).pop(key, None):
inputs[key] = value
+
+ if prediction := kwargs.get("prediction"):
+ inputs["prediction"] = prediction
+
return inputs
def _extract_attributes(self, kwargs):
@@ -178,20 +184,21 @@ class MlflowLogger(CustomLogger):
"call_type": kwargs.get("call_type"),
"model": kwargs.get("model"),
}
- standard_obj = kwargs.get("standard_logging_object")
+ standard_obj: Optional[StandardLoggingPayload] = kwargs.get("standard_logging_object")
if standard_obj:
attributes.update(
{
"api_base": standard_obj.get("api_base"),
"cache_hit": standard_obj.get("cache_hit"),
- "usage": {
- "completion_tokens": standard_obj.get("completion_tokens"),
- "prompt_tokens": standard_obj.get("prompt_tokens"),
+ "mlflow.chat.tokenUsage": {
+ "input_tokens": standard_obj.get("prompt_tokens"),
+ "output_tokens": standard_obj.get("completion_tokens"),
"total_tokens": standard_obj.get("total_tokens"),
},
"raw_llm_response": standard_obj.get("response"),
"response_cost": standard_obj.get("response_cost"),
"saved_cache_cost": standard_obj.get("saved_cache_cost"),
+ "request_tags": standard_obj.get("request_tags"),
}
)
else:
@@ -237,7 +244,7 @@ class MlflowLogger(CustomLogger):
if active_span := mlflow.get_current_active_span(): # type: ignore
return self._client.start_span(
name=span_name,
- request_id=active_span.request_id,
+ trace_id=active_span.request_id,
parent_id=active_span.span_id,
span_type=span_type,
inputs=inputs,
@@ -250,21 +257,25 @@ class MlflowLogger(CustomLogger):
span_type=span_type,
inputs=inputs,
attributes=attributes,
+ tags=self._transform_tag_list_to_dict(attributes.get("request_tags", [])),
start_time_ns=start_time_ns,
)
+ def _transform_tag_list_to_dict(self, tag_list: list) -> dict:
+ return {tag: "" for tag in tag_list}
+
def _end_span_or_trace(self, span, outputs, end_time_ns, status):
"""End an MLflow span or a trace."""
if span.parent_id is None:
self._client.end_trace(
- request_id=span.request_id,
+ trace_id=span.request_id,
outputs=outputs,
status=status,
end_time_ns=end_time_ns,
)
else:
self._client.end_span(
- request_id=span.request_id,
+ trace_id=span.request_id,
span_id=span.span_id,
outputs=outputs,
status=status,
diff --git a/litellm/integrations/openmeter.py b/litellm/integrations/openmeter.py
index 19010daf831..b8fb64ec287 100644
--- a/litellm/integrations/openmeter.py
+++ b/litellm/integrations/openmeter.py
@@ -66,8 +66,18 @@ class OpenMeterLogger(CustomLogger):
}
user_param = kwargs.get("user", None) # end-user passed in via 'user' param
+
+ # If no user provided directly, try to get it from token user_id
if user_param is None:
- raise Exception("OpenMeter: user is required")
+ # Check if user_id is available from the API key metadata
+ litellm_params = kwargs.get("litellm_params", {})
+ metadata = litellm_params.get("metadata", {})
+ user_api_key_user_id = metadata.get("user_api_key_user_id", None)
+
+ if user_api_key_user_id is not None:
+ user_param = user_api_key_user_id
+ else:
+ raise Exception("OpenMeter: user is required")
# Ensure subject is always a string for OpenMeter API
subject = str(user_param)
diff --git a/litellm/integrations/opentelemetry.py b/litellm/integrations/opentelemetry.py
index 22ab3092901..e825f89f56e 100644
--- a/litellm/integrations/opentelemetry.py
+++ b/litellm/integrations/opentelemetry.py
@@ -15,6 +15,8 @@ from litellm.types.utils import (
StandardLoggingPayload,
)
+# OpenTelemetry imports moved to individual functions to avoid import errors when not installed
+
if TYPE_CHECKING:
from opentelemetry.sdk.trace.export import SpanExporter as _SpanExporter
from opentelemetry.trace import Context as _Context
@@ -41,6 +43,8 @@ else:
Context = Any
LITELLM_TRACER_NAME = os.getenv("OTEL_TRACER_NAME", "litellm")
+LITELLM_METER_NAME = os.getenv("LITELLM_METER_NAME", "litellm")
+LITELLM_LOGGER_NAME = os.getenv("LITELLM_LOGGER_NAME", "litellm")
# Remove the hardcoded LITELLM_RESOURCE dictionary - we'll create it properly later
RAW_REQUEST_SPAN_NAME = "raw_gen_ai_request"
LITELLM_REQUEST_SPAN_NAME = "litellm_request"
@@ -83,6 +87,8 @@ class OpenTelemetryConfig:
exporter: Union[str, SpanExporter] = "console"
endpoint: Optional[str] = None
headers: Optional[str] = None
+ enable_metrics: bool = False
+ enable_events: bool = False
@classmethod
def from_env(cls):
@@ -104,6 +110,14 @@ class OpenTelemetryConfig:
headers = os.getenv(
"OTEL_EXPORTER_OTLP_HEADERS", os.getenv("OTEL_HEADERS")
) # example: OTEL_HEADERS=x-honeycomb-team=B85YgLm96***"
+ enable_metrics: bool = (
+ os.getenv("LITELLM_OTEL_INTEGRATION_ENABLE_METRICS", "false").lower()
+ == "true"
+ )
+ enable_events: bool = (
+ os.getenv("LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS", "false").lower()
+ == "true"
+ )
if exporter == "in_memory":
return cls(exporter=InMemorySpanExporter())
@@ -111,6 +125,8 @@ class OpenTelemetryConfig:
exporter=exporter,
endpoint=endpoint,
headers=headers, # example: OTEL_HEADERS=x-honeycomb-team=B85YgLm96***"
+ enable_metrics=enable_metrics,
+ enable_events=enable_events,
)
@@ -119,27 +135,22 @@ class OpenTelemetry(CustomLogger):
self,
config: Optional[OpenTelemetryConfig] = None,
callback_name: Optional[str] = None,
+ # injection points for testing
+ tracer_provider: Optional[Any] = None,
+ logger_provider: Optional[Any] = None,
+ meter_provider: Optional[Any] = None,
**kwargs,
):
- from opentelemetry import trace
- from opentelemetry.sdk.trace import TracerProvider
- from opentelemetry.trace import SpanKind
if config is None:
config = OpenTelemetryConfig.from_env()
self.config = config
+ self.callback_name = callback_name
self.OTEL_EXPORTER = self.config.exporter
self.OTEL_ENDPOINT = self.config.endpoint
self.OTEL_HEADERS = self.config.headers
- provider = TracerProvider(resource=_get_litellm_resource())
- provider.add_span_processor(self._get_span_processor())
- self.callback_name = callback_name
-
- trace.set_tracer_provider(provider)
- self.tracer = trace.get_tracer(LITELLM_TRACER_NAME)
-
- self.span_kind = SpanKind
+ self._init_tracing(tracer_provider)
_debug_otel = str(os.getenv("DEBUG_OTEL", "False")).lower()
@@ -156,6 +167,8 @@ class OpenTelemetry(CustomLogger):
# init CustomLogger params
super().__init__(**kwargs)
+ self._init_metrics(meter_provider)
+ self._init_logs(logger_provider)
self._init_otel_logger_on_litellm_proxy()
def _init_otel_logger_on_litellm_proxy(self):
@@ -178,14 +191,109 @@ class OpenTelemetry(CustomLogger):
litellm.service_callback.append("otel")
setattr(proxy_server, "open_telemetry_logger", self)
+ def _init_tracing(self, tracer_provider):
+ from opentelemetry import trace
+ from opentelemetry.sdk.trace import TracerProvider
+ from opentelemetry.trace import SpanKind
+
+ # use provided tracer or create a new one
+ if tracer_provider is None:
+ tracer_provider = TracerProvider(resource=_get_litellm_resource())
+ # Only add OTLP span processor if we created the tracer provider ourselves
+ tracer_provider.add_span_processor(self._get_span_processor())
+
+ # register global provider and grab our tracer
+ trace.set_tracer_provider(tracer_provider)
+ self.tracer = trace.get_tracer(LITELLM_TRACER_NAME)
+ self.span_kind = SpanKind
+
+ def _init_metrics(self, meter_provider):
+ if not self.config.enable_metrics:
+ self._operation_duration_histogram = None
+ self._token_usage_histogram = None
+ self._cost_histogram = None
+ return
+
+ from opentelemetry import metrics
+ from opentelemetry.sdk.metrics import Histogram, MeterProvider
+
+ # Only create OTLP infrastructure if no custom meter provider is provided
+ if meter_provider is None:
+ from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
+ OTLPMetricExporter,
+ )
+ from opentelemetry.sdk.metrics.export import (
+ AggregationTemporality,
+ PeriodicExportingMetricReader,
+ )
+
+ _metric_exporter = OTLPMetricExporter(
+ endpoint=self.config.endpoint,
+ headers=OpenTelemetry._get_headers_dictionary(self.config.headers),
+ preferred_temporality={Histogram: AggregationTemporality.DELTA},
+ )
+ _metric_reader = PeriodicExportingMetricReader(
+ _metric_exporter, export_interval_millis=10000
+ )
+
+ meter_provider = MeterProvider(
+ metric_readers=[_metric_reader], resource=_get_litellm_resource()
+ )
+ meter = meter_provider.get_meter(__name__)
+ else:
+ # Use the provided meter provider as-is, without creating additional OTLP infrastructure
+ meter = meter_provider.get_meter(__name__)
+
+ metrics.set_meter_provider(meter_provider)
+
+ self._operation_duration_histogram = meter.create_histogram(
+ name="gen_ai.client.operation.duration", # Replace with semconv constant in otel 1.38
+ description="GenAI operation duration",
+ unit="s",
+ )
+ self._token_usage_histogram = meter.create_histogram(
+ name="gen_ai.client.token.usage", # Replace with semconv constant in otel 1.38
+ description="GenAI token usage",
+ unit="{token}",
+ )
+ self._cost_histogram = meter.create_histogram(
+ name="gen_ai.client.token.cost",
+ description="GenAI request cost",
+ unit="USD",
+ )
+
+ def _init_logs(self, logger_provider):
+ # nothing to do if events disabled
+ if not self.config.enable_events:
+ return
+
+ from opentelemetry._logs import set_logger_provider
+ from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
+ from opentelemetry.sdk._logs import LoggerProvider as OTLoggerProvider
+ from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
+
+ # set up log pipeline
+ if logger_provider is None:
+ logger_provider = OTLoggerProvider()
+ # Only add OTLP exporter if we created the logger provider ourselves
+ logger_provider.add_log_record_processor(
+ BatchLogRecordProcessor(
+ OTLPLogExporter(
+ endpoint=self.config.endpoint,
+ headers=self._get_headers_dictionary(self.config.headers),
+ )
+ )
+ )
+ set_logger_provider(logger_provider)
+
def log_success_event(self, kwargs, response_obj, start_time, end_time):
- self._handle_sucess(kwargs, response_obj, start_time, end_time)
+ self._handle_success(kwargs, response_obj, start_time, end_time)
def log_failure_event(self, kwargs, response_obj, start_time, end_time):
self._handle_failure(kwargs, response_obj, start_time, end_time)
async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
- self._handle_sucess(kwargs, response_obj, start_time, end_time)
+ self._handle_success(kwargs, response_obj, start_time, end_time)
async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
self._handle_failure(kwargs, response_obj, start_time, end_time)
@@ -372,9 +480,9 @@ class OpenTelemetry(CustomLogger):
def _get_dynamic_otel_headers_from_kwargs(self, kwargs) -> Optional[dict]:
"""Extract dynamic headers from kwargs if available."""
- standard_callback_dynamic_params: Optional[StandardCallbackDynamicParams] = (
- kwargs.get("standard_callback_dynamic_params")
- )
+ standard_callback_dynamic_params: Optional[
+ StandardCallbackDynamicParams
+ ] = kwargs.get("standard_callback_dynamic_params")
if not standard_callback_dynamic_params:
return None
@@ -414,50 +522,192 @@ class OpenTelemetry(CustomLogger):
# End of Team/Key Based Logging Control Flow
#########################################################
- def _handle_sucess(self, kwargs, response_obj, start_time, end_time):
- from opentelemetry import trace
- from opentelemetry.trace import Status, StatusCode
+ def _handle_success(self, kwargs, response_obj, start_time, end_time):
verbose_logger.debug(
"OpenTelemetry Logger: Logging kwargs: %s, OTEL config settings=%s",
kwargs,
self.config,
)
+ ctx, parent_span = self._get_span_context(kwargs)
+
+ # 1. Primary span
+ span = self._start_primary_span(kwargs, response_obj, start_time, end_time, ctx)
+
+ # 2. Raw‐request sub-span (if enabled)
+ self._maybe_log_raw_request(kwargs, response_obj, start_time, end_time, span)
+
+ # 3. Guardrail span
+ self._create_guardrail_span(kwargs=kwargs, context=ctx)
+
+ # 4. Metrics & cost recording
+ self._record_metrics(kwargs, response_obj, start_time, end_time)
+
+ # 5. Semantic logs.
+ if self.config.enable_events:
+ self._emit_semantic_logs(kwargs, response_obj, span)
+
+ # 6. End parent span
+ if parent_span is not None:
+ parent_span.end(end_time=self._to_ns(datetime.now()))
+
+ def _start_primary_span(self, kwargs, response_obj, start_time, end_time, context):
+ from opentelemetry.trace import Status, StatusCode
- _parent_context, parent_otel_span = self._get_span_context(kwargs)
- # Span 1: Request sent to litellm SDK
otel_tracer: Tracer = self.get_tracer_to_use_for_request(kwargs)
span = otel_tracer.start_span(
name=self._get_span_name(kwargs),
start_time=self._to_ns(start_time),
- context=_parent_context,
+ context=context,
)
span.set_status(Status(StatusCode.OK))
self.set_attributes(span, kwargs, response_obj)
+ span.end(end_time=self._to_ns(end_time))
+ return span
- if litellm.turn_off_message_logging is True:
- pass
- elif self.message_logging is not True:
- pass
- else:
- # Span 2: Raw Request / Response to LLM
- raw_request_span = otel_tracer.start_span(
- name=RAW_REQUEST_SPAN_NAME,
- start_time=self._to_ns(start_time),
- context=trace.set_span_in_context(span),
+ def _maybe_log_raw_request(
+ self, kwargs, response_obj, start_time, end_time, parent_span
+ ):
+ from opentelemetry import trace
+ from opentelemetry.trace import Status, StatusCode
+
+ # only log raw LLM request/response if message_logging is on and not globally turned off
+ if litellm.turn_off_message_logging or not self.message_logging:
+ return
+
+ litellm_params = kwargs.get("litellm_params", {})
+ metadata = litellm_params.get("metadata") or {}
+ generation_name = metadata.get("generation_name")
+
+ raw_span_name = generation_name if generation_name else RAW_REQUEST_SPAN_NAME
+
+
+ otel_tracer: Tracer = self.get_tracer_to_use_for_request(kwargs)
+ raw_span = otel_tracer.start_span(
+ name=raw_span_name,
+ start_time=self._to_ns(start_time),
+ context=trace.set_span_in_context(parent_span),
+ )
+ raw_span.set_status(Status(StatusCode.OK))
+ self.set_raw_request_attributes(raw_span, kwargs, response_obj)
+ raw_span.end(end_time=self._to_ns(end_time))
+
+ def _record_metrics(self, kwargs, response_obj, start_time, end_time):
+ duration_s = (end_time - start_time).total_seconds()
+ params = kwargs.get("litellm_params") or {}
+ provider = params.get("custom_llm_provider", "Unknown")
+
+ common_attrs = {
+ "gen_ai.operation.name": "chat",
+ "gen_ai.system": provider,
+ "gen_ai.request.model": kwargs.get("model"),
+ "gen_ai.framework": "litellm",
+ }
+
+ std_log = kwargs.get("standard_logging_object")
+ md = getattr(std_log, "metadata", None) or (std_log or {}).get("metadata", {})
+ for key in [
+ "user_api_key_hash",
+ "user_api_key_alias",
+ "user_api_key_team_id",
+ "user_api_key_org_id",
+ "user_api_key_user_id",
+ "user_api_key_team_alias",
+ "user_api_key_user_email",
+ "spend_logs_metadata",
+ "requester_ip_address",
+ "requester_metadata",
+ "user_api_key_end_user_id",
+ "prompt_management_metadata",
+ "applied_guardrails",
+ "mcp_tool_call_metadata",
+ "vector_store_request_metadata",
+ ]:
+ if md.get(key) is not None:
+ common_attrs[f"metadata.{key}"] = str(md[key])
+
+ if self._operation_duration_histogram:
+ self._operation_duration_histogram.record(
+ duration_s, attributes=common_attrs
+ )
+ if (
+ response_obj
+ and (usage := response_obj.get("usage"))
+ and self._token_usage_histogram
+ ):
+ in_attrs = {**common_attrs, "gen_ai.token.type": "input"}
+ out_attrs = {**common_attrs, "gen_ai.token.type": "completion"}
+ self._token_usage_histogram.record(
+ usage.get("prompt_tokens", 0), attributes=in_attrs
+ )
+ self._token_usage_histogram.record(
+ usage.get("completion_tokens", 0), attributes=out_attrs
+ )
+
+ cost = kwargs.get("response_cost")
+ if self._cost_histogram and cost:
+ self._cost_histogram.record(cost, attributes=common_attrs)
+
+ def _emit_semantic_logs(self, kwargs, response_obj, span: Span):
+ if not self.config.enable_events:
+ return
+
+ from opentelemetry._logs import LogRecord, get_logger
+ otel_logger = get_logger(LITELLM_LOGGER_NAME)
+
+ parent_ctx = span.get_span_context()
+ provider = (kwargs.get("litellm_params") or {}).get(
+ "custom_llm_provider", "Unknown"
+ )
+
+ # per-message events
+ for msg in kwargs.get("messages", []):
+ role = msg.get("role", "user")
+ attrs = {"event_name": "gen_ai.content.prompt", "gen_ai.system": provider}
+ if role == "tool" and msg.get("id"):
+ attrs["id"] = msg["id"]
+ if self.message_logging and msg.get("content"):
+ attrs["gen_ai.prompt"] = msg["content"]
+
+ otel_logger.emit(
+ LogRecord(
+ attributes=attrs,
+ body=msg.copy(),
+ trace_id=parent_ctx.trace_id,
+ span_id=parent_ctx.span_id,
+ trace_flags=parent_ctx.trace_flags,
+ )
)
- raw_request_span.set_status(Status(StatusCode.OK))
- self.set_raw_request_attributes(raw_request_span, kwargs, response_obj)
- raw_request_span.end(end_time=self._to_ns(end_time))
+ # per-choice events
+ for idx, choice in enumerate(response_obj.get("choices", [])):
+ attrs = {
+ "event_name": "gen_ai.content.completion",
+ "gen_ai.system": provider,
+ "index": idx,
+ "finish_reason": choice.get("finish_reason"),
+ }
+ body_msg = choice.get("message", {})
+ if self.message_logging and body_msg.get("content"):
+ attrs["message.content"] = body_msg["content"]
+ body = {
+ "index": idx,
+ "finish_reason": choice.get("finish_reason"),
+ "message": {"role": body_msg.get("role", "assistant")},
+ }
+ if self.message_logging and body_msg.get("content"):
+ body["message"]["content"] = body_msg["content"]
- span.end(end_time=self._to_ns(end_time))
+ otel_logger.emit(
+ LogRecord(
+ attributes=attrs,
+ body=body,
+ trace_id=parent_ctx.trace_id,
+ span_id=parent_ctx.span_id,
+ trace_flags=parent_ctx.trace_flags,
+ )
+ )
- # Create span for guardrail information
- self._create_guardrail_span(kwargs=kwargs, context=_parent_context)
-
- if parent_otel_span is not None:
- parent_otel_span.end(end_time=self._to_ns(datetime.now()))
def _create_guardrail_span(
self, kwargs: Optional[dict], context: Optional[Context]
@@ -872,56 +1122,68 @@ class OpenTelemetry(CustomLogger):
span.set_attribute(key, primitive_value)
def set_raw_request_attributes(self, span: Span, kwargs, response_obj):
- kwargs.get("optional_params", {})
- litellm_params = kwargs.get("litellm_params", {}) or {}
- custom_llm_provider = litellm_params.get("custom_llm_provider", "Unknown")
+ try:
+ kwargs.get("optional_params", {})
+ litellm_params = kwargs.get("litellm_params", {}) or {}
+ custom_llm_provider = litellm_params.get("custom_llm_provider", "Unknown")
- _raw_response = kwargs.get("original_response")
- _additional_args = kwargs.get("additional_args", {}) or {}
- complete_input_dict = _additional_args.get("complete_input_dict")
- #############################################
- ########## LLM Request Attributes ###########
- #############################################
+ _raw_response = kwargs.get("original_response")
+ _additional_args = kwargs.get("additional_args", {}) or {}
+ complete_input_dict = _additional_args.get("complete_input_dict")
+ #############################################
+ ########## LLM Request Attributes ###########
+ #############################################
- # OTEL Attributes for the RAW Request to https://docs.anthropic.com/en/api/messages
- if complete_input_dict and isinstance(complete_input_dict, dict):
- for param, val in complete_input_dict.items():
- self.safe_set_attribute(
- span=span, key=f"llm.{custom_llm_provider}.{param}", value=val
- )
+ # OTEL Attributes for the RAW Request to https://docs.anthropic.com/en/api/messages
+ if complete_input_dict and isinstance(complete_input_dict, dict):
+ for param, val in complete_input_dict.items():
+ self.safe_set_attribute(
+ span=span, key=f"llm.{custom_llm_provider}.{param}", value=val
+ )
- #############################################
- ########## LLM Response Attributes ##########
- #############################################
- if _raw_response and isinstance(_raw_response, str):
- # cast sr -> dict
- import json
+ #############################################
+ ########## LLM Response Attributes ##########
+ #############################################
+ if _raw_response and isinstance(_raw_response, str):
+ # cast sr -> dict
+ import json
+
+ try:
+ _raw_response = json.loads(_raw_response)
+ for param, val in _raw_response.items():
+ self.safe_set_attribute(
+ span=span,
+ key=f"llm.{custom_llm_provider}.{param}",
+ value=val,
+ )
+ except json.JSONDecodeError:
+ verbose_logger.debug(
+ "litellm.integrations.opentelemetry.py::set_raw_request_attributes() - raw_response not json string - {}".format(
+ _raw_response
+ )
+ )
- try:
- _raw_response = json.loads(_raw_response)
- for param, val in _raw_response.items():
self.safe_set_attribute(
span=span,
- key=f"llm.{custom_llm_provider}.{param}",
- value=val,
+ key=f"llm.{custom_llm_provider}.stringified_raw_response",
+ value=_raw_response,
)
- except json.JSONDecodeError:
- verbose_logger.debug(
- "litellm.integrations.opentelemetry.py::set_raw_request_attributes() - raw_response not json string - {}".format(
- _raw_response
- )
- )
-
- self.safe_set_attribute(
- span=span,
- key=f"llm.{custom_llm_provider}.stringified_raw_response",
- value=_raw_response,
- )
+ except Exception as e:
+ verbose_logger.exception(
+ "OpenTelemetry logging error in set_raw_request_attributes %s", str(e)
+ )
def _to_ns(self, dt):
return int(dt.timestamp() * 1e9)
def _get_span_name(self, kwargs):
+ litellm_params = kwargs.get("litellm_params", {})
+ metadata = litellm_params.get("metadata") or {}
+ generation_name = metadata.get("generation_name")
+
+ if generation_name:
+ return generation_name
+
return LITELLM_REQUEST_SPAN_NAME
def get_traceparent_from_header(self, headers):
diff --git a/litellm/integrations/opik/opik.py b/litellm/integrations/opik/opik.py
index 8cbfb9e6535..9fa3482f663 100644
--- a/litellm/integrations/opik/opik.py
+++ b/litellm/integrations/opik/opik.py
@@ -3,6 +3,7 @@ Opik Logger that logs LLM events to an Opik server
"""
import asyncio
+from datetime import timezone
import json
import traceback
from typing import Dict, List
@@ -191,9 +192,25 @@ class OpikLogger(CustomBatchLogger):
# Extract opik metadata
litellm_opik_metadata = litellm_params_metadata.get("opik", {})
+
+ # Use standard_logging_object to create metadata and input/output data
+ standard_logging_object = kwargs.get("standard_logging_object", None)
+ if standard_logging_object is None:
+ verbose_logger.debug(
+ "OpikLogger skipping event; no standard_logging_object found"
+ )
+ return []
+
+ # Update litellm_opik_metadata with opik metadata from requester
+ standard_logging_metadata = standard_logging_object.get("metadata", {}) or {}
+ requester_metadata = standard_logging_metadata.get("requester_metadata", {}) or {}
+ requester_opik_metadata = requester_metadata.get("opik", {}) or {}
+ litellm_opik_metadata.update(requester_opik_metadata)
+
verbose_logger.debug(
f"litellm_opik_metadata - {json.dumps(litellm_opik_metadata, default=str)}"
)
+
project_name = litellm_opik_metadata.get("project_name", self.opik_project_name)
# Extract trace_id and parent_span_id
@@ -207,19 +224,33 @@ class OpikLogger(CustomBatchLogger):
else:
trace_id = None
parent_span_id = None
+
# Create Opik tags
opik_tags = litellm_opik_metadata.get("tags", [])
if kwargs.get("custom_llm_provider"):
opik_tags.append(kwargs["custom_llm_provider"])
+
+ # Get thread_id if present
+ thread_id = litellm_opik_metadata.get("thread_id", None)
- # Use standard_logging_object to create metadata and input/output data
- standard_logging_object = kwargs.get("standard_logging_object", None)
- if standard_logging_object is None:
- verbose_logger.debug(
- "OpikLogger skipping event; no standard_logging_object found"
- )
- return []
-
+ # Override with any opik_ headers from proxy request
+ proxy_server_request = _litellm_params.get("proxy_server_request", {}) or {}
+ proxy_headers = proxy_server_request.get("headers", {}) or {}
+ for key, value in proxy_headers.items():
+ if key.startswith("opik_"):
+ param_key = key.replace("opik_", "", 1)
+ if param_key == "project_name" and value:
+ project_name = value
+ elif param_key == "thread_id" and value:
+ thread_id = value
+ elif param_key == "tags" and value:
+ try:
+ parsed_tags = json.loads(value)
+ if isinstance(parsed_tags, list):
+ opik_tags.extend(parsed_tags)
+ except (json.JSONDecodeError, TypeError):
+ pass
+
# Create input and output data
input_data = standard_logging_object.get("messages", {})
output_data = standard_logging_object.get("response", {})
@@ -242,7 +273,7 @@ class OpikLogger(CustomBatchLogger):
del metadata["current_span_data"]
metadata["created_from"] = "litellm"
- metadata.update(standard_logging_object.get("metadata", {}))
+ metadata.update(standard_logging_metadata)
if "call_type" in standard_logging_object:
metadata["type"] = standard_logging_object["call_type"]
if "status" in standard_logging_object:
@@ -285,20 +316,20 @@ class OpikLogger(CustomBatchLogger):
verbose_logger.debug(
f"OpikLogger creating payload for trace with id {trace_id}"
)
-
- payload.append(
- {
- "project_name": project_name,
- "id": trace_id,
- "name": trace_name,
- "start_time": start_time.isoformat() + "Z",
- "end_time": end_time.isoformat() + "Z",
- "input": input_data,
- "output": output_data,
- "metadata": metadata,
- "tags": opik_tags,
- }
- )
+ payload.append(
+ {
+ "project_name": project_name,
+ "id": trace_id,
+ "name": trace_name,
+ "start_time": start_time.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"),
+ "end_time": end_time.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"),
+ "input": input_data,
+ "output": output_data,
+ "metadata": metadata,
+ "tags": opik_tags,
+ "thread_id": thread_id,
+ }
+ )
span_id = create_uuid7()
verbose_logger.debug(
@@ -312,12 +343,13 @@ class OpikLogger(CustomBatchLogger):
"parent_span_id": parent_span_id,
"name": span_name,
"type": "llm",
- "start_time": start_time.isoformat() + "Z",
- "end_time": end_time.isoformat() + "Z",
+ "start_time": start_time.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"),
+ "end_time": end_time.astimezone(timezone.utc).isoformat().replace("+00:00", "Z"),
"input": input_data,
"output": output_data,
"metadata": metadata,
"tags": opik_tags,
+ "thread_id": thread_id,
"usage": usage,
}
)
diff --git a/litellm/integrations/posthog.py b/litellm/integrations/posthog.py
new file mode 100644
index 00000000000..c609d30ccff
--- /dev/null
+++ b/litellm/integrations/posthog.py
@@ -0,0 +1,379 @@
+"""
+PostHog Integration - sends LLM analytics events to PostHog
+
+Follows PostHog's LLM Analytics format: https://posthog.com/docs/llm-analytics/manual-capture
+
+async_log_success_event: stores batch of events in memory and flushes to PostHog
+async_log_failure_event: logs failed LLM calls with error information
+
+For batching specific details see CustomBatchLogger class
+"""
+
+import asyncio
+import os
+from typing import Any, Dict, Optional, Tuple
+
+from litellm._logging import verbose_logger
+from litellm._uuid import uuid
+from litellm.integrations.custom_batch_logger import CustomBatchLogger
+from litellm.llms.custom_httpx.http_handler import (
+ _get_httpx_client,
+ get_async_httpx_client,
+ httpxSpecialProvider,
+)
+from litellm.types.integrations.posthog import (
+ POSTHOG_MAX_BATCH_SIZE,
+ PostHogEventPayload,
+)
+from litellm.types.utils import StandardCallbackDynamicParams, StandardLoggingPayload
+
+
+class PostHogLogger(CustomBatchLogger):
+ def __init__(self, **kwargs):
+ """
+ Initializes the PostHog logger, checks if the correct env variables are set
+
+ Required environment variables:
+ `POSTHOG_API_KEY` - your PostHog API key
+ `POSTHOG_API_URL` - your PostHog API URL (defaults to https://app.posthog.com)
+ """
+ try:
+ verbose_logger.debug("PostHog: in init posthog logger")
+ if os.getenv("POSTHOG_API_KEY", None) is None:
+ raise Exception("POSTHOG_API_KEY is not set, set 'POSTHOG_API_KEY=<>'")
+
+ self.async_client = get_async_httpx_client(
+ llm_provider=httpxSpecialProvider.LoggingCallback
+ )
+ self.sync_client = _get_httpx_client()
+
+ self.POSTHOG_API_KEY = os.getenv("POSTHOG_API_KEY")
+ posthog_api_url = os.getenv("POSTHOG_API_URL", "https://us.i.posthog.com")
+ self.posthog_host = posthog_api_url.rstrip('/')
+ self.capture_url = f"{self.posthog_host}/batch/"
+
+ self._async_initialized = False
+ self.flush_lock = None
+ self.log_queue = []
+
+ super().__init__(
+ **kwargs, flush_lock=None, batch_size=POSTHOG_MAX_BATCH_SIZE
+ )
+
+ except Exception as e:
+ verbose_logger.exception(
+ f"PostHog: Got exception on init PostHog client {str(e)}"
+ )
+ raise e
+
+ def log_success_event(self, kwargs, response_obj, start_time, end_time):
+ try:
+ verbose_logger.debug(
+ "PostHog: Sync logging - Enters logging function for model %s", kwargs
+ )
+
+ api_key, api_url = self._get_credentials_for_request(kwargs)
+ if api_key is None or api_url is None:
+ raise Exception("PostHog credentials not found in kwargs")
+ event_payload = self.create_posthog_event_payload(kwargs)
+
+ headers = {
+ "Content-Type": "application/json",
+ }
+
+ payload = self._create_posthog_payload([event_payload], api_key)
+ capture_url = f"{api_url.rstrip('/')}/batch/"
+
+ response = self.sync_client.post(
+ url=capture_url,
+ json=payload,
+ headers=headers,
+ )
+ response.raise_for_status()
+
+ if response.status_code != 200:
+ raise Exception(
+ f"Response from PostHog API status_code: {response.status_code}, text: {response.text}"
+ )
+
+ verbose_logger.debug("PostHog: Sync event successfully sent")
+
+ except Exception as e:
+ verbose_logger.exception(f"PostHog Sync Layer Error - {str(e)}")
+
+ async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
+ try:
+ verbose_logger.debug(
+ "PostHog: Async logging - Enters logging function for model %s", kwargs
+ )
+ self._ensure_async_setup() # Lazy initialization
+ await self._log_async_event(kwargs, response_obj, start_time, end_time)
+ except Exception as e:
+ verbose_logger.exception(f"PostHog Layer Error - {str(e)}")
+ pass
+
+ async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
+ try:
+ verbose_logger.debug(
+ "PostHog: Async logging - Enters logging function for model %s", kwargs
+ )
+ self._ensure_async_setup() # Lazy initialization
+ await self._log_async_event(kwargs, response_obj, start_time, end_time)
+ except Exception as e:
+ verbose_logger.exception(f"PostHog Layer Error - {str(e)}")
+ pass
+
+ async def _log_async_event(self, kwargs, response_obj=None, start_time=0.0, end_time=0.0):
+ # Note: response_obj, start_time, end_time not used - all data comes from kwargs
+ api_key, api_url = self._get_credentials_for_request(kwargs)
+ event_payload = self.create_posthog_event_payload(kwargs)
+
+ # Store event with its credentials for batch sending
+ self.log_queue.append({
+ "event": event_payload,
+ "api_key": api_key,
+ "api_url": api_url
+ })
+ verbose_logger.debug(
+ f"PostHog, event added to queue. Will flush in {self.flush_interval} seconds..."
+ )
+
+ if len(self.log_queue) >= self.batch_size:
+ await self.flush_queue()
+
+ def create_posthog_event_payload(self, kwargs: Dict[str, Any]) -> PostHogEventPayload:
+ """
+ Helper function to create a PostHog event payload for logging
+
+ Args:
+ kwargs (Dict[str, Any]): request kwargs containing standard_logging_object
+
+ Returns:
+ PostHogEventPayload: defined in types.py
+ """
+ standard_logging_object: Optional[StandardLoggingPayload] = kwargs.get(
+ "standard_logging_object", None
+ )
+ if standard_logging_object is None:
+ raise ValueError("standard_logging_object not found in kwargs")
+
+ call_type = standard_logging_object.get("call_type", "")
+ event_name = "$ai_embedding" if call_type == "embedding" else "$ai_generation"
+
+ properties = self._create_posthog_properties(
+ standard_logging_object=standard_logging_object,
+ kwargs=kwargs,
+ event_name=event_name,
+ )
+
+ distinct_id = self._get_distinct_id(standard_logging_object, kwargs)
+
+ return PostHogEventPayload(
+ event=event_name,
+ properties=properties,
+ distinct_id=distinct_id,
+ )
+
+ def _create_posthog_properties(
+ self,
+ standard_logging_object: StandardLoggingPayload,
+ kwargs: Dict[str, Any],
+ event_name: str,
+ ) -> Dict[str, Any]:
+ """Create PostHog properties following LLM Analytics spec"""
+ properties = {}
+
+ # Core model information
+ properties["$ai_model"] = self._safe_get(standard_logging_object, "model", "")
+ properties["$ai_provider"] = self._safe_get(standard_logging_object, "custom_llm_provider", "")
+
+ # Input/Output data
+ messages = self._safe_get(standard_logging_object, "messages")
+ if messages is not None:
+ properties["$ai_input"] = messages
+
+ if event_name == "$ai_generation":
+ response = self._safe_get(standard_logging_object, "response")
+ if response is not None:
+ properties["$ai_output_choices"] = response
+
+ # Token information
+ properties["$ai_input_tokens"] = self._safe_get(standard_logging_object, "prompt_tokens", 0)
+ if event_name == "$ai_generation":
+ properties["$ai_output_tokens"] = self._safe_get(standard_logging_object, "completion_tokens", 0)
+
+ # Cost and performance
+ response_cost = self._safe_get(standard_logging_object, "response_cost")
+ if response_cost is not None:
+ properties["$ai_total_cost_usd"] = response_cost
+
+ properties["$ai_latency"] = self._safe_get(standard_logging_object, "response_time", 0.0)
+
+ # Error handling
+ if self._safe_get(standard_logging_object, "status") == "failure":
+ properties["$ai_is_error"] = True
+ error_str = self._safe_get(standard_logging_object, "error_str")
+ if error_str is not None:
+ properties["$ai_error"] = error_str
+
+ # Add trace properties
+ self._add_trace_properties(properties, kwargs)
+
+ # Add custom metadata fields
+ self._add_custom_metadata_properties(properties, kwargs)
+
+ return properties
+
+ def _add_trace_properties(self, properties: Dict[str, Any], kwargs: Dict[str, Any]):
+ standard_logging_object = self._safe_get(kwargs, "standard_logging_object", {})
+
+ trace_id = self._safe_get(standard_logging_object, "trace_id", self._safe_uuid())
+ properties["$ai_trace_id"] = trace_id
+
+ span_id = self._safe_get(standard_logging_object, "id", self._safe_uuid())
+ properties["$ai_span_id"] = span_id
+
+ metadata = self._extract_metadata(kwargs)
+ parent_id = metadata.get("parent_run_id") or metadata.get("parent_id")
+ if parent_id:
+ properties["$ai_parent_id"] = parent_id
+
+ def _add_custom_metadata_properties(self, properties: Dict[str, Any], kwargs: Dict[str, Any]):
+ """Add custom metadata fields to PostHog properties"""
+ metadata = self._extract_metadata(kwargs)
+ if not isinstance(metadata, dict):
+ return
+
+ litellm_internal_fields = {
+ "endpoint", "caching_groups", "user_api_key_hash", "user_api_key_alias",
+ "user_api_key_team_id", "user_api_key_user_id", "user_api_key_org_id",
+ "user_api_key_team_alias", "user_api_key_end_user_id", "user_api_key_user_email",
+ "user_api_key", "user_api_end_user_max_budget", "litellm_api_version",
+ "global_max_parallel_requests", "user_api_key_team_max_budget", "user_api_key_team_spend",
+ "user_api_key_spend", "user_api_key_max_budget", "user_api_key_model_max_budget",
+ "user_api_key_metadata", "headers", "litellm_parent_otel_span", "requester_ip_address",
+ "model_group", "model_group_size", "deployment", "model_info", "api_base",
+ "caching_groups", "hidden_params", "parent_run_id", "parent_id", "user_id"
+ }
+
+ for key, value in metadata.items():
+ if key not in litellm_internal_fields:
+ properties[key] = value
+
+ def _get_distinct_id(
+ self, standard_logging_object: StandardLoggingPayload, kwargs: Dict[str, Any]
+ ) -> str:
+ metadata = self._extract_metadata(kwargs)
+ user_id = self._safe_get(metadata, "user_id")
+ if user_id:
+ return str(user_id)
+ end_user = self._safe_get(standard_logging_object, "end_user")
+ if end_user:
+ return str(end_user)
+ trace_id = self._safe_get(standard_logging_object, "trace_id")
+ if trace_id:
+ return str(trace_id)
+
+ return self._safe_uuid()
+
+ def _get_credentials_for_request(self, kwargs: Dict[str, Any]) -> Tuple[Optional[str], Optional[str]]:
+ """
+ Get PostHog credentials for this request.
+
+ Checks for per-request credentials in standard_callback_dynamic_params,
+ falls back to instance defaults from environment variables.
+
+ Args:
+ kwargs: Request kwargs containing standard_callback_dynamic_params
+
+ Returns:
+ tuple[str, str]: (api_key, api_url)
+ """
+ standard_callback_dynamic_params: Optional[StandardCallbackDynamicParams] = (
+ kwargs.get("standard_callback_dynamic_params", None)
+ )
+
+ if standard_callback_dynamic_params is not None:
+ api_key = standard_callback_dynamic_params.get("posthog_api_key") or self.POSTHOG_API_KEY
+ api_url = standard_callback_dynamic_params.get("posthog_api_url") or self.posthog_host
+ else:
+ api_key = self.POSTHOG_API_KEY
+ api_url = self.posthog_host
+
+ return api_key, api_url
+
+ async def async_send_batch(self):
+ """
+ Sends the in memory logs queue to PostHog API
+
+ Raises:
+ Raises a NON Blocking verbose_logger.exception if an error occurs
+ """
+ try:
+ if not self.log_queue:
+ return
+
+ verbose_logger.debug(
+ f"PostHog: Sending batch of {len(self.log_queue)} events"
+ )
+
+ # Group events by credentials for batch sending
+ batches_by_credentials: Dict[tuple[str, str], list] = {}
+ for item in self.log_queue:
+ key = (item["api_key"], item["api_url"])
+ if key not in batches_by_credentials:
+ batches_by_credentials[key] = []
+ batches_by_credentials[key].append(item["event"])
+
+ # Send each batch to its respective PostHog instance
+ for (api_key, api_url), events in batches_by_credentials.items():
+ headers = {
+ "Content-Type": "application/json",
+ }
+
+ payload = self._create_posthog_payload(events, api_key)
+ capture_url = f"{api_url.rstrip('/')}/batch/"
+
+ response = await self.async_client.post(
+ url=capture_url,
+ json=payload,
+ headers=headers,
+ )
+ response.raise_for_status()
+
+ if response.status_code != 200:
+ raise Exception(
+ f"Response from PostHog API status_code: {response.status_code}, text: {response.text}"
+ )
+
+ verbose_logger.debug(
+ f"PostHog: Batch of {len(self.log_queue)} events successfully sent"
+ )
+ except Exception as e:
+ verbose_logger.exception(f"PostHog Error sending batch API - {str(e)}")
+
+ def _ensure_async_setup(self):
+ if not self._async_initialized:
+ try:
+ self.flush_lock = asyncio.Lock()
+ asyncio.create_task(self.periodic_flush())
+ self._async_initialized = True
+ verbose_logger.debug("PostHog: Async components initialized")
+ except Exception as e:
+ verbose_logger.error(f"PostHog: Failed to initialize async components: {str(e)}")
+ raise
+
+ def _extract_metadata(self, kwargs: Dict[str, Any]) -> Dict[str, Any]:
+ litellm_params = kwargs.get("litellm_params", {}) or {}
+ return litellm_params.get("metadata", {}) or {}
+
+ def _safe_uuid(self) -> str:
+ return str(uuid.uuid4())
+
+ def _create_posthog_payload(self, events: list, api_key: str) -> Dict[str, Any]:
+ return {"api_key": api_key, "batch": events}
+
+ def _safe_get(self, obj: Any, key: str, default: Any = None) -> Any:
+ if obj is None or not hasattr(obj, 'get'):
+ return default
+ return obj.get(key, default)
diff --git a/litellm/integrations/prompt_management_base.py b/litellm/integrations/prompt_management_base.py
index 4a8bcd2e249..7754ca435ca 100644
--- a/litellm/integrations/prompt_management_base.py
+++ b/litellm/integrations/prompt_management_base.py
@@ -1,5 +1,7 @@
from abc import ABC, abstractmethod
-from typing import Any, Dict, List, Optional, Tuple, TypedDict
+from typing import Any, Dict, List, Optional, Tuple
+
+from typing_extensions import TypedDict
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import StandardCallbackDynamicParams
@@ -54,6 +56,7 @@ class PromptManagementBase(ABC):
prompt_label: Optional[str] = None,
prompt_version: Optional[int] = None,
) -> PromptManagementClient:
+
compiled_prompt_client = self._compile_prompt_helper(
prompt_id=prompt_id,
prompt_variables=prompt_variables,
@@ -91,6 +94,7 @@ class PromptManagementBase(ABC):
prompt_label: Optional[str] = None,
prompt_version: Optional[int] = None,
) -> Tuple[str, List[AllMessageValues], dict]:
+
if prompt_id is None:
raise ValueError("prompt_id is required for Prompt Management Base class")
if not self.should_run_prompt_management(
diff --git a/litellm/integrations/s3_v2.py b/litellm/integrations/s3_v2.py
index 7df3e58b2da..a65500c80dc 100644
--- a/litellm/integrations/s3_v2.py
+++ b/litellm/integrations/s3_v2.py
@@ -203,7 +203,7 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
start_time=start_time,
end_time=end_time,
)
-
+
async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
await self._async_log_event_base(
kwargs=kwargs,
@@ -212,7 +212,6 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
end_time=end_time,
)
pass
-
async def _async_log_event_base(self, kwargs, response_obj, start_time, end_time):
try:
@@ -242,7 +241,6 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
verbose_logger.exception(f"s3 Layer Error - {str(e)}")
pass
-
async def async_upload_data_to_s3(
self, batch_logging_element: s3BatchLoggingElement
):
@@ -277,8 +275,14 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
# Prepare the URL
url = f"https://{self.s3_bucket_name}.s3.{self.s3_region_name}.amazonaws.com/{batch_logging_element.s3_object_key}"
- if self.s3_endpoint_url:
- url = self.s3_endpoint_url + "/" + batch_logging_element.s3_object_key
+ if self.s3_endpoint_url and self.s3_bucket_name:
+ url = (
+ self.s3_endpoint_url
+ + "/"
+ + self.s3_bucket_name
+ + "/"
+ + batch_logging_element.s3_object_key
+ )
# Convert JSON to string
json_string = safe_dumps(batch_logging_element.payload)
@@ -304,7 +308,10 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
data=prepped.body,
headers=prepped.headers,
)
- SigV4Auth(credentials, "s3", self.s3_region_name).add_auth(aws_request)
+ aws_region_name = self.get_aws_region_name_for_non_llm_api_calls(
+ aws_region_name=self.s3_region_name
+ )
+ SigV4Auth(credentials, "s3", aws_region_name).add_auth(aws_request)
# Prepare the signed headers
signed_headers = dict(aws_request.headers.items())
@@ -417,8 +424,14 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
# Prepare the URL
url = f"https://{self.s3_bucket_name}.s3.{self.s3_region_name}.amazonaws.com/{batch_logging_element.s3_object_key}"
- if self.s3_endpoint_url:
- url = self.s3_endpoint_url + "/" + batch_logging_element.s3_object_key
+ if self.s3_endpoint_url and self.s3_bucket_name:
+ url = (
+ self.s3_endpoint_url
+ + "/"
+ + self.s3_bucket_name
+ + "/"
+ + batch_logging_element.s3_object_key
+ )
# Convert JSON to string
json_string = safe_dumps(batch_logging_element.payload)
@@ -444,7 +457,10 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
data=prepped.body,
headers=prepped.headers,
)
- SigV4Auth(credentials, "s3", self.s3_region_name).add_auth(aws_request)
+ aws_region_name = self.get_aws_region_name_for_non_llm_api_calls(
+ aws_region_name=self.s3_region_name
+ )
+ SigV4Auth(credentials, "s3", aws_region_name).add_auth(aws_request)
# Prepare the signed headers
signed_headers = dict(aws_request.headers.items())
@@ -455,3 +471,117 @@ class S3Logger(CustomBatchLogger, BaseAWSLLM):
response.raise_for_status()
except Exception as e:
verbose_logger.exception(f"Error uploading to s3: {str(e)}")
+
+ async def _download_object_from_s3(self, s3_object_key: str) -> Optional[dict]:
+ """
+ Download and parse JSON object from S3.
+
+ Args:
+ s3_object_key: The S3 object key to download
+
+ Returns:
+ Optional[dict]: The parsed JSON object or None if not found/error
+ """
+ try:
+ import hashlib
+
+ import requests
+ from botocore.auth import SigV4Auth
+ from botocore.awsrequest import AWSRequest
+ except ImportError:
+ raise ImportError("Missing boto3 to call S3. Run 'pip install boto3'.")
+
+ try:
+ from litellm.litellm_core_utils.asyncify import asyncify
+
+ # Get AWS credentials
+ asyncified_get_credentials = asyncify(self.get_credentials)
+ credentials = await asyncified_get_credentials(
+ aws_access_key_id=self.s3_aws_access_key_id,
+ aws_secret_access_key=self.s3_aws_secret_access_key,
+ aws_session_token=self.s3_aws_session_token,
+ aws_region_name=self.s3_region_name,
+ aws_session_name=self.s3_aws_session_name,
+ aws_profile_name=self.s3_aws_profile_name,
+ aws_role_name=self.s3_aws_role_name,
+ aws_web_identity_token=self.s3_aws_web_identity_token,
+ aws_sts_endpoint=self.s3_aws_sts_endpoint,
+ )
+
+ verbose_logger.debug(
+ f"s3_v2 logger - downloading data from s3 - {s3_object_key}"
+ )
+
+ # Prepare the URL
+ url = f"https://{self.s3_bucket_name}.s3.{self.s3_region_name}.amazonaws.com/{s3_object_key}"
+
+ if self.s3_endpoint_url and self.s3_bucket_name:
+ url = (
+ self.s3_endpoint_url
+ + "/"
+ + self.s3_bucket_name
+ + "/"
+ + s3_object_key
+ )
+
+ # Prepare the request for GET operation
+ # For GET requests, we need x-amz-content-sha256 with hash of empty string
+ empty_string_hash = hashlib.sha256(b"").hexdigest()
+ headers = {
+ "x-amz-content-sha256": empty_string_hash,
+ }
+ req = requests.Request("GET", url, headers=headers)
+ prepped = req.prepare()
+
+ # Sign the request
+ aws_request = AWSRequest(
+ method=prepped.method,
+ url=prepped.url,
+ headers=prepped.headers,
+ )
+ SigV4Auth(credentials, "s3", self.s3_region_name).add_auth(aws_request)
+
+ # Prepare the signed headers
+ signed_headers = dict(aws_request.headers.items())
+
+ # Make the request
+ response = await self.async_httpx_client.get(url, headers=signed_headers)
+
+ if response.status_code != 200:
+ verbose_logger.exception(
+ "S3 object not found, saw response=", response.text
+ )
+ return None
+
+ # Parse JSON response
+ return response.json()
+
+ except Exception as e:
+ verbose_logger.exception(f"Error downloading from S3: {str(e)}")
+ return None
+
+ async def get_proxy_server_request_from_cold_storage_with_object_key(
+ self,
+ object_key: str,
+ ) -> Optional[dict]:
+ """
+ Get the proxy server request from cold storage
+
+ Allows fetching a dict of the proxy server request from s3 or GCS bucket.
+
+ Args:
+ request_id: The unique request ID to search for
+ start_time: The start time of the request (datetime or ISO string)
+
+ Returns:
+ Optional[dict]: The request data dictionary or None if not found
+ """
+ try:
+ # Download and return the object from S3
+ downloaded_object = await self._download_object_from_s3(object_key)
+ return downloaded_object
+ except Exception as e:
+ verbose_logger.exception(
+ f"Error retrieving object {object_key} from cold storage: {str(e)}"
+ )
+ return None
diff --git a/litellm/integrations/sqs.py b/litellm/integrations/sqs.py
index 2a0c73dfdbf..8a2ebf8d344 100644
--- a/litellm/integrations/sqs.py
+++ b/litellm/integrations/sqs.py
@@ -7,6 +7,7 @@ This logger sends ``StandardLoggingPayload`` entries to an AWS SQS queue.
from __future__ import annotations
import asyncio
+import traceback
from typing import List, Optional
import litellm
@@ -200,6 +201,25 @@ class SQSLogger(CustomBatchLogger, BaseAWSLLM):
except Exception as e:
verbose_logger.exception(f"sqs Layer Error - {str(e)}")
+ async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
+ try:
+ standard_logging_payload = kwargs.get("standard_logging_object")
+ if standard_logging_payload is None:
+ raise ValueError("standard_logging_payload is None")
+
+ self.log_queue.append(standard_logging_payload)
+ verbose_logger.debug(
+ "sqs logging: queue length %s, batch size %s",
+ len(self.log_queue),
+ self.batch_size,
+ )
+
+ except Exception as e:
+ verbose_logger.exception(
+ f"Datadog Layer Error - {str(e)}\n{traceback.format_exc()}"
+ )
+ pass
+
async def async_send_batch(self) -> None:
verbose_logger.debug(
f"sqs logger - sending batch of {len(self.log_queue)}"
diff --git a/litellm/integrations/vector_store_integrations/vector_store_pre_call_hook.py b/litellm/integrations/vector_store_integrations/vector_store_pre_call_hook.py
index 59c378f8204..8ef160dd783 100644
--- a/litellm/integrations/vector_store_integrations/vector_store_pre_call_hook.py
+++ b/litellm/integrations/vector_store_integrations/vector_store_pre_call_hook.py
@@ -8,6 +8,7 @@ It searches the vector store for relevant context and appends it to the messages
from typing import TYPE_CHECKING, Dict, List, Optional, Tuple, cast
import litellm
+import litellm.vector_stores
from litellm._logging import verbose_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.llms.openai import AllMessageValues, ChatCompletionUserMessage
@@ -192,4 +193,4 @@ class VectorStorePreCallHook(CustomLogger):
modified_messages.insert(-1, cast(AllMessageValues, context_message))
return modified_messages
- return messages
\ No newline at end of file
+ return messages
diff --git a/litellm/integrations/weights_biases.py b/litellm/integrations/weights_biases.py
index 63d87c9bd90..0d011e26aef 100644
--- a/litellm/integrations/weights_biases.py
+++ b/litellm/integrations/weights_biases.py
@@ -44,7 +44,7 @@ try:
request, response, time_elapsed
)
else:
- logger.info(f"Unknown OpenAI response object: {response['object']}")
+ logger.debug(f"Unknown OpenAI response object: {response['object']}")
except Exception as e:
logger.warning(f"Failed to resolve request/response: {e}")
return None
diff --git a/litellm/litellm_core_utils/cached_imports.py b/litellm/litellm_core_utils/cached_imports.py
new file mode 100644
index 00000000000..c3ab292e9c5
--- /dev/null
+++ b/litellm/litellm_core_utils/cached_imports.py
@@ -0,0 +1,56 @@
+"""
+Cached imports module for LiteLLM.
+
+This module provides cached import functionality to avoid repeated imports
+inside functions that are critical to performance.
+"""
+
+from typing import TYPE_CHECKING, Callable, Optional, Type
+
+# Type annotations for cached imports
+if TYPE_CHECKING:
+ from litellm.litellm_core_utils.litellm_logging import Logging
+ from litellm.litellm_core_utils.coroutine_checker import CoroutineChecker
+
+# Global cache variables
+_LiteLLMLogging: Optional[Type["Logging"]] = None
+_coroutine_checker: Optional["CoroutineChecker"] = None
+_set_callbacks: Optional[Callable] = None
+
+
+def get_litellm_logging_class() -> Type["Logging"]:
+ """Get the cached LiteLLM Logging class, initializing if needed."""
+ global _LiteLLMLogging
+ if _LiteLLMLogging is not None:
+ return _LiteLLMLogging
+ from litellm.litellm_core_utils.litellm_logging import Logging
+ _LiteLLMLogging = Logging
+ return _LiteLLMLogging
+
+
+def get_coroutine_checker() -> "CoroutineChecker":
+ """Get the cached coroutine checker instance, initializing if needed."""
+ global _coroutine_checker
+ if _coroutine_checker is not None:
+ return _coroutine_checker
+ from litellm.litellm_core_utils.coroutine_checker import coroutine_checker
+ _coroutine_checker = coroutine_checker
+ return _coroutine_checker
+
+
+def get_set_callbacks() -> Callable:
+ """Get the cached set_callbacks function, initializing if needed."""
+ global _set_callbacks
+ if _set_callbacks is not None:
+ return _set_callbacks
+ from litellm.litellm_core_utils.litellm_logging import set_callbacks
+ _set_callbacks = set_callbacks
+ return _set_callbacks
+
+
+def clear_cached_imports() -> None:
+ """Clear all cached imports. Useful for testing or memory management."""
+ global _LiteLLMLogging, _coroutine_checker, _set_callbacks
+ _LiteLLMLogging = None
+ _coroutine_checker = None
+ _set_callbacks = None
diff --git a/litellm/litellm_core_utils/cli_token_utils.py b/litellm/litellm_core_utils/cli_token_utils.py
new file mode 100644
index 00000000000..2aedb1c19d2
--- /dev/null
+++ b/litellm/litellm_core_utils/cli_token_utils.py
@@ -0,0 +1,58 @@
+"""
+CLI Token Utilities
+
+SDK-level utilities for reading CLI authentication tokens.
+This module has no dependencies on proxy code and can be safely imported at the SDK level.
+"""
+
+import json
+import os
+from pathlib import Path
+from typing import Optional
+
+
+def get_cli_token_file_path() -> str:
+ """Get the path to the CLI token file"""
+ home_dir = Path.home()
+ config_dir = home_dir / ".litellm"
+ return str(config_dir / "token.json")
+
+
+def load_cli_token() -> Optional[dict]:
+ """Load CLI token data from file"""
+ token_file = get_cli_token_file_path()
+ if not os.path.exists(token_file):
+ return None
+
+ try:
+ with open(token_file, 'r') as f:
+ return json.load(f)
+ except (json.JSONDecodeError, IOError):
+ return None
+
+
+def get_litellm_gateway_api_key() -> Optional[str]:
+ """
+ Get the stored CLI API key for use with LiteLLM SDK.
+
+ This function reads the token file created by `litellm-proxy login`
+ and returns the API key for use in Python scripts.
+
+ Returns:
+ str: The API key if found, None otherwise
+
+ Example:
+ >>> import litellm
+ >>> api_key = litellm.get_litellm_gateway_api_key()
+ >>> if api_key:
+ >>> response = litellm.completion(
+ >>> model="gpt-3.5-turbo",
+ >>> messages=[{"role": "user", "content": "Hello"}],
+ >>> api_key=api_key,
+ >>> base_url="https://your-proxy.com/v1"
+ >>> )
+ """
+ token_data = load_cli_token()
+ if token_data and 'key' in token_data:
+ return token_data['key']
+ return None
diff --git a/litellm/litellm_core_utils/core_helpers.py b/litellm/litellm_core_utils/core_helpers.py
index 86e7eb89a21..7423e55b626 100644
--- a/litellm/litellm_core_utils/core_helpers.py
+++ b/litellm/litellm_core_utils/core_helpers.py
@@ -18,27 +18,46 @@ else:
def safe_divide_seconds(
- seconds: float,
- denominator: float,
- default: Optional[float] = None
+ seconds: float, denominator: float, default: Optional[float] = None
) -> Optional[float]:
"""
Safely divide seconds by denominator, handling zero division.
-
+
Args:
seconds: Time duration in seconds
denominator: The divisor (e.g., number of tokens)
default: Value to return if division by zero (defaults to None)
-
+
Returns:
The result of the division as a float (seconds per unit), or default if denominator is zero
"""
if denominator <= 0:
return default
-
+
return float(seconds / denominator)
+def safe_divide(
+ numerator: Union[int, float],
+ denominator: Union[int, float],
+ default: Union[int, float] = 0
+) -> Union[int, float]:
+ """
+ Safely divide two numbers, returning a default value if denominator is zero.
+
+ Args:
+ numerator: The number to divide
+ denominator: The number to divide by
+ default: Value to return if denominator is zero (defaults to 0)
+
+ Returns:
+ The result of numerator/denominator, or default if denominator is zero
+ """
+ if denominator == 0:
+ return default
+ return numerator / denominator
+
+
def map_finish_reason(
finish_reason: str,
): # openai supports 5 stop sequences - 'stop', 'length', 'function_call', 'content_filter', 'null'
@@ -203,3 +222,65 @@ def preserve_upstream_non_openai_attributes(
for key, value in original_chunk.model_dump().items():
if key not in expected_keys:
setattr(model_response, key, value)
+
+
+def safe_deep_copy(data):
+ """
+ Safe Deep Copy
+
+ The LiteLLM request may contain objects that cannot be pickled/deep-copied
+ (e.g., tracing spans, locks, clients).
+
+ This helper deep-copies each top-level key independently; on failure keeps
+ original ref
+ """
+ import copy
+
+ import litellm
+
+ if litellm.safe_memory_mode is True:
+ return data
+
+ litellm_parent_otel_span: Optional[Any] = None
+ # Step 1: Remove the litellm_parent_otel_span
+ litellm_parent_otel_span = None
+ if isinstance(data, dict):
+ # remove litellm_parent_otel_span since this is not picklable
+ if "metadata" in data and "litellm_parent_otel_span" in data["metadata"]:
+ litellm_parent_otel_span = data["metadata"].pop("litellm_parent_otel_span")
+ data["metadata"]["litellm_parent_otel_span"] = "placeholder"
+ if (
+ "litellm_metadata" in data
+ and "litellm_parent_otel_span" in data["litellm_metadata"]
+ ):
+ litellm_parent_otel_span = data["litellm_metadata"].pop(
+ "litellm_parent_otel_span"
+ )
+ data["litellm_metadata"]["litellm_parent_otel_span"] = "placeholder"
+
+ # Step 2: Per-key deepcopy with fallback
+ if isinstance(data, dict):
+ new_data = {}
+ for k, v in data.items():
+ try:
+ new_data[k] = copy.deepcopy(v)
+ except Exception:
+ new_data[k] = v
+ else:
+ try:
+ new_data = copy.deepcopy(data)
+ except Exception:
+ new_data = data
+
+ # Step 3: re-add the litellm_parent_otel_span after doing a deep copy
+ if isinstance(data, dict) and litellm_parent_otel_span is not None:
+ if "metadata" in data and "litellm_parent_otel_span" in data["metadata"]:
+ data["metadata"]["litellm_parent_otel_span"] = litellm_parent_otel_span
+ if (
+ "litellm_metadata" in data
+ and "litellm_parent_otel_span" in data["litellm_metadata"]
+ ):
+ data["litellm_metadata"][
+ "litellm_parent_otel_span"
+ ] = litellm_parent_otel_span
+ return new_data
\ No newline at end of file
diff --git a/litellm/litellm_core_utils/coroutine_checker.py b/litellm/litellm_core_utils/coroutine_checker.py
new file mode 100644
index 00000000000..368aee62ed0
--- /dev/null
+++ b/litellm/litellm_core_utils/coroutine_checker.py
@@ -0,0 +1,63 @@
+# CoroutineChecker utility for checking if functions/callables are coroutines or coroutine functions
+
+import inspect
+from typing import Any
+from weakref import WeakKeyDictionary
+from litellm.constants import (
+ COROUTINE_CHECKER_MAX_SIZE_IN_MEMORY,
+)
+
+
+class CoroutineChecker:
+ """Utility class for checking coroutine status of functions and callables.
+
+ Simple bounded cache using WeakKeyDictionary to avoid memory leaks.
+ """
+
+ def __init__(self):
+ self._cache = WeakKeyDictionary()
+ self._max_size = COROUTINE_CHECKER_MAX_SIZE_IN_MEMORY
+
+ def is_async_callable(self, callback: Any) -> bool:
+ """Fast, cached check for whether a callback is an async function.
+ Falls back gracefully if the object cannot be weak-referenced or cached.
+ 2.59x speedup.
+ """
+ # Fast path: check cache first (most common case)
+ try:
+ cached = self._cache.get(callback)
+ if cached is not None:
+ return cached
+ except Exception:
+ pass
+
+ # Determine target - optimized path for common cases
+ target = callback
+ if not inspect.isfunction(target) and not inspect.ismethod(target):
+ try:
+ call_attr = getattr(target, "__call__", None)
+ if call_attr is not None:
+ target = call_attr
+ except Exception:
+ pass
+
+ # Compute result
+ try:
+ result = inspect.iscoroutinefunction(target)
+ except Exception:
+ result = False
+
+ # Cache the result with size enforcement
+ try:
+ # Simple size enforcement: clear cache if it gets too large
+ if len(self._cache) >= self._max_size:
+ self._cache.clear()
+
+ self._cache[callback] = result
+ except Exception:
+ pass
+
+ return result
+
+# Global instance for backward compatibility and convenience
+coroutine_checker = CoroutineChecker()
diff --git a/litellm/litellm_core_utils/custom_logger_registry.py b/litellm/litellm_core_utils/custom_logger_registry.py
index 252fb29eb3a..09794bf2677 100644
--- a/litellm/litellm_core_utils/custom_logger_registry.py
+++ b/litellm/litellm_core_utils/custom_logger_registry.py
@@ -7,12 +7,16 @@ Example:
"datadog" -> DataDogLogger
"prometheus" -> PrometheusLogger
"""
+
from typing import Union
+from litellm import _custom_logger_compatible_callbacks_literal
from litellm.integrations.agentops import AgentOps
from litellm.integrations.anthropic_cache_control_hook import AnthropicCacheControlHook
from litellm.integrations.argilla import ArgillaLogger
from litellm.integrations.azure_storage.azure_storage import AzureBlobStorageLogger
+from litellm.integrations.bitbucket import BitBucketPromptManager
+from litellm.integrations.gitlab import GitLabPromptManager
from litellm.integrations.braintrust_logging import BraintrustLogger
from litellm.integrations.datadog.datadog import DataDogLogger
from litellm.integrations.datadog.datadog_llm_obs import DataDogLLMObsLogger
@@ -31,22 +35,28 @@ from litellm.integrations.mlflow import MlflowLogger
from litellm.integrations.openmeter import OpenMeterLogger
from litellm.integrations.opentelemetry import OpenTelemetry
from litellm.integrations.opik.opik import OpikLogger
+from litellm.integrations.posthog import PostHogLogger
+
try:
from litellm_enterprise.integrations.prometheus import PrometheusLogger
except Exception:
PrometheusLogger = None
+from litellm.integrations.cloudzero.cloudzero import CloudZeroLogger
+from litellm.integrations.dotprompt import DotpromptManager
from litellm.integrations.s3_v2 import S3Logger
from litellm.integrations.sqs import SQSLogger
from litellm.integrations.vector_store_integrations.vector_store_pre_call_hook import (
VectorStorePreCallHook,
)
from litellm.proxy.hooks.dynamic_rate_limiter import _PROXY_DynamicRateLimitHandler
+from litellm.proxy.hooks.dynamic_rate_limiter_v3 import _PROXY_DynamicRateLimitHandlerV3
class CustomLoggerRegistry:
"""
Registry mapping the callback class string to the class type.
"""
+
CALLBACK_CLASS_STR_TO_CLASS_TYPE = {
"lago": LagoLogger,
"openmeter": OpenMeterLogger,
@@ -79,7 +89,13 @@ class CustomLoggerRegistry:
"s3_v2": S3Logger,
"aws_sqs": SQSLogger,
"dynamic_rate_limiter": _PROXY_DynamicRateLimitHandler,
+ "dynamic_rate_limiter_v3": _PROXY_DynamicRateLimitHandlerV3,
"vector_store_pre_call_hook": VectorStorePreCallHook,
+ "dotprompt": DotpromptManager,
+ "bitbucket": BitBucketPromptManager,
+ "gitlab": GitLabPromptManager,
+ "cloudzero": CloudZeroLogger,
+ "posthog": PostHogLogger,
}
try:
@@ -110,14 +126,17 @@ class CustomLoggerRegistry:
def get_callback_str_from_class_type(cls, class_type: type) -> Union[str, None]:
"""
Get the callback string from the class type.
-
+
Args:
class_type: The class type to find the string for
-
+
Returns:
str: The callback string, or None if not found
"""
- for callback_str, callback_class in cls.CALLBACK_CLASS_STR_TO_CLASS_TYPE.items():
+ for (
+ callback_str,
+ callback_class,
+ ) in cls.CALLBACK_CLASS_STR_TO_CLASS_TYPE.items():
if callback_class == class_type:
return callback_str
return None
@@ -127,15 +146,28 @@ class CustomLoggerRegistry:
"""
Get all callback strings that map to the same class type.
Some class types (like OpenTelemetry) have multiple string mappings.
-
+
Args:
class_type: The class type to find all strings for
-
+
Returns:
list: List of callback strings that map to the class type
"""
callback_strs: list[str] = []
- for callback_str, callback_class in cls.CALLBACK_CLASS_STR_TO_CLASS_TYPE.items():
+ for (
+ callback_str,
+ callback_class,
+ ) in cls.CALLBACK_CLASS_STR_TO_CLASS_TYPE.items():
if callback_class == class_type:
callback_strs.append(callback_str)
- return callback_strs
\ No newline at end of file
+ return callback_strs
+
+ @classmethod
+ def get_class_type_for_custom_logger_name(
+ cls,
+ custom_logger_name: _custom_logger_compatible_callbacks_literal,
+ ) -> type:
+ """
+ Get the class type for a given custom logger name
+ """
+ return cls.CALLBACK_CLASS_STR_TO_CLASS_TYPE[custom_logger_name]
diff --git a/litellm/litellm_core_utils/duration_parser.py b/litellm/litellm_core_utils/duration_parser.py
index 08f1d4c82d0..9a317cfcf0d 100644
--- a/litellm/litellm_core_utils/duration_parser.py
+++ b/litellm/litellm_core_utils/duration_parser.py
@@ -1,7 +1,7 @@
"""
Helper utilities for parsing durations - 1s, 1d, 10d, 30d, 1mo, 2mo
-duration_in_seconds is used in diff parts of the code base, example
+duration_in_seconds is used in diff parts of the code base, example
- Router - Provider budget routing
- Proxy - Key, Team Generation
"""
@@ -158,6 +158,7 @@ def _setup_timezone(
"US/Eastern": timezone(timedelta(hours=-4)), # EDT
"US/Pacific": timezone(timedelta(hours=-7)), # PDT
"Asia/Kolkata": timezone(timedelta(hours=5, minutes=30)), # IST
+ "Asia/Bangkok": timezone(timedelta(hours=7)), # ICT (Indochina Time)
"Europe/London": timezone(timedelta(hours=1)), # BST
"UTC": timezone.utc,
}
@@ -192,6 +193,10 @@ def _handle_day_reset(
current_time: datetime, base_midnight: datetime, value: int, timezone: timezone
) -> datetime:
"""Handle day-based reset times."""
+ # Handle zero value - immediate expiration
+ if value == 0:
+ return current_time
+
if value == 1: # Daily reset at midnight
return base_midnight + timedelta(days=1)
elif value == 7: # Weekly reset on Monday at midnight
@@ -234,6 +239,10 @@ def _handle_hour_reset(
current_time: datetime, base_midnight: datetime, value: int
) -> datetime:
"""Handle hour-based reset times."""
+ # Handle zero value - immediate expiration
+ if value == 0:
+ return current_time
+
current_hour = current_time.hour
current_minute = current_time.minute
current_second = current_time.second
@@ -266,6 +275,10 @@ def _handle_minute_reset(
current_time: datetime, base_midnight: datetime, value: int
) -> datetime:
"""Handle minute-based reset times."""
+ # Handle zero value - immediate expiration
+ if value == 0:
+ return current_time
+
current_hour = current_time.hour
current_minute = current_time.minute
current_second = current_time.second
@@ -306,6 +319,10 @@ def _handle_second_reset(
current_time: datetime, base_midnight: datetime, value: int
) -> datetime:
"""Handle second-based reset times."""
+ # Handle zero value - immediate expiration
+ if value == 0:
+ return current_time
+
current_hour = current_time.hour
current_minute = current_time.minute
current_second = current_time.second
diff --git a/litellm/litellm_core_utils/exception_mapping_utils.py b/litellm/litellm_core_utils/exception_mapping_utils.py
index 25ae0269ab3..c6d3637ffcb 100644
--- a/litellm/litellm_core_utils/exception_mapping_utils.py
+++ b/litellm/litellm_core_utils/exception_mapping_utils.py
@@ -6,6 +6,7 @@ import httpx
import litellm
from litellm._logging import verbose_logger
+from litellm.types.utils import LlmProviders
from ..exceptions import (
APIConnectionError,
@@ -556,7 +557,7 @@ def exception_type( # type: ignore # noqa: PLR0915
model=model,
llm_provider="anthropic",
)
- elif "overloaded_error" in error_str:
+ elif "overloaded_error" in error_str or "Overloaded" in error_str:
exception_mapping_worked = True
raise InternalServerError(
message="AnthropicError - {}".format(error_str),
@@ -762,7 +763,7 @@ def exception_type( # type: ignore # noqa: PLR0915
error_str += "XXXXXXX" + '"'
raise AuthenticationError(
- message=f"{custom_llm_provider}Exception: Authentication Error - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception: Authentication Error - {error_str}",
llm_provider=custom_llm_provider,
model=model,
response=getattr(original_exception, "response", None),
@@ -771,14 +772,14 @@ def exception_type( # type: ignore # noqa: PLR0915
elif "model's maximum context limit" in error_str:
exception_mapping_worked = True
raise ContextWindowExceededError(
- message=f"{custom_llm_provider}Exception: Context Window Error - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception: Context Window Error - {error_str}",
model=model,
llm_provider=custom_llm_provider,
)
elif "token_quota_reached" in error_str:
exception_mapping_worked = True
raise RateLimitError(
- message=f"{custom_llm_provider}Exception: Rate Limit Errror - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception: Rate Limit Errror - {error_str}",
llm_provider=custom_llm_provider,
model=model,
response=getattr(original_exception, "response", None),
@@ -789,14 +790,14 @@ def exception_type( # type: ignore # noqa: PLR0915
):
exception_mapping_worked = True
raise litellm.InternalServerError(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
llm_provider=custom_llm_provider,
model=model,
)
elif "model_no_support_for_function" in error_str:
exception_mapping_worked = True
raise BadRequestError(
- message=f"{custom_llm_provider}Exception - Use 'watsonx_text' route instead. IBM WatsonX does not support `/text/chat` endpoint. - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception - Use 'watsonx_text' route instead. IBM WatsonX does not support `/text/chat` endpoint. - {error_str}",
llm_provider=custom_llm_provider,
model=model,
)
@@ -804,7 +805,7 @@ def exception_type( # type: ignore # noqa: PLR0915
if original_exception.status_code == 500:
exception_mapping_worked = True
raise litellm.InternalServerError(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
llm_provider=custom_llm_provider,
model=model,
)
@@ -814,28 +815,28 @@ def exception_type( # type: ignore # noqa: PLR0915
):
exception_mapping_worked = True
raise AuthenticationError(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
llm_provider=custom_llm_provider,
model=model,
)
elif original_exception.status_code == 400:
exception_mapping_worked = True
raise BadRequestError(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
llm_provider=custom_llm_provider,
model=model,
)
elif original_exception.status_code == 404:
exception_mapping_worked = True
raise NotFoundError(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
llm_provider=custom_llm_provider,
model=model,
)
elif original_exception.status_code == 408:
exception_mapping_worked = True
raise Timeout(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
model=model,
llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
@@ -846,7 +847,7 @@ def exception_type( # type: ignore # noqa: PLR0915
):
exception_mapping_worked = True
raise BadRequestError(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
model=model,
llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
@@ -854,7 +855,7 @@ def exception_type( # type: ignore # noqa: PLR0915
elif original_exception.status_code == 429:
exception_mapping_worked = True
raise RateLimitError(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
model=model,
llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
@@ -862,7 +863,7 @@ def exception_type( # type: ignore # noqa: PLR0915
elif original_exception.status_code == 503:
exception_mapping_worked = True
raise ServiceUnavailableError(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
model=model,
llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
@@ -870,7 +871,7 @@ def exception_type( # type: ignore # noqa: PLR0915
elif original_exception.status_code == 504: # gateway timeout error
exception_mapping_worked = True
raise Timeout(
- message=f"{custom_llm_provider}Exception - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {original_exception.message}",
model=model,
llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
@@ -1168,9 +1169,9 @@ def exception_type( # type: ignore # noqa: PLR0915
exception_status_code=original_exception.status_code,
)
elif (
- custom_llm_provider == "vertex_ai"
- or custom_llm_provider == "vertex_ai_beta"
- or custom_llm_provider == "gemini"
+ custom_llm_provider == LlmProviders.VERTEX_AI
+ or custom_llm_provider == LlmProviders.VERTEX_AI_BETA
+ or custom_llm_provider == LlmProviders.GEMINI
):
if (
"Vertex AI API has not been used in project" in error_str
@@ -1178,9 +1179,9 @@ def exception_type( # type: ignore # noqa: PLR0915
):
exception_mapping_worked = True
raise BadRequestError(
- message=f"litellm.BadRequestError: VertexAIException - {error_str}",
+ message=f"litellm.BadRequestError: {custom_llm_provider}Exception - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
response=httpx.Response(
status_code=400,
request=httpx.Request(
@@ -1193,7 +1194,7 @@ def exception_type( # type: ignore # noqa: PLR0915
if "400 Request payload size exceeds" in error_str:
exception_mapping_worked = True
raise ContextWindowExceededError(
- message=f"VertexException - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {error_str}",
model=model,
llm_provider=custom_llm_provider,
)
@@ -1203,9 +1204,9 @@ def exception_type( # type: ignore # noqa: PLR0915
):
exception_mapping_worked = True
raise litellm.InternalServerError(
- message=f"litellm.InternalServerError: VertexAIException - {error_str}",
+ message=f"litellm.InternalServerError: {custom_llm_provider}Exception - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
response=httpx.Response(
status_code=500,
content=str(original_exception),
@@ -1216,7 +1217,7 @@ def exception_type( # type: ignore # noqa: PLR0915
elif "API key not valid." in error_str:
exception_mapping_worked = True
raise AuthenticationError(
- message=f"{custom_llm_provider}Exception - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {error_str}",
model=model,
llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
@@ -1224,9 +1225,9 @@ def exception_type( # type: ignore # noqa: PLR0915
elif "403" in error_str:
exception_mapping_worked = True
raise BadRequestError(
- message=f"VertexAIException BadRequestError - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception BadRequestError - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
response=httpx.Response(
status_code=403,
request=httpx.Request(
@@ -1243,9 +1244,9 @@ def exception_type( # type: ignore # noqa: PLR0915
):
exception_mapping_worked = True
raise ContentPolicyViolationError(
- message=f"VertexAIException ContentPolicyViolationError - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception ContentPolicyViolationError - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
response=httpx.Response(
status_code=400,
@@ -1264,9 +1265,9 @@ def exception_type( # type: ignore # noqa: PLR0915
):
exception_mapping_worked = True
raise RateLimitError(
- message=f"litellm.RateLimitError: VertexAIException - {error_str}",
+ message=f"litellm.RateLimitError: {custom_llm_provider}Exception - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
response=httpx.Response(
status_code=429,
@@ -1282,18 +1283,18 @@ def exception_type( # type: ignore # noqa: PLR0915
):
exception_mapping_worked = True
raise litellm.InternalServerError(
- message=f"litellm.InternalServerError: VertexAIException - {error_str}",
+ message=f"litellm.InternalServerError: {custom_llm_provider}Exception - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
)
if hasattr(original_exception, "status_code"):
if original_exception.status_code == 400:
exception_mapping_worked = True
raise BadRequestError(
- message=f"VertexAIException BadRequestError - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception BadRequestError - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
response=httpx.Response(
status_code=400,
@@ -1306,21 +1307,35 @@ def exception_type( # type: ignore # noqa: PLR0915
if original_exception.status_code == 401:
exception_mapping_worked = True
raise AuthenticationError(
- message=f"VertexAIException - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {error_str}",
llm_provider=custom_llm_provider,
model=model,
)
+ if original_exception.status_code == 403:
+ exception_mapping_worked = True
+ raise PermissionDeniedError(
+ message=f"{custom_llm_provider.capitalize()}Exception - {error_str}",
+ llm_provider=custom_llm_provider,
+ model=model,
+ response=httpx.Response(
+ status_code=403,
+ request=httpx.Request(
+ method="POST",
+ url="https://cloud.google.com/vertex-ai/",
+ ),
+ ),
+ )
if original_exception.status_code == 404:
exception_mapping_worked = True
raise NotFoundError(
- message=f"VertexAIException - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {error_str}",
llm_provider=custom_llm_provider,
model=model,
)
if original_exception.status_code == 408:
exception_mapping_worked = True
raise Timeout(
- message=f"VertexAIException - {original_exception.message}",
+ message=f"{custom_llm_provider.capitalize()}Exception - {error_str}",
llm_provider=custom_llm_provider,
model=model,
)
@@ -1328,9 +1343,9 @@ def exception_type( # type: ignore # noqa: PLR0915
if original_exception.status_code == 429:
exception_mapping_worked = True
raise RateLimitError(
- message=f"litellm.RateLimitError: VertexAIException - {error_str}",
+ message=f"litellm.RateLimitError: {custom_llm_provider.capitalize()}Exception - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
response=httpx.Response(
status_code=429,
@@ -1343,9 +1358,9 @@ def exception_type( # type: ignore # noqa: PLR0915
if original_exception.status_code == 500:
exception_mapping_worked = True
raise litellm.InternalServerError(
- message=f"VertexAIException InternalServerError - {error_str}",
+ message=f"{custom_llm_provider.capitalize()}Exception InternalServerError - {error_str}",
model=model,
- llm_provider="vertex_ai",
+ llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
response=httpx.Response(
status_code=500,
@@ -1353,71 +1368,20 @@ def exception_type( # type: ignore # noqa: PLR0915
request=httpx.Request(method="completion", url="https://github.com/BerriAI/litellm"), # type: ignore
),
)
- if original_exception.status_code == 503:
+ if original_exception.status_code == 502:
exception_mapping_worked = True
- raise ServiceUnavailableError(
- message=f"VertexAIException - {original_exception.message}",
+ raise APIConnectionError(
+ message=f"{custom_llm_provider.capitalize()}Exception - {error_str}",
llm_provider=custom_llm_provider,
model=model,
)
- elif custom_llm_provider == "palm" or custom_llm_provider == "gemini":
- if "503 Getting metadata" in error_str:
- # auth errors look like this
- # 503 Getting metadata from plugin failed with error: Reauthentication is needed. Please run `gcloud auth application-default login` to reauthenticate.
- exception_mapping_worked = True
- raise BadRequestError(
- message="GeminiException - Invalid api key",
- model=model,
- llm_provider="palm",
- response=getattr(original_exception, "response", None),
- )
- if (
- "504 Deadline expired before operation could complete." in error_str
- or "504 Deadline Exceeded" in error_str
- ):
- exception_mapping_worked = True
- raise Timeout(
- message=f"GeminiException - {original_exception.message}",
- model=model,
- llm_provider="palm",
- exception_status_code=original_exception.status_code,
- )
- if "400 Request payload size exceeds" in error_str:
- exception_mapping_worked = True
- raise ContextWindowExceededError(
- message=f"GeminiException - {error_str}",
- model=model,
- llm_provider="palm",
- response=getattr(original_exception, "response", None),
- )
- if (
- "500 An internal error has occurred." in error_str
- or "list index out of range" in error_str
- ):
- exception_mapping_worked = True
- raise APIError(
- status_code=getattr(original_exception, "status_code", 500),
- message=f"GeminiException - {original_exception.message}",
- llm_provider="palm",
- model=model,
- request=httpx.Response(
- status_code=429,
- request=httpx.Request(
- method="POST",
- url=" https://cloud.google.com/vertex-ai/",
- ),
- ),
- )
- if hasattr(original_exception, "status_code"):
- if original_exception.status_code == 400:
+ if original_exception.status_code == 503:
exception_mapping_worked = True
- raise BadRequestError(
- message=f"GeminiException - {error_str}",
+ raise ServiceUnavailableError(
+ message=f"{custom_llm_provider.capitalize()}Exception - {error_str}",
+ llm_provider=custom_llm_provider,
model=model,
- llm_provider="palm",
- response=getattr(original_exception, "response", None),
)
- # Dailed: Error occurred: 400 Request payload size exceeds the limit: 20000 bytes
elif custom_llm_provider == "cloudflare":
if "Authentication error" in error_str:
exception_mapping_worked = True
@@ -1449,6 +1413,14 @@ def exception_type( # type: ignore # noqa: PLR0915
model=model,
response=getattr(original_exception, "response", None),
)
+ elif "invalid type: parameter" in error_str:
+ exception_mapping_worked = True
+ raise BadRequestError(
+ message=f"CohereException - {original_exception.message}",
+ llm_provider="cohere",
+ model=model,
+ response=getattr(original_exception, "response", None),
+ )
elif "too many tokens" in error_str:
exception_mapping_worked = True
raise ContextWindowExceededError(
@@ -1526,7 +1498,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"CohereException - {original_exception.message}",
llm_provider="cohere",
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
raise original_exception
elif custom_llm_provider == "huggingface":
@@ -1601,7 +1573,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"HuggingfaceException - {original_exception.message}",
llm_provider="huggingface",
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
elif custom_llm_provider == "ai21":
if hasattr(original_exception, "message"):
@@ -1660,7 +1632,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"AI21Exception - {original_exception.message}",
llm_provider="ai21",
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
elif custom_llm_provider == "nlp_cloud":
if "detail" in error_str:
@@ -1687,7 +1659,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"NLPCloudException - {error_str}",
model=model,
llm_provider="nlp_cloud",
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
if hasattr(
original_exception, "status_code"
@@ -1747,7 +1719,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"NLPCloudException - {original_exception.message}",
llm_provider="nlp_cloud",
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
elif (
original_exception.status_code == 504
@@ -1767,7 +1739,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"NLPCloudException - {original_exception.message}",
llm_provider="nlp_cloud",
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
elif custom_llm_provider == "together_ai":
try:
@@ -1876,7 +1848,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"TogetherAIException - {original_exception.message}",
llm_provider="together_ai",
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
elif custom_llm_provider == "aleph_alpha":
if (
@@ -1981,7 +1953,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"VLLMException - {original_exception.message}",
llm_provider="vllm",
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
elif custom_llm_provider == "azure" or custom_llm_provider == "azure_text":
message = get_error_message(error_obj=original_exception)
@@ -2236,7 +2208,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message=f"APIError: {exception_provider} - {error_str}",
llm_provider=custom_llm_provider,
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
litellm_debug_info=extra_information,
)
else:
@@ -2271,7 +2243,7 @@ def exception_type( # type: ignore # noqa: PLR0915
message="{} - {}".format(exception_provider, error_str),
llm_provider=custom_llm_provider,
model=model,
- request=original_exception.request,
+ request=getattr(original_exception, "request", None),
)
else:
raise APIConnectionError(
diff --git a/litellm/litellm_core_utils/fallback_utils.py b/litellm/litellm_core_utils/fallback_utils.py
index d5610d5fddf..7ce53862089 100644
--- a/litellm/litellm_core_utils/fallback_utils.py
+++ b/litellm/litellm_core_utils/fallback_utils.py
@@ -1,9 +1,9 @@
-import uuid
-from copy import deepcopy
+from litellm._uuid import uuid
from typing import Optional
import litellm
from litellm._logging import verbose_logger
+from litellm.litellm_core_utils.core_helpers import safe_deep_copy
from .asyncify import run_async_function
@@ -41,7 +41,7 @@ async def async_completion_with_fallbacks(**kwargs):
most_recent_exception_str: Optional[str] = None
for fallback in fallbacks:
try:
- completion_kwargs = deepcopy(base_kwargs)
+ completion_kwargs = safe_deep_copy(base_kwargs)
# Handle dictionary fallback configurations
if isinstance(fallback, dict):
model = fallback.pop("model", original_model)
diff --git a/litellm/litellm_core_utils/get_litellm_params.py b/litellm/litellm_core_utils/get_litellm_params.py
index c354dea0241..c167c202e5d 100644
--- a/litellm/litellm_core_utils/get_litellm_params.py
+++ b/litellm/litellm_core_utils/get_litellm_params.py
@@ -62,6 +62,7 @@ def get_litellm_params(
use_litellm_proxy: Optional[bool] = None,
api_version: Optional[str] = None,
max_retries: Optional[int] = None,
+ litellm_request_debug: Optional[bool] = None,
**kwargs,
) -> dict:
litellm_params = {
@@ -118,5 +119,6 @@ def get_litellm_params(
"vertex_credentials": kwargs.get("vertex_credentials"),
"vertex_project": kwargs.get("vertex_project"),
"use_litellm_proxy": use_litellm_proxy,
+ "litellm_request_debug": litellm_request_debug,
}
return litellm_params
diff --git a/litellm/litellm_core_utils/get_llm_provider_logic.py b/litellm/litellm_core_utils/get_llm_provider_logic.py
index 4e0a2efb0c6..f209aed483c 100644
--- a/litellm/litellm_core_utils/get_llm_provider_logic.py
+++ b/litellm/litellm_core_utils/get_llm_provider_logic.py
@@ -196,6 +196,9 @@ def get_llm_provider( # noqa: PLR0915
elif endpoint == "https://api.cerebras.ai/v1":
custom_llm_provider = "cerebras"
dynamic_api_key = get_secret_str("CEREBRAS_API_KEY")
+ elif endpoint == "https://inference.baseten.co/v1":
+ custom_llm_provider = "baseten"
+ dynamic_api_key = get_secret_str("BASETEN_API_KEY")
elif endpoint == "https://api.sambanova.ai/v1":
custom_llm_provider = "sambanova"
dynamic_api_key = get_secret_str("SAMBANOVA_API_KEY")
@@ -246,6 +249,12 @@ def get_llm_provider( # noqa: PLR0915
elif endpoint == "https://api.hyperbolic.xyz/v1":
custom_llm_provider = "hyperbolic"
dynamic_api_key = get_secret_str("HYPERBOLIC_API_KEY")
+ elif endpoint == "https://ai-gateway.vercel.sh/v1":
+ custom_llm_provider = "vercel_ai_gateway"
+ dynamic_api_key = get_secret_str("VERCEL_AI_GATEWAY_API_KEY")
+ elif endpoint == "https://api.inference.wandb.ai/v1":
+ custom_llm_provider = "wandb"
+ dynamic_api_key = get_secret_str("WANDB_API_KEY")
if api_base is not None and not isinstance(api_base, str):
raise Exception(
@@ -314,6 +323,7 @@ def get_llm_provider( # noqa: PLR0915
or model in litellm.vertex_embedding_models
or model in litellm.vertex_vision_models
or model in litellm.vertex_ai_image_models
+ or model in litellm.vertex_ai_video_models
):
custom_llm_provider = "vertex_ai"
## ai21
@@ -351,11 +361,28 @@ def get_llm_provider( # noqa: PLR0915
custom_llm_provider = "openai"
elif model in litellm.empower_models:
custom_llm_provider = "empower"
+ elif model in litellm.gradient_ai_models:
+ custom_llm_provider = "gradient_ai"
elif model == "*":
custom_llm_provider = "openai"
# bytez models
elif model.startswith("bytez/"):
custom_llm_provider = "bytez"
+ elif model.startswith("lemonade/"):
+ custom_llm_provider = "lemonade"
+ elif model.startswith("heroku/"):
+ custom_llm_provider = "heroku"
+ # cometapi models
+ elif model.startswith("cometapi/"):
+ custom_llm_provider = "cometapi"
+ elif model.startswith("oci/"):
+ custom_llm_provider = "oci"
+ elif model.startswith("compactifai/"):
+ custom_llm_provider = "compactifai"
+ elif model.startswith("ovhcloud/"):
+ custom_llm_provider = "ovhcloud"
+ elif model.startswith("lemonade/"):
+ custom_llm_provider = "lemonade"
if not custom_llm_provider:
if litellm.suppress_debug_info is False:
print() # noqa
@@ -471,6 +498,13 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
api_base or get_secret("CEREBRAS_API_BASE") or "https://api.cerebras.ai/v1"
) # type: ignore
dynamic_api_key = api_key or get_secret_str("CEREBRAS_API_KEY")
+ elif custom_llm_provider == "baseten":
+ # Use BasetenConfig to determine the appropriate API base URL
+ if api_base is None:
+ api_base = litellm.BasetenConfig.get_api_base_for_model(model)
+ else:
+ api_base = api_base or get_secret_str("BASETEN_API_BASE") or "https://inference.baseten.co/v1"
+ dynamic_api_key = api_key or get_secret_str("BASETEN_API_KEY")
elif custom_llm_provider == "sambanova":
api_base = (
api_base
@@ -662,6 +696,13 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
or f"https://{get_secret('SNOWFLAKE_ACCOUNT_ID')}.snowflakecomputing.com/api/v2/cortex/inference:complete"
) # type: ignore
dynamic_api_key = api_key or get_secret_str("SNOWFLAKE_JWT")
+ elif custom_llm_provider == "gradient_ai":
+ (
+ api_base,
+ dynamic_api_key,
+ ) = litellm.GradientAIConfig()._get_openai_compatible_provider_info(
+ api_base, api_key
+ )
elif custom_llm_provider == "featherless_ai":
(
api_base,
@@ -676,6 +717,13 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
) = litellm.NscaleConfig()._get_openai_compatible_provider_info(
api_base=api_base, api_key=api_key
)
+ elif custom_llm_provider == "heroku":
+ (
+ api_base,
+ dynamic_api_key,
+ ) = litellm.HerokuChatConfig()._get_openai_compatible_provider_info(
+ api_base, api_key
+ )
elif custom_llm_provider == "dashscope":
(
api_base,
@@ -718,6 +766,34 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
) = litellm.HyperbolicChatConfig()._get_openai_compatible_provider_info(
api_base, api_key
)
+ elif custom_llm_provider == "vercel_ai_gateway":
+ (
+ api_base,
+ dynamic_api_key,
+ ) = litellm.VercelAIGatewayConfig()._get_openai_compatible_provider_info(
+ api_base, api_key
+ )
+ elif custom_llm_provider == "aiml":
+ (
+ api_base,
+ dynamic_api_key,
+ ) = litellm.AIMLChatConfig()._get_openai_compatible_provider_info(
+ api_base, api_key
+ )
+ elif custom_llm_provider == "wandb":
+ api_base = (
+ api_base
+ or get_secret("WANDB_API_BASE")
+ or "https://api.inference.wandb.ai/v1"
+ ) # type: ignore
+ dynamic_api_key = api_key or get_secret_str("WANDB_API_KEY")
+ elif custom_llm_provider == "lemonade":
+ (
+ api_base,
+ dynamic_api_key,
+ ) = litellm.LemonadeChatConfig()._get_openai_compatible_provider_info(
+ api_base, api_key
+ )
if api_base is not None and not isinstance(api_base, str):
raise Exception("api base needs to be a string. api_base={}".format(api_base))
diff --git a/litellm/litellm_core_utils/get_provider_specific_headers.py b/litellm/litellm_core_utils/get_provider_specific_headers.py
new file mode 100644
index 00000000000..cf9165cfda9
--- /dev/null
+++ b/litellm/litellm_core_utils/get_provider_specific_headers.py
@@ -0,0 +1,23 @@
+from typing import Dict, Optional
+
+from litellm.types.utils import ProviderSpecificHeader
+
+
+class ProviderSpecificHeaderUtils:
+ @staticmethod
+ def get_provider_specific_headers(
+ provider_specific_header: Optional[ProviderSpecificHeader],
+ custom_llm_provider: Optional[str],
+ ) -> Dict:
+ """
+ Get the provider specific headers for the given custom llm provider
+
+ Returns:
+ Optional[Dict]: The provider specific headers for the given custom llm provider
+ """
+ if (
+ provider_specific_header is not None
+ and provider_specific_header.get("custom_llm_provider") == custom_llm_provider
+ ):
+ return provider_specific_header.get("extra_headers", {})
+ return {}
\ No newline at end of file
diff --git a/litellm/litellm_core_utils/get_supported_openai_params.py b/litellm/litellm_core_utils/get_supported_openai_params.py
index f1901fa2ce9..06e650f938d 100644
--- a/litellm/litellm_core_utils/get_supported_openai_params.py
+++ b/litellm/litellm_core_utils/get_supported_openai_params.py
@@ -78,6 +78,8 @@ def get_supported_openai_params( # noqa: PLR0915
return litellm.nvidiaNimEmbeddingConfig.get_supported_openai_params()
elif custom_llm_provider == "cerebras":
return litellm.CerebrasConfig().get_supported_openai_params(model=model)
+ elif custom_llm_provider == "baseten":
+ return litellm.BasetenConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "xai":
return litellm.XAIChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "ai21_chat" or custom_llm_provider == "ai21":
@@ -92,9 +94,7 @@ def get_supported_openai_params( # noqa: PLR0915
return litellm.VLLMConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "deepseek":
return litellm.DeepSeekChatConfig().get_supported_openai_params(model=model)
- elif custom_llm_provider == "cohere":
- return litellm.CohereConfig().get_supported_openai_params(model=model)
- elif custom_llm_provider == "cohere_chat":
+ elif custom_llm_provider == "cohere_chat" or custom_llm_provider == "cohere":
return litellm.CohereChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "maritalk":
return litellm.MaritalkConfig().get_supported_openai_params(model=model)
@@ -121,10 +121,16 @@ def get_supported_openai_params( # noqa: PLR0915
return litellm.AzureOpenAIO1Config().get_supported_openai_params(
model=model
)
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ return litellm.AzureOpenAIGPT5Config().get_supported_openai_params(
+ model=model
+ )
else:
return litellm.AzureOpenAIConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "openrouter":
return litellm.OpenrouterConfig().get_supported_openai_params(model=model)
+ elif custom_llm_provider == "vercel_ai_gateway":
+ return litellm.VercelAIGatewayConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "mistral" or custom_llm_provider == "codestral":
# mistal and codestral api have the exact same params
if request_type == "chat_completion":
@@ -136,17 +142,25 @@ def get_supported_openai_params( # noqa: PLR0915
model=model
)
elif custom_llm_provider == "sambanova":
- return litellm.SambanovaConfig().get_supported_openai_params(model=model)
+ if request_type == "embeddings":
+ litellm.SambaNovaEmbeddingConfig().get_supported_openai_params(model=model)
+ else:
+ return litellm.SambanovaConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "nebius":
if request_type == "chat_completion":
return litellm.NebiusConfig().get_supported_openai_params(model=model)
+ elif custom_llm_provider == "wandb":
+ if request_type == "chat_completion":
+ return litellm.WandbConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "replicate":
return litellm.ReplicateConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "huggingface":
return litellm.HuggingFaceChatConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "jina_ai":
if request_type == "embeddings":
- return litellm.JinaAIEmbeddingConfig().get_supported_openai_params()
+ return litellm.JinaAIEmbeddingConfig().get_supported_openai_params(
+ model=model
+ )
elif custom_llm_provider == "together_ai":
return litellm.TogetherAIConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "databricks":
@@ -257,10 +271,9 @@ def get_supported_openai_params( # noqa: PLR0915
from litellm.llms.elevenlabs.audio_transcription.transformation import (
ElevenLabsAudioTranscriptionConfig,
)
- return (
- ElevenLabsAudioTranscriptionConfig().get_supported_openai_params(
- model=model
- )
+
+ return ElevenLabsAudioTranscriptionConfig().get_supported_openai_params(
+ model=model
)
elif custom_llm_provider in litellm._custom_providers:
if request_type == "chat_completion":
diff --git a/litellm/litellm_core_utils/health_check_helpers.py b/litellm/litellm_core_utils/health_check_helpers.py
index 7a2c005e8f6..2f412479937 100644
--- a/litellm/litellm_core_utils/health_check_helpers.py
+++ b/litellm/litellm_core_utils/health_check_helpers.py
@@ -1,12 +1,13 @@
-
"""
Helper functions for health check calls.
"""
+
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging
+
class HealthCheckHelpers:
@staticmethod
@@ -38,10 +39,9 @@ class HealthCheckHelpers:
model_params["model"] = cheapest_models[0]
model_params["litellm_logging_obj"] = litellm_logging_obj
model_params["fallbacks"] = fallback_models
- model_params["max_tokens"] = 1
+ model_params["max_tokens"] = 10 # gpt-5-nano throws errors for max_tokens=1
await acompletion(**model_params)
return {}
-
@staticmethod
def _update_model_params_with_health_check_tracking_information(
@@ -57,6 +57,7 @@ class HealthCheckHelpers:
"""
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.litellm_pre_call_utils import LiteLLMProxyRequestSetup
+
_metadata_variable_name = "litellm_metadata"
litellm_metadata = HealthCheckHelpers._get_metadata_for_health_check_call()
model_params[_metadata_variable_name] = litellm_metadata
@@ -66,13 +67,14 @@ class HealthCheckHelpers:
_metadata_variable_name=_metadata_variable_name,
)
return model_params
-
+
@staticmethod
def _get_metadata_for_health_check_call():
"""
Returns the metadata for the health check call.
"""
from litellm.constants import LITTELM_INTERNAL_HEALTH_SERVICE_ACCOUNT_NAME
+
return {
"tags": [LITTELM_INTERNAL_HEALTH_SERVICE_ACCOUNT_NAME],
- }
\ No newline at end of file
+ }
diff --git a/litellm/litellm_core_utils/litellm_logging.py b/litellm/litellm_core_utils/litellm_logging.py
index 8bbbd6d9d40..eafcab88557 100644
--- a/litellm/litellm_core_utils/litellm_logging.py
+++ b/litellm/litellm_core_utils/litellm_logging.py
@@ -10,7 +10,6 @@ import subprocess
import sys
import time
import traceback
-import uuid
from datetime import datetime as dt_object
from functools import lru_cache
from typing import (
@@ -38,6 +37,7 @@ from litellm import (
turn_off_message_logging,
)
from litellm._logging import _is_debugging_on, verbose_logger
+from litellm._uuid import uuid
from litellm.batches.batch_utils import _handle_completed_batch
from litellm.caching.caching import DualCache, InMemoryCache
from litellm.caching.caching_handler import LLMCachingHandler
@@ -81,12 +81,15 @@ from litellm.types.llms.openai import (
)
from litellm.types.mcp import MCPPostCallResponseObject
from litellm.types.rerank import RerankResponse
-from litellm.types.router import CustomPricingLiteLLMParams
from litellm.types.utils import (
+ CachingDetails,
CallTypes,
+ CostBreakdown,
CostResponseTypes,
+ CustomPricingLiteLLMParams,
DynamicPromptManagementParamLiteral,
EmbeddingResponse,
+ GuardrailStatus,
ImageResponse,
LiteLLMBatch,
LiteLLMLoggingBaseClass,
@@ -105,6 +108,7 @@ from litellm.types.utils import (
StandardLoggingPayload,
StandardLoggingPayloadErrorInformation,
StandardLoggingPayloadStatus,
+ StandardLoggingPayloadStatusFields,
StandardLoggingPromptManagementMetadata,
StandardLoggingVectorStoreRequest,
TextCompletionResponse,
@@ -120,6 +124,7 @@ from ..integrations.azure_storage.azure_storage import AzureBlobStorageLogger
from ..integrations.custom_prompt_management import CustomPromptManagement
from ..integrations.datadog.datadog import DataDogLogger
from ..integrations.datadog.datadog_llm_obs import DataDogLLMObsLogger
+from ..integrations.dotprompt import DotpromptManager
from ..integrations.dynamodb import DyanmoDBLogger
from ..integrations.galileo import GalileoObserve
from ..integrations.gcs_bucket.gcs_bucket import GCSBucketLogger
@@ -130,7 +135,6 @@ from ..integrations.humanloop import HumanloopLogger
from ..integrations.lago import LagoLogger
from ..integrations.langfuse.langfuse import LangFuseLogger
from ..integrations.langfuse.langfuse_handler import LangFuseHandler
-from ..integrations.langfuse.langfuse_otel import LangfuseOtelLogger
from ..integrations.langfuse.langfuse_prompt_management import LangfusePromptManagement
from ..integrations.langsmith import LangsmithLogger
from ..integrations.literal_ai import LiteralAILogger
@@ -138,6 +142,7 @@ from ..integrations.logfire_logger import LogfireLevel, LogfireLogger
from ..integrations.lunary import LunaryLogger
from ..integrations.openmeter import OpenMeterLogger
from ..integrations.opik.opik import OpikLogger
+from ..integrations.posthog import PostHogLogger
from ..integrations.prompt_layer import PromptLayerLogger
from ..integrations.s3 import S3Logger
from ..integrations.s3_v2 import S3Logger as S3V2Logger
@@ -167,11 +172,10 @@ try:
from litellm_enterprise.enterprise_callbacks.send_emails.smtp_email import (
SMTPEmailLogger,
)
+ from litellm_enterprise.integrations.prometheus import PrometheusLogger
from litellm_enterprise.litellm_core_utils.litellm_logging import (
StandardLoggingPayloadSetup as EnterpriseStandardLoggingPayloadSetup,
)
- from litellm_enterprise.integrations.prometheus import PrometheusLogger
-
EnterpriseStandardLoggingPayloadSetupVAR: Optional[
Type[EnterpriseStandardLoggingPayloadSetup]
@@ -194,7 +198,6 @@ _in_memory_loggers: List[Any] = []
sentry_sdk_instance = None
capture_exception = None
add_breadcrumb = None
-posthog = None
slack_app = None
alerts_channel = None
heliconeLogger = None
@@ -246,6 +249,7 @@ class Logging(LiteLLMLoggingBaseClass):
global supabaseClient, promptLayerLogger, weightsBiasesLogger, logfireLogger, capture_exception, add_breadcrumb, lunaryLogger, logfireLogger, prometheusLogger, slack_app
custom_pricing: bool = False
stream_options = None
+ litellm_request_debug: bool = False
def __init__(
self,
@@ -344,6 +348,12 @@ class Logging(LiteLLMLoggingBaseClass):
self.litellm_params = litellm_params
+ # Initialize cost breakdown field
+ self.cost_breakdown: Optional[CostBreakdown] = None
+
+ # Init Caching related details
+ self.caching_details: Optional[CachingDetails] = None
+
self.model_call_details: Dict[str, Any] = {
"litellm_trace_id": litellm_trace_id,
"litellm_call_id": litellm_call_id,
@@ -471,6 +481,7 @@ class Logging(LiteLLMLoggingBaseClass):
**self.litellm_params,
**scrub_sensitive_keys_in_metadata(litellm_params),
}
+ self.litellm_request_debug = litellm_params.get("litellm_request_debug", False)
self.logger_fn = litellm_params.get("logger_fn", None)
verbose_logger.debug(f"self.optional_params: {self.optional_params}")
@@ -504,6 +515,15 @@ class Logging(LiteLLMLoggingBaseClass):
if "custom_llm_provider" in self.model_call_details:
self.custom_llm_provider = self.model_call_details["custom_llm_provider"]
+ def update_messages(self, messages: List[AllMessageValues]):
+ """
+ Update the logged value of the messages in the model_call_details
+
+ Allows pre-call hooks to update the messages before the call is made
+ """
+ self.messages = messages
+ self.model_call_details["messages"] = messages
+
def should_run_prompt_management_hooks(
self,
non_default_params: Dict,
@@ -599,9 +619,7 @@ class Logging(LiteLLMLoggingBaseClass):
custom_logger = (
prompt_management_logger
or self.get_custom_logger_for_prompt_management(
- model=model,
- tools=tools,
- non_default_params=non_default_params
+ model=model, tools=tools, non_default_params=non_default_params
)
)
@@ -673,16 +691,15 @@ class Logging(LiteLLMLoggingBaseClass):
# Vector Store / Knowledge Base hooks
#########################################################
if litellm.vector_store_registry is not None:
-
- vector_store_custom_logger = _init_custom_logger_compatible_class(
- logging_integration="vector_store_pre_call_hook",
- internal_usage_cache=None,
- llm_router=None,
- )
- self.model_call_details["prompt_integration"] = (
- vector_store_custom_logger.__class__.__name__
- )
- return vector_store_custom_logger
+ vector_store_custom_logger = _init_custom_logger_compatible_class(
+ logging_integration="vector_store_pre_call_hook",
+ internal_usage_cache=None,
+ llm_router=None,
+ )
+ self.model_call_details["prompt_integration"] = (
+ vector_store_custom_logger.__class__.__name__
+ )
+ return vector_store_custom_logger
return None
@@ -805,7 +822,7 @@ class Logging(LiteLLMLoggingBaseClass):
str(e)
)
)
- if self.logger_fn and callable(self.logger_fn):
+ if getattr(self, "logger_fn", None) and callable(self.logger_fn):
try:
self.logger_fn(
self.model_call_details
@@ -901,13 +918,19 @@ class Logging(LiteLLMLoggingBaseClass):
Prints the RAW curl command sent from LiteLLM
"""
- if _is_debugging_on():
+ if _is_debugging_on() or self.litellm_request_debug:
if json_logs:
masked_headers = self._get_masked_headers(headers)
- verbose_logger.debug(
- "POST Request Sent from LiteLLM",
- extra={"api_base": {api_base}, **masked_headers},
- )
+ if self.litellm_request_debug:
+ verbose_logger.warning( # .warning ensures this shows up in all environments
+ "POST Request Sent from LiteLLM",
+ extra={"api_base": {api_base}, **masked_headers},
+ )
+ else:
+ verbose_logger.debug(
+ "POST Request Sent from LiteLLM",
+ extra={"api_base": {api_base}, **masked_headers},
+ )
else:
headers = additional_args.get("headers", {})
if headers is None:
@@ -920,7 +943,12 @@ class Logging(LiteLLMLoggingBaseClass):
additional_args=additional_args,
data=data,
)
- verbose_logger.debug(f"\033[92m{curl_command}\033[0m\n")
+ if self.litellm_request_debug:
+ verbose_logger.warning(
+ f"\033[92m{curl_command}\033[0m\n"
+ ) # .warning ensures this shows up in all environments
+ else:
+ verbose_logger.debug(f"\033[92m{curl_command}\033[0m\n")
def _get_request_body(self, data: dict) -> str:
return str(data)
@@ -945,7 +973,8 @@ class Logging(LiteLLMLoggingBaseClass):
if additional_args.get("request_str", None) is not None:
# print the sagemaker / bedrock client request
curl_command = "\nRequest Sent from LiteLLM:\n"
- curl_command += additional_args.get("request_str", None)
+ request_str = additional_args.get("request_str", "")
+ curl_command += request_str
elif api_base == "":
curl_command = str(self.model_call_details)
return curl_command
@@ -976,8 +1005,14 @@ class Logging(LiteLLMLoggingBaseClass):
self.model_call_details["additional_args"] = additional_args
self.model_call_details["log_event_type"] = "post_api_call"
+ if self.litellm_request_debug:
+ attr = "warning"
+ else:
+ attr = "debug"
+
if json_logs:
- verbose_logger.debug(
+ callattr = getattr(verbose_logger, attr)
+ callattr(
"RAW RESPONSE:\n{}\n\n".format(
self.model_call_details.get(
"original_response", self.model_call_details
@@ -985,14 +1020,15 @@ class Logging(LiteLLMLoggingBaseClass):
),
)
else:
- print_verbose(
+ callattr = getattr(verbose_logger, attr)
+ callattr(
"RAW RESPONSE:\n{}\n\n".format(
self.model_call_details.get(
"original_response", self.model_call_details
)
)
)
- if self.logger_fn and callable(self.logger_fn):
+ if getattr(self, "logger_fn", None) and callable(self.logger_fn):
try:
self.logger_fn(
self.model_call_details
@@ -1129,6 +1165,33 @@ class Logging(LiteLLMLoggingBaseClass):
- self.model_call_details.get("start_time", datetime.datetime.now())
).total_seconds() * 1000
+ def set_cost_breakdown(
+ self,
+ input_cost: float,
+ output_cost: float,
+ total_cost: float,
+ cost_for_built_in_tools_cost_usd_dollar: float,
+ ) -> None:
+ """
+ Helper method to store cost breakdown in the logging object.
+
+ Args:
+ input_cost: Cost of input/prompt tokens
+ output_cost: Cost of output/completion tokens
+ cost_for_built_in_tools_cost_usd_dollar: Cost of built-in tools
+ total_cost: Total cost of request
+ """
+
+ self.cost_breakdown = CostBreakdown(
+ input_cost=input_cost,
+ output_cost=output_cost,
+ total_cost=total_cost,
+ tool_usage_cost=cost_for_built_in_tools_cost_usd_dollar,
+ )
+ verbose_logger.debug(
+ f"Cost breakdown set - input: {input_cost}, output: {output_cost}, cost_for_built_in_tools_cost_usd_dollar: {cost_for_built_in_tools_cost_usd_dollar}, total: {total_cost}"
+ )
+
def _response_cost_calculator(
self,
result: Union[
@@ -1157,7 +1220,6 @@ class Logging(LiteLLMLoggingBaseClass):
used for consistent cost calculation across response headers + logging integrations.
"""
-
if isinstance(result, BaseModel) and hasattr(result, "_hidden_params"):
hidden_params = getattr(result, "_hidden_params", {})
if (
@@ -1203,6 +1265,11 @@ class Logging(LiteLLMLoggingBaseClass):
"standard_built_in_tools_params": self.standard_built_in_tools_params,
"router_model_id": router_model_id,
"litellm_logging_obj": self,
+ "service_tier": (
+ self.optional_params.get("service_tier")
+ if self.optional_params
+ else None
+ ),
}
except Exception as e: # error creating kwargs for cost calculation
debug_info = StandardLoggingModelCostFailureDebugInformation(
@@ -1314,9 +1381,9 @@ class Logging(LiteLLMLoggingBaseClass):
if (
EnterpriseCallbackControls is not None
and EnterpriseCallbackControls.is_callback_disabled_dynamically(
- callback=callback,
+ callback=callback,
litellm_params=litellm_params,
- standard_callback_dynamic_params = self.standard_callback_dynamic_params
+ standard_callback_dynamic_params=self.standard_callback_dynamic_params,
)
):
verbose_logger.debug(
@@ -1571,7 +1638,6 @@ class Logging(LiteLLMLoggingBaseClass):
)
if complete_streaming_response is not None:
-
self.success_handler(result=complete_streaming_response)
return
@@ -1708,8 +1774,15 @@ class Logging(LiteLLMLoggingBaseClass):
response_obj=result,
start_time=start_time,
end_time=end_time,
- litellm_call_id=litellm_params.get(
- "litellm_call_id", str(uuid.uuid4())
+ litellm_call_id=(
+ current_call_id
+ if (
+ current_call_id := litellm_params.get(
+ "litellm_call_id"
+ )
+ )
+ is not None
+ else str(uuid.uuid4())
),
print_verbose=print_verbose,
)
@@ -2108,10 +2181,12 @@ class Logging(LiteLLMLoggingBaseClass):
result.usage = batch_usage
elif not is_base64_unified_file_id: # only run for non-unified file ids
- response_cost, batch_usage, batch_models = (
- await _handle_completed_batch(
- batch=result, custom_llm_provider=self.custom_llm_provider
- )
+ (
+ response_cost,
+ batch_usage,
+ batch_models,
+ ) = await _handle_completed_batch(
+ batch=result, custom_llm_provider=self.custom_llm_provider
)
result._hidden_params["response_cost"] = response_cost
@@ -2265,15 +2340,20 @@ class Logging(LiteLLMLoggingBaseClass):
start_time=start_time,
end_time=end_time,
)
+
if isinstance(callback, CustomLogger): # custom logger class
+ model_call_details: Dict = self.model_call_details
+ ##################################
+ # call redaction hook for custom logger
+ model_call_details = callback.redact_standard_logging_payload_from_model_call_details(
+ model_call_details=model_call_details
+ )
+ ##################################
if self.stream is True:
- if (
- "async_complete_streaming_response"
- in self.model_call_details
- ):
+ if "async_complete_streaming_response" in model_call_details:
await callback.async_log_success_event(
- kwargs=self.model_call_details,
- response_obj=self.model_call_details[
+ kwargs=model_call_details,
+ response_obj=model_call_details[
"async_complete_streaming_response"
],
start_time=start_time,
@@ -2281,14 +2361,14 @@ class Logging(LiteLLMLoggingBaseClass):
)
else:
await callback.async_log_stream_event( # [TODO]: move this to being an async log stream event function
- kwargs=self.model_call_details,
+ kwargs=model_call_details,
response_obj=result,
start_time=start_time,
end_time=end_time,
)
else:
await callback.async_log_success_event(
- kwargs=self.model_call_details,
+ kwargs=model_call_details,
response_obj=result,
start_time=start_time,
end_time=end_time,
@@ -2763,6 +2843,7 @@ class Logging(LiteLLMLoggingBaseClass):
result: Any,
start_time: datetime.datetime,
end_time: datetime.datetime,
+ cache_hit: Optional[Any] = None,
) -> None:
"""
Handles calling success callbacks for Async calls.
@@ -2777,6 +2858,7 @@ class Logging(LiteLLMLoggingBaseClass):
result,
start_time,
end_time,
+ cache_hit,
)
def _should_run_sync_callbacks_for_async_calls(self) -> bool:
@@ -2905,14 +2987,17 @@ class Logging(LiteLLMLoggingBaseClass):
- For Non-streaming responses, we need to transform the response to a ModelResponse object.
- For streaming responses, anthropic_messages handler calls success_handler with a assembled ModelResponse.
"""
+ import httpx
+
if self.stream and isinstance(result, ModelResponse):
return result
elif isinstance(result, ModelResponse):
return result
- if "httpx_response" in self.model_call_details:
+ httpx_response = self.model_call_details.get("httpx_response", None)
+ if httpx_response and isinstance(httpx_response, httpx.Response):
result = litellm.AnthropicConfig().transform_response(
- raw_response=self.model_call_details.get("httpx_response", None),
+ raw_response=httpx_response,
model_response=litellm.ModelResponse(),
model=self.model,
messages=[],
@@ -3028,7 +3113,7 @@ def set_callbacks(callback_list, function_id=None): # noqa: PLR0915
"""
Globally sets the callback client
"""
- global sentry_sdk_instance, capture_exception, add_breadcrumb, posthog, slack_app, alerts_channel, traceloopLogger, athinaLogger, heliconeLogger, supabaseClient, lunaryLogger, promptLayerLogger, langFuseLogger, customLogger, weightsBiasesLogger, logfireLogger, dynamoLogger, s3Logger, dataDogLogger, prometheusLogger, greenscaleLogger, openMeterLogger, deepevalLogger
+ global sentry_sdk_instance, capture_exception, add_breadcrumb, slack_app, alerts_channel, traceloopLogger, athinaLogger, heliconeLogger, supabaseClient, lunaryLogger, promptLayerLogger, langFuseLogger, customLogger, weightsBiasesLogger, logfireLogger, dynamoLogger, s3Logger, dataDogLogger, prometheusLogger, greenscaleLogger, openMeterLogger, deepevalLogger
try:
for callback in callback_list:
@@ -3067,19 +3152,6 @@ def set_callbacks(callback_list, function_id=None): # noqa: PLR0915
)
capture_exception = sentry_sdk_instance.capture_exception
add_breadcrumb = sentry_sdk_instance.add_breadcrumb
- elif callback == "posthog":
- try:
- from posthog import Posthog
- except ImportError:
- print_verbose("Package 'posthog' is missing. Installing it...")
- subprocess.check_call(
- [sys.executable, "-m", "pip", "install", "posthog"]
- )
- from posthog import Posthog
- posthog = Posthog(
- project_api_key=os.environ.get("POSTHOG_API_KEY"),
- host=os.environ.get("POSTHOG_API_URL"),
- )
elif callback == "slack":
try:
from slack_bolt import App
@@ -3174,6 +3246,14 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
_openmeter_logger = OpenMeterLogger()
_in_memory_loggers.append(_openmeter_logger)
return _openmeter_logger # type: ignore
+ elif logging_integration == "posthog":
+ for callback in _in_memory_loggers:
+ if isinstance(callback, PostHogLogger):
+ return callback # type: ignore
+
+ _posthog_logger = PostHogLogger()
+ _in_memory_loggers.append(_posthog_logger)
+ return _posthog_logger # type: ignore
elif logging_integration == "braintrust":
from litellm.integrations.braintrust_logging import BraintrustLogger
@@ -3209,6 +3289,8 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
_in_memory_loggers.append(_literalai_logger)
return _literalai_logger # type: ignore
elif logging_integration == "prometheus":
+ if PrometheusLogger is None:
+ raise ValueError("PrometheusLogger is not initialized")
for callback in _in_memory_loggers:
if isinstance(callback, PrometheusLogger):
return callback # type: ignore
@@ -3347,7 +3429,15 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
galileo_logger = GalileoObserve()
_in_memory_loggers.append(galileo_logger)
return galileo_logger # type: ignore
+ elif logging_integration == "cloudzero":
+ from litellm.integrations.cloudzero.cloudzero import CloudZeroLogger
+ for callback in _in_memory_loggers:
+ if isinstance(callback, CloudZeroLogger):
+ return callback # type: ignore
+ cloudzero_logger = CloudZeroLogger()
+ _in_memory_loggers.append(cloudzero_logger)
+ return cloudzero_logger # type: ignore
elif logging_integration == "deepeval":
for callback in _in_memory_loggers:
if isinstance(callback, DeepEvalLogger):
@@ -3399,6 +3489,30 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
dynamic_rate_limiter_obj.update_variables(llm_router=llm_router)
_in_memory_loggers.append(dynamic_rate_limiter_obj)
return dynamic_rate_limiter_obj # type: ignore
+ elif logging_integration == "dynamic_rate_limiter_v3":
+ from litellm.proxy.hooks.dynamic_rate_limiter_v3 import (
+ _PROXY_DynamicRateLimitHandlerV3,
+ )
+
+ for callback in _in_memory_loggers:
+ if isinstance(callback, _PROXY_DynamicRateLimitHandlerV3):
+ return callback # type: ignore
+
+ if internal_usage_cache is None:
+ raise Exception(
+ "Internal Error: Cache cannot be empty - internal_usage_cache={}".format(
+ internal_usage_cache
+ )
+ )
+
+ dynamic_rate_limiter_obj_v3 = _PROXY_DynamicRateLimitHandlerV3(
+ internal_usage_cache=internal_usage_cache
+ )
+
+ if llm_router is not None and isinstance(llm_router, litellm.Router):
+ dynamic_rate_limiter_obj_v3.update_variables(llm_router=llm_router)
+ _in_memory_loggers.append(dynamic_rate_limiter_obj_v3)
+ return dynamic_rate_limiter_obj_v3 # type: ignore
elif logging_integration == "langtrace":
if "LANGTRACE_API_KEY" not in os.environ:
raise ValueError("LANGTRACE_API_KEY not found in environment variables")
@@ -3442,6 +3556,7 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
_in_memory_loggers.append(langfuse_logger)
return langfuse_logger # type: ignore
elif logging_integration == "langfuse_otel":
+ from litellm.integrations.langfuse.langfuse_otel import LangfuseOtelLogger
from litellm.integrations.opentelemetry import (
OpenTelemetry,
OpenTelemetryConfig,
@@ -3452,15 +3567,16 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
# The endpoint and headers are now set as environment variables by get_langfuse_otel_config()
otel_config = OpenTelemetryConfig(
exporter=langfuse_otel_config.protocol,
+ headers=langfuse_otel_config.otlp_auth_headers,
)
for callback in _in_memory_loggers:
if (
- isinstance(callback, OpenTelemetry)
+ isinstance(callback, LangfuseOtelLogger)
and callback.callback_name == "langfuse_otel"
):
return callback # type: ignore
- _otel_logger = OpenTelemetry(
+ _otel_logger = LangfuseOtelLogger(
config=otel_config, callback_name="langfuse_otel"
)
_in_memory_loggers.append(_otel_logger)
@@ -3483,7 +3599,7 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
from litellm.integrations.vector_store_integrations.vector_store_pre_call_hook import (
VectorStorePreCallHook,
)
-
+
for callback in _in_memory_loggers:
if isinstance(callback, VectorStorePreCallHook):
return callback
@@ -3526,11 +3642,59 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
humanloop_logger = HumanloopLogger()
_in_memory_loggers.append(humanloop_logger)
return humanloop_logger # type: ignore
+ elif logging_integration == "dotprompt":
+ for callback in _in_memory_loggers:
+ if isinstance(callback, DotpromptManager):
+ return callback
+
+ dotprompt_logger = DotpromptManager()
+ _in_memory_loggers.append(dotprompt_logger)
+ return dotprompt_logger # type: ignore
+ elif logging_integration == "bitbucket":
+ from litellm.integrations.bitbucket.bitbucket_prompt_manager import (
+ BitBucketPromptManager,
+ )
+
+ for callback in _in_memory_loggers:
+ if isinstance(callback, BitBucketPromptManager):
+ return callback
+
+ # Get global BitBucket config
+ bitbucket_config = getattr(litellm, "global_bitbucket_config", None)
+ if bitbucket_config is None:
+ raise ValueError(
+ "BitBucket configuration not found. Please set litellm.global_bitbucket_config first."
+ )
+
+ bitbucket_logger = BitBucketPromptManager(bitbucket_config=bitbucket_config)
+ _in_memory_loggers.append(bitbucket_logger)
+ return bitbucket_logger # type: ignore
+ elif logging_integration == "gitlab":
+ from litellm.integrations.gitlab.gitlab_prompt_manager import (
+ GitLabPromptManager,
+ )
+
+ for callback in _in_memory_loggers:
+ if isinstance(callback, GitLabPromptManager):
+ return callback
+
+ # Get global BitBucket config
+ gitlab_config = getattr(litellm, "global_gitlab_config", None)
+ if gitlab_config is None:
+ raise ValueError(
+ "Gitlab configuration not found. Please set litellm.global_gitlab_config first."
+ )
+
+ gitlab_logger = GitLabPromptManager(gitlab_config=gitlab_config)
+ _in_memory_loggers.append(gitlab_logger)
+ return gitlab_logger # type: ignore
+ return None
except Exception as e:
verbose_logger.exception(
f"[Non-Blocking Error] Error initializing custom logger: {e}"
)
return None
+ return None
def get_custom_logger_compatible_class( # noqa: PLR0915
@@ -3555,6 +3719,12 @@ def get_custom_logger_compatible_class( # noqa: PLR0915
for callback in _in_memory_loggers:
if isinstance(callback, GalileoObserve):
return callback
+ elif logging_integration == "cloudzero":
+ from litellm.integrations.cloudzero.cloudzero import CloudZeroLogger
+
+ for callback in _in_memory_loggers:
+ if isinstance(callback, CloudZeroLogger):
+ return callback
elif logging_integration == "deepeval":
for callback in _in_memory_loggers:
if isinstance(callback, DeepEvalLogger):
@@ -3571,7 +3741,7 @@ def get_custom_logger_compatible_class( # noqa: PLR0915
for callback in _in_memory_loggers:
if isinstance(callback, LiteralAILogger):
return callback
- elif logging_integration == "prometheus":
+ elif logging_integration == "prometheus" and PrometheusLogger is not None:
for callback in _in_memory_loggers:
if isinstance(callback, PrometheusLogger):
return callback
@@ -3644,6 +3814,14 @@ def get_custom_logger_compatible_class( # noqa: PLR0915
for callback in _in_memory_loggers:
if isinstance(callback, _PROXY_DynamicRateLimitHandler):
return callback # type: ignore
+ elif logging_integration == "dynamic_rate_limiter_v3":
+ from litellm.proxy.hooks.dynamic_rate_limiter_v3 import (
+ _PROXY_DynamicRateLimitHandlerV3,
+ )
+
+ for callback in _in_memory_loggers:
+ if isinstance(callback, _PROXY_DynamicRateLimitHandlerV3):
+ return callback # type: ignore
elif logging_integration == "langtrace":
from litellm.integrations.opentelemetry import OpenTelemetry
@@ -3674,7 +3852,7 @@ def get_custom_logger_compatible_class( # noqa: PLR0915
from litellm.integrations.vector_store_integrations.vector_store_pre_call_hook import (
VectorStorePreCallHook,
)
-
+
for callback in _in_memory_loggers:
if isinstance(callback, VectorStorePreCallHook):
return callback
@@ -3805,6 +3983,8 @@ class StandardLoggingPayloadSetup:
] = None,
usage_object: Optional[dict] = None,
proxy_server_request: Optional[dict] = None,
+ start_time: Optional[dt_object] = None,
+ response_id: Optional[str] = None,
) -> StandardLoggingMetadata:
"""
Clean and filter the metadata dictionary to include only the specified keys in StandardLoggingMetadata.
@@ -3840,22 +4020,27 @@ class StandardLoggingPayloadSetup:
clean_metadata = StandardLoggingMetadata(
user_api_key_hash=None,
user_api_key_alias=None,
+ user_api_key_spend=None,
+ user_api_key_max_budget=None,
+ user_api_key_budget_reset_at=None,
user_api_key_team_id=None,
user_api_key_org_id=None,
user_api_key_user_id=None,
user_api_key_team_alias=None,
user_api_key_user_email=None,
+ user_api_key_end_user_id=None,
+ user_api_key_request_route=None,
spend_logs_metadata=None,
requester_ip_address=None,
requester_metadata=None,
- user_api_key_end_user_id=None,
prompt_management_metadata=prompt_management_metadata,
applied_guardrails=applied_guardrails,
mcp_tool_call_metadata=mcp_tool_call_metadata,
vector_store_request_metadata=vector_store_request_metadata,
usage_object=usage_object,
requester_custom_headers=None,
- user_api_key_request_route=None,
+ cold_storage_object_key=None,
+ user_api_key_auth_metadata=None,
)
if isinstance(metadata, dict):
# Filter the metadata dictionary to include only the specified keys
@@ -3888,6 +4073,18 @@ class StandardLoggingPayloadSetup:
proxy_server_request=proxy_server_request,
)
+ # Generate cold storage object key if cold storage is configured
+ if start_time is not None and response_id is not None:
+ cold_storage_object_key = (
+ StandardLoggingPayloadSetup._generate_cold_storage_object_key(
+ start_time=start_time,
+ response_id=response_id,
+ team_alias=clean_metadata.get("user_api_key_team_alias"),
+ )
+ )
+ if cold_storage_object_key:
+ clean_metadata["cold_storage_object_key"] = cold_storage_object_key
+
return clean_metadata
@staticmethod
@@ -4046,6 +4243,65 @@ class StandardLoggingPayloadSetup:
return api_base.rstrip("/")
return api_base
+ @staticmethod
+ def _generate_cold_storage_object_key(
+ start_time: dt_object,
+ response_id: str,
+ team_alias: Optional[str] = None,
+ ) -> Optional[str]:
+ """
+ Generate cold storage object key in the same format as S3Logger.
+
+ Args:
+ start_time: The start time of the request
+ response_id: The response ID
+ team_alias: Optional team alias for team-based prefixing
+
+ Returns:
+ Optional[str]: The generated object key or None if cold storage not configured
+ """
+ # Generate object key in same format as S3Logger
+ from litellm.integrations.s3 import get_s3_object_key
+
+ # Only generate object key if cold storage is configured
+ configured_cold_storage_logger = litellm.configured_cold_storage_logger
+ if configured_cold_storage_logger is None:
+ return None
+
+ try:
+ # Generate file name in same format as litellm.utils.get_logging_id
+ s3_file_name = f"time-{start_time.strftime('%H-%M-%S-%f')}_{response_id}"
+
+ # Get the actual s3_path from the configured cold storage logger instance
+ s3_path = "" # default value
+
+ # Try to get the actual logger instance from the logger name
+ try:
+ custom_logger = litellm.logging_callback_manager.get_active_custom_logger_for_callback_name(
+ configured_cold_storage_logger
+ )
+ if (
+ custom_logger
+ and hasattr(custom_logger, "s3_path")
+ and getattr(custom_logger, "s3_path")
+ ):
+ s3_path = getattr(custom_logger, "s3_path")
+ except Exception:
+ # If any error occurs in getting the logger instance, use default empty s3_path
+ pass
+
+ s3_object_key = get_s3_object_key(
+ s3_path=s3_path, # Use actual s3_path from logger configuration
+ team_alias_prefix="", # Don't split by team alias for cold storage
+ start_time=start_time,
+ s3_file_name=s3_file_name,
+ )
+
+ return s3_object_key
+ except Exception:
+ # If any error occurs in generating the key, return None
+ return None
+
@staticmethod
def get_error_information(
original_exception: Optional[Exception],
@@ -4191,6 +4447,51 @@ class StandardLoggingPayloadSetup:
return request_tags
+
+def _get_status_fields(
+ status: StandardLoggingPayloadStatus,
+ guardrail_information: Optional[dict],
+ error_str: Optional[str]
+) -> "StandardLoggingPayloadStatusFields":
+ """
+ Determine status fields based on request status and guardrail information.
+
+ Args:
+ status: Overall request status ("success" or "failure")
+ guardrail_information: Guardrail information from metadata
+ error_str: Error string if any
+
+ Returns:
+ StandardLoggingPayloadStatusFields with llm_api_status and guardrail_status
+ """
+ # Mapping for legacy guardrail status values to new GuardrailStatus values
+ GUARDRAIL_STATUS_MAP: Dict[str, GuardrailStatus] = {
+ "success": "success",
+ "blocked": "guardrail_intervened", # legacy
+ "guardrail_intervened": "guardrail_intervened", # direct
+ "failure": "guardrail_failed_to_respond", # legacy
+ "guardrail_failed_to_respond": "guardrail_failed_to_respond", # direct
+ "not_run": "not_run"
+ }
+
+ # Set LLM API status
+ llm_api_status: StandardLoggingPayloadStatus = status
+
+
+ #########################################################
+ # Map - guardrail_information.guardrail_status to guardrail_status
+ #########################################################
+ guardrail_status: GuardrailStatus = "not_run"
+ if guardrail_information and isinstance(guardrail_information, dict):
+ raw_status = guardrail_information.get("guardrail_status", "not_run")
+ guardrail_status = GUARDRAIL_STATUS_MAP.get(raw_status, "not_run")
+
+ return StandardLoggingPayloadStatusFields(
+ llm_api_status=llm_api_status,
+ guardrail_status=guardrail_status
+ )
+
+
def get_standard_logging_object_payload(
kwargs: Optional[dict],
init_response_obj: Union[Any, BaseModel, dict],
@@ -4297,8 +4598,9 @@ def get_standard_logging_object_payload(
),
usage_object=usage.model_dump(),
proxy_server_request=proxy_server_request,
+ start_time=start_time,
+ response_id=id,
)
-
_request_body = proxy_server_request.get("body", {})
end_user_id = clean_metadata["user_api_key_end_user_id"] or _request_body.get(
"user", None
@@ -4354,6 +4656,11 @@ def get_standard_logging_object_payload(
cache_hit=cache_hit,
stream=stream,
status=status,
+ status_fields=_get_status_fields(
+ status=status,
+ guardrail_information=metadata.get("standard_logging_guardrail_information", None),
+ error_str=error_str
+ ),
custom_llm_provider=cast(Optional[str], kwargs.get("custom_llm_provider")),
saved_cache_cost=saved_cache_cost,
startTime=start_time_float,
@@ -4364,6 +4671,7 @@ def get_standard_logging_object_payload(
metadata=clean_metadata,
cache_key=clean_hidden_params["cache_key"],
response_cost=response_cost,
+ cost_breakdown=logging_obj.cost_breakdown,
total_tokens=usage.total_tokens,
prompt_tokens=usage.prompt_tokens,
completion_tokens=usage.completion_tokens,
@@ -4405,7 +4713,7 @@ def get_standard_logging_object_payload(
def emit_standard_logging_payload(payload: StandardLoggingPayload):
if os.getenv("LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD"):
- verbose_logger.info(json.dumps(payload, indent=4))
+ print(json.dumps(payload, indent=4)) # noqa
def get_standard_logging_metadata(
@@ -4428,6 +4736,9 @@ def get_standard_logging_metadata(
clean_metadata = StandardLoggingMetadata(
user_api_key_hash=None,
user_api_key_alias=None,
+ user_api_key_spend=None,
+ user_api_key_max_budget=None,
+ user_api_key_budget_reset_at=None,
user_api_key_team_id=None,
user_api_key_org_id=None,
user_api_key_user_id=None,
@@ -4444,16 +4755,14 @@ def get_standard_logging_metadata(
usage_object=None,
requester_custom_headers=None,
user_api_key_request_route=None,
+ cold_storage_object_key=None,
+ user_api_key_auth_metadata=None,
)
if isinstance(metadata, dict):
- # Filter the metadata dictionary to include only the specified keys
- clean_metadata = StandardLoggingMetadata(
- **{ # type: ignore
- key: metadata[key]
- for key in StandardLoggingMetadata.__annotations__.keys()
- if key in metadata
- }
- )
+ # Update the clean_metadata with values from input metadata that match StandardLoggingMetadata fields
+ for key in StandardLoggingMetadata.__annotations__.keys():
+ if key in metadata:
+ clean_metadata[key] = metadata[key] # type: ignore
if metadata.get("user_api_key") is not None:
if is_valid_sha256_hash(str(metadata.get("user_api_key"))):
diff --git a/litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py b/litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py
index 75bb699292e..b6113661777 100644
--- a/litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py
+++ b/litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py
@@ -47,7 +47,7 @@ class StandardBuiltInToolCostTracking:
- Code Interpreter (Azure)
"""
standard_built_in_tools_params = standard_built_in_tools_params or {}
-
+
# Handle web search
if StandardBuiltInToolCostTracking.response_object_includes_web_search_call(
response_object=response_object, usage=usage
@@ -58,7 +58,7 @@ class StandardBuiltInToolCostTracking:
usage=usage,
standard_built_in_tools_params=standard_built_in_tools_params,
)
-
+
# Handle file search
if StandardBuiltInToolCostTracking.response_object_includes_file_search_call(
response_object=response_object
@@ -68,7 +68,7 @@ class StandardBuiltInToolCostTracking:
custom_llm_provider=custom_llm_provider,
standard_built_in_tools_params=standard_built_in_tools_params,
)
-
+
# Handle Azure assistant features
return StandardBuiltInToolCostTracking._handle_azure_assistant_costs(
model=model,
@@ -85,14 +85,14 @@ class StandardBuiltInToolCostTracking:
) -> float:
"""Handle web search cost calculation."""
from litellm.llms import get_cost_for_web_search_request
-
+
model_info = StandardBuiltInToolCostTracking._safe_get_model_info(
model=model, custom_llm_provider=custom_llm_provider
)
-
+
if custom_llm_provider is None and model_info is not None:
custom_llm_provider = model_info["litellm_provider"]
-
+
if (
model_info is not None
and usage is not None
@@ -105,9 +105,11 @@ class StandardBuiltInToolCostTracking:
)
if result is not None:
return result
-
+
return StandardBuiltInToolCostTracking.get_cost_for_web_search(
- web_search_options=standard_built_in_tools_params.get("web_search_options", None),
+ web_search_options=standard_built_in_tools_params.get(
+ "web_search_options", None
+ ),
model_info=model_info,
)
@@ -121,12 +123,17 @@ class StandardBuiltInToolCostTracking:
model_info = StandardBuiltInToolCostTracking._safe_get_model_info(
model=model, custom_llm_provider=custom_llm_provider
)
- file_search_usage = standard_built_in_tools_params.get("file_search", {})
-
+ file_search_raw: Any = standard_built_in_tools_params.get("file_search", {})
+ file_search_usage: Optional[FileSearchTool] = (
+ FileSearchTool(**file_search_raw) if file_search_raw else None
+ )
+
# Convert model_info to dict and extract usage parameters
model_info_dict = dict(model_info) if model_info is not None else None
- storage_gb, days = StandardBuiltInToolCostTracking._extract_file_search_params(file_search_usage)
-
+ storage_gb, days = StandardBuiltInToolCostTracking._extract_file_search_params(
+ file_search_usage
+ )
+
return StandardBuiltInToolCostTracking.get_cost_for_file_search(
file_search=file_search_usage,
provider=custom_llm_provider,
@@ -144,11 +151,11 @@ class StandardBuiltInToolCostTracking:
"""Handle Azure assistant features cost calculation."""
if custom_llm_provider != "azure":
return 0.0
-
+
model_info = StandardBuiltInToolCostTracking._safe_get_model_info(
model=model, custom_llm_provider=custom_llm_provider
)
-
+
total_cost = 0.0
total_cost += StandardBuiltInToolCostTracking._get_vector_store_cost(
model_info, custom_llm_provider, standard_built_in_tools_params
@@ -159,31 +166,33 @@ class StandardBuiltInToolCostTracking:
total_cost += StandardBuiltInToolCostTracking._get_code_interpreter_cost(
model_info, custom_llm_provider, standard_built_in_tools_params
)
-
+
return total_cost
@staticmethod
- def _extract_file_search_params(file_search_usage: Any) -> Tuple[Optional[float], Optional[float]]:
+ def _extract_file_search_params(
+ file_search_usage: Any,
+ ) -> Tuple[Optional[float], Optional[float]]:
"""Extract and convert file search parameters safely."""
storage_gb = None
days = None
-
+
if isinstance(file_search_usage, dict):
storage_gb_val = file_search_usage.get("storage_gb")
days_val = file_search_usage.get("days")
-
+
if storage_gb_val is not None:
try:
storage_gb = float(storage_gb_val) # type: ignore
except (TypeError, ValueError):
storage_gb = None
-
+
if days_val is not None:
try:
days = float(days_val) # type: ignore
except (TypeError, ValueError):
days = None
-
+
return storage_gb, days
@staticmethod
@@ -193,13 +202,17 @@ class StandardBuiltInToolCostTracking:
standard_built_in_tools_params: StandardBuiltInToolsParams,
) -> float:
"""Calculate vector store cost."""
- vector_store_usage = standard_built_in_tools_params.get("vector_store_usage", None)
+ vector_store_usage = standard_built_in_tools_params.get(
+ "vector_store_usage", None
+ )
if not vector_store_usage:
return 0.0
-
+
model_info_dict = dict(model_info) if model_info is not None else None
- vector_store_dict = vector_store_usage if isinstance(vector_store_usage, dict) else {}
-
+ vector_store_dict = (
+ vector_store_usage if isinstance(vector_store_usage, dict) else {}
+ )
+
return StandardBuiltInToolCostTracking.get_cost_for_vector_store(
vector_store_usage=vector_store_dict,
provider=custom_llm_provider,
@@ -213,13 +226,17 @@ class StandardBuiltInToolCostTracking:
standard_built_in_tools_params: StandardBuiltInToolsParams,
) -> float:
"""Calculate computer use cost."""
- computer_use_usage = standard_built_in_tools_params.get("computer_use_usage", {})
+ computer_use_usage = standard_built_in_tools_params.get(
+ "computer_use_usage", {}
+ )
if not computer_use_usage:
return 0.0
-
+
model_info_dict = dict(model_info) if model_info is not None else None
- input_tokens, output_tokens = StandardBuiltInToolCostTracking._extract_token_counts(computer_use_usage)
-
+ input_tokens, output_tokens = (
+ StandardBuiltInToolCostTracking._extract_token_counts(computer_use_usage)
+ )
+
return StandardBuiltInToolCostTracking.get_cost_for_computer_use(
input_tokens=input_tokens,
output_tokens=output_tokens,
@@ -234,13 +251,17 @@ class StandardBuiltInToolCostTracking:
standard_built_in_tools_params: StandardBuiltInToolsParams,
) -> float:
"""Calculate code interpreter cost."""
- code_interpreter_sessions = standard_built_in_tools_params.get("code_interpreter_sessions", None)
+ code_interpreter_sessions = standard_built_in_tools_params.get(
+ "code_interpreter_sessions", None
+ )
if not code_interpreter_sessions:
return 0.0
-
+
model_info_dict = dict(model_info) if model_info is not None else None
- sessions = StandardBuiltInToolCostTracking._safe_convert_to_int(code_interpreter_sessions)
-
+ sessions = StandardBuiltInToolCostTracking._safe_convert_to_int(
+ code_interpreter_sessions
+ )
+
return StandardBuiltInToolCostTracking.get_cost_for_code_interpreter(
sessions=sessions,
provider=custom_llm_provider,
@@ -248,18 +269,24 @@ class StandardBuiltInToolCostTracking:
)
@staticmethod
- def _extract_token_counts(computer_use_usage: Any) -> Tuple[Optional[int], Optional[int]]:
+ def _extract_token_counts(
+ computer_use_usage: Any,
+ ) -> Tuple[Optional[int], Optional[int]]:
"""Extract and convert token counts safely."""
input_tokens = None
output_tokens = None
-
+
if isinstance(computer_use_usage, dict):
input_tokens_val = computer_use_usage.get("input_tokens")
output_tokens_val = computer_use_usage.get("output_tokens")
-
- input_tokens = StandardBuiltInToolCostTracking._safe_convert_to_int(input_tokens_val)
- output_tokens = StandardBuiltInToolCostTracking._safe_convert_to_int(output_tokens_val)
-
+
+ input_tokens = StandardBuiltInToolCostTracking._safe_convert_to_int(
+ input_tokens_val
+ )
+ output_tokens = StandardBuiltInToolCostTracking._safe_convert_to_int(
+ output_tokens_val
+ )
+
return input_tokens, output_tokens
@staticmethod
@@ -400,8 +427,11 @@ class StandardBuiltInToolCostTracking:
if model_info is None:
return 0.0
+ search_context_raw: Any = model_info.get("search_context_cost_per_query", {})
search_context_pricing: SearchContextCostPerQuery = (
- model_info.get("search_context_cost_per_query", {}) or {}
+ SearchContextCostPerQuery(**search_context_raw)
+ if search_context_raw
+ else SearchContextCostPerQuery()
)
if web_search_options.get("search_context_size", None) == "low":
return search_context_pricing.get("search_context_size_low", 0.0)
@@ -424,9 +454,12 @@ class StandardBuiltInToolCostTracking:
"""
if model_info is None:
return 0.0
+ search_context_raw: Any = model_info.get("search_context_cost_per_query", {}) or {}
search_context_pricing: SearchContextCostPerQuery = (
- model_info.get("search_context_cost_per_query", {}) or {}
- ) or {}
+ SearchContextCostPerQuery(**search_context_raw)
+ if search_context_raw
+ else SearchContextCostPerQuery()
+ )
return search_context_pricing.get("search_context_size_medium", 0.0)
@staticmethod
@@ -445,22 +478,27 @@ class StandardBuiltInToolCostTracking:
"""
if file_search is None:
return 0.0
-
+
# Check if model-specific pricing is available
- if model_info and "file_search_cost_per_gb_per_day" in model_info and provider == "azure":
+ if (
+ model_info
+ and "file_search_cost_per_gb_per_day" in model_info
+ and provider == "azure"
+ ):
if storage_gb and days:
return storage_gb * days * model_info["file_search_cost_per_gb_per_day"]
elif model_info and "file_search_cost_per_1k_calls" in model_info:
return model_info["file_search_cost_per_1k_calls"]
-
+
# Azure has storage-based pricing for file search
if provider == "azure":
from litellm.constants import AZURE_FILE_SEARCH_COST_PER_GB_PER_DAY
+
if storage_gb and days:
return storage_gb * days * AZURE_FILE_SEARCH_COST_PER_GB_PER_DAY
# Default to 0 if no storage info provided
return 0.0
-
+
# Default to OpenAI pricing (per-call based)
return OPENAI_FILE_SEARCH_COST_PER_1K_CALLS
@@ -472,24 +510,25 @@ class StandardBuiltInToolCostTracking:
) -> float:
"""
Calculate cost for vector store usage.
-
+
Azure charges based on storage size and duration.
"""
if vector_store_usage is None:
return 0.0
-
+
storage_gb = vector_store_usage.get("storage_gb", 0.0)
days = vector_store_usage.get("days", 0.0)
-
+
# Check if model-specific pricing is available
if model_info and "vector_store_cost_per_gb_per_day" in model_info:
return storage_gb * days * model_info["vector_store_cost_per_gb_per_day"]
-
+
# Azure has different pricing structure for vector store
if provider == "azure":
from litellm.constants import AZURE_VECTOR_STORE_COST_PER_GB_PER_DAY
+
return storage_gb * days * AZURE_VECTOR_STORE_COST_PER_GB_PER_DAY
-
+
# OpenAI doesn't charge separately for vector store (included in embeddings)
return 0.0
@@ -502,14 +541,18 @@ class StandardBuiltInToolCostTracking:
) -> float:
"""
Calculate cost for computer use feature.
-
+
Azure: $0.003 USD per 1K input tokens, $0.012 USD per 1K output tokens
"""
if provider == "azure" and (input_tokens or output_tokens):
# Check if model-specific pricing is available
if model_info:
- input_cost = model_info.get("computer_use_input_cost_per_1k_tokens", 0.0)
- output_cost = model_info.get("computer_use_output_cost_per_1k_tokens", 0.0)
+ input_cost = model_info.get(
+ "computer_use_input_cost_per_1k_tokens", 0.0
+ )
+ output_cost = model_info.get(
+ "computer_use_output_cost_per_1k_tokens", 0.0
+ )
if input_cost or output_cost:
total_cost = 0.0
if input_tokens:
@@ -517,19 +560,24 @@ class StandardBuiltInToolCostTracking:
if output_tokens:
total_cost += (output_tokens / 1000.0) * output_cost
return total_cost
-
+
# Azure default pricing
from litellm.constants import (
AZURE_COMPUTER_USE_INPUT_COST_PER_1K_TOKENS,
AZURE_COMPUTER_USE_OUTPUT_COST_PER_1K_TOKENS,
)
+
total_cost = 0.0
if input_tokens:
- total_cost += (input_tokens / 1000.0) * AZURE_COMPUTER_USE_INPUT_COST_PER_1K_TOKENS
+ total_cost += (
+ input_tokens / 1000.0
+ ) * AZURE_COMPUTER_USE_INPUT_COST_PER_1K_TOKENS
if output_tokens:
- total_cost += (output_tokens / 1000.0) * AZURE_COMPUTER_USE_OUTPUT_COST_PER_1K_TOKENS
+ total_cost += (
+ output_tokens / 1000.0
+ ) * AZURE_COMPUTER_USE_OUTPUT_COST_PER_1K_TOKENS
return total_cost
-
+
# OpenAI doesn't charge separately for computer use yet
return 0.0
@@ -541,21 +589,22 @@ class StandardBuiltInToolCostTracking:
) -> float:
"""
Calculate cost for code interpreter feature.
-
+
Azure: $0.03 USD per session
"""
if sessions is None or sessions == 0:
return 0.0
-
+
# Check if model-specific pricing is available
if model_info and "code_interpreter_cost_per_session" in model_info:
return sessions * model_info["code_interpreter_cost_per_session"]
-
+
# Azure pricing for code interpreter
if provider == "azure":
from litellm.constants import AZURE_CODE_INTERPRETER_COST_PER_SESSION
+
return sessions * AZURE_CODE_INTERPRETER_COST_PER_SESSION
-
+
# OpenAI doesn't charge separately for code interpreter yet
return 0.0
@@ -580,7 +629,9 @@ class StandardBuiltInToolCostTracking:
return WebSearchOptions(**kwargs.get("web_search_options", {}))
tools = StandardBuiltInToolCostTracking._get_tools_from_kwargs(
- kwargs, "web_search_preview"
+ kwargs=kwargs, tool_type="web_search_preview"
+ ) or StandardBuiltInToolCostTracking._get_tools_from_kwargs(
+ kwargs=kwargs, tool_type="web_search"
)
if tools:
# Look for web search tool in the tools array
@@ -612,6 +663,8 @@ class StandardBuiltInToolCostTracking:
def _is_web_search_tool_call(tool: Dict) -> bool:
if tool.get("type", None) == "web_search_preview":
return True
+ if tool.get("type", None) == "web_search":
+ return True
if "search_context_size" in tool:
return True
return False
diff --git a/litellm/litellm_core_utils/llm_cost_calc/utils.py b/litellm/litellm_core_utils/llm_cost_calc/utils.py
index 737e3f7f982..626a3f3625f 100644
--- a/litellm/litellm_core_utils/llm_cost_calc/utils.py
+++ b/litellm/litellm_core_utils/llm_cost_calc/utils.py
@@ -1,11 +1,19 @@
# What is this?
## Helper utilities for cost_per_token()
-from typing import Literal, Optional, Tuple, cast
+from typing import Any, Literal, Optional, Tuple, TypedDict, cast
import litellm
from litellm._logging import verbose_logger
-from litellm.types.utils import CallTypes, ModelInfo, PassthroughCallTypes, Usage
+from litellm.types.utils import (
+ CacheCreationTokenDetails,
+ CallTypes,
+ ImageResponse,
+ ModelInfo,
+ PassthroughCallTypes,
+ Usage,
+ ServiceTier,
+)
from litellm.utils import get_model_info
@@ -107,15 +115,62 @@ def _generic_cost_per_character(
return prompt_cost, completion_cost
-def _get_token_base_cost(model_info: ModelInfo, usage: Usage) -> Tuple[float, float]:
+def _get_service_tier_cost_key(base_key: str, service_tier: Optional[str]) -> str:
"""
- Return prompt cost for a given model and usage.
+ Get the appropriate cost key based on service tier.
+
+ Args:
+ base_key: The base cost key (e.g., "input_cost_per_token")
+ service_tier: The service tier ("flex", "priority", or None for standard)
+
+ Returns:
+ str: The cost key to use (e.g., "input_cost_per_token_flex" or "input_cost_per_token")
+ """
+ if service_tier is None:
+ return base_key
+
+ # Only use service tier specific keys for "flex" and "priority"
+ if service_tier.lower() in [ServiceTier.FLEX.value, ServiceTier.PRIORITY.value]:
+ return f"{base_key}_{service_tier.lower()}"
+
+ # For any other service tier, use standard pricing
+ return base_key
+
+
+def _get_token_base_cost(
+ model_info: ModelInfo, usage: Usage, service_tier: Optional[str] = None
+) -> Tuple[float, float, float, float, float]:
+ """
+ Return prompt cost, completion cost, and cache costs for a given model and usage.
If input_tokens > threshold and `input_cost_per_token_above_[x]k_tokens` or `input_cost_per_token_above_[x]_tokens` is set,
- then we use the corresponding threshold cost.
+ then we use the corresponding threshold cost for all token types.
+
+ Returns:
+ Tuple[float, float, float, float] - (prompt_cost, completion_cost, cache_creation_cost, cache_read_cost)
"""
- prompt_base_cost = cast(float, _get_cost_per_unit(model_info, "input_cost_per_token"))
- completion_base_cost = cast(float, _get_cost_per_unit(model_info, "output_cost_per_token"))
+ # Get service tier aware cost keys
+ input_cost_key = _get_service_tier_cost_key("input_cost_per_token", service_tier)
+ output_cost_key = _get_service_tier_cost_key("output_cost_per_token", service_tier)
+ cache_creation_cost_key = _get_service_tier_cost_key("cache_creation_input_token_cost", service_tier)
+ cache_read_cost_key = _get_service_tier_cost_key("cache_read_input_token_cost", service_tier)
+
+ prompt_base_cost = cast(
+ float, _get_cost_per_unit(model_info, input_cost_key)
+ )
+ completion_base_cost = cast(
+ float, _get_cost_per_unit(model_info, output_cost_key)
+ )
+ cache_creation_cost = cast(
+ float, _get_cost_per_unit(model_info, cache_creation_cost_key)
+ )
+ cache_creation_cost_above_1hr = cast(
+ float,
+ _get_cost_per_unit(model_info, "cache_creation_input_token_cost_above_1hr"),
+ )
+ cache_read_cost = cast(
+ float, _get_cost_per_unit(model_info, cache_read_cost_key)
+ )
## CHECK IF ABOVE THRESHOLD
threshold: Optional[float] = None
@@ -129,19 +184,57 @@ def _get_token_base_cost(model_info: ModelInfo, usage: Usage) -> Tuple[float, fl
)
if usage.prompt_tokens > threshold:
- prompt_base_cost = cast(float, _get_cost_per_unit(model_info, key, prompt_base_cost))
- completion_base_cost = cast(float, _get_cost_per_unit(
- model_info,
- f"output_cost_per_token_above_{threshold_str}_tokens",
- completion_base_cost,
- ))
+ prompt_base_cost = cast(
+ float, _get_cost_per_unit(model_info, key, prompt_base_cost)
+ )
+ completion_base_cost = cast(
+ float,
+ _get_cost_per_unit(
+ model_info,
+ f"output_cost_per_token_above_{threshold_str}_tokens",
+ completion_base_cost,
+ ),
+ )
+
+ # Apply tiered pricing to cache costs
+ cache_creation_tiered_key = (
+ f"cache_creation_input_token_cost_above_{threshold_str}_tokens"
+ )
+ cache_read_tiered_key = (
+ f"cache_read_input_token_cost_above_{threshold_str}_tokens"
+ )
+
+ if cache_creation_tiered_key in model_info:
+ cache_creation_cost = cast(
+ float,
+ _get_cost_per_unit(
+ model_info,
+ cache_creation_tiered_key,
+ cache_creation_cost,
+ ),
+ )
+
+ if cache_read_tiered_key in model_info:
+ cache_read_cost = cast(
+ float,
+ _get_cost_per_unit(
+ model_info, cache_read_tiered_key, cache_read_cost
+ ),
+ )
+
break
except (IndexError, ValueError):
continue
except Exception:
continue
- return prompt_base_cost, completion_base_cost
+ return (
+ prompt_base_cost,
+ completion_base_cost,
+ cache_creation_cost,
+ cache_creation_cost_above_1hr,
+ cache_read_cost,
+ )
def calculate_cost_component(
@@ -169,7 +262,9 @@ def calculate_cost_component(
return 0.0
-def _get_cost_per_unit(model_info: ModelInfo, cost_key: str, default_value: Optional[float] = 0.0) -> Optional[float]:
+def _get_cost_per_unit(
+ model_info: ModelInfo, cost_key: str, default_value: Optional[float] = 0.0
+) -> Optional[float]:
# Sometimes the cost per unit is a string (e.g.: If a value like "3e-7" was read from the config.yaml)
cost_per_unit = model_info.get(cost_key)
if isinstance(cost_per_unit, float):
@@ -183,12 +278,224 @@ def _get_cost_per_unit(model_info: ModelInfo, cost_key: str, default_value: Opti
verbose_logger.exception(
f"litellm.litellm_core_utils.llm_cost_calc.utils.py::calculate_cost_per_component(): Exception occured - {cost_per_unit}\nDefaulting to 0.0"
)
- return default_value
+ # If the service tier key doesn't exist or is None, try to fall back to the standard key
+ if cost_per_unit is None:
+ # Check if any service tier suffix exists in the cost key using ServiceTier enum
+ for service_tier in ServiceTier:
+ suffix = f"_{service_tier.value}"
+ if suffix in cost_key:
+ # Extract the base key by removing the matched suffix
+ base_key = cost_key.replace(suffix, '')
+ fallback_cost = model_info.get(base_key)
+ if isinstance(fallback_cost, float):
+ return fallback_cost
+ if isinstance(fallback_cost, int):
+ return float(fallback_cost)
+ if isinstance(fallback_cost, str):
+ try:
+ return float(fallback_cost)
+ except ValueError:
+ verbose_logger.exception(
+ f"litellm.litellm_core_utils.llm_cost_calc.utils.py::_get_cost_per_unit(): Exception occured - {fallback_cost}\nDefaulting to 0.0"
+ )
+ break # Only try the first matching suffix
+
+ return default_value
+
+
+def calculate_cache_writing_cost(
+ cache_creation_tokens: int,
+ cache_creation_token_details: Optional[CacheCreationTokenDetails],
+ cache_creation_cost_above_1hr: float,
+ cache_creation_cost: float,
+) -> float:
+ """
+ Adjust cost of cache creation tokens based on the cache creation token details.
+ """
+ total_cost: float = 0.0
+ if cache_creation_token_details is not None:
+ # get the number of 5m and 1h cache creation tokens
+ cache_creation_tokens_5m = (
+ cache_creation_token_details.ephemeral_5m_input_tokens
+ )
+ cache_creation_tokens_1h = (
+ cache_creation_token_details.ephemeral_1h_input_tokens
+ )
+ # add the number of 5m and 1h cache creation tokens to the cache creation tokens
+ total_cost += (
+ cache_creation_tokens_5m * cache_creation_cost
+ if cache_creation_tokens_5m is not None
+ else 0.0
+ )
+ total_cost += (
+ cache_creation_tokens_1h * cache_creation_cost_above_1hr
+ if cache_creation_tokens_1h is not None
+ else 0.0
+ )
+ else:
+ total_cost += cache_creation_tokens * cache_creation_cost
+ return total_cost
+
+
+class PromptTokensDetailsResult(TypedDict):
+ cache_hit_tokens: int
+ cache_creation_tokens: int
+ cache_creation_token_details: Optional[CacheCreationTokenDetails]
+ text_tokens: int
+ audio_tokens: int
+ character_count: int
+ image_count: int
+ video_length_seconds: int
+
+
+def _parse_prompt_tokens_details(usage: Usage) -> PromptTokensDetailsResult:
+ cache_hit_tokens = (
+ cast(Optional[int], getattr(usage.prompt_tokens_details, "cached_tokens", 0))
+ or 0
+ )
+ cache_creation_tokens = (
+ cast(
+ Optional[int],
+ getattr(usage.prompt_tokens_details, "cache_creation_tokens", 0),
+ )
+ or 0
+ )
+ cache_creation_token_details = (
+ cast(
+ Optional[CacheCreationTokenDetails],
+ getattr(usage.prompt_tokens_details, "cache_creation_token_details", None),
+ )
+ or None
+ )
+ text_tokens = (
+ cast(Optional[int], getattr(usage.prompt_tokens_details, "text_tokens", None))
+ or 0 # default to prompt tokens, if this field is not set
+ )
+ audio_tokens = (
+ cast(Optional[int], getattr(usage.prompt_tokens_details, "audio_tokens", 0))
+ or 0
+ )
+ character_count = (
+ cast(
+ Optional[int],
+ getattr(usage.prompt_tokens_details, "character_count", 0),
+ )
+ or 0
+ )
+ image_count = (
+ cast(Optional[int], getattr(usage.prompt_tokens_details, "image_count", 0)) or 0
+ )
+ video_length_seconds = (
+ cast(
+ Optional[int],
+ getattr(usage.prompt_tokens_details, "video_length_seconds", 0),
+ )
+ or 0
+ )
+
+ return PromptTokensDetailsResult(
+ cache_hit_tokens=cache_hit_tokens,
+ cache_creation_tokens=cache_creation_tokens,
+ cache_creation_token_details=cache_creation_token_details,
+ text_tokens=text_tokens,
+ audio_tokens=audio_tokens,
+ character_count=character_count,
+ image_count=image_count,
+ video_length_seconds=video_length_seconds,
+ )
+
+
+class CompletionTokensDetailsResult(TypedDict):
+ audio_tokens: int
+ text_tokens: int
+ reasoning_tokens: int
+
+
+def _parse_completion_tokens_details(usage: Usage) -> CompletionTokensDetailsResult:
+ audio_tokens = (
+ cast(
+ Optional[int],
+ getattr(usage.completion_tokens_details, "audio_tokens", 0),
+ )
+ or 0
+ )
+ text_tokens = (
+ cast(
+ Optional[int],
+ getattr(usage.completion_tokens_details, "text_tokens", None),
+ )
+ or 0 # default to completion tokens, if this field is not set
+ )
+ reasoning_tokens = (
+ cast(
+ Optional[int],
+ getattr(usage.completion_tokens_details, "reasoning_tokens", 0),
+ )
+ or 0
+ )
+
+ return CompletionTokensDetailsResult(
+ audio_tokens=audio_tokens,
+ text_tokens=text_tokens,
+ reasoning_tokens=reasoning_tokens,
+ )
+
+
+def _calculate_input_cost(
+ prompt_tokens_details: PromptTokensDetailsResult,
+ model_info: ModelInfo,
+ prompt_base_cost: float,
+ cache_read_cost: float,
+ cache_creation_cost: float,
+ cache_creation_cost_above_1hr: float,
+) -> float:
+ """
+ Calculates the input cost for a given model, prompt tokens, and completion tokens.
+ """
+ prompt_cost = float(prompt_tokens_details["text_tokens"]) * prompt_base_cost
+
+ ### CACHE READ COST - Now uses tiered pricing
+ prompt_cost += float(prompt_tokens_details["cache_hit_tokens"]) * cache_read_cost
+
+ ### AUDIO COST
+ prompt_cost += calculate_cost_component(
+ model_info, "input_cost_per_audio_token", prompt_tokens_details["audio_tokens"]
+ )
+
+ ### CACHE WRITING COST - Now uses tiered pricing
+ prompt_cost += calculate_cache_writing_cost(
+ cache_creation_tokens=prompt_tokens_details["cache_creation_tokens"],
+ cache_creation_token_details=prompt_tokens_details[
+ "cache_creation_token_details"
+ ],
+ cache_creation_cost_above_1hr=cache_creation_cost_above_1hr,
+ cache_creation_cost=cache_creation_cost,
+ )
+
+ ### CHARACTER COST
+
+ prompt_cost += calculate_cost_component(
+ model_info, "input_cost_per_character", prompt_tokens_details["character_count"]
+ )
+
+ ### IMAGE COUNT COST
+ prompt_cost += calculate_cost_component(
+ model_info, "input_cost_per_image", prompt_tokens_details["image_count"]
+ )
+
+ ### VIDEO LENGTH COST
+ prompt_cost += calculate_cost_component(
+ model_info,
+ "input_cost_per_video_per_second",
+ prompt_tokens_details["video_length_seconds"],
+ )
+
+ return prompt_cost
def generic_cost_per_token(
- model: str, usage: Usage, custom_llm_provider: str
+ model: str, usage: Usage, custom_llm_provider: str, service_tier: Optional[str] = None
) -> Tuple[float, float]:
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@@ -210,89 +517,45 @@ def generic_cost_per_token(
### Cost of processing (non-cache hit + cache hit) + Cost of cache-writing (cache writing)
prompt_cost = 0.0
### PROCESSING COST
- text_tokens = usage.prompt_tokens
- cache_hit_tokens = 0
- audio_tokens = 0
- character_count = 0
- image_count = 0
- video_length_seconds = 0
+ prompt_tokens_details = PromptTokensDetailsResult(
+ cache_hit_tokens=0,
+ cache_creation_tokens=0,
+ cache_creation_token_details=None,
+ text_tokens=usage.prompt_tokens,
+ audio_tokens=0,
+ character_count=0,
+ image_count=0,
+ video_length_seconds=0,
+ )
if usage.prompt_tokens_details:
- cache_hit_tokens = (
- cast(
- Optional[int], getattr(usage.prompt_tokens_details, "cached_tokens", 0)
- )
- or 0
- )
- text_tokens = (
- cast(
- Optional[int], getattr(usage.prompt_tokens_details, "text_tokens", None)
- )
- or 0 # default to prompt tokens, if this field is not set
- )
- audio_tokens = (
- cast(Optional[int], getattr(usage.prompt_tokens_details, "audio_tokens", 0))
- or 0
- )
- character_count = (
- cast(
- Optional[int],
- getattr(usage.prompt_tokens_details, "character_count", 0),
- )
- or 0
- )
- image_count = (
- cast(Optional[int], getattr(usage.prompt_tokens_details, "image_count", 0))
- or 0
- )
- video_length_seconds = (
- cast(
- Optional[int],
- getattr(usage.prompt_tokens_details, "video_length_seconds", 0),
- )
- or 0
- )
+ prompt_tokens_details = _parse_prompt_tokens_details(usage)
## EDGE CASE - text tokens not set inside PromptTokensDetails
- if text_tokens == 0:
- text_tokens = usage.prompt_tokens - cache_hit_tokens - audio_tokens
- prompt_base_cost, completion_base_cost = _get_token_base_cost(
- model_info=model_info, usage=usage
- )
+ if prompt_tokens_details["text_tokens"] == 0:
+ text_tokens = (
+ usage.prompt_tokens
+ - prompt_tokens_details["cache_hit_tokens"]
+ - prompt_tokens_details["audio_tokens"]
+ - prompt_tokens_details["cache_creation_tokens"]
+ )
+ prompt_tokens_details["text_tokens"] = text_tokens
- prompt_cost = float(text_tokens) * prompt_base_cost
+ (
+ prompt_base_cost,
+ completion_base_cost,
+ cache_creation_cost,
+ cache_creation_cost_above_1hr,
+ cache_read_cost,
+ ) = _get_token_base_cost(model_info=model_info, usage=usage, service_tier=service_tier)
- ### CACHE READ COST
- prompt_cost += calculate_cost_component(
- model_info, "cache_read_input_token_cost", cache_hit_tokens
- )
-
- ### AUDIO COST
- prompt_cost += calculate_cost_component(
- model_info, "input_cost_per_audio_token", audio_tokens
- )
-
- ### CACHE WRITING COST
- prompt_cost += calculate_cost_component(
- model_info,
- "cache_creation_input_token_cost",
- usage._cache_creation_input_tokens,
- )
-
- ### CHARACTER COST
-
- prompt_cost += calculate_cost_component(
- model_info, "input_cost_per_character", character_count
- )
-
- ### IMAGE COUNT COST
- prompt_cost += calculate_cost_component(
- model_info, "input_cost_per_image", image_count
- )
-
- ### VIDEO LENGTH COST
- prompt_cost += calculate_cost_component(
- model_info, "input_cost_per_video_per_second", video_length_seconds
+ prompt_cost = _calculate_input_cost(
+ prompt_tokens_details=prompt_tokens_details,
+ model_info=model_info,
+ prompt_base_cost=prompt_base_cost,
+ cache_read_cost=cache_read_cost,
+ cache_creation_cost=cache_creation_cost,
+ cache_creation_cost_above_1hr=cache_creation_cost_above_1hr,
)
## CALCULATE OUTPUT COST
@@ -301,27 +564,10 @@ def generic_cost_per_token(
reasoning_tokens = 0
is_text_tokens_total = False
if usage.completion_tokens_details is not None:
- audio_tokens = (
- cast(
- Optional[int],
- getattr(usage.completion_tokens_details, "audio_tokens", 0),
- )
- or 0
- )
- text_tokens = (
- cast(
- Optional[int],
- getattr(usage.completion_tokens_details, "text_tokens", None),
- )
- or 0 # default to completion tokens, if this field is not set
- )
- reasoning_tokens = (
- cast(
- Optional[int],
- getattr(usage.completion_tokens_details, "reasoning_tokens", 0),
- )
- or 0
- )
+ completion_tokens_details = _parse_completion_tokens_details(usage)
+ audio_tokens = completion_tokens_details["audio_tokens"]
+ text_tokens = completion_tokens_details["text_tokens"]
+ reasoning_tokens = completion_tokens_details["reasoning_tokens"]
if text_tokens == 0:
text_tokens = usage.completion_tokens
@@ -330,8 +576,12 @@ def generic_cost_per_token(
## TEXT COST
completion_cost = float(text_tokens) * completion_base_cost
- _output_cost_per_audio_token = _get_cost_per_unit(model_info, "output_cost_per_audio_token", None)
- _output_cost_per_reasoning_token = _get_cost_per_unit(model_info, "output_cost_per_reasoning_token", None)
+ _output_cost_per_audio_token = _get_cost_per_unit(
+ model_info, "output_cost_per_audio_token", None
+ )
+ _output_cost_per_reasoning_token = _get_cost_per_unit(
+ model_info, "output_cost_per_reasoning_token", None
+ )
## AUDIO COST
if not is_text_tokens_total and audio_tokens is not None and audio_tokens > 0:
@@ -377,3 +627,93 @@ class CostCalculatorUtils:
]:
return True
return False
+
+ @staticmethod
+ def route_image_generation_cost_calculator(
+ model: str,
+ completion_response: Any,
+ custom_llm_provider: Optional[str] = None,
+ quality: Optional[str] = None,
+ n: Optional[int] = None,
+ size: Optional[str] = None,
+ optional_params: Optional[dict] = None,
+ ) -> float:
+ """
+ Route the image generation cost calculator based on the custom_llm_provider
+ """
+ from litellm.cost_calculator import default_image_cost_calculator
+ from litellm.llms.azure_ai.image_generation.cost_calculator import (
+ cost_calculator as azure_ai_image_cost_calculator,
+ )
+ from litellm.llms.bedrock.image.cost_calculator import (
+ cost_calculator as bedrock_image_cost_calculator,
+ )
+ from litellm.llms.gemini.image_generation.cost_calculator import (
+ cost_calculator as gemini_image_cost_calculator,
+ )
+ from litellm.llms.recraft.cost_calculator import (
+ cost_calculator as recraft_image_cost_calculator,
+ )
+ from litellm.llms.vertex_ai.image_generation.cost_calculator import (
+ cost_calculator as vertex_ai_image_cost_calculator,
+ )
+
+ if custom_llm_provider == litellm.LlmProviders.VERTEX_AI.value:
+ if isinstance(completion_response, ImageResponse):
+ return vertex_ai_image_cost_calculator(
+ model=model,
+ image_response=completion_response,
+ )
+ elif custom_llm_provider == litellm.LlmProviders.BEDROCK.value:
+ if isinstance(completion_response, ImageResponse):
+ return bedrock_image_cost_calculator(
+ model=model,
+ size=size,
+ image_response=completion_response,
+ optional_params=optional_params,
+ )
+ raise TypeError(
+ "completion_response must be of type ImageResponse for bedrock image cost calculation"
+ )
+ elif custom_llm_provider == litellm.LlmProviders.RECRAFT.value:
+ from litellm.llms.recraft.cost_calculator import (
+ cost_calculator as recraft_image_cost_calculator,
+ )
+
+ return recraft_image_cost_calculator(
+ model=model,
+ image_response=completion_response,
+ )
+ elif custom_llm_provider == litellm.LlmProviders.AIML.value:
+ from litellm.llms.aiml.image_generation.cost_calculator import (
+ cost_calculator as aiml_image_cost_calculator,
+ )
+
+ return aiml_image_cost_calculator(
+ model=model,
+ image_response=completion_response,
+ )
+ elif custom_llm_provider == litellm.LlmProviders.GEMINI.value:
+ from litellm.llms.gemini.image_generation.cost_calculator import (
+ cost_calculator as gemini_image_cost_calculator,
+ )
+
+ return gemini_image_cost_calculator(
+ model=model,
+ image_response=completion_response,
+ )
+ elif custom_llm_provider == litellm.LlmProviders.AZURE_AI.value:
+ return azure_ai_image_cost_calculator(
+ model=model,
+ image_response=completion_response,
+ )
+ else:
+ return default_image_cost_calculator(
+ model=model,
+ quality=quality,
+ custom_llm_provider=custom_llm_provider,
+ n=n,
+ size=size,
+ optional_params=optional_params,
+ )
+ return 0.0
diff --git a/litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py b/litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py
index 54adef9c958..6ed9d5725e9 100644
--- a/litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py
+++ b/litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py
@@ -1,14 +1,16 @@
import asyncio
import json
-import re
import time
import traceback
-import uuid
from typing import Dict, Iterable, List, Literal, Optional, Tuple, Union
import litellm
from litellm._logging import verbose_logger
+from litellm._uuid import uuid
from litellm.constants import RESPONSE_FORMAT_TOOL_NAME
+from litellm.litellm_core_utils.prompt_templates.common_utils import (
+ _extract_reasoning_content,
+)
from litellm.types.llms.databricks import DatabricksTool
from litellm.types.llms.openai import (
ChatCompletionThinkingBlock,
@@ -29,6 +31,7 @@ from litellm.types.utils import Logprobs as TextCompletionLogprobs
from litellm.types.utils import (
Message,
ModelResponse,
+ ModelResponseStream,
RerankResponse,
StreamingChoices,
TextChoices,
@@ -43,13 +46,13 @@ from .get_headers import get_response_headers
def _safe_convert_created_field(created_value) -> int:
"""
Safely convert a 'created' field value to an integer.
-
- Some providers (like SambaNova) return the 'created' field as a float
+
+ Some providers (like SambaNova) return the 'created' field as a float
(Unix timestamp with fractional seconds), but LiteLLM expects an integer.
-
+
Args:
created_value: The value from response_object["created"]
-
+
Returns:
int: Unix timestamp as integer
"""
@@ -106,12 +109,12 @@ async def convert_to_streaming_response_async(response_object: Optional[dict] =
if response_object is None:
raise Exception("Error in response object format")
- model_response_object = ModelResponse(stream=True)
+ model_response_object = ModelResponseStream()
if model_response_object is None:
raise Exception("Error in response creating model response object")
- choice_list = []
+ choice_list: List[StreamingChoices] = []
for idx, choice in enumerate(response_object["choices"]):
if (
@@ -161,7 +164,9 @@ async def convert_to_streaming_response_async(response_object: Optional[dict] =
model_response_object.id = response_object["id"]
if "created" in response_object:
- model_response_object.created = _safe_convert_created_field(response_object["created"])
+ model_response_object.created = _safe_convert_created_field(
+ response_object["created"]
+ )
if "system_fingerprint" in response_object:
model_response_object.system_fingerprint = response_object["system_fingerprint"]
@@ -178,8 +183,8 @@ def convert_to_streaming_response(response_object: Optional[dict] = None):
if response_object is None:
raise Exception("Error in response object format")
- model_response_object = ModelResponse(stream=True)
- choice_list = []
+ model_response_object = ModelResponseStream()
+ choice_list: List[StreamingChoices] = []
for idx, choice in enumerate(response_object["choices"]):
delta = Delta(**choice["message"])
finish_reason = choice.get("finish_reason", None)
@@ -209,7 +214,9 @@ def convert_to_streaming_response(response_object: Optional[dict] = None):
model_response_object.id = response_object["id"]
if "created" in response_object:
- model_response_object.created = _safe_convert_created_field(response_object["created"])
+ model_response_object.created = _safe_convert_created_field(
+ response_object["created"]
+ )
if "system_fingerprint" in response_object:
model_response_object.system_fingerprint = response_object["system_fingerprint"]
@@ -270,49 +277,6 @@ def _handle_invalid_parallel_tool_calls(
return tool_calls
-def _parse_content_for_reasoning(
- message_text: Optional[str],
-) -> Tuple[Optional[str], Optional[str]]:
- """
- Parse the content for reasoning
-
- Returns:
- - reasoning_content: The content of the reasoning
- - content: The content of the message
- """
- if not message_text:
- return None, message_text
-
- reasoning_match = re.match(
- r"<(?:think|thinking)>(.*?)(?:think|thinking)>(.*)", message_text, re.DOTALL
- )
-
- if reasoning_match:
- return reasoning_match.group(1), reasoning_match.group(2)
-
- return None, message_text
-
-
-def _extract_reasoning_content(message: dict) -> Tuple[Optional[str], Optional[str]]:
- """
- Extract reasoning content and main content from a message.
-
- Args:
- message (dict): The message dictionary that may contain reasoning_content
-
- Returns:
- tuple[Optional[str], Optional[str]]: A tuple of (reasoning_content, content)
- """
- message_content = message.get("content")
- if "reasoning_content" in message:
- return message["reasoning_content"], message["content"]
- elif "reasoning" in message:
- return message["reasoning"], message["content"]
- elif isinstance(message_content, str):
- return _parse_content_for_reasoning(message_content)
- return None, message_content
-
-
class LiteLLMResponseObjectHandler:
@staticmethod
def convert_to_image_response(
@@ -497,7 +461,7 @@ def convert_to_model_response_object( # noqa: PLR0915
if stream is True:
# for returning cached responses, we need to yield a generator
return convert_to_streaming_response(response_object=response_object)
- choice_list = []
+ choice_list: List[Choices] = []
assert response_object["choices"] is not None and isinstance(
response_object["choices"], Iterable
@@ -557,9 +521,9 @@ def convert_to_model_response_object( # noqa: PLR0915
provider_specific_fields["thinking_blocks"] = thinking_blocks
if reasoning_content:
- provider_specific_fields[
- "reasoning_content"
- ] = reasoning_content
+ provider_specific_fields["reasoning_content"] = (
+ reasoning_content
+ )
message = Message(
content=content,
@@ -571,6 +535,7 @@ def convert_to_model_response_object( # noqa: PLR0915
reasoning_content=reasoning_content,
thinking_blocks=thinking_blocks,
annotations=choice["message"].get("annotations", None),
+ images=choice["message"].get("images", None),
)
finish_reason = choice.get("finish_reason", None)
if finish_reason is None:
@@ -600,13 +565,15 @@ def convert_to_model_response_object( # noqa: PLR0915
provider_specific_fields=provider_specific_fields,
)
choice_list.append(choice)
- model_response_object.choices = choice_list
+ model_response_object.choices = choice_list # type: ignore
if "usage" in response_object and response_object["usage"] is not None:
usage_object = litellm.Usage(**response_object["usage"])
setattr(model_response_object, "usage", usage_object)
if "created" in response_object:
- model_response_object.created = _safe_convert_created_field(response_object["created"])
+ model_response_object.created = _safe_convert_created_field(
+ response_object["created"]
+ )
if "id" in response_object:
model_response_object.id = response_object["id"] or str(uuid.uuid4())
diff --git a/litellm/litellm_core_utils/llm_response_utils/response_metadata.py b/litellm/litellm_core_utils/llm_response_utils/response_metadata.py
index b1085c684fc..c5ef7237628 100644
--- a/litellm/litellm_core_utils/llm_response_utils/response_metadata.py
+++ b/litellm/litellm_core_utils/llm_response_utils/response_metadata.py
@@ -85,15 +85,37 @@ class ResponseMetadata:
# Set total response time if supported
if self.supports_response_time:
self.result._response_ms = total_response_time_ms
+
+ #########################################################
+ # 1. Add _response_ms total duration
+ #########################################################
+ self._update_hidden_params(
+ {
+ "_response_ms": total_response_time_ms,
+ }
+ )
- # Calculate LiteLLM overhead
+ #########################################################
+ # 2. Add LiteLLM overhead duration
+ #########################################################
llm_api_duration_ms = logging_obj.model_call_details.get("llm_api_duration_ms")
if llm_api_duration_ms is not None:
overhead_ms = round(total_response_time_ms - llm_api_duration_ms, 4)
self._update_hidden_params(
{
"litellm_overhead_time_ms": overhead_ms,
- "_response_ms": total_response_time_ms,
+ }
+ )
+
+ #########################################################
+ # 3. Add duration for reading from cache
+ # In this case overhead from litellm is the difference between the cache read duration and the total response time
+ #########################################################
+ if logging_obj.caching_details is not None and logging_obj.caching_details.get("cache_hit") is True and (cache_duration_ms := logging_obj.caching_details.get("cache_duration_ms")) is not None:
+ overhead_ms = total_response_time_ms - cache_duration_ms
+ self._update_hidden_params(
+ {
+ "litellm_overhead_time_ms": overhead_ms,
}
)
@@ -113,6 +135,10 @@ def update_response_metadata(
) -> None:
"""
Updates response metadata including hidden params and timing metrics
+ Updates response metadata, adds the following:
+ - response._hidden_params
+ - response._hidden_params["litellm_overhead_time_ms"]
+ - response.response_time_ms
"""
if result is None:
return
diff --git a/litellm/litellm_core_utils/logging_callback_manager.py b/litellm/litellm_core_utils/logging_callback_manager.py
index 44cb146f91a..9ec346c20a1 100644
--- a/litellm/litellm_core_utils/logging_callback_manager.py
+++ b/litellm/litellm_core_utils/logging_callback_manager.py
@@ -1,4 +1,4 @@
-from typing import Callable, List, Set, Type, Union
+from typing import TYPE_CHECKING, Callable, List, Optional, Set, Type, Union
import litellm
from litellm._logging import verbose_logger
@@ -6,6 +6,11 @@ from litellm.integrations.additional_logging_utils import AdditionalLoggingUtils
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.utils import CallbacksByType
+if TYPE_CHECKING:
+ from litellm import _custom_logger_compatible_callbacks_literal
+else:
+ _custom_logger_compatible_callbacks_literal = str
+
class LoggingCallbackManager:
"""
@@ -343,3 +348,26 @@ class LoggingCallbackManager:
elif callable(callback):
return getattr(callback, "__name__", str(callback))
return str(callback)
+
+
+ def get_active_custom_logger_for_callback_name(
+ self,
+ callback_name: _custom_logger_compatible_callbacks_literal,
+ ) -> Optional[CustomLogger]:
+ """
+ Get the active custom logger for a given callback name
+ """
+ from litellm.litellm_core_utils.custom_logger_registry import (
+ CustomLoggerRegistry,
+ )
+
+ # get the custom logger class type
+ custom_logger_class_type = CustomLoggerRegistry.get_class_type_for_custom_logger_name(callback_name)
+
+ # get the active custom logger
+ custom_logger = self.get_custom_loggers_for_type(custom_logger_class_type)
+
+ if len(custom_logger) == 0:
+ raise ValueError(f"No active custom logger found for callback name: {callback_name}")
+
+ return custom_logger[0]
diff --git a/litellm/litellm_core_utils/logging_utils.py b/litellm/litellm_core_utils/logging_utils.py
index c7512ea146b..bf43519afc6 100644
--- a/litellm/litellm_core_utils/logging_utils.py
+++ b/litellm/litellm_core_utils/logging_utils.py
@@ -1,5 +1,6 @@
import asyncio
import functools
+import time
from datetime import datetime
from typing import TYPE_CHECKING, Any, List, Optional, Union
@@ -11,15 +12,19 @@ from litellm.types.utils import (
)
if TYPE_CHECKING:
+ from opentelemetry.trace import Span as _Span
+
from litellm import ModelResponse as _ModelResponse
from litellm.litellm_core_utils.litellm_logging import (
Logging as LiteLLMLoggingObject,
)
LiteLLMModelResponse = _ModelResponse
+ Span = Union[_Span, Any]
else:
LiteLLMModelResponse = Any
LiteLLMLoggingObject = Any
+ Span = Any
import litellm
@@ -28,9 +33,52 @@ import litellm
Helper utils used for logging callbacks
"""
+# Global service logger instance to avoid recreating it
+_service_logger = None
+
+
+def _get_service_logger():
+ """Get or create the global ServiceLogging instance"""
+ global _service_logger
+ if _service_logger is None:
+ from litellm._service_logger import ServiceLogging
+
+ _service_logger = ServiceLogging()
+ return _service_logger
+
+
+def _get_parent_otel_span_from_logging_obj(
+ logging_obj: Optional[LiteLLMLoggingObject] = None,
+) -> Optional[Span]:
+ """
+ Extract the parent OTEL span from the logging object using existing helper.
+
+ Args:
+ logging_obj: The LiteLLM logging object containing model call details
+
+ Returns:
+ The parent OTEL span if found, None otherwise
+ """
+ try:
+ if logging_obj is None or not hasattr(logging_obj, "model_call_details"):
+ return None
+
+ # Reuse existing function by passing model_call_details as kwargs
+ from litellm.litellm_core_utils.core_helpers import (
+ _get_parent_otel_span_from_kwargs,
+ )
+
+ return _get_parent_otel_span_from_kwargs(logging_obj.model_call_details)
+
+ except Exception as e:
+ verbose_logger.exception(
+ f"Error in _get_parent_otel_span_from_logging_obj: {str(e)}"
+ )
+ return None
+
def convert_litellm_response_object_to_str(
- response_obj: Union[Any, LiteLLMModelResponse]
+ response_obj: Union[Any, LiteLLMModelResponse],
) -> Optional[str]:
"""
Get the string of the response object from LiteLLM
@@ -125,37 +173,102 @@ def track_llm_api_timing():
"""
Decorator to track LLM API call timing for both sync and async functions.
The logging_obj is expected to be passed as an argument to the decorated function.
+ Logs timing using ServiceLogging similar to Redis cache.
"""
def decorator(func):
@functools.wraps(func)
async def async_wrapper(*args, **kwargs):
start_time = datetime.now()
+ start_time_float = time.time()
+ logging_obj = kwargs.get("logging_obj", None)
+
+ # Extract parent OTEL span from logging object
+ parent_otel_span = _get_parent_otel_span_from_logging_obj(logging_obj)
+
try:
result = await func(*args, **kwargs)
return result
finally:
end_time = datetime.now()
+ end_time_float = time.time()
+ duration = end_time_float - start_time_float
+
+ # Set duration in model call details
_set_duration_in_model_call_details(
- logging_obj=kwargs.get("logging_obj", None),
+ logging_obj=logging_obj,
start_time=start_time,
end_time=end_time,
)
+ # Log timing using ServiceLogging (like Redis cache)
+ try:
+ from litellm.types.services import ServiceTypes
+
+ service_logger = _get_service_logger()
+
+ # Get function name for call_type
+ call_type = f"{func.__name__} <- track_llm_api_timing"
+
+ # Create async task for service logging (similar to Redis cache pattern)
+ asyncio.create_task(
+ service_logger.async_service_success_hook(
+ service=ServiceTypes.LITELLM,
+ duration=duration,
+ call_type=call_type,
+ start_time=start_time_float,
+ end_time=end_time_float,
+ parent_otel_span=parent_otel_span,
+ )
+ )
+ except Exception as e:
+ verbose_logger.debug(f"Error in service logging: {str(e)}")
+
@functools.wraps(func)
def sync_wrapper(*args, **kwargs):
start_time = datetime.now()
+ start_time_float = time.time()
+ logging_obj = kwargs.get("logging_obj", None)
+
+ # Extract parent OTEL span from logging object
+ parent_otel_span = _get_parent_otel_span_from_logging_obj(logging_obj)
+
try:
result = func(*args, **kwargs)
return result
finally:
end_time = datetime.now()
+ end_time_float = time.time()
+ duration = end_time_float - start_time_float
+
+ # Set duration in model call details
_set_duration_in_model_call_details(
- logging_obj=kwargs.get("logging_obj", None),
+ logging_obj=logging_obj,
start_time=start_time,
end_time=end_time,
)
+ # Log timing using ServiceLogging (like Redis cache)
+ try:
+ from litellm.types.services import ServiceTypes
+
+ service_logger = _get_service_logger()
+
+ # Get function name for call_type
+ call_type = f"{func.__name__} <- track_llm_api_timing"
+
+ # Use sync service logging for sync functions
+ service_logger.service_success_hook(
+ service=ServiceTypes.LITELLM,
+ duration=duration,
+ call_type=call_type,
+ start_time=start_time_float,
+ end_time=end_time_float,
+ parent_otel_span=parent_otel_span,
+ )
+ except Exception as e:
+ verbose_logger.debug(f"Error in service logging: {str(e)}")
+
# Check if the function is async or sync
if asyncio.iscoroutinefunction(func):
return async_wrapper
diff --git a/litellm/litellm_core_utils/logging_worker.py b/litellm/litellm_core_utils/logging_worker.py
new file mode 100644
index 00000000000..3c475f133a8
--- /dev/null
+++ b/litellm/litellm_core_utils/logging_worker.py
@@ -0,0 +1,159 @@
+import asyncio
+import contextlib
+import contextvars
+from typing import Coroutine, Optional
+
+from typing_extensions import TypedDict
+
+from litellm._logging import verbose_logger
+
+
+class LoggingTask(TypedDict):
+ """
+ A logging task with its associated context to ensure logging is executed in
+ the original task's context.
+ """
+
+ coroutine: Coroutine
+ context: contextvars.Context
+
+
+class LoggingWorker:
+ """
+ A simple, async logging worker that processes log coroutines in the background.
+ Designed to be best-effort with bounded queues to prevent backpressure.
+
+ This leads to a +200 RPS performance improvement when using LiteLLM Python SDK or Proxy Server.
+ - Use this to queue coroutine tasks that are not critical to the main flow of the application. e.g Success/Error callbacks, logging, etc.
+ """
+
+ LOGGING_WORKER_MAX_QUEUE_SIZE = 50_000
+ LOGGING_WORKER_MAX_TIME_PER_COROUTINE = 20.0
+
+ MAX_ITERATIONS_TO_CLEAR_QUEUE = 200
+ MAX_TIME_TO_CLEAR_QUEUE = 5.0
+
+ def __init__(
+ self,
+ timeout: float = LOGGING_WORKER_MAX_TIME_PER_COROUTINE,
+ max_queue_size: int = LOGGING_WORKER_MAX_QUEUE_SIZE,
+ ):
+ self.timeout = timeout
+ self.max_queue_size = max_queue_size
+ self._queue: Optional[asyncio.Queue[LoggingTask]] = None
+ self._worker_task: Optional[asyncio.Task] = None
+
+ def _ensure_queue(self) -> None:
+ """Initialize the queue if it doesn't exist."""
+ if self._queue is None:
+ self._queue = asyncio.Queue(maxsize=self.max_queue_size)
+
+ def start(self) -> None:
+ """Start the logging worker. Idempotent - safe to call multiple times."""
+ self._ensure_queue()
+ if self._worker_task is None or self._worker_task.done():
+ self._worker_task = asyncio.create_task(self._worker_loop())
+
+ async def _worker_loop(self) -> None:
+ """Main worker loop that processes log coroutines sequentially."""
+ try:
+ if self._queue is None:
+ return
+
+ while True:
+ # Process one coroutine at a time to keep event loop load predictable
+ task = await self._queue.get()
+ try:
+ # Run the coroutine in its original context
+ await asyncio.wait_for(
+ task["context"].run(asyncio.create_task, task["coroutine"]),
+ timeout=self.timeout,
+ )
+ except Exception as e:
+ verbose_logger.exception(f"LoggingWorker error: {e}")
+ pass
+ finally:
+ self._queue.task_done()
+
+ except asyncio.CancelledError:
+ verbose_logger.debug("LoggingWorker cancelled during shutdown")
+ # Attempt to clear remaining items to prevent "never awaited" warnings
+ await self.clear_queue()
+
+ def enqueue(self, coroutine: Coroutine) -> None:
+ """
+ Add a coroutine to the logging queue.
+ Hot path: never blocks, drops logs if queue is full.
+ """
+ if self._queue is None:
+ return
+
+ try:
+ # Capture the current context when enqueueing
+ task = LoggingTask(coroutine=coroutine, context=contextvars.copy_context())
+ self._queue.put_nowait(task)
+ except asyncio.QueueFull as e:
+ verbose_logger.exception(f"LoggingWorker queue is full: {e}")
+ # Drop logs on overload to protect request throughput
+ pass
+
+ def ensure_initialized_and_enqueue(self, async_coroutine: Coroutine):
+ """
+ Ensure the logging worker is initialized and enqueue the coroutine.
+ """
+ self.start()
+ self.enqueue(async_coroutine)
+
+ async def stop(self) -> None:
+ """Stop the logging worker and clean up resources."""
+ if self._worker_task:
+ self._worker_task.cancel()
+ with contextlib.suppress(Exception):
+ await self._worker_task
+ self._worker_task = None
+
+ async def flush(self) -> None:
+ """Flush the logging queue."""
+ if self._queue is None:
+ return
+ while not self._queue.empty():
+ await self._queue.join()
+
+ async def clear_queue(self):
+ """
+ Clear the queue with a maximum time limit.
+ """
+ if self._queue is None:
+ return
+
+ start_time = asyncio.get_event_loop().time()
+
+ for _ in range(self.MAX_ITERATIONS_TO_CLEAR_QUEUE):
+ # Check if we've exceeded the maximum time
+ if (
+ asyncio.get_event_loop().time() - start_time
+ >= self.MAX_TIME_TO_CLEAR_QUEUE
+ ):
+ verbose_logger.warning(
+ f"clear_queue exceeded max_time of {self.MAX_TIME_TO_CLEAR_QUEUE}s, stopping early"
+ )
+ break
+
+ try:
+ task = self._queue.get_nowait()
+ # Await the coroutine to properly execute and avoid "never awaited" warnings
+ try:
+ await asyncio.wait_for(
+ task["context"].run(asyncio.create_task, task["coroutine"]),
+ timeout=self.timeout,
+ )
+ except Exception:
+ # Suppress errors during cleanup
+ pass
+ self._queue.task_done() # If you're using join() elsewhere
+ except asyncio.QueueEmpty:
+ break
+
+
+# Global instance for backward compatibility
+GLOBAL_LOGGING_WORKER = LoggingWorker()
diff --git a/litellm/litellm_core_utils/model_response_utils.py b/litellm/litellm_core_utils/model_response_utils.py
new file mode 100644
index 00000000000..974d12aef6f
--- /dev/null
+++ b/litellm/litellm_core_utils/model_response_utils.py
@@ -0,0 +1,215 @@
+"""
+Utility functions for ModelResponse and ModelResponseStream objects.
+"""
+
+from typing import Any
+
+from litellm.types.utils import Delta, ModelResponseBase, ModelResponseStream
+
+
+def is_model_response_stream_empty(model_response: ModelResponseStream) -> bool:
+ """
+ Check if a ModelResponseStream is empty based on:
+ - If finish_reason is set -> it's non empty
+ - If any field in choices is set (e.g. content, tool calls, etc.) it's non empty
+ - If usage exists -> it's non empty
+
+ This function is robust and ignores fields that are always set (from ModelResponseBase)
+ and checks for any meaningful content in other fields.
+
+ Args:
+ model_response: The ModelResponseStream to check
+
+ Returns:
+ bool: True if the stream is empty, False if it contains meaningful data
+ """
+ # Fields that are always set in ModelResponseBase and should be ignored
+ # These are structural fields that don't indicate content
+ BASE_FIELDS = ModelResponseBase.model_fields.keys()
+
+ # Check if usage exists - this indicates meaningful data
+ if getattr(model_response, "usage", None) is not None:
+ return False
+
+ # Check provider_specific_fields at the top level
+ if (
+ hasattr(model_response, "provider_specific_fields")
+ and model_response.provider_specific_fields is not None
+ and model_response.provider_specific_fields != {}
+ ):
+ return False
+
+ # Check model_extra for dynamically added fields (this is where Pydantic stores them)
+ if hasattr(model_response, "model_extra") and model_response.model_extra:
+ for extra_field_name, extra_field_value in model_response.model_extra.items():
+ if _has_meaningful_content(extra_field_value):
+ return False
+
+ # Check for any non-base fields that are set
+ for model_response_field in model_response.model_fields.keys():
+ # Skip base fields that are always set
+ if model_response_field in BASE_FIELDS:
+ continue
+
+ # Skip choices - we'll handle them separately with deep inspection
+ if model_response_field == "choices":
+ continue
+
+ # Check if any other field has meaningful content
+ model_response_value = getattr(model_response, model_response_field, None)
+ if _has_meaningful_content(model_response_value):
+ return False
+
+ # Deep check of choices for any meaningful content
+ if hasattr(model_response, "choices") and model_response.choices:
+ for choice in model_response.choices:
+ if _is_choice_non_empty(choice):
+ return False
+
+ # If we get here, the stream is empty
+ return True
+
+
+def _has_meaningful_content(value: Any) -> bool:
+ """
+ Check if a value contains meaningful content.
+
+ Args:
+ value: The value to check
+
+ Returns:
+ bool: True if the value has meaningful content, False otherwise
+ """
+ if value is None:
+ return False
+
+ if isinstance(value, str):
+ # Don't strip whitespace - preserve all content including newlines, spaces, etc.
+ # Even pure whitespace characters like '\n' or ' ' are meaningful content
+ return len(value) > 0
+
+ if isinstance(value, (list, dict)):
+ return len(value) > 0
+
+ if isinstance(value, bool):
+ return True # Any boolean value is meaningful
+
+ if isinstance(value, (int, float)):
+ return True # Any numeric value is meaningful
+
+ # For other types (objects), consider them meaningful if they exist
+ return True
+
+
+def _is_choice_non_empty(choice: Any) -> bool:
+ """
+ Deep check if a choice contains any meaningful content.
+
+ Args:
+ choice: The choice object to check
+
+ Returns:
+ bool: True if the choice has meaningful content, False otherwise
+ """
+ # Check finish_reason
+ if hasattr(choice, "finish_reason") and choice.finish_reason is not None:
+
+ return True
+
+ # Check logprobs
+ if hasattr(choice, "logprobs") and choice.logprobs is not None:
+
+ return True
+
+ # Check enhancements (if present)
+ if hasattr(choice, "enhancements") and choice.enhancements is not None:
+
+ return True
+
+ # Deep check delta object
+ if hasattr(choice, "delta") and choice.delta is not None:
+ if _is_delta_non_empty(choice.delta):
+
+ return True
+
+ # Check model_extra for dynamically added fields on the choice
+ if hasattr(choice, "model_extra") and choice.model_extra:
+ for extra_field_name, extra_field_value in choice.model_extra.items():
+ # Skip certain structural fields that are just default/None placeholders
+ if extra_field_name == "index" and extra_field_value == 0:
+
+ continue
+ if (
+ extra_field_name in {"finish_reason", "logprobs"}
+ and extra_field_value is None
+ ):
+
+ continue
+ if extra_field_name == "delta":
+
+ continue
+ if _has_meaningful_content(extra_field_value):
+
+ return True
+
+ # Check for any other non-standard fields on the choice
+ for attr_name in dir(choice):
+ # Skip private attributes, methods, and known empty fields
+ if (
+ attr_name.startswith("_")
+ or callable(getattr(choice, attr_name))
+ or attr_name.startswith("model_")
+ or attr_name
+ in {
+ "finish_reason",
+ "index",
+ "delta",
+ "logprobs",
+ "enhancements",
+ }
+ ):
+
+ continue
+
+ attr_value = getattr(choice, attr_name, None)
+ if _has_meaningful_content(attr_value):
+
+ return True
+
+ return False
+
+
+def _is_delta_non_empty(delta: Delta) -> bool:
+ """
+ Deep check if a delta object contains any meaningful content.
+
+ Args:
+ delta: The delta object to check
+
+ Returns:
+ bool: True if the delta has meaningful content, False otherwise
+ """
+ # Check model_extra for dynamically added fields (this is where Pydantic stores them)
+ if hasattr(delta, "model_extra") and delta.model_extra:
+ for extra_field_name, extra_field_value in delta.model_extra.items():
+ # Even structural fields are meaningful if they have actual content
+ if _has_meaningful_content(extra_field_value):
+
+ return True
+
+ # Check all regular attributes of the delta object
+ for attr_name in dir(delta):
+ # Skip private attributes, methods, and Pydantic-specific fields
+ if (
+ attr_name.startswith("_")
+ or callable(getattr(delta, attr_name))
+ or attr_name.startswith("model_")
+ ):
+ continue
+
+ attr_value = getattr(delta, attr_name, None)
+ if _has_meaningful_content(attr_value):
+
+ return True
+
+ return False
diff --git a/litellm/litellm_core_utils/prompt_templates/common_utils.py b/litellm/litellm_core_utils/prompt_templates/common_utils.py
index 258601ff5a0..19d5932ff28 100644
--- a/litellm/litellm_core_utils/prompt_templates/common_utils.py
+++ b/litellm/litellm_core_utils/prompt_templates/common_utils.py
@@ -14,10 +14,12 @@ from typing import (
Literal,
Mapping,
Optional,
+ Tuple,
Union,
cast,
)
+from litellm.router_utils.batch_utils import InMemoryFile
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionAssistantMessage,
@@ -453,6 +455,10 @@ def extract_file_data(file_data: FileTypes) -> ExtractedFileData:
filename, file_content, content_type = file_data
elif len(file_data) == 4:
filename, file_content, content_type, file_headers = file_data
+ elif isinstance(file_data, InMemoryFile):
+ filename = file_data.name
+ file_content = file_data
+ content_type = file_data.content_type
else:
file_content = file_data
# Convert content to bytes
@@ -519,25 +525,25 @@ def unpack_defs(schema: dict, defs: dict) -> None:
}
# Use iterative approach with queue to avoid recursion
- # Each item in queue is (node, parent_container, key/index, active_defs, seen_ids)
+ # Each item in queue is (node, parent_container, key/index, active_defs, ref_chain)
queue: deque[
tuple[Any, Union[dict, list, None], Union[str, int, None], dict, set]
] = deque([(schema, None, None, root_defs, set())])
while queue:
- node, parent, key, active_defs, seen = queue.popleft()
-
- # Avoid infinite loops on self-referential schemas
- if id(node) in seen:
- continue
- seen = seen.copy() # Create new set for this branch
- seen.add(id(node))
+ node, parent, key, active_defs, ref_chain = queue.popleft()
# ----------------------------- dict -----------------------------
if isinstance(node, dict):
# --- Case 1: this node *is* a reference ---
if "$ref" in node:
ref_name = node["$ref"].split("/")[-1]
+
+ # Check for circular reference in the resolution chain
+ if ref_name in ref_chain:
+ # Circular reference detected - leave as-is to prevent infinite recursion
+ continue
+
target_schema = active_defs.get(ref_name)
# Unknown reference – leave untouched
if target_schema is None:
@@ -563,8 +569,12 @@ def unpack_defs(schema: dict, defs: dict) -> None:
schema.update(resolved)
resolved = schema
+ # Add to ref chain to track circular references
+ new_ref_chain = ref_chain.copy()
+ new_ref_chain.add(ref_name)
+
# Add resolved node to queue for further processing
- queue.append((resolved, parent, key, child_defs, seen))
+ queue.append((resolved, parent, key, child_defs, new_ref_chain))
continue
# --- Case 2: regular dict – process its values ---
@@ -577,13 +587,13 @@ def unpack_defs(schema: dict, defs: dict) -> None:
# Add all dict values to queue
for k, v in node.items():
- queue.append((v, node, k, current_defs, seen))
+ queue.append((v, node, k, current_defs, ref_chain))
# ---------------------------- list ------------------------------
elif isinstance(node, list):
# Add all list items to queue
for idx, item in enumerate(node):
- queue.append((item, node, idx, active_defs, seen))
+ queue.append((item, node, idx, active_defs, ref_chain))
def _get_image_mime_type_from_url(url: str) -> Optional[str]:
@@ -822,3 +832,101 @@ def set_last_user_message(
messages.reverse()
messages.append({"role": "user", "content": content})
return messages
+
+
+def convert_prefix_message_to_non_prefix_messages(
+ messages: List[AllMessageValues],
+) -> List[AllMessageValues]:
+ """
+ For models that don't support {prefix: true} in messages, we need to convert the prefix message to a non-prefix message.
+
+ Use prompt:
+
+ {"role": "assistant", "content": "value", "prefix": true} -> [
+ {
+ "role": "system",
+ "content": "You are a helpful assistant. You are given a message and you need to respond to it. You are also given a generated content. You need to respond to the message in continuation of the generated content. Do not repeat the same content. Your response should be in continuation of this text: ",
+ },
+ {
+ "role": "assistant",
+ "content": message["content"],
+ },
+ ]
+
+ do this in place
+ """
+ new_messages: List[AllMessageValues] = []
+ for message in messages:
+ if message.get("prefix"):
+ new_messages.append(
+ {
+ "role": "system",
+ "content": "You are a helpful assistant. You are given a message and you need to respond to it. You are also given a generated content. You need to respond to the message in continuation of the generated content. Do not repeat the same content. Your response should be in continuation of this text: ",
+ }
+ )
+ new_messages.append(
+ {**{k: v for k, v in message.items() if k != "prefix"}} # type: ignore
+ )
+ else:
+ new_messages.append(message)
+ return new_messages
+
+
+def _extract_reasoning_content(message: dict) -> Tuple[Optional[str], Optional[str]]:
+ """
+ Extract reasoning content and main content from a message.
+
+ Args:
+ message (dict): The message dictionary that may contain reasoning_content
+
+ Returns:
+ tuple[Optional[str], Optional[str]]: A tuple of (reasoning_content, content)
+ """
+ message_content = message.get("content")
+ if "reasoning_content" in message:
+ return message["reasoning_content"], message["content"]
+ elif "reasoning" in message:
+ return message["reasoning"], message["content"]
+ elif isinstance(message_content, str):
+ return _parse_content_for_reasoning(message_content)
+ return None, message_content
+
+
+def _parse_content_for_reasoning(
+ message_text: Optional[str],
+) -> Tuple[Optional[str], Optional[str]]:
+ """
+ Parse the content for reasoning
+
+ Returns:
+ - reasoning_content: The content of the reasoning
+ - content: The content of the message
+ """
+ if not message_text:
+ return None, message_text
+
+ reasoning_match = re.match(
+ r"<(?:think|thinking)>(.*?)(?:think|thinking)>(.*)", message_text, re.DOTALL
+ )
+
+ if reasoning_match:
+ return reasoning_match.group(1), reasoning_match.group(2)
+
+ return None, message_text
+
+
+def extract_images_from_message(message: AllMessageValues) -> List[str]:
+ """
+ Extract images from a message
+ """
+ images = []
+ message_content = message.get("content")
+ if isinstance(message_content, list):
+ for m in message_content:
+ image_url = m.get("image_url")
+ if image_url:
+ if isinstance(image_url, str):
+ images.append(image_url)
+ elif isinstance(image_url, dict) and "url" in image_url:
+ images.append(image_url["url"])
+ return images
diff --git a/litellm/litellm_core_utils/prompt_templates/factory.py b/litellm/litellm_core_utils/prompt_templates/factory.py
index 91a5b317fd6..d2cad0abd93 100644
--- a/litellm/litellm_core_utils/prompt_templates/factory.py
+++ b/litellm/litellm_core_utils/prompt_templates/factory.py
@@ -1,7 +1,7 @@
import copy
import json
+import mimetypes
import re
-import uuid
import xml.etree.ElementTree as ET
from enum import Enum
from typing import Any, List, Optional, Tuple, cast, overload
@@ -12,8 +12,11 @@ import litellm
import litellm.types
import litellm.types.llms
from litellm import verbose_logger
+from litellm._uuid import uuid
from litellm.llms.custom_httpx.http_handler import HTTPHandler, get_async_httpx_client
+from litellm.types.files import get_file_extension_from_mime_type
from litellm.types.llms.anthropic import *
+from litellm.types.llms.bedrock import CachePointBlock
from litellm.types.llms.bedrock import MessageBlock as BedrockMessageBlock
from litellm.types.llms.custom_http import httpxSpecialProvider
from litellm.types.llms.ollama import OllamaVisionModelObject
@@ -229,7 +232,6 @@ def ollama_pt(
## MERGE CONSECUTIVE ASSISTANT CONTENT ##
while msg_i < len(messages) and messages[msg_i]["role"] == "assistant":
assistant_content_str += convert_content_list_to_str(messages[msg_i])
- msg_i += 1
tool_calls = messages[msg_i].get("tool_calls")
ollama_tool_calls = []
@@ -255,7 +257,7 @@ def ollama_pt(
f"Tool Calls: {json.dumps(ollama_tool_calls, indent=2)}"
)
- msg_i += 1
+ msg_i += 1
if assistant_content_str:
prompt += f"### Assistant:\n{assistant_content_str}\n\n"
@@ -362,62 +364,20 @@ def phind_codellama_pt(messages):
return prompt
-def hf_chat_template( # noqa: PLR0915
- model: str, messages: list, chat_template: Optional[Any] = None
-):
- # Define Jinja2 environment
- env = ImmutableSandboxedEnvironment()
-
- def raise_exception(message):
- raise Exception(f"Error message - {message}")
-
- # Create a template object from the template text
- env.globals["raise_exception"] = raise_exception
-
- ## get the tokenizer config from huggingface
- bos_token = ""
- eos_token = ""
- if chat_template is None:
-
- def _get_tokenizer_config(hf_model_name):
- try:
- url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
- # Make a GET request to fetch the JSON data
- client = HTTPHandler(concurrent_limit=1)
-
- response = client.get(url)
- except Exception as e:
- raise e
- if response.status_code == 200:
- # Parse the JSON data
- tokenizer_config = json.loads(response.content)
- return {"status": "success", "tokenizer": tokenizer_config}
- else:
- return {"status": "failure"}
-
- if model in litellm.known_tokenizer_config:
- tokenizer_config = litellm.known_tokenizer_config[model]
- else:
- tokenizer_config = _get_tokenizer_config(model)
- litellm.known_tokenizer_config.update({model: tokenizer_config})
-
- if (
- tokenizer_config["status"] == "failure"
- or "chat_template" not in tokenizer_config["tokenizer"]
- ):
- raise Exception("No chat template found")
- ## read the bos token, eos token and chat template from the json
- tokenizer_config = tokenizer_config["tokenizer"] # type: ignore
-
- bos_token = tokenizer_config["bos_token"] # type: ignore
- if bos_token is not None and not isinstance(bos_token, str):
- if isinstance(bos_token, dict):
- bos_token = bos_token.get("content", None)
- eos_token = tokenizer_config["eos_token"] # type: ignore
- if eos_token is not None and not isinstance(eos_token, str):
- if isinstance(eos_token, dict):
- eos_token = eos_token.get("content", None)
- chat_template = tokenizer_config["chat_template"] # type: ignore
+def _render_chat_template(env, chat_template: str, bos_token: str, eos_token: str, messages: list) -> str:
+ """
+ Shared template rendering logic for both sync and async hf_chat_template
+
+ Args:
+ env: Jinja2 environment
+ chat_template: Chat template string
+ bos_token: Beginning of sequence token
+ eos_token: End of sequence token
+ messages: Messages to render
+
+ Returns:
+ Rendered template string
+ """
try:
template = env.from_string(chat_template) # type: ignore
except Exception as e:
@@ -432,7 +392,6 @@ def hf_chat_template( # noqa: PLR0915
bos_token="",
)
return True
-
# This will be raised if Jinja attempts to render the system message and it can't
except Exception:
return False
@@ -466,7 +425,7 @@ def hf_chat_template( # noqa: PLR0915
)
except Exception as e:
if "Conversation roles must alternate user/assistant" in str(e):
- # reformat messages to ensure user/assistant are alternating, if there's either 2 consecutive 'user' messages or 2 consecutive 'assistant' message, add a blank 'user' or 'assistant' message to ensure compatibility
+ # reformat messages to ensure user/assistant are alternating
new_messages = []
for i in range(len(reformatted_messages) - 1):
new_messages.append(reformatted_messages[i])
@@ -492,6 +451,188 @@ def hf_chat_template( # noqa: PLR0915
) # don't use verbose_logger.exception, if exception is raised
+async def _afetch_and_extract_template(
+ model: str, chat_template: Optional[Any], get_config_fn, get_template_fn
+) -> Tuple[str, str, str]:
+ """
+ Async version: Fetch template and tokens from HuggingFace.
+
+ Returns: (chat_template, bos_token, eos_token)
+ """
+ from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
+ _extract_token_value,
+ )
+
+ bos_token = ""
+ eos_token = ""
+
+ if chat_template is None:
+ # Fetch or retrieve cached tokenizer config
+ if model in litellm.known_tokenizer_config:
+ tokenizer_config = litellm.known_tokenizer_config[model]
+ else:
+ tokenizer_config = await get_config_fn(hf_model_name=model)
+ litellm.known_tokenizer_config.update({model: tokenizer_config})
+
+ # Try to get chat template from tokenizer_config.json first
+ if (
+ tokenizer_config.get("status") == "success"
+ and "tokenizer" in tokenizer_config
+ and isinstance(tokenizer_config["tokenizer"], dict)
+ and "chat_template" in tokenizer_config["tokenizer"]
+ ):
+ tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
+ bos_token = _extract_token_value(
+ token_value=tokenizer_data.get("bos_token")
+ )
+ eos_token = _extract_token_value(
+ token_value=tokenizer_data.get("eos_token")
+ )
+ chat_template = tokenizer_data["chat_template"]
+ else:
+ # Fallback: Try to fetch chat template from separate .jinja file
+ template_result = await get_template_fn(hf_model_name=model)
+ if template_result.get("status") == "success":
+ chat_template = template_result["chat_template"]
+ # Still try to get tokens from tokenizer_config if available
+ if (
+ tokenizer_config.get("status") == "success"
+ and "tokenizer" in tokenizer_config
+ and isinstance(tokenizer_config["tokenizer"], dict)
+ ):
+ tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
+ bos_token = _extract_token_value(
+ token_value=tokenizer_data.get("bos_token")
+ )
+ eos_token = _extract_token_value(
+ token_value=tokenizer_data.get("eos_token")
+ )
+ else:
+ raise Exception("No chat template found")
+
+ return chat_template, bos_token, eos_token # type: ignore
+
+
+def _fetch_and_extract_template(
+ model: str, chat_template: Optional[Any], get_config_fn, get_template_fn
+) -> Tuple[str, str, str]:
+ """
+ Sync version: Fetch template and tokens from HuggingFace.
+
+ Returns: (chat_template, bos_token, eos_token)
+ """
+ from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
+ _extract_token_value,
+ )
+
+ bos_token = ""
+ eos_token = ""
+
+ if chat_template is None:
+ # Fetch or retrieve cached tokenizer config
+ if model in litellm.known_tokenizer_config:
+ tokenizer_config = litellm.known_tokenizer_config[model]
+ else:
+ tokenizer_config = get_config_fn(hf_model_name=model)
+ litellm.known_tokenizer_config.update({model: tokenizer_config})
+
+ # Try to get chat template from tokenizer_config.json first
+ if (
+ tokenizer_config.get("status") == "success"
+ and "tokenizer" in tokenizer_config
+ and isinstance(tokenizer_config["tokenizer"], dict)
+ and "chat_template" in tokenizer_config["tokenizer"]
+ ):
+ tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
+ bos_token = _extract_token_value(
+ token_value=tokenizer_data.get("bos_token")
+ )
+ eos_token = _extract_token_value(
+ token_value=tokenizer_data.get("eos_token")
+ )
+ chat_template = tokenizer_data["chat_template"]
+ else:
+ # Fallback: Try to fetch chat template from separate .jinja file
+ template_result = get_template_fn(hf_model_name=model)
+ if template_result.get("status") == "success":
+ chat_template = template_result["chat_template"]
+ # Still try to get tokens from tokenizer_config if available
+ if (
+ tokenizer_config.get("status") == "success"
+ and "tokenizer" in tokenizer_config
+ and isinstance(tokenizer_config["tokenizer"], dict)
+ ):
+ tokenizer_data: dict = tokenizer_config["tokenizer"] # type: ignore
+ bos_token = _extract_token_value(
+ token_value=tokenizer_data.get("bos_token")
+ )
+ eos_token = _extract_token_value(
+ token_value=tokenizer_data.get("eos_token")
+ )
+ else:
+ raise Exception("No chat template found")
+
+ return chat_template, bos_token, eos_token # type: ignore
+
+
+async def ahf_chat_template(
+ model: str, messages: list, chat_template: Optional[Any] = None
+):
+ """HuggingFace chat template (async version)"""
+ from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
+ _aget_chat_template_file,
+ _aget_tokenizer_config,
+ strftime_now,
+ )
+
+ env = ImmutableSandboxedEnvironment()
+ env.globals["raise_exception"] = lambda msg: Exception(f"Error message - {msg}")
+ env.globals["strftime_now"] = strftime_now
+
+ template, bos_token, eos_token = await _afetch_and_extract_template(
+ model=model,
+ chat_template=chat_template,
+ get_config_fn=_aget_tokenizer_config,
+ get_template_fn=_aget_chat_template_file,
+ )
+ return _render_chat_template(
+ env=env,
+ chat_template=template,
+ bos_token=bos_token,
+ eos_token=eos_token,
+ messages=messages,
+ )
+
+
+def hf_chat_template(
+ model: str, messages: list, chat_template: Optional[Any] = None
+):
+ """HuggingFace chat template (sync version)"""
+ from litellm.litellm_core_utils.prompt_templates.huggingface_template_handler import (
+ _get_chat_template_file,
+ _get_tokenizer_config,
+ strftime_now,
+ )
+
+ env = ImmutableSandboxedEnvironment()
+ env.globals["raise_exception"] = lambda msg: Exception(f"Error message - {msg}")
+ env.globals["strftime_now"] = strftime_now
+
+ template, bos_token, eos_token = _fetch_and_extract_template(
+ model=model,
+ chat_template=chat_template,
+ get_config_fn=_get_tokenizer_config,
+ get_template_fn=_get_chat_template_file,
+ )
+ return _render_chat_template(
+ env=env,
+ chat_template=template,
+ bos_token=bos_token,
+ eos_token=eos_token,
+ messages=messages,
+ )
+
+
def deepseek_r1_pt(messages):
return hf_chat_template(
model="deepseek-r1/deepseek-r1-7b-instruct", messages=messages
@@ -1064,10 +1205,10 @@ def convert_to_gemini_tool_call_invoke(
if tool_calls is not None:
for tool in tool_calls:
if "function" in tool:
- gemini_function_call: Optional[VertexFunctionCall] = (
- _gemini_tool_call_invoke_helper(
- function_call_params=tool["function"]
- )
+ gemini_function_call: Optional[
+ VertexFunctionCall
+ ] = _gemini_tool_call_invoke_helper(
+ function_call_params=tool["function"]
)
if gemini_function_call is not None:
_parts_list.append(
@@ -1121,13 +1262,14 @@ def convert_to_gemini_tool_call_result(
}
"""
content_str: str = ""
- if isinstance(message["content"], str):
- content_str = message["content"]
- elif isinstance(message["content"], List):
- content_list = message["content"]
- for content in content_list:
- if content["type"] == "text":
- content_str += content["text"]
+ if "content" in message:
+ if isinstance(message["content"], str):
+ content_str = message["content"]
+ elif isinstance(message["content"], List):
+ content_list = message["content"]
+ for content in content_list:
+ if content["type"] == "text":
+ content_str += content["text"]
name: Optional[str] = message.get("name", "") # type: ignore
# Recover name from last message with tool calls
@@ -1585,9 +1727,9 @@ def anthropic_messages_pt( # noqa: PLR0915
)
if "cache_control" in _content_element:
- _anthropic_content_element["cache_control"] = (
- _content_element["cache_control"]
- )
+ _anthropic_content_element[
+ "cache_control"
+ ] = _content_element["cache_control"]
user_content.append(_anthropic_content_element)
elif m.get("type", "") == "text":
m = cast(ChatCompletionTextObject, m)
@@ -1625,9 +1767,9 @@ def anthropic_messages_pt( # noqa: PLR0915
)
if "cache_control" in _content_element:
- _anthropic_content_text_element["cache_control"] = (
- _content_element["cache_control"]
- )
+ _anthropic_content_text_element[
+ "cache_control"
+ ] = _content_element["cache_control"]
user_content.append(_anthropic_content_text_element)
@@ -2350,7 +2492,6 @@ def stringify_json_tool_call_content(messages: List) -> List:
###### AMAZON BEDROCK #######
import base64
-import mimetypes
from email.message import Message
import httpx
@@ -2478,20 +2619,10 @@ class BedrockImageProcessor:
)
if is_document:
- potential_extensions = mimetypes.guess_all_extensions(mime_type)
- valid_extensions = [
- ext[1:]
- for ext in potential_extensions
- if ext[1:] in supported_doc_formats
- ]
+ return BedrockImageProcessor._get_document_format(
+ mime_type=mime_type, supported_doc_formats=supported_doc_formats
+ )
- if not valid_extensions:
- raise ValueError(
- f"No supported extensions for MIME type: {mime_type}. Supported formats: {supported_doc_formats}"
- )
-
- # Use first valid extension instead of provided image_format
- return valid_extensions[0]
else:
#########################################################
# Check if image_format is an image or video
@@ -2502,6 +2633,53 @@ class BedrockImageProcessor:
)
return image_format
+ @staticmethod
+ def _get_document_format(mime_type: str, supported_doc_formats: List[str]) -> str:
+ """
+ Get the document format from the mime type
+
+ - Primary method - uses `mimetypes.guess_all_extensions`
+ - Fallback method - uses `get_file_extension_from_mime_type`
+
+ Relevant Issue: https://github.com/BerriAI/litellm/issues/12260
+
+ `mimetypes` is not available in docker containers, so we fallback to `get_file_extension_from_mime_type`
+
+ Args:
+ mime_type: The mime type of the document
+ supported_doc_formats: The supported document formats for the current model
+
+ Returns:
+ The document format
+ """
+ valid_extensions: Optional[List[str]] = None
+ potential_extensions = mimetypes.guess_all_extensions(mime_type, strict=False)
+ valid_extensions = [
+ ext[1:] for ext in potential_extensions if ext[1:] in supported_doc_formats
+ ]
+
+ # Fallback to types/files.py if mimetypes doesn't return valid extensions
+ #################
+ # litellm runs on docker containers and `mimetypes` depends on the installed mimetypes of the OS
+ # we fallback to well known mime types in types/files.py if mimetypes doesn't return valid extensions
+ if not valid_extensions:
+ try:
+ fallback_extension = get_file_extension_from_mime_type(mime_type)
+ if fallback_extension in supported_doc_formats:
+ valid_extensions = [fallback_extension]
+ except ValueError:
+ # Neither mimetypes nor files.py could handle this MIME type
+ # get_file_extension_from_mime_type raises ValueError if the mime type is not supported
+ pass
+
+ if not valid_extensions:
+ raise ValueError(
+ f"No supported extensions for MIME type: {mime_type}. Supported formats: {supported_doc_formats}"
+ )
+
+ # Use first valid extension instead of provided image_format
+ return valid_extensions[0]
+
@staticmethod
def _create_bedrock_block(
image_bytes: str, mime_type: str, image_format: str
@@ -2632,12 +2810,22 @@ def _convert_to_bedrock_tool_call_invoke(
id = tool["id"]
name = tool["function"].get("name", "")
arguments = tool["function"].get("arguments", "")
- arguments_dict = json.loads(arguments) if arguments else {}
+ if not arguments or not arguments.strip():
+ arguments_dict = {}
+ else:
+ arguments_dict = json.loads(arguments)
bedrock_tool = BedrockToolUseBlock(
input=arguments_dict, name=name, toolUseId=id
)
bedrock_content_block = BedrockContentBlock(toolUse=bedrock_tool)
_parts_list.append(bedrock_content_block)
+
+ # Check for cache_control and add a separate cachePoint block
+ if tool.get("cache_control", None) is not None:
+ cache_point_block = BedrockContentBlock(
+ cachePoint=CachePointBlock(type="default")
+ )
+ _parts_list.append(cache_point_block)
return _parts_list
except Exception as e:
raise Exception(
@@ -2698,6 +2886,7 @@ def _convert_to_bedrock_tool_call_result(
for content in content_list:
if content["type"] == "text":
content_str += content["text"]
+
message.get("name", "")
id = str(message.get("tool_call_id", str(uuid.uuid4())))
@@ -2706,6 +2895,7 @@ def _convert_to_bedrock_tool_call_result(
content=[tool_result_content_block],
toolUseId=id,
)
+
content_block = BedrockContentBlock(toolResult=tool_result)
return content_block
@@ -2949,7 +3139,10 @@ def process_empty_text_blocks(
]
modified_message = message.copy()
- modified_message["content"] = modified_content_block
+ modified_message["content"] = cast(
+ Union[List[ChatCompletionTextObject], List[ChatCompletionThinkingBlock]],
+ modified_content_block,
+ )
return modified_message
@@ -3067,6 +3260,12 @@ class BedrockConverseMessagesProcessor:
if element["type"] == "text":
_part = BedrockContentBlock(text=element["text"])
_parts.append(_part)
+ elif element["type"] == "guarded_text":
+ # Wrap guarded_text in guardContent block
+ _part = BedrockContentBlock(
+ guardContent={"text": {"text": element["text"]}}
+ )
+ _parts.append(_part)
elif element["type"] == "image_url":
format: Optional[str] = None
if isinstance(element["image_url"], dict):
@@ -3135,9 +3334,33 @@ class BedrockConverseMessagesProcessor:
## MERGE CONSECUTIVE TOOL CALL MESSAGES ##
tool_content: List[BedrockContentBlock] = []
while msg_i < len(messages) and messages[msg_i]["role"] == "tool":
- tool_call_result = _convert_to_bedrock_tool_call_result(messages[msg_i])
-
+ current_message = messages[msg_i]
+ tool_call_result = _convert_to_bedrock_tool_call_result(current_message)
tool_content.append(tool_call_result)
+
+ # Check if we need to add a separate cachePoint block
+ has_cache_control = False
+
+ # Check for message-level cache_control
+ if current_message.get("cache_control", None) is not None:
+ has_cache_control = True
+ # Check for content-level cache_control in list content
+ elif isinstance(current_message.get("content"), list):
+ for content_element in current_message["content"]:
+ if (
+ isinstance(content_element, dict)
+ and content_element.get("cache_control", None) is not None
+ ):
+ has_cache_control = True
+ break
+
+ # Add a separate cachePoint block if cache_control is present
+ if has_cache_control:
+ cache_point_block = BedrockContentBlock(
+ cachePoint=CachePointBlock(type="default")
+ )
+ tool_content.append(cache_point_block)
+
msg_i += 1
if tool_content:
# if last message was a 'user' message, then add a blank assistant message (bedrock requires alternating roles)
@@ -3217,6 +3440,17 @@ class BedrockConverseMessagesProcessor:
image_url=image_url
)
assistants_parts.append(assistants_part)
+ # Add cache point block for assistant content elements
+ _cache_point_block = (
+ litellm.AmazonConverseConfig()._get_cache_point_block(
+ message_block=cast(
+ OpenAIMessageContentListBlock, element
+ ),
+ block_type="content_block",
+ )
+ )
+ if _cache_point_block is not None:
+ assistants_parts.append(_cache_point_block)
assistant_content.extend(assistants_parts)
elif _assistant_content is not None and isinstance(
_assistant_content, str
@@ -3224,6 +3458,15 @@ class BedrockConverseMessagesProcessor:
assistant_content.append(
BedrockContentBlock(text=_assistant_content)
)
+ # Add cache point block for assistant string content
+ _cache_point_block = (
+ litellm.AmazonConverseConfig()._get_cache_point_block(
+ assistant_message_block, block_type="content_block"
+ )
+ )
+ if _cache_point_block is not None:
+ assistant_content.append(_cache_point_block)
+
_tool_calls = assistant_message_block.get("tool_calls", [])
if _tool_calls:
assistant_content.extend(
@@ -3398,6 +3641,12 @@ def _bedrock_converse_messages_pt( # noqa: PLR0915
if element["type"] == "text":
_part = BedrockContentBlock(text=element["text"])
_parts.append(_part)
+ elif element["type"] == "guarded_text":
+ # Wrap guarded_text in guardContent block
+ _part = BedrockContentBlock(
+ guardContent={"text": {"text": element["text"]}}
+ )
+ _parts.append(_part)
elif element["type"] == "image_url":
format: Optional[str] = None
if isinstance(element["image_url"], dict):
@@ -3466,8 +3715,34 @@ def _bedrock_converse_messages_pt( # noqa: PLR0915
tool_content: List[BedrockContentBlock] = []
while msg_i < len(messages) and messages[msg_i]["role"] == "tool":
tool_call_result = _convert_to_bedrock_tool_call_result(messages[msg_i])
+ current_message = messages[msg_i]
+ # Add the tool result first
tool_content.append(tool_call_result)
+
+ # Check if we need to add a separate cachePoint block
+ has_cache_control = False
+
+ # Check for message-level cache_control
+ if current_message.get("cache_control", None) is not None:
+ has_cache_control = True
+ # Check for content-level cache_control in list content
+ elif isinstance(current_message.get("content"), list):
+ for content_element in current_message["content"]:
+ if (
+ isinstance(content_element, dict)
+ and content_element.get("cache_control", None) is not None
+ ):
+ has_cache_control = True
+ break
+
+ # Add a separate cachePoint block if cache_control is present
+ if has_cache_control:
+ cache_point_block = BedrockContentBlock(
+ cachePoint=CachePointBlock(type="default")
+ )
+ tool_content.append(cache_point_block)
+
msg_i += 1
if tool_content:
# if last message was a 'user' message, then add a blank assistant message (bedrock requires alternating roles)
@@ -3539,9 +3814,28 @@ def _bedrock_converse_messages_pt( # noqa: PLR0915
image_url=image_url
)
assistants_parts.append(assistants_part)
+ # Add cache point block for assistant content elements
+ _cache_point_block = (
+ litellm.AmazonConverseConfig()._get_cache_point_block(
+ message_block=cast(
+ OpenAIMessageContentListBlock, element
+ ),
+ block_type="content_block",
+ )
+ )
+ if _cache_point_block is not None:
+ assistants_parts.append(_cache_point_block)
assistant_content.extend(assistants_parts)
elif _assistant_content is not None and isinstance(_assistant_content, str):
assistant_content.append(BedrockContentBlock(text=_assistant_content))
+ # Add cache point block for assistant string content
+ _cache_point_block = (
+ litellm.AmazonConverseConfig()._get_cache_point_block(
+ assistant_message_block, block_type="content_block"
+ )
+ )
+ if _cache_point_block is not None:
+ assistant_content.append(_cache_point_block)
_tool_calls = assistant_message_block.get("tool_calls", [])
if _tool_calls:
assistant_content.extend(
@@ -3710,7 +4004,12 @@ def function_call_prompt(messages: list, functions: list):
function_added_to_prompt = False
for message in messages:
if "system" in message["role"]:
- message["content"] += f""" {function_prompt}"""
+ if isinstance(message["content"], str):
+ message["content"] += f""" {function_prompt}"""
+ else:
+ message["content"].append(
+ {"type": "text", "text": f""" {function_prompt}"""}
+ )
function_added_to_prompt = True
if function_added_to_prompt is False:
@@ -3871,33 +4170,9 @@ def prompt_factory(
elif custom_llm_provider == "azure_text":
return azure_text_pt(messages=messages)
elif custom_llm_provider == "watsonx":
- if "granite" in model and "chat" in model:
- # granite-13b-chat-v1 and granite-13b-chat-v2 use a specific prompt template
- return ibm_granite_pt(messages=messages)
- elif "ibm-mistral" in model and "instruct" in model:
- # models like ibm-mistral/mixtral-8x7b-instruct-v01-q use the mistral instruct prompt template
- return mistral_instruct_pt(messages=messages)
- elif "meta-llama/llama-3" in model and "instruct" in model:
- # https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/
- return custom_prompt(
- role_dict={
- "system": {
- "pre_message": "<|start_header_id|>system<|end_header_id|>\n",
- "post_message": "<|eot_id|>",
- },
- "user": {
- "pre_message": "<|start_header_id|>user<|end_header_id|>\n",
- "post_message": "<|eot_id|>",
- },
- "assistant": {
- "pre_message": "<|start_header_id|>assistant<|end_header_id|>\n",
- "post_message": "<|eot_id|>",
- },
- },
- messages=messages,
- initial_prompt_value="<|begin_of_text|>",
- final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
- )
+ from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
+ return IBMWatsonXChatConfig.apply_prompt_template(model=model, messages=messages)
+
try:
if "meta-llama/llama-2" in model and "chat" in model:
return llama_2_chat_pt(messages=messages)
diff --git a/litellm/litellm_core_utils/prompt_templates/huggingface_template_handler.py b/litellm/litellm_core_utils/prompt_templates/huggingface_template_handler.py
new file mode 100644
index 00000000000..9305d5bbfc1
--- /dev/null
+++ b/litellm/litellm_core_utils/prompt_templates/huggingface_template_handler.py
@@ -0,0 +1,139 @@
+import json
+from datetime import datetime
+from typing import Any, Dict, Union
+
+from litellm.llms.custom_httpx.http_handler import (
+ _get_httpx_client,
+ get_async_httpx_client,
+)
+from litellm.types.llms.custom_http import httpxSpecialProvider
+
+
+def strftime_now(fmt: str) -> str:
+ """
+ Custom function for templates that need current date/time formatting (e.g., gpt-oss)
+
+ Args:
+ fmt: Format string for datetime.now().strftime()
+
+ Returns:
+ Formatted string
+ """
+ return datetime.now().strftime(fmt)
+
+
+def _get_tokenizer_config(hf_model_name: str) -> Dict[str, Any]:
+ """
+ Fetch tokenizer_config.json from HuggingFace (sync)
+
+ Args:
+ hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
+
+ Returns:
+ Dict with 'status' and optionally 'tokenizer' keys
+ """
+ try:
+ url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
+ client = _get_httpx_client()
+ response = client.get(url=url)
+ except Exception as e:
+ raise e
+ if response.status_code == 200:
+ tokenizer_config = json.loads(response.content)
+ return {"status": "success", "tokenizer": tokenizer_config}
+ else:
+ return {"status": "failure"}
+
+
+async def _aget_tokenizer_config(hf_model_name: str) -> Dict[str, Any]:
+ """
+ Fetch tokenizer_config.json from HuggingFace (async)
+
+ Args:
+ hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
+
+ Returns:
+ Dict with 'status' and optionally 'tokenizer' keys
+ """
+ try:
+ url = f"https://huggingface.co/{hf_model_name}/raw/main/tokenizer_config.json"
+ client = get_async_httpx_client(
+ llm_provider=httpxSpecialProvider.PromptFactory,
+ )
+ response = await client.get(url=url)
+ except Exception as e:
+ raise e
+ if response.status_code == 200:
+ tokenizer_config = json.loads(response.content)
+ return {"status": "success", "tokenizer": tokenizer_config}
+ else:
+ return {"status": "failure"}
+
+
+def _get_chat_template_file(hf_model_name: str) -> Dict[str, Any]:
+ """
+ Fetch chat template from separate .jinja file (sync)
+
+ Args:
+ hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
+
+ Returns:
+ Dict with 'status' and optionally 'chat_template' keys
+ """
+ template_filenames = ["chat_template.jinja", "chat_template.jinja2"]
+ client = _get_httpx_client()
+
+ for filename in template_filenames:
+ try:
+ url = f"https://huggingface.co/{hf_model_name}/raw/main/{filename}"
+ response = client.get(url=url)
+ if response.status_code == 200:
+ return {"status": "success", "chat_template": response.content.decode("utf-8")}
+ except Exception:
+ continue
+
+ return {"status": "failure"}
+
+
+async def _aget_chat_template_file(hf_model_name: str) -> Dict[str, Any]:
+ """
+ Fetch chat template from separate .jinja file (async)
+
+ Args:
+ hf_model_name: HuggingFace model name (e.g., 'openai/gpt-oss-120b')
+
+ Returns:
+ Dict with 'status' and optionally 'chat_template' keys
+ """
+ template_filenames = ["chat_template.jinja", "chat_template.jinja2"]
+ client = get_async_httpx_client(
+ llm_provider=httpxSpecialProvider.PromptFactory,
+ )
+
+ for filename in template_filenames:
+ try:
+ url = f"https://huggingface.co/{hf_model_name}/raw/main/{filename}"
+ response = await client.get(url=url)
+ if response.status_code == 200:
+ return {"status": "success", "chat_template": response.content.decode("utf-8")}
+ except Exception:
+ continue
+
+ return {"status": "failure"}
+
+
+def _extract_token_value(token_value: Union[None, str, Dict[str, Any]]) -> str:
+ """
+ Extract token string from various formats (string, dict, etc.)
+
+ Args:
+ token_value: Token value in various formats (None, str, or dict with 'content' key)
+
+ Returns:
+ Extracted token string
+ """
+ if token_value is None or isinstance(token_value, str):
+ return token_value or ""
+ if isinstance(token_value, dict):
+ return token_value.get("content", "")
+ return ""
\ No newline at end of file
diff --git a/litellm/litellm_core_utils/prompt_templates/image_handling.py b/litellm/litellm_core_utils/prompt_templates/image_handling.py
index a9ff14d6c82..4fa10e42111 100644
--- a/litellm/litellm_core_utils/prompt_templates/image_handling.py
+++ b/litellm/litellm_core_utils/prompt_templates/image_handling.py
@@ -17,7 +17,7 @@ in_memory_cache = InMemoryCache(max_size_in_memory=MAX_IMGS_IN_MEMORY)
def _process_image_response(response: Response, url: str) -> str:
if response.status_code != 200:
- raise Exception(
+ raise litellm.ImageFetchError(
f"Error: Unable to fetch image from URL. Status code: {response.status_code}, url={url}"
)
@@ -57,9 +57,11 @@ async def async_convert_url_to_base64(url: str) -> str:
try:
response = await client.get(url, follow_redirects=True)
return _process_image_response(response, url)
+ except litellm.ImageFetchError:
+ raise
except Exception:
pass
- raise Exception(
+ raise litellm.ImageFetchError(
f"Error: Unable to fetch image from URL after 3 attempts. url={url}"
)
@@ -74,10 +76,11 @@ def convert_url_to_base64(url: str) -> str:
try:
response = client.get(url, follow_redirects=True)
return _process_image_response(response, url)
+ except litellm.ImageFetchError:
+ raise
except Exception as e:
verbose_logger.exception(e)
- # print(e)
pass
- raise Exception(
- f"Error: Unable to fetch image from URL after 3 attempts. url={url}"
+ raise litellm.ImageFetchError(
+ f"Error: Unable to fetch image from URL after 3 attempts. url={url}",
)
diff --git a/litellm/litellm_core_utils/rules.py b/litellm/litellm_core_utils/rules.py
index beeb012d032..717ff55ab22 100644
--- a/litellm/litellm_core_utils/rules.py
+++ b/litellm/litellm_core_utils/rules.py
@@ -23,6 +23,11 @@ class Rules:
def __init__(self) -> None:
pass
+ @staticmethod
+ def has_pre_call_rules() -> bool:
+ """Check if any pre-call rules are configured"""
+ return len(litellm.pre_call_rules) > 0
+
def pre_call_rules(self, input: str, model: str):
for rule in litellm.pre_call_rules:
if callable(rule):
diff --git a/litellm/litellm_core_utils/safe_json_dumps.py b/litellm/litellm_core_utils/safe_json_dumps.py
index 7ad0038ecb2..c714e36b5f9 100644
--- a/litellm/litellm_core_utils/safe_json_dumps.py
+++ b/litellm/litellm_core_utils/safe_json_dumps.py
@@ -1,5 +1,6 @@
import json
from typing import Any, Union
+
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH
diff --git a/litellm/litellm_core_utils/sensitive_data_masker.py b/litellm/litellm_core_utils/sensitive_data_masker.py
index 900239602df..ea0bed30416 100644
--- a/litellm/litellm_core_utils/sensitive_data_masker.py
+++ b/litellm/litellm_core_utils/sensitive_data_masker.py
@@ -21,6 +21,8 @@ class SensitiveDataMasker:
"access",
"private",
"certificate",
+ "fingerprint",
+ "tenancy",
}
self.visible_prefix = visible_prefix
@@ -33,11 +35,23 @@ class SensitiveDataMasker:
value_str = str(value)
masked_length = len(value_str) - (self.visible_prefix + self.visible_suffix)
- return f"{value_str[:self.visible_prefix]}{self.mask_char * masked_length}{value_str[-self.visible_suffix:]}"
+
+ # Handle the case where visible_suffix is 0 to avoid showing the entire string
+ if self.visible_suffix == 0:
+ return f"{value_str[:self.visible_prefix]}{self.mask_char * masked_length}"
+ else:
+ return f"{value_str[:self.visible_prefix]}{self.mask_char * masked_length}{value_str[-self.visible_suffix:]}"
def is_sensitive_key(self, key: str) -> bool:
key_lower = str(key).lower()
- result = any(pattern in key_lower for pattern in self.sensitive_patterns)
+ # Split on underscores and check if any segment matches the pattern
+ # This avoids false positives like "max_tokens" matching "token"
+ # but still catches "api_key", "access_token", etc.
+ key_segments = key_lower.replace('-', '_').split('_')
+ result = any(
+ pattern in key_segments
+ for pattern in self.sensitive_patterns
+ )
return result
def mask_dict(
@@ -61,7 +75,7 @@ class SensitiveDataMasker:
masked_data[k] = self._mask_value(str_value)
else:
masked_data[k] = (
- v if isinstance(v, (int, float, bool, str)) else str(v)
+ v if isinstance(v, (int, float, bool, str, list)) else str(v)
)
except Exception:
masked_data[k] = ""
@@ -75,12 +89,14 @@ masker = SensitiveDataMasker()
data = {
"api_key": "sk-1234567890abcdef",
"redis_password": "very_secret_pass",
- "port": 6379
+ "port": 6379,
+ "tags": ["East US 2", "production", "test"]
}
masked = masker.mask_dict(data)
# Result: {
# "api_key": "sk-1****cdef",
# "redis_password": "very****pass",
-# "port": 6379
+# "port": 6379,
+# "tags": ["East US 2", "production", "test"]
# }
"""
diff --git a/litellm/litellm_core_utils/streaming_chunk_builder_utils.py b/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
index fb919afd49d..2f85c7aef60 100644
--- a/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
+++ b/litellm/litellm_core_utils/streaming_chunk_builder_utils.py
@@ -527,7 +527,12 @@ class ChunkProcessor:
returned_usage, "cache_read_input_tokens", cache_read_input_tokens
) # for anthropic
if completion_tokens_details is not None:
- returned_usage.completion_tokens_details = completion_tokens_details
+ if isinstance(completion_tokens_details, CompletionTokensDetails):
+ returned_usage.completion_tokens_details = CompletionTokensDetailsWrapper(
+ **completion_tokens_details.model_dump()
+ )
+ else:
+ returned_usage.completion_tokens_details = completion_tokens_details
if reasoning_tokens is not None:
if returned_usage.completion_tokens_details is None:
diff --git a/litellm/litellm_core_utils/streaming_handler.py b/litellm/litellm_core_utils/streaming_handler.py
index 17f5a8d1deb..1daf543cfcb 100644
--- a/litellm/litellm_core_utils/streaming_handler.py
+++ b/litellm/litellm_core_utils/streaming_handler.py
@@ -5,7 +5,6 @@ import json
import threading
import time
import traceback
-import uuid
from typing import Any, Callable, Dict, List, Optional, Union, cast
import httpx
@@ -13,6 +12,10 @@ from pydantic import BaseModel
import litellm
from litellm import verbose_logger
+from litellm._uuid import uuid
+from litellm.litellm_core_utils.model_response_utils import (
+ is_model_response_stream_empty,
+)
from litellm.litellm_core_utils.redact_messages import LiteLLMLoggingObject
from litellm.litellm_core_utils.thread_pool_executor import executor
from litellm.types.llms.openai import ChatCompletionChunk
@@ -32,6 +35,12 @@ from .exception_mapping_utils import exception_type
from .llm_response_utils.get_api_base import get_api_base
from .rules import Rules
+# Constants for special delta attribute names
+AUDIO_ATTRIBUTE = "audio"
+IMAGE_ATTRIBUTE = "images"
+TOOL_CALLS_ATTRIBUTE = "tool_calls"
+FUNCTION_CALL_ATTRIBUTE = "function_call"
+
def is_async_iterable(obj: Any) -> bool:
"""
@@ -763,6 +772,83 @@ class CustomStreamWrapper:
model_response.choices[0].delta = Delta(**_initial_delta)
return model_response
+ def _has_special_delta_content(self, model_response: ModelResponseStream) -> bool:
+ """
+ Check if the delta contains special content types (tool_calls, function_call, audio, or image).
+ """
+ if len(model_response.choices) == 0:
+ return False
+
+ delta = model_response.choices[0].delta
+
+ # Check for tool_calls or function_call
+ if (
+ getattr(delta, TOOL_CALLS_ATTRIBUTE, None) is not None
+ or getattr(delta, FUNCTION_CALL_ATTRIBUTE, None) is not None
+ ):
+ return True
+
+ # Check for audio
+ if (
+ hasattr(delta, AUDIO_ATTRIBUTE)
+ and getattr(delta, AUDIO_ATTRIBUTE, None) is not None
+ ):
+ return True
+
+ # Check for image
+ if (
+ hasattr(delta, IMAGE_ATTRIBUTE)
+ and getattr(delta, IMAGE_ATTRIBUTE, None) is not None
+ ):
+ return True
+
+ return False
+
+ def _handle_special_delta_content(
+ self, model_response: ModelResponseStream
+ ) -> ModelResponseStream:
+ """
+ Handle special delta content types by stripping role and returning the response.
+ """
+ return self.strip_role_from_delta(model_response)
+
+ def _has_special_delta_attribute(self, delta, attribute_name: str) -> bool:
+ """
+ Check if delta has a specific attribute and it's not None.
+ """
+ return delta is not None and getattr(delta, attribute_name, None) is not None
+
+ def _copy_delta_attribute(
+ self, source_delta, target_delta, attribute_name: str
+ ) -> None:
+ """
+ Copy a specific attribute from source delta to target delta.
+ """
+ setattr(target_delta, attribute_name, getattr(source_delta, attribute_name))
+
+ def _has_any_special_delta_attributes(self, delta) -> bool:
+ """
+ Check if delta has any special attributes (audio, image).
+ """
+ special_attributes = [AUDIO_ATTRIBUTE, IMAGE_ATTRIBUTE]
+ for attribute in special_attributes:
+ if self._has_special_delta_attribute(delta, attribute):
+ return True
+ return False
+
+ def _handle_special_delta_attributes(
+ self, delta, model_response: "ModelResponseStream"
+ ) -> None:
+ """
+ Handle special delta attributes (audio, image) by copying them to model_response.
+ """
+ special_attributes = [AUDIO_ATTRIBUTE, IMAGE_ATTRIBUTE]
+ for attribute in special_attributes:
+ if self._has_special_delta_attribute(delta, attribute):
+ self._copy_delta_attribute(
+ delta, model_response.choices[0].delta, attribute
+ )
+
def return_processed_chunk_logic( # noqa
self,
completion_obj: Dict[str, Any],
@@ -885,20 +971,8 @@ class CustomStreamWrapper:
self.sent_last_chunk = True
return model_response
- elif (
- model_response.choices[0].delta.tool_calls is not None
- or model_response.choices[0].delta.function_call is not None
- ):
- model_response = self.strip_role_from_delta(model_response)
-
- return model_response
- elif (
- len(model_response.choices) > 0
- and hasattr(model_response.choices[0].delta, "audio")
- and model_response.choices[0].delta.audio is not None
- ):
- model_response = self.strip_role_from_delta(model_response)
- return model_response
+ elif self._has_special_delta_content(model_response):
+ return self._handle_special_delta_content(model_response)
else:
if hasattr(model_response, "usage"):
self.chunks.append(model_response)
@@ -940,8 +1014,8 @@ class CustomStreamWrapper:
and not self.sent_last_thinking_block
and model_response.choices[0].delta.content
):
- model_response.choices[0].delta.content = (
- "" + (model_response.choices[0].delta.content or "")
+ model_response.choices[0].delta.content = "" + (
+ model_response.choices[0].delta.content or ""
)
self.sent_last_thinking_block = True
@@ -950,6 +1024,8 @@ class CustomStreamWrapper:
return
def chunk_creator(self, chunk: Any): # type: ignore # noqa: PLR0915
+ if hasattr(chunk, "id"):
+ self.response_id = chunk.id
model_response = self.model_response_creator()
response_obj: Dict[str, Any] = {}
try:
@@ -1289,12 +1365,13 @@ class CustomStreamWrapper:
f"model_response finish reason 3: {self.received_finish_reason}; response_obj={response_obj}"
)
## FUNCTION CALL PARSING
+ original_chunk = (
+ response_obj.get("original_chunk") if response_obj is not None else None
+ )
if (
- response_obj is not None
- and response_obj.get("original_chunk", None) is not None
+ original_chunk is not None
): # function / tool calling branch - only set for openai/azure compatible endpoints
# enter this branch when no content has been passed in response
- original_chunk = response_obj.get("original_chunk", None)
if hasattr(original_chunk, "id"):
model_response = self.set_model_id(
original_chunk.id, model_response
@@ -1371,10 +1448,8 @@ class CustomStreamWrapper:
)
)
model_response.choices[0].delta = Delta()
- elif (
- delta is not None and getattr(delta, "audio", None) is not None
- ):
- model_response.choices[0].delta.audio = delta.audio
+ elif self._has_any_special_delta_attributes(delta):
+ self._handle_special_delta_attributes(delta, model_response)
else:
try:
delta = (
@@ -1545,11 +1620,12 @@ class CustomStreamWrapper:
completion_start_time=datetime.datetime.now()
)
## LOGGING
- executor.submit(
- self.run_success_logging_and_cache_storage,
- response,
- cache_hit,
- ) # log response
+ if not litellm.disable_streaming_logging:
+ executor.submit(
+ self.run_success_logging_and_cache_storage,
+ response,
+ cache_hit,
+ ) # log response
choice = response.choices[0]
if isinstance(choice, StreamingChoices):
self.response_uptil_now += choice.delta.get("content", "") or ""
@@ -1574,6 +1650,13 @@ class CustomStreamWrapper:
response = self.model_response_creator(
chunk=obj_dict, hidden_params=response._hidden_params
)
+ ## check if empty
+ is_empty = is_model_response_stream_empty(
+ model_response=cast(ModelResponseStream, response)
+ )
+
+ if is_empty:
+ continue
# add usage as hidden param
if self.sent_last_chunk is True and self.stream_options is None:
usage = calculate_total_usage(chunks=self.chunks)
@@ -1584,7 +1667,9 @@ class CustomStreamWrapper:
except StopIteration:
if self.sent_last_chunk is True:
complete_streaming_response = litellm.stream_chunk_builder(
- chunks=self.chunks, messages=self.messages
+ chunks=self.chunks,
+ messages=self.messages,
+ logging_obj=self.logging_obj,
)
response = self.model_response_creator()
@@ -1728,7 +1813,18 @@ class CustomStreamWrapper:
# Create a new object without the removed attribute
processed_chunk = self.model_response_creator(chunk=obj_dict)
+ is_empty = is_model_response_stream_empty(
+ model_response=cast(ModelResponseStream, processed_chunk)
+ )
+
+ if is_empty:
+ continue
print_verbose(f"final returned processed chunk: {processed_chunk}")
+
+ # add usage as hidden param
+ if self.sent_last_chunk is True and self.stream_options is None:
+ usage = calculate_total_usage(chunks=self.chunks)
+ processed_chunk._hidden_params["usage"] = usage
return processed_chunk
raise StopAsyncIteration
else: # temporary patch for non-aiohttp async calls
@@ -1768,8 +1864,11 @@ class CustomStreamWrapper:
if self.sent_last_chunk is True:
# log the final chunk with accurate streaming values
complete_streaming_response = litellm.stream_chunk_builder(
- chunks=self.chunks, messages=self.messages
+ chunks=self.chunks,
+ messages=self.messages,
+ logging_obj=self.logging_obj,
)
+
response = self.model_response_creator()
if complete_streaming_response is not None:
setattr(
@@ -1841,13 +1940,25 @@ class CustomStreamWrapper:
self.logging_obj.async_failure_handler(e, traceback_exception) # type: ignore
)
## Map to OpenAI Exception
- raise exception_type(
- model=self.model,
- custom_llm_provider=self.custom_llm_provider,
- original_exception=e,
- completion_kwargs={},
- extra_kwargs={},
- )
+ try:
+ raise exception_type(
+ model=self.model,
+ custom_llm_provider=self.custom_llm_provider,
+ original_exception=e,
+ completion_kwargs={},
+ extra_kwargs={},
+ )
+ except Exception as e:
+ from litellm.exceptions import MidStreamFallbackError
+
+ raise MidStreamFallbackError(
+ message=str(e),
+ model=self.model,
+ llm_provider=self.custom_llm_provider or "anthropic",
+ original_exception=e,
+ generated_content=self.response_uptil_now,
+ is_pre_first_chunk=not self.sent_first_chunk,
+ )
@staticmethod
def _strip_sse_data_from_chunk(chunk: Optional[str]) -> Optional[str]:
diff --git a/litellm/litellm_core_utils/token_counter.py b/litellm/litellm_core_utils/token_counter.py
index 1eb4fc9016a..fab2c1e76ee 100644
--- a/litellm/litellm_core_utils/token_counter.py
+++ b/litellm/litellm_core_utils/token_counter.py
@@ -462,9 +462,8 @@ def _count_messages(
default_token_count,
)
else:
- raise ValueError(
- f"Unsupported type {type(value)} for key {key} in message {message}"
- )
+ # Skip unsupported keys instead of raising an error
+ continue
return num_tokens
@@ -530,7 +529,7 @@ def _get_count_function(
encoding = tiktoken.get_encoding("cl100k_base")
def count_tokens(text: str) -> int:
- return len(encoding.encode(text))
+ return len(encoding.encode(text, disallowed_special=()))
else:
raise ValueError("Unsupported tokenizer type")
diff --git a/litellm/llms/aiml/__init__.py b/litellm/llms/aiml/__init__.py
new file mode 100644
index 00000000000..42482760cda
--- /dev/null
+++ b/litellm/llms/aiml/__init__.py
@@ -0,0 +1,5 @@
+from .image_generation import get_aiml_image_generation_config
+
+__all__ = [
+ "get_aiml_image_generation_config",
+]
diff --git a/litellm/llms/aiml/chat/transformation.py b/litellm/llms/aiml/chat/transformation.py
new file mode 100644
index 00000000000..0f3e333343d
--- /dev/null
+++ b/litellm/llms/aiml/chat/transformation.py
@@ -0,0 +1,23 @@
+from typing import Optional, Tuple
+
+from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
+from litellm.secret_managers.main import get_secret_str
+
+
+class AIMLChatConfig(OpenAIGPTConfig):
+ @property
+ def custom_llm_provider(self) -> Optional[str]:
+ return "aiml"
+
+ def _get_openai_compatible_provider_info(
+ self, api_base: Optional[str], api_key: Optional[str]
+ ) -> Tuple[Optional[str], Optional[str]]:
+ # AIML is openai compatible, we just need to set the api_base
+ api_base = (
+ api_base
+ or get_secret_str("AIML_API_BASE")
+ or "https://api.aimlapi.com/v1" # Default AIML API base URL
+ ) # type: ignore
+ dynamic_api_key = api_key or get_secret_str("AIML_API_KEY")
+ return api_base, dynamic_api_key
+ pass
\ No newline at end of file
diff --git a/litellm/llms/aiml/image_generation/__init__.py b/litellm/llms/aiml/image_generation/__init__.py
new file mode 100644
index 00000000000..4548bd1b3f8
--- /dev/null
+++ b/litellm/llms/aiml/image_generation/__init__.py
@@ -0,0 +1,13 @@
+from litellm.llms.base_llm.image_generation.transformation import (
+ BaseImageGenerationConfig,
+)
+
+from .transformation import AimlImageGenerationConfig
+
+__all__ = [
+ "AimlImageGenerationConfig",
+]
+
+
+def get_aiml_image_generation_config(model: str) -> BaseImageGenerationConfig:
+ return AimlImageGenerationConfig()
diff --git a/litellm/llms/aiml/image_generation/cost_calculator.py b/litellm/llms/aiml/image_generation/cost_calculator.py
new file mode 100644
index 00000000000..1fecfb6a9a5
--- /dev/null
+++ b/litellm/llms/aiml/image_generation/cost_calculator.py
@@ -0,0 +1,25 @@
+from typing import Any
+
+import litellm
+from litellm.types.utils import ImageResponse
+
+
+def cost_calculator(
+ model: str,
+ image_response: Any,
+) -> float:
+ """
+ AI/ML flux image generation cost calculator
+ """
+ _model_info = litellm.get_model_info(
+ model=model,
+ custom_llm_provider=litellm.LlmProviders.AIML.value,
+ )
+ output_cost_per_image: float = _model_info.get("output_cost_per_image") or 0.0
+ num_images: int = 0
+ if isinstance(image_response, ImageResponse):
+ if image_response.data:
+ num_images = len(image_response.data)
+ return output_cost_per_image * num_images
+ else:
+ raise ValueError(f"image_response must be of type ImageResponse got type={type(image_response)}")
diff --git a/litellm/llms/aiml/image_generation/transformation.py b/litellm/llms/aiml/image_generation/transformation.py
new file mode 100644
index 00000000000..3b586689ea7
--- /dev/null
+++ b/litellm/llms/aiml/image_generation/transformation.py
@@ -0,0 +1,204 @@
+from typing import TYPE_CHECKING, Any, List, Optional
+
+import httpx
+
+from litellm.llms.base_llm.image_generation.transformation import (
+ BaseImageGenerationConfig,
+)
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.aiml import AimlImageGenerationRequestParams
+from litellm.types.llms.openai import (
+ AllMessageValues,
+ OpenAIImageGenerationOptionalParams,
+)
+from litellm.types.utils import ImageObject, ImageResponse
+
+if TYPE_CHECKING:
+ from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
+
+ LiteLLMLoggingObj = _LiteLLMLoggingObj
+else:
+ LiteLLMLoggingObj = Any
+
+
+class AimlImageGenerationConfig(BaseImageGenerationConfig):
+ DEFAULT_BASE_URL: str = "https://api.aimlapi.com"
+ IMAGE_GENERATION_ENDPOINT: str = "v1/images/generations"
+
+ def get_supported_openai_params(
+ self, model: str
+ ) -> List[OpenAIImageGenerationOptionalParams]:
+ """
+ https://api.aimlapi.com/v1/images/generations
+ """
+ return [
+ "n",
+ "response_format",
+ "size"
+ ]
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ supported_params = self.get_supported_openai_params(model)
+
+ for k in non_default_params.keys():
+ if k not in optional_params.keys():
+ if k in supported_params:
+ # Map OpenAI params to AI/ML params
+ if k == "n":
+ optional_params["num_images"] = non_default_params[k]
+ elif k == "response_format":
+ optional_params["output_format"] = non_default_params[k]
+ elif k == "size":
+ # Map OpenAI size format to AI/ML image_size
+ size_value = non_default_params[k]
+ if isinstance(size_value, str):
+ # Handle standard OpenAI sizes like "1024x1024"
+ if "x" in size_value:
+ width, height = map(int, size_value.split("x"))
+ optional_params["image_size"] = {"width": width, "height": height}
+ else:
+ # Pass through predefined sizes
+ optional_params["image_size"] = size_value
+ else:
+ optional_params["image_size"] = size_value
+ else:
+ optional_params[k] = non_default_params[k]
+ elif drop_params:
+ pass
+ else:
+ raise ValueError(
+ f"Parameter {k} is not supported for model {model}. Supported parameters are {supported_params}. Set drop_params=True to drop unsupported parameters."
+ )
+
+ return optional_params
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ """
+ Get the complete url for the request
+ """
+ complete_url: str = (
+ api_base
+ or get_secret_str("AIML_API_BASE")
+ or self.DEFAULT_BASE_URL
+ )
+
+ complete_url = complete_url.rstrip("/")
+ complete_url = f"{complete_url}/{self.IMAGE_GENERATION_ENDPOINT}"
+ return complete_url
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ final_api_key: Optional[str] = (
+ api_key or
+ get_secret_str("AIML_API_KEY") or
+ get_secret_str("AIMLAPI_KEY") # Alternative name
+ )
+ if not final_api_key:
+ raise ValueError("AIML_API_KEY or AIMLAPI_KEY is not set")
+
+ headers["Authorization"] = f"Bearer {final_api_key}"
+ headers["Content-Type"] = "application/json"
+ return headers
+
+ def transform_image_generation_request(
+ self,
+ model: str,
+ prompt: str,
+ optional_params: dict,
+ litellm_params: dict,
+ headers: dict,
+ ) -> dict:
+ """
+ Transform the image generation request to the AI/ML flux image generation request body
+
+ https://api.aimlapi.com/v1/images/generations
+ """
+ aiml_image_generation_request_body: AimlImageGenerationRequestParams = AimlImageGenerationRequestParams(
+ prompt=prompt,
+ model=model,
+ **optional_params,
+ )
+ return dict(aiml_image_generation_request_body)
+
+ def transform_image_generation_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: ImageResponse,
+ logging_obj: LiteLLMLoggingObj,
+ request_data: dict,
+ optional_params: dict,
+ litellm_params: dict,
+ encoding: Any,
+ api_key: Optional[str] = None,
+ json_mode: Optional[bool] = None,
+ ) -> ImageResponse:
+ """
+ Transform the image generation response to the litellm image response
+
+ https://api.aimlapi.com/v1/images/generations
+ """
+ try:
+ response_data = raw_response.json()
+ except Exception as e:
+ raise self.get_error_class(
+ error_message=f"Error transforming image generation response: {e}",
+ status_code=raw_response.status_code,
+ headers=raw_response.headers,
+ )
+
+ if not model_response.data:
+ model_response.data = []
+
+ # AI/ML API can return images in two different formats:
+ # 1. output.choices array with image_base64
+ # 2. images array with url (and optional width, height, content_type)
+
+ if "output" in response_data and "choices" in response_data["output"]:
+ for choice in response_data["output"]["choices"]:
+ if "image_base64" in choice:
+ model_response.data.append(ImageObject(
+ b64_json=choice["image_base64"],
+ url=None, # AI/ML API returns base64, not URLs
+ ))
+ elif "url" in choice:
+ model_response.data.append(ImageObject(
+ b64_json=None,
+ url=choice["url"],
+ ))
+ elif "images" in response_data:
+ # Handle alternative format: {"images": [{"url": "...", "width": 1024, "height": 768, "content_type": "image/jpeg"}]}
+ for image in response_data["images"]:
+ if "url" in image:
+ model_response.data.append(ImageObject(
+ b64_json=None,
+ url=image["url"],
+ ))
+ elif "image_base64" in image:
+ model_response.data.append(ImageObject(
+ b64_json=image["image_base64"],
+ url=None,
+ ))
+ return model_response
diff --git a/litellm/llms/anthropic/chat/handler.py b/litellm/llms/anthropic/chat/handler.py
index 620d2f05119..b7b39f10395 100644
--- a/litellm/llms/anthropic/chat/handler.py
+++ b/litellm/llms/anthropic/chat/handler.py
@@ -51,6 +51,7 @@ from litellm.types.utils import (
ModelResponseStream,
StreamingChoices,
Usage,
+ _generate_id,
)
from ...base import BaseLLM
@@ -490,6 +491,8 @@ class ModelResponseIterator:
self.content_blocks: List[ContentBlockDelta] = []
self.tool_index = -1
self.json_mode = json_mode
+ # Generate response ID once per stream to match OpenAI-compatible behavior
+ self.response_id = _generate_id()
# Track if we're currently streaming a response_format tool
self.is_response_format_tool: bool = False
@@ -640,7 +643,8 @@ class ModelResponseIterator:
]
] = None
- index = int(chunk.get("index", 0))
+ # Always use index=0 for OpenAI choice format (fixes multi-choice errors)
+ index = 0
if type_chunk == "content_block_delta":
"""
Anthropic content chunk
@@ -764,6 +768,7 @@ class ModelResponseIterator:
)
],
usage=usage,
+ id=self.response_id,
)
return returned_chunk
@@ -935,4 +940,4 @@ class ModelResponseIterator:
data_json = json.loads(str_line[5:])
return self.chunk_parser(chunk=data_json)
else:
- return ModelResponseStream()
+ return ModelResponseStream(id=self.response_id)
diff --git a/litellm/llms/anthropic/chat/transformation.py b/litellm/llms/anthropic/chat/transformation.py
index ce874bfde9a..691b46af8da 100644
--- a/litellm/llms/anthropic/chat/transformation.py
+++ b/litellm/llms/anthropic/chat/transformation.py
@@ -18,6 +18,8 @@ from litellm.litellm_core_utils.core_helpers import map_finish_reason
from litellm.llms.base_llm.base_utils import type_to_response_format_param
from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
from litellm.types.llms.anthropic import (
+ ANTHROPIC_BETA_HEADER_VALUES,
+ ANTHROPIC_HOSTED_TOOLS,
AllAnthropicMessageValues,
AllAnthropicToolsValues,
AnthropicCodeExecutionTool,
@@ -45,9 +47,15 @@ from litellm.types.llms.openai import (
OpenAIMcpServerTool,
OpenAIWebSearchOptions,
)
-from litellm.types.utils import CompletionTokensDetailsWrapper
+from litellm.types.utils import (
+ CacheCreationTokenDetails,
+ CompletionTokensDetailsWrapper,
+)
from litellm.types.utils import Message as LitellmMessage
-from litellm.types.utils import PromptTokensDetailsWrapper, ServerToolUse
+from litellm.types.utils import (
+ PromptTokensDetailsWrapper,
+ ServerToolUse,
+)
from litellm.utils import (
ModelResponse,
Usage,
@@ -67,9 +75,6 @@ else:
LoggingClass = Any
-ANTHROPIC_HOSTED_TOOLS = ["web_search", "bash", "text_editor", "code_execution"]
-
-
class AnthropicConfig(AnthropicModelInfo, BaseConfig):
"""
Reference: https://docs.anthropic.com/claude/reference/messages_post
@@ -200,8 +205,12 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
_allowed_properties = set(AnthropicInputSchema.__annotations__.keys())
- input_schema_filtered = {k: v for k, v in _input_schema.items() if k in _allowed_properties}
- input_anthropic_schema: AnthropicInputSchema = AnthropicInputSchema(**input_schema_filtered)
+ input_schema_filtered = {
+ k: v for k, v in _input_schema.items() if k in _allowed_properties
+ }
+ input_anthropic_schema: AnthropicInputSchema = AnthropicInputSchema(
+ **input_schema_filtered
+ )
_tool = AnthropicMessagesTool(
name=tool["function"]["name"],
@@ -632,6 +641,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
)
)
return tools
+
+ def update_headers_with_optional_anthropic_beta(self, headers: dict, optional_params: dict) -> dict:
+ """Update headers with optional anthropic beta."""
+ _tools = optional_params.get("tools", [])
+ for tool in _tools:
+ if tool.get("type", None) and tool.get("type").startswith(ANTHROPIC_HOSTED_TOOLS.WEB_FETCH.value):
+ headers["anthropic-beta"] = ANTHROPIC_BETA_HEADER_VALUES.WEB_FETCH_2025_09_10.value
+ return headers
def transform_request(
self,
@@ -668,6 +685,8 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
llm_provider="anthropic",
)
+ headers = self.update_headers_with_optional_anthropic_beta(headers=headers, optional_params=optional_params)
+
# Separate system prompt from rest of message
anthropic_system_message_list = self.translate_system_message(messages=messages)
# Handling anthropic API Prompt Caching
@@ -797,7 +816,15 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
if content.get("citations") is not None:
if citations is None:
citations = []
- citations.append(content["citations"])
+ citations.append(
+ [
+ {
+ **citation,
+ "supported_text": content.get("text", ""),
+ }
+ for citation in content["citations"]
+ ]
+ )
if thinking_blocks is not None:
reasoning_content = ""
for block in thinking_blocks:
@@ -816,12 +843,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
_usage = usage_object
cache_creation_input_tokens: int = 0
cache_read_input_tokens: int = 0
+ cache_creation_token_details: Optional[CacheCreationTokenDetails] = None
web_search_requests: Optional[int] = None
if (
"cache_creation_input_tokens" in _usage
and _usage["cache_creation_input_tokens"] is not None
):
cache_creation_input_tokens = _usage["cache_creation_input_tokens"]
+ prompt_tokens += cache_creation_input_tokens
if (
"cache_read_input_tokens" in _usage
and _usage["cache_read_input_tokens"] is not None
@@ -837,8 +866,20 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
int, _usage["server_tool_use"]["web_search_requests"]
)
+ if "cache_creation" in _usage and _usage["cache_creation"] is not None:
+ cache_creation_token_details = CacheCreationTokenDetails(
+ ephemeral_5m_input_tokens=_usage["cache_creation"].get(
+ "ephemeral_5m_input_tokens"
+ ),
+ ephemeral_1h_input_tokens=_usage["cache_creation"].get(
+ "ephemeral_1h_input_tokens"
+ ),
+ )
+
prompt_tokens_details = PromptTokensDetailsWrapper(
cached_tokens=cache_read_input_tokens,
+ cache_creation_tokens=cache_creation_input_tokens,
+ cache_creation_token_details=cache_creation_token_details,
)
completion_token_details = (
CompletionTokensDetailsWrapper(
diff --git a/litellm/llms/anthropic/common_utils.py b/litellm/llms/anthropic/common_utils.py
index c263d903188..68b5341e954 100644
--- a/litellm/llms/anthropic/common_utils.py
+++ b/litellm/llms/anthropic/common_utils.py
@@ -2,7 +2,7 @@
This file contains common utils for anthropic calls.
"""
-from typing import Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Union
import httpx
@@ -10,10 +10,11 @@ import litellm
from litellm.litellm_core_utils.prompt_templates.common_utils import (
get_file_ids_from_messages,
)
-from litellm.llms.base_llm.base_utils import BaseLLMModelInfo
+from litellm.llms.base_llm.base_utils import BaseLLMModelInfo, BaseTokenCounter
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.types.llms.anthropic import AllAnthropicToolsValues, AnthropicMcpServerTool
from litellm.types.llms.openai import AllMessageValues
+from litellm.types.utils import TokenCountResponse
class AnthropicError(BaseLLMException):
@@ -229,6 +230,53 @@ class AnthropicModelInfo(BaseLLMModelInfo):
litellm_model_names.append(litellm_model_name)
return litellm_model_names
+ def get_token_counter(self) -> Optional[BaseTokenCounter]:
+ """
+ Factory method to create an Anthropic token counter.
+
+ Returns:
+ AnthropicTokenCounter instance for this provider.
+ """
+ return AnthropicTokenCounter()
+
+
+class AnthropicTokenCounter(BaseTokenCounter):
+ """Token counter implementation for Anthropic provider."""
+
+ def should_use_token_counting_api(
+ self,
+ custom_llm_provider: Optional[str] = None,
+ ) -> bool:
+ from litellm.types.utils import LlmProviders
+ return custom_llm_provider == LlmProviders.ANTHROPIC.value
+
+ async def count_tokens(
+ self,
+ model_to_use: str,
+ messages: Optional[List[Dict[str, Any]]],
+ contents: Optional[List[Dict[str, Any]]],
+ deployment: Optional[Dict[str, Any]] = None,
+ request_model: str = "",
+ ) -> Optional[TokenCountResponse]:
+ from litellm.proxy.utils import count_tokens_with_anthropic_api
+
+ result = await count_tokens_with_anthropic_api(
+ model_to_use=model_to_use,
+ messages=messages,
+ deployment=deployment,
+ )
+
+ if result is not None:
+ return TokenCountResponse(
+ total_tokens=result.get("total_tokens", 0),
+ request_model=request_model,
+ model_used=model_to_use,
+ tokenizer_type=result.get("tokenizer_used", ""),
+ original_response=result,
+ )
+
+ return None
+
def process_anthropic_headers(headers: Union[httpx.Headers, dict]) -> dict:
openai_headers = {}
diff --git a/litellm/llms/anthropic/completion/transformation.py b/litellm/llms/anthropic/completion/transformation.py
index 9e3287aa8a1..a8798cd5d0e 100644
--- a/litellm/llms/anthropic/completion/transformation.py
+++ b/litellm/llms/anthropic/completion/transformation.py
@@ -55,9 +55,9 @@ class AnthropicTextConfig(BaseConfig):
to pass metadata to anthropic, it's {"user_id": "any-relevant-information"}
"""
- max_tokens_to_sample: Optional[
- int
- ] = litellm.max_tokens # anthropic requires a default
+ max_tokens_to_sample: Optional[int] = (
+ litellm.max_tokens
+ ) # anthropic requires a default
stop_sequences: Optional[list] = None
temperature: Optional[int] = None
top_p: Optional[int] = None
@@ -291,7 +291,7 @@ class AnthropicTextCompletionResponseIterator(BaseModelResponseIterator):
_chunk_text = chunk.get("completion", None)
if _chunk_text is not None and isinstance(_chunk_text, str):
text = _chunk_text
- finish_reason = chunk.get("stop_reason", None)
+ finish_reason = chunk.get("stop_reason") or ""
if finish_reason is not None:
is_finished = True
returned_chunk = GenericStreamingChunk(
diff --git a/litellm/llms/anthropic/cost_calculation.py b/litellm/llms/anthropic/cost_calculation.py
index 56a83324d91..8f34eb00ce5 100644
--- a/litellm/llms/anthropic/cost_calculation.py
+++ b/litellm/llms/anthropic/cost_calculation.py
@@ -49,7 +49,7 @@ def get_cost_for_anthropic_web_search(
## Get the cost per web search request
search_context_pricing: SearchContextCostPerQuery = (
- model_info.get("search_context_cost_per_query", {}) or {}
+ model_info.get("search_context_cost_per_query") or SearchContextCostPerQuery()
)
cost_per_web_search_request = search_context_pricing.get(
"search_context_size_medium", 0.0
diff --git a/litellm/llms/anthropic/experimental_pass_through/adapters/handler.py b/litellm/llms/anthropic/experimental_pass_through/adapters/handler.py
index 5e0dfa9238a..88a63fc6f5d 100644
--- a/litellm/llms/anthropic/experimental_pass_through/adapters/handler.py
+++ b/litellm/llms/anthropic/experimental_pass_through/adapters/handler.py
@@ -133,7 +133,6 @@ class LiteLLMMessagesToCompletionTransformationHandler:
**kwargs,
) -> Union[AnthropicMessagesResponse, AsyncIterator]:
"""Handle non-Anthropic models asynchronously using the adapter"""
-
completion_kwargs = (
LiteLLMMessagesToCompletionTransformationHandler._prepare_completion_kwargs(
max_tokens=max_tokens,
diff --git a/litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py b/litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
index aa95183bb6c..47263dc1748 100644
--- a/litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
+++ b/litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py
@@ -2,7 +2,7 @@
## Translates OpenAI call to Anthropic `/v1/messages` format
import json
import traceback
-import uuid
+from litellm._uuid import uuid
from collections import deque
from typing import TYPE_CHECKING, Any, AsyncIterator, Iterator, Literal, Optional
@@ -28,10 +28,6 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
TextBlock,
)
- def __init__(self, completion_stream: Any, model: str):
- super().__init__(completion_stream)
- self.model = model
-
sent_first_chunk: bool = False
sent_content_block_start: bool = False
sent_content_block_finish: bool = False
@@ -39,6 +35,7 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
sent_last_message: bool = False
holding_chunk: Optional[Any] = None
holding_stop_reason_chunk: Optional[Any] = None
+ queued_usage_chunk: bool = False
current_content_block_index: int = 0
current_content_block_start: ContentBlockContentBlockDict = TextBlock(
type="text",
@@ -47,6 +44,10 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
pending_new_content_block: bool = False
chunk_queue: deque = deque() # Queue for buffering multiple chunks
+ def __init__(self, completion_stream: Any, model: str):
+ super().__init__(completion_stream)
+ self.model = model
+
def __next__(self):
from .transformation import LiteLLMAnthropicMessagesAdapter
@@ -217,77 +218,82 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
# Queue the merged chunk and reset
self.chunk_queue.append(merged_chunk)
+ self.queued_usage_chunk = True
self.holding_stop_reason_chunk = None
return self.chunk_queue.popleft()
# Check if this processed chunk has a stop_reason - hold it for next chunk
- if should_start_new_block and not self.sent_content_block_finish:
- # Queue the sequence: content_block_stop -> content_block_start -> current_chunk
+ if not self.queued_usage_chunk:
+ if should_start_new_block and not self.sent_content_block_finish:
+ # Queue the sequence: content_block_stop -> content_block_start -> current_chunk
- # 1. Stop current content block
- self.chunk_queue.append(
- {
- "type": "content_block_stop",
- "index": max(self.current_content_block_index - 1, 0),
- }
- )
+ # 1. Stop current content block
+ self.chunk_queue.append(
+ {
+ "type": "content_block_stop",
+ "index": max(self.current_content_block_index - 1, 0),
+ }
+ )
- # 2. Start new content block
- self.chunk_queue.append(
- {
- "type": "content_block_start",
- "index": self.current_content_block_index,
- "content_block": self.current_content_block_start,
- }
- )
+ # 2. Start new content block
+ self.chunk_queue.append(
+ {
+ "type": "content_block_start",
+ "index": self.current_content_block_index,
+ "content_block": self.current_content_block_start,
+ }
+ )
- # 3. Queue the current chunk (don't lose it!)
- self.chunk_queue.append(processed_chunk)
-
- # Reset state for new block
- self.sent_content_block_finish = False
-
- # Return the first queued item
- return self.chunk_queue.popleft()
-
- if (
- processed_chunk["type"] == "message_delta"
- and self.sent_content_block_finish is False
- ):
- # Queue both the content_block_stop and the holding chunk
- self.chunk_queue.append(
- {
- "type": "content_block_stop",
- "index": self.current_content_block_index,
- }
- )
- self.sent_content_block_finish = True
- if processed_chunk.get("delta", {}).get("stop_reason") is not None:
-
- self.holding_stop_reason_chunk = processed_chunk
- else:
+ # 3. Queue the current chunk (don't lose it!)
self.chunk_queue.append(processed_chunk)
- return self.chunk_queue.popleft()
- elif self.holding_chunk is not None:
- # Queue both chunks
- self.chunk_queue.append(self.holding_chunk)
- self.chunk_queue.append(processed_chunk)
- self.holding_chunk = None
- return self.chunk_queue.popleft()
- else:
- # Queue the current chunk
- self.chunk_queue.append(processed_chunk)
- return self.chunk_queue.popleft()
+
+ # Reset state for new block
+ self.sent_content_block_finish = False
+
+ # Return the first queued item
+ return self.chunk_queue.popleft()
+
+ if (
+ processed_chunk["type"] == "message_delta"
+ and self.sent_content_block_finish is False
+ ):
+ # Queue both the content_block_stop and the holding chunk
+ self.chunk_queue.append(
+ {
+ "type": "content_block_stop",
+ "index": self.current_content_block_index,
+ }
+ )
+ self.sent_content_block_finish = True
+ if (
+ processed_chunk.get("delta", {}).get("stop_reason")
+ is not None
+ ):
+ self.holding_stop_reason_chunk = processed_chunk
+ else:
+ self.chunk_queue.append(processed_chunk)
+ return self.chunk_queue.popleft()
+ elif self.holding_chunk is not None:
+ # Queue both chunks
+ self.chunk_queue.append(self.holding_chunk)
+ self.chunk_queue.append(processed_chunk)
+ self.holding_chunk = None
+ return self.chunk_queue.popleft()
+ else:
+ # Queue the current chunk
+ self.chunk_queue.append(processed_chunk)
+ return self.chunk_queue.popleft()
# Handle any remaining held chunks after stream ends
- if self.holding_stop_reason_chunk is not None:
- self.chunk_queue.append(self.holding_stop_reason_chunk)
- self.holding_stop_reason_chunk = None
+ if not self.queued_usage_chunk:
+ if self.holding_stop_reason_chunk is not None:
+ self.chunk_queue.append(self.holding_stop_reason_chunk)
+ self.holding_stop_reason_chunk = None
- if self.holding_chunk is not None:
- self.chunk_queue.append(self.holding_chunk)
- self.holding_chunk = None
+ if self.holding_chunk is not None:
+ self.chunk_queue.append(self.holding_chunk)
+ self.holding_chunk = None
if not self.sent_last_message:
self.sent_last_message = True
@@ -373,4 +379,11 @@ class AnthropicStreamWrapper(AdapterCompletionStreamWrapper):
self.current_content_block_start = content_block_start
return True
+ # For parallel tool calls, we'll necessarily have a new content block
+ # if we get a function name since it signals a new tool call
+ if block_type == "tool_use" and content_block_start.get("name"):
+ self.current_content_block_type = block_type
+ self.current_content_block_start = content_block_start
+ return True
+
return False
diff --git a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
index 990d613ecf0..7de2a1e1c66 100644
--- a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
+++ b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
@@ -458,7 +458,7 @@ class LiteLLMAnthropicMessagesAdapter:
Literal["text", "tool_use"],
"ContentBlockContentBlockDict",
]:
- import uuid
+ from litellm._uuid import uuid
from litellm.types.llms.anthropic import TextBlock, ToolUseBlock
@@ -489,7 +489,7 @@ class LiteLLMAnthropicMessagesAdapter:
text: str = ""
partial_json: Optional[str] = None
for choice in choices:
- if choice.delta.content is not None:
+ if choice.delta.content is not None and len(choice.delta.content) > 0:
text += choice.delta.content
elif choice.delta.tool_calls is not None:
partial_json = ""
@@ -499,7 +499,6 @@ class LiteLLMAnthropicMessagesAdapter:
and tool.function.arguments is not None
):
partial_json += tool.function.arguments
-
if partial_json is not None:
return "input_json_delta", ContentJsonBlockDelta(
type="input_json_delta", partial_json=partial_json
diff --git a/litellm/llms/azure/audio_transcriptions.py b/litellm/llms/azure/audio_transcriptions.py
index 1f09ac7574a..8519b1c35a5 100644
--- a/litellm/llms/azure/audio_transcriptions.py
+++ b/litellm/llms/azure/audio_transcriptions.py
@@ -1,4 +1,4 @@
-import uuid
+from litellm._uuid import uuid
from typing import Any, Coroutine, Optional, Union
from openai import AsyncAzureOpenAI, AzureOpenAI
diff --git a/litellm/llms/azure/azure.py b/litellm/llms/azure/azure.py
index 285f176026d..7c5b693b453 100644
--- a/litellm/llms/azure/azure.py
+++ b/litellm/llms/azure/azure.py
@@ -182,12 +182,12 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
model: str,
messages: list,
model_response: ModelResponse,
- api_key: str,
+ api_key: Optional[str],
api_base: str,
api_version: str,
api_type: str,
- azure_ad_token: str,
- azure_ad_token_provider: Callable,
+ azure_ad_token: Optional[str],
+ azure_ad_token_provider: Optional[Callable],
dynamic_params: bool,
print_verbose: Callable,
timeout: Union[float, httpx.Timeout],
@@ -230,6 +230,14 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
)
data = {"model": None, "messages": messages, **optional_params}
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ data = litellm.AzureOpenAIGPT5Config().transform_request(
+ model=model,
+ messages=messages,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ headers=headers or {},
+ )
else:
data = litellm.AzureOpenAIConfig().transform_request(
model=model,
@@ -364,7 +372,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
async def acompletion(
self,
- api_key: str,
+ api_key: Optional[str],
api_version: str,
model: str,
api_base: str,
@@ -469,7 +477,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
self,
logging_obj,
api_base: str,
- api_key: str,
+ api_key: Optional[str],
api_version: str,
dynamic_params: bool,
data: dict,
@@ -547,7 +555,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
self,
logging_obj: LiteLLMLoggingObj,
api_base: str,
- api_key: str,
+ api_key: Optional[str],
api_version: str,
dynamic_params: bool,
data: dict,
@@ -1109,6 +1117,14 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
status_code=422, message="max retries must be an int"
)
+ if api_key is None and azure_ad_token_provider is not None:
+ azure_ad_token = azure_ad_token_provider()
+ if azure_ad_token:
+ headers.pop(
+ "api-key", None
+ )
+ headers["Authorization"] = f"Bearer {azure_ad_token}"
+
# init AzureOpenAI Client
azure_client_params: Dict[str, Any] = self.initialize_azure_sdk_client(
litellm_params=litellm_params or {},
diff --git a/litellm/llms/azure/chat/gpt_5_transformation.py b/litellm/llms/azure/chat/gpt_5_transformation.py
new file mode 100644
index 00000000000..d563a2889ca
--- /dev/null
+++ b/litellm/llms/azure/chat/gpt_5_transformation.py
@@ -0,0 +1,59 @@
+"""Support for Azure OpenAI gpt-5 model family."""
+
+from typing import List
+
+from litellm.llms.openai.chat.gpt_5_transformation import OpenAIGPT5Config
+from litellm.types.llms.openai import AllMessageValues
+
+from .gpt_transformation import AzureOpenAIConfig
+
+
+class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
+ """Azure specific handling for gpt-5 models."""
+
+ GPT5_SERIES_ROUTE = "gpt5_series/"
+
+ @classmethod
+ def is_model_gpt_5_model(cls, model: str) -> bool:
+ """Check if the Azure model string refers to a gpt-5 variant.
+
+ Accepts both explicit gpt-5 model names and the ``gpt5_series/`` prefix
+ used for manual routing.
+ """
+ return "gpt-5" in model or "gpt5_series" in model
+
+ def get_supported_openai_params(self, model: str) -> List[str]:
+ return OpenAIGPT5Config.get_supported_openai_params(self, model=model)
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ api_version: str = "",
+ ) -> dict:
+ return OpenAIGPT5Config.map_openai_params(
+ self,
+ non_default_params=non_default_params,
+ optional_params=optional_params,
+ model=model,
+ drop_params=drop_params,
+ )
+
+ def transform_request(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ headers: dict,
+ ) -> dict:
+ model = model.replace(self.GPT5_SERIES_ROUTE, "")
+ return super().transform_request(
+ model=model,
+ messages=messages,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ headers=headers,
+ )
diff --git a/litellm/llms/azure/chat/o_series_handler.py b/litellm/llms/azure/chat/o_series_handler.py
index 2f3e9e63996..d0f5153b0eb 100644
--- a/litellm/llms/azure/chat/o_series_handler.py
+++ b/litellm/llms/azure/chat/o_series_handler.py
@@ -4,7 +4,7 @@ Handler file for calls to Azure OpenAI's o1/o3 family of models
Written separately to handle faking streaming for o1 and o3 models.
"""
-from typing import Any, Callable, Optional, Union
+from typing import TYPE_CHECKING, Any, Callable, Optional, Union
import httpx
@@ -13,6 +13,9 @@ from litellm.types.utils import ModelResponse
from ...openai.openai import OpenAIChatCompletion
from ..common_utils import BaseAzureLLM
+if TYPE_CHECKING:
+ from aiohttp import ClientSession
+
class AzureOpenAIO1ChatCompletion(BaseAzureLLM, OpenAIChatCompletion):
def completion(
@@ -38,6 +41,7 @@ class AzureOpenAIO1ChatCompletion(BaseAzureLLM, OpenAIChatCompletion):
organization: Optional[str] = None,
custom_llm_provider: Optional[str] = None,
drop_params: Optional[bool] = None,
+ shared_session: Optional["ClientSession"] = None,
):
client = self.get_azure_openai_client(
litellm_params=litellm_params,
@@ -69,4 +73,5 @@ class AzureOpenAIO1ChatCompletion(BaseAzureLLM, OpenAIChatCompletion):
organization=organization,
custom_llm_provider=custom_llm_provider,
drop_params=drop_params,
+ shared_session=shared_session,
)
diff --git a/litellm/llms/azure/common_utils.py b/litellm/llms/azure/common_utils.py
index 0ed4627908d..dfe662cc165 100644
--- a/litellm/llms/azure/common_utils.py
+++ b/litellm/llms/azure/common_utils.py
@@ -162,8 +162,8 @@ def get_azure_ad_token_from_username_password(
def get_azure_ad_token_from_oidc(
azure_ad_token: str,
- azure_client_id: Optional[str],
- azure_tenant_id: Optional[str],
+ azure_client_id: Optional[str] = None,
+ azure_tenant_id: Optional[str] = None,
scope: Optional[str] = None,
) -> str:
"""
@@ -365,14 +365,21 @@ def get_azure_ad_token(
azure_ad_token_provider = get_azure_ad_token_provider(azure_scope=scope)
except ValueError:
verbose_logger.debug("Azure AD Token Provider could not be used.")
-
+ except Exception as e:
+ verbose_logger.error(
+ f"Error calling Azure AD token provider: {str(e)}. Follow docs - https://docs.litellm.ai/docs/providers/azure/#azure-ad-token-refresh---defaultazurecredential"
+ )
+ raise e
+
#########################################################
# If litellm.enable_azure_ad_token_refresh is True and no other token provider is available,
# try to get DefaultAzureCredential provider
#########################################################
if azure_ad_token_provider is None and azure_ad_token is None:
- azure_ad_token_provider = BaseAzureLLM._try_get_default_azure_credential_provider(
- scope=scope,
+ azure_ad_token_provider = (
+ BaseAzureLLM._try_get_default_azure_credential_provider(
+ scope=scope,
+ )
)
# Execute the token provider to get the token if available
@@ -403,27 +410,27 @@ class BaseAzureLLM(BaseOpenAILLM):
) -> Optional[Callable[[], str]]:
"""
Try to get DefaultAzureCredential provider
-
+
Args:
scope: Azure scope for the token
-
+
Returns:
Token provider callable if DefaultAzureCredential is enabled and available, None otherwise
"""
from litellm.types.secret_managers.get_azure_ad_token_provider import (
AzureCredentialType,
)
-
- verbose_logger.debug(
- "Attempting to use DefaultAzureCredential for Azure Auth"
- )
-
+
+ verbose_logger.debug("Attempting to use DefaultAzureCredential for Azure Auth")
+
try:
azure_ad_token_provider = get_azure_ad_token_provider(
azure_scope=scope,
azure_credential=AzureCredentialType.DefaultAzureCredential,
)
- verbose_logger.debug("Successfully obtained Azure AD token provider using DefaultAzureCredential")
+ verbose_logger.debug(
+ "Successfully obtained Azure AD token provider using DefaultAzureCredential"
+ )
return azure_ad_token_provider
except Exception as e:
verbose_logger.debug(f"DefaultAzureCredential failed: {str(e)}")
@@ -559,7 +566,9 @@ class BaseAzureLLM(BaseOpenAILLM):
"Using Azure AD token provider based on Service Principal with Secret workflow for Azure Auth"
)
try:
- azure_ad_token_provider = get_azure_ad_token_provider(azure_scope=scope)
+ azure_ad_token_provider = get_azure_ad_token_provider(
+ azure_scope=scope,
+ )
except ValueError:
verbose_logger.debug("Azure AD Token Provider could not be used.")
if api_version is None:
@@ -656,12 +665,17 @@ class BaseAzureLLM(BaseOpenAILLM):
else:
client = AzureOpenAI(**azure_client_params) # type: ignore
return client
-
+
@staticmethod
def _base_validate_azure_environment(
- headers: dict, litellm_params: Optional[GenericLiteLLMParams]
+ headers: dict, litellm_params: Optional[GenericLiteLLMParams]
) -> dict:
litellm_params = litellm_params or GenericLiteLLMParams()
+
+ # Check if api-key is already in headers; if so, use it
+ if "api-key" in headers:
+ return headers
+
api_key = (
litellm_params.api_key
or litellm.api_key
@@ -681,13 +695,24 @@ class BaseAzureLLM(BaseOpenAILLM):
headers["Authorization"] = f"Bearer {azure_ad_token}"
return headers
-
+
@staticmethod
def _get_base_azure_url(
api_base: Optional[str],
litellm_params: Optional[Union[GenericLiteLLMParams, Dict[str, Any]]],
- route: Literal["/openai/responses", "/openai/vector_stores"]
+ route: Union[Literal["/openai/responses", "/openai/vector_stores"], str],
+ default_api_version: Optional[Union[str, Literal["latest", "preview"]]] = None,
) -> str:
+ """
+ Get the base Azure URL for the given route and API version.
+
+ Args:
+ api_base: The base URL of the Azure API.
+ litellm_params: The litellm parameters.
+ route: The route to the API.
+ default_api_version: The default API version to use if no api_version is provided. If 'latest', it will use `openai/v1/...` route.
+ """
+
api_base = api_base or litellm.api_base or get_secret_str("AZURE_API_BASE")
if api_base is None:
raise ValueError(
@@ -697,7 +722,10 @@ class BaseAzureLLM(BaseOpenAILLM):
# Extract api_version or use default
litellm_params = litellm_params or {}
- api_version = cast(Optional[str], litellm_params.get("api_version"))
+ api_version = (
+ cast(Optional[str], litellm_params.get("api_version"))
+ or default_api_version
+ )
# Create a new dictionary with existing params
query_params = dict(original_url.params)
@@ -705,27 +733,28 @@ class BaseAzureLLM(BaseOpenAILLM):
# Add api_version if needed
if "api-version" not in query_params and api_version:
query_params["api-version"] = api_version
-
+
# Add the path to the base URL
if route not in api_base:
- new_url = _add_path_to_api_base(
- api_base=api_base, ending_path=route
- )
+ new_url = _add_path_to_api_base(api_base=api_base, ending_path=route)
else:
new_url = api_base
-
+
if BaseAzureLLM._is_azure_v1_api_version(api_version):
# ensure the request go to /openai/v1 and not just /openai
if "/openai/v1" not in new_url:
parsed_url = httpx.URL(new_url)
- new_url = str(parsed_url.copy_with(path=parsed_url.path.replace("/openai", "/openai/v1")))
-
+ new_url = str(
+ parsed_url.copy_with(
+ path=parsed_url.path.replace("/openai", "/openai/v1")
+ )
+ )
# Use the new query_params dictionary
final_url = httpx.URL(new_url).copy_with(params=query_params)
return str(final_url)
-
+
@staticmethod
def _is_azure_v1_api_version(api_version: Optional[str]) -> bool:
if api_version is None:
diff --git a/litellm/llms/azure/completion/handler.py b/litellm/llms/azure/completion/handler.py
index a44f9045712..05d5e2f6c68 100644
--- a/litellm/llms/azure/completion/handler.py
+++ b/litellm/llms/azure/completion/handler.py
@@ -30,11 +30,11 @@ class AzureTextCompletion(BaseAzureLLM):
model: str,
messages: list,
model_response: ModelResponse,
- api_key: str,
+ api_key: Optional[str],
api_base: str,
api_version: str,
api_type: str,
- azure_ad_token: str,
+ azure_ad_token: Optional[str],
azure_ad_token_provider: Optional[Callable],
print_verbose: Callable,
timeout,
@@ -59,7 +59,7 @@ class AzureTextCompletion(BaseAzureLLM):
### CHECK IF CLOUDFLARE AI GATEWAY ###
### if so - set the model as part of the base url
- if "gateway.ai.cloudflare.com" in api_base:
+ if api_base is not None and "gateway.ai.cloudflare.com" in api_base:
## build base url - assume api base includes resource name
client = self._init_azure_client_for_cloudflare_ai_gateway(
api_key=api_key,
@@ -196,7 +196,7 @@ class AzureTextCompletion(BaseAzureLLM):
async def acompletion(
self,
- api_key: str,
+ api_key: Optional[str],
api_version: str,
model: str,
api_base: str,
@@ -263,7 +263,7 @@ class AzureTextCompletion(BaseAzureLLM):
self,
logging_obj,
api_base: str,
- api_key: str,
+ api_key: Optional[str],
api_version: str,
data: dict,
model: str,
@@ -320,7 +320,7 @@ class AzureTextCompletion(BaseAzureLLM):
self,
logging_obj,
api_base: str,
- api_key: str,
+ api_key: Optional[str],
api_version: str,
data: dict,
model: str,
diff --git a/litellm/llms/azure/passthrough/transformation.py b/litellm/llms/azure/passthrough/transformation.py
new file mode 100644
index 00000000000..4e9de4b314f
--- /dev/null
+++ b/litellm/llms/azure/passthrough/transformation.py
@@ -0,0 +1,85 @@
+from typing import TYPE_CHECKING, List, Optional, Tuple
+
+import httpx
+
+from litellm.llms.azure.common_utils import BaseAzureLLM
+from litellm.llms.base_llm.passthrough.transformation import BasePassthroughConfig
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.openai import AllMessageValues
+from litellm.types.router import GenericLiteLLMParams
+
+if TYPE_CHECKING:
+ from httpx import URL
+
+
+class AzurePassthroughConfig(BasePassthroughConfig):
+ def is_streaming_request(self, endpoint: str, request_data: dict) -> bool:
+ return "stream" in request_data
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ endpoint: str,
+ request_query_params: Optional[dict],
+ litellm_params: dict,
+ ) -> Tuple["URL", str]:
+ base_target_url = self.get_api_base(api_base)
+
+ if base_target_url is None:
+ raise Exception("Azure api base not found")
+
+ litellm_metadata = litellm_params.get("litellm_metadata") or {}
+ model_group = litellm_metadata.get("model_group")
+ if model_group and model_group in endpoint:
+ endpoint = endpoint.replace(model_group, model)
+
+ complete_url = BaseAzureLLM._get_base_azure_url(
+ api_base=base_target_url,
+ litellm_params=litellm_params,
+ route=endpoint,
+ default_api_version=litellm_params.get("api_version"),
+ )
+ return (
+ httpx.URL(complete_url),
+ base_target_url,
+ )
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ return BaseAzureLLM._base_validate_azure_environment(
+ headers=headers,
+ litellm_params=GenericLiteLLMParams(
+ **{**litellm_params, "api_key": api_key}
+ ),
+ )
+
+ @staticmethod
+ def get_api_base(
+ api_base: Optional[str] = None,
+ ) -> Optional[str]:
+ return api_base or get_secret_str("AZURE_API_BASE")
+
+ @staticmethod
+ def get_api_key(
+ api_key: Optional[str] = None,
+ ) -> Optional[str]:
+ return api_key or get_secret_str("AZURE_API_KEY")
+
+ @staticmethod
+ def get_base_model(model: str) -> Optional[str]:
+ return model
+
+ def get_models(
+ self, api_key: Optional[str] = None, api_base: Optional[str] = None
+ ) -> List[str]:
+ return super().get_models(api_key, api_base)
diff --git a/litellm/llms/azure/responses/o_series_transformation.py b/litellm/llms/azure/responses/o_series_transformation.py
new file mode 100644
index 00000000000..a0b2ef16300
--- /dev/null
+++ b/litellm/llms/azure/responses/o_series_transformation.py
@@ -0,0 +1,93 @@
+"""
+Support for Azure OpenAI O-series models (o1, o3, etc.) in Responses API
+
+https://platform.openai.com/docs/guides/reasoning
+
+Translations handled by LiteLLM:
+- temperature => drop param (if user opts in to dropping param)
+- Other parameters follow base Azure OpenAI Responses API behavior
+"""
+
+from typing import TYPE_CHECKING, Any, Dict
+
+from litellm._logging import verbose_logger
+from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams
+from litellm.utils import supports_reasoning
+
+from .transformation import AzureOpenAIResponsesAPIConfig
+
+if TYPE_CHECKING:
+ from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
+
+ LiteLLMLoggingObj = _LiteLLMLoggingObj
+else:
+ LiteLLMLoggingObj = Any
+
+
+class AzureOpenAIOSeriesResponsesAPIConfig(AzureOpenAIResponsesAPIConfig):
+ """
+ Configuration for Azure OpenAI O-series models in Responses API.
+
+ O-series models (o1, o3, etc.) do not support the temperature parameter
+ in the responses API, so we need to drop it when drop_params is enabled.
+ """
+
+ def get_supported_openai_params(self, model: str) -> list:
+ """
+ Get supported parameters for Azure OpenAI O-series Responses API.
+
+ O-series models don't support temperature parameter in responses API.
+ """
+ # Get the base Azure supported params
+ base_supported_params = super().get_supported_openai_params(model)
+
+ # O-series models don't support temperature parameter in responses API
+ o_series_unsupported_params = ["temperature"]
+
+ # Filter out unsupported parameters for O-series models
+ o_series_supported_params = [
+ param for param in base_supported_params
+ if param not in o_series_unsupported_params
+ ]
+
+ return o_series_supported_params
+
+ def map_openai_params(
+ self,
+ response_api_optional_params: ResponsesAPIOptionalRequestParams,
+ model: str,
+ drop_params: bool,
+ ) -> Dict:
+ """
+ Map OpenAI parameters for Azure OpenAI O-series Responses API.
+
+ Drops temperature parameter if drop_params is True since O-series models
+ don't support temperature in the responses API.
+ """
+ mapped_params = dict(response_api_optional_params)
+
+ # If drop_params is enabled, remove temperature parameter for O-series models
+ if drop_params and "temperature" in mapped_params:
+ verbose_logger.debug(
+ f"Dropping unsupported parameter 'temperature' for Azure OpenAI O-series responses API model {model}"
+ )
+ mapped_params.pop("temperature", None)
+
+ return mapped_params
+
+ def is_o_series_model(self, model: str) -> bool:
+ """
+ Check if the model is an O-series model.
+
+ Args:
+ model: The model name to check
+
+ Returns:
+ True if it's an O-series model, False otherwise
+ """
+ # Check if model name contains o_series or if it's a known O-series model
+ if "o_series" in model.lower():
+ return True
+
+ # Check if the model supports reasoning (which is O-series specific)
+ return supports_reasoning(model)
\ No newline at end of file
diff --git a/litellm/llms/azure/responses/transformation.py b/litellm/llms/azure/responses/transformation.py
index e3d37c8a15a..1516ed089ee 100644
--- a/litellm/llms/azure/responses/transformation.py
+++ b/litellm/llms/azure/responses/transformation.py
@@ -1,4 +1,7 @@
-from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Tuple
+from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple, Union
+
+import httpx
+from openai.types.responses import ResponseReasoningItem
from litellm._logging import verbose_logger
from litellm.llms.azure.common_utils import BaseAzureLLM
@@ -6,6 +9,7 @@ from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfi
from litellm.types.llms.openai import *
from litellm.types.responses.main import *
from litellm.types.router import GenericLiteLLMParams
+from litellm.types.utils import LlmProviders
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@@ -16,6 +20,10 @@ else:
class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
+ @property
+ def custom_llm_provider(self) -> LlmProviders:
+ return LlmProviders.AZURE
+
def validate_environment(
self, headers: dict, model: str, litellm_params: Optional[GenericLiteLLMParams]
) -> dict:
@@ -31,6 +39,74 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
model = model.replace("o_series/", "")
return model
+ def _handle_reasoning_item(self, item: Dict[str, Any]) -> Dict[str, Any]:
+ """
+ Handle reasoning items to filter out the status field.
+ Issue: https://github.com/BerriAI/litellm/issues/13484
+
+ Azure OpenAI API does not accept 'status' field in reasoning input items.
+ """
+ if item.get("type") == "reasoning":
+ try:
+ # Ensure required fields are present for ResponseReasoningItem
+ item_data = dict(item)
+ if "id" not in item_data:
+ item_data["id"] = f"rs_{hash(str(item_data))}"
+ if "summary" not in item_data:
+ item_data["summary"] = (
+ item_data.get("reasoning_content", "")[:100] + "..."
+ if len(item_data.get("reasoning_content", "")) > 100
+ else item_data.get("reasoning_content", "")
+ )
+
+ # Create ResponseReasoningItem object from the item data
+ reasoning_item = ResponseReasoningItem(**item_data)
+
+ # Convert back to dict with exclude_none=True to exclude None fields
+ dict_reasoning_item = reasoning_item.model_dump(exclude_none=True)
+ dict_reasoning_item.pop("status", None)
+
+ return dict_reasoning_item
+ except Exception as e:
+ verbose_logger.debug(
+ f"Failed to create ResponseReasoningItem, falling back to manual filtering: {e}"
+ )
+ # Fallback: manually filter out known None fields
+ filtered_item = {
+ k: v
+ for k, v in item.items()
+ if v is not None
+ or k not in {"status", "content", "encrypted_content"}
+ }
+ return filtered_item
+ return item
+
+ def _validate_input_param(
+ self, input: Union[str, ResponseInputParam]
+ ) -> Union[str, ResponseInputParam]:
+ """
+ Override parent method to also filter out 'status' field from message items.
+ Azure OpenAI API does not accept 'status' field in input messages.
+ """
+ from typing import cast
+
+ # First call parent's validation
+ validated_input = super()._validate_input_param(input)
+
+ # Then filter out status from message items
+ if isinstance(validated_input, list):
+ filtered_input: List[Any] = []
+ for item in validated_input:
+ if isinstance(item, dict) and item.get("type") == "message":
+ # Filter out status field from message items
+ filtered_item = {k: v for k, v in item.items() if k != "status"}
+ filtered_input.append(filtered_item)
+ else:
+ filtered_input.append(item)
+ return cast(ResponseInputParam, filtered_input)
+
+ return validated_input
+
def transform_responses_api_request(
self,
model: str,
@@ -41,12 +117,13 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
) -> Dict:
"""No transform applied since inputs are in OpenAI spec already"""
stripped_model_name = self.get_stripped_model_name(model)
- return dict(
- ResponsesAPIRequestParams(
- model=stripped_model_name,
- input=input,
- **response_api_optional_request_params,
- )
+
+ return super().transform_responses_api_request(
+ model=stripped_model_name,
+ input=input,
+ response_api_optional_request_params=response_api_optional_request_params,
+ litellm_params=litellm_params,
+ headers=headers,
)
def get_complete_url(
@@ -70,8 +147,13 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
- A complete URL string, e.g.,
"https://litellm8397336933.openai.azure.com/openai/responses?api-version=2024-05-01-preview"
"""
+ from litellm.constants import AZURE_DEFAULT_RESPONSES_API_VERSION
+
return BaseAzureLLM._get_base_azure_url(
- api_base=api_base, litellm_params=litellm_params, route="/openai/responses"
+ api_base=api_base,
+ litellm_params=litellm_params,
+ route="/openai/responses",
+ default_api_version=AZURE_DEFAULT_RESPONSES_API_VERSION,
)
#########################################################
@@ -184,3 +266,66 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
params["order"] = order
verbose_logger.debug(f"list input items url={url}")
return url, params
+
+ #########################################################
+ ########## CANCEL RESPONSE API TRANSFORMATION ##########
+ #########################################################
+ def transform_cancel_response_api_request(
+ self,
+ response_id: str,
+ api_base: str,
+ litellm_params: GenericLiteLLMParams,
+ headers: dict,
+ ) -> Tuple[str, Dict]:
+ """
+ Transform the cancel response API request into a URL and data
+
+ Azure OpenAI API expects the following request:
+ - POST /openai/responses/{response_id}/cancel?api-version=xxx
+
+ This function handles URLs with query parameters by inserting the response_id
+ at the correct location (before any query parameters).
+ """
+ from urllib.parse import urlparse, urlunparse
+
+ # Parse the URL to separate its components
+ parsed_url = urlparse(api_base)
+
+ # Insert the response_id and /cancel at the end of the path component
+ # Remove trailing slash if present to avoid double slashes
+ path = parsed_url.path.rstrip("/")
+ new_path = f"{path}/{response_id}/cancel"
+
+ # Reconstruct the URL with all original components but with the modified path
+ cancel_url = urlunparse(
+ (
+ parsed_url.scheme, # http, https
+ parsed_url.netloc, # domain name, port
+ new_path, # path with response_id and /cancel added
+ parsed_url.params, # parameters
+ parsed_url.query, # query string
+ parsed_url.fragment, # fragment
+ )
+ )
+
+ data: Dict = {}
+ verbose_logger.debug(f"cancel response url={cancel_url}")
+ return cancel_url, data
+
+ def transform_cancel_response_api_response(
+ self,
+ raw_response: httpx.Response,
+ logging_obj: LiteLLMLoggingObj,
+ ) -> ResponsesAPIResponse:
+ """
+ Transform the cancel response API response into a ResponsesAPIResponse
+ """
+ try:
+ raw_response_json = raw_response.json()
+ except Exception:
+ from litellm.llms.azure.chat.gpt_transformation import AzureOpenAIError
+
+ raise AzureOpenAIError(
+ message=raw_response.text, status_code=raw_response.status_code
+ )
+ return ResponsesAPIResponse(**raw_response_json)
diff --git a/litellm/llms/azure_ai/chat/transformation.py b/litellm/llms/azure_ai/chat/transformation.py
index 7eb7b767d04..04d2b3a2769 100644
--- a/litellm/llms/azure_ai/chat/transformation.py
+++ b/litellm/llms/azure_ai/chat/transformation.py
@@ -14,6 +14,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
from litellm.llms.base_llm.chat.transformation import LiteLLMLoggingObj
from litellm.llms.openai.common_utils import drop_params_from_unprocessable_entity_error
from litellm.llms.openai.openai import OpenAIConfig
+from litellm.llms.xai.chat.transformation import XAIChatConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse, ProviderField
@@ -35,9 +36,24 @@ class AzureAIStudioConfig(OpenAIConfig):
for param in supported_params:
if param != "tool_choice":
filtered_supported_params.append(param)
- return filtered_supported_params
+ supported_params = filtered_supported_params
+
+ # Filter out unsupported parameters for specific models
+ if not self._supports_stop_reason(model):
+ supported_params = [param for param in supported_params if param != "stop"]
+
return supported_params
+ def _supports_stop_reason(self, model: str) -> bool:
+ """
+ Check if the model supports stop tokens.
+ """
+ if "grok" in model:
+ # Reuse Xai method for Grok model
+ xai_config = XAIChatConfig()
+ return xai_config._supports_stop_reason(model)
+ return True
+
def validate_environment(
self,
headers: dict,
@@ -53,9 +69,7 @@ class AzureAIStudioConfig(OpenAIConfig):
else:
headers["Authorization"] = f"Bearer {api_key}"
- headers["Content-Type"] = (
- "application/json" # tell Azure AI Studio to expect JSON
- )
+ headers["Content-Type"] = "application/json" # tell Azure AI Studio to expect JSON
return headers
@@ -65,10 +79,7 @@ class AzureAIStudioConfig(OpenAIConfig):
"""
parsed_url = urlparse(api_base)
host = parsed_url.hostname
- if host and (
- host.endswith(".services.ai.azure.com")
- or host.endswith(".openai.azure.com")
- ):
+ if host and (host.endswith(".services.ai.azure.com") or host.endswith(".openai.azure.com")):
return True
return False
@@ -115,13 +126,9 @@ class AzureAIStudioConfig(OpenAIConfig):
# Add the path to the base URL
if "services.ai.azure.com" in api_base:
- new_url = _add_path_to_api_base(
- api_base=api_base, ending_path="/models/chat/completions"
- )
+ new_url = _add_path_to_api_base(api_base=api_base, ending_path="/models/chat/completions")
else:
- new_url = _add_path_to_api_base(
- api_base=api_base, ending_path="/chat/completions"
- )
+ new_url = _add_path_to_api_base(api_base=api_base, ending_path="/chat/completions")
# Use the new query_params dictionary
final_url = httpx.URL(new_url).copy_with(params=query_params)
@@ -191,11 +198,7 @@ class AzureAIStudioConfig(OpenAIConfig):
dynamic_api_key = api_key or get_secret_str("AZURE_AI_API_KEY")
if self._is_azure_openai_model(model=model, api_base=api_base):
- verbose_logger.debug(
- "Model={} is Azure OpenAI model. Setting custom_llm_provider='azure'.".format(
- model
- )
- )
+ verbose_logger.debug("Model={} is Azure OpenAI model. Setting custom_llm_provider='azure'.".format(model))
custom_llm_provider = "azure"
return api_base, dynamic_api_key, custom_llm_provider
@@ -211,9 +214,7 @@ class AzureAIStudioConfig(OpenAIConfig):
if extra_body and isinstance(extra_body, dict):
optional_params.update(extra_body)
optional_params.pop("max_retries", None)
- return super().transform_request(
- model, messages, optional_params, litellm_params, headers
- )
+ return super().transform_request(model, messages, optional_params, litellm_params, headers)
def transform_response(
self,
@@ -252,47 +253,30 @@ class AzureAIStudioConfig(OpenAIConfig):
if should_drop_params and "Extra inputs are not permitted" in error_text:
return True
- elif (
- "unknown field: parameter index is not a valid field" in error_text
- ): # remove index from tool calls
+ elif "unknown field: parameter index is not a valid field" in error_text: # remove index from tool calls
return True
elif (
- AzureFoundryErrorStrings.SET_EXTRA_PARAMETERS_TO_PASS_THROUGH.value
- in error_text
+ AzureFoundryErrorStrings.SET_EXTRA_PARAMETERS_TO_PASS_THROUGH.value in error_text
): # remove extra-parameters from tool calls
return True
- return super().should_retry_llm_api_inside_llm_translation_on_http_error(
- e=e, litellm_params=litellm_params
- )
+ return super().should_retry_llm_api_inside_llm_translation_on_http_error(e=e, litellm_params=litellm_params)
@property
def max_retry_on_unprocessable_entity_error(self) -> int:
return 2
- def transform_request_on_unprocessable_entity_error(
- self, e: httpx.HTTPStatusError, request_data: dict
- ) -> dict:
+ def transform_request_on_unprocessable_entity_error(self, e: httpx.HTTPStatusError, request_data: dict) -> dict:
_messages = cast(Optional[List[AllMessageValues]], request_data.get("messages"))
- if (
- "unknown field: parameter index is not a valid field" in e.response.text
- and _messages is not None
- ):
+ if "unknown field: parameter index is not a valid field" in e.response.text and _messages is not None:
litellm.remove_index_from_tool_calls(
messages=_messages,
)
- elif (
- AzureFoundryErrorStrings.SET_EXTRA_PARAMETERS_TO_PASS_THROUGH.value
- in e.response.text
- ):
- request_data = self._drop_extra_params_from_request_data(
- request_data, e.response.text
- )
+ elif AzureFoundryErrorStrings.SET_EXTRA_PARAMETERS_TO_PASS_THROUGH.value in e.response.text:
+ request_data = self._drop_extra_params_from_request_data(request_data, e.response.text)
data = drop_params_from_unprocessable_entity_error(e=e, data=request_data)
return data
- def _drop_extra_params_from_request_data(
- self, request_data: dict, error_text: str
- ) -> dict:
+ def _drop_extra_params_from_request_data(self, request_data: dict, error_text: str) -> dict:
params_to_drop = self._extract_params_to_drop_from_error_text(error_text)
if params_to_drop:
for param in params_to_drop:
@@ -300,9 +284,7 @@ class AzureAIStudioConfig(OpenAIConfig):
request_data.pop(param, None)
return request_data
- def _extract_params_to_drop_from_error_text(
- self, error_text: str
- ) -> Optional[List[str]]:
+ def _extract_params_to_drop_from_error_text(self, error_text: str) -> Optional[List[str]]:
"""
Error text looks like this"
"Extra parameters ['stream_options', 'extra-parameters'] are not allowed when extra-parameters is not set or set to be 'error'.
diff --git a/litellm/llms/azure_ai/common_utils.py b/litellm/llms/azure_ai/common_utils.py
new file mode 100644
index 00000000000..dcc9335e42d
--- /dev/null
+++ b/litellm/llms/azure_ai/common_utils.py
@@ -0,0 +1,56 @@
+from typing import List, Optional
+
+import litellm
+from litellm.llms.base_llm.base_utils import BaseLLMModelInfo
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.openai import AllMessageValues
+
+
+class AzureFoundryModelInfo(BaseLLMModelInfo):
+ @staticmethod
+ def get_api_base(api_base: Optional[str] = None) -> Optional[str]:
+ return (
+ api_base
+ or litellm.api_base
+ or get_secret_str("AZURE_AI_API_BASE")
+ )
+
+ @staticmethod
+ def get_api_key(api_key: Optional[str] = None) -> Optional[str]:
+ return (
+ api_key
+ or litellm.api_key
+ or litellm.openai_key
+ or get_secret_str("AZURE_AI_API_KEY")
+ )
+
+ @property
+ def api_version(self, api_version: Optional[str] = None) -> Optional[str]:
+ api_version = (
+ api_version
+ or litellm.api_version
+ or get_secret_str("AZURE_API_VERSION")
+ )
+ return api_version
+
+ #########################################################
+ # Not implemented methods
+ #########################################################
+
+
+ @staticmethod
+ def get_base_model(model: str) -> Optional[str]:
+ raise NotImplementedError("Azure Foundry does not support base model")
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ """Azure Foundry sends api key in query params"""
+ raise NotImplementedError("Azure Foundry does not support environment validation")
diff --git a/litellm/llms/azure_ai/embed/handler.py b/litellm/llms/azure_ai/embed/handler.py
index da39c5f3b89..13b8cc4cf29 100644
--- a/litellm/llms/azure_ai/embed/handler.py
+++ b/litellm/llms/azure_ai/embed/handler.py
@@ -210,6 +210,7 @@ class AzureAIEmbedding(OpenAIChatCompletion):
client=None,
aembedding=None,
max_retries: Optional[int] = None,
+ shared_session=None,
) -> EmbeddingResponse:
"""
- Separate image url from text
@@ -275,6 +276,7 @@ class AzureAIEmbedding(OpenAIChatCompletion):
else None
),
aembedding=aembedding,
+ shared_session=shared_session,
)
text_embedding_responses = response.data
diff --git a/litellm/llms/azure_ai/image_edit/__init__.py b/litellm/llms/azure_ai/image_edit/__init__.py
new file mode 100644
index 00000000000..e0e57bec403
--- /dev/null
+++ b/litellm/llms/azure_ai/image_edit/__init__.py
@@ -0,0 +1,15 @@
+from litellm.llms.base_llm.image_edit.transformation import BaseImageEditConfig
+
+from .transformation import AzureFoundryFluxImageEditConfig
+
+__all__ = ["AzureFoundryFluxImageEditConfig"]
+
+
+def get_azure_ai_image_edit_config(model: str) -> BaseImageEditConfig:
+ model = model.lower()
+ model = model.replace("-", "")
+ model = model.replace("_", "")
+ if model == "" or "flux" in model: # empty model is flux
+ return AzureFoundryFluxImageEditConfig()
+ else:
+ raise ValueError(f"Model {model} is not supported for Azure AI image editing.")
diff --git a/litellm/llms/azure_ai/image_edit/transformation.py b/litellm/llms/azure_ai/image_edit/transformation.py
new file mode 100644
index 00000000000..47f612912ce
--- /dev/null
+++ b/litellm/llms/azure_ai/image_edit/transformation.py
@@ -0,0 +1,99 @@
+from typing import Optional
+
+import httpx
+
+import litellm
+from litellm.llms.azure_ai.common_utils import AzureFoundryModelInfo
+from litellm.llms.openai.image_edit.transformation import OpenAIImageEditConfig
+from litellm.secret_managers.main import get_secret_str
+from litellm.utils import _add_path_to_api_base
+
+
+class AzureFoundryFluxImageEditConfig(OpenAIImageEditConfig):
+ """
+ Azure AI Foundry FLUX image edit config
+
+ Supports FLUX models including FLUX-1-kontext-pro for image editing.
+
+ Azure AI Foundry FLUX models handle image editing through the /images/edits endpoint,
+ same as standard Azure OpenAI models. The request format uses multipart/form-data
+ with image files and prompt.
+ """
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ api_key: Optional[str] = None,
+ ) -> dict:
+ """
+ Validate Azure AI Foundry environment and set up authentication
+ Uses Api-Key header format
+ """
+ api_key = AzureFoundryModelInfo.get_api_key(api_key)
+
+ if not api_key:
+ raise ValueError(
+ f"Azure AI API key is required for model {model}. Set AZURE_AI_API_KEY environment variable or pass api_key parameter."
+ )
+
+ headers.update(
+ {
+ "Api-Key": api_key, # Azure AI Foundry uses Api-Key header format
+ }
+ )
+ return headers
+
+ def get_complete_url(
+ self,
+ model: str,
+ api_base: Optional[str],
+ litellm_params: dict,
+ ) -> str:
+ """
+ Constructs a complete URL for Azure AI Foundry image edits API request.
+
+ Azure AI Foundry FLUX models handle image editing through the /images/edits
+ endpoint.
+
+ Args:
+ - model: Model name (deployment name for Azure AI Foundry)
+ - api_base: Base URL for Azure AI endpoint
+ - litellm_params: Additional parameters including api_version
+
+ Returns:
+ - Complete URL for the image edits endpoint
+ """
+ api_base = AzureFoundryModelInfo.get_api_base(api_base)
+
+ if api_base is None:
+ raise ValueError(
+ "Azure AI API base is required. Set AZURE_AI_API_BASE environment variable or pass api_base parameter."
+ )
+
+ api_version = (litellm_params.get("api_version") or litellm.api_version
+ or get_secret_str("AZURE_AI_API_VERSION")
+ )
+ if api_version is None:
+ # API version is mandatory for Azure AI Foundry
+ raise ValueError(
+ "Azure API version is required. Set AZURE_AI_API_VERSION environment variable or pass api_version parameter."
+ )
+
+ # Add the path to the base URL using the model as deployment name
+ # Azure AI Foundry FLUX models use /images/edits for editing
+ if "/openai/deployments/" in api_base:
+ new_url = _add_path_to_api_base(
+ api_base=api_base,
+ ending_path="/images/edits",
+ )
+ else:
+ new_url = _add_path_to_api_base(
+ api_base=api_base,
+ ending_path=f"/openai/deployments/{model}/images/edits",
+ )
+
+ # Use the new query_params dictionary
+ final_url = httpx.URL(new_url).copy_with(params={"api-version": api_version})
+
+ return str(final_url)
diff --git a/litellm/llms/azure_ai/image_generation/__init__.py b/litellm/llms/azure_ai/image_generation/__init__.py
new file mode 100644
index 00000000000..cebab3de16e
--- /dev/null
+++ b/litellm/llms/azure_ai/image_generation/__init__.py
@@ -0,0 +1,33 @@
+from litellm._logging import verbose_logger
+from litellm.llms.base_llm.image_generation.transformation import (
+ BaseImageGenerationConfig,
+)
+
+from .dall_e_2_transformation import AzureFoundryDallE2ImageGenerationConfig
+from .dall_e_3_transformation import AzureFoundryDallE3ImageGenerationConfig
+from .flux_transformation import AzureFoundryFluxImageGenerationConfig
+from .gpt_transformation import AzureFoundryGPTImageGenerationConfig
+
+__all__ = [
+ "AzureFoundryFluxImageGenerationConfig",
+ "AzureFoundryGPTImageGenerationConfig",
+ "AzureFoundryDallE2ImageGenerationConfig",
+ "AzureFoundryDallE3ImageGenerationConfig",
+]
+
+
+def get_azure_ai_image_generation_config(model: str) -> BaseImageGenerationConfig:
+ model = model.lower()
+ model = model.replace("-", "")
+ model = model.replace("_", "")
+ if model == "" or "dalle2" in model: # empty model is dall-e-2
+ return AzureFoundryDallE2ImageGenerationConfig()
+ elif "dalle3" in model:
+ return AzureFoundryDallE3ImageGenerationConfig()
+ elif "flux" in model:
+ return AzureFoundryFluxImageGenerationConfig()
+ else:
+ verbose_logger.debug(
+ f"Using AzureGPTImageGenerationConfig for model: {model}. This follows the gpt-image-1 model format."
+ )
+ return AzureFoundryGPTImageGenerationConfig()
diff --git a/litellm/llms/azure_ai/image_generation/cost_calculator.py b/litellm/llms/azure_ai/image_generation/cost_calculator.py
new file mode 100644
index 00000000000..2fc7c554a34
--- /dev/null
+++ b/litellm/llms/azure_ai/image_generation/cost_calculator.py
@@ -0,0 +1,25 @@
+from typing import Any
+
+import litellm
+from litellm.types.utils import ImageResponse
+
+
+def cost_calculator(
+ model: str,
+ image_response: Any,
+) -> float:
+ """
+ Recraft image generation cost calculator
+ """
+ _model_info = litellm.get_model_info(
+ model=model,
+ custom_llm_provider=litellm.LlmProviders.AZURE_AI.value,
+ )
+ output_cost_per_image: float = _model_info.get("output_cost_per_image") or 0.0
+ num_images: int = 0
+ if isinstance(image_response, ImageResponse):
+ if image_response.data:
+ num_images = len(image_response.data)
+ return output_cost_per_image * num_images
+ else:
+ raise ValueError(f"image_response must be of type ImageResponse got type={type(image_response)}")
diff --git a/litellm/llms/azure_ai/image_generation/dall_e_2_transformation.py b/litellm/llms/azure_ai/image_generation/dall_e_2_transformation.py
new file mode 100644
index 00000000000..1ef93366f71
--- /dev/null
+++ b/litellm/llms/azure_ai/image_generation/dall_e_2_transformation.py
@@ -0,0 +1,9 @@
+from litellm.llms.openai.image_generation import DallE2ImageGenerationConfig
+
+
+class AzureFoundryDallE2ImageGenerationConfig(DallE2ImageGenerationConfig):
+ """
+ Azure dall-e-2 image generation config
+ """
+
+ pass
diff --git a/litellm/llms/azure_ai/image_generation/dall_e_3_transformation.py b/litellm/llms/azure_ai/image_generation/dall_e_3_transformation.py
new file mode 100644
index 00000000000..4688a5c3caa
--- /dev/null
+++ b/litellm/llms/azure_ai/image_generation/dall_e_3_transformation.py
@@ -0,0 +1,9 @@
+from litellm.llms.openai.image_generation import DallE3ImageGenerationConfig
+
+
+class AzureFoundryDallE3ImageGenerationConfig(DallE3ImageGenerationConfig):
+ """
+ Azure dall-e-3 image generation config
+ """
+
+ pass
diff --git a/litellm/llms/azure_ai/image_generation/flux_transformation.py b/litellm/llms/azure_ai/image_generation/flux_transformation.py
new file mode 100644
index 00000000000..5325f32ef63
--- /dev/null
+++ b/litellm/llms/azure_ai/image_generation/flux_transformation.py
@@ -0,0 +1,14 @@
+from litellm.llms.openai.image_generation import GPTImageGenerationConfig
+
+
+class AzureFoundryFluxImageGenerationConfig(GPTImageGenerationConfig):
+ """
+ Azure Foundry flux image generation config
+
+ From manual testing it follows the gpt-image-1 image generation config
+
+ (Azure Foundry does not have any docs on supported params at the time of writing)
+
+ From our test suite - following GPTImageGenerationConfig is working for this model
+ """
+ pass
diff --git a/litellm/llms/azure_ai/image_generation/gpt_transformation.py b/litellm/llms/azure_ai/image_generation/gpt_transformation.py
new file mode 100644
index 00000000000..3eead307463
--- /dev/null
+++ b/litellm/llms/azure_ai/image_generation/gpt_transformation.py
@@ -0,0 +1,9 @@
+from litellm.llms.openai.image_generation import GPTImageGenerationConfig
+
+
+class AzureFoundryGPTImageGenerationConfig(GPTImageGenerationConfig):
+ """
+ Azure gpt-image-1 image generation config
+ """
+
+ pass
diff --git a/litellm/llms/base_llm/__init__.py b/litellm/llms/base_llm/__init__.py
index 187c985fd67..665e242969c 100644
--- a/litellm/llms/base_llm/__init__.py
+++ b/litellm/llms/base_llm/__init__.py
@@ -1,5 +1,6 @@
from .anthropic_messages.transformation import BaseAnthropicMessagesConfig
from .audio_transcription.transformation import BaseAudioTranscriptionConfig
+from .batches.transformation import BaseBatchesConfig
from .chat.transformation import BaseConfig
from .embedding.transformation import BaseEmbeddingConfig
from .image_edit.transformation import BaseImageEditConfig
@@ -12,4 +13,5 @@ __all__ = [
"BaseAnthropicMessagesConfig",
"BaseEmbeddingConfig",
"BaseImageEditConfig",
+ "BaseBatchesConfig",
]
diff --git a/litellm/llms/base_llm/audio_transcription/transformation.py b/litellm/llms/base_llm/audio_transcription/transformation.py
index 179b8d0fb02..3574996e48e 100644
--- a/litellm/llms/base_llm/audio_transcription/transformation.py
+++ b/litellm/llms/base_llm/audio_transcription/transformation.py
@@ -1,6 +1,6 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
+from typing import TYPE_CHECKING, Any, List, Optional, Union
import httpx
@@ -23,12 +23,13 @@ else:
class AudioTranscriptionRequestData:
"""
Structured data for audio transcription requests.
-
+
Attributes:
data: The request data (form data for multipart, json data for regular requests)
files: Optional files dict for multipart form data
content_type: Optional content type override
"""
+
data: Union[dict, bytes]
files: Optional[dict] = None
content_type: Optional[str] = None
@@ -66,13 +67,11 @@ class BaseAudioTranscriptionConfig(BaseConfig, ABC):
audio_file: FileTypes,
optional_params: dict,
litellm_params: dict,
- ) -> Union[AudioTranscriptionRequestData, Dict]:
+ ) -> AudioTranscriptionRequestData:
raise NotImplementedError(
"AudioTranscriptionConfig needs a request transformation for audio transcription models"
)
-
-
def transform_audio_transcription_response(
self,
raw_response: httpx.Response,
@@ -110,7 +109,6 @@ class BaseAudioTranscriptionConfig(BaseConfig, ABC):
raise NotImplementedError(
"AudioTranscriptionConfig does not need a response transformation for audio transcription models"
)
-
def get_provider_specific_params(
self,
@@ -141,7 +139,7 @@ class BaseAudioTranscriptionConfig(BaseConfig, ABC):
provider_specific_params[key] = value
return provider_specific_params
-
+
def _should_exclude_param(
self,
param_name: str,
diff --git a/litellm/llms/base_llm/base_utils.py b/litellm/llms/base_llm/base_utils.py
index 35959f0d083..9172a05e385 100644
--- a/litellm/llms/base_llm/base_utils.py
+++ b/litellm/llms/base_llm/base_utils.py
@@ -5,14 +5,37 @@ Utility functions for base LLM classes.
import copy
import json
from abc import ABC, abstractmethod
-from typing import List, Optional, Type, Union
+from typing import Any, Dict, List, Optional, Type, Union
from openai.lib import _parsing, _pydantic
from pydantic import BaseModel
from litellm._logging import verbose_logger
from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolCallChunk
-from litellm.types.utils import Message, ProviderSpecificModelInfo
+from litellm.types.utils import Message, ProviderSpecificModelInfo, TokenCountResponse
+
+
+class BaseTokenCounter(ABC):
+ @abstractmethod
+ async def count_tokens(
+ self,
+ model_to_use: str,
+ messages: Optional[List[Dict[str, Any]]],
+ contents: Optional[List[Dict[str, Any]]],
+ deployment: Optional[Dict[str, Any]] = None,
+ request_model: str = "",
+ ) -> Optional[TokenCountResponse]:
+ pass
+
+ @abstractmethod
+ def should_use_token_counting_api(
+ self,
+ custom_llm_provider: Optional[str] = None,
+ ) -> bool:
+ """
+ Returns True if we should the this API for token counting for the selected `custom_llm_provider`
+ """
+ return False
class BaseLLMModelInfo(ABC):
@@ -70,6 +93,16 @@ class BaseLLMModelInfo(ABC):
"""
pass
+ def get_token_counter(self) -> Optional[BaseTokenCounter]:
+ """
+ Factory method to create a token counter for this provider.
+
+ Returns:
+ Optional TokenCounterInterface implementation for this provider,
+ or None if token counting is not supported.
+ """
+ return None
+
def _convert_tool_response_to_message(
tool_calls: List[ChatCompletionToolCallChunk],
diff --git a/litellm/llms/base_llm/batches/transformation.py b/litellm/llms/base_llm/batches/transformation.py
new file mode 100644
index 00000000000..9e67689fcd9
--- /dev/null
+++ b/litellm/llms/base_llm/batches/transformation.py
@@ -0,0 +1,218 @@
+import types
+from abc import ABC, abstractmethod
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
+
+import httpx
+from httpx import Headers
+
+from litellm.types.llms.openai import (
+ AllMessageValues,
+ CreateBatchRequest,
+)
+from litellm.types.utils import LiteLLMBatch, LlmProviders
+
+if TYPE_CHECKING:
+ from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
+
+ from ..chat.transformation import BaseLLMException as _BaseLLMException
+
+ LiteLLMLoggingObj = _LiteLLMLoggingObj
+ BaseLLMException = _BaseLLMException
+else:
+ LiteLLMLoggingObj = Any
+ BaseLLMException = Any
+
+
+class BaseBatchesConfig(ABC):
+ """
+ Abstract base class for batch processing configurations across different LLM providers.
+
+ This class defines the interface that all provider-specific batch configurations
+ must implement to work with LiteLLM's unified batch processing system.
+ """
+
+ def __init__(self):
+ pass
+
+ @property
+ @abstractmethod
+ def custom_llm_provider(self) -> LlmProviders:
+ """Return the LLM provider type for this configuration."""
+ pass
+
+ @classmethod
+ def get_config(cls):
+ """Get configuration dictionary for this class."""
+ return {
+ k: v
+ for k, v in cls.__dict__.items()
+ if not k.startswith("__")
+ and not k.startswith("_abc")
+ and not isinstance(
+ v,
+ (
+ types.FunctionType,
+ types.BuiltinFunctionType,
+ classmethod,
+ staticmethod,
+ ),
+ )
+ and v is not None
+ }
+
+ @abstractmethod
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ """
+ Validate and prepare environment-specific headers and parameters.
+
+ Args:
+ headers: HTTP headers dictionary
+ model: Model name
+ messages: List of messages
+ optional_params: Optional parameters
+ litellm_params: LiteLLM parameters
+ api_key: API key
+ api_base: API base URL
+
+ Returns:
+ Updated headers dictionary
+ """
+ pass
+
+ @abstractmethod
+ def get_complete_batch_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: Dict,
+ litellm_params: Dict,
+ data: CreateBatchRequest,
+ ) -> str:
+ """
+ Get the complete URL for batch creation request.
+
+ Args:
+ api_base: Base API URL
+ api_key: API key
+ model: Model name
+ optional_params: Optional parameters
+ litellm_params: LiteLLM parameters
+ data: Batch creation request data
+
+ Returns:
+ Complete URL for the batch request
+ """
+ pass
+
+ @abstractmethod
+ def transform_create_batch_request(
+ self,
+ model: str,
+ create_batch_data: CreateBatchRequest,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> Union[bytes, str, Dict[str, Any]]:
+ """
+ Transform the batch creation request to provider-specific format.
+
+ Args:
+ model: Model name
+ create_batch_data: Batch creation request data
+ optional_params: Optional parameters
+ litellm_params: LiteLLM parameters
+
+ Returns:
+ Transformed request data
+ """
+ pass
+
+ @abstractmethod
+ def transform_create_batch_response(
+ self,
+ model: Optional[str],
+ raw_response: httpx.Response,
+ logging_obj: LiteLLMLoggingObj,
+ litellm_params: dict,
+ ) -> LiteLLMBatch:
+ """
+ Transform provider-specific batch response to LiteLLM format.
+
+ Args:
+ model: Model name
+ raw_response: Raw HTTP response
+ logging_obj: Logging object
+ litellm_params: LiteLLM parameters
+
+ Returns:
+ LiteLLM batch object
+ """
+ pass
+
+ @abstractmethod
+ def transform_retrieve_batch_request(
+ self,
+ batch_id: str,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> Union[bytes, str, Dict[str, Any]]:
+ """
+ Transform the batch retrieval request to provider-specific format.
+
+ Args:
+ batch_id: Batch ID to retrieve
+ optional_params: Optional parameters
+ litellm_params: LiteLLM parameters
+
+ Returns:
+ Transformed request data
+ """
+ pass
+
+ @abstractmethod
+ def transform_retrieve_batch_response(
+ self,
+ model: Optional[str],
+ raw_response: httpx.Response,
+ logging_obj: LiteLLMLoggingObj,
+ litellm_params: dict,
+ ) -> LiteLLMBatch:
+ """
+ Transform provider-specific batch retrieval response to LiteLLM format.
+
+ Args:
+ model: Model name
+ raw_response: Raw HTTP response
+ logging_obj: Logging object
+ litellm_params: LiteLLM parameters
+
+ Returns:
+ LiteLLM batch object
+ """
+ pass
+
+ @abstractmethod
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[Dict, Headers]
+ ) -> "BaseLLMException":
+ """
+ Get the appropriate error class for this provider.
+
+ Args:
+ error_message: Error message
+ status_code: HTTP status code
+ headers: Response headers
+
+ Returns:
+ Provider-specific exception class
+ """
+ pass
diff --git a/litellm/llms/base_llm/files/transformation.py b/litellm/llms/base_llm/files/transformation.py
index 5c37a8b7547..35b76479cdc 100644
--- a/litellm/llms/base_llm/files/transformation.py
+++ b/litellm/llms/base_llm/files/transformation.py
@@ -35,6 +35,16 @@ class BaseFilesConfig(BaseConfig):
def custom_llm_provider(self) -> LlmProviders:
pass
+ @property
+ def file_upload_http_method(self) -> str:
+ """
+ HTTP method to use for file uploads.
+ Override this in provider configs if they need different methods.
+ Default is POST (used by most providers like OpenAI, Anthropic).
+ S3-based providers like Bedrock should return "PUT".
+ """
+ return "POST"
+
@abstractmethod
def get_supported_openai_params(
self, model: str
diff --git a/litellm/llms/base_llm/google_genai/transformation.py b/litellm/llms/base_llm/google_genai/transformation.py
index 9706b226c47..6dbccaada9a 100644
--- a/litellm/llms/base_llm/google_genai/transformation.py
+++ b/litellm/llms/base_llm/google_genai/transformation.py
@@ -10,12 +10,14 @@ if TYPE_CHECKING:
GenerateContentConfigDict,
GenerateContentContentListUnionDict,
GenerateContentResponse,
+ ToolConfigDict,
)
else:
GenerateContentConfigDict = Any
GenerateContentContentListUnionDict = Any
GenerateContentResponse = Any
LiteLLMLoggingObj = Any
+ ToolConfigDict = Any
from litellm.types.router import GenericLiteLLMParams
@@ -145,6 +147,7 @@ class BaseGoogleGenAIGenerateContentConfig(ABC):
self,
model: str,
contents: GenerateContentContentListUnionDict,
+ tools: Optional[ToolConfigDict],
generate_content_config_dict: Dict,
) -> dict:
"""
@@ -153,6 +156,7 @@ class BaseGoogleGenAIGenerateContentConfig(ABC):
Args:
model: The model name
contents: Input contents
+ tools: Tools
generate_content_request_params: Request parameters
litellm_params: LiteLLM parameters
headers: Request headers
diff --git a/litellm/llms/base_llm/passthrough/transformation.py b/litellm/llms/base_llm/passthrough/transformation.py
index 60d89c1610f..f925e6819dc 100644
--- a/litellm/llms/base_llm/passthrough/transformation.py
+++ b/litellm/llms/base_llm/passthrough/transformation.py
@@ -31,30 +31,26 @@ class BasePassthroughConfig(BaseLLMModelInfo):
Args:
endpoint: str - the endpoint to add to the url
base_target_url: str - the base url to add the endpoint to
- request_query_params: dict - the query params to add to the url
+ request_query_params: Optional[dict] - the query params to add to the url
Returns:
- str - the formatted url
+ httpx.URL - the formatted url
"""
from urllib.parse import urlencode
import httpx
- encoded_endpoint = httpx.URL(endpoint).path
+ base = base_target_url.rstrip('/')
+ endpoint = endpoint.lstrip('/')
+ full_url = f"{base}/{endpoint}"
- # Ensure endpoint starts with '/' for proper URL construction
- if not encoded_endpoint.startswith("/"):
- encoded_endpoint = "/" + encoded_endpoint
-
- # Construct the full target URL using httpx
- base_url = httpx.URL(base_target_url)
- updated_url = base_url.copy_with(path=encoded_endpoint)
+ url = httpx.URL(full_url)
if request_query_params:
- # Create a new URL with the merged query params
- updated_url = updated_url.copy_with(
+ url = url.copy_with(
query=urlencode(request_query_params).encode("ascii")
)
- return updated_url
+
+ return url
@abstractmethod
def get_complete_url(
diff --git a/litellm/llms/base_llm/rerank/transformation.py b/litellm/llms/base_llm/rerank/transformation.py
index 8701fe57bfd..6e9c03dee89 100644
--- a/litellm/llms/base_llm/rerank/transformation.py
+++ b/litellm/llms/base_llm/rerank/transformation.py
@@ -3,7 +3,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
import httpx
-from litellm.types.rerank import OptionalRerankParams, RerankBilledUnits, RerankResponse
+from litellm.types.rerank import RerankBilledUnits, RerankResponse
from litellm.types.utils import ModelInfo
from ..chat.transformation import BaseLLMException
@@ -30,7 +30,7 @@ class BaseRerankConfig(ABC):
def transform_rerank_request(
self,
model: str,
- optional_rerank_params: OptionalRerankParams,
+ optional_rerank_params: Dict,
headers: dict,
) -> dict:
return {}
@@ -78,7 +78,7 @@ class BaseRerankConfig(ABC):
return_documents: Optional[bool] = True,
max_chunks_per_doc: Optional[int] = None,
max_tokens_per_doc: Optional[int] = None,
- ) -> OptionalRerankParams:
+ ) -> Dict:
pass
def get_error_class(
diff --git a/litellm/llms/base_llm/responses/transformation.py b/litellm/llms/base_llm/responses/transformation.py
index e2f89da5e86..facabbda72a 100644
--- a/litellm/llms/base_llm/responses/transformation.py
+++ b/litellm/llms/base_llm/responses/transformation.py
@@ -12,6 +12,7 @@ from litellm.types.llms.openai import (
)
from litellm.types.responses.main import *
from litellm.types.router import GenericLiteLLMParams
+from litellm.types.utils import LlmProviders
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@@ -29,6 +30,11 @@ class BaseResponsesAPIConfig(ABC):
def __init__(self):
pass
+ @property
+ @abstractmethod
+ def custom_llm_provider(self) -> LlmProviders:
+ pass
+
@classmethod
def get_config(cls):
return {
@@ -211,3 +217,28 @@ class BaseResponsesAPIConfig(ABC):
) -> bool:
"""Returns True if litellm should fake a stream for the given model and stream value"""
return False
+
+ #########################################################
+ ########## CANCEL RESPONSE API TRANSFORMATION ##########
+ #########################################################
+ @abstractmethod
+ def transform_cancel_response_api_request(
+ self,
+ response_id: str,
+ api_base: str,
+ litellm_params: GenericLiteLLMParams,
+ headers: dict,
+ ) -> Tuple[str, Dict]:
+ pass
+
+ @abstractmethod
+ def transform_cancel_response_api_response(
+ self,
+ raw_response: httpx.Response,
+ logging_obj: LiteLLMLoggingObj,
+ ) -> ResponsesAPIResponse:
+ pass
+
+ #########################################################
+ ########## END CANCEL RESPONSE API TRANSFORMATION #######
+ #########################################################
diff --git a/litellm/llms/baseten.py b/litellm/llms/baseten.py
deleted file mode 100644
index e1d513d6d11..00000000000
--- a/litellm/llms/baseten.py
+++ /dev/null
@@ -1,172 +0,0 @@
-import json
-import time
-from typing import Callable
-
-import litellm
-from litellm.types.utils import ModelResponse, Usage
-
-
-class BasetenError(Exception):
- def __init__(self, status_code, message):
- self.status_code = status_code
- self.message = message
- super().__init__(
- self.message
- ) # Call the base class constructor with the parameters it needs
-
-
-def validate_environment(api_key):
- headers = {
- "accept": "application/json",
- "content-type": "application/json",
- }
- if api_key:
- headers["Authorization"] = f"Api-Key {api_key}"
- return headers
-
-
-def completion(
- model: str,
- messages: list,
- model_response: ModelResponse,
- print_verbose: Callable,
- encoding,
- api_key,
- logging_obj,
- optional_params: dict,
- litellm_params=None,
- logger_fn=None,
-):
- headers = validate_environment(api_key)
- completion_url_fragment_1 = "https://app.baseten.co/models/"
- completion_url_fragment_2 = "/predict"
- model = model
- prompt = ""
- for message in messages:
- if "role" in message:
- if message["role"] == "user":
- prompt += f"{message['content']}"
- else:
- prompt += f"{message['content']}"
- else:
- prompt += f"{message['content']}"
- data = {
- "inputs": prompt,
- "prompt": prompt,
- "parameters": optional_params,
- "stream": (
- True
- if "stream" in optional_params and optional_params["stream"] is True
- else False
- ),
- }
-
- ## LOGGING
- logging_obj.pre_call(
- input=prompt,
- api_key=api_key,
- additional_args={"complete_input_dict": data},
- )
- ## COMPLETION CALL
- response = litellm.module_level_client.post(
- completion_url_fragment_1 + model + completion_url_fragment_2,
- headers=headers,
- data=json.dumps(data),
- stream=(
- True
- if "stream" in optional_params and optional_params["stream"] is True
- else False
- ),
- )
- if "text/event-stream" in response.headers["Content-Type"] or (
- "stream" in optional_params and optional_params["stream"] is True
- ):
- return response.iter_lines()
- else:
- ## LOGGING
- logging_obj.post_call(
- input=prompt,
- api_key=api_key,
- original_response=response.text,
- additional_args={"complete_input_dict": data},
- )
- print_verbose(f"raw model_response: {response.text}")
- ## RESPONSE OBJECT
- completion_response = response.json()
- if "error" in completion_response:
- raise BasetenError(
- message=completion_response["error"],
- status_code=response.status_code,
- )
- else:
- if "model_output" in completion_response:
- if (
- isinstance(completion_response["model_output"], dict)
- and "data" in completion_response["model_output"]
- and isinstance(completion_response["model_output"]["data"], list)
- ):
- model_response.choices[0].message.content = completion_response[ # type: ignore
- "model_output"
- ][
- "data"
- ][
- 0
- ]
- elif isinstance(completion_response["model_output"], str):
- model_response.choices[0].message.content = completion_response[ # type: ignore
- "model_output"
- ]
- elif "completion" in completion_response and isinstance(
- completion_response["completion"], str
- ):
- model_response.choices[0].message.content = completion_response[ # type: ignore
- "completion"
- ]
- elif isinstance(completion_response, list) and len(completion_response) > 0:
- if "generated_text" not in completion_response:
- raise BasetenError(
- message=f"Unable to parse response. Original response: {response.text}",
- status_code=response.status_code,
- )
- model_response.choices[0].message.content = completion_response[0][ # type: ignore
- "generated_text"
- ]
- ## GETTING LOGPROBS
- if (
- "details" in completion_response[0]
- and "tokens" in completion_response[0]["details"]
- ):
- model_response.choices[0].finish_reason = completion_response[0][
- "details"
- ]["finish_reason"]
- sum_logprob = 0
- for token in completion_response[0]["details"]["tokens"]:
- sum_logprob += token["logprob"]
- model_response.choices[0].logprobs = sum_logprob # type: ignore
- else:
- raise BasetenError(
- message=f"Unable to parse response. Original response: {response.text}",
- status_code=response.status_code,
- )
-
- ## CALCULATING USAGE - baseten charges on time, not tokens - have some mapping of cost here.
- prompt_tokens = len(encoding.encode(prompt))
- completion_tokens = len(
- encoding.encode(model_response["choices"][0]["message"]["content"])
- )
-
- model_response.created = int(time.time())
- model_response.model = model
- usage = Usage(
- prompt_tokens=prompt_tokens,
- completion_tokens=completion_tokens,
- total_tokens=prompt_tokens + completion_tokens,
- )
-
- setattr(model_response, "usage", usage)
- return model_response
-
-
-def embedding():
- # logic for parsing in - calling - parsing out model embedding calls
- pass
diff --git a/litellm/llms/baseten/chat.py b/litellm/llms/baseten/chat.py
new file mode 100644
index 00000000000..05fc9961ac5
--- /dev/null
+++ b/litellm/llms/baseten/chat.py
@@ -0,0 +1,118 @@
+from typing import Optional
+from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
+
+
+class BasetenConfig(OpenAIGPTConfig):
+ """
+ Reference: https://inference.baseten.co/v1
+
+ Below are the parameters:
+ """
+
+ max_tokens: Optional[int] = None
+ response_format: Optional[dict] = None
+ seed: Optional[int] = None
+ stream: Optional[bool] = None
+ top_p: Optional[int] = None
+ tool_choice: Optional[str] = None
+ tools: Optional[list] = None
+ user: Optional[str] = None
+ presence_penalty: Optional[int] = None
+ frequency_penalty: Optional[int] = None
+ stream_options: Optional[dict] = None
+
+ def __init__(
+ self,
+ max_tokens: Optional[int] = None,
+ response_format: Optional[dict] = None,
+ seed: Optional[int] = None,
+ stop: Optional[list] = None,
+ stream: Optional[bool] = None,
+ temperature: Optional[float] = None,
+ top_p: Optional[int] = None,
+ tool_choice: Optional[str] = None,
+ tools: Optional[list] = None,
+ user: Optional[str] = None,
+ presence_penalty: Optional[int] = None,
+ frequency_penalty: Optional[int] = None,
+ stream_options: Optional[dict] = None,
+ ) -> None:
+ locals_ = locals().copy()
+ for key, value in locals_.items():
+ if key != "self" and value is not None:
+ setattr(self.__class__, key, value)
+
+ @classmethod
+ def get_config(cls):
+ return super().get_config()
+
+ def get_supported_openai_params(self, model: str) -> list:
+ """
+ Get the supported OpenAI params for the given model
+ """
+ return [
+ "max_tokens",
+ "max_completion_tokens",
+ "response_format",
+ "seed",
+ "stop",
+ "stream",
+ "temperature",
+ "top_p",
+ "tool_choice",
+ "tools",
+ "user",
+ "presence_penalty",
+ "frequency_penalty",
+ "stream_options",
+ ]
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ supported_openai_params = self.get_supported_openai_params(model=model)
+ for param, value in non_default_params.items():
+ if param == "max_completion_tokens":
+ optional_params["max_tokens"] = value
+ elif param in supported_openai_params:
+ optional_params[param] = value
+ return optional_params
+
+ def _get_openai_compatible_provider_info(self, api_base: str, api_key: str) -> tuple:
+ """
+ Get the OpenAI compatible provider info for Baseten
+ """
+ # Default to Model API
+ default_api_base = "https://inference.baseten.co/v1"
+ default_api_key = api_key or "BASETEN_API_KEY"
+
+ return default_api_base, default_api_key
+
+ @staticmethod
+ def is_dedicated_deployment(model: str) -> bool:
+ """
+ Check if the model is a dedicated deployment (8-digit alphanumeric code)
+ """
+ # Remove 'baseten/' prefix if present
+ model_id = model.replace("baseten/", "")
+
+ # Check if it's an 8-digit alphanumeric code
+ import re
+ return bool(re.match(r'^[a-zA-Z0-9]{8}$', model_id))
+
+ @staticmethod
+ def get_api_base_for_model(model: str) -> str:
+ """
+ Get the appropriate API base URL for the given model
+ """
+ if BasetenConfig.is_dedicated_deployment(model):
+ # Extract the model ID (remove 'baseten/' prefix if present)
+ model_id = model.replace("baseten/", "")
+ return f"https://model-{model_id}.api.baseten.co/environments/production/sync/v1"
+ else:
+ # Use Model API
+ return "https://inference.baseten.co/v1"
\ No newline at end of file
diff --git a/litellm/llms/bedrock/base_aws_llm.py b/litellm/llms/bedrock/base_aws_llm.py
index cc205e62dc9..8211addaf95 100644
--- a/litellm/llms/bedrock/base_aws_llm.py
+++ b/litellm/llms/bedrock/base_aws_llm.py
@@ -20,7 +20,11 @@ from pydantic import BaseModel
from litellm._logging import verbose_logger
from litellm.caching.caching import DualCache
-from litellm.constants import BEDROCK_INVOKE_PROVIDERS_LITERAL, BEDROCK_MAX_POLICY_SIZE
+from litellm.constants import (
+ BEDROCK_EMBEDDING_PROVIDERS_LITERAL,
+ BEDROCK_INVOKE_PROVIDERS_LITERAL,
+ BEDROCK_MAX_POLICY_SIZE,
+)
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.secret_managers.main import get_secret, get_secret_str
@@ -66,6 +70,7 @@ class BaseAWSLLM:
"aws_web_identity_token",
"aws_sts_endpoint",
"aws_bedrock_runtime_endpoint",
+ "aws_external_id",
]
def get_cache_key(self, credential_args: Dict[str, Optional[str]]) -> str:
@@ -88,6 +93,7 @@ class BaseAWSLLM:
aws_role_name: Optional[str] = None,
aws_web_identity_token: Optional[str] = None,
aws_sts_endpoint: Optional[str] = None,
+ aws_external_id: Optional[str] = None,
):
"""
Return a boto3.Credentials object
@@ -103,6 +109,7 @@ class BaseAWSLLM:
aws_role_name,
aws_web_identity_token,
aws_sts_endpoint,
+ aws_external_id,
]
# Iterate over parameters and update if needed
@@ -127,6 +134,7 @@ class BaseAWSLLM:
aws_role_name,
aws_web_identity_token,
aws_sts_endpoint,
+ aws_external_id,
) = params_to_check
verbose_logger.debug(
@@ -139,7 +147,8 @@ class BaseAWSLLM:
"aws_profile_name=%s\n"
"aws_role_name=%s\n"
"aws_web_identity_token=%s\n"
- "aws_sts_endpoint=%s",
+ "aws_sts_endpoint=%s\n"
+ "aws_external_id=%s",
aws_access_key_id,
aws_secret_access_key,
aws_session_token,
@@ -149,6 +158,7 @@ class BaseAWSLLM:
aws_role_name,
aws_web_identity_token,
aws_sts_endpoint,
+ aws_external_id,
)
# create cache key for non-expiring auth flows
@@ -177,17 +187,46 @@ class BaseAWSLLM:
aws_session_name=aws_session_name,
aws_region_name=aws_region_name,
aws_sts_endpoint=aws_sts_endpoint,
+ aws_external_id=aws_external_id,
)
elif aws_role_name is not None:
- # If aws_session_name is not provided, generate a default one
- if aws_session_name is None:
- aws_session_name = f"litellm-session-{int(datetime.now().timestamp())}"
- credentials, _cache_ttl = self._auth_with_aws_role(
- aws_access_key_id=aws_access_key_id,
- aws_secret_access_key=aws_secret_access_key,
- aws_role_name=aws_role_name,
- aws_session_name=aws_session_name,
- )
+ # Check if we're in IRSA and trying to assume the same role we already have
+ current_role_arn = os.getenv("AWS_ROLE_ARN")
+ web_identity_token_file = os.getenv("AWS_WEB_IDENTITY_TOKEN_FILE")
+
+ # In IRSA environments, we should skip role assumption if we're already running as the target role
+ # This is true when:
+ # 1. We have AWS_ROLE_ARN set (current role)
+ # 2. We have AWS_WEB_IDENTITY_TOKEN_FILE set (IRSA environment)
+ # 3. The current role matches the requested role
+ if (
+ current_role_arn
+ and web_identity_token_file
+ and current_role_arn == aws_role_name
+ ):
+ verbose_logger.debug(
+ "Using IRSA same-role optimization: calling _auth_with_env_vars"
+ )
+ # We're already running as this role via IRSA, no need to assume it again
+ # Use the default boto3 credentials (which will use the IRSA credentials)
+ credentials, _cache_ttl = self._auth_with_env_vars()
+ else:
+ verbose_logger.debug(
+ "Using role assumption: calling _auth_with_aws_role"
+ )
+ # If aws_session_name is not provided, generate a default one
+ if aws_session_name is None:
+ aws_session_name = (
+ f"litellm-session-{int(datetime.now().timestamp())}"
+ )
+ credentials, _cache_ttl = self._auth_with_aws_role(
+ aws_access_key_id=aws_access_key_id,
+ aws_secret_access_key=aws_secret_access_key,
+ aws_session_token=aws_session_token,
+ aws_role_name=aws_role_name,
+ aws_session_name=aws_session_name,
+ aws_external_id=aws_external_id,
+ )
elif aws_profile_name is not None: ### CHECK SESSION ###
credentials, _cache_ttl = self._auth_with_aws_profile(aws_profile_name)
@@ -292,6 +331,40 @@ class BaseAWSLLM:
return provider
return None
+ @staticmethod
+ def get_bedrock_embedding_provider(
+ model: str,
+ ) -> Optional[BEDROCK_EMBEDDING_PROVIDERS_LITERAL]:
+ """
+ Helper function to get the bedrock embedding provider from the model
+
+ Handles scenarios like:
+ 1. model=cohere.embed-english-v3:0 -> Returns `cohere`
+ 2. model=amazon.titan-embed-text-v1 -> Returns `amazon`
+ 3. model=us.twelvelabs.marengo-embed-2-7-v1:0 -> Returns `twelvelabs`
+ 4. model=twelvelabs.marengo-embed-2-7-v1:0 -> Returns `twelvelabs`
+ """
+ # Handle regional models like us.twelvelabs.marengo-embed-2-7-v1:0
+ if "." in model:
+ parts = model.split(".")
+ # Check if the second part (after potential region) is a known provider
+ if len(parts) >= 2:
+ potential_provider = parts[1] # e.g., "twelvelabs" from "us.twelvelabs.marengo-embed-2-7-v1:0"
+ if potential_provider in get_args(BEDROCK_EMBEDDING_PROVIDERS_LITERAL):
+ return cast(BEDROCK_EMBEDDING_PROVIDERS_LITERAL, potential_provider)
+
+ # Check if the first part is a known provider (standard format)
+ potential_provider = parts[0] # e.g., "cohere" from "cohere.embed-english-v3:0"
+ if potential_provider in get_args(BEDROCK_EMBEDDING_PROVIDERS_LITERAL):
+ return cast(BEDROCK_EMBEDDING_PROVIDERS_LITERAL, potential_provider)
+
+ # Fallback: check if any provider name appears in the model string
+ for provider in get_args(BEDROCK_EMBEDDING_PROVIDERS_LITERAL):
+ if provider in model:
+ return cast(BEDROCK_EMBEDDING_PROVIDERS_LITERAL, provider)
+
+ return None
+
def _get_aws_region_name(
self,
optional_params: dict,
@@ -388,6 +461,7 @@ class BaseAWSLLM:
aws_session_name: str,
aws_region_name: Optional[str],
aws_sts_endpoint: Optional[str],
+ aws_external_id: Optional[str] = None,
) -> Tuple[Credentials, Optional[int]]:
"""
Authenticate with AWS Web Identity Token
@@ -420,13 +494,19 @@ class BaseAWSLLM:
# https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts/client/assume_role_with_web_identity.html
- sts_response = sts_client.assume_role_with_web_identity(
- RoleArn=aws_role_name,
- RoleSessionName=aws_session_name,
- WebIdentityToken=oidc_token,
- DurationSeconds=3600,
- Policy='{"Version":"2012-10-17","Statement":[{"Sid":"BedrockLiteLLM","Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream"],"Resource":"*","Condition":{"Bool":{"aws:SecureTransport":"true"},"StringLike":{"aws:UserAgent":"litellm/*"}}}]}',
- )
+ assume_role_params = {
+ "RoleArn": aws_role_name,
+ "RoleSessionName": aws_session_name,
+ "WebIdentityToken": oidc_token,
+ "DurationSeconds": 3600,
+ "Policy": '{"Version":"2012-10-17","Statement":[{"Sid":"BedrockLiteLLM","Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream"],"Resource":"*","Condition":{"Bool":{"aws:SecureTransport":"true"},"StringLike":{"aws:UserAgent":"litellm/*"}}}]}',
+ }
+
+ # Add ExternalId parameter if provided
+ if aws_external_id is not None:
+ assume_role_params["ExternalId"] = aws_external_id
+
+ sts_response = sts_client.assume_role_with_web_identity(**assume_role_params)
iam_creds_dict = {
"aws_access_key_id": sts_response["Credentials"]["AccessKeyId"],
@@ -446,13 +526,142 @@ class BaseAWSLLM:
iam_creds = session.get_credentials()
return iam_creds, self._get_default_ttl_for_boto3_credentials()
+ def _handle_irsa_cross_account(
+ self,
+ irsa_role_arn: str,
+ aws_role_name: str,
+ aws_session_name: str,
+ region: str,
+ web_identity_token_file: str,
+ aws_external_id: Optional[str] = None,
+ ) -> dict:
+ """Handle cross-account role assumption for IRSA."""
+ import boto3
+
+ verbose_logger.debug("Cross-account role assumption detected")
+
+ # Read the web identity token
+ with open(web_identity_token_file, "r") as f:
+ web_identity_token = f.read().strip()
+
+ # Create an STS client without credentials
+ with tracer.trace("boto3.client(sts) for manual IRSA"):
+ sts_client = boto3.client("sts", region_name=region)
+
+ # Manually assume the IRSA role with the session name
+ verbose_logger.debug(
+ f"Manually assuming IRSA role {irsa_role_arn} with session {aws_session_name}"
+ )
+ irsa_response = sts_client.assume_role_with_web_identity(
+ RoleArn=irsa_role_arn,
+ RoleSessionName=aws_session_name,
+ WebIdentityToken=web_identity_token,
+ )
+
+ # Extract the credentials from the IRSA assumption
+ irsa_creds = irsa_response["Credentials"]
+
+ # Create a new STS client with the IRSA credentials
+ with tracer.trace("boto3.client(sts) with manual IRSA credentials"):
+ sts_client_with_creds = boto3.client(
+ "sts",
+ region_name=region,
+ aws_access_key_id=irsa_creds["AccessKeyId"],
+ aws_secret_access_key=irsa_creds["SecretAccessKey"],
+ aws_session_token=irsa_creds["SessionToken"],
+ )
+
+ # Get current caller identity for debugging
+ try:
+ caller_identity = sts_client_with_creds.get_caller_identity()
+ verbose_logger.debug(
+ f"Current identity after manual IRSA assumption: {caller_identity.get('Arn', 'unknown')}"
+ )
+ except Exception as e:
+ verbose_logger.debug(f"Failed to get caller identity: {e}")
+
+ # Now assume the target role
+ verbose_logger.debug(
+ f"Attempting to assume target role: {aws_role_name} with session: {aws_session_name}"
+ )
+ assume_role_params = {
+ "RoleArn": aws_role_name,
+ "RoleSessionName": aws_session_name,
+ }
+
+ # Add ExternalId parameter if provided
+ if aws_external_id is not None:
+ assume_role_params["ExternalId"] = aws_external_id
+
+ return sts_client_with_creds.assume_role(**assume_role_params)
+
+ def _handle_irsa_same_account(
+ self,
+ aws_role_name: str,
+ aws_session_name: str,
+ region: str,
+ aws_external_id: Optional[str] = None,
+ ) -> dict:
+ """Handle same-account role assumption for IRSA."""
+ import boto3
+
+ verbose_logger.debug("Same account role assumption, using automatic IRSA")
+ with tracer.trace("boto3.client(sts) with automatic IRSA"):
+ sts_client = boto3.client("sts", region_name=region)
+
+ # Get current caller identity for debugging
+ try:
+ caller_identity = sts_client.get_caller_identity()
+ verbose_logger.debug(
+ f"Current IRSA identity: {caller_identity.get('Arn', 'unknown')}"
+ )
+ except Exception as e:
+ verbose_logger.debug(f"Failed to get caller identity: {e}")
+
+ # Assume the role
+ verbose_logger.debug(
+ f"Attempting to assume role: {aws_role_name} with session: {aws_session_name}"
+ )
+ assume_role_params = {
+ "RoleArn": aws_role_name,
+ "RoleSessionName": aws_session_name,
+ }
+
+ # Add ExternalId parameter if provided
+ if aws_external_id is not None:
+ assume_role_params["ExternalId"] = aws_external_id
+
+ return sts_client.assume_role(**assume_role_params)
+
+ def _extract_credentials_and_ttl(
+ self, sts_response: dict
+ ) -> Tuple[Credentials, Optional[int]]:
+ """Extract credentials and TTL from STS response."""
+ from botocore.credentials import Credentials
+
+ sts_credentials = sts_response["Credentials"]
+ credentials = Credentials(
+ access_key=sts_credentials["AccessKeyId"],
+ secret_key=sts_credentials["SecretAccessKey"],
+ token=sts_credentials["SessionToken"],
+ )
+
+ expiration_time = sts_credentials["Expiration"]
+ ttl = int(
+ (expiration_time - datetime.now(expiration_time.tzinfo)).total_seconds()
+ )
+
+ return credentials, ttl
+
@tracer.wrap()
def _auth_with_aws_role(
self,
aws_access_key_id: Optional[str],
aws_secret_access_key: Optional[str],
+ aws_session_token: Optional[str],
aws_role_name: str,
aws_session_name: str,
+ aws_external_id: Optional[str] = None,
) -> Tuple[Credentials, Optional[int]]:
"""
Authenticate with AWS Role
@@ -460,16 +669,87 @@ class BaseAWSLLM:
import boto3
from botocore.credentials import Credentials
- with tracer.trace("boto3.client(sts)"):
- sts_client = boto3.client(
- "sts",
- aws_access_key_id=aws_access_key_id, # [OPTIONAL]
- aws_secret_access_key=aws_secret_access_key, # [OPTIONAL]
+ # Check if we're in an EKS/IRSA environment
+ web_identity_token_file = os.getenv("AWS_WEB_IDENTITY_TOKEN_FILE")
+ irsa_role_arn = os.getenv("AWS_ROLE_ARN")
+
+ # If we have IRSA environment variables and no explicit credentials,
+ # we need to use the web identity token flow
+ if (
+ web_identity_token_file
+ and irsa_role_arn
+ and aws_access_key_id is None
+ and aws_secret_access_key is None
+ ):
+ # For cross-account role assumption with specific session names,
+ # we need to manually assume the IRSA role first with the correct session name
+ verbose_logger.debug(
+ f"IRSA detected: using web identity token from {web_identity_token_file}"
)
- sts_response = sts_client.assume_role(
- RoleArn=aws_role_name, RoleSessionName=aws_session_name
- )
+ try:
+ # Get region from environment
+ region = (
+ os.getenv("AWS_REGION")
+ or os.getenv("AWS_DEFAULT_REGION")
+ or "us-east-1"
+ )
+
+ # Check if we need to do cross-account role assumption
+ if aws_role_name != irsa_role_arn:
+ sts_response = self._handle_irsa_cross_account(
+ irsa_role_arn,
+ aws_role_name,
+ aws_session_name,
+ region,
+ web_identity_token_file,
+ aws_external_id,
+ )
+ else:
+ sts_response = self._handle_irsa_same_account(
+ aws_role_name, aws_session_name, region, aws_external_id
+ )
+
+ return self._extract_credentials_and_ttl(sts_response)
+
+ except Exception as e:
+ verbose_logger.debug(f"Failed to assume role via IRSA: {e}")
+ if "AccessDenied" in str(
+ e
+ ) and "is not authorized to perform: sts:AssumeRole" in str(e):
+ # Provide a more helpful error message for trust policy issues
+ verbose_logger.error(
+ f"Access denied when trying to assume role {aws_role_name}. "
+ f"Please ensure the trust policy of {aws_role_name} allows "
+ f"the current role to assume it. Current identity: check logs with verbose mode."
+ )
+ # Re-raise the exception instead of falling through
+ raise
+
+ # In EKS/IRSA environments, use ambient credentials (no explicit keys needed)
+ # This allows the web identity token to work automatically
+ if aws_access_key_id is None and aws_secret_access_key is None:
+ with tracer.trace("boto3.client(sts)"):
+ sts_client = boto3.client("sts")
+ else:
+ with tracer.trace("boto3.client(sts)"):
+ sts_client = boto3.client(
+ "sts",
+ aws_access_key_id=aws_access_key_id,
+ aws_secret_access_key=aws_secret_access_key,
+ aws_session_token=aws_session_token,
+ )
+
+ assume_role_params = {
+ "RoleArn": aws_role_name,
+ "RoleSessionName": aws_session_name,
+ }
+
+ # Add ExternalId parameter if provided
+ if aws_external_id is not None:
+ assume_role_params["ExternalId"] = aws_external_id
+
+ sts_response = sts_client.assume_role(**assume_role_params)
# Extract the credentials from the response and convert to Session Credentials
sts_credentials = sts_response["Credentials"]
@@ -591,14 +871,14 @@ class BaseAWSLLM:
)
# Determine proxy_endpoint_url
- if env_aws_bedrock_runtime_endpoint and isinstance(
- env_aws_bedrock_runtime_endpoint, str
- ):
- proxy_endpoint_url = env_aws_bedrock_runtime_endpoint
- elif aws_bedrock_runtime_endpoint is not None and isinstance(
+ if aws_bedrock_runtime_endpoint is not None and isinstance(
aws_bedrock_runtime_endpoint, str
):
proxy_endpoint_url = aws_bedrock_runtime_endpoint
+ elif env_aws_bedrock_runtime_endpoint and isinstance(
+ env_aws_bedrock_runtime_endpoint, str
+ ):
+ proxy_endpoint_url = env_aws_bedrock_runtime_endpoint
else:
proxy_endpoint_url = endpoint_url
@@ -648,6 +928,7 @@ class BaseAWSLLM:
aws_bedrock_runtime_endpoint = optional_params.pop(
"aws_bedrock_runtime_endpoint", None
) # https://bedrock-runtime.{region_name}.amazonaws.com
+ aws_external_id = optional_params.pop("aws_external_id", None)
credentials: Credentials = self.get_credentials(
aws_access_key_id=aws_access_key_id,
@@ -659,6 +940,7 @@ class BaseAWSLLM:
aws_role_name=aws_role_name,
aws_web_identity_token=aws_web_identity_token,
aws_sts_endpoint=aws_sts_endpoint,
+ aws_external_id=aws_external_id,
)
return Boto3CredentialsInfo(
@@ -763,6 +1045,7 @@ class BaseAWSLLM:
aws_profile_name = optional_params.get("aws_profile_name", None)
aws_web_identity_token = optional_params.get("aws_web_identity_token", None)
aws_sts_endpoint = optional_params.get("aws_sts_endpoint", None)
+ aws_external_id = optional_params.get("aws_external_id", None)
aws_region_name = self._get_aws_region_name(
optional_params=optional_params, model=model
)
@@ -777,6 +1060,7 @@ class BaseAWSLLM:
aws_role_name=aws_role_name,
aws_web_identity_token=aws_web_identity_token,
aws_sts_endpoint=aws_sts_endpoint,
+ aws_external_id=aws_external_id,
)
sigv4 = SigV4Auth(credentials, service_name, aws_region_name)
diff --git a/litellm/llms/bedrock/batches/transformation.py b/litellm/llms/bedrock/batches/transformation.py
new file mode 100644
index 00000000000..2f3d00dddda
--- /dev/null
+++ b/litellm/llms/bedrock/batches/transformation.py
@@ -0,0 +1,452 @@
+import os
+import time
+from typing import Any, Dict, List, Literal, Optional, Union, cast
+
+from httpx import Headers, Response
+
+from litellm.llms.base_llm.batches.transformation import BaseBatchesConfig
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.types.llms.bedrock import (
+ BedrockCreateBatchRequest,
+ BedrockCreateBatchResponse,
+ BedrockInputDataConfig,
+ BedrockOutputDataConfig,
+ BedrockS3InputDataConfig,
+ BedrockS3OutputDataConfig,
+)
+from litellm.types.llms.openai import (
+ AllMessageValues,
+ CreateBatchRequest,
+)
+from litellm.types.utils import LiteLLMBatch, LlmProviders
+
+from ..base_aws_llm import BaseAWSLLM
+from ..common_utils import CommonBatchFilesUtils
+
+
+class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
+ """
+ Config for Bedrock Batches - handles batch job creation and management for Bedrock
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.common_utils = CommonBatchFilesUtils()
+
+ @property
+ def custom_llm_provider(self) -> LlmProviders:
+ return LlmProviders.BEDROCK
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ """
+ Validate and prepare environment for Bedrock batch requests.
+ AWS credentials are handled by BaseAWSLLM.
+ """
+ # Add any Bedrock-specific headers if needed
+ return headers
+
+ def get_complete_batch_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: Dict,
+ litellm_params: Dict,
+ data: CreateBatchRequest,
+ ) -> str:
+ """
+ Get the complete URL for Bedrock batch creation.
+ Bedrock batch jobs are created via the model invocation job API.
+ """
+ aws_region_name = self._get_aws_region_name(optional_params, model)
+
+ # Bedrock model invocation job endpoint
+ # Format: https://bedrock.{region}.amazonaws.com/model-invocation-job
+ bedrock_endpoint = f"https://bedrock.{aws_region_name}.amazonaws.com/model-invocation-job"
+
+ return bedrock_endpoint
+
+
+
+
+
+
+
+ def transform_create_batch_request(
+ self,
+ model: str,
+ create_batch_data: CreateBatchRequest,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> Dict[str, Any]:
+ """
+ Transform the batch creation request to Bedrock format.
+
+ Bedrock batch inference requires:
+ - modelId: The Bedrock model ID
+ - jobName: Unique name for the batch job
+ - inputDataConfig: Configuration for input data (S3 location)
+ - outputDataConfig: Configuration for output data (S3 location)
+ - roleArn: IAM role ARN for the batch job
+ """
+ # Get required parameters
+ input_file_id = create_batch_data.get("input_file_id")
+ if not input_file_id:
+ raise ValueError("input_file_id is required for Bedrock batch creation")
+
+ # Extract S3 information from file ID using common utility
+ input_bucket, input_key = self.common_utils.parse_s3_uri(input_file_id)
+
+ # Get output S3 configuration
+ output_bucket = litellm_params.get("s3_output_bucket_name") or os.getenv("AWS_S3_OUTPUT_BUCKET_NAME")
+ if not output_bucket:
+ # Use same bucket as input if no output bucket specified
+ output_bucket = input_bucket
+
+ # Get IAM role ARN
+ role_arn = (
+ litellm_params.get("aws_batch_role_arn")
+ or optional_params.get("aws_batch_role_arn")
+ or os.getenv("AWS_BATCH_ROLE_ARN")
+ )
+ if not role_arn:
+ raise ValueError(
+ "AWS IAM role ARN is required for Bedrock batch jobs. "
+ "Set 'aws_batch_role_arn' in litellm_params or AWS_BATCH_ROLE_ARN env var"
+ )
+
+
+ if not model:
+ raise ValueError("Could not determine Bedrock model ID. Please pass `model` in your request body.")
+
+ # Generate job name with the correct model ID using common utility
+ job_name = self.common_utils.generate_unique_job_name(model, prefix="litellm")
+ output_key = f"litellm-batch-outputs/{job_name}/"
+
+ # Build input data config
+ input_data_config: BedrockInputDataConfig = {
+ "s3InputDataConfig": BedrockS3InputDataConfig(
+ s3Uri=f"s3://{input_bucket}/{input_key}"
+ )
+ }
+
+ # Build output data config
+ output_data_config: BedrockOutputDataConfig = {
+ "s3OutputDataConfig": BedrockS3OutputDataConfig(
+ s3Uri=f"s3://{output_bucket}/{output_key}"
+ )
+ }
+
+ # Create Bedrock batch request with proper typing
+ bedrock_request: BedrockCreateBatchRequest = {
+ "modelId": model,
+ "jobName": job_name,
+ "inputDataConfig": input_data_config,
+ "outputDataConfig": output_data_config,
+ "roleArn": role_arn
+ }
+
+ # Add optional parameters if provided
+ completion_window = create_batch_data.get("completion_window")
+ if completion_window:
+ # Map OpenAI completion window to Bedrock timeout
+ # OpenAI uses "24h", Bedrock expects timeout in hours
+ if completion_window == "24h":
+ bedrock_request["timeoutDurationInHours"] = 24
+
+ # For Bedrock, we need to return a pre-signed request with AWS auth headers
+ # Use common utility for AWS signing
+ endpoint_url = f"https://bedrock.{self._get_aws_region_name(optional_params, model)}.amazonaws.com/model-invocation-job"
+ signed_headers, signed_data = self.common_utils.sign_aws_request(
+ service_name="bedrock",
+ data=bedrock_request,
+ endpoint_url=endpoint_url,
+ optional_params=optional_params,
+ method="POST"
+ )
+
+ # Return a pre-signed request format that the HTTP handler can use
+ return {
+ "method": "POST",
+ "url": endpoint_url,
+ "headers": signed_headers,
+ "data": signed_data.decode('utf-8')
+ }
+
+ def transform_create_batch_response(
+ self,
+ model: Optional[str],
+ raw_response: Response,
+ logging_obj: Any,
+ litellm_params: dict,
+ ) -> LiteLLMBatch:
+ """
+ Transform Bedrock batch creation response to LiteLLM format.
+ """
+ try:
+ response_data: BedrockCreateBatchResponse = raw_response.json()
+ except Exception as e:
+ raise ValueError(f"Failed to parse Bedrock batch response: {e}")
+
+ # Extract information from typed Bedrock response
+ job_arn = response_data.get("jobArn", "")
+ status_str: str = str(response_data.get("status", "Submitted"))
+
+ # Map Bedrock status to OpenAI-compatible status
+ status_mapping: Dict[str, str] = {
+ "Submitted": "validating",
+ "Validating": "validating",
+ "Scheduled": "in_progress",
+ "InProgress": "in_progress",
+ "PartiallyCompleted": "completed",
+ "Completed": "completed",
+ "Failed": "failed",
+ "Stopping": "cancelling",
+ "Stopped": "cancelled",
+ "Expired": "expired",
+ }
+
+ openai_status = cast(Literal["validating", "failed", "in_progress", "finalizing", "completed", "expired", "cancelling", "cancelled"], status_mapping.get(status_str, "validating"))
+
+ # Get original request data from litellm_params if available
+ original_request = litellm_params.get("original_batch_request", {})
+
+ # Create LiteLLM batch object
+ return LiteLLMBatch(
+ id=job_arn, # Use ARN as the batch ID
+ object="batch",
+ endpoint=original_request.get("endpoint", "/v1/chat/completions"),
+ errors=None,
+ input_file_id=original_request.get("input_file_id", ""),
+ completion_window=original_request.get("completion_window", "24h"),
+ status=openai_status,
+ output_file_id=None, # Will be populated when job completes
+ error_file_id=None,
+ created_at=int(time.time()),
+ in_progress_at=int(time.time()) if status_str == "InProgress" else None,
+ expires_at=None,
+ finalizing_at=None,
+ completed_at=None,
+ failed_at=None,
+ expired_at=None,
+ cancelling_at=None,
+ cancelled_at=None,
+ request_counts=None,
+ metadata=original_request.get("metadata", {}),
+ )
+
+ def transform_retrieve_batch_request(
+ self,
+ batch_id: str,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> Dict[str, Any]:
+ """
+ Transform batch retrieval request for Bedrock.
+
+ Args:
+ batch_id: Bedrock job ARN
+ optional_params: Optional parameters
+ litellm_params: LiteLLM parameters
+
+ Returns:
+ Transformed request data for Bedrock GetModelInvocationJob API
+ """
+ # For Bedrock, batch_id should be the full job ARN
+ # The GetModelInvocationJob API expects the full ARN as the identifier
+ if not batch_id.startswith("arn:aws:bedrock:"):
+ raise ValueError(f"Invalid batch_id format. Expected ARN, got: {batch_id}")
+
+ # Extract the job identifier from the ARN - use the full ARN path part
+ # ARN format: arn:aws:bedrock:region:account:model-invocation-job/job-name
+ arn_parts = batch_id.split(":")
+ if len(arn_parts) < 6:
+ raise ValueError(f"Invalid ARN format: {batch_id}")
+
+ region = arn_parts[3]
+ # arn_parts[5] contains "model-invocation-job/{jobId}"
+
+ # Build the endpoint URL for GetModelInvocationJob
+ # AWS API format: GET /model-invocation-job/{jobIdentifier}
+ # Use the FULL ARN as jobIdentifier and URL-encode it (includes ':' and '/')
+ import urllib.parse as _ul
+ encoded_arn = _ul.quote(batch_id, safe="")
+ endpoint_url = f"https://bedrock.{region}.amazonaws.com/model-invocation-job/{encoded_arn}"
+
+ # Use common utility for AWS signing
+ signed_headers, _ = self.common_utils.sign_aws_request(
+ service_name="bedrock",
+ data={}, # GET request has no body
+ endpoint_url=endpoint_url,
+ optional_params=optional_params,
+ method="GET"
+ )
+
+ # Return pre-signed request format
+ return {
+ "method": "GET",
+ "url": endpoint_url,
+ "headers": signed_headers,
+ "data": None
+ }
+
+ def _parse_timestamps_and_status(self, response_data, status_str: str):
+ """Helper to parse timestamps based on status."""
+ import datetime
+ def parse_timestamp(ts_str: Optional[str]) -> Optional[int]:
+ if not ts_str:
+ return None
+ try:
+ dt = datetime.datetime.fromisoformat(ts_str.replace('Z', '+00:00'))
+ return int(dt.timestamp())
+ except Exception:
+ return None
+
+ created_at = parse_timestamp(str(response_data.get("submitTime")) if response_data.get("submitTime") is not None else None)
+ in_progress_states = {"InProgress", "Validating", "Scheduled"}
+ in_progress_at = (
+ parse_timestamp(str(response_data.get("lastModifiedTime")) if response_data.get("lastModifiedTime") is not None else None)
+ if status_str in in_progress_states
+ else None
+ )
+ completed_at = parse_timestamp(str(response_data.get("endTime")) if response_data.get("endTime") is not None else None) if status_str in {"Completed", "PartiallyCompleted"} else None
+ failed_at = parse_timestamp(str(response_data.get("endTime")) if response_data.get("endTime") is not None else None) if status_str == "Failed" else None
+ cancelled_at = parse_timestamp(str(response_data.get("endTime")) if response_data.get("endTime") is not None else None) if status_str == "Stopped" else None
+ expires_at = parse_timestamp(str(response_data.get("jobExpirationTime")) if response_data.get("jobExpirationTime") is not None else None)
+
+ return created_at, in_progress_at, completed_at, failed_at, cancelled_at, expires_at
+
+ def _extract_file_configs(self, response_data):
+ """Helper to extract input and output file configurations."""
+ # Extract input file ID
+ input_file_id = ""
+ input_data_config = response_data.get("inputDataConfig", {})
+ if isinstance(input_data_config, dict):
+ s3_input_config = input_data_config.get("s3InputDataConfig", {})
+ if isinstance(s3_input_config, dict):
+ input_file_id = s3_input_config.get("s3Uri", "")
+
+ # Extract output file ID
+ output_file_id = None
+ output_data_config = response_data.get("outputDataConfig", {})
+ if isinstance(output_data_config, dict):
+ s3_output_config = output_data_config.get("s3OutputDataConfig", {})
+ if isinstance(s3_output_config, dict):
+ output_file_id = s3_output_config.get("s3Uri", "")
+
+ return input_file_id, output_file_id
+
+ def _extract_errors_and_metadata(self, response_data, raw_response):
+ """Helper to extract errors and enriched metadata."""
+ # Extract errors
+ message = response_data.get("message")
+ errors = None
+ if message:
+ from openai.types.batch import Errors
+ from openai.types.batch_error import BatchError
+ errors = Errors(
+ data=[BatchError(message=message, code=str(raw_response.status_code))],
+ object="list"
+ )
+
+ # Enrich metadata with useful Bedrock fields
+ enriched_metadata_raw: Dict[str, Any] = {
+ "jobName": response_data.get("jobName"),
+ "clientRequestToken": response_data.get("clientRequestToken"),
+ "modelId": response_data.get("modelId"),
+ "roleArn": response_data.get("roleArn"),
+ "timeoutDurationInHours": response_data.get("timeoutDurationInHours"),
+ "vpcConfig": response_data.get("vpcConfig"),
+ }
+ import json as _json
+ enriched_metadata: Dict[str, str] = {}
+ for _k, _v in enriched_metadata_raw.items():
+ if _v is None:
+ continue
+ if isinstance(_v, (dict, list)):
+ try:
+ enriched_metadata[_k] = _json.dumps(_v)
+ except Exception:
+ enriched_metadata[_k] = str(_v)
+ else:
+ enriched_metadata[_k] = str(_v)
+
+ return errors, enriched_metadata
+
+ def transform_retrieve_batch_response(
+ self,
+ model: Optional[str],
+ raw_response: Response,
+ logging_obj: Any,
+ litellm_params: dict,
+ ) -> LiteLLMBatch:
+ """
+ Transform Bedrock batch retrieval response to LiteLLM format.
+ """
+ from litellm.types.llms.bedrock import BedrockGetBatchResponse
+ try:
+ response_data: BedrockGetBatchResponse = raw_response.json()
+ except Exception as e:
+ raise ValueError(f"Failed to parse Bedrock batch response: {e}")
+
+ job_arn = response_data.get("jobArn", "")
+ status_str: str = str(response_data.get("status", "Submitted"))
+
+ # Map Bedrock status to OpenAI-compatible status
+ status_mapping: Dict[str, str] = {
+ "Submitted": "validating", "Validating": "validating", "Scheduled": "in_progress",
+ "InProgress": "in_progress", "PartiallyCompleted": "completed", "Completed": "completed",
+ "Failed": "failed", "Stopping": "cancelling", "Stopped": "cancelled", "Expired": "expired"
+ }
+ openai_status = cast(Literal["validating", "failed", "in_progress", "finalizing", "completed", "expired", "cancelling", "cancelled"], status_mapping.get(status_str, "validating"))
+
+ # Parse timestamps
+ created_at, in_progress_at, completed_at, failed_at, cancelled_at, expires_at = self._parse_timestamps_and_status(response_data, status_str)
+
+ # Extract file configurations
+ input_file_id, output_file_id = self._extract_file_configs(response_data)
+
+ # Extract errors and metadata
+ errors, enriched_metadata = self._extract_errors_and_metadata(response_data, raw_response)
+
+ return LiteLLMBatch(
+ id=job_arn,
+ object="batch",
+ endpoint="/v1/chat/completions",
+ errors=errors,
+ input_file_id=input_file_id,
+ completion_window="24h",
+ status=openai_status,
+ output_file_id=output_file_id,
+ error_file_id=None,
+ created_at=created_at or int(time.time()),
+ in_progress_at=in_progress_at,
+ expires_at=expires_at,
+ finalizing_at=None,
+ completed_at=completed_at,
+ failed_at=failed_at,
+ expired_at=None,
+ cancelling_at=None,
+ cancelled_at=cancelled_at,
+ request_counts=None,
+ metadata=enriched_metadata,
+ )
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[Dict, Headers]
+ ) -> BaseLLMException:
+ """
+ Get Bedrock-specific error class using common utility.
+ """
+ return self.common_utils.get_error_class(error_message, status_code, headers)
+
+
diff --git a/litellm/llms/bedrock/chat/converse_handler.py b/litellm/llms/bedrock/chat/converse_handler.py
index 900fad3d043..54c603e5960 100644
--- a/litellm/llms/bedrock/chat/converse_handler.py
+++ b/litellm/llms/bedrock/chat/converse_handler.py
@@ -119,6 +119,7 @@ class BedrockConverseLLM(BaseAWSLLM):
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
+ headers=headers,
)
data = json.dumps(request_data)
@@ -185,8 +186,10 @@ class BedrockConverseLLM(BaseAWSLLM):
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
+ headers=headers,
)
data = json.dumps(request_data)
+
prepped = self.get_request_headers(
credentials=credentials,
aws_region_name=litellm_params.get("aws_region_name") or "us-west-2",
@@ -276,8 +279,13 @@ class BedrockConverseLLM(BaseAWSLLM):
else:
modelId = self.encode_model_id(model_id=model)
- if stream is True and "ai21" in modelId:
- fake_stream = True
+ fake_stream = litellm.AmazonConverseConfig().should_fake_stream(
+ fake_stream=fake_stream,
+ model=model,
+ stream=stream,
+ custom_llm_provider="bedrock",
+ )
+
### SET REGION NAME ###
aws_region_name = self._get_aws_region_name(
@@ -299,6 +307,7 @@ class BedrockConverseLLM(BaseAWSLLM):
) # https://bedrock-runtime.{region_name}.amazonaws.com
aws_web_identity_token = optional_params.pop("aws_web_identity_token", None)
aws_sts_endpoint = optional_params.pop("aws_sts_endpoint", None)
+ aws_external_id = optional_params.pop("aws_external_id", None)
optional_params.pop("aws_region_name", None)
litellm_params[
@@ -315,6 +324,7 @@ class BedrockConverseLLM(BaseAWSLLM):
aws_role_name=aws_role_name,
aws_web_identity_token=aws_web_identity_token,
aws_sts_endpoint=aws_sts_endpoint,
+ aws_external_id=aws_external_id,
)
### SET RUNTIME ENDPOINT ###
@@ -385,8 +395,10 @@ class BedrockConverseLLM(BaseAWSLLM):
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
+ headers=extra_headers,
)
data = json.dumps(_data)
+
prepped = self.get_request_headers(
credentials=credentials,
aws_region_name=aws_region_name,
diff --git a/litellm/llms/bedrock/chat/converse_transformation.py b/litellm/llms/bedrock/chat/converse_transformation.py
index ec378ddbb85..d099c9813d6 100644
--- a/litellm/llms/bedrock/chat/converse_transformation.py
+++ b/litellm/llms/bedrock/chat/converse_transformation.py
@@ -10,9 +10,11 @@ from typing import List, Literal, Optional, Tuple, Union, cast, overload
import httpx
import litellm
+from litellm._logging import verbose_logger
+from litellm.constants import RESPONSE_FORMAT_TOOL_NAME
from litellm.litellm_core_utils.core_helpers import map_finish_reason
from litellm.litellm_core_utils.litellm_logging import Logging
-from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response import (
+from litellm.litellm_core_utils.prompt_templates.common_utils import (
_parse_content_for_reasoning,
)
from litellm.litellm_core_utils.prompt_templates.factory import (
@@ -25,6 +27,7 @@ from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMExcepti
from litellm.types.llms.bedrock import *
from litellm.types.llms.openai import (
AllMessageValues,
+ ChatCompletionAssistantMessage,
ChatCompletionRedactedThinkingBlock,
ChatCompletionResponseMessage,
ChatCompletionSystemMessage,
@@ -47,7 +50,20 @@ from litellm.types.utils import (
)
from litellm.utils import add_dummy_tool, has_tool_call_blocks, supports_reasoning
-from ..common_utils import BedrockError, BedrockModelInfo, get_bedrock_tool_name
+from ..common_utils import (
+ BedrockError,
+ BedrockModelInfo,
+ get_anthropic_beta_from_headers,
+ get_bedrock_tool_name,
+)
+
+# Computer use tool prefixes supported by Bedrock
+BEDROCK_COMPUTER_USE_TOOLS = [
+ "computer_use_preview",
+ "computer_",
+ "bash_",
+ "text_editor_",
+]
class AmazonConverseConfig(BaseConfig):
@@ -86,6 +102,61 @@ class AmazonConverseConfig(BaseConfig):
"performanceConfig": PerformanceConfigBlock,
}
+ @staticmethod
+ def _convert_consecutive_user_messages_to_guarded_text(
+ messages: List[AllMessageValues], optional_params: dict
+ ) -> List[AllMessageValues]:
+ """
+ Convert consecutive user messages at the end to guarded_text type if guardrailConfig is present
+ and no guarded_text is already present in those messages.
+ """
+ # Check if guardrailConfig is present
+ if "guardrailConfig" not in optional_params:
+ return messages
+
+ # Find all consecutive user messages at the end
+ consecutive_user_message_indices = []
+ for i in range(len(messages) - 1, -1, -1):
+ if messages[i].get("role") == "user":
+ consecutive_user_message_indices.append(i)
+ else:
+ break
+
+ if not consecutive_user_message_indices:
+ return messages
+
+ # Process each consecutive user message
+ messages_copy = copy.deepcopy(messages)
+ for user_message_index in consecutive_user_message_indices:
+ user_message = messages_copy[user_message_index]
+ content = user_message.get("content", [])
+
+ if isinstance(content, list):
+ has_guarded_text = any(
+ isinstance(item, dict) and item.get("type") == "guarded_text"
+ for item in content
+ )
+ if has_guarded_text:
+ continue # Skip this message if it already has guarded_text
+
+ # Convert text elements to guarded_text
+ new_content = []
+ for item in content:
+ if isinstance(item, dict) and item.get("type") == "text":
+ new_item = {"type": "guarded_text", "text": item["text"]} # type: ignore
+ new_content.append(new_item)
+ else:
+ new_content.append(item)
+
+ messages_copy[user_message_index]["content"] = new_content # type: ignore
+ elif isinstance(content, str):
+ # If content is a string, convert it to guarded_text
+ messages_copy[user_message_index]["content"] = [ # type: ignore
+ {"type": "guarded_text", "text": content} # type: ignore
+ ]
+
+ return messages_copy
+
@classmethod
def get_config(cls):
return {
@@ -104,6 +175,77 @@ class AmazonConverseConfig(BaseConfig):
and v is not None
}
+ def _validate_request_metadata(self, metadata: dict) -> None:
+ """
+ Validate requestMetadata according to AWS Bedrock Converse API constraints.
+
+ Constraints:
+ - Maximum of 16 items
+ - Keys: 1-256 characters, pattern [a-zA-Z0-9\\s:_@$#=/+,-.]{1,256}
+ - Values: 0-256 characters, pattern [a-zA-Z0-9\\s:_@$#=/+,-.]{0,256}
+ """
+ import re
+
+ if not isinstance(metadata, dict):
+ raise litellm.exceptions.BadRequestError(
+ message="requestMetadata must be a dictionary",
+ model="bedrock",
+ llm_provider="bedrock",
+ )
+
+ if len(metadata) > 16:
+ raise litellm.exceptions.BadRequestError(
+ message="requestMetadata can contain a maximum of 16 items",
+ model="bedrock",
+ llm_provider="bedrock",
+ )
+
+ key_pattern = re.compile(r"^[a-zA-Z0-9\s:_@$#=/+,.-]{1,256}$")
+ value_pattern = re.compile(r"^[a-zA-Z0-9\s:_@$#=/+,.-]{0,256}$")
+
+ for key, value in metadata.items():
+ if not isinstance(key, str):
+ raise litellm.exceptions.BadRequestError(
+ message="requestMetadata keys must be strings",
+ model="bedrock",
+ llm_provider="bedrock",
+ )
+
+ if not isinstance(value, str):
+ raise litellm.exceptions.BadRequestError(
+ message="requestMetadata values must be strings",
+ model="bedrock",
+ llm_provider="bedrock",
+ )
+
+ if len(key) == 0 or len(key) > 256:
+ raise litellm.exceptions.BadRequestError(
+ message="requestMetadata key length must be 1-256 characters",
+ model="bedrock",
+ llm_provider="bedrock",
+ )
+
+ if len(value) > 256:
+ raise litellm.exceptions.BadRequestError(
+ message="requestMetadata value length must be 0-256 characters",
+ model="bedrock",
+ llm_provider="bedrock",
+ )
+
+ if not key_pattern.match(key):
+ raise litellm.exceptions.BadRequestError(
+ message=f"requestMetadata key '{key}' contains invalid characters. Allowed: [a-zA-Z0-9\\s:_@$#=/+,.-]",
+ model="bedrock",
+ llm_provider="bedrock",
+ )
+
+ if not value_pattern.match(value):
+ raise litellm.exceptions.BadRequestError(
+ message=f"requestMetadata value '{value}' contains invalid characters. Allowed: [a-zA-Z0-9\\s:_@$#=/+,.-]",
+ model="bedrock",
+ llm_provider="bedrock",
+ )
+
def get_supported_openai_params(self, model: str) -> List[str]:
from litellm.utils import supports_function_calling
@@ -117,6 +259,7 @@ class AmazonConverseConfig(BaseConfig):
"top_p",
"extra_headers",
"response_format",
+ "requestMetadata",
]
if (
@@ -154,7 +297,9 @@ class AmazonConverseConfig(BaseConfig):
# only anthropic and mistral support tool choice config. otherwise (E.g. cohere) will fail the call - https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ToolChoice.html
supported_params.append("tool_choice")
- if (
+ if "gpt-oss" in model:
+ supported_params.append("reasoning_effort")
+ elif (
"claude-3-7" in model
or "claude-sonnet-4" in model
or "claude-opus-4" in model
@@ -218,10 +363,101 @@ class AmazonConverseConfig(BaseConfig):
+ self.get_supported_video_types()
)
+ def is_computer_use_tool_used(
+ self, tools: Optional[List[OpenAIChatCompletionToolParam]], model: str
+ ) -> bool:
+ """Check if computer use tools are being used in the request."""
+ if tools is None:
+ return False
+
+ for tool in tools:
+ if "type" in tool:
+ tool_type = tool["type"]
+ for computer_use_prefix in BEDROCK_COMPUTER_USE_TOOLS:
+ if tool_type.startswith(computer_use_prefix):
+ return True
+ return False
+
+ def _transform_computer_use_tools(
+ self, computer_use_tools: List[OpenAIChatCompletionToolParam]
+ ) -> List[dict]:
+ """Transform computer use tools to Bedrock format."""
+ transformed_tools: List[dict] = []
+
+ for tool in computer_use_tools:
+ tool_type = tool.get("type", "")
+
+ # Check if this is a computer use tool with the startswith method
+ is_computer_use_tool = False
+ for computer_use_prefix in BEDROCK_COMPUTER_USE_TOOLS:
+ if tool_type.startswith(computer_use_prefix):
+ is_computer_use_tool = True
+ break
+
+ transformed_tool: dict = {}
+ if is_computer_use_tool:
+ if tool_type.startswith("computer_") and "function" in tool:
+ # Computer use tool with function format
+ func = tool["function"]
+ transformed_tool = {
+ "type": tool_type,
+ "name": func.get("name", "computer"),
+ **func.get("parameters", {}),
+ }
+ else:
+ # Direct tools - just need to ensure name is present
+ transformed_tool = dict(tool)
+ if "name" not in transformed_tool:
+ if tool_type.startswith("bash_"):
+ transformed_tool["name"] = "bash"
+ elif tool_type.startswith("text_editor_"):
+ transformed_tool["name"] = "str_replace_editor"
+ else:
+ # Pass through other tools as-is
+ transformed_tool = dict(tool)
+
+ transformed_tools.append(transformed_tool)
+
+ return transformed_tools
+
+ def _separate_computer_use_tools(
+ self, tools: List[OpenAIChatCompletionToolParam], model: str
+ ) -> Tuple[
+ List[OpenAIChatCompletionToolParam], List[OpenAIChatCompletionToolParam]
+ ]:
+ """
+ Separate computer use tools from regular function tools.
+
+ Args:
+ tools: List of tools to separate
+ model: The model name to check if it supports computer use
+
+ Returns:
+ Tuple of (computer_use_tools, regular_tools)
+ """
+ computer_use_tools = []
+ regular_tools = []
+
+ for tool in tools:
+ if "type" in tool:
+ tool_type = tool["type"]
+ is_computer_use_tool = False
+ for computer_use_prefix in BEDROCK_COMPUTER_USE_TOOLS:
+ if tool_type.startswith(computer_use_prefix):
+ is_computer_use_tool = True
+ break
+ if is_computer_use_tool:
+ computer_use_tools.append(tool)
+ else:
+ regular_tools.append(tool)
+ else:
+ regular_tools.append(tool)
+
+ return computer_use_tools, regular_tools
+
def _create_json_tool_call_for_response_format(
self,
json_schema: Optional[dict] = None,
- schema_name: str = "json_tool_call",
description: Optional[str] = None,
) -> ChatCompletionToolParam:
"""
@@ -243,10 +479,12 @@ class AmazonConverseConfig(BaseConfig):
"properties": {},
}
else:
+ # Use the schema as-is for Bedrock
+ # Bedrock requires the tool schema to be of type "object" and doesn't need unwrapping
_input_schema = json_schema
tool_param_function_chunk = ChatCompletionToolParamFunctionChunk(
- name=schema_name, parameters=_input_schema
+ name=RESPONSE_FORMAT_TOOL_NAME, parameters=_input_schema
)
if description:
tool_param_function_chunk["description"] = description
@@ -285,56 +523,13 @@ class AmazonConverseConfig(BaseConfig):
for param, value in non_default_params.items():
if param == "response_format" and isinstance(value, dict):
- ignore_response_format_types = ["text"]
- if value["type"] in ignore_response_format_types: # value is a no-op
- continue
-
- json_schema: Optional[dict] = None
- schema_name: str = ""
- description: Optional[str] = None
- if "response_schema" in value:
- json_schema = value["response_schema"]
- schema_name = "json_tool_call"
- elif "json_schema" in value:
- json_schema = value["json_schema"]["schema"]
- schema_name = value["json_schema"]["name"]
- description = value["json_schema"].get("description")
-
- if "type" in value and value["type"] == "text":
- continue
-
- """
- Follow similar approach to anthropic - translate to a single tool call.
-
- When using tools in this way: - https://docs.anthropic.com/en/docs/build-with-claude/tool-use#json-mode
- - You usually want to provide a single tool
- - You should set tool_choice (see Forcing tool use) to instruct the model to explicitly use that tool
- - Remember that the model will pass the input to the tool, so the name of the tool and description should be from the model’s perspective.
- """
- _tool = self._create_json_tool_call_for_response_format(
- json_schema=json_schema,
- schema_name=schema_name if schema_name != "" else "json_tool_call",
- description=description,
+ optional_params = self._translate_response_format_param(
+ value=value,
+ model=model,
+ optional_params=optional_params,
+ non_default_params=non_default_params,
+ is_thinking_enabled=is_thinking_enabled,
)
- optional_params = self._add_tools_to_optional_params(
- optional_params=optional_params, tools=[_tool]
- )
-
- if (
- litellm.utils.supports_tool_choice(
- model=model, custom_llm_provider=self.custom_llm_provider
- )
- and not is_thinking_enabled
- ):
-
- optional_params["tool_choice"] = ToolChoiceValuesBlock(
- tool=SpecificToolChoiceBlock(
- name=schema_name if schema_name != "" else "json_tool_call"
- )
- )
- optional_params["json_mode"] = True
- if non_default_params.get("stream", False) is True:
- optional_params["fake_stream"] = True
if param == "max_tokens" or param == "max_completion_tokens":
optional_params["maxTokens"] = value
if param == "stream":
@@ -365,13 +560,84 @@ class AmazonConverseConfig(BaseConfig):
if param == "thinking":
optional_params["thinking"] = value
elif param == "reasoning_effort" and isinstance(value, str):
- optional_params["thinking"] = AnthropicConfig._map_reasoning_effort(
- value
- )
+ if "gpt-oss" in model:
+ # GPT-OSS models: keep reasoning_effort as-is
+ # It will be passed through to additionalModelRequestFields
+ optional_params["reasoning_effort"] = value
+ else:
+ # Anthropic and other models: convert to thinking parameter
+ optional_params["thinking"] = AnthropicConfig._map_reasoning_effort(
+ value
+ )
+ if param == "requestMetadata":
+ if value is not None and isinstance(value, dict):
+ self._validate_request_metadata(value) # type: ignore
+ optional_params["requestMetadata"] = value
- self.update_optional_params_with_thinking_tokens(
- non_default_params=non_default_params, optional_params=optional_params
+ # Only update thinking tokens for non-GPT-OSS models
+ if "gpt-oss" not in model:
+ self.update_optional_params_with_thinking_tokens(
+ non_default_params=non_default_params, optional_params=optional_params
+ )
+
+ return optional_params
+
+ def _translate_response_format_param(
+ self,
+ value: dict,
+ model: str,
+ optional_params: dict,
+ non_default_params: dict,
+ is_thinking_enabled: bool,
+ ) -> dict:
+ """
+ Handles translation of response_format parameter to Bedrock format.
+
+ Returns `optional_params` with the translated response_format parameter.
+ """
+ ignore_response_format_types = ["text"]
+ if value["type"] in ignore_response_format_types: # value is a no-op
+ return optional_params
+
+ json_schema: Optional[dict] = None
+ description: Optional[str] = None
+ if "response_schema" in value:
+ json_schema = value["response_schema"]
+ elif "json_schema" in value:
+ json_schema = value["json_schema"]["schema"]
+ description = value["json_schema"].get("description")
+
+ if "type" in value and value["type"] == "text":
+ return optional_params
+
+ """
+ Follow similar approach to anthropic - translate to a single tool call.
+
+ When using tools in this way: - https://docs.anthropic.com/en/docs/build-with-claude/tool-use#json-mode
+ - You usually want to provide a single tool
+ - You should set tool_choice (see Forcing tool use) to instruct the model to explicitly use that tool
+ - Remember that the model will pass the input to the tool, so the name of the tool and description should be from the model’s perspective.
+ """
+ _tool = self._create_json_tool_call_for_response_format(
+ json_schema=json_schema,
+ description=description,
)
+ optional_params = self._add_tools_to_optional_params(
+ optional_params=optional_params, tools=[_tool]
+ )
+
+ if (
+ litellm.utils.supports_tool_choice(
+ model=model, custom_llm_provider=self.custom_llm_provider
+ )
+ and not is_thinking_enabled
+ ):
+ optional_params["tool_choice"] = ToolChoiceValuesBlock(
+ tool=SpecificToolChoiceBlock(name=RESPONSE_FORMAT_TOOL_NAME)
+ )
+ optional_params["json_mode"] = True
+ if non_default_params.get("stream", False) is True:
+ optional_params["fake_stream"] = True
return optional_params
@@ -405,6 +671,7 @@ class AmazonConverseConfig(BaseConfig):
OpenAIMessageContentListBlock,
ChatCompletionUserMessage,
ChatCompletionSystemMessage,
+ ChatCompletionAssistantMessage,
],
block_type: Literal["system"],
) -> Optional[SystemContentBlock]:
@@ -417,6 +684,7 @@ class AmazonConverseConfig(BaseConfig):
OpenAIMessageContentListBlock,
ChatCompletionUserMessage,
ChatCompletionSystemMessage,
+ ChatCompletionAssistantMessage,
],
block_type: Literal["content_block"],
) -> Optional[ContentBlock]:
@@ -428,6 +696,7 @@ class AmazonConverseConfig(BaseConfig):
OpenAIMessageContentListBlock,
ChatCompletionUserMessage,
ChatCompletionSystemMessage,
+ ChatCompletionAssistantMessage,
],
block_type: Literal["system", "content_block"],
) -> Optional[Union[SystemContentBlock, ContentBlock]]:
@@ -493,12 +762,101 @@ class AmazonConverseConfig(BaseConfig):
return {}
+ def _prepare_request_params(
+ self, optional_params: dict, model: str
+ ) -> Tuple[dict, dict, dict]:
+ """Prepare and separate request parameters."""
+ inference_params = copy.deepcopy(optional_params)
+ supported_converse_params = list(
+ AmazonConverseConfig.__annotations__.keys()
+ ) + ["top_k"]
+ supported_tool_call_params = ["tools", "tool_choice"]
+ supported_config_params = list(self.get_config_blocks().keys())
+ total_supported_params = (
+ supported_converse_params
+ + supported_tool_call_params
+ + supported_config_params
+ )
+ inference_params.pop("json_mode", None) # used for handling json_schema
+
+ # Extract requestMetadata before processing other parameters
+ request_metadata = inference_params.pop("requestMetadata", None)
+ if request_metadata is not None:
+ self._validate_request_metadata(request_metadata)
+
+ # keep supported params in 'inference_params', and set all model-specific params in 'additional_request_params'
+ additional_request_params = {
+ k: v for k, v in inference_params.items() if k not in total_supported_params
+ }
+ inference_params = {
+ k: v for k, v in inference_params.items() if k in total_supported_params
+ }
+
+ # Only set the topK value in for models that support it
+ additional_request_params.update(
+ self._handle_top_k_value(model, inference_params)
+ )
+
+ return inference_params, additional_request_params, request_metadata
+
+ def _process_tools_and_beta(
+ self,
+ original_tools: list,
+ model: str,
+ headers: Optional[dict],
+ additional_request_params: dict,
+ ) -> Tuple[List[ToolBlock], list]:
+ """Process tools and collect anthropic_beta values."""
+ bedrock_tools: List[ToolBlock] = []
+
+ # Collect anthropic_beta values from user headers
+ anthropic_beta_list = []
+ if headers:
+ user_betas = get_anthropic_beta_from_headers(headers)
+ anthropic_beta_list.extend(user_betas)
+
+ # Only separate tools if computer use tools are actually present
+ if original_tools and self.is_computer_use_tool_used(original_tools, model):
+ # Separate computer use tools from regular function tools
+ computer_use_tools, regular_tools = self._separate_computer_use_tools(
+ original_tools, model
+ )
+
+ # Process regular function tools using existing logic
+ bedrock_tools = _bedrock_tools_pt(regular_tools)
+
+ # Add computer use tools and anthropic_beta if needed (only when computer use tools are present)
+ if computer_use_tools:
+ anthropic_beta_list.append("computer-use-2024-10-22")
+ # Transform computer use tools to proper Bedrock format
+ transformed_computer_tools = self._transform_computer_use_tools(
+ computer_use_tools
+ )
+ additional_request_params["tools"] = transformed_computer_tools
+ else:
+ # No computer use tools, process all tools as regular tools
+ bedrock_tools = _bedrock_tools_pt(original_tools)
+
+ # Set anthropic_beta in additional_request_params if we have any beta features
+ if anthropic_beta_list:
+ # Remove duplicates while preserving order
+ unique_betas = []
+ seen = set()
+ for beta in anthropic_beta_list:
+ if beta not in seen:
+ unique_betas.append(beta)
+ seen.add(beta)
+ additional_request_params["anthropic_beta"] = unique_betas
+
+ return bedrock_tools, anthropic_beta_list
+
def _transform_request_helper(
self,
model: str,
system_content_blocks: List[SystemContentBlock],
optional_params: dict,
messages: Optional[List[AllMessageValues]] = None,
+ headers: Optional[dict] = None,
) -> CommonRequestObject:
## VALIDATE REQUEST
"""
@@ -520,35 +878,18 @@ class AmazonConverseConfig(BaseConfig):
llm_provider="bedrock",
)
- inference_params = copy.deepcopy(optional_params)
- supported_converse_params = list(
- AmazonConverseConfig.__annotations__.keys()
- ) + ["top_k"]
- supported_tool_call_params = ["tools", "tool_choice"]
- supported_config_params = list(self.get_config_blocks().keys())
- total_supported_params = (
- supported_converse_params
- + supported_tool_call_params
- + supported_config_params
- )
- inference_params.pop("json_mode", None) # used for handling json_schema
-
- # keep supported params in 'inference_params', and set all model-specific params in 'additional_request_params'
- additional_request_params = {
- k: v for k, v in inference_params.items() if k not in total_supported_params
- }
- inference_params = {
- k: v for k, v in inference_params.items() if k in total_supported_params
- }
-
- # Only set the topK value in for models that support it
- additional_request_params.update(
- self._handle_top_k_value(model, inference_params)
+ # Prepare and separate parameters
+ inference_params, additional_request_params, request_metadata = (
+ self._prepare_request_params(optional_params, model)
)
- bedrock_tools: List[ToolBlock] = _bedrock_tools_pt(
- inference_params.pop("tools", [])
+ original_tools = inference_params.pop("tools", [])
+
+ # Process tools and collect beta values
+ bedrock_tools, anthropic_beta_list = self._process_tools_and_beta(
+ original_tools, model, headers, additional_request_params
)
+
bedrock_tool_config: Optional[ToolConfigBlock] = None
if len(bedrock_tools) > 0:
tool_choice_values: ToolChoiceValuesBlock = inference_params.pop(
@@ -578,6 +919,10 @@ class AmazonConverseConfig(BaseConfig):
if bedrock_tool_config is not None:
data["toolConfig"] = bedrock_tool_config
+ # Request Metadata (top-level field)
+ if request_metadata is not None:
+ data["requestMetadata"] = request_metadata
+
return data
async def _async_transform_request(
@@ -586,8 +931,14 @@ class AmazonConverseConfig(BaseConfig):
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
+ headers: Optional[dict] = None,
) -> RequestObject:
messages, system_content_blocks = self._transform_system_message(messages)
+
+ # Convert last user message to guarded_text if guardrailConfig is present
+ messages = self._convert_consecutive_user_messages_to_guarded_text(
+ messages, optional_params
+ )
## TRANSFORMATION ##
_data: CommonRequestObject = self._transform_request_helper(
@@ -595,6 +946,7 @@ class AmazonConverseConfig(BaseConfig):
system_content_blocks=system_content_blocks,
optional_params=optional_params,
messages=messages,
+ headers=headers,
)
bedrock_messages = (
@@ -625,6 +977,7 @@ class AmazonConverseConfig(BaseConfig):
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
+ headers=headers,
),
)
@@ -634,14 +987,21 @@ class AmazonConverseConfig(BaseConfig):
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
+ headers: Optional[dict] = None,
) -> RequestObject:
messages, system_content_blocks = self._transform_system_message(messages)
+ # Convert last user message to guarded_text if guardrailConfig is present
+ messages = self._convert_consecutive_user_messages_to_guarded_text(
+ messages, optional_params
+ )
+
_data: CommonRequestObject = self._transform_request_helper(
model=model,
system_content_blocks=system_content_blocks,
optional_params=optional_params,
messages=messages,
+ headers=headers,
)
## TRANSFORMATION ##
@@ -732,10 +1092,8 @@ class AmazonConverseConfig(BaseConfig):
cache_read_input_tokens = usage["cacheReadInputTokens"]
input_tokens += cache_read_input_tokens
if "cacheWriteInputTokens" in usage:
- """
- Do not increment prompt_tokens with cacheWriteInputTokens
- """
cache_creation_input_tokens = usage["cacheWriteInputTokens"]
+ input_tokens += cache_creation_input_tokens
prompt_tokens_details = PromptTokensDetailsWrapper(
cached_tokens=cache_read_input_tokens
@@ -969,10 +1327,36 @@ class AmazonConverseConfig(BaseConfig):
self._transform_thinking_blocks(reasoningContentBlocks)
)
chat_completion_message["content"] = content_str
- if json_mode is True and tools is not None and len(tools) == 1:
- # to support 'json_schema' logic on bedrock models
+ if (
+ json_mode is True
+ and tools is not None
+ and len(tools) == 1
+ and tools[0]["function"].get("name") == RESPONSE_FORMAT_TOOL_NAME
+ ):
+ verbose_logger.debug(
+ "Processing JSON tool call response for response_format"
+ )
json_mode_content_str: Optional[str] = tools[0]["function"].get("arguments")
if json_mode_content_str is not None:
+ import json
+
+ # Bedrock returns the response wrapped in a "properties" object
+ # We need to extract the actual content from this wrapper
+ try:
+ response_data = json.loads(json_mode_content_str)
+
+ # If Bedrock wrapped the response in "properties", extract the content
+ if (
+ isinstance(response_data, dict)
+ and "properties" in response_data
+ and len(response_data) == 1
+ ):
+ response_data = response_data["properties"]
+ json_mode_content_str = json.dumps(response_data)
+ except json.JSONDecodeError:
+ # If parsing fails, use the original response
+ pass
+
chat_completion_message["content"] = json_mode_content_str
else:
chat_completion_message["tool_calls"] = tools
@@ -1032,3 +1416,36 @@ class AmazonConverseConfig(BaseConfig):
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
return headers
+
+ def should_fake_stream(
+ self,
+ model: Optional[str],
+ stream: Optional[bool],
+ custom_llm_provider: Optional[str] = None,
+ fake_stream: Optional[bool] = None,
+ ) -> bool:
+ """
+ Returns True if the model/provider should fake stream
+ """
+ ###################################################################
+ # If an upstream method already set fake_stream to True, return True
+ ###################################################################
+ if fake_stream is True:
+ return True
+
+ ###################################################################
+ # Bedrock Converse Specific Logic
+ ###################################################################
+ if stream is True:
+ if model is not None:
+ ###################################################################
+ # GPT-OSS models do not support streaming
+ ###################################################################
+ if "gpt-oss" in model:
+ return True
+ ###################################################################
+ # AI21 models do not support streaming
+ ###################################################################
+ if "ai21" in model:
+ return True
+ return False
diff --git a/litellm/llms/bedrock/chat/invoke_agent/transformation.py b/litellm/llms/bedrock/chat/invoke_agent/transformation.py
index e4ff6d398ea..2c7135f4d83 100644
--- a/litellm/llms/bedrock/chat/invoke_agent/transformation.py
+++ b/litellm/llms/bedrock/chat/invoke_agent/transformation.py
@@ -3,14 +3,15 @@ Transformation for Bedrock Invoke Agent
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_InvokeAgent.html
"""
+
import base64
import json
-import uuid
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
import httpx
from litellm._logging import verbose_logger
+from litellm._uuid import uuid
from litellm.litellm_core_utils.prompt_templates.common_utils import (
convert_content_list_to_str,
)
@@ -22,6 +23,11 @@ from litellm.types.llms.bedrock_invoke_agents import (
InvokeAgentEvent,
InvokeAgentEventHeaders,
InvokeAgentEventList,
+ InvokeAgentMetadata,
+ InvokeAgentModelInvocationInput,
+ InvokeAgentModelInvocationOutput,
+ InvokeAgentOrchestrationTrace,
+ InvokeAgentPreProcessingTrace,
InvokeAgentTrace,
InvokeAgentTracePayload,
InvokeAgentUsage,
@@ -389,15 +395,22 @@ class AmazonInvokeAgentConfig(BaseConfig, BaseAWSLLM):
self, trace_data: InvokeAgentTrace, usage_info: InvokeAgentUsage
) -> None:
"""Extract usage information from preprocessing trace."""
- pre_processing = trace_data.get("preProcessingTrace", {})
+ pre_processing: Optional[InvokeAgentPreProcessingTrace] = trace_data.get(
+ "preProcessingTrace"
+ )
if not pre_processing:
return
- model_output = pre_processing.get("modelInvocationOutput", {})
+ model_output: Optional[InvokeAgentModelInvocationOutput] = (
+ pre_processing.get("modelInvocationOutput")
+ or InvokeAgentModelInvocationOutput()
+ )
if not model_output:
return
- metadata = model_output.get("metadata", {})
+ metadata: Optional[InvokeAgentMetadata] = (
+ model_output.get("metadata") or InvokeAgentMetadata()
+ )
if not metadata:
return
@@ -412,11 +425,16 @@ class AmazonInvokeAgentConfig(BaseConfig, BaseAWSLLM):
self, trace_data: InvokeAgentTrace
) -> Optional[str]:
"""Extract model information from orchestration trace."""
- orchestration_trace = trace_data.get("orchestrationTrace", {})
+ orchestration_trace: Optional[InvokeAgentOrchestrationTrace] = trace_data.get(
+ "orchestrationTrace"
+ )
if not orchestration_trace:
return None
- model_invocation = orchestration_trace.get("modelInvocationInput", {})
+ model_invocation: Optional[InvokeAgentModelInvocationInput] = (
+ orchestration_trace.get("modelInvocationInput")
+ or InvokeAgentModelInvocationInput()
+ )
if not model_invocation:
return None
diff --git a/litellm/llms/bedrock/chat/invoke_handler.py b/litellm/llms/bedrock/chat/invoke_handler.py
index e33535043d5..71aadffe5bb 100644
--- a/litellm/llms/bedrock/chat/invoke_handler.py
+++ b/litellm/llms/bedrock/chat/invoke_handler.py
@@ -7,7 +7,6 @@ import json
import time
import types
import urllib.parse
-import uuid
from functools import partial
from typing import (
Any,
@@ -26,6 +25,7 @@ import httpx # type: ignore
import litellm
from litellm import verbose_logger
+from litellm._uuid import uuid
from litellm.caching.caching import InMemoryCache
from litellm.litellm_core_utils.core_helpers import map_finish_reason
from litellm.litellm_core_utils.litellm_logging import Logging
@@ -498,9 +498,9 @@ class BedrockLLM(BaseAWSLLM):
content=None,
)
model_response.choices[0].message = _message # type: ignore
- model_response._hidden_params[
- "original_response"
- ] = outputText # allow user to access raw anthropic tool calling response
+ model_response._hidden_params["original_response"] = (
+ outputText # allow user to access raw anthropic tool calling response
+ )
if (
_is_function_call is True
and stream is not None
@@ -808,9 +808,9 @@ class BedrockLLM(BaseAWSLLM):
): # completion(top_k=3) > anthropic_config(top_k=3) <- allows for dynamic variables to be passed in
inference_params[k] = v
if stream is True:
- inference_params[
- "stream"
- ] = True # cohere requires stream = True in inference params
+ inference_params["stream"] = (
+ True # cohere requires stream = True in inference params
+ )
data = json.dumps({"prompt": prompt, **inference_params})
elif provider == "anthropic":
if model.startswith("anthropic.claude-3"):
@@ -831,7 +831,7 @@ class BedrockLLM(BaseAWSLLM):
model=model, messages=messages, custom_llm_provider="anthropic_xml"
) # type: ignore
## LOAD CONFIG
- config = litellm.AmazonAnthropicClaude3Config.get_config()
+ config = litellm.AmazonAnthropicClaudeConfig.get_config()
for k, v in config.items():
if (
k not in inference_params
@@ -1352,9 +1352,11 @@ class AWSEventStreamDecoder:
"name": None,
"arguments": delta_obj["toolUse"]["input"],
},
- "index": self.tool_calls_index
- if self.tool_calls_index is not None
- else index,
+ "index": (
+ self.tool_calls_index
+ if self.tool_calls_index is not None
+ else index
+ ),
}
elif "reasoningContent" in delta_obj:
provider_specific_fields = {
@@ -1384,7 +1386,11 @@ class AWSEventStreamDecoder:
"name": None,
"arguments": "{}",
},
- "index": chunk_data["contentBlockIndex"],
+ "index": (
+ self.tool_calls_index
+ if self.tool_calls_index is not None
+ else index
+ ),
}
elif "stopReason" in chunk_data:
finish_reason = map_finish_reason(chunk_data.get("stopReason", "stop"))
@@ -1446,7 +1452,7 @@ class AWSEventStreamDecoder:
######### /bedrock/invoke nova mappings ###############
elif "contentBlockDelta" in chunk_data:
# when using /bedrock/invoke/nova, the chunk_data is nested under "contentBlockDelta"
- _chunk_data = chunk_data.get("contentBlockDelta", None)
+ _chunk_data = chunk_data.get("contentBlockDelta", {})
return self.converse_chunk_parser(chunk_data=_chunk_data)
######## bedrock.mistral mappings ###############
elif "outputs" in chunk_data:
diff --git a/litellm/llms/bedrock/chat/invoke_transformations/amazon_deepseek_transformation.py b/litellm/llms/bedrock/chat/invoke_transformations/amazon_deepseek_transformation.py
index d7ceec1f1c1..0fe84b0ce0c 100644
--- a/litellm/llms/bedrock/chat/invoke_transformations/amazon_deepseek_transformation.py
+++ b/litellm/llms/bedrock/chat/invoke_transformations/amazon_deepseek_transformation.py
@@ -3,7 +3,7 @@ from typing import Any, List, Optional, cast
from httpx import Response
from litellm import verbose_logger
-from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response import (
+from litellm.litellm_core_utils.prompt_templates.common_utils import (
_parse_content_for_reasoning,
)
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
diff --git a/litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py b/litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
index 738490aa7bb..9b13d3df08e 100644
--- a/litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
+++ b/litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py
@@ -6,6 +6,7 @@ from litellm.llms.anthropic.chat.transformation import AnthropicConfig
from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation import (
AmazonInvokeConfig,
)
+from litellm.llms.bedrock.common_utils import get_anthropic_beta_from_headers
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
@@ -17,13 +18,22 @@ else:
LiteLLMLoggingObj = Any
-class AmazonAnthropicClaude3Config(AmazonInvokeConfig, AnthropicConfig):
+class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
"""
Reference:
https://us-west-2.console.aws.amazon.com/bedrock/home?region=us-west-2#/providers?model=claude
https://docs.anthropic.com/claude/docs/models-overview#model-comparison
+ https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages-request-response.html
- Supported Params for the Amazon / Anthropic Claude 3 models:
+ Supported Params for the Amazon / Anthropic Claude models (Claude 3, Claude 4, etc.):
+ Supports anthropic_beta parameter for beta features like:
+ - computer-use-2025-01-24 (Claude 3.7 Sonnet)
+ - computer-use-2024-10-22 (Claude 3.5 Sonnet v2)
+ - token-efficient-tools-2025-02-19 (Claude 3.7 Sonnet)
+ - interleaved-thinking-2025-05-14 (Claude 4 models)
+ - output-128k-2025-02-19 (Claude 3.7 Sonnet)
+ - dev-full-thinking-2025-05-14 (Claude 4 models)
+ - context-1m-2025-08-07 (Claude Sonnet 4)
"""
anthropic_version: str = "bedrock-2023-05-31"
@@ -50,6 +60,7 @@ class AmazonAnthropicClaude3Config(AmazonInvokeConfig, AnthropicConfig):
drop_params,
)
+
def transform_request(
self,
model: str,
@@ -72,6 +83,11 @@ class AmazonAnthropicClaude3Config(AmazonInvokeConfig, AnthropicConfig):
if "anthropic_version" not in _anthropic_request:
_anthropic_request["anthropic_version"] = self.anthropic_version
+ # Handle anthropic_beta from user headers
+ anthropic_beta_list = get_anthropic_beta_from_headers(headers)
+ if anthropic_beta_list:
+ _anthropic_request["anthropic_beta"] = anthropic_beta_list
+
return _anthropic_request
def transform_response(
diff --git a/litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py b/litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py
index 16f146206b1..08a0690716b 100644
--- a/litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py
+++ b/litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py
@@ -190,13 +190,15 @@ class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
] = True # cohere requires stream = True in inference params
request_data = {"prompt": prompt, **inference_params}
elif provider == "anthropic":
- return litellm.AmazonAnthropicClaude3Config().transform_request(
+ transformed_request = litellm.AmazonAnthropicClaudeConfig().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers,
)
+
+ return transformed_request
elif provider == "nova":
return litellm.AmazonInvokeNovaConfig().transform_request(
model=model,
@@ -293,7 +295,7 @@ class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
completion_response["generations"][0]["finish_reason"]
)
elif provider == "anthropic":
- return litellm.AmazonAnthropicClaude3Config().transform_response(
+ return litellm.AmazonAnthropicClaudeConfig().transform_response(
model=model,
raw_response=raw_response,
model_response=model_response,
diff --git a/litellm/llms/bedrock/common_utils.py b/litellm/llms/bedrock/common_utils.py
index 2a8fdc148bd..1a599fda59f 100644
--- a/litellm/llms/bedrock/common_utils.py
+++ b/litellm/llms/bedrock/common_utils.py
@@ -4,11 +4,17 @@ Common utilities used across bedrock chat/embedding/image generation
import json
import os
-from typing import TYPE_CHECKING, List, Literal, Optional, Union
+from typing import TYPE_CHECKING, Dict, List, Literal, Optional, Union
+
+if TYPE_CHECKING:
+ from litellm.types.llms.bedrock import BedrockCreateBatchRequest
import httpx
import litellm
+from litellm.llms.base_llm.anthropic_messages.transformation import (
+ BaseAnthropicMessagesConfig,
+)
from litellm.llms.base_llm.base_utils import BaseLLMModelInfo
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.secret_managers.main import get_secret
@@ -434,32 +440,103 @@ class BedrockModelInfo(BaseLLMModelInfo):
"""
Abbreviations of regions AWS Bedrock supports for cross region inference
"""
- return ["us", "eu", "apac"]
+ return ["global", "us", "eu", "apac", "jp", "au"]
@staticmethod
def get_bedrock_route(
model: str,
- ) -> Literal["converse", "invoke", "converse_like", "agent"]:
+ ) -> Literal["converse", "invoke", "converse_like", "agent", "async_invoke"]:
"""
Get the bedrock route for the given model.
"""
+ route_mappings: Dict[
+ str, Literal["invoke", "converse_like", "converse", "agent", "async_invoke"]
+ ] = {
+ "invoke/": "invoke",
+ "converse_like/": "converse_like",
+ "converse/": "converse",
+ "agent/": "agent",
+ "async_invoke/": "async_invoke",
+ }
+
+ # Check explicit routes first
+ for prefix, route_type in route_mappings.items():
+ if prefix in model:
+ return route_type
+
base_model = BedrockModelInfo.get_base_model(model)
alt_model = BedrockModelInfo.get_non_litellm_routing_model_name(model=model)
- if "invoke/" in model:
- return "invoke"
- elif "converse_like" in model:
- return "converse_like"
- elif "converse/" in model:
- return "converse"
- elif "agent/" in model:
- return "agent"
- elif (
+ if (
base_model in litellm.bedrock_converse_models
or alt_model in litellm.bedrock_converse_models
):
return "converse"
return "invoke"
+ @staticmethod
+ def _explicit_converse_route(model: str) -> bool:
+ """
+ Check if the model is an explicit converse route.
+ """
+ return "converse/" in model
+
+ @staticmethod
+ def _explicit_invoke_route(model: str) -> bool:
+ """
+ Check if the model is an explicit invoke route.
+ """
+ return "invoke/" in model
+
+ @staticmethod
+ def _explicit_agent_route(model: str) -> bool:
+ """
+ Check if the model is an explicit agent route.
+ """
+ return "agent/" in model
+
+ @staticmethod
+ def _explicit_converse_like_route(model: str) -> bool:
+ """
+ Check if the model is an explicit converse like route.
+ """
+ return "converse_like/" in model
+
+ @staticmethod
+ def _explicit_async_invoke_route(model: str) -> bool:
+ """
+ Check if the model is an explicit async invoke route.
+ """
+ return "async_invoke/" in model
+
+ @staticmethod
+ def get_bedrock_provider_config_for_messages_api(
+ model: str,
+ ) -> Optional[BaseAnthropicMessagesConfig]:
+ """
+ Get the bedrock provider config for the given model.
+
+ Only route to AmazonAnthropicClaude3MessagesConfig() for BaseMessagesConfig
+
+ All other routes should return None since they will go through litellm.completion
+ """
+
+ #########################################################
+ # Converse routes should go through litellm.completion()
+ if BedrockModelInfo._explicit_converse_route(model):
+ return None
+
+ #########################################################
+ # This goes through litellm.AmazonAnthropicClaude3MessagesConfig()
+ # Since bedrock Invoke supports Native Anthropic Messages API
+ #########################################################
+ if "claude" in model:
+ return litellm.AmazonAnthropicClaudeMessagesConfig()
+
+ #########################################################
+ # These routes will go through litellm.completion()
+ #########################################################
+ return None
+
class BedrockEventStreamDecoderBase:
"""
@@ -524,3 +601,257 @@ class BedrockEventStreamDecoderBase:
return None
return chunk.decode() # type: ignore[no-any-return]
+
+
+def get_anthropic_beta_from_headers(headers: dict) -> List[str]:
+ """
+ Extract anthropic-beta header values and convert them to a list.
+ Supports comma-separated values from user headers.
+
+ Used by both converse and invoke transformations for consistent handling
+ of anthropic-beta headers that should be passed to AWS Bedrock.
+
+ Args:
+ headers (dict): Request headers dictionary
+
+ Returns:
+ List[str]: List of anthropic beta feature strings, empty list if no header
+ """
+ anthropic_beta_header = headers.get("anthropic-beta")
+ if not anthropic_beta_header:
+ return []
+
+ # Split comma-separated values and strip whitespace
+ return [beta.strip() for beta in anthropic_beta_header.split(",")]
+
+
+class CommonBatchFilesUtils:
+ """
+ Common utilities for Bedrock batch and file operations.
+ Provides shared functionality to reduce code duplication between batches and files.
+ """
+
+ def __init__(self):
+ # Import here to avoid circular imports
+ from .base_aws_llm import BaseAWSLLM
+
+ self._base_aws = BaseAWSLLM()
+
+ def get_bedrock_model_id_from_litellm_model(self, model: str) -> str:
+ """
+ Extract the actual Bedrock model ID from LiteLLM model name.
+
+ Args:
+ model: LiteLLM model name (e.g., "bedrock/anthropic.claude-3-sonnet-20240229-v1:0")
+
+ Returns:
+ Bedrock model ID (e.g., "anthropic.claude-3-sonnet-20240229-v1:0")
+ """
+ if model.startswith("bedrock/"):
+ return model[8:] # Remove "bedrock/" prefix
+ return model
+
+ def parse_s3_uri(self, s3_uri: str) -> tuple:
+ """
+ Parse S3 URI into bucket and key components.
+
+ Args:
+ s3_uri: S3 URI (e.g., "s3://bucket/key/path")
+
+ Returns:
+ Tuple of (bucket, key)
+
+ Raises:
+ ValueError: If URI format is invalid
+ """
+ if not s3_uri.startswith("s3://"):
+ raise ValueError(f"Invalid S3 URI format: {s3_uri}")
+
+ s3_parts = s3_uri[5:].split("/", 1) # Remove "s3://" and split on first "/"
+ if len(s3_parts) != 2:
+ raise ValueError(f"Invalid S3 URI format: {s3_uri}")
+
+ return s3_parts[0], s3_parts[1] # bucket, key
+
+ def extract_model_from_s3_file_path(
+ self, s3_uri: str, optional_params: dict
+ ) -> str:
+ """
+ Extract model ID from S3 file path.
+
+ The Bedrock file transformation creates S3 objects with the model name embedded:
+ Format: s3://bucket/litellm-bedrock-files-{model}-{uuid}.jsonl
+ """
+ # Check if model is provided in optional_params first
+ if "model" in optional_params and optional_params["model"]:
+ return self.get_bedrock_model_id_from_litellm_model(
+ optional_params["model"]
+ )
+
+ # Extract model from S3 URI path
+ # Expected format: s3://bucket/litellm-bedrock-files-{model}-{uuid}.jsonl
+ try:
+ bucket, object_key = self.parse_s3_uri(s3_uri)
+
+ # Extract model from object key if it follows our naming pattern
+ if object_key.startswith("litellm-bedrock-files-"):
+ # Remove prefix and suffix to get model part
+ model_part = object_key[22:] # Remove "litellm-bedrock-files-"
+ # Find the last dash before the UUID
+ parts = model_part.split("-")
+ if len(parts) > 1:
+ # Reconstruct model name (everything except the last UUID part and .jsonl)
+ model_name = "-".join(parts[:-1])
+ if model_name.endswith(".jsonl"):
+ model_name = model_name[:-6] # Remove .jsonl
+ return model_name
+ except Exception:
+ pass
+
+ # Fallback to default model
+ return "anthropic.claude-3-5-sonnet-20240620-v1:0"
+
+ def sign_aws_request(
+ self,
+ service_name: str,
+ data: Union[str, dict, "BedrockCreateBatchRequest"],
+ endpoint_url: str,
+ optional_params: dict,
+ method: str = "POST",
+ ) -> tuple:
+ """
+ Sign AWS request using Signature Version 4.
+
+ Args:
+ service_name: AWS service name ("bedrock" or "s3")
+ data: Request data (string or dict)
+ endpoint_url: Full endpoint URL
+ optional_params: Optional parameters containing AWS credentials
+ method: HTTP method (default: POST)
+
+ Returns:
+ Tuple of (signed_headers, signed_data)
+ """
+ try:
+ from botocore.auth import SigV4Auth
+ from botocore.awsrequest import AWSRequest
+ except ImportError:
+ raise ImportError("Missing boto3 to call bedrock. Run 'pip install boto3'.")
+
+ # Get AWS credentials using existing methods
+ aws_region_name = self._base_aws._get_aws_region_name(
+ optional_params=optional_params, model=""
+ )
+ credentials = self._base_aws.get_credentials(
+ aws_access_key_id=optional_params.get("aws_access_key_id"),
+ aws_secret_access_key=optional_params.get("aws_secret_access_key"),
+ aws_session_token=optional_params.get("aws_session_token"),
+ aws_region_name=aws_region_name,
+ aws_session_name=optional_params.get("aws_session_name"),
+ aws_profile_name=optional_params.get("aws_profile_name"),
+ aws_role_name=optional_params.get("aws_role_name"),
+ aws_web_identity_token=optional_params.get("aws_web_identity_token"),
+ aws_sts_endpoint=optional_params.get("aws_sts_endpoint"),
+ )
+
+ # Prepare the request data
+ method_upper = method.upper()
+ if method_upper == "GET":
+ # GET requests should be signed with an empty payload
+ request_data = ""
+ headers = {}
+ else:
+ if isinstance(data, dict):
+ import json
+
+ request_data = json.dumps(data)
+ else:
+ request_data = data
+ # Prepare headers for non-GET requests
+ headers = {"Content-Type": "application/json"}
+
+ # Create AWS request and sign it
+ sigv4 = SigV4Auth(credentials, service_name, aws_region_name)
+ request = AWSRequest(
+ method=method_upper, url=endpoint_url, data=request_data, headers=headers
+ )
+ sigv4.add_auth(request)
+ prepped = request.prepare()
+
+ return (
+ dict(prepped.headers),
+ request_data.encode("utf-8")
+ if isinstance(request_data, str)
+ else request_data,
+ )
+
+ def generate_unique_job_name(self, model: str, prefix: str = "litellm") -> str:
+ """
+ Generate a unique job name for AWS services.
+ AWS services often have length limits, so this creates a concise name.
+
+ Args:
+ model: Model name to include in the job name
+ prefix: Prefix for the job name
+
+ Returns:
+ Unique job name (≤ 63 characters for Bedrock compatibility)
+ """
+ from litellm._uuid import uuid
+
+ unique_id = str(uuid.uuid4())[:8]
+ # Format: {prefix}-batch-{model}-{uuid}
+ # Example: litellm-batch-claude-266c398e
+ job_name = f"{prefix}-batch-{unique_id}"
+
+ return job_name
+
+ def get_s3_bucket_and_key_from_config(
+ self,
+ litellm_params: dict,
+ optional_params: dict,
+ bucket_env_var: str = "AWS_S3_BUCKET_NAME",
+ key_prefix: str = "litellm",
+ ) -> tuple:
+ """
+ Get S3 bucket and generate a unique key from configuration.
+
+ Args:
+ litellm_params: LiteLLM parameters
+ optional_params: Optional parameters
+ bucket_env_var: Environment variable name for bucket
+ key_prefix: Prefix for the S3 key
+
+ Returns:
+ Tuple of (bucket_name, object_key)
+ """
+ import time
+ from litellm._uuid import uuid
+
+ # Get bucket name
+ bucket_name = (
+ litellm_params.get("s3_bucket_name")
+ or optional_params.get("s3_bucket_name")
+ or os.getenv(bucket_env_var)
+ )
+ if not bucket_name:
+ raise ValueError(
+ f"S3 bucket name is required. Set 's3_bucket_name' parameter or {bucket_env_var} env var"
+ )
+
+ # Generate unique object key
+ timestamp = int(time.time())
+ unique_id = str(uuid.uuid4())[:8]
+ object_key = f"{key_prefix}-{timestamp}-{unique_id}"
+
+ return bucket_name, object_key
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[Dict, httpx.Headers]
+ ) -> BaseLLMException:
+ """
+ Get Bedrock-specific error class.
+ """
+ return BedrockError(
+ status_code=status_code, message=error_message, headers=headers
+ )
diff --git a/litellm/llms/bedrock/count_tokens/handler.py b/litellm/llms/bedrock/count_tokens/handler.py
new file mode 100644
index 00000000000..d4355c0c360
--- /dev/null
+++ b/litellm/llms/bedrock/count_tokens/handler.py
@@ -0,0 +1,123 @@
+"""
+AWS Bedrock CountTokens API handler.
+
+Simplified handler leveraging existing LiteLLM Bedrock infrastructure.
+"""
+
+from typing import Any, Dict
+
+from fastapi import HTTPException
+
+import litellm
+from litellm._logging import verbose_logger
+from litellm.llms.bedrock.count_tokens.transformation import BedrockCountTokensConfig
+from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
+
+
+class BedrockCountTokensHandler(BedrockCountTokensConfig):
+ """
+ Simplified handler for AWS Bedrock CountTokens API requests.
+
+ Uses existing LiteLLM infrastructure for authentication and request handling.
+ """
+
+ async def handle_count_tokens_request(
+ self,
+ request_data: Dict[str, Any],
+ litellm_params: Dict[str, Any],
+ resolved_model: str,
+ ) -> Dict[str, Any]:
+ """
+ Handle a CountTokens request using existing LiteLLM patterns.
+
+ Args:
+ request_data: The incoming request payload
+ litellm_params: LiteLLM configuration parameters
+ resolved_model: The actual model ID resolved from router
+
+ Returns:
+ Dictionary containing token count response
+ """
+ try:
+ # Validate the request
+ self.validate_count_tokens_request(request_data)
+
+ verbose_logger.debug(
+ f"Processing CountTokens request for resolved model: {resolved_model}"
+ )
+
+ # Get AWS region using existing LiteLLM function
+ aws_region_name = self._get_aws_region_name(
+ optional_params=litellm_params,
+ model=resolved_model,
+ model_id=None,
+ )
+
+ verbose_logger.debug(f"Retrieved AWS region: {aws_region_name}")
+
+ # Transform request to Bedrock format (supports both Converse and InvokeModel)
+ bedrock_request = self.transform_anthropic_to_bedrock_count_tokens(
+ request_data=request_data
+ )
+
+ verbose_logger.debug(f"Transformed request: {bedrock_request}")
+
+ # Get endpoint URL using simplified function
+ endpoint_url = self.get_bedrock_count_tokens_endpoint(
+ resolved_model, aws_region_name
+ )
+
+ verbose_logger.debug(f"Making request to: {endpoint_url}")
+
+ # Use existing _sign_request method from BaseAWSLLM
+ headers = {"Content-Type": "application/json"}
+ signed_headers, signed_body = self._sign_request(
+ service_name="bedrock",
+ headers=headers,
+ optional_params=litellm_params,
+ request_data=bedrock_request,
+ api_base=endpoint_url,
+ model=resolved_model,
+ )
+
+ async_client = get_async_httpx_client(llm_provider=litellm.LlmProviders.BEDROCK)
+
+ response = await async_client.post(
+ endpoint_url,
+ headers=signed_headers,
+ data=signed_body,
+ timeout=30.0,
+ )
+
+ verbose_logger.debug(f"Response status: {response.status_code}")
+
+ if response.status_code != 200:
+ error_text = response.text
+ verbose_logger.error(f"AWS Bedrock error: {error_text}")
+ raise HTTPException(
+ status_code=400,
+ detail={"error": f"AWS Bedrock error: {error_text}"},
+ )
+
+ bedrock_response = response.json()
+
+ verbose_logger.debug(f"Bedrock response: {bedrock_response}")
+
+ # Transform response back to expected format
+ final_response = self.transform_bedrock_response_to_anthropic(
+ bedrock_response
+ )
+
+ verbose_logger.debug(f"Final response: {final_response}")
+
+ return final_response
+
+ except HTTPException:
+ # Re-raise HTTP exceptions as-is
+ raise
+ except Exception as e:
+ verbose_logger.error(f"Error in CountTokens handler: {str(e)}")
+ raise HTTPException(
+ status_code=500,
+ detail={"error": f"CountTokens processing error: {str(e)}"},
+ )
diff --git a/litellm/llms/bedrock/count_tokens/transformation.py b/litellm/llms/bedrock/count_tokens/transformation.py
new file mode 100644
index 00000000000..d46ed3aa452
--- /dev/null
+++ b/litellm/llms/bedrock/count_tokens/transformation.py
@@ -0,0 +1,213 @@
+"""
+AWS Bedrock CountTokens API transformation logic.
+
+This module handles the transformation of requests from Anthropic Messages API format
+to AWS Bedrock's CountTokens API format and vice versa.
+"""
+
+from typing import Any, Dict, List
+
+from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
+from litellm.llms.bedrock.common_utils import BedrockModelInfo
+
+
+class BedrockCountTokensConfig(BaseAWSLLM):
+ """
+ Configuration and transformation logic for AWS Bedrock CountTokens API.
+
+ AWS Bedrock CountTokens API Specification:
+ - Endpoint: POST /model/{modelId}/count-tokens
+ - Input formats: 'invokeModel' or 'converse'
+ - Response: {"inputTokens": }
+ """
+
+ def _detect_input_type(self, request_data: Dict[str, Any]) -> str:
+ """
+ Detect whether to use 'converse' or 'invokeModel' input format.
+
+ Args:
+ request_data: The original request data
+
+ Returns:
+ 'converse' or 'invokeModel'
+ """
+ # If the request has messages in the expected Anthropic format, use converse
+ if "messages" in request_data and isinstance(request_data["messages"], list):
+ return "converse"
+
+ # For raw text or other formats, use invokeModel
+ # This handles cases where the input is prompt-based or already in raw Bedrock format
+ return "invokeModel"
+
+ def transform_anthropic_to_bedrock_count_tokens(
+ self,
+ request_data: Dict[str, Any],
+ ) -> Dict[str, Any]:
+ """
+ Transform request to Bedrock CountTokens format.
+ Supports both Converse and InvokeModel input types.
+
+ Input (Anthropic format):
+ {
+ "model": "claude-3-5-sonnet",
+ "messages": [{"role": "user", "content": "Hello!"}]
+ }
+
+ Output (Bedrock CountTokens format for Converse):
+ {
+ "input": {
+ "converse": {
+ "messages": [...],
+ "system": [...] (if present)
+ }
+ }
+ }
+
+ Output (Bedrock CountTokens format for InvokeModel):
+ {
+ "input": {
+ "invokeModel": {
+ "body": "{...raw model input...}"
+ }
+ }
+ }
+ """
+ input_type = self._detect_input_type(request_data)
+
+ if input_type == "converse":
+ return self._transform_to_converse_format(request_data.get("messages", []))
+ else:
+ return self._transform_to_invoke_model_format(request_data)
+
+ def _transform_to_converse_format(
+ self, messages: List[Dict[str, Any]]
+ ) -> Dict[str, Any]:
+ """Transform to Converse input format."""
+ # Extract system messages if present
+ system_messages = []
+ user_messages = []
+
+ for message in messages:
+ if message.get("role") == "system":
+ system_messages.append({"text": message.get("content", "")})
+ else:
+ # Transform message content to Bedrock format
+ transformed_message: Dict[str, Any] = {"role": message.get("role"), "content": []}
+
+ # Handle content - ensure it's in the correct array format
+ content = message.get("content", "")
+ if isinstance(content, str):
+ # String content -> convert to text block
+ transformed_message["content"].append({"text": content})
+ elif isinstance(content, list):
+ # Already in blocks format - use as is
+ transformed_message["content"] = content
+
+ user_messages.append(transformed_message)
+
+ # Build the converse input format
+ converse_input = {"messages": user_messages}
+
+ # Add system messages if present
+ if system_messages:
+ converse_input["system"] = system_messages
+
+ # Build the complete request
+ return {"input": {"converse": converse_input}}
+
+ def _transform_to_invoke_model_format(
+ self, request_data: Dict[str, Any]
+ ) -> Dict[str, Any]:
+ """Transform to InvokeModel input format."""
+ import json
+
+ # For InvokeModel, we need to provide the raw body that would be sent to the model
+ # Remove the 'model' field from the body as it's not part of the model input
+ body_data = {k: v for k, v in request_data.items() if k != "model"}
+
+ return {"input": {"invokeModel": {"body": json.dumps(body_data)}}}
+
+ def get_bedrock_count_tokens_endpoint(
+ self, model: str, aws_region_name: str
+ ) -> str:
+ """
+ Construct the AWS Bedrock CountTokens API endpoint using existing LiteLLM functions.
+
+ Args:
+ model: The resolved model ID from router lookup
+ aws_region_name: AWS region (e.g., "eu-west-1")
+
+ Returns:
+ Complete endpoint URL for CountTokens API
+ """
+ # Use existing LiteLLM function to get the base model ID (removes region prefix)
+ model_id = BedrockModelInfo.get_base_model(model)
+
+ # Remove bedrock/ prefix if present
+ if model_id.startswith("bedrock/"):
+ model_id = model_id[8:] # Remove "bedrock/" prefix
+
+ base_url = f"https://bedrock-runtime.{aws_region_name}.amazonaws.com"
+ endpoint = f"{base_url}/model/{model_id}/count-tokens"
+
+ return endpoint
+
+ def transform_bedrock_response_to_anthropic(
+ self, bedrock_response: Dict[str, Any]
+ ) -> Dict[str, Any]:
+ """
+ Transform Bedrock CountTokens response to Anthropic format.
+
+ Input (Bedrock response):
+ {
+ "inputTokens": 123
+ }
+
+ Output (Anthropic format):
+ {
+ "input_tokens": 123
+ }
+ """
+ input_tokens = bedrock_response.get("inputTokens", 0)
+
+ return {"input_tokens": input_tokens}
+
+ def validate_count_tokens_request(self, request_data: Dict[str, Any]) -> None:
+ """
+ Validate the incoming count tokens request.
+ Supports both Converse and InvokeModel input formats.
+
+ Args:
+ request_data: The request payload
+
+ Raises:
+ ValueError: If the request is invalid
+ """
+ if not request_data.get("model"):
+ raise ValueError("model parameter is required")
+
+ input_type = self._detect_input_type(request_data)
+
+ if input_type == "converse":
+ # Validate Converse format (messages-based)
+ messages = request_data.get("messages", [])
+ if not messages:
+ raise ValueError("messages parameter is required for Converse input")
+
+ if not isinstance(messages, list):
+ raise ValueError("messages must be a list")
+
+ for i, message in enumerate(messages):
+ if not isinstance(message, dict):
+ raise ValueError(f"Message {i} must be a dictionary")
+
+ if "role" not in message:
+ raise ValueError(f"Message {i} must have a 'role' field")
+
+ if "content" not in message:
+ raise ValueError(f"Message {i} must have a 'content' field")
+ else:
+ # For InvokeModel format, we need at least some content to count tokens
+ # The content structure varies by model, so we do minimal validation
+ if len(request_data) <= 1: # Only has 'model' field
+ raise ValueError("Request must contain content to count tokens")
diff --git a/litellm/llms/bedrock/embed/amazon_titan_v2_transformation.py b/litellm/llms/bedrock/embed/amazon_titan_v2_transformation.py
index 8056e9e9b2c..ff748b58e8e 100644
--- a/litellm/llms/bedrock/embed/amazon_titan_v2_transformation.py
+++ b/litellm/llms/bedrock/embed/amazon_titan_v2_transformation.py
@@ -10,7 +10,7 @@ Docs - https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-tit
"""
import types
-from typing import List, Optional
+from typing import List, Optional, Union
from litellm.types.llms.bedrock import (
AmazonTitanV2EmbeddingRequest,
@@ -30,9 +30,7 @@ class AmazonTitanV2Config:
normalize: Optional[bool] = None
dimensions: Optional[int] = None
- def __init__(
- self, normalize: Optional[bool] = None, dimensions: Optional[int] = None
- ) -> None:
+ def __init__(self, normalize: Optional[bool] = None, dimensions: Optional[int] = None) -> None:
locals_ = locals().copy()
for key, value in locals_.items():
if key != "self" and value is not None:
@@ -57,32 +55,56 @@ class AmazonTitanV2Config:
}
def get_supported_openai_params(self) -> List[str]:
- return ["dimensions"]
+ return ["dimensions", "encoding_format"]
- def map_openai_params(
- self, non_default_params: dict, optional_params: dict
- ) -> dict:
+ def map_openai_params(self, non_default_params: dict, optional_params: dict) -> dict:
for k, v in non_default_params.items():
if k == "dimensions":
optional_params["dimensions"] = v
+ elif k == "encoding_format":
+ # Map OpenAI encoding_format to AWS embeddingTypes
+ if v == "float":
+ optional_params["embeddingTypes"] = ["float"]
+ elif v == "base64":
+ # base64 maps to binary format in AWS
+ optional_params["embeddingTypes"] = ["binary"]
+ else:
+ # For any other encoding format, default to float
+ optional_params["embeddingTypes"] = ["float"]
return optional_params
- def _transform_request(
- self, input: str, inference_params: dict
- ) -> AmazonTitanV2EmbeddingRequest:
+ def _transform_request(self, input: str, inference_params: dict) -> AmazonTitanV2EmbeddingRequest:
return AmazonTitanV2EmbeddingRequest(inputText=input, **inference_params) # type: ignore
- def _transform_response(
- self, response_list: List[dict], model: str
- ) -> EmbeddingResponse:
+ def _transform_response(self, response_list: List[dict], model: str) -> EmbeddingResponse:
total_prompt_tokens = 0
transformed_responses: List[Embedding] = []
for index, response in enumerate(response_list):
_parsed_response = AmazonTitanV2EmbeddingResponse(**response) # type: ignore
+
+ # According to AWS docs, embeddingsByType is always present
+ # If binary was requested (encoding_format="base64"), use binary data
+ # Otherwise, use float data from embeddingsByType or fallback to embedding field
+ embedding_data: Union[List[float], List[int]]
+
+ if ("embeddingsByType" in _parsed_response and
+ "binary" in _parsed_response["embeddingsByType"]):
+ # Use binary data if available (for encoding_format="base64")
+ embedding_data = _parsed_response["embeddingsByType"]["binary"]
+ elif ("embeddingsByType" in _parsed_response and
+ "float" in _parsed_response["embeddingsByType"]):
+ # Use float data from embeddingsByType
+ embedding_data = _parsed_response["embeddingsByType"]["float"]
+ elif "embedding" in _parsed_response:
+ # Fallback to legacy embedding field
+ embedding_data = _parsed_response["embedding"]
+ else:
+ raise ValueError(f"No embedding data found in response: {response}")
+
transformed_responses.append(
Embedding(
- embedding=_parsed_response["embedding"],
+ embedding=embedding_data,
index=index,
object="embedding",
)
diff --git a/litellm/llms/bedrock/embed/embedding.py b/litellm/llms/bedrock/embed/embedding.py
index 91c71e86f1a..3edd6d6741b 100644
--- a/litellm/llms/bedrock/embed/embedding.py
+++ b/litellm/llms/bedrock/embed/embedding.py
@@ -4,11 +4,13 @@ Handles embedding calls to Bedrock's `/invoke` endpoint
import copy
import json
-from typing import Any, Callable, List, Optional, Tuple, Union
+import urllib.parse
+from typing import Any, Callable, List, Optional, Tuple, Union, get_args
import httpx
import litellm
+from litellm.constants import BEDROCK_EMBEDDING_PROVIDERS_LITERAL
from litellm.llms.cohere.embed.handler import embedding as cohere_embedding
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
@@ -17,8 +19,11 @@ from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
)
from litellm.secret_managers.main import get_secret
-from litellm.types.llms.bedrock import AmazonEmbeddingRequest, CohereEmbeddingRequest
-from litellm.types.utils import EmbeddingResponse
+from litellm.types.llms.bedrock import (
+ AmazonEmbeddingRequest,
+ CohereEmbeddingRequest,
+)
+from litellm.types.utils import EmbeddingResponse, LlmProviders
from ..base_aws_llm import BaseAWSLLM
from ..common_utils import BedrockError
@@ -28,6 +33,7 @@ from .amazon_titan_multimodal_transformation import (
)
from .amazon_titan_v2_transformation import AmazonTitanV2Config
from .cohere_transformation import BedrockCohereEmbeddingConfig
+from .twelvelabs_marengo_transformation import TwelveLabsMarengoEmbeddingConfig
class BedrockEmbedding(BaseAWSLLM):
@@ -70,7 +76,7 @@ class BedrockEmbedding(BaseAWSLLM):
if aws_region_name is None:
aws_region_name = "us-west-2"
- credentials: Credentials = self.get_credentials(
+ credentials: Credentials = self.get_credentials( # type: ignore
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
aws_session_token=aws_session_token,
@@ -145,6 +151,89 @@ class BedrockEmbedding(BaseAWSLLM):
return response.json()
+ def _transform_response(
+ self,
+ response_list: List[dict],
+ model: str,
+ provider: BEDROCK_EMBEDDING_PROVIDERS_LITERAL,
+ is_async_invoke: Optional[bool] = False,
+ ) -> Optional[EmbeddingResponse]:
+ """
+ Transforms the response from the Bedrock embedding provider to the OpenAI format.
+ """
+ returned_response: Optional[EmbeddingResponse] = None
+
+ # Handle async invoke responses (single response with invocationArn)
+ if (
+ is_async_invoke
+ and len(response_list) == 1
+ and "invocationArn" in response_list[0]
+ ):
+ if provider == "twelvelabs":
+ returned_response = (
+ TwelveLabsMarengoEmbeddingConfig()._transform_async_invoke_response(
+ response=response_list[0], model=model
+ )
+ )
+ else:
+ # For other providers, create a generic async response
+ invocation_arn = response_list[0].get("invocationArn", "")
+
+ from litellm.types.utils import Embedding, Usage
+
+ embedding = Embedding(
+ embedding=[],
+ index=0,
+ object="embedding", # Must be literal "embedding"
+ )
+ usage = Usage(prompt_tokens=0, total_tokens=0)
+
+ # Create hidden params with job ID
+ from litellm.types.llms.base import HiddenParams
+
+ hidden_params = HiddenParams()
+ setattr(hidden_params, "_invocation_arn", invocation_arn)
+
+ returned_response = EmbeddingResponse(
+ data=[embedding],
+ model=model,
+ usage=usage,
+ hidden_params=hidden_params,
+ )
+ else:
+ # Handle regular invoke responses
+ if model == "amazon.titan-embed-image-v1":
+ returned_response = (
+ AmazonTitanMultimodalEmbeddingG1Config()._transform_response(
+ response_list=response_list, model=model
+ )
+ )
+ elif model == "amazon.titan-embed-text-v1":
+ returned_response = AmazonTitanG1Config()._transform_response(
+ response_list=response_list, model=model
+ )
+ elif model == "amazon.titan-embed-text-v2:0":
+ returned_response = AmazonTitanV2Config()._transform_response(
+ response_list=response_list, model=model
+ )
+ elif provider == "twelvelabs":
+ returned_response = (
+ TwelveLabsMarengoEmbeddingConfig()._transform_response(
+ response_list=response_list, model=model
+ )
+ )
+
+ ##########################################################
+ # Validate returned response
+ ##########################################################
+ if returned_response is None:
+ raise Exception(
+ "Unable to map model response to known provider format. model={}".format(
+ model
+ )
+ )
+ return returned_response
+
def _single_func_embeddings(
self,
client: Optional[HTTPHandler],
@@ -156,23 +245,25 @@ class BedrockEmbedding(BaseAWSLLM):
aws_region_name: str,
model: str,
logging_obj: Any,
+ provider: BEDROCK_EMBEDDING_PROVIDERS_LITERAL,
api_key: Optional[str] = None,
+ is_async_invoke: Optional[bool] = False,
):
responses: List[dict] = []
for data in batch_data:
headers = {"Content-Type": "application/json"}
if extra_headers is not None:
headers = {"Content-Type": "application/json", **extra_headers}
-
- prepped = self.get_request_headers(
- credentials=credentials,
- aws_region_name=aws_region_name,
- extra_headers=extra_headers,
- endpoint_url=endpoint_url,
- data=json.dumps(data),
- headers=headers,
- api_key=api_key
- )
+
+ prepped = self.get_request_headers( # type: ignore # type: ignore
+ credentials=credentials,
+ aws_region_name=aws_region_name,
+ extra_headers=extra_headers,
+ endpoint_url=endpoint_url,
+ data=json.dumps(data),
+ headers=headers,
+ api_key=api_key,
+ )
## LOGGING
logging_obj.pre_call(
@@ -202,32 +293,12 @@ class BedrockEmbedding(BaseAWSLLM):
responses.append(response)
- returned_response: Optional[EmbeddingResponse] = None
-
- ## TRANSFORM RESPONSE ##
- if model == "amazon.titan-embed-image-v1":
- returned_response = (
- AmazonTitanMultimodalEmbeddingG1Config()._transform_response(
- response_list=responses, model=model
- )
- )
- elif model == "amazon.titan-embed-text-v1":
- returned_response = AmazonTitanG1Config()._transform_response(
- response_list=responses, model=model
- )
- elif model == "amazon.titan-embed-text-v2:0":
- returned_response = AmazonTitanV2Config()._transform_response(
- response_list=responses, model=model
- )
-
- if returned_response is None:
- raise Exception(
- "Unable to map model response to known provider format. model={}".format(
- model
- )
- )
-
- return returned_response
+ return self._transform_response(
+ response_list=responses,
+ model=model,
+ provider=provider,
+ is_async_invoke=is_async_invoke,
+ )
async def _async_single_func_embeddings(
self,
@@ -240,23 +311,25 @@ class BedrockEmbedding(BaseAWSLLM):
aws_region_name: str,
model: str,
logging_obj: Any,
+ provider: BEDROCK_EMBEDDING_PROVIDERS_LITERAL,
api_key: Optional[str] = None,
+ is_async_invoke: Optional[bool] = False,
):
responses: List[dict] = []
for data in batch_data:
headers = {"Content-Type": "application/json"}
if extra_headers is not None:
headers = {"Content-Type": "application/json", **extra_headers}
-
- prepped = self.get_request_headers(
- credentials=credentials,
- aws_region_name=aws_region_name,
- extra_headers=extra_headers,
- endpoint_url=endpoint_url,
- data=json.dumps(data),
- headers=headers,
- api_key=api_key,
- )
+
+ prepped = self.get_request_headers( # type: ignore # type: ignore
+ credentials=credentials,
+ aws_region_name=aws_region_name,
+ extra_headers=extra_headers,
+ endpoint_url=endpoint_url,
+ data=json.dumps(data),
+ headers=headers,
+ api_key=api_key,
+ )
## LOGGING
logging_obj.pre_call(
@@ -285,33 +358,13 @@ class BedrockEmbedding(BaseAWSLLM):
)
responses.append(response)
-
- returned_response: Optional[EmbeddingResponse] = None
-
## TRANSFORM RESPONSE ##
- if model == "amazon.titan-embed-image-v1":
- returned_response = (
- AmazonTitanMultimodalEmbeddingG1Config()._transform_response(
- response_list=responses, model=model
- )
- )
- elif model == "amazon.titan-embed-text-v1":
- returned_response = AmazonTitanG1Config()._transform_response(
- response_list=responses, model=model
- )
- elif model == "amazon.titan-embed-text-v2:0":
- returned_response = AmazonTitanV2Config()._transform_response(
- response_list=responses, model=model
- )
-
- if returned_response is None:
- raise Exception(
- "Unable to map model response to known provider format. model={}".format(
- model
- )
- )
-
- return returned_response
+ return self._transform_response(
+ response_list=responses,
+ model=model,
+ provider=provider,
+ is_async_invoke=is_async_invoke,
+ )
def embeddings(
self,
@@ -333,7 +386,25 @@ class BedrockEmbedding(BaseAWSLLM):
credentials, aws_region_name = self._load_credentials(optional_params)
### TRANSFORMATION ###
- provider = model.split(".")[0]
+ unencoded_model_id = (
+ optional_params.pop("model_id", None) or model
+ ) # default to model if not passed
+ modelId = urllib.parse.quote(unencoded_model_id, safe="")
+ aws_region_name = self._get_aws_region_name(
+ optional_params=optional_params,
+ model=model,
+ model_id=unencoded_model_id,
+ )
+ # Check async invoke needs to be used
+ has_async_invoke = "async_invoke/" in model
+ if has_async_invoke:
+ model = model.replace("async_invoke/", "", 1)
+ provider = self.get_bedrock_embedding_provider(model)
+ if provider is None:
+ raise Exception(
+ f"Unable to determine bedrock embedding provider for model: {model}. "
+ f"Supported providers: {list(get_args(BEDROCK_EMBEDDING_PROVIDERS_LITERAL))}"
+ )
inference_params = copy.deepcopy(optional_params)
inference_params = {
k: v
@@ -343,9 +414,6 @@ class BedrockEmbedding(BaseAWSLLM):
inference_params.pop(
"user", None
) # make sure user is not passed in for bedrock call
- modelId = (
- optional_params.pop("model_id", None) or model
- ) # default to model if not passed
data: Optional[CohereEmbeddingRequest] = None
batch_data: Optional[List] = None
@@ -386,6 +454,19 @@ class BedrockEmbedding(BaseAWSLLM):
)
)
batch_data.append(transformed_request)
+ elif provider == "twelvelabs":
+ batch_data = []
+ for i in input:
+ twelvelabs_request = (
+ TwelveLabsMarengoEmbeddingConfig()._transform_request(
+ input=i,
+ inference_params=inference_params,
+ async_invoke_route=has_async_invoke,
+ model_id=modelId,
+ output_s3_uri=inference_params.get("output_s3_uri"),
+ )
+ )
+ batch_data.append(twelvelabs_request)
### SET RUNTIME ENDPOINT ###
endpoint_url, proxy_endpoint_url = self.get_runtime_endpoint(
@@ -395,7 +476,10 @@ class BedrockEmbedding(BaseAWSLLM):
),
aws_region_name=aws_region_name,
)
- endpoint_url = f"{endpoint_url}/model/{modelId}/invoke"
+ if has_async_invoke:
+ endpoint_url = f"{endpoint_url}/async-invoke"
+ else:
+ endpoint_url = f"{endpoint_url}/model/{modelId}/invoke"
if batch_data is not None:
if aembedding:
@@ -414,8 +498,10 @@ class BedrockEmbedding(BaseAWSLLM):
model=model,
logging_obj=logging_obj,
api_key=api_key,
+ provider=provider,
+ is_async_invoke=has_async_invoke,
)
- return self._single_func_embeddings(
+ returned_response = self._single_func_embeddings(
client=(
client
if client is not None and isinstance(client, HTTPHandler)
@@ -430,15 +516,20 @@ class BedrockEmbedding(BaseAWSLLM):
model=model,
logging_obj=logging_obj,
api_key=api_key,
+ provider=provider,
+ is_async_invoke=has_async_invoke,
)
+ if returned_response is None:
+ raise Exception("Unable to map Bedrock request to provider")
+ return returned_response
elif data is None:
raise Exception("Unable to map Bedrock request to provider")
headers = {"Content-Type": "application/json"}
if extra_headers is not None:
headers = {"Content-Type": "application/json", **extra_headers}
-
- prepped = self.get_request_headers(
+
+ prepped = self.get_request_headers( # type: ignore
credentials=credentials,
aws_region_name=aws_region_name,
extra_headers=extra_headers,
@@ -464,3 +555,94 @@ class BedrockEmbedding(BaseAWSLLM):
client=client,
headers=prepped.headers, # type: ignore
)
+
+ async def _get_async_invoke_status(
+ self, invocation_arn: str, aws_region_name: str, logging_obj=None, **kwargs
+ ) -> dict:
+ """
+ Get the status of an async invoke job using the GetAsyncInvoke operation.
+
+ Args:
+ invocation_arn: The invocation ARN from the async invoke response
+ aws_region_name: AWS region name
+ **kwargs: Additional parameters (credentials, etc.)
+
+ Returns:
+ dict: Status response from AWS Bedrock
+ """
+
+ # Get AWS credentials using the same method as other Bedrock methods
+ credentials, _ = self._load_credentials(kwargs)
+
+ # Get the runtime endpoint
+ endpoint_url, _ = self.get_runtime_endpoint(
+ api_base=None,
+ aws_bedrock_runtime_endpoint=kwargs.get("aws_bedrock_runtime_endpoint"),
+ aws_region_name=aws_region_name,
+ )
+
+ # Construct the status check URL
+ status_url = f"{endpoint_url}/async-invoke/{invocation_arn}"
+
+ # Prepare headers
+ headers = {"Content-Type": "application/json"}
+
+ # Get AWS signed headers
+ prepped = self.get_request_headers( # type: ignore
+ credentials=credentials,
+ aws_region_name=aws_region_name,
+ extra_headers=None,
+ endpoint_url=status_url,
+ data="", # GET request, no body
+ headers=headers,
+ api_key=None,
+ )
+
+ # LOGGING
+ if logging_obj is not None:
+ # Create custom curl command for GET request
+ masked_headers = logging_obj._get_masked_headers(prepped.headers)
+ formatted_headers = " ".join(
+ [f"-H '{k}: {v}'" for k, v in masked_headers.items()]
+ )
+ custom_curl = "\n\nGET Request Sent from LiteLLM:\n"
+ custom_curl += "curl -X GET \\\n"
+ custom_curl += f"{prepped.url} \\\n"
+ custom_curl += f"{formatted_headers}\n"
+
+ logging_obj.pre_call(
+ input=invocation_arn,
+ api_key="",
+ additional_args={
+ "complete_input_dict": {"invocation_arn": invocation_arn},
+ "api_base": prepped.url,
+ "headers": prepped.headers,
+ "request_str": custom_curl, # Override with custom GET curl command
+ },
+ )
+
+ # Make the GET request
+ client = get_async_httpx_client(llm_provider=LlmProviders.BEDROCK)
+ response = await client.get(
+ url=prepped.url,
+ headers=prepped.headers,
+ )
+
+ # LOGGING
+ if logging_obj is not None:
+ logging_obj.post_call(
+ input=invocation_arn,
+ api_key="",
+ original_response=response,
+ additional_args={
+ "complete_input_dict": {"invocation_arn": invocation_arn}
+ },
+ )
+
+ # Parse response
+ if response.status_code == 200:
+ return response.json()
+ else:
+ raise Exception(
+ f"Failed to get async invoke status: {response.status_code} - {response.text}"
+ )
diff --git a/litellm/llms/bedrock/embed/twelvelabs_marengo_transformation.py b/litellm/llms/bedrock/embed/twelvelabs_marengo_transformation.py
new file mode 100644
index 00000000000..c85c388eebc
--- /dev/null
+++ b/litellm/llms/bedrock/embed/twelvelabs_marengo_transformation.py
@@ -0,0 +1,301 @@
+"""
+Transformation logic from OpenAI /v1/embeddings format to Bedrock TwelveLabs Marengo /invoke and /async-invoke format.
+
+Why separate file? Make it easy to see how transformation works
+
+Docs - https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-marengo.html
+"""
+
+from typing import List, Optional, Union, cast
+
+from litellm.types.llms.bedrock import (
+ TWELVELABS_EMBEDDING_INPUT_TYPES,
+ TwelveLabsAsyncInvokeRequest,
+ TwelveLabsMarengoEmbeddingRequest,
+ TwelveLabsOutputDataConfig,
+ TwelveLabsS3Location,
+ TwelveLabsS3OutputDataConfig,
+)
+from litellm.types.utils import Embedding, EmbeddingResponse, Usage
+
+
+class TwelveLabsMarengoEmbeddingConfig:
+ """
+ Reference - https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-marengo.html
+
+ Supports text, image, video, and audio inputs.
+ - InvokeModel: text and image inputs
+ - StartAsyncInvoke: video, audio, image, and text inputs
+ """
+
+ def __init__(self) -> None:
+ pass
+
+ def get_supported_openai_params(self) -> List[str]:
+ return [
+ "encoding_format",
+ "textTruncate",
+ "embeddingOption",
+ "startSec",
+ "lengthSec",
+ "useFixedLengthSec",
+ "minClipSec",
+ "input_type",
+ ]
+
+ def map_openai_params(
+ self, non_default_params: dict, optional_params: dict
+ ) -> dict:
+ for k, v in non_default_params.items():
+ if k == "encoding_format":
+ # TwelveLabs doesn't have encoding_format, but we can map it to embeddingOption
+ if v == "float":
+ optional_params["embeddingOption"] = ["visual-text", "visual-image"]
+ elif k == "textTruncate":
+ optional_params["textTruncate"] = v
+ elif k == "embeddingOption":
+ optional_params["embeddingOption"] = v
+ elif k == "input_type":
+ # Map input_type to inputType for Bedrock
+ optional_params["inputType"] = v
+ elif k in ["startSec", "lengthSec", "useFixedLengthSec", "minClipSec"]:
+ optional_params[k] = v
+ return optional_params
+
+ def _extract_bucket_owner_from_params(self, inference_params: dict) -> str:
+ """
+ Extract bucket owner from inference parameters.
+ """
+ return inference_params.get("bucketOwner", "")
+
+ def _is_s3_url(self, input: str) -> bool:
+ """Check if input is an S3 URL."""
+ return input.startswith("s3://")
+
+ def _transform_request(
+ self,
+ input: str,
+ inference_params: dict,
+ async_invoke_route: bool = False,
+ model_id: Optional[str] = None,
+ output_s3_uri: Optional[str] = None,
+ ) -> Union[TwelveLabsMarengoEmbeddingRequest, TwelveLabsAsyncInvokeRequest]:
+ """
+ Transform OpenAI-style input to TwelveLabs Marengo format/async-invoke format.
+
+ Supports:
+ - Text inputs (for both invoke and async-invoke)
+ - Image inputs (for both invoke and async-invoke)
+ - Video inputs (async-invoke only)
+ - Audio inputs (async-invoke only)
+ - S3 URLs for all media types (async-invoke only)
+ """
+ # Get input_type or default to "text"
+ input_type = cast(
+ TWELVELABS_EMBEDDING_INPUT_TYPES,
+ inference_params.get("inputType") or inference_params.get("input_type") or "text"
+ )
+
+ # Validate that async-invoke is used for video/audio
+ if input_type in ["video", "audio"] and not async_invoke_route:
+ raise ValueError(
+ f"Input type '{input_type}' requires async_invoke route. "
+ f"Use model format: 'bedrock/async_invoke/model_id'"
+ )
+
+ transformed_request: TwelveLabsMarengoEmbeddingRequest = {
+ "inputType": input_type
+ }
+
+ if input_type == "text":
+ transformed_request["inputText"] = input
+ # Set default textTruncate if not specified
+ if "textTruncate" not in inference_params:
+ transformed_request["textTruncate"] = "end"
+
+ elif input_type in ["image", "video", "audio"]:
+ if self._is_s3_url(input):
+ # S3 URL input
+ s3_location: TwelveLabsS3Location = {"uri": input}
+ bucket_owner = self._extract_bucket_owner_from_params(inference_params)
+ if bucket_owner:
+ s3_location["bucketOwner"] = bucket_owner
+
+ transformed_request["mediaSource"] = {"s3Location": s3_location}
+ else:
+ # Base64 encoded input
+ if input.startswith("data:"):
+ # Extract base64 data from data URL
+ b64_str = input.split(",", 1)[1] if "," in input else input
+ else:
+ # Direct base64 string
+ from litellm.utils import get_base64_str
+ b64_str = get_base64_str(input)
+
+ transformed_request["mediaSource"] = {"base64String": b64_str}
+
+ # Apply any additional inference parameters
+ for k, v in inference_params.items():
+ if k not in [
+ "inputType",
+ "input_type", # Exclude both camelCase and snake_case
+ "inputText",
+ "mediaSource",
+ "bucketOwner", # Don't include bucketOwner in the request
+ ]: # Don't override core fields
+ transformed_request[k] = v # type: ignore
+
+ # If async invoke route, wrap in the async invoke format
+ if async_invoke_route and model_id:
+ return self._wrap_async_invoke_request(
+ model_input=transformed_request,
+ model_id=model_id,
+ output_s3_uri=output_s3_uri,
+ )
+
+ return transformed_request
+
+ def _wrap_async_invoke_request(
+ self,
+ model_input: TwelveLabsMarengoEmbeddingRequest,
+ model_id: str,
+ output_s3_uri: Optional[str] = None,
+ ) -> TwelveLabsAsyncInvokeRequest:
+ """
+ Wrap the transformed request in the correct AWS Bedrock async invoke format.
+
+ Args:
+ model_input: The transformed TwelveLabs Marengo embedding request
+ model_id: The model identifier (without async_invoke prefix)
+ output_s3_uri: Optional S3 URI for output data config
+
+ Returns:
+ TwelveLabsAsyncInvokeRequest: The wrapped async invoke request
+ """
+ import urllib.parse
+
+ # Clean the model ID
+ unquoted_model_id = urllib.parse.unquote(model_id)
+ if unquoted_model_id.startswith("async_invoke/"):
+ unquoted_model_id = unquoted_model_id.replace("async_invoke/", "")
+
+ # Validate that the S3 URI is not empty
+ if not output_s3_uri or output_s3_uri.strip() == "":
+ raise ValueError("output_s3_uri cannot be empty for async invoke requests")
+
+ return TwelveLabsAsyncInvokeRequest(
+ modelId=unquoted_model_id,
+ modelInput=model_input,
+ outputDataConfig=TwelveLabsOutputDataConfig(
+ s3OutputDataConfig=TwelveLabsS3OutputDataConfig(s3Uri=output_s3_uri)
+ ),
+ )
+
+ def _transform_response(
+ self, response_list: List[dict], model: str
+ ) -> EmbeddingResponse:
+ """
+ Transform TwelveLabs response to OpenAI format.
+ Handles the actual TwelveLabs response format: {"data": [{"embedding": [...]}]}
+ """
+ embeddings: List[Embedding] = []
+ total_tokens = 0
+
+ for response in response_list:
+ # TwelveLabs response format has a "data" field containing the embeddings
+ if "data" in response and isinstance(response["data"], list):
+ for item in response["data"]:
+ if "embedding" in item:
+ # Single embedding response
+ embedding = Embedding(
+ embedding=item["embedding"],
+ index=len(embeddings),
+ object="embedding",
+ )
+ embeddings.append(embedding)
+
+ # Estimate token count (rough approximation)
+ if "inputTextTokenCount" in item:
+ total_tokens += item["inputTextTokenCount"]
+ else:
+ # Rough estimate: 1 token per 4 characters for text, or use embedding size
+ total_tokens += len(item["embedding"]) // 4
+ elif "embedding" in response:
+ # Direct embedding response (fallback for other formats)
+ embedding = Embedding(
+ embedding=response["embedding"],
+ index=len(embeddings),
+ object="embedding",
+ )
+ embeddings.append(embedding)
+
+ # Estimate token count (rough approximation)
+ if "inputTextTokenCount" in response:
+ total_tokens += response["inputTextTokenCount"]
+ else:
+ # Rough estimate: 1 token per 4 characters for text
+ total_tokens += len(response.get("inputText", "")) // 4
+ elif "embeddings" in response:
+ # Multiple embeddings response (from video/audio)
+ for i, emb in enumerate(response["embeddings"]):
+ embedding = Embedding(
+ embedding=emb["embedding"],
+ index=len(embeddings),
+ object="embedding",
+ )
+ embeddings.append(embedding)
+ total_tokens += len(emb["embedding"]) // 4 # Rough estimate
+
+ usage = Usage(prompt_tokens=total_tokens, total_tokens=total_tokens)
+
+ return EmbeddingResponse(data=embeddings, model=model, usage=usage)
+
+ def _transform_async_invoke_response(
+ self, response: dict, model: str
+ ) -> EmbeddingResponse:
+ """
+ Transform async invoke response (invocation ARN) to OpenAI format.
+
+ AWS async invoke returns:
+ {
+ "invocationArn": "arn:aws:bedrock:us-east-1:123456789012:async-invoke/abc123"
+ }
+
+ We transform this to a job-like embedding response:
+ {
+ "object": "list",
+ "data": [
+ {
+ "object": "embedding_job_id:1234567890",
+ "embedding": [],
+ "index": 0
+ }
+ ],
+ "model": "model",
+ "usage": {}
+ }
+ """
+ invocation_arn = response.get("invocationArn", "")
+
+ # Create a placeholder embedding object for the job
+ embedding = Embedding(
+ embedding=[], # Empty embedding for async jobs
+ index=0,
+ object="embedding",
+ )
+
+ # Create usage object (empty for async jobs)
+ usage = Usage(prompt_tokens=0, total_tokens=0)
+
+ # Create hidden params with job ID
+ from litellm.types.llms.base import HiddenParams
+
+ hidden_params = HiddenParams()
+ setattr(hidden_params, "_invocation_arn", invocation_arn)
+
+ return EmbeddingResponse(
+ data=[embedding],
+ model=model,
+ usage=usage,
+ hidden_params=hidden_params,
+ )
diff --git a/litellm/llms/bedrock/files/transformation.py b/litellm/llms/bedrock/files/transformation.py
new file mode 100644
index 00000000000..0a95cf9168f
--- /dev/null
+++ b/litellm/llms/bedrock/files/transformation.py
@@ -0,0 +1,662 @@
+import json
+import os
+import time
+from litellm._uuid import uuid
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+from httpx import Headers, Response
+
+from litellm._logging import verbose_logger
+from litellm.files.utils import FilesAPIUtils
+from litellm.litellm_core_utils.prompt_templates.common_utils import extract_file_data
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.base_llm.files.transformation import (
+ BaseFilesConfig,
+ LiteLLMLoggingObj,
+)
+from litellm.types.llms.openai import (
+ AllMessageValues,
+ CreateFileRequest,
+ FileTypes,
+ OpenAICreateFileRequestOptionalParams,
+ OpenAIFileObject,
+ PathLike,
+)
+from litellm.types.utils import ExtractedFileData, LlmProviders
+from litellm.utils import get_llm_provider
+
+from ..base_aws_llm import BaseAWSLLM
+from ..common_utils import BedrockError
+
+
+class BedrockFilesConfig(BaseAWSLLM, BaseFilesConfig):
+ """
+ Config for Bedrock Files - handles S3 uploads for Bedrock batch processing
+ """
+
+ def __init__(self):
+ self.jsonl_transformation = BedrockJsonlFilesTransformation()
+ super().__init__()
+
+ @property
+ def custom_llm_provider(self) -> LlmProviders:
+ return LlmProviders.BEDROCK
+
+ @property
+ def file_upload_http_method(self) -> str:
+ """
+ Bedrock files are uploaded to S3, which requires PUT requests
+ """
+ return "PUT"
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ # No additional headers needed for S3 uploads - AWS credentials handled by BaseAWSLLM
+ return headers
+
+
+
+ def _get_content_from_openai_file(self, openai_file_content: FileTypes) -> str:
+ """
+ Helper to extract content from various OpenAI file types and return as string.
+
+ Handles:
+ - Direct content (str, bytes, IO[bytes])
+ - Tuple formats: (filename, content, [content_type], [headers])
+ - PathLike objects
+ """
+ content: Union[str, bytes] = b""
+ # Extract file content from tuple if necessary
+ if isinstance(openai_file_content, tuple):
+ # Take the second element which is always the file content
+ file_content = openai_file_content[1]
+ else:
+ file_content = openai_file_content
+
+ # Handle different file content types
+ if isinstance(file_content, str):
+ # String content can be used directly
+ content = file_content
+ elif isinstance(file_content, bytes):
+ # Bytes content can be decoded
+ content = file_content
+ elif isinstance(file_content, PathLike): # PathLike
+ with open(str(file_content), "rb") as f:
+ content = f.read()
+ elif hasattr(file_content, "read"): # IO[bytes]
+ # File-like objects need to be read
+ content = file_content.read()
+
+ # Ensure content is string
+ if isinstance(content, bytes):
+ content = content.decode("utf-8")
+
+ return content
+
+ def _get_s3_object_name_from_batch_jsonl(
+ self,
+ openai_jsonl_content: List[Dict[str, Any]],
+ ) -> str:
+ """
+ Gets a unique S3 object name for the Bedrock batch processing job
+
+ named as: litellm-bedrock-files/{model}/{uuid}
+ """
+ _model = openai_jsonl_content[0].get("body", {}).get("model", "")
+ # Remove bedrock/ prefix if present
+ if _model.startswith("bedrock/"):
+ _model = _model[8:]
+
+ # Replace colons with hyphens for Bedrock S3 URI compliance
+ _model = _model.replace(":", "-")
+
+ object_name = f"litellm-bedrock-files-{_model}-{uuid.uuid4()}.jsonl"
+ return object_name
+
+ def get_object_name(
+ self, extracted_file_data: ExtractedFileData, purpose: str
+ ) -> str:
+ """
+ Get the object name for the request
+ """
+ extracted_file_data_content = extracted_file_data.get("content")
+
+ if extracted_file_data_content is None:
+ raise ValueError("file content is required")
+
+ if purpose == "batch":
+ ## 1. If jsonl, check if there's a model name
+ file_content = self._get_content_from_openai_file(
+ extracted_file_data_content
+ )
+
+ # Split into lines and parse each line as JSON
+ openai_jsonl_content = [
+ json.loads(line) for line in file_content.splitlines() if line.strip()
+ ]
+ if len(openai_jsonl_content) > 0:
+ return self._get_s3_object_name_from_batch_jsonl(openai_jsonl_content)
+
+ ## 2. If not jsonl, return the filename
+ filename = extracted_file_data.get("filename")
+ if filename:
+ return filename
+ ## 3. If no file name, return timestamp
+ return str(int(time.time()))
+
+ def get_complete_file_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: Dict,
+ litellm_params: Dict,
+ data: CreateFileRequest,
+ ) -> str:
+ """
+ Get the complete S3 URL for the file upload request
+ """
+ bucket_name = litellm_params.get("s3_bucket_name") or os.getenv("AWS_S3_BUCKET_NAME")
+ if not bucket_name:
+ raise ValueError("S3 bucket_name is required. Set 's3_bucket_name' in litellm_params or AWS_S3_BUCKET_NAME env var")
+
+ aws_region_name = self._get_aws_region_name(optional_params, model)
+
+ file_data = data.get("file")
+ purpose = data.get("purpose")
+ if file_data is None:
+ raise ValueError("file is required")
+ if purpose is None:
+ raise ValueError("purpose is required")
+ extracted_file_data = extract_file_data(file_data)
+ object_name = self.get_object_name(extracted_file_data, purpose)
+
+ # S3 endpoint URL format
+ s3_endpoint_url = optional_params.get("s3_endpoint_url") or f"https://s3.{aws_region_name}.amazonaws.com"
+
+ return f"{s3_endpoint_url}/{bucket_name}/{object_name}"
+
+ def get_supported_openai_params(
+ self, model: str
+ ) -> List[OpenAICreateFileRequestOptionalParams]:
+ return []
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ return optional_params
+
+
+ def _map_openai_to_bedrock_params(
+ self,
+ openai_request_body: Dict[str, Any],
+ provider: Optional[str] = None,
+ ) -> Dict[str, Any]:
+ """
+ Transform OpenAI request body to Bedrock-compatible modelInput parameters using existing transformation logic
+ """
+ from litellm.types.utils import LlmProviders
+ _model = openai_request_body.get("model", "")
+ messages = openai_request_body.get("messages", [])
+
+ # Use existing Anthropic transformation logic for Anthropic models
+ if provider == LlmProviders.ANTHROPIC:
+ from litellm.llms.bedrock.chat.invoke_transformations.anthropic_claude3_transformation import (
+ AmazonAnthropicClaudeConfig,
+ )
+
+ anthropic_config = AmazonAnthropicClaudeConfig()
+
+ # Extract optional params (everything except model and messages)
+ optional_params = {k: v for k, v in openai_request_body.items() if k not in ["model", "messages"]}
+ mapped_params = anthropic_config.map_openai_params(
+ non_default_params={},
+ optional_params=optional_params,
+ model=_model,
+ drop_params=False
+ )
+
+ # Transform using existing Anthropic logic
+ bedrock_params = anthropic_config.transform_request(
+ model=_model,
+ messages=messages,
+ optional_params=mapped_params,
+ litellm_params={},
+ headers={}
+ )
+
+ return bedrock_params
+ else:
+ # For other providers, use basic mapping
+ bedrock_params = {
+ "messages": messages,
+ **{k: v for k, v in openai_request_body.items() if k not in ["model", "messages"]}
+ }
+ return bedrock_params
+
+ def _transform_openai_jsonl_content_to_bedrock_jsonl_content(
+ self, openai_jsonl_content: List[Dict[str, Any]]
+ ) -> List[Dict[str, Any]]:
+ """
+ Transforms OpenAI JSONL content to Bedrock batch format
+
+ Bedrock batch format: { "recordId": "alphanumeric string", "modelInput": {JSON body} }
+ Example:
+ {
+ "recordId": "CALL0000001",
+ "modelInput": {
+ "anthropic_version": "bedrock-2023-05-31",
+ "max_tokens": 1024,
+ "messages": [
+ {
+ "role": "user",
+ "content": [{"type": "text", "text": "Hello"}]
+ }
+ ]
+ }
+ }
+ """
+
+ bedrock_jsonl_content = []
+ for idx, _openai_jsonl_content in enumerate(openai_jsonl_content):
+ # Extract the request body from OpenAI format
+ openai_body = _openai_jsonl_content.get("body", {})
+ model = openai_body.get("model", "")
+
+ try:
+ model, _, _, _ = get_llm_provider(
+ model=model,
+ custom_llm_provider=None,
+ )
+ except Exception as e:
+ verbose_logger.exception(f"litellm.llms.bedrock.files.transformation.py::_transform_openai_jsonl_content_to_bedrock_jsonl_content() - Error inferring custom_llm_provider - {str(e)}")
+
+ # Determine provider from model name
+ provider = self.get_bedrock_invoke_provider(model)
+
+ # Transform to Bedrock modelInput format
+ model_input = self._map_openai_to_bedrock_params(
+ openai_request_body=openai_body,
+ provider=provider
+ )
+
+ # Create Bedrock batch record
+ record_id = _openai_jsonl_content.get("custom_id", f"CALL{str(idx).zfill(7)}")
+ bedrock_record = {
+ "recordId": record_id,
+ "modelInput": model_input
+ }
+
+ bedrock_jsonl_content.append(bedrock_record)
+ return bedrock_jsonl_content
+
+ def transform_create_file_request(
+ self,
+ model: str,
+ create_file_data: CreateFileRequest,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> Union[bytes, str, dict]:
+ """
+ Transform file request and return a pre-signed request for S3.
+ This keeps the HTTP handler clean by doing all the signing here.
+ """
+ file_data = create_file_data.get("file")
+ if file_data is None:
+ raise ValueError("file is required")
+ extracted_file_data = extract_file_data(file_data)
+ extracted_file_data_content = extracted_file_data.get("content")
+
+ if extracted_file_data_content is None:
+ raise ValueError("file content is required")
+
+ # Get and transform the file content
+ if FilesAPIUtils.is_batch_jsonl_file(
+ create_file_data=create_file_data,
+ extracted_file_data=extracted_file_data,
+ ):
+ ## Transform JSONL content to Bedrock format
+ original_file_content = self._get_content_from_openai_file(
+ extracted_file_data_content
+ )
+ openai_jsonl_content = [
+ json.loads(line) for line in original_file_content.splitlines() if line.strip()
+ ]
+ bedrock_jsonl_content = (
+ self._transform_openai_jsonl_content_to_bedrock_jsonl_content(
+ openai_jsonl_content
+ )
+ )
+ file_content = "\n".join(json.dumps(item) for item in bedrock_jsonl_content)
+ elif isinstance(extracted_file_data_content, bytes):
+ file_content = extracted_file_data_content.decode('utf-8')
+ elif isinstance(extracted_file_data_content, str):
+ file_content = extracted_file_data_content
+ else:
+ raise ValueError("Unsupported file content type")
+
+ # Get the S3 URL for upload
+ api_base = self.get_complete_file_url(
+ api_base=None,
+ api_key=None,
+ model=model,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ data=create_file_data,
+ )
+
+ # Sign the request and return a pre-signed request object
+ signed_headers, signed_body = self._sign_s3_request(
+ content=file_content,
+ api_base=api_base,
+ optional_params=optional_params,
+ )
+
+ litellm_params["upload_url"] = api_base
+
+ # Return a dict that tells the HTTP handler exactly what to do
+ return {
+ "method": "PUT",
+ "url": api_base,
+ "headers": signed_headers,
+ "data": signed_body or file_content,
+ }
+
+ def _sign_s3_request(
+ self,
+ content: str,
+ api_base: str,
+ optional_params: dict,
+ ) -> Tuple[dict, str]:
+ """
+ Sign S3 PUT request using the same proven logic as S3Logger.
+ Reuses the exact pattern from litellm/integrations/s3_v2.py
+ """
+ try:
+ import hashlib
+
+ import requests
+ from botocore.auth import SigV4Auth
+ from botocore.awsrequest import AWSRequest
+ except ImportError:
+ raise ImportError("Missing boto3 to call bedrock. Run 'pip install boto3'.")
+
+ # Get AWS credentials using existing methods
+ aws_region_name = self._get_aws_region_name(
+ optional_params=optional_params, model=""
+ )
+ credentials = self.get_credentials(
+ aws_access_key_id=optional_params.get("aws_access_key_id"),
+ aws_secret_access_key=optional_params.get("aws_secret_access_key"),
+ aws_session_token=optional_params.get("aws_session_token"),
+ aws_region_name=aws_region_name,
+ aws_session_name=optional_params.get("aws_session_name"),
+ aws_profile_name=optional_params.get("aws_profile_name"),
+ aws_role_name=optional_params.get("aws_role_name"),
+ aws_web_identity_token=optional_params.get("aws_web_identity_token"),
+ aws_sts_endpoint=optional_params.get("aws_sts_endpoint"),
+ )
+
+ # Calculate SHA256 hash of the content (REQUIRED for S3)
+ content_hash = hashlib.sha256(content.encode("utf-8")).hexdigest()
+
+ # Prepare headers with required S3 headers (same as s3_v2.py)
+ request_headers = {
+ "Content-Type": "application/json", # JSONL files are JSON content
+ "x-amz-content-sha256": content_hash, # REQUIRED by S3
+ "Content-Language": "en",
+ "Cache-Control": "private, immutable, max-age=31536000, s-maxage=0",
+ }
+
+ # Use requests.Request to prepare the request (same pattern as s3_v2.py)
+ req = requests.Request("PUT", api_base, data=content, headers=request_headers)
+ prepped = req.prepare()
+
+ # Sign the request with S3 service
+ aws_request = AWSRequest(
+ method=prepped.method,
+ url=prepped.url,
+ data=prepped.body,
+ headers=prepped.headers,
+ )
+
+ # Get region name for non-LLM API calls (same as s3_v2.py)
+ signing_region = self.get_aws_region_name_for_non_llm_api_calls(
+ aws_region_name=aws_region_name
+ )
+
+ SigV4Auth(credentials, "s3", signing_region).add_auth(aws_request)
+
+ # Return signed headers and body
+ signed_body = aws_request.body
+ if isinstance(signed_body, bytes):
+ signed_body = signed_body.decode('utf-8')
+ elif signed_body is None:
+ signed_body = content # Fallback to original content
+
+ return dict(aws_request.headers), signed_body
+
+ def _convert_https_url_to_s3_uri(self, https_url: str) -> tuple[str, str]:
+ """
+ Convert HTTPS S3 URL to s3:// URI format.
+
+ Args:
+ https_url: HTTPS S3 URL (e.g., "https://s3.us-west-2.amazonaws.com/bucket/key")
+
+ Returns:
+ Tuple of (s3_uri, filename)
+
+ Example:
+ Input: "https://s3.us-west-2.amazonaws.com/litellm-proxy/file.jsonl"
+ Output: ("s3://litellm-proxy/file.jsonl", "file.jsonl")
+ """
+ import re
+
+ # Match HTTPS S3 URL patterns
+ # Pattern 1: https://s3.region.amazonaws.com/bucket/key
+ # Pattern 2: https://bucket.s3.region.amazonaws.com/key
+
+ pattern1 = r"https://s3\.([^.]+)\.amazonaws\.com/([^/]+)/(.+)"
+ pattern2 = r"https://([^.]+)\.s3\.([^.]+)\.amazonaws\.com/(.+)"
+
+ match1 = re.match(pattern1, https_url)
+ match2 = re.match(pattern2, https_url)
+
+ if match1:
+ # Pattern: https://s3.region.amazonaws.com/bucket/key
+ region, bucket, key = match1.groups()
+ s3_uri = f"s3://{bucket}/{key}"
+ elif match2:
+ # Pattern: https://bucket.s3.region.amazonaws.com/key
+ bucket, region, key = match2.groups()
+ s3_uri = f"s3://{bucket}/{key}"
+ else:
+ # Fallback: try to extract bucket and key from URL path
+ from urllib.parse import urlparse
+ parsed = urlparse(https_url)
+ path_parts = parsed.path.lstrip('/').split('/', 1)
+ if len(path_parts) >= 2:
+ bucket, key = path_parts[0], path_parts[1]
+ s3_uri = f"s3://{bucket}/{key}"
+ else:
+ raise ValueError(f"Unable to parse S3 URL: {https_url}")
+
+ # Extract filename from key
+ filename = key.split("/")[-1] if "/" in key else key
+
+ return s3_uri, filename
+
+ def transform_create_file_response(
+ self,
+ model: Optional[str],
+ raw_response: Response,
+ logging_obj: LiteLLMLoggingObj,
+ litellm_params: dict,
+ ) -> OpenAIFileObject:
+ """
+ Transform S3 File upload response into OpenAI-style FileObject
+ """
+ # For S3 uploads, we typically get an ETag and other metadata
+ response_headers = raw_response.headers
+ # Extract S3 object information from the response
+ # S3 PUT object returns ETag and other metadata in headers
+ content_length = response_headers.get("Content-Length", "0")
+
+ # Use the actual upload URL that was used for the S3 upload
+ upload_url = litellm_params.get("upload_url")
+ file_id: str = ""
+ filename: str = ""
+ if upload_url:
+ # Convert HTTPS S3 URL to s3:// URI format
+ file_id, filename = self._convert_https_url_to_s3_uri(upload_url)
+
+ return OpenAIFileObject(
+ purpose="batch", # Default purpose for Bedrock files
+ id=file_id,
+ filename=filename,
+ created_at=int(time.time()), # Current timestamp
+ status="uploaded",
+ bytes=int(content_length) if content_length.isdigit() else 0,
+ object="file",
+ )
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[Dict, Headers]
+ ) -> BaseLLMException:
+ return BedrockError(
+ status_code=status_code, message=error_message, headers=headers
+ )
+
+
+class BedrockJsonlFilesTransformation:
+ """
+ Transforms OpenAI /v1/files/* requests to Bedrock S3 file uploads for batch processing
+ """
+
+ def transform_openai_file_content_to_bedrock_file_content(
+ self, openai_file_content: Optional[FileTypes] = None
+ ) -> Tuple[str, str]:
+ """
+ Transforms OpenAI FileContentRequest to Bedrock S3 file format
+ """
+
+ if openai_file_content is None:
+ raise ValueError("contents of file are None")
+ # Read the content of the file
+ file_content = self._get_content_from_openai_file(openai_file_content)
+
+ # Split into lines and parse each line as JSON
+ openai_jsonl_content = [
+ json.loads(line) for line in file_content.splitlines() if line.strip()
+ ]
+ bedrock_jsonl_content = (
+ self._transform_openai_jsonl_content_to_bedrock_jsonl_content(
+ openai_jsonl_content
+ )
+ )
+ bedrock_jsonl_string = "\n".join(
+ json.dumps(item) for item in bedrock_jsonl_content
+ )
+ object_name = self._get_s3_object_name(
+ openai_jsonl_content=openai_jsonl_content
+ )
+ return bedrock_jsonl_string, object_name
+
+ def _transform_openai_jsonl_content_to_bedrock_jsonl_content(
+ self, openai_jsonl_content: List[Dict[str, Any]]
+ ):
+ """
+ Delegate to the main BedrockFilesConfig transformation method
+ """
+ config = BedrockFilesConfig()
+ return config._transform_openai_jsonl_content_to_bedrock_jsonl_content(openai_jsonl_content)
+
+ def _get_s3_object_name(
+ self,
+ openai_jsonl_content: List[Dict[str, Any]],
+ ) -> str:
+ """
+ Gets a unique S3 object name for the Bedrock batch processing job
+
+ named as: litellm-bedrock-files-{model}-{uuid}
+ """
+ _model = openai_jsonl_content[0].get("body", {}).get("model", "")
+ # Remove bedrock/ prefix if present
+ if _model.startswith("bedrock/"):
+ _model = _model[8:]
+ object_name = f"litellm-bedrock-files-{_model}-{uuid.uuid4()}.jsonl"
+ return object_name
+
+
+
+ def _get_content_from_openai_file(self, openai_file_content: FileTypes) -> str:
+ """
+ Helper to extract content from various OpenAI file types and return as string.
+
+ Handles:
+ - Direct content (str, bytes, IO[bytes])
+ - Tuple formats: (filename, content, [content_type], [headers])
+ - PathLike objects
+ """
+ content: Union[str, bytes] = b""
+ # Extract file content from tuple if necessary
+ if isinstance(openai_file_content, tuple):
+ # Take the second element which is always the file content
+ file_content = openai_file_content[1]
+ else:
+ file_content = openai_file_content
+
+ # Handle different file content types
+ if isinstance(file_content, str):
+ # String content can be used directly
+ content = file_content
+ elif isinstance(file_content, bytes):
+ # Bytes content can be decoded
+ content = file_content
+ elif isinstance(file_content, PathLike): # PathLike
+ with open(str(file_content), "rb") as f:
+ content = f.read()
+ elif hasattr(file_content, "read"): # IO[bytes]
+ # File-like objects need to be read
+ content = file_content.read()
+
+ # Ensure content is string
+ if isinstance(content, bytes):
+ content = content.decode("utf-8")
+
+ return content
+
+ def transform_s3_bucket_response_to_openai_file_object(
+ self, create_file_data: CreateFileRequest, s3_upload_response: Dict[str, Any]
+ ) -> OpenAIFileObject:
+ """
+ Transforms S3 Bucket upload file response to OpenAI FileObject
+ """
+ # S3 response typically contains ETag, key, etc.
+ object_key = s3_upload_response.get("Key", "")
+ bucket_name = s3_upload_response.get("Bucket", "")
+
+ # Extract filename from object key
+ filename = object_key.split("/")[-1] if "/" in object_key else object_key
+
+ return OpenAIFileObject(
+ purpose=create_file_data.get("purpose", "batch"),
+ id=f"s3://{bucket_name}/{object_key}",
+ filename=filename,
+ created_at=int(time.time()), # Current timestamp
+ status="uploaded",
+ bytes=s3_upload_response.get("ContentLength", 0),
+ object="file",
+ )
diff --git a/litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py b/litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py
index 3ef7a40e9a9..cd33e62af16 100644
--- a/litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py
+++ b/litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py
@@ -7,12 +7,12 @@ from litellm.types.llms.bedrock import (
AmazonNovaCanvasColorGuidedGenerationParams,
AmazonNovaCanvasColorGuidedRequest,
AmazonNovaCanvasImageGenerationConfig,
+ AmazonNovaCanvasInpaintingParams,
+ AmazonNovaCanvasInpaintingRequest,
AmazonNovaCanvasRequestBase,
AmazonNovaCanvasTextToImageParams,
AmazonNovaCanvasTextToImageRequest,
AmazonNovaCanvasTextToImageResponse,
- AmazonNovaCanvasInpaintingParams,
- AmazonNovaCanvasInpaintingRequest,
)
from litellm.types.utils import ImageResponse
@@ -67,6 +67,11 @@ class AmazonNovaCanvasConfig:
"""
task_type = optional_params.pop("taskType", "TEXT_IMAGE")
image_generation_config = optional_params.pop("imageGenerationConfig", {})
+
+ # Extract model_id parameter to prevent "extraneous key" error from Bedrock API
+ # Following the same pattern as chat completions and embeddings
+ unencoded_model_id = optional_params.pop("model_id", None) # noqa: F841
+
image_generation_config = {**image_generation_config, **optional_params}
if task_type == "TEXT_IMAGE":
text_to_image_params: Dict[str, Any] = image_generation_config.pop(
diff --git a/litellm/llms/bedrock/image/image_handler.py b/litellm/llms/bedrock/image/image_handler.py
index 55d94675d14..0103f190d36 100644
--- a/litellm/llms/bedrock/image/image_handler.py
+++ b/litellm/llms/bedrock/image/image_handler.py
@@ -233,7 +233,17 @@ class BedrockImageGeneration(BaseAWSLLM):
Returns:
dict: The request body to use for the Bedrock Image Generation API
"""
- provider = model.split(".")[0]
+ # Use the existing ARN-aware provider detection method
+ bedrock_provider = self.get_bedrock_invoke_provider(model)
+
+ if bedrock_provider == "amazon" or bedrock_provider == "nova":
+ # Handle Amazon Nova Canvas models
+ provider = "amazon"
+ elif bedrock_provider == "stability":
+ provider = "stability"
+ else:
+ # Fallback to original logic for backward compatibility
+ provider = model.split(".")[0]
inference_params = copy.deepcopy(optional_params)
inference_params.pop(
"user", None
diff --git a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
index 09c6673cc5d..4fa8517a090 100644
--- a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
+++ b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
@@ -12,6 +12,7 @@ from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation import (
AmazonInvokeConfig,
)
+from litellm.llms.bedrock.common_utils import get_anthropic_beta_from_headers
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import GenericStreamingChunk
from litellm.types.utils import GenericStreamingChunk as GChunk
@@ -25,12 +26,13 @@ else:
LiteLLMLoggingObj = Any
-class AmazonAnthropicClaude3MessagesConfig(
+class AmazonAnthropicClaudeMessagesConfig(
AnthropicMessagesConfig,
AmazonInvokeConfig,
):
"""
Call Claude model family in the /v1/messages API spec
+ Supports anthropic_beta parameter for beta features.
"""
DEFAULT_BEDROCK_ANTHROPIC_API_VERSION = "bedrock-2023-05-31"
@@ -127,6 +129,12 @@ class AmazonAnthropicClaude3MessagesConfig(
# 3. `model` is not allowed in request body for bedrock invoke
if "model" in anthropic_messages_request:
anthropic_messages_request.pop("model", None)
+
+ # 4. Handle anthropic_beta from user headers
+ anthropic_beta_list = get_anthropic_beta_from_headers(headers)
+ if anthropic_beta_list:
+ anthropic_messages_request["anthropic_beta"] = anthropic_beta_list
+
return anthropic_messages_request
def get_async_streaming_response_iterator(
diff --git a/litellm/llms/bedrock/passthrough/transformation.py b/litellm/llms/bedrock/passthrough/transformation.py
index d7221ff4b7a..5791bfb8013 100644
--- a/litellm/llms/bedrock/passthrough/transformation.py
+++ b/litellm/llms/bedrock/passthrough/transformation.py
@@ -41,9 +41,15 @@ class BedrockPassthroughConfig(
model_id=None,
)
- api_base = f"https://bedrock-runtime.{aws_region_name}.amazonaws.com"
+ aws_bedrock_runtime_endpoint = optional_params.get("aws_bedrock_runtime_endpoint")
+ endpoint_url, _ = self.get_runtime_endpoint(
+ api_base=api_base,
+ aws_bedrock_runtime_endpoint=aws_bedrock_runtime_endpoint,
+ aws_region_name=aws_region_name,
+ endpoint_type="runtime",
+ )
- return self.format_url(endpoint, api_base, request_query_params or {}), api_base
+ return self.format_url(endpoint, endpoint_url, request_query_params or {}), endpoint_url
def sign_request(
self,
diff --git a/litellm/llms/bedrock/rerank/transformation.py b/litellm/llms/bedrock/rerank/transformation.py
index be8250a9671..b5d33eda49f 100644
--- a/litellm/llms/bedrock/rerank/transformation.py
+++ b/litellm/llms/bedrock/rerank/transformation.py
@@ -4,7 +4,7 @@ Translates from Cohere's `/v1/rerank` input format to Bedrock's `/rerank` input
Why separate file? Make it easy to see how transformation works
"""
-import uuid
+from litellm._uuid import uuid
from typing import List, Optional, Union
from litellm.types.llms.bedrock import (
diff --git a/litellm/llms/cohere/common_utils.py b/litellm/llms/cohere/common_utils.py
index 6dbe52d575e..d194d9556b6 100644
--- a/litellm/llms/cohere/common_utils.py
+++ b/litellm/llms/cohere/common_utils.py
@@ -31,7 +31,7 @@ def validate_environment(
"Request-Source": "unspecified:litellm",
"accept": "application/json",
"content-type": "application/json",
- "Authorization": "bearer $CO_API_KEY"
+ "Authorization": "Bearer $CO_API_KEY"
}
"""
headers.update(
@@ -42,7 +42,7 @@ def validate_environment(
}
)
if api_key:
- headers["Authorization"] = f"bearer {api_key}"
+ headers["Authorization"] = f"Bearer {api_key}"
return headers
diff --git a/litellm/llms/cohere/completion/transformation.py b/litellm/llms/cohere/completion/transformation.py
deleted file mode 100644
index f96ef89d3c5..00000000000
--- a/litellm/llms/cohere/completion/transformation.py
+++ /dev/null
@@ -1,265 +0,0 @@
-import time
-from typing import TYPE_CHECKING, Any, AsyncIterator, Iterator, List, Optional, Union
-
-import httpx
-
-import litellm
-from litellm.litellm_core_utils.prompt_templates.common_utils import (
- convert_content_list_to_str,
-)
-from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
-from litellm.types.llms.openai import AllMessageValues
-from litellm.types.utils import Choices, Message, ModelResponse, Usage
-
-from ..common_utils import CohereError
-from ..common_utils import ModelResponseIterator as CohereModelResponseIterator
-from ..common_utils import validate_environment as cohere_validate_environment
-
-if TYPE_CHECKING:
- from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
-
- LiteLLMLoggingObj = _LiteLLMLoggingObj
-else:
- LiteLLMLoggingObj = Any
-
-
-class CohereTextConfig(BaseConfig):
- """
- Reference: https://docs.cohere.com/reference/generate
-
- The class `CohereConfig` provides configuration for the Cohere's API interface. Below are the parameters:
-
- - `num_generations` (integer): Maximum number of generations returned. Default is 1, with a minimum value of 1 and a maximum value of 5.
-
- - `max_tokens` (integer): Maximum number of tokens the model will generate as part of the response. Default value is 20.
-
- - `truncate` (string): Specifies how the API handles inputs longer than maximum token length. Options include NONE, START, END. Default is END.
-
- - `temperature` (number): A non-negative float controlling the randomness in generation. Lower temperatures result in less random generations. Default is 0.75.
-
- - `preset` (string): Identifier of a custom preset, a combination of parameters such as prompt, temperature etc.
-
- - `end_sequences` (array of strings): The generated text gets cut at the beginning of the earliest occurrence of an end sequence, which will be excluded from the text.
-
- - `stop_sequences` (array of strings): The generated text gets cut at the end of the earliest occurrence of a stop sequence, which will be included in the text.
-
- - `k` (integer): Limits generation at each step to top `k` most likely tokens. Default is 0.
-
- - `p` (number): Limits generation at each step to most likely tokens with total probability mass of `p`. Default is 0.
-
- - `frequency_penalty` (number): Reduces repetitiveness of generated tokens. Higher values apply stronger penalties to previously occurred tokens.
-
- - `presence_penalty` (number): Reduces repetitiveness of generated tokens. Similar to frequency_penalty, but this penalty applies equally to all tokens that have already appeared.
-
- - `return_likelihoods` (string): Specifies how and if token likelihoods are returned with the response. Options include GENERATION, ALL and NONE.
-
- - `logit_bias` (object): Used to prevent the model from generating unwanted tokens or to incentivize it to include desired tokens. e.g. {"hello_world": 1233}
- """
-
- num_generations: Optional[int] = None
- max_tokens: Optional[int] = None
- truncate: Optional[str] = None
- temperature: Optional[int] = None
- preset: Optional[str] = None
- end_sequences: Optional[list] = None
- stop_sequences: Optional[list] = None
- k: Optional[int] = None
- p: Optional[int] = None
- frequency_penalty: Optional[int] = None
- presence_penalty: Optional[int] = None
- return_likelihoods: Optional[str] = None
- logit_bias: Optional[dict] = None
-
- def __init__(
- self,
- num_generations: Optional[int] = None,
- max_tokens: Optional[int] = None,
- truncate: Optional[str] = None,
- temperature: Optional[int] = None,
- preset: Optional[str] = None,
- end_sequences: Optional[list] = None,
- stop_sequences: Optional[list] = None,
- k: Optional[int] = None,
- p: Optional[int] = None,
- frequency_penalty: Optional[int] = None,
- presence_penalty: Optional[int] = None,
- return_likelihoods: Optional[str] = None,
- logit_bias: Optional[dict] = None,
- ) -> None:
- locals_ = locals().copy()
- for key, value in locals_.items():
- if key != "self" and value is not None:
- setattr(self.__class__, key, value)
-
- @classmethod
- def get_config(cls):
- return super().get_config()
-
- def validate_environment(
- self,
- headers: dict,
- model: str,
- messages: List[AllMessageValues],
- optional_params: dict,
- litellm_params: dict,
- api_key: Optional[str] = None,
- api_base: Optional[str] = None,
- ) -> dict:
- return cohere_validate_environment(
- headers=headers,
- model=model,
- messages=messages,
- optional_params=optional_params,
- api_key=api_key,
- )
-
- def get_error_class(
- self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
- ) -> BaseLLMException:
- return CohereError(status_code=status_code, message=error_message)
-
- def get_supported_openai_params(self, model: str) -> List:
- return [
- "stream",
- "temperature",
- "max_tokens",
- "logit_bias",
- "top_p",
- "frequency_penalty",
- "presence_penalty",
- "stop",
- "n",
- "extra_headers",
- ]
-
- def map_openai_params(
- self,
- non_default_params: dict,
- optional_params: dict,
- model: str,
- drop_params: bool,
- ) -> dict:
- for param, value in non_default_params.items():
- if param == "stream":
- optional_params["stream"] = value
- elif param == "temperature":
- optional_params["temperature"] = value
- elif param == "max_tokens":
- optional_params["max_tokens"] = value
- elif param == "n":
- optional_params["num_generations"] = value
- elif param == "logit_bias":
- optional_params["logit_bias"] = value
- elif param == "top_p":
- optional_params["p"] = value
- elif param == "frequency_penalty":
- optional_params["frequency_penalty"] = value
- elif param == "presence_penalty":
- optional_params["presence_penalty"] = value
- elif param == "stop":
- optional_params["stop_sequences"] = value
- return optional_params
-
- def transform_request(
- self,
- model: str,
- messages: List[AllMessageValues],
- optional_params: dict,
- litellm_params: dict,
- headers: dict,
- ) -> dict:
- prompt = " ".join(
- convert_content_list_to_str(message=message) for message in messages
- )
-
- ## Load Config
- config = litellm.CohereConfig.get_config()
- for k, v in config.items():
- if (
- k not in optional_params
- ): # completion(top_k=3) > cohere_config(top_k=3) <- allows for dynamic variables to be passed in
- optional_params[k] = v
-
- ## Handle Tool Calling
- if "tools" in optional_params:
- _is_function_call = True
- tool_calling_system_prompt = self._construct_cohere_tool_for_completion_api(
- tools=optional_params["tools"]
- )
- optional_params["tools"] = tool_calling_system_prompt
-
- data = {
- "model": model,
- "prompt": prompt,
- **optional_params,
- }
-
- return data
-
- def transform_response(
- self,
- model: str,
- raw_response: httpx.Response,
- model_response: ModelResponse,
- logging_obj: LiteLLMLoggingObj,
- request_data: dict,
- messages: List[AllMessageValues],
- optional_params: dict,
- litellm_params: dict,
- encoding: Any,
- api_key: Optional[str] = None,
- json_mode: Optional[bool] = None,
- ) -> ModelResponse:
- prompt = " ".join(
- convert_content_list_to_str(message=message) for message in messages
- )
- completion_response = raw_response.json()
- choices_list = []
- for idx, item in enumerate(completion_response["generations"]):
- if len(item["text"]) > 0:
- message_obj = Message(content=item["text"])
- else:
- message_obj = Message(content=None)
- choice_obj = Choices(
- finish_reason=item["finish_reason"],
- index=idx + 1,
- message=message_obj,
- )
- choices_list.append(choice_obj)
- model_response.choices = choices_list # type: ignore
-
- ## CALCULATING USAGE
- prompt_tokens = len(encoding.encode(prompt))
- completion_tokens = len(
- encoding.encode(model_response["choices"][0]["message"].get("content", ""))
- )
-
- model_response.created = int(time.time())
- model_response.model = model
- usage = Usage(
- prompt_tokens=prompt_tokens,
- completion_tokens=completion_tokens,
- total_tokens=prompt_tokens + completion_tokens,
- )
- setattr(model_response, "usage", usage)
- return model_response
-
- def _construct_cohere_tool_for_completion_api(
- self,
- tools: Optional[List] = None,
- ) -> dict:
- if tools is None:
- tools = []
- return {"tools": tools}
-
- def get_model_response_iterator(
- self,
- streaming_response: Union[Iterator[str], AsyncIterator[str], ModelResponse],
- sync_stream: bool,
- json_mode: Optional[bool] = False,
- ):
- return CohereModelResponseIterator(
- streaming_response=streaming_response,
- sync_stream=sync_stream,
- json_mode=json_mode,
- )
diff --git a/litellm/llms/cohere/rerank/transformation.py b/litellm/llms/cohere/rerank/transformation.py
index 5371b9a4b61..f9c979712da 100644
--- a/litellm/llms/cohere/rerank/transformation.py
+++ b/litellm/llms/cohere/rerank/transformation.py
@@ -1,8 +1,8 @@
from typing import Any, Dict, List, Optional, Union
import httpx
-import litellm
+import litellm
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig
@@ -52,20 +52,20 @@ class CohereRerankConfig(BaseRerankConfig):
return_documents: Optional[bool] = True,
max_chunks_per_doc: Optional[int] = None,
max_tokens_per_doc: Optional[int] = None,
- ) -> OptionalRerankParams:
+ ) -> Dict:
"""
Map Cohere rerank params
No mapping required - returns all supported params
"""
- return OptionalRerankParams(
+ return dict(OptionalRerankParams(
query=query,
documents=documents,
top_n=top_n,
rank_fields=rank_fields,
return_documents=return_documents,
max_chunks_per_doc=max_chunks_per_doc,
- )
+ ))
def validate_environment(
self,
@@ -86,7 +86,7 @@ class CohereRerankConfig(BaseRerankConfig):
)
default_headers = {
- "Authorization": f"bearer {api_key}",
+ "Authorization": f"Bearer {api_key}",
"accept": "application/json",
"content-type": "application/json",
}
@@ -101,7 +101,7 @@ class CohereRerankConfig(BaseRerankConfig):
def transform_rerank_request(
self,
model: str,
- optional_rerank_params: OptionalRerankParams,
+ optional_rerank_params: Dict,
headers: dict,
) -> dict:
if "query" not in optional_rerank_params:
diff --git a/litellm/llms/cohere/rerank_v2/transformation.py b/litellm/llms/cohere/rerank_v2/transformation.py
index 74e760460d0..eb551a8a949 100644
--- a/litellm/llms/cohere/rerank_v2/transformation.py
+++ b/litellm/llms/cohere/rerank_v2/transformation.py
@@ -44,25 +44,25 @@ class CohereRerankV2Config(CohereRerankConfig):
return_documents: Optional[bool] = True,
max_chunks_per_doc: Optional[int] = None,
max_tokens_per_doc: Optional[int] = None,
- ) -> OptionalRerankParams:
+ ) -> Dict:
"""
Map Cohere rerank params
No mapping required - returns all supported params
"""
- return OptionalRerankParams(
+ return dict(OptionalRerankParams(
query=query,
documents=documents,
top_n=top_n,
rank_fields=rank_fields,
return_documents=return_documents,
max_tokens_per_doc=max_tokens_per_doc,
- )
+ ))
def transform_rerank_request(
self,
model: str,
- optional_rerank_params: OptionalRerankParams,
+ optional_rerank_params: Dict,
headers: dict,
) -> dict:
if "query" not in optional_rerank_params:
diff --git a/litellm/llms/cometapi/chat/transformation.py b/litellm/llms/cometapi/chat/transformation.py
new file mode 100644
index 00000000000..fedb8f61e5b
--- /dev/null
+++ b/litellm/llms/cometapi/chat/transformation.py
@@ -0,0 +1,207 @@
+"""
+Support for CometAPI's `/v1/chat/completions` endpoint.
+
+Based on OpenAI-compatible API interface implementation
+Documentation: [CometAPI Documentation Link]
+"""
+
+from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union
+
+import httpx
+
+from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolParam
+from litellm.types.utils import ModelResponse, ModelResponseStream
+
+from ...openai.chat.gpt_transformation import OpenAIGPTConfig
+from ..common_utils import CometAPIException
+
+
+class CometAPIConfig(OpenAIGPTConfig):
+ """
+ CometAPI configuration class, inherits from OpenAIGPTConfig
+
+ Since CometAPI is OpenAI-compatible API, we inherit from OpenAIGPTConfig
+ and only need to override necessary methods to handle CometAPI-specific features
+ """
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ """
+ Map OpenAI format parameters to CometAPI format
+ """
+ mapped_openai_params = super().map_openai_params(
+ non_default_params, optional_params, model, drop_params
+ )
+
+ # CometAPI-specific parameters (if any)
+ extra_body: dict[str, Any] = {}
+ # TODO: Add CometAPI-specific parameter handling here
+ # Example:
+ # custom_param = non_default_params.pop("custom_param", None)
+ # if custom_param is not None:
+ # extra_body["custom_param"] = custom_param
+
+ if extra_body:
+ mapped_openai_params["extra_body"] = extra_body
+
+ return mapped_openai_params
+
+ def remove_cache_control_flag_from_messages_and_tools(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ tools: Optional[List["ChatCompletionToolParam"]] = None,
+ ) -> Tuple[List[AllMessageValues], Optional[List["ChatCompletionToolParam"]]]:
+ """
+ Remove cache control flags from messages and tools if not supported
+ """
+ # For CometAPI, use default behavior (remove cache control)
+ return super().remove_cache_control_flag_from_messages_and_tools(
+ model, messages, tools
+ )
+
+ def transform_request(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ headers: dict,
+ ) -> dict:
+ """
+ Transform the overall request to be sent to the API.
+
+ Returns:
+ dict: The transformed request. Sent as the body of the API call.
+ """
+ extra_body = optional_params.pop("extra_body", {})
+ response = super().transform_request(
+ model, messages, optional_params, litellm_params, headers
+ )
+ response.update(extra_body)
+ return response
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ """
+ Get the complete URL for the CometAPI call.
+
+ Returns:
+ str: The complete URL for the API call.
+ """
+ # Default base
+ if api_base is None:
+ api_base = "https://api.cometapi.com/v1"
+ endpoint = "chat/completions"
+
+ # Normalize
+ api_base = api_base.rstrip("/")
+
+ # If endpoint already present, return as-is
+ if endpoint in api_base:
+ return api_base
+
+ # Ensure we include /v1 prefix when missing
+ if api_base.endswith("/v1"):
+ return f"{api_base}/{endpoint}"
+ if api_base.endswith("/v1/"):
+ return f"{api_base}{endpoint}"
+ # If user provided https://api.cometapi.com, add /v1
+ if api_base == "https://api.cometapi.com":
+ return f"{api_base}/v1/{endpoint}"
+ # Generic fallback: if '/v1' not in path, add it
+ if "/v1" not in api_base.split("//", 1)[-1]:
+ return f"{api_base}/v1/{endpoint}"
+ return f"{api_base}/{endpoint}"
+
+ def get_error_class(
+ self,
+ error_message: str,
+ status_code: int,
+ headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ """
+ Return CometAPI-specific error class
+ """
+ return CometAPIException(
+ message=error_message,
+ status_code=status_code,
+ headers=headers,
+ )
+
+ def get_model_response_iterator(
+ self,
+ streaming_response: Union[Iterator[str], AsyncIterator[str], ModelResponse],
+ sync_stream: bool,
+ json_mode: Optional[bool] = False,
+ ) -> Any:
+ """
+ Get model response iterator for streaming responses
+ """
+ return CometAPIChatCompletionStreamingHandler(
+ streaming_response=streaming_response,
+ sync_stream=sync_stream,
+ json_mode=json_mode,
+ )
+
+
+class CometAPIChatCompletionStreamingHandler(BaseModelResponseIterator):
+ """
+ Handler for CometAPI streaming chat completion responses
+ """
+
+ def chunk_parser(self, chunk: dict) -> ModelResponseStream:
+ """
+ Parse individual chunks from streaming response
+ """
+ try:
+ # Handle error in chunk
+ if "error" in chunk:
+ error_chunk = chunk["error"]
+ error_message = "CometAPI Error: {}".format(
+ error_chunk.get("message", "Unknown error")
+ )
+ raise CometAPIException(
+ message=error_message,
+ status_code=error_chunk.get("code", 400),
+ headers={"Content-Type": "application/json"},
+ )
+
+ # Process choices
+ new_choices = []
+ for choice in chunk["choices"]:
+ # Handle reasoning content if present
+ if "delta" in choice and "reasoning" in choice["delta"]:
+ choice["delta"]["reasoning_content"] = choice["delta"].get("reasoning")
+ new_choices.append(choice)
+
+ return ModelResponseStream(
+ id=chunk["id"],
+ object="chat.completion.chunk",
+ created=chunk["created"],
+ usage=chunk.get("usage"),
+ model=chunk["model"],
+ choices=new_choices,
+ )
+ except KeyError as e:
+ raise CometAPIException(
+ message=f"KeyError: {e}, Got unexpected response from CometAPI: {chunk}",
+ status_code=400,
+ headers={"Content-Type": "application/json"},
+ )
+ except Exception as e:
+ raise e
diff --git a/litellm/llms/cometapi/common_utils.py b/litellm/llms/cometapi/common_utils.py
new file mode 100644
index 00000000000..2e5e3e5fab7
--- /dev/null
+++ b/litellm/llms/cometapi/common_utils.py
@@ -0,0 +1,6 @@
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+
+
+class CometAPIException(BaseLLMException):
+ """CometAPI exception handling class"""
+ pass
diff --git a/litellm/llms/compactifai/__init__.py b/litellm/llms/compactifai/__init__.py
new file mode 100644
index 00000000000..16b0c04cdab
--- /dev/null
+++ b/litellm/llms/compactifai/__init__.py
@@ -0,0 +1 @@
+# CompactifAI provider for LiteLLM
\ No newline at end of file
diff --git a/litellm/llms/compactifai/chat/__init__.py b/litellm/llms/compactifai/chat/__init__.py
new file mode 100644
index 00000000000..d1a4463166b
--- /dev/null
+++ b/litellm/llms/compactifai/chat/__init__.py
@@ -0,0 +1 @@
+# CompactifAI chat completions
\ No newline at end of file
diff --git a/litellm/llms/compactifai/chat/transformation.py b/litellm/llms/compactifai/chat/transformation.py
new file mode 100644
index 00000000000..5cb8cd9a4ab
--- /dev/null
+++ b/litellm/llms/compactifai/chat/transformation.py
@@ -0,0 +1,100 @@
+"""
+CompactifAI chat completion transformation
+"""
+
+from typing import TYPE_CHECKING, Any, List, Optional, Tuple, Union
+
+import httpx
+
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.utils import ModelResponse
+from litellm.llms.openai.common_utils import OpenAIError
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+
+from ...openai.chat.gpt_transformation import OpenAIGPTConfig
+
+if TYPE_CHECKING:
+ from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
+
+ LiteLLMLoggingObj = _LiteLLMLoggingObj
+else:
+ LiteLLMLoggingObj = Any
+
+
+class CompactifAIChatConfig(OpenAIGPTConfig):
+ """
+ Configuration class for CompactifAI chat completions.
+ Since CompactifAI is OpenAI-compatible, we extend OpenAIGPTConfig.
+ """
+
+ def _get_openai_compatible_provider_info(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ ) -> Tuple[Optional[str], Optional[str]]:
+ """
+ Get API base and key for CompactifAI provider.
+ """
+ api_base = api_base or "https://api.compactif.ai/v1"
+ dynamic_api_key = api_key or get_secret_str("COMPACTIFAI_API_KEY") or ""
+ return api_base, dynamic_api_key
+
+ def transform_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: ModelResponse,
+ logging_obj: LiteLLMLoggingObj,
+ request_data: dict,
+ messages: List,
+ optional_params: dict,
+ litellm_params: dict,
+ encoding: Any,
+ api_key: Optional[str] = None,
+ json_mode: Optional[bool] = None,
+ ) -> ModelResponse:
+ """
+ Transform CompactifAI response to LiteLLM format.
+ Since CompactifAI is OpenAI-compatible, we can use the standard OpenAI transformation.
+ """
+ ## LOGGING
+ logging_obj.post_call(
+ input=messages,
+ api_key=api_key,
+ original_response=raw_response.text,
+ additional_args={"complete_input_dict": request_data},
+ )
+
+ ## RESPONSE OBJECT
+ response_json = raw_response.json()
+
+ # Handle JSON mode if needed
+ if json_mode:
+ for choice in response_json["choices"]:
+ message = choice.get("message")
+ if message and message.get("tool_calls"):
+ # Convert tool calls to content for JSON mode
+ tool_calls = message.get("tool_calls", [])
+ if len(tool_calls) == 1:
+ message["content"] = tool_calls[0]["function"].get("arguments", "")
+ message["tool_calls"] = None
+
+ returned_response = ModelResponse(**response_json)
+
+ # Set model name with provider prefix
+ returned_response.model = f"compactifai/{model}"
+
+ return returned_response
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ """
+ Get the appropriate error class for CompactifAI errors.
+ Since CompactifAI is OpenAI-compatible, we use OpenAI error handling.
+ """
+ return OpenAIError(
+ status_code=status_code,
+ message=error_message,
+ headers=headers,
+ )
\ No newline at end of file
diff --git a/litellm/llms/custom_httpx/aiohttp_handler.py b/litellm/llms/custom_httpx/aiohttp_handler.py
index d9fc85877c3..c7a04a49fc2 100644
--- a/litellm/llms/custom_httpx/aiohttp_handler.py
+++ b/litellm/llms/custom_httpx/aiohttp_handler.py
@@ -17,6 +17,7 @@ from litellm.llms.custom_httpx.http_handler import (
HTTPHandler,
_get_httpx_client,
)
+from litellm.llms.custom_httpx.aiohttp_transport import LiteLLMAiohttpTransport
from litellm.types.llms.openai import FileTypes
from litellm.types.utils import HttpHandlerRequestFields, ImageResponse, LlmProviders
from litellm.utils import CustomStreamWrapper, ModelResponse, ProviderConfigManager
@@ -32,8 +33,71 @@ DEFAULT_TIMEOUT = 600
class BaseLLMAIOHTTPHandler:
- def __init__(self):
- self.client_session: Optional[aiohttp.ClientSession] = None
+ def __init__(
+ self,
+ client_session: Optional[aiohttp.ClientSession] = None,
+ transport: Optional[LiteLLMAiohttpTransport] = None,
+ connector: Optional[aiohttp.BaseConnector] = None,
+ ):
+ self.client_session = client_session
+ self._owns_session = (
+ client_session is None
+ ) # Track if we own the session for cleanup
+
+ self.transport = transport
+ self._owns_transport = (
+ transport is None
+ ) # Track if we own the transport for cleanup
+
+ self.connector = connector
+ self._owns_connector = (
+ connector is None
+ ) # Track if we own the connector for cleanup
+
+ def _get_or_create_transport(self) -> Optional[LiteLLMAiohttpTransport]:
+ """Get existing transport or create a new one if needed."""
+ if self.transport:
+ return self.transport
+
+ # Create a transport using AsyncHTTPHandler's logic
+ try:
+ self.transport = AsyncHTTPHandler._create_aiohttp_transport()
+ self._owns_transport = True
+ return self.transport
+ except Exception:
+ # If transport creation fails, return None (will use direct session)
+ return None
+
+ def _get_connector(self) -> Optional[aiohttp.BaseConnector]:
+ """Get or create a connector for the client session."""
+ if self.connector:
+ return self.connector
+ elif self.transport and hasattr(self.transport, "client"):
+ # Extract connector from transport if available
+ client = self.transport.client
+ if callable(client):
+ # If client is a factory, we can't extract connector directly
+ return None
+ elif hasattr(client, "connector"):
+ return client.connector
+ return None
+
+ def _create_client_session_with_transport(self) -> ClientSession:
+ """Create a new client session using transport or connector configuration."""
+ connector = self._get_connector()
+
+ if self.transport and hasattr(self.transport, "_get_valid_client_session"):
+ # Use transport's session creation if available
+ session = self.transport._get_valid_client_session()
+ return session
+ elif connector:
+ # Use provided connector
+ session = aiohttp.ClientSession(connector=connector)
+ return session
+ else:
+ # Default session creation
+ session = aiohttp.ClientSession()
+ return session
def _get_async_client_session(
self, dynamic_client_session: Optional[ClientSession] = None
@@ -43,15 +107,33 @@ class BaseLLMAIOHTTPHandler:
elif self.client_session:
return self.client_session
else:
- # init client session, and then return new session
- self.client_session = aiohttp.ClientSession()
+ # Create client session using transport/connector if available
+ self.client_session = self._create_client_session_with_transport()
+ self._owns_session = True # We created this session, so we own it
return self.client_session
async def close(self):
- """Close the aiohttp client session if it exists."""
- if self.client_session and not self.client_session.closed:
+ """Close the aiohttp client session and transport if we own them."""
+ # Close client session if we own it
+ if (
+ self.client_session
+ and not self.client_session.closed
+ and self._owns_session
+ ):
await self.client_session.close()
+ # Close transport if we own it
+ if (
+ self.transport
+ and self._owns_transport
+ and hasattr(self.transport, "aclose")
+ ):
+ try:
+ await self.transport.aclose()
+ except Exception:
+ # Ignore errors during transport cleanup
+ pass
+
async def _make_common_async_call(
self,
async_client_session: Optional[ClientSession],
diff --git a/litellm/llms/custom_httpx/aiohttp_transport.py b/litellm/llms/custom_httpx/aiohttp_transport.py
index 3ed7d04bde6..50bbccd6a4b 100644
--- a/litellm/llms/custom_httpx/aiohttp_transport.py
+++ b/litellm/llms/custom_httpx/aiohttp_transport.py
@@ -3,7 +3,7 @@ import contextlib
import os
import typing
import urllib.request
-from typing import Callable, Dict, Union
+from typing import Callable, Dict, Optional, Union
import aiohttp
import aiohttp.client_exceptions
@@ -115,6 +115,12 @@ class AiohttpTransport(httpx.AsyncBaseTransport):
) -> None:
self.client = client
+ #########################################################
+ # Class variables for proxy settings
+ #########################################################
+ self.proxy: Optional[str] = None
+ self.checked_proxy_env_settings: bool = False
+
async def aclose(self) -> None:
if isinstance(self.client, ClientSession):
await self.client.close()
@@ -146,6 +152,16 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
# If we don't have a client or it's not a ClientSession, create one
if not isinstance(self.client, ClientSession):
+ if hasattr(self, "_client_factory") and callable(self._client_factory):
+ self.client = self._client_factory()
+ else:
+ self.client = ClientSession()
+ # Don't return yet - check if the newly created session is valid
+
+ # Check if the session itself is closed
+ if self.client.closed:
+ verbose_logger.debug("Session is closed, creating new session")
+ # Create a new session
if hasattr(self, "_client_factory") and callable(self._client_factory):
self.client = self._client_factory()
else:
@@ -163,14 +179,17 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
or session_loop != current_loop
or session_loop.is_closed()
):
- # Clean up the old session
+ # Close old session to prevent leaks
+ old_session = self.client
try:
- # Note: not awaiting close() here as it might be from a different loop
- # The session will be garbage collected
- pass
+ if not old_session.closed:
+ try:
+ asyncio.create_task(old_session.close())
+ except RuntimeError:
+ # Different event loop - can't schedule task, rely on GC
+ verbose_logger.debug("Old session from different loop, relying on GC")
except Exception as e:
verbose_logger.debug(f"Error closing old session: {e}")
- pass
# Create a new session in the current event loop
if hasattr(self, "_client_factory") and callable(self._client_factory):
@@ -187,13 +206,58 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
return self.client
+ async def _make_aiohttp_request(
+ self,
+ client_session: ClientSession,
+ request: httpx.Request,
+ timeout: dict,
+ proxy: Optional[str],
+ sni_hostname: Optional[str],
+ ) -> ClientResponse:
+ """
+ Helper function to make an aiohttp request with the given parameters.
+
+ Args:
+ client_session: The aiohttp ClientSession to use
+ request: The httpx Request to send
+ timeout: Timeout settings dict with 'connect', 'read', 'pool' keys
+ proxy: Optional proxy URL
+ sni_hostname: Optional SNI hostname for SSL
+
+ Returns:
+ ClientResponse from aiohttp
+ """
+ from aiohttp import ClientTimeout
+ from yarl import URL as YarlURL
+
+ try:
+ data = request.content
+ except httpx.RequestNotRead:
+ data = request.stream # type: ignore
+ request.headers.pop("transfer-encoding", None) # handled by aiohttp
+
+ response = await client_session.request(
+ method=request.method,
+ url=YarlURL(str(request.url), encoded=True),
+ headers=request.headers,
+ data=data,
+ allow_redirects=False,
+ auto_decompress=False,
+ timeout=ClientTimeout(
+ sock_connect=timeout.get("connect"),
+ sock_read=timeout.get("read"),
+ connect=timeout.get("pool"),
+ ),
+ proxy=proxy,
+ server_hostname=sni_hostname,
+ ).__aenter__()
+
+ return response
+
async def handle_async_request(
self,
request: httpx.Request,
) -> httpx.Response:
- from aiohttp import ClientTimeout
- from yarl import URL as YarlURL
-
timeout = request.extensions.get("timeout", {})
sni_hostname = request.extensions.get("sni_hostname")
@@ -203,28 +267,38 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
# Resolve proxy settings from environment variables
proxy = await self._get_proxy_settings(request)
- with map_aiohttp_exceptions():
- try:
- data = request.content
- except httpx.RequestNotRead:
- data = request.stream # type: ignore
- request.headers.pop("transfer-encoding", None) # handled by aiohttp
-
- response = await client_session.request(
- method=request.method,
- url=YarlURL(str(request.url), encoded=True),
- headers=request.headers,
- data=data,
- allow_redirects=False,
- auto_decompress=False,
- timeout=ClientTimeout(
- sock_connect=timeout.get("connect"),
- sock_read=timeout.get("read"),
- connect=timeout.get("pool"),
- ),
- proxy=proxy,
- server_hostname=sni_hostname,
- ).__aenter__()
+ try:
+ with map_aiohttp_exceptions():
+ response = await self._make_aiohttp_request(
+ client_session=client_session,
+ request=request,
+ timeout=timeout,
+ proxy=proxy,
+ sni_hostname=sni_hostname,
+ )
+ except RuntimeError as e:
+ # Handle the case where session was closed between our check and actual use
+ if "Session is closed" in str(e):
+ verbose_logger.debug(f"Session closed during request, retrying with new session: {e}")
+ # Force creation of a new session
+ if hasattr(self, "_client_factory") and callable(self._client_factory):
+ self.client = self._client_factory()
+ else:
+ self.client = ClientSession()
+ client_session = self.client
+
+ # Retry the request with the new session
+ with map_aiohttp_exceptions():
+ response = await self._make_aiohttp_request(
+ client_session=client_session,
+ request=request,
+ timeout=timeout,
+ proxy=proxy,
+ sni_hostname=sni_hostname,
+ )
+ else:
+ # Re-raise if it's a different RuntimeError
+ raise
return httpx.Response(
status_code=response.status,
@@ -249,7 +323,22 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
def _proxy_from_env(self, url: httpx.URL) -> typing.Optional[str]:
- """Return proxy URL from env for the given request URL."""
+ """
+ Return proxy URL from env for the given request URL
+
+ Only check the proxy env settings once, this is a costly operation for CPU % usage
+
+ ."""
+ #########################################################
+ # Check if we've already checked the proxy env settings
+ #########################################################
+ if self.checked_proxy_env_settings is True:
+ return self.proxy
+
+ #########################################################
+ # set self.checked_proxy_env_settings to True
+ #########################################################
+ self.checked_proxy_env_settings = True
proxies = urllib.request.getproxies()
if urllib.request.proxy_bypass(url.host):
return None
@@ -257,4 +346,5 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
proxy = proxies.get(url.scheme) or proxies.get("all")
if proxy and "://" not in proxy:
proxy = f"http://{proxy}"
- return proxy
+ self.proxy = proxy
+ return self.proxy
diff --git a/litellm/llms/custom_httpx/http_handler.py b/litellm/llms/custom_httpx/http_handler.py
index cf2187153a9..a3ad2c67272 100644
--- a/litellm/llms/custom_httpx/http_handler.py
+++ b/litellm/llms/custom_httpx/http_handler.py
@@ -12,7 +12,13 @@ from httpx._types import RequestFiles
import litellm
from litellm._logging import verbose_logger
-from litellm.constants import _DEFAULT_TTL_FOR_HTTPX_CLIENTS
+from litellm.constants import (
+ _DEFAULT_TTL_FOR_HTTPX_CLIENTS,
+ AIOHTTP_CONNECTOR_LIMIT,
+ AIOHTTP_KEEPALIVE_TIMEOUT,
+ AIOHTTP_TTL_DNS_CACHE,
+ DEFAULT_SSL_CIPHERS
+)
from litellm.litellm_core_utils.logging_utils import track_llm_api_timing
from litellm.types.llms.custom_http import *
@@ -40,7 +46,9 @@ headers = {
_DEFAULT_TIMEOUT = httpx.Timeout(timeout=5.0, connect=5.0)
-def get_ssl_configuration(ssl_verify: Optional[VerifyTypes] = None) -> Union[bool, str, ssl.SSLContext]:
+def get_ssl_configuration(
+ ssl_verify: Optional[VerifyTypes] = None,
+) -> Union[bool, str, ssl.SSLContext]:
"""
Unified SSL configuration function that handles ssl_context and ssl_verify logic.
@@ -59,7 +67,7 @@ def get_ssl_configuration(ssl_verify: Optional[VerifyTypes] = None) -> Union[boo
- False: Disable SSL verification
- True: Enable SSL verification
- str: Path to CA bundle file
-
+
Returns:
Union[bool, str, ssl.SSLContext]: Appropriate SSL configuration
"""
@@ -72,7 +80,9 @@ def get_ssl_configuration(ssl_verify: Optional[VerifyTypes] = None) -> Union[boo
# Get ssl_verify from environment or litellm settings if not provided
if ssl_verify is None:
ssl_verify = os.getenv("SSL_VERIFY", litellm.ssl_verify)
- ssl_verify_bool = str_to_bool(ssl_verify) if isinstance(ssl_verify, str) else ssl_verify
+ ssl_verify_bool = (
+ str_to_bool(ssl_verify) if isinstance(ssl_verify, str) else ssl_verify
+ )
if ssl_verify_bool is not None:
ssl_verify = ssl_verify_bool
@@ -89,16 +99,20 @@ def get_ssl_configuration(ssl_verify: Optional[VerifyTypes] = None) -> Union[boo
cafile = certifi.where()
if ssl_verify is not False:
- custom_ssl_context = ssl.create_default_context(
- cafile=cafile
- )
- # If security level is set, apply it to the SSL context
- if (
- ssl_security_level
- and isinstance(ssl_security_level, str)
- ):
- # Create a custom SSL context with reduced security level
+ custom_ssl_context = ssl.create_default_context(cafile=cafile)
+
+ # Optimize SSL handshake performance
+ # Set minimum TLS version to 1.2 for better performance
+ custom_ssl_context.minimum_version = ssl.TLSVersion.TLSv1_2
+
+ # Configure cipher suites for optimal performance
+ if ssl_security_level and isinstance(ssl_security_level, str):
+ # User provided custom cipher configuration (e.g., via SSL_SECURITY_LEVEL env var)
custom_ssl_context.set_ciphers(ssl_security_level)
+ else:
+ # Use optimized cipher list that strongly prefers fast ciphers
+ # but falls back to widely compatible ones
+ custom_ssl_context.set_ciphers(DEFAULT_SSL_CIPHERS)
# Use our custom SSL context instead of the original ssl_verify value
return custom_ssl_context
@@ -165,26 +179,27 @@ class AsyncHTTPHandler:
self,
timeout: Optional[Union[float, httpx.Timeout]] = None,
event_hooks: Optional[Mapping[str, List[Callable[..., Any]]]] = None,
- concurrent_limit=1000,
+ concurrent_limit=None, # Kept for backward compatibility, but ignored (no limits)
client_alias: Optional[str] = None, # name for client in logs
ssl_verify: Optional[VerifyTypes] = None,
+ shared_session: Optional["ClientSession"] = None,
):
self.timeout = timeout
self.event_hooks = event_hooks
self.client = self.create_client(
timeout=timeout,
- concurrent_limit=concurrent_limit,
event_hooks=event_hooks,
ssl_verify=ssl_verify,
+ shared_session=shared_session,
)
self.client_alias = client_alias
def create_client(
self,
timeout: Optional[Union[float, httpx.Timeout]],
- concurrent_limit: int,
event_hooks: Optional[Mapping[str, List[Callable[..., Any]]]],
ssl_verify: Optional[VerifyTypes] = None,
+ shared_session: Optional["ClientSession"] = None,
) -> httpx.AsyncClient:
# Get unified SSL configuration
ssl_config = get_ssl_configuration(ssl_verify)
@@ -200,19 +215,17 @@ class AsyncHTTPHandler:
transport = AsyncHTTPHandler._create_async_transport(
ssl_context=ssl_config if isinstance(ssl_config, ssl.SSLContext) else None,
ssl_verify=ssl_config if isinstance(ssl_config, bool) else None,
+ shared_session=shared_session,
)
return httpx.AsyncClient(
transport=transport,
event_hooks=event_hooks,
timeout=timeout,
- limits=httpx.Limits(
- max_connections=concurrent_limit,
- max_keepalive_connections=concurrent_limit,
- ),
verify=ssl_config,
cert=cert,
headers=headers,
+ follow_redirects=True,
)
async def close(self):
@@ -282,7 +295,7 @@ class AsyncHTTPHandler:
except (httpx.RemoteProtocolError, httpx.ConnectError):
# Retry the request with a new session if there is a connection error
new_client = self.create_client(
- timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
+ timeout=timeout, event_hooks=self.event_hooks
)
try:
return await self.single_connection_post_request(
@@ -348,7 +361,7 @@ class AsyncHTTPHandler:
except (httpx.RemoteProtocolError, httpx.ConnectError):
# Retry the request with a new session if there is a connection error
new_client = self.create_client(
- timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
+ timeout=timeout, event_hooks=self.event_hooks
)
try:
return await self.single_connection_post_request(
@@ -408,7 +421,7 @@ class AsyncHTTPHandler:
except (httpx.RemoteProtocolError, httpx.ConnectError):
# Retry the request with a new session if there is a connection error
new_client = self.create_client(
- timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
+ timeout=timeout, event_hooks=self.event_hooks
)
try:
return await self.single_connection_post_request(
@@ -467,7 +480,7 @@ class AsyncHTTPHandler:
except (httpx.RemoteProtocolError, httpx.ConnectError):
# Retry the request with a new session if there is a connection error
new_client = self.create_client(
- timeout=timeout, concurrent_limit=1, event_hooks=self.event_hooks
+ timeout=timeout, event_hooks=self.event_hooks
)
try:
return await self.single_connection_post_request(
@@ -522,7 +535,9 @@ class AsyncHTTPHandler:
@staticmethod
def _create_async_transport(
- ssl_context: Optional[ssl.SSLContext] = None, ssl_verify: Optional[bool] = None
+ ssl_context: Optional[ssl.SSLContext] = None,
+ ssl_verify: Optional[bool] = None,
+ shared_session: Optional["ClientSession"] = None,
) -> Optional[Union[LiteLLMAiohttpTransport, AsyncHTTPTransport]]:
"""
- Creates a transport for httpx.AsyncClient
@@ -543,7 +558,9 @@ class AsyncHTTPHandler:
#########################################################
if AsyncHTTPHandler._should_use_aiohttp_transport():
return AsyncHTTPHandler._create_aiohttp_transport(
- ssl_context=ssl_context, ssl_verify=ssl_verify
+ ssl_context=ssl_context,
+ ssl_verify=ssl_verify,
+ shared_session=shared_session,
)
#########################################################
@@ -586,7 +603,7 @@ class AsyncHTTPHandler:
) -> Dict[str, Any]:
"""
Helper method to get SSL connector initialization arguments for aiohttp TCPConnector.
-
+
SSL Configuration Priority:
1. If ssl_context is provided -> use the custom SSL context
2. If ssl_verify is False -> disable SSL verification (ssl=False)
@@ -597,20 +614,21 @@ class AsyncHTTPHandler:
connector_kwargs: Dict[str, Any] = {
"local_addr": ("0.0.0.0", 0) if litellm.force_ipv4 else None,
}
-
+
if ssl_context is not None:
# Priority 1: Use the provided custom SSL context
connector_kwargs["ssl"] = ssl_context
elif ssl_verify is False:
# Priority 2: Explicitly disable SSL verification
connector_kwargs["verify_ssl"] = False
-
+
return connector_kwargs
@staticmethod
def _create_aiohttp_transport(
ssl_verify: Optional[bool] = None,
ssl_context: Optional[ssl.SSLContext] = None,
+ shared_session: Optional["ClientSession"] = None,
) -> LiteLLMAiohttpTransport:
"""
Creates an AiohttpTransport with RequestNotRead error handling
@@ -634,9 +652,27 @@ class AsyncHTTPHandler:
trust_env = True
verbose_logger.debug("Creating AiohttpTransport...")
+
+ # Use shared session if provided and valid
+ if shared_session is not None and not shared_session.closed:
+ verbose_logger.debug(
+ f"SHARED SESSION: Reusing existing ClientSession (ID: {id(shared_session)})"
+ )
+ return LiteLLMAiohttpTransport(client=shared_session)
+
+ # Create new session only if none provided or existing one is invalid
+ verbose_logger.debug(
+ "NEW SESSION: Creating new ClientSession (no shared session provided)"
+ )
return LiteLLMAiohttpTransport(
client=lambda: ClientSession(
- connector=TCPConnector(**connector_kwargs),
+ connector=TCPConnector(
+ limit=AIOHTTP_CONNECTOR_LIMIT,
+ keepalive_timeout=AIOHTTP_KEEPALIVE_TIMEOUT,
+ ttl_dns_cache=AIOHTTP_TTL_DNS_CACHE,
+ enable_cleanup_closed=True,
+ **connector_kwargs
+ ),
trust_env=trust_env,
),
)
@@ -659,7 +695,7 @@ class HTTPHandler:
def __init__(
self,
timeout: Optional[Union[float, httpx.Timeout]] = None,
- concurrent_limit=1000,
+ concurrent_limit=None, # Kept for backward compatibility, but ignored (no limits)
client: Optional[httpx.Client] = None,
ssl_verify: Optional[Union[bool, str]] = None,
):
@@ -680,13 +716,10 @@ class HTTPHandler:
self.client = httpx.Client(
transport=transport,
timeout=timeout,
- limits=httpx.Limits(
- max_connections=concurrent_limit,
- max_keepalive_connections=concurrent_limit,
- ),
verify=ssl_config,
cert=cert,
headers=headers,
+ follow_redirects=True,
)
else:
self.client = client
@@ -913,12 +946,13 @@ class HTTPHandler:
if litellm.force_ipv4:
return HTTPTransport(local_address="0.0.0.0")
else:
- return None
+ return getattr(litellm, 'sync_transport', None)
def get_async_httpx_client(
llm_provider: Union[LlmProviders, httpxSpecialProvider],
params: Optional[dict] = None,
+ shared_session: Optional["ClientSession"] = None,
) -> AsyncHTTPHandler:
"""
Retrieves the async HTTP client from the cache
@@ -940,10 +974,12 @@ def get_async_httpx_client(
return _cached_client
if params is not None:
+ params["shared_session"] = shared_session
_new_client = AsyncHTTPHandler(**params)
else:
_new_client = AsyncHTTPHandler(
- timeout=httpx.Timeout(timeout=600.0, connect=5.0)
+ timeout=httpx.Timeout(timeout=600.0, connect=5.0),
+ shared_session=shared_session,
)
litellm.in_memory_llm_clients_cache.set_cache(
diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py
index 46fd866be2b..5037f4d8d44 100644
--- a/litellm/llms/custom_httpx/llm_http_handler.py
+++ b/litellm/llms/custom_httpx/llm_http_handler.py
@@ -28,6 +28,7 @@ from litellm.llms.base_llm.audio_transcription.transformation import (
BaseAudioTranscriptionConfig,
)
from litellm.llms.base_llm.base_model_iterator import MockResponseIterator
+from litellm.llms.base_llm.batches.transformation import BaseBatchesConfig
from litellm.llms.base_llm.chat.transformation import BaseConfig
from litellm.llms.base_llm.embedding.transformation import BaseEmbeddingConfig
from litellm.llms.base_llm.files.transformation import BaseFilesConfig
@@ -58,15 +59,21 @@ from litellm.types.llms.anthropic_messages.anthropic_response import (
AnthropicMessagesResponse,
)
from litellm.types.llms.openai import (
+ CreateBatchRequest,
CreateFileRequest,
OpenAIFileObject,
ResponseInputParam,
ResponsesAPIResponse,
)
-from litellm.types.rerank import OptionalRerankParams, RerankResponse
+from litellm.types.rerank import RerankResponse
from litellm.types.responses.main import DeleteResponseResult
from litellm.types.router import GenericLiteLLMParams
-from litellm.types.utils import EmbeddingResponse, FileTypes, TranscriptionResponse
+from litellm.types.utils import (
+ EmbeddingResponse,
+ FileTypes,
+ LiteLLMBatch,
+ TranscriptionResponse,
+)
from litellm.types.vector_stores import (
VectorStoreCreateOptionalRequestParams,
VectorStoreCreateResponse,
@@ -81,6 +88,8 @@ from litellm.utils import (
)
if TYPE_CHECKING:
+ from aiohttp import ClientSession
+
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
from litellm.llms.base_llm.passthrough.transformation import BasePassthroughConfig
@@ -229,11 +238,16 @@ class BaseLLMHTTPHandler:
client: Optional[AsyncHTTPHandler] = None,
json_mode: bool = False,
signed_json_body: Optional[bytes] = None,
+ shared_session: Optional["ClientSession"] = None,
):
if client is None:
+ verbose_logger.debug(
+ f"Creating HTTP client with shared_session: {id(shared_session) if shared_session else None}"
+ )
async_httpx_client = get_async_httpx_client(
llm_provider=litellm.LlmProviders(custom_llm_provider),
params={"ssl_verify": litellm_params.get("ssl_verify", None)},
+ shared_session=shared_session,
)
else:
async_httpx_client = client
@@ -268,7 +282,7 @@ class BaseLLMHTTPHandler:
self,
model: str,
messages: list,
- api_base: str,
+ api_base: Optional[str],
custom_llm_provider: str,
model_response: ModelResponse,
encoding,
@@ -283,6 +297,7 @@ class BaseLLMHTTPHandler:
headers: Optional[Dict[str, Any]] = None,
client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
provider_config: Optional[BaseConfig] = None,
+ shared_session: Optional["ClientSession"] = None,
):
json_mode: bool = optional_params.pop("json_mode", False)
extra_body: Optional[dict] = optional_params.pop("extra_body", None)
@@ -411,6 +426,7 @@ class BaseLLMHTTPHandler:
),
json_mode=json_mode,
signed_json_body=signed_json_body,
+ shared_session=shared_session,
)
if stream is True:
@@ -462,7 +478,7 @@ class BaseLLMHTTPHandler:
if client is None or not isinstance(client, HTTPHandler):
sync_httpx_client = _get_httpx_client(
- params={"ssl_verify": litellm_params.get("ssl_verify", None)}
+ params={"ssl_verify": litellm_params.get("ssl_verify", None)},
)
else:
sync_httpx_client = client
@@ -736,7 +752,7 @@ class BaseLLMHTTPHandler:
model_response: EmbeddingResponse,
api_key: Optional[str] = None,
client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
- aembedding: bool = False,
+ aembedding: Optional[bool] = False,
headers: Optional[Dict[str, Any]] = None,
) -> EmbeddingResponse:
provider_config = ProviderConfigManager.get_provider_embedding_config(
@@ -878,7 +894,7 @@ class BaseLLMHTTPHandler:
custom_llm_provider: str,
logging_obj: LiteLLMLoggingObj,
provider_config: BaseRerankConfig,
- optional_rerank_params: OptionalRerankParams,
+ optional_rerank_params: Dict,
timeout: Optional[Union[float, httpx.Timeout]],
model_response: RerankResponse,
_is_async: bool = False,
@@ -1256,6 +1272,10 @@ class BaseLLMHTTPHandler:
stream: Optional[bool] = False,
kwargs: Optional[Dict[str, Any]] = None,
) -> Union[AnthropicMessagesResponse, AsyncIterator]:
+ from litellm.litellm_core_utils.get_provider_specific_headers import (
+ ProviderSpecificHeaderUtils,
+ )
+
if client is None or not isinstance(client, AsyncHTTPHandler):
async_httpx_client = get_async_httpx_client(
llm_provider=litellm.LlmProviders.ANTHROPIC
@@ -1269,10 +1289,9 @@ class BaseLLMHTTPHandler:
Optional[litellm.types.utils.ProviderSpecificHeader],
kwargs.get("provider_specific_header", None),
)
- extra_headers = (
- provider_specific_header.get("extra_headers", {})
- if provider_specific_header
- else {}
+ extra_headers = ProviderSpecificHeaderUtils.get_provider_specific_headers(
+ provider_specific_header=provider_specific_header,
+ custom_llm_provider=custom_llm_provider,
)
(
headers,
@@ -1514,6 +1533,7 @@ class BaseLLMHTTPHandler:
data=data,
fake_stream=fake_stream,
)
+
response = sync_httpx_client.post(
url=api_base,
headers=headers,
@@ -2208,15 +2228,40 @@ class BaseLLMHTTPHandler:
else:
sync_httpx_client = client
- if isinstance(transformed_request, str) or isinstance(
- transformed_request, bytes
- ):
- upload_response = sync_httpx_client.post(
- url=api_base,
- headers=headers,
- data=transformed_request,
+ if isinstance(transformed_request, dict) and "method" in transformed_request:
+ # Handle pre-signed requests (e.g., from Bedrock S3 uploads)
+ upload_response = getattr(
+ sync_httpx_client, transformed_request["method"].lower()
+ )(
+ url=transformed_request["url"],
+ headers=transformed_request["headers"],
+ data=transformed_request["data"],
timeout=timeout,
)
+ elif isinstance(transformed_request, str) or isinstance(
+ transformed_request, bytes
+ ):
+ # Handle traditional file uploads
+ # Ensure transformed_request is a string for httpx compatibility
+ if isinstance(transformed_request, bytes):
+ transformed_request = transformed_request.decode("utf-8")
+
+ # Use the HTTP method specified by the provider config
+ http_method = provider_config.file_upload_http_method.upper()
+ if http_method == "PUT":
+ upload_response = sync_httpx_client.put(
+ url=api_base,
+ headers=headers,
+ data=transformed_request,
+ timeout=timeout,
+ )
+ else: # Default to POST
+ upload_response = sync_httpx_client.post(
+ url=api_base,
+ headers=headers,
+ data=transformed_request,
+ timeout=timeout,
+ )
else:
try:
# Step 1: Initial request to get upload URL
@@ -2249,11 +2294,15 @@ class BaseLLMHTTPHandler:
provider_config=provider_config,
)
+ # Store the upload URL in litellm_params for the transformation method
+ litellm_params_with_url = dict(litellm_params)
+ litellm_params_with_url["upload_url"] = api_base
+
return provider_config.transform_create_file_response(
model=None,
raw_response=upload_response,
logging_obj=logging_obj,
- litellm_params=litellm_params,
+ litellm_params=litellm_params_with_url,
)
async def async_create_file(
@@ -2277,15 +2326,53 @@ class BaseLLMHTTPHandler:
else:
async_httpx_client = client
- if isinstance(transformed_request, str) or isinstance(
- transformed_request, bytes
- ):
- upload_response = await async_httpx_client.post(
- url=api_base,
- headers=headers,
- data=transformed_request,
+ #########################################################
+ # Debug Logging
+ #########################################################
+ logging_obj.pre_call(
+ input="",
+ api_key="",
+ additional_args={
+ "complete_input_dict": transformed_request,
+ "api_base": api_base,
+ "headers": headers,
+ },
+ )
+
+ if isinstance(transformed_request, dict) and "method" in transformed_request:
+ # Handle pre-signed requests (e.g., from Bedrock S3 uploads)
+ upload_response = await getattr(
+ async_httpx_client, transformed_request["method"].lower()
+ )(
+ url=transformed_request["url"],
+ headers=transformed_request["headers"],
+ data=transformed_request["data"],
timeout=timeout,
)
+ elif isinstance(transformed_request, str) or isinstance(
+ transformed_request, bytes
+ ):
+ # Handle traditional file uploads
+ # Ensure transformed_request is a string for httpx compatibility
+ if isinstance(transformed_request, bytes):
+ transformed_request = transformed_request.decode("utf-8")
+
+ # Use the HTTP method specified by the provider config
+ http_method = provider_config.file_upload_http_method.upper()
+ if http_method == "PUT":
+ upload_response = await async_httpx_client.put(
+ url=api_base,
+ headers=headers,
+ data=transformed_request,
+ timeout=timeout,
+ )
+ else: # Default to POST
+ upload_response = await async_httpx_client.post(
+ url=api_base,
+ headers=headers,
+ data=transformed_request,
+ timeout=timeout,
+ )
else:
try:
# Step 1: Initial request to get upload URL
@@ -2326,6 +2413,536 @@ class BaseLLMHTTPHandler:
litellm_params=litellm_params,
)
+ def create_batch(
+ self,
+ create_batch_data: "CreateBatchRequest",
+ litellm_params: dict,
+ provider_config: "BaseBatchesConfig",
+ headers: dict,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ logging_obj: "LiteLLMLoggingObj",
+ _is_async: bool = False,
+ client: Optional[Union["HTTPHandler", "AsyncHTTPHandler"]] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ model: Optional[str] = None,
+ ) -> Union["LiteLLMBatch", Coroutine[Any, Any, "LiteLLMBatch"]]:
+ """
+ Creates a batch using provider-specific batch creation process
+ """
+ # get config from model, custom llm provider
+ if model is None:
+ raise ValueError("model is required for create_batch")
+
+ headers = provider_config.validate_environment(
+ api_key=api_key,
+ headers=headers,
+ model=model,
+ messages=[],
+ optional_params={},
+ litellm_params=litellm_params,
+ )
+
+ api_base = provider_config.get_complete_batch_url(
+ api_base=api_base,
+ api_key=api_key,
+ model=model,
+ optional_params={},
+ litellm_params=litellm_params,
+ data=create_batch_data,
+ )
+ if api_base is None:
+ raise ValueError("api_base is required for create_batch")
+
+ # Get the transformed request data
+ transformed_request = provider_config.transform_create_batch_request(
+ model=model,
+ create_batch_data=create_batch_data,
+ litellm_params=litellm_params,
+ optional_params={},
+ )
+
+ if _is_async:
+ return self.async_create_batch(
+ transformed_request=transformed_request,
+ litellm_params=litellm_params,
+ provider_config=provider_config,
+ headers=headers,
+ api_base=api_base,
+ logging_obj=logging_obj,
+ client=client,
+ timeout=timeout,
+ create_batch_data=create_batch_data,
+ )
+
+ if client is None or not isinstance(client, HTTPHandler):
+ sync_httpx_client = _get_httpx_client()
+ else:
+ sync_httpx_client = client
+
+ try:
+ if (
+ isinstance(transformed_request, dict)
+ and "method" in transformed_request
+ ):
+ # Handle pre-signed requests (e.g., from Bedrock with AWS auth)
+ batch_response = getattr(
+ sync_httpx_client, transformed_request["method"].lower()
+ )(
+ url=transformed_request["url"],
+ headers=transformed_request["headers"],
+ data=transformed_request["data"],
+ timeout=timeout,
+ )
+ elif isinstance(transformed_request, dict):
+ # For other providers that use JSON requests
+ batch_response = sync_httpx_client.post(
+ url=api_base,
+ headers={**headers, "Content-Type": "application/json"},
+ json=transformed_request,
+ timeout=timeout,
+ )
+ else:
+ # Handle other request types if needed
+ batch_response = sync_httpx_client.post(
+ url=api_base,
+ headers=headers,
+ data=transformed_request,
+ timeout=timeout,
+ )
+ except Exception as e:
+ verbose_logger.exception(f"Error creating batch: {e}")
+ raise self._handle_error(
+ e=e,
+ provider_config=provider_config,
+ )
+
+ # Store original request for response transformation
+ litellm_params_with_request = {
+ **litellm_params,
+ "original_batch_request": create_batch_data,
+ }
+
+ return provider_config.transform_create_batch_response(
+ model=model,
+ raw_response=batch_response,
+ logging_obj=logging_obj,
+ litellm_params=litellm_params_with_request,
+ )
+
+ def retrieve_batch(
+ self,
+ batch_id: str,
+ litellm_params: dict,
+ provider_config: "BaseBatchesConfig",
+ headers: dict,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ logging_obj: "LiteLLMLoggingObj",
+ _is_async: bool = False,
+ client: Optional[Union["HTTPHandler", "AsyncHTTPHandler"]] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ model: Optional[str] = None,
+ ) -> Union["LiteLLMBatch", Coroutine[Any, Any, "LiteLLMBatch"]]:
+ """
+ Retrieve a batch using provider-specific configuration.
+ """
+ # Transform the request using provider config
+ transformed_request = provider_config.transform_retrieve_batch_request(
+ batch_id=batch_id,
+ optional_params=litellm_params,
+ litellm_params=litellm_params,
+ )
+
+ if _is_async:
+ return self.async_retrieve_batch(
+ transformed_request=transformed_request,
+ litellm_params=litellm_params,
+ provider_config=provider_config,
+ headers=headers,
+ api_base=api_base,
+ logging_obj=logging_obj,
+ client=client,
+ timeout=timeout,
+ batch_id=batch_id,
+ model=model,
+ )
+
+ if client is None or not isinstance(client, HTTPHandler):
+ sync_httpx_client = _get_httpx_client()
+ else:
+ sync_httpx_client = client
+
+ try:
+ if (
+ isinstance(transformed_request, dict)
+ and "method" in transformed_request
+ ):
+ # Handle pre-signed requests (e.g., from Bedrock with AWS auth)
+ method = transformed_request["method"].lower()
+ request_kwargs = {
+ "url": transformed_request["url"],
+ "headers": transformed_request["headers"],
+ }
+
+ # Only add data for non-GET requests
+ if method != "get" and transformed_request.get("data") is not None:
+ request_kwargs["data"] = transformed_request["data"]
+
+ batch_response = getattr(sync_httpx_client, method)(**request_kwargs)
+ elif isinstance(transformed_request, dict) and api_base:
+ # For other providers that use JSON requests
+ batch_response = sync_httpx_client.get(
+ url=api_base,
+ headers={**headers, "Content-Type": "application/json"},
+ params=transformed_request,
+ )
+ else:
+ # Handle other request types if needed
+ if not api_base:
+ raise ValueError("api_base is required for non-pre-signed requests")
+ batch_response = sync_httpx_client.get(
+ url=api_base,
+ headers=headers,
+ )
+ except Exception as e:
+ verbose_logger.exception(f"Error retrieving batch: {e}")
+ raise self._handle_error(
+ e=e,
+ provider_config=provider_config,
+ )
+
+ return provider_config.transform_retrieve_batch_response(
+ model=model,
+ raw_response=batch_response,
+ logging_obj=logging_obj,
+ litellm_params=litellm_params,
+ )
+
+ async def async_create_batch(
+ self,
+ transformed_request: Union[bytes, str, dict],
+ litellm_params: dict,
+ provider_config: "BaseBatchesConfig",
+ headers: dict,
+ api_base: str,
+ logging_obj: "LiteLLMLoggingObj",
+ client: Optional[Union["HTTPHandler", "AsyncHTTPHandler"]] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ create_batch_data: Optional["CreateBatchRequest"] = None,
+ model: Optional[str] = None,
+ ):
+ """
+ Async version of create_batch
+ """
+ if client is None or not isinstance(client, AsyncHTTPHandler):
+ async_httpx_client = get_async_httpx_client(
+ llm_provider=provider_config.custom_llm_provider
+ )
+ else:
+ async_httpx_client = client
+
+ #########################################################
+ # Debug Logging
+ #########################################################
+ logging_obj.pre_call(
+ input="",
+ api_key="",
+ additional_args={
+ "complete_input_dict": transformed_request,
+ "api_base": api_base,
+ "headers": headers,
+ },
+ )
+
+ try:
+ if (
+ isinstance(transformed_request, dict)
+ and "method" in transformed_request
+ ):
+ # Handle pre-signed requests (e.g., from Bedrock with AWS auth)
+ batch_response = await getattr(
+ async_httpx_client, transformed_request["method"].lower()
+ )(
+ url=transformed_request["url"],
+ headers=transformed_request["headers"],
+ data=transformed_request["data"],
+ timeout=timeout,
+ )
+ elif isinstance(transformed_request, dict):
+ # For other providers that use JSON requests
+ batch_response = await async_httpx_client.post(
+ url=api_base,
+ headers={**headers, "Content-Type": "application/json"},
+ json=transformed_request,
+ timeout=timeout,
+ )
+ else:
+ # Handle other request types if needed
+ batch_response = await async_httpx_client.post(
+ url=api_base,
+ headers=headers,
+ data=transformed_request,
+ timeout=timeout,
+ )
+ except Exception as e:
+ verbose_logger.exception(f"Error creating batch: {e}")
+ raise self._handle_error(
+ e=e,
+ provider_config=provider_config,
+ )
+
+ # Store original request for response transformation (for async version)
+ litellm_params_with_request = {
+ **litellm_params,
+ "original_batch_request": create_batch_data or {},
+ }
+
+ return provider_config.transform_create_batch_response(
+ model=model,
+ raw_response=batch_response,
+ logging_obj=logging_obj,
+ litellm_params=litellm_params_with_request,
+ )
+
+ async def async_retrieve_batch(
+ self,
+ transformed_request: Union[bytes, str, dict],
+ litellm_params: dict,
+ provider_config: "BaseBatchesConfig",
+ headers: dict,
+ api_base: Optional[str],
+ logging_obj: "LiteLLMLoggingObj",
+ client: Optional[Union["HTTPHandler", "AsyncHTTPHandler"]] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ batch_id: Optional[str] = None,
+ model: Optional[str] = None,
+ ):
+ """
+ Async version of retrieve_batch
+ """
+ if client is None or not isinstance(client, AsyncHTTPHandler):
+ async_httpx_client = get_async_httpx_client(
+ llm_provider=provider_config.custom_llm_provider
+ )
+ else:
+ async_httpx_client = client
+
+ #########################################################
+ # Debug Logging
+ #########################################################
+ logging_obj.pre_call(
+ input="",
+ api_key="",
+ additional_args={
+ "complete_input_dict": transformed_request,
+ "api_base": api_base,
+ "headers": headers,
+ "batch_id": batch_id,
+ },
+ )
+
+ try:
+ if (
+ isinstance(transformed_request, dict)
+ and "method" in transformed_request
+ ):
+ # Handle pre-signed requests (e.g., from Bedrock with AWS auth)
+ method = transformed_request["method"].lower()
+ request_kwargs = {
+ "url": transformed_request["url"],
+ "headers": transformed_request["headers"],
+ }
+
+ # Only add data for non-GET requests
+ if method != "get" and transformed_request.get("data") is not None:
+ request_kwargs["data"] = transformed_request["data"]
+
+ batch_response = await getattr(async_httpx_client, method)(
+ **request_kwargs
+ )
+ elif isinstance(transformed_request, dict) and api_base:
+ # For other providers that use JSON requests
+ batch_response = await async_httpx_client.get(
+ url=api_base,
+ headers={**headers, "Content-Type": "application/json"},
+ params=transformed_request,
+ )
+ else:
+ # Handle other request types if needed
+ if not api_base:
+ raise ValueError("api_base is required for non-pre-signed requests")
+ batch_response = await async_httpx_client.get(
+ url=api_base,
+ headers=headers,
+ )
+ except Exception as e:
+ verbose_logger.exception(f"Error retrieving batch: {e}")
+ raise self._handle_error(
+ e=e,
+ provider_config=provider_config,
+ )
+
+ return provider_config.transform_retrieve_batch_response(
+ model=model,
+ raw_response=batch_response,
+ logging_obj=logging_obj,
+ litellm_params=litellm_params,
+ )
+
+ def cancel_response_api_handler(
+ self,
+ response_id: str,
+ responses_api_provider_config: BaseResponsesAPIConfig,
+ litellm_params: GenericLiteLLMParams,
+ logging_obj: LiteLLMLoggingObj,
+ custom_llm_provider: Optional[str],
+ extra_headers: Optional[Dict[str, Any]] = None,
+ extra_body: Optional[Dict[str, Any]] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
+ _is_async: bool = False,
+ ) -> Union[ResponsesAPIResponse, Coroutine[Any, Any, ResponsesAPIResponse]]:
+ """
+ Async version of the responses API handler.
+ Uses async HTTP client to make requests.
+ """
+ if _is_async:
+ return self.async_cancel_response_api_handler(
+ response_id=response_id,
+ responses_api_provider_config=responses_api_provider_config,
+ litellm_params=litellm_params,
+ logging_obj=logging_obj,
+ custom_llm_provider=custom_llm_provider,
+ extra_headers=extra_headers,
+ extra_body=extra_body,
+ timeout=timeout,
+ client=client,
+ )
+ if client is None or not isinstance(client, HTTPHandler):
+ sync_httpx_client = _get_httpx_client(
+ params={"ssl_verify": litellm_params.get("ssl_verify", None)}
+ )
+ else:
+ sync_httpx_client = client
+
+ headers = responses_api_provider_config.validate_environment(
+ headers=extra_headers or {}, model="None", litellm_params=litellm_params
+ )
+
+ if extra_headers:
+ headers.update(extra_headers)
+
+ api_base = responses_api_provider_config.get_complete_url(
+ api_base=litellm_params.api_base,
+ litellm_params=dict(litellm_params),
+ )
+
+ url, data = responses_api_provider_config.transform_cancel_response_api_request(
+ response_id=response_id,
+ api_base=api_base,
+ litellm_params=litellm_params,
+ headers=headers,
+ )
+
+ ## LOGGING
+ logging_obj.pre_call(
+ input=response_id,
+ api_key="",
+ additional_args={
+ "complete_input_dict": data,
+ "api_base": url,
+ "headers": headers,
+ },
+ )
+
+ try:
+ response = sync_httpx_client.post(
+ url=url, headers=headers, json=data, timeout=timeout
+ )
+
+ except Exception as e:
+ raise self._handle_error(
+ e=e,
+ provider_config=responses_api_provider_config,
+ )
+
+ return responses_api_provider_config.transform_cancel_response_api_response(
+ raw_response=response,
+ logging_obj=logging_obj,
+ )
+
+ async def async_cancel_response_api_handler(
+ self,
+ response_id: str,
+ responses_api_provider_config: BaseResponsesAPIConfig,
+ litellm_params: GenericLiteLLMParams,
+ logging_obj: LiteLLMLoggingObj,
+ custom_llm_provider: Optional[str],
+ extra_headers: Optional[Dict[str, Any]] = None,
+ extra_body: Optional[Dict[str, Any]] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
+ _is_async: bool = False,
+ ) -> ResponsesAPIResponse:
+ """
+ Async version of the cancel response API handler.
+ Uses async HTTP client to make requests.
+ """
+ if client is None or not isinstance(client, AsyncHTTPHandler):
+ async_httpx_client = get_async_httpx_client(
+ llm_provider=litellm.LlmProviders(custom_llm_provider),
+ params={"ssl_verify": litellm_params.get("ssl_verify", None)},
+ )
+ else:
+ async_httpx_client = client
+
+ headers = responses_api_provider_config.validate_environment(
+ headers=extra_headers or {}, model="None", litellm_params=litellm_params
+ )
+
+ if extra_headers:
+ headers.update(extra_headers)
+
+ api_base = responses_api_provider_config.get_complete_url(
+ api_base=litellm_params.api_base,
+ litellm_params=dict(litellm_params),
+ )
+
+ url, data = responses_api_provider_config.transform_cancel_response_api_request(
+ response_id=response_id,
+ api_base=api_base,
+ litellm_params=litellm_params,
+ headers=headers,
+ )
+
+ ## LOGGING
+ logging_obj.pre_call(
+ input=response_id,
+ api_key="",
+ additional_args={
+ "complete_input_dict": data,
+ "api_base": url,
+ "headers": headers,
+ },
+ )
+
+ try:
+ response = await async_httpx_client.post(
+ url=url, headers=headers, json=data, timeout=timeout
+ )
+
+ except Exception as e:
+ raise self._handle_error(
+ e=e,
+ provider_config=responses_api_provider_config,
+ )
+
+ return responses_api_provider_config.transform_cancel_response_api_response(
+ raw_response=response,
+ logging_obj=logging_obj,
+ )
+
def list_files(self):
"""
Lists all files
@@ -2377,6 +2994,7 @@ class BaseLLMHTTPHandler:
BaseVectorStoreConfig,
BaseGoogleGenAIGenerateContentConfig,
BaseAnthropicMessagesConfig,
+ BaseBatchesConfig,
"BasePassthroughConfig",
],
):
@@ -2677,6 +3295,7 @@ class BaseLLMHTTPHandler:
_is_async: bool = False,
fake_stream: bool = False,
litellm_metadata: Optional[Dict[str, Any]] = None,
+ api_key: Optional[str] = None,
) -> Union[
ImageResponse,
Coroutine[Any, Any, ImageResponse],
@@ -2701,6 +3320,7 @@ class BaseLLMHTTPHandler:
client=client if isinstance(client, AsyncHTTPHandler) else None,
fake_stream=fake_stream,
litellm_metadata=litellm_metadata,
+ api_key=api_key,
)
if client is None or not isinstance(client, HTTPHandler):
@@ -2711,8 +3331,9 @@ class BaseLLMHTTPHandler:
sync_httpx_client = client
headers = image_generation_provider_config.validate_environment(
- api_key=litellm_params.get("api_key", None),
- headers=image_generation_optional_request_params.get("extra_headers", {}) or {},
+ api_key=api_key,
+ headers=image_generation_optional_request_params.get("extra_headers", {})
+ or {},
model=model,
messages=[],
optional_params=image_generation_optional_request_params,
@@ -2763,15 +3384,17 @@ class BaseLLMHTTPHandler:
provider_config=image_generation_provider_config,
)
- model_response: ImageResponse = image_generation_provider_config.transform_image_generation_response(
- model=model,
- raw_response=response,
- model_response=litellm.ImageResponse(),
- logging_obj=logging_obj,
- request_data=data,
- optional_params=image_generation_optional_request_params,
- litellm_params=dict(litellm_params),
- encoding=None,
+ model_response: ImageResponse = (
+ image_generation_provider_config.transform_image_generation_response(
+ model=model,
+ raw_response=response,
+ model_response=litellm.ImageResponse(),
+ logging_obj=logging_obj,
+ request_data=data,
+ optional_params=image_generation_optional_request_params,
+ litellm_params=dict(litellm_params),
+ encoding=None,
+ )
)
return model_response
@@ -2791,6 +3414,7 @@ class BaseLLMHTTPHandler:
client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
fake_stream: bool = False,
litellm_metadata: Optional[Dict[str, Any]] = None,
+ api_key: Optional[str] = None,
) -> ImageResponse:
"""
Async version of the image generation handler.
@@ -2804,10 +3428,10 @@ class BaseLLMHTTPHandler:
else:
async_httpx_client = client
-
headers = image_generation_provider_config.validate_environment(
- api_key=litellm_params.get("api_key", None),
- headers=image_generation_optional_request_params.get("extra_headers", {}) or {},
+ api_key=api_key,
+ headers=image_generation_optional_request_params.get("extra_headers", {})
+ or {},
model=model,
messages=[],
optional_params=image_generation_optional_request_params,
@@ -2858,17 +3482,19 @@ class BaseLLMHTTPHandler:
provider_config=image_generation_provider_config,
)
- model_response: ImageResponse = image_generation_provider_config.transform_image_generation_response(
- model=model,
- raw_response=response,
- model_response=litellm.ImageResponse(),
- logging_obj=logging_obj,
- request_data=data,
- optional_params=image_generation_optional_request_params,
- litellm_params=dict(litellm_params),
- encoding=None,
+ model_response: ImageResponse = (
+ image_generation_provider_config.transform_image_generation_response(
+ model=model,
+ raw_response=response,
+ model_response=litellm.ImageResponse(),
+ logging_obj=logging_obj,
+ request_data=data,
+ optional_params=image_generation_optional_request_params,
+ litellm_params=dict(litellm_params),
+ encoding=None,
+ )
)
-
+
return model_response
###### VECTOR STORE HANDLER ######
@@ -2907,15 +3533,16 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
)
- url, request_body = (
- vector_store_provider_config.transform_search_vector_store_request(
- vector_store_id=vector_store_id,
- query=query,
- vector_store_search_optional_params=vector_store_search_optional_params,
- api_base=api_base,
- litellm_logging_obj=logging_obj,
- litellm_params=dict(litellm_params),
- )
+ (
+ url,
+ request_body,
+ ) = vector_store_provider_config.transform_search_vector_store_request(
+ vector_store_id=vector_store_id,
+ query=query,
+ vector_store_search_optional_params=vector_store_search_optional_params,
+ api_base=api_base,
+ litellm_logging_obj=logging_obj,
+ litellm_params=dict(litellm_params),
)
all_optional_params: Dict[str, Any] = dict(litellm_params)
all_optional_params.update(vector_store_search_optional_params or {})
@@ -2936,7 +3563,9 @@ class BaseLLMHTTPHandler:
},
)
- request_data = json.dumps(request_body) if signed_json_body is None else signed_json_body
+ request_data = (
+ json.dumps(request_body) if signed_json_body is None else signed_json_body
+ )
try:
response = await async_httpx_client.post(
@@ -3004,15 +3633,16 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
)
- url, request_body = (
- vector_store_provider_config.transform_search_vector_store_request(
- vector_store_id=vector_store_id,
- query=query,
- vector_store_search_optional_params=vector_store_search_optional_params,
- api_base=api_base,
- litellm_logging_obj=logging_obj,
- litellm_params=dict(litellm_params),
- )
+ (
+ url,
+ request_body,
+ ) = vector_store_provider_config.transform_search_vector_store_request(
+ vector_store_id=vector_store_id,
+ query=query,
+ vector_store_search_optional_params=vector_store_search_optional_params,
+ api_base=api_base,
+ litellm_logging_obj=logging_obj,
+ litellm_params=dict(litellm_params),
)
all_optional_params: Dict[str, Any] = dict(litellm_params)
@@ -3035,7 +3665,9 @@ class BaseLLMHTTPHandler:
},
)
- request_data = json.dumps(request_body) if signed_json_body is None else signed_json_body
+ request_data = (
+ json.dumps(request_body) if signed_json_body is None else signed_json_body
+ )
try:
response = sync_httpx_client.post(
@@ -3084,11 +3716,12 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
)
- url, request_body = (
- vector_store_provider_config.transform_create_vector_store_request(
- vector_store_create_optional_params=vector_store_create_optional_params,
- api_base=api_base,
- )
+ (
+ url,
+ request_body,
+ ) = vector_store_provider_config.transform_create_vector_store_request(
+ vector_store_create_optional_params=vector_store_create_optional_params,
+ api_base=api_base,
)
logging_obj.pre_call(
@@ -3159,11 +3792,12 @@ class BaseLLMHTTPHandler:
litellm_params=dict(litellm_params),
)
- url, request_body = (
- vector_store_provider_config.transform_create_vector_store_request(
- vector_store_create_optional_params=vector_store_create_optional_params,
- api_base=api_base,
- )
+ (
+ url,
+ request_body,
+ ) = vector_store_provider_config.transform_create_vector_store_request(
+ vector_store_create_optional_params=vector_store_create_optional_params,
+ api_base=api_base,
)
logging_obj.pre_call(
@@ -3196,6 +3830,7 @@ class BaseLLMHTTPHandler:
contents: Any,
generate_content_provider_config: BaseGoogleGenAIGenerateContentConfig,
generate_content_config_dict: Dict,
+ tools: Any,
custom_llm_provider: str,
litellm_params: GenericLiteLLMParams,
logging_obj: LiteLLMLoggingObj,
@@ -3221,6 +3856,7 @@ class BaseLLMHTTPHandler:
contents=contents,
generate_content_provider_config=generate_content_provider_config,
generate_content_config_dict=generate_content_config_dict,
+ tools=tools,
custom_llm_provider=custom_llm_provider,
litellm_params=litellm_params,
logging_obj=logging_obj,
@@ -3240,13 +3876,14 @@ class BaseLLMHTTPHandler:
sync_httpx_client = client
# Get headers and URL from the provider config
- headers, api_base = (
- generate_content_provider_config.sync_get_auth_token_and_url(
- api_base=litellm_params.api_base,
- model=model,
- litellm_params=dict(litellm_params),
- stream=stream,
- )
+ (
+ headers,
+ api_base,
+ ) = generate_content_provider_config.sync_get_auth_token_and_url(
+ api_base=litellm_params.api_base,
+ model=model,
+ litellm_params=dict(litellm_params),
+ stream=stream,
)
if extra_headers:
@@ -3256,6 +3893,7 @@ class BaseLLMHTTPHandler:
data = generate_content_provider_config.transform_generate_content_request(
model=model,
contents=contents,
+ tools=tools,
generate_content_config_dict=generate_content_config_dict,
)
@@ -3317,6 +3955,7 @@ class BaseLLMHTTPHandler:
contents: Any,
generate_content_provider_config: BaseGoogleGenAIGenerateContentConfig,
generate_content_config_dict: Dict,
+ tools: Any,
custom_llm_provider: str,
litellm_params: GenericLiteLLMParams,
logging_obj: LiteLLMLoggingObj,
@@ -3344,13 +3983,14 @@ class BaseLLMHTTPHandler:
async_httpx_client = client
# Get headers and URL from the provider config
- headers, api_base = (
- await generate_content_provider_config.get_auth_token_and_url(
- model=model,
- litellm_params=dict(litellm_params),
- stream=stream,
- api_base=litellm_params.api_base,
- )
+ (
+ headers,
+ api_base,
+ ) = await generate_content_provider_config.get_auth_token_and_url(
+ model=model,
+ litellm_params=dict(litellm_params),
+ stream=stream,
+ api_base=litellm_params.api_base,
)
if extra_headers:
@@ -3360,6 +4000,7 @@ class BaseLLMHTTPHandler:
data = generate_content_provider_config.transform_generate_content_request(
model=model,
contents=contents,
+ tools=tools,
generate_content_config_dict=generate_content_config_dict,
)
diff --git a/litellm/llms/dashscope/cost_calculator.py b/litellm/llms/dashscope/cost_calculator.py
index 0f4490cb3df..107eb7f5adf 100644
--- a/litellm/llms/dashscope/cost_calculator.py
+++ b/litellm/llms/dashscope/cost_calculator.py
@@ -1,21 +1,155 @@
"""
-Cost calculator for DeepSeek Chat models.
+Cost calculator for Dashscope Chat models.
-Handles prompt caching scenario.
+Handles tiered pricing and prompt caching scenarios.
"""
-from typing import Tuple
+from dataclasses import dataclass
+from typing import List, Optional, Tuple
-from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token
-from litellm.types.utils import Usage
+from litellm.types.utils import ModelInfo, Usage
+from litellm.utils import get_model_info
+
+
+@dataclass
+class TokenBreakdown:
+ """Token breakdown for cost calculation."""
+ text_tokens: int
+ cached_tokens: int
+ completion_tokens: int
+ reasoning_tokens: int
+
+
+def _extract_token_breakdown(usage: Usage) -> TokenBreakdown:
+ """Extract token counts from usage, handling cached and reasoning tokens."""
+ cached_tokens = 0
+ if usage.prompt_tokens_details and hasattr(usage.prompt_tokens_details, "cached_tokens"):
+ cached_tokens = usage.prompt_tokens_details.cached_tokens or 0
+
+ text_tokens = usage.prompt_tokens - cached_tokens
+
+ reasoning_tokens = 0
+ if (hasattr(usage, "completion_tokens_details") and
+ usage.completion_tokens_details and
+ hasattr(usage.completion_tokens_details, "reasoning_tokens")):
+ reasoning_tokens = usage.completion_tokens_details.reasoning_tokens or 0
+
+ completion_tokens = (usage.completion_tokens or 0) - reasoning_tokens
+
+ return TokenBreakdown(text_tokens, cached_tokens, completion_tokens, reasoning_tokens)
+
+
+def _calculate_tiered_cost(
+ tokens: int,
+ tiered_pricing: List[dict],
+ cost_key: str,
+ fallback_cost_key: Optional[str] = None
+) -> float:
+ """Calculate cost using tiered pricing structure.
+
+ Finds the appropriate tier based on token count and applies that tier's rate to all tokens.
+ """
+ if not tiered_pricing or tokens <= 0:
+ return 0.0
+
+ # Find the appropriate tier for the token count
+ for tier in tiered_pricing:
+ tier_range = tier.get("range", [])
+ if len(tier_range) != 2:
+ continue
+
+ range_start, range_end = tier_range
+
+ # Check if tokens fall within this tier's range
+ if range_start <= tokens <= range_end:
+ cost_per_token = tier.get(cost_key) or tier.get(fallback_cost_key, 0)
+ return tokens * cost_per_token
+
+ # If no tier matches, use the last tier (highest tier)
+ if tiered_pricing:
+ last_tier = tiered_pricing[-1]
+ cost_per_token = last_tier.get(cost_key) or last_tier.get(fallback_cost_key, 0)
+ return tokens * cost_per_token
+
+ return 0.0
+
+
+def _calculate_flat_cost(tokens: int, cost_per_token: float) -> float:
+ """Calculate cost using flat pricing."""
+ return tokens * cost_per_token
+
+
+def _calculate_prompt_cost(breakdown: TokenBreakdown, model_info: ModelInfo, tiered_pricing: Optional[List[dict]]) -> float:
+ """Calculate total prompt cost including cached tokens."""
+ if tiered_pricing:
+ text_cost = _calculate_tiered_cost(
+ tokens=breakdown.text_tokens,
+ tiered_pricing=tiered_pricing,
+ cost_key="input_cost_per_token"
+ )
+ cache_cost = _calculate_tiered_cost(
+ tokens=breakdown.cached_tokens,
+ tiered_pricing=tiered_pricing,
+ cost_key="cache_read_input_token_cost"
+ )
+ return text_cost + cache_cost
+
+ input_cost = model_info.get("input_cost_per_token", 0.0)
+ cache_cost = model_info.get("cache_read_input_token_cost", input_cost) or input_cost
+
+ return (_calculate_flat_cost(tokens=breakdown.text_tokens, cost_per_token=input_cost) +
+ _calculate_flat_cost(tokens=breakdown.cached_tokens, cost_per_token=cache_cost))
+
+
+def _calculate_completion_cost(breakdown: TokenBreakdown, model_info: ModelInfo, tiered_pricing: Optional[List[dict]]) -> float:
+ """Calculate total completion cost including reasoning tokens."""
+ if tiered_pricing:
+ completion_cost = _calculate_tiered_cost(
+ tokens=breakdown.completion_tokens,
+ tiered_pricing=tiered_pricing,
+ cost_key="output_cost_per_token"
+ )
+ reasoning_cost = _calculate_tiered_cost(
+ tokens=breakdown.reasoning_tokens,
+ tiered_pricing=tiered_pricing,
+ cost_key="output_cost_per_reasoning_token",
+ fallback_cost_key="output_cost_per_token"
+ )
+ return completion_cost + reasoning_cost
+
+ output_cost = model_info.get("output_cost_per_token", 0.0)
+ reasoning_cost = model_info.get("output_cost_per_reasoning_token", output_cost) or output_cost
+
+ return (_calculate_flat_cost(tokens=breakdown.completion_tokens, cost_per_token=output_cost) +
+ _calculate_flat_cost(tokens=breakdown.reasoning_tokens, cost_per_token=reasoning_cost))
def cost_per_token(model: str, usage: Usage) -> Tuple[float, float]:
"""
- Calculates the cost per token for a given model, prompt tokens, and completion tokens.
-
- Follows the same logic as Anthropic's cost per token calculation.
+ Calculate cost per token for Dashscope models.
+
+ Supports both tiered and flat pricing with cached and reasoning tokens.
+
+ Args:
+ model: Model name without provider prefix
+ usage: LiteLLM Usage block
+
+ Returns:
+ Tuple[float, float] - (prompt_cost_in_usd, completion_cost_in_usd)
"""
- return generic_cost_per_token(
- model=model, usage=usage, custom_llm_provider="deepseek"
+ model_info = get_model_info(model=model, custom_llm_provider="dashscope")
+ breakdown = _extract_token_breakdown(usage)
+ tiered_pricing = model_info.get("tiered_pricing") if isinstance(model_info.get("tiered_pricing"), list) else None
+
+ prompt_cost = _calculate_prompt_cost(
+ breakdown=breakdown,
+ model_info=model_info,
+ tiered_pricing=tiered_pricing
)
+ completion_cost = _calculate_completion_cost(
+ breakdown=breakdown,
+ model_info=model_info,
+ tiered_pricing=tiered_pricing
+ )
+
+ return prompt_cost, completion_cost
diff --git a/litellm/llms/databricks/chat/transformation.py b/litellm/llms/databricks/chat/transformation.py
index e7d7920769f..a1370074238 100644
--- a/litellm/llms/databricks/chat/transformation.py
+++ b/litellm/llms/databricks/chat/transformation.py
@@ -26,7 +26,6 @@ from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response impo
_should_convert_tool_call_to_json_mode,
)
from litellm.litellm_core_utils.prompt_templates.common_utils import (
- handle_messages_with_content_list_to_str_conversion,
strip_name_from_messages,
)
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
@@ -170,12 +169,20 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
if tool is None:
return None
+ # Build DatabricksFunction explicitly to avoid parameter conflicts
+ function_params: DatabricksFunction = {
+ "name": tool["name"],
+ "parameters": cast(dict, tool.get("input_schema") or {})
+ }
+
+ # Only add description if it exists
+ description = tool.get("description")
+ if description is not None:
+ function_params["description"] = cast(Union[dict, str], description)
+
return DatabricksTool(
type="function",
- function=DatabricksFunction(
- name=tool["name"],
- parameters=cast(dict, tool.get("input_schema") or {}),
- ),
+ function=function_params,
)
def _map_openai_to_dbrx_tool(self, model: str, tools: List) -> List[DatabricksTool]:
@@ -301,7 +308,6 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
) -> Union[List[AllMessageValues], Coroutine[Any, Any, List[AllMessageValues]]]:
"""
Databricks does not support:
- - content in list format.
- 'name' in user message.
"""
new_messages = []
@@ -311,7 +317,6 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
else:
_message = message
new_messages.append(_message)
- new_messages = handle_messages_with_content_list_to_str_conversion(new_messages)
new_messages = strip_name_from_messages(new_messages)
if is_async:
@@ -334,8 +339,9 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
elif isinstance(content, list):
content_str = ""
for item in content:
- if item["type"] == "text":
- content_str += item["text"]
+ if item.get("type") == "text":
+ text_value = item.get("text", "")
+ content_str += str(text_value) if text_value is not None else ""
return content_str
else:
raise Exception(f"Unsupported content type: {type(content)}")
@@ -364,21 +370,42 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
reasoning_content: Optional[str] = None
if isinstance(content, list):
for item in content:
- if item["type"] == "reasoning":
- for sum in item["summary"]:
- if reasoning_content is None:
- reasoning_content = ""
- reasoning_content += sum["text"]
- thinking_block = ChatCompletionThinkingBlock(
- type="thinking",
- thinking=sum["text"],
- signature=sum["signature"],
- )
- if thinking_blocks is None:
- thinking_blocks = []
- thinking_blocks.append(thinking_block)
+ if item.get("type") == "reasoning":
+ summary_list = item.get("summary", [])
+ if isinstance(summary_list, list):
+ for sum in summary_list:
+ if reasoning_content is None:
+ reasoning_content = ""
+ reasoning_content += sum["text"]
+ thinking_block = ChatCompletionThinkingBlock(
+ type="thinking",
+ thinking=sum.get("text", ""),
+ signature=sum.get("signature", ""),
+ )
+ if thinking_blocks is None:
+ thinking_blocks = []
+ thinking_blocks.append(thinking_block)
return reasoning_content, thinking_blocks
+ @staticmethod
+ def extract_citations(
+ content: Optional[AllDatabricksContentValues],
+ ) -> Optional[List[Any]]:
+ if content is None:
+ return None
+ citations = []
+ if isinstance(content, list):
+ for item in content:
+ text = item.get("text", None)
+ if citations_item := item.get("citations"):
+ citations.append(
+ [
+ {**citation, "supported_text": text}
+ for citation in citations_item
+ ]
+ )
+ return citations or None
+
def _transform_dbrx_choices(
self, choices: List[DatabricksChoice], json_mode: Optional[bool] = None
) -> List[Choices]:
@@ -427,12 +454,19 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
choice["message"].get("content")
)
+ citations = DatabricksConfig.extract_citations(
+ choice["message"].get("content")
+ )
+
translated_message = Message(
role="assistant",
content=content_str,
reasoning_content=reasoning_content,
thinking_blocks=thinking_blocks,
tool_calls=choice["message"].get("tool_calls"),
+ provider_specific_fields={"citations": citations}
+ if citations is not None
+ else None,
)
if finish_reason is None:
@@ -561,6 +595,17 @@ class DatabricksChatResponseIterator(BaseModelResponseIterator):
for _tc in tool_calls:
if _tc.get("function", {}).get("arguments") == "{}":
_tc["function"]["arguments"] = "" # avoid invalid json
+ if isinstance(choice["delta"]["content"], list) and (
+ content := choice["delta"]["content"]
+ ):
+ if citations := content[0].get("citations"):
+ # TODO: Databricks delta does not include supported text or chunk type.
+ # Add either here once Databricks supports it to enable citation linkage.
+ choice["delta"].setdefault("provider_specific_fields", {})[
+ "citation"
+ ] = citations[
+ 0
+ ] # Databricks Content item always has citation as a list of list
# extract the content str
content_str = DatabricksConfig.extract_content_str(
choice["delta"].get("content")
diff --git a/litellm/llms/datarobot/chat/transformation.py b/litellm/llms/datarobot/chat/transformation.py
index e334c94e517..23ce63c25b2 100644
--- a/litellm/llms/datarobot/chat/transformation.py
+++ b/litellm/llms/datarobot/chat/transformation.py
@@ -6,8 +6,11 @@ Calls done in OpenAI/openai.py as DataRobot is openai-compatible.
from typing import Optional, Tuple
from litellm.secret_managers.main import get_secret_str
+from urllib.parse import urlparse, urlunparse
from ...openai_like.chat.transformation import OpenAILikeChatConfig
+LLMGW_PATH = "/genai/llmgw/chat/completions"
+
class DataRobotConfig(OpenAILikeChatConfig):
@staticmethod
@@ -32,22 +35,28 @@ class DataRobotConfig(OpenAILikeChatConfig):
if api_base is None:
api_base = "https://app.datarobot.com"
- # If the api_base is a deployment URL, we do not append the chat completions path
- if "api/v2/deployments" not in api_base:
- # If the api_base is not a deployment URL, we need to append the chat completions path
- if "api/v2/genai/llmgw/chat/completions" not in api_base:
- api_base += "/api/v2/genai/llmgw/chat/completions"
+ parsed = urlparse(api_base)
+ path = parsed.path
+
+ if not path or path == "/": # Add full path to LLMGW
+ path += f"/api/v2/{LLMGW_PATH}"
+ elif "api/v2/deployments" in path: # Dedicated deployment, leave it
+ pass
+ elif (
+ "api/v2" in path and LLMGW_PATH not in path
+ ): # Standard ENDPOINT path, add LLMGW
+ path += LLMGW_PATH
# Ensure the url ends with a trailing slash
- if not api_base.endswith("/"):
- api_base += "/"
+ if not path.endswith("/"):
+ path += "/"
+ path = path.replace("//", "/")
+ updated_parsed = parsed._replace(path=path)
- return api_base # type: ignore
+ return urlunparse(updated_parsed)
def _get_openai_compatible_provider_info(
- self,
- api_base: Optional[str],
- api_key: Optional[str]
+ self, api_base: Optional[str], api_key: Optional[str]
) -> Tuple[Optional[str], Optional[str]]:
"""Attempts to ensure that the API base and key are set, preferring user-provided values,
before falling back to secret manager values (``DATAROBOT_ENDPOINT`` and ``DATAROBOT_API_TOKEN``
diff --git a/litellm/llms/deepinfra/chat/transformation.py b/litellm/llms/deepinfra/chat/transformation.py
index 0d446d39b92..09cdabcdd82 100644
--- a/litellm/llms/deepinfra/chat/transformation.py
+++ b/litellm/llms/deepinfra/chat/transformation.py
@@ -12,6 +12,9 @@ class DeepInfraConfig(OpenAIGPTConfig):
The class `DeepInfra` provides configuration for the DeepInfra's Chat Completions API interface. Below are the parameters:
"""
+ @property
+ def custom_llm_provider(self) -> Optional[str]:
+ return "deepinfra"
frequency_penalty: Optional[int] = None
function_call: Optional[Union[str, dict]] = None
@@ -53,7 +56,7 @@ class DeepInfraConfig(OpenAIGPTConfig):
return super().get_config()
def get_supported_openai_params(self, model: str):
- return [
+ supported_openai_params = [
"stream",
"frequency_penalty",
"function_call",
@@ -68,9 +71,16 @@ class DeepInfraConfig(OpenAIGPTConfig):
"top_p",
"response_format",
"tools",
- "tool_choice",
+ "tool_choice"
]
+ if litellm.supports_reasoning(
+ model=model,
+ custom_llm_provider=self.custom_llm_provider,
+ ):
+ supported_openai_params.append("reasoning_effort")
+ return supported_openai_params
+
def map_openai_params(
self,
non_default_params: dict,
diff --git a/litellm/llms/deepinfra/rerank/transformation.py b/litellm/llms/deepinfra/rerank/transformation.py
new file mode 100644
index 00000000000..69c7dabebd8
--- /dev/null
+++ b/litellm/llms/deepinfra/rerank/transformation.py
@@ -0,0 +1,239 @@
+"""
+Translate between Cohere's `/rerank` format and Deepinfra's `/rerank` format.
+"""
+
+from typing import Any, Dict, List, Optional, Union
+
+import httpx
+
+from litellm._uuid import uuid
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.base_llm.rerank.transformation import (
+ BaseLLMException,
+ BaseRerankConfig,
+)
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.rerank import (
+ OptionalRerankParams,
+ RerankBilledUnits,
+ RerankResponse,
+ RerankResponseMeta,
+ RerankResponseResult,
+ RerankTokens,
+)
+
+
+class DeepinfraRerankConfig(BaseRerankConfig):
+ """
+ Deepinfra Rerank - Follows the same Spec as Cohere Rerank
+ """
+
+ def get_complete_url(self, api_base: Optional[str], model: str) -> str:
+ """
+ Constructs the complete DeepInfra inference endpoint URL for rerank.
+
+ Args:
+ api_base (Optional[str]): The base URL for the DeepInfra API.
+ model (str): The model identifier.
+
+ Returns:
+ str: The complete URL for the DeepInfra rerank inference endpoint.
+
+ Raises:
+ ValueError: If api_base is None.
+ """
+ if not api_base:
+ raise ValueError(
+ "Deepinfra API Base is required. api_base=None. Set in call or via `DEEPINFRA_API_BASE` env var."
+ )
+
+ # Remove 'openai' from the base if present
+ api_base_clean = (
+ api_base.replace("openai", "") if "openai" in api_base else api_base
+ )
+
+ # Remove any trailing slashes for consistency, then add one
+ api_base_clean = api_base_clean.rstrip("/") + "/"
+
+ # Compose the full endpoint
+ return f"{api_base_clean}inference/{model}"
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ api_key: Optional[str] = None,
+ ) -> dict:
+ if api_key is None:
+ api_key = get_secret_str("DEEPINFRA_API_KEY")
+
+ if api_key is None:
+ raise ValueError(
+ "Deepinfra API key is required. Please set 'DEEPINFRA_API_KEY' environment variable"
+ )
+
+ default_headers = {
+ "Authorization": f"Bearer {api_key}",
+ "accept": "application/json",
+ "content-type": "application/json",
+ }
+
+ # If 'Authorization' is provided in headers, it overrides the default.
+ if "Authorization" in headers:
+ default_headers["Authorization"] = headers["Authorization"]
+
+ # Merge other headers, overriding any default ones except Authorization
+ return {**default_headers, **headers}
+
+ def map_cohere_rerank_params(
+ self,
+ non_default_params: dict,
+ model: str,
+ drop_params: bool,
+ query: str,
+ documents: List[Union[str, Dict[str, Any]]],
+ custom_llm_provider: Optional[str] = None,
+ top_n: Optional[int] = None,
+ rank_fields: Optional[List[str]] = None,
+ return_documents: Optional[bool] = True,
+ max_chunks_per_doc: Optional[int] = None,
+ max_tokens_per_doc: Optional[int] = None,
+ ) -> Dict:
+ # Start with the basic parameters
+ optional_rerank_params = {}
+ if query:
+ optional_rerank_params["queries"] = [query] * len(
+ documents
+ ) # Deepinfra rerank requires queries to be of same length as documents
+
+ if non_default_params is not None:
+ for k, v in non_default_params.items():
+ if k == "queries" and v is not None:
+ # This should override the query parameter if it is provided
+ optional_rerank_params["queries"] = v
+ elif k == "documents" and v is not None:
+ optional_rerank_params["documents"] = v
+ elif k == "service_tier" and v is not None:
+ optional_rerank_params["service_tier"] = v
+ elif k == "instruction" and v is not None:
+ optional_rerank_params["instruction"] = v
+ elif k == "webhook" and v is not None:
+ optional_rerank_params["webhook"] = v
+ return OptionalRerankParams(**optional_rerank_params) # type: ignore
+
+ def transform_rerank_request(
+ self,
+ model: str,
+ optional_rerank_params: Dict,
+ headers: dict,
+ ) -> dict:
+ # Convert OptionalRerankParams to dict as expected by parent class
+ if optional_rerank_params is None:
+ return {}
+ return dict(optional_rerank_params)
+
+ def transform_rerank_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: RerankResponse,
+ logging_obj: LiteLLMLoggingObj,
+ api_key: Optional[str] = None,
+ request_data: dict = {},
+ optional_params: dict = {},
+ litellm_params: dict = {},
+ ) -> RerankResponse:
+ try:
+ response_json = raw_response.json()
+ logging_obj.post_call(original_response=raw_response.text)
+
+ # Extract the scores from the response
+ scores = response_json.get("scores", [])
+ input_tokens = response_json.get("input_tokens", 0)
+ request_id = response_json.get("request_id")
+
+ # Create inference status information
+ inference_status = response_json.get("inference_status", {})
+ status = inference_status.get("status", "unknown")
+ runtime_ms = inference_status.get("runtime_ms", 0)
+ cost = inference_status.get("cost", 0.0)
+ tokens_generated = inference_status.get("tokens_generated", 0)
+ tokens_input = inference_status.get("tokens_input", 0)
+
+ # Create RerankResponse
+ results = []
+ for i, score in enumerate(scores):
+ results.append(
+ RerankResponseResult(index=i, relevance_score=float(score))
+ )
+
+ # Create metadata for the response
+ tokens = RerankTokens(
+ input_tokens=input_tokens,
+ output_tokens=0, # DeepInfra doesn't provide output tokens for rerank
+ )
+ billed_units = RerankBilledUnits(total_tokens=input_tokens)
+ meta = RerankResponseMeta(tokens=tokens, billed_units=billed_units)
+
+ rerank_response = RerankResponse(
+ id=request_id or str(uuid.uuid4()), results=results, meta=meta
+ )
+
+ # Store additional information in hidden params
+ rerank_response._hidden_params = {
+ "status": status,
+ "runtime_ms": runtime_ms,
+ "cost": cost,
+ "tokens_generated": tokens_generated,
+ "tokens_input": tokens_input,
+ "model": model,
+ }
+
+ return rerank_response
+
+ except Exception:
+ # If there's an error parsing the response, fall back to the parent implementation
+ rerank_response = super().transform_rerank_response(
+ model=model,
+ raw_response=raw_response,
+ model_response=model_response,
+ logging_obj=logging_obj,
+ api_key=api_key,
+ request_data=request_data,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ )
+
+ rerank_response._hidden_params["model"] = model
+ return rerank_response
+
+ def get_supported_cohere_rerank_params(self, model: str) -> list:
+ return ["query", "documents"]
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ # Deepinfra errors may come as JSON: {"detail": {"error": "..."}}
+ import json
+
+ # Try to extract a more specific error message if possible
+ try:
+ error_data = error_message
+ if isinstance(error_message, str):
+ error_data = json.loads(error_message)
+ if isinstance(error_data, dict):
+ # Check for {"detail": {"error": "..."}}
+ detail = error_data.get("detail")
+ if isinstance(detail, dict) and "error" in detail:
+ error_message = detail["error"]
+ elif isinstance(detail, str):
+ error_message = detail
+ except Exception:
+ # If parsing fails, just use the original error_message
+ pass
+
+ raise BaseLLMException(
+ status_code=status_code,
+ message=error_message,
+ headers=headers,
+ )
diff --git a/litellm/llms/fireworks_ai/chat/transformation.py b/litellm/llms/fireworks_ai/chat/transformation.py
index 31d749032b4..524b1c97145 100644
--- a/litellm/llms/fireworks_ai/chat/transformation.py
+++ b/litellm/llms/fireworks_ai/chat/transformation.py
@@ -1,5 +1,5 @@
import json
-import uuid
+from litellm._uuid import uuid
from typing import Any, List, Literal, Optional, Tuple, Union, cast
import httpx
diff --git a/litellm/llms/gemini/chat/transformation.py b/litellm/llms/gemini/chat/transformation.py
index 37217ebfaab..e889126883c 100644
--- a/litellm/llms/gemini/chat/transformation.py
+++ b/litellm/llms/gemini/chat/transformation.py
@@ -1,10 +1,13 @@
-from typing import List, Optional
+from typing import List, Optional, cast
from litellm.litellm_core_utils.prompt_templates.factory import (
convert_generic_image_chunk_to_openai_image_obj,
convert_to_anthropic_image_obj,
)
-from litellm.types.llms.openai import AllMessageValues
+from litellm.litellm_core_utils.prompt_templates.image_handling import (
+ convert_url_to_base64,
+)
+from litellm.types.llms.openai import AllMessageValues, ChatCompletionFileObject
from litellm.types.llms.vertex_ai import ContentType, PartType
from litellm.utils import supports_reasoning
@@ -99,7 +102,8 @@ class GoogleAIStudioGeminiConfig(VertexGeminiConfig):
self, messages: List[AllMessageValues]
) -> List[ContentType]:
"""
- Google AI Studio Gemini does not support image urls in messages.
+ Google AI Studio Gemini does not support HTTP/HTTPS URLs for files.
+ Convert them to base64 data instead.
"""
for message in messages:
_message_content = message.get("content")
@@ -124,4 +128,16 @@ class GoogleAIStudioGeminiConfig(VertexGeminiConfig):
image_obj
)
)
+ elif element.get("type") == "file":
+ file_element = cast(ChatCompletionFileObject, element)
+ file_id = file_element["file"].get("file_id")
+ if file_id and ("http://" in file_id or "https://" in file_id):
+ # Convert HTTP/HTTPS file URL to base64 data
+ try:
+ base64_data = convert_url_to_base64(file_id)
+ file_element["file"]["file_data"] = base64_data # type: ignore
+ file_element["file"].pop("file_id", None) # type: ignore
+ except Exception:
+ # If conversion fails, leave as is and let the API handle it
+ pass
return _gemini_convert_messages_with_history(messages=messages)
diff --git a/litellm/llms/gemini/common_utils.py b/litellm/llms/gemini/common_utils.py
index 31b57434e10..e53829d3329 100644
--- a/litellm/llms/gemini/common_utils.py
+++ b/litellm/llms/gemini/common_utils.py
@@ -1,15 +1,16 @@
import base64
import datetime
-from typing import Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Union
import httpx
import litellm
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH
-from litellm.llms.base_llm.base_utils import BaseLLMModelInfo
+from litellm.llms.base_llm.base_utils import BaseLLMModelInfo, BaseTokenCounter
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
+from litellm.types.utils import TokenCountResponse
class GeminiError(BaseLLMException):
@@ -44,7 +45,7 @@ class GeminiModelInfo(BaseLLMModelInfo):
@staticmethod
def get_api_key(api_key: Optional[str] = None) -> Optional[str]:
- return api_key or (get_secret_str("GEMINI_API_KEY"))
+ return api_key or (get_secret_str("GOOGLE_API_KEY")) or (get_secret_str("GEMINI_API_KEY"))
@staticmethod
def get_base_model(model: str) -> Optional[str]:
@@ -66,7 +67,7 @@ class GeminiModelInfo(BaseLLMModelInfo):
endpoint = f"/{self.api_version}/models"
if api_base is None or api_key is None:
raise ValueError(
- "GEMINI_API_BASE or GEMINI_API_KEY is not set. Please set the environment variable, to query Gemini's `/models` endpoint."
+ "GEMINI_API_BASE or GEMINI_API_KEY/GOOGLE_API_KEY is not set. Please set the environment variable, to query Gemini's `/models` endpoint."
)
response = litellm.module_level_client.get(
@@ -89,6 +90,16 @@ class GeminiModelInfo(BaseLLMModelInfo):
return GeminiError(
status_code=status_code, message=error_message, headers=headers
)
+
+ def get_token_counter(self) -> Optional[BaseTokenCounter]:
+ """
+ Factory method to create a token counter for this provider.
+
+ Returns:
+ Optional TokenCounterInterface implementation for this provider,
+ or None if token counting is not supported.
+ """
+ return GoogleAIStudioTokenCounter()
def encode_unserializable_types(
@@ -133,3 +144,50 @@ def encode_unserializable_types(
else:
processed_data[key] = value
return processed_data
+
+
+def get_api_key_from_env() -> Optional[str]:
+ return get_secret_str("GOOGLE_API_KEY") or get_secret_str("GEMINI_API_KEY")
+
+
+class GoogleAIStudioTokenCounter(BaseTokenCounter):
+ """Token counter implementation for Google AI Studio provider."""
+ def should_use_token_counting_api(
+ self,
+ custom_llm_provider: Optional[str] = None,
+ ) -> bool:
+ from litellm.types.utils import LlmProviders
+ return custom_llm_provider == LlmProviders.GEMINI.value
+
+ async def count_tokens(
+ self,
+ model_to_use: str,
+ messages: Optional[List[Dict[str, Any]]],
+ contents: Optional[List[Dict[str, Any]]],
+ deployment: Optional[Dict[str, Any]] = None,
+ request_model: str = "",
+ ) -> Optional[TokenCountResponse]:
+ import copy
+
+ from litellm.llms.gemini.count_tokens.handler import GoogleAIStudioTokenCounter
+ deployment = deployment or {}
+ count_tokens_params_request = copy.deepcopy(deployment.get("litellm_params", {}))
+ count_tokens_params = {
+ "model": model_to_use,
+ "contents": contents,
+ }
+ count_tokens_params_request.update(count_tokens_params)
+ result = await GoogleAIStudioTokenCounter().acount_tokens(
+ **count_tokens_params_request,
+ )
+
+ if result is not None:
+ return TokenCountResponse(
+ total_tokens=result.get("totalTokens", 0),
+ request_model=request_model,
+ model_used=model_to_use,
+ tokenizer_type=result.get("tokenizer_used", ""),
+ original_response=result,
+ )
+
+ return None
\ No newline at end of file
diff --git a/litellm/llms/gemini/count_tokens/handler.py b/litellm/llms/gemini/count_tokens/handler.py
new file mode 100644
index 00000000000..4d6c7fd8864
--- /dev/null
+++ b/litellm/llms/gemini/count_tokens/handler.py
@@ -0,0 +1,164 @@
+from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, Union
+
+import httpx
+
+import litellm
+from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
+from litellm.types.utils import LlmProviders
+
+if TYPE_CHECKING:
+ from litellm.types.google_genai.main import GenerateContentContentListUnionDict
+else:
+ GenerateContentContentListUnionDict = Any
+
+
+class GoogleAIStudioTokenCounter:
+ def _clean_contents_for_gemini_api(self, contents: Any) -> Any:
+ """
+ Clean up contents to remove unsupported fields for the Gemini API.
+
+ The Google Gemini API doesn't recognize the 'id' field in function responses,
+ so we need to remove it to prevent 400 Bad Request errors.
+
+ Args:
+ contents: The contents to clean up
+
+ Returns:
+ Cleaned contents with unsupported fields removed
+ """
+ import copy
+
+ from google.genai.types import FunctionResponse
+
+ cleaned_contents = copy.deepcopy(contents)
+
+ for content in cleaned_contents:
+ parts = content["parts"]
+ for part in parts:
+ if "functionResponse" in part:
+ function_response_data = part["functionResponse"]
+ function_response_part = FunctionResponse(**function_response_data)
+ function_response_part.id = None
+ part["functionResponse"] = function_response_part.model_dump(
+ exclude_none=True
+ )
+
+ return cleaned_contents
+
+ def _construct_url(self, model: str, api_base: Optional[str] = None) -> str:
+ """
+ Construct the URL for the Google Gen AI Studio countTokens endpoint.
+ """
+ base_url = api_base or "https://generativelanguage.googleapis.com"
+ return f"{base_url}/v1beta/models/{model}:countTokens"
+
+ async def validate_environment(
+ self,
+ api_base: Optional[str] = None,
+ api_key: Optional[str] = None,
+ headers: Optional[Dict[str, Any]] = None,
+ model: str = "",
+ litellm_params: Optional[Dict[str, Any]] = None,
+ ) -> Tuple[Dict[str, Any], str]:
+ """
+ Returns a Tuple of headers and url for the Google Gen AI Studio countTokens endpoint.
+ """
+ from litellm.llms.gemini.google_genai.transformation import GoogleGenAIConfig
+
+ headers = GoogleGenAIConfig().validate_environment(
+ api_key=api_key,
+ headers=headers,
+ model=model,
+ litellm_params=litellm_params,
+ )
+
+ url = self._construct_url(model=model, api_base=api_base)
+ return headers, url
+
+ async def acount_tokens(
+ self,
+ contents: Any,
+ model: str,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ **kwargs,
+ ) -> Dict[str, Any]:
+ """
+ Count tokens using Google Gen AI Studio countTokens endpoint.
+
+ Args:
+ contents: The content to count tokens for (Google Gen AI format)
+ Example: [{"parts": [{"text": "Hello world"}]}]
+ model: The model name (e.g. "gemini-1.5-flash")
+ api_key: Optional Google API key (will fall back to environment)
+ api_base: Optional API base URL (defaults to Google Gen AI Studio)
+ timeout: Optional timeout for the request
+ **kwargs: Additional parameters
+
+ Returns:
+ Dict containing token count information from Google Gen AI Studio API.
+ Example response:
+ {
+ "totalTokens": 31,
+ "totalBillableCharacters": 96,
+ "promptTokensDetails": [
+ {
+ "modality": "TEXT",
+ "tokenCount": 31
+ }
+ ]
+ }
+
+ Raises:
+ ValueError: If API key is missing
+ litellm.APIError: If the API call fails
+ litellm.APIConnectionError: If the connection fails
+ Exception: For any other unexpected errors
+ """
+
+ # Prepare headers
+ headers, url = await self.validate_environment(
+ api_key=api_key,
+ api_base=api_base,
+ headers={},
+ model=model,
+ litellm_params=kwargs,
+ )
+
+ # Prepare request body - clean up contents to remove unsupported fields
+ cleaned_contents = self._clean_contents_for_gemini_api(contents)
+ request_body = {"contents": cleaned_contents}
+
+ async_httpx_client = get_async_httpx_client(
+ llm_provider=LlmProviders.GEMINI,
+ )
+
+ try:
+ response = await async_httpx_client.post(
+ url=url, headers=headers, json=request_body
+ )
+
+ # Check for HTTP errors
+ response.raise_for_status()
+
+ # Parse response
+ result = response.json()
+ return result
+
+ except httpx.HTTPStatusError as e:
+ error_msg = f"Google Gen AI Studio API error: {e.response.status_code} - {e.response.text}"
+ raise litellm.APIError(
+ message=error_msg,
+ llm_provider="gemini",
+ model=model,
+ status_code=e.response.status_code,
+ ) from e
+ except httpx.RequestError as e:
+ error_msg = f"Request to Google Gen AI Studio failed: {str(e)}"
+ raise litellm.APIConnectionError(
+ message=error_msg, llm_provider="gemini", model=model
+ ) from e
+ except Exception as e:
+ error_msg = f"Unexpected error during token counting: {str(e)}"
+ raise Exception(error_msg) from e
diff --git a/litellm/llms/gemini/google_genai/transformation.py b/litellm/llms/gemini/google_genai/transformation.py
index 9910e478063..94dfea5f58a 100644
--- a/litellm/llms/gemini/google_genai/transformation.py
+++ b/litellm/llms/gemini/google_genai/transformation.py
@@ -1,6 +1,7 @@
"""
Transformation for Calling Google models in their native format.
"""
+
from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple, Union, cast
import httpx
@@ -11,7 +12,6 @@ from litellm.llms.base_llm.google_genai.transformation import (
BaseGoogleGenAIGenerateContentConfig,
)
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexLLM
-from litellm.secret_managers.main import get_secret_str
from litellm.types.router import GenericLiteLLMParams
if TYPE_CHECKING:
@@ -19,25 +19,36 @@ if TYPE_CHECKING:
GenerateContentConfigDict,
GenerateContentContentListUnionDict,
GenerateContentResponse,
+ ToolConfigDict,
)
else:
GenerateContentConfigDict = Any
GenerateContentContentListUnionDict = Any
GenerateContentResponse = Any
+ ToolConfigDict = Any
+
+from ..common_utils import get_api_key_from_env
class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
"""
Configuration for calling Google models in their native format.
"""
+
+ ##############################
+ # Constants
+ ##############################
+ XGOOGLE_API_KEY = "x-goog-api-key"
+ ##############################
+
@property
def custom_llm_provider(self) -> Literal["gemini", "vertex_ai"]:
return "gemini"
-
+
def __init__(self):
super().__init__()
VertexLLM.__init__(self)
-
+
def get_supported_generate_content_optional_params(self, model: str) -> List[str]:
"""
Get the list of supported Google GenAI parameters for the model.
@@ -50,7 +61,7 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
"""
return [
"http_options",
- "system_instruction",
+ "system_instruction",
"temperature",
"top_p",
"top_k",
@@ -76,10 +87,9 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
"speech_config",
"audio_timestamp",
"automatic_function_calling",
- "thinking_config"
+ "thinking_config",
]
-
def map_generate_content_optional_params(
self,
generate_content_config_dict: GenerateContentConfigDict,
@@ -95,26 +105,31 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
Returns:
Mapped parameters for the provider
"""
- from litellm.types.google_genai.main import GenerateContentConfigDict
- _generate_content_config_dict = GenerateContentConfigDict()
- supported_google_genai_params = self.get_supported_generate_content_optional_params(model)
+ _generate_content_config_dict: Dict[str, Any] = {}
+ supported_google_genai_params = (
+ self.get_supported_generate_content_optional_params(model)
+ )
for param, value in generate_content_config_dict.items():
if param in supported_google_genai_params:
_generate_content_config_dict[param] = value
- return dict(_generate_content_config_dict)
-
+ return _generate_content_config_dict
+
def validate_environment(
- self,
+ self,
api_key: Optional[str],
headers: Optional[dict],
model: str,
- litellm_params: Optional[Union[GenericLiteLLMParams, dict]]
+ litellm_params: Optional[Union[GenericLiteLLMParams, dict]],
) -> dict:
default_headers = {
"Content-Type": "application/json",
}
- if api_key is not None:
- default_headers["Authorization"] = f"Bearer {api_key}"
+ # Use the passed api_key first, then fall back to litellm_params and environment
+ gemini_api_key = api_key or self._get_google_ai_studio_api_key(
+ dict(litellm_params or {})
+ )
+ if gemini_api_key is not None:
+ default_headers[self.XGOOGLE_API_KEY] = gemini_api_key
if headers is not None:
default_headers.update(headers)
@@ -124,17 +139,17 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
return (
litellm_params.pop("api_key", None)
or litellm_params.pop("gemini_api_key", None)
- or get_secret_str("GEMINI_API_KEY")
+ or get_api_key_from_env()
or litellm.api_key
)
-
+
def _get_common_auth_components(
self,
litellm_params: dict,
) -> Tuple[Any, Optional[str], Optional[str]]:
"""
Get common authentication components used by both sync and async methods.
-
+
Returns:
Tuple of (vertex_credentials, vertex_project, vertex_location)
"""
@@ -142,7 +157,7 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
vertex_project = self.get_vertex_ai_project(litellm_params)
vertex_location = self.get_vertex_ai_location(litellm_params)
return vertex_credentials, vertex_project, vertex_location
-
+
def _build_final_headers_and_url(
self,
model: str,
@@ -158,7 +173,7 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
Build final headers and API URL from auth components.
"""
gemini_api_key = self._get_google_ai_studio_api_key(litellm_params)
-
+
auth_header, api_base = self._get_token_and_url(
model=model,
gemini_api_key=gemini_api_key,
@@ -191,7 +206,9 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
"""
Sync version of get_auth_token_and_url.
"""
- vertex_credentials, vertex_project, vertex_location = self._get_common_auth_components(litellm_params)
+ vertex_credentials, vertex_project, vertex_location = (
+ self._get_common_auth_components(litellm_params)
+ )
_auth_header, vertex_project = self._ensure_access_token(
credentials=vertex_credentials,
@@ -228,7 +245,9 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
Returns:
Tuple of headers and API base
"""
- vertex_credentials, vertex_project, vertex_location = self._get_common_auth_components(litellm_params)
+ vertex_credentials, vertex_project, vertex_location = (
+ self._get_common_auth_components(litellm_params)
+ )
_auth_header, vertex_project = await self._ensure_access_token_async(
credentials=vertex_credentials,
@@ -246,28 +265,30 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
api_base=api_base,
litellm_params=litellm_params,
)
-
def transform_generate_content_request(
self,
model: str,
contents: GenerateContentContentListUnionDict,
+ tools: Optional[ToolConfigDict],
generate_content_config_dict: Dict,
) -> dict:
from litellm.types.google_genai.main import (
GenerateContentConfigDict,
GenerateContentRequestDict,
)
+
typed_generate_content_request = GenerateContentRequestDict(
model=model,
contents=contents,
+ tools=tools,
generationConfig=GenerateContentConfigDict(**generate_content_config_dict),
)
request_dict = cast(dict, typed_generate_content_request)
return request_dict
-
+
def transform_generate_content_response(
self,
model: str,
@@ -285,6 +306,7 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
Transformed response data
"""
from litellm.types.google_genai.main import GenerateContentResponse
+
try:
response = raw_response.json()
except Exception as e:
@@ -293,7 +315,7 @@ class GoogleGenAIConfig(BaseGoogleGenAIGenerateContentConfig, VertexLLM):
status_code=raw_response.status_code,
headers=raw_response.headers,
)
-
+
logging_obj.model_call_details["httpx_response"] = raw_response
-
- return GenerateContentResponse(**response)
\ No newline at end of file
+
+ return GenerateContentResponse(**response)
diff --git a/litellm/llms/gemini/image_generation/transformation.py b/litellm/llms/gemini/image_generation/transformation.py
index 72ba5bcf1ee..f136bd0a404 100644
--- a/litellm/llms/gemini/image_generation/transformation.py
+++ b/litellm/llms/gemini/image_generation/transformation.py
@@ -85,17 +85,25 @@ class GoogleImageGenConfig(BaseImageGenerationConfig):
) -> str:
"""
Get the complete url for the request
-
- Google AI API format: https://generativelanguage.googleapis.com/v1beta/models/{model}:predict
+
+ Gemini 2.5 Flash Image Preview: :generateContent
+ Other Imagen models: :predict
"""
complete_url: str = (
- api_base
- or get_secret_str("GEMINI_API_BASE")
+ api_base
+ or get_secret_str("GEMINI_API_BASE")
or self.DEFAULT_BASE_URL
)
complete_url = complete_url.rstrip("/")
- complete_url = f"{complete_url}/models/{model}:predict"
+
+ # Gemini 2.5 Flash Image Preview uses generateContent endpoint
+ if "2.5-flash-image-preview" in model:
+ complete_url = f"{complete_url}/models/{model}:generateContent"
+ else:
+ # All other Imagen models use predict endpoint
+ complete_url = f"{complete_url}/models/{model}:predict"
+
return complete_url
def validate_environment(
@@ -128,35 +136,52 @@ class GoogleImageGenConfig(BaseImageGenerationConfig):
headers: dict,
) -> dict:
"""
- Transform the image generation request to Google AI Imagen format
-
- Google AI API format:
+ Transform the image generation request to Gemini format
+
+ For Gemini 2.5 Flash Image Preview, use the standard Gemini format with response_modalities:
{
- "instances": [
+ "contents": [
{
- "prompt": "Robot holding a red skateboard"
+ "parts": [
+ {"text": "Generate an image of..."}
+ ]
}
],
- "parameters": {
- "sampleCount": 4,
- "aspectRatio": "1:1",
- "personGeneration": "allow_adult"
+ "generationConfig": {
+ "response_modalities": ["IMAGE", "TEXT"]
}
}
"""
- from litellm.types.llms.gemini import (
- GeminiImageGenerationInstance,
- GeminiImageGenerationParameters,
- )
- request_body: GeminiImageGenerationRequest = GeminiImageGenerationRequest(
- instances=[
- GeminiImageGenerationInstance(
- prompt=prompt
- )
- ],
- parameters=GeminiImageGenerationParameters(**optional_params)
- )
- return request_body.model_dump(exclude_none=True)
+ # For Gemini 2.5 Flash Image Preview, use standard Gemini format
+ if "2.5-flash-image-preview" in model:
+ request_body: dict = {
+ "contents": [
+ {
+ "parts": [
+ {"text": prompt}
+ ]
+ }
+ ],
+ "generationConfig": {
+ "response_modalities": ["IMAGE", "TEXT"]
+ }
+ }
+ return request_body
+ else:
+ # For other Imagen models, use the original Imagen format
+ from litellm.types.llms.gemini import (
+ GeminiImageGenerationInstance,
+ GeminiImageGenerationParameters,
+ )
+ request_body_obj: GeminiImageGenerationRequest = GeminiImageGenerationRequest(
+ instances=[
+ GeminiImageGenerationInstance(
+ prompt=prompt
+ )
+ ],
+ parameters=GeminiImageGenerationParameters(**optional_params)
+ )
+ return request_body_obj.model_dump(exclude_none=True)
def transform_image_generation_response(
self,
@@ -185,16 +210,30 @@ class GoogleImageGenConfig(BaseImageGenerationConfig):
if not model_response.data:
model_response.data = []
-
- # Google AI returns predictions with generated images
- predictions = response_data.get("predictions", [])
- for prediction in predictions:
- # Google AI returns base64 encoded images in the prediction
- generated_images = prediction.get("generatedImages", [])
- for image_data in generated_images:
+
+ # Handle different response formats based on model
+ if "2.5-flash-image-preview" in model:
+ # Gemini 2.5 Flash Image Preview returns in candidates format
+ candidates = response_data.get("candidates", [])
+ for candidate in candidates:
+ content = candidate.get("content", {})
+ parts = content.get("parts", [])
+ for part in parts:
+ # Look for inlineData with image
+ if "inlineData" in part:
+ inline_data = part["inlineData"]
+ if "data" in inline_data:
+ model_response.data.append(ImageObject(
+ b64_json=inline_data["data"],
+ url=None,
+ ))
+ else:
+ # Original Imagen format - predictions with generated images
+ predictions = response_data.get("predictions", [])
+ for prediction in predictions:
+ # Google AI returns base64 encoded images in the prediction
model_response.data.append(ImageObject(
- b64_json=image_data.get("bytesBase64Encoded", None),
+ b64_json=prediction.get("bytesBase64Encoded", None),
url=None, # Google AI returns base64, not URLs
))
-
return model_response
\ No newline at end of file
diff --git a/litellm/llms/gemini/realtime/transformation.py b/litellm/llms/gemini/realtime/transformation.py
index 980723eb3fe..62329358e47 100644
--- a/litellm/llms/gemini/realtime/transformation.py
+++ b/litellm/llms/gemini/realtime/transformation.py
@@ -3,11 +3,10 @@ This file contains the transformation logic for the Gemini realtime API.
"""
import json
-import os
-import uuid
from typing import Any, Dict, List, Optional, Union, cast
from litellm import verbose_logger
+from litellm._uuid import uuid
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.realtime.transformation import BaseRealtimeConfig
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
@@ -55,7 +54,7 @@ from litellm.types.realtime import (
)
from litellm.utils import get_empty_usage
-from ..common_utils import encode_unserializable_types
+from ..common_utils import encode_unserializable_types, get_api_key_from_env
MAP_GEMINI_FIELD_TO_OPENAI_EVENT: Dict[str, OpenAIRealtimeEventTypes] = {
"setupComplete": OpenAIRealtimeEventTypes.SESSION_CREATED,
@@ -81,7 +80,7 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
if api_base is None:
api_base = "wss://generativelanguage.googleapis.com"
if api_key is None:
- api_key = os.environ.get("GEMINI_API_KEY")
+ api_key = get_api_key_from_env()
if api_key is None:
raise ValueError("api_key is required for Gemini API calls")
api_base = api_base.replace("https://", "wss://")
@@ -187,10 +186,11 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
)
vertex_gemini_config = VertexGeminiConfig()
- vertex_gemini_config._map_function(value)
- optional_params["generationConfig"][
- "tools"
- ] = vertex_gemini_config._map_function(value)
+ optional_params["generationConfig"]["tools"] = (
+ vertex_gemini_config._map_function(
+ value=value, optional_params=optional_params
+ )
+ )
elif key == "input_audio_transcription" and value is not None:
optional_params["inputAudioTranscription"] = {}
elif key == "turn_detection":
@@ -201,10 +201,10 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
if (
len(transformed_audio_activity_config) > 0
): # if the config is not empty, add it to the optional params
- optional_params[
- "realtimeInputConfig"
- ] = BidiGenerateContentRealtimeInputConfig(
- automaticActivityDetection=transformed_audio_activity_config
+ optional_params["realtimeInputConfig"] = (
+ BidiGenerateContentRealtimeInputConfig(
+ automaticActivityDetection=transformed_audio_activity_config
+ )
)
if len(optional_params["generationConfig"]) == 0:
optional_params.pop("generationConfig")
@@ -405,15 +405,17 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
output_index=0,
event_id="event_{}".format(uuid.uuid4()),
item_id=output_item_id,
- part={
- "type": "text",
- "text": "",
- }
- if delta_type == "text"
- else {
- "type": "audio",
- "transcript": "",
- },
+ part=(
+ {
+ "type": "text",
+ "text": "",
+ }
+ if delta_type == "text"
+ else {
+ "type": "audio",
+ "transcript": "",
+ }
+ ),
response_id=response_id,
)
response_items.append(response_content_part_added)
@@ -440,9 +442,11 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
)
return OpenAIRealtimeResponseDelta(
- type="response.text.delta"
- if delta_type == "text"
- else "response.audio.delta",
+ type=(
+ "response.text.delta"
+ if delta_type == "text"
+ else "response.audio.delta"
+ ),
content_index=0,
event_id="event_{}".format(uuid.uuid4()),
item_id=output_item_id,
@@ -513,12 +517,14 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
event_id="event_{}".format(uuid.uuid4()),
item_id=current_output_item_id,
output_index=0,
- part={"type": "text", "text": delta_done_event_text}
- if delta_done_event_text and delta_type == "text"
- else {
- "type": "audio",
- "transcript": "", # gemini doesn't return transcript for audio
- },
+ part=(
+ {"type": "text", "text": delta_done_event_text}
+ if delta_done_event_text and delta_type == "text"
+ else {
+ "type": "audio",
+ "transcript": "", # gemini doesn't return transcript for audio
+ }
+ ),
response_id=current_response_id,
)
returned_items.append(response_content_part_done)
@@ -535,12 +541,14 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
"status": "completed",
"role": "assistant",
"content": [
- {"type": "text", "text": delta_done_event_text}
- if delta_done_event_text and delta_type == "text"
- else {
- "type": "audio",
- "transcript": "",
- }
+ (
+ {"type": "text", "text": delta_done_event_text}
+ if delta_done_event_text and delta_type == "text"
+ else {
+ "type": "audio",
+ "transcript": "",
+ }
+ )
],
},
)
@@ -674,9 +682,11 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
object="realtime.response",
id=current_response_id,
status="completed",
- output=[output_item["item"] for output_item in output_items]
- if output_items
- else [],
+ output=(
+ [output_item["item"] for output_item in output_items]
+ if output_items
+ else []
+ ),
conversation_id=current_conversation_id,
modalities=_modalities,
usage=responses_api_usage.model_dump(),
@@ -828,9 +838,9 @@ class GeminiRealtimeConfig(BaseRealtimeConfig):
"session_configuration_request"
]
current_item_chunks = realtime_response_transform_input["current_item_chunks"]
- current_delta_type: Optional[
- ALL_DELTA_TYPES
- ] = realtime_response_transform_input["current_delta_type"]
+ current_delta_type: Optional[ALL_DELTA_TYPES] = (
+ realtime_response_transform_input["current_delta_type"]
+ )
returned_message: List[OpenAIRealtimeEvents] = []
for key, value in json_message.items():
diff --git a/litellm/llms/github_copilot/chat/transformation.py b/litellm/llms/github_copilot/chat/transformation.py
index 4526e6247b4..66227ac21d8 100644
--- a/litellm/llms/github_copilot/chat/transformation.py
+++ b/litellm/llms/github_copilot/chat/transformation.py
@@ -75,8 +75,36 @@ class GithubCopilotConfig(OpenAIConfig):
initiator = self._determine_initiator(messages)
validated_headers["X-Initiator"] = initiator
+ # Add Copilot-Vision-Request header if request contains images
+ if self._has_vision_content(messages):
+ validated_headers["Copilot-Vision-Request"] = "true"
+
return validated_headers
+ def get_supported_openai_params(self, model: str) -> list:
+ """
+ Get supported OpenAI parameters for GitHub Copilot.
+
+ For Claude models that support extended thinking (Claude 4 family and Claude 3-7), includes thinking and reasoning_effort parameters.
+ For other models, returns standard OpenAI parameters (which may include reasoning_effort for o-series models).
+ """
+ from litellm.utils import supports_reasoning
+
+ # Get base OpenAI parameters
+ base_params = super().get_supported_openai_params(model)
+
+ # Add Claude-specific parameters for models that support extended thinking
+ if "claude" in model.lower() and supports_reasoning(
+ model=model.lower(),
+ ):
+ if "thinking" not in base_params:
+ base_params.append("thinking")
+ # reasoning_effort is not included by parent for Claude models, so add it
+ if "reasoning_effort" not in base_params:
+ base_params.append("reasoning_effort")
+
+ return base_params
+
def _determine_initiator(self, messages: List[AllMessageValues]) -> str:
"""
Determine if request is user or agent initiated based on message roles.
@@ -87,3 +115,27 @@ class GithubCopilotConfig(OpenAIConfig):
if role in ["tool", "assistant"]:
return "agent"
return "user"
+
+ def _has_vision_content(self, messages: List[AllMessageValues]) -> bool:
+ """
+ Check if any message contains vision content (images).
+ Returns True if any message has content with vision-related types, otherwise False.
+
+ Checks for:
+ - image_url content type (OpenAI format)
+ - Content items with type 'image_url'
+ """
+ for message in messages:
+ content = message.get("content")
+ if isinstance(content, list):
+ # Check if any content item indicates vision content
+ for content_item in content:
+ if isinstance(content_item, dict):
+ # Check for image_url field (direct image URL)
+ if "image_url" in content_item:
+ return True
+ # Check for type field indicating image content
+ content_type = content_item.get("type")
+ if content_type == "image_url":
+ return True
+ return False
diff --git a/litellm/llms/github_copilot/common_utils.py b/litellm/llms/github_copilot/common_utils.py
index 4c9a4b6dad0..86fbb706e52 100644
--- a/litellm/llms/github_copilot/common_utils.py
+++ b/litellm/llms/github_copilot/common_utils.py
@@ -28,7 +28,6 @@ class GithubCopilotError(BaseLLMException):
)
-
class GetDeviceCodeError(GithubCopilotError):
pass
diff --git a/litellm/llms/gradient_ai/chat/transformation.py b/litellm/llms/gradient_ai/chat/transformation.py
new file mode 100644
index 00000000000..d631affdef8
--- /dev/null
+++ b/litellm/llms/gradient_ai/chat/transformation.py
@@ -0,0 +1,147 @@
+from typing import List, Optional, Tuple, Union, Dict, Literal
+
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.openai import (
+ AllMessageValues,
+)
+
+from ...openai_like.chat.transformation import OpenAILikeChatConfig
+
+# Default GradientAI endpoint
+GRADIENT_AI_SERVERLESS_ENDPOINT = "https://inference.do-ai.run"
+
+
+class GradientAIConfig(OpenAILikeChatConfig):
+
+ k: Optional[int] = None
+ kb_filters: Optional[List[Dict]] = None
+ filter_kb_content_by_query_metadata: Optional[bool] = None
+ instruction_override: Optional[str] = None
+ include_functions_info: Optional[bool] = None
+ include_retrieval_info: Optional[bool] = None
+ include_guardrails_info: Optional[bool] = None
+ provide_citations: Optional[bool] = None
+ retrieval_method: Optional[Literal["rewrite", "step_back", "sub_queries", "none"]] = None
+
+ def __init__(
+ self,
+ frequency_penalty: Optional[float] = None,
+ max_tokens: Optional[int] = None,
+ max_completion_tokens: Optional[int] = None,
+ presence_penalty: Optional[float] = None,
+ retrieval_method: Optional[str] = None,
+ stop: Optional[Union[str, List[str]]] = None,
+ stream: Optional[bool] = None,
+ temperature: Optional[float] = None,
+ top_p: Optional[float] = None,
+ k: Optional[int] = None,
+ kb_filters: Optional[List[Dict]] = None,
+ filter_kb_content_by_query_metadata: Optional[bool] = None,
+ instruction_override: Optional[str] = None,
+ include_functions_info: Optional[bool] = None,
+ include_retrieval_info: Optional[bool] = None,
+ include_guardrails_info: Optional[bool] = None,
+ provide_citations: Optional[bool] = None,
+ ) -> None:
+ locals_ = locals().copy()
+ for key, value in locals_.items():
+ if key != "self" and value is not None:
+ setattr(self.__class__, key, value)
+
+ @classmethod
+ def get_config(cls):
+ return super().get_config()
+
+ def get_supported_openai_params(self, model: str) -> list:
+ supported_params = [
+ "frequency_penalty",
+ "max_tokens",
+ "max_completion_tokens",
+ "presence_penalty",
+ "stop",
+ "stream",
+ "stream_options",
+ "temperature",
+ "top_p",
+ # GradientAI specific parameters
+ "k",
+ "kb_filters",
+ "filter_kb_content_by_query_metadata",
+ "instruction_override",
+ "include_functions_info",
+ "include_retrieval_info",
+ "include_guardrails_info",
+ "provide_citations",
+ "retrieval_method",
+ ]
+ return supported_params
+
+ def validate_environment(self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None):
+ api_key = api_key or get_secret_str("GRADIENT_AI_API_KEY")
+ if api_key is None:
+ raise ValueError("GradientAI API key not found")
+ if headers is None:
+ headers = {}
+ headers["Authorization"] = f"Bearer {api_key}"
+ headers["Content-Type"] = "application/json"
+ return headers
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ gradient_ai_endpoint = get_secret_str("GRADIENT_AI_AGENT_ENDPOINT")
+ complete_url = f"{GRADIENT_AI_SERVERLESS_ENDPOINT}/v1/chat/completions"
+
+ if api_base and api_base != GRADIENT_AI_SERVERLESS_ENDPOINT:
+ complete_url = f"{api_base}/api/v1/chat/completions"
+ elif gradient_ai_endpoint and gradient_ai_endpoint != GRADIENT_AI_SERVERLESS_ENDPOINT:
+ complete_url = f"{gradient_ai_endpoint}/api/v1/chat/completions"
+
+ return complete_url
+
+ def _get_openai_compatible_provider_info(
+ self, api_base: Optional[str], api_key: Optional[str]
+ ) -> Tuple[Optional[str], Optional[str]]:
+ gradient_ai_endpoint = get_secret_str("GRADIENT_AI_AGENT_ENDPOINT")
+
+ if not api_base and not gradient_ai_endpoint:
+ api_base = GRADIENT_AI_SERVERLESS_ENDPOINT
+ else:
+ api_base = api_base or gradient_ai_endpoint
+
+ dynamic_api_key = api_key or get_secret_str("GRADIENT_AI_API_KEY")
+ return api_base, dynamic_api_key
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool = False,
+ replace_max_completion_tokens_with_max_tokens: bool = False,
+ ) -> dict:
+ supported_openai_params = self.get_supported_openai_params(model=model)
+ for param, value in non_default_params.items():
+ if param in supported_openai_params:
+ optional_params[param] = value
+ elif not drop_params:
+ from litellm.utils import UnsupportedParamsError
+ raise UnsupportedParamsError(
+ status_code=400,
+ message=f"GradientAI does not support parameter '{param}'. To drop unsupported params, set `drop_params=True`."
+ )
+
+ return optional_params
diff --git a/litellm/llms/groq/chat/transformation.py b/litellm/llms/groq/chat/transformation.py
index 86fa323f9e3..165301efb5c 100644
--- a/litellm/llms/groq/chat/transformation.py
+++ b/litellm/llms/groq/chat/transformation.py
@@ -6,6 +6,8 @@ from typing import Any, Coroutine, List, Literal, Optional, Tuple, Union, cast,
import httpx
from pydantic import BaseModel
+import litellm
+from litellm._logging import verbose_logger
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import (
@@ -55,6 +57,10 @@ class GroqChatConfig(OpenAILikeChatConfig):
if key != "self" and value is not None:
setattr(self.__class__, key, value)
+ @property
+ def custom_llm_provider(self) -> Optional[str]:
+ return "groq"
+
@classmethod
def get_config(cls):
return super().get_config()
@@ -65,6 +71,15 @@ class GroqChatConfig(OpenAILikeChatConfig):
base_params.remove("max_retries")
except ValueError:
pass
+
+ try:
+ if litellm.supports_reasoning(
+ model=model, custom_llm_provider=self.custom_llm_provider
+ ):
+ base_params.append("reasoning_effort")
+ except Exception as e:
+ verbose_logger.debug(f"Error checking if model supports reasoning: {e}")
+
return base_params
@overload
diff --git a/litellm/llms/heroku/chat/transformation.py b/litellm/llms/heroku/chat/transformation.py
new file mode 100644
index 00000000000..a64d8afe63a
--- /dev/null
+++ b/litellm/llms/heroku/chat/transformation.py
@@ -0,0 +1,67 @@
+"""
+Heroku Chat Completions API
+
+this is OpenAI compatible - no translation needed / occurs
+"""
+import os
+
+from typing import Optional, List, Tuple, Union, Coroutine, Any, Literal, overload
+from litellm.litellm_core_utils.prompt_templates.common_utils import (
+ handle_messages_with_content_list_to_str_conversion,
+)
+from litellm.types.llms.openai import AllMessageValues
+from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
+
+# Base error class for Heroku
+class HerokuError(Exception):
+ pass
+
+class HerokuChatConfig(OpenAIGPTConfig):
+ @overload
+ def _transform_messages(
+ self, messages: List[AllMessageValues], model: str, is_async: Literal[True]
+ ) -> Coroutine[Any, Any, List[AllMessageValues]]:
+ ...
+
+ @overload
+ def _transform_messages(
+ self,
+ messages: List[AllMessageValues],
+ model: str,
+ is_async: Literal[False] = False,
+ ) -> List[AllMessageValues]:
+ ...
+
+ def _transform_messages(
+ self, messages: List[AllMessageValues], model: str, is_async: bool = False
+ ) -> Union[List[AllMessageValues], Coroutine[Any, Any, List[AllMessageValues]]]:
+ """
+ Heroku does not support content in list format.
+ See: https://devcenter.heroku.com/articles/heroku-inference-api-v1-chat-completions#content-object
+ """
+ messages = handle_messages_with_content_list_to_str_conversion(messages)
+ if is_async:
+ return super()._transform_messages(
+ messages=messages, model=model, is_async=True
+ )
+ else:
+ return super()._transform_messages(
+ messages=messages, model=model, is_async=False
+ )
+
+ def _get_openai_compatible_provider_info(self, api_base: Optional[str], api_key: Optional[str]) -> Tuple[Optional[str], Optional[str]]:
+ api_base = api_base or os.getenv("HEROKU_API_BASE")
+ api_key = api_key or os.getenv("HEROKU_API_KEY")
+
+ return api_base, api_key
+
+ def get_complete_url(self, api_base: Optional[str], api_key: Optional[str], model: str, optional_params: dict, litellm_params: dict, stream: Optional[bool] = None) -> str:
+ api_base, _ = self._get_openai_compatible_provider_info(api_base, api_key)
+
+ if not api_base:
+ raise HerokuError("No api base was set. Please provide an api_base, or set the HEROKU_API_BASE environment variable.")
+
+ if not api_base.endswith("/v1/chat/completions"):
+ api_base = f"{api_base}/v1/chat/completions"
+
+ return api_base
\ No newline at end of file
diff --git a/litellm/llms/hosted_vllm/chat/transformation.py b/litellm/llms/hosted_vllm/chat/transformation.py
index 529354f80eb..1d21490ea31 100644
--- a/litellm/llms/hosted_vllm/chat/transformation.py
+++ b/litellm/llms/hosted_vllm/chat/transformation.py
@@ -21,6 +21,11 @@ from ...openai.chat.gpt_transformation import OpenAIGPTConfig
class HostedVLLMChatConfig(OpenAIGPTConfig):
+ def get_supported_openai_params(self, model: str) -> List[str]:
+ params = super().get_supported_openai_params(model)
+ params.append("reasoning_effort")
+ return params
+
def map_openai_params(
self,
non_default_params: dict,
diff --git a/litellm/llms/hosted_vllm/rerank/transformation.py b/litellm/llms/hosted_vllm/rerank/transformation.py
index 419327d9d5c..2faef2c4c73 100644
--- a/litellm/llms/hosted_vllm/rerank/transformation.py
+++ b/litellm/llms/hosted_vllm/rerank/transformation.py
@@ -2,27 +2,26 @@
Transformation logic for Hosted VLLM rerank
"""
-import uuid
from typing import Any, Dict, List, Optional, Union
+import httpx
+
+from litellm._uuid import uuid
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig
+from litellm.secret_managers.main import get_secret_str
from litellm.types.rerank import (
+ OptionalRerankParams,
RerankBilledUnits,
+ RerankRequest,
RerankResponse,
RerankResponseDocument,
RerankResponseMeta,
RerankResponseResult,
RerankTokens,
- OptionalRerankParams,
- RerankRequest,
)
-import httpx
-
-from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
-from litellm.llms.base_llm.chat.transformation import BaseLLMException
-from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig
-from litellm.secret_managers.main import get_secret_str
-
class HostedVLLMRerankError(BaseLLMException):
def __init__(
@@ -42,8 +41,11 @@ class HostedVLLMRerankConfig(BaseRerankConfig):
if api_base:
# Remove trailing slashes and ensure clean base URL
api_base = api_base.rstrip("/")
- if not api_base.endswith("/v1/rerank"):
- api_base = f"{api_base}/v1/rerank"
+ # Preserve backward compatibility
+ if api_base.endswith("/v1/rerank"):
+ api_base = api_base.replace("/v1/rerank", "/rerank")
+ elif not api_base.endswith("/rerank"):
+ api_base = f"{api_base}/rerank"
return api_base
raise ValueError("api_base must be provided for Hosted VLLM rerank")
@@ -69,20 +71,20 @@ class HostedVLLMRerankConfig(BaseRerankConfig):
return_documents: Optional[bool] = True,
max_chunks_per_doc: Optional[int] = None,
max_tokens_per_doc: Optional[int] = None,
- ) -> OptionalRerankParams:
+ ) -> Dict:
"""
Map parameters for Hosted VLLM rerank
"""
if max_chunks_per_doc is not None:
raise ValueError("Hosted VLLM does not support max_chunks_per_doc")
- return OptionalRerankParams(
+ return dict(OptionalRerankParams(
query=query,
documents=documents,
top_n=top_n,
rank_fields=rank_fields,
return_documents=return_documents,
- )
+ ))
def validate_environment(
self,
@@ -109,7 +111,7 @@ class HostedVLLMRerankConfig(BaseRerankConfig):
def transform_rerank_request(
self,
model: str,
- optional_rerank_params: OptionalRerankParams,
+ optional_rerank_params: Dict,
headers: dict,
) -> dict:
if "query" not in optional_rerank_params:
diff --git a/litellm/llms/hosted_vllm/transcriptions/transformation.py b/litellm/llms/hosted_vllm/transcriptions/transformation.py
new file mode 100644
index 00000000000..e726ee33abf
--- /dev/null
+++ b/litellm/llms/hosted_vllm/transcriptions/transformation.py
@@ -0,0 +1,65 @@
+"""
+Transformation logic for Hosted VLLM rerank
+"""
+
+from typing import Optional, Union
+
+import httpx
+
+from litellm.llms.base_llm.audio_transcription.transformation import (
+ AudioTranscriptionRequestData,
+)
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.openai.transcriptions.whisper_transformation import (
+ OpenAIWhisperAudioTranscriptionConfig,
+)
+from litellm.types.utils import FileTypes
+
+
+class HostedVLLMAudioTranscriptionError(BaseLLMException):
+ def __init__(
+ self,
+ status_code: int,
+ message: str,
+ headers: Optional[Union[dict, httpx.Headers]] = None,
+ ):
+ super().__init__(status_code=status_code, message=message, headers=headers)
+
+
+class HostedVLLMAudioTranscriptionConfig(OpenAIWhisperAudioTranscriptionConfig):
+ def __init__(self) -> None:
+ pass
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ if api_base:
+ # Remove trailing slashes and ensure clean base URL
+ api_base = api_base.rstrip("/")
+ if not api_base.endswith("/v1/audio/transcriptions"):
+ api_base = f"{api_base}/v1/audio/transcriptions"
+ return api_base
+ raise ValueError("api_base must be provided for Hosted VLLM rerank")
+
+ def transform_audio_transcription_request(
+ self,
+ model: str,
+ audio_file: FileTypes,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> AudioTranscriptionRequestData:
+ """
+ Transform the audio transcription request
+ """
+
+ data = {"model": model, "file": audio_file, **optional_params}
+
+ return AudioTranscriptionRequestData(
+ data=data,
+ )
diff --git a/litellm/llms/huggingface/embedding/transformation.py b/litellm/llms/huggingface/embedding/transformation.py
index 60bd5dcd617..88d42cfcdcc 100644
--- a/litellm/llms/huggingface/embedding/transformation.py
+++ b/litellm/llms/huggingface/embedding/transformation.py
@@ -40,17 +40,17 @@ class HuggingFaceEmbeddingConfig(BaseConfig):
Reference: https://huggingface.github.io/text-generation-inference/#/Text%20Generation%20Inference/compat_generate
"""
- hf_task: Optional[
- hf_tasks
- ] = None # litellm-specific param, used to know the api spec to use when calling huggingface api
+ hf_task: Optional[hf_tasks] = (
+ None # litellm-specific param, used to know the api spec to use when calling huggingface api
+ )
best_of: Optional[int] = None
decoder_input_details: Optional[bool] = None
details: Optional[bool] = True # enables returning logprobs + best of
max_new_tokens: Optional[int] = None
repetition_penalty: Optional[float] = None
- return_full_text: Optional[
- bool
- ] = False # by default don't return the input as part of the output
+ return_full_text: Optional[bool] = (
+ False # by default don't return the input as part of the output
+ )
seed: Optional[int] = None
temperature: Optional[float] = None
top_k: Optional[int] = None
@@ -120,9 +120,9 @@ class HuggingFaceEmbeddingConfig(BaseConfig):
optional_params["top_p"] = value
if param == "n":
optional_params["best_of"] = value
- optional_params[
- "do_sample"
- ] = True # Need to sample if you want best of for hf inference endpoints
+ optional_params["do_sample"] = (
+ True # Need to sample if you want best of for hf inference endpoints
+ )
if param == "stream":
optional_params["stream"] = value
if param == "stop":
@@ -268,7 +268,7 @@ class HuggingFaceEmbeddingConfig(BaseConfig):
# check if the model has a registered custom prompt
model_prompt_details = litellm.custom_prompt_dict[model]
prompt = custom_prompt(
- role_dict=model_prompt_details.get("roles", None),
+ role_dict=model_prompt_details.get("roles") or {},
initial_prompt_value=model_prompt_details.get(
"initial_prompt_value", ""
),
@@ -363,9 +363,9 @@ class HuggingFaceEmbeddingConfig(BaseConfig):
"content-type": "application/json",
}
if api_key is not None:
- default_headers[
- "Authorization"
- ] = f"Bearer {api_key}" # Huggingface Inference Endpoint default is to accept bearer tokens
+ default_headers["Authorization"] = (
+ f"Bearer {api_key}" # Huggingface Inference Endpoint default is to accept bearer tokens
+ )
headers = {**headers, **default_headers}
return headers
diff --git a/litellm/llms/huggingface/rerank/transformation.py b/litellm/llms/huggingface/rerank/transformation.py
index 3f5c44fec05..1454328cc13 100644
--- a/litellm/llms/huggingface/rerank/transformation.py
+++ b/litellm/llms/huggingface/rerank/transformation.py
@@ -1,10 +1,11 @@
import os
-import uuid
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, TypedDict, Union
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
import httpx
+from typing_extensions import TypedDict
import litellm
+from litellm._uuid import uuid
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig
from litellm.secret_managers.main import get_secret_str
@@ -94,7 +95,7 @@ class HuggingFaceRerankConfig(BaseRerankConfig):
return_documents: Optional[bool] = True,
max_chunks_per_doc: Optional[int] = None,
max_tokens_per_doc: Optional[int] = None,
- ) -> OptionalRerankParams:
+ ) -> Dict:
optional_rerank_params = {}
if non_default_params is not None:
for k, v in non_default_params.items():
diff --git a/litellm/llms/infinity/rerank/transformation.py b/litellm/llms/infinity/rerank/transformation.py
index 4b75fa121b2..55aac6033d5 100644
--- a/litellm/llms/infinity/rerank/transformation.py
+++ b/litellm/llms/infinity/rerank/transformation.py
@@ -4,7 +4,7 @@ Transformation logic from Cohere's /v1/rerank format to Infinity's `/v1/rerank`
Why separate file? Make it easy to see how transformation works
"""
-import uuid
+from litellm._uuid import uuid
from typing import List, Optional
import httpx
@@ -49,7 +49,7 @@ class InfinityRerankConfig(CohereRerankConfig):
)
default_headers = {
- "Authorization": f"bearer {api_key}",
+ "Authorization": f"Bearer {api_key}",
"accept": "application/json",
"content-type": "application/json",
}
diff --git a/litellm/llms/jina_ai/common_utils.py b/litellm/llms/jina_ai/common_utils.py
new file mode 100644
index 00000000000..cd9fd402afb
--- /dev/null
+++ b/litellm/llms/jina_ai/common_utils.py
@@ -0,0 +1,6 @@
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+
+
+class JinaAIError(BaseLLMException):
+ def __init__(self, status_code, message):
+ super().__init__(status_code=status_code, message=message)
diff --git a/litellm/llms/jina_ai/embedding/transformation.py b/litellm/llms/jina_ai/embedding/transformation.py
index 5263be900fa..7a634903005 100644
--- a/litellm/llms/jina_ai/embedding/transformation.py
+++ b/litellm/llms/jina_ai/embedding/transformation.py
@@ -1,5 +1,5 @@
"""
-Transformation logic from OpenAI /v1/embeddings format to Jina AI's `/v1/embeddings` format.
+Transformation logic from OpenAI /v1/embeddings format to Jina AI's `/v1/embeddings` format.
Why separate file? Make it easy to see how transformation works
@@ -7,13 +7,23 @@ Docs - https://jina.ai/embeddings/
"""
import types
-from typing import List, Optional, Tuple
+from typing import List, Optional, Tuple, Union, cast
+
+import httpx
from litellm import LlmProviders
from litellm.secret_managers.main import get_secret_str
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.base_llm import BaseEmbeddingConfig
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.types.llms.openai import AllEmbeddingInputValues, AllMessageValues
+from litellm.types.utils import EmbeddingResponse
+from litellm.utils import is_base64_encoded
+
+from ..common_utils import JinaAIError
-class JinaAIEmbeddingConfig:
+class JinaAIEmbeddingConfig(BaseEmbeddingConfig):
"""
Reference: https://jina.ai/embeddings/
"""
@@ -44,11 +54,15 @@ class JinaAIEmbeddingConfig:
and v is not None
}
- def get_supported_openai_params(self) -> List[str]:
+ def get_supported_openai_params(self, model: str) -> List[str]:
return ["dimensions"]
def map_openai_params(
- self, non_default_params: dict, optional_params: dict
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
) -> dict:
if "dimensions" in non_default_params:
optional_params["dimensions"] = non_default_params["dimensions"]
@@ -76,3 +90,88 @@ class JinaAIEmbeddingConfig:
or get_secret_str("JINA_AI_TOKEN")
)
return LlmProviders.JINA_AI.value, api_base, dynamic_api_key
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ return (
+ f"{api_base}/embeddings"
+ if api_base
+ else "https://api.jina.ai/v1/embeddings"
+ )
+
+ def transform_embedding_request(
+ self,
+ model: str,
+ input: AllEmbeddingInputValues,
+ optional_params: dict,
+ headers: dict,
+ ) -> dict:
+ data = {"model": model, **optional_params}
+ input = cast(List[str], input) if isinstance(input, List) else [input]
+ if any((is_base64_encoded(x) for x in input)):
+ transformed_input = []
+ for value in input:
+ if isinstance(value, str):
+ if is_base64_encoded(value):
+ img_data = value.split(",")[1]
+ transformed_input.append({"image": img_data})
+ else:
+ transformed_input.append({"text": value})
+ data["input"] = transformed_input
+ else:
+ data["input"] = input
+ return data
+
+ def transform_embedding_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: EmbeddingResponse,
+ logging_obj: LiteLLMLoggingObj,
+ api_key: Optional[str],
+ request_data: dict,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> EmbeddingResponse:
+ response_json = raw_response.json()
+ ## LOGGING
+ logging_obj.post_call(
+ input=input,
+ api_key=api_key,
+ additional_args={"complete_input_dict": request_data},
+ original_response=response_json,
+ )
+ return EmbeddingResponse(**response_json)
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ default_headers = {
+ "Content-Type": "application/json",
+ }
+ if api_key:
+ default_headers["Authorization"] = f"Bearer {api_key}"
+ headers = {**default_headers, **headers}
+ return headers
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ return JinaAIError(
+ status_code=status_code,
+ message=error_message,
+ )
diff --git a/litellm/llms/jina_ai/rerank/transformation.py b/litellm/llms/jina_ai/rerank/transformation.py
index 8d0a9b1431c..3ba24680fd4 100644
--- a/litellm/llms/jina_ai/rerank/transformation.py
+++ b/litellm/llms/jina_ai/rerank/transformation.py
@@ -6,11 +6,11 @@ Why separate file? Make it easy to see how transformation works
Docs - https://jina.ai/reranker
"""
-import uuid
from typing import Any, Dict, List, Optional, Tuple, Union
from httpx import URL, Response
+from litellm._uuid import uuid
from litellm.llms.base_llm.chat.transformation import LiteLLMLoggingObj
from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig
from litellm.types.rerank import (
@@ -45,15 +45,15 @@ class JinaAIRerankConfig(BaseRerankConfig):
return_documents: Optional[bool] = True,
max_chunks_per_doc: Optional[int] = None,
max_tokens_per_doc: Optional[int] = None,
- ) -> OptionalRerankParams:
+ ) -> Dict:
optional_params = {}
supported_params = self.get_supported_cohere_rerank_params(model)
for k, v in non_default_params.items():
if k in supported_params:
optional_params[k] = v
- return OptionalRerankParams(
+ return dict(OptionalRerankParams(
**optional_params,
- )
+ ))
def get_complete_url(self, api_base: Optional[str], model: str) -> str:
base_path = "/v1/rerank"
@@ -67,7 +67,7 @@ class JinaAIRerankConfig(BaseRerankConfig):
return cleaned_base
def transform_rerank_request(
- self, model: str, optional_rerank_params: OptionalRerankParams, headers: Dict
+ self, model: str, optional_rerank_params: Dict, headers: Dict
) -> Dict:
return {"model": model, **optional_rerank_params}
@@ -98,9 +98,26 @@ class JinaAIRerankConfig(BaseRerankConfig):
if _results is None:
raise ValueError(f"No results found in the response={_json_response}")
+ # Transform Jina AI's response format to match LiteLLM's expected format
+ # Jina AI returns: {"index": 0, "relevance_score": 0.72, "document": "hello"}
+ # LiteLLM expects: {"index": 0, "relevance_score": 0.72, "document": {"text": "hello"}}
+ transformed_results = []
+ for result in _results:
+ transformed_result = {
+ "index": result["index"],
+ "relevance_score": result["relevance_score"],
+ }
+ # Convert document from string to dict format if it exists
+ if "document" in result and isinstance(result["document"], str):
+ transformed_result["document"] = {"text": result["document"]}
+ elif "document" in result:
+ # If it's already a dict, keep it as is
+ transformed_result["document"] = result["document"]
+ transformed_results.append(transformed_result)
+
return RerankResponse(
id=_json_response.get("id") or str(uuid.uuid4()),
- results=_results, # type: ignore
+ results=transformed_results, # type: ignore
meta=rerank_meta,
) # Return response
diff --git a/litellm/llms/lemonade/chat/transformation.py b/litellm/llms/lemonade/chat/transformation.py
new file mode 100644
index 00000000000..8cba844435e
--- /dev/null
+++ b/litellm/llms/lemonade/chat/transformation.py
@@ -0,0 +1,149 @@
+"""
+Translate from OpenAI's `/v1/chat/completions` to Lemonade's `/v1/chat/completions`
+"""
+from typing import Any, List, Optional, Tuple, Union
+
+import httpx
+
+import litellm
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.openai import (
+ AllMessageValues,
+)
+from litellm.types.utils import ModelResponse
+
+from ...openai_like.chat.transformation import OpenAILikeChatConfig
+
+
+class LemonadeChatConfig(OpenAILikeChatConfig):
+ repeat_penalty: Optional[float] = None
+ functions: Optional[list] = None
+ logit_bias: Optional[dict] = None
+ max_tokens: Optional[int] = None
+ max_completion_tokens: Optional[int] = None
+ n: Optional[int] = None
+ presence_penalty: Optional[int] = None
+ stop: Optional[Union[str, list]] = None
+ temperature: Optional[int] = None
+ top_p: Optional[int] = None
+ top_k: Optional[int] = None
+ response_format: Optional[dict] = None
+ tools: Optional[list] = None
+
+ def __init__(
+ self,
+ repeat_penalty: Optional[float] = None,
+ functions: Optional[list] = None,
+ logit_bias: Optional[dict] = None,
+ max_completion_tokens: Optional[int] = None,
+ max_tokens: Optional[int] = None,
+ n: Optional[int] = None,
+ presence_penalty: Optional[int] = None,
+ stop: Optional[Union[str, list]] = None,
+ temperature: Optional[int] = None,
+ top_p: Optional[int] = None,
+ top_k: Optional[int] = None,
+ response_format: Optional[dict] = None,
+ tools: Optional[list] = None,
+ ) -> None:
+ locals_ = locals().copy()
+ for key, value in locals_.items():
+ if key != "self" and value is not None:
+ setattr(self.__class__, key, value)
+
+ @property
+ def custom_llm_provider(self) -> Optional[str]:
+ return "lemonade"
+
+ @classmethod
+ def get_config(cls):
+ return super().get_config()
+
+ def get_models(self, api_key: Optional[str] = None, api_base: Optional[str] = None):
+ """
+ Get available models from Lemonade API.
+
+ This method queries the Lemonade /models endpoint to retrieve the list of available models.
+
+ Args:
+ api_key: Optional API key (Lemonade doesn't require authentication)
+ api_base: Optional API base URL (defaults to LEMONADE_API_BASE env var or http://localhost:8000)
+
+ Returns:
+ List of model names prefixed with "lemonade/"
+ """
+ api_base, api_key = self._get_openai_compatible_provider_info(
+ api_base=api_base, api_key=api_key
+ )
+
+ if api_base is None:
+ raise ValueError(
+ "LEMONADE_API_BASE is not set. Please set the environment variable to query Lemonade's /models endpoint."
+ )
+
+ # Getting the list of models from lemonade
+ try:
+ response = litellm.module_level_client.get(
+ url=f"{api_base}/models",
+ )
+ except Exception as e:
+ raise ValueError(
+ f"Failed to fetch models from Lemonade. Set Lemonade API Base via `LEMONADE_API_BASE` environment variable. Error: {e}"
+ )
+
+ if response.status_code != 200:
+ raise ValueError(
+ f"Failed to fetch models from Lemonade. Status code: {response.status_code}, Response: {response.text}"
+ )
+
+ model_list = response.json().get("data", [])
+ return ["lemonade/" + model["id"] for model in model_list]
+
+ def _get_openai_compatible_provider_info(
+ self, api_base: Optional[str], api_key: Optional[str]
+ ) -> Tuple[Optional[str], Optional[str]]:
+ # lemonade is openai compatible, we just need to set this to custom_openai and have the api_base be lemonade's endpoint
+ api_base = (
+ api_base
+ or get_secret_str("LEMONADE_API_BASE")
+ or "http://localhost:8000/api/v1"
+ ) # type: ignore
+ # Lemonade doesn't check the key
+ key = "lemonade"
+ return api_base, key
+
+
+ def transform_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: ModelResponse,
+ logging_obj: LiteLLMLoggingObj,
+ request_data: dict,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ encoding: Any,
+ api_key: Optional[str] = None,
+ json_mode: Optional[bool] = None,
+ ) -> ModelResponse:
+ model_response = super().transform_response(
+ model=model,
+ model_response=model_response,
+ raw_response=raw_response,
+ messages=messages,
+ logging_obj=logging_obj,
+ request_data=request_data,
+ encoding=encoding,
+ optional_params=optional_params,
+ json_mode=json_mode,
+ litellm_params=litellm_params,
+ api_key=api_key,
+ )
+
+ # Storing lemonade in the model response for easier cost calculation later
+ setattr(model_response, "model", "lemonade/" + model)
+
+ return model_response
+
\ No newline at end of file
diff --git a/litellm/llms/lemonade/cost_calculator.py b/litellm/llms/lemonade/cost_calculator.py
new file mode 100644
index 00000000000..27e1ca275f8
--- /dev/null
+++ b/litellm/llms/lemonade/cost_calculator.py
@@ -0,0 +1,35 @@
+"""
+Cost calculation for Lemonade LLM provider.
+
+Since Lemonade is a local/self-hosted service, all costs default to 0.
+This prevents cost calculation errors when using models not in model_prices_and_context_window.json
+"""
+from typing import Tuple
+
+from litellm.types.utils import Usage
+
+
+def cost_per_token(
+ model: str,
+ usage: Usage,
+) -> Tuple[float, float]:
+ """
+ Calculate cost per token for Lemonade models.
+
+ Since Lemonade is a local/self-hosted deployment, there are no per-token costs.
+ This function returns (0.0, 0.0) for all models to allow cost tracking to work
+ without errors for any Lemonade model, regardless of whether it's in the
+ model_prices_and_context_window.json file.
+
+ Args:
+ model: The model name (with or without "lemonade/" prefix)
+ usage: Usage object containing token counts
+
+ Returns:
+ Tuple of (prompt_cost, completion_cost) - always (0.0, 0.0) for Lemonade
+ """
+ # Lemonade is self-hosted/local, so cost is always 0
+ prompt_cost = 0.0
+ completion_cost = 0.0
+
+ return prompt_cost, completion_cost
diff --git a/litellm/llms/litellm_proxy/chat/transformation.py b/litellm/llms/litellm_proxy/chat/transformation.py
index ea89c4c3bc7..cf6a6ed7a54 100644
--- a/litellm/llms/litellm_proxy/chat/transformation.py
+++ b/litellm/llms/litellm_proxy/chat/transformation.py
@@ -4,6 +4,7 @@ Translate from OpenAI's `/v1/chat/completions` to VLLM's `/v1/chat/completions`
from typing import TYPE_CHECKING, List, Optional, Tuple
+from litellm.constants import OPENAI_CHAT_COMPLETION_PARAMS
from litellm.secret_managers.main import get_secret_bool, get_secret_str
from litellm.types.router import LiteLLM_Params
@@ -16,8 +17,7 @@ if TYPE_CHECKING:
class LiteLLMProxyChatConfig(OpenAIGPTConfig):
def get_supported_openai_params(self, model: str) -> List:
params_list = super().get_supported_openai_params(model)
- params_list.append("thinking")
- params_list.append("reasoning_effort")
+ params_list.extend(OPENAI_CHAT_COMPLETION_PARAMS)
return params_list
def _map_openai_params(
diff --git a/litellm/llms/litellm_proxy/image_edit/transformation.py b/litellm/llms/litellm_proxy/image_edit/transformation.py
new file mode 100644
index 00000000000..5f5e2bdb24d
--- /dev/null
+++ b/litellm/llms/litellm_proxy/image_edit/transformation.py
@@ -0,0 +1,26 @@
+from typing import Optional
+
+from litellm.llms.openai.image_edit.transformation import OpenAIImageEditConfig
+from litellm.secret_managers.main import get_secret_str
+
+
+class LiteLLMProxyImageEditConfig(OpenAIImageEditConfig):
+ """Configuration for image edit requests routed through LiteLLM Proxy."""
+
+ def validate_environment(
+ self, headers: dict, model: str, api_key: Optional[str] = None
+ ) -> dict:
+ api_key = api_key or get_secret_str("LITELLM_PROXY_API_KEY")
+ headers.update({"Authorization": f"Bearer {api_key}"})
+ return headers
+
+ def get_complete_url(
+ self, model: str, api_base: Optional[str], litellm_params: dict
+ ) -> str:
+ api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
+ if api_base is None:
+ raise ValueError(
+ "api_base not set for LiteLLM Proxy route. Set in env via `LITELLM_PROXY_API_BASE`"
+ )
+ api_base = api_base.rstrip("/")
+ return f"{api_base}/images/edits"
diff --git a/litellm/llms/litellm_proxy/image_generation/transformation.py b/litellm/llms/litellm_proxy/image_generation/transformation.py
new file mode 100644
index 00000000000..6174424154d
--- /dev/null
+++ b/litellm/llms/litellm_proxy/image_generation/transformation.py
@@ -0,0 +1,40 @@
+from typing import Optional
+
+from litellm.llms.openai.image_generation.gpt_transformation import (
+ GPTImageGenerationConfig,
+)
+from litellm.secret_managers.main import get_secret_str
+
+
+class LiteLLMProxyImageGenerationConfig(GPTImageGenerationConfig):
+ """Configuration for image generation requests routed through LiteLLM Proxy."""
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages,
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ api_key = api_key or get_secret_str("LITELLM_PROXY_API_KEY")
+ headers.update({"Authorization": f"Bearer {api_key}"})
+ return headers
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
+ if api_base is None:
+ raise ValueError(
+ "api_base not set for LiteLLM Proxy route. Set in env via `LITELLM_PROXY_API_BASE`"
+ )
+ api_base = api_base.rstrip("/")
+ return f"{api_base}/images/generations"
diff --git a/litellm/llms/litellm_proxy/responses/transformation.py b/litellm/llms/litellm_proxy/responses/transformation.py
new file mode 100644
index 00000000000..0b81d8be7d8
--- /dev/null
+++ b/litellm/llms/litellm_proxy/responses/transformation.py
@@ -0,0 +1,48 @@
+"""
+Responses API transformation for LiteLLM Proxy provider.
+
+LiteLLM Proxy supports the OpenAI Responses API natively when the underlying model supports it.
+This config enables pass-through behavior to the proxy's /v1/responses endpoint.
+"""
+
+from typing import Optional
+
+from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.utils import LlmProviders
+
+
+class LiteLLMProxyResponsesAPIConfig(OpenAIResponsesAPIConfig):
+ """
+ Configuration for LiteLLM Proxy Responses API support.
+
+ Extends OpenAI's config since the proxy follows OpenAI's API spec,
+ but uses LITELLM_PROXY_API_BASE for the base URL.
+ """
+
+ @property
+ def custom_llm_provider(self) -> LlmProviders:
+ return LlmProviders.LITELLM_PROXY
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ litellm_params: dict,
+ ) -> str:
+ """
+ Get the endpoint for LiteLLM Proxy responses API.
+
+ Uses LITELLM_PROXY_API_BASE environment variable if api_base is not provided.
+ """
+ api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
+
+ if api_base is None:
+ raise ValueError(
+ "api_base not set for LiteLLM Proxy responses API. "
+ "Set via api_base parameter or LITELLM_PROXY_API_BASE environment variable"
+ )
+
+ # Remove trailing slashes
+ api_base = api_base.rstrip("/")
+
+ return f"{api_base}/responses"
diff --git a/litellm/llms/lm_studio/chat/transformation.py b/litellm/llms/lm_studio/chat/transformation.py
index f7a2cc0f28a..7b188ff33f8 100644
--- a/litellm/llms/lm_studio/chat/transformation.py
+++ b/litellm/llms/lm_studio/chat/transformation.py
@@ -15,8 +15,8 @@ class LMStudioChatConfig(OpenAIGPTConfig):
) -> Tuple[Optional[str], Optional[str]]:
api_base = api_base or get_secret_str("LM_STUDIO_API_BASE") # type: ignore
dynamic_api_key = (
- api_key or get_secret_str("LM_STUDIO_API_KEY") or " "
- ) # vllm does not require an api key
+ api_key or get_secret_str("LM_STUDIO_API_KEY") or "fake-api-key"
+ ) # LM Studio does not require an api key, but OpenAI client requires non-None value
return api_base, dynamic_api_key
def map_openai_params(
diff --git a/litellm/llms/mistral/chat/transformation.py b/litellm/llms/mistral/chat/transformation.py
index 0441e75beec..51fa65244a0 100644
--- a/litellm/llms/mistral/chat/transformation.py
+++ b/litellm/llms/mistral/chat/transformation.py
@@ -6,9 +6,21 @@ Why separate file? Make it easy to see how transformation works
Docs - https://docs.mistral.ai/api/
"""
-from typing import Any, Coroutine, List, Literal, Optional, Tuple, Union, cast, overload
+from typing import (
+ Any,
+ Coroutine,
+ List,
+ Literal,
+ Optional,
+ Tuple,
+ Union,
+ cast,
+ get_type_hints,
+ overload,
+)
import httpx
+
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.prompt_templates.common_utils import (
handle_messages_with_content_list_to_str_conversion,
@@ -16,7 +28,7 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
)
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.secret_managers.main import get_secret_str
-from litellm.types.llms.mistral import MistralToolCallMessage
+from litellm.types.llms.mistral import MistralThinkingBlock, MistralToolCallMessage
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
from litellm.utils import convert_to_model_response_object
@@ -144,10 +156,13 @@ class MistralConfig(OpenAIGPTConfig):
for param, value in non_default_params.items():
if param == "max_tokens":
optional_params["max_tokens"] = value
- if param == "max_completion_tokens": # max_completion_tokens should take priority
+ if (
+ param == "max_completion_tokens"
+ ): # max_completion_tokens should take priority
optional_params["max_tokens"] = value
if param == "tools":
- optional_params["tools"] = value
+ # Clean tools to remove problematic schema fields for Mistral API
+ optional_params["tools"] = self._clean_tool_schema_for_mistral(value)
if param == "stream" and value is True:
optional_params["stream"] = value
if param == "temperature":
@@ -157,7 +172,9 @@ class MistralConfig(OpenAIGPTConfig):
if param == "stop":
optional_params["stop"] = value
if param == "tool_choice" and isinstance(value, str):
- optional_params["tool_choice"] = self._map_tool_choice(tool_choice=value)
+ optional_params["tool_choice"] = self._map_tool_choice(
+ tool_choice=value
+ )
if param == "seed":
optional_params["extra_body"] = {"random_seed": value}
if param == "response_format":
@@ -183,7 +200,9 @@ class MistralConfig(OpenAIGPTConfig):
) # type: ignore
# if api_base does not end with /v1 we add it
- if api_base is not None and not api_base.endswith("/v1"): # Mistral always needs a /v1 at the end
+ if api_base is not None and not api_base.endswith(
+ "/v1"
+ ): # Mistral always needs a /v1 at the end
api_base = api_base + "/v1"
dynamic_api_key = (
api_key
@@ -192,10 +211,13 @@ class MistralConfig(OpenAIGPTConfig):
)
return api_base, dynamic_api_key
+ # fmt: off
+
@overload
def _transform_messages(
self, messages: List[AllMessageValues], model: str, is_async: Literal[True]
- ) -> Coroutine[Any, Any, List[AllMessageValues]]: ...
+ ) -> Coroutine[Any, Any, List[AllMessageValues]]:
+ ...
@overload
def _transform_messages(
@@ -203,7 +225,9 @@ class MistralConfig(OpenAIGPTConfig):
messages: List[AllMessageValues],
model: str,
is_async: Literal[False] = False,
- ) -> List[AllMessageValues]: ...
+ ) -> List[AllMessageValues]:
+ ...
+ # fmt: on
def _transform_messages(
self, messages: List[AllMessageValues], model: str, is_async: bool = False
@@ -214,18 +238,20 @@ class MistralConfig(OpenAIGPTConfig):
- if image passed in, then just return as is (user-intended)
- if `name` is passed, then drop it for mistral API: https://github.com/BerriAI/litellm/issues/6696
- Motivation: mistral api doesn't support content as a list
+ Motivation: mistral api doesn't support content as a list.
+ The above statement is not valid now. Need to plan to remove all the #1,2,3
+ Mistral API supports content as a list.
"""
- ## 1. If 'image_url' in content, then return as is
+ ## 1. If 'image_url' or 'file' in content, then transform with base class and mistral-specific handling
for m in messages:
_content_block = m.get("content")
if _content_block and isinstance(_content_block, list):
- for c in _content_block:
- if c.get("type") == "image_url":
- if is_async:
- return super()._transform_messages(messages, model, True)
- else:
- return super()._transform_messages(messages, model, False)
+ if any(c.get("type") in ["image_url", "file"] for c in _content_block):
+ if is_async:
+ return self._transform_messages_async(messages, model)
+ else:
+ messages = self._transform_messages_sync(messages, model)
+ return messages
## 2. If content is list, then convert to string
messages = handle_messages_with_content_list_to_str_conversion(messages)
@@ -235,6 +261,8 @@ class MistralConfig(OpenAIGPTConfig):
for m in messages:
m = MistralConfig._handle_name_in_message(m)
m = MistralConfig._handle_tool_call_message(m)
+ if MistralConfig._is_empty_assistant_message(m):
+ continue
m = strip_none_values_from_message(m) # prevents 'extra_forbidden' error
new_messages.append(m)
@@ -243,6 +271,51 @@ class MistralConfig(OpenAIGPTConfig):
else:
return super()._transform_messages(new_messages, model, False)
+ async def _transform_messages_async(self,
+ messages: List[AllMessageValues], model: str
+ ) -> List[AllMessageValues]:
+ """
+ Handle modification of messages for Mistral API in an async context.
+ """
+ # Call parent async method to handle basic transformations
+ # and then apply Mistral-specific handling for files
+ messages = await super()._transform_messages(messages, model, True)
+ messages = self._handle_message_with_file(messages)
+ return messages
+
+ def _transform_messages_sync(self,
+ messages: List[AllMessageValues], model: str
+ ) -> List[AllMessageValues]:
+ """ Handle modification of messages for Mistral API in a sync context.
+ """
+ # Call parent sync method to handle basic transformations
+ # and then apply Mistral-specific handling for files
+ # This is the sync version of the async method above
+ messages = super()._transform_messages(messages, model, False)
+ messages = self._handle_message_with_file(messages)
+ return messages
+
+ def _handle_message_with_file(
+ self,
+ messages: List[AllMessageValues]) -> List[AllMessageValues]:
+ """
+ Mistral API supports only 'file_id' in message content with type 'file'.
+ """
+ for m in messages:
+ _content_block = m.get("content")
+ if _content_block and isinstance(_content_block, list):
+ if any(c.get("type") == "file" for c in _content_block):
+ # If file content is present, we get file_id from 'file' attribute of content block
+ # then replace 'file' with 'file_id' and assign the value of 'file_id' attribute to it.
+ file_contents = [c for c in _content_block if c.get("type") == "file"]
+ for file_content in file_contents:
+ file_id = file_content.get("file", {}).get("file_id")
+ if file_id:
+ # Replace 'file' with 'file_id'
+ file_content["file_id"] = file_id # type: ignore
+ file_content.pop("file", None)
+ return messages
+
def _add_reasoning_system_prompt_if_needed(
self, messages: List[AllMessageValues], optional_params: dict
) -> List[AllMessageValues]:
@@ -265,20 +338,30 @@ class MistralConfig(OpenAIGPTConfig):
# Handle both string and list content, preserving original format
if isinstance(existing_content, str):
# String content - prepend reasoning prompt
- new_content: Union[str, list] = f"{reasoning_prompt}\n\n{existing_content}"
+ new_content: Union[str, list] = (
+ f"{reasoning_prompt}\n\n{existing_content}"
+ )
elif isinstance(existing_content, list):
# List content - prepend reasoning prompt as text block
- new_content = [{"type": "text", "text": reasoning_prompt + "\n\n"}] + existing_content
+ new_content = [
+ {"type": "text", "text": reasoning_prompt + "\n\n"}
+ ] + existing_content
else:
# Fallback for any other type - convert to string
new_content = f"{reasoning_prompt}\n\n{str(existing_content)}"
- messages[i] = cast(AllMessageValues, {**msg, "content": new_content})
+ messages[i] = cast(
+ AllMessageValues, {**msg, "content": new_content}
+ )
break
else:
# Add new system message with reasoning instructions
reasoning_message: AllMessageValues = cast(
- AllMessageValues, {"role": "system", "content": self._get_mistral_reasoning_system_prompt()}
+ AllMessageValues,
+ {
+ "role": "system",
+ "content": self._get_mistral_reasoning_system_prompt(),
+ },
)
messages = [reasoning_message] + messages
@@ -286,6 +369,40 @@ class MistralConfig(OpenAIGPTConfig):
optional_params.pop("_add_reasoning_prompt", None)
return messages
+ @classmethod
+ def _clean_tool_schema_for_mistral(cls, tools: list) -> list:
+ """
+ Clean tool schemas to remove fields that cause issues with Mistral API.
+
+ Removes:
+ - $id and $schema fields (cause grammar validation errors)
+ - additionalProperties=False (causes OpenAI API schema errors)
+ - strict field (not supported by Mistral)
+
+ Args:
+ tools: List of tool definitions
+ max_depth: Maximum recursion depth for schema cleaning (default: 10)
+
+ Returns:
+ Cleaned tools list
+ """
+ if not tools:
+ return tools
+
+ import copy
+
+ from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH
+ from litellm.utils import _remove_json_schema_refs
+
+ cleaned_tools = copy.deepcopy(tools)
+
+ # Apply all cleaning functions with max_depth protection
+ cleaned_tools = _remove_json_schema_refs(
+ cleaned_tools, max_depth=DEFAULT_MAX_RECURSE_DEPTH
+ )
+
+ return cleaned_tools
+
@classmethod
def _handle_name_in_message(cls, message: AllMessageValues) -> AllMessageValues:
"""
@@ -324,6 +441,25 @@ class MistralConfig(OpenAIGPTConfig):
message["tool_calls"] = mistral_tool_calls # type: ignore
return message
+ @classmethod
+ def _is_empty_assistant_message(cls, message: AllMessageValues) -> bool:
+ """
+ Mistral API does not support empty string in assistant content.
+ """
+ from litellm.types.llms.openai import ChatCompletionAssistantMessage
+
+ set_keys = get_type_hints(ChatCompletionAssistantMessage).keys()
+
+ all_expected_values_are_empty = True
+ for key in set_keys:
+ if key != "role" and message.get(key) is not None:
+ if key == "content" and message.get(key) == "":
+ continue
+ else:
+ all_expected_values_are_empty = False
+ break
+ return all_expected_values_are_empty
+
@staticmethod
def _handle_empty_content_response(response_data: dict) -> dict:
"""
@@ -344,6 +480,58 @@ class MistralConfig(OpenAIGPTConfig):
choice["message"]["content"] = None
return response_data
+ @staticmethod
+ def _convert_thinking_block_to_reasoning_content(
+ thinking_blocks: MistralThinkingBlock,
+ ) -> str:
+ """
+ Convert Mistral thinking blocks to reasoning content.
+ """
+ return "\n".join(
+ [block.get("text", "") for block in thinking_blocks["thinking"]]
+ )
+
+ @staticmethod
+ def _handle_content_list_to_str_conversion(response_data: dict) -> dict:
+ """
+ Handle Mistral's content list format and extract thinking content.
+
+ Map mistral's content list to string and extract thinking blocks:
+ - Thinking block -> reasoning_content field
+ - Text block -> content field
+ """
+
+ if response_data.get("choices") and len(response_data["choices"]) > 0:
+ for choice in response_data["choices"]:
+ if choice.get("message") and choice["message"].get("content"):
+ content = choice["message"]["content"]
+
+ # Only process if content is a list
+ if isinstance(content, list):
+ thinking_content = ""
+ text_content = ""
+
+ # Process each content block
+ for block in content:
+ if block.get("type") == "thinking":
+ thinking_blocks = block.get("thinking", [])
+ thinking_texts = []
+ for thinking_block in thinking_blocks:
+ if thinking_block.get("type") == "text":
+ thinking_texts.append(
+ thinking_block.get("text", "")
+ )
+ thinking_content = "\n".join(thinking_texts)
+ elif block.get("type") == "text":
+ text_content = block.get("text", "")
+
+ # Set the extracted content
+ choice["message"]["content"] = text_content
+ if thinking_content:
+ choice["message"]["reasoning_content"] = thinking_content
+
+ return response_data
+
def transform_request(
self,
model: str,
@@ -360,8 +548,12 @@ class MistralConfig(OpenAIGPTConfig):
dict: The transformed request. Sent as the body of the API call.
"""
# Add reasoning system prompt if needed (for magistral models)
- if "magistral" in model.lower() and optional_params.get("_add_reasoning_prompt", False):
- messages = self._add_reasoning_system_prompt_if_needed(messages, optional_params)
+ if "magistral" in model.lower() and optional_params.get(
+ "_add_reasoning_prompt", False
+ ):
+ messages = self._add_reasoning_system_prompt_if_needed(
+ messages, optional_params
+ )
# Call parent transform_request which handles _transform_messages
return super().transform_request(
@@ -388,14 +580,16 @@ class MistralConfig(OpenAIGPTConfig):
) -> ModelResponse:
"""
Transform the raw response from Mistral API.
- Handles Mistral-specific behavior like converting empty string content to None.
+ Handles Mistral-specific behavior like converting empty string content to None
+ and extracting thinking content from content lists.
"""
logging_obj.post_call(original_response=raw_response.text)
logging_obj.model_call_details["response_headers"] = raw_response.headers
- # Handle Mistral-specific empty string content conversion to None
+ # Handle Mistral-specific response transformations
response_data = raw_response.json()
response_data = self._handle_empty_content_response(response_data)
+ response_data = self._handle_content_list_to_str_conversion(response_data)
final_response_obj = cast(
ModelResponse,
diff --git a/litellm/llms/nvidia_nim/rerank/transformation.py b/litellm/llms/nvidia_nim/rerank/transformation.py
new file mode 100644
index 00000000000..cb9fd4bebaa
--- /dev/null
+++ b/litellm/llms/nvidia_nim/rerank/transformation.py
@@ -0,0 +1,325 @@
+from typing import Any, Dict, List, Literal, Optional, Union
+
+import httpx
+from typing_extensions import Required, TypedDict
+
+import litellm
+from litellm._uuid import uuid
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.rerank import (
+ RerankBilledUnits,
+ RerankResponse,
+ RerankResponseMeta,
+ RerankResponseResult,
+)
+
+
+class NvidiaNimQueryObject(TypedDict):
+ text: Required[str]
+
+
+class NvidiaNimPassageObject(TypedDict):
+ text: Required[str]
+
+
+class NvidiaNimRerankRequest(TypedDict, total=False):
+ model: Required[str]
+ query: Required[NvidiaNimQueryObject]
+ passages: Required[List[NvidiaNimPassageObject]]
+ truncate: Literal["NONE", "END"]
+ top_k: int
+
+
+class NvidiaNimRankingResult(TypedDict):
+ index: Required[int]
+ logit: Required[float]
+
+
+class NvidiaNimRerankResponse(TypedDict):
+ rankings: Required[List[NvidiaNimRankingResult]]
+
+
+class NvidiaNimRerankConfig(BaseRerankConfig):
+ """
+ Reference: https://docs.api.nvidia.com/nim/reference/nvidia-llama-3_2-nv-rerankqa-1b-v2-infer
+
+ Nvidia NIM rerank API uses a different format:
+ - query is an object with 'text' field
+ - documents are called 'passages' and have 'text' field
+ """
+ DEFAULT_NIM_RERANK_API_BASE = "https://ai.api.nvidia.com"
+
+ def __init__(self) -> None:
+ pass
+
+ def get_complete_url(self, api_base: Optional[str], model: str) -> str:
+ """
+ Construct the Nvidia NIM rerank URL.
+
+ Format: {api_base}/v1/retrieval/{model}/reranking
+
+ If the user provides a full URL (e.g., {api_base}/v1/retrieval/{model}/reranking),
+ it will be used as-is.
+ """
+ if not api_base:
+ api_base = self.DEFAULT_NIM_RERANK_API_BASE
+
+ api_base = api_base.rstrip("/")
+
+ # Check if user already provided the full URL with /retrieval/ path
+ if "/retrieval/" in api_base:
+ return api_base
+
+ # Ensure we don't have duplicate /v1
+ if api_base.endswith("/v1"):
+ api_base = api_base[:-3]
+
+ return f"{api_base}/v1/retrieval/{model}/reranking"
+
+ def get_supported_cohere_rerank_params(self, model: str) -> list:
+ """
+ Nvidia NIM supports these rerank parameters.
+ """
+ return [
+ "query",
+ "documents",
+ "top_n",
+ ]
+
+ def map_cohere_rerank_params(
+ self,
+ non_default_params: Optional[dict],
+ model: str,
+ drop_params: bool,
+ query: str,
+ documents: List[Union[str, Dict[str, Any]]],
+ custom_llm_provider: Optional[str] = None,
+ top_n: Optional[int] = None,
+ rank_fields: Optional[List[str]] = None,
+ return_documents: Optional[bool] = True,
+ max_chunks_per_doc: Optional[int] = None,
+ max_tokens_per_doc: Optional[int] = None,
+ ) -> Dict:
+ """
+ Map Cohere/OpenAI rerank params to Nvidia NIM format.
+
+ Parameter mapping:
+ - top_n (Cohere) -> top_k (Nvidia)
+
+ Nvidia NIM specific params (passed through as-is from non_default_params):
+ - truncate: How to truncate input if too long (NONE, END)
+ """
+ optional_nvidia_nim_rerank_params: Dict[str, Any] = {
+ "query": query,
+ "documents": documents,
+ }
+
+ # Map Cohere's top_n to Nvidia's top_k
+ if top_n is not None:
+ optional_nvidia_nim_rerank_params["top_k"] = top_n
+
+ # Pass through Nvidia-specific params from non_default_params
+ if non_default_params:
+ optional_nvidia_nim_rerank_params.update(non_default_params)
+ return dict(optional_nvidia_nim_rerank_params)
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ api_key: Optional[str] = None,
+ ) -> dict:
+ """
+ Validate that the Nvidia NIM API key is present.
+ """
+ if api_key is None:
+ api_key = (
+ get_secret_str("NVIDIA_NIM_API_KEY")
+ or litellm.api_key
+ )
+
+ if api_key is None:
+ raise ValueError(
+ "Nvidia NIM API key is required. Please set 'NVIDIA_NIM_API_KEY' in your environment"
+ )
+
+ default_headers = {
+ "Authorization": f"Bearer {api_key}",
+ "accept": "application/json",
+ "content-type": "application/json",
+ }
+
+ # If 'Authorization' is provided in headers, it overrides the default
+ if "Authorization" in headers:
+ default_headers["Authorization"] = headers["Authorization"]
+
+ # Merge other headers, overriding any default ones except Authorization
+ return {**default_headers, **headers}
+
+ def transform_rerank_request(
+ self,
+ model: str,
+ optional_rerank_params: Dict,
+ headers: dict,
+ ) -> dict:
+ """
+ Transform request to Nvidia NIM format.
+
+ Nvidia NIM expects:
+ - query as {text: "..."}
+ - documents as passages: [{text: "..."}, ...]
+ - Optional: truncate (NONE or END), top_k
+
+ Note: optional_rerank_params may contain provider-specific params like 'top_k' and 'truncate'
+ that aren't in the OptionalRerankParams TypedDict but are passed through at runtime.
+ The mapping from Cohere's 'top_n' to Nvidia's 'top_k' already happened in map_cohere_rerank_params.
+ """
+ if "query" not in optional_rerank_params:
+ raise ValueError("query is required for Nvidia NIM rerank")
+ if "documents" not in optional_rerank_params:
+ raise ValueError("documents is required for Nvidia NIM rerank")
+
+ query = optional_rerank_params["query"]
+ documents = optional_rerank_params["documents"]
+
+ # Transform query to object format
+ query_obj: NvidiaNimQueryObject = {"text": query}
+
+ # Transform documents to passages format
+ passages: List[NvidiaNimPassageObject] = []
+ for doc in documents:
+ if isinstance(doc, str):
+ passages.append({"text": doc})
+ elif isinstance(doc, dict):
+ # If document is already a dict, check if it has 'text' field
+ if "text" in doc:
+ passages.append({"text": doc["text"]})
+ else:
+ # Otherwise, stringify the dict
+ import json
+ passages.append({"text": json.dumps(doc)})
+ else:
+ passages.append({"text": str(doc)})
+
+ # Note: URL path uses underscores (llama-3_2) but JSON body uses periods (llama-3.2)
+ # Convert underscores back to periods for the model field in request body
+ model_for_body = model.replace("_", ".")
+
+ # Build request using TypedDict
+ request_data: NvidiaNimRerankRequest = {
+ "model": model_for_body,
+ "query": query_obj,
+ "passages": passages,
+ }
+
+ # Add optional top_k parameter if provided (already mapped from top_n in map_cohere_rerank_params)
+ if "top_k" in optional_rerank_params and optional_rerank_params.get("top_k") is not None: # type: ignore
+ request_data["top_k"] = optional_rerank_params.get("top_k") # type: ignore
+
+ # Add Nvidia-specific truncate parameter if provided
+ # This is passed through from non_default_params, not in base OptionalRerankParams
+ if "truncate" in optional_rerank_params and optional_rerank_params.get("truncate") is not None: # type: ignore
+ truncate_value = optional_rerank_params.get("truncate") # type: ignore
+ if truncate_value in ["NONE", "END"]:
+ request_data["truncate"] = truncate_value # type: ignore
+
+ return dict(request_data)
+
+ def transform_rerank_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: RerankResponse,
+ logging_obj: LiteLLMLoggingObj,
+ api_key: Optional[str] = None,
+ request_data: dict = {},
+ optional_params: dict = {},
+ litellm_params: dict = {},
+ ) -> RerankResponse:
+ """
+ Transform Nvidia NIM rerank response to LiteLLM format.
+
+ Nvidia NIM returns (NvidiaNimRerankResponse):
+ {
+ "rankings": [
+ {
+ "index": 0,
+ "logit": 0.123
+ }
+ ]
+ }
+
+ LiteLLM expects (RerankResponse):
+ {
+ "results": [
+ {
+ "index": 0,
+ "relevance_score": 0.123,
+ "document": {"text": "..."} # optional
+ }
+ ]
+ }
+ """
+ try:
+ raw_response_json = raw_response.json()
+ except Exception:
+ raise BaseLLMException(
+ status_code=raw_response.status_code,
+ message=raw_response.text,
+ headers=raw_response.headers,
+ )
+
+ # Parse as NvidiaNimRerankResponse
+ nvidia_response: NvidiaNimRerankResponse = raw_response_json
+
+ # Transform Nvidia NIM response to LiteLLM format
+ results: List[RerankResponseResult] = []
+ rankings = nvidia_response.get("rankings", [])
+
+ # Get original documents from request if we need to include them
+ original_passages: List[NvidiaNimPassageObject] = request_data.get("passages", [])
+
+ for ranking in rankings:
+ result_item: RerankResponseResult = {
+ "index": ranking["index"],
+ "relevance_score": ranking["logit"],
+ }
+
+ # Include document if it was in the original request
+ index: int = ranking["index"]
+ if index < len(original_passages):
+ result_item["document"] = {"text": original_passages[index]["text"]} # type: ignore
+
+ results.append(result_item)
+
+ # Construct metadata with billed_units
+ # Nvidia NIM uses "usage" field with "total_tokens"
+ usage = raw_response_json.get("usage", {})
+ total_tokens = usage.get("total_tokens", 0)
+
+ billed_units: RerankBilledUnits = {
+ "total_tokens": total_tokens if total_tokens > 0 else len(results)
+ }
+
+ meta: RerankResponseMeta = {
+ "billed_units": billed_units
+ }
+
+ return RerankResponse(
+ id=raw_response_json.get("id") or str(uuid.uuid4()),
+ results=results,
+ meta=meta,
+ )
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ return BaseLLMException(
+ status_code=status_code,
+ message=error_message,
+ headers=headers,
+ )
+
diff --git a/litellm/llms/oci/chat/transformation.py b/litellm/llms/oci/chat/transformation.py
new file mode 100644
index 00000000000..3ab827797c5
--- /dev/null
+++ b/litellm/llms/oci/chat/transformation.py
@@ -0,0 +1,1179 @@
+import base64
+import datetime
+import hashlib
+import json
+from typing import TYPE_CHECKING, Any, AsyncIterator, Dict, List, Optional, Tuple, Union
+from urllib.parse import urlparse
+
+import httpx
+
+import litellm
+from litellm.litellm_core_utils.logging_utils import track_llm_api_timing
+from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
+from litellm.llms.custom_httpx.http_handler import (
+ AsyncHTTPHandler,
+ HTTPHandler,
+ _get_httpx_client,
+ get_async_httpx_client,
+ version,
+)
+from litellm.llms.oci.common_utils import OCIError
+from litellm.types.llms.oci import (
+ CohereChatRequest,
+ CohereMessage,
+ CohereChatResult,
+ CohereParameterDefinition,
+ CohereStreamChunk,
+ CohereTool,
+ CohereToolCall,
+ OCIChatRequestPayload,
+ OCICompletionPayload,
+ OCICompletionResponse,
+ OCIContentPartUnion,
+ OCIImageContentPart,
+ OCIMessage,
+ OCIRoles,
+ OCIServingMode,
+ OCIStreamChunk,
+ OCITextContentPart,
+ OCIToolCall,
+ OCIToolDefinition,
+ OCIVendors,
+)
+from litellm.types.llms.openai import AllMessageValues
+from litellm.types.utils import (
+ Delta,
+ LlmProviders,
+ ModelResponse,
+ ModelResponseStream,
+ StreamingChoices,
+)
+from litellm.utils import (
+ ChatCompletionMessageToolCall,
+ CustomStreamWrapper,
+ Usage,
+)
+
+if TYPE_CHECKING:
+ from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
+
+ LiteLLMLoggingObj = _LiteLLMLoggingObj
+else:
+ LiteLLMLoggingObj = Any
+
+
+def sha256_base64(data: bytes) -> str:
+ digest = hashlib.sha256(data).digest()
+ return base64.b64encode(digest).decode()
+
+
+def build_signature_string(method, path, headers, signed_headers):
+ lines = []
+ for header in signed_headers:
+ if header == "(request-target)":
+ value = f"{method.lower()} {path}"
+ else:
+ value = headers[header]
+ lines.append(f"{header}: {value}")
+ return "\n".join(lines)
+
+
+def load_private_key_from_str(key_str: str):
+ try:
+ from cryptography.hazmat.primitives import serialization
+ from cryptography.hazmat.primitives.asymmetric import rsa
+ except ImportError as e:
+ raise ImportError(
+ "cryptography package is required for OCI authentication. "
+ "Please install it with: pip install cryptography"
+ ) from e
+
+ key = serialization.load_pem_private_key(
+ key_str.encode("utf-8"),
+ password=None,
+ )
+ if not isinstance(key, rsa.RSAPrivateKey):
+ raise TypeError(
+ "The provided private key is not an RSA key, which is required for OCI signing."
+ )
+ return key
+
+
+def load_private_key_from_file(file_path: str):
+ """Loads a private key from a file path"""
+ try:
+ with open(file_path, "r", encoding="utf-8") as f:
+ key_str = f.read().strip()
+ except FileNotFoundError:
+ raise FileNotFoundError(f"Private key file not found: {file_path}")
+ except OSError as e:
+ raise OSError(f"Failed to read private key file '{file_path}': {e}") from e
+
+ if not key_str:
+ raise ValueError(f"Private key file is empty: {file_path}")
+
+ return load_private_key_from_str(key_str)
+
+
+def get_vendor_from_model(model: str) -> OCIVendors:
+ """
+ Extracts the vendor from the model name.
+ Args:
+ model (str): The model name.
+ Returns:
+ str: The vendor name.
+ """
+ vendor = model.split(".")[0].lower()
+ if vendor == "cohere":
+ return OCIVendors.COHERE
+ else:
+ return OCIVendors.GENERIC
+
+
+# 5 minute timeout (models may need to load)
+STREAMING_TIMEOUT = 60 * 5
+
+
+class OCIChatConfig(BaseConfig):
+ """
+ Configuration class for OCI's API interface.
+ """
+
+ def __init__(
+ self,
+ ) -> None:
+ locals_ = locals().copy()
+ for key, value in locals_.items():
+ if key != "self" and value is not None:
+ setattr(self.__class__, key, value)
+ # mark the class as using a custom stream wrapper because the default only iterates on lines
+ setattr(self.__class__, "has_custom_stream_wrapper", True)
+
+ self.openai_to_oci_generic_param_map = {
+ "stream": "isStream",
+ "max_tokens": "maxTokens",
+ "max_completion_tokens": "maxTokens",
+ "temperature": "temperature",
+ "tools": "tools",
+ "frequency_penalty": "frequencyPenalty",
+ "logprobs": "logProbs",
+ "logit_bias": "logitBias",
+ "n": "numGenerations",
+ "presence_penalty": "presencePenalty",
+ "seed": "seed",
+ "stop": "stop",
+ "tool_choice": "toolChoice",
+ "top_p": "topP",
+ "max_retries": False,
+ "top_logprobs": False,
+ "modalities": False,
+ "prediction": False,
+ "stream_options": False,
+ "function_call": False,
+ "functions": False,
+ "extra_headers": False,
+ "parallel_tool_calls": False,
+ "audio": False,
+ "web_search_options": False,
+ }
+
+ # Cohere and Gemini use the same parameter mapping as GENERIC
+ self.openai_to_oci_cohere_param_map = self.openai_to_oci_generic_param_map.copy()
+
+ def get_supported_openai_params(self, model: str) -> List[str]:
+ supported_params = []
+ vendor = get_vendor_from_model(model)
+ if vendor == OCIVendors.COHERE:
+ open_ai_to_oci_param_map = self.openai_to_oci_cohere_param_map
+ open_ai_to_oci_param_map.pop("tool_choice")
+ open_ai_to_oci_param_map.pop("max_retries")
+ else:
+ open_ai_to_oci_param_map = self.openai_to_oci_generic_param_map
+ for key, value in open_ai_to_oci_param_map.items():
+ if value:
+ supported_params.append(key)
+
+ return supported_params
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ adapted_params = {}
+ vendor = get_vendor_from_model(model)
+ if vendor == OCIVendors.COHERE:
+ open_ai_to_oci_param_map = self.openai_to_oci_cohere_param_map
+ else:
+ open_ai_to_oci_param_map = self.openai_to_oci_generic_param_map
+
+ all_params = {**non_default_params, **optional_params}
+
+ for key, value in all_params.items():
+ alias = open_ai_to_oci_param_map.get(key)
+
+ if alias is False:
+ # Workaround for mypy issue
+ if drop_params or litellm.drop_params:
+ continue
+ raise Exception(f"param `{key}` is not supported on OCI")
+
+ if alias is None:
+ adapted_params[key] = value
+ continue
+
+ adapted_params[alias] = value
+
+ return adapted_params
+
+ def sign_request(
+ self,
+ headers: dict,
+ optional_params: dict,
+ request_data: dict,
+ api_base: str,
+ api_key: Optional[str] = None,
+ model: Optional[str] = None,
+ stream: Optional[bool] = None,
+ fake_stream: Optional[bool] = None,
+ ) -> Tuple[dict, Optional[bytes]]:
+ """
+ Some providers like Bedrock require signing the request. The sign request funtion needs access to `request_data` and `complete_url`
+ Args:
+ headers: dict
+ optional_params: dict
+ request_data: dict - the request body being sent in http request
+ api_base: str - the complete url being sent in http request
+ Returns:
+ dict - the signed headers
+ """
+ import json
+
+ oci_region = optional_params.get("oci_region", "us-ashburn-1")
+ api_base = (
+ api_base
+ or litellm.api_base
+ or f"https://inference.generativeai.{oci_region}.oci.oraclecloud.com"
+ )
+ oci_user = optional_params.get("oci_user")
+ oci_fingerprint = optional_params.get("oci_fingerprint")
+ oci_tenancy = optional_params.get("oci_tenancy")
+ oci_key = optional_params.get("oci_key")
+ oci_key_file = optional_params.get("oci_key_file")
+
+ if (
+ not oci_user
+ or not oci_fingerprint
+ or not oci_tenancy
+ or not (oci_key or oci_key_file)
+ ):
+ raise Exception(
+ "Missing required parameters: oci_user, oci_fingerprint, oci_tenancy, "
+ "and at least one of oci_key or oci_key_file."
+ )
+
+ method = str(optional_params.get("method", "POST")).upper()
+ body = json.dumps(request_data).encode("utf-8")
+ parsed = urlparse(api_base)
+ path = parsed.path or "/"
+ host = parsed.netloc
+
+ date = datetime.datetime.utcnow().strftime("%a, %d %b %Y %H:%M:%S GMT")
+ content_type = headers.get("content-type", "application/json")
+ content_length = str(len(body))
+ x_content_sha256 = sha256_base64(body)
+
+ headers_to_sign = {
+ "date": date,
+ "host": host,
+ "content-type": content_type,
+ "content-length": content_length,
+ "x-content-sha256": x_content_sha256,
+ }
+
+ signed_headers = [
+ "date",
+ "(request-target)",
+ "host",
+ "content-length",
+ "content-type",
+ "x-content-sha256",
+ ]
+ signing_string = build_signature_string(
+ method, path, headers_to_sign, signed_headers
+ )
+
+ try:
+ from cryptography.hazmat.primitives import hashes
+ from cryptography.hazmat.primitives.asymmetric import padding
+ except ImportError as e:
+ raise ImportError(
+ "cryptography package is required for OCI authentication. "
+ "Please install it with: pip install cryptography"
+ ) from e
+
+ private_key = (
+ load_private_key_from_str(oci_key)
+ if oci_key
+ else load_private_key_from_file(oci_key_file) if oci_key_file else None
+ )
+
+ if private_key is None:
+ raise Exception(
+ "Private key is required for OCI authentication. Please provide either oci_key or oci_key_file."
+ )
+
+ signature = private_key.sign(
+ signing_string.encode("utf-8"),
+ padding.PKCS1v15(),
+ hashes.SHA256(),
+ )
+ signature_b64 = base64.b64encode(signature).decode()
+
+ key_id = f"{oci_tenancy}/{oci_user}/{oci_fingerprint}"
+
+ authorization = (
+ 'Signature version="1",'
+ f'keyId="{key_id}",'
+ 'algorithm="rsa-sha256",'
+ f'headers="{" ".join(signed_headers)}",'
+ f'signature="{signature_b64}"'
+ )
+
+ headers.update(
+ {
+ "authorization": authorization,
+ "date": date,
+ "host": host,
+ "content-type": content_type,
+ "content-length": content_length,
+ "x-content-sha256": x_content_sha256,
+ }
+ )
+
+ return headers, None
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ oci_region = optional_params.get("oci_region", "us-ashburn-1")
+ api_base = (
+ api_base
+ or litellm.api_base
+ or f"https://inference.generativeai.{oci_region}.oci.oraclecloud.com"
+ )
+ oci_user = optional_params.get("oci_user")
+ oci_fingerprint = optional_params.get("oci_fingerprint")
+ oci_tenancy = optional_params.get("oci_tenancy")
+ oci_key = optional_params.get("oci_key")
+ oci_key_file = optional_params.get("oci_key_file")
+ oci_compartment_id = optional_params.get("oci_compartment_id")
+
+ if (
+ not oci_user
+ or not oci_fingerprint
+ or not oci_tenancy
+ or not (oci_key or oci_key_file)
+ or not oci_compartment_id
+ ):
+ raise Exception(
+ "Missing required parameters: oci_user, oci_fingerprint, oci_tenancy, oci_compartment_id "
+ "and at least one of oci_key or oci_key_file."
+ )
+
+ if not api_base:
+ raise Exception(
+ "Either `api_base` must be provided or `litellm.api_base` must be set. Alternatively, you can set the `oci_region` optional parameter to use the default OCI region."
+ )
+
+ headers.update(
+ {
+ "content-type": "application/json",
+ "user-agent": f"litellm/{version}",
+ }
+ )
+
+ if not messages:
+ raise Exception(
+ "kwarg `messages` must be an array of messages that follow the openai chat standard"
+ )
+
+ return headers
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ oci_region = optional_params.get("oci_region", "us-ashburn-1")
+ return f"https://inference.generativeai.{oci_region}.oci.oraclecloud.com/20231130/actions/chat"
+
+ def _get_optional_params(self, vendor: OCIVendors, optional_params: dict) -> Dict:
+ selected_params = {}
+ if vendor == OCIVendors.COHERE:
+ open_ai_to_oci_param_map = self.openai_to_oci_cohere_param_map
+ # remove tool_choice from the map
+ open_ai_to_oci_param_map.pop("tool_choice")
+ # Add default values for Cohere API
+ selected_params = {
+ "maxTokens": 600,
+ "temperature": 1,
+ "topK": 0,
+ "topP": 0.75,
+ "frequencyPenalty": 0
+ }
+ else:
+ open_ai_to_oci_param_map = self.openai_to_oci_generic_param_map
+
+ # Map OpenAI params to OCI params
+ for openai_key, oci_key in open_ai_to_oci_param_map.items():
+ if oci_key and openai_key in optional_params:
+ selected_params[oci_key] = optional_params[openai_key] # type: ignore[index]
+
+ # Also check for already-mapped OCI params (for backward compatibility)
+ for oci_value in open_ai_to_oci_param_map.values():
+ if oci_value and oci_value in optional_params and oci_value not in selected_params:
+ selected_params[oci_value] = optional_params[oci_value] # type: ignore[index]
+
+ if "tools" in selected_params:
+ if vendor == OCIVendors.COHERE:
+ selected_params["tools"] = self.adapt_tool_definitions_to_cohere_standard( # type: ignore[assignment]
+ selected_params["tools"] # type: ignore[arg-type]
+ )
+ else:
+ selected_params["tools"] = adapt_tool_definition_to_oci_standard( # type: ignore[assignment]
+ selected_params["tools"], vendor # type: ignore[arg-type]
+ )
+ return selected_params
+
+ def adapt_messages_to_cohere_standard(self, messages: List[AllMessageValues]) -> List[CohereMessage]:
+ """Build chat history for Cohere models."""
+ chat_history = []
+ for msg in messages[:-1]: # All messages except the last one
+ role = msg.get("role")
+ content = msg.get("content")
+
+ if isinstance(content, list):
+ # Extract text from content array
+ text_content = ""
+ for content_item in content:
+ if isinstance(content_item, dict) and content_item.get("type") == "text":
+ text_content += content_item.get("text", "")
+ content = text_content
+
+ # Ensure content is a string
+ if not isinstance(content, str):
+ content = str(content) if content is not None else ""
+
+ # Handle tool calls
+ tool_calls: Optional[List[CohereToolCall]] = None
+ if role == "assistant" and "tool_calls" in msg and msg.get("tool_calls"): # type: ignore[union-attr,typeddict-item]
+ tool_calls = []
+ for tool_call in msg["tool_calls"]: # type: ignore[union-attr,typeddict-item]
+ # Parse arguments if they're a JSON string
+ raw_arguments: Any = tool_call.get("function", {}).get("arguments", {})
+ if isinstance(raw_arguments, str):
+ try:
+ arguments: Dict[str, Any] = json.loads(raw_arguments)
+ except json.JSONDecodeError:
+ arguments = {}
+ else:
+ arguments = raw_arguments
+
+ tool_calls.append(CohereToolCall(
+ name=str(tool_call.get("function", {}).get("name", "")),
+ parameters=arguments
+ ))
+
+ if role == "user":
+ chat_history.append(CohereMessage(role="USER", message=content))
+ elif role == "assistant":
+ chat_history.append(CohereMessage(role="CHATBOT", message=content, toolCalls=tool_calls))
+ elif role == "tool":
+ # Tool messages need special handling
+ chat_history.append(CohereMessage(
+ role="TOOL",
+ message=content,
+ toolCalls=None # Tool messages don't have tool calls
+ ))
+
+ return chat_history
+
+ def adapt_tool_definitions_to_cohere_standard(self, tools: List[Dict[str, Any]]) -> List[CohereTool]:
+ """Adapt tool definitions to Cohere format."""
+ cohere_tools = []
+ for tool in tools:
+ function_def = tool.get("function", {})
+ parameters = function_def.get("parameters", {}).get("properties", {})
+ required = function_def.get("parameters", {}).get("required", [])
+
+ parameter_definitions = {}
+ for param_name, param_schema in parameters.items():
+ parameter_definitions[param_name] = CohereParameterDefinition(
+ description=param_schema.get("description", ""),
+ type=param_schema.get("type", "string"),
+ isRequired=param_name in required
+ )
+
+ cohere_tools.append(CohereTool(
+ name=function_def.get("name", ""),
+ description=function_def.get("description", ""),
+ parameterDefinitions=parameter_definitions
+ ))
+
+ return cohere_tools
+
+ def _extract_text_content(self, content: Any) -> str:
+ """Extract text content from message content."""
+ if isinstance(content, str):
+ return content
+ elif isinstance(content, list):
+ text_content = ""
+ for content_item in content:
+ if isinstance(content_item, dict) and content_item.get("type") == "text":
+ text_content += content_item.get("text", "")
+ return text_content
+ return str(content)
+
+ def transform_request(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ headers: dict,
+ ) -> dict:
+ oci_compartment_id = optional_params.get("oci_compartment_id", None)
+ if not oci_compartment_id:
+ raise Exception("kwarg `oci_compartment_id` is required for OCI requests")
+
+ vendor = get_vendor_from_model(model)
+
+ oci_serving_mode = optional_params.get("oci_serving_mode", "ON_DEMAND")
+ if oci_serving_mode not in ["ON_DEMAND", "DEDICATED"]:
+ raise Exception(
+ "kwarg `oci_serving_mode` must be either 'ON_DEMAND' or 'DEDICATED'"
+ )
+
+ if oci_serving_mode == "DEDICATED":
+ servingMode = OCIServingMode(
+ servingType="DEDICATED",
+ endpointId=model,
+ )
+ else:
+ servingMode = OCIServingMode(
+ servingType="ON_DEMAND",
+ modelId=model,
+ )
+
+ # Build request based on vendor type
+ if vendor == OCIVendors.COHERE:
+ # For Cohere, we need to use the specific Cohere format
+ # Extract the last user message as the main message
+ user_messages = [msg for msg in messages if msg.get("role") == "user"]
+ if not user_messages:
+ raise Exception("No user message found for Cohere model")
+
+
+ # Create Cohere-specific chat request
+ chat_request = CohereChatRequest(
+ apiFormat="COHERE",
+ message=self._extract_text_content(user_messages[-1]["content"]),
+ chatHistory=self.adapt_messages_to_cohere_standard(messages),
+ **self._get_optional_params(OCIVendors.COHERE, optional_params)
+ )
+
+ data = OCICompletionPayload(
+ compartmentId=oci_compartment_id,
+ servingMode=servingMode,
+ chatRequest=chat_request
+ )
+ else:
+ # Use generic format for other vendors
+ data = OCICompletionPayload(
+ compartmentId=oci_compartment_id,
+ servingMode=servingMode,
+ chatRequest=OCIChatRequestPayload(
+ apiFormat=vendor.value,
+ messages=adapt_messages_to_generic_oci_standard(messages),
+ **self._get_optional_params(vendor, optional_params),
+ ),
+ )
+
+ return data.model_dump(exclude_none=True)
+
+ def _handle_cohere_response(
+ self,
+ json_response: dict,
+ model: str,
+ model_response: ModelResponse
+ ) -> ModelResponse:
+ """Handle Cohere-specific response format."""
+ cohere_response = CohereChatResult(**json_response)
+ # Cohere response format (uses camelCase)
+ model_id = model
+
+ # Set basic response info
+ model_response.model = model_id
+ model_response.created = int(datetime.datetime.now().timestamp())
+
+ # Extract the response text
+ response_text = cohere_response.chatResponse.text
+ oci_finish_reason = cohere_response.chatResponse.finishReason
+
+ # Map finish reason
+ if oci_finish_reason == "COMPLETE":
+ finish_reason = "stop"
+ elif oci_finish_reason == "MAX_TOKENS":
+ finish_reason = "length"
+ else:
+ finish_reason = "stop"
+
+ # Handle tool calls
+ tool_calls: Optional[List[Dict[str, Any]]] = None
+ if cohere_response.chatResponse.toolCalls:
+ tool_calls = []
+ for tool_call in cohere_response.chatResponse.toolCalls:
+ tool_calls.append({
+ "id": f"call_{len(tool_calls)}", # Generate a simple ID
+ "type": "function",
+ "function": {
+ "name": tool_call.name,
+ "arguments": json.dumps(tool_call.parameters)
+ }
+ })
+
+ # Create choice
+ from litellm.types.utils import Choices
+ choice = Choices(
+ index=0,
+ message={
+ "role": "assistant",
+ "content": response_text,
+ "tool_calls": tool_calls
+ },
+ finish_reason=finish_reason
+ )
+ model_response.choices = [choice]
+
+ # Extract usage info
+ usage_info = cohere_response.chatResponse.usage
+ from litellm.types.utils import Usage
+ model_response.usage = Usage( # type: ignore[attr-defined]
+ prompt_tokens=usage_info.promptTokens, # type: ignore[union-attr]
+ completion_tokens=usage_info.completionTokens, # type: ignore[union-attr]
+ total_tokens=usage_info.totalTokens # type: ignore[union-attr]
+ )
+
+ return model_response
+
+ def _handle_generic_response(
+ self,
+ json: dict,
+ model: str,
+ model_response: ModelResponse,
+ raw_response: httpx.Response
+ ) -> ModelResponse:
+ """Handle generic OCI response format."""
+ try:
+ completion_response = OCICompletionResponse(**json)
+ except TypeError as e:
+ raise OCIError(
+ message=f"Response cannot be casted to OCICompletionResponse: {str(e)}",
+ status_code=raw_response.status_code,
+ )
+
+ iso_str = completion_response.chatResponse.timeCreated
+ dt = datetime.datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
+ model_response.created = int(dt.timestamp())
+
+ model_response.model = completion_response.modelId
+
+ message = model_response.choices[0].message # type: ignore
+ response_message = completion_response.chatResponse.choices[0].message
+ if response_message.content and response_message.content[0].type == "TEXT":
+ message.content = response_message.content[0].text
+ if response_message.toolCalls:
+ message.tool_calls = adapt_tools_to_openai_standard(
+ response_message.toolCalls
+ )
+
+ usage = Usage(
+ prompt_tokens=completion_response.chatResponse.usage.promptTokens,
+ completion_tokens=completion_response.chatResponse.usage.completionTokens,
+ total_tokens=completion_response.chatResponse.usage.totalTokens,
+ )
+ model_response.usage = usage # type: ignore
+
+ return model_response
+
+ def transform_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: ModelResponse,
+ logging_obj: LiteLLMLoggingObj,
+ request_data: dict,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ encoding: Any,
+ api_key: Optional[str] = None,
+ json_mode: Optional[bool] = None,
+ ) -> ModelResponse:
+ json = raw_response.json() # noqa: F811
+
+ error = json.get("error")
+
+ if error is not None:
+ raise OCIError(
+ message=str(json["error"]),
+ status_code=raw_response.status_code,
+ )
+
+ if not isinstance(json, dict):
+ raise OCIError(
+ message="Invalid response format from OCI",
+ status_code=raw_response.status_code,
+ )
+
+ vendor = get_vendor_from_model(model)
+
+ # Handle response based on vendor type
+ if vendor == OCIVendors.COHERE:
+ model_response = self._handle_cohere_response(json, model, model_response)
+ else:
+ model_response = self._handle_generic_response(json, model, model_response, raw_response)
+
+ model_response._hidden_params["additional_headers"] = raw_response.headers
+
+ return model_response
+
+ @track_llm_api_timing()
+ def get_sync_custom_stream_wrapper(
+ self,
+ model: str,
+ custom_llm_provider: str,
+ logging_obj: LiteLLMLoggingObj,
+ api_base: str,
+ headers: dict,
+ data: dict,
+ messages: list,
+ client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
+ json_mode: Optional[bool] = None,
+ signed_json_body: Optional[bytes] = None,
+ ) -> "OCIStreamWrapper":
+ if "stream" in data:
+ del data["stream"]
+ if client is None or isinstance(client, AsyncHTTPHandler):
+ client = _get_httpx_client(params={})
+
+ try:
+ response = client.post(
+ api_base,
+ headers=headers,
+ data=json.dumps(data),
+ stream=True,
+ logging_obj=logging_obj,
+ timeout=STREAMING_TIMEOUT,
+ )
+ except httpx.HTTPStatusError as e:
+ raise OCIError(status_code=e.response.status_code, message=e.response.text)
+
+ if response.status_code != 200:
+ raise OCIError(status_code=response.status_code, message=response.text)
+
+ completion_stream = response.iter_text()
+
+ streaming_response = OCIStreamWrapper(
+ completion_stream=completion_stream,
+ model=model,
+ custom_llm_provider=custom_llm_provider,
+ logging_obj=logging_obj,
+ )
+ return streaming_response
+
+ @track_llm_api_timing()
+ async def get_async_custom_stream_wrapper(
+ self,
+ model: str,
+ custom_llm_provider: str,
+ logging_obj: LiteLLMLoggingObj,
+ api_base: str,
+ headers: dict,
+ data: dict,
+ messages: list,
+ client: Optional[Union[HTTPHandler, AsyncHTTPHandler]] = None,
+ json_mode: Optional[bool] = None,
+ signed_json_body: Optional[bytes] = None,
+ ) -> "OCIStreamWrapper":
+ if "stream" in data:
+ del data["stream"]
+
+ if client is None or isinstance(client, HTTPHandler):
+ client = get_async_httpx_client(llm_provider=LlmProviders.BYTEZ, params={})
+
+ try:
+ response = await client.post(
+ api_base,
+ headers=headers,
+ data=json.dumps(data),
+ stream=True,
+ logging_obj=logging_obj,
+ timeout=STREAMING_TIMEOUT,
+ )
+ except httpx.HTTPStatusError as e:
+ raise OCIError(status_code=e.response.status_code, message=e.response.text)
+
+ if response.status_code != 200:
+ raise OCIError(status_code=response.status_code, message=response.text)
+
+ completion_stream = response.aiter_text()
+
+ async def split_chunks(completion_stream: AsyncIterator[str]):
+ async for item in completion_stream:
+ for chunk in item.split("\n\n"):
+ if not chunk:
+ continue
+ yield chunk.strip()
+
+ streaming_response = OCIStreamWrapper(
+ completion_stream=split_chunks(completion_stream),
+ model=model,
+ custom_llm_provider=custom_llm_provider,
+ logging_obj=logging_obj,
+ )
+ return streaming_response
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ return OCIError(status_code=status_code, message=error_message)
+
+
+open_ai_to_generic_oci_role_map: Dict[str, OCIRoles] = {
+ "system": "SYSTEM",
+ "user": "USER",
+ "assistant": "ASSISTANT",
+ "tool": "TOOL",
+}
+
+
+def adapt_messages_to_generic_oci_standard_content_message(
+ role: str, content: Union[str, list]
+) -> OCIMessage:
+ new_content: List[OCIContentPartUnion] = []
+ if isinstance(content, str):
+ return OCIMessage(
+ role=open_ai_to_generic_oci_role_map[role],
+ content=[OCITextContentPart(text=content)],
+ toolCalls=None,
+ toolCallId=None,
+ )
+
+ # content is a list of content items:
+ # [
+ # {"type": "text", "text": "Hello"},
+ # {"type": "image_url", "image_url": "https://example.com/image.png"}
+ # ]
+ for content_item in content:
+ if not isinstance(content_item, dict):
+ raise Exception("Each content item must be a dictionary")
+
+ type = content_item.get("type")
+ if not isinstance(type, str):
+ raise Exception("Prop `type` is not a string")
+
+ if type not in ["text", "image_url"]:
+ raise Exception(f"Prop `{type}` is not supported")
+
+ if type == "text":
+ text = content_item.get("text")
+ if not isinstance(text, str):
+ raise Exception("Prop `text` is not a string")
+ new_content.append(OCITextContentPart(text=text))
+
+ elif type == "image_url":
+ image_url = content_item.get("image_url")
+ if not isinstance(image_url, str):
+ raise Exception("Prop `image_url` is not a string")
+ new_content.append(OCIImageContentPart(imageUrl=image_url))
+
+ return OCIMessage(
+ role=open_ai_to_generic_oci_role_map[role],
+ content=new_content,
+ toolCalls=None,
+ toolCallId=None,
+ )
+
+
+def adapt_messages_to_generic_oci_standard_tool_call(
+ role: str, tool_calls: list
+) -> OCIMessage:
+ tool_calls_formated = []
+ for tool_call in tool_calls:
+ if not isinstance(tool_call, dict):
+ raise Exception("Each tool call must be a dictionary")
+
+ if tool_call.get("type") != "function":
+ raise Exception("OCI only supports function tools")
+
+ tool_call_id = tool_call.get("id")
+ if not isinstance(tool_call_id, str):
+ raise Exception("Prop `id` is not a string")
+
+ tool_function = tool_call.get("function")
+ if not isinstance(tool_function, dict):
+ raise Exception("Prop `function` is not a dictionary")
+
+ function_name = tool_function.get("name")
+ if not isinstance(function_name, str):
+ raise Exception("Prop `name` is not a string")
+
+ arguments = tool_call["function"].get("arguments", "{}")
+ if not isinstance(arguments, str):
+ raise Exception("Prop `arguments` is not a string")
+
+ # tool_calls_formated.append(OCIToolCall(
+ # id=tool_call_id,
+ # type="FUNCTION",
+ # function=OCIFunction(
+ # name=function_name,
+ # arguments=arguments
+ # )
+ # ))
+
+ tool_calls_formated.append(
+ OCIToolCall(
+ id=tool_call_id,
+ type="FUNCTION",
+ name=function_name,
+ arguments=arguments,
+ )
+ )
+
+ return OCIMessage(
+ role=open_ai_to_generic_oci_role_map[role],
+ content=None,
+ toolCalls=tool_calls_formated,
+ toolCallId=None,
+ )
+
+
+def adapt_messages_to_generic_oci_standard_tool_response(
+ role: str, tool_call_id: str, content: str
+) -> OCIMessage:
+ return OCIMessage(
+ role=open_ai_to_generic_oci_role_map[role],
+ content=[OCITextContentPart(text=content)],
+ toolCalls=None,
+ toolCallId=tool_call_id,
+ )
+
+
+def adapt_messages_to_generic_oci_standard(
+ messages: List[AllMessageValues],
+) -> List[OCIMessage]:
+ new_messages = []
+ for message in messages:
+ role = message["role"]
+ content = message.get("content")
+ tool_calls = message.get("tool_calls")
+ tool_call_id = message.get("tool_call_id")
+
+ if role == "assistant" and tool_calls is not None:
+ if not isinstance(tool_calls, list):
+ raise Exception("Prop `tool_calls` must be a list of tool calls")
+ new_messages.append(
+ adapt_messages_to_generic_oci_standard_tool_call(role, tool_calls)
+ )
+
+ elif role in ["system", "user", "assistant"] and content is not None:
+ if not isinstance(content, (str, list)):
+ raise Exception(
+ "Prop `content` must be a string or a list of content items"
+ )
+ new_messages.append(
+ adapt_messages_to_generic_oci_standard_content_message(role, content)
+ )
+
+ elif role == "tool":
+ if not isinstance(tool_call_id, str):
+ raise Exception("Prop `tool_call_id` is required and must be a string")
+ if not isinstance(content, str):
+ raise Exception("Prop `content` is not a string")
+ new_messages.append(
+ adapt_messages_to_generic_oci_standard_tool_response(
+ role, tool_call_id, content
+ )
+ )
+
+ return new_messages
+
+
+def adapt_tool_definition_to_oci_standard(tools: List[Dict], vendor: OCIVendors):
+ new_tools = []
+ for tool in tools:
+ if tool["type"] != "function":
+ raise Exception("OCI only supports function tools")
+
+ tool_function = tool.get("function")
+ if not isinstance(tool_function, dict):
+ raise Exception("Prop `function` is not a dictionary")
+
+ new_tool = OCIToolDefinition(
+ type="FUNCTION",
+ name=tool_function.get("name"),
+ description=tool_function.get("description", ""),
+ parameters=tool_function.get("parameters", {}),
+ )
+ new_tools.append(new_tool)
+
+ return new_tools
+
+
+def adapt_tools_to_openai_standard(
+ tools: List[OCIToolCall],
+) -> List[ChatCompletionMessageToolCall]:
+ new_tools = []
+ for tool in tools:
+ new_tool = ChatCompletionMessageToolCall(
+ id=tool.id,
+ type="function",
+ function={
+ "name": tool.name,
+ "arguments": tool.arguments,
+ },
+ )
+ new_tools.append(new_tool)
+ return new_tools
+
+
+class OCIStreamWrapper(CustomStreamWrapper):
+ """
+ Custom stream wrapper for OCI responses.
+ This class is used to handle streaming responses from OCI's API.
+ """
+
+ def __init__(
+ self,
+ **kwargs: Any,
+ ):
+ super().__init__(**kwargs)
+
+ def chunk_creator(self, chunk: Any):
+ if not isinstance(chunk, str):
+ raise ValueError(f"Chunk is not a string: {chunk}")
+ if not chunk.startswith("data:"):
+ raise ValueError(f"Chunk does not start with 'data:': {chunk}")
+ dict_chunk = json.loads(chunk[5:]) # Remove 'data: ' prefix and parse JSON
+
+ # Check if this is a Cohere stream chunk
+ if "apiFormat" in dict_chunk and dict_chunk.get("apiFormat") == "COHERE":
+ return self._handle_cohere_stream_chunk(dict_chunk)
+ else:
+ return self._handle_generic_stream_chunk(dict_chunk)
+
+ def _handle_cohere_stream_chunk(self, dict_chunk: dict):
+ """Handle Cohere-specific streaming chunks."""
+ try:
+ typed_chunk = CohereStreamChunk(**dict_chunk)
+ except TypeError as e:
+ raise ValueError(f"Chunk cannot be casted to CohereStreamChunk: {str(e)}")
+
+ if typed_chunk.index is None:
+ typed_chunk.index = 0
+
+ # Extract text content
+ text = typed_chunk.text or ""
+
+ # Map finish reason to standard format
+ finish_reason = typed_chunk.finishReason
+ if finish_reason == "COMPLETE":
+ finish_reason = "stop"
+ elif finish_reason == "MAX_TOKENS":
+ finish_reason = "length"
+ elif finish_reason is None:
+ finish_reason = None
+ else:
+ finish_reason = "stop"
+
+ # For Cohere, we don't have tool calls in the streaming format
+ tool_calls = None
+
+ return ModelResponseStream(
+ choices=[
+ StreamingChoices(
+ index=typed_chunk.index if typed_chunk.index else 0,
+ delta=Delta(
+ content=text,
+ tool_calls=tool_calls,
+ provider_specific_fields=None,
+ thinking_blocks=None,
+ reasoning_content=None,
+ ),
+ finish_reason=finish_reason,
+ )
+ ]
+ )
+
+ def _handle_generic_stream_chunk(self, dict_chunk: dict):
+ """Handle generic OCI streaming chunks."""
+ try:
+ typed_chunk = OCIStreamChunk(**dict_chunk)
+ except TypeError as e:
+ raise ValueError(f"Chunk cannot be casted to OCIStreamChunk: {str(e)}")
+
+ if typed_chunk.index is None:
+ typed_chunk.index = 0
+
+ text = ""
+ if typed_chunk.message and typed_chunk.message.content:
+ for item in typed_chunk.message.content:
+ if isinstance(item, OCITextContentPart):
+ text += item.text
+ elif isinstance(item, OCIImageContentPart):
+ raise ValueError(
+ "OCI does not support image content in streaming responses"
+ )
+ else:
+ raise ValueError(
+ f"Unsupported content type in OCI response: {item.type}"
+ )
+
+ tool_calls = None
+ if typed_chunk.message and typed_chunk.message.toolCalls:
+ tool_calls = adapt_tools_to_openai_standard(typed_chunk.message.toolCalls)
+
+ return ModelResponseStream(
+ choices=[
+ StreamingChoices(
+ index=typed_chunk.index if typed_chunk.index else 0,
+ delta=Delta(
+ content=text,
+ tool_calls=(
+ [tool.model_dump() for tool in tool_calls]
+ if tool_calls
+ else None
+ ),
+ provider_specific_fields=None, # OCI does not have provider specific fields in the response
+ thinking_blocks=None, # OCI does not have thinking blocks in the response
+ reasoning_content=None, # OCI does not have reasoning content in the response
+ ),
+ finish_reason=typed_chunk.finishReason,
+ )
+ ]
+ )
diff --git a/litellm/llms/oci/common_utils.py b/litellm/llms/oci/common_utils.py
new file mode 100644
index 00000000000..661a6c89e4b
--- /dev/null
+++ b/litellm/llms/oci/common_utils.py
@@ -0,0 +1,19 @@
+from typing import Optional
+
+import httpx
+
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+
+
+class OCIError(BaseLLMException):
+ def __init__(
+ self,
+ status_code: int,
+ message: str,
+ headers: Optional[httpx.Headers] = None,
+ ):
+ super().__init__(
+ status_code=status_code,
+ message=message,
+ headers=headers,
+ )
diff --git a/litellm/llms/ollama/chat/transformation.py b/litellm/llms/ollama/chat/transformation.py
index d4ce4052a7e..b740eb122fd 100644
--- a/litellm/llms/ollama/chat/transformation.py
+++ b/litellm/llms/ollama/chat/transformation.py
@@ -1,6 +1,6 @@
import json
import time
-import uuid
+from litellm._uuid import uuid
from typing import (
TYPE_CHECKING,
Any,
@@ -16,9 +16,18 @@ from httpx._models import Headers, Response
from pydantic import BaseModel
import litellm
+from litellm.litellm_core_utils.prompt_templates.common_utils import (
+ _extract_reasoning_content,
+ convert_content_list_to_str,
+ extract_images_from_message,
+)
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
-from litellm.types.llms.ollama import OllamaToolCall, OllamaToolCallFunction
+from litellm.types.llms.ollama import (
+ OllamaChatCompletionMessage,
+ OllamaToolCall,
+ OllamaToolCallFunction,
+)
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionAssistantToolCall,
@@ -137,6 +146,7 @@ class OllamaChatConfig(BaseConfig):
"tool_choice",
"functions",
"response_format",
+ "reasoning_effort",
]
def map_openai_params(
@@ -174,6 +184,11 @@ class OllamaChatConfig(BaseConfig):
):
if value.get("json_schema") and value["json_schema"].get("schema"):
optional_params["format"] = value["json_schema"]["schema"]
+ if param == "reasoning_effort" and value is not None:
+ if model.startswith("gpt-oss"):
+ optional_params["think"] = value
+ else:
+ optional_params["think"] = True
### FUNCTION CALLING LOGIC ###
if param == "tools":
## CHECK IF MODEL SUPPORTS TOOL CALLING ##
@@ -212,9 +227,9 @@ class OllamaChatConfig(BaseConfig):
litellm.add_function_to_prompt = (
True # so that main.py adds the function call to the prompt
)
- optional_params[
- "functions_unsupported_model"
- ] = non_default_params.get("functions")
+ optional_params["functions_unsupported_model"] = (
+ non_default_params.get("functions")
+ )
non_default_params.pop("tool_choice", None) # causes ollama requests to hang
non_default_params.pop("functions", None) # causes ollama requests to hang
return optional_params
@@ -229,6 +244,8 @@ class OllamaChatConfig(BaseConfig):
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
+ if api_key is not None and "Authorization" not in headers:
+ headers["Authorization"] = f"Bearer {api_key}"
return headers
def get_complete_url(
@@ -267,6 +284,7 @@ class OllamaChatConfig(BaseConfig):
stream = optional_params.pop("stream", False)
format = optional_params.pop("format", None)
keep_alive = optional_params.pop("keep_alive", None)
+ think = optional_params.pop("think", None)
function_name = optional_params.pop("function_name", None)
litellm_params["function_name"] = function_name
tools = optional_params.pop("tools", None)
@@ -294,7 +312,23 @@ class OllamaChatConfig(BaseConfig):
)
new_tools.append(ollama_tool_call)
cast(dict, m)["tool_calls"] = new_tools
- new_messages.append(m)
+ reasoning_content, parsed_content = _extract_reasoning_content(
+ cast(dict, m)
+ )
+ content_str = convert_content_list_to_str(cast(AllMessageValues, m))
+ images = extract_images_from_message(cast(AllMessageValues, m))
+
+ ollama_message = OllamaChatCompletionMessage(
+ role=cast(str, m.get("role")),
+ )
+ if reasoning_content is not None:
+ ollama_message["thinking"] = reasoning_content
+ if content_str is not None:
+ ollama_message["content"] = content_str
+ if images is not None:
+ ollama_message["images"] = images
+
+ new_messages.append(ollama_message)
# Load Config
config = self.get_config()
@@ -314,6 +348,8 @@ class OllamaChatConfig(BaseConfig):
data["tools"] = tools
if keep_alive is not None:
data["keep_alive"] = keep_alive
+ if think is not None:
+ data["think"] = think
return data
@@ -346,11 +382,31 @@ class OllamaChatConfig(BaseConfig):
## RESPONSE OBJECT
model_response.choices[0].finish_reason = "stop"
+ response_json_message = response_json.get("message")
+ if response_json_message is not None:
+ if "thinking" in response_json_message:
+ # remap 'thinking' to 'reasoning_content'
+ response_json_message["reasoning_content"] = response_json_message[
+ "thinking"
+ ]
+ del response_json_message["thinking"]
+ elif response_json_message.get("content") is not None:
+ # parse reasoning content from content
+ from litellm.litellm_core_utils.prompt_templates.common_utils import (
+ _parse_content_for_reasoning,
+ )
+
+ reasoning_content, content = _parse_content_for_reasoning(
+ response_json_message["content"]
+ )
+ response_json_message["reasoning_content"] = reasoning_content
+ response_json_message["content"] = content
+
if (
request_data.get("format", "") == "json"
and litellm_params.get("function_name") is not None
):
- function_call = json.loads(response_json["message"]["content"])
+ function_call = json.loads(response_json_message["content"])
message = litellm.Message(
content=None,
tool_calls=[
@@ -367,11 +423,13 @@ class OllamaChatConfig(BaseConfig):
"type": "function",
}
],
+ reasoning_content=response_json_message.get("reasoning_content"),
)
model_response.choices[0].message = message # type: ignore
model_response.choices[0].finish_reason = "tool_calls"
else:
- _message = litellm.Message(**response_json["message"])
+
+ _message = litellm.Message(**response_json_message)
model_response.choices[0].message = _message # type: ignore
model_response.created = int(time.time())
model_response.model = "ollama_chat/" + model
@@ -412,6 +470,9 @@ class OllamaChatConfig(BaseConfig):
class OllamaChatCompletionResponseIterator(BaseModelResponseIterator):
+ started_reasoning_content: bool = False
+ finished_reasoning_content: bool = False
+
def _is_function_call_complete(self, function_args: Union[str, dict]) -> bool:
if isinstance(function_args, dict):
return True
@@ -465,8 +526,38 @@ class OllamaChatCompletionResponseIterator(BaseModelResponseIterator):
if is_function_call_complete:
tool_call["id"] = str(uuid.uuid4())
+ # PROCESS REASONING CONTENT
+ reasoning_content: Optional[str] = None
+ content: Optional[str] = None
+ if chunk["message"].get("thinking") is not None:
+ if self.started_reasoning_content is False:
+ reasoning_content = chunk["message"].get("thinking")
+ self.started_reasoning_content = True
+ elif self.finished_reasoning_content is False:
+ reasoning_content = chunk["message"].get("thinking")
+ self.finished_reasoning_content = True
+ elif chunk["message"].get("content") is not None:
+ message_content = chunk["message"].get("content")
+ if "" in message_content:
+ message_content = message_content.replace("", "")
+
+ self.started_reasoning_content = True
+
+ if "" in message_content and self.started_reasoning_content:
+ message_content = message_content.replace("", "")
+ self.finished_reasoning_content = True
+
+ if (
+ self.started_reasoning_content
+ and not self.finished_reasoning_content
+ ):
+ reasoning_content = message_content
+ else:
+ content = message_content
+
delta = Delta(
- content=chunk["message"].get("content", ""),
+ content=content,
+ reasoning_content=reasoning_content,
tool_calls=tool_calls,
)
diff --git a/litellm/llms/ollama/common_utils.py b/litellm/llms/ollama/common_utils.py
index daff7a12065..166ceee27fc 100644
--- a/litellm/llms/ollama/common_utils.py
+++ b/litellm/llms/ollama/common_utils.py
@@ -57,8 +57,20 @@ class OllamaModelInfo(BaseLLMModelInfo):
"""
@staticmethod
- def get_api_key(api_key=None) -> None:
- return None # Ollama does not use an API key by default
+ def get_api_key(api_key=None) -> Optional[str]:
+ """Get API key from environment variables or litellm configuration"""
+ import os
+
+ import litellm
+ from litellm.secret_managers.main import get_secret_str
+
+ return (
+ os.environ.get("OLLAMA_API_KEY")
+ or litellm.api_key
+ or litellm.openai_key
+ or get_secret_str("OLLAMA_API_KEY")
+ )
+
@staticmethod
def get_api_base(api_base: Optional[str] = None) -> str:
@@ -73,9 +85,12 @@ class OllamaModelInfo(BaseLLMModelInfo):
"""
base = self.get_api_base(api_base)
+ api_key = self.get_api_key()
+ headers = { "Authorization": f"Bearer {api_key}" } if api_key else {}
+
names: set[str] = set()
try:
- resp = httpx.get(f"{base}/api/tags")
+ resp = httpx.get(f"{base}/api/tags", headers=headers)
resp.raise_for_status()
data = resp.json()
# Expecting a dict with a 'models' list
diff --git a/litellm/llms/ollama/completion/transformation.py b/litellm/llms/ollama/completion/transformation.py
index aa1da616d89..b476e5c8a63 100644
--- a/litellm/llms/ollama/completion/transformation.py
+++ b/litellm/llms/ollama/completion/transformation.py
@@ -1,6 +1,6 @@
import json
import time
-import uuid
+from litellm._uuid import uuid
from typing import TYPE_CHECKING, Any, AsyncIterator, Iterator, List, Optional, Union
from httpx._models import Headers, Response
@@ -19,11 +19,13 @@ from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMExcepti
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues, ChatCompletionUsageBlock
from litellm.types.utils import (
+ Delta,
GenericStreamingChunk,
ModelInfoBase,
ModelResponse,
ModelResponseStream,
ProviderField,
+ StreamingChoices,
)
from ..common_utils import OllamaError, _convert_image
@@ -90,9 +92,9 @@ class OllamaConfig(BaseConfig):
repeat_penalty: Optional[float] = None
temperature: Optional[float] = None
seed: Optional[int] = None
- stop: Optional[
- list
- ] = None # stop is a list based on this - https://github.com/ollama/ollama/pull/442
+ stop: Optional[list] = (
+ None # stop is a list based on this - https://github.com/ollama/ollama/pull/442
+ )
tfs_z: Optional[float] = None
num_predict: Optional[int] = None
top_k: Optional[int] = None
@@ -152,6 +154,7 @@ class OllamaConfig(BaseConfig):
"stop",
"response_format",
"max_completion_tokens",
+ "reasoning_effort",
]
def map_openai_params(
@@ -164,19 +167,24 @@ class OllamaConfig(BaseConfig):
for param, value in non_default_params.items():
if param == "max_tokens" or param == "max_completion_tokens":
optional_params["num_predict"] = value
- if param == "stream":
+ elif param == "stream":
optional_params["stream"] = value
- if param == "temperature":
+ elif param == "temperature":
optional_params["temperature"] = value
- if param == "seed":
+ elif param == "seed":
optional_params["seed"] = value
- if param == "top_p":
+ elif param == "top_p":
optional_params["top_p"] = value
- if param == "frequency_penalty":
+ elif param == "frequency_penalty":
optional_params["frequency_penalty"] = value
- if param == "stop":
+ elif param == "stop":
optional_params["stop"] = value
- if param == "response_format" and isinstance(value, dict):
+ elif param == "reasoning_effort" and value is not None:
+ if model.startswith("gpt-oss"):
+ optional_params["think"] = value
+ else:
+ optional_params["think"] = True
+ elif param == "response_format" and isinstance(value, dict):
if value["type"] == "json_object":
optional_params["format"] = "json"
elif value["type"] == "json_schema":
@@ -199,6 +207,21 @@ class OllamaConfig(BaseConfig):
return v
return None
+ @staticmethod
+ def get_api_key() -> Optional[str]:
+ """Get API key from environment variables or litellm configuration"""
+ import os
+
+ import litellm
+ from litellm.secret_managers.main import get_secret_str
+
+ return (
+ os.environ.get("OLLAMA_API_KEY")
+ or litellm.api_key
+ or litellm.openai_key
+ or get_secret_str("OLLAMA_API_KEY")
+ )
+
def get_model_info(self, model: str) -> ModelInfoBase:
"""
curl http://localhost:11434/api/show -d '{
@@ -208,11 +231,14 @@ class OllamaConfig(BaseConfig):
if model.startswith("ollama/") or model.startswith("ollama_chat/"):
model = model.split("/", 1)[1]
api_base = get_secret_str("OLLAMA_API_BASE") or "http://localhost:11434"
+ api_key = self.get_api_key()
+ headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}
try:
response = litellm.module_level_client.post(
url=f"{api_base}/api/show",
json={"name": model},
+ headers=headers,
)
except Exception as e:
raise Exception(
@@ -256,44 +282,82 @@ class OllamaConfig(BaseConfig):
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ModelResponse:
+ from litellm.litellm_core_utils.prompt_templates.common_utils import (
+ _parse_content_for_reasoning,
+ )
+
response_json = raw_response.json()
## RESPONSE OBJECT
model_response.choices[0].finish_reason = "stop"
if request_data.get("format", "") == "json":
- response_content = json.loads(response_json["response"])
+ # Check if response field exists and is not empty before parsing JSON
+ response_text = response_json.get("response", "")
- # Check if this is a function call format with name/arguments structure
- if (
- isinstance(response_content, dict)
- and "name" in response_content
- and "arguments" in response_content
- ):
- # Handle as function call (original behavior)
- function_call = response_content
- message = litellm.Message(
- content=None,
- tool_calls=[
- {
- "id": f"call_{str(uuid.uuid4())}",
- "function": {
- "name": function_call["name"],
- "arguments": json.dumps(function_call["arguments"]),
- },
- "type": "function",
- }
- ],
- )
- model_response.choices[0].message = message # type: ignore
- model_response.choices[0].finish_reason = "tool_calls"
- else:
- # Handle as regular JSON (new behavior)
- message = litellm.Message(
- content=json.dumps(response_content),
- )
+ if not response_text or not response_text.strip():
+ # Handle empty response gracefully - set empty content
+ message = litellm.Message(content="")
model_response.choices[0].message = message # type: ignore
model_response.choices[0].finish_reason = "stop"
+ else:
+ try:
+ response_content = json.loads(response_text)
+
+ # Check if this is a function call format with name/arguments structure
+ if (
+ isinstance(response_content, dict)
+ and "name" in response_content
+ and "arguments" in response_content
+ ):
+ # Handle as function call (original behavior)
+ function_call = response_content
+ message = litellm.Message(
+ content=None,
+ tool_calls=[
+ {
+ "id": f"call_{str(uuid.uuid4())}",
+ "function": {
+ "name": function_call["name"],
+ "arguments": json.dumps(
+ function_call["arguments"]
+ ),
+ },
+ "type": "function",
+ }
+ ],
+ )
+ model_response.choices[0].message = message # type: ignore
+ model_response.choices[0].finish_reason = "tool_calls"
+ else:
+ # Handle as regular JSON (new behavior)
+ message = litellm.Message(
+ content=json.dumps(response_content),
+ )
+ model_response.choices[0].message = message # type: ignore
+ model_response.choices[0].finish_reason = "stop"
+ except json.JSONDecodeError:
+ # If JSON parsing fails, treat as regular text response
+ ## output parse reasoning content from response_text
+ reasoning_content: Optional[str] = None
+ content: Optional[str] = None
+ if response_text is not None:
+ reasoning_content, content = _parse_content_for_reasoning(
+ response_text
+ )
+ message = litellm.Message(
+ content=content, reasoning_content=reasoning_content
+ )
+ model_response.choices[0].message = message # type: ignore
+ model_response.choices[0].finish_reason = "stop"
else:
- model_response.choices[0].message.content = response_json["response"] # type: ignore
+ response_text = response_json.get("response", "")
+ content = None
+ reasoning_content = None
+ if response_text is not None and isinstance(response_text, str):
+ reasoning_content, content = _parse_content_for_reasoning(response_text)
+ else:
+ content = response_text # type: ignore
+ model_response.choices[0].message.content = content # type: ignore
+ model_response.choices[0].message.reasoning_content = reasoning_content # type: ignore
model_response.created = int(time.time())
model_response.model = "ollama/" + model
_prompt = request_data.get("prompt", "")
@@ -351,6 +415,7 @@ class OllamaConfig(BaseConfig):
stream = optional_params.pop("stream", False)
format = optional_params.pop("format", None)
images = optional_params.pop("images", None)
+ think = optional_params.pop("think", None)
data = {
"model": model,
"prompt": ollama_prompt,
@@ -364,6 +429,8 @@ class OllamaConfig(BaseConfig):
data["images"] = [
_convert_image(convert_to_ollama_image(image)) for image in images
]
+ if think is not None:
+ data["think"] = think
return data
@@ -418,12 +485,21 @@ class OllamaConfig(BaseConfig):
class OllamaTextCompletionResponseIterator(BaseModelResponseIterator):
+ def __init__(
+ self, streaming_response, sync_stream: bool, json_mode: Optional[bool] = False
+ ):
+ super().__init__(streaming_response, sync_stream, json_mode)
+ self.started_reasoning_content: bool = False
+ self.finished_reasoning_content: bool = False
+
def _handle_string_chunk(
self, str_line: str
) -> Union[GenericStreamingChunk, ModelResponseStream]:
return self.chunk_parser(json.loads(str_line))
- def chunk_parser(self, chunk: dict) -> GenericStreamingChunk:
+ def chunk_parser(
+ self, chunk: dict
+ ) -> Union[GenericStreamingChunk, ModelResponseStream]:
try:
if "error" in chunk:
raise Exception(f"Ollama Error - {chunk}")
@@ -453,12 +529,53 @@ class OllamaTextCompletionResponseIterator(BaseModelResponseIterator):
)
elif chunk["response"]:
text = chunk["response"]
- return GenericStreamingChunk(
- text=text,
- is_finished=is_finished,
- finish_reason="stop",
+ reasoning_content: Optional[str] = None
+ content: Optional[str] = None
+ if text is not None:
+ if "" in text:
+ text = text.replace("", "")
+ self.started_reasoning_content = True
+ elif "" in text:
+ text = text.replace("", "")
+ self.finished_reasoning_content = True
+
+ if (
+ self.started_reasoning_content
+ and not self.finished_reasoning_content
+ ):
+ reasoning_content = text
+ else:
+ content = text
+
+ return ModelResponseStream(
+ choices=[
+ StreamingChoices(
+ index=0,
+ delta=Delta(
+ reasoning_content=reasoning_content, content=content
+ ),
+ )
+ ],
+ finish_reason=finish_reason,
usage=None,
)
+ # return GenericStreamingChunk(
+ # text=text,
+ # is_finished=is_finished,
+ # finish_reason="stop",
+ # usage=None,
+ # )
+ elif "thinking" in chunk and not chunk["response"]:
+ # Return reasoning content as ModelResponseStream so UIs can render it
+ thinking_content = chunk.get("thinking") or ""
+ return ModelResponseStream(
+ choices=[
+ StreamingChoices(
+ index=0,
+ delta=Delta(reasoning_content=thinking_content),
+ )
+ ]
+ )
else:
raise Exception(f"Unable to parse ollama chunk - {chunk}")
except Exception as e:
diff --git a/litellm/llms/ollama_chat.py b/litellm/llms/ollama_chat.py
index d46e7145194..e186636de99 100644
--- a/litellm/llms/ollama_chat.py
+++ b/litellm/llms/ollama_chat.py
@@ -1,6 +1,6 @@
import json
import time
-import uuid
+from litellm._uuid import uuid
from typing import Any, List, Optional, Union
import aiohttp
@@ -59,6 +59,7 @@ def get_ollama_response( # noqa: PLR0915
stream = optional_params.pop("stream", False)
format = optional_params.pop("format", None)
keep_alive = optional_params.pop("keep_alive", None)
+ think = optional_params.pop("think", None)
function_name = optional_params.pop("function_name", None)
tools = optional_params.pop("tools", None)
@@ -98,6 +99,8 @@ def get_ollama_response( # noqa: PLR0915
data["tools"] = tools
if keep_alive is not None:
data["keep_alive"] = keep_alive
+ if think is not None:
+ data["think"] = think
## LOGGING
logging_obj.pre_call(
input=None,
diff --git a/litellm/llms/openai/chat/gpt_5_transformation.py b/litellm/llms/openai/chat/gpt_5_transformation.py
new file mode 100644
index 00000000000..183f60debbd
--- /dev/null
+++ b/litellm/llms/openai/chat/gpt_5_transformation.py
@@ -0,0 +1,88 @@
+"""Support for OpenAI gpt-5 model family."""
+
+from typing import Optional
+
+import litellm
+
+from .gpt_transformation import OpenAIGPTConfig
+
+
+class OpenAIGPT5Config(OpenAIGPTConfig):
+ """Configuration for gpt-5 models including GPT-5-Codex variants.
+
+ Handles OpenAI API quirks for the gpt-5 series like:
+
+ - Mapping ``max_tokens`` -> ``max_completion_tokens``.
+ - Dropping unsupported ``temperature`` values when requested.
+ - Support for GPT-5-Codex models optimized for code generation.
+ """
+
+ @classmethod
+ def is_model_gpt_5_model(cls, model: str) -> bool:
+ return "gpt-5" in model
+
+ @classmethod
+ def is_model_gpt_5_codex_model(cls, model: str) -> bool:
+ """Check if the model is specifically a GPT-5 Codex variant."""
+ return "gpt-5-codex" in model
+
+ def get_supported_openai_params(self, model: str) -> list:
+ from litellm.utils import supports_tool_choice
+
+ base_gpt_series_params = super().get_supported_openai_params(model=model)
+ gpt_5_only_params = ["reasoning_effort"]
+ base_gpt_series_params.extend(gpt_5_only_params)
+ if not supports_tool_choice(model=model):
+ base_gpt_series_params.remove("tool_choice")
+
+ non_supported_params = [
+ "logprobs",
+ "top_p",
+ "presence_penalty",
+ "frequency_penalty",
+ "top_logprobs",
+ "stop",
+ ]
+
+ return [
+ param
+ for param in base_gpt_series_params
+ if param not in non_supported_params
+ ]
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ ################################################################
+ # max_tokens is not supported for gpt-5 models on OpenAI API
+ # Relevant issue: https://github.com/BerriAI/litellm/issues/13381
+ ################################################################
+ if "max_tokens" in non_default_params:
+ optional_params["max_completion_tokens"] = non_default_params.pop(
+ "max_tokens"
+ )
+
+ if "temperature" in non_default_params:
+ temperature_value: Optional[float] = non_default_params.pop("temperature")
+ if temperature_value is not None:
+ if temperature_value == 1:
+ optional_params["temperature"] = temperature_value
+ elif litellm.drop_params or drop_params:
+ pass
+ else:
+ raise litellm.utils.UnsupportedParamsError(
+ message=(
+ "gpt-5 models (including gpt-5-codex) don't support temperature={}. Only temperature=1 is supported. To drop unsupported params set `litellm.drop_params = True`"
+ ).format(temperature_value),
+ status_code=400,
+ )
+ return super()._map_openai_params(
+ non_default_params=non_default_params,
+ optional_params=optional_params,
+ model=model,
+ drop_params=drop_params,
+ )
diff --git a/litellm/llms/openai/chat/gpt_transformation.py b/litellm/llms/openai/chat/gpt_transformation.py
index 551f870aca9..3e18617905c 100644
--- a/litellm/llms/openai/chat/gpt_transformation.py
+++ b/litellm/llms/openai/chat/gpt_transformation.py
@@ -158,6 +158,7 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
"parallel_tool_calls",
"audio",
"web_search_options",
+ "safety_identifier",
] # works across all models
model_specific_params = []
@@ -348,6 +349,7 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
for message in messages:
message_content = message.get("content")
message_role = message.get("role")
+
if (
message_role == "user"
and message_content
@@ -395,13 +397,13 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
)
from litellm.types.llms.openai import ChatCompletionToolParam
- for message in messages:
- message = cast(
+ for i, message in enumerate(messages):
+ messages[i] = cast(
AllMessageValues, filter_value_from_dict(message, "cache_control") # type: ignore
)
if tools is not None:
- for tool in tools:
- tool = cast(
+ for i, tool in enumerate(tools):
+ tools[i] = cast(
ChatCompletionToolParam,
filter_value_from_dict(tool, "cache_control"), # type: ignore
)
@@ -428,6 +430,8 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
if tools is not None and len(tools) > 0:
optional_params["tools"] = tools
+ optional_params.pop("max_retries", None)
+
return {
"model": model,
"messages": messages,
@@ -705,8 +709,14 @@ class OpenAIGPTConfig(BaseLLMModelInfo, BaseConfig):
if api_key is None:
api_key = get_secret_str("OPENAI_API_KEY")
+ # Strip api_base to just the base URL (scheme + host + port)
+ parsed_url = httpx.URL(api_base)
+ base_url = f"{parsed_url.scheme}://{parsed_url.host}"
+ if parsed_url.port:
+ base_url += f":{parsed_url.port}"
+
response = litellm.module_level_client.get(
- url=f"{api_base}/v1/models",
+ url=f"{base_url}/v1/models",
headers={"Authorization": f"Bearer {api_key}"},
)
diff --git a/litellm/llms/openai/common_utils.py b/litellm/llms/openai/common_utils.py
index aa670df0531..ce470f04aca 100644
--- a/litellm/llms/openai/common_utils.py
+++ b/litellm/llms/openai/common_utils.py
@@ -5,12 +5,15 @@ Common helpers / utils across al OpenAI endpoints
import hashlib
import json
import ssl
-from typing import Any, Dict, List, Literal, Optional, Union
+from typing import Any, Dict, List, Literal, Optional, TYPE_CHECKING, Union
import httpx
import openai
from openai import AsyncAzureOpenAI, AsyncOpenAI, AzureOpenAI, OpenAI
+if TYPE_CHECKING:
+ from aiohttp import ClientSession
+
import litellm
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.custom_httpx.http_handler import (
@@ -194,7 +197,9 @@ class BaseOpenAILLM:
return param_names
@staticmethod
- def _get_async_http_client() -> Optional[httpx.AsyncClient]:
+ def _get_async_http_client(
+ shared_session: Optional["ClientSession"] = None,
+ ) -> Optional[httpx.AsyncClient]:
if litellm.aclient_session is not None:
return litellm.aclient_session
@@ -202,11 +207,13 @@ class BaseOpenAILLM:
ssl_config = get_ssl_configuration()
return httpx.AsyncClient(
- limits=httpx.Limits(max_connections=1000, max_keepalive_connections=100),
verify=ssl_config,
transport=AsyncHTTPHandler._create_async_transport(
- ssl_context=ssl_config if isinstance(ssl_config, ssl.SSLContext) else None,
+ ssl_context=ssl_config
+ if isinstance(ssl_config, ssl.SSLContext)
+ else None,
ssl_verify=ssl_config if isinstance(ssl_config, bool) else None,
+ shared_session=shared_session,
),
follow_redirects=True,
)
@@ -215,12 +222,11 @@ class BaseOpenAILLM:
def _get_sync_http_client() -> Optional[httpx.Client]:
if litellm.client_session is not None:
return litellm.client_session
-
+
# Get unified SSL configuration
ssl_config = get_ssl_configuration()
-
+
return httpx.Client(
- limits=httpx.Limits(max_connections=1000, max_keepalive_connections=100),
verify=ssl_config,
follow_redirects=True,
)
diff --git a/litellm/llms/openai/completion/transformation.py b/litellm/llms/openai/completion/transformation.py
index 43fbc1f2192..77dc0b54fe0 100644
--- a/litellm/llms/openai/completion/transformation.py
+++ b/litellm/llms/openai/completion/transformation.py
@@ -1,5 +1,5 @@
"""
-Support for gpt model family
+Support for gpt model family
"""
from typing import List, Optional, Union
@@ -87,7 +87,7 @@ class OpenAITextCompletionConfig(BaseTextCompletionConfig, OpenAIGPTConfig):
## RESPONSE OBJECT
if response_object is None or model_response_object is None:
raise ValueError("Error in response object format")
- choice_list = []
+ choice_list: List[Choices] = []
for idx, choice in enumerate(response_object["choices"]):
message = Message(
content=choice["text"],
@@ -100,7 +100,7 @@ class OpenAITextCompletionConfig(BaseTextCompletionConfig, OpenAIGPTConfig):
logprobs=choice.get("logprobs", None),
)
choice_list.append(choice)
- model_response_object.choices = choice_list
+ model_response_object.choices = choice_list # type: ignore
if "usage" in response_object:
setattr(model_response_object, "usage", response_object["usage"])
@@ -111,9 +111,9 @@ class OpenAITextCompletionConfig(BaseTextCompletionConfig, OpenAIGPTConfig):
if "model" in response_object:
model_response_object.model = response_object["model"]
- model_response_object._hidden_params[
- "original_response"
- ] = response_object # track original response, if users make a litellm.text_completion() request, we can return the original response
+ model_response_object._hidden_params["original_response"] = (
+ response_object # track original response, if users make a litellm.text_completion() request, we can return the original response
+ )
return model_response_object
except Exception as e:
raise e
diff --git a/litellm/llms/openai/cost_calculation.py b/litellm/llms/openai/cost_calculation.py
index 304c444e37a..229f75f2657 100644
--- a/litellm/llms/openai/cost_calculation.py
+++ b/litellm/llms/openai/cost_calculation.py
@@ -18,7 +18,7 @@ def cost_router(call_type: CallTypes) -> Literal["cost_per_token", "cost_per_sec
return "cost_per_token"
-def cost_per_token(model: str, usage: Usage) -> Tuple[float, float]:
+def cost_per_token(model: str, usage: Usage, service_tier: Optional[str] = None) -> Tuple[float, float]:
"""
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
@@ -31,7 +31,7 @@ def cost_per_token(model: str, usage: Usage) -> Tuple[float, float]:
"""
## CALCULATE INPUT COST
return generic_cost_per_token(
- model=model, usage=usage, custom_llm_provider="openai"
+ model=model, usage=usage, custom_llm_provider="openai", service_tier=service_tier
)
# ### Non-cached text tokens
# non_cached_text_tokens = usage.prompt_tokens
diff --git a/litellm/llms/openai/image_edit/transformation.py b/litellm/llms/openai/image_edit/transformation.py
index c8a1e8f0e1c..be960641154 100644
--- a/litellm/llms/openai/image_edit/transformation.py
+++ b/litellm/llms/openai/image_edit/transformation.py
@@ -80,24 +80,49 @@ class OpenAIImageEditConfig(BaseImageEditConfig):
request_dict = cast(Dict, request)
#########################################################
- # Separate images as `files` and send other parameters as `data`
+ # Separate images and masks as `files` and send other parameters as `data`
#########################################################
- _images = request_dict.get("image") or []
- data_without_images = {k: v for k, v in request_dict.items() if k != "image"}
+ _image_list = request_dict.get("image")
+ _mask = request_dict.get("mask")
+ data_without_files = {
+ k: v for k, v in request_dict.items() if k not in ["image", "mask"]
+ }
files_list: List[Tuple[str, Any]] = []
- for _image in _images:
- image_content_type: str = ImageEditRequestUtils.get_image_content_type(
- _image
+
+ # Handle image parameter
+ if _image_list is not None:
+ image_list = (
+ [_image_list] if not isinstance(_image_list, list) else _image_list
)
- if isinstance(_image, BufferedReader):
- files_list.append(
- ("image[]", (_image.name, _image, image_content_type))
+ for _image in image_list:
+ if _image is not None:
+ image_content_type: str = (
+ ImageEditRequestUtils.get_image_content_type(_image)
+ )
+ if isinstance(_image, BufferedReader):
+ files_list.append(
+ ("image[]", (_image.name, _image, image_content_type))
+ )
+ else:
+ files_list.append(
+ ("image[]", ("image.png", _image, image_content_type))
+ )
+ # Handle mask parameter if provided
+ if _mask is not None:
+ # Handle case where mask can be a list (extract first mask)
+ if isinstance(_mask, list):
+ _mask = _mask[0] if _mask else None
+
+ if _mask is not None:
+ mask_content_type: str = ImageEditRequestUtils.get_image_content_type(
+ _mask
)
- else:
- files_list.append(
- ("image[]", ("image.png", _image, image_content_type))
- )
- return data_without_images, files_list
+ if isinstance(_mask, BufferedReader):
+ files_list.append(("mask", (_mask.name, _mask, mask_content_type)))
+ else:
+ files_list.append(("mask", ("mask.png", _mask, mask_content_type)))
+
+ return data_without_files, files_list
def transform_image_edit_response(
self,
diff --git a/litellm/llms/openai/image_generation/gpt_transformation.py b/litellm/llms/openai/image_generation/gpt_transformation.py
index 150cffba21c..1cee13784e7 100644
--- a/litellm/llms/openai/image_generation/gpt_transformation.py
+++ b/litellm/llms/openai/image_generation/gpt_transformation.py
@@ -16,7 +16,6 @@ class GPTImageGenerationConfig(BaseImageGenerationConfig):
) -> List[OpenAIImageGenerationOptionalParams]:
return [
"background",
- "input_fidelity",
"moderation",
"n",
"output_compression",
diff --git a/litellm/llms/openai/openai.py b/litellm/llms/openai/openai.py
index e9bed019a91..324205237dc 100644
--- a/litellm/llms/openai/openai.py
+++ b/litellm/llms/openai/openai.py
@@ -10,12 +10,16 @@ from typing import (
List,
Literal,
Optional,
+ TYPE_CHECKING,
Union,
cast,
)
from urllib.parse import urlparse
import httpx
+
+if TYPE_CHECKING:
+ from aiohttp import ClientSession
import openai
from openai import AsyncOpenAI, OpenAI
from openai.types.beta.assistant_deleted import AssistantDeleted
@@ -47,6 +51,7 @@ from litellm.utils import (
from ...types.llms.openai import *
from ..base import BaseLLM
+from .chat.gpt_5_transformation import OpenAIGPT5Config
from .chat.o_series_transformation import OpenAIOSeriesConfig
from .common_utils import (
BaseOpenAILLM,
@@ -55,6 +60,7 @@ from .common_utils import (
)
openaiOSeriesConfig = OpenAIOSeriesConfig()
+openAIGPT5Config = OpenAIGPT5Config()
class MistralEmbeddingConfig:
@@ -183,6 +189,8 @@ class OpenAIConfig(BaseConfig):
"""
if openaiOSeriesConfig.is_model_o_series_model(model=model):
return openaiOSeriesConfig.get_supported_openai_params(model=model)
+ elif openAIGPT5Config.is_model_gpt_5_model(model=model):
+ return openAIGPT5Config.get_supported_openai_params(model=model)
elif litellm.openAIGPTAudioConfig.is_model_gpt_audio_model(model=model):
return litellm.openAIGPTAudioConfig.get_supported_openai_params(model=model)
else:
@@ -217,6 +225,13 @@ class OpenAIConfig(BaseConfig):
model=model,
drop_params=drop_params,
)
+ elif openAIGPT5Config.is_model_gpt_5_model(model=model):
+ return openAIGPT5Config.map_openai_params(
+ non_default_params=non_default_params,
+ optional_params=optional_params,
+ model=model,
+ drop_params=drop_params,
+ )
elif litellm.openAIGPTAudioConfig.is_model_gpt_audio_model(model=model):
return litellm.openAIGPTAudioConfig.map_openai_params(
non_default_params=non_default_params,
@@ -344,6 +359,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
max_retries: Optional[int] = DEFAULT_MAX_RETRIES,
organization: Optional[str] = None,
client: Optional[Union[OpenAI, AsyncOpenAI]] = None,
+ shared_session: Optional["ClientSession"] = None,
) -> Optional[Union[OpenAI, AsyncOpenAI]]:
client_initialization_params: Dict = locals()
if client is None:
@@ -368,7 +384,9 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
_new_client: Union[OpenAI, AsyncOpenAI] = AsyncOpenAI(
api_key=api_key,
base_url=api_base,
- http_client=OpenAIChatCompletion._get_async_http_client(),
+ http_client=OpenAIChatCompletion._get_async_http_client(
+ shared_session=shared_session
+ ),
timeout=timeout,
max_retries=max_retries,
organization=organization,
@@ -511,8 +529,9 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
organization: Optional[str] = None,
custom_llm_provider: Optional[str] = None,
drop_params: Optional[bool] = None,
+ shared_session: Optional["ClientSession"] = None,
):
- super().completion()
+ super().completion(shared_session=shared_session)
try:
fake_stream: bool = False
inference_params = optional_params.copy()
@@ -595,6 +614,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
organization=organization,
drop_params=drop_params,
fake_stream=fake_stream,
+ shared_session=shared_session,
)
data = provider_config.transform_request(
@@ -760,6 +780,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
drop_params: Optional[bool] = None,
stream_options: Optional[dict] = None,
fake_stream: bool = False,
+ shared_session: Optional["ClientSession"] = None,
):
response = None
data = await provider_config.async_transform_request(
@@ -782,6 +803,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
max_retries=max_retries,
organization=organization,
client=client,
+ shared_session=shared_session,
)
## LOGGING
@@ -1103,6 +1125,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
api_base: Optional[str] = None,
client: Optional[AsyncOpenAI] = None,
max_retries=None,
+ shared_session: Optional["ClientSession"] = None,
):
try:
openai_aclient: AsyncOpenAI = self._get_openai_client( # type: ignore
@@ -1112,6 +1135,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
timeout=timeout,
max_retries=max_retries,
client=client,
+ shared_session=shared_session,
)
headers, response = await self.make_openai_embedding_request(
openai_aclient=openai_aclient,
@@ -1175,6 +1199,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
client=None,
aembedding=None,
max_retries: Optional[int] = None,
+ shared_session: Optional["ClientSession"] = None,
) -> EmbeddingResponse:
super().embedding()
try:
@@ -1201,6 +1226,7 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
timeout=timeout,
client=client,
max_retries=max_retries,
+ shared_session=shared_session,
)
openai_client: OpenAI = self._get_openai_client( # type: ignore
diff --git a/litellm/llms/openai/realtime/handler.py b/litellm/llms/openai/realtime/handler.py
index aca32e1404a..e0c85d18178 100644
--- a/litellm/llms/openai/realtime/handler.py
+++ b/litellm/llms/openai/realtime/handler.py
@@ -1,5 +1,5 @@
"""
-This file contains the calling Azure OpenAI's `/openai/realtime` endpoint.
+This file contains the calling OpenAI's `/v1/realtime` endpoint.
This requires websockets, and is currently only supported on LiteLLM Proxy.
"""
@@ -15,7 +15,7 @@ from litellm.types.realtime import RealtimeQueryParams
class OpenAIRealtime(OpenAIChatCompletion):
def _construct_url(self, api_base: str, query_params: RealtimeQueryParams) -> str:
"""
- Construct the backend websocket URL with all query parameters (excluding 'model' if present).
+ Construct the backend websocket URL with all query parameters (including 'model').
"""
from httpx import URL
@@ -24,10 +24,9 @@ class OpenAIRealtime(OpenAIChatCompletion):
url = URL(api_base)
# Set the correct path
url = url.copy_with(path="/v1/realtime")
- # Build query dict excluding 'model'
- query_dict = {k: v for k, v in query_params.items() if k != "model"}
- if query_dict:
- url = url.copy_with(params=query_dict)
+ # Include all query parameters including 'model'
+ if query_params:
+ url = url.copy_with(params=query_params)
return str(url)
async def async_realtime(
@@ -43,11 +42,10 @@ class OpenAIRealtime(OpenAIChatCompletion):
):
import websockets
from websockets.asyncio.client import ClientConnection
-
if api_base is None:
- raise ValueError("api_base is required for Azure OpenAI calls")
+ api_base = "https://api.openai.com/"
if api_key is None:
- raise ValueError("api_key is required for Azure OpenAI calls")
+ raise ValueError("api_key is required for OpenAI realtime calls")
# Use all query params if provided, else fallback to just model
if query_params is None:
diff --git a/litellm/llms/openai/responses/transformation.py b/litellm/llms/openai/responses/transformation.py
index 527ae4a9d49..1e949e434d3 100644
--- a/litellm/llms/openai/responses/transformation.py
+++ b/litellm/llms/openai/responses/transformation.py
@@ -1,17 +1,22 @@
-from typing import TYPE_CHECKING, Any, Dict, Optional, Union, cast
+from typing import TYPE_CHECKING, Any, Dict, Optional, Union, cast, get_type_hints
import httpx
+from openai.types.responses import ResponseReasoningItem
+from pydantic import BaseModel
import litellm
from litellm._logging import verbose_logger
+from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response import (
+ _safe_convert_created_field,
+)
from litellm.llms.base_llm.responses.transformation import BaseResponsesAPIConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import *
from litellm.types.responses.main import *
from litellm.types.router import GenericLiteLLMParams
+from litellm.types.utils import LlmProviders
from ..common_utils import OpenAIError
-from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response import _safe_convert_created_field
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@@ -22,36 +27,28 @@ else:
class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
+ @property
+ def custom_llm_provider(self) -> LlmProviders:
+ return LlmProviders.OPENAI
+
def get_supported_openai_params(self, model: str) -> list:
"""
All OpenAI Responses API params are supported
"""
- return [
- "input",
- "model",
- "include",
- "instructions",
- "max_output_tokens",
- "metadata",
- "parallel_tool_calls",
- "previous_response_id",
- "reasoning",
- "store",
- "background",
- "stream",
- "prompt",
- "temperature",
- "text",
- "tool_choice",
- "tools",
- "top_p",
- "truncation",
- "user",
- "extra_headers",
- "extra_query",
- "extra_body",
- "timeout",
- ]
+ supported_params = get_type_hints(ResponsesAPIRequestParams).keys()
+ return list(
+ set(
+ [
+ "input",
+ "model",
+ "extra_headers",
+ "extra_query",
+ "extra_body",
+ "timeout",
+ ]
+ + list(supported_params)
+ )
+ )
def map_openai_params(
self,
@@ -71,12 +68,91 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
headers: dict,
) -> Dict:
"""No transform applied since inputs are in OpenAI spec already"""
- return dict(
+
+ input = self._validate_input_param(input)
+ final_request_params = dict(
ResponsesAPIRequestParams(
model=model, input=input, **response_api_optional_request_params
)
)
+ return final_request_params
+
+ def _validate_input_param(
+ self, input: Union[str, ResponseInputParam]
+ ) -> Union[str, ResponseInputParam]:
+ """
+ Ensure all input fields if pydantic are converted to dict
+
+ OpenAI API Fails when we try to JSON dumps specific input pydantic fields.
+ This function ensures all input fields are converted to dict.
+ """
+ if isinstance(input, list):
+ validated_input = []
+ for item in input:
+ # if it's pydantic, convert to dict
+ if isinstance(item, BaseModel):
+ validated_input.append(item.model_dump(exclude_none=True))
+ elif isinstance(item, dict):
+ # Handle reasoning items specifically to filter out status=None
+ verbose_logger.debug(f"Handling reasoning item: {item}")
+ if item.get("type") == "reasoning":
+ # Type assertion since we know it's a dict at this point
+ dict_item = cast(Dict[str, Any], item)
+ filtered_item = self._handle_reasoning_item(dict_item)
+ else:
+ # For other dict items, just pass through
+ filtered_item = cast(Dict[str, Any], item)
+ validated_input.append(filtered_item)
+ else:
+ validated_input.append(item)
+ return validated_input # type: ignore
+ # Input is expected to be either str or List, no single BaseModel expected
+ return input
+
+ def _handle_reasoning_item(self, item: Dict[str, Any]) -> Dict[str, Any]:
+ """
+ Handle reasoning items specifically to filter out status=None using OpenAI's model.
+ Issue: https://github.com/BerriAI/litellm/issues/13484
+ OpenAI API does not accept ReasoningItem(status=None), so we need to:
+ 1. Check if the item is a reasoning type
+ 2. Create a ResponseReasoningItem object with the item data
+ 3. Convert it back to dict with exclude_none=True to filter None values
+ """
+ if item.get("type") == "reasoning":
+ try:
+ # Ensure required fields are present for ResponseReasoningItem
+ item_data = dict(item)
+ if "id" not in item_data:
+ item_data["id"] = f"rs_{hash(str(item_data))}"
+ if "summary" not in item_data:
+ item_data["summary"] = (
+ item_data.get("reasoning_content", "")[:100] + "..."
+ if len(item_data.get("reasoning_content", "")) > 100
+ else item_data.get("reasoning_content", "")
+ )
+
+ # Create ResponseReasoningItem object from the item data
+ reasoning_item = ResponseReasoningItem(**item_data)
+
+ # Convert back to dict with exclude_none=True to exclude None fields
+ dict_reasoning_item = reasoning_item.model_dump(exclude_none=True)
+
+ return dict_reasoning_item
+ except Exception as e:
+ verbose_logger.debug(
+ f"Failed to create ResponseReasoningItem, falling back to manual filtering: {e}"
+ )
+ # Fallback: manually filter out known None fields
+ filtered_item = {
+ k: v
+ for k, v in item.items()
+ if v is not None
+ or k not in {"status", "content", "encrypted_content"}
+ }
+ return filtered_item
+ return item
+
def transform_response_api_response(
self,
model: str,
@@ -85,13 +161,25 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
) -> ResponsesAPIResponse:
"""No transform applied since outputs are in OpenAI spec already"""
try:
+ logging_obj.post_call(
+ original_response=raw_response.text,
+ additional_args={"complete_input_dict": {}},
+ )
raw_response_json = raw_response.json()
- raw_response_json["created_at"] = _safe_convert_created_field(raw_response_json["created_at"])
+ raw_response_json["created_at"] = _safe_convert_created_field(
+ raw_response_json["created_at"]
+ )
except Exception:
raise OpenAIError(
message=raw_response.text, status_code=raw_response.status_code
)
- return ResponsesAPIResponse(**raw_response_json)
+ try:
+ return ResponsesAPIResponse(**raw_response_json)
+ except Exception:
+ verbose_logger.debug(
+ f"Error constructing ResponsesAPIResponse: {raw_response_json}, using model_construct"
+ )
+ return ResponsesAPIResponse.model_construct(**raw_response_json)
def validate_environment(
self, headers: dict, model: str, litellm_params: Optional[GenericLiteLLMParams]
@@ -185,6 +273,15 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
ResponsesAPIStreamEvents.WEB_SEARCH_CALL_IN_PROGRESS: WebSearchCallInProgressEvent,
ResponsesAPIStreamEvents.WEB_SEARCH_CALL_SEARCHING: WebSearchCallSearchingEvent,
ResponsesAPIStreamEvents.WEB_SEARCH_CALL_COMPLETED: WebSearchCallCompletedEvent,
+ ResponsesAPIStreamEvents.MCP_LIST_TOOLS_IN_PROGRESS: MCPListToolsInProgressEvent,
+ ResponsesAPIStreamEvents.MCP_LIST_TOOLS_COMPLETED: MCPListToolsCompletedEvent,
+ ResponsesAPIStreamEvents.MCP_LIST_TOOLS_FAILED: MCPListToolsFailedEvent,
+ ResponsesAPIStreamEvents.MCP_CALL_IN_PROGRESS: MCPCallInProgressEvent,
+ ResponsesAPIStreamEvents.MCP_CALL_ARGUMENTS_DELTA: MCPCallArgumentsDeltaEvent,
+ ResponsesAPIStreamEvents.MCP_CALL_ARGUMENTS_DONE: MCPCallArgumentsDoneEvent,
+ ResponsesAPIStreamEvents.MCP_CALL_COMPLETED: MCPCallCompletedEvent,
+ ResponsesAPIStreamEvents.MCP_CALL_FAILED: MCPCallFailedEvent,
+ ResponsesAPIStreamEvents.IMAGE_GENERATION_PARTIAL_IMAGE: ImageGenerationPartialImageEvent,
ResponsesAPIStreamEvents.ERROR: ErrorEvent,
}
@@ -330,3 +427,39 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
raise OpenAIError(
message=raw_response.text, status_code=raw_response.status_code
)
+
+ #########################################################
+ ########## CANCEL RESPONSE API TRANSFORMATION ##########
+ #########################################################
+ def transform_cancel_response_api_request(
+ self,
+ response_id: str,
+ api_base: str,
+ litellm_params: GenericLiteLLMParams,
+ headers: dict,
+ ) -> Tuple[str, Dict]:
+ """
+ Transform the cancel response API request into a URL and data
+
+ OpenAI API expects the following request
+ - POST /v1/responses/{response_id}/cancel
+ """
+ url = f"{api_base}/{response_id}/cancel"
+ data: Dict = {}
+ return url, data
+
+ def transform_cancel_response_api_response(
+ self,
+ raw_response: httpx.Response,
+ logging_obj: LiteLLMLoggingObj,
+ ) -> ResponsesAPIResponse:
+ """
+ Transform the cancel response API response into a ResponsesAPIResponse
+ """
+ try:
+ raw_response_json = raw_response.json()
+ except Exception:
+ raise OpenAIError(
+ message=raw_response.text, status_code=raw_response.status_code
+ )
+ return ResponsesAPIResponse(**raw_response_json)
diff --git a/litellm/llms/openai/transcriptions/gpt_transformation.py b/litellm/llms/openai/transcriptions/gpt_transformation.py
index 796e10f5153..34621c44e22 100644
--- a/litellm/llms/openai/transcriptions/gpt_transformation.py
+++ b/litellm/llms/openai/transcriptions/gpt_transformation.py
@@ -1,5 +1,8 @@
from typing import List
+from litellm.llms.base_llm.audio_transcription.transformation import (
+ AudioTranscriptionRequestData,
+)
from litellm.types.llms.openai import OpenAIAudioTranscriptionOptionalParams
from litellm.types.utils import FileTypes
@@ -27,8 +30,12 @@ class OpenAIGPTAudioTranscriptionConfig(OpenAIWhisperAudioTranscriptionConfig):
audio_file: FileTypes,
optional_params: dict,
litellm_params: dict,
- ) -> dict:
+ ) -> AudioTranscriptionRequestData:
"""
Transform the audio transcription request
"""
- return {"model": model, "file": audio_file, **optional_params}
+ data = {"model": model, "file": audio_file, **optional_params}
+
+ return AudioTranscriptionRequestData(
+ data=data,
+ )
diff --git a/litellm/llms/openai/transcriptions/handler.py b/litellm/llms/openai/transcriptions/handler.py
index 4fe48dd3c6c..19b303bb968 100644
--- a/litellm/llms/openai/transcriptions/handler.py
+++ b/litellm/llms/openai/transcriptions/handler.py
@@ -1,4 +1,4 @@
-from typing import Optional, Union
+from typing import Optional, Union, cast
import httpx
from openai import AsyncOpenAI, OpenAI
@@ -34,6 +34,7 @@ class OpenAIAudioTranscription(OpenAIChatCompletion):
- call openai_aclient.audio.transcriptions.create by default
"""
try:
+
raw_response = (
await openai_aclient.audio.transcriptions.with_raw_response.create(
**data, timeout=timeout
@@ -93,15 +94,14 @@ class OpenAIAudioTranscription(OpenAIChatCompletion):
Handle audio transcription request
"""
if provider_config is not None:
- data = provider_config.transform_audio_transcription_request(
+ transformed_data = provider_config.transform_audio_transcription_request(
model=model,
audio_file=audio_file,
optional_params=optional_params,
litellm_params=litellm_params,
)
- if not isinstance(data, dict):
- raise ValueError("OpenAI transformation route requires a dict")
+ data = cast(dict, transformed_data.data)
else:
data = {"model": model, "file": audio_file, **optional_params}
diff --git a/litellm/llms/openai/transcriptions/whisper_transformation.py b/litellm/llms/openai/transcriptions/whisper_transformation.py
index c0ccc71579f..fa507e1bc26 100644
--- a/litellm/llms/openai/transcriptions/whisper_transformation.py
+++ b/litellm/llms/openai/transcriptions/whisper_transformation.py
@@ -1,8 +1,9 @@
from typing import List, Optional, Union
-from httpx import Headers
+from httpx import Headers, Response
from litellm.llms.base_llm.audio_transcription.transformation import (
+ AudioTranscriptionRequestData,
BaseAudioTranscriptionConfig,
)
from litellm.llms.base_llm.chat.transformation import BaseLLMException
@@ -11,12 +12,40 @@ from litellm.types.llms.openai import (
AllMessageValues,
OpenAIAudioTranscriptionOptionalParams,
)
-from litellm.types.utils import FileTypes
+from litellm.types.utils import FileTypes, TranscriptionResponse
from ..common_utils import OpenAIError
class OpenAIWhisperAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ """
+ OPTIONAL
+
+ Get the complete url for the request
+
+ Some providers need `model` in `api_base`
+ """
+ ## get the api base, attach the endpoint - v1/audio/transcriptions
+ # strip trailing slash if present
+ api_base = api_base.rstrip("/") if api_base else ""
+
+ # if endswith "/v1"
+ if api_base and api_base.endswith("/v1"):
+ api_base = f"{api_base}/audio/transcriptions"
+ else:
+ api_base = f"{api_base}/v1/audio/transcriptions"
+
+ return api_base or ""
+
def get_supported_openai_params(
self, model: str
) -> List[OpenAIAudioTranscriptionOptionalParams]:
@@ -72,21 +101,22 @@ class OpenAIWhisperAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
audio_file: FileTypes,
optional_params: dict,
litellm_params: dict,
- ) -> dict:
+ ) -> AudioTranscriptionRequestData:
"""
Transform the audio transcription request
"""
-
data = {"model": model, "file": audio_file, **optional_params}
if "response_format" not in data or (
data["response_format"] == "text" or data["response_format"] == "json"
):
- data[
- "response_format"
- ] = "verbose_json" # ensures 'duration' is received - used for cost calculation
+ data["response_format"] = (
+ "verbose_json" # ensures 'duration' is received - used for cost calculation
+ )
- return data
+ return AudioTranscriptionRequestData(
+ data=data,
+ )
def get_error_class(
self, error_message: str, status_code: int, headers: Union[dict, Headers]
@@ -96,3 +126,25 @@ class OpenAIWhisperAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
message=error_message,
headers=headers,
)
+
+ def transform_audio_transcription_response(
+ self,
+ raw_response: Response,
+ ) -> TranscriptionResponse:
+ try:
+ raw_response_json = raw_response.json()
+ except Exception as e:
+ raise ValueError(
+ f"Error transforming response to json: {str(e)}\nResponse: {raw_response.text}"
+ )
+
+ if any(
+ key in raw_response_json
+ for key in TranscriptionResponse.model_fields.keys()
+ ):
+ return TranscriptionResponse(**raw_response_json)
+ else:
+ raise ValueError(
+ "Invalid response format. Received response does not match the expected format. Got: ",
+ raw_response_json,
+ )
diff --git a/litellm/llms/openai/vector_stores/transformation.py b/litellm/llms/openai/vector_stores/transformation.py
index 0e890f0fd51..76cd12be8ee 100644
--- a/litellm/llms/openai/vector_stores/transformation.py
+++ b/litellm/llms/openai/vector_stores/transformation.py
@@ -14,6 +14,7 @@ from litellm.types.vector_stores import (
VectorStoreSearchRequest,
VectorStoreSearchResponse,
)
+from litellm.utils import add_openai_metadata
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
@@ -119,12 +120,13 @@ class OpenAIVectorStoreConfig(BaseVectorStoreConfig):
api_base: str,
) -> Tuple[str, Dict]:
url = api_base # Base URL for creating vector stores
+ metadata = vector_store_create_optional_params.get("metadata", None)
typed_request_body = VectorStoreCreateRequest(
name=vector_store_create_optional_params.get("name", None),
file_ids=vector_store_create_optional_params.get("file_ids", None),
expires_after=vector_store_create_optional_params.get("expires_after", None),
chunking_strategy=vector_store_create_optional_params.get("chunking_strategy", None),
- metadata=vector_store_create_optional_params.get("metadata", None),
+ metadata=add_openai_metadata(metadata) if metadata is not None else None,
)
dict_request_body = cast(dict, typed_request_body)
diff --git a/litellm/llms/openrouter/chat/transformation.py b/litellm/llms/openrouter/chat/transformation.py
index bf57218c91d..f1eafe4e294 100644
--- a/litellm/llms/openrouter/chat/transformation.py
+++ b/litellm/llms/openrouter/chat/transformation.py
@@ -6,7 +6,8 @@ Calls done in OpenAI/openai.py as OpenRouter is openai-compatible.
Docs: https://openrouter.ai/docs/parameters
"""
-from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union
+from enum import Enum
+from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union, cast
import httpx
@@ -20,6 +21,12 @@ from ...openai.chat.gpt_transformation import OpenAIGPTConfig
from ..common_utils import OpenRouterException
+class CacheControlSupportedModels(str, Enum):
+ """Models that support cache_control in content blocks."""
+ CLAUDE = "claude"
+ GEMINI = "gemini"
+
+
class OpenrouterConfig(OpenAIGPTConfig):
def map_openai_params(
self,
@@ -48,19 +55,76 @@ class OpenrouterConfig(OpenAIGPTConfig):
)
return mapped_openai_params
+ def _supports_cache_control_in_content(self, model: str) -> bool:
+ """
+ Check if the model supports cache_control in content blocks.
+
+ Returns:
+ bool: True if model supports cache_control (Claude or Gemini models)
+ """
+ model_lower = model.lower()
+ return any(
+ supported_model.value in model_lower
+ for supported_model in CacheControlSupportedModels
+ )
+
def remove_cache_control_flag_from_messages_and_tools(
self,
model: str,
messages: List[AllMessageValues],
tools: Optional[List["ChatCompletionToolParam"]] = None,
) -> Tuple[List[AllMessageValues], Optional[List["ChatCompletionToolParam"]]]:
- if "claude" in model.lower(): # don't remove 'cache_control' flag
+ if self._supports_cache_control_in_content(model):
return messages, tools
else:
return super().remove_cache_control_flag_from_messages_and_tools(
model, messages, tools
)
+ def _move_cache_control_to_content(
+ self, messages: List[AllMessageValues]
+ ) -> List[AllMessageValues]:
+ """
+ Move cache_control from message level to content blocks.
+ OpenRouter requires cache_control to be inside content blocks, not at message level.
+
+ To avoid exceeding Anthropic's limit of 4 cache breakpoints, cache_control is only
+ added to the LAST content block in each message.
+ """
+ transformed_messages: List[AllMessageValues] = []
+ for message in messages:
+ message_dict = dict(message)
+ cache_control = message_dict.pop("cache_control", None)
+
+ if cache_control is not None:
+ content = message_dict.get("content")
+
+ if isinstance(content, list):
+ # Content is already a list, add cache_control only to the last block
+ if len(content) > 0:
+ content_copy = []
+ for i, block in enumerate(content):
+ block_dict = dict(block)
+ # Only add cache_control to the last content block
+ if i == len(content) - 1:
+ block_dict["cache_control"] = cache_control
+ content_copy.append(block_dict)
+ message_dict["content"] = content_copy
+ else:
+ # Content is a string, convert to structured format
+ message_dict["content"] = [
+ {
+ "type": "text",
+ "text": content,
+ "cache_control": cache_control,
+ }
+ ]
+
+ # Cast back to AllMessageValues after modification
+ transformed_messages.append(cast(AllMessageValues, message_dict))
+
+ return transformed_messages
+
def transform_request(
self,
model: str,
@@ -75,13 +139,78 @@ class OpenrouterConfig(OpenAIGPTConfig):
Returns:
dict: The transformed request. Sent as the body of the API call.
"""
+ if self._supports_cache_control_in_content(model):
+ messages = self._move_cache_control_to_content(messages)
+
extra_body = optional_params.pop("extra_body", {})
response = super().transform_request(
model, messages, optional_params, litellm_params, headers
)
response.update(extra_body)
+
+ # ALWAYS add usage parameter to get cost data from OpenRouter
+ # This ensures cost tracking works for all OpenRouter models
+ if "usage" not in response:
+ response["usage"] = {"include": True}
+
return response
+ def transform_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: ModelResponse,
+ logging_obj: Any,
+ request_data: dict,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ encoding: Any,
+ api_key: Optional[str] = None,
+ json_mode: Optional[bool] = None,
+ ) -> ModelResponse:
+ """
+ Transform the response from OpenRouter API.
+
+ Extracts cost information from response headers if available.
+
+ Returns:
+ ModelResponse: The transformed response with cost information.
+ """
+ # Call parent transform_response to get the standard ModelResponse
+ model_response = super().transform_response(
+ model=model,
+ raw_response=raw_response,
+ model_response=model_response,
+ logging_obj=logging_obj,
+ request_data=request_data,
+ messages=messages,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ encoding=encoding,
+ api_key=api_key,
+ json_mode=json_mode,
+ )
+
+ # Extract cost from OpenRouter response body
+ # OpenRouter returns cost information in the usage object when usage.include=true
+ try:
+ response_json = raw_response.json()
+ if "usage" in response_json and response_json["usage"]:
+ response_cost = response_json["usage"].get("cost")
+ if response_cost is not None:
+ # Store cost in hidden params for the cost calculator to use
+ if not hasattr(model_response, "_hidden_params"):
+ model_response._hidden_params = {}
+ if "additional_headers" not in model_response._hidden_params:
+ model_response._hidden_params["additional_headers"] = {}
+ model_response._hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"] = float(response_cost)
+ except Exception:
+ # If we can't extract cost, continue without it - don't fail the response
+ pass
+
+ return model_response
+
def get_error_class(
self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
) -> BaseLLMException:
diff --git a/litellm/llms/ovhcloud/chat/transformation.py b/litellm/llms/ovhcloud/chat/transformation.py
new file mode 100644
index 00000000000..6bdc28620ff
--- /dev/null
+++ b/litellm/llms/ovhcloud/chat/transformation.py
@@ -0,0 +1,141 @@
+"""
+Support for OVHCloud AI Endpoints `/v1/chat/completions` endpoint.
+
+Our unified API follows the OpenAI standard.
+More information on our website: https://endpoints.ai.cloud.ovh.net
+"""
+from typing import Optional, Union, List
+
+import httpx
+from litellm import ModelResponseStream, OpenAIGPTConfig, get_model_info, verbose_logger
+from litellm.llms.ovhcloud.utils import OVHCloudException
+from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.types.llms.openai import AllMessageValues
+
+class OVHCloudChatConfig(OpenAIGPTConfig):
+ @property
+ def custom_llm_provider(self) -> Optional[str]:
+ return "ovhcloud"
+
+ def get_supported_openai_params(self, model: str) -> list:
+ """
+ Details about function calling support can be found here:
+ https://help.ovhcloud.com/csm/en-gb-public-cloud-ai-endpoints-function-calling?id=kb_article_view&sysparm_article=KB0071907
+ """
+ supports_function_calling: Optional[bool] = None
+ try:
+ model_info = get_model_info(model, custom_llm_provider="ovhcloud")
+ supports_function_calling = model_info.get(
+ "supports_function_calling", False
+ )
+ except Exception as e:
+ verbose_logger.debug(f"Error getting supported OpenAI params: {e}")
+ pass
+
+ optional_params = super().get_supported_openai_params(model)
+ if supports_function_calling is not True:
+ verbose_logger.debug(
+ "You can see our models supporting function_calling in our catalog: https://endpoints.ai.cloud.ovh.net/catalog "
+ )
+ optional_params.remove("tools")
+ optional_params.remove("tool_choice")
+ optional_params.remove("function_call")
+ optional_params.remove("response_format")
+ return optional_params
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ api_base = "https://oai.endpoints.kepler.ai.cloud.ovh.net/v1" if api_base is None else api_base.rstrip("/")
+ complete_url = f"{api_base}/chat/completions"
+ return complete_url
+
+ def get_error_class(
+ self,
+ error_message: str,
+ status_code: int,
+ headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ return OVHCloudException(
+ message=error_message,
+ status_code=status_code,
+ headers=headers,
+ )
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ mapped_openai_params = super().map_openai_params(
+ non_default_params, optional_params, model, drop_params
+ )
+ return mapped_openai_params
+
+ def transform_request(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ headers: dict,
+ ) -> dict:
+ extra_body = optional_params.pop("extra_body", {})
+ response = super().transform_request(
+ model, messages, optional_params, litellm_params, headers
+ )
+ response.update(extra_body)
+ return response
+
+class OVHCloudChatCompletionStreamingHandler(BaseModelResponseIterator):
+ """
+ Handler for OVHCloud AI Endpoints streaming chat completion responses
+ """
+
+ def chunk_parser(self, chunk: dict) -> ModelResponseStream:
+ """
+ Parse individual chunks from streaming response
+ """
+ try:
+ if "error" in chunk:
+ error_chunk = chunk["error"]
+ error_message = "OVHCloud Error: {}".format(
+ error_chunk.get("message", "Unknown error")
+ )
+ raise OVHCloudException(
+ message=error_message,
+ status_code=error_chunk.get("code", 400),
+ headers={"Content-Type": "application/json"},
+ )
+
+ new_choices = []
+ for choice in chunk["choices"]:
+ if "delta" in choice and "reasoning" in choice["delta"]:
+ choice["delta"]["reasoning_content"] = choice["delta"].get("reasoning")
+ new_choices.append(choice)
+
+ return ModelResponseStream(
+ id=chunk["id"],
+ object="chat.completion.chunk",
+ created=chunk["created"],
+ usage=chunk.get("usage"),
+ model=chunk["model"],
+ choices=new_choices,
+ )
+ except KeyError as e:
+ raise OVHCloudException(
+ message=f"KeyError: {e}, Got unexpected response from CometAPI: {chunk}",
+ status_code=400,
+ headers={"Content-Type": "application/json"},
+ )
+ except Exception as e:
+ raise e
\ No newline at end of file
diff --git a/litellm/llms/ovhcloud/embedding/transformation.py b/litellm/llms/ovhcloud/embedding/transformation.py
new file mode 100644
index 00000000000..1266f74c0a2
--- /dev/null
+++ b/litellm/llms/ovhcloud/embedding/transformation.py
@@ -0,0 +1,122 @@
+"""
+This is OpenAI compatible - no transformation is applied
+
+"""
+from typing import List, Optional, Union
+
+import httpx
+
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.base_llm.embedding.transformation import BaseEmbeddingConfig
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.openai import AllEmbeddingInputValues, AllMessageValues
+from litellm.types.utils import EmbeddingResponse, Usage
+
+from ..utils import OVHCloudException
+
+
+class OVHCloudEmbeddingConfig(BaseEmbeddingConfig):
+ def __init__(self) -> None:
+ pass
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ api_base = "https://oai.endpoints.kepler.ai.cloud.ovh.net/v1" if api_base is None else api_base.rstrip("/")
+ complete_url = f"{api_base}/embeddings"
+ return complete_url
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ if api_key is None:
+ api_key = get_secret_str("OVHCLOUD_API_KEY")
+
+ default_headers = {
+ "Authorization": f"Bearer {api_key}",
+ "accept": "application/json",
+ "Content-Type": "application/json",
+ }
+
+ if "Authorization" in headers:
+ default_headers["Authorization"] = headers["Authorization"]
+
+ return {**default_headers, **headers}
+
+ def get_supported_openai_params(self, model: str):
+ return []
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ):
+ supported_openai_params = self.get_supported_openai_params(model)
+ for param, value in non_default_params.items():
+ if param in supported_openai_params:
+ optional_params[param] = value
+ return optional_params
+
+ def transform_embedding_request(
+ self,
+ model: str,
+ input: AllEmbeddingInputValues,
+ optional_params: dict,
+ headers: dict,
+ ) -> dict:
+ return {"input": input, "model": model, **optional_params}
+
+ def transform_embedding_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: EmbeddingResponse,
+ logging_obj: LiteLLMLoggingObj,
+ api_key: Optional[str],
+ request_data: dict,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> EmbeddingResponse:
+ try:
+ raw_response_json = raw_response.json()
+ except Exception:
+ raise OVHCloudException(
+ message=raw_response.text,
+ status_code=raw_response.status_code,
+ headers=raw_response.headers,
+ )
+
+ model_response.model = raw_response_json.get("model")
+ model_response.data = raw_response_json.get("data")
+ model_response.object = raw_response_json.get("object")
+
+ usage = Usage(
+ prompt_tokens=raw_response_json.get("usage", {}).get("prompt_tokens", 0),
+ total_tokens=raw_response_json.get("usage", {}).get("total_tokens", 0),
+ )
+
+ model_response.usage = usage
+ return model_response
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ return OVHCloudException(
+ message=error_message, status_code=status_code, headers=headers
+ )
diff --git a/litellm/llms/ovhcloud/utils.py b/litellm/llms/ovhcloud/utils.py
new file mode 100644
index 00000000000..9ae4dfb1efd
--- /dev/null
+++ b/litellm/llms/ovhcloud/utils.py
@@ -0,0 +1,6 @@
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+
+
+class OVHCloudException(BaseLLMException):
+ """OVHCloud AI Endpoints exception handling class"""
+ pass
\ No newline at end of file
diff --git a/litellm/llms/perplexity/chat/transformation.py b/litellm/llms/perplexity/chat/transformation.py
index 955fdff0818..27e6415ff8b 100644
--- a/litellm/llms/perplexity/chat/transformation.py
+++ b/litellm/llms/perplexity/chat/transformation.py
@@ -13,6 +13,8 @@ from litellm.types.utils import Usage, PromptTokensDetailsWrapper
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.types.utils import ModelResponse
+from litellm.types.llms.openai import ChatCompletionAnnotation
+from litellm.types.llms.openai import ChatCompletionAnnotationURLCitation
class PerplexityChatConfig(OpenAIGPTConfig):
@@ -102,7 +104,10 @@ class PerplexityChatConfig(OpenAIGPTConfig):
# Extract and enhance usage with Perplexity-specific fields
try:
raw_response_json = raw_response.json()
- self._enhance_usage_with_perplexity_fields(model_response, raw_response_json)
+ self._enhance_usage_with_perplexity_fields(
+ model_response, raw_response_json
+ )
+ self._add_citations_as_annotations(model_response, raw_response_json)
except Exception as e:
verbose_logger.debug(f"Error extracting Perplexity-specific usage fields: {e}")
@@ -131,7 +136,9 @@ class PerplexityChatConfig(OpenAIGPTConfig):
if citations:
# Count total characters in citations as a proxy for citation tokens
# This is an estimation - in practice, you might want to use proper tokenization
- total_citation_chars = sum(len(str(citation)) for citation in citations if citation)
+ total_citation_chars = sum(
+ len(str(citation)) for citation in citations if citation
+ )
# Rough estimation: ~4 characters per token (OpenAI's general rule)
if total_citation_chars > 0:
citation_tokens = max(1, total_citation_chars // 4)
@@ -150,7 +157,9 @@ class PerplexityChatConfig(OpenAIGPTConfig):
num_search_queries = raw_response_json.get("search_queries")
# Create or update prompt_tokens_details to include web search requests and citation tokens
- if citation_tokens > 0 or (num_search_queries is not None and num_search_queries > 0):
+ if citation_tokens > 0 or (
+ num_search_queries is not None and num_search_queries > 0
+ ):
if usage.prompt_tokens_details is None:
usage.prompt_tokens_details = PromptTokensDetailsWrapper()
@@ -161,3 +170,82 @@ class PerplexityChatConfig(OpenAIGPTConfig):
# Store search queries count in the standard web_search_requests field
if num_search_queries is not None and num_search_queries > 0:
usage.prompt_tokens_details.web_search_requests = num_search_queries
+
+ def _add_citations_as_annotations(
+ self, model_response: ModelResponse, raw_response_json: dict
+ ) -> None:
+ """
+ Extract citations and search_results from Perplexity API response
+ and add them as ChatCompletionAnnotation objects to the message.
+ """
+ if not model_response.choices:
+ return
+
+ # Get the first choice (assuming single response)
+ choice = model_response.choices[0]
+ if not hasattr(choice, "message") or choice.message is None:
+ return
+
+ message = choice.message
+ annotations = []
+
+ # Extract citations from the response
+ citations = raw_response_json.get("citations", [])
+ search_results = raw_response_json.get("search_results", [])
+
+ # Create a mapping of URLs to search result titles
+ url_to_title = {}
+ for result in search_results:
+ if isinstance(result, dict) and "url" in result and "title" in result:
+ url_to_title[result["url"]] = result["title"]
+
+ # Get the message content to find citation positions
+ content = getattr(message, "content", "")
+ if not content:
+ return
+
+ # Find all citation markers like [1], [2], [3], [4] in the text
+ import re
+
+ citation_pattern = r"\[(\d+)\]"
+ citation_matches = list(re.finditer(citation_pattern, content))
+
+ # Create a mapping of citation numbers to URLs
+ citation_number_to_url = {}
+ for i, citation in enumerate(citations):
+ if isinstance(citation, str):
+ citation_number_to_url[i + 1] = citation # 1-indexed
+
+ # Create annotations for each citation match found in the text
+ for match in citation_matches:
+ citation_number = int(match.group(1))
+ if citation_number in citation_number_to_url:
+ url = citation_number_to_url[citation_number]
+ title = url_to_title.get(url, "")
+
+ # Create the URL citation annotation with actual text positions
+ url_citation: ChatCompletionAnnotationURLCitation = {
+ "url": url,
+ "title": title,
+ "start_index": match.start(),
+ "end_index": match.end(),
+ }
+
+ annotation: ChatCompletionAnnotation = {
+ "type": "url_citation",
+ "url_citation": url_citation,
+ }
+
+ annotations.append(annotation)
+
+ # Add annotations to the message if we have any
+ if annotations:
+ if not hasattr(message, "annotations") or message.annotations is None:
+ message.annotations = []
+ message.annotations.extend(annotations)
+
+ # Also add the raw citations and search_results as attributes for backward compatibility
+ if citations:
+ setattr(model_response, "citations", citations)
+ if search_results:
+ setattr(model_response, "search_results", search_results)
\ No newline at end of file
diff --git a/litellm/llms/sambanova/common_utils.py b/litellm/llms/sambanova/common_utils.py
new file mode 100644
index 00000000000..b622f705845
--- /dev/null
+++ b/litellm/llms/sambanova/common_utils.py
@@ -0,0 +1,6 @@
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+
+
+class SambaNovaError(BaseLLMException):
+ def __init__(self, status_code, message, headers):
+ super().__init__(status_code=status_code, message=message, headers=headers)
diff --git a/litellm/llms/cohere/completion/handler.py b/litellm/llms/sambanova/embedding/handler.py
similarity index 50%
rename from litellm/llms/cohere/completion/handler.py
rename to litellm/llms/sambanova/embedding/handler.py
index 6a77951146f..c3629e4d75f 100644
--- a/litellm/llms/cohere/completion/handler.py
+++ b/litellm/llms/sambanova/embedding/handler.py
@@ -1,5 +1,5 @@
"""
-Cohere /generate API - uses `llm_http_handler.py` to make httpx requests
+SambaNova Embedding - uses `llm_http_handler.py` to make httpx requests
Request/Response transformation is handled in `transformation.py`
"""
diff --git a/litellm/llms/sambanova/embedding/transformation.py b/litellm/llms/sambanova/embedding/transformation.py
new file mode 100644
index 00000000000..eca44c7c039
--- /dev/null
+++ b/litellm/llms/sambanova/embedding/transformation.py
@@ -0,0 +1,139 @@
+"""
+This is OpenAI compatible - no transformation is applied
+
+"""
+from typing import List, Optional, Union
+
+import httpx
+
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.base_llm.embedding.transformation import BaseEmbeddingConfig
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.openai import AllEmbeddingInputValues, AllMessageValues
+from litellm.types.utils import EmbeddingResponse, Usage
+
+from ..common_utils import SambaNovaError
+
+
+class SambaNovaEmbeddingConfig(BaseEmbeddingConfig):
+ def __init__(self) -> None:
+ pass
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ if api_base is None:
+ raise ValueError("api_base is required for SambaNova embeddings")
+ # Remove trailing slashes and ensure clean base URL
+ api_base = api_base.rstrip("/")
+ if not api_base.endswith("/embeddings"):
+ api_base = f"{api_base}/embeddings"
+ return api_base
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ if api_key is None:
+ api_key = get_secret_str("SAMBANOVA_API_KEY")
+
+ default_headers = {
+ "Authorization": f"Bearer {api_key}",
+ "accept": "application/json",
+ "Content-Type": "application/json",
+ }
+
+ # If 'Authorization' is provided in headers, it overrides the default.
+ if "Authorization" in headers:
+ default_headers["Authorization"] = headers["Authorization"]
+
+ # Merge other headers, overriding any default ones except Authorization
+ return {**default_headers, **headers}
+
+ def get_supported_openai_params(self, model: str):
+ """
+ Non additional params supported, placeholder method for future supported params
+ https://docs.sambanova.ai/cloud/api-reference/endpoints/embeddings-api
+ """
+ return []
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ):
+ """
+ No transformation is applied - SambaNova is openai compatible
+ """
+ supported_openai_params = self.get_supported_openai_params(model)
+ for param, value in non_default_params.items():
+ if param in supported_openai_params:
+ optional_params[param] = value
+ return optional_params
+
+ def transform_embedding_request(
+ self,
+ model: str,
+ input: AllEmbeddingInputValues,
+ optional_params: dict,
+ headers: dict,
+ ) -> dict:
+ return {
+ "input": input,
+ "model": model,
+ **optional_params,
+ }
+
+ def transform_embedding_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: EmbeddingResponse,
+ logging_obj: LiteLLMLoggingObj,
+ api_key: Optional[str],
+ request_data: dict,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> EmbeddingResponse:
+ try:
+ raw_response_json = raw_response.json()
+ except Exception:
+ raise SambaNovaError(
+ message=raw_response.text,
+ status_code=raw_response.status_code,
+ headers=raw_response.headers,
+ )
+
+ model_response.model = raw_response_json.get("model")
+ model_response.data = raw_response_json.get("data")
+ model_response.object = raw_response_json.get("object")
+
+ usage = Usage(
+ prompt_tokens=raw_response_json.get("usage", {}).get("prompt_tokens", 0),
+ total_tokens=raw_response_json.get("usage", {}).get("total_tokens", 0),
+ )
+
+ model_response.usage = usage
+ return model_response
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ return SambaNovaError(
+ message=error_message, status_code=status_code, headers=headers
+ )
diff --git a/litellm/llms/snowflake/chat/transformation.py b/litellm/llms/snowflake/chat/transformation.py
index 2b92911b055..4c0258d9f4b 100644
--- a/litellm/llms/snowflake/chat/transformation.py
+++ b/litellm/llms/snowflake/chat/transformation.py
@@ -1,14 +1,15 @@
"""
-Support for Snowflake REST API
+Support for Snowflake REST API
"""
-from typing import TYPE_CHECKING, Any, List, Optional, Tuple
+import json
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
import httpx
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
-from litellm.types.utils import ModelResponse
+from litellm.types.utils import ChatCompletionMessageToolCall, Function, ModelResponse
from ...openai_like.chat.transformation import OpenAIGPTConfig
@@ -22,15 +23,25 @@ else:
class SnowflakeConfig(OpenAIGPTConfig):
"""
- source: https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex
+ Reference: https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-llm-rest-api
+
+ Snowflake Cortex LLM REST API supports function calling with specific models (e.g., Claude 3.5 Sonnet).
+ This config handles transformation between OpenAI format and Snowflake's tool_spec format.
"""
@classmethod
def get_config(cls):
return super().get_config()
- def get_supported_openai_params(self, model: str) -> List:
- return ["temperature", "max_tokens", "top_p", "response_format"]
+ def get_supported_openai_params(self, model: str) -> List[str]:
+ return [
+ "temperature",
+ "max_tokens",
+ "top_p",
+ "response_format",
+ "tools",
+ "tool_choice",
+ ]
def map_openai_params(
self,
@@ -56,6 +67,57 @@ class SnowflakeConfig(OpenAIGPTConfig):
optional_params[param] = value
return optional_params
+ def _transform_tool_calls_from_snowflake_to_openai(
+ self, content_list: List[Dict[str, Any]]
+ ) -> Tuple[str, Optional[List[ChatCompletionMessageToolCall]]]:
+ """
+ Transform Snowflake tool calls to OpenAI format.
+
+ Args:
+ content_list: Snowflake's content_list array containing text and tool_use items
+
+ Returns:
+ Tuple of (text_content, tool_calls)
+
+ Snowflake format in content_list:
+ {
+ "type": "tool_use",
+ "tool_use": {
+ "tool_use_id": "tooluse_...",
+ "name": "get_weather",
+ "input": {"location": "Paris"}
+ }
+ }
+
+ OpenAI format (returned tool_calls):
+ ChatCompletionMessageToolCall(
+ id="tooluse_...",
+ type="function",
+ function=Function(name="get_weather", arguments='{"location": "Paris"}')
+ )
+ """
+ text_content = ""
+ tool_calls: List[ChatCompletionMessageToolCall] = []
+
+ for idx, content_item in enumerate(content_list):
+ if content_item.get("type") == "text":
+ text_content += content_item.get("text", "")
+
+ ## TOOL CALLING
+ elif content_item.get("type") == "tool_use":
+ tool_use_data = content_item.get("tool_use", {})
+ tool_call = ChatCompletionMessageToolCall(
+ id=tool_use_data.get("tool_use_id", ""),
+ type="function",
+ function=Function(
+ name=tool_use_data.get("name", ""),
+ arguments=json.dumps(tool_use_data.get("input", {})),
+ ),
+ )
+ tool_calls.append(tool_call)
+
+ return text_content, tool_calls if tool_calls else None
+
def transform_response(
self,
model: str,
@@ -71,6 +133,7 @@ class SnowflakeConfig(OpenAIGPTConfig):
json_mode: Optional[bool] = None,
) -> ModelResponse:
response_json = raw_response.json()
+
logging_obj.post_call(
input=messages,
api_key="",
@@ -78,6 +141,26 @@ class SnowflakeConfig(OpenAIGPTConfig):
additional_args={"complete_input_dict": request_data},
)
+ ## RESPONSE TRANSFORMATION
+ # Snowflake returns content_list (not content) with tool_use objects
+ # We need to transform this to OpenAI's format with content + tool_calls
+ if "choices" in response_json and len(response_json["choices"]) > 0:
+ choice = response_json["choices"][0]
+ if "message" in choice and "content_list" in choice["message"]:
+ content_list = choice["message"]["content_list"]
+ (
+ text_content,
+ tool_calls,
+ ) = self._transform_tool_calls_from_snowflake_to_openai(content_list)
+
+ # Update the choice message with OpenAI format
+ choice["message"]["content"] = text_content
+ if tool_calls:
+ choice["message"]["tool_calls"] = tool_calls
+
+ # Remove Snowflake-specific content_list
+ del choice["message"]["content_list"]
+
returned_response = ModelResponse(**response_json)
returned_response.model = "snowflake/" + (returned_response.model or "")
@@ -150,6 +233,95 @@ class SnowflakeConfig(OpenAIGPTConfig):
return api_base
+ def _transform_tools(self, tools: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+ """
+ Transform OpenAI tool format to Snowflake tool format.
+
+ Args:
+ tools: List of tools in OpenAI format
+
+ Returns:
+ List of tools in Snowflake format
+
+ OpenAI format:
+ {
+ "type": "function",
+ "function": {
+ "name": "get_weather",
+ "description": "...",
+ "parameters": {...}
+ }
+ }
+
+ Snowflake format:
+ {
+ "tool_spec": {
+ "type": "generic",
+ "name": "get_weather",
+ "description": "...",
+ "input_schema": {...}
+ }
+ }
+ """
+ snowflake_tools: List[Dict[str, Any]] = []
+ for tool in tools:
+ if tool.get("type") == "function":
+ function = tool.get("function", {})
+ snowflake_tool: Dict[str, Any] = {
+ "tool_spec": {
+ "type": "generic",
+ "name": function.get("name"),
+ "input_schema": function.get(
+ "parameters",
+ {"type": "object", "properties": {}},
+ ),
+ }
+ }
+ # Add description if present
+ if "description" in function:
+ snowflake_tool["tool_spec"]["description"] = function[
+ "description"
+ ]
+
+ snowflake_tools.append(snowflake_tool)
+
+ return snowflake_tools
+
+ def _transform_tool_choice(
+ self, tool_choice: Union[str, Dict[str, Any]]
+ ) -> Union[str, Dict[str, Any]]:
+ """
+ Transform OpenAI tool_choice format to Snowflake format.
+
+ Args:
+ tool_choice: Tool choice in OpenAI format (str or dict)
+
+ Returns:
+ Tool choice in Snowflake format
+
+ OpenAI format:
+ {"type": "function", "function": {"name": "get_weather"}}
+
+ Snowflake format:
+ {"type": "tool", "name": ["get_weather"]}
+
+ Note: String values ("auto", "required", "none") pass through unchanged.
+ """
+ if isinstance(tool_choice, str):
+ # "auto", "required", "none" pass through as-is
+ return tool_choice
+
+ if isinstance(tool_choice, dict):
+ if tool_choice.get("type") == "function":
+ function_name = tool_choice.get("function", {}).get("name")
+ if function_name:
+ return {
+ "type": "tool",
+ "name": [function_name], # Snowflake expects array
+ }
+
+ return tool_choice
+
def transform_request(
self,
model: str,
@@ -160,6 +332,18 @@ class SnowflakeConfig(OpenAIGPTConfig):
) -> dict:
stream: bool = optional_params.pop("stream", None) or False
extra_body = optional_params.pop("extra_body", {})
+
+ ## TOOL CALLING
+ # Transform tools from OpenAI format to Snowflake's tool_spec format
+ tools = optional_params.pop("tools", None)
+ if tools:
+ optional_params["tools"] = self._transform_tools(tools)
+
+ # Transform tool_choice from OpenAI format to Snowflake's tool name array format
+ tool_choice = optional_params.pop("tool_choice", None)
+ if tool_choice:
+ optional_params["tool_choice"] = self._transform_tool_choice(tool_choice)
+
return {
"model": model,
"messages": messages,
diff --git a/litellm/llms/together_ai/rerank/transformation.py b/litellm/llms/together_ai/rerank/transformation.py
index 1fdb772adde..63b593dfe42 100644
--- a/litellm/llms/together_ai/rerank/transformation.py
+++ b/litellm/llms/together_ai/rerank/transformation.py
@@ -4,7 +4,7 @@ Transformation logic from Cohere's /v1/rerank format to Together AI's `/v1/rera
Why separate file? Make it easy to see how transformation works
"""
-import uuid
+from litellm._uuid import uuid
from typing import List, Optional
from litellm.types.rerank import (
diff --git a/litellm/llms/vercel_ai_gateway/chat/transformation.py b/litellm/llms/vercel_ai_gateway/chat/transformation.py
new file mode 100644
index 00000000000..13a88377489
--- /dev/null
+++ b/litellm/llms/vercel_ai_gateway/chat/transformation.py
@@ -0,0 +1,112 @@
+"""
+Support for OpenAI's `/v1/chat/completions` endpoint.
+
+Calls done in OpenAI/openai.py as Vercel AI Gateway is openai-compatible.
+
+Docs: https://vercel.com/docs/ai-gateway
+"""
+
+from typing import List, Optional, Tuple, Union
+
+import httpx
+
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.types.llms.openai import AllMessageValues
+from litellm.secret_managers.main import get_secret_str
+import litellm
+
+from ...openai.chat.gpt_transformation import OpenAIGPTConfig
+from ..common_utils import VercelAIGatewayException
+
+
+class VercelAIGatewayConfig(OpenAIGPTConfig):
+ @property
+ def custom_llm_provider(self) -> Optional[str]:
+ return "vercel_ai_gateway"
+
+ def get_supported_openai_params(self, model: str) -> list:
+ base_params = super().get_supported_openai_params(model)
+ if "extra_body" not in base_params:
+ base_params.append("extra_body")
+ return base_params
+
+ def _get_openai_compatible_provider_info(
+ self, api_base: Optional[str], api_key: Optional[str]
+ ) -> Tuple[Optional[str], Optional[str]]:
+
+ api_base = (
+ api_base
+ or get_secret_str("VERCEL_AI_GATEWAY_API_BASE")
+ or "https://ai-gateway.vercel.sh/v1"
+ )
+ user_api_key = (
+ api_key
+ or get_secret_str("VERCEL_AI_GATEWAY_API_KEY")
+ or get_secret_str("VERCEL_OIDC_TOKEN")
+ )
+ return api_base, user_api_key
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ mapped_openai_params = super().map_openai_params(
+ non_default_params, optional_params, model, drop_params
+ )
+
+ # Vercel AI Gateway-only parameters
+ extra_body = {}
+ provider_options = non_default_params.pop("providerOptions", None)
+
+ if provider_options is not None:
+ extra_body["providerOptions"] = provider_options
+
+ mapped_openai_params["extra_body"] = extra_body # openai client supports `extra_body` param
+ return mapped_openai_params
+
+ def transform_request(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ headers: dict,
+ ) -> dict:
+ """
+ Transform the overall request to be sent to the API.
+
+ Returns:
+ dict: The transformed request. Sent as the body of the API call.
+ """
+ return super().transform_request(
+ model, messages, optional_params, litellm_params, headers
+ )
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ return VercelAIGatewayException(
+ message=error_message,
+ status_code=status_code,
+ headers=headers,
+ )
+
+ def get_models(
+ self, api_key: Optional[str] = None, api_base: Optional[str] = None
+ ) -> List[str]:
+ api_base, _ = self._get_openai_compatible_provider_info(api_base, api_key)
+
+ if api_base is None:
+ api_base = "https://ai-gateway.vercel.sh/v1"
+
+ models_url = f"{api_base}/models"
+ response = litellm.module_level_client.get(url=models_url)
+
+ if response.status_code != 200:
+ raise Exception(f"Failed to get models: {response.text}")
+
+ models = response.json()["data"]
+ return [model["id"] for model in models]
diff --git a/litellm/llms/vercel_ai_gateway/common_utils.py b/litellm/llms/vercel_ai_gateway/common_utils.py
new file mode 100644
index 00000000000..93e792be05e
--- /dev/null
+++ b/litellm/llms/vercel_ai_gateway/common_utils.py
@@ -0,0 +1,5 @@
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+
+
+class VercelAIGatewayException(BaseLLMException):
+ pass
diff --git a/litellm/llms/vertex_ai/batches/transformation.py b/litellm/llms/vertex_ai/batches/transformation.py
index a97f312d486..22cd0bd402a 100644
--- a/litellm/llms/vertex_ai/batches/transformation.py
+++ b/litellm/llms/vertex_ai/batches/transformation.py
@@ -1,4 +1,4 @@
-import uuid
+from litellm._uuid import uuid
from typing import Dict
from litellm.llms.vertex_ai.common_utils import (
@@ -114,7 +114,14 @@ class VertexAIBatchTransformation:
"""
Gets the output file id from the Vertex AI Batch response
"""
- output_file_id: str = ""
+
+ output_file_id: str = (
+ response.get("outputInfo", OutputInfo()).get("gcsOutputDirectory", "")
+ + "/predictions.jsonl"
+ )
+ if output_file_id != "/predictions.jsonl":
+ return output_file_id
+
output_config = response.get("outputConfig")
if output_config is None:
return output_file_id
diff --git a/litellm/llms/vertex_ai/common_utils.py b/litellm/llms/vertex_ai/common_utils.py
index cceac0ea794..3e650ecd111 100644
--- a/litellm/llms/vertex_ai/common_utils.py
+++ b/litellm/llms/vertex_ai/common_utils.py
@@ -1,4 +1,5 @@
import re
+from enum import Enum
from typing import Any, Dict, List, Literal, Optional, Set, Tuple, Union, get_type_hints
import httpx
@@ -7,8 +8,11 @@ import litellm
from litellm import supports_response_schema, supports_system_messages, verbose_logger
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH
from litellm.litellm_core_utils.prompt_templates.common_utils import unpack_defs
+from litellm.llms.base_llm.base_utils import BaseLLMModelInfo, BaseTokenCounter
from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.types.llms.openai import AllMessageValues
from litellm.types.llms.vertex_ai import PartType, Schema
+from litellm.types.utils import TokenCountResponse
class VertexAIError(BaseLLMException):
@@ -21,6 +25,68 @@ class VertexAIError(BaseLLMException):
super().__init__(message=message, status_code=status_code, headers=headers)
+class VertexAIModelRoute(str, Enum):
+ """Enum for Vertex AI model routing"""
+ PARTNER_MODELS = "partner_models"
+ GEMINI = "gemini"
+ GEMMA = "gemma"
+ MODEL_GARDEN = "model_garden"
+ NON_GEMINI = "non_gemini"
+
+
+def get_vertex_ai_model_route(model: str, litellm_params: Optional[dict] = None) -> VertexAIModelRoute:
+ """
+ Determine which handler to use for a Vertex AI model based on the model name.
+
+ Args:
+ model: The model name (e.g., "llama3-405b", "gemini-pro", "gemma/gemma-3-12b-it", "openai/gpt-oss-120b")
+ litellm_params: Optional litellm parameters dict that may contain base_model for routing
+
+ Returns:
+ VertexAIModelRoute: The route enum indicating which handler should be used
+
+ Examples:
+ >>> get_vertex_ai_model_route("llama3-405b")
+ VertexAIModelRoute.PARTNER_MODELS
+
+ >>> get_vertex_ai_model_route("gemini-pro")
+ VertexAIModelRoute.GEMINI
+
+ >>> get_vertex_ai_model_route("gemma/gemma-3-12b-it")
+ VertexAIModelRoute.GEMMA
+
+ >>> get_vertex_ai_model_route("openai/gpt-oss-120b")
+ VertexAIModelRoute.MODEL_GARDEN
+ """
+ from litellm.llms.vertex_ai.vertex_ai_partner_models.main import (
+ VertexAIPartnerModels,
+ )
+
+ # Check base_model in litellm_params for gemini override
+ if litellm_params and litellm_params.get("base_model") is not None:
+ if "gemini" in litellm_params["base_model"]:
+ return VertexAIModelRoute.GEMINI
+
+ # Check for partner models (llama, mistral, claude, etc.)
+ if VertexAIPartnerModels.is_vertex_partner_model(model=model):
+ return VertexAIModelRoute.PARTNER_MODELS
+
+ # Check for gemma models
+ if "gemma/" in model:
+ return VertexAIModelRoute.GEMMA
+
+ # Check for model garden openai models
+ if "openai" in model:
+ return VertexAIModelRoute.MODEL_GARDEN
+
+ # Check for gemini models
+ if "gemini" in model:
+ return VertexAIModelRoute.GEMINI
+
+ # Default to non-gemini (legacy vertex models like chat-bison, text-bison, etc.)
+ return VertexAIModelRoute.NON_GEMINI
+
+
def get_supports_system_message(
model: str, custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"]
) -> bool:
@@ -63,7 +129,7 @@ def get_supports_response_schema(
from typing import Literal, Optional
all_gemini_url_modes = Literal[
- "chat", "embedding", "batch_embedding", "image_generation"
+ "chat", "embedding", "batch_embedding", "image_generation", "count_tokens"
]
@@ -113,6 +179,12 @@ def _get_vertex_url(
url = f"https://{vertex_location}-aiplatform.googleapis.com/v1/projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{model}:{endpoint}"
if model.isdigit():
url = f"https://{vertex_location}-aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/{vertex_location}/endpoints/{model}:{endpoint}"
+ elif mode == "count_tokens":
+ endpoint = "countTokens"
+ if vertex_location == "global":
+ url = f"https://aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/global/publishers/google/models/{model}:{endpoint}"
+ else:
+ url = f"https://{vertex_location}-aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{model}:{endpoint}"
if not url or not endpoint:
raise ValueError(f"Unable to get vertex url/endpoint for mode: {mode}")
return url, endpoint
@@ -148,10 +220,17 @@ def _get_gemini_url(
url = "https://generativelanguage.googleapis.com/v1beta/{}:{}?key={}".format(
_gemini_model_name, endpoint, gemini_api_key
)
+ elif mode == "count_tokens":
+ endpoint = "countTokens"
+ url = "https://generativelanguage.googleapis.com/v1beta/{}:{}?key={}".format(
+ _gemini_model_name, endpoint, gemini_api_key
+ )
elif mode == "image_generation":
raise ValueError(
"LiteLLM's `gemini/` route does not support image generation yet. Let us know if you need this feature by opening an issue at https://github.com/BerriAI/litellm/issues"
)
+ else:
+ raise ValueError(f"Unsupported mode: {mode}")
return url, endpoint
@@ -171,6 +250,25 @@ def _check_text_in_content(parts: List[PartType]) -> bool:
return has_text_param
+def _fix_enum_empty_strings(schema, depth=0):
+ """Fix empty strings in enum values by replacing them with None. Gemini doesn't accept empty strings in enums."""
+ if depth > DEFAULT_MAX_RECURSE_DEPTH:
+ raise ValueError(f"Max depth of {DEFAULT_MAX_RECURSE_DEPTH} exceeded while processing schema.")
+
+ if "enum" in schema and isinstance(schema["enum"], list):
+ schema["enum"] = [None if value == "" else value for value in schema["enum"]]
+
+ # Reuse existing recursion pattern from convert_anyof_null_to_nullable
+ properties = schema.get("properties", None)
+ if properties is not None:
+ for _, value in properties.items():
+ _fix_enum_empty_strings(value, depth=depth + 1)
+
+ items = schema.get("items", None)
+ if items is not None:
+ _fix_enum_empty_strings(items, depth=depth + 1)
+
+
def _build_vertex_schema(parameters: dict, add_property_ordering: bool = False):
"""
This is a modified version of https://github.com/google-gemini/generative-ai-python/blob/8f77cc6ac99937cd3a81299ecf79608b91b06bbb/google/generativeai/types/content_types.py#L419
@@ -199,6 +297,11 @@ def _build_vertex_schema(parameters: dict, add_property_ordering: bool = False):
# * https://github.com/pydantic/pydantic/discussions/4872
convert_anyof_null_to_nullable(parameters)
+ _convert_schema_types(parameters)
+
+ # Handle empty strings in enum values - Gemini doesn't accept empty strings in enums
+ _fix_enum_empty_strings(parameters)
+
# Handle empty items objects
process_items(parameters)
add_object_type(parameters)
@@ -238,9 +341,7 @@ def _filter_anyof_fields(schema_dict: Dict[str, Any]) -> Dict[str, Any]:
item["title"] = title
if description:
item["description"] = description
- return {"anyOf": any_of}
- else:
- return schema_dict
+ return {"anyOf": any_of}
return schema_dict
@@ -425,6 +526,47 @@ def _convert_vertex_datetime_to_openai_datetime(vertex_datetime: str) -> int:
return int(dt.timestamp())
+def _convert_schema_types(schema, depth=0):
+ """
+ Convert type arrays and lowercase types for Vertex AI compatibility.
+
+ Transforms OpenAI-style schemas to Vertex AI format by converting type arrays
+ like ["string", "number"] to anyOf format and converting all types to uppercase.
+ """
+ if depth > DEFAULT_MAX_RECURSE_DEPTH:
+ raise ValueError(
+ f"Max depth of {DEFAULT_MAX_RECURSE_DEPTH} exceeded while processing schema. Please check the schema for excessive nesting."
+ )
+
+ if not isinstance(schema, dict):
+ return
+
+
+ # Handle type field
+ if "type" in schema:
+ type_val = schema["type"]
+ if isinstance(type_val, list) and len(type_val) > 1:
+ # Convert ["string", "number"] -> {"anyOf": [{"type": "STRING"}, {"type": "NUMBER"}]}
+ schema["anyOf"] = [{"type": t} for t in type_val if isinstance(t, str)]
+ schema.pop("type")
+ elif isinstance(type_val, list) and len(type_val) == 1:
+ schema["type"] = type_val[0]
+ elif isinstance(type_val, str):
+ schema["type"] = type_val
+
+ # Recursively process nested properties, items, and anyOf
+ for key in ["properties", "items", "anyOf"]:
+ if key in schema:
+ value = schema[key]
+ if key == "properties" and isinstance(value, dict):
+ for prop_schema in value.values():
+ _convert_schema_types(prop_schema, depth + 1)
+ elif key == "items":
+ _convert_schema_types(value, depth + 1)
+ elif key == "anyOf" and isinstance(value, list):
+ for anyof_schema in value:
+ _convert_schema_types(anyof_schema, depth + 1)
+
def get_vertex_project_id_from_url(url: str) -> Optional[str]:
"""
Get the vertex project id from the url
@@ -522,3 +664,99 @@ def is_global_only_vertex_model(model: str) -> bool:
if supported_regions is None:
return False
return "global" in supported_regions
+
+class VertexAIModelInfo(BaseLLMModelInfo):
+ def get_token_counter(self) -> Optional[BaseTokenCounter]:
+ """
+ Factory method to create a token counter for this provider.
+
+ Returns:
+ Optional TokenCounterInterface implementation for this provider,
+ or None if token counting is not supported.
+ """
+ return VertexAITokenCounter()
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ raise NotImplementedError("Vertex AI models are not supported yet")
+
+ def get_models(
+ self, api_key: Optional[str] = None, api_base: Optional[str] = None
+ ) -> List[str]:
+ """
+ Returns a list of models supported by this provider.
+ """
+ raise NotImplementedError("Vertex AI models are not supported yet")
+
+ @staticmethod
+ def get_api_key(api_key: Optional[str] = None) -> Optional[str]:
+ raise NotImplementedError("Vertex AI models are not supported yet")
+
+ @staticmethod
+ def get_api_base(
+ api_base: Optional[str] = None,
+ ) -> Optional[str]:
+ raise NotImplementedError("Vertex AI models are not supported yet")
+
+
+
+ @staticmethod
+ def get_base_model(model: str) -> Optional[str]:
+ """
+ Returns the base model name from the given model name.
+
+ Some providers like bedrock - can receive model=`invoke/anthropic.claude-3-opus-20240229-v1:0` or `converse/anthropic.claude-3-opus-20240229-v1:0`
+ This function will return `anthropic.claude-3-opus-20240229-v1:0`
+ """
+ raise NotImplementedError("Vertex AI models are not supported yet")
+
+
+class VertexAITokenCounter(BaseTokenCounter):
+ """Token counter implementation for Google AI Studio provider."""
+ def should_use_token_counting_api(
+ self,
+ custom_llm_provider: Optional[str] = None,
+ ) -> bool:
+ from litellm.types.utils import LlmProviders
+ return custom_llm_provider == LlmProviders.VERTEX_AI.value
+
+ async def count_tokens(
+ self,
+ model_to_use: str,
+ messages: Optional[List[Dict[str, Any]]],
+ contents: Optional[List[Dict[str, Any]]],
+ deployment: Optional[Dict[str, Any]] = None,
+ request_model: str = "",
+ ) -> Optional[TokenCountResponse]:
+ import copy
+
+ from litellm.llms.vertex_ai.count_tokens.handler import VertexAITokenCounter
+ deployment = deployment or {}
+ count_tokens_params_request = copy.deepcopy(deployment.get("litellm_params", {}))
+ count_tokens_params = {
+ "model": model_to_use,
+ "contents": contents,
+ }
+ count_tokens_params_request.update(count_tokens_params)
+ result = await VertexAITokenCounter().acount_tokens(
+ **count_tokens_params_request,
+ )
+
+ if result is not None:
+ return TokenCountResponse(
+ total_tokens=result.get("totalTokens", 0),
+ request_model=request_model,
+ model_used=model_to_use,
+ tokenizer_type=result.get("tokenizer_used", ""),
+ original_response=result,
+ )
+
+ return None
\ No newline at end of file
diff --git a/litellm/llms/vertex_ai/context_caching/transformation.py b/litellm/llms/vertex_ai/context_caching/transformation.py
index f3ca699546f..bb40b7665c1 100644
--- a/litellm/llms/vertex_ai/context_caching/transformation.py
+++ b/litellm/llms/vertex_ai/context_caching/transformation.py
@@ -5,7 +5,7 @@ Why separate file? Make it easy to see how transformation works
"""
import re
-from typing import List, Optional, Tuple
+from typing import List, Optional, Tuple, Literal
from litellm.types.llms.openai import AllMessageValues
from litellm.types.llms.vertex_ai import CachedContentRequestBody
@@ -155,13 +155,18 @@ def separate_cached_messages(
def transform_openai_messages_to_gemini_context_caching(
- model: str, messages: List[AllMessageValues], cache_key: str
+ model: str,
+ messages: List[AllMessageValues],
+ custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
+ cache_key: str,
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
) -> CachedContentRequestBody:
# Extract TTL from cached messages BEFORE system message transformation
ttl = extract_ttl_from_cached_messages(messages)
supports_system_message = get_supports_system_message(
- model=model, custom_llm_provider="gemini"
+ model=model, custom_llm_provider=custom_llm_provider
)
transformed_system_messages, new_messages = _transform_system_message(
@@ -170,9 +175,14 @@ def transform_openai_messages_to_gemini_context_caching(
transformed_messages = _gemini_convert_messages_with_history(messages=new_messages)
+ model_name = "models/{}".format(model)
+
+ if custom_llm_provider == "vertex_ai" or custom_llm_provider == "vertex_ai_beta":
+ model_name = f"projects/{vertex_project}/locations/{vertex_location}/publishers/google/{model_name}"
+
data = CachedContentRequestBody(
contents=transformed_messages,
- model="models/{}".format(model),
+ model=model_name,
displayName=cache_key,
)
diff --git a/litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py b/litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
index 33a480aa6bb..70b068b5a4d 100644
--- a/litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
+++ b/litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py
@@ -41,8 +41,11 @@ class ContextCachingEndpoints(VertexBase):
def _get_token_and_url_context_caching(
self,
gemini_api_key: Optional[str],
- custom_llm_provider: Literal["gemini"],
+ custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
api_base: Optional[str],
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ vertex_auth_header: Optional[str],
) -> Tuple[Optional[str], str]:
"""
Internal function. Returns the token and url for the call.
@@ -58,9 +61,15 @@ class ContextCachingEndpoints(VertexBase):
url = "https://generativelanguage.googleapis.com/v1beta/{}?key={}".format(
endpoint, gemini_api_key
)
-
+ elif custom_llm_provider == "vertex_ai":
+ auth_header = vertex_auth_header
+ endpoint = "cachedContents"
+ url = f"https://{vertex_location}-aiplatform.googleapis.com/v1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
else:
- raise NotImplementedError
+ auth_header = vertex_auth_header
+ endpoint = "cachedContents"
+ url = f"https://{vertex_location}-aiplatform.googleapis.com/v1beta1/projects/{vertex_project}/locations/{vertex_location}/{endpoint}"
+
return self._check_custom_proxy(
api_base=api_base,
@@ -80,6 +89,10 @@ class ContextCachingEndpoints(VertexBase):
api_key: str,
api_base: Optional[str],
logging_obj: Logging,
+ custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ vertex_auth_header: Optional[str],
) -> Optional[str]:
"""
Checks if content already cached.
@@ -94,8 +107,11 @@ class ContextCachingEndpoints(VertexBase):
_, url = self._get_token_and_url_context_caching(
gemini_api_key=api_key,
- custom_llm_provider="gemini",
+ custom_llm_provider=custom_llm_provider,
api_base=api_base,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=vertex_auth_header
)
try:
## LOGGING
@@ -145,6 +161,10 @@ class ContextCachingEndpoints(VertexBase):
api_key: str,
api_base: Optional[str],
logging_obj: Logging,
+ custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ vertex_auth_header: Optional[str]
) -> Optional[str]:
"""
Checks if content already cached.
@@ -159,8 +179,11 @@ class ContextCachingEndpoints(VertexBase):
_, url = self._get_token_and_url_context_caching(
gemini_api_key=api_key,
- custom_llm_provider="gemini",
+ custom_llm_provider=custom_llm_provider,
api_base=api_base,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=vertex_auth_header
)
try:
## LOGGING
@@ -212,6 +235,10 @@ class ContextCachingEndpoints(VertexBase):
client: Optional[HTTPHandler],
timeout: Optional[Union[float, httpx.Timeout]],
logging_obj: Logging,
+ custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ vertex_auth_header: Optional[str],
extra_headers: Optional[dict] = None,
cached_content: Optional[str] = None,
) -> Tuple[List[AllMessageValues], dict, Optional[str]]:
@@ -240,8 +267,11 @@ class ContextCachingEndpoints(VertexBase):
## AUTHORIZATION ##
token, url = self._get_token_and_url_context_caching(
gemini_api_key=api_key,
- custom_llm_provider="gemini",
+ custom_llm_provider=custom_llm_provider,
api_base=api_base,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=vertex_auth_header
)
headers = {
@@ -273,6 +303,10 @@ class ContextCachingEndpoints(VertexBase):
api_key=api_key,
api_base=api_base,
logging_obj=logging_obj,
+ custom_llm_provider=custom_llm_provider,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=vertex_auth_header
)
if google_cache_name:
return non_cached_messages, optional_params, google_cache_name
@@ -280,7 +314,12 @@ class ContextCachingEndpoints(VertexBase):
## TRANSFORM REQUEST
cached_content_request_body = (
transform_openai_messages_to_gemini_context_caching(
- model=model, messages=cached_messages, cache_key=generated_cache_key
+ model=model,
+ messages=cached_messages,
+ cache_key=generated_cache_key,
+ custom_llm_provider=custom_llm_provider,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
)
)
@@ -328,6 +367,10 @@ class ContextCachingEndpoints(VertexBase):
client: Optional[AsyncHTTPHandler],
timeout: Optional[Union[float, httpx.Timeout]],
logging_obj: Logging,
+ custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ vertex_auth_header: Optional[str],
extra_headers: Optional[dict] = None,
cached_content: Optional[str] = None,
) -> Tuple[List[AllMessageValues], dict, Optional[str]]:
@@ -356,8 +399,11 @@ class ContextCachingEndpoints(VertexBase):
## AUTHORIZATION ##
token, url = self._get_token_and_url_context_caching(
gemini_api_key=api_key,
- custom_llm_provider="gemini",
+ custom_llm_provider=custom_llm_provider,
api_base=api_base,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=vertex_auth_header
)
headers = {
@@ -386,6 +432,10 @@ class ContextCachingEndpoints(VertexBase):
api_key=api_key,
api_base=api_base,
logging_obj=logging_obj,
+ custom_llm_provider=custom_llm_provider,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=vertex_auth_header
)
if google_cache_name:
@@ -394,7 +444,12 @@ class ContextCachingEndpoints(VertexBase):
## TRANSFORM REQUEST
cached_content_request_body = (
transform_openai_messages_to_gemini_context_caching(
- model=model, messages=cached_messages, cache_key=generated_cache_key
+ model=model,
+ messages=cached_messages,
+ cache_key=generated_cache_key,
+ custom_llm_provider=custom_llm_provider,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
)
)
diff --git a/litellm/llms/vertex_ai/cost_calculator.py b/litellm/llms/vertex_ai/cost_calculator.py
index 119ba2b0366..e98dc75915d 100644
--- a/litellm/llms/vertex_ai/cost_calculator.py
+++ b/litellm/llms/vertex_ai/cost_calculator.py
@@ -44,6 +44,7 @@ def cost_router(
or "mistral" in model
or "jamba" in model
or "codestral" in model
+ or "gemma" in model
):
return "cost_per_token"
elif custom_llm_provider == "vertex_ai" and (
diff --git a/litellm/llms/vertex_ai/count_tokens/handler.py b/litellm/llms/vertex_ai/count_tokens/handler.py
new file mode 100644
index 00000000000..d95c6801e57
--- /dev/null
+++ b/litellm/llms/vertex_ai/count_tokens/handler.py
@@ -0,0 +1,46 @@
+from typing import Any, Dict, Optional, Tuple
+
+from litellm.llms.gemini.count_tokens.handler import GoogleAIStudioTokenCounter
+from litellm.llms.vertex_ai.vertex_llm_base import VertexBase
+
+
+class VertexAITokenCounter(GoogleAIStudioTokenCounter, VertexBase):
+ async def validate_environment(
+ self,
+ api_base: Optional[str] = None,
+ api_key: Optional[str] = None,
+ headers: Optional[Dict[str, Any]] = None,
+ model: str = "",
+ litellm_params: Optional[Dict[str, Any]] = None,
+ ) -> Tuple[Dict[str, Any], str]:
+ """
+ Returns a Tuple of headers and url for the Vertex AI countTokens endpoint.
+ """
+ litellm_params = litellm_params or {}
+ vertex_credentials = self.get_vertex_ai_credentials(litellm_params=litellm_params)
+ vertex_project = self.get_vertex_ai_project(litellm_params=litellm_params)
+ vertex_location = self.get_vertex_ai_location(litellm_params=litellm_params)
+ should_use_v1beta1_features = self.is_using_v1beta1_features(litellm_params)
+ _auth_header, vertex_project = await self._ensure_access_token_async(
+ credentials=vertex_credentials,
+ project_id=vertex_project,
+ custom_llm_provider="vertex_ai",
+ )
+
+ auth_header, api_base = self._get_token_and_url(
+ model=model,
+ gemini_api_key=None,
+ auth_header=_auth_header,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_credentials=vertex_credentials,
+ stream=False,
+ custom_llm_provider="vertex_ai",
+ api_base=None,
+ should_use_v1beta1_features=should_use_v1beta1_features,
+ mode="count_tokens",
+ )
+ headers = {
+ "Authorization": f"Bearer {auth_header}",
+ }
+ return headers, api_base
\ No newline at end of file
diff --git a/litellm/llms/vertex_ai/files/handler.py b/litellm/llms/vertex_ai/files/handler.py
index a666a2c37fb..6636bccd6a3 100644
--- a/litellm/llms/vertex_ai/files/handler.py
+++ b/litellm/llms/vertex_ai/files/handler.py
@@ -1,5 +1,6 @@
import asyncio
-from typing import Any, Coroutine, Optional, Union
+import urllib.parse
+from typing import Any, Coroutine, Optional, Tuple, Union
import httpx
@@ -9,7 +10,12 @@ from litellm.integrations.gcs_bucket.gcs_bucket_base import (
GCSLoggingConfig,
)
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
-from litellm.types.llms.openai import CreateFileRequest, OpenAIFileObject
+from litellm.types.llms.openai import (
+ CreateFileRequest,
+ FileContentRequest,
+ HttpxBinaryResponseContent,
+ OpenAIFileObject,
+)
from litellm.types.llms.vertex_ai import VERTEX_CREDENTIALS_TYPES
from .transformation import VertexAIJsonlFilesTransformation
@@ -105,3 +111,136 @@ class VertexAIFilesHandler(GCSBucketBase):
max_retries=max_retries,
)
)
+
+ def _extract_bucket_and_object_from_file_id(self, file_id: str) -> Tuple[str, str]:
+ """
+ Extract bucket name and object path from URL-encoded file_id.
+
+ Expected format: gs%3A%2F%2Fbucket-name%2Fpath%2Fto%2Ffile
+ Which decodes to: gs://bucket-name/path/to/file
+
+ Returns:
+ tuple: (bucket_name, url_encoded_object_path)
+ - bucket_name: "bucket-name"
+ - url_encoded_object_path: "path%2Fto%2Ffile"
+ """
+ decoded_path = urllib.parse.unquote(file_id)
+
+ if decoded_path.startswith("gs://"):
+ full_path = decoded_path[5:] # Remove 'gs://' prefix
+ else:
+ full_path = decoded_path
+
+ if "/" in full_path:
+ bucket_name, object_path = full_path.split("/", 1)
+ else:
+ bucket_name = full_path
+ object_path = ""
+
+ encoded_object_path = urllib.parse.quote(object_path, safe="")
+
+ return bucket_name, encoded_object_path
+
+ async def afile_content(
+ self,
+ file_content_request: FileContentRequest,
+ vertex_credentials: Optional[VERTEX_CREDENTIALS_TYPES],
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ timeout: Union[float, httpx.Timeout],
+ max_retries: Optional[int],
+ ) -> HttpxBinaryResponseContent:
+ """
+ Download file content from GCS bucket for VertexAI files.
+
+ Args:
+ file_content_request: Contains file_id (URL-encoded GCS path)
+ vertex_credentials: VertexAI credentials
+ vertex_project: VertexAI project ID
+ vertex_location: VertexAI location
+ timeout: Request timeout
+ max_retries: Max retry attempts
+
+ Returns:
+ HttpxBinaryResponseContent: Binary content wrapped in compatible response format
+ """
+ file_id = file_content_request.get("file_id")
+ if not file_id:
+ raise ValueError("file_id is required in file_content_request")
+
+ bucket_name, encoded_object_path = self._extract_bucket_and_object_from_file_id(
+ file_id
+ )
+
+ download_kwargs = {
+ "standard_callback_dynamic_params": {"gcs_bucket_name": bucket_name}
+ }
+
+ file_content = await self.download_gcs_object(
+ object_name=encoded_object_path, **download_kwargs
+ )
+
+ if file_content is None:
+ decoded_path = urllib.parse.unquote(file_id)
+ raise ValueError(f"Failed to download file from GCS: {decoded_path}")
+
+ decoded_path = urllib.parse.unquote(file_id)
+ mock_response = httpx.Response(
+ status_code=200,
+ content=file_content,
+ headers={"content-type": "application/octet-stream"},
+ request=httpx.Request(method="GET", url=decoded_path),
+ )
+
+ return HttpxBinaryResponseContent(response=mock_response)
+
+ def file_content(
+ self,
+ _is_async: bool,
+ file_content_request: FileContentRequest,
+ api_base: Optional[str],
+ vertex_credentials: Optional[VERTEX_CREDENTIALS_TYPES],
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ timeout: Union[float, httpx.Timeout],
+ max_retries: Optional[int],
+ ) -> Union[
+ HttpxBinaryResponseContent, Coroutine[Any, Any, HttpxBinaryResponseContent]
+ ]:
+ """
+ Download file content from GCS bucket for VertexAI files.
+ Supports both sync and async operations.
+
+ Args:
+ _is_async: Whether to run asynchronously
+ file_content_request: Contains file_id (URL-encoded GCS path)
+ api_base: API base (unused for GCS operations)
+ vertex_credentials: VertexAI credentials
+ vertex_project: VertexAI project ID
+ vertex_location: VertexAI location
+ timeout: Request timeout
+ max_retries: Max retry attempts
+
+ Returns:
+ HttpxBinaryResponseContent or Coroutine: Binary content wrapped in compatible response format
+ """
+ if _is_async:
+ return self.afile_content(
+ file_content_request=file_content_request,
+ vertex_credentials=vertex_credentials,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ timeout=timeout,
+ max_retries=max_retries,
+ )
+ else:
+ return asyncio.run(
+ self.afile_content(
+ file_content_request=file_content_request,
+ vertex_credentials=vertex_credentials,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ timeout=timeout,
+ max_retries=max_retries,
+ )
+ )
diff --git a/litellm/llms/vertex_ai/files/transformation.py b/litellm/llms/vertex_ai/files/transformation.py
index c795367e486..01f6c86fd4d 100644
--- a/litellm/llms/vertex_ai/files/transformation.py
+++ b/litellm/llms/vertex_ai/files/transformation.py
@@ -1,11 +1,12 @@
import json
import os
import time
-import uuid
+from litellm._uuid import uuid
from typing import Any, Dict, List, Optional, Tuple, Union
from httpx import Headers, Response
+from litellm.files.utils import FilesAPIUtils
from litellm.litellm_core_utils.prompt_templates.common_utils import extract_file_data
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.base_llm.files.transformation import (
@@ -260,10 +261,13 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
raise ValueError("file is required")
extracted_file_data = extract_file_data(file_data)
extracted_file_data_content = extracted_file_data.get("content")
- if (
- create_file_data.get("purpose") == "batch"
- and extracted_file_data.get("content_type") == "application/jsonl"
- and extracted_file_data_content is not None
+
+ if extracted_file_data_content is None:
+ raise ValueError("file content is required")
+
+ if FilesAPIUtils.is_batch_jsonl_file(
+ create_file_data=create_file_data,
+ extracted_file_data=extracted_file_data,
):
## 1. If jsonl, check if there's a model name
file_content = self._get_content_from_openai_file(
@@ -279,7 +283,7 @@ class VertexAIFilesConfig(VertexBase, BaseFilesConfig):
openai_jsonl_content
)
)
- return json.dumps(vertex_jsonl_content)
+ return "\n".join(json.dumps(item) for item in vertex_jsonl_content)
elif isinstance(extracted_file_data_content, bytes):
return extracted_file_data_content
else:
diff --git a/litellm/llms/vertex_ai/fine_tuning/handler.py b/litellm/llms/vertex_ai/fine_tuning/handler.py
index 4d7f8cec02d..6372f8ea305 100644
--- a/litellm/llms/vertex_ai/fine_tuning/handler.py
+++ b/litellm/llms/vertex_ai/fine_tuning/handler.py
@@ -64,9 +64,9 @@ class VertexFineTuningAPI(VertexLLM):
)
if create_fine_tuning_job_data.validation_file:
- supervised_tuning_spec[
- "validation_dataset"
- ] = create_fine_tuning_job_data.validation_file
+ supervised_tuning_spec["validation_dataset"] = (
+ create_fine_tuning_job_data.validation_file
+ )
_vertex_hyperparameters = (
self._transform_openai_hyperparameters_to_vertex_hyperparameters(
@@ -140,7 +140,9 @@ class VertexFineTuningAPI(VertexLLM):
fine_tuned_model=response.get("tunedModelDisplayName", ""),
finished_at=None,
hyperparameters=self._translate_vertex_response_hyperparameters(
- vertex_hyper_parameters=_supervisedTuningSpec.get("hyperParameters", {})
+ vertex_hyper_parameters=_supervisedTuningSpec.get(
+ "hyperParameters", FineTuneHyperparameters()
+ )
or {}
),
model=response.get("baseModel", "") or "",
@@ -343,9 +345,9 @@ class VertexFineTuningAPI(VertexLLM):
elif "cachedContents" in request_route:
_model = request_data.get("model")
if _model is not None and "/publishers/google/models/" not in _model:
- request_data[
- "model"
- ] = f"projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{_model}"
+ request_data["model"] = (
+ f"projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{_model}"
+ )
url = f"https://{vertex_location}-aiplatform.googleapis.com/v1beta1/projects/{vertex_project}/locations/{vertex_location}{request_route}"
else:
diff --git a/litellm/llms/vertex_ai/gemini/transformation.py b/litellm/llms/vertex_ai/gemini/transformation.py
index 85e3f15364b..3d313456d19 100644
--- a/litellm/llms/vertex_ai/gemini/transformation.py
+++ b/litellm/llms/vertex_ai/gemini/transformation.py
@@ -28,6 +28,7 @@ from litellm.types.files import (
get_file_type_from_extension,
is_gemini_1_5_accepted_file_type,
)
+from litellm.types.utils import LlmProviders
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionAssistantMessage,
@@ -35,6 +36,7 @@ from litellm.types.llms.openai import (
ChatCompletionFileObject,
ChatCompletionImageObject,
ChatCompletionTextObject,
+ ChatCompletionUserMessage,
)
from litellm.types.llms.vertex_ai import *
from litellm.types.llms.vertex_ai import (
@@ -104,6 +106,64 @@ def _process_gemini_image(image_url: str, format: Optional[str] = None) -> PartT
raise e
+def _snake_to_camel(snake_str: str) -> str:
+ """Convert snake_case to camelCase"""
+ components = snake_str.split("_")
+ return components[0] + "".join(x.capitalize() for x in components[1:])
+
+
+def _camel_to_snake(camel_str: str) -> str:
+ """Convert camelCase to snake_case"""
+ import re
+
+ return re.sub(r"(? Optional[str]:
+ """
+ Get the equivalent key from available keys, checking both camelCase and snake_case variants
+ """
+ if key in available_keys:
+ return key
+
+ # Try camelCase version
+ camel_key = _snake_to_camel(key)
+ if camel_key in available_keys:
+ return camel_key
+
+ # Try snake_case version
+ snake_key = _camel_to_snake(key)
+ if snake_key in available_keys:
+ return snake_key
+
+ return None
+
+
+def check_if_part_exists_in_parts(
+ parts: List[PartType], part: PartType, excluded_keys: List[str] = []
+) -> bool:
+ """
+ Check if a part exists in a list of parts
+ Handles both camelCase and snake_case key variations (e.g., function_call vs functionCall)
+ """
+ keys_to_compare = set(part.keys()) - set(excluded_keys)
+ for p in parts:
+ p_keys = set(p.keys())
+ # Check if all keys in part have equivalent values in p
+ match_found = True
+ for key in keys_to_compare:
+ equivalent_key = _get_equivalent_key(key, p_keys)
+ if equivalent_key is None or p.get(equivalent_key, None) != part.get(
+ key, None
+ ):
+ match_found = False
+ break
+
+ if match_found:
+ return True
+ return False
+
+
def _gemini_convert_messages_with_history( # noqa: PLR0915
messages: List[AllMessageValues],
) -> List[ContentType]:
@@ -235,10 +295,33 @@ def _gemini_convert_messages_with_history( # noqa: PLR0915
assistant_msg = ChatCompletionAssistantMessage(**msg_dict) # type: ignore
_message_content = assistant_msg.get("content", None)
reasoning_content = assistant_msg.get("reasoning_content", None)
+ thinking_blocks = assistant_msg.get("thinking_blocks")
if reasoning_content is not None:
assistant_content.append(
PartType(thought=True, text=reasoning_content)
)
+ if thinking_blocks is not None:
+ for block in thinking_blocks:
+ block_thinking_str = block.get("thinking")
+ block_signature = block.get("signature")
+ if (
+ block_thinking_str is not None
+ and block_signature is not None
+ ):
+ try:
+ assistant_content.append(
+ PartType(
+ thoughtSignature=block_signature,
+ **json.loads(block_thinking_str),
+ )
+ )
+ except Exception:
+ assistant_content.append(
+ PartType(
+ thoughtSignature=block_signature,
+ text=block_thinking_str,
+ )
+ )
if _message_content is not None and isinstance(_message_content, list):
_parts = []
for element in _message_content:
@@ -261,9 +344,17 @@ def _gemini_convert_messages_with_history( # noqa: PLR0915
assistant_msg.get("tool_calls", []) is not None
or assistant_msg.get("function_call") is not None
): # support assistant tool invoke conversion
- assistant_content.extend(
- convert_to_gemini_tool_call_invoke(assistant_msg)
+ gemini_tool_call_parts = convert_to_gemini_tool_call_invoke(
+ assistant_msg
)
+ ## check if gemini_tool_call already exists in assistant_content
+ for gemini_tool_call_part in gemini_tool_call_parts:
+ if not check_if_part_exists_in_parts(
+ assistant_content,
+ gemini_tool_call_part,
+ excluded_keys=["thoughtSignature"],
+ ):
+ assistant_content.append(gemini_tool_call_part)
last_message_with_tool_calls = assistant_msg
msg_i += 1
@@ -297,6 +388,19 @@ def _gemini_convert_messages_with_history( # noqa: PLR0915
)
if len(tool_call_responses) > 0:
contents.append(ContentType(parts=tool_call_responses))
+
+ if len(contents) == 0:
+ verbose_logger.warning(
+ """
+ No contents in messages. Contents are required. See
+ https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.publishers.models/generateContent#request-body.
+ If the original request did not comply to OpenAI API requirements it should have failed by now,
+ but LiteLLM does not check for missing messages.
+ Setting an empty content to prevent an 400 error.
+ Relevant Issue - https://github.com/BerriAI/litellm/issues/9733
+ """
+ )
+ contents.append(ContentType(role="user", parts=[PartType(text=" ")]))
return contents
except Exception as e:
raise e
@@ -358,6 +462,17 @@ def _transform_request_body(
) # type: ignore
config_fields = GenerationConfig.__annotations__.keys()
+ # If the LiteLLM client sends Gemini-supported parameter "labels", add it
+ # as "labels" field to the request sent to the Gemini backend.
+ labels: Optional[dict[str, str]] = optional_params.pop("labels", None)
+ # If the LiteLLM client sends OpenAI-supported parameter "metadata", add it
+ # as "labels" field to the request sent to the Gemini backend.
+ if labels is None and "metadata" in litellm_params:
+ metadata = litellm_params["metadata"]
+ if metadata is not None and "requester_metadata" in metadata:
+ rm = metadata["requester_metadata"]
+ labels = {k: v for k, v in rm.items() if isinstance(v, str)}
+
filtered_params = {
k: v for k, v in optional_params.items() if k in config_fields
}
@@ -378,6 +493,9 @@ def _transform_request_body(
data["generationConfig"] = generation_config
if cached_content is not None:
data["cachedContent"] = cached_content
+ # Only add labels for Vertex AI endpoints (not Google GenAI/AI Studio) and only if non-empty
+ if labels and custom_llm_provider != LlmProviders.GEMINI:
+ data["labels"] = labels
except Exception as e:
raise e
@@ -396,28 +514,35 @@ def sync_transform_request_body(
logging_obj: LiteLLMLoggingObj,
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
litellm_params: dict,
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ vertex_auth_header: Optional[str],
) -> RequestBody:
from ..context_caching.vertex_ai_context_caching import ContextCachingEndpoints
context_caching_endpoints = ContextCachingEndpoints()
- if gemini_api_key is not None:
- messages, optional_params, cached_content = (
- context_caching_endpoints.check_and_create_cache(
- messages=messages,
- optional_params=optional_params,
- api_key=gemini_api_key,
- api_base=api_base,
- model=model,
- client=client,
- timeout=timeout,
- extra_headers=extra_headers,
- cached_content=optional_params.pop("cached_content", None),
- logging_obj=logging_obj,
- )
- )
- else: # [TODO] implement context caching for gemini as well
- cached_content = optional_params.pop("cached_content", None)
+ (
+ messages,
+ optional_params,
+ cached_content,
+ ) = context_caching_endpoints.check_and_create_cache(
+ messages=messages,
+ optional_params=optional_params,
+ api_key=gemini_api_key or "dummy",
+ api_base=api_base,
+ model=model,
+ client=client,
+ timeout=timeout,
+ extra_headers=extra_headers,
+ cached_content=optional_params.pop("cached_content", None),
+ logging_obj=logging_obj,
+ custom_llm_provider=custom_llm_provider,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=vertex_auth_header,
+ )
+
return _transform_request_body(
messages=messages,
@@ -441,30 +566,34 @@ async def async_transform_request_body(
logging_obj: litellm.litellm_core_utils.litellm_logging.Logging, # type: ignore
custom_llm_provider: Literal["vertex_ai", "vertex_ai_beta", "gemini"],
litellm_params: dict,
+ vertex_project: Optional[str],
+ vertex_location: Optional[str],
+ vertex_auth_header: Optional[str],
) -> RequestBody:
from ..context_caching.vertex_ai_context_caching import ContextCachingEndpoints
context_caching_endpoints = ContextCachingEndpoints()
- if gemini_api_key is not None:
- (
- messages,
- optional_params,
- cached_content,
- ) = await context_caching_endpoints.async_check_and_create_cache(
- messages=messages,
- optional_params=optional_params,
- api_key=gemini_api_key,
- api_base=api_base,
- model=model,
- client=client,
- timeout=timeout,
- extra_headers=extra_headers,
- cached_content=optional_params.pop("cached_content", None),
- logging_obj=logging_obj,
- )
- else: # [TODO] implement context caching for gemini as well
- cached_content = optional_params.pop("cached_content", None)
+ (
+ messages,
+ optional_params,
+ cached_content,
+ ) = await context_caching_endpoints.async_check_and_create_cache(
+ messages=messages,
+ optional_params=optional_params,
+ api_key=gemini_api_key or "dummy",
+ api_base=api_base,
+ model=model,
+ client=client,
+ timeout=timeout,
+ extra_headers=extra_headers,
+ cached_content=optional_params.pop("cached_content", None),
+ logging_obj=logging_obj,
+ custom_llm_provider=custom_llm_provider,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=vertex_auth_header,
+ )
return _transform_request_body(
messages=messages,
@@ -476,6 +605,15 @@ async def async_transform_request_body(
)
+def _default_user_message_when_system_message_passed() -> ChatCompletionUserMessage:
+ """
+ Returns a default user message when a "system" message is passed in gemini fails.
+
+ This adds a blank user message to the messages list, to ensure that gemini doesn't fail the request.
+ """
+ return ChatCompletionUserMessage(content=".", role="user")
+
+
def _transform_system_message(
supports_system_message: bool, messages: List[AllMessageValues]
) -> Tuple[Optional[SystemInstructions], List[AllMessageValues]]:
@@ -510,6 +648,13 @@ def _transform_system_message(
messages.pop(idx)
if len(system_content_blocks) > 0:
+ #########################################################
+ # If no messages are passed in, add a blank user message
+ # Relevant Issue - https://github.com/BerriAI/litellm/issues/13769
+ #########################################################
+ if len(messages) == 0:
+ messages.append(_default_user_message_when_system_message_passed())
+ #########################################################
return SystemInstructions(parts=system_content_blocks), messages
return None, messages
diff --git a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
index 37a4ab84dda..cd7ebaca790 100644
--- a/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
+++ b/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
@@ -3,7 +3,6 @@
## Initial implementation - covers gemini + image gen calls
import json
import time
-import uuid
from copy import deepcopy
from functools import partial
from typing import (
@@ -25,11 +24,16 @@ import litellm
import litellm.litellm_core_utils
import litellm.litellm_core_utils.litellm_logging
from litellm import verbose_logger
+from litellm._uuid import uuid
from litellm.constants import (
DEFAULT_REASONING_EFFORT_DISABLE_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET,
DEFAULT_REASONING_EFFORT_MEDIUM_THINKING_BUDGET,
+ DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET,
+ DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH,
+ DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH_LITE,
+ DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_PRO,
)
from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
from litellm.llms.custom_httpx.http_handler import (
@@ -43,9 +47,12 @@ from litellm.types.llms.gemini import BidiGenerateContentServerMessage
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionResponseMessage,
+ ChatCompletionThinkingBlock,
ChatCompletionToolCallChunk,
ChatCompletionToolCallFunctionChunk,
ChatCompletionToolParamFunctionChunk,
+ ImageURLListItem,
+ ImageURLObject,
OpenAIChatCompletionFinishReason,
)
from litellm.types.llms.vertex_ai import (
@@ -61,6 +68,7 @@ from litellm.types.llms.vertex_ai import (
ToolConfig,
Tools,
UsageMetadata,
+ VertexToolName,
)
from litellm.types.utils import (
ChatCompletionAudioResponse,
@@ -89,11 +97,12 @@ from .transformation import (
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
- from litellm.types.utils import ModelResponseStream
+ from litellm.types.utils import ModelResponseStream, StreamingChoices
LoggingClass = LiteLLMLoggingObj
else:
LoggingClass = Any
+ StreamingChoices = Any
class VertexAIBaseConfig:
@@ -268,42 +277,106 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
"""
return Tools(googleSearch={})
- def _map_function(self, value: List[dict]) -> List[Tools]: # noqa: PLR0915
+ def _extract_google_maps_retrieval_config(
+ self, google_maps_config: dict
+ ) -> Tuple[dict, Optional[dict]]:
+ """
+ Extract location configuration from googleMaps tool for Vertex AI toolConfig.
+
+ Supports two interface styles:
+ 1. Nested (recommended): {"enableWidget": "...", "retrievalConfig": {"latitude": ..., "longitude": ...}}
+ 2. Flat (backward compat): {"enableWidget": "...", "latitude": ..., "longitude": ...}
+
+ Args:
+ google_maps_config: The googleMaps tool configuration from LiteLLM
+
+ Returns:
+ Tuple of (cleaned_google_maps_config, retrieval_config):
+ - cleaned_google_maps_config: googleMaps config without location fields
+ - retrieval_config: Location config for toolConfig.retrievalConfig or None
+ """
+ retrieval_config = None
+ latitude = google_maps_config.get("latitude")
+ longitude = google_maps_config.get("longitude")
+ language_code = google_maps_config.get("languageCode")
+
+ if latitude is not None and longitude is not None:
+ retrieval_config = {
+ "latLng": {
+ "latitude": latitude,
+ "longitude": longitude,
+ }
+ }
+ if language_code is not None:
+ retrieval_config["languageCode"] = language_code
+
+ # Remove location fields from tool definition
+ cleaned_config = {
+ k: v
+ for k, v in google_maps_config.items()
+ if k not in ["latitude", "longitude", "languageCode"]
+ }
+
+ return cleaned_config, retrieval_config
+
+ def get_tool_value(
+ self,
+ tool: dict,
+ tool_name: str
+ ) -> Optional[dict]:
+ """
+ Helper function to get tool value handling both camelCase and underscore_case variants
+
+ Args:
+ tool (dict): The tool dictionary
+ tool_name (str): The base tool name (e.g. "codeExecution")
+
+ Returns:
+ Optional[dict]: The tool value if found, None otherwise
+ """
+ # Convert camelCase to underscore_case
+ underscore_name = "".join(
+ ["_" + c.lower() if c.isupper() else c for c in tool_name]
+ ).lstrip("_")
+ # Try both camelCase and underscore_case variants
+
+ if tool.get(tool_name) is not None:
+ return tool.get(tool_name)
+ elif tool.get(underscore_name) is not None:
+ return tool.get(underscore_name)
+ else:
+ return None
+
+ def _map_function( # noqa: PLR0915
+ self, value: List[dict], optional_params: dict
+ ) -> List[Tools]:
+ """
+ Map OpenAI-style tools/functions to Vertex AI format.
+
+ Args:
+ value: List of tool definitions
+ optional_params: Request-scoped parameters to store retrieval config
+
+ Returns:
+ List of mapped tools in Vertex AI format
+
+ Side effects:
+ May add 'toolConfig' with 'retrievalConfig' to optional_params if
+ googleMaps tools contain location data
+ """
gtool_func_declarations = []
googleSearch: Optional[dict] = None
googleSearchRetrieval: Optional[dict] = None
enterpriseWebSearch: Optional[dict] = None
urlContext: Optional[dict] = None
code_execution: Optional[dict] = None
+ googleMaps: Optional[dict] = None
+ google_maps_retrieval_config: Optional[dict] = None
# remove 'additionalProperties' from tools
value = _remove_additional_properties(value)
# remove 'strict' from tools
value = _remove_strict_from_schema(value)
- def get_tool_value(tool: dict, tool_name: str) -> Optional[dict]:
- """
- Helper function to get tool value handling both camelCase and underscore_case variants
-
- Args:
- tool (dict): The tool dictionary
- tool_name (str): The base tool name (e.g. "codeExecution")
-
- Returns:
- Optional[dict]: The tool value if found, None otherwise
- """
- # Convert camelCase to underscore_case
- underscore_name = "".join(
- ["_" + c.lower() if c.isupper() else c for c in tool_name]
- ).lstrip("_")
- # Try both camelCase and underscore_case variants
-
- if tool.get(tool_name) is not None:
- return tool.get(tool_name)
- elif tool.get(underscore_name) is not None:
- return tool.get(underscore_name)
- else:
- return None
-
for tool in value:
openai_function_object: Optional[
ChatCompletionToolParamFunctionChunk
@@ -327,19 +400,33 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
elif "name" in tool: # functions list
openai_function_object = ChatCompletionToolParamFunctionChunk(**tool) # type: ignore
+ # Handle tools with 'type' field (OpenAI spec compliance) Ignore this field -> https://github.com/BerriAI/litellm/issues/14644#issuecomment-3342061838
+ if "type" in tool:
+ del tool["type"] # type: ignore
+
tool_name = list(tool.keys())[0] if len(tool.keys()) == 1 else None
if tool_name and (
- tool_name == "codeExecution" or tool_name == "code_execution"
+ tool_name == "codeExecution" or tool_name == VertexToolName.CODE_EXECUTION.value
): # code_execution maintained for backwards compatibility
- code_execution = get_tool_value(tool, "codeExecution")
- elif tool_name and tool_name == "googleSearch":
- googleSearch = get_tool_value(tool, "googleSearch")
- elif tool_name and tool_name == "googleSearchRetrieval":
- googleSearchRetrieval = get_tool_value(tool, "googleSearchRetrieval")
- elif tool_name and tool_name == "enterpriseWebSearch":
- enterpriseWebSearch = get_tool_value(tool, "enterpriseWebSearch")
- elif tool_name and tool_name == "urlContext":
- urlContext = get_tool_value(tool, "urlContext")
+ code_execution = self.get_tool_value(tool, "codeExecution")
+ elif tool_name and tool_name == VertexToolName.GOOGLE_SEARCH.value:
+ googleSearch = self.get_tool_value(tool, VertexToolName.GOOGLE_SEARCH.value)
+ elif tool_name and tool_name == VertexToolName.GOOGLE_SEARCH_RETRIEVAL.value:
+ googleSearchRetrieval = self.get_tool_value(tool, VertexToolName.GOOGLE_SEARCH_RETRIEVAL.value)
+ elif tool_name and tool_name == VertexToolName.ENTERPRISE_WEB_SEARCH.value:
+ enterpriseWebSearch = self.get_tool_value(tool, VertexToolName.ENTERPRISE_WEB_SEARCH.value)
+ elif tool_name and (tool_name == VertexToolName.URL_CONTEXT.value or tool_name == "urlContext"):
+ urlContext = self.get_tool_value(tool, tool_name)
+ elif tool_name and (
+ tool_name == VertexToolName.GOOGLE_MAPS.value or tool_name == "google_maps"
+ ):
+ google_maps_value = self.get_tool_value(tool, VertexToolName.GOOGLE_MAPS.value)
+
+ # Extract and transform location configuration for toolConfig
+ if google_maps_value is not None:
+ googleMaps, google_maps_retrieval_config = self._extract_google_maps_retrieval_config(
+ google_maps_config=google_maps_value
+ )
elif openai_function_object is not None:
gtool_func_declaration = FunctionDeclaration(
name=openai_function_object["name"],
@@ -361,19 +448,29 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
"Invalid tool={}. Use `litellm.set_verbose` or `litellm --detailed_debug` to see raw request."
)
- _tools = Tools(
- function_declarations=gtool_func_declarations,
- )
+ # Only include function_declarations if there are actual functions
+ _tools = Tools()
+ if gtool_func_declarations:
+ _tools["function_declarations"] = gtool_func_declarations
if googleSearch is not None:
- _tools["googleSearch"] = googleSearch
+ _tools[VertexToolName.GOOGLE_SEARCH.value] = googleSearch
if googleSearchRetrieval is not None:
- _tools["googleSearchRetrieval"] = googleSearchRetrieval
+ _tools[VertexToolName.GOOGLE_SEARCH_RETRIEVAL.value] = googleSearchRetrieval
if enterpriseWebSearch is not None:
- _tools["enterpriseWebSearch"] = enterpriseWebSearch
+ _tools[VertexToolName.ENTERPRISE_WEB_SEARCH.value] = enterpriseWebSearch
if code_execution is not None:
- _tools["code_execution"] = code_execution
+ _tools[VertexToolName.CODE_EXECUTION.value] = code_execution
if urlContext is not None:
- _tools["url_context"] = urlContext
+ _tools[VertexToolName.URL_CONTEXT.value] = urlContext
+ if googleMaps is not None:
+ _tools[VertexToolName.GOOGLE_MAPS.value] = googleMaps
+
+ # Add retrieval config to toolConfig if googleMaps has location data
+ if google_maps_retrieval_config is not None:
+ if "toolConfig" not in optional_params:
+ optional_params["toolConfig"] = {}
+ optional_params["toolConfig"]["retrievalConfig"] = google_maps_retrieval_config
+
return [_tools]
def _map_response_schema(self, value: dict) -> dict:
@@ -419,8 +516,27 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
@staticmethod
def _map_reasoning_effort_to_thinking_budget(
reasoning_effort: str,
+ model: Optional[str] = None,
) -> GeminiThinkingConfig:
- if reasoning_effort == "low":
+ if reasoning_effort == "minimal":
+ # Use model-specific minimum thinking budget or fallback
+ # Check for exact matches first, then partial matches
+ if model and "gemini-2.5-flash-lite" in model.lower():
+ budget = DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH_LITE
+ elif model and "gemini-2.5-pro" in model.lower():
+ budget = DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_PRO
+ elif model and "gemini-2.5-flash" in model.lower():
+ budget = (
+ DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET_GEMINI_2_5_FLASH
+ )
+ else:
+ budget = DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET
+
+ return {
+ "thinkingBudget": budget,
+ "includeThoughts": True,
+ }
+ elif reasoning_effort == "low":
return {
"thinkingBudget": DEFAULT_REASONING_EFFORT_LOW_THINKING_BUDGET,
"includeThoughts": True,
@@ -461,7 +577,6 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
params["includeThoughts"] = True
if thinking_budget is not None and isinstance(thinking_budget, int):
params["thinkingBudget"] = thinking_budget
-
return params
def map_response_modalities(self, value: list) -> list:
@@ -576,8 +691,12 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
and isinstance(value, list)
and value
):
+ # Pass optional_params so _map_function can add toolConfig if needed
+ mapped_tools = self._map_function(
+ value=value, optional_params=optional_params
+ )
optional_params = self._add_tools_to_optional_params(
- optional_params, self._map_function(value=value)
+ optional_params, mapped_tools
)
elif param == "tool_choice" and (
isinstance(value, str) or isinstance(value, dict)
@@ -599,7 +718,9 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
elif param == "reasoning_effort" and isinstance(value, str):
optional_params[
"thinkingConfig"
- ] = VertexGeminiConfig._map_reasoning_effort_to_thinking_budget(value)
+ ] = VertexGeminiConfig._map_reasoning_effort_to_thinking_budget(
+ value, model
+ )
elif param == "thinking":
optional_params[
"thinkingConfig"
@@ -774,8 +895,9 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
elif "inlineData" in part:
mime_type = part["inlineData"]["mimeType"]
data = part["inlineData"]["data"]
- # Check if inline data is audio - if so, exclude from text content
- if mime_type.startswith("audio/"):
+ # Check if inline data is audio or image - if so, exclude from text content
+ # Images and audio are now handled separately in their respective response fields
+ if mime_type.startswith("audio/") or mime_type.startswith("image/"):
continue
_content_str += "data:{};base64,{}".format(mime_type, data)
@@ -791,6 +913,45 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
return content_str, reasoning_content_str
+ def _extract_thinking_blocks_from_parts(
+ self, parts: List[HttpxPartType]
+ ) -> List[ChatCompletionThinkingBlock]:
+ """Extract thinking blocks from parts if present"""
+ thinking_blocks: List[ChatCompletionThinkingBlock] = []
+ for part in parts:
+ if "thoughtSignature" in part:
+ part_copy = part.copy()
+ part_copy.pop("thoughtSignature")
+ thinking_blocks.append(
+ ChatCompletionThinkingBlock(
+ type="thinking",
+ thinking=json.dumps(part_copy),
+ signature=part["thoughtSignature"],
+ )
+ )
+ return thinking_blocks
+
+ def _extract_image_response_from_parts(
+ self, parts: List[HttpxPartType]
+ ) -> Optional[List[ImageURLListItem]]:
+ """Extract image response from parts if present"""
+ images: List[ImageURLListItem] = []
+ for part in parts:
+ if "inlineData" in part:
+ mime_type = part["inlineData"]["mimeType"]
+ data = part["inlineData"]["data"]
+ if mime_type.startswith("image/"):
+ # Convert base64 data to data URI format
+ data_uri = f"data:{mime_type};base64,{data}"
+ images.append(
+ ImageURLListItem(
+ image_url=ImageURLObject(url=data_uri, detail="auto"),
+ index=0,
+ type="image_url",
+ )
+ )
+ return images
+
def _extract_audio_response_from_parts(
self, parts: List[HttpxPartType]
) -> Optional[ChatCompletionAudioResponse]:
@@ -1038,6 +1199,16 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
text_tokens = detail.get("tokenCount", 0)
if "thoughtsTokenCount" in usage_metadata:
reasoning_tokens = usage_metadata["thoughtsTokenCount"]
+
+ ## adjust 'text_tokens' to subtract cached tokens
+ if (
+ (audio_tokens is None or audio_tokens == 0)
+ and text_tokens is not None
+ and text_tokens > 0
+ and cached_tokens is not None
+ ):
+ text_tokens = text_tokens - cached_tokens
+
prompt_tokens_details = PromptTokensDetailsWrapper(
cached_tokens=cached_tokens,
audio_tokens=audio_tokens,
@@ -1098,6 +1269,80 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
web_search_requests = len(grounding_metadata)
return web_search_requests
+ @staticmethod
+ def _create_streaming_choice(
+ chat_completion_message: ChatCompletionResponseMessage,
+ candidate: Candidates,
+ idx: int,
+ tools: Optional[List[ChatCompletionToolCallChunk]],
+ functions: Optional[ChatCompletionToolCallFunctionChunk],
+ chat_completion_logprobs: Optional[ChoiceLogprobs],
+ image_response: Optional[List[ImageURLListItem]],
+ ) -> StreamingChoices:
+ """
+ Helper method to create a streaming choice object for Vertex AI
+ """
+ from litellm.types.utils import Delta, StreamingChoices
+
+ # create a streaming choice object
+ choice = StreamingChoices(
+ finish_reason=VertexGeminiConfig._check_finish_reason(
+ chat_completion_message, candidate.get("finishReason")
+ ),
+ index=candidate.get("index", idx),
+ delta=Delta(
+ content=chat_completion_message.get("content"),
+ reasoning_content=chat_completion_message.get("reasoning_content"),
+ tool_calls=tools,
+ images=image_response,
+ function_call=functions,
+ ),
+ logprobs=chat_completion_logprobs,
+ enhancements=None,
+ )
+ return choice
+
+ @staticmethod
+ def _extract_candidate_metadata(
+ candidate: Candidates,
+ ) -> Tuple[List[dict], List[dict], List, List]:
+ """
+ Extract metadata from a single candidate response.
+
+ Returns:
+ grounding_metadata: List[dict]
+ url_context_metadata: List[dict]
+ safety_ratings: List
+ citation_metadata: List
+ """
+ grounding_metadata: List[dict] = []
+ url_context_metadata: List[dict] = []
+ safety_ratings: List = []
+ citation_metadata: List = []
+
+ if "groundingMetadata" in candidate:
+ if isinstance(candidate["groundingMetadata"], list):
+ grounding_metadata.extend(candidate["groundingMetadata"]) # type: ignore
+ else:
+ grounding_metadata.append(candidate["groundingMetadata"]) # type: ignore
+
+ if "safetyRatings" in candidate:
+ safety_ratings.append(candidate["safetyRatings"])
+
+ if "citationMetadata" in candidate:
+ citation_metadata.append(candidate["citationMetadata"])
+
+ if "urlContextMetadata" in candidate:
+ # Add URL context metadata to grounding metadata
+ url_context_metadata.append(cast(dict, candidate["urlContextMetadata"]))
+
+ return (
+ grounding_metadata,
+ url_context_metadata,
+ safety_ratings,
+ citation_metadata,
+ )
+
@staticmethod
def _process_candidates(
_candidates: List[Candidates],
@@ -1120,6 +1365,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
grounding_metadata: List[dict] = []
url_context_metadata: List[dict] = []
+ image_response: Optional[List[ImageURLListItem]] = None
safety_ratings: List = []
citation_metadata: List = []
chat_completion_message: ChatCompletionResponseMessage = {"role": "assistant"}
@@ -1127,26 +1373,24 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
tools: Optional[List[ChatCompletionToolCallChunk]] = []
functions: Optional[ChatCompletionToolCallFunctionChunk] = None
cumulative_tool_call_index: int = 0
+ thinking_blocks: Optional[List[ChatCompletionThinkingBlock]] = None
for idx, candidate in enumerate(_candidates):
if "content" not in candidate:
continue
- if "groundingMetadata" in candidate:
- if isinstance(candidate["groundingMetadata"], list):
- grounding_metadata.extend(candidate["groundingMetadata"]) # type: ignore
- else:
- grounding_metadata.append(candidate["groundingMetadata"]) # type: ignore
+ # Extract metadata using helper function
+ (
+ candidate_grounding_metadata,
+ candidate_url_context_metadata,
+ candidate_safety_ratings,
+ candidate_citation_metadata,
+ ) = VertexGeminiConfig._extract_candidate_metadata(candidate)
- if "safetyRatings" in candidate:
- safety_ratings.append(candidate["safetyRatings"])
-
- if "citationMetadata" in candidate:
- citation_metadata.append(candidate["citationMetadata"])
-
- if "urlContextMetadata" in candidate:
- # Add URL context metadata to grounding metadata
- url_context_metadata.append(cast(dict, candidate["urlContextMetadata"]))
+ grounding_metadata.extend(candidate_grounding_metadata)
+ url_context_metadata.extend(candidate_url_context_metadata)
+ safety_ratings.extend(candidate_safety_ratings)
+ citation_metadata.extend(candidate_citation_metadata)
if "parts" in candidate["content"]:
(
@@ -1161,18 +1405,33 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
parts=candidate["content"]["parts"]
)
)
+ image_response = (
+ VertexGeminiConfig()._extract_image_response_from_parts(
+ parts=candidate["content"]["parts"]
+ )
+ )
+
+ thinking_blocks = (
+ VertexGeminiConfig()._extract_thinking_blocks_from_parts(
+ parts=candidate["content"]["parts"]
+ )
+ )
if audio_response is not None:
cast(Dict[str, Any], chat_completion_message)[
"audio"
] = audio_response
chat_completion_message["content"] = None # OpenAI spec
- elif content is not None:
+ if image_response is not None:
+ # Handle image response - combine with text content into structured format
+ cast(Dict[str, Any], chat_completion_message)[
+ "images"
+ ] = image_response
+ if content is not None:
chat_completion_message["content"] = content
if reasoning_content is not None:
chat_completion_message["reasoning_content"] = reasoning_content
-
(
functions,
tools,
@@ -1194,25 +1453,18 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
if functions is not None:
chat_completion_message["function_call"] = functions
- if isinstance(model_response, ModelResponseStream):
- from litellm.types.utils import Delta, StreamingChoices
+ if thinking_blocks is not None:
+ chat_completion_message["thinking_blocks"] = thinking_blocks # type: ignore
- # create a streaming choice object
- choice = StreamingChoices(
- finish_reason=VertexGeminiConfig._check_finish_reason(
- chat_completion_message, candidate.get("finishReason")
- ),
- index=candidate.get("index", idx),
- delta=Delta(
- content=chat_completion_message.get("content"),
- reasoning_content=chat_completion_message.get(
- "reasoning_content"
- ),
- tool_calls=tools,
- function_call=functions,
- ),
- logprobs=chat_completion_logprobs,
- enhancements=None,
+ if isinstance(model_response, ModelResponseStream):
+ choice = VertexGeminiConfig._create_streaming_choice(
+ chat_completion_message=chat_completion_message,
+ candidate=candidate,
+ idx=idx,
+ tools=tools,
+ functions=functions,
+ chat_completion_logprobs=chat_completion_logprobs,
+ image_response=image_response,
)
model_response.choices.append(choice)
elif isinstance(model_response, ModelResponse):
@@ -1438,7 +1690,7 @@ async def make_call(
)
try:
- response = await client.post(api_base, headers=headers, data=data, stream=True)
+ response = await client.post(api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj)
response.raise_for_status()
except httpx.HTTPStatusError as e:
exception_string = str(await e.response.aread())
@@ -1485,7 +1737,7 @@ def make_sync_call(
if client is None:
client = HTTPHandler() # Create a new client if none provided
- response = client.post(api_base, headers=headers, data=data, stream=True)
+ response = client.post(api_base, headers=headers, data=data, stream=True, logging_obj=logging_obj)
if response.status_code != 200 and response.status_code != 201:
raise VertexAIError(
@@ -1540,7 +1792,6 @@ class VertexLLM(VertexBase):
gemini_api_key: Optional[str] = None,
extra_headers: Optional[dict] = None,
) -> CustomStreamWrapper:
- request_body = await async_transform_request_body(**data) # type: ignore
should_use_v1beta1_features = self.is_using_v1beta1_features(
optional_params=optional_params
@@ -1574,6 +1825,13 @@ class VertexLLM(VertexBase):
litellm_params=litellm_params,
)
+ request_body = await async_transform_request_body(
+ **data,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=auth_header) # type: ignore
+
+
## LOGGING
logging_obj.pre_call(
input=messages,
@@ -1661,7 +1919,12 @@ class VertexLLM(VertexBase):
litellm_params=litellm_params,
)
- request_body = await async_transform_request_body(**data) # type: ignore
+ request_body = await async_transform_request_body(
+ **data,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=auth_header) # type: ignore
+
_async_client_params = {}
if timeout:
_async_client_params["timeout"] = timeout
@@ -1684,7 +1947,7 @@ class VertexLLM(VertexBase):
try:
response = await client.post(
- api_base, headers=headers, json=cast(dict, request_body)
+ api_base, headers=headers, json=cast(dict, request_body), logging_obj=logging_obj
) # type: ignore
response.raise_for_status()
except httpx.HTTPStatusError as err:
@@ -1836,7 +2099,11 @@ class VertexLLM(VertexBase):
)
## TRANSFORMATION ##
- data = sync_transform_request_body(**transform_request_params)
+ data = sync_transform_request_body(
+ **transform_request_params,
+ vertex_project=vertex_project,
+ vertex_location=vertex_location,
+ vertex_auth_header=auth_header)
## LOGGING
logging_obj.pre_call(
@@ -1887,7 +2154,7 @@ class VertexLLM(VertexBase):
client = client
try:
- response = client.post(url=url, headers=headers, json=data) # type: ignore
+ response = client.post(url=url, headers=headers, json=data, logging_obj=logging_obj) # type: ignore
response.raise_for_status()
except httpx.HTTPStatusError as err:
error_code = err.response.status_code
diff --git a/litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py b/litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py
index ecfe2ee8b4b..af9af71fef4 100644
--- a/litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py
+++ b/litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py
@@ -43,7 +43,7 @@ class GoogleBatchEmbeddings(VertexLLM):
vertex_project=None,
vertex_location=None,
vertex_credentials=None,
- aembedding=False,
+ aembedding: Optional[bool] = False,
timeout=300,
client=None,
) -> EmbeddingResponse:
diff --git a/litellm/llms/vertex_ai/google_genai/transformation.py b/litellm/llms/vertex_ai/google_genai/transformation.py
index 02825026e1b..d7a4ceeb3e7 100644
--- a/litellm/llms/vertex_ai/google_genai/transformation.py
+++ b/litellm/llms/vertex_ai/google_genai/transformation.py
@@ -1,16 +1,100 @@
"""
Transformation for Calling Google models in their native format.
"""
-from typing import Literal
+
+from typing import Any, Dict, Literal, Optional, Union
from litellm.llms.gemini.google_genai.transformation import GoogleGenAIConfig
+from litellm.types.router import GenericLiteLLMParams
class VertexAIGoogleGenAIConfig(GoogleGenAIConfig):
"""
Configuration for calling Google models in their native format.
"""
+
+ HEADER_NAME = "Authorization"
+ BEARER_PREFIX = "Bearer"
+
@property
def custom_llm_provider(self) -> Literal["gemini", "vertex_ai"]:
return "vertex_ai"
-
\ No newline at end of file
+
+ def validate_environment(
+ self,
+ api_key: Optional[str],
+ headers: Optional[dict],
+ model: str,
+ litellm_params: Optional[Union[GenericLiteLLMParams, dict]],
+ ) -> dict:
+ default_headers = {
+ "Content-Type": "application/json",
+ }
+
+ if api_key is not None:
+ default_headers[self.HEADER_NAME] = f"{self.BEARER_PREFIX} {api_key}"
+ if headers is not None:
+ default_headers.update(headers)
+
+ return default_headers
+
+ def _camel_to_snake(self, camel_str: str) -> str:
+ """Convert camelCase to snake_case"""
+ import re
+
+ return re.sub(r"(? dict:
+ """
+ Transform the generate content request for Vertex AI.
+ Since Vertex AI natively supports Google GenAI format, we can pass most fields directly.
+ """
+ # Build the request in Google GenAI format that Vertex AI expects
+ result = {
+ "model": model,
+ "contents": contents,
+ }
+
+ # Add tools if provided
+ if tools:
+ result["tools"] = tools
+
+ # Add systemInstruction if provided
+ if system_instruction:
+ result["systemInstruction"] = system_instruction
+
+ # Handle generationConfig - Vertex AI expects it in the same format
+ if generate_content_config_dict:
+ result["generationConfig"] = generate_content_config_dict
+
+ return result
diff --git a/litellm/llms/vertex_ai/multimodal_embeddings/embedding_handler.py b/litellm/llms/vertex_ai/multimodal_embeddings/embedding_handler.py
index 8aebd83cc44..582d7a4c569 100644
--- a/litellm/llms/vertex_ai/multimodal_embeddings/embedding_handler.py
+++ b/litellm/llms/vertex_ai/multimodal_embeddings/embedding_handler.py
@@ -46,7 +46,7 @@ class VertexMultimodalEmbedding(VertexLLM):
vertex_project=None,
vertex_location=None,
vertex_credentials=None,
- aembedding=False,
+ aembedding: Optional[bool] = False,
timeout=300,
client=None,
) -> EmbeddingResponse:
diff --git a/litellm/llms/vertex_ai/text_to_speech/text_to_speech_handler.py b/litellm/llms/vertex_ai/text_to_speech/text_to_speech_handler.py
index 18bc72db46a..9d9015c2b91 100644
--- a/litellm/llms/vertex_ai/text_to_speech/text_to_speech_handler.py
+++ b/litellm/llms/vertex_ai/text_to_speech/text_to_speech_handler.py
@@ -1,6 +1,7 @@
-from typing import Optional, TypedDict, Union
+from typing import Optional, Union
import httpx
+from typing_extensions import TypedDict
import litellm
from litellm.llms.custom_httpx.http_handler import (
diff --git a/litellm/llms/vertex_ai/vertex_ai_partner_models/gpt_oss/transformation.py b/litellm/llms/vertex_ai/vertex_ai_partner_models/gpt_oss/transformation.py
new file mode 100644
index 00000000000..86e36e802ed
--- /dev/null
+++ b/litellm/llms/vertex_ai/vertex_ai_partner_models/gpt_oss/transformation.py
@@ -0,0 +1,27 @@
+import litellm
+from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
+
+
+class VertexAIGPTOSSTransformation(OpenAIGPTConfig):
+ """
+ Transformation for GPT-OSS model from VertexAI
+
+ https://console.cloud.google.com/vertex-ai/publishers/openai/model-garden/gpt-oss-120b-maas?hl=id
+ """
+ def __init__(self):
+ super().__init__()
+
+ def get_supported_openai_params(self, model: str) -> list:
+ base_gpt_series_params = super().get_supported_openai_params(model=model)
+ gpt_oss_only_params = ["reasoning_effort"]
+ base_gpt_series_params.extend(gpt_oss_only_params)
+
+ #########################################################
+ # VertexAI - GPT-OSS does not support tool calls
+ #########################################################
+ if litellm.supports_function_calling(model=model) is False:
+ TOOL_CALLING_PARAMS_TO_REMOVE = ["tool", "tool_choice", "function_call", "functions"]
+ base_gpt_series_params = [param for param in base_gpt_series_params if param not in TOOL_CALLING_PARAMS_TO_REMOVE]
+
+ return base_gpt_series_params
+
diff --git a/litellm/llms/vertex_ai/vertex_ai_partner_models/llama3/transformation.py b/litellm/llms/vertex_ai/vertex_ai_partner_models/llama3/transformation.py
index 7e965313a0b..748a5f5fb40 100644
--- a/litellm/llms/vertex_ai/vertex_ai_partner_models/llama3/transformation.py
+++ b/litellm/llms/vertex_ai/vertex_ai_partner_models/llama3/transformation.py
@@ -113,10 +113,10 @@ class VertexAILlama3Config(OpenAIGPTConfig):
status_code=raw_response.status_code,
headers=response_headers,
)
- model_response.model = completion_response["model"]
- model_response.id = completion_response["id"]
- model_response.created = completion_response["created"]
- setattr(model_response, "usage", Usage(**completion_response["usage"]))
+ model_response.model = completion_response.get("model", model)
+ model_response.id = completion_response.get("id", "")
+ model_response.created = completion_response.get("created", 0)
+ setattr(model_response, "usage", Usage(**completion_response.get("usage", {})))
model_response.choices = self._transform_choices( # type: ignore
choices=completion_response["choices"],
diff --git a/litellm/llms/vertex_ai/vertex_ai_partner_models/main.py b/litellm/llms/vertex_ai/vertex_ai_partner_models/main.py
index 7303ab0786c..ea29970f0aa 100644
--- a/litellm/llms/vertex_ai/vertex_ai_partner_models/main.py
+++ b/litellm/llms/vertex_ai/vertex_ai_partner_models/main.py
@@ -1,5 +1,6 @@
# What is this?
## API Handler for calling Vertex AI Partner Models
+from enum import Enum
from typing import Callable, Optional, Union
import httpx # type: ignore
@@ -27,6 +28,16 @@ class VertexAIError(Exception):
self.message
) # Call the base class constructor with the parameters it needs
+class PartnerModelPrefixes(str, Enum):
+ META_PREFIX = "meta/"
+ DEEPSEEK_PREFIX = "deepseek-ai"
+ MISTRAL_PREFIX = "mistral"
+ CODERESTAL_PREFIX = "codestral"
+ JAMBA_PREFIX = "jamba"
+ CLAUDE_PREFIX = "claude"
+ QWEN_PREFIX = "qwen"
+ GPT_OSS_PREFIX = "openai/gpt-oss-"
+
class VertexAIPartnerModels(VertexBase):
def __init__(self) -> None:
@@ -42,15 +53,29 @@ class VertexAIPartnerModels(VertexBase):
bool: True if the model string is a Vertex AI Partner Model, False otherwise
"""
if (
- model.startswith("meta/")
- or model.startswith("deepseek-ai")
- or model.startswith("mistral")
- or model.startswith("codestral")
- or model.startswith("jamba")
- or model.startswith("claude")
+ model.startswith(PartnerModelPrefixes.META_PREFIX)
+ or model.startswith(PartnerModelPrefixes.DEEPSEEK_PREFIX)
+ or model.startswith(PartnerModelPrefixes.MISTRAL_PREFIX)
+ or model.startswith(PartnerModelPrefixes.CODERESTAL_PREFIX)
+ or model.startswith(PartnerModelPrefixes.JAMBA_PREFIX)
+ or model.startswith(PartnerModelPrefixes.CLAUDE_PREFIX)
+ or model.startswith(PartnerModelPrefixes.QWEN_PREFIX)
+ or model.startswith(PartnerModelPrefixes.GPT_OSS_PREFIX)
):
return True
return False
+
+ @staticmethod
+ def should_use_openai_handler(model: str):
+ OPENAI_LIKE_VERTEX_PROVIDERS = [
+ "llama",
+ PartnerModelPrefixes.DEEPSEEK_PREFIX,
+ PartnerModelPrefixes.QWEN_PREFIX,
+ PartnerModelPrefixes.GPT_OSS_PREFIX,
+ ]
+ if any(provider in model for provider in OPENAI_LIKE_VERTEX_PROVIDERS):
+ return True
+ return False
def completion(
self,
@@ -115,7 +140,7 @@ class VertexAIPartnerModels(VertexBase):
optional_params["stream"] = stream
- if "llama" in model or "deepseek-ai" in model:
+ if self.should_use_openai_handler(model):
partner = VertexPartnerProvider.llama
elif "mistral" in model or "codestral" in model:
partner = VertexPartnerProvider.mistralai
@@ -191,7 +216,7 @@ class VertexAIPartnerModels(VertexBase):
client=client,
custom_llm_provider=LlmProviders.VERTEX_AI.value,
)
- elif "llama" in model:
+ elif self.should_use_openai_handler(model):
return base_llm_http_handler.completion(
model=model,
stream=stream,
diff --git a/litellm/llms/vertex_ai/vertex_embeddings/embedding_handler.py b/litellm/llms/vertex_ai/vertex_embeddings/embedding_handler.py
index 1167ca285fc..a170e6cc7f2 100644
--- a/litellm/llms/vertex_ai/vertex_embeddings/embedding_handler.py
+++ b/litellm/llms/vertex_ai/vertex_embeddings/embedding_handler.py
@@ -36,7 +36,7 @@ class VertexEmbedding(VertexBase):
timeout: Optional[Union[float, httpx.Timeout]],
api_key: Optional[str] = None,
encoding=None,
- aembedding=False,
+ aembedding: Optional[bool] = False,
api_base: Optional[str] = None,
client: Optional[Union[AsyncHTTPHandler, HTTPHandler]] = None,
vertex_project: Optional[str] = None,
@@ -86,8 +86,10 @@ class VertexEmbedding(VertexBase):
mode="embedding",
)
headers = self.set_headers(auth_header=auth_header, extra_headers=extra_headers)
- vertex_request: VertexEmbeddingRequest = litellm.vertexAITextEmbeddingConfig.transform_openai_request_to_vertex_embedding_request(
- input=input, optional_params=optional_params, model=model
+ vertex_request: VertexEmbeddingRequest = (
+ litellm.vertexAITextEmbeddingConfig.transform_openai_request_to_vertex_embedding_request(
+ input=input, optional_params=optional_params, model=model
+ )
)
_client_params = {}
@@ -176,8 +178,10 @@ class VertexEmbedding(VertexBase):
mode="embedding",
)
headers = self.set_headers(auth_header=auth_header, extra_headers=extra_headers)
- vertex_request: VertexEmbeddingRequest = litellm.vertexAITextEmbeddingConfig.transform_openai_request_to_vertex_embedding_request(
- input=input, optional_params=optional_params, model=model
+ vertex_request: VertexEmbeddingRequest = (
+ litellm.vertexAITextEmbeddingConfig.transform_openai_request_to_vertex_embedding_request(
+ input=input, optional_params=optional_params, model=model
+ )
)
_async_client_params = {}
diff --git a/litellm/llms/vertex_ai/vertex_embeddings/types.py b/litellm/llms/vertex_ai/vertex_embeddings/types.py
index c0c53b170c4..7f85ea46f31 100644
--- a/litellm/llms/vertex_ai/vertex_embeddings/types.py
+++ b/litellm/llms/vertex_ai/vertex_embeddings/types.py
@@ -3,7 +3,9 @@ Types for Vertex Embeddings Requests
"""
from enum import Enum
-from typing import List, Optional, TypedDict, Union
+from typing import List, Optional, Union
+
+from typing_extensions import TypedDict
class TaskType(str, Enum):
diff --git a/litellm/llms/vertex_ai/vertex_gemma_models/__init__.py b/litellm/llms/vertex_ai/vertex_gemma_models/__init__.py
new file mode 100644
index 00000000000..d06c7a5cd7a
--- /dev/null
+++ b/litellm/llms/vertex_ai/vertex_gemma_models/__init__.py
@@ -0,0 +1,2 @@
+"""Vertex AI Gemma-AI Models Handler"""
+
diff --git a/litellm/llms/vertex_ai/vertex_gemma_models/main.py b/litellm/llms/vertex_ai/vertex_gemma_models/main.py
new file mode 100644
index 00000000000..8203b285ebd
--- /dev/null
+++ b/litellm/llms/vertex_ai/vertex_gemma_models/main.py
@@ -0,0 +1,145 @@
+"""
+API Handler for calling Vertex AI Gemma Models
+
+These models use a custom prediction endpoint format that wraps messages in 'instances'
+with @requestFormat: "chatCompletions" and returns responses wrapped in 'predictions'.
+
+Usage:
+
+response = litellm.completion(
+ model="vertex_ai/gemma/gemma-3-12b-it-1222199011122",
+ messages=[{"role": "user", "content": "What is machine learning?"}],
+ vertex_project="your-project-id",
+ vertex_location="us-central1",
+)
+
+Sent to this route when `model` is in the format `vertex_ai/gemma/{MODEL_NAME}`
+
+The API expects a custom endpoint URL format:
+https://{ENDPOINT_NUMBER}.{location}-{REGION_NUMBER}.prediction.vertexai.goog/v1/projects/{PROJECT_ID}/locations/{location}/endpoints/{ENDPOINT_ID}:predict
+"""
+
+from typing import Callable, Optional, Union
+
+import httpx # type: ignore
+
+from litellm.utils import ModelResponse
+
+from ..common_utils import VertexAIError
+from ..vertex_llm_base import VertexBase
+
+
+class VertexAIGemmaModels(VertexBase):
+ def __init__(self) -> None:
+ pass
+
+ def completion(
+ self,
+ model: str,
+ messages: list,
+ model_response: ModelResponse,
+ print_verbose: Callable,
+ encoding,
+ logging_obj,
+ api_base: Optional[str],
+ optional_params: dict,
+ custom_prompt_dict: dict,
+ headers: Optional[dict],
+ timeout: Union[float, httpx.Timeout],
+ litellm_params: dict,
+ vertex_project=None,
+ vertex_location=None,
+ vertex_credentials=None,
+ logger_fn=None,
+ acompletion: bool = False,
+ client=None,
+ ):
+ """
+ Handles calling Vertex AI Gemma Models
+
+ Sent to this route when `model` is in the format `vertex_ai/gemma/{MODEL_NAME}`
+ """
+ try:
+ import vertexai
+
+ from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
+ VertexLLM,
+ )
+ from litellm.llms.vertex_ai.vertex_gemma_models.transformation import (
+ VertexGemmaConfig,
+ )
+ except Exception as e:
+ raise VertexAIError(
+ status_code=400,
+ message=f"""vertexai import failed please run `pip install -U "google-cloud-aiplatform>=1.38"`. Got error: {e}""",
+ )
+
+ if not (
+ hasattr(vertexai, "preview") or hasattr(vertexai.preview, "language_models")
+ ):
+ raise VertexAIError(
+ status_code=400,
+ message="""Upgrade vertex ai. Run `pip install "google-cloud-aiplatform>=1.38"`""",
+ )
+ try:
+ model = model.replace("gemma/", "")
+ vertex_httpx_logic = VertexLLM()
+
+ access_token, project_id = vertex_httpx_logic._ensure_access_token(
+ credentials=vertex_credentials,
+ project_id=vertex_project,
+ custom_llm_provider="vertex_ai",
+ )
+
+ gemma_transformation = VertexGemmaConfig()
+
+ ## CONSTRUCT API BASE
+ stream: bool = optional_params.get("stream", False) or False
+ optional_params["stream"] = stream
+
+ # If api_base is not provided, it should be set as an environment variable
+ # or passed explicitly because the endpoint URL is unique per deployment
+ if api_base is None:
+ raise VertexAIError(
+ status_code=400,
+ message="api_base is required for Vertex AI Gemma models. Please provide the full endpoint URL.",
+ )
+
+ # Check if we need to append :predict
+ if not api_base.endswith(":predict"):
+ _, api_base = self._check_custom_proxy(
+ api_base=api_base,
+ custom_llm_provider="vertex_ai",
+ gemini_api_key=None,
+ endpoint="predict",
+ stream=stream,
+ auth_header=None,
+ url=api_base,
+ )
+ # If api_base already ends with :predict, use it as-is
+
+ # Use the custom transformation handler for gemma models
+ return gemma_transformation.completion(
+ model=model,
+ messages=messages,
+ api_base=api_base,
+ api_key=access_token,
+ custom_prompt_dict=custom_prompt_dict,
+ model_response=model_response,
+ print_verbose=print_verbose,
+ logging_obj=logging_obj,
+ optional_params=optional_params,
+ acompletion=acompletion,
+ litellm_params=litellm_params,
+ logger_fn=logger_fn,
+ client=client,
+ timeout=timeout,
+ encoding=encoding,
+ custom_llm_provider="vertex_ai",
+ )
+
+ except Exception as e:
+ if hasattr(e, "status_code"):
+ raise e
+ raise VertexAIError(status_code=500, message=str(e))
+
diff --git a/litellm/llms/vertex_ai/vertex_gemma_models/transformation.py b/litellm/llms/vertex_ai/vertex_gemma_models/transformation.py
new file mode 100644
index 00000000000..24b53f0ba4f
--- /dev/null
+++ b/litellm/llms/vertex_ai/vertex_gemma_models/transformation.py
@@ -0,0 +1,354 @@
+"""
+Transformation logic for Vertex AI Gemma Models
+
+Handles the custom request/response format:
+- Request: Wraps messages in 'instances' with @requestFormat: "chatCompletions"
+- Response: Extracts data from 'predictions' wrapper
+
+The actual message transformation reuses OpenAIGPTConfig since Gemma uses OpenAI-compatible format.
+"""
+
+from typing import Any, Callable, Dict, List, Optional, Union, cast
+
+import httpx
+
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
+from litellm.types.llms.openai import AllMessageValues
+from litellm.types.utils import ModelResponse
+
+
+class VertexGemmaConfig(OpenAIGPTConfig):
+ """
+ Configuration and transformation class for Vertex AI Gemma models
+
+ Extends OpenAIGPTConfig to wrap/unwrap the instances/predictions format
+ used by Vertex AI's Gemma deployment endpoint.
+ """
+
+ def __init__(self) -> None:
+ super().__init__()
+
+ def should_fake_stream(
+ self,
+ model: Optional[str],
+ stream: Optional[bool],
+ custom_llm_provider: Optional[str] = None,
+ ) -> bool:
+ """
+ Vertex AI Gemma models do not support streaming.
+ Return True to enable fake streaming on the client side.
+ """
+ return True
+
+ def _handle_fake_stream_response(
+ self,
+ model_response: ModelResponse,
+ stream: bool,
+ ) -> Union[ModelResponse, Any]:
+ """
+ Helper method to return fake stream iterator if streaming is requested.
+
+ Args:
+ model_response: The completed model response
+ stream: Whether streaming was requested
+
+ Returns:
+ MockResponseIterator if stream=True, otherwise the model_response
+ """
+ if stream:
+ from litellm.llms.base_llm.base_model_iterator import MockResponseIterator
+ return MockResponseIterator(model_response=model_response)
+ return model_response
+
+ def transform_request(
+ self,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ headers: dict,
+ ) -> dict:
+ """
+ Transform request to Vertex Gemma format.
+
+ Uses parent class to create OpenAI-compatible request, then wraps it
+ in the Vertex Gemma instances format.
+ """
+ # Get the base OpenAI request from parent class
+ openai_request = super().transform_request(
+ model=model,
+ messages=messages,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ headers=headers,
+ )
+
+ # Remove params not needed/supported by Vertex Gemma
+ openai_request.pop("model", None)
+ openai_request.pop("stream", None) # Streaming not supported, will be faked client-side
+ openai_request.pop("stream_options", None) # Stream options not supported
+
+ # Wrap in Vertex Gemma format
+ return {
+ "instances": [
+ {
+ "@requestFormat": "chatCompletions",
+ **openai_request,
+ }
+ ]
+ }
+
+ def _unwrap_predictions_response(
+ self,
+ response_json: Dict[str, Any],
+ ) -> Dict[str, Any]:
+ """
+ Unwrap the Vertex Gemma predictions format to OpenAI format.
+
+ Vertex Gemma wraps the OpenAI-compatible response in a 'predictions' field.
+ This method extracts it so the parent class can process it normally.
+ """
+ if "predictions" not in response_json:
+ raise BaseLLMException(
+ status_code=422,
+ message="Invalid response format: missing 'predictions' field",
+ )
+
+ return response_json["predictions"]
+
+ def completion(
+ self,
+ model: str,
+ messages: list,
+ api_base: str,
+ api_key: str,
+ custom_prompt_dict: dict,
+ model_response: ModelResponse,
+ print_verbose: Callable,
+ logging_obj: Any,
+ optional_params: dict,
+ acompletion: bool,
+ litellm_params: dict,
+ logger_fn: Optional[Callable] = None,
+ client: Optional[httpx.Client] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ encoding=None,
+ custom_llm_provider: str = "vertex_ai",
+ ):
+ """
+ Make completion request to Vertex Gemma endpoint.
+ Supports both sync and async requests with fake streaming.
+ """
+ if acompletion:
+ return self._async_completion(
+ model=model,
+ messages=messages,
+ api_base=api_base,
+ api_key=api_key,
+ model_response=model_response,
+ print_verbose=print_verbose,
+ logging_obj=logging_obj,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ timeout=timeout,
+ encoding=encoding,
+ )
+ else:
+ return self._sync_completion(
+ model=model,
+ messages=messages,
+ api_base=api_base,
+ api_key=api_key,
+ model_response=model_response,
+ print_verbose=print_verbose,
+ logging_obj=logging_obj,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ timeout=timeout,
+ encoding=encoding,
+ )
+
+ def _sync_completion(
+ self,
+ model: str,
+ messages: list,
+ api_base: str,
+ api_key: str,
+ model_response: ModelResponse,
+ print_verbose: Callable,
+ logging_obj: Any,
+ optional_params: dict,
+ litellm_params: dict,
+ timeout: Optional[Union[float, httpx.Timeout]],
+ encoding: Any,
+ ):
+ """Synchronous completion request"""
+ from litellm.llms.custom_httpx.http_handler import HTTPHandler
+ from litellm.utils import convert_to_model_response_object
+
+ # Check if streaming is requested (will be faked)
+ stream = optional_params.get("stream", False)
+
+ # Transform the request using parent class methods
+ request_data = self.transform_request(
+ model=model,
+ messages=messages,
+ optional_params=optional_params.copy(),
+ litellm_params=litellm_params,
+ headers={},
+ )
+
+ # Set up headers
+ headers = {
+ "Authorization": f"Bearer {api_key}",
+ "Content-Type": "application/json",
+ }
+
+ # Log the request
+ logging_obj.pre_call(
+ input=messages,
+ api_key=api_key,
+ additional_args={
+ "complete_input_dict": request_data,
+ "api_base": api_base,
+ },
+ )
+
+ # Make the HTTP request
+ http_handler = HTTPHandler(concurrent_limit=1)
+ response = http_handler.post(
+ url=api_base,
+ headers=headers,
+ json=request_data,
+ timeout=timeout,
+ )
+
+ if response.status_code != 200:
+ raise BaseLLMException(
+ status_code=response.status_code,
+ message=f"Request failed: {response.text}",
+ )
+
+ response_json = response.json()
+
+ # Unwrap predictions to get OpenAI-compatible response
+ openai_response = self._unwrap_predictions_response(response_json)
+
+ # Use litellm's standard response converter
+ model_response = cast(
+ ModelResponse,
+ convert_to_model_response_object(
+ response_object=openai_response,
+ model_response_object=model_response,
+ _response_headers={},
+ ),
+ )
+
+ # Ensure model is set correctly
+ model_response.model = model
+
+ # Log the response
+ logging_obj.post_call(
+ input=messages,
+ api_key=api_key,
+ original_response=response_json,
+ additional_args={"complete_input_dict": request_data},
+ )
+
+ # Return fake stream iterator if streaming was requested
+ return self._handle_fake_stream_response(model_response=model_response, stream=stream)
+
+ async def _async_completion(
+ self,
+ model: str,
+ messages: list,
+ api_base: str,
+ api_key: str,
+ model_response: ModelResponse,
+ print_verbose: Callable,
+ logging_obj: Any,
+ optional_params: dict,
+ litellm_params: dict,
+ timeout: Optional[Union[float, httpx.Timeout]],
+ encoding: Any,
+ ):
+ """Asynchronous completion request"""
+ from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
+ from litellm.types.utils import LlmProviders
+ from litellm.utils import convert_to_model_response_object
+
+ # Check if streaming is requested (will be faked)
+ stream = optional_params.get("stream", False)
+
+ # Transform the request using parent class async methods
+ request_data = await self.async_transform_request(
+ model=model,
+ messages=messages,
+ optional_params=optional_params.copy(),
+ litellm_params=litellm_params,
+ headers={},
+ )
+
+ # Set up headers
+ headers = {
+ "Authorization": f"Bearer {api_key}",
+ "Content-Type": "application/json",
+ }
+
+ # Log the request
+ logging_obj.pre_call(
+ input=messages,
+ api_key=api_key,
+ additional_args={
+ "complete_input_dict": request_data,
+ "api_base": api_base,
+ },
+ )
+
+ # Make the HTTP request
+ http_handler = get_async_httpx_client(
+ llm_provider=LlmProviders.VERTEX_AI,
+ )
+ response = await http_handler.post(
+ url=api_base,
+ headers=headers,
+ json=request_data,
+ timeout=timeout,
+ )
+
+ if response.status_code != 200:
+ raise BaseLLMException(
+ status_code=response.status_code,
+ message=f"Request failed: {response.text}",
+ )
+
+ response_json = response.json()
+
+ # Unwrap predictions to get OpenAI-compatible response
+ openai_response = self._unwrap_predictions_response(response_json)
+
+ # Use litellm's standard response converter
+ model_response = cast(
+ ModelResponse,
+ convert_to_model_response_object(
+ response_object=openai_response,
+ model_response_object=model_response,
+ _response_headers={},
+ ),
+ )
+
+ # Ensure model is set correctly
+ model_response.model = model
+
+ # Log the response
+ logging_obj.post_call(
+ input=messages,
+ api_key=api_key,
+ original_response=response_json,
+ additional_args={"complete_input_dict": request_data},
+ )
+
+ # Return fake stream iterator if streaming was requested
+ return self._handle_fake_stream_response(model_response=model_response, stream=stream)
+
diff --git a/litellm/llms/vertex_ai/vertex_llm_base.py b/litellm/llms/vertex_ai/vertex_llm_base.py
index ae43e1fd167..6d194d41add 100644
--- a/litellm/llms/vertex_ai/vertex_llm_base.py
+++ b/litellm/llms/vertex_ai/vertex_llm_base.py
@@ -239,6 +239,7 @@ class VertexBase:
stream=stream,
auth_header=None,
url=default_api_base,
+ model=model,
)
return api_base
@@ -270,17 +271,11 @@ class VertexBase:
def is_using_v1beta1_features(self, optional_params: dict) -> bool:
"""
- VertexAI only supports ContextCaching on v1beta1
-
use this helper to decide if request should be sent to v1 or v1beta1
- Returns v1beta1 if context caching is enabled
- Returns v1 in all other cases
+ Returns true if any beta feature is enabled
+ Returns false in all other cases
"""
- if "cached_content" in optional_params:
- return True
- if "CachedContent" in optional_params:
- return True
return False
def _check_custom_proxy(
@@ -292,6 +287,7 @@ class VertexBase:
stream: Optional[bool],
auth_header: Optional[str],
url: str,
+ model: Optional[str] = None,
) -> Tuple[Optional[str], str]:
"""
for cloudflare ai gateway - https://github.com/BerriAI/litellm/issues/4317
@@ -301,7 +297,12 @@ class VertexBase:
"""
if api_base:
if custom_llm_provider == "gemini":
- url = "{}:{}".format(api_base, endpoint)
+ # For Gemini (Google AI Studio), construct the full path like other providers
+ if model is None:
+ raise ValueError(
+ "Model parameter is required for Gemini custom API base URLs"
+ )
+ url = "{}/models/{}:{}".format(api_base, model, endpoint)
if gemini_api_key is None:
raise ValueError(
"Missing gemini_api_key, please set `GEMINI_API_KEY`"
@@ -373,12 +374,63 @@ class VertexBase:
endpoint=endpoint,
stream=stream,
url=url,
+ model=model,
)
+ def _handle_reauthentication(
+ self,
+ credentials: Optional[VERTEX_CREDENTIALS_TYPES],
+ project_id: Optional[str],
+ credential_cache_key: Tuple,
+ error: Exception,
+ ) -> Tuple[str, str]:
+ """
+ Handle reauthentication when credentials refresh fails.
+
+ This method clears the cached credentials and attempts to reload them once.
+ It should only be called when "Reauthentication is needed" error occurs.
+
+ Args:
+ credentials: The original credentials
+ project_id: The project ID
+ credential_cache_key: The cache key to clear
+ error: The original error that triggered reauthentication
+
+ Returns:
+ Tuple of (access_token, project_id)
+
+ Raises:
+ The original error if reauthentication fails
+ """
+ verbose_logger.debug(
+ f"Handling reauthentication for project_id: {project_id}. "
+ f"Clearing cache and retrying once."
+ )
+
+ # Clear the cached credentials
+ if credential_cache_key in self._credentials_project_mapping:
+ del self._credentials_project_mapping[credential_cache_key]
+
+ # Retry once with _retry_reauth=True to prevent infinite recursion
+ try:
+ return self.get_access_token(
+ credentials=credentials,
+ project_id=project_id,
+ _retry_reauth=True,
+ )
+ except Exception as retry_error:
+ verbose_logger.error(
+ f"Reauthentication retry failed for project_id: {project_id}. "
+ f"Original error: {str(error)}. Retry error: {str(retry_error)}"
+ )
+ # Re-raise the original error for better context
+ raise error
+
def get_access_token(
self,
credentials: Optional[VERTEX_CREDENTIALS_TYPES],
project_id: Optional[str],
+ _retry_reauth: bool = False,
) -> Tuple[str, str]:
"""
Get access token and project id
@@ -388,6 +440,14 @@ class VertexBase:
3. Check if loaded credentials have expired
4. If expired, refresh credentials
5. Return access token and project id
+
+ Args:
+ credentials: The credentials to use for authentication
+ project_id: The Google Cloud project ID
+ _retry_reauth: Internal flag to prevent infinite recursion during reauthentication
+
+ Returns:
+ Tuple of (access_token, project_id)
"""
# Convert dict credentials to string for caching
@@ -469,14 +529,26 @@ class VertexBase:
raise ValueError("Credentials are None after loading")
if _credentials.expired:
- verbose_logger.debug(
- f"Credentials expired, refreshing for project_id: {project_id}"
- )
- self.refresh_auth(_credentials)
- self._credentials_project_mapping[credential_cache_key] = (
- _credentials,
- credential_project_id,
- )
+ try:
+ verbose_logger.debug(
+ f"Credentials expired, refreshing for project_id: {project_id}"
+ )
+ self.refresh_auth(_credentials)
+ self._credentials_project_mapping[credential_cache_key] = (
+ _credentials,
+ credential_project_id,
+ )
+ except Exception as e:
+ # if refresh fails, it's possible the user has re-authenticated via `gcloud auth application-default login`
+ # in this case, we should try to reload the credentials by clearing the cache and retrying
+ if "Reauthentication is needed" in str(e) and not _retry_reauth:
+ return self._handle_reauthentication(
+ credentials=credentials,
+ project_id=project_id,
+ credential_cache_key=credential_cache_key,
+ error=e,
+ )
+ raise e
## VALIDATION STEP
if _credentials.token is None or not isinstance(_credentials.token, str):
diff --git a/litellm/llms/vllm/common_utils.py b/litellm/llms/vllm/common_utils.py
index 8dca3e1de25..e2ed0daafe4 100644
--- a/litellm/llms/vllm/common_utils.py
+++ b/litellm/llms/vllm/common_utils.py
@@ -11,7 +11,21 @@ from litellm.utils import _add_path_to_api_base
class VLLMError(BaseLLMException):
- pass
+ def __init__(
+ self,
+ status_code: int,
+ message: str,
+ request: Optional[httpx.Request] = None,
+ response: Optional[httpx.Response] = None,
+ headers: Optional[Union[httpx.Headers, dict]] = None,
+ ):
+ super().__init__(
+ status_code=status_code,
+ message=message,
+ request=request,
+ response=response,
+ headers=headers,
+ )
class VLLMModelInfo(BaseLLMModelInfo):
@@ -25,7 +39,8 @@ class VLLMModelInfo(BaseLLMModelInfo):
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
- """Google AI Studio sends api key in query params"""
+ if api_key is not None:
+ headers["x-api-key"] = api_key
return headers
@staticmethod
@@ -53,7 +68,7 @@ class VLLMModelInfo(BaseLLMModelInfo):
endpoint = "/v1/models"
if api_base is None or api_key is None:
raise ValueError(
- "GEMINI_API_BASE or GEMINI_API_KEY is not set. Please set the environment variable, to query Gemini's `/models` endpoint."
+ "VLLM_API_BASE or VLLM_API_KEY is not set. Please set the environment variable, to query VLLM's `/models` endpoint."
)
url = _add_path_to_api_base(api_base, endpoint)
diff --git a/litellm/llms/volcengine/__init__.py b/litellm/llms/volcengine/__init__.py
new file mode 100644
index 00000000000..0887937bed5
--- /dev/null
+++ b/litellm/llms/volcengine/__init__.py
@@ -0,0 +1,24 @@
+"""
+Volcengine LLM Provider
+Support for Volcengine (ByteDance) chat and embedding models
+"""
+
+from .chat.transformation import VolcEngineChatConfig
+from .common_utils import (
+ VolcEngineError,
+ get_volcengine_base_url,
+ get_volcengine_headers,
+)
+from .embedding import VolcEngineEmbeddingConfig
+
+# For backward compatibility, keep the old class name
+VolcEngineConfig = VolcEngineChatConfig
+
+__all__ = [
+ "VolcEngineChatConfig",
+ "VolcEngineConfig", # backward compatibility
+ "VolcEngineEmbeddingConfig",
+ "VolcEngineError",
+ "get_volcengine_base_url",
+ "get_volcengine_headers",
+]
diff --git a/litellm/llms/volcengine.py b/litellm/llms/volcengine/chat/transformation.py
similarity index 71%
rename from litellm/llms/volcengine.py
rename to litellm/llms/volcengine/chat/transformation.py
index 58d2371af53..6df1cd38267 100644
--- a/litellm/llms/volcengine.py
+++ b/litellm/llms/volcengine/chat/transformation.py
@@ -3,7 +3,10 @@ from typing import Optional, Union
from litellm.llms.openai_like.chat.transformation import OpenAILikeChatConfig
-class VolcEngineConfig(OpenAILikeChatConfig):
+class VolcEngineChatConfig(OpenAILikeChatConfig):
+ """
+ Reference: https://www.volcengine.com/docs/82379/1494384
+ """
frequency_penalty: Optional[int] = None
function_call: Optional[Union[str, dict]] = None
functions: Optional[list] = None
@@ -81,8 +84,22 @@ class VolcEngineConfig(OpenAILikeChatConfig):
)
if "thinking" in optional_params:
- optional_params.setdefault("extra_body", {})["thinking"] = (
- optional_params.pop("thinking")
- )
+ """
+ The `thinking` parameters of VolcEngine model has different default values.
+ See the docs for details.
+ Refrence: https://www.volcengine.com/docs/82379/1449737#0002
+ """
+ thinking_value = optional_params.pop("thinking")
+ # Handle using thinking params case - add to extra_body if value is legal
+ if (
+ thinking_value is not None
+ and isinstance(thinking_value, dict)
+ and thinking_value.get("type", None) in ["enabled", "disabled", "auto"] # legal values, see docs
+ ):
+ # Add thinking parameter to extra_body for all legal cases
+ optional_params.setdefault("extra_body", {})["thinking"] = thinking_value
+ else:
+ # Skip adding thinking parameter when it's not set or has invalid value
+ pass
return optional_params
diff --git a/litellm/llms/volcengine/common_utils.py b/litellm/llms/volcengine/common_utils.py
new file mode 100644
index 00000000000..0c8d3daebdc
--- /dev/null
+++ b/litellm/llms/volcengine/common_utils.py
@@ -0,0 +1,62 @@
+"""
+Common utilities for Volcengine LLM provider
+"""
+
+from typing import Optional
+
+import httpx
+
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+
+
+class VolcEngineError(BaseLLMException):
+ """
+ Custom exception class for Volcengine provider errors.
+ """
+
+ def __init__(
+ self, status_code: int, message: str, headers: Optional[httpx.Headers] = None
+ ):
+ self.status_code = status_code
+ self.message = message
+ self.headers = headers or httpx.Headers()
+ super().__init__(
+ status_code=status_code, message=message, headers=dict(self.headers)
+ )
+
+
+def get_volcengine_base_url(api_base: Optional[str] = None) -> str:
+ """
+ Get the base URL for Volcengine API calls.
+
+ Args:
+ api_base: Optional custom API base URL
+
+ Returns:
+ The base URL to use for API calls
+ """
+ if api_base:
+ return api_base
+ return "https://ark.cn-beijing.volces.com"
+
+
+def get_volcengine_headers(api_key: str, extra_headers: Optional[dict] = None) -> dict:
+ """
+ Get headers for Volcengine API calls.
+
+ Args:
+ api_key: The API key for authentication
+ extra_headers: Optional additional headers
+
+ Returns:
+ Dictionary of headers
+ """
+ headers = {
+ "Content-Type": "application/json",
+ "Authorization": f"Bearer {api_key}",
+ }
+
+ if extra_headers:
+ headers.update(extra_headers)
+
+ return headers
diff --git a/litellm/llms/volcengine/embedding/__init__.py b/litellm/llms/volcengine/embedding/__init__.py
new file mode 100644
index 00000000000..7b3efc4f961
--- /dev/null
+++ b/litellm/llms/volcengine/embedding/__init__.py
@@ -0,0 +1,7 @@
+"""
+Volcengine Embedding Module
+"""
+
+from .transformation import VolcEngineEmbeddingConfig
+
+__all__ = ["VolcEngineEmbeddingConfig"]
diff --git a/litellm/llms/volcengine/embedding/transformation.py b/litellm/llms/volcengine/embedding/transformation.py
new file mode 100644
index 00000000000..20747b76725
--- /dev/null
+++ b/litellm/llms/volcengine/embedding/transformation.py
@@ -0,0 +1,211 @@
+"""
+Volcengine Embedding Transformation
+Transforms OpenAI embedding requests to Volcengine format
+"""
+
+from typing import List, Optional, Union, Dict, Any
+import httpx
+from litellm.types.llms.openai import AllEmbeddingInputValues, AllMessageValues
+from litellm.types.utils import EmbeddingResponse
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.base_llm.embedding.transformation import BaseEmbeddingConfig
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from ..common_utils import get_volcengine_base_url, get_volcengine_headers
+
+
+class VolcEngineEmbeddingConfig(BaseEmbeddingConfig):
+ """
+ Configuration class for Volcengine embedding models.
+ Reference: https://ark.cn-beijing.volces.com/api/v3/embeddings
+ """
+
+ def __init__(
+ self,
+ encoding_format: Optional[str] = None,
+ ) -> None:
+ locals_ = locals().copy()
+ for key, value in locals_.items():
+ if key != "self" and value is not None:
+ setattr(self.__class__, key, value)
+
+ @classmethod
+ def get_config(cls):
+ return super().get_config()
+
+ def get_supported_openai_params(self, model: str) -> List[str]:
+ """
+ Get the list of OpenAI parameters supported by Volcengine embedding models.
+
+ Args:
+ model: The model name
+
+ Returns:
+ List of supported parameter names
+ """
+ return [
+ "encoding_format",
+ "user",
+ "extra_headers",
+ ]
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ """
+ Get the complete URL for volcengine embedding API calls.
+
+ Args:
+ api_base: Optional custom API base URL
+ api_key: API key (not used for URL construction)
+ model: Model name (not used for URL construction)
+ optional_params: Optional parameters (not used for URL construction)
+ litellm_params: LiteLLM parameters (not used for URL construction)
+ stream: Stream parameter (not used for URL construction)
+
+ Returns:
+ Complete URL for the embedding API endpoint
+ """
+ base_url = get_volcengine_base_url(api_base)
+ # Construct the complete URL with /embeddings endpoint
+ if base_url.endswith("/api/v3"):
+ return f"{base_url}/embeddings"
+ else:
+ return f"{base_url}/api/v3/embeddings"
+
+ def map_openai_params(
+ self,
+ non_default_params: Dict[str, Any],
+ optional_params: Dict[str, Any],
+ model: str,
+ drop_params: bool,
+ ) -> Dict[str, Any]:
+ """
+ Map OpenAI embedding parameters to Volcengine format.
+
+ Args:
+ non_default_params: Parameters that are not default values
+ optional_params: Optional parameters dict to update
+ model: The model name
+ drop_params: Whether to drop unsupported parameters
+
+ Returns:
+ Updated optional_params dict
+ """
+ for param, value in non_default_params.items():
+ if param == "encoding_format":
+ # Volcengine supports: float, base64, null
+ if value in ["float", "base64", None]:
+ optional_params["encoding_format"] = value
+ else:
+ if not drop_params:
+ raise ValueError(
+ f"Unsupported encoding_format: {value}. Volcengine supports: float, base64, null"
+ )
+ elif param == "user":
+ # Keep user parameter as-is
+ optional_params["user"] = value
+ elif param in self.get_supported_openai_params(model):
+ optional_params[param] = value
+ elif not drop_params:
+ raise ValueError(f"Unsupported parameter for Volcengine: {param}")
+
+ return optional_params
+
+
+
+ def transform_embedding_request(
+ self,
+ model: str,
+ input: AllEmbeddingInputValues,
+ optional_params: dict,
+ headers: dict,
+ ) -> dict:
+ """Transform embedding request to Volcengine format"""
+ # Prepare request data (only the JSON body, not the full request)
+ data = {
+ "model": model,
+ "input": input if isinstance(input, list) else [input],
+ }
+
+ # Add optional parameters from optional_params
+ if "encoding_format" in optional_params:
+ encoding_format = optional_params["encoding_format"]
+ if encoding_format is not None:
+ data["encoding_format"] = encoding_format
+
+ if "user" in optional_params:
+ user = optional_params["user"]
+ if user is not None:
+ data["user"] = user
+
+ return data
+
+ def transform_embedding_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: EmbeddingResponse,
+ logging_obj: LiteLLMLoggingObj,
+ api_key: Optional[str],
+ request_data: dict,
+ optional_params: dict,
+ litellm_params: dict,
+ ) -> EmbeddingResponse:
+ """Transform Volcengine response to EmbeddingResponse"""
+ try:
+ response_json = raw_response.json()
+ except Exception as e:
+ raise ValueError(f"Failed to parse Volcengine response as JSON: {str(e)}")
+
+ # Volcengine response format matches OpenAI format closely
+ # Just need to ensure all required fields are present
+ transformed_response = {
+ "object": "list",
+ "data": response_json.get("data", []),
+ "model": response_json.get("model", model),
+ "usage": response_json.get("usage", {}),
+ }
+
+ # Add id if present
+ if "id" in response_json:
+ transformed_response["id"] = response_json["id"]
+
+ # Create EmbeddingResponse from transformed data
+ return EmbeddingResponse(**transformed_response)
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ """Validate environment and return headers"""
+ # Get Volcengine headers
+ if api_key is None:
+ raise ValueError("api_key is required for Volcengine authentication")
+ volcengine_headers = get_volcengine_headers(api_key)
+ return {**headers, **volcengine_headers}
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ """Get error class for Volcengine errors"""
+ from ..common_utils import VolcEngineError
+ # Convert dict to httpx.Headers if needed
+ if isinstance(headers, dict):
+ headers = httpx.Headers(headers)
+ return VolcEngineError(
+ status_code=status_code,
+ message=error_message,
+ headers=headers,
+ )
diff --git a/litellm/llms/voyage/embedding/transformation_contextual.py b/litellm/llms/voyage/embedding/transformation_contextual.py
new file mode 100644
index 00000000000..4df2fa4ba31
--- /dev/null
+++ b/litellm/llms/voyage/embedding/transformation_contextual.py
@@ -0,0 +1,153 @@
+"""
+This module is used to transform the request and response for the Voyage contextualized embeddings API.
+This would be used for all the contextualized embeddings models in Voyage.
+"""
+from typing import List, Optional, Union
+
+import httpx
+
+from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
+from litellm.llms.base_llm.chat.transformation import BaseLLMException
+from litellm.llms.base_llm.embedding.transformation import BaseEmbeddingConfig
+from litellm.secret_managers.main import get_secret_str
+from litellm.types.llms.openai import AllEmbeddingInputValues, AllMessageValues
+from litellm.types.utils import EmbeddingResponse, Usage
+
+
+class VoyageError(BaseLLMException):
+ def __init__(
+ self,
+ status_code: int,
+ message: str,
+ headers: Union[dict, httpx.Headers] = {},
+ ):
+ self.status_code = status_code
+ self.message = message
+ self.request = httpx.Request(
+ method="POST", url="https://api.voyageai.com/v1/contextualizedembeddings"
+ )
+ self.response = httpx.Response(status_code=status_code, request=self.request)
+ super().__init__(
+ status_code=status_code,
+ message=message,
+ headers=headers,
+ )
+
+
+class VoyageContextualEmbeddingConfig(BaseEmbeddingConfig):
+ """
+ Reference: https://docs.voyageai.com/reference/embeddings-api
+ """
+
+ def __init__(self) -> None:
+ pass
+
+ def get_complete_url(
+ self,
+ api_base: Optional[str],
+ api_key: Optional[str],
+ model: str,
+ optional_params: dict,
+ litellm_params: dict,
+ stream: Optional[bool] = None,
+ ) -> str:
+ if api_base:
+ if not api_base.endswith("/contextualizedembeddings"):
+ api_base = f"{api_base}/contextualizedembeddings"
+ return api_base
+ return "https://api.voyageai.com/v1/contextualizedembeddings"
+
+ def get_supported_openai_params(self, model: str) -> list:
+ return ["encoding_format", "dimensions"]
+
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ """
+ Map OpenAI params to Voyage params
+
+ Reference: https://docs.voyageai.com/reference/contextualized-embeddings-api
+ """
+ if "encoding_format" in non_default_params:
+ optional_params["encoding_format"] = non_default_params["encoding_format"]
+ if "dimensions" in non_default_params:
+ optional_params["output_dimension"] = non_default_params["dimensions"]
+ return optional_params
+
+ def validate_environment(
+ self,
+ headers: dict,
+ model: str,
+ messages: List[AllMessageValues],
+ optional_params: dict,
+ litellm_params: dict,
+ api_key: Optional[str] = None,
+ api_base: Optional[str] = None,
+ ) -> dict:
+ if api_key is None:
+ api_key = (
+ get_secret_str("VOYAGE_API_KEY")
+ or get_secret_str("VOYAGE_AI_API_KEY")
+ or get_secret_str("VOYAGE_AI_TOKEN")
+ )
+ return {
+ "Authorization": f"Bearer {api_key}",
+ }
+
+ def transform_embedding_request(
+ self,
+ model: str,
+ input: Union[AllEmbeddingInputValues, List[List[str]]],
+ optional_params: dict,
+ headers: dict,
+ ) -> dict:
+ return {
+ "inputs": input,
+ "model": model,
+ **optional_params,
+ }
+
+ def transform_embedding_response(
+ self,
+ model: str,
+ raw_response: httpx.Response,
+ model_response: EmbeddingResponse,
+ logging_obj: LiteLLMLoggingObj,
+ api_key: Optional[str] = None,
+ request_data: dict = {},
+ optional_params: dict = {},
+ litellm_params: dict = {},
+ ) -> EmbeddingResponse:
+ try:
+ raw_response_json = raw_response.json()
+ except Exception:
+ raise VoyageError(
+ message=raw_response.text, status_code=raw_response.status_code
+ )
+
+ # model_response.usage
+ model_response.model = raw_response_json.get("model")
+ model_response.data = raw_response_json.get("data")
+ model_response.object = raw_response_json.get("object")
+
+ usage = Usage(
+ prompt_tokens=raw_response_json.get("usage", {}).get("total_tokens", 0),
+ total_tokens=raw_response_json.get("usage", {}).get("total_tokens", 0),
+ )
+ model_response.usage = usage
+ return model_response
+
+ def get_error_class(
+ self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
+ ) -> BaseLLMException:
+ return VoyageError(
+ message=error_message, status_code=status_code, headers=headers
+ )
+
+ @staticmethod
+ def is_contextualized_embeddings(model: str) -> bool:
+ return "context" in model.lower()
diff --git a/litellm/llms/wandb/__init__.py b/litellm/llms/wandb/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/litellm/llms/wandb/chat/__init__.py b/litellm/llms/wandb/chat/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/litellm/llms/wandb/chat/transformation.py b/litellm/llms/wandb/chat/transformation.py
new file mode 100644
index 00000000000..1cb2ab492bc
--- /dev/null
+++ b/litellm/llms/wandb/chat/transformation.py
@@ -0,0 +1,27 @@
+"""
+Wandb Chat Completions API - Transformation
+
+This is OpenAI compatible - no translation needed / occurs
+"""
+
+from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
+
+
+class WandbConfig(OpenAIGPTConfig):
+ def map_openai_params(
+ self,
+ non_default_params: dict,
+ optional_params: dict,
+ model: str,
+ drop_params: bool,
+ ) -> dict:
+ """
+ map max_completion_tokens param to max_tokens
+ """
+ supported_openai_params = self.get_supported_openai_params(model=model)
+ for param, value in non_default_params.items():
+ if param == "max_completion_tokens":
+ optional_params["max_tokens"] = value
+ elif param in supported_openai_params:
+ optional_params[param] = value
+ return optional_params
diff --git a/litellm/llms/watsonx/chat/handler.py b/litellm/llms/watsonx/chat/handler.py
index 5c19757fecb..bc0effe4a1a 100644
--- a/litellm/llms/watsonx/chat/handler.py
+++ b/litellm/llms/watsonx/chat/handler.py
@@ -21,7 +21,7 @@ class WatsonXChatHandler(OpenAILikeChatHandler):
*,
model: str,
messages: list,
- api_base: str,
+ api_base: Optional[str],
custom_llm_provider: str,
custom_prompt_dict: dict,
model_response: ModelResponse,
@@ -70,7 +70,7 @@ class WatsonXChatHandler(OpenAILikeChatHandler):
)
return super().completion(
- model=watsonx_auth_payload.get("model_id", None),
+ model=watsonx_auth_payload.get("model_id") or "",
messages=messages,
api_base=api_base,
custom_llm_provider=custom_llm_provider,
diff --git a/litellm/llms/watsonx/chat/transformation.py b/litellm/llms/watsonx/chat/transformation.py
index 6b0dd5a39ae..2c096cafced 100644
--- a/litellm/llms/watsonx/chat/transformation.py
+++ b/litellm/llms/watsonx/chat/transformation.py
@@ -4,10 +4,14 @@ Translation from OpenAI's `/chat/completions` endpoint to IBM WatsonX's `/text/c
Docs: https://cloud.ibm.com/apidocs/watsonx-ai#text-chat
"""
-from typing import List, Optional, Tuple, Union
+from typing import Dict, List, Optional, Tuple, Union
from litellm.secret_managers.main import get_secret_str
-from litellm.types.llms.watsonx import WatsonXAIEndpoint, WatsonXAPIParams
+from litellm.types.llms.watsonx import (
+ WatsonXAIEndpoint,
+ WatsonXAPIParams,
+ WatsonXModelPattern,
+)
from ....utils import _remove_additional_properties, _remove_strict_from_schema
from ...openai.chat.gpt_transformation import OpenAIGPTConfig
@@ -120,3 +124,95 @@ class IBMWatsonXChatConfig(IBMWatsonXMixin, OpenAIGPTConfig):
None if model.startswith("deployment/") else api_params["project_id"]
)
return payload
+
+ @staticmethod
+ def _apply_prompt_template_core(model: str, messages: List[Dict[str, str]], hf_template_fn) -> Optional[str]:
+ """Core logic for applying prompt templates"""
+ from litellm.litellm_core_utils.prompt_templates.factory import (
+ custom_prompt,
+ ibm_granite_pt,
+ mistral_instruct_pt,
+ )
+
+ if WatsonXModelPattern.GRANITE_CHAT.value in model:
+ return ibm_granite_pt(messages=messages)
+ elif WatsonXModelPattern.IBM_MISTRAL.value in model:
+ return mistral_instruct_pt(messages=messages)
+ elif WatsonXModelPattern.GPT_OSS.value in model:
+ hf_model = model.split("watsonx/")[-1] if "watsonx/" in model else model
+ try:
+ return hf_template_fn(model=hf_model, messages=messages)
+ except Exception:
+ pass
+ elif WatsonXModelPattern.LLAMA3_INSTRUCT.value in model:
+ return custom_prompt(
+ role_dict={
+ "system": {"pre_message": "<|start_header_id|>system<|end_header_id|>\n", "post_message": "<|eot_id|>"},
+ "user": {"pre_message": "<|start_header_id|>user<|end_header_id|>\n", "post_message": "<|eot_id|>"},
+ "assistant": {"pre_message": "<|start_header_id|>assistant<|end_header_id|>\n", "post_message": "<|eot_id|>"},
+ },
+ messages=messages,
+ initial_prompt_value="<|begin_of_text|>",
+ final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
+ )
+ return None
+
+ @staticmethod
+ async def aapply_prompt_template(model: str, messages: List[Dict[str, str]]) -> Optional[str]:
+ """Apply prompt template (async version)"""
+ import litellm
+ from litellm.litellm_core_utils.prompt_templates.factory import (
+ ahf_chat_template,
+ custom_prompt,
+ hf_chat_template,
+ ibm_granite_pt,
+ mistral_instruct_pt,
+ )
+
+ if WatsonXModelPattern.GRANITE_CHAT.value in model:
+ return ibm_granite_pt(messages=messages)
+ elif WatsonXModelPattern.IBM_MISTRAL.value in model:
+ return mistral_instruct_pt(messages=messages)
+ elif WatsonXModelPattern.GPT_OSS.value in model:
+ hf_model = model.split("watsonx/")[-1] if "watsonx/" in model else model
+ try:
+ # Use sync if cached, async if not
+ if hf_model in litellm.known_tokenizer_config:
+ return hf_chat_template(model=hf_model, messages=messages)
+ else:
+ return await ahf_chat_template(model=hf_model, messages=messages)
+ except Exception:
+ pass
+ elif WatsonXModelPattern.LLAMA3_INSTRUCT.value in model:
+ return custom_prompt(
+ role_dict={
+ "system": {
+ "pre_message": "<|start_header_id|>system<|end_header_id|>\n",
+ "post_message": "<|eot_id|>",
+ },
+ "user": {
+ "pre_message": "<|start_header_id|>user<|end_header_id|>\n",
+ "post_message": "<|eot_id|>",
+ },
+ "assistant": {
+ "pre_message": "<|start_header_id|>assistant<|end_header_id|>\n",
+ "post_message": "<|eot_id|>",
+ },
+ },
+ messages=messages,
+ initial_prompt_value="<|begin_of_text|>",
+ final_prompt_value="<|start_header_id|>assistant<|end_header_id|>\n",
+ )
+ return None
+
+ @staticmethod
+ def apply_prompt_template(model: str, messages: List[Dict[str, str]]) -> Optional[str]:
+ """Apply prompt template (sync version)"""
+ from litellm.litellm_core_utils.prompt_templates.factory import (
+ hf_chat_template,
+ )
+
+ return IBMWatsonXChatConfig._apply_prompt_template_core(
+ model=model, messages=messages, hf_template_fn=hf_chat_template
+ )
+
diff --git a/litellm/llms/watsonx/common_utils.py b/litellm/llms/watsonx/common_utils.py
index c756be6d458..58b33097cbd 100644
--- a/litellm/llms/watsonx/common_utils.py
+++ b/litellm/llms/watsonx/common_utils.py
@@ -131,34 +131,102 @@ def _get_api_params(
)
-def convert_watsonx_messages_to_prompt(
+async def _aconvert_watsonx_messages_core(
model: str,
messages: List[AllMessageValues],
provider: str,
custom_prompt_dict: Dict,
+ apply_template_fn,
) -> str:
+ """Async core logic for converting watsonx messages to prompt"""
+ from litellm.types.llms.watsonx import WatsonXModelPattern
+
# handle anthropic prompts and amazon titan prompts
if model in custom_prompt_dict:
- # check if the model has a registered custom prompt
model_prompt_dict = custom_prompt_dict[model]
- prompt = ptf.custom_prompt(
+ return ptf.custom_prompt(
messages=messages,
- role_dict=model_prompt_dict.get(
- "role_dict", model_prompt_dict.get("roles")
- ),
+ role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
bos_token=model_prompt_dict.get("bos_token", ""),
eos_token=model_prompt_dict.get("eos_token", ""),
)
- return prompt
- elif provider == "ibm-mistralai":
- prompt = ptf.mistral_instruct_pt(messages=messages)
+ elif provider == WatsonXModelPattern.IBM_MISTRALAI.value:
+ return ptf.mistral_instruct_pt(messages=messages)
else:
- prompt: str = ptf.prompt_factory( # type: ignore
+ # Try applying specific template first
+ result = await apply_template_fn(model=model, messages=messages)
+ if result:
+ return result
+ # Fallback to default
+ return ptf.prompt_factory(
model=model, messages=messages, custom_llm_provider="watsonx"
+ ) # type: ignore
+
+
+def _convert_watsonx_messages_core(
+ model: str,
+ messages: List[AllMessageValues],
+ provider: str,
+ custom_prompt_dict: Dict,
+ apply_template_fn,
+) -> str:
+ """Sync core logic for converting watsonx messages to prompt"""
+ from litellm.types.llms.watsonx import WatsonXModelPattern
+
+ # handle anthropic prompts and amazon titan prompts
+ if model in custom_prompt_dict:
+ model_prompt_dict = custom_prompt_dict[model]
+ return ptf.custom_prompt(
+ messages=messages,
+ role_dict=model_prompt_dict.get("role_dict", model_prompt_dict.get("roles")),
+ initial_prompt_value=model_prompt_dict.get("initial_prompt_value", ""),
+ final_prompt_value=model_prompt_dict.get("final_prompt_value", ""),
+ bos_token=model_prompt_dict.get("bos_token", ""),
+ eos_token=model_prompt_dict.get("eos_token", ""),
)
- return prompt
+ elif provider == WatsonXModelPattern.IBM_MISTRALAI.value:
+ return ptf.mistral_instruct_pt(messages=messages)
+ else:
+ # Try applying specific template first
+ result = apply_template_fn(model=model, messages=messages)
+ if result:
+ return result
+ # Fallback to default
+ return ptf.prompt_factory(
+ model=model, messages=messages, custom_llm_provider="watsonx"
+ ) # type: ignore
+
+
+async def aconvert_watsonx_messages_to_prompt(
+ model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
+) -> str:
+ """Async version of convert_watsonx_messages_to_prompt"""
+ from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
+
+ return await _aconvert_watsonx_messages_core(
+ model=model,
+ messages=messages,
+ provider=provider,
+ custom_prompt_dict=custom_prompt_dict,
+ apply_template_fn=IBMWatsonXChatConfig.aapply_prompt_template,
+ )
+
+
+def convert_watsonx_messages_to_prompt(
+ model: str, messages: List[AllMessageValues], provider: str, custom_prompt_dict: Dict
+) -> str:
+ """Sync version of convert_watsonx_messages_to_prompt"""
+ from litellm.llms.watsonx.chat.transformation import IBMWatsonXChatConfig
+
+ return _convert_watsonx_messages_core(
+ model=model,
+ messages=messages,
+ provider=provider,
+ custom_prompt_dict=custom_prompt_dict,
+ apply_template_fn=IBMWatsonXChatConfig.apply_prompt_template,
+ )
# Mixin class for shared IBM Watson X functionality
diff --git a/litellm/llms/watsonx/completion/transformation.py b/litellm/llms/watsonx/completion/transformation.py
index a0b9735a990..3c1229ecd2b 100644
--- a/litellm/llms/watsonx/completion/transformation.py
+++ b/litellm/llms/watsonx/completion/transformation.py
@@ -228,39 +228,35 @@ class IBMWatsonXAIConfig(IBMWatsonXMixin, BaseConfig):
"us-south",
]
- def transform_request(
- self,
- model: str,
- messages: List[AllMessageValues],
- optional_params: Dict,
- litellm_params: Dict,
- headers: Dict,
- ) -> Dict:
- provider = model.split("/")[0]
- prompt = convert_watsonx_messages_to_prompt(
- model=model,
- messages=messages,
- provider=provider,
- custom_prompt_dict={},
- )
+ def _build_request_payload(self, model: str, prompt: str, optional_params: Dict) -> Dict:
+ """Shared logic to build request payload"""
extra_body_params = optional_params.pop("extra_body", {})
optional_params.update(extra_body_params)
watsonx_api_params = _get_api_params(params=optional_params)
-
- watsonx_auth_payload = self._prepare_payload(
- model=model,
- api_params=watsonx_api_params,
- )
-
- # init the payload to the text generation call
- payload = {
+ watsonx_auth_payload = self._prepare_payload(model=model, api_params=watsonx_api_params)
+
+ return {
"input": prompt,
"moderations": optional_params.pop("moderations", {}),
"parameters": optional_params,
**watsonx_auth_payload,
}
- return payload
+ async def atransform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
+ """Async version of transform_request"""
+ from litellm.llms.watsonx.common_utils import (
+ aconvert_watsonx_messages_to_prompt,
+ )
+
+ provider = model.split("/")[0]
+ prompt = await aconvert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
+ return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
+
+ def transform_request(self, model: str, messages: List[AllMessageValues], optional_params: Dict, litellm_params: Dict, headers: Dict) -> Dict:
+ """Sync version of transform_request"""
+ provider = model.split("/")[0]
+ prompt = convert_watsonx_messages_to_prompt(model=model, messages=messages, provider=provider, custom_prompt_dict={})
+ return self._build_request_payload(model=model, prompt=prompt, optional_params=optional_params)
def transform_response(
self,
diff --git a/litellm/llms/xai/chat/transformation.py b/litellm/llms/xai/chat/transformation.py
index 5a488876cd9..b01f6c18466 100644
--- a/litellm/llms/xai/chat/transformation.py
+++ b/litellm/llms/xai/chat/transformation.py
@@ -31,7 +31,6 @@ class XAIChatConfig(OpenAIGPTConfig):
def get_supported_openai_params(self, model: str) -> list:
base_openai_params = [
- "frequency_penalty",
"logit_bias",
"logprobs",
"max_tokens",
@@ -50,8 +49,22 @@ class XAIChatConfig(OpenAIGPTConfig):
"web_search_options",
]
# for some reason, grok-3-mini does not support stop tokens
+ #########################################################
+ # stop tokens check
+ #########################################################
if self._supports_stop_reason(model):
base_openai_params.append("stop")
+
+
+ #########################################################
+ # frequency penalty check
+ #########################################################
+ if self._supports_frequency_penalty(model):
+ base_openai_params.append("frequency_penalty")
+
+ #########################################################
+ # reasoning check
+ #########################################################
try:
if litellm.supports_reasoning(
model=model, custom_llm_provider=self.custom_llm_provider
@@ -67,6 +80,20 @@ class XAIChatConfig(OpenAIGPTConfig):
return False
elif "grok-4" in model:
return False
+ elif "grok-code-fast" in model:
+ return False
+ return True
+
+ def _supports_frequency_penalty(self, model: str) -> bool:
+ """
+ From manual testing grok-4 does not support `frequency_penalty`
+
+ When sent the model fails from xAI API
+ """
+ if "grok-4" in model:
+ return False
+ if "grok-code-fast" in model:
+ return False
return True
def map_openai_params(
diff --git a/litellm/llms/xai/cost_calculator.py b/litellm/llms/xai/cost_calculator.py
new file mode 100644
index 00000000000..62a48080d1c
--- /dev/null
+++ b/litellm/llms/xai/cost_calculator.py
@@ -0,0 +1,54 @@
+"""
+Helper util for handling XAI-specific cost calculation
+- e.g.: reasoning tokens for grok models
+"""
+
+from typing import Tuple, Union
+
+from litellm.types.utils import Usage
+from litellm.utils import get_model_info
+
+
+def cost_per_token(model: str, usage: Usage) -> Tuple[float, float]:
+ """
+ Calculates the cost per token for a given XAI model, prompt tokens, and completion tokens.
+
+ Input:
+ - model: str, the model name without provider prefix
+ - usage: LiteLLM Usage block, containing XAI-specific usage information
+
+ Returns:
+ Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
+ """
+ ## GET MODEL INFO
+ model_info = get_model_info(model=model, custom_llm_provider="xai")
+
+ def _safe_float_cast(
+ value: Union[str, int, float, None, object], default: float = 0.0
+ ) -> float:
+ """Safely cast a value to float with proper type handling for mypy."""
+ if value is None:
+ return default
+ try:
+ return float(value) # type: ignore
+ except (ValueError, TypeError):
+ return default
+
+ ## CALCULATE INPUT COST
+ input_cost_per_token = _safe_float_cast(model_info.get("input_cost_per_token"))
+ prompt_cost: float = (usage.prompt_tokens or 0) * input_cost_per_token
+
+ ## CALCULATE OUTPUT COST
+ output_cost_per_token = _safe_float_cast(model_info.get("output_cost_per_token"))
+
+ # For XAI models, completion is billed as (visible completion tokens + reasoning tokens)
+ completion_tokens = int(getattr(usage, "completion_tokens", 0) or 0)
+ reasoning_tokens = 0
+ if hasattr(usage, "completion_tokens_details") and usage.completion_tokens_details:
+ reasoning_tokens = int(
+ getattr(usage.completion_tokens_details, "reasoning_tokens", 0) or 0
+ )
+
+ completion_cost = (completion_tokens + reasoning_tokens) * output_cost_per_token
+
+ return prompt_cost, completion_cost
diff --git a/litellm/main.py b/litellm/main.py
index fb6204c5bf0..3955a0f32f8 100644
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -17,13 +17,14 @@ import random
import sys
import time
import traceback
-import uuid
from concurrent import futures
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
from copy import deepcopy
from functools import partial
from typing import (
+ TYPE_CHECKING,
Any,
+ AsyncIterator,
Callable,
Coroutine,
Dict,
@@ -38,6 +39,11 @@ from typing import (
get_args,
)
+from litellm._uuid import uuid
+
+if TYPE_CHECKING:
+ from aiohttp import ClientSession
+
import dotenv
import httpx
import openai
@@ -61,6 +67,9 @@ from litellm.exceptions import LiteLLMUnknownProvider
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.audio_utils.utils import get_audio_file_for_health_check
from litellm.litellm_core_utils.dd_tracing import tracer
+from litellm.litellm_core_utils.get_provider_specific_headers import (
+ ProviderSpecificHeaderUtils,
+)
from litellm.litellm_core_utils.health_check_utils import (
_create_health_check_response,
_filter_model_params,
@@ -76,8 +85,12 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
from litellm.llms.base_llm import BaseConfig, BaseImageGenerationConfig
from litellm.llms.bedrock.common_utils import BedrockModelInfo
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
+from litellm.llms.vertex_ai.common_utils import (
+ VertexAIModelRoute,
+ get_vertex_ai_model_route,
+)
from litellm.realtime_api.main import _realtime_health_check
-from litellm.secret_managers.main import get_secret_str
+from litellm.secret_managers.main import get_secret_bool, get_secret_str
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import RawRequestTypedDict
from litellm.utils import (
@@ -107,11 +120,13 @@ from litellm.utils import (
supports_httpx_timeout,
token_counter,
validate_and_fix_openai_messages,
+ validate_and_fix_openai_tools,
validate_chat_completion_tool_choice,
)
from ._logging import verbose_logger
from .caching.caching import disable_cache, enable_cache, update_cache
+from .litellm_core_utils.core_helpers import safe_deep_copy
from .litellm_core_utils.fallback_utils import (
async_completion_with_fallbacks,
completion_with_fallbacks,
@@ -129,7 +144,6 @@ from .litellm_core_utils.prompt_templates.factory import (
stringify_json_tool_call_content,
)
from .litellm_core_utils.streaming_chunk_builder_utils import ChunkProcessor
-from .llms import baseten
from .llms.anthropic.chat import AnthropicChatCompletion
from .llms.azure.audio_transcriptions import AzureAudioTranscription
from .llms.azure.azure import AzureChatCompletion, _check_dynamic_azure_params
@@ -147,9 +161,13 @@ from .llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from .llms.custom_llm import CustomLLM, custom_chat_llm_router
from .llms.databricks.embed.handler import DatabricksEmbeddingHandler
from .llms.deprecated_providers import aleph_alpha, palm
+from .llms.gemini.common_utils import get_api_key_from_env
from .llms.groq.chat.handler import GroqChatCompletion
+from .llms.heroku.chat.transformation import HerokuChatConfig
from .llms.huggingface.embedding.handler import HuggingFaceEmbedding
+from .llms.lemonade.chat.transformation import LemonadeChatConfig
from .llms.nlp_cloud.chat.handler import completion as nlp_cloud_chat_completion
+from .llms.oci.chat.transformation import OCIChatConfig
from .llms.ollama.completion import handler as ollama
from .llms.oobabooga.chat import oobabooga
from .llms.openai.completion.handler import OpenAITextCompletion
@@ -158,6 +176,7 @@ from .llms.openai.openai import OpenAIChatCompletion
from .llms.openai.transcriptions.handler import OpenAIAudioTranscription
from .llms.openai_like.chat.handler import OpenAILikeChatHandler
from .llms.openai_like.embedding.handler import OpenAILikeEmbeddingHandler
+from .llms.ovhcloud.chat.transformation import OVHCloudChatConfig
from .llms.petals.completion import handler as petals_handler
from .llms.predibase.chat.handler import PredibaseChatCompletion
from .llms.replicate.chat.handler import completion as replicate_chat_completion
@@ -177,6 +196,7 @@ from .llms.vertex_ai.multimodal_embeddings.embedding_handler import (
from .llms.vertex_ai.text_to_speech.text_to_speech_handler import VertexTextToSpeechAPI
from .llms.vertex_ai.vertex_ai_partner_models.main import VertexAIPartnerModels
from .llms.vertex_ai.vertex_embeddings.embedding_handler import VertexEmbedding
+from .llms.vertex_ai.vertex_gemma_models.main import VertexAIGemmaModels
from .llms.vertex_ai.vertex_model_garden.main import VertexAIModelGardenModels
from .llms.vllm.completion import handler as vllm_handler
from .llms.watsonx.chat.handler import WatsonXChatHandler
@@ -240,6 +260,7 @@ vertex_multimodal_embedding = VertexMultimodalEmbedding()
vertex_image_generation = VertexImageGeneration()
google_batch_embeddings = GoogleBatchEmbeddings()
vertex_partner_models_chat_completion = VertexAIPartnerModels()
+vertex_gemma_chat_completion = VertexAIGemmaModels()
vertex_model_garden_chat_completion = VertexAIModelGardenModels()
vertex_text_to_speech = VertexTextToSpeechAPI()
sagemaker_llm = SagemakerLLM()
@@ -251,6 +272,10 @@ base_llm_http_handler = BaseLLMHTTPHandler()
base_llm_aiohttp_handler = BaseLLMAIOHTTPHandler()
sagemaker_chat_completion = SagemakerChatHandler()
bytez_transformation = BytezChatConfig()
+heroku_transformation = HerokuChatConfig()
+oci_transformation = OCIChatConfig()
+ovhcloud_transformation = OVHCloudChatConfig()
+lemonade_transformation = LemonadeChatConfig()
####### COMPLETION ENDPOINTS ################
@@ -350,7 +375,10 @@ async def acompletion(
logprobs: Optional[bool] = None,
top_logprobs: Optional[int] = None,
deployment_id=None,
- reasoning_effort: Optional[Literal["low", "medium", "high"]] = None,
+ reasoning_effort: Optional[
+ Literal["none", "minimal", "low", "medium", "high", "default"]
+ ] = None,
+ safety_identifier: Optional[str] = None,
# set api_base, api_version, api_key
base_url: Optional[str] = None,
api_version: Optional[str] = None,
@@ -360,6 +388,8 @@ async def acompletion(
# Optional liteLLM function params
thinking: Optional[AnthropicThinkingParam] = None,
web_search_options: Optional[OpenAIWebSearchOptions] = None,
+ # Session management
+ shared_session: Optional["ClientSession"] = None,
**kwargs,
) -> Union[ModelResponse, CustomStreamWrapper]:
"""
@@ -452,6 +482,16 @@ async def acompletion(
#########################################################
#########################################################
+ # Log shared session usage
+ if shared_session is not None:
+ verbose_logger.debug(
+ f"🔄 SHARED SESSION: acompletion called with shared_session (ID: {id(shared_session)})"
+ )
+ else:
+ verbose_logger.debug(
+ "🔄 NO SHARED SESSION: acompletion called without shared_session"
+ )
+
# Adjusted to use explicit arguments instead of *args and **kwargs
completion_kwargs = {
"model": model,
@@ -487,14 +527,18 @@ async def acompletion(
"api_key": api_key,
"model_list": model_list,
"reasoning_effort": reasoning_effort,
+ "safety_identifier": safety_identifier,
"extra_headers": extra_headers,
"acompletion": True, # assuming this is a required parameter
"thinking": thinking,
"web_search_options": web_search_options,
+ "shared_session": shared_session,
}
if custom_llm_provider is None:
_, custom_llm_provider, _, _ = get_llm_provider(
- model=model, api_base=completion_kwargs.get("base_url", None)
+ model=model,
+ custom_llm_provider=custom_llm_provider,
+ api_base=completion_kwargs.get("base_url", None),
)
fallbacks = fallbacks or litellm.model_fallbacks
@@ -687,12 +731,15 @@ async def _sleep_for_timeout_async(timeout: Union[float, str, httpx.Timeout]):
await asyncio.sleep(timeout.connect)
+MOCK_RESPONSE_TYPE = Union[str, Exception, dict]
+
+
def mock_completion(
model: str,
messages: List,
stream: Optional[bool] = False,
n: Optional[int] = None,
- mock_response: Union[str, Exception, dict] = "This is a mock request",
+ mock_response: Optional[MOCK_RESPONSE_TYPE] = "This is a mock request",
mock_tool_calls: Optional[List] = None,
mock_timeout: Optional[bool] = False,
logging=None,
@@ -889,7 +936,9 @@ def completion( # type: ignore # noqa: PLR0915
logit_bias: Optional[dict] = None,
user: Optional[str] = None,
# openai v1.0+ new params
- reasoning_effort: Optional[Literal["low", "medium", "high"]] = None,
+ reasoning_effort: Optional[
+ Literal["none", "minimal", "low", "medium", "high", "default"]
+ ] = None,
response_format: Optional[Union[dict, Type[BaseModel]]] = None,
seed: Optional[int] = None,
tools: Optional[List] = None,
@@ -900,6 +949,7 @@ def completion( # type: ignore # noqa: PLR0915
web_search_options: Optional[OpenAIWebSearchOptions] = None,
deployment_id=None,
extra_headers: Optional[dict] = None,
+ safety_identifier: Optional[str] = None,
# soon to be deprecated params by OpenAI
functions: Optional[List] = None,
function_call: Optional[str] = None,
@@ -910,6 +960,8 @@ def completion( # type: ignore # noqa: PLR0915
model_list: Optional[list] = None, # pass in a list of api_base,keys, etc.
# Optional liteLLM function params
thinking: Optional[AnthropicThinkingParam] = None,
+ # Session management
+ shared_session: Optional["ClientSession"] = None,
**kwargs,
) -> Union[ModelResponse, CustomStreamWrapper]:
"""
@@ -962,12 +1014,13 @@ def completion( # type: ignore # noqa: PLR0915
raise ValueError("model param not passed in.")
# validate messages
messages = validate_and_fix_openai_messages(messages=messages)
+ tools = validate_and_fix_openai_tools(tools=tools)
# validate tool_choice
tool_choice = validate_chat_completion_tool_choice(tool_choice=tool_choice)
######### unpacking kwargs #####################
args = locals()
api_base = kwargs.get("api_base", None)
- mock_response = kwargs.get("mock_response", None)
+ mock_response: Optional[MOCK_RESPONSE_TYPE] = kwargs.get("mock_response", None)
mock_tool_calls = kwargs.get("mock_tool_calls", None)
mock_timeout = cast(Optional[bool], kwargs.get("mock_timeout", None))
force_timeout = kwargs.get("force_timeout", 600) ## deprecated
@@ -1048,11 +1101,13 @@ def completion( # type: ignore # noqa: PLR0915
non_default_params = get_non_default_completion_params(kwargs=kwargs)
litellm_params = {} # used to prevent unbound var errors
## PROMPT MANAGEMENT HOOKS ##
+
if isinstance(litellm_logging_obj, LiteLLMLoggingObj) and (
litellm_logging_obj.should_run_prompt_management_hooks(
prompt_id=prompt_id, non_default_params=non_default_params
)
):
+
(
model,
messages,
@@ -1072,7 +1127,7 @@ def completion( # type: ignore # noqa: PLR0915
api_base = base_url
if num_retries is not None:
max_retries = num_retries
- logging = litellm_logging_obj
+ logging: Logging = cast(Logging, litellm_logging_obj)
fallbacks = fallbacks or litellm.model_fallbacks
if fallbacks is not None:
return completion_with_fallbacks(**args)
@@ -1101,11 +1156,13 @@ def completion( # type: ignore # noqa: PLR0915
api_key=api_key,
)
- if (
- provider_specific_header is not None
- and provider_specific_header["custom_llm_provider"] == custom_llm_provider
- ):
- headers.update(provider_specific_header["extra_headers"])
+ if provider_specific_header is not None:
+ headers.update(
+ ProviderSpecificHeaderUtils.get_provider_specific_headers(
+ provider_specific_header=provider_specific_header,
+ custom_llm_provider=custom_llm_provider,
+ )
+ )
if model_response is not None and hasattr(model_response, "_hidden_params"):
model_response._hidden_params["custom_llm_provider"] = custom_llm_provider
@@ -1234,6 +1291,7 @@ def completion( # type: ignore # noqa: PLR0915
"reasoning_effort": reasoning_effort,
"thinking": thinking,
"web_search_options": web_search_options,
+ "safety_identifier": safety_identifier,
"allowed_openai_params": kwargs.get("allowed_openai_params"),
}
optional_params = get_optional_params(
@@ -1247,6 +1305,7 @@ def completion( # type: ignore # noqa: PLR0915
additional_drop_params=kwargs.get("additional_drop_params"),
remove_sensitive_keys=True,
add_provider_specific_params=True,
+ provider_config=provider_config,
)
if litellm.add_function_to_prompt and optional_params.get(
@@ -1309,6 +1368,7 @@ def completion( # type: ignore # noqa: PLR0915
azure_scope=kwargs.get("azure_scope"),
max_retries=max_retries,
timeout=timeout,
+ litellm_request_debug=kwargs.get("litellm_request_debug", False),
)
cast(LiteLLMLoggingObj, logging).update_environment_variables(
model=model,
@@ -1380,7 +1440,7 @@ def completion( # type: ignore # noqa: PLR0915
api_version = (
api_version
or litellm.api_version
- or get_secret("AZURE_API_VERSION")
+ or get_secret_str("AZURE_API_VERSION")
or litellm.AZURE_DEFAULT_API_VERSION
)
@@ -1388,13 +1448,13 @@ def completion( # type: ignore # noqa: PLR0915
api_key
or litellm.api_key
or litellm.azure_key
- or get_secret("AZURE_OPENAI_API_KEY")
- or get_secret("AZURE_API_KEY")
+ or get_secret_str("AZURE_OPENAI_API_KEY")
+ or get_secret_str("AZURE_API_KEY")
)
azure_ad_token = optional_params.get("extra_body", {}).pop(
"azure_ad_token", None
- ) or get_secret("AZURE_AD_TOKEN")
+ ) or get_secret_str("AZURE_AD_TOKEN")
azure_ad_token_provider = litellm_params.get(
"azure_ad_token_provider", None
@@ -1482,25 +1542,32 @@ def completion( # type: ignore # noqa: PLR0915
)
elif custom_llm_provider == "azure_text":
# azure configs
- api_type = get_secret("AZURE_API_TYPE") or "azure"
+ api_type = get_secret_str("AZURE_API_TYPE") or "azure"
- api_base = api_base or litellm.api_base or get_secret("AZURE_API_BASE")
+ api_base = api_base or litellm.api_base or get_secret_str("AZURE_API_BASE")
+
+ if api_base is None:
+ raise ValueError(
+ "api_base is required for Azure OpenAI LLM provider. Either set it dynamically or set the AZURE_API_BASE environment variable."
+ )
api_version = (
- api_version or litellm.api_version or get_secret("AZURE_API_VERSION")
+ api_version
+ or litellm.api_version
+ or get_secret_str("AZURE_API_VERSION")
)
api_key = (
api_key
or litellm.api_key
or litellm.azure_key
- or get_secret("AZURE_OPENAI_API_KEY")
- or get_secret("AZURE_API_KEY")
+ or get_secret_str("AZURE_OPENAI_API_KEY")
+ or get_secret_str("AZURE_API_KEY")
)
azure_ad_token = optional_params.get("extra_body", {}).pop(
"azure_ad_token", None
- ) or get_secret("AZURE_AD_TOKEN")
+ ) or get_secret_str("AZURE_AD_TOKEN")
azure_ad_token_provider = litellm_params.get(
"azure_ad_token_provider", None
@@ -1526,7 +1593,7 @@ def completion( # type: ignore # noqa: PLR0915
headers=headers,
api_key=api_key,
api_base=api_base,
- api_version=api_version,
+ api_version=cast(str, api_version),
api_type=api_type,
azure_ad_token=azure_ad_token,
azure_ad_token_provider=azure_ad_token_provider,
@@ -1555,6 +1622,7 @@ def completion( # type: ignore # noqa: PLR0915
)
elif custom_llm_provider == "deepseek":
## COMPLETION CALL
+
try:
response = base_llm_http_handler.completion(
model=model,
@@ -1567,6 +1635,7 @@ def completion( # type: ignore # noqa: PLR0915
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
timeout=timeout, # type: ignore
client=client,
custom_llm_provider=custom_llm_provider,
@@ -1585,18 +1654,11 @@ def completion( # type: ignore # noqa: PLR0915
raise e
elif custom_llm_provider == "azure_ai":
- api_base = (
- api_base # for deepinfra/perplexity/anyscale/groq/friendliai we check in get_llm_provider and pass in the api base from there
- or litellm.api_base
- or get_secret("AZURE_AI_API_BASE")
- )
+ from litellm.llms.azure_ai.common_utils import AzureFoundryModelInfo
+
+ api_base = AzureFoundryModelInfo.get_api_base(api_base)
# set API KEY
- api_key = (
- api_key
- or litellm.api_key # for deepinfra/perplexity/anyscale/friendliai we check in get_llm_provider and pass in the api key from there
- or litellm.openai_key
- or get_secret("AZURE_AI_API_KEY")
- )
+ api_key = AzureFoundryModelInfo.get_api_key(api_key)
headers = headers or litellm.headers
@@ -1620,6 +1682,7 @@ def completion( # type: ignore # noqa: PLR0915
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
timeout=timeout, # type: ignore
client=client, # pass AsyncOpenAI, OpenAI client
custom_llm_provider=custom_llm_provider,
@@ -1749,6 +1812,7 @@ def completion( # type: ignore # noqa: PLR0915
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
timeout=timeout, # type: ignore
client=client,
custom_llm_provider=custom_llm_provider,
@@ -1765,6 +1829,36 @@ def completion( # type: ignore # noqa: PLR0915
additional_args={"headers": headers},
)
raise e
+ elif custom_llm_provider == "heroku":
+ try:
+ response = base_llm_http_handler.completion(
+ model=model,
+ messages=messages,
+ headers=headers,
+ model_response=model_response,
+ api_key=api_key,
+ api_base=api_base,
+ acompletion=acompletion,
+ logging_obj=logging,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ shared_session=shared_session,
+ timeout=timeout,
+ client=client,
+ custom_llm_provider=custom_llm_provider,
+ encoding=encoding,
+ stream=stream,
+ provider_config=provider_config,
+ )
+ except Exception as e:
+ logging.post_call(
+ input=messages,
+ api_key=api_key,
+ original_response=str(e),
+ additional_args={"headers": headers},
+ )
+ raise e
+
elif custom_llm_provider == "xai":
## COMPLETION CALL
try:
@@ -1779,6 +1873,7 @@ def completion( # type: ignore # noqa: PLR0915
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
timeout=timeout, # type: ignore
client=client,
custom_llm_provider=custom_llm_provider,
@@ -1830,6 +1925,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider=custom_llm_provider,
timeout=timeout,
headers=headers,
@@ -1876,6 +1972,46 @@ def completion( # type: ignore # noqa: PLR0915
encoding=encoding,
stream=stream,
)
+ elif custom_llm_provider == "cometapi":
+ api_key = (
+ api_key
+ or litellm.cometapi_key
+ or get_secret_str("COMETAPI_KEY")
+ or litellm.api_key
+ )
+
+ api_base = (
+ api_base
+ or litellm.api_base
+ or get_secret_str("COMETAPI_API_BASE")
+ or "https://api.cometapi.com/v1"
+ )
+
+ ## COMPLETION CALL
+ response = base_llm_http_handler.completion(
+ model=model,
+ messages=messages,
+ headers=headers,
+ model_response=model_response,
+ api_key=api_key,
+ api_base=api_base,
+ acompletion=acompletion,
+ logging_obj=logging,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ shared_session=shared_session,
+ timeout=timeout,
+ client=client,
+ custom_llm_provider=custom_llm_provider,
+ encoding=encoding,
+ stream=stream,
+ provider_config=provider_config,
+ )
+
+ ## LOGGING
+ logging.post_call(
+ input=messages, api_key=api_key, original_response=response
+ )
elif (
model in litellm.open_ai_chat_completion_models
or custom_llm_provider == "custom_openai"
@@ -1883,12 +2019,14 @@ def completion( # type: ignore # noqa: PLR0915
or custom_llm_provider == "perplexity"
or custom_llm_provider == "nvidia_nim"
or custom_llm_provider == "cerebras"
+ or custom_llm_provider == "baseten"
or custom_llm_provider == "sambanova"
or custom_llm_provider == "volcengine"
or custom_llm_provider == "anyscale"
or custom_llm_provider == "openai"
or custom_llm_provider == "together_ai"
or custom_llm_provider == "nebius"
+ or custom_llm_provider == "wandb"
or custom_llm_provider in litellm.openai_compatible_providers
or "ft:gpt-3.5-turbo" in model # finetune gpt-3.5-turbo
): # allow user to make an openai call with a custom base
@@ -1935,26 +2073,53 @@ def completion( # type: ignore # noqa: PLR0915
optional_params[k] = v
## COMPLETION CALL
+ use_base_llm_http_handler = get_secret_bool(
+ "EXPERIMENTAL_OPENAI_BASE_LLM_HTTP_HANDLER"
+ )
+
try:
- response = openai_chat_completions.completion(
- model=model,
- messages=messages,
- headers=headers,
- model_response=model_response,
- print_verbose=print_verbose,
- api_key=api_key,
- api_base=api_base,
- acompletion=acompletion,
- logging_obj=logging,
- optional_params=optional_params,
- litellm_params=litellm_params,
- logger_fn=logger_fn,
- timeout=timeout, # type: ignore
- custom_prompt_dict=custom_prompt_dict,
- client=client, # pass AsyncOpenAI, OpenAI client
- organization=organization,
- custom_llm_provider=custom_llm_provider,
- )
+ if use_base_llm_http_handler:
+
+ response = base_llm_http_handler.completion(
+ model=model,
+ messages=messages,
+ api_base=api_base,
+ custom_llm_provider=custom_llm_provider,
+ model_response=model_response,
+ encoding=encoding,
+ logging_obj=logging,
+ optional_params=optional_params,
+ timeout=timeout,
+ litellm_params=litellm_params,
+ shared_session=shared_session,
+ acompletion=acompletion,
+ stream=stream,
+ api_key=api_key,
+ headers=headers,
+ client=client,
+ provider_config=provider_config,
+ )
+ else:
+ response = openai_chat_completions.completion(
+ model=model,
+ messages=messages,
+ headers=headers,
+ model_response=model_response,
+ print_verbose=print_verbose,
+ api_key=api_key,
+ api_base=api_base,
+ acompletion=acompletion,
+ logging_obj=logging,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ logger_fn=logger_fn,
+ timeout=timeout, # type: ignore
+ custom_prompt_dict=custom_prompt_dict,
+ client=client, # pass AsyncOpenAI, OpenAI client
+ organization=organization,
+ custom_llm_provider=custom_llm_provider,
+ shared_session=shared_session,
+ )
except Exception as e:
## LOGGING - log the original exception returned
logging.post_call(
@@ -1994,6 +2159,7 @@ def completion( # type: ignore # noqa: PLR0915
optional_params=optional_params,
timeout=timeout,
litellm_params=litellm_params,
+ shared_session=shared_session,
acompletion=acompletion,
stream=stream,
api_key=api_key,
@@ -2081,6 +2247,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider="clarifai",
timeout=timeout,
headers=headers,
@@ -2104,8 +2271,18 @@ def completion( # type: ignore # noqa: PLR0915
or "https://api.anthropic.com/v1/complete"
)
- if api_base is not None and not api_base.endswith("/v1/complete"):
+ # Check if we should disable automatic URL suffix appending
+ disable_url_suffix = get_secret_bool("LITELLM_ANTHROPIC_DISABLE_URL_SUFFIX")
+ if (
+ api_base is not None
+ and not disable_url_suffix
+ and not api_base.endswith("/v1/complete")
+ ):
api_base += "/v1/complete"
+ elif disable_url_suffix:
+ verbose_logger.debug(
+ "LITELLM_ANTHROPIC_DISABLE_URL_SUFFIX is set, skipping /v1/complete suffix"
+ )
response = base_llm_http_handler.completion(
model=model,
@@ -2116,6 +2293,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider="anthropic_text",
timeout=timeout,
headers=headers,
@@ -2141,8 +2319,18 @@ def completion( # type: ignore # noqa: PLR0915
or "https://api.anthropic.com/v1/messages"
)
- if api_base is not None and not api_base.endswith("/v1/messages"):
+ # Check if we should disable automatic URL suffix appending
+ disable_url_suffix = get_secret_bool("LITELLM_ANTHROPIC_DISABLE_URL_SUFFIX")
+ if (
+ api_base is not None
+ and not disable_url_suffix
+ and not api_base.endswith("/v1/messages")
+ ):
api_base += "/v1/messages"
+ elif disable_url_suffix:
+ verbose_logger.debug(
+ "LITELLM_ANTHROPIC_DISABLE_URL_SUFFIX is set, skipping /v1/messages suffix"
+ )
response = anthropic_chat_completions.completion(
model=model,
@@ -2259,47 +2447,7 @@ def completion( # type: ignore # noqa: PLR0915
)
return response
response = model_response
- elif custom_llm_provider == "cohere":
- cohere_key = (
- api_key
- or litellm.cohere_key
- or get_secret("COHERE_API_KEY")
- or get_secret("CO_API_KEY")
- or litellm.api_key
- )
-
- api_base = (
- api_base
- or litellm.api_base
- or get_secret("COHERE_API_BASE")
- or "https://api.cohere.ai/v1/generate"
- )
-
- headers = headers or litellm.headers or {}
- if headers is None:
- headers = {}
-
- if extra_headers is not None:
- headers.update(extra_headers)
-
- response = base_llm_http_handler.completion(
- model=model,
- stream=stream,
- messages=messages,
- acompletion=acompletion,
- api_base=api_base,
- model_response=model_response,
- optional_params=optional_params,
- litellm_params=litellm_params,
- custom_llm_provider="cohere",
- timeout=timeout,
- headers=headers,
- encoding=encoding,
- api_key=cohere_key,
- logging_obj=logging, # model call logging done inside the class as we make need to modify I/O to fit aleph alpha's requirements
- client=client,
- )
- elif custom_llm_provider == "cohere_chat":
+ elif custom_llm_provider == "cohere_chat" or custom_llm_provider == "cohere":
cohere_key = (
api_key
or litellm.cohere_key
@@ -2331,6 +2479,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider="cohere_chat",
timeout=timeout,
headers=headers,
@@ -2396,6 +2545,50 @@ def completion( # type: ignore # noqa: PLR0915
encoding=encoding,
stream=stream,
)
+ elif custom_llm_provider == "oci":
+ response = base_llm_http_handler.completion(
+ model=model,
+ messages=messages,
+ headers=headers,
+ model_response=model_response,
+ api_key=api_key,
+ api_base=api_base,
+ acompletion=acompletion,
+ logging_obj=logging,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ timeout=timeout, # type: ignore
+ client=client,
+ custom_llm_provider=custom_llm_provider,
+ encoding=encoding,
+ stream=stream,
+ )
+ elif custom_llm_provider == "compactifai":
+ api_key = (
+ api_key or get_secret_str("COMPACTIFAI_API_KEY") or litellm.api_key
+ )
+
+ api_base = api_base or "https://api.compactif.ai/v1"
+
+ ## COMPLETION CALL
+ response = base_llm_http_handler.completion(
+ model=model,
+ messages=messages,
+ headers=headers,
+ model_response=model_response,
+ api_key=api_key,
+ api_base=api_base,
+ acompletion=acompletion,
+ logging_obj=logging,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ timeout=timeout,
+ client=client,
+ custom_llm_provider=custom_llm_provider,
+ encoding=encoding,
+ stream=stream,
+ provider_config=provider_config,
+ )
elif custom_llm_provider == "oobabooga":
custom_llm_provider = "oobabooga"
model_response = oobabooga.completion(
@@ -2549,6 +2742,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider="openrouter",
timeout=timeout,
headers=headers,
@@ -2561,6 +2755,69 @@ def completion( # type: ignore # noqa: PLR0915
logging.post_call(
input=messages, api_key=openai.api_key, original_response=response
)
+ elif custom_llm_provider == "vercel_ai_gateway":
+ api_base = (
+ api_base
+ or litellm.api_base
+ or get_secret_str("VERCEL_AI_GATEWAY_API_BASE")
+ or "https://ai-gateway.vercel.sh/v1"
+ )
+
+ api_key = (
+ api_key or litellm.api_key or get_secret("VERCEL_AI_GATEWAY_API_KEY")
+ )
+
+ vercel_site_url = get_secret("VERCEL_SITE_URL") or "https://litellm.ai"
+ vercel_app_name = get_secret("VERCEL_APP_NAME") or "liteLLM"
+
+ vercel_headers = {
+ "http-referer": vercel_site_url,
+ "x-title": vercel_app_name,
+ }
+
+ _headers = headers or litellm.headers
+ if _headers:
+ vercel_headers.update(_headers)
+
+ headers = vercel_headers
+
+ ## Load Config
+ config = litellm.VercelAIGatewayConfig.get_config()
+ for k, v in config.items():
+ if k == "extra_body":
+ # we use openai 'extra_body' to pass vercel specific params - providerOptions
+ if "extra_body" in optional_params:
+ optional_params[k].update(v)
+ else:
+ optional_params[k] = v
+ elif k not in optional_params:
+ optional_params[k] = v
+
+ data = {"model": model, "messages": messages, **optional_params}
+
+ ## COMPLETION CALL
+ response = base_llm_http_handler.completion(
+ model=model,
+ stream=stream,
+ messages=messages,
+ acompletion=acompletion,
+ api_base=api_base,
+ model_response=model_response,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ shared_session=shared_session,
+ custom_llm_provider="vercel_ai_gateway",
+ timeout=timeout,
+ headers=headers,
+ encoding=encoding,
+ api_key=api_key,
+ logging_obj=logging, # model call logging done inside the class as we make need to modify I/O to fit aleph alpha's requirements
+ client=client,
+ )
+ ## LOGGING
+ logging.post_call(
+ input=messages, api_key=openai.api_key, original_response=response
+ )
elif (
custom_llm_provider == "together_ai"
or ("togethercomputer" in model)
@@ -2595,14 +2852,13 @@ def completion( # type: ignore # noqa: PLR0915
gemini_api_key = (
api_key
- or get_secret("GEMINI_API_KEY")
+ or get_api_key_from_env()
or get_secret("PALM_API_KEY") # older palm api key should also work
or litellm.api_key
)
api_base = api_base or litellm.api_base or get_secret("GEMINI_API_BASE")
-
- new_params = deepcopy(optional_params)
+ new_params = safe_deep_copy(optional_params or {})
response = vertex_chat_completion.completion( # type: ignore
model=model,
messages=messages,
@@ -2619,13 +2875,13 @@ def completion( # type: ignore # noqa: PLR0915
logging_obj=logging,
acompletion=acompletion,
timeout=timeout,
- custom_llm_provider=custom_llm_provider,
+ custom_llm_provider=custom_llm_provider, # type: ignore
client=client,
api_base=api_base,
- extra_headers=extra_headers,
+ extra_headers=headers,
)
- elif custom_llm_provider == "vertex_ai":
+ elif custom_llm_provider == "vertex_ai":
vertex_ai_project = (
optional_params.pop("vertex_project", None)
or optional_params.pop("vertex_ai_project", None)
@@ -2646,8 +2902,10 @@ def completion( # type: ignore # noqa: PLR0915
api_base = api_base or litellm.api_base or get_secret("VERTEXAI_API_BASE")
- new_params = deepcopy(optional_params)
- if vertex_partner_models_chat_completion.is_vertex_partner_model(model):
+ new_params = safe_deep_copy(optional_params or {})
+ model_route = get_vertex_ai_model_route(model=model, litellm_params=litellm_params)
+
+ if model_route == VertexAIModelRoute.PARTNER_MODELS:
model_response = vertex_partner_models_chat_completion.completion(
model=model,
messages=messages,
@@ -2668,10 +2926,7 @@ def completion( # type: ignore # noqa: PLR0915
timeout=timeout,
client=client,
)
- elif "gemini" in model or (
- litellm_params.get("base_model") is not None
- and "gemini" in litellm_params["base_model"]
- ):
+ elif model_route == VertexAIModelRoute.GEMINI:
model_response = vertex_chat_completion.completion( # type: ignore
model=model,
messages=messages,
@@ -2688,12 +2943,34 @@ def completion( # type: ignore # noqa: PLR0915
logging_obj=logging,
acompletion=acompletion,
timeout=timeout,
- custom_llm_provider=custom_llm_provider,
+ custom_llm_provider=custom_llm_provider, # type: ignore
client=client,
api_base=api_base,
- extra_headers=extra_headers,
+ extra_headers=headers,
)
- elif "openai" in model:
+ elif model_route == VertexAIModelRoute.GEMMA:
+ # Vertex Gemma Models with custom prediction endpoint
+ model_response = vertex_gemma_chat_completion.completion(
+ model=model,
+ messages=messages,
+ model_response=model_response,
+ print_verbose=print_verbose,
+ optional_params=new_params,
+ litellm_params=litellm_params, # type: ignore
+ logger_fn=logger_fn,
+ encoding=encoding,
+ api_base=api_base,
+ vertex_location=vertex_ai_location,
+ vertex_project=vertex_ai_project,
+ vertex_credentials=vertex_credentials,
+ logging_obj=logging,
+ acompletion=acompletion,
+ headers=headers,
+ custom_prompt_dict=custom_prompt_dict,
+ timeout=timeout,
+ client=client,
+ )
+ elif model_route == VertexAIModelRoute.MODEL_GARDEN:
# Vertex Model Garden - OpenAI compatible models
model_response = vertex_model_garden_chat_completion.completion(
model=model,
@@ -2715,7 +2992,7 @@ def completion( # type: ignore # noqa: PLR0915
timeout=timeout,
client=client,
)
- else:
+ else: # VertexAIModelRoute.NON_GEMINI
model_response = vertex_ai_non_gemini.completion(
model=model,
messages=messages,
@@ -2917,7 +3194,7 @@ def completion( # type: ignore # noqa: PLR0915
logger_fn=logger_fn,
encoding=encoding,
logging_obj=logging,
- extra_headers=extra_headers,
+ extra_headers=headers, # Use merged headers instead of original extra_headers
timeout=timeout,
acompletion=acompletion,
client=client,
@@ -3036,6 +3313,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider="watsonx_text",
timeout=timeout,
headers=headers,
@@ -3089,6 +3367,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider="ollama",
timeout=timeout,
headers=headers,
@@ -3122,6 +3401,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider="ollama_chat",
timeout=timeout,
headers=headers,
@@ -3142,6 +3422,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider=custom_llm_provider,
timeout=timeout,
headers=headers,
@@ -3174,6 +3455,7 @@ def completion( # type: ignore # noqa: PLR0915
model_response=model_response,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
custom_llm_provider="cloudflare",
timeout=timeout,
headers=headers,
@@ -3181,42 +3463,7 @@ def completion( # type: ignore # noqa: PLR0915
api_key=api_key,
logging_obj=logging, # model call logging done inside the class as we make need to modify I/O to fit aleph alpha's requirements
)
- elif (
- custom_llm_provider == "baseten"
- or litellm.api_base == "https://app.baseten.co"
- ):
- custom_llm_provider = "baseten"
- baseten_key = (
- api_key
- or litellm.baseten_key
- or os.environ.get("BASETEN_API_KEY")
- or litellm.api_key
- )
- model_response = baseten.completion(
- model=model,
- messages=messages,
- model_response=model_response,
- print_verbose=print_verbose,
- optional_params=optional_params,
- litellm_params=litellm_params,
- logger_fn=logger_fn,
- encoding=encoding,
- api_key=baseten_key,
- logging_obj=logging,
- )
- if inspect.isgenerator(model_response) or (
- "stream" in optional_params and optional_params["stream"] is True
- ):
- # don't try to access stream object,
- response = CustomStreamWrapper(
- model_response,
- model,
- custom_llm_provider="baseten",
- logging_obj=logging,
- )
- return response
- response = model_response
elif custom_llm_provider == "petals" or model in litellm.petals_models:
api_base = api_base or litellm.api_base
@@ -3262,6 +3509,7 @@ def completion( # type: ignore # noqa: PLR0915
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
+ shared_session=shared_session,
timeout=timeout, # type: ignore
client=client,
custom_llm_provider=custom_llm_provider,
@@ -3278,6 +3526,26 @@ def completion( # type: ignore # noqa: PLR0915
additional_args={"headers": headers},
)
raise e
+ elif custom_llm_provider == "gradient_ai":
+
+ api_base = litellm.api_base or api_base
+ response = base_llm_http_handler.completion(
+ model=model,
+ stream=stream,
+ messages=messages,
+ acompletion=acompletion,
+ api_base=api_base,
+ model_response=model_response,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ shared_session=shared_session,
+ custom_llm_provider="gradient_ai",
+ timeout=timeout,
+ headers=headers,
+ encoding=encoding,
+ api_key=api_key,
+ logging_obj=logging,
+ )
elif custom_llm_provider == "bytez":
api_key = (
@@ -3306,6 +3574,71 @@ def completion( # type: ignore # noqa: PLR0915
provider_config=bytez_transformation,
)
+ pass
+ elif custom_llm_provider == "lemonade":
+ api_key = (
+ api_key
+ or litellm.lemonade_key
+ or get_secret_str("LEMONADE_API_KEY")
+ or litellm.api_key
+ )
+
+ response = base_llm_http_handler.completion(
+ model=model,
+ messages=messages,
+ headers=headers,
+ model_response=model_response,
+ api_key=api_key,
+ api_base=api_base,
+ acompletion=acompletion,
+ logging_obj=logging,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ timeout=timeout, # type: ignore
+ client=client,
+ custom_llm_provider=custom_llm_provider,
+ encoding=encoding,
+ stream=stream,
+ provider_config=lemonade_transformation,
+ )
+
+ pass
+
+
+ elif custom_llm_provider == "ovhcloud" or model in litellm.ovhcloud_models:
+ api_key = (
+ api_key
+ or litellm.ovhcloud_key
+ or get_secret_str("OVHCLOUD_API_KEY")
+ or litellm.api_key
+ )
+
+ api_base = (
+ api_base
+ or litellm.api_base
+ or get_secret_str("OVHCLOUD_API_BASE")
+ or "https://oai.endpoints.kepler.ai.cloud.ovh.net/v1"
+ )
+
+ response = base_llm_http_handler.completion(
+ model=model,
+ messages=messages,
+ headers=headers,
+ model_response=model_response,
+ api_key=api_key,
+ api_base=api_base,
+ acompletion=acompletion,
+ logging_obj=logging,
+ optional_params=optional_params,
+ litellm_params=litellm_params,
+ timeout=timeout, # type: ignore
+ client=client,
+ custom_llm_provider=custom_llm_provider,
+ encoding=encoding,
+ stream=stream,
+ provider_config=ovhcloud_transformation,
+ )
+
pass
elif custom_llm_provider == "custom":
@@ -3388,7 +3721,7 @@ def completion( # type: ignore # noqa: PLR0915
async_fn=acompletion, stream=stream, custom_llm=custom_handler
)
- headers = headers or litellm.headers
+ headers = headers or litellm.headers or {}
## CALL FUNCTION
response = handler_fn(
@@ -3512,7 +3845,7 @@ async def aembedding(*args, **kwargs) -> EmbeddingResponse:
model = args[0] if len(args) > 0 else kwargs["model"]
### PASS ARGS TO Embedding ###
kwargs["aembedding"] = True
- custom_llm_provider = None
+ custom_llm_provider = kwargs.get("custom_llm_provider", None)
try:
# Use a partial function to pass your keyword arguments
func = partial(embedding, *args, **kwargs)
@@ -3522,7 +3855,9 @@ async def aembedding(*args, **kwargs) -> EmbeddingResponse:
func_with_context = partial(ctx.run, func)
_, custom_llm_provider, _, _ = get_llm_provider(
- model=model, api_base=kwargs.get("api_base", None)
+ model=model,
+ custom_llm_provider=custom_llm_provider,
+ api_base=kwargs.get("api_base", None),
)
# Await normally
@@ -3661,11 +3996,12 @@ def embedding( # noqa: PLR0915
"""
azure = kwargs.get("azure", None)
client = kwargs.pop("client", None)
+ shared_session = kwargs.get("shared_session", None)
max_retries = kwargs.get("max_retries", None)
litellm_logging_obj: LiteLLMLoggingObj = kwargs.get("litellm_logging_obj") # type: ignore
mock_response: Optional[List[float]] = kwargs.get("mock_response", None) # type: ignore
- azure_ad_token_provider = kwargs.pop("azure_ad_token_provider", None)
- aembedding = kwargs.get("aembedding", None)
+ azure_ad_token_provider = kwargs.get("azure_ad_token_provider", None)
+ aembedding: Optional[bool] = kwargs.get("aembedding", None)
extra_headers = kwargs.get("extra_headers", None)
headers = kwargs.get("headers", None)
### CUSTOM MODEL COST ###
@@ -3850,6 +4186,7 @@ def embedding( # noqa: PLR0915
client=client,
aembedding=aembedding,
max_retries=max_retries,
+ shared_session=shared_session,
)
elif custom_llm_provider == "databricks":
api_base = api_base or litellm.api_base or get_secret("DATABRICKS_API_BASE") # type: ignore
@@ -3877,7 +4214,6 @@ def embedding( # noqa: PLR0915
)
elif (
custom_llm_provider == "openai_like"
- or custom_llm_provider == "jina_ai"
or custom_llm_provider == "hosted_vllm"
or custom_llm_provider == "llamafile"
or custom_llm_provider == "lm_studio"
@@ -3895,6 +4231,9 @@ def embedding( # noqa: PLR0915
or get_secret_str("OPENAI_LIKE_API_KEY")
)
+ if extra_headers is not None:
+ optional_params["extra_headers"] = extra_headers
+
## EMBEDDING CALL
response = openai_like_embedding.embedding(
model=model,
@@ -3998,9 +4337,7 @@ def embedding( # noqa: PLR0915
litellm_params={},
)
elif custom_llm_provider == "gemini":
- gemini_api_key = (
- api_key or get_secret_str("GEMINI_API_KEY") or litellm.api_key
- )
+ gemini_api_key = api_key or get_api_key_from_env() or litellm.api_key
api_base = api_base or litellm.api_base or get_secret_str("GEMINI_API_BASE")
@@ -4196,6 +4533,49 @@ def embedding( # noqa: PLR0915
client=client,
aembedding=aembedding,
)
+ elif custom_llm_provider == "wandb":
+ api_key = api_key or litellm.api_key or get_secret_str("WANDB_API_KEY")
+ api_base = (
+ api_base
+ or litellm.api_base
+ or get_secret_str("WANDB_API_BASE")
+ or "https://api.inference.wandb.ai/v1"
+ )
+
+ response = openai_chat_completions.embedding(
+ model=model,
+ input=input,
+ api_base=api_base,
+ api_key=api_key,
+ logging_obj=logging,
+ timeout=timeout,
+ model_response=EmbeddingResponse(),
+ optional_params=optional_params,
+ client=client,
+ aembedding=aembedding,
+ )
+ elif custom_llm_provider == "sambanova":
+ api_key = api_key or litellm.api_key or get_secret_str("SAMBANOVA_API_KEY")
+ api_base = (
+ api_base
+ or litellm.api_base
+ or get_secret_str("SAMBANOVA_API_BASE")
+ or "https://api.sambanova.ai/v1"
+ )
+ response = base_llm_http_handler.embedding(
+ model=model,
+ input=input,
+ custom_llm_provider=custom_llm_provider,
+ api_base=api_base,
+ api_key=api_key,
+ logging_obj=logging,
+ timeout=timeout,
+ model_response=EmbeddingResponse(),
+ optional_params=optional_params,
+ client=client,
+ aembedding=aembedding,
+ litellm_params={},
+ )
elif custom_llm_provider == "voyage":
response = base_llm_http_handler.embedding(
model=model,
@@ -4303,6 +4683,77 @@ def embedding( # noqa: PLR0915
client=client,
aembedding=aembedding,
)
+ elif custom_llm_provider == "jina_ai":
+ if isinstance(input, str):
+ transformed_input = [input]
+ else:
+ transformed_input = input
+ response = base_llm_http_handler.embedding(
+ model=model,
+ input=transformed_input,
+ custom_llm_provider=custom_llm_provider,
+ api_base=api_base,
+ api_key=api_key,
+ logging_obj=logging,
+ timeout=timeout,
+ model_response=EmbeddingResponse(),
+ optional_params=optional_params,
+ litellm_params={},
+ client=client,
+ aembedding=aembedding,
+ )
+ elif custom_llm_provider == "volcengine":
+ volcengine_key = (
+ api_key
+ or litellm.api_key
+ or get_secret_str("ARK_API_KEY")
+ or get_secret_str("VOLCENGINE_API_KEY")
+ )
+ if volcengine_key is None:
+ raise ValueError(
+ "Missing API key for Volcengine. Set ARK_API_KEY or VOLCENGINE_API_KEY environment variable or pass api_key parameter."
+ )
+ if extra_headers is not None and isinstance(extra_headers, dict):
+ headers = extra_headers
+ else:
+ headers = {}
+ response = base_llm_http_handler.embedding(
+ model=model,
+ input=input,
+ timeout=timeout,
+ custom_llm_provider=custom_llm_provider,
+ logging_obj=logging,
+ api_base=api_base,
+ optional_params=optional_params,
+ litellm_params={},
+ model_response=EmbeddingResponse(),
+ api_key=volcengine_key,
+ client=client,
+ aembedding=aembedding,
+ headers=headers,
+ )
+ elif custom_llm_provider == "ovhcloud":
+ api_key = api_key or litellm.api_key or get_secret_str("OVHCLOUD_API_KEY")
+ api_base = (
+ api_base
+ or litellm.api_base
+ or get_secret_str("OVHCLOUD_API_BASE")
+ or "https://oai.endpoints.kepler.ai.cloud.ovh.net/v1"
+ )
+ response = base_llm_http_handler.embedding(
+ model=model,
+ input=input,
+ custom_llm_provider=custom_llm_provider,
+ api_base=api_base,
+ api_key=api_key,
+ logging_obj=logging,
+ timeout=timeout,
+ model_response=EmbeddingResponse(),
+ optional_params=optional_params,
+ client=client,
+ aembedding=aembedding,
+ litellm_params={},
+ )
elif custom_llm_provider in litellm._custom_providers:
custom_handler: Optional[CustomLLM] = None
for item in litellm.custom_provider_map:
@@ -4749,6 +5200,21 @@ async def aadapter_completion(
except Exception as e:
raise e
+async def aadapter_generate_content(
+ **kwargs,
+) -> Union[Dict[str, Any], AsyncIterator[bytes]]:
+ from litellm.google_genai.adapters.handler import (
+ GenerateContentToCompletionHandler,
+ )
+
+ coro = cast(
+ Coroutine[Any, Any, Union[Dict[str, Any], AsyncIterator[bytes]]],
+ GenerateContentToCompletionHandler.generate_content_handler(
+ **kwargs, _is_async=True
+ ),
+ )
+ return await coro
+
def adapter_completion(
*, adapter_id: str, **kwargs
@@ -4972,8 +5438,7 @@ def transcription(
proxy_server_request = kwargs.get("proxy_server_request", None)
model_info = kwargs.get("model_info", None)
metadata = kwargs.get("metadata", None)
- atranscription = kwargs.get("atranscription", False)
- atranscription = kwargs.get("atranscription", False)
+ atranscription = kwargs.pop("atranscription", False)
litellm_logging_obj: LiteLLMLoggingObj = kwargs.get("litellm_logging_obj") # type: ignore
extra_headers = kwargs.get("extra_headers", None)
kwargs.pop("tags", [])
@@ -4997,7 +5462,10 @@ def transcription(
model_response = litellm.utils.TranscriptionResponse()
model, custom_llm_provider, dynamic_api_key, api_base = get_llm_provider(
- model=model, custom_llm_provider=custom_llm_provider, api_base=api_base
+ model=model,
+ custom_llm_provider=custom_llm_provider,
+ api_base=api_base,
+ api_key=api_key,
) # type: ignore
if dynamic_api_key is not None:
@@ -5013,6 +5481,7 @@ def transcription(
custom_llm_provider=custom_llm_provider,
**non_default_params,
)
+
litellm_params_dict = get_litellm_params(**kwargs)
litellm_logging_obj.update_environment_variables(
@@ -5077,9 +5546,8 @@ def transcription(
max_retries=max_retries,
litellm_params=litellm_params_dict,
)
- elif (
- custom_llm_provider == "openai"
- or custom_llm_provider in litellm.openai_compatible_providers
+ elif custom_llm_provider == "openai" or (
+ custom_llm_provider in litellm.openai_compatible_providers
):
api_base = (
api_base
@@ -5094,6 +5562,7 @@ def transcription(
or None # default - https://github.com/openai/openai-python/blob/284c1799070c723c6a553337134148a7ab088dd8/openai/util.py#L105
)
# set API KEY
+
api_key = api_key or litellm.api_key or litellm.openai_key or get_secret("OPENAI_API_KEY") # type: ignore
response = openai_audio_transcriptions.audio_transcriptions(
model=model,
@@ -5110,10 +5579,7 @@ def transcription(
provider_config=provider_config,
litellm_params=litellm_params_dict,
)
- elif custom_llm_provider in [
- LlmProviders.DEEPGRAM.value,
- LlmProviders.ELEVENLABS.value,
- ]:
+ elif provider_config is not None:
response = base_llm_http_handler.audio_transcriptions(
model=model,
audio_file=file,
@@ -5230,7 +5696,7 @@ def speech( # noqa: PLR0915
if max_retries is None:
max_retries = litellm.num_retries or openai.DEFAULT_MAX_RETRIES
litellm_params_dict = get_litellm_params(**kwargs)
- logging_obj = kwargs.get("litellm_logging_obj", None)
+ logging_obj: Logging = cast(Logging, kwargs.get("litellm_logging_obj"))
logging_obj.update_environment_variables(
model=model,
user=user,
@@ -5430,6 +5896,7 @@ def speech( # noqa: PLR0915
##### Health Endpoints #######################
+
async def ahealth_check(
model_params: dict,
mode: Optional[
@@ -5469,13 +5936,17 @@ async def ahealth_check(
messages=[],
stream=False,
call_type="acompletion",
- litellm_call_id="1234",
+ litellm_call_id=str(uuid.uuid4()),
start_time=datetime.datetime.now(),
- function_id="1234",
+ function_id=str(uuid.uuid4()),
log_raw_request_response=True,
)
model_params["litellm_logging_obj"] = litellm_logging_obj
- model_params = HealthCheckHelpers._update_model_params_with_health_check_tracking_information(model_params=model_params)
+ model_params = (
+ HealthCheckHelpers._update_model_params_with_health_check_tracking_information(
+ model_params=model_params
+ )
+ )
#########################################################
try:
model: Optional[str] = model_params.get("model", None)
@@ -5514,9 +5985,15 @@ async def ahealth_check(
input=input or ["test"],
),
"audio_speech": lambda: litellm.aspeech(
- **_filter_model_params(model_params),
+ **{
+ **_filter_model_params(model_params),
+ **(
+ {"voice": "alloy"}
+ if "voice" not in _filter_model_params(model_params)
+ else {}
+ ),
+ },
input=prompt or "test",
- voice="alloy",
),
"audio_transcription": lambda: litellm.atranscription(
**_filter_model_params(model_params),
@@ -5673,7 +6150,11 @@ def stream_chunk_builder_text_completion(
def stream_chunk_builder( # noqa: PLR0915
- chunks: list, messages: Optional[list] = None, start_time=None, end_time=None
+ chunks: list,
+ messages: Optional[list] = None,
+ start_time=None,
+ end_time=None,
+ logging_obj: Optional[Logging] = None,
) -> Optional[Union[ModelResponse, TextCompletionResponse]]:
try:
if chunks is None:
@@ -5798,6 +6279,12 @@ def stream_chunk_builder( # noqa: PLR0915
setattr(response, "usage", usage)
+ # Add cost to usage object if include_cost_in_streaming_usage is True
+ if litellm.include_cost_in_streaming_usage and logging_obj is not None:
+ setattr(
+ usage, "cost", logging_obj._response_cost_calculator(result=response)
+ )
+
return response
except Exception as e:
verbose_logger.exception(
diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json
index 40a07a74189..91a23b7f00f 100644
--- a/litellm/model_prices_and_context_window_backup.json
+++ b/litellm/model_prices_and_context_window_backup.json
@@ -1,2038 +1,956 @@
{
- "sample_spec": {
- "max_tokens": "LEGACY parameter. set to max_output_tokens if provider specifies it. IF not set to max_input_tokens, if provider specifies it.",
- "max_input_tokens": "max input tokens, if the provider specifies it. if not default to max_tokens",
- "max_output_tokens": "max output tokens, if the provider specifies it. if not default to max_tokens",
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "output_cost_per_reasoning_token": 0.0,
- "litellm_provider": "one of https://docs.litellm.ai/docs/providers",
- "mode": "one of: chat, embedding, completion, image_generation, audio_transcription, audio_speech, image_generation, moderation, rerank",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_system_messages": true,
- "supports_reasoning": true,
- "supports_web_search": true,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.0,
- "search_context_size_medium": 0.0,
- "search_context_size_high": 0.0
- },
- "file_search_cost_per_1k_calls": 0.0,
- "file_search_cost_per_gb_per_day": 0.0,
- "vector_store_cost_per_gb_per_day": 0.0,
- "computer_use_input_cost_per_1k_tokens": 0.0,
- "computer_use_output_cost_per_1k_tokens": 0.0,
- "code_interpreter_cost_per_session": 0.0,
- "supported_regions": [
- "global",
- "us-west-2",
- "eu-west-1",
- "ap-southeast-1",
- "ap-northeast-1"
- ],
- "deprecation_date": "date when the model becomes deprecated in the format YYYY-MM-DD"
- },
- "omni-moderation-latest": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 0,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openai",
- "mode": "moderation"
- },
- "omni-moderation-latest-intents": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 0,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openai",
- "mode": "moderation"
- },
- "omni-moderation-2024-09-26": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 0,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openai",
- "mode": "moderation"
- },
- "gpt-4": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 6e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4.1": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "input_cost_per_token_batches": 1e-06,
- "output_cost_per_token_batches": 4e-06,
- "cache_read_input_token_cost": 5e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "gpt-4.1-2025-04-14": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "input_cost_per_token_batches": 1e-06,
- "output_cost_per_token_batches": 4e-06,
- "cache_read_input_token_cost": 5e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "gpt-4.1-mini": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 1.6e-06,
- "input_cost_per_token_batches": 2e-07,
- "output_cost_per_token_batches": 8e-07,
- "cache_read_input_token_cost": 1e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "gpt-4.1-mini-2025-04-14": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 1.6e-06,
- "input_cost_per_token_batches": 2e-07,
- "output_cost_per_token_batches": 8e-07,
- "cache_read_input_token_cost": 1e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "gpt-4.1-nano": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "input_cost_per_token_batches": 5e-08,
- "output_cost_per_token_batches": 2e-07,
- "cache_read_input_token_cost": 2.5e-08,
- "litellm_provider": "openai",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "gpt-4.1-nano-2025-04-14": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "input_cost_per_token_batches": 5e-08,
- "output_cost_per_token_batches": 2e-07,
- "cache_read_input_token_cost": 2.5e-08,
- "litellm_provider": "openai",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "gpt-4o": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "input_cost_per_token_batches": 1.25e-06,
- "output_cost_per_token_batches": 5e-06,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "watsonx/ibm/granite-3-8b-instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 0.0002,
- "output_cost_per_token": 0.0002,
- "litellm_provider": "watsonx",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_parallel_function_calling": false,
- "supports_vision": false,
- "supports_audio_input": false,
- "supports_audio_output": false,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_system_messages": true
- },
- "watsonx/mistralai/mistral-large": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 16384,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "watsonx",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_parallel_function_calling": false,
- "supports_vision": false,
- "supports_audio_input": false,
- "supports_audio_output": false,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_system_messages": true
- },
- "gpt-4o-search-preview-2025-03-11": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "input_cost_per_token_batches": 1.25e-06,
- "output_cost_per_token_batches": 5e-06,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-search-preview": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "input_cost_per_token_batches": 1.25e-06,
- "output_cost_per_token_batches": 5e-06,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_web_search": true,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.03,
- "search_context_size_medium": 0.035,
- "search_context_size_high": 0.05
- }
- },
- "gpt-4.5-preview": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 7.5e-05,
- "output_cost_per_token": 0.00015,
- "input_cost_per_token_batches": 3.75e-05,
- "output_cost_per_token_batches": 7.5e-05,
- "cache_read_input_token_cost": 3.75e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4.5-preview-2025-02-27": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 7.5e-05,
- "output_cost_per_token": 0.00015,
- "input_cost_per_token_batches": 3.75e-05,
- "output_cost_per_token_batches": 7.5e-05,
- "cache_read_input_token_cost": 3.75e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "deprecation_date": "2025-07-14"
- },
- "gpt-4o-audio-preview": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "input_cost_per_audio_token": 0.0001,
- "output_cost_per_token": 1e-05,
- "output_cost_per_audio_token": 0.0002,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-audio-preview-2024-12-17": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "input_cost_per_audio_token": 4e-05,
- "output_cost_per_token": 1e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-audio-preview-2024-10-01": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "input_cost_per_audio_token": 0.0001,
- "output_cost_per_token": 1e-05,
- "output_cost_per_audio_token": 0.0002,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-audio-preview-2025-06-03": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "input_cost_per_audio_token": 4e-05,
- "output_cost_per_token": 1e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-mini-audio-preview": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "input_cost_per_audio_token": 1e-05,
- "output_cost_per_token": 6e-07,
- "output_cost_per_audio_token": 2e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-mini-audio-preview-2024-12-17": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "input_cost_per_audio_token": 1e-05,
- "output_cost_per_token": 6e-07,
- "output_cost_per_audio_token": 2e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-mini": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "input_cost_per_token_batches": 7.5e-08,
- "output_cost_per_token_batches": 3e-07,
- "cache_read_input_token_cost": 7.5e-08,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-mini-search-preview-2025-03-11": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "input_cost_per_token_batches": 7.5e-08,
- "output_cost_per_token_batches": 3e-07,
- "cache_read_input_token_cost": 7.5e-08,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-mini-search-preview": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "input_cost_per_token_batches": 7.5e-08,
- "output_cost_per_token_batches": 3e-07,
- "cache_read_input_token_cost": 7.5e-08,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_web_search": true,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.025,
- "search_context_size_medium": 0.0275,
- "search_context_size_high": 0.03
- }
- },
- "gpt-4o-mini-2024-07-18": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "input_cost_per_token_batches": 7.5e-08,
- "output_cost_per_token_batches": 3e-07,
- "cache_read_input_token_cost": 7.5e-08,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "search_context_cost_per_query": {
- "search_context_size_low": 30.0,
- "search_context_size_medium": 35.0,
- "search_context_size_high": 50.0
- }
- },
- "codex-mini-latest": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 6e-06,
- "cache_read_input_token_cost": 3.75e-07,
- "litellm_provider": "openai",
- "mode": "responses",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supported_endpoints": [
- "/v1/responses"
- ]
- },
- "o1-pro": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 0.00015,
- "output_cost_per_token": 0.0006,
- "input_cost_per_token_batches": 7.5e-05,
- "output_cost_per_token_batches": 0.0003,
- "litellm_provider": "openai",
- "mode": "responses",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_native_streaming": false,
- "supports_reasoning": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supported_endpoints": [
- "/v1/responses",
- "/v1/batch"
- ]
- },
- "o1-pro-2025-03-19": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 0.00015,
- "output_cost_per_token": 0.0006,
- "input_cost_per_token_batches": 7.5e-05,
- "output_cost_per_token_batches": 0.0003,
- "litellm_provider": "openai",
- "mode": "responses",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_native_streaming": false,
- "supports_reasoning": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supported_endpoints": [
- "/v1/responses",
- "/v1/batch"
- ]
- },
- "o1": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "o1-mini": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 5.5e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true
- },
- "computer-use-preview": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.2e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": false,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_reasoning": true
- },
- "o3-deep-research": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 4e-05,
- "input_cost_per_token_batches": 5e-06,
- "output_cost_per_token_batches": 2e-05,
- "cache_read_input_token_cost": 2.5e-06,
- "litellm_provider": "openai",
- "mode": "responses",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "o3-deep-research-2025-06-26": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 4e-05,
- "input_cost_per_token_batches": 5e-06,
- "output_cost_per_token_batches": 2e-05,
- "cache_read_input_token_cost": 2.5e-06,
- "litellm_provider": "openai",
- "mode": "responses",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "o3-pro": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-05,
- "input_cost_per_token_batches": 1e-05,
- "output_cost_per_token_batches": 4e-05,
- "output_cost_per_token": 8e-05,
- "litellm_provider": "openai",
- "mode": "responses",
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/responses",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ]
- },
- "o3-pro-2025-06-10": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-05,
- "input_cost_per_token_batches": 1e-05,
- "output_cost_per_token_batches": 4e-05,
- "output_cost_per_token": 8e-05,
- "litellm_provider": "openai",
- "mode": "responses",
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/responses",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ]
- },
- "o3": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "cache_read_input_token_cost": 5e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/responses",
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ]
- },
- "o3-2025-04-16": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "cache_read_input_token_cost": 5e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/responses",
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ]
- },
- "o3-mini": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 5.5e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": false,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "o3-mini-2025-01-31": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 5.5e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": false,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "o4-mini": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 2.75e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "o4-mini-deep-research": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "input_cost_per_token_batches": 1e-06,
- "output_cost_per_token_batches": 4e-06,
- "cache_read_input_token_cost": 5e-07,
- "litellm_provider": "openai",
- "mode": "responses",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "o4-mini-deep-research-2025-06-26": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "input_cost_per_token_batches": 1e-06,
- "output_cost_per_token_batches": 4e-06,
- "cache_read_input_token_cost": 5e-07,
- "litellm_provider": "openai",
- "mode": "responses",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "o4-mini-2025-04-16": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 2.75e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "o1-mini-2024-09-12": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.2e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "o1-preview": {
- "max_tokens": 32768,
- "max_input_tokens": 128000,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "o1-preview-2024-09-12": {
- "max_tokens": 32768,
- "max_input_tokens": 128000,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "o1-2024-12-17": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "chatgpt-4o-latest": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-2024-05-13": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 1.5e-05,
- "input_cost_per_token_batches": 2.5e-06,
- "output_cost_per_token_batches": 7.5e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-2024-08-06": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "input_cost_per_token_batches": 1.25e-06,
- "output_cost_per_token_batches": 5e-06,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-2024-11-20": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "input_cost_per_token_batches": 1.25e-06,
- "output_cost_per_token_batches": 5e-06,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-realtime-preview-2024-10-01": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "input_cost_per_audio_token": 0.0001,
- "cache_read_input_token_cost": 2.5e-06,
- "cache_creation_input_audio_token_cost": 2e-05,
- "output_cost_per_token": 2e-05,
- "output_cost_per_audio_token": 0.0002,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-realtime-preview": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "input_cost_per_audio_token": 4e-05,
- "cache_read_input_token_cost": 2.5e-06,
- "output_cost_per_token": 2e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-realtime-preview-2024-12-17": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "input_cost_per_audio_token": 4e-05,
- "cache_read_input_token_cost": 2.5e-06,
- "output_cost_per_token": 2e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-realtime-preview-2025-06-03": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "input_cost_per_audio_token": 4e-05,
- "cache_read_input_token_cost": 2.5e-06,
- "output_cost_per_token": 2e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-mini-realtime-preview": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-07,
- "input_cost_per_audio_token": 1e-05,
- "cache_read_input_token_cost": 3e-07,
- "cache_creation_input_audio_token_cost": 3e-07,
- "output_cost_per_token": 2.4e-06,
- "output_cost_per_audio_token": 2e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4o-mini-realtime-preview-2024-12-17": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-07,
- "input_cost_per_audio_token": 1e-05,
- "cache_read_input_token_cost": 3e-07,
- "cache_creation_input_audio_token_cost": 3e-07,
- "output_cost_per_token": 2.4e-06,
- "output_cost_per_audio_token": 2e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-turbo-preview": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-0314": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 6e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-0613": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 6e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "deprecation_date": "2025-06-06",
- "supports_tool_choice": true
- },
- "gpt-4-32k": {
- "max_tokens": 4096,
- "max_input_tokens": 32768,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-05,
- "output_cost_per_token": 0.00012,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-32k-0314": {
- "max_tokens": 4096,
- "max_input_tokens": 32768,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-05,
- "output_cost_per_token": 0.00012,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-32k-0613": {
- "max_tokens": 4096,
- "max_input_tokens": 32768,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-05,
- "output_cost_per_token": 0.00012,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-turbo": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-turbo-2024-04-09": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-1106-preview": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-0125-preview": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-4-vision-preview": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "deprecation_date": "2024-12-06",
- "supports_tool_choice": true
- },
- "gpt-4-1106-vision-preview": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "deprecation_date": "2024-12-06",
- "supports_tool_choice": true
- },
- "gpt-3.5-turbo": {
- "max_tokens": 4097,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-3.5-turbo-0301": {
- "max_tokens": 4097,
- "max_input_tokens": 4097,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-3.5-turbo-0613": {
- "max_tokens": 4097,
- "max_input_tokens": 4097,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-3.5-turbo-1106": {
- "max_tokens": 16385,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-3.5-turbo-0125": {
- "max_tokens": 16385,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-3.5-turbo-16k": {
- "max_tokens": 16385,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 4e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "gpt-3.5-turbo-16k-0613": {
- "max_tokens": 16385,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 4e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:gpt-3.5-turbo": {
- "max_tokens": 4096,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 6e-06,
- "input_cost_per_token_batches": 1.5e-06,
- "output_cost_per_token_batches": 3e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:gpt-3.5-turbo-0125": {
- "max_tokens": 4096,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:gpt-3.5-turbo-1106": {
- "max_tokens": 4096,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:gpt-3.5-turbo-0613": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:gpt-4-0613": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 6e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "OpenAI needs to add pricing for this ft model, will be updated when added by OpenAI. Defaulting to base model pricing",
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:gpt-4o-2024-08-06": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 3.75e-06,
- "output_cost_per_token": 1.5e-05,
- "input_cost_per_token_batches": 1.875e-06,
- "output_cost_per_token_batches": 7.5e-06,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:gpt-4o-2024-11-20": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 3.75e-06,
- "cache_creation_input_token_cost": 1.875e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:gpt-4o-mini-2024-07-18": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 1.2e-06,
- "input_cost_per_token_batches": 1.5e-07,
- "output_cost_per_token_batches": 6e-07,
- "cache_read_input_token_cost": 1.5e-07,
- "litellm_provider": "openai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "ft:davinci-002": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 2e-06,
- "input_cost_per_token_batches": 1e-06,
- "output_cost_per_token_batches": 1e-06,
- "litellm_provider": "text-completion-openai",
- "mode": "completion"
- },
- "ft:babbage-002": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 4096,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 4e-07,
- "input_cost_per_token_batches": 2e-07,
- "output_cost_per_token_batches": 2e-07,
- "litellm_provider": "text-completion-openai",
- "mode": "completion"
- },
- "text-embedding-3-large": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "output_vector_size": 3072,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 0.0,
- "input_cost_per_token_batches": 6.5e-08,
- "output_cost_per_token_batches": 0.0,
- "litellm_provider": "openai",
- "mode": "embedding"
- },
- "text-embedding-3-small": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "output_vector_size": 1536,
- "input_cost_per_token": 2e-08,
- "output_cost_per_token": 0.0,
- "input_cost_per_token_batches": 1e-08,
- "output_cost_per_token_batches": 0.0,
- "litellm_provider": "openai",
- "mode": "embedding"
- },
- "text-embedding-ada-002": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "output_vector_size": 1536,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openai",
- "mode": "embedding"
- },
- "text-embedding-ada-002-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "input_cost_per_token_batches": 5e-08,
- "output_cost_per_token_batches": 0.0,
- "litellm_provider": "openai",
- "mode": "embedding"
- },
- "text-moderation-stable": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 0,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openai",
- "mode": "moderation"
- },
- "text-moderation-007": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 0,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openai",
- "mode": "moderation"
- },
- "text-moderation-latest": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 0,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openai",
- "mode": "moderation"
- },
- "256-x-256/dall-e-2": {
+ "1024-x-1024/50-steps/bedrock/amazon.nova-canvas-v1:0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 2600,
"mode": "image_generation",
- "input_cost_per_pixel": 2.4414e-07,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
+ "output_cost_per_image": 0.06
},
- "512-x-512/dall-e-2": {
+ "1024-x-1024/50-steps/stability.stable-diffusion-xl-v1": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
"mode": "image_generation",
- "input_cost_per_pixel": 6.86e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
+ "output_cost_per_image": 0.04
},
"1024-x-1024/dall-e-2": {
- "mode": "image_generation",
"input_cost_per_pixel": 1.9e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
- },
- "hd/1024-x-1792/dall-e-3": {
- "mode": "image_generation",
- "input_cost_per_pixel": 6.539e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
- },
- "hd/1792-x-1024/dall-e-3": {
- "mode": "image_generation",
- "input_cost_per_pixel": 6.539e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
- },
- "hd/1024-x-1024/dall-e-3": {
- "mode": "image_generation",
- "input_cost_per_pixel": 7.629e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
- },
- "standard/1024-x-1792/dall-e-3": {
- "mode": "image_generation",
- "input_cost_per_pixel": 4.359e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
- },
- "standard/1792-x-1024/dall-e-3": {
- "mode": "image_generation",
- "input_cost_per_pixel": 4.359e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
- },
- "standard/1024-x-1024/dall-e-3": {
- "mode": "image_generation",
- "input_cost_per_pixel": 3.81469e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai"
- },
- "gpt-image-1": {
- "mode": "image_generation",
- "input_cost_per_pixel": 4.0054321e-08,
- "output_cost_per_pixel": 0.0,
"litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
+ },
+ "1024-x-1024/max-steps/stability.stable-diffusion-xl-v1": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.08
+ },
+ "256-x-256/dall-e-2": {
+ "input_cost_per_pixel": 2.4414e-07,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
+ },
+ "512-x-512/50-steps/stability.stable-diffusion-xl-v0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.018
+ },
+ "512-x-512/dall-e-2": {
+ "input_cost_per_pixel": 6.86e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
+ },
+ "512-x-512/max-steps/stability.stable-diffusion-xl-v0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.036
+ },
+ "ai21.j2-mid-v1": {
+ "input_cost_per_token": 1.25e-05,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8191,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-05
+ },
+ "ai21.j2-ultra-v1": {
+ "input_cost_per_token": 1.88e-05,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8191,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 1.88e-05
+ },
+ "ai21.jamba-1-5-large-v1:0": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06
+ },
+ "ai21.jamba-1-5-mini-v1:0": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07
+ },
+ "ai21.jamba-instruct-v1:0": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 70000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
+ "supports_system_messages": true
+ },
+ "aiml/dall-e-2": {
+ "litellm_provider": "aiml",
+ "metadata": {
+ "notes": "DALL-E 2 via AI/ML API - Reliable text-to-image generation"
+ },
+ "mode": "image_generation",
+ "output_cost_per_image": 0.021,
+ "source": "https://docs.aimlapi.com/",
"supported_endpoints": [
"/v1/images/generations"
]
},
- "low/1024-x-1024/gpt-image-1": {
+ "aiml/dall-e-3": {
+ "litellm_provider": "aiml",
+ "metadata": {
+ "notes": "DALL-E 3 via AI/ML API - High-quality text-to-image generation"
+ },
"mode": "image_generation",
- "input_cost_per_pixel": 1.0490417e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.042,
+ "source": "https://docs.aimlapi.com/",
"supported_endpoints": [
"/v1/images/generations"
]
},
- "medium/1024-x-1024/gpt-image-1": {
+ "aiml/flux-pro": {
+ "litellm_provider": "aiml",
+ "metadata": {
+ "notes": "Flux Dev - Development version optimized for experimentation"
+ },
"mode": "image_generation",
- "input_cost_per_pixel": 4.0054321e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.053,
+ "source": "https://docs.aimlapi.com/",
"supported_endpoints": [
"/v1/images/generations"
]
},
- "high/1024-x-1024/gpt-image-1": {
+ "aiml/flux-pro/v1.1": {
+ "litellm_provider": "aiml",
"mode": "image_generation",
- "input_cost_per_pixel": 1.59263611e-07,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.042,
"supported_endpoints": [
"/v1/images/generations"
]
},
- "low/1024-x-1536/gpt-image-1": {
+ "aiml/flux-pro/v1.1-ultra": {
+ "litellm_provider": "aiml",
"mode": "image_generation",
- "input_cost_per_pixel": 1.0172526e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.063,
"supported_endpoints": [
"/v1/images/generations"
]
},
- "medium/1024-x-1536/gpt-image-1": {
+ "aiml/flux-realism": {
+ "litellm_provider": "aiml",
+ "metadata": {
+ "notes": "Flux Pro - Professional-grade image generation model"
+ },
"mode": "image_generation",
- "input_cost_per_pixel": 4.0054321e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.037,
+ "source": "https://docs.aimlapi.com/",
"supported_endpoints": [
"/v1/images/generations"
]
},
- "high/1024-x-1536/gpt-image-1": {
+ "aiml/flux/dev": {
+ "litellm_provider": "aiml",
+ "metadata": {
+ "notes": "Flux Dev - Development version optimized for experimentation"
+ },
"mode": "image_generation",
- "input_cost_per_pixel": 1.58945719e-07,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.026,
+ "source": "https://docs.aimlapi.com/",
"supported_endpoints": [
"/v1/images/generations"
]
},
- "low/1536-x-1024/gpt-image-1": {
+ "aiml/flux/kontext-max/text-to-image": {
+ "litellm_provider": "aiml",
+ "metadata": {
+ "notes": "Flux Pro v1.1 - Enhanced version with improved capabilities and 6x faster inference speed"
+ },
"mode": "image_generation",
- "input_cost_per_pixel": 1.0172526e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.084,
+ "source": "https://docs.aimlapi.com/",
"supported_endpoints": [
"/v1/images/generations"
]
},
- "medium/1536-x-1024/gpt-image-1": {
+ "aiml/flux/kontext-pro/text-to-image": {
+ "litellm_provider": "aiml",
+ "metadata": {
+ "notes": "Flux Pro v1.1 - Enhanced version with improved capabilities and 6x faster inference speed"
+ },
"mode": "image_generation",
- "input_cost_per_pixel": 4.0054321e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.042,
+ "source": "https://docs.aimlapi.com/",
"supported_endpoints": [
"/v1/images/generations"
]
},
- "high/1536-x-1024/gpt-image-1": {
+ "aiml/flux/schnell": {
+ "litellm_provider": "aiml",
+ "metadata": {
+ "notes": "Flux Schnell - Fast generation model optimized for speed"
+ },
"mode": "image_generation",
- "input_cost_per_pixel": 1.58945719e-07,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "openai",
+ "output_cost_per_image": 0.003,
+ "source": "https://docs.aimlapi.com/",
"supported_endpoints": [
"/v1/images/generations"
]
},
- "gpt-4o-transcribe": {
+ "amazon.nova-lite-v1:0": {
+ "input_cost_per_token": 6e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-07,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "amazon.nova-micro-v1:0": {
+ "input_cost_per_token": 3.5e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 1.4e-07,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true
+ },
+ "amazon.nova-pro-v1:0": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 3.2e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "amazon.rerank-v1:0": {
+ "input_cost_per_query": 0.001,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "bedrock",
+ "max_document_chunks_per_query": 100,
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_query_tokens": 32000,
+ "max_tokens": 32000,
+ "max_tokens_per_document_chunk": 512,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "amazon.titan-embed-image-v1": {
+ "input_cost_per_image": 6e-05,
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128,
+ "max_tokens": 128,
+ "metadata": {
+ "notes": "'supports_image_input' is a deprecated field. Use 'supports_embedding_image_input' instead."
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024,
+ "source": "https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/providers?model=amazon.titan-image-generator-v1",
+ "supports_embedding_image_input": true,
+ "supports_image_input": true
+ },
+ "amazon.titan-embed-text-v1": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1536
+ },
+ "amazon.titan-embed-text-v2:0": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024
+ },
+ "twelvelabs.marengo-embed-2-7-v1:0": {
+ "input_cost_per_token": 7e-05,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024,
+ "supports_embedding_image_input": true,
+ "supports_image_input": true
+ },
+ "us.twelvelabs.marengo-embed-2-7-v1:0": {
+ "input_cost_per_token": 7e-05,
+ "input_cost_per_video_per_second": 0.0007,
+ "input_cost_per_audio_per_second": 0.00014,
+ "input_cost_per_image": 0.0001,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024,
+ "supports_embedding_image_input": true,
+ "supports_image_input": true
+ },
+ "eu.twelvelabs.marengo-embed-2-7-v1:0": {
+ "input_cost_per_token": 7e-05,
+ "input_cost_per_video_per_second": 0.0007,
+ "input_cost_per_audio_per_second": 0.00014,
+ "input_cost_per_image": 0.0001,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024,
+ "supports_embedding_image_input": true,
+ "supports_image_input": true
+ },
+ "twelvelabs.pegasus-1-2-v1:0": {
+ "input_cost_per_video_per_second": 0.00049,
+ "output_cost_per_token": 7.5e-06,
+ "litellm_provider": "bedrock",
+ "mode": "chat",
+ "supports_video_input": true
+ },
+ "us.twelvelabs.pegasus-1-2-v1:0": {
+ "input_cost_per_video_per_second": 0.00049,
+ "output_cost_per_token": 7.5e-06,
+ "litellm_provider": "bedrock",
+ "mode": "chat",
+ "supports_video_input": true
+ },
+ "eu.twelvelabs.pegasus-1-2-v1:0": {
+ "input_cost_per_video_per_second": 0.00049,
+ "output_cost_per_token": 7.5e-06,
+ "litellm_provider": "bedrock",
+ "mode": "chat",
+ "supports_video_input": true
+ },
+ "amazon.titan-text-express-v1": {
+ "input_cost_per_token": 1.3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 8000,
+ "max_tokens": 8000,
+ "mode": "chat",
+ "output_cost_per_token": 1.7e-06
+ },
+ "amazon.titan-text-lite-v1": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 4000,
+ "max_tokens": 4000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07
+ },
+ "amazon.titan-text-premier-v1:0": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06
+ },
+ "anthropic.claude-3-5-haiku-20241022-v1:0": {
+ "cache_creation_input_token_cost": 1e-06,
+ "cache_read_input_token_cost": 8e-08,
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "anthropic.claude-3-5-sonnet-20240620-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "anthropic.claude-3-5-sonnet-20241022-v2:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "anthropic.claude-3-7-sonnet-20250219-v1:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "anthropic.claude-3-haiku-20240307-v1:0": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "anthropic.claude-3-opus-20240229-v1:0": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "anthropic.claude-3-sonnet-20240229-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "anthropic.claude-instant-v1": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-06,
+ "supports_tool_choice": true
+ },
+ "anthropic.claude-opus-4-1-20250805-v1:0": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "anthropic.claude-opus-4-20250514-v1:0": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "anthropic.claude-sonnet-4-20250514-v1:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "anthropic.claude-v1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05
+ },
+ "anthropic.claude-v2:1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "anyscale/HuggingFaceH4/zephyr-7b-beta": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07
+ },
+ "anyscale/codellama/CodeLlama-34b-Instruct-hf": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06
+ },
+ "anyscale/codellama/CodeLlama-70b-Instruct-hf": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06,
+ "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/codellama-CodeLlama-70b-Instruct-hf"
+ },
+ "anyscale/google/gemma-7b-it": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/google-gemma-7b-it"
+ },
+ "anyscale/meta-llama/Llama-2-13b-chat-hf": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07
+ },
+ "anyscale/meta-llama/Llama-2-70b-chat-hf": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06
+ },
+ "anyscale/meta-llama/Llama-2-7b-chat-hf": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07
+ },
+ "anyscale/meta-llama/Meta-Llama-3-70B-Instruct": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06,
+ "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/meta-llama-Meta-Llama-3-70B-Instruct"
+ },
+ "anyscale/meta-llama/Meta-Llama-3-8B-Instruct": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/meta-llama-Meta-Llama-3-8B-Instruct"
+ },
+ "anyscale/mistralai/Mistral-7B-Instruct-v0.1": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/mistralai-Mistral-7B-Instruct-v0.1",
+ "supports_function_calling": true
+ },
+ "anyscale/mistralai/Mixtral-8x22B-Instruct-v0.1": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/mistralai-Mixtral-8x22B-Instruct-v0.1",
+ "supports_function_calling": true
+ },
+ "anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "anyscale",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/mistralai-Mixtral-8x7B-Instruct-v0.1",
+ "supports_function_calling": true
+ },
+ "apac.amazon.nova-lite-v1:0": {
+ "input_cost_per_token": 6.3e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 2.52e-07,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "apac.amazon.nova-micro-v1:0": {
+ "input_cost_per_token": 3.7e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 1.48e-07,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true
+ },
+ "apac.amazon.nova-pro-v1:0": {
+ "input_cost_per_token": 8.4e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 3.36e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "apac.anthropic.claude-3-5-sonnet-20240620-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "apac.anthropic.claude-3-5-sonnet-20241022-v2:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "apac.anthropic.claude-3-haiku-20240307-v1:0": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "apac.anthropic.claude-3-sonnet-20240229-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "apac.anthropic.claude-sonnet-4-20250514-v1:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "assemblyai/best": {
+ "input_cost_per_second": 3.333e-05,
+ "litellm_provider": "assemblyai",
"mode": "audio_transcription",
- "max_input_tokens": 16000,
- "max_output_tokens": 2000,
- "input_cost_per_token": 2.5e-06,
- "input_cost_per_audio_token": 6e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "openai",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ]
+ "output_cost_per_second": 0.0
},
- "gpt-4o-mini-transcribe": {
+ "assemblyai/nano": {
+ "input_cost_per_second": 0.00010278,
+ "litellm_provider": "assemblyai",
"mode": "audio_transcription",
- "max_input_tokens": 16000,
- "max_output_tokens": 2000,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_audio_token": 3e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "openai",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ]
+ "output_cost_per_second": 0.0
},
- "whisper-1": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.0001,
- "output_cost_per_second": 0.0001,
- "litellm_provider": "openai",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ]
+ "au.anthropic.claude-sonnet-4-5-20250929-v1:0": {
+ "cache_creation_input_token_cost": 4.125e-06,
+ "cache_read_input_token_cost": 3.3e-07,
+ "input_cost_per_token": 3.3e-06,
+ "input_cost_per_token_above_200k_tokens": 6.6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.475e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.65e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 346
},
- "tts-1": {
- "mode": "audio_speech",
- "input_cost_per_character": 1.5e-05,
- "litellm_provider": "openai",
- "supported_endpoints": [
- "/v1/audio/speech"
- ]
- },
- "tts-1-hd": {
- "mode": "audio_speech",
- "input_cost_per_character": 3e-05,
- "litellm_provider": "openai",
- "supported_endpoints": [
- "/v1/audio/speech"
- ]
- },
- "gpt-4o-mini-tts": {
- "mode": "audio_speech",
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_audio_token": 1.2e-05,
- "output_cost_per_second": 0.00025,
- "litellm_provider": "openai",
- "supported_modalities": [
- "text",
- "audio"
- ],
- "supported_output_modalities": [
- "audio"
- ],
- "supported_endpoints": [
- "/v1/audio/speech"
- ]
- },
- "azure/gpt-4o-mini-tts": {
- "mode": "audio_speech",
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_audio_token": 1.2e-05,
- "output_cost_per_second": 0.00025,
+ "azure/ada": {
+ "input_cost_per_token": 1e-07,
"litellm_provider": "azure",
+ "max_input_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "azure/codex-mini": {
+ "cache_read_input_token_cost": 3.75e-07,
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 6e-06,
+ "supported_endpoints": [
+ "/v1/responses"
+ ],
"supported_modalities": [
"text",
- "audio"
+ "image"
],
"supported_output_modalities": [
- "audio"
+ "text"
],
- "supported_endpoints": [
- "/v1/audio/speech"
- ]
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/command-r-plus": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true
},
"azure/computer-use-preview": {
- "max_tokens": 1024,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 8192,
"max_output_tokens": 1024,
- "input_cost_per_token": 3e-06,
+ "max_tokens": 1024,
+ "mode": "chat",
"output_cost_per_token": 1.2e-05,
- "litellm_provider": "azure",
- "mode": "chat",
"supported_endpoints": [
"/v1/responses"
],
@@ -2045,2229 +963,2350 @@
],
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
"supports_prompt_caching": false,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
- "supports_reasoning": true
+ "supports_vision": true
},
- "azure/gpt-4o-audio-preview-2024-12-17": {
- "max_tokens": 16384,
+ "azure/eu/gpt-4o-2024-08-06": {
+ "cache_read_input_token_cost": 1.375e-06,
+ "input_cost_per_token": 2.75e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "input_cost_per_audio_token": 4e-05,
- "output_cost_per_token": 1e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "azure",
+ "max_tokens": 16384,
"mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions"
- ],
- "supported_modalities": [
- "text",
- "audio"
- ],
- "supported_output_modalities": [
- "text",
- "audio"
- ],
+ "output_cost_per_token": 1.1e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_response_schema": false,
- "supports_vision": false,
- "supports_prompt_caching": false,
- "supports_system_messages": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
"supports_tool_choice": true,
- "supports_native_streaming": true,
- "supports_reasoning": false
+ "supports_vision": true
},
- "azure/gpt-4o-mini-audio-preview-2024-12-17": {
- "max_tokens": 16384,
+ "azure/eu/gpt-4o-2024-11-20": {
+ "cache_creation_input_token_cost": 1.38e-06,
+ "input_cost_per_token": 2.75e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "input_cost_per_audio_token": 4e-05,
- "output_cost_per_token": 1e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "azure",
+ "max_tokens": 16384,
"mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions"
- ],
- "supported_modalities": [
- "text",
- "audio"
- ],
- "supported_output_modalities": [
- "text",
- "audio"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": false,
- "supports_vision": false,
- "supports_prompt_caching": false,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true,
- "supports_reasoning": false
- },
- "azure/gpt-4.1": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "input_cost_per_token_batches": 1e-06,
- "output_cost_per_token_batches": 4e-06,
- "cache_read_input_token_cost": 5e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
+ "output_cost_per_token": 1.1e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
"supports_tool_choice": true,
- "supports_native_streaming": true,
- "supports_web_search": true,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.03,
- "search_context_size_medium": 0.035,
- "search_context_size_high": 0.05
- }
+ "supports_vision": true
},
- "azure/gpt-4.1-2025-04-14": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "input_cost_per_token_batches": 1e-06,
- "output_cost_per_token_batches": 4e-06,
- "cache_read_input_token_cost": 5e-07,
+ "azure/eu/gpt-4o-mini-2024-07-18": {
+ "cache_read_input_token_cost": 8.3e-08,
+ "input_cost_per_token": 1.65e-07,
"litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true,
- "supports_web_search": true,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.03,
- "search_context_size_medium": 0.035,
- "search_context_size_high": 0.05
- }
- },
- "azure/gpt-4.1-mini": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 1.6e-06,
- "input_cost_per_token_batches": 2e-07,
- "output_cost_per_token_batches": 8e-07,
- "cache_read_input_token_cost": 1e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true,
- "supports_web_search": true,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.025,
- "search_context_size_medium": 0.0275,
- "search_context_size_high": 0.03
- }
- },
- "azure/gpt-4.1-mini-2025-04-14": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 1.6e-06,
- "input_cost_per_token_batches": 2e-07,
- "output_cost_per_token_batches": 8e-07,
- "cache_read_input_token_cost": 1e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true,
- "supports_web_search": true,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.025,
- "search_context_size_medium": 0.0275,
- "search_context_size_high": 0.03
- }
- },
- "azure/gpt-4.1-nano": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "input_cost_per_token_batches": 5e-08,
- "output_cost_per_token_batches": 2e-07,
- "cache_read_input_token_cost": 2.5e-08,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "azure/gpt-4.1-nano-2025-04-14": {
- "max_tokens": 32768,
- "max_input_tokens": 1047576,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "input_cost_per_token_batches": 5e-08,
- "output_cost_per_token_batches": 2e-07,
- "cache_read_input_token_cost": 2.5e-08,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_native_streaming": true
- },
- "azure/o3-pro": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-05,
- "output_cost_per_token": 8e-05,
- "input_cost_per_token_batches": 1e-05,
- "output_cost_per_token_batches": 4e-05,
- "litellm_provider": "azure",
- "mode": "responses",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_prompt_caching": false,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "azure/o3-pro-2025-06-10": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-05,
- "output_cost_per_token": 8e-05,
- "input_cost_per_token_batches": 1e-05,
- "output_cost_per_token_batches": 4e-05,
- "litellm_provider": "azure",
- "mode": "responses",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_prompt_caching": false,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "azure/o3": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "cache_read_input_token_cost": 5e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "azure/o3-2025-04-16": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 4e-05,
- "cache_read_input_token_cost": 2.5e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "azure/o3-deep-research": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 4e-05,
- "cache_read_input_token_cost": 2.5e-06,
- "litellm_provider": "azure",
- "mode": "responses",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_web_search": true
- },
- "azure/o4-mini": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 2.75e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/batch",
- "/v1/responses"
- ],
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "azure/gpt-4o-mini-realtime-preview-2024-12-17": {
- "max_tokens": 4096,
"max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-07,
- "input_cost_per_audio_token": 1e-05,
- "cache_read_input_token_cost": 3e-07,
- "cache_creation_input_audio_token_cost": 3e-07,
- "output_cost_per_token": 2.4e-06,
- "output_cost_per_audio_token": 2e-05,
- "litellm_provider": "azure",
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_token": 6.6e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
"azure/eu/gpt-4o-mini-realtime-preview-2024-12-17": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6.6e-07,
- "input_cost_per_audio_token": 1.1e-05,
- "cache_read_input_token_cost": 3.3e-07,
"cache_creation_input_audio_token_cost": 3.3e-07,
- "output_cost_per_token": 2.64e-06,
- "output_cost_per_audio_token": 2.2e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "azure/us/gpt-4o-mini-realtime-preview-2024-12-17": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6.6e-07,
- "input_cost_per_audio_token": 1.1e-05,
"cache_read_input_token_cost": 3.3e-07,
- "cache_creation_input_audio_token_cost": 3.3e-07,
- "output_cost_per_token": 2.64e-06,
+ "input_cost_per_audio_token": 1.1e-05,
+ "input_cost_per_token": 6.6e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
"output_cost_per_audio_token": 2.2e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
+ "output_cost_per_token": 2.64e-06,
"supports_audio_input": true,
"supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "azure/gpt-4o-realtime-preview-2024-12-17": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "input_cost_per_audio_token": 4e-05,
- "cache_read_input_token_cost": 2.5e-06,
- "output_cost_per_token": 2e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_modalities": [
- "text",
- "audio"
- ],
- "supported_output_modalities": [
- "text",
- "audio"
- ],
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "azure/us/gpt-4o-realtime-preview-2024-12-17": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5.5e-06,
- "input_cost_per_audio_token": 4.4e-05,
- "cache_read_input_token_cost": 2.75e-06,
- "cache_read_input_audio_token_cost": 2.5e-06,
- "output_cost_per_token": 2.2e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_modalities": [
- "text",
- "audio"
- ],
- "supported_output_modalities": [
- "text",
- "audio"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "azure/eu/gpt-4o-realtime-preview-2024-12-17": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5.5e-06,
- "input_cost_per_audio_token": 4.4e-05,
- "cache_read_input_token_cost": 2.75e-06,
- "cache_read_input_audio_token_cost": 2.5e-06,
- "output_cost_per_token": 2.2e-05,
- "output_cost_per_audio_token": 8e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supported_modalities": [
- "text",
- "audio"
- ],
- "supported_output_modalities": [
- "text",
- "audio"
- ],
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "azure/gpt-4o-realtime-preview-2024-10-01": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "input_cost_per_audio_token": 0.0001,
- "cache_read_input_token_cost": 2.5e-06,
- "cache_creation_input_audio_token_cost": 2e-05,
- "output_cost_per_token": 2e-05,
- "output_cost_per_audio_token": 0.0002,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "azure/us/gpt-4o-realtime-preview-2024-10-01": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5.5e-06,
- "input_cost_per_audio_token": 0.00011,
- "cache_read_input_token_cost": 2.75e-06,
- "cache_creation_input_audio_token_cost": 2.2e-05,
- "output_cost_per_token": 2.2e-05,
- "output_cost_per_audio_token": 0.00022,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_audio_input": true,
- "supports_audio_output": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"azure/eu/gpt-4o-realtime-preview-2024-10-01": {
- "max_tokens": 4096,
+ "cache_creation_input_audio_token_cost": 2.2e-05,
+ "cache_read_input_token_cost": 2.75e-06,
+ "input_cost_per_audio_token": 0.00011,
+ "input_cost_per_token": 5.5e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 5.5e-06,
- "input_cost_per_audio_token": 0.00011,
- "cache_read_input_token_cost": 2.75e-06,
- "cache_creation_input_audio_token_cost": 2.2e-05,
- "output_cost_per_token": 2.2e-05,
- "output_cost_per_audio_token": 0.00022,
- "litellm_provider": "azure",
+ "max_tokens": 4096,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
+ "output_cost_per_audio_token": 0.00022,
+ "output_cost_per_token": 2.2e-05,
"supports_audio_input": true,
"supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
- "azure/o4-mini-2025-04-16": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 2.75e-07,
+ "azure/eu/gpt-4o-realtime-preview-2024-12-17": {
+ "cache_read_input_audio_token_cost": 2.5e-06,
+ "cache_read_input_token_cost": 2.75e-06,
+ "input_cost_per_audio_token": 4.4e-05,
+ "input_cost_per_token": 5.5e-06,
"litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": false,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "azure/o3-mini-2025-01-31": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 5.5e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_reasoning": true,
- "supports_vision": false,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/us/o3-mini-2025-01-31": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.21e-06,
- "input_cost_per_token_batches": 6.05e-07,
- "output_cost_per_token": 4.84e-06,
- "output_cost_per_token_batches": 2.42e-06,
- "cache_read_input_token_cost": 6.05e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_vision": false,
- "supports_reasoning": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/eu/o3-mini-2025-01-31": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.21e-06,
- "input_cost_per_token_batches": 6.05e-07,
- "output_cost_per_token": 4.84e-06,
- "output_cost_per_token_batches": 2.42e-06,
- "cache_read_input_token_cost": 6.05e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_vision": false,
- "supports_reasoning": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/tts-1": {
- "mode": "audio_speech",
- "input_cost_per_character": 1.5e-05,
- "litellm_provider": "azure"
- },
- "azure/tts-1-hd": {
- "mode": "audio_speech",
- "input_cost_per_character": 3e-05,
- "litellm_provider": "azure"
- },
- "azure/whisper-1": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.0001,
- "output_cost_per_second": 0.0001,
- "litellm_provider": "azure"
- },
- "azure/gpt-4o-transcribe": {
- "mode": "audio_transcription",
- "max_input_tokens": 16000,
- "max_output_tokens": 2000,
- "input_cost_per_token": 2.5e-06,
- "input_cost_per_audio_token": 6e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "azure",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ]
- },
- "azure/gpt-4o-mini-transcribe": {
- "mode": "audio_transcription",
- "max_input_tokens": 16000,
- "max_output_tokens": 2000,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_audio_token": 3e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "azure",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ]
- },
- "azure/o3-mini": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 5.5e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_vision": false,
- "supports_prompt_caching": true,
- "supports_reasoning": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "azure/o1-mini": {
- "max_tokens": 65536,
"max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.21e-06,
- "output_cost_per_token": 4.84e-06,
- "cache_read_input_token_cost": 6.05e-07,
- "litellm_provider": "azure",
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
"mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 2.2e-05,
+ "supported_modalities": [
+ "text",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "azure/o1-mini-2024-09-12": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "cache_read_input_token_cost": 5.5e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "azure/us/o1-mini-2024-09-12": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.21e-06,
- "input_cost_per_token_batches": 6.05e-07,
- "output_cost_per_token": 4.84e-06,
- "output_cost_per_token_batches": 2.42e-06,
- "cache_read_input_token_cost": 6.05e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_prompt_caching": true
- },
- "azure/eu/o1-mini-2024-09-12": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.21e-06,
- "input_cost_per_token_batches": 6.05e-07,
- "output_cost_per_token": 4.84e-06,
- "output_cost_per_token_batches": 2.42e-06,
- "cache_read_input_token_cost": 6.05e-07,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_prompt_caching": true
- },
- "azure/o1": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/o1-2024-12-17": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/us/o1-2024-12-17": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.65e-05,
- "output_cost_per_token": 6.6e-05,
- "cache_read_input_token_cost": 8.25e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
"azure/eu/o1-2024-12-17": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.65e-05,
- "output_cost_per_token": 6.6e-05,
"cache_read_input_token_cost": 8.25e-06,
+ "input_cost_per_token": 1.65e-05,
"litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/codex-mini": {
- "max_tokens": 100000,
"max_input_tokens": 200000,
"max_output_tokens": 100000,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 6e-06,
- "cache_read_input_token_cost": 3.75e-07,
- "litellm_provider": "azure",
- "mode": "responses",
- "supports_pdf_input": true,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 6.6e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_vision": true,
"supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
"supports_tool_choice": true,
- "supports_reasoning": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "supported_endpoints": [
- "/v1/responses"
- ]
+ "supports_vision": true
},
- "azure/o1-preview": {
- "max_tokens": 32768,
- "max_input_tokens": 128000,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
+ "azure/eu/o1-mini-2024-09-12": {
+ "cache_read_input_token_cost": 6.05e-07,
+ "input_cost_per_token": 1.21e-06,
+ "input_cost_per_token_batches": 6.05e-07,
"litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
"mode": "chat",
+ "output_cost_per_token": 4.84e-06,
+ "output_cost_per_token_batches": 2.42e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "azure/o1-preview-2024-09-12": {
- "max_tokens": 32768,
- "max_input_tokens": 128000,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "azure/us/o1-preview-2024-09-12": {
- "max_tokens": 32768,
- "max_input_tokens": 128000,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.65e-05,
- "output_cost_per_token": 6.6e-05,
- "cache_read_input_token_cost": 8.25e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_prompt_caching": true
+ "supports_prompt_caching": true,
+ "supports_vision": false
},
"azure/eu/o1-preview-2024-09-12": {
- "max_tokens": 32768,
+ "cache_read_input_token_cost": 8.25e-06,
+ "input_cost_per_token": 1.65e-05,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 32768,
- "input_cost_per_token": 1.65e-05,
+ "max_tokens": 32768,
+ "mode": "chat",
"output_cost_per_token": 6.6e-05,
- "cache_read_input_token_cost": 8.25e-06,
- "litellm_provider": "azure",
- "mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_prompt_caching": true
- },
- "azure/gpt-4.5-preview": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 7.5e-05,
- "output_cost_per_token": 0.00015,
- "input_cost_per_token_batches": 3.75e-05,
- "output_cost_per_token_batches": 7.5e-05,
- "cache_read_input_token_cost": 3.75e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
"supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "supports_vision": false
},
- "azure/gpt-4o": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "cache_read_input_token_cost": 1.25e-06,
+ "azure/eu/o3-mini-2025-01-31": {
+ "cache_read_input_token_cost": 6.05e-07,
+ "input_cost_per_token": 1.21e-06,
+ "input_cost_per_token_batches": 6.05e-07,
"litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
+ "output_cost_per_token": 4.84e-06,
+ "output_cost_per_token_batches": 2.42e-06,
"supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/global/gpt-4o-2024-11-20": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/gpt-4o-2024-08-06": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/global/gpt-4o-2024-08-06": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/gpt-4o-2024-11-20": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.75e-06,
- "output_cost_per_token": 1.1e-05,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/us/gpt-4o-2024-11-20": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.75e-06,
- "cache_creation_input_token_cost": 1.38e-06,
- "output_cost_per_token": 1.1e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "azure/eu/gpt-4o-2024-11-20": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.75e-06,
- "cache_creation_input_token_cost": 1.38e-06,
- "output_cost_per_token": 1.1e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "azure/gpt-4o-2024-05-13": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
},
"azure/global-standard/gpt-4o-2024-08-06": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
"cache_read_input_token_cost": 1.25e-06,
+ "deprecation_date": "2025-08-20",
+ "input_cost_per_token": 2.5e-06,
"litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_token": 1e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
"supports_prompt_caching": true,
+ "supports_response_schema": true,
"supports_tool_choice": true,
- "deprecation_date": "2025-08-20"
- },
- "azure/us/gpt-4o-2024-08-06": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.75e-06,
- "output_cost_per_token": 1.1e-05,
- "cache_read_input_token_cost": 1.375e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/eu/gpt-4o-2024-08-06": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2.75e-06,
- "output_cost_per_token": 1.1e-05,
- "cache_read_input_token_cost": 1.375e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
+ "supports_vision": true
},
"azure/global-standard/gpt-4o-2024-11-20": {
- "max_tokens": 16384,
+ "cache_read_input_token_cost": 1.25e-06,
+ "deprecation_date": "2025-12-20",
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "cache_read_input_token_cost": 1.25e-06,
- "litellm_provider": "azure",
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_token": 1e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_response_schema": true,
- "supports_vision": true,
"supports_tool_choice": true,
- "deprecation_date": "2025-12-20"
+ "supports_vision": true
},
"azure/global-standard/gpt-4o-mini": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
"input_cost_per_token": 1.5e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
"output_cost_per_token": 6e-07,
- "litellm_provider": "azure",
- "mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "azure/gpt-4o-mini": {
- "max_tokens": 16384,
+ "azure/global/gpt-4o-2024-08-06": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
- "input_cost_per_token": 1.65e-07,
- "output_cost_per_token": 6.6e-07,
- "cache_read_input_token_cost": 7.5e-08,
- "litellm_provider": "azure",
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_token": 1e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
"supports_prompt_caching": true,
- "supports_tool_choice": true
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "azure/gpt-4o-mini-2024-07-18": {
- "max_tokens": 16384,
+ "azure/global/gpt-4o-2024-11-20": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
- "input_cost_per_token": 1.65e-07,
- "output_cost_per_token": 6.6e-07,
- "cache_read_input_token_cost": 7.5e-08,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/us/gpt-4o-mini-2024-07-18": {
"max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.65e-07,
- "output_cost_per_token": 6.6e-07,
- "cache_read_input_token_cost": 8.3e-08,
- "litellm_provider": "azure",
"mode": "chat",
+ "output_cost_per_token": 1e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
"supports_prompt_caching": true,
- "supports_tool_choice": true
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "azure/eu/gpt-4o-mini-2024-07-18": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.65e-07,
- "output_cost_per_token": 6.6e-07,
- "cache_read_input_token_cost": 8.3e-08,
+ "azure/gpt-3.5-turbo": {
+ "input_cost_per_token": 5e-07,
"litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "azure/gpt-4-turbo-2024-04-09": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
+ "max_input_tokens": 4097,
"max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "azure",
+ "max_tokens": 4096,
"mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-3.5-turbo-0125": {
+ "deprecation_date": "2025-03-31",
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_vision": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-3.5-turbo-instruct-0914": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "azure_text",
+ "max_input_tokens": 4097,
+ "max_tokens": 4097,
+ "mode": "completion",
+ "output_cost_per_token": 2e-06
+ },
+ "azure/gpt-35-turbo": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 4097,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-35-turbo-0125": {
+ "deprecation_date": "2025-05-31",
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-35-turbo-0301": {
+ "deprecation_date": "2025-02-13",
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 4097,
+ "max_output_tokens": 4096,
+ "max_tokens": 4097,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-35-turbo-0613": {
+ "deprecation_date": "2025-02-13",
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 4097,
+ "max_output_tokens": 4096,
+ "max_tokens": 4097,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-35-turbo-1106": {
+ "deprecation_date": "2025-03-31",
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-35-turbo-16k": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-35-turbo-16k-0613": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-35-turbo-instruct": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "azure_text",
+ "max_input_tokens": 4097,
+ "max_tokens": 4097,
+ "mode": "completion",
+ "output_cost_per_token": 2e-06
+ },
+ "azure/gpt-35-turbo-instruct-0914": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "azure_text",
+ "max_input_tokens": 4097,
+ "max_tokens": 4097,
+ "mode": "completion",
+ "output_cost_per_token": 2e-06
+ },
+ "azure/gpt-4": {
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
"supports_tool_choice": true
},
"azure/gpt-4-0125-preview": {
- "max_tokens": 4096,
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_tool_choice": true
- },
- "azure/gpt-4-1106-preview": {
"max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "azure",
"mode": "chat",
+ "output_cost_per_token": 3e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"azure/gpt-4-0613": {
- "max_tokens": 4096,
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "azure",
"max_input_tokens": 8192,
"max_output_tokens": 4096,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 6e-05,
- "litellm_provider": "azure",
+ "max_tokens": 4096,
"mode": "chat",
+ "output_cost_per_token": 6e-05,
"supports_function_calling": true,
"supports_tool_choice": true
},
- "azure/gpt-4-32k-0613": {
- "max_tokens": 4096,
- "max_input_tokens": 32768,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-05,
- "output_cost_per_token": 0.00012,
+ "azure/gpt-4-1106-preview": {
+ "input_cost_per_token": 1e-05,
"litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
"mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"azure/gpt-4-32k": {
- "max_tokens": 4096,
+ "input_cost_per_token": 6e-05,
+ "litellm_provider": "azure",
"max_input_tokens": 32768,
"max_output_tokens": 4096,
- "input_cost_per_token": 6e-05,
- "output_cost_per_token": 0.00012,
- "litellm_provider": "azure",
+ "max_tokens": 4096,
"mode": "chat",
+ "output_cost_per_token": 0.00012,
"supports_tool_choice": true
},
- "azure/gpt-4": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 6e-05,
+ "azure/gpt-4-32k-0613": {
+ "input_cost_per_token": 6e-05,
"litellm_provider": "azure",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
"mode": "chat",
- "supports_function_calling": true,
+ "output_cost_per_token": 0.00012,
"supports_tool_choice": true
},
"azure/gpt-4-turbo": {
- "max_tokens": 4096,
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "azure",
+ "max_tokens": 4096,
"mode": "chat",
+ "output_cost_per_token": 3e-05,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_tool_choice": true
},
+ "azure/gpt-4-turbo-2024-04-09": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
"azure/gpt-4-turbo-vision-preview": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
"input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
"litellm_provider": "azure",
- "mode": "chat",
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "azure/gpt-35-turbo-16k-0613": {
- "max_tokens": 4096,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 4e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "azure/gpt-35-turbo-1106": {
- "max_tokens": 4096,
- "max_input_tokens": 16384,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "deprecation_date": "2025-03-31",
- "supports_tool_choice": true
- },
- "azure/gpt-35-turbo-0613": {
- "max_tokens": 4097,
- "max_input_tokens": 4097,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "deprecation_date": "2025-02-13",
- "supports_tool_choice": true
- },
- "azure/gpt-35-turbo-0301": {
- "max_tokens": 4097,
- "max_input_tokens": 4097,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "deprecation_date": "2025-02-13",
- "supports_tool_choice": true
- },
- "azure/gpt-35-turbo-0125": {
- "max_tokens": 4096,
- "max_input_tokens": 16384,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "deprecation_date": "2025-05-31",
- "supports_tool_choice": true
- },
- "azure/gpt-3.5-turbo-0125": {
- "max_tokens": 4096,
- "max_input_tokens": 16384,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "deprecation_date": "2025-03-31",
- "supports_tool_choice": true
- },
- "azure/gpt-35-turbo-16k": {
- "max_tokens": 4096,
- "max_input_tokens": 16385,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 4e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "azure/gpt-35-turbo": {
- "max_tokens": 4096,
- "max_input_tokens": 4097,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "azure/gpt-3.5-turbo": {
- "max_tokens": 4096,
- "max_input_tokens": 4097,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "azure/gpt-3.5-turbo-instruct-0914": {
- "max_tokens": 4097,
- "max_input_tokens": 4097,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "azure_text",
- "mode": "completion"
- },
- "azure/gpt-35-turbo-instruct": {
- "max_tokens": 4097,
- "max_input_tokens": 4097,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "azure_text",
- "mode": "completion"
- },
- "azure/gpt-35-turbo-instruct-0914": {
- "max_tokens": 4097,
- "max_input_tokens": 4097,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "azure_text",
- "mode": "completion"
- },
- "azure/mistral-large-latest": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true
- },
- "azure/mistral-large-2402": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "azure",
- "mode": "chat",
- "supports_function_calling": true
- },
- "azure/command-r-plus": {
- "max_tokens": 4096,
"max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "azure",
+ "max_tokens": 4096,
"mode": "chat",
- "supports_function_calling": true
+ "output_cost_per_token": 3e-05,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "azure/ada": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
+ "azure/gpt-4.1": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "input_cost_per_token_batches": 1e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "output_cost_per_token_batches": 4e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": false
+ },
+ "azure/gpt-4.1-2025-04-14": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "input_cost_per_token_batches": 1e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "output_cost_per_token_batches": 4e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": false
+ },
+ "azure/gpt-4.1-mini": {
+ "cache_read_input_token_cost": 1e-07,
+ "input_cost_per_token": 4e-07,
+ "input_cost_per_token_batches": 2e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-06,
+ "output_cost_per_token_batches": 8e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": false
+ },
+ "azure/gpt-4.1-mini-2025-04-14": {
+ "cache_read_input_token_cost": 1e-07,
+ "input_cost_per_token": 4e-07,
+ "input_cost_per_token_batches": 2e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-06,
+ "output_cost_per_token_batches": 8e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": false
+ },
+ "azure/gpt-4.1-nano": {
+ "cache_read_input_token_cost": 2.5e-08,
"input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
+ "input_cost_per_token_batches": 5e-08,
"litellm_provider": "azure",
- "mode": "embedding"
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "output_cost_per_token_batches": 2e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "azure/text-embedding-ada-002": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
+ "azure/gpt-4.1-nano-2025-04-14": {
+ "cache_read_input_token_cost": 2.5e-08,
"input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
+ "input_cost_per_token_batches": 5e-08,
"litellm_provider": "azure",
- "mode": "embedding"
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "output_cost_per_token_batches": 2e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "azure/text-embedding-3-large": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 0.0,
+ "azure/gpt-4.5-preview": {
+ "cache_read_input_token_cost": 3.75e-05,
+ "input_cost_per_token": 7.5e-05,
+ "input_cost_per_token_batches": 3.75e-05,
"litellm_provider": "azure",
- "mode": "embedding"
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 0.00015,
+ "output_cost_per_token_batches": 7.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "azure/text-embedding-3-small": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "input_cost_per_token": 2e-08,
- "output_cost_per_token": 0.0,
+ "azure/gpt-4o": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
"litellm_provider": "azure",
- "mode": "embedding"
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-4o-2024-05-13": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-4o-2024-08-06": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-4o-2024-11-20": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.75e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.1e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-4o-audio-preview-2024-12-17": {
+ "input_cost_per_audio_token": 4e-05,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": false,
+ "supports_reasoning": false,
+ "supports_response_schema": false,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure/gpt-4o-mini": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_token": 1.65e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 6.6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-4o-mini-2024-07-18": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_token": 1.65e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 6.6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-4o-mini-audio-preview-2024-12-17": {
+ "input_cost_per_audio_token": 4e-05,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": false,
+ "supports_reasoning": false,
+ "supports_response_schema": false,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure/gpt-4o-mini-realtime-preview-2024-12-17": {
+ "cache_creation_input_audio_token_cost": 3e-07,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_audio_token": 1e-05,
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 2e-05,
+ "output_cost_per_token": 2.4e-06,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-4o-mini-transcribe": {
+ "input_cost_per_audio_token": 3e-06,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 16000,
+ "max_output_tokens": 2000,
+ "mode": "audio_transcription",
+ "output_cost_per_token": 5e-06,
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "azure/gpt-4o-mini-tts": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "azure",
+ "mode": "audio_speech",
+ "output_cost_per_audio_token": 1.2e-05,
+ "output_cost_per_second": 0.00025,
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/audio/speech"
+ ],
+ "supported_modalities": [
+ "text",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "audio"
+ ]
+ },
+ "azure/gpt-4o-realtime-preview-2024-10-01": {
+ "cache_creation_input_audio_token_cost": 2e-05,
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_audio_token": 0.0001,
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 0.0002,
+ "output_cost_per_token": 2e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-4o-realtime-preview-2024-12-17": {
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_audio_token": 4e-05,
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 2e-05,
+ "supported_modalities": [
+ "text",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "azure/gpt-4o-transcribe": {
+ "input_cost_per_audio_token": 6e-06,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 16000,
+ "max_output_tokens": 2000,
+ "mode": "audio_transcription",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "azure/gpt-5": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-5-2025-08-07": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-5-chat": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "source": "https://azure.microsoft.com/en-us/blog/gpt-5-in-azure-ai-foundry-the-future-of-ai-apps-and-agents-starts-here/",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "azure/gpt-5-chat-latest": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "azure/gpt-5-codex": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "responses",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-5-mini": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-5-mini-2025-08-07": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-5-nano": {
+ "cache_read_input_token_cost": 5e-09,
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "azure",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/gpt-5-nano-2025-08-07": {
+ "cache_read_input_token_cost": 5e-09,
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "azure",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
"azure/gpt-image-1": {
- "mode": "image_generation",
"input_cost_per_pixel": 4.0054321e-08,
- "output_cost_per_pixel": 0.0,
"litellm_provider": "azure",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
"supported_endpoints": [
"/v1/images/generations"
]
},
- "azure/low/1024-x-1024/gpt-image-1": {
- "mode": "image_generation",
- "input_cost_per_pixel": 1.0490417e-08,
- "output_cost_per_pixel": 0.0,
+ "azure/hd/1024-x-1024/dall-e-3": {
+ "input_cost_per_pixel": 7.629e-08,
"litellm_provider": "azure",
- "supported_endpoints": [
- "/v1/images/generations"
- ]
+ "mode": "image_generation",
+ "output_cost_per_token": 0.0
},
- "azure/medium/1024-x-1024/gpt-image-1": {
- "mode": "image_generation",
- "input_cost_per_pixel": 4.0054321e-08,
- "output_cost_per_pixel": 0.0,
+ "azure/hd/1024-x-1792/dall-e-3": {
+ "input_cost_per_pixel": 6.539e-08,
"litellm_provider": "azure",
- "supported_endpoints": [
- "/v1/images/generations"
- ]
+ "mode": "image_generation",
+ "output_cost_per_token": 0.0
+ },
+ "azure/hd/1792-x-1024/dall-e-3": {
+ "input_cost_per_pixel": 6.539e-08,
+ "litellm_provider": "azure",
+ "mode": "image_generation",
+ "output_cost_per_token": 0.0
},
"azure/high/1024-x-1024/gpt-image-1": {
- "mode": "image_generation",
"input_cost_per_pixel": 1.59263611e-07,
- "output_cost_per_pixel": 0.0,
"litellm_provider": "azure",
- "supported_endpoints": [
- "/v1/images/generations"
- ]
- },
- "azure/low/1024-x-1536/gpt-image-1": {
"mode": "image_generation",
- "input_cost_per_pixel": 1.0172526e-08,
"output_cost_per_pixel": 0.0,
- "litellm_provider": "azure",
- "supported_endpoints": [
- "/v1/images/generations"
- ]
- },
- "azure/medium/1024-x-1536/gpt-image-1": {
- "mode": "image_generation",
- "input_cost_per_pixel": 4.0054321e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "azure",
"supported_endpoints": [
"/v1/images/generations"
]
},
"azure/high/1024-x-1536/gpt-image-1": {
- "mode": "image_generation",
"input_cost_per_pixel": 1.58945719e-07,
- "output_cost_per_pixel": 0.0,
"litellm_provider": "azure",
- "supported_endpoints": [
- "/v1/images/generations"
- ]
- },
- "azure/low/1536-x-1024/gpt-image-1": {
"mode": "image_generation",
- "input_cost_per_pixel": 1.0172526e-08,
"output_cost_per_pixel": 0.0,
- "litellm_provider": "azure",
- "supported_endpoints": [
- "/v1/images/generations"
- ]
- },
- "azure/medium/1536-x-1024/gpt-image-1": {
- "mode": "image_generation",
- "input_cost_per_pixel": 4.0054321e-08,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "azure",
"supported_endpoints": [
"/v1/images/generations"
]
},
"azure/high/1536-x-1024/gpt-image-1": {
- "mode": "image_generation",
"input_cost_per_pixel": 1.58945719e-07,
- "output_cost_per_pixel": 0.0,
"litellm_provider": "azure",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
"supported_endpoints": [
"/v1/images/generations"
]
},
- "azure/standard/1024-x-1024/dall-e-3": {
- "input_cost_per_pixel": 3.81469e-08,
- "output_cost_per_token": 0.0,
+ "azure/low/1024-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 1.0490417e-08,
"litellm_provider": "azure",
- "mode": "image_generation"
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
},
- "azure/hd/1024-x-1024/dall-e-3": {
- "input_cost_per_pixel": 7.629e-08,
- "output_cost_per_token": 0.0,
+ "azure/low/1024-x-1536/gpt-image-1": {
+ "input_cost_per_pixel": 1.0172526e-08,
"litellm_provider": "azure",
- "mode": "image_generation"
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
},
- "azure/standard/1024-x-1792/dall-e-3": {
- "input_cost_per_pixel": 4.359e-08,
- "output_cost_per_token": 0.0,
+ "azure/low/1536-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 1.0172526e-08,
"litellm_provider": "azure",
- "mode": "image_generation"
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
},
- "azure/standard/1792-x-1024/dall-e-3": {
- "input_cost_per_pixel": 4.359e-08,
- "output_cost_per_token": 0.0,
+ "azure/medium/1024-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 4.0054321e-08,
"litellm_provider": "azure",
- "mode": "image_generation"
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
},
- "azure/hd/1024-x-1792/dall-e-3": {
- "input_cost_per_pixel": 6.539e-08,
- "output_cost_per_token": 0.0,
+ "azure/medium/1024-x-1536/gpt-image-1": {
+ "input_cost_per_pixel": 4.0054321e-08,
"litellm_provider": "azure",
- "mode": "image_generation"
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
},
- "azure/hd/1792-x-1024/dall-e-3": {
- "input_cost_per_pixel": 6.539e-08,
- "output_cost_per_token": 0.0,
+ "azure/medium/1536-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 4.0054321e-08,
"litellm_provider": "azure",
- "mode": "image_generation"
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "azure/mistral-large-2402": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_function_calling": true
+ },
+ "azure/mistral-large-latest": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_function_calling": true
+ },
+ "azure/o1": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/o1-2024-12-17": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/o1-mini": {
+ "cache_read_input_token_cost": 6.05e-07,
+ "input_cost_per_token": 1.21e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 4.84e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_vision": false
+ },
+ "azure/o1-mini-2024-09-12": {
+ "cache_read_input_token_cost": 5.5e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_vision": false
+ },
+ "azure/o1-preview": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_vision": false
+ },
+ "azure/o1-preview-2024-09-12": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_vision": false
+ },
+ "azure/o3": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/o3-2025-04-16": {
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/o3-deep-research": {
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 4e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "azure/o3-mini": {
+ "cache_read_input_token_cost": 5.5e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure/o3-mini-2025-01-31": {
+ "cache_read_input_token_cost": 5.5e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure/o3-pro": {
+ "input_cost_per_token": 2e-05,
+ "input_cost_per_token_batches": 1e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 8e-05,
+ "output_cost_per_token_batches": 4e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": false,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/o3-pro-2025-06-10": {
+ "input_cost_per_token": 2e-05,
+ "input_cost_per_token_batches": 1e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 8e-05,
+ "output_cost_per_token_batches": 4e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": false,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/o4-mini": {
+ "cache_read_input_token_cost": 2.75e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/o4-mini-2025-04-16": {
+ "cache_read_input_token_cost": 2.75e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
"azure/standard/1024-x-1024/dall-e-2": {
"input_cost_per_pixel": 0.0,
- "output_cost_per_token": 0.0,
"litellm_provider": "azure",
- "mode": "image_generation"
+ "mode": "image_generation",
+ "output_cost_per_token": 0.0
},
- "azure_ai/grok-3": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3.3e-06,
- "output_cost_per_token": 16.5e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": false,
- "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
- "supports_web_search": true
+ "azure/standard/1024-x-1024/dall-e-3": {
+ "input_cost_per_pixel": 3.81469e-08,
+ "litellm_provider": "azure",
+ "mode": "image_generation",
+ "output_cost_per_token": 0.0
},
- "azure_ai/global/grok-3": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 15e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": false,
- "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
- "supports_web_search": true
+ "azure/standard/1024-x-1792/dall-e-3": {
+ "input_cost_per_pixel": 4.359e-08,
+ "litellm_provider": "azure",
+ "mode": "image_generation",
+ "output_cost_per_token": 0.0
},
- "azure_ai/global/grok-3-mini": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 0.25e-06,
- "output_cost_per_token": 1.27e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": false,
- "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
- "supports_web_search": true
+ "azure/standard/1792-x-1024/dall-e-3": {
+ "input_cost_per_pixel": 4.359e-08,
+ "litellm_provider": "azure",
+ "mode": "image_generation",
+ "output_cost_per_token": 0.0
},
- "azure_ai/grok-3-mini": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 0.275e-06,
- "output_cost_per_token": 1.38e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": false,
- "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
- "supports_web_search": true
- },
- "azure_ai/deepseek-r1": {
- "max_tokens": 8192,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.35e-06,
- "output_cost_per_token": 5.4e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "source": "https://techcommunity.microsoft.com/blog/machinelearningblog/deepseek-r1-improved-performance-higher-limits-and-transparent-pricing/4386367"
- },
- "azure_ai/deepseek-v3": {
- "max_tokens": 8192,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.14e-06,
- "output_cost_per_token": 4.56e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_tool_choice": true,
- "source": "https://techcommunity.microsoft.com/blog/machinelearningblog/announcing-deepseek-v3-on-azure-ai-foundry-and-github/4390438"
- },
- "azure_ai/deepseek-v3-0324": {
- "max_tokens": 8192,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.14e-06,
- "output_cost_per_token": 4.56e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "source": "https://techcommunity.microsoft.com/blog/machinelearningblog/announcing-deepseek-v3-on-azure-ai-foundry-and-github/4390438"
- },
- "azure_ai/jamba-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 70000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "azure_ai/jais-30b-chat": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0032,
- "output_cost_per_token": 0.00971,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "source": "https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/models/jais-30b-chat"
- },
- "azure_ai/mistral-nemo": {
- "max_tokens": 4096,
- "max_input_tokens": 131072,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/000-000.mistral-nemo-12b-2407?tab=PlansAndPrice"
- },
- "azure_ai/mistral-medium-2505": {
+ "azure/text-embedding-3-large": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 8191,
"max_tokens": 8191,
- "max_input_tokens": 131072,
- "max_output_tokens": 8191,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
},
- "azure_ai/mistral-large": {
+ "azure/text-embedding-3-small": {
+ "input_cost_per_token": 2e-08,
+ "litellm_provider": "azure",
+ "max_input_tokens": 8191,
"max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 4e-06,
- "output_cost_per_token": 1.2e-05,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
},
- "azure_ai/mistral-small": {
+ "azure/text-embedding-ada-002": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 8191,
"max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "azure_ai",
- "supports_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
},
- "azure_ai/mistral-small-2503": {
- "max_tokens": 128000,
+ "azure/tts-1": {
+ "input_cost_per_character": 1.5e-05,
+ "litellm_provider": "azure",
+ "mode": "audio_speech"
+ },
+ "azure/tts-1-hd": {
+ "input_cost_per_character": 3e-05,
+ "litellm_provider": "azure",
+ "mode": "audio_speech"
+ },
+ "azure/us/gpt-4o-2024-08-06": {
+ "cache_read_input_token_cost": 1.375e-06,
+ "input_cost_per_token": 2.75e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "azure_ai",
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_token": 1.1e-05,
"supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "azure_ai/mistral-large-2407": {
- "max_tokens": 4096,
+ "azure/us/gpt-4o-2024-11-20": {
+ "cache_creation_input_token_cost": 1.38e-06,
+ "input_cost_per_token": 2.75e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.1e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/us/gpt-4o-mini-2024-07-18": {
+ "cache_read_input_token_cost": 8.3e-08,
+ "input_cost_per_token": 1.65e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 6.6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/us/gpt-4o-mini-realtime-preview-2024-12-17": {
+ "cache_creation_input_audio_token_cost": 3.3e-07,
+ "cache_read_input_token_cost": 3.3e-07,
+ "input_cost_per_audio_token": 1.1e-05,
+ "input_cost_per_token": 6.6e-07,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "azure_ai",
- "supports_function_calling": true,
+ "max_tokens": 4096,
"mode": "chat",
- "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/000-000.mistral-ai-large-2407-offer?tab=Overview",
+ "output_cost_per_audio_token": 2.2e-05,
+ "output_cost_per_token": 2.64e-06,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "azure_ai/mistral-large-latest": {
- "max_tokens": 4096,
+ "azure/us/gpt-4o-realtime-preview-2024-10-01": {
+ "cache_creation_input_audio_token_cost": 2.2e-05,
+ "cache_read_input_token_cost": 2.75e-06,
+ "input_cost_per_audio_token": 0.00011,
+ "input_cost_per_token": 5.5e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "azure_ai",
- "supports_function_calling": true,
+ "max_tokens": 4096,
"mode": "chat",
- "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/000-000.mistral-ai-large-2407-offer?tab=Overview",
+ "output_cost_per_audio_token": 0.00022,
+ "output_cost_per_token": 2.2e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "azure_ai/ministral-3b": {
- "max_tokens": 4096,
+ "azure/us/gpt-4o-realtime-preview-2024-12-17": {
+ "cache_read_input_audio_token_cost": 2.5e-06,
+ "cache_read_input_token_cost": 2.75e-06,
+ "input_cost_per_audio_token": 4.4e-05,
+ "input_cost_per_token": 5.5e-06,
+ "litellm_provider": "azure",
"max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 4e-08,
- "output_cost_per_token": 4e-08,
- "litellm_provider": "azure_ai",
- "supports_function_calling": true,
+ "max_tokens": 4096,
"mode": "chat",
- "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/000-000.ministral-3b-2410-offer?tab=Overview",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 2.2e-05,
+ "supported_modalities": [
+ "text",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
+ "azure/us/o1-2024-12-17": {
+ "cache_read_input_token_cost": 8.25e-06,
+ "input_cost_per_token": 1.65e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 6.6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure/us/o1-mini-2024-09-12": {
+ "cache_read_input_token_cost": 6.05e-07,
+ "input_cost_per_token": 1.21e-06,
+ "input_cost_per_token_batches": 6.05e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 4.84e-06,
+ "output_cost_per_token_batches": 2.42e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_vision": false
+ },
+ "azure/us/o1-preview-2024-09-12": {
+ "cache_read_input_token_cost": 8.25e-06,
+ "input_cost_per_token": 1.65e-05,
+ "litellm_provider": "azure",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6.6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_vision": false
+ },
+ "azure/us/o3-mini-2025-01-31": {
+ "cache_read_input_token_cost": 6.05e-07,
+ "input_cost_per_token": 1.21e-06,
+ "input_cost_per_token_batches": 6.05e-07,
+ "litellm_provider": "azure",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.84e-06,
+ "output_cost_per_token_batches": 2.42e-06,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure/whisper-1": {
+ "input_cost_per_second": 0.0001,
+ "litellm_provider": "azure",
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0001
+ },
+ "azure_ai/Cohere-embed-v3-english": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 512,
+ "max_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024,
+ "source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cohere.cohere-embed-v3-english-offer?tab=PlansAndPrice",
+ "supports_embedding_image_input": true
+ },
+ "azure_ai/Cohere-embed-v3-multilingual": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 512,
+ "max_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024,
+ "source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cohere.cohere-embed-v3-english-offer?tab=PlansAndPrice",
+ "supports_embedding_image_input": true
+ },
+ "azure_ai/FLUX-1.1-pro": {
+ "litellm_provider": "azure_ai",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.04,
+ "source": "https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/black-forest-labs-flux-1-kontext-pro-and-flux1-1-pro-now-available-in-azure-ai-f/4434659",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "azure_ai/FLUX.1-Kontext-pro": {
+ "litellm_provider": "azure_ai",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.04,
+ "source": "https://azuremarketplace.microsoft.com/pt-br/marketplace/apps/cohere.cohere-embed-4-offer?tab=PlansAndPrice",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
"azure_ai/Llama-3.2-11B-Vision-Instruct": {
- "max_tokens": 2048,
+ "input_cost_per_token": 3.7e-07,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 128000,
"max_output_tokens": 2048,
- "input_cost_per_token": 3.7e-07,
- "output_cost_per_token": 3.7e-07,
- "litellm_provider": "azure_ai",
- "supports_function_calling": true,
- "supports_vision": true,
+ "max_tokens": 2048,
"mode": "chat",
+ "output_cost_per_token": 3.7e-07,
"source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/metagenai.meta-llama-3-2-11b-vision-instruct-offer?tab=Overview",
- "supports_tool_choice": true
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure_ai/Llama-3.2-90B-Vision-Instruct": {
+ "input_cost_per_token": 2.04e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 2.04e-06,
+ "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/metagenai.meta-llama-3-2-90b-vision-instruct-offer?tab=Overview",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
"azure_ai/Llama-3.3-70B-Instruct": {
- "max_tokens": 2048,
+ "input_cost_per_token": 7.1e-07,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 128000,
"max_output_tokens": 2048,
- "input_cost_per_token": 7.1e-07,
+ "max_tokens": 2048,
+ "mode": "chat",
"output_cost_per_token": 7.1e-07,
- "litellm_provider": "azure_ai",
- "supports_function_calling": true,
- "mode": "chat",
"source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/metagenai.llama-3-3-70b-instruct-offer?tab=Overview",
- "supports_tool_choice": true
- },
- "azure_ai/Llama-4-Scout-17B-16E-Instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 10000000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 7.8e-07,
- "litellm_provider": "azure_ai",
"supports_function_calling": true,
- "supports_vision": true,
- "mode": "chat",
- "source": "https://azure.microsoft.com/en-us/blog/introducing-the-llama-4-herd-in-azure-ai-foundry-and-azure-databricks/",
"supports_tool_choice": true
},
"azure_ai/Llama-4-Maverick-17B-128E-Instruct-FP8": {
- "max_tokens": 16384,
+ "input_cost_per_token": 1.41e-06,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 1000000,
"max_output_tokens": 16384,
- "input_cost_per_token": 1.41e-06,
+ "max_tokens": 16384,
+ "mode": "chat",
"output_cost_per_token": 3.5e-07,
- "litellm_provider": "azure_ai",
- "supports_function_calling": true,
- "supports_vision": true,
- "mode": "chat",
"source": "https://azure.microsoft.com/en-us/blog/introducing-the-llama-4-herd-in-azure-ai-foundry-and-azure-databricks/",
- "supports_tool_choice": true
- },
- "azure_ai/Llama-3.2-90B-Vision-Instruct": {
- "max_tokens": 2048,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 2.04e-06,
- "output_cost_per_token": 2.04e-06,
- "litellm_provider": "azure_ai",
"supports_function_calling": true,
- "supports_vision": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure_ai/Llama-4-Scout-17B-16E-Instruct": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 10000000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
- "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/metagenai.meta-llama-3-2-90b-vision-instruct-offer?tab=Overview",
- "supports_tool_choice": true
+ "output_cost_per_token": 7.8e-07,
+ "source": "https://azure.microsoft.com/en-us/blog/introducing-the-llama-4-herd-in-azure-ai-foundry-and-azure-databricks/",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
"azure_ai/Meta-Llama-3-70B-Instruct": {
- "max_tokens": 2048,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 8192,
"max_output_tokens": 2048,
- "input_cost_per_token": 1.1e-06,
+ "max_tokens": 2048,
+ "mode": "chat",
"output_cost_per_token": 3.7e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "azure_ai/Meta-Llama-3.1-8B-Instruct": {
- "max_tokens": 2048,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 6.1e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/metagenai.meta-llama-3-1-8b-instruct-offer?tab=PlansAndPrice",
- "supports_tool_choice": true
- },
- "azure_ai/Meta-Llama-3.1-70B-Instruct": {
- "max_tokens": 2048,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 2.68e-06,
- "output_cost_per_token": 3.54e-06,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/metagenai.meta-llama-3-1-70b-instruct-offer?tab=PlansAndPrice",
"supports_tool_choice": true
},
"azure_ai/Meta-Llama-3.1-405B-Instruct": {
- "max_tokens": 2048,
+ "input_cost_per_token": 5.33e-06,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 128000,
"max_output_tokens": 2048,
- "input_cost_per_token": 5.33e-06,
- "output_cost_per_token": 1.6e-05,
- "litellm_provider": "azure_ai",
+ "max_tokens": 2048,
"mode": "chat",
+ "output_cost_per_token": 1.6e-05,
"source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/metagenai.meta-llama-3-1-405b-instruct-offer?tab=PlansAndPrice",
"supports_tool_choice": true
},
- "azure_ai/Phi-4-mini-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 131072,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7.5e-08,
- "output_cost_per_token": 3e-07,
+ "azure_ai/Meta-Llama-3.1-70B-Instruct": {
+ "input_cost_per_token": 2.68e-06,
"litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://techcommunity.microsoft.com/blog/Azure-AI-Services-blog/announcing-new-phi-pricing-empowering-your-business-with-small-language-models/4395112"
- },
- "azure_ai/Phi-4-multimodal-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 131072,
- "max_output_tokens": 4096,
- "input_cost_per_token": 8e-08,
- "input_cost_per_audio_token": 4e-06,
- "output_cost_per_token": 3.2e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_audio_input": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "source": "https://techcommunity.microsoft.com/blog/Azure-AI-Services-blog/announcing-new-phi-pricing-empowering-your-business-with-small-language-models/4395112"
- },
- "azure_ai/Phi-4": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_vision": false,
- "source": "https://techcommunity.microsoft.com/blog/machinelearningblog/affordable-innovation-unveiling-the-pricing-of-phi-3-slms-on-models-as-a-service/4156495",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "azure_ai/Phi-3.5-mini-instruct": {
- "max_tokens": 4096,
"max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 5.2e-07,
- "litellm_provider": "azure_ai",
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
"mode": "chat",
- "supports_vision": false,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "output_cost_per_token": 3.54e-06,
+ "source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/metagenai.meta-llama-3-1-70b-instruct-offer?tab=PlansAndPrice",
"supports_tool_choice": true
},
- "azure_ai/Phi-3.5-vision-instruct": {
- "max_tokens": 4096,
+ "azure_ai/Meta-Llama-3.1-8B-Instruct": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 5.2e-07,
- "litellm_provider": "azure_ai",
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
"mode": "chat",
- "supports_vision": true,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
- "supports_tool_choice": true
- },
- "azure_ai/Phi-3.5-MoE-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.6e-07,
- "output_cost_per_token": 6.4e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_vision": false,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
- "supports_tool_choice": true
- },
- "azure_ai/Phi-3-mini-4k-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 5.2e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_vision": false,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
- "supports_tool_choice": true
- },
- "azure_ai/Phi-3-mini-128k-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 5.2e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_vision": false,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
- "supports_tool_choice": true
- },
- "azure_ai/Phi-3-small-8k-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_vision": false,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
- "supports_tool_choice": true
- },
- "azure_ai/Phi-3-small-128k-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_vision": false,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
- "supports_tool_choice": true
- },
- "azure_ai/Phi-3-medium-4k-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.7e-07,
- "output_cost_per_token": 6.8e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_vision": false,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "output_cost_per_token": 6.1e-07,
+ "source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/metagenai.meta-llama-3-1-8b-instruct-offer?tab=PlansAndPrice",
"supports_tool_choice": true
},
"azure_ai/Phi-3-medium-128k-instruct": {
- "max_tokens": 4096,
+ "input_cost_per_token": 1.7e-07,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 1.7e-07,
- "output_cost_per_token": 6.8e-07,
- "litellm_provider": "azure_ai",
- "mode": "chat",
- "supports_vision": false,
- "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
- "supports_tool_choice": true
- },
- "azure_ai/cohere-rerank-v3.5": {
"max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6.8e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-3-medium-4k-instruct": {
+ "input_cost_per_token": 1.7e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6.8e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-3-mini-128k-instruct": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 5.2e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-3-mini-4k-instruct": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 5.2e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-3-small-128k-instruct": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-3-small-8k-instruct": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-3.5-MoE-instruct": {
+ "input_cost_per_token": 1.6e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6.4e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-3.5-mini-instruct": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 5.2e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-3.5-vision-instruct": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 5.2e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/phi-3/",
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "azure_ai/Phi-4": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "source": "https://techcommunity.microsoft.com/blog/machinelearningblog/affordable-innovation-unveiling-the-pricing-of-phi-3-slms-on-models-as-a-service/4156495",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "azure_ai/Phi-4-mini-instruct": {
+ "input_cost_per_token": 7.5e-08,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "source": "https://techcommunity.microsoft.com/blog/Azure-AI-Services-blog/announcing-new-phi-pricing-empowering-your-business-with-small-language-models/4395112",
+ "supports_function_calling": true
+ },
+ "azure_ai/Phi-4-multimodal-instruct": {
+ "input_cost_per_audio_token": 4e-06,
+ "input_cost_per_token": 8e-08,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3.2e-07,
+ "source": "https://techcommunity.microsoft.com/blog/Azure-AI-Services-blog/announcing-new-phi-pricing-empowering-your-business-with-small-language-models/4395112",
+ "supports_audio_input": true,
+ "supports_function_calling": true,
+ "supports_vision": true
+ },
+ "azure_ai/Phi-4-mini-reasoning": {
+ "input_cost_per_token": 8e-08,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3.2e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
+ "supports_function_calling": true
+ },
+ "azure_ai/Phi-4-reasoning": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_reasoning": true
+ },
+ "azure_ai/MAI-DS-R1": {
+ "input_cost_per_token": 1.35e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 5.4e-06,
+ "source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "azure_ai/cohere-rerank-v3-english": {
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 4096,
"max_output_tokens": 4096,
"max_query_tokens": 2048,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "azure_ai",
- "mode": "rerank"
+ "max_tokens": 4096,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
},
"azure_ai/cohere-rerank-v3-multilingual": {
- "max_tokens": 4096,
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 4096,
"max_output_tokens": 4096,
"max_query_tokens": 2048,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "azure_ai",
- "mode": "rerank"
- },
- "azure_ai/cohere-rerank-v3-english": {
"max_tokens": 4096,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "azure_ai/cohere-rerank-v3.5": {
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "azure_ai",
"max_input_tokens": 4096,
"max_output_tokens": 4096,
"max_query_tokens": 2048,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "azure_ai",
- "mode": "rerank"
+ "max_tokens": 4096,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
},
- "azure_ai/Cohere-embed-v3-english": {
- "max_tokens": 512,
- "max_input_tokens": 512,
- "output_vector_size": 1024,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
+ "azure_ai/deepseek-r1": {
+ "input_cost_per_token": 1.35e-06,
"litellm_provider": "azure_ai",
- "mode": "embedding",
- "supports_embedding_image_input": true,
- "source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cohere.cohere-embed-v3-english-offer?tab=PlansAndPrice"
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 5.4e-06,
+ "source": "https://techcommunity.microsoft.com/blog/machinelearningblog/deepseek-r1-improved-performance-higher-limits-and-transparent-pricing/4386367",
+ "supports_reasoning": true,
+ "supports_tool_choice": true
},
- "azure_ai/Cohere-embed-v3-multilingual": {
- "max_tokens": 512,
- "max_input_tokens": 512,
- "output_vector_size": 1024,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
+ "azure_ai/deepseek-v3": {
+ "input_cost_per_token": 1.14e-06,
"litellm_provider": "azure_ai",
- "mode": "embedding",
- "supports_embedding_image_input": true,
- "source": "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cohere.cohere-embed-v3-english-offer?tab=PlansAndPrice"
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4.56e-06,
+ "source": "https://techcommunity.microsoft.com/blog/machinelearningblog/announcing-deepseek-v3-on-azure-ai-foundry-and-github/4390438",
+ "supports_tool_choice": true
+ },
+ "azure_ai/deepseek-v3-0324": {
+ "input_cost_per_token": 1.14e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4.56e-06,
+ "source": "https://techcommunity.microsoft.com/blog/machinelearningblog/announcing-deepseek-v3-on-azure-ai-foundry-and-github/4390438",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
},
"azure_ai/embed-v-4-0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "output_vector_size": 3072,
"input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 0.0,
"litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_tokens": 128000,
"mode": "embedding",
- "supports_embedding_image_input": true,
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 3072,
+ "source": "https://azuremarketplace.microsoft.com/pt-br/marketplace/apps/cohere.cohere-embed-4-offer?tab=PlansAndPrice",
"supported_endpoints": [
"/v1/embeddings"
],
@@ -4275,3529 +3314,5729 @@
"text",
"image"
],
- "source": "https://azuremarketplace.microsoft.com/pt-br/marketplace/apps/cohere.cohere-embed-4-offer?tab=PlansAndPrice"
+ "supports_embedding_image_input": true
+ },
+ "azure_ai/global/grok-3": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
+ "supports_function_calling": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "azure_ai/global/grok-3-mini": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.27e-06,
+ "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "azure_ai/grok-3": {
+ "input_cost_per_token": 3.3e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.65e-05,
+ "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
+ "supports_function_calling": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "azure_ai/grok-3-mini": {
+ "input_cost_per_token": 2.75e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.38e-06,
+ "source": "https://devblogs.microsoft.com/foundry/announcing-grok-3-and-grok-3-mini-on-azure-ai-foundry/",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "azure_ai/grok-4": {
+ "input_cost_per_token": 5.5e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.75e-05,
+ "source": "https://azure.microsoft.com/en-us/blog/grok-4-is-now-available-in-azure-ai-foundry-unlock-frontier-intelligence-and-business-ready-capabilities/",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "azure_ai/grok-4-fast-non-reasoning": {
+ "input_cost_per_token": 0.43e-06,
+ "output_cost_per_token": 1.73e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "azure_ai/grok-4-fast-reasoning": {
+ "input_cost_per_token": 0.43e-06,
+ "output_cost_per_token": 1.73e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "source": "https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/announcing-the-grok-4-fast-models-from-xai-now-available-in-azure-ai-foundry/4456701",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "azure_ai/grok-code-fast-1": {
+ "input_cost_per_token": 3.5e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.75e-05,
+ "source": "https://azure.microsoft.com/en-us/blog/grok-4-is-now-available-in-azure-ai-foundry-unlock-frontier-intelligence-and-business-ready-capabilities/",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "azure_ai/jais-30b-chat": {
+ "input_cost_per_token": 0.0032,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0.00971,
+ "source": "https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/models/jais-30b-chat"
+ },
+ "azure_ai/jamba-instruct": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 70000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
+ "supports_tool_choice": true
+ },
+ "azure_ai/ministral-3b": {
+ "input_cost_per_token": 4e-08,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 4e-08,
+ "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/000-000.ministral-3b-2410-offer?tab=Overview",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure_ai/mistral-large": {
+ "input_cost_per_token": 4e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure_ai/mistral-large-2407": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/000-000.mistral-ai-large-2407-offer?tab=Overview",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure_ai/mistral-large-latest": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/000-000.mistral-ai-large-2407-offer?tab=Overview",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure_ai/mistral-medium-2505": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure_ai/mistral-nemo": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://azuremarketplace.microsoft.com/en/marketplace/apps/000-000.mistral-nemo-12b-2407?tab=PlansAndPrice",
+ "supports_function_calling": true
+ },
+ "azure_ai/mistral-small": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "azure_ai/mistral-small-2503": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "azure_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
"babbage-002": {
- "max_tokens": 16384,
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "text-completion-openai",
- "mode": "completion"
- },
- "davinci-002": {
"max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "text-completion-openai",
- "mode": "completion"
- },
- "gpt-3.5-turbo-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "text-completion-openai",
- "mode": "completion"
- },
- "gpt-3.5-turbo-instruct-0914": {
- "max_tokens": 4097,
- "max_input_tokens": 8192,
- "max_output_tokens": 4097,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "text-completion-openai",
- "mode": "completion"
- },
- "mistral/mistral-tiny": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-small": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "mistral",
- "supports_function_calling": true,
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-small-latest": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "mistral",
- "supports_function_calling": true,
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-medium": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2.7e-06,
- "output_cost_per_token": 8.1e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-medium-latest": {
- "max_tokens": 8191,
- "max_input_tokens": 131072,
- "max_output_tokens": 8191,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-medium-2505": {
- "max_tokens": 8191,
- "max_input_tokens": 131072,
- "max_output_tokens": 8191,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-medium-2312": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2.7e-06,
- "output_cost_per_token": 8.1e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-large-latest": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-large-2411": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-large-2402": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 4e-06,
- "output_cost_per_token": 1.2e-05,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/mistral-large-2407": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 9e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/pixtral-large-latest": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/pixtral-large-2411": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/pixtral-12b-2409": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/open-mistral-7b": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/open-mixtral-8x7b": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/open-mixtral-8x22b": {
- "max_tokens": 8191,
- "max_input_tokens": 65336,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/codestral-latest": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/codestral-2405": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/open-mistral-nemo": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/technology/",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/open-mistral-nemo-2407": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/technology/",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/open-codestral-mamba": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/technology/",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "mistral/codestral-mamba-latest": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/technology/",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "mistral/devstral-small-2505": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/news/devstral",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/devstral-small-2507": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/news/devstral",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/devstral-medium-2507": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/news/devstral",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_response_schema": true
- },
- "mistral/magistral-medium-latest": {
- "max_tokens": 40000,
- "max_input_tokens": 40000,
- "max_output_tokens": 40000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/news/magistral",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": true
- },
- "mistral/magistral-medium-2506": {
- "max_tokens": 40000,
- "max_input_tokens": 40000,
- "max_output_tokens": 40000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/news/magistral",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": true
- },
- "mistral/magistral-small-latest": {
- "max_tokens": 40000,
- "max_input_tokens": 40000,
- "max_output_tokens": 40000,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/pricing#api-pricing",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": true
- },
- "mistral/magistral-small-2506": {
- "max_tokens": 40000,
- "max_input_tokens": 40000,
- "max_output_tokens": 40000,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "mistral",
- "mode": "chat",
- "source": "https://mistral.ai/pricing#api-pricing",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": true
- },
- "mistral/mistral-embed": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "input_cost_per_token": 1e-07,
- "litellm_provider": "mistral",
- "mode": "embedding"
- },
- "deepseek/deepseek-reasoner": {
- "max_tokens": 8192,
- "max_input_tokens": 65536,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5.5e-07,
- "input_cost_per_token_cache_hit": 1.4e-07,
- "output_cost_per_token": 2.19e-06,
- "litellm_provider": "deepseek",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "deepseek/deepseek-chat": {
- "max_tokens": 8192,
- "max_input_tokens": 65536,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2.7e-07,
- "input_cost_per_token_cache_hit": 7e-08,
- "cache_read_input_token_cost": 7e-08,
- "cache_creation_input_token_cost": 0.0,
- "output_cost_per_token": 1.1e-06,
- "litellm_provider": "deepseek",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_prompt_caching": true
- },
- "deepseek/deepseek-r1": {
- "max_tokens": 8192,
- "max_input_tokens": 65536,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5.5e-07,
- "input_cost_per_token_cache_hit": 1.4e-07,
- "output_cost_per_token": 2.19e-06,
- "litellm_provider": "deepseek",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_prompt_caching": true
- },
- "deepseek/deepseek-v3": {
- "max_tokens": 8192,
- "max_input_tokens": 65536,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2.7e-07,
- "input_cost_per_token_cache_hit": 7e-08,
- "cache_read_input_token_cost": 7e-08,
- "cache_creation_input_token_cost": 0.0,
- "output_cost_per_token": 1.1e-06,
- "litellm_provider": "deepseek",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_prompt_caching": true
- },
- "codestral/codestral-latest": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "codestral",
- "mode": "chat",
- "source": "https://docs.mistral.ai/capabilities/code_generation/",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "codestral/codestral-2405": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "codestral",
- "mode": "chat",
- "source": "https://docs.mistral.ai/capabilities/code_generation/",
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "text-completion-codestral/codestral-latest": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "text-completion-codestral",
"mode": "completion",
- "source": "https://docs.mistral.ai/capabilities/code_generation/"
+ "output_cost_per_token": 4e-07
},
- "text-completion-codestral/codestral-2405": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "text-completion-codestral",
- "mode": "completion",
- "source": "https://docs.mistral.ai/capabilities/code_generation/"
- },
- "xai/grok-beta": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "xai/grok-2-vision-1212": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2e-06,
- "input_cost_per_image": 2e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "xai/grok-2-vision-latest": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2e-06,
- "input_cost_per_image": 2e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "xai/grok-2-vision": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2e-06,
- "input_cost_per_image": 2e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "xai/grok-3": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-latest": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-beta": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-fast-beta": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 2.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-fast-latest": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 2.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-mini": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-mini-latest": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-mini-fast": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 4e-06,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-mini-fast-latest": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 4e-06,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_reasoning": true,
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-mini-beta": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-3-mini-fast-beta": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 4e-06,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_response_schema": false,
- "source": "https://x.ai/api#pricing",
- "supports_web_search": true
- },
- "xai/grok-vision-beta": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5e-06,
- "input_cost_per_image": 5e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "xai/grok-2-1212": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "xai/grok-2": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "xai/grok-2-latest": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "xai/grok-4": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "source": "https://docs.x.ai/docs/models",
- "supports_web_search": true
- },
- "xai/grok-4-0709": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "source": "https://docs.x.ai/docs/models",
- "supports_web_search": true
- },
- "xai/grok-4-latest": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "xai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "source": "https://docs.x.ai/docs/models",
- "supports_web_search": true
- },
- "deepseek/deepseek-coder": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.4e-07,
- "input_cost_per_token_cache_hit": 1.4e-08,
- "output_cost_per_token": 2.8e-07,
- "litellm_provider": "deepseek",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true,
- "supports_prompt_caching": true
- },
- "groq/deepseek-r1-distill-llama-70b": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 7.5e-07,
- "output_cost_per_token": 9.9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "groq/llama-3.3-70b-versatile": {
- "max_tokens": 32768,
- "max_input_tokens": 128000,
- "max_output_tokens": 32768,
- "input_cost_per_token": 5.9e-07,
- "output_cost_per_token": 7.9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "groq/llama-3.3-70b-specdec": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5.9e-07,
- "output_cost_per_token": 9.9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_tool_choice": true,
- "deprecation_date": "2025-04-14"
- },
- "groq/llama-guard-3-8b": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "groq",
- "mode": "chat"
- },
- "groq/llama2-70b-4096": {
- "max_tokens": 4096,
+ "bedrock/*/1-month-commitment/cohere.command-light-text-v14": {
+ "input_cost_per_second": 0.001902,
+ "litellm_provider": "bedrock",
"max_input_tokens": 4096,
"max_output_tokens": 4096,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 8e-07,
- "litellm_provider": "groq",
+ "max_tokens": 4096,
"mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
+ "output_cost_per_second": 0.001902,
"supports_tool_choice": true
},
- "groq/llama3-8b-8192": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 8e-08,
- "litellm_provider": "groq",
+ "bedrock/*/1-month-commitment/cohere.command-text-v14": {
+ "input_cost_per_second": 0.011,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
"mode": "chat",
+ "output_cost_per_second": 0.011,
"supports_tool_choice": true
},
- "groq/llama-3.2-1b-preview": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 4e-08,
- "output_cost_per_token": 4e-08,
- "litellm_provider": "groq",
+ "bedrock/*/6-month-commitment/cohere.command-light-text-v14": {
+ "input_cost_per_second": 0.0011416,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
"mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "deprecation_date": "2025-04-14"
- },
- "groq/llama-3.2-3b-preview": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 6e-08,
- "output_cost_per_token": 6e-08,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "deprecation_date": "2025-04-14"
- },
- "groq/llama-3.2-11b-text-preview": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.8e-07,
- "output_cost_per_token": 1.8e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "deprecation_date": "2024-10-28"
- },
- "groq/llama-3.2-11b-vision-preview": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.8e-07,
- "output_cost_per_token": 1.8e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "deprecation_date": "2025-04-14"
- },
- "groq/llama-3.2-90b-text-preview": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "deprecation_date": "2024-11-25"
- },
- "groq/llama-3.2-90b-vision-preview": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "deprecation_date": "2025-04-14"
- },
- "groq/llama3-70b-8192": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5.9e-07,
- "output_cost_per_token": 7.9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_response_schema": true,
+ "output_cost_per_second": 0.0011416,
"supports_tool_choice": true
},
- "groq/llama-3.1-8b-instant": {
- "max_tokens": 8192,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 8e-08,
- "litellm_provider": "groq",
+ "bedrock/*/6-month-commitment/cohere.command-text-v14": {
+ "input_cost_per_second": 0.0066027,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
"mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
+ "output_cost_per_second": 0.0066027,
"supports_tool_choice": true
},
- "groq/llama-3.1-70b-versatile": {
- "max_tokens": 8192,
+ "bedrock/ap-northeast-1/1-month-commitment/anthropic.claude-instant-v1": {
+ "input_cost_per_second": 0.01475,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.01475,
+ "supports_tool_choice": true
+ },
+ "bedrock/ap-northeast-1/1-month-commitment/anthropic.claude-v1": {
+ "input_cost_per_second": 0.0455,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.0455
+ },
+ "bedrock/ap-northeast-1/1-month-commitment/anthropic.claude-v2:1": {
+ "input_cost_per_second": 0.0455,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.0455,
+ "supports_tool_choice": true
+ },
+ "bedrock/ap-northeast-1/6-month-commitment/anthropic.claude-instant-v1": {
+ "input_cost_per_second": 0.008194,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.008194,
+ "supports_tool_choice": true
+ },
+ "bedrock/ap-northeast-1/6-month-commitment/anthropic.claude-v1": {
+ "input_cost_per_second": 0.02527,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.02527
+ },
+ "bedrock/ap-northeast-1/6-month-commitment/anthropic.claude-v2:1": {
+ "input_cost_per_second": 0.02527,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.02527,
+ "supports_tool_choice": true
+ },
+ "bedrock/ap-northeast-1/anthropic.claude-instant-v1": {
+ "input_cost_per_token": 2.23e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 7.55e-06,
+ "supports_tool_choice": true
+ },
+ "bedrock/ap-northeast-1/anthropic.claude-v1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "bedrock/ap-northeast-1/anthropic.claude-v2:1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "bedrock/ap-south-1/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 3.18e-06,
+ "litellm_provider": "bedrock",
"max_input_tokens": 8192,
"max_output_tokens": 8192,
- "input_cost_per_token": 5.9e-07,
- "output_cost_per_token": 7.9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "deprecation_date": "2025-01-24"
- },
- "groq/llama-3.1-405b-reasoning": {
"max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4.2e-06
+ },
+ "bedrock/ap-south-1/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3.6e-07,
+ "litellm_provider": "bedrock",
"max_input_tokens": 8192,
"max_output_tokens": 8192,
- "input_cost_per_token": 5.9e-07,
- "output_cost_per_token": 7.9e-07,
- "litellm_provider": "groq",
+ "max_tokens": 8192,
"mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
+ "output_cost_per_token": 7.2e-07
+ },
+ "bedrock/ca-central-1/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 3.05e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4.03e-06
+ },
+ "bedrock/ca-central-1/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6.9e-07
+ },
+ "bedrock/eu-central-1/1-month-commitment/anthropic.claude-instant-v1": {
+ "input_cost_per_second": 0.01635,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.01635,
"supports_tool_choice": true
},
- "groq/meta-llama/llama-4-scout-17b-16e-instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 131072,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.1e-07,
- "output_cost_per_token": 3.4e-07,
- "litellm_provider": "groq",
+ "bedrock/eu-central-1/1-month-commitment/anthropic.claude-v1": {
+ "input_cost_per_second": 0.0415,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
"mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
+ "output_cost_per_second": 0.0415
+ },
+ "bedrock/eu-central-1/1-month-commitment/anthropic.claude-v2:1": {
+ "input_cost_per_second": 0.0415,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.0415,
"supports_tool_choice": true
},
- "groq/meta-llama/llama-4-maverick-17b-128e-instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 131072,
+ "bedrock/eu-central-1/6-month-commitment/anthropic.claude-instant-v1": {
+ "input_cost_per_second": 0.009083,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.009083,
+ "supports_tool_choice": true
+ },
+ "bedrock/eu-central-1/6-month-commitment/anthropic.claude-v1": {
+ "input_cost_per_second": 0.02305,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.02305
+ },
+ "bedrock/eu-central-1/6-month-commitment/anthropic.claude-v2:1": {
+ "input_cost_per_second": 0.02305,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.02305,
+ "supports_tool_choice": true
+ },
+ "bedrock/eu-central-1/anthropic.claude-instant-v1": {
+ "input_cost_per_token": 2.48e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 8.38e-06,
+ "supports_tool_choice": true
+ },
+ "bedrock/eu-central-1/anthropic.claude-v1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05
+ },
+ "bedrock/eu-central-1/anthropic.claude-v2:1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "bedrock/eu-west-1/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 2.86e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
"max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 3.78e-06
+ },
+ "bedrock/eu-west-1/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3.2e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6.5e-07
+ },
+ "bedrock/eu-west-2/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 3.45e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4.55e-06
+ },
+ "bedrock/eu-west-2/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3.9e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 7.8e-07
+ },
+ "bedrock/eu-west-3/mistral.mistral-7b-instruct-v0:2": {
"input_cost_per_token": 2e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "groq/mistral-saba-24b": {
- "max_tokens": 32000,
+ "litellm_provider": "bedrock",
"max_input_tokens": 32000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 7.9e-07,
- "output_cost_per_token": 7.9e-07,
- "litellm_provider": "groq",
- "mode": "chat"
- },
- "groq/mixtral-8x7b-32768": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2.4e-07,
- "output_cost_per_token": 2.4e-07,
- "litellm_provider": "groq",
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
"mode": "chat",
+ "output_cost_per_token": 2.6e-07,
+ "supports_tool_choice": true
+ },
+ "bedrock/eu-west-3/mistral.mistral-large-2402-v1:0": {
+ "input_cost_per_token": 1.04e-05,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 3.12e-05,
+ "supports_function_calling": true
+ },
+ "bedrock/eu-west-3/mistral.mixtral-8x7b-instruct-v0:1": {
+ "input_cost_per_token": 5.9e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 9.1e-07,
+ "supports_tool_choice": true
+ },
+ "bedrock/invoke/anthropic.claude-3-5-sonnet-20240620-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "metadata": {
+ "notes": "Anthropic via Invoke route does not currently support pdf input."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
"supports_function_calling": true,
"supports_response_schema": true,
"supports_tool_choice": true,
- "deprecation_date": "2025-03-20"
+ "supports_vision": true
},
- "groq/gemma-7b-it": {
- "max_tokens": 8192,
+ "bedrock/sa-east-1/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 4.45e-06,
+ "litellm_provider": "bedrock",
"max_input_tokens": 8192,
"max_output_tokens": 8192,
- "input_cost_per_token": 7e-08,
- "output_cost_per_token": 7e-08,
- "litellm_provider": "groq",
+ "max_tokens": 8192,
"mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "deprecation_date": "2024-12-18"
+ "output_cost_per_token": 5.88e-06
},
- "groq/gemma2-9b-it": {
- "max_tokens": 8192,
+ "bedrock/sa-east-1/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "bedrock",
"max_input_tokens": 8192,
"max_output_tokens": 8192,
- "input_cost_per_token": 2e-07,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.01e-06
+ },
+ "bedrock/us-east-1/1-month-commitment/anthropic.claude-instant-v1": {
+ "input_cost_per_second": 0.011,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.011,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-east-1/1-month-commitment/anthropic.claude-v1": {
+ "input_cost_per_second": 0.0175,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.0175
+ },
+ "bedrock/us-east-1/1-month-commitment/anthropic.claude-v2:1": {
+ "input_cost_per_second": 0.0175,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.0175,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-east-1/6-month-commitment/anthropic.claude-instant-v1": {
+ "input_cost_per_second": 0.00611,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.00611,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-east-1/6-month-commitment/anthropic.claude-v1": {
+ "input_cost_per_second": 0.00972,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.00972
+ },
+ "bedrock/us-east-1/6-month-commitment/anthropic.claude-v2:1": {
+ "input_cost_per_second": 0.00972,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.00972,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-east-1/anthropic.claude-instant-v1": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-06,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-east-1/anthropic.claude-v1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-east-1/anthropic.claude-v2:1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-east-1/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 2.65e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 3.5e-06
+ },
+ "bedrock/us-east-1/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07
+ },
+ "bedrock/us-east-1/mistral.mistral-7b-instruct-v0:2": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
"output_cost_per_token": 2e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_response_schema": true,
- "supports_tool_choice": false
- },
- "groq/llama3-groq-70b-8192-tool-use-preview": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 8.9e-07,
- "output_cost_per_token": 8.9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "deprecation_date": "2025-01-06"
- },
- "groq/llama3-groq-8b-8192-tool-use-preview": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.9e-07,
- "output_cost_per_token": 1.9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "deprecation_date": "2025-01-06"
- },
- "groq/qwen/qwen3-32b": {
- "max_tokens": 131000,
- "max_input_tokens": 131000,
- "max_output_tokens": 131000,
- "input_cost_per_token": 2.9e-07,
- "output_cost_per_token": 5.9e-07,
- "litellm_provider": "groq",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_reasoning": true,
"supports_tool_choice": true
},
- "groq/moonshotai/kimi-k2-instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "groq",
+ "bedrock/us-east-1/mistral.mistral-large-2402-v1:0": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
"mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
+ "output_cost_per_token": 2.4e-05,
+ "supports_function_calling": true
+ },
+ "bedrock/us-east-1/mistral.mixtral-8x7b-instruct-v0:1": {
+ "input_cost_per_token": 4.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
"supports_tool_choice": true
},
- "groq/playai-tts": {
- "max_tokens": 10000,
- "max_input_tokens": 10000,
+ "bedrock/us-gov-east-1/amazon.nova-pro-v1:0": {
+ "input_cost_per_token": 9.6e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 300000,
"max_output_tokens": 10000,
- "input_cost_per_character": 5e-05,
- "litellm_provider": "groq",
- "mode": "audio_speech"
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 3.84e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
},
- "groq/whisper-large-v3": {
- "input_cost_per_second": 3.083e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "groq",
- "mode": "audio_transcription"
+ "bedrock/us-gov-east-1/amazon.titan-embed-text-v1": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1536
},
- "groq/whisper-large-v3-turbo": {
- "input_cost_per_second": 1.111e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "groq",
- "mode": "audio_transcription"
+ "bedrock/us-gov-east-1/amazon.titan-embed-text-v2:0": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024
},
- "groq/distil-whisper-large-v3-en": {
- "input_cost_per_second": 5.56e-06,
- "output_cost_per_second": 0.0,
- "litellm_provider": "groq",
- "mode": "audio_transcription"
+ "bedrock/us-gov-east-1/amazon.titan-text-express-v1": {
+ "input_cost_per_token": 1.3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 8000,
+ "max_tokens": 8000,
+ "mode": "chat",
+ "output_cost_per_token": 1.7e-06
},
- "cerebras/llama3.1-8b": {
- "max_tokens": 128000,
+ "bedrock/us-gov-east-1/amazon.titan-text-lite-v1": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 4000,
+ "max_tokens": 4000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07
+ },
+ "bedrock/us-gov-east-1/amazon.titan-text-premier-v1:0": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06
+ },
+ "bedrock/us-gov-east-1/anthropic.claude-3-5-sonnet-20240620-v1:0": {
+ "input_cost_per_token": 3.6e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "bedrock/us-gov-east-1/anthropic.claude-3-haiku-20240307-v1:0": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "bedrock/us-gov-east-1/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 2.65e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8000,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 3.5e-06,
+ "supports_pdf_input": true
+ },
+ "bedrock/us-gov-east-1/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8000,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 2.65e-06,
+ "supports_pdf_input": true
+ },
+ "bedrock/us-gov-west-1/amazon.nova-pro-v1:0": {
+ "input_cost_per_token": 9.6e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 3.84e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "bedrock/us-gov-west-1/amazon.titan-embed-text-v1": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1536
+ },
+ "bedrock/us-gov-west-1/amazon.titan-embed-text-v2:0": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1024
+ },
+ "bedrock/us-gov-west-1/amazon.titan-text-express-v1": {
+ "input_cost_per_token": 1.3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 8000,
+ "max_tokens": 8000,
+ "mode": "chat",
+ "output_cost_per_token": 1.7e-06
+ },
+ "bedrock/us-gov-west-1/amazon.titan-text-lite-v1": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 4000,
+ "max_tokens": 4000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07
+ },
+ "bedrock/us-gov-west-1/amazon.titan-text-premier-v1:0": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 42000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06
+ },
+ "bedrock/us-gov-west-1/anthropic.claude-3-5-sonnet-20240620-v1:0": {
+ "input_cost_per_token": 3.6e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "bedrock/us-gov-west-1/anthropic.claude-3-haiku-20240307-v1:0": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "bedrock/us-gov-west-1/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 2.65e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8000,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 3.5e-06,
+ "supports_pdf_input": true
+ },
+ "bedrock/us-gov-west-1/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8000,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 2.65e-06,
+ "supports_pdf_input": true
+ },
+ "bedrock/us-west-1/meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 2.65e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 3.5e-06
+ },
+ "bedrock/us-west-1/meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07
+ },
+ "bedrock/us-west-2/1-month-commitment/anthropic.claude-instant-v1": {
+ "input_cost_per_second": 0.011,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.011,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-west-2/1-month-commitment/anthropic.claude-v1": {
+ "input_cost_per_second": 0.0175,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.0175
+ },
+ "bedrock/us-west-2/1-month-commitment/anthropic.claude-v2:1": {
+ "input_cost_per_second": 0.0175,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.0175,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-west-2/6-month-commitment/anthropic.claude-instant-v1": {
+ "input_cost_per_second": 0.00611,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.00611,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-west-2/6-month-commitment/anthropic.claude-v1": {
+ "input_cost_per_second": 0.00972,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.00972
+ },
+ "bedrock/us-west-2/6-month-commitment/anthropic.claude-v2:1": {
+ "input_cost_per_second": 0.00972,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_second": 0.00972,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-west-2/anthropic.claude-instant-v1": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-06,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-west-2/anthropic.claude-v1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-west-2/anthropic.claude-v2:1": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-west-2/mistral.mistral-7b-instruct-v0:2": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "supports_tool_choice": true
+ },
+ "bedrock/us-west-2/mistral.mistral-large-2402-v1:0": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_function_calling": true
+ },
+ "bedrock/us-west-2/mistral.mixtral-8x7b-instruct-v0:1": {
+ "input_cost_per_token": 4.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
+ "supports_tool_choice": true
+ },
+ "bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0": {
+ "cache_creation_input_token_cost": 1e-06,
+ "cache_read_input_token_cost": 8e-08,
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "cerebras/llama-3.3-70b": {
+ "input_cost_per_token": 8.5e-07,
+ "litellm_provider": "cerebras",
"max_input_tokens": 128000,
"max_output_tokens": 128000,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "cerebras",
+ "max_tokens": 128000,
"mode": "chat",
+ "output_cost_per_token": 1.2e-06,
"supports_function_calling": true,
"supports_tool_choice": true
},
"cerebras/llama3.1-70b": {
- "max_tokens": 128000,
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "cerebras",
"max_input_tokens": 128000,
"max_output_tokens": 128000,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "cerebras",
+ "max_tokens": 128000,
"mode": "chat",
+ "output_cost_per_token": 6e-07,
"supports_function_calling": true,
"supports_tool_choice": true
},
- "cerebras/llama-3.3-70b": {
- "max_tokens": 128000,
+ "cerebras/llama3.1-8b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "cerebras",
"max_input_tokens": 128000,
"max_output_tokens": 128000,
- "input_cost_per_token": 8.5e-07,
- "output_cost_per_token": 1.2e-06,
- "litellm_provider": "cerebras",
+ "max_tokens": 128000,
"mode": "chat",
+ "output_cost_per_token": 1e-07,
"supports_function_calling": true,
"supports_tool_choice": true
},
+ "cerebras/openai/gpt-oss-120b": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "cerebras",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6.9e-07,
+ "source": "https://www.cerebras.ai/blog/openai-gpt-oss-120b-runs-fastest-on-cerebras",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
"cerebras/qwen-3-32b": {
- "max_tokens": 128000,
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "cerebras",
"max_input_tokens": 128000,
"max_output_tokens": 128000,
- "input_cost_per_token": 4e-07,
+ "max_tokens": 128000,
+ "mode": "chat",
"output_cost_per_token": 8e-07,
- "litellm_provider": "cerebras",
- "mode": "chat",
+ "source": "https://inference-docs.cerebras.ai/support/pricing",
"supports_function_calling": true,
- "supports_tool_choice": true,
- "source": "https://inference-docs.cerebras.ai/support/pricing"
- },
- "friendliai/meta-llama-3.1-8b-instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "friendliai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
"supports_tool_choice": true
},
- "friendliai/meta-llama-3.1-70b-instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "friendliai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "claude-3-haiku-20240307": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "cache_creation_input_token_cost": 3e-07,
- "cache_read_input_token_cost": 3e-08,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 264,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-03-01",
- "supports_tool_choice": true
- },
- "claude-3-5-haiku-20241022": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 4e-06,
- "cache_creation_input_token_cost": 1e-06,
- "cache_read_input_token_cost": 8e-08,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 264,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-10-01",
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "claude-3-5-haiku-latest": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 5e-06,
- "cache_creation_input_token_cost": 1.25e-06,
- "cache_read_input_token_cost": 1e-07,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 264,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-10-01",
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "claude-3-opus-latest": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 395,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-03-01",
- "supports_tool_choice": true
- },
- "claude-3-opus-20240229": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 395,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-03-01",
- "supports_tool_choice": true
- },
- "claude-3-5-sonnet-latest": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-06-01",
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "claude-3-5-sonnet-20240620": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-06-01",
- "supports_tool_choice": true
- },
- "claude-opus-4-20250514": {
- "max_tokens": 32000,
- "max_input_tokens": 200000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "claude-sonnet-4-20250514": {
- "max_tokens": 64000,
- "max_input_tokens": 200000,
- "max_output_tokens": 64000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "claude-4-opus-20250514": {
- "max_tokens": 32000,
- "max_input_tokens": 200000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "claude-4-sonnet-20250514": {
- "max_tokens": 64000,
- "max_input_tokens": 200000,
- "max_output_tokens": 64000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "claude-3-7-sonnet-latest": {
- "supports_computer_use": true,
- "max_tokens": 128000,
- "max_input_tokens": 200000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-06-01",
- "supports_tool_choice": true,
- "supports_reasoning": true
- },
- "claude-3-7-sonnet-20250219": {
- "supports_computer_use": true,
- "max_tokens": 128000,
- "max_input_tokens": 200000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2026-02-01",
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_web_search": true
- },
- "claude-3-5-sonnet-20241022": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "litellm_provider": "anthropic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-10-01",
- "supports_tool_choice": true,
- "supports_web_search": true
- },
- "text-bison": {
- "max_tokens": 2048,
- "max_input_tokens": 8192,
- "max_output_tokens": 2048,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-text-models",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "text-bison@001": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-text-models",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "text-bison@002": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-text-models",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "text-bison32k": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-text-models",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "text-bison32k@002": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-text-models",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "text-unicorn": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 2.8e-05,
- "litellm_provider": "vertex_ai-text-models",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "text-unicorn@001": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 2.8e-05,
- "litellm_provider": "vertex_ai-text-models",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
"chat-bison": {
- "max_tokens": 4096,
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-chat-models",
"max_input_tokens": 8192,
"max_output_tokens": 4096,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-chat-models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
- },
- "chat-bison@001": {
"max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-chat-models",
"mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
- },
- "chat-bison@002": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
"output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-chat-models",
- "mode": "chat",
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "deprecation_date": "2025-04-09",
"supports_tool_choice": true
},
"chat-bison-32k": {
- "max_tokens": 8192,
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-chat-models",
"max_input_tokens": 32000,
"max_output_tokens": 8192,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-chat-models",
+ "max_tokens": 8192,
"mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_tool_choice": true
},
"chat-bison-32k@002": {
- "max_tokens": 8192,
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-chat-models",
"max_input_tokens": 32000,
"max_output_tokens": 8192,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-chat-models",
+ "max_tokens": 8192,
"mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_tool_choice": true
},
+ "chat-bison@001": {
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-chat-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_tool_choice": true
+ },
+ "chat-bison@002": {
+ "deprecation_date": "2025-04-09",
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-chat-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_tool_choice": true
+ },
+ "chatdolphin": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "nlp_cloud",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07
+ },
+ "chatgpt-4o-latest": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "claude-3-5-haiku-20241022": {
+ "cache_creation_input_token_cost": 1e-06,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 8e-08,
+ "deprecation_date": "2025-10-01",
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tool_use_system_prompt_tokens": 264
+ },
+ "claude-3-5-haiku-latest": {
+ "cache_creation_input_token_cost": 1.25e-06,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 1e-07,
+ "deprecation_date": "2025-10-01",
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tool_use_system_prompt_tokens": 264
+ },
+ "claude-3-5-sonnet-20240620": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "deprecation_date": "2025-06-01",
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-3-5-sonnet-20241022": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "deprecation_date": "2025-10-01",
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-3-5-sonnet-latest": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "deprecation_date": "2025-06-01",
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-3-7-sonnet-20250219": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "deprecation_date": "2026-02-01",
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-3-7-sonnet-latest": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "deprecation_date": "2025-06-01",
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-3-haiku-20240307": {
+ "cache_creation_input_token_cost": 3e-07,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 3e-08,
+ "deprecation_date": "2025-03-01",
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 264
+ },
+ "claude-3-opus-20240229": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 1.5e-06,
+ "deprecation_date": "2025-03-01",
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 395
+ },
+ "claude-3-opus-latest": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 1.5e-06,
+ "deprecation_date": "2025-03-01",
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 395
+ },
+ "claude-4-opus-20250514": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-4-sonnet-20250514": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-sonnet-4-5": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 346
+ },
+ "claude-sonnet-4-5-20250929": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 346
+ },
+ "claude-opus-4-1": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_creation_input_token_cost_above_1hr": 3e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-opus-4-1-20250805": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_creation_input_token_cost_above_1hr": 3e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-opus-4-20250514": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_creation_input_token_cost_above_1hr": 3e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "claude-sonnet-4-20250514": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_creation_input_token_cost_above_1hr": 6e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "anthropic",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "cloudflare/@cf/meta/llama-2-7b-chat-fp16": {
+ "input_cost_per_token": 1.923e-06,
+ "litellm_provider": "cloudflare",
+ "max_input_tokens": 3072,
+ "max_output_tokens": 3072,
+ "max_tokens": 3072,
+ "mode": "chat",
+ "output_cost_per_token": 1.923e-06
+ },
+ "cloudflare/@cf/meta/llama-2-7b-chat-int8": {
+ "input_cost_per_token": 1.923e-06,
+ "litellm_provider": "cloudflare",
+ "max_input_tokens": 2048,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 1.923e-06
+ },
+ "cloudflare/@cf/mistral/mistral-7b-instruct-v0.1": {
+ "input_cost_per_token": 1.923e-06,
+ "litellm_provider": "cloudflare",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.923e-06
+ },
+ "cloudflare/@hf/thebloke/codellama-7b-instruct-awq": {
+ "input_cost_per_token": 1.923e-06,
+ "litellm_provider": "cloudflare",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.923e-06
+ },
"code-bison": {
- "max_tokens": 1024,
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-text-models",
"max_input_tokens": 6144,
"max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-text-models",
+ "max_tokens": 1024,
"mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_tool_choice": true
},
- "code-bison@001": {
- "max_tokens": 1024,
+ "code-bison-32k@002": {
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-text-models",
"max_input_tokens": 6144,
"max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-text-models",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "code-bison@002": {
"max_tokens": 1024,
- "max_input_tokens": 6144,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-text-models",
"mode": "completion",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
},
"code-bison32k": {
- "max_tokens": 1024,
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-text-models",
"max_input_tokens": 6144,
"max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-text-models",
+ "max_tokens": 1024,
"mode": "completion",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
},
- "code-bison-32k@002": {
- "max_tokens": 1024,
+ "code-bison@001": {
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-text-models",
"max_input_tokens": 6144,
"max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
+ "max_tokens": 1024,
+ "mode": "completion",
"output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-text-models",
- "mode": "completion",
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
},
- "code-gecko@001": {
- "max_tokens": 64,
- "max_input_tokens": 2048,
- "max_output_tokens": 64,
+ "code-bison@002": {
+ "input_cost_per_character": 2.5e-07,
"input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
"litellm_provider": "vertex_ai-code-text-models",
+ "max_input_tokens": 6144,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
"mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "code-gecko@002": {
- "max_tokens": 64,
- "max_input_tokens": 2048,
- "max_output_tokens": 64,
- "input_cost_per_token": 1.25e-07,
+ "output_cost_per_character": 5e-07,
"output_cost_per_token": 1.25e-07,
- "litellm_provider": "vertex_ai-code-text-models",
- "mode": "completion",
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
},
"code-gecko": {
- "max_tokens": 64,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-text-models",
"max_input_tokens": 2048,
"max_output_tokens": 64,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "litellm_provider": "vertex_ai-code-text-models",
+ "max_tokens": 64,
"mode": "completion",
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
},
"code-gecko-latest": {
- "max_tokens": 64,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-text-models",
"max_input_tokens": 2048,
"max_output_tokens": 64,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "litellm_provider": "vertex_ai-code-text-models",
+ "max_tokens": 64,
"mode": "completion",
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
},
- "codechat-bison@latest": {
- "max_tokens": 1024,
- "max_input_tokens": 6144,
- "max_output_tokens": 1024,
+ "code-gecko@001": {
"input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-text-models",
+ "max_input_tokens": 2048,
+ "max_output_tokens": 64,
+ "max_tokens": 64,
+ "mode": "completion",
"output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-chat-models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "code-gecko@002": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-text-models",
+ "max_input_tokens": 2048,
+ "max_output_tokens": 64,
+ "max_tokens": 64,
+ "mode": "completion",
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
},
"codechat-bison": {
- "max_tokens": 1024,
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-chat-models",
"max_input_tokens": 6144,
"max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-chat-models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
- },
- "codechat-bison@001": {
"max_tokens": 1024,
- "max_input_tokens": 6144,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-chat-models",
"mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
- },
- "codechat-bison@002": {
- "max_tokens": 1024,
- "max_input_tokens": 6144,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
"output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-chat-models",
- "mode": "chat",
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_tool_choice": true
},
"codechat-bison-32k": {
- "max_tokens": 8192,
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-chat-models",
"max_input_tokens": 32000,
"max_output_tokens": 8192,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-chat-models",
+ "max_tokens": 8192,
"mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_tool_choice": true
},
"codechat-bison-32k@002": {
- "max_tokens": 8192,
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-chat-models",
"max_input_tokens": 32000,
"max_output_tokens": 8192,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "input_cost_per_character": 2.5e-07,
- "output_cost_per_character": 5e-07,
- "litellm_provider": "vertex_ai-code-chat-models",
+ "max_tokens": 8192,
"mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_tool_choice": true
},
- "meta_llama/Llama-4-Scout-17B-16E-Instruct-FP8": {
- "max_tokens": 128000,
- "max_input_tokens": 10000000,
- "max_output_tokens": 4028,
- "litellm_provider": "meta_llama",
+ "codechat-bison@001": {
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-chat-models",
+ "max_input_tokens": 6144,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
"mode": "chat",
- "supports_function_calling": true,
- "source": "https://llama.developer.meta.com/docs/models",
- "supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ]
- },
- "meta_llama/Llama-4-Maverick-17B-128E-Instruct-FP8": {
- "max_tokens": 128000,
- "max_input_tokens": 1000000,
- "max_output_tokens": 4028,
- "litellm_provider": "meta_llama",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://llama.developer.meta.com/docs/models",
- "supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text"
- ]
- },
- "meta_llama/Llama-3.3-70B-Instruct": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4028,
- "litellm_provider": "meta_llama",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://llama.developer.meta.com/docs/models",
- "supports_tool_choice": true,
- "supported_modalities": [
- "text"
- ],
- "supported_output_modalities": [
- "text"
- ]
- },
- "meta_llama/Llama-3.3-8B-Instruct": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4028,
- "litellm_provider": "meta_llama",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://llama.developer.meta.com/docs/models",
- "supports_tool_choice": true,
- "supported_modalities": [
- "text"
- ],
- "supported_output_modalities": [
- "text"
- ]
- },
- "gemini-pro": {
- "max_tokens": 8192,
- "max_input_tokens": 32760,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0.0025,
- "input_cost_per_video_per_second": 0.002,
- "input_cost_per_token": 5e-07,
- "input_cost_per_character": 1.25e-07,
- "output_cost_per_token": 1.5e-06,
- "output_cost_per_character": 3.75e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_tool_choice": true
},
- "gemini-1.0-pro": {
- "max_tokens": 8192,
- "max_input_tokens": 32760,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0.0025,
- "input_cost_per_video_per_second": 0.002,
- "input_cost_per_token": 5e-07,
- "input_cost_per_character": 1.25e-07,
- "output_cost_per_token": 1.5e-06,
- "output_cost_per_character": 3.75e-07,
- "litellm_provider": "vertex_ai-language-models",
+ "codechat-bison@002": {
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-chat-models",
+ "max_input_tokens": 6144,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
"mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_tool_choice": true
+ },
+ "codechat-bison@latest": {
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-code-chat-models",
+ "max_input_tokens": 6144,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "chat",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_tool_choice": true
+ },
+ "codestral/codestral-2405": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "codestral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "source": "https://docs.mistral.ai/capabilities/code_generation/",
+ "supports_assistant_prefill": true,
+ "supports_tool_choice": true
+ },
+ "codestral/codestral-latest": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "codestral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "source": "https://docs.mistral.ai/capabilities/code_generation/",
+ "supports_assistant_prefill": true,
+ "supports_tool_choice": true
+ },
+ "codex-mini-latest": {
+ "cache_read_input_token_cost": 3.75e-07,
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 6e-06,
+ "supported_endpoints": [
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
"supports_function_calling": true,
"supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "cohere.command-light-text-v14": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_tool_choice": true
+ },
+ "cohere.command-r-plus-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_tool_choice": true
+ },
+ "cohere.command-r-v1:0": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_tool_choice": true
+ },
+ "cohere.command-text-v14": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_tool_choice": true
+ },
+ "cohere.embed-english-v3": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 512,
+ "max_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "supports_embedding_image_input": true
+ },
+ "cohere.embed-multilingual-v3": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 512,
+ "max_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "supports_embedding_image_input": true
+ },
+ "cohere.embed-v4:0": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1536,
+ "supports_embedding_image_input": true
+ },
+ "cohere.rerank-v3-5:0": {
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "bedrock",
+ "max_document_chunks_per_query": 100,
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_query_tokens": 32000,
+ "max_tokens": 32000,
+ "max_tokens_per_document_chunk": 512,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "command": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "completion",
+ "output_cost_per_token": 2e-06
+ },
+ "command-a-03-2025": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "cohere_chat",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 8000,
+ "max_tokens": 8000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "command-light": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "cohere_chat",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_tool_choice": true
+ },
+ "command-nightly": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "completion",
+ "output_cost_per_token": 2e-06
+ },
+ "command-r": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "cohere_chat",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "command-r-08-2024": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "cohere_chat",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "command-r-plus": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "cohere_chat",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "command-r-plus-08-2024": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "cohere_chat",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "command-r7b-12-2024": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "cohere_chat",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3.75e-08,
+ "source": "https://docs.cohere.com/v2/docs/command-r7b",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "computer-use-preview": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "azure",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-05,
+ "supported_endpoints": [
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": false,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "deepseek-chat": {
+ "cache_read_input_token_cost": 6e-08,
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "deepseek",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8192,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.7e-06,
+ "source": "https://api-docs.deepseek.com/quick_start/pricing",
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "deepseek-reasoner": {
+ "cache_read_input_token_cost": 6e-08,
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "deepseek",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 65536,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.7e-06,
+ "source": "https://api-docs.deepseek.com/quick_start/pricing",
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supports_function_calling": false,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": false
+ },
+ "dashscope/qwen-coder": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 16384,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-flash": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 32768,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 4e-07,
+ "range": [
+ 0,
+ 256000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 2.5e-07,
+ "output_cost_per_token": 2e-06,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen-flash-2025-07-28": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 32768,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 4e-07,
+ "range": [
+ 0,
+ 256000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 2.5e-07,
+ "output_cost_per_token": 2e-06,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen-max": {
+ "input_cost_per_token": 1.6e-06,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 30720,
+ "max_output_tokens": 8192,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6.4e-06,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-plus": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 129024,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-plus-2025-01-25": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 129024,
+ "max_output_tokens": 8192,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-plus-2025-04-28": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 129024,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 4e-06,
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-plus-2025-07-14": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 129024,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 4e-06,
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-plus-2025-07-28": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 32768,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "input_cost_per_token": 4e-07,
+ "output_cost_per_reasoning_token": 4e-06,
+ "output_cost_per_token": 1.2e-06,
+ "range": [
+ 0,
+ 256000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 1.2e-06,
+ "output_cost_per_reasoning_token": 1.2e-05,
+ "output_cost_per_token": 3.6e-06,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen-plus-2025-09-11": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 32768,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "input_cost_per_token": 4e-07,
+ "output_cost_per_reasoning_token": 4e-06,
+ "output_cost_per_token": 1.2e-06,
+ "range": [
+ 0,
+ 256000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 1.2e-06,
+ "output_cost_per_reasoning_token": 1.2e-05,
+ "output_cost_per_token": 3.6e-06,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen-plus-latest": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 32768,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "input_cost_per_token": 4e-07,
+ "output_cost_per_reasoning_token": 4e-06,
+ "output_cost_per_token": 1.2e-06,
+ "range": [
+ 0,
+ 256000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 1.2e-06,
+ "output_cost_per_reasoning_token": 1.2e-05,
+ "output_cost_per_token": 3.6e-06,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen-turbo": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 129024,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 5e-07,
+ "output_cost_per_token": 2e-07,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-turbo-2024-11-01": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-turbo-2025-04-28": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 16384,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 5e-07,
+ "output_cost_per_token": 2e-07,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen-turbo-latest": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 16384,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 5e-07,
+ "output_cost_per_token": 2e-07,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen3-30b-a3b": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 129024,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dashscope/qwen3-coder-flash": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 65536,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "cache_read_input_token_cost": 8e-08,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 1.5e-06,
+ "range": [
+ 0,
+ 32000.0
+ ]
+ },
+ {
+ "cache_read_input_token_cost": 1.2e-07,
+ "input_cost_per_token": 5e-07,
+ "output_cost_per_token": 2.5e-06,
+ "range": [
+ 32000.0,
+ 128000.0
+ ]
+ },
+ {
+ "cache_read_input_token_cost": 2e-07,
+ "input_cost_per_token": 8e-07,
+ "output_cost_per_token": 4e-06,
+ "range": [
+ 128000.0,
+ 256000.0
+ ]
+ },
+ {
+ "cache_read_input_token_cost": 4e-07,
+ "input_cost_per_token": 1.6e-06,
+ "output_cost_per_token": 9.6e-06,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen3-coder-flash-2025-07-28": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 65536,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 1.5e-06,
+ "range": [
+ 0,
+ 32000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 5e-07,
+ "output_cost_per_token": 2.5e-06,
+ "range": [
+ 32000.0,
+ 128000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 8e-07,
+ "output_cost_per_token": 4e-06,
+ "range": [
+ 128000.0,
+ 256000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 1.6e-06,
+ "output_cost_per_token": 9.6e-06,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen3-coder-plus": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 65536,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "cache_read_input_token_cost": 1e-07,
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 5e-06,
+ "range": [
+ 0,
+ 32000.0
+ ]
+ },
+ {
+ "cache_read_input_token_cost": 1.8e-07,
+ "input_cost_per_token": 1.8e-06,
+ "output_cost_per_token": 9e-06,
+ "range": [
+ 32000.0,
+ 128000.0
+ ]
+ },
+ {
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "output_cost_per_token": 1.5e-05,
+ "range": [
+ 128000.0,
+ 256000.0
+ ]
+ },
+ {
+ "cache_read_input_token_cost": 6e-07,
+ "input_cost_per_token": 6e-06,
+ "output_cost_per_token": 6e-05,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen3-coder-plus-2025-07-22": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 997952,
+ "max_output_tokens": 65536,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 5e-06,
+ "range": [
+ 0,
+ 32000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 1.8e-06,
+ "output_cost_per_token": 9e-06,
+ "range": [
+ 32000.0,
+ 128000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 3e-06,
+ "output_cost_per_token": 1.5e-05,
+ "range": [
+ 128000.0,
+ 256000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 6e-06,
+ "output_cost_per_token": 6e-05,
+ "range": [
+ 256000.0,
+ 1000000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwen3-max-preview": {
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 258048,
+ "max_output_tokens": 65536,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "tiered_pricing": [
+ {
+ "input_cost_per_token": 1.2e-06,
+ "output_cost_per_token": 6e-06,
+ "range": [
+ 0,
+ 32000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 2.4e-06,
+ "output_cost_per_token": 1.2e-05,
+ "range": [
+ 32000.0,
+ 128000.0
+ ]
+ },
+ {
+ "input_cost_per_token": 3e-06,
+ "output_cost_per_token": 1.5e-05,
+ "range": [
+ 128000.0,
+ 252000.0
+ ]
+ }
+ ]
+ },
+ "dashscope/qwq-plus": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "dashscope",
+ "max_input_tokens": 98304,
+ "max_output_tokens": 8192,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-06,
+ "source": "https://www.alibabacloud.com/help/en/model-studio/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-bge-large-en": {
+ "input_cost_per_token": 1.0003e-07,
+ "input_dbu_cost_per_token": 1.429e-06,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 512,
+ "max_tokens": 512,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_dbu_cost_per_token": 0.0,
+ "output_vector_size": 1024,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving"
+ },
+ "databricks/databricks-claude-3-7-sonnet": {
+ "input_cost_per_token": 2.5e-06,
+ "input_dbu_cost_per_token": 3.571e-05,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 128000,
+ "max_tokens": 200000,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Claude 3.7 conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 1.7857e-05,
+ "output_db_cost_per_token": 0.000214286,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-gte-large-en": {
+ "input_cost_per_token": 1.2999e-07,
+ "input_dbu_cost_per_token": 1.857e-06,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_dbu_cost_per_token": 0.0,
+ "output_vector_size": 1024,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving"
+ },
+ "databricks/databricks-llama-2-70b-chat": {
+ "input_cost_per_token": 5.0001e-07,
+ "input_dbu_cost_per_token": 7.143e-06,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "output_dbu_cost_per_token": 2.1429e-05,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-llama-4-maverick": {
+ "input_cost_per_token": 5e-06,
+ "input_dbu_cost_per_token": 7.143e-05,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "metadata": {
+ "notes": "Databricks documentation now provides both DBU costs (_dbu_cost_per_token) and dollar costs(_cost_per_token)."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "output_dbu_cost_per_token": 0.00021429,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-meta-llama-3-1-405b-instruct": {
+ "input_cost_per_token": 5e-06,
+ "input_dbu_cost_per_token": 7.1429e-05,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 1.500002e-05,
+ "output_db_cost_per_token": 0.000214286,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-meta-llama-3-3-70b-instruct": {
+ "input_cost_per_token": 1.00002e-06,
+ "input_dbu_cost_per_token": 1.4286e-05,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 2.99999e-06,
+ "output_dbu_cost_per_token": 4.2857e-05,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-meta-llama-3-70b-instruct": {
+ "input_cost_per_token": 1.00002e-06,
+ "input_dbu_cost_per_token": 1.4286e-05,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 2.99999e-06,
+ "output_dbu_cost_per_token": 4.2857e-05,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-mixtral-8x7b-instruct": {
+ "input_cost_per_token": 5.0001e-07,
+ "input_dbu_cost_per_token": 7.143e-06,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 9.9902e-07,
+ "output_dbu_cost_per_token": 1.4286e-05,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-mpt-30b-instruct": {
+ "input_cost_per_token": 9.9902e-07,
+ "input_dbu_cost_per_token": 1.4286e-05,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 9.9902e-07,
+ "output_dbu_cost_per_token": 1.4286e-05,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_tool_choice": true
+ },
+ "databricks/databricks-mpt-7b-instruct": {
+ "input_cost_per_token": 5.0001e-07,
+ "input_dbu_cost_per_token": 7.143e-06,
+ "litellm_provider": "databricks",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "metadata": {
+ "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "output_dbu_cost_per_token": 0.0,
+ "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
+ "supports_tool_choice": true
+ },
+ "davinci-002": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "text-completion-openai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 4096,
+ "max_tokens": 16384,
+ "mode": "completion",
+ "output_cost_per_token": 2e-06
+ },
+ "deepgram/base": {
+ "input_cost_per_second": 0.00020833,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0125/60 seconds = $0.00020833 per second",
+ "original_pricing_per_minute": 0.0125
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/base-conversationalai": {
+ "input_cost_per_second": 0.00020833,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0125/60 seconds = $0.00020833 per second",
+ "original_pricing_per_minute": 0.0125
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/base-finance": {
+ "input_cost_per_second": 0.00020833,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0125/60 seconds = $0.00020833 per second",
+ "original_pricing_per_minute": 0.0125
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/base-general": {
+ "input_cost_per_second": 0.00020833,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0125/60 seconds = $0.00020833 per second",
+ "original_pricing_per_minute": 0.0125
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/base-meeting": {
+ "input_cost_per_second": 0.00020833,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0125/60 seconds = $0.00020833 per second",
+ "original_pricing_per_minute": 0.0125
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/base-phonecall": {
+ "input_cost_per_second": 0.00020833,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0125/60 seconds = $0.00020833 per second",
+ "original_pricing_per_minute": 0.0125
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/base-video": {
+ "input_cost_per_second": 0.00020833,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0125/60 seconds = $0.00020833 per second",
+ "original_pricing_per_minute": 0.0125
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/base-voicemail": {
+ "input_cost_per_second": 0.00020833,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0125/60 seconds = $0.00020833 per second",
+ "original_pricing_per_minute": 0.0125
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/enhanced": {
+ "input_cost_per_second": 0.00024167,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0145/60 seconds = $0.00024167 per second",
+ "original_pricing_per_minute": 0.0145
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/enhanced-finance": {
+ "input_cost_per_second": 0.00024167,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0145/60 seconds = $0.00024167 per second",
+ "original_pricing_per_minute": 0.0145
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/enhanced-general": {
+ "input_cost_per_second": 0.00024167,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0145/60 seconds = $0.00024167 per second",
+ "original_pricing_per_minute": 0.0145
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/enhanced-meeting": {
+ "input_cost_per_second": 0.00024167,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0145/60 seconds = $0.00024167 per second",
+ "original_pricing_per_minute": 0.0145
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/enhanced-phonecall": {
+ "input_cost_per_second": 0.00024167,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0145/60 seconds = $0.00024167 per second",
+ "original_pricing_per_minute": 0.0145
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-atc": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-automotive": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-conversationalai": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-drivethru": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-finance": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-general": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-meeting": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-phonecall": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-video": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-2-voicemail": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-3": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-3-general": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-3-medical": {
+ "input_cost_per_second": 8.667e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0052/60 seconds = $0.00008667 per second (multilingual)",
+ "original_pricing_per_minute": 0.0052
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-general": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/nova-phonecall": {
+ "input_cost_per_second": 7.167e-05,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "calculation": "$0.0043/60 seconds = $0.00007167 per second",
+ "original_pricing_per_minute": 0.0043
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/whisper": {
+ "input_cost_per_second": 0.0001,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/whisper-base": {
+ "input_cost_per_second": 0.0001,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/whisper-large": {
+ "input_cost_per_second": 0.0001,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/whisper-medium": {
+ "input_cost_per_second": 0.0001,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/whisper-small": {
+ "input_cost_per_second": 0.0001,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepgram/whisper-tiny": {
+ "input_cost_per_second": 0.0001,
+ "litellm_provider": "deepgram",
+ "metadata": {
+ "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://deepgram.com/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "deepinfra/Gryphe/MythoMax-L2-13b": {
+ "max_tokens": 4096,
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "input_cost_per_token": 8e-08,
+ "output_cost_per_token": 9e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/NousResearch/Hermes-3-Llama-3.1-405B": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 1e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/NousResearch/Hermes-3-Llama-3.1-70B": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 3e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/Qwen/QwQ-32B": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 1.5e-07,
+ "output_cost_per_token": 4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen2.5-72B-Instruct": {
+ "max_tokens": 32768,
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "input_cost_per_token": 1.2e-07,
+ "output_cost_per_token": 3.9e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen2.5-7B-Instruct": {
+ "max_tokens": 32768,
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "input_cost_per_token": 4e-08,
+ "output_cost_per_token": 1e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/Qwen/Qwen2.5-VL-32B-Instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 2e-07,
+ "output_cost_per_token": 6e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-14B": {
+ "max_tokens": 40960,
+ "max_input_tokens": 40960,
+ "max_output_tokens": 40960,
+ "input_cost_per_token": 6e-08,
+ "output_cost_per_token": 2.4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-235B-A22B": {
+ "max_tokens": 40960,
+ "max_input_tokens": 40960,
+ "max_output_tokens": 40960,
+ "input_cost_per_token": 1.8e-07,
+ "output_cost_per_token": 5.4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-235B-A22B-Instruct-2507": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 9e-08,
+ "output_cost_per_token": 6e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-235B-A22B-Thinking-2507": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 2.9e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-30B-A3B": {
+ "max_tokens": 40960,
+ "max_input_tokens": 40960,
+ "max_output_tokens": 40960,
+ "input_cost_per_token": 8e-08,
+ "output_cost_per_token": 2.9e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-32B": {
+ "max_tokens": 40960,
+ "max_input_tokens": 40960,
+ "max_output_tokens": 40960,
+ "input_cost_per_token": 1e-07,
+ "output_cost_per_token": 2.8e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-Coder-480B-A35B-Instruct": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 4e-07,
+ "output_cost_per_token": 1.6e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-Coder-480B-A35B-Instruct-Turbo": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 2.9e-07,
+ "output_cost_per_token": 1.2e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-Next-80B-A3B-Instruct": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 1.4e-07,
+ "output_cost_per_token": 1.4e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Qwen/Qwen3-Next-80B-A3B-Thinking": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 1.4e-07,
+ "output_cost_per_token": 1.4e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/Sao10K/L3-8B-Lunaris-v1-Turbo": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 4e-08,
+ "output_cost_per_token": 5e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/Sao10K/L3.1-70B-Euryale-v2.2": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 6.5e-07,
+ "output_cost_per_token": 7.5e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/Sao10K/L3.3-70B-Euryale-v2.3": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 6.5e-07,
+ "output_cost_per_token": 7.5e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/allenai/olmOCR-7B-0725-FP8": {
+ "max_tokens": 16384,
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 1.5e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/anthropic/claude-3-7-sonnet-latest": {
+ "max_tokens": 200000,
+ "max_input_tokens": 200000,
+ "max_output_tokens": 200000,
+ "input_cost_per_token": 3.3e-06,
+ "output_cost_per_token": 1.65e-05,
+ "cache_read_input_token_cost": 3.3e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/anthropic/claude-4-opus": {
+ "max_tokens": 200000,
+ "max_input_tokens": 200000,
+ "max_output_tokens": 200000,
+ "input_cost_per_token": 1.65e-05,
+ "output_cost_per_token": 8.25e-05,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/anthropic/claude-4-sonnet": {
+ "max_tokens": 200000,
+ "max_input_tokens": 200000,
+ "max_output_tokens": 200000,
+ "input_cost_per_token": 3.3e-06,
+ "output_cost_per_token": 1.65e-05,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-R1": {
+ "max_tokens": 163840,
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "input_cost_per_token": 7e-07,
+ "output_cost_per_token": 2.4e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-R1-0528": {
+ "max_tokens": 163840,
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "input_cost_per_token": 5e-07,
+ "output_cost_per_token": 2.15e-06,
+ "cache_read_input_token_cost": 4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-R1-0528-Turbo": {
+ "max_tokens": 32768,
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 3e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-R1-Distill-Llama-70B": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 2e-07,
+ "output_cost_per_token": 6e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 2.7e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-R1-Turbo": {
+ "max_tokens": 40960,
+ "max_input_tokens": 40960,
+ "max_output_tokens": 40960,
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 3e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-V3": {
+ "max_tokens": 163840,
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "input_cost_per_token": 3.8e-07,
+ "output_cost_per_token": 8.9e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-V3-0324": {
+ "max_tokens": 163840,
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "input_cost_per_token": 2.5e-07,
+ "output_cost_per_token": 8.8e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-V3.1": {
+ "max_tokens": 163840,
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 1e-06,
+ "cache_read_input_token_cost": 2.16e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true,
+ "supports_reasoning": true
+ },
+ "deepinfra/deepseek-ai/DeepSeek-V3.1-Terminus": {
+ "max_tokens": 163840,
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "input_cost_per_token": 2.7e-07,
+ "output_cost_per_token": 1e-06,
+ "cache_read_input_token_cost": 2.16e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/google/gemini-2.0-flash-001": {
+ "max_tokens": 1000000,
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 1000000,
+ "input_cost_per_token": 1e-07,
+ "output_cost_per_token": 4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/google/gemini-2.5-flash": {
+ "max_tokens": 1000000,
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 1000000,
+ "input_cost_per_token": 3e-07,
+ "output_cost_per_token": 2.5e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/google/gemini-2.5-pro": {
+ "max_tokens": 1000000,
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 1000000,
+ "input_cost_per_token": 1.25e-06,
+ "output_cost_per_token": 1e-05,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/google/gemma-3-12b-it": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 1e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/google/gemma-3-27b-it": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 9e-08,
+ "output_cost_per_token": 1.6e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/google/gemma-3-4b-it": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 4e-08,
+ "output_cost_per_token": 8e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Llama-3.2-11B-Vision-Instruct": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 4.9e-08,
+ "output_cost_per_token": 4.9e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/meta-llama/Llama-3.2-3B-Instruct": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 2e-08,
+ "output_cost_per_token": 2e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Llama-3.3-70B-Instruct": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 2.3e-07,
+ "output_cost_per_token": 4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Llama-3.3-70B-Instruct-Turbo": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 1.3e-07,
+ "output_cost_per_token": 3.9e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8": {
+ "max_tokens": 1048576,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 1048576,
+ "input_cost_per_token": 1.5e-07,
+ "output_cost_per_token": 6e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct": {
+ "max_tokens": 327680,
+ "max_input_tokens": 327680,
+ "max_output_tokens": 327680,
+ "input_cost_per_token": 8e-08,
+ "output_cost_per_token": 3e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Llama-Guard-3-8B": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 5.5e-08,
+ "output_cost_per_token": 5.5e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/meta-llama/Llama-Guard-4-12B": {
+ "max_tokens": 163840,
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "input_cost_per_token": 1.8e-07,
+ "output_cost_per_token": 1.8e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/meta-llama/Meta-Llama-3-8B-Instruct": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 3e-08,
+ "output_cost_per_token": 6e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Meta-Llama-3.1-70B-Instruct": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 1e-07,
+ "output_cost_per_token": 2.8e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Meta-Llama-3.1-8B-Instruct": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 3e-08,
+ "output_cost_per_token": 5e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 2e-08,
+ "output_cost_per_token": 3e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/microsoft/WizardLM-2-8x22B": {
+ "max_tokens": 65536,
+ "max_input_tokens": 65536,
+ "max_output_tokens": 65536,
+ "input_cost_per_token": 4.8e-07,
+ "output_cost_per_token": 4.8e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": false
+ },
+ "deepinfra/microsoft/phi-4": {
+ "max_tokens": 16384,
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "input_cost_per_token": 7e-08,
+ "output_cost_per_token": 1.4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/mistralai/Mistral-Nemo-Instruct-2407": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 2e-08,
+ "output_cost_per_token": 4e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/mistralai/Mistral-Small-24B-Instruct-2501": {
+ "max_tokens": 32768,
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 8e-08,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/mistralai/Mistral-Small-3.2-24B-Instruct-2506": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 7.5e-08,
+ "output_cost_per_token": 2e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/mistralai/Mixtral-8x7B-Instruct-v0.1": {
+ "max_tokens": 32768,
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "input_cost_per_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/moonshotai/Kimi-K2-Instruct": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 5e-07,
+ "output_cost_per_token": 2e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/moonshotai/Kimi-K2-Instruct-0905": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 5e-07,
+ "output_cost_per_token": 2e-06,
+ "cache_read_input_token_cost": 4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/nvidia/Llama-3.1-Nemotron-70B-Instruct": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 6e-07,
+ "output_cost_per_token": 6e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/nvidia/Llama-3.3-Nemotron-Super-49B-v1.5": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 1e-07,
+ "output_cost_per_token": 4e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/nvidia/NVIDIA-Nemotron-Nano-9B-v2": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 4e-08,
+ "output_cost_per_token": 1.6e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/openai/gpt-oss-120b": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 5e-08,
+ "output_cost_per_token": 4.5e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/openai/gpt-oss-20b": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 4e-08,
+ "output_cost_per_token": 1.5e-07,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepinfra/zai-org/GLM-4.5": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 4e-07,
+ "output_cost_per_token": 1.6e-06,
+ "litellm_provider": "deepinfra",
+ "mode": "chat",
+ "supports_tool_choice": true
+ },
+ "deepseek/deepseek-chat": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 7e-08,
+ "input_cost_per_token": 2.7e-07,
+ "input_cost_per_token_cache_hit": 7e-08,
+ "litellm_provider": "deepseek",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.1e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_tool_choice": true
+ },
+ "deepseek/deepseek-coder": {
+ "input_cost_per_token": 1.4e-07,
+ "input_cost_per_token_cache_hit": 1.4e-08,
+ "litellm_provider": "deepseek",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_tool_choice": true
+ },
+ "deepseek/deepseek-r1": {
+ "input_cost_per_token": 5.5e-07,
+ "input_cost_per_token_cache_hit": 1.4e-07,
+ "litellm_provider": "deepseek",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.19e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "deepseek/deepseek-reasoner": {
+ "input_cost_per_token": 5.5e-07,
+ "input_cost_per_token_cache_hit": 1.4e-07,
+ "litellm_provider": "deepseek",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.19e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "deepseek/deepseek-v3": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 7e-08,
+ "input_cost_per_token": 2.7e-07,
+ "input_cost_per_token_cache_hit": 7e-08,
+ "litellm_provider": "deepseek",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.1e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_tool_choice": true
+ },
+ "deepseek.v3-v1:0": {
+ "input_cost_per_token": 5.8e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 163840,
+ "max_output_tokens": 81920,
+ "max_tokens": 163840,
+ "mode": "chat",
+ "output_cost_per_token": 1.68e-06,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "dolphin": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "nlp_cloud",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "completion",
+ "output_cost_per_token": 5e-07
+ },
+ "doubao-embedding": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "volcengine",
+ "max_input_tokens": 4096,
+ "max_tokens": 4096,
+ "metadata": {
+ "notes": "Volcengine Doubao embedding model - standard version with 2560 dimensions"
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 2560
+ },
+ "doubao-embedding-large": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "volcengine",
+ "max_input_tokens": 4096,
+ "max_tokens": 4096,
+ "metadata": {
+ "notes": "Volcengine Doubao embedding model - large version with 2048 dimensions"
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 2048
+ },
+ "doubao-embedding-large-text-240915": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "volcengine",
+ "max_input_tokens": 4096,
+ "max_tokens": 4096,
+ "metadata": {
+ "notes": "Volcengine Doubao embedding model - text-240915 version with 4096 dimensions"
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 4096
+ },
+ "doubao-embedding-large-text-250515": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "volcengine",
+ "max_input_tokens": 4096,
+ "max_tokens": 4096,
+ "metadata": {
+ "notes": "Volcengine Doubao embedding model - text-250515 version with 2048 dimensions"
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 2048
+ },
+ "doubao-embedding-text-240715": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "volcengine",
+ "max_input_tokens": 4096,
+ "max_tokens": 4096,
+ "metadata": {
+ "notes": "Volcengine Doubao embedding model - text-240715 version with 2560 dimensions"
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 2560
+ },
+ "elevenlabs/scribe_v1": {
+ "input_cost_per_second": 6.11e-05,
+ "litellm_provider": "elevenlabs",
+ "metadata": {
+ "calculation": "$0.22/hour = $0.00366/minute = $0.0000611 per second (enterprise pricing)",
+ "notes": "ElevenLabs Scribe v1 - state-of-the-art speech recognition model with 99 language support",
+ "original_pricing_per_hour": 0.22
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://elevenlabs.io/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "elevenlabs/scribe_v1_experimental": {
+ "input_cost_per_second": 6.11e-05,
+ "litellm_provider": "elevenlabs",
+ "metadata": {
+ "calculation": "$0.22/hour = $0.00366/minute = $0.0000611 per second (enterprise pricing)",
+ "notes": "ElevenLabs Scribe v1 experimental - enhanced version of the main Scribe model",
+ "original_pricing_per_hour": 0.22
+ },
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0,
+ "source": "https://elevenlabs.io/pricing",
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "embed-english-light-v2.0": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "embed-english-light-v3.0": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "embed-english-v2.0": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "embed-english-v3.0": {
+ "input_cost_per_image": 0.0001,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "metadata": {
+ "notes": "'supports_image_input' is a deprecated field. Use 'supports_embedding_image_input' instead."
+ },
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "supports_embedding_image_input": true,
+ "supports_image_input": true
+ },
+ "embed-multilingual-v2.0": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 768,
+ "max_tokens": 768,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "embed-multilingual-v3.0": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "supports_embedding_image_input": true
+ },
+ "eu.amazon.nova-lite-v1:0": {
+ "input_cost_per_token": 7.8e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 3.12e-07,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "eu.amazon.nova-micro-v1:0": {
+ "input_cost_per_token": 4.6e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 1.84e-07,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true
+ },
+ "eu.amazon.nova-pro-v1:0": {
+ "input_cost_per_token": 1.05e-06,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 4.2e-06,
+ "source": "https://aws.amazon.com/bedrock/pricing/",
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "eu.anthropic.claude-3-5-haiku-20241022-v1:0": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "eu.anthropic.claude-3-5-sonnet-20240620-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "eu.anthropic.claude-3-5-sonnet-20241022-v2:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "eu.anthropic.claude-3-7-sonnet-20250219-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "eu.anthropic.claude-3-haiku-20240307-v1:0": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "eu.anthropic.claude-3-opus-20240229-v1:0": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "eu.anthropic.claude-3-sonnet-20240229-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "eu.anthropic.claude-opus-4-1-20250805-v1:0": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "eu.anthropic.claude-opus-4-20250514-v1:0": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "eu.anthropic.claude-sonnet-4-20250514-v1:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "eu.anthropic.claude-sonnet-4-5-20250929-v1:0": {
+ "cache_creation_input_token_cost": 4.125e-06,
+ "cache_read_input_token_cost": 3.3e-07,
+ "input_cost_per_token": 3.3e-06,
+ "input_cost_per_token_above_200k_tokens": 6.6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.475e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.65e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 346
+ },
+ "eu.meta.llama3-2-1b-instruct-v1:0": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.3e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "eu.meta.llama3-2-3b-instruct-v1:0": {
+ "input_cost_per_token": 1.9e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.9e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "eu.mistral.pixtral-large-2502-v1:0": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "featherless_ai/featherless-ai/Qwerky-72B": {
+ "litellm_provider": "featherless_ai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 4096,
+ "max_tokens": 32768,
+ "mode": "chat"
+ },
+ "featherless_ai/featherless-ai/Qwerky-QwQ-32B": {
+ "litellm_provider": "featherless_ai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 4096,
+ "max_tokens": 32768,
+ "mode": "chat"
+ },
+ "fireworks-ai-4.1b-to-16b": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "fireworks_ai",
+ "output_cost_per_token": 2e-07
+ },
+ "fireworks-ai-56b-to-176b": {
+ "input_cost_per_token": 1.2e-06,
+ "litellm_provider": "fireworks_ai",
+ "output_cost_per_token": 1.2e-06
+ },
+ "fireworks-ai-above-16b": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "fireworks_ai",
+ "output_cost_per_token": 9e-07
+ },
+ "fireworks-ai-default": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "fireworks_ai",
+ "output_cost_per_token": 0.0
+ },
+ "fireworks-ai-embedding-150m-to-350m": {
+ "input_cost_per_token": 1.6e-08,
+ "litellm_provider": "fireworks_ai-embedding-models",
+ "output_cost_per_token": 0.0
+ },
+ "fireworks-ai-embedding-up-to-150m": {
+ "input_cost_per_token": 8e-09,
+ "litellm_provider": "fireworks_ai-embedding-models",
+ "output_cost_per_token": 0.0
+ },
+ "fireworks-ai-moe-up-to-56b": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "fireworks_ai",
+ "output_cost_per_token": 5e-07
+ },
+ "fireworks-ai-up-to-4b": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "fireworks_ai",
+ "output_cost_per_token": 2e-07
+ },
+ "fireworks_ai/WhereIsAI/UAE-Large-V1": {
+ "input_cost_per_token": 1.6e-08,
+ "litellm_provider": "fireworks_ai-embedding-models",
+ "max_input_tokens": 512,
+ "max_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "source": "https://fireworks.ai/pricing"
+ },
+ "fireworks_ai/accounts/fireworks/models/deepseek-coder-v2-instruct": {
+ "input_cost_per_token": 1.2e-06,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/deepseek-r1": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 20480,
+ "max_tokens": 20480,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "source": "https://fireworks.ai/pricing",
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/deepseek-r1-0528": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 160000,
+ "max_output_tokens": 160000,
+ "max_tokens": 160000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "source": "https://fireworks.ai/pricing",
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/deepseek-r1-basic": {
+ "input_cost_per_token": 5.5e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 20480,
+ "max_tokens": 20480,
+ "mode": "chat",
+ "output_cost_per_token": 2.19e-06,
+ "source": "https://fireworks.ai/pricing",
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/deepseek-v3": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/deepseek-v3-0324": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "max_tokens": 163840,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "source": "https://fireworks.ai/models/fireworks/deepseek-v3-0324",
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/deepseek-v3p1": {
+ "input_cost_per_token": 5.6e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.68e-06,
+ "source": "https://fireworks.ai/pricing",
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/firefunction-v2": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/glm-4p5": {
+ "input_cost_per_token": 5.5e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 96000,
+ "max_tokens": 96000,
+ "mode": "chat",
+ "output_cost_per_token": 2.19e-06,
+ "source": "https://fireworks.ai/models/fireworks/glm-4p5",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/glm-4p5-air": {
+ "input_cost_per_token": 2.2e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 96000,
+ "max_tokens": 96000,
+ "mode": "chat",
+ "output_cost_per_token": 8.8e-07,
+ "source": "https://artificialanalysis.ai/models/glm-4-5-air",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/gpt-oss-120b": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/gpt-oss-20b": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/kimi-k2-instruct": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-06,
+ "source": "https://fireworks.ai/models/fireworks/kimi-k2-instruct",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/llama-v3p1-405b-instruct": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/llama-v3p1-8b-instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/llama-v3p2-11b-vision-instruct": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "fireworks_ai/accounts/fireworks/models/llama-v3p2-1b-instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/llama-v3p2-3b-instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/llama-v3p2-90b-vision-instruct": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_response_schema": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "fireworks_ai/accounts/fireworks/models/llama4-maverick-instruct-basic": {
+ "input_cost_per_token": 2.2e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 8.8e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/llama4-scout-instruct-basic": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/mixtral-8x22b-instruct-hf": {
+ "input_cost_per_token": 1.2e-06,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "fireworks_ai/accounts/fireworks/models/qwen2-72b-instruct": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/qwen2p5-coder-32b-instruct": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/accounts/fireworks/models/yi-large": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "fireworks_ai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "source": "https://fireworks.ai/pricing",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "fireworks_ai/nomic-ai/nomic-embed-text-v1": {
+ "input_cost_per_token": 8e-09,
+ "litellm_provider": "fireworks_ai-embedding-models",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "source": "https://fireworks.ai/pricing"
+ },
+ "fireworks_ai/nomic-ai/nomic-embed-text-v1.5": {
+ "input_cost_per_token": 8e-09,
+ "litellm_provider": "fireworks_ai-embedding-models",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "source": "https://fireworks.ai/pricing"
+ },
+ "fireworks_ai/thenlper/gte-base": {
+ "input_cost_per_token": 8e-09,
+ "litellm_provider": "fireworks_ai-embedding-models",
+ "max_input_tokens": 512,
+ "max_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "source": "https://fireworks.ai/pricing"
+ },
+ "fireworks_ai/thenlper/gte-large": {
+ "input_cost_per_token": 1.6e-08,
+ "litellm_provider": "fireworks_ai-embedding-models",
+ "max_input_tokens": 512,
+ "max_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "source": "https://fireworks.ai/pricing"
+ },
+ "friendliai/meta-llama-3.1-70b-instruct": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "friendliai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "friendliai/meta-llama-3.1-8b-instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "friendliai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "ft:babbage-002": {
+ "input_cost_per_token": 4e-07,
+ "input_cost_per_token_batches": 2e-07,
+ "litellm_provider": "text-completion-openai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 4096,
+ "max_tokens": 16384,
+ "mode": "completion",
+ "output_cost_per_token": 4e-07,
+ "output_cost_per_token_batches": 2e-07
+ },
+ "ft:davinci-002": {
+ "input_cost_per_token": 2e-06,
+ "input_cost_per_token_batches": 1e-06,
+ "litellm_provider": "text-completion-openai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 4096,
+ "max_tokens": 16384,
+ "mode": "completion",
+ "output_cost_per_token": 2e-06,
+ "output_cost_per_token_batches": 1e-06
+ },
+ "ft:gpt-3.5-turbo": {
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_batches": 1.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "output_cost_per_token_batches": 3e-06,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "ft:gpt-3.5-turbo-0125": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "ft:gpt-3.5-turbo-0613": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "ft:gpt-3.5-turbo-1106": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "ft:gpt-4-0613": {
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "source": "OpenAI needs to add pricing for this ft model, will be updated when added by OpenAI. Defaulting to base model pricing",
+ "supports_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "ft:gpt-4o-2024-08-06": {
+ "input_cost_per_token": 3.75e-06,
+ "input_cost_per_token_batches": 1.875e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "output_cost_per_token_batches": 7.5e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "ft:gpt-4o-2024-11-20": {
+ "cache_creation_input_token_cost": 1.875e-06,
+ "input_cost_per_token": 3.75e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "ft:gpt-4o-mini-2024-07-18": {
+ "cache_read_input_token_cost": 1.5e-07,
+ "input_cost_per_token": 3e-07,
+ "input_cost_per_token_batches": 1.5e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "output_cost_per_token_batches": 6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini-1.0-pro": {
+ "input_cost_per_character": 1.25e-07,
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "input_cost_per_video_per_second": 0.002,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 32760,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_character": 3.75e-07,
+ "output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#google_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"gemini-1.0-pro-001": {
- "max_tokens": 8192,
+ "deprecation_date": "2025-04-09",
+ "input_cost_per_character": 1.25e-07,
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "input_cost_per_video_per_second": 0.002,
+ "litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 32760,
"max_output_tokens": 8192,
- "input_cost_per_image": 0.0025,
- "input_cost_per_video_per_second": 0.002,
- "input_cost_per_token": 5e-07,
- "input_cost_per_character": 1.25e-07,
- "output_cost_per_token": 1.5e-06,
- "output_cost_per_character": 3.75e-07,
- "litellm_provider": "vertex_ai-language-models",
+ "max_tokens": 8192,
"mode": "chat",
- "supports_function_calling": true,
+ "output_cost_per_character": 3.75e-07,
+ "output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "deprecation_date": "2025-04-09",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
- },
- "gemini-1.0-ultra": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 2048,
- "input_cost_per_image": 0.0025,
- "input_cost_per_video_per_second": 0.002,
- "input_cost_per_token": 5e-07,
- "input_cost_per_character": 1.25e-07,
- "output_cost_per_token": 1.5e-06,
- "output_cost_per_character": 3.75e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
"supports_function_calling": true,
- "source": "As of Jun, 2024. There is no available doc on vertex ai pricing gemini-1.0-ultra-001. Using gemini-1.0-pro pricing. Got max_tokens info here: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
- },
- "gemini-1.0-ultra-001": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 2048,
- "input_cost_per_image": 0.0025,
- "input_cost_per_video_per_second": 0.002,
- "input_cost_per_token": 5e-07,
- "input_cost_per_character": 1.25e-07,
- "output_cost_per_token": 1.5e-06,
- "output_cost_per_character": 3.75e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "As of Jun, 2024. There is no available doc on vertex ai pricing gemini-1.0-ultra-001. Using gemini-1.0-pro pricing. Got max_tokens info here: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
},
"gemini-1.0-pro-002": {
- "max_tokens": 8192,
+ "deprecation_date": "2025-04-09",
+ "input_cost_per_character": 1.25e-07,
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "input_cost_per_video_per_second": 0.002,
+ "litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 32760,
"max_output_tokens": 8192,
- "input_cost_per_image": 0.0025,
- "input_cost_per_video_per_second": 0.002,
- "input_cost_per_token": 5e-07,
- "input_cost_per_character": 1.25e-07,
- "output_cost_per_token": 1.5e-06,
+ "max_tokens": 8192,
+ "mode": "chat",
"output_cost_per_character": 3.75e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "deprecation_date": "2025-04-09",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-pro": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0.00032875,
- "input_cost_per_audio_per_second": 3.125e-05,
- "input_cost_per_video_per_second": 0.00032875,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_character": 3.125e-07,
- "input_cost_per_image_above_128k_tokens": 0.0006575,
- "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
- "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
- "input_cost_per_token_above_128k_tokens": 2.5e-06,
- "input_cost_per_character_above_128k_tokens": 6.25e-07,
- "output_cost_per_token": 5e-06,
- "output_cost_per_character": 1.25e-06,
- "output_cost_per_token_above_128k_tokens": 1e-05,
- "output_cost_per_character_above_128k_tokens": 2.5e-06,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-pro-002": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0.00032875,
- "input_cost_per_audio_per_second": 3.125e-05,
- "input_cost_per_video_per_second": 0.00032875,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_character": 3.125e-07,
- "input_cost_per_image_above_128k_tokens": 0.0006575,
- "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
- "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
- "input_cost_per_token_above_128k_tokens": 2.5e-06,
- "input_cost_per_character_above_128k_tokens": 6.25e-07,
- "output_cost_per_token": 5e-06,
- "output_cost_per_character": 1.25e-06,
- "output_cost_per_token_above_128k_tokens": 1e-05,
- "output_cost_per_character_above_128k_tokens": 2.5e-06,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_vision": true,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-1.5-pro",
- "deprecation_date": "2025-09-24",
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-pro-001": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0.00032875,
- "input_cost_per_audio_per_second": 3.125e-05,
- "input_cost_per_video_per_second": 0.00032875,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_character": 3.125e-07,
- "input_cost_per_image_above_128k_tokens": 0.0006575,
- "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
- "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
- "input_cost_per_token_above_128k_tokens": 2.5e-06,
- "input_cost_per_character_above_128k_tokens": 6.25e-07,
- "output_cost_per_token": 5e-06,
- "output_cost_per_character": 1.25e-06,
- "output_cost_per_token_above_128k_tokens": 1e-05,
- "output_cost_per_character_above_128k_tokens": 2.5e-06,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_vision": true,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "deprecation_date": "2025-05-24",
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-pro-preview-0514": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0.00032875,
- "input_cost_per_audio_per_second": 3.125e-05,
- "input_cost_per_video_per_second": 0.00032875,
- "input_cost_per_token": 7.8125e-08,
- "input_cost_per_character": 3.125e-07,
- "input_cost_per_image_above_128k_tokens": 0.0006575,
- "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
- "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
- "input_cost_per_token_above_128k_tokens": 1.5625e-07,
- "input_cost_per_character_above_128k_tokens": 6.25e-07,
- "output_cost_per_token": 3.125e-07,
- "output_cost_per_character": 1.25e-06,
- "output_cost_per_token_above_128k_tokens": 6.25e-07,
- "output_cost_per_character_above_128k_tokens": 2.5e-06,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-pro-preview-0215": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0.00032875,
- "input_cost_per_audio_per_second": 3.125e-05,
- "input_cost_per_video_per_second": 0.00032875,
- "input_cost_per_token": 7.8125e-08,
- "input_cost_per_character": 3.125e-07,
- "input_cost_per_image_above_128k_tokens": 0.0006575,
- "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
- "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
- "input_cost_per_token_above_128k_tokens": 1.5625e-07,
- "input_cost_per_character_above_128k_tokens": 6.25e-07,
- "output_cost_per_token": 3.125e-07,
- "output_cost_per_character": 1.25e-06,
- "output_cost_per_token_above_128k_tokens": 6.25e-07,
- "output_cost_per_character_above_128k_tokens": 2.5e-06,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-pro-preview-0409": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0.00032875,
- "input_cost_per_audio_per_second": 3.125e-05,
- "input_cost_per_video_per_second": 0.00032875,
- "input_cost_per_token": 7.8125e-08,
- "input_cost_per_character": 3.125e-07,
- "input_cost_per_image_above_128k_tokens": 0.0006575,
- "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
- "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
- "input_cost_per_token_above_128k_tokens": 1.5625e-07,
- "input_cost_per_character_above_128k_tokens": 6.25e-07,
- "output_cost_per_token": 3.125e-07,
- "output_cost_per_character": 1.25e-06,
- "output_cost_per_token_above_128k_tokens": 6.25e-07,
- "output_cost_per_character_above_128k_tokens": 2.5e-06,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-flash": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 2e-05,
- "input_cost_per_video_per_second": 2e-05,
- "input_cost_per_audio_per_second": 2e-06,
- "input_cost_per_token": 7.5e-08,
- "input_cost_per_character": 1.875e-08,
- "input_cost_per_token_above_128k_tokens": 1e-06,
- "input_cost_per_character_above_128k_tokens": 2.5e-07,
- "input_cost_per_image_above_128k_tokens": 4e-05,
- "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
- "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
- "output_cost_per_token": 3e-07,
- "output_cost_per_character": 7.5e-08,
- "output_cost_per_token_above_128k_tokens": 6e-07,
- "output_cost_per_character_above_128k_tokens": 1.5e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-flash-exp-0827": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 2e-05,
- "input_cost_per_video_per_second": 2e-05,
- "input_cost_per_audio_per_second": 2e-06,
- "input_cost_per_token": 4.688e-09,
- "input_cost_per_character": 1.875e-08,
- "input_cost_per_token_above_128k_tokens": 1e-06,
- "input_cost_per_character_above_128k_tokens": 2.5e-07,
- "input_cost_per_image_above_128k_tokens": 4e-05,
- "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
- "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
- "output_cost_per_token": 4.6875e-09,
- "output_cost_per_character": 1.875e-08,
- "output_cost_per_token_above_128k_tokens": 9.375e-09,
- "output_cost_per_character_above_128k_tokens": 3.75e-08,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-flash-002": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 2e-05,
- "input_cost_per_video_per_second": 2e-05,
- "input_cost_per_audio_per_second": 2e-06,
- "input_cost_per_token": 7.5e-08,
- "input_cost_per_character": 1.875e-08,
- "input_cost_per_token_above_128k_tokens": 1e-06,
- "input_cost_per_character_above_128k_tokens": 2.5e-07,
- "input_cost_per_image_above_128k_tokens": 4e-05,
- "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
- "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
- "output_cost_per_token": 3e-07,
- "output_cost_per_character": 7.5e-08,
- "output_cost_per_token_above_128k_tokens": 6e-07,
- "output_cost_per_character_above_128k_tokens": 1.5e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-1.5-flash",
- "deprecation_date": "2025-09-24",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-flash-001": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 2e-05,
- "input_cost_per_video_per_second": 2e-05,
- "input_cost_per_audio_per_second": 2e-06,
- "input_cost_per_token": 7.5e-08,
- "input_cost_per_character": 1.875e-08,
- "input_cost_per_token_above_128k_tokens": 1e-06,
- "input_cost_per_character_above_128k_tokens": 2.5e-07,
- "input_cost_per_image_above_128k_tokens": 4e-05,
- "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
- "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
- "output_cost_per_token": 3e-07,
- "output_cost_per_character": 7.5e-08,
- "output_cost_per_token_above_128k_tokens": 6e-07,
- "output_cost_per_character_above_128k_tokens": 1.5e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "deprecation_date": "2025-05-24",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
- },
- "gemini-1.5-flash-preview-0514": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 2e-05,
- "input_cost_per_video_per_second": 2e-05,
- "input_cost_per_audio_per_second": 2e-06,
- "input_cost_per_token": 7.5e-08,
- "input_cost_per_character": 1.875e-08,
- "input_cost_per_token_above_128k_tokens": 1e-06,
- "input_cost_per_character_above_128k_tokens": 2.5e-07,
- "input_cost_per_image_above_128k_tokens": 4e-05,
- "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
- "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
- "output_cost_per_token": 4.6875e-09,
- "output_cost_per_character": 1.875e-08,
- "output_cost_per_token_above_128k_tokens": 9.375e-09,
- "output_cost_per_character_above_128k_tokens": 3.75e-08,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
- },
- "gemini-pro-experimental": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0,
- "output_cost_per_token": 0,
- "input_cost_per_character": 0,
- "output_cost_per_character": 0,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_tool_choice": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/gemini-experimental",
- "supports_parallel_function_calling": true
- },
- "gemini-flash-experimental": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0,
- "output_cost_per_token": 0,
- "input_cost_per_character": 0,
- "output_cost_per_character": 0,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_tool_choice": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/gemini-experimental",
- "supports_parallel_function_calling": true
- },
- "gemini-pro-vision": {
- "max_tokens": 2048,
- "max_input_tokens": 16384,
- "max_output_tokens": 2048,
- "max_images_per_prompt": 16,
- "max_videos_per_prompt": 1,
- "max_video_length": 2,
- "input_cost_per_token": 5e-07,
"output_cost_per_token": 1.5e-06,
- "input_cost_per_image": 0.0025,
- "litellm_provider": "vertex_ai-vision-models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
},
"gemini-1.0-pro-vision": {
- "max_tokens": 2048,
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "vertex_ai-vision-models",
+ "max_images_per_prompt": 16,
"max_input_tokens": 16384,
"max_output_tokens": 2048,
- "max_images_per_prompt": 16,
- "max_videos_per_prompt": 1,
+ "max_tokens": 2048,
"max_video_length": 2,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "input_cost_per_image": 0.0025,
- "litellm_provider": "vertex_ai-vision-models",
+ "max_videos_per_prompt": 1,
"mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
+ "output_cost_per_token": 1.5e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
"supports_tool_choice": true,
- "supports_parallel_function_calling": true
+ "supports_vision": true
},
"gemini-1.0-pro-vision-001": {
- "max_tokens": 2048,
+ "deprecation_date": "2025-04-09",
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "vertex_ai-vision-models",
+ "max_images_per_prompt": 16,
"max_input_tokens": 16384,
"max_output_tokens": 2048,
- "max_images_per_prompt": 16,
- "max_videos_per_prompt": 1,
+ "max_tokens": 2048,
"max_video_length": 2,
- "input_cost_per_token": 5e-07,
+ "max_videos_per_prompt": 1,
+ "mode": "chat",
"output_cost_per_token": 1.5e-06,
- "input_cost_per_image": 0.0025,
- "litellm_provider": "vertex_ai-vision-models",
- "mode": "chat",
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_function_calling": true,
- "supports_vision": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "deprecation_date": "2025-04-09",
+ "supports_parallel_function_calling": true,
"supports_tool_choice": true,
- "supports_parallel_function_calling": true
+ "supports_vision": true
},
- "medlm-medium": {
- "max_tokens": 8192,
- "max_input_tokens": 32768,
- "max_output_tokens": 8192,
- "input_cost_per_character": 5e-07,
- "output_cost_per_character": 1e-06,
+ "gemini-1.0-ultra": {
+ "input_cost_per_character": 1.25e-07,
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "input_cost_per_video_per_second": 0.002,
"litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
- },
- "medlm-large": {
- "max_tokens": 1024,
"max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_character": 5e-06,
- "output_cost_per_character": 1.5e-05,
- "litellm_provider": "vertex_ai-language-models",
+ "max_output_tokens": 2048,
+ "max_tokens": 8192,
"mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "output_cost_per_character": 3.75e-07,
+ "output_cost_per_token": 1.5e-06,
+ "source": "As of Jun, 2024. There is no available doc on vertex ai pricing gemini-1.0-ultra-001. Using gemini-1.0-pro pricing. Got max_tokens info here: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
"supports_tool_choice": true
},
- "gemini-2.5-pro-exp-03-25": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
+ "gemini-1.0-ultra-001": {
+ "input_cost_per_character": 1.25e-07,
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "input_cost_per_video_per_second": 0.002,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 2048,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_character": 3.75e-07,
+ "output_cost_per_token": 1.5e-06,
+ "source": "As of Jun, 2024. There is no available doc on vertex ai pricing gemini-1.0-ultra-001. Using gemini-1.0-pro pricing. Got max_tokens info here: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "gemini-1.5-flash": {
+ "input_cost_per_audio_per_second": 2e-06,
+ "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
+ "input_cost_per_character": 1.875e-08,
+ "input_cost_per_character_above_128k_tokens": 2.5e-07,
+ "input_cost_per_image": 2e-05,
+ "input_cost_per_image_above_128k_tokens": 4e-05,
+ "input_cost_per_token": 7.5e-08,
+ "input_cost_per_token_above_128k_tokens": 1e-06,
+ "input_cost_per_video_per_second": 2e-05,
+ "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
+ "litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
"max_pdf_size_mb": 30,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_audio_input": true,
- "supports_video_input": true,
- "supports_pdf_input": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini-2.0-pro-exp-02-05": {
"max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 7.5e-08,
+ "output_cost_per_character_above_128k_tokens": 1.5e-07,
+ "output_cost_per_token": 3e-07,
+ "output_cost_per_token_above_128k_tokens": 6e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini-1.5-flash-001": {
+ "deprecation_date": "2025-05-24",
+ "input_cost_per_audio_per_second": 2e-06,
+ "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
+ "input_cost_per_character": 1.875e-08,
+ "input_cost_per_character_above_128k_tokens": 2.5e-07,
+ "input_cost_per_image": 2e-05,
+ "input_cost_per_image_above_128k_tokens": 4e-05,
+ "input_cost_per_token": 7.5e-08,
+ "input_cost_per_token_above_128k_tokens": 1e-06,
+ "input_cost_per_video_per_second": 2e-05,
+ "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 7.5e-08,
+ "output_cost_per_character_above_128k_tokens": 1.5e-07,
+ "output_cost_per_token": 3e-07,
+ "output_cost_per_token_above_128k_tokens": 6e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini-1.5-flash-002": {
+ "deprecation_date": "2025-09-24",
+ "input_cost_per_audio_per_second": 2e-06,
+ "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
+ "input_cost_per_character": 1.875e-08,
+ "input_cost_per_character_above_128k_tokens": 2.5e-07,
+ "input_cost_per_image": 2e-05,
+ "input_cost_per_image_above_128k_tokens": 4e-05,
+ "input_cost_per_token": 7.5e-08,
+ "input_cost_per_token_above_128k_tokens": 1e-06,
+ "input_cost_per_video_per_second": 2e-05,
+ "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 7.5e-08,
+ "output_cost_per_character_above_128k_tokens": 1.5e-07,
+ "output_cost_per_token": 3e-07,
+ "output_cost_per_token_above_128k_tokens": 6e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-1.5-flash",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini-1.5-flash-exp-0827": {
+ "input_cost_per_audio_per_second": 2e-06,
+ "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
+ "input_cost_per_character": 1.875e-08,
+ "input_cost_per_character_above_128k_tokens": 2.5e-07,
+ "input_cost_per_image": 2e-05,
+ "input_cost_per_image_above_128k_tokens": 4e-05,
+ "input_cost_per_token": 4.688e-09,
+ "input_cost_per_token_above_128k_tokens": 1e-06,
+ "input_cost_per_video_per_second": 2e-05,
+ "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 1.875e-08,
+ "output_cost_per_character_above_128k_tokens": 3.75e-08,
+ "output_cost_per_token": 4.6875e-09,
+ "output_cost_per_token_above_128k_tokens": 9.375e-09,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini-1.5-flash-preview-0514": {
+ "input_cost_per_audio_per_second": 2e-06,
+ "input_cost_per_audio_per_second_above_128k_tokens": 4e-06,
+ "input_cost_per_character": 1.875e-08,
+ "input_cost_per_character_above_128k_tokens": 2.5e-07,
+ "input_cost_per_image": 2e-05,
+ "input_cost_per_image_above_128k_tokens": 4e-05,
+ "input_cost_per_token": 7.5e-08,
+ "input_cost_per_token_above_128k_tokens": 1e-06,
+ "input_cost_per_video_per_second": 2e-05,
+ "input_cost_per_video_per_second_above_128k_tokens": 4e-05,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 1.875e-08,
+ "output_cost_per_character_above_128k_tokens": 3.75e-08,
+ "output_cost_per_token": 4.6875e-09,
+ "output_cost_per_token_above_128k_tokens": 9.375e-09,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini-1.5-pro": {
+ "input_cost_per_audio_per_second": 3.125e-05,
+ "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
+ "input_cost_per_character": 3.125e-07,
+ "input_cost_per_character_above_128k_tokens": 6.25e-07,
+ "input_cost_per_image": 0.00032875,
+ "input_cost_per_image_above_128k_tokens": 0.0006575,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_128k_tokens": 2.5e-06,
+ "input_cost_per_video_per_second": 0.00032875,
+ "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
+ "litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 2097152,
"max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_audio_input": true,
- "supports_video_input": true,
- "supports_pdf_input": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini-2.0-flash-exp": {
"max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 1.5e-07,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 6e-07,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "vertex_ai-language-models",
"mode": "chat",
- "supports_system_messages": true,
+ "output_cost_per_character": 1.25e-06,
+ "output_cost_per_character_above_128k_tokens": 2.5e-06,
+ "output_cost_per_token": 5e-06,
+ "output_cost_per_token_above_128k_tokens": 1e-05,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_function_calling": true,
- "supports_vision": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
"supports_response_schema": true,
- "supports_audio_output": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.75e-08,
- "supports_prompt_caching": true
- },
- "gemini-2.0-flash-001": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1e-06,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
"supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
"supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
- "deprecation_date": "2026-02-05",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.75e-08,
- "supports_prompt_caching": true
+ "supports_vision": true
},
- "gemini-2.0-flash-thinking-exp": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 0,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 0.0,
- "supports_prompt_caching": true
- },
- "gemini-2.0-flash-thinking-exp-01-21": {
- "max_tokens": 65536,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65536,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 0,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": false,
- "supports_vision": true,
- "supports_response_schema": false,
- "supports_audio_output": false,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
- "supports_tool_choice": true,
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 0.0,
- "supports_prompt_caching": true
- },
- "gemini-2.5-pro": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
+ "gemini-1.5-pro-001": {
+ "deprecation_date": "2025-05-24",
+ "input_cost_per_audio_per_second": 3.125e-05,
+ "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
+ "input_cost_per_character": 3.125e-07,
+ "input_cost_per_character_above_128k_tokens": 6.25e-07,
+ "input_cost_per_image": 0.00032875,
+ "input_cost_per_image_above_128k_tokens": 0.0006575,
"input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "input_cost_per_token_above_128k_tokens": 2.5e-06,
+ "input_cost_per_video_per_second": 0.00032875,
+ "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
"litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
"mode": "chat",
- "supports_system_messages": true,
+ "output_cost_per_character": 1.25e-06,
+ "output_cost_per_character_above_128k_tokens": 2.5e-06,
+ "output_cost_per_token": 5e-06,
+ "output_cost_per_token_above_128k_tokens": 1e-05,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_function_calling": true,
- "supports_vision": true,
- "supports_audio_input": true,
- "supports_video_input": true,
- "supports_pdf_input": true,
+ "supports_parallel_function_calling": true,
"supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-pro-exp-03-25": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 0.0,
- "input_cost_per_token_above_200k_tokens": 0.0,
- "output_cost_per_token": 0.0,
- "output_cost_per_token_above_200k_tokens": 0.0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 5,
- "tpm": 250000,
"supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_audio_input": true,
- "supports_video_input": true,
- "supports_pdf_input": true,
- "supports_response_schema": true,
"supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
- "supports_web_search": true,
- "cache_read_input_token_cost": 0.0,
- "supports_prompt_caching": true
+ "supports_vision": true
},
- "gemini/gemini-2.5-pro": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
+ "gemini-1.5-pro-002": {
+ "deprecation_date": "2025-09-24",
+ "input_cost_per_audio_per_second": 3.125e-05,
+ "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
+ "input_cost_per_character": 3.125e-07,
+ "input_cost_per_character_above_128k_tokens": 6.25e-07,
+ "input_cost_per_image": 0.00032875,
+ "input_cost_per_image_above_128k_tokens": 0.0006575,
"input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 2000,
- "tpm": 800000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_audio_input": true,
- "supports_video_input": true,
- "supports_pdf_input": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-flash": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1e-06,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 2.5e-06,
- "output_cost_per_reasoning_token": 2.5e-06,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_reasoning": true,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "supports_url_context": true,
- "tpm": 8000000,
- "rpm": 100000,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 7.5e-08,
- "supports_prompt_caching": true
- },
- "gemini-2.5-flash": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1e-06,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 2.5e-06,
- "output_cost_per_reasoning_token": 2.5e-06,
+ "input_cost_per_token_above_128k_tokens": 2.5e-06,
+ "input_cost_per_video_per_second": 0.00032875,
+ "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
"litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
"mode": "chat",
- "supports_reasoning": true,
- "supports_system_messages": true,
+ "output_cost_per_character": 1.25e-06,
+ "output_cost_per_character_above_128k_tokens": 2.5e-06,
+ "output_cost_per_token": 5e-06,
+ "output_cost_per_token_above_128k_tokens": 1e-05,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-1.5-pro",
"supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supports_parallel_function_calling": true,
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 7.5e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash-live-001": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 3.5e-07,
- "input_cost_per_audio_token": 2.1e-06,
- "input_cost_per_image": 2.1e-06,
- "input_cost_per_video_per_second": 2.1e-06,
- "output_cost_per_token": 1.5e-06,
- "output_cost_per_audio_token": 8.5e-06,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10,
- "tpm": 250000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
"supports_response_schema": true,
- "supports_audio_output": true,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "audio"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2-0-flash-live-001",
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 7.5e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-flash-preview-tts": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1e-06,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "output_cost_per_reasoning_token": 3.5e-06,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10,
- "tpm": 250000,
"supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
"supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text"
- ],
- "supported_output_modalities": [
- "audio"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.75e-08,
- "supports_prompt_caching": true
+ "supports_vision": true
},
- "gemini/gemini-2.5-flash-preview-05-20": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1e-06,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 2.5e-06,
- "output_cost_per_reasoning_token": 2.5e-06,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10,
- "tpm": 250000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 7.5e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-flash-preview-04-17": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1e-06,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "output_cost_per_reasoning_token": 3.5e-06,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10,
- "tpm": 250000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
- "supports_web_search": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 3.75e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-flash-lite-preview-06-17": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 5e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "output_cost_per_reasoning_token": 4e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 15,
- "tpm": 250000,
- "supports_reasoning": true,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-lite",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-flash-lite": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 5e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "output_cost_per_reasoning_token": 4e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 15,
- "tpm": 250000,
- "supports_reasoning": true,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-lite",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
- },
- "gemini-2.5-flash-preview-05-20": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1e-06,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 2.5e-06,
- "output_cost_per_reasoning_token": 2.5e-06,
+ "gemini-1.5-pro-preview-0215": {
+ "input_cost_per_audio_per_second": 3.125e-05,
+ "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
+ "input_cost_per_character": 3.125e-07,
+ "input_cost_per_character_above_128k_tokens": 6.25e-07,
+ "input_cost_per_image": 0.00032875,
+ "input_cost_per_image_above_128k_tokens": 0.0006575,
+ "input_cost_per_token": 7.8125e-08,
+ "input_cost_per_token_above_128k_tokens": 1.5625e-07,
+ "input_cost_per_video_per_second": 0.00032875,
+ "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
"litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
"mode": "chat",
- "supports_reasoning": true,
- "supports_system_messages": true,
+ "output_cost_per_character": 1.25e-06,
+ "output_cost_per_character_above_128k_tokens": 2.5e-06,
+ "output_cost_per_token": 3.125e-07,
+ "output_cost_per_token_above_128k_tokens": 6.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supports_parallel_function_calling": true,
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 7.5e-08,
- "supports_prompt_caching": true
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
},
- "gemini-2.5-flash-preview-04-17": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1e-06,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "output_cost_per_reasoning_token": 3.5e-06,
+ "gemini-1.5-pro-preview-0409": {
+ "input_cost_per_audio_per_second": 3.125e-05,
+ "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
+ "input_cost_per_character": 3.125e-07,
+ "input_cost_per_character_above_128k_tokens": 6.25e-07,
+ "input_cost_per_image": 0.00032875,
+ "input_cost_per_image_above_128k_tokens": 0.0006575,
+ "input_cost_per_token": 7.8125e-08,
+ "input_cost_per_token_above_128k_tokens": 1.5625e-07,
+ "input_cost_per_video_per_second": 0.00032875,
+ "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
"litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
"mode": "chat",
- "supports_reasoning": true,
- "supports_system_messages": true,
+ "output_cost_per_character": 1.25e-06,
+ "output_cost_per_character_above_128k_tokens": 2.5e-06,
+ "output_cost_per_token": 3.125e-07,
+ "output_cost_per_token_above_128k_tokens": 6.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supports_parallel_function_calling": true,
- "supports_web_search": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 3.75e-08,
- "supports_prompt_caching": true
+ "supports_response_schema": true,
+ "supports_tool_choice": true
},
- "gemini-2.5-flash-lite-preview-06-17": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 5e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "output_cost_per_reasoning_token": 4e-07,
+ "gemini-1.5-pro-preview-0514": {
+ "input_cost_per_audio_per_second": 3.125e-05,
+ "input_cost_per_audio_per_second_above_128k_tokens": 6.25e-05,
+ "input_cost_per_character": 3.125e-07,
+ "input_cost_per_character_above_128k_tokens": 6.25e-07,
+ "input_cost_per_image": 0.00032875,
+ "input_cost_per_image_above_128k_tokens": 0.0006575,
+ "input_cost_per_token": 7.8125e-08,
+ "input_cost_per_token_above_128k_tokens": 1.5625e-07,
+ "input_cost_per_video_per_second": 0.00032875,
+ "input_cost_per_video_per_second_above_128k_tokens": 0.0006575,
"litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
"mode": "chat",
- "supports_reasoning": true,
- "supports_system_messages": true,
+ "output_cost_per_character": 1.25e-06,
+ "output_cost_per_character_above_128k_tokens": 2.5e-06,
+ "output_cost_per_token": 3.125e-07,
+ "output_cost_per_token_above_128k_tokens": 6.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
"supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supports_parallel_function_calling": true,
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
- },
- "gemini-2.5-flash-lite": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 5e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "output_cost_per_reasoning_token": 4e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_reasoning": true,
+ "supports_response_schema": true,
"supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_endpoints": [
- "/v1/chat/completions",
- "/v1/completions",
- "/v1/batch"
- ],
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
+ "supports_tool_choice": true
},
"gemini-2.0-flash": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
+ "cache_read_input_token_cost": 2.5e-08,
"input_cost_per_audio_token": 7e-07,
"input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
"litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supports_audio_input": true,
+ "output_cost_per_token": 4e-07,
+ "source": "https://ai.google.dev/pricing#2_0flash",
"supported_modalities": [
"text",
"image",
@@ -7808,33 +9047,120 @@
"text",
"image"
],
- "supports_tool_choice": true,
- "source": "https://ai.google.dev/pricing#2_0flash",
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_web_search": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
"supports_url_context": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.0-flash-001": {
+ "cache_read_input_token_cost": 3.75e-08,
+ "deprecation_date": "2026-02-05",
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.0-flash-exp": {
+ "cache_read_input_token_cost": 3.75e-08,
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
+ "input_cost_per_token": 1.5e-07,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
+ "output_cost_per_token": 6e-07,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
},
"gemini-2.0-flash-lite": {
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 50,
+ "cache_read_input_token_cost": 1.875e-08,
"input_cost_per_audio_token": 7.5e-08,
"input_cost_per_token": 7.5e-08,
- "output_cost_per_token": 3e-07,
"litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 50,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
+ "output_cost_per_token": 3e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
"supported_modalities": [
"text",
"image",
@@ -7844,32 +9170,33 @@
"supported_output_modalities": [
"text"
],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
- "supports_tool_choice": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 1.875e-08,
- "supports_prompt_caching": true
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
},
"gemini-2.0-flash-lite-001": {
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 50,
+ "cache_read_input_token_cost": 1.875e-08,
+ "deprecation_date": "2026-02-25",
"input_cost_per_audio_token": 7.5e-08,
"input_cost_per_token": 7.5e-08,
- "output_cost_per_token": 3e-07,
"litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 50,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
+ "output_cost_per_token": 3e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
"supported_modalities": [
"text",
"image",
@@ -7879,38 +9206,259 @@
"supported_output_modalities": [
"text"
],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
- "supports_tool_choice": true,
- "deprecation_date": "2026-02-25",
+ "supports_audio_output": true,
+ "supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 1.875e-08,
- "supports_prompt_caching": true
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
},
- "gemini-2.5-pro-preview-06-05": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
+ "gemini-2.0-flash-live-preview-04-09": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 3e-06,
+ "input_cost_per_image": 3e-06,
+ "input_cost_per_token": 5e-07,
+ "input_cost_per_video_per_second": 3e-06,
+ "litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
"max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 1.25e-06,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_audio_token": 1.2e-05,
+ "output_cost_per_token": 2e-06,
+ "rpm": 10,
+ "source": "https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#gemini-2-0-flash-live-preview-04-09",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini-2.0-flash-preview-image-generation": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "source": "https://ai.google.dev/pricing#2_0flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.0-flash-thinking-exp": {
+ "cache_read_input_token_cost": 0.0,
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.0-flash-thinking-exp-01-21": {
+ "cache_read_input_token_cost": 0.0,
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65536,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65536,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": false,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.0-pro-exp-02-05": {
+ "cache_read_input_token_cost": 3.125e-07,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
"output_cost_per_token": 1e-05,
"output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_reasoning": true,
- "supports_system_messages": true,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_input": true,
"supports_function_calling": true,
- "supports_vision": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
"supports_response_schema": true,
- "supports_audio_output": false,
+ "supports_system_messages": true,
"supports_tool_choice": true,
+ "supports_video_input": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-flash": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 2.5e-06,
+ "output_cost_per_token": 2.5e-06,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
@@ -7925,37 +9473,486 @@
"supported_output_modalities": [
"text"
],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supports_audio_output": false,
+ "supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_web_search": true,
"supports_pdf_input": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true
},
- "gemini-2.5-pro-preview-05-06": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
+ "gemini-2.5-flash-image-preview": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
"max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.039,
+ "output_cost_per_reasoning_token": 3e-05,
+ "output_cost_per_token": 3e-05,
+ "rpm": 100000,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 8000000
+ },
+ "gemini-2.5-flash-lite": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-flash-lite-preview-09-2025": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 3e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "source": "https://developers.googleblog.com/en/continuing-to-bring-you-our-latest-models-with-an-improved-gemini-2-5-flash-and-flash-lite-release/",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-flash-preview-09-2025": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 2.5e-06,
+ "output_cost_per_token": 2.5e-06,
+ "source": "https://developers.googleblog.com/en/continuing-to-bring-you-our-latest-models-with-an-improved-gemini-2-5-flash-and-flash-lite-release/",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-flash-lite-preview-06-17": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-flash-preview-04-17": {
+ "cache_read_input_token_cost": 3.75e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 3.5e-06,
+ "output_cost_per_token": 6e-07,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-flash-preview-05-20": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 2.5e-06,
+ "output_cost_per_token": 2.5e-06,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-pro": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_input": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_video_input": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-pro-exp-03-25": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_input": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_video_input": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-pro-preview-03-25": {
+ "cache_read_input_token_cost": 3.125e-07,
"input_cost_per_audio_token": 1.25e-06,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
"output_cost_per_token": 1e-05,
"output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_reasoning": true,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
"supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
"supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-pro-preview-05-06": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_audio_token": 1.25e-06,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
@@ -7973,37 +9970,37 @@
"supported_regions": [
"global"
],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supports_audio_output": false,
+ "supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_web_search": true,
"supports_pdf_input": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
},
- "gemini-2.5-pro-preview-03-25": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
+ "gemini-2.5-pro-preview-06-05": {
+ "cache_read_input_token_cost": 3.125e-07,
"input_cost_per_audio_token": 1.25e-06,
"input_cost_per_token": 1.25e-06,
"input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
"output_cost_per_token": 1e-05,
"output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_reasoning": true,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
@@ -8018,718 +10015,678 @@
"supported_output_modalities": [
"text"
],
- "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supports_audio_output": false,
+ "supports_function_calling": true,
"supports_parallel_function_calling": true,
- "supports_web_search": true,
"supports_pdf_input": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini-2.0-flash-preview-image-generation": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supports_audio_input": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "supports_tool_choice": true,
- "source": "https://ai.google.dev/pricing#2_0flash",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
- },
- "gemini-2.5-pro-preview-tts": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_modalities": [
- "text"
- ],
- "supported_output_modalities": [
- "audio"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
- "supports_parallel_function_calling": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-pro-exp-02-05": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 0,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 2,
- "tpm": 1000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_audio_input": true,
- "supports_video_input": true,
- "supports_pdf_input": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
- "supports_web_search": true,
- "cache_read_input_token_cost": 0.0,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash-preview-image-generation": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10000,
- "tpm": 10000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supports_audio_input": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "supports_tool_choice": true,
- "source": "https://ai.google.dev/pricing#2_0flash",
- "supports_web_search": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10000,
- "tpm": 10000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supports_audio_input": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "supports_tool_choice": true,
- "source": "https://ai.google.dev/pricing#2_0flash",
- "supports_web_search": true,
- "supports_url_context": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash-lite": {
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 50,
- "input_cost_per_audio_token": 7.5e-08,
- "input_cost_per_token": 7.5e-08,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "tpm": 4000000,
- "rpm": 4000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.0-flash-lite",
- "supports_web_search": true,
- "cache_read_input_token_cost": 1.875e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash-001": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10000,
- "tpm": 10000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "source": "https://ai.google.dev/pricing#2_0flash",
- "supports_web_search": true,
- "cache_read_input_token_cost": 2.5e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-pro-preview-tts": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10000,
- "tpm": 10000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_modalities": [
- "text"
- ],
- "supported_output_modalities": [
- "audio"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
- "supports_web_search": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-pro-preview-06-05": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10000,
- "tpm": 10000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-pro-preview-05-06": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10000,
- "tpm": 10000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
- "supports_web_search": true,
- "supports_url_context": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.5-pro-preview-03-25": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1.25e-06,
- "input_cost_per_token_above_200k_tokens": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "output_cost_per_token_above_200k_tokens": 1.5e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 10000,
- "tpm": 10000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
- "supports_web_search": true,
- "supports_pdf_input": true,
- "cache_read_input_token_cost": 3.125e-07,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash-exp": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 0,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "tpm": 4000000,
- "rpm": 10,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
- "supports_tool_choice": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 0.0,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash-lite-preview-02-05": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7.5e-08,
- "input_cost_per_token": 7.5e-08,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "rpm": 60000,
- "tpm": 10000000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "supports_tool_choice": true,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash-lite",
- "supports_web_search": true,
- "cache_read_input_token_cost": 1.875e-08,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash-thinking-exp": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65536,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 0,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "tpm": 4000000,
- "rpm": 10,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
- "supports_tool_choice": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 0.0,
- "supports_prompt_caching": true
- },
- "gemini/gemini-2.0-flash-thinking-exp-01-21": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65536,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 0,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "tpm": 4000000,
- "rpm": 10,
- "supported_modalities": [
- "text",
- "image",
- "audio",
- "video"
- ],
- "supported_output_modalities": [
- "text",
- "image"
- ],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
- "supports_tool_choice": true,
- "supports_web_search": true,
- "cache_read_input_token_cost": 0.0,
- "supports_prompt_caching": true
- },
- "gemini/gemma-3-27b-it": {
- "max_tokens": 8192,
- "max_input_tokens": 131072,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 0,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "source": "https://aistudio.google.com",
- "supports_tool_choice": true
- },
- "gemini/learnlm-1.5-pro-experimental": {
- "max_tokens": 8192,
- "max_input_tokens": 32767,
- "max_output_tokens": 8192,
- "input_cost_per_image": 0,
- "input_cost_per_video_per_second": 0,
- "input_cost_per_audio_per_second": 0,
- "input_cost_per_token": 0,
- "input_cost_per_character": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "input_cost_per_character_above_128k_tokens": 0,
- "input_cost_per_image_above_128k_tokens": 0,
- "input_cost_per_video_per_second_above_128k_tokens": 0,
- "input_cost_per_audio_per_second_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_character": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_character_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": false,
- "source": "https://aistudio.google.com",
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-sonnet": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "gemini-2.0-flash-live-preview-04-09": {
- "max_tokens": 65535,
- "max_input_tokens": 1048576,
- "max_output_tokens": 65535,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 5e-07,
- "input_cost_per_audio_token": 3e-06,
- "input_cost_per_image": 3e-06,
- "input_cost_per_video_per_second": 3e-06,
- "output_cost_per_token": 2e-06,
- "output_cost_per_audio_token": 1.2e-05,
- "litellm_provider": "vertex_ai-language-models",
- "mode": "chat",
- "rpm": 10,
- "tpm": 250000,
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
+ "supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
- "supports_audio_output": true,
+ "supports_system_messages": true,
"supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-2.5-pro-preview-tts": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "audio"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gemini-embedding-001": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 3072,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
+ },
+ "gemini-flash-experimental": {
+ "input_cost_per_character": 0,
+ "input_cost_per_token": 0,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_token": 0,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/gemini-experimental",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "gemini-pro": {
+ "input_cost_per_character": 1.25e-07,
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "input_cost_per_video_per_second": 0.002,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 32760,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_character": 3.75e-07,
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "gemini-pro-experimental": {
+ "input_cost_per_character": 0,
+ "input_cost_per_token": 0,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_token": 0,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/gemini-experimental",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "gemini-pro-vision": {
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "vertex_ai-vision-models",
+ "max_images_per_prompt": 16,
+ "max_input_tokens": 16384,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "max_video_length": 2,
+ "max_videos_per_prompt": 1,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini/gemini-1.5-flash": {
+ "input_cost_per_token": 7.5e-08,
+ "input_cost_per_token_above_128k_tokens": 1.5e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "output_cost_per_token_above_128k_tokens": 6e-07,
+ "rpm": 2000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-flash-001": {
+ "cache_creation_input_token_cost": 1e-06,
+ "cache_read_input_token_cost": 1.875e-08,
+ "deprecation_date": "2025-05-24",
+ "input_cost_per_token": 7.5e-08,
+ "input_cost_per_token_above_128k_tokens": 1.5e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "output_cost_per_token_above_128k_tokens": 6e-07,
+ "rpm": 2000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-flash-002": {
+ "cache_creation_input_token_cost": 1e-06,
+ "cache_read_input_token_cost": 1.875e-08,
+ "deprecation_date": "2025-09-24",
+ "input_cost_per_token": 7.5e-08,
+ "input_cost_per_token_above_128k_tokens": 1.5e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "output_cost_per_token_above_128k_tokens": 6e-07,
+ "rpm": 2000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-flash-8b": {
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 4000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-flash-8b-exp-0827": {
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 4000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-flash-8b-exp-0924": {
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 4000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-flash-exp-0827": {
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 2000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-flash-latest": {
+ "input_cost_per_token": 7.5e-08,
+ "input_cost_per_token_above_128k_tokens": 1.5e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "output_cost_per_token_above_128k_tokens": 6e-07,
+ "rpm": 2000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-pro": {
+ "input_cost_per_token": 3.5e-06,
+ "input_cost_per_token_above_128k_tokens": 7e-06,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.05e-05,
+ "output_cost_per_token_above_128k_tokens": 2.1e-05,
+ "rpm": 1000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-pro-001": {
+ "deprecation_date": "2025-05-24",
+ "input_cost_per_token": 3.5e-06,
+ "input_cost_per_token_above_128k_tokens": 7e-06,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.05e-05,
+ "output_cost_per_token_above_128k_tokens": 2.1e-05,
+ "rpm": 1000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-pro-002": {
+ "deprecation_date": "2025-09-24",
+ "input_cost_per_token": 3.5e-06,
+ "input_cost_per_token_above_128k_tokens": 7e-06,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.05e-05,
+ "output_cost_per_token_above_128k_tokens": 2.1e-05,
+ "rpm": 1000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-pro-exp-0801": {
+ "input_cost_per_token": 3.5e-06,
+ "input_cost_per_token_above_128k_tokens": 7e-06,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.05e-05,
+ "output_cost_per_token_above_128k_tokens": 2.1e-05,
+ "rpm": 1000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-pro-exp-0827": {
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 1000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-1.5-pro-latest": {
+ "input_cost_per_token": 3.5e-06,
+ "input_cost_per_token_above_128k_tokens": 7e-06,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.05e-06,
+ "output_cost_per_token_above_128k_tokens": 2.1e-05,
+ "rpm": 1000,
+ "source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-2.0-flash": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "rpm": 10000,
+ "source": "https://ai.google.dev/pricing#2_0flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 10000000
+ },
+ "gemini/gemini-2.0-flash-001": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "rpm": 10000,
+ "source": "https://ai.google.dev/pricing#2_0flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 10000000
+ },
+ "gemini/gemini-2.0-flash-exp": {
+ "cache_read_input_token_cost": 0.0,
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 10,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-2.0-flash-lite": {
+ "cache_read_input_token_cost": 1.875e-08,
+ "input_cost_per_audio_token": 7.5e-08,
+ "input_cost_per_token": 7.5e-08,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 50,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "rpm": 4000,
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.0-flash-lite",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-2.0-flash-lite-preview-02-05": {
+ "cache_read_input_token_cost": 1.875e-08,
+ "input_cost_per_audio_token": 7.5e-08,
+ "input_cost_per_token": 7.5e-08,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "rpm": 60000,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash-lite",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 10000000
+ },
+ "gemini/gemini-2.0-flash-live-001": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 2.1e-06,
+ "input_cost_per_image": 2.1e-06,
+ "input_cost_per_token": 3.5e-07,
+ "input_cost_per_video_per_second": 2.1e-06,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_audio_token": 8.5e-06,
+ "output_cost_per_token": 1.5e-06,
+ "rpm": 10,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2-0-flash-live-001",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions"
@@ -8744,7295 +10701,8386 @@
"text",
"audio"
],
- "source": "https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#gemini-2-0-flash-live-preview-04-09",
- "supports_web_search": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
"supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-2.0-flash-preview-image-generation": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "rpm": 10000,
+ "source": "https://ai.google.dev/pricing#2_0flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 10000000
+ },
+ "gemini/gemini-2.0-flash-thinking-exp": {
+ "cache_read_input_token_cost": 0.0,
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65536,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 10,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-2.0-flash-thinking-exp-01-21": {
+ "cache_read_input_token_cost": 0.0,
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65536,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 10,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-2.0-flash",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-2.0-pro-exp-02-05": {
+ "cache_read_input_token_cost": 0.0,
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "rpm": 2,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supports_audio_input": true,
+ "supports_function_calling": true,
"supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_video_input": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 1000000
+ },
+ "gemini/gemini-2.5-flash": {
"cache_read_input_token_cost": 7.5e-08,
- "supports_prompt_caching": true
- },
- "vertex_ai/claude-3-sonnet@20240229": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "vertex_ai-anthropic_models",
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-5-sonnet": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_pdf_input": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-5-sonnet@20240620": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_pdf_input": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-5-sonnet-v2": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_pdf_input": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-5-sonnet-v2@20241022": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_pdf_input": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-7-sonnet@20250219": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_pdf_input": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "deprecation_date": "2025-06-01",
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-opus-4": {
- "max_tokens": 32000,
- "max_input_tokens": 200000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "vertex_ai/claude-opus-4@20250514": {
- "max_tokens": 32000,
- "max_input_tokens": 200000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "vertex_ai/claude-sonnet-4": {
- "max_tokens": 64000,
- "max_input_tokens": 200000,
- "max_output_tokens": 64000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "vertex_ai/claude-sonnet-4@20250514": {
- "max_tokens": 64000,
- "max_input_tokens": 200000,
- "max_output_tokens": 64000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "vertex_ai/claude-3-haiku": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-haiku@20240307": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-5-haiku": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_pdf_input": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-5-haiku@20241022": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_pdf_input": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-opus": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/claude-3-opus@20240229": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "litellm_provider": "vertex_ai-anthropic_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "vertex_ai/meta/llama3-405b-instruct-maas": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
- "supports_tool_choice": true
- },
- "vertex_ai/meta/llama-4-scout-17b-16e-instruct-maas": {
- "max_tokens": 10000000.0,
- "max_input_tokens": 10000000.0,
- "max_output_tokens": 10000000.0,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
- "supports_tool_choice": true,
- "supports_function_calling": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text",
- "code"
- ]
- },
- "vertex_ai/meta/llama-4-scout-17b-128e-instruct-maas": {
- "max_tokens": 10000000.0,
- "max_input_tokens": 10000000.0,
- "max_output_tokens": 10000000.0,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
- "supports_tool_choice": true,
- "supports_function_calling": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text",
- "code"
- ]
- },
- "vertex_ai/meta/llama-4-maverick-17b-128e-instruct-maas": {
- "max_tokens": 1000000.0,
- "max_input_tokens": 1000000.0,
- "max_output_tokens": 1000000.0,
- "input_cost_per_token": 3.5e-07,
- "output_cost_per_token": 1.15e-06,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
- "supports_tool_choice": true,
- "supports_function_calling": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text",
- "code"
- ]
- },
- "vertex_ai/meta/llama-4-maverick-17b-16e-instruct-maas": {
- "max_tokens": 1000000.0,
- "max_input_tokens": 1000000.0,
- "max_output_tokens": 1000000.0,
- "input_cost_per_token": 3.5e-07,
- "output_cost_per_token": 1.15e-06,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
- "supports_tool_choice": true,
- "supports_function_calling": true,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text",
- "code"
- ]
- },
- "vertex_ai/meta/llama3-70b-instruct-maas": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
- "supports_tool_choice": true
- },
- "vertex_ai/meta/llama3-8b-instruct-maas": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
- "supports_tool_choice": true
- },
- "vertex_ai/meta/llama-3.1-8b-instruct-maas": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_vision": true,
- "source": "https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-3.2-90b-vision-instruct-maas",
- "supports_tool_choice": true,
- "metadata": {
- "notes": "VertexAI states that The Llama 3.1 API service for llama-3.1-70b-instruct-maas and llama-3.1-8b-instruct-maas are in public preview and at no cost."
- }
- },
- "vertex_ai/meta/llama-3.1-70b-instruct-maas": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_vision": true,
- "source": "https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-3.2-90b-vision-instruct-maas",
- "supports_tool_choice": true
- },
- "vertex_ai/meta/llama-3.1-405b-instruct-maas": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 16e-06,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_vision": true,
- "source": "https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-3.2-90b-vision-instruct-maas",
- "supports_tool_choice": true
- },
- "vertex_ai/meta/llama-3.2-90b-vision-instruct-maas": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "vertex_ai-llama_models",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_vision": true,
- "source": "https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-3.2-90b-vision-instruct-maas",
- "supports_tool_choice": true,
- "metadata": {
- "notes": "VertexAI states that The Llama 3.2 API service is at no cost during public preview, and will be priced as per dollar-per-1M-tokens at GA."
- }
- },
- "vertex_ai/mistral-large@latest": {
- "max_tokens": 8191,
- "max_input_tokens": 128000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/mistral-large@2411-001": {
- "max_tokens": 8191,
- "max_input_tokens": 128000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/mistral-large-2411": {
- "max_tokens": 8191,
- "max_input_tokens": 128000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/mistral-large@2407": {
- "max_tokens": 8191,
- "max_input_tokens": 128000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/mistral-nemo@latest": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/mistral-small-2503@001": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "vertex_ai-mistral_models",
- "supports_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "vertex_ai/mistral-small-2503": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "vertex_ai/jamba-1.5-mini@001": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "vertex_ai-ai21_models",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "vertex_ai/jamba-1.5-large@001": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "vertex_ai-ai21_models",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "vertex_ai/jamba-1.5": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "vertex_ai-ai21_models",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "vertex_ai/jamba-1.5-mini": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "vertex_ai-ai21_models",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "vertex_ai/jamba-1.5-large": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "vertex_ai-ai21_models",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "vertex_ai/mistral-nemo@2407": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/codestral@latest": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/codestral@2405": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/codestral-2501": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "vertex_ai-mistral_models",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "vertex_ai/imagegeneration@006": {
- "output_cost_per_image": 0.02,
- "litellm_provider": "vertex_ai-image-models",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "vertex_ai/imagen-4.0-generate-preview-06-06": {
- "output_cost_per_image": 0.04,
- "litellm_provider": "vertex_ai-image-models",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "vertex_ai/imagen-4.0-ultra-generate-preview-06-06": {
- "output_cost_per_image": 0.06,
- "litellm_provider": "vertex_ai-image-models",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "vertex_ai/imagen-4.0-fast-generate-preview-06-06": {
- "output_cost_per_image": 0.02,
- "litellm_provider": "vertex_ai-image-models",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "vertex_ai/imagen-3.0-generate-002": {
- "output_cost_per_image": 0.04,
- "litellm_provider": "vertex_ai-image-models",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "vertex_ai/imagen-3.0-generate-001": {
- "output_cost_per_image": 0.04,
- "litellm_provider": "vertex_ai-image-models",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "vertex_ai/imagen-3.0-fast-generate-001": {
- "output_cost_per_image": 0.02,
- "litellm_provider": "vertex_ai-image-models",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "text-embedding-004": {
- "max_tokens": 2048,
- "max_input_tokens": 2048,
- "output_vector_size": 768,
- "input_cost_per_character": 2.5e-08,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
- },
- "gemini-embedding-001": {
- "max_tokens": 2048,
- "max_input_tokens": 2048,
- "output_vector_size": 3072,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
- },
- "text-embedding-005": {
- "max_tokens": 2048,
- "max_input_tokens": 2048,
- "output_vector_size": 768,
- "input_cost_per_character": 2.5e-08,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
- },
- "text-multilingual-embedding-002": {
- "max_tokens": 2048,
- "max_input_tokens": 2048,
- "output_vector_size": 768,
- "input_cost_per_character": 2.5e-08,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
- },
- "multimodalembedding": {
- "max_tokens": 2048,
- "max_input_tokens": 2048,
- "output_vector_size": 768,
- "input_cost_per_character": 2e-07,
- "input_cost_per_image": 0.0001,
- "input_cost_per_video_per_second": 0.0005,
- "input_cost_per_video_per_second_above_8s_interval": 0.001,
- "input_cost_per_video_per_second_above_15s_interval": 0.002,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
+ "output_cost_per_reasoning_token": 2.5e-06,
+ "output_cost_per_token": 2.5e-06,
+ "rpm": 100000,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supported_endpoints": [
- "/v1/embeddings"
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
],
"supported_modalities": [
"text",
"image",
+ "audio",
"video"
],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 8000000
},
- "multimodalembedding@001": {
- "max_tokens": 2048,
- "max_input_tokens": 2048,
- "output_vector_size": 768,
- "input_cost_per_character": 2e-07,
- "input_cost_per_image": 0.0001,
- "input_cost_per_video_per_second": 0.0005,
- "input_cost_per_video_per_second_above_8s_interval": 0.001,
- "input_cost_per_video_per_second_above_15s_interval": 0.002,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
+ "gemini/gemini-2.5-flash-image-preview": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.039,
+ "output_cost_per_reasoning_token": 3e-05,
+ "output_cost_per_token": 3e-05,
+ "rpm": 100000,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
"supported_endpoints": [
- "/v1/embeddings"
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
],
"supported_modalities": [
"text",
"image",
+ "audio",
"video"
],
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
+ "supported_output_modalities": [
+ "text",
+ "image"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 8000000
},
- "text-embedding-large-exp-03-07": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "output_vector_size": 3072,
- "input_cost_per_character": 2.5e-08,
+ "gemini/gemini-2.5-flash-lite": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 5e-07,
"input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
- },
- "textembedding-gecko": {
- "max_tokens": 3072,
- "max_input_tokens": 3072,
- "output_vector_size": 768,
- "input_cost_per_character": 2.5e-08,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "textembedding-gecko-multilingual": {
- "max_tokens": 3072,
- "max_input_tokens": 3072,
- "output_vector_size": 768,
- "input_cost_per_character": 2.5e-08,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "textembedding-gecko-multilingual@001": {
- "max_tokens": 3072,
- "max_input_tokens": 3072,
- "output_vector_size": 768,
- "input_cost_per_character": 2.5e-08,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "textembedding-gecko@001": {
- "max_tokens": 3072,
- "max_input_tokens": 3072,
- "output_vector_size": 768,
- "input_cost_per_character": 2.5e-08,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "textembedding-gecko@003": {
- "max_tokens": 3072,
- "max_input_tokens": 3072,
- "output_vector_size": 768,
- "input_cost_per_character": 2.5e-08,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "text-embedding-preview-0409": {
- "max_tokens": 3072,
- "max_input_tokens": 3072,
- "output_vector_size": 768,
- "input_cost_per_token": 6.25e-09,
- "input_cost_per_token_batch_requests": 5e-09,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "text-multilingual-embedding-preview-0409": {
- "max_tokens": 3072,
- "max_input_tokens": 3072,
- "output_vector_size": 768,
- "input_cost_per_token": 6.25e-09,
- "output_cost_per_token": 0,
- "litellm_provider": "vertex_ai-embedding-models",
- "mode": "embedding",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "palm/chat-bison": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "litellm_provider": "palm",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "palm/chat-bison-001": {
- "max_tokens": 4096,
- "max_input_tokens": 8192,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "litellm_provider": "palm",
- "mode": "chat",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "palm/text-bison": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "litellm_provider": "palm",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "palm/text-bison-001": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "litellm_provider": "palm",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "palm/text-bison-safety-off": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "litellm_provider": "palm",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "palm/text-bison-safety-recitation-off": {
- "max_tokens": 1024,
- "max_input_tokens": 8192,
- "max_output_tokens": 1024,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 1.25e-07,
- "litellm_provider": "palm",
- "mode": "completion",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
- },
- "gemini/gemini-1.5-flash-002": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
+ "litellm_provider": "gemini",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
"max_pdf_size_mb": 30,
- "cache_read_input_token_cost": 1.875e-08,
- "cache_creation_input_token_cost": 1e-06,
- "input_cost_per_token": 7.5e-08,
- "input_cost_per_token_above_128k_tokens": 1.5e-07,
- "output_cost_per_token": 3e-07,
- "output_cost_per_token_above_128k_tokens": 6e-07,
- "litellm_provider": "gemini",
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"mode": "chat",
- "supports_system_messages": true,
+ "output_cost_per_reasoning_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "rpm": 15,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-lite",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
"supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
"supports_prompt_caching": true,
- "tpm": 4000000,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-2.5-flash-lite-preview-09-2025": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 3e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "rpm": 15,
+ "source": "https://developers.googleblog.com/en/continuing-to-bring-you-our-latest-models-with-an-improved-gemini-2-5-flash-and-flash-lite-release/",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-2.5-flash-preview-09-2025": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 2.5e-06,
+ "output_cost_per_token": 2.5e-06,
+ "rpm": 15,
+ "source": "https://developers.googleblog.com/en/continuing-to-bring-you-our-latest-models-with-an-improved-gemini-2-5-flash-and-flash-lite-release/",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-flash-latest": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 2.5e-06,
+ "output_cost_per_token": 2.5e-06,
+ "rpm": 15,
+ "source": "https://developers.googleblog.com/en/continuing-to-bring-you-our-latest-models-with-an-improved-gemini-2-5-flash-and-flash-lite-release/",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-flash-lite-latest": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 3e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "rpm": 15,
+ "source": "https://developers.googleblog.com/en/continuing-to-bring-you-our-latest-models-with-an-improved-gemini-2-5-flash-and-flash-lite-release/",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-2.5-flash-lite-preview-06-17": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 4e-07,
+ "output_cost_per_token": 4e-07,
+ "rpm": 15,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-lite",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-2.5-flash-preview-04-17": {
+ "cache_read_input_token_cost": 3.75e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 3.5e-06,
+ "output_cost_per_token": 6e-07,
+ "rpm": 10,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-2.5-flash-preview-05-20": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 2.5e-06,
+ "output_cost_per_token": 2.5e-06,
+ "rpm": 10,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-2.5-flash-preview-tts": {
+ "cache_read_input_token_cost": 3.75e-08,
+ "input_cost_per_audio_token": 1e-06,
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 3.5e-06,
+ "output_cost_per_token": 6e-07,
+ "rpm": 10,
+ "source": "https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-preview",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "audio"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
+ },
+ "gemini/gemini-2.5-pro": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
"rpm": 2000,
- "source": "https://ai.google.dev/pricing",
- "deprecation_date": "2025-09-24",
- "supports_tool_choice": true
- },
- "gemini/gemini-1.5-flash-001": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "cache_read_input_token_cost": 1.875e-08,
- "cache_creation_input_token_cost": 1e-06,
- "input_cost_per_token": 7.5e-08,
- "input_cost_per_token_above_128k_tokens": 1.5e-07,
- "output_cost_per_token": 3e-07,
- "output_cost_per_token_above_128k_tokens": 6e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_input": true,
"supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
+ "supports_pdf_input": true,
"supports_prompt_caching": true,
- "tpm": 4000000,
- "rpm": 2000,
- "source": "https://ai.google.dev/pricing",
- "deprecation_date": "2025-05-24",
- "supports_tool_choice": true
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_video_input": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 800000
},
- "gemini/gemini-1.5-flash": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
+ "gemini/gemini-2.5-pro-exp-03-25": {
+ "cache_read_input_token_cost": 0.0,
+ "input_cost_per_token": 0.0,
+ "input_cost_per_token_above_200k_tokens": 0.0,
+ "litellm_provider": "gemini",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 7.5e-08,
- "input_cost_per_token_above_128k_tokens": 1.5e-07,
- "output_cost_per_token": 3e-07,
- "output_cost_per_token_above_128k_tokens": 6e-07,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "tpm": 4000000,
- "rpm": 2000,
- "source": "https://ai.google.dev/pricing",
- "supports_tool_choice": true
- },
- "gemini/gemini-1.5-flash-latest": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
"max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
"max_pdf_size_mb": 30,
- "input_cost_per_token": 7.5e-08,
- "input_cost_per_token_above_128k_tokens": 1.5e-07,
- "output_cost_per_token": 3e-07,
- "output_cost_per_token_above_128k_tokens": 6e-07,
- "litellm_provider": "gemini",
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"mode": "chat",
- "supports_system_messages": true,
+ "output_cost_per_token": 0.0,
+ "output_cost_per_token_above_200k_tokens": 0.0,
+ "rpm": 5,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/completions"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_input": true,
"supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
+ "supports_pdf_input": true,
"supports_prompt_caching": true,
- "tpm": 4000000,
- "rpm": 2000,
- "source": "https://ai.google.dev/pricing",
- "supports_tool_choice": true
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_video_input": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 250000
},
- "gemini/gemini-1.5-flash-8b": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
+ "gemini/gemini-2.5-pro-preview-03-25": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "gemini",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_prompt_caching": true,
- "tpm": 4000000,
- "rpm": 4000,
- "source": "https://ai.google.dev/pricing",
- "supports_tool_choice": true
- },
- "gemini/gemini-1.5-flash-8b-exp-0924": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
"max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
"max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "rpm": 10000,
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 10000000
+ },
+ "gemini/gemini-2.5-pro-preview-05-06": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "gemini",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
"max_pdf_size_mb": 30,
- "input_cost_per_token": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "litellm_provider": "gemini",
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"mode": "chat",
- "supports_system_messages": true,
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "rpm": 10000,
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
"supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
+ "supports_pdf_input": true,
"supports_prompt_caching": true,
- "tpm": 4000000,
- "rpm": 4000,
- "source": "https://ai.google.dev/pricing",
- "supports_tool_choice": true
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 10000000
+ },
+ "gemini/gemini-2.5-pro-preview-06-05": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "rpm": 10000,
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio",
+ "video"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_url_context": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 10000000
+ },
+ "gemini/gemini-2.5-pro-preview-tts": {
+ "cache_read_input_token_cost": 3.125e-07,
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_above_200k_tokens": 2.5e-06,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65535,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 65535,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_above_200k_tokens": 1.5e-05,
+ "rpm": 10000,
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "audio"
+ ],
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "tpm": 10000000
},
"gemini/gemini-exp-1114": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
"input_cost_per_token": 0,
"input_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_token_above_128k_tokens": 0,
"litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "tpm": 4000000,
- "rpm": 1000,
- "source": "https://ai.google.dev/pricing",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"metadata": {
"notes": "Rate limits not documented for gemini-exp-1114. Assuming same as gemini-1.5-pro.",
"supports_tool_choice": true
- }
- },
- "gemini/gemini-exp-1206": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 0,
- "input_cost_per_token_above_128k_tokens": 0,
+ },
+ "mode": "chat",
"output_cost_per_token": 0,
"output_cost_per_token_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "tpm": 4000000,
"rpm": 1000,
"source": "https://ai.google.dev/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 4000000
+ },
+ "gemini/gemini-exp-1206": {
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 2097152,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
"metadata": {
"notes": "Rate limits not documented for gemini-exp-1206. Assuming same as gemini-1.5-pro.",
"supports_tool_choice": true
- }
- },
- "gemini/gemini-1.5-flash-exp-0827": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 0,
- "input_cost_per_token_above_128k_tokens": 0,
+ },
+ "mode": "chat",
"output_cost_per_token": 0,
"output_cost_per_token_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "tpm": 4000000,
- "rpm": 2000,
- "source": "https://ai.google.dev/pricing",
- "supports_tool_choice": true
- },
- "gemini/gemini-1.5-flash-8b-exp-0827": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_token": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "tpm": 4000000,
- "rpm": 4000,
- "source": "https://ai.google.dev/pricing",
- "supports_tool_choice": true
- },
- "gemini/gemini-pro": {
- "max_tokens": 8192,
- "max_input_tokens": 32760,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.5e-07,
- "input_cost_per_token_above_128k_tokens": 7e-07,
- "output_cost_per_token": 1.05e-06,
- "output_cost_per_token_above_128k_tokens": 2.1e-06,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_function_calling": true,
- "rpd": 30000,
- "tpm": 120000,
- "rpm": 360,
- "source": "https://ai.google.dev/gemini-api/docs/models/gemini",
- "supports_tool_choice": true
- },
- "gemini/gemini-1.5-pro": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.5e-06,
- "input_cost_per_token_above_128k_tokens": 7e-06,
- "output_cost_per_token": 1.05e-05,
- "output_cost_per_token_above_128k_tokens": 2.1e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "tpm": 4000000,
- "rpm": 1000,
- "source": "https://ai.google.dev/pricing"
- },
- "gemini/gemini-1.5-pro-002": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.5e-06,
- "input_cost_per_token_above_128k_tokens": 7e-06,
- "output_cost_per_token": 1.05e-05,
- "output_cost_per_token_above_128k_tokens": 2.1e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "supports_prompt_caching": true,
- "tpm": 4000000,
"rpm": 1000,
"source": "https://ai.google.dev/pricing",
- "deprecation_date": "2025-09-24"
- },
- "gemini/gemini-1.5-pro-001": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.5e-06,
- "input_cost_per_token_above_128k_tokens": 7e-06,
- "output_cost_per_token": 1.05e-05,
- "output_cost_per_token_above_128k_tokens": 2.1e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
"supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
"supports_response_schema": true,
- "supports_prompt_caching": true,
- "tpm": 4000000,
- "rpm": 1000,
- "source": "https://ai.google.dev/pricing",
- "deprecation_date": "2025-05-24"
- },
- "gemini/gemini-1.5-pro-exp-0801": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.5e-06,
- "input_cost_per_token_above_128k_tokens": 7e-06,
- "output_cost_per_token": 1.05e-05,
- "output_cost_per_token_above_128k_tokens": 2.1e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
"supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
"supports_tool_choice": true,
- "supports_response_schema": true,
- "tpm": 4000000,
- "rpm": 1000,
- "source": "https://ai.google.dev/pricing"
- },
- "gemini/gemini-1.5-pro-exp-0827": {
- "max_tokens": 8192,
- "max_input_tokens": 2097152,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0,
- "input_cost_per_token_above_128k_tokens": 0,
- "output_cost_per_token": 0,
- "output_cost_per_token_above_128k_tokens": 0,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
"supports_vision": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "tpm": 4000000,
- "rpm": 1000,
- "source": "https://ai.google.dev/pricing"
- },
- "gemini/gemini-1.5-pro-latest": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.5e-06,
- "input_cost_per_token_above_128k_tokens": 7e-06,
- "output_cost_per_token": 1.05e-06,
- "output_cost_per_token_above_128k_tokens": 2.1e-05,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "tpm": 4000000,
- "rpm": 1000,
- "source": "https://ai.google.dev/pricing"
- },
- "gemini/gemini-pro-vision": {
- "max_tokens": 2048,
- "max_input_tokens": 30720,
- "max_output_tokens": 2048,
- "input_cost_per_token": 3.5e-07,
- "input_cost_per_token_above_128k_tokens": 7e-07,
- "output_cost_per_token": 1.05e-06,
- "output_cost_per_token_above_128k_tokens": 2.1e-06,
- "litellm_provider": "gemini",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "rpd": 30000,
- "tpm": 120000,
- "rpm": 360,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
+ "tpm": 4000000
},
"gemini/gemini-gemma-2-27b-it": {
- "max_tokens": 8192,
- "max_output_tokens": 8192,
"input_cost_per_token": 3.5e-07,
- "output_cost_per_token": 1.05e-06,
"litellm_provider": "gemini",
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
"mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
+ "output_cost_per_token": 1.05e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
"gemini/gemini-gemma-2-9b-it": {
- "max_tokens": 8192,
- "max_output_tokens": 8192,
"input_cost_per_token": 3.5e-07,
- "output_cost_per_token": 1.05e-06,
"litellm_provider": "gemini",
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
"mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
+ "output_cost_per_token": 1.05e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
- "supports_tool_choice": true
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "gemini/imagen-4.0-generate-preview-06-06": {
- "output_cost_per_image": 0.04,
+ "gemini/gemini-pro": {
+ "input_cost_per_token": 3.5e-07,
+ "input_cost_per_token_above_128k_tokens": 7e-07,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 32760,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.05e-06,
+ "output_cost_per_token_above_128k_tokens": 2.1e-06,
+ "rpd": 30000,
+ "rpm": 360,
+ "source": "https://ai.google.dev/gemini-api/docs/models/gemini",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "tpm": 120000
+ },
+ "gemini/gemini-pro-vision": {
+ "input_cost_per_token": 3.5e-07,
+ "input_cost_per_token_above_128k_tokens": 7e-07,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 30720,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 1.05e-06,
+ "output_cost_per_token_above_128k_tokens": 2.1e-06,
+ "rpd": 30000,
+ "rpm": 360,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tpm": 120000
+ },
+ "gemini/gemma-3-27b-it": {
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
+ "input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
+ "output_cost_per_token": 0,
+ "output_cost_per_token_above_128k_tokens": 0,
+ "source": "https://aistudio.google.com",
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini/imagen-3.0-fast-generate-001": {
"litellm_provider": "gemini",
"mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "gemini/imagen-4.0-ultra-generate-preview-06-06": {
- "output_cost_per_image": 0.06,
- "litellm_provider": "gemini",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "gemini/imagen-4.0-fast-generate-preview-06-06": {
"output_cost_per_image": 0.02,
- "litellm_provider": "gemini",
- "mode": "image_generation",
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
- },
- "gemini/imagen-3.0-generate-002": {
- "output_cost_per_image": 0.04,
- "litellm_provider": "gemini",
- "mode": "image_generation",
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
},
"gemini/imagen-3.0-generate-001": {
- "output_cost_per_image": 0.04,
"litellm_provider": "gemini",
"mode": "image_generation",
+ "output_cost_per_image": 0.04,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
},
- "gemini/imagen-3.0-fast-generate-001": {
+ "gemini/imagen-3.0-generate-002": {
+ "litellm_provider": "gemini",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.04,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "gemini/imagen-4.0-fast-generate-001": {
+ "litellm_provider": "gemini",
+ "mode": "image_generation",
"output_cost_per_image": 0.02,
- "litellm_provider": "gemini",
- "mode": "image_generation",
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
},
- "command-a-03-2025": {
- "max_tokens": 8000,
- "max_input_tokens": 256000,
- "max_output_tokens": 8000,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "cohere_chat",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "command-r": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "cohere_chat",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "command-r-08-2024": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "cohere_chat",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "command-r7b-12-2024": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 3.75e-08,
- "litellm_provider": "cohere_chat",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://docs.cohere.com/v2/docs/command-r7b",
- "supports_tool_choice": true
- },
- "command-light": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "cohere_chat",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "command-r-plus": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "cohere_chat",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "command-r-plus-08-2024": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "cohere_chat",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "command-nightly": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "cohere",
- "mode": "completion"
- },
- "command": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "cohere",
- "mode": "completion"
- },
- "rerank-v3.5": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "max_query_tokens": 2048,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "rerank"
- },
- "rerank-english-v3.0": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "max_query_tokens": 2048,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "rerank"
- },
- "rerank-multilingual-v3.0": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "max_query_tokens": 2048,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "rerank"
- },
- "rerank-english-v2.0": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "max_query_tokens": 2048,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "rerank"
- },
- "rerank-multilingual-v2.0": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "max_query_tokens": 2048,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "rerank"
- },
- "embed-english-light-v3.0": {
- "max_tokens": 1024,
- "max_input_tokens": 1024,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "embedding"
- },
- "embed-multilingual-v3.0": {
- "max_tokens": 1024,
- "max_input_tokens": 1024,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "supports_embedding_image_input": true,
- "mode": "embedding"
- },
- "embed-english-v2.0": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "embedding"
- },
- "embed-english-light-v2.0": {
- "max_tokens": 1024,
- "max_input_tokens": 1024,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "embedding"
- },
- "embed-multilingual-v2.0": {
- "max_tokens": 768,
- "max_input_tokens": 768,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "embedding"
- },
- "embed-english-v3.0": {
- "max_tokens": 1024,
- "max_input_tokens": 1024,
- "input_cost_per_token": 1e-07,
- "input_cost_per_image": 0.0001,
- "output_cost_per_token": 0.0,
- "litellm_provider": "cohere",
- "mode": "embedding",
- "supports_image_input": true,
- "supports_embedding_image_input": true,
- "metadata": {
- "notes": "'supports_image_input' is a deprecated field. Use 'supports_embedding_image_input' instead."
- }
- },
- "replicate/meta/llama-2-13b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-2-13b-chat": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-2-70b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6.5e-07,
- "output_cost_per_token": 2.75e-06,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-2-70b-chat": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6.5e-07,
- "output_cost_per_token": 2.75e-06,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-2-7b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-2-7b-chat": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-3-70b": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 6.5e-07,
- "output_cost_per_token": 2.75e-06,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-3-70b-instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 6.5e-07,
- "output_cost_per_token": 2.75e-06,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-3-8b": {
- "max_tokens": 8086,
- "max_input_tokens": 8086,
- "max_output_tokens": 8086,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/meta/llama-3-8b-instruct": {
- "max_tokens": 8086,
- "max_input_tokens": 8086,
- "max_output_tokens": 8086,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/mistralai/mistral-7b-v0.1": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/mistralai/mistral-7b-instruct-v0.2": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "replicate/mistralai/mixtral-8x7b-instruct-v0.1": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "replicate",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/deepseek/deepseek-r1-0528": {
- "max_tokens": 8192,
- "max_input_tokens": 65336,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5e-07,
- "input_cost_per_token_cache_hit": 1.4e-07,
- "output_cost_per_token": 2.15e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_reasoning": true,
- "supports_tool_choice": true,
- "supports_prompt_caching": true
- },
- "openrouter/bytedance/ui-tars-1.5-7b":{
- "max_tokens": 2048,
- "max_input_tokens": 131072,
- "max_output_tokens": 2048,
- "input_cost_per_token": 0.1e-06,
- "output_cost_per_token": 0.2e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "source": "https://openrouter.ai/api/v1/models/bytedance/ui-tars-1.5-7b",
- "supports_tool_choice": true
- },
- "openrouter/deepseek/deepseek-r1": {
- "max_tokens": 8192,
- "max_input_tokens": 65336,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5.5e-07,
- "input_cost_per_token_cache_hit": 1.4e-07,
- "output_cost_per_token": 2.19e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_reasoning": true,
- "supports_tool_choice": true,
- "supports_prompt_caching": true
- },
- "openrouter/deepseek/deepseek-chat": {
- "max_tokens": 8192,
- "max_input_tokens": 65536,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.4e-07,
- "output_cost_per_token": 2.8e-07,
- "litellm_provider": "openrouter",
- "supports_prompt_caching": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/deepseek/deepseek-coder": {
- "max_tokens": 8192,
- "max_input_tokens": 66000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.4e-07,
- "output_cost_per_token": 2.8e-07,
- "litellm_provider": "openrouter",
- "supports_prompt_caching": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/microsoft/wizardlm-2-8x22b:nitro": {
- "max_tokens": 65536,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/google/gemini-2.5-pro": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1.25e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supports_tool_choice": true
- },
- "openrouter/google/gemini-pro-1.5": {
- "max_tokens": 8192,
- "max_input_tokens": 1000000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 7.5e-06,
- "input_cost_per_image": 0.00265,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "openrouter/google/gemini-2.0-flash-001": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supports_tool_choice": true
- },
- "openrouter/google/gemini-2.5-flash": {
- "max_tokens": 8192,
- "max_input_tokens": 1048576,
- "max_output_tokens": 8192,
- "max_images_per_prompt": 3000,
- "max_videos_per_prompt": 10,
- "max_video_length": 1,
- "max_audio_length_hours": 8.4,
- "max_audio_per_prompt": 1,
- "max_pdf_size_mb": 30,
- "input_cost_per_audio_token": 7e-07,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 2.5e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_system_messages": true,
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_response_schema": true,
- "supports_audio_output": true,
- "supports_tool_choice": true
- },
- "openrouter/mistralai/mixtral-8x22b-instruct": {
- "max_tokens": 65536,
- "input_cost_per_token": 6.5e-07,
- "output_cost_per_token": 6.5e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/cohere/command-r-plus": {
- "max_tokens": 128000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/databricks/dbrx-instruct": {
- "max_tokens": 32768,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3-haiku": {
- "max_tokens": 200000,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "input_cost_per_image": 0.0004,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3-5-haiku": {
- "max_tokens": 200000,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3-haiku-20240307": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 264,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3-5-haiku-20241022": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "tool_use_system_prompt_tokens": 264,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3.5-sonnet": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3.5-sonnet:beta": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3.7-sonnet": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "input_cost_per_image": 0.0048,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3.7-sonnet:beta": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "input_cost_per_image": 0.0048,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3-sonnet": {
- "max_tokens": 200000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "input_cost_per_image": 0.0048,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-sonnet-4": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "input_cost_per_image": 0.0048,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_reasoning": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_tool_choice": true
- },
- "openrouter/mistralai/mistral-large": {
- "max_tokens": 32000,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/mistralai/mistral-small-3.1-24b-instruct": {
- "max_tokens": 32000,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/mistralai/mistral-small-3.2-24b-instruct": {
- "max_tokens": 32000,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/cognitivecomputations/dolphin-mixtral-8x7b": {
- "max_tokens": 32769,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/google/gemini-pro-vision": {
- "max_tokens": 45875,
- "input_cost_per_token": 1.25e-07,
- "output_cost_per_token": 3.75e-07,
- "input_cost_per_image": 0.0025,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "openrouter/fireworks/firellava-13b": {
- "max_tokens": 4096,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/meta-llama/llama-3-8b-instruct:free": {
- "max_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/meta-llama/llama-3-8b-instruct:extended": {
- "max_tokens": 16384,
- "input_cost_per_token": 2.25e-07,
- "output_cost_per_token": 2.25e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/meta-llama/llama-3-70b-instruct:nitro": {
- "max_tokens": 8192,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/meta-llama/llama-3-70b-instruct": {
- "max_tokens": 8192,
- "input_cost_per_token": 5.9e-07,
- "output_cost_per_token": 7.9e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/openai/o1": {
- "max_tokens": 100000,
- "max_input_tokens": 200000,
- "max_output_tokens": 100000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "cache_read_input_token_cost": 7.5e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_prompt_caching": true,
- "supports_system_messages": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "openrouter/openai/o1-mini": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.2e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_tool_choice": true
- },
- "openrouter/openai/o1-mini-2024-09-12": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.2e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_tool_choice": true
- },
- "openrouter/openai/o1-preview": {
- "max_tokens": 32768,
- "max_input_tokens": 128000,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_tool_choice": true
- },
- "openrouter/openai/o1-preview-2024-09-12": {
- "max_tokens": 32768,
- "max_input_tokens": 128000,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 6e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_tool_choice": true
- },
- "openrouter/openai/o3-mini": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_reasoning": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_tool_choice": true
- },
- "openrouter/openai/o3-mini-high": {
- "max_tokens": 65536,
- "max_input_tokens": 128000,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.1e-06,
- "output_cost_per_token": 4.4e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_reasoning": true,
- "supports_parallel_function_calling": true,
- "supports_vision": false,
- "supports_tool_choice": true
- },
- "openrouter/openai/gpt-4o": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "openrouter/openai/gpt-4o-2024-05-13": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "openrouter/openai/gpt-4-vision-preview": {
- "max_tokens": 130000,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 3e-05,
- "input_cost_per_image": 0.01445,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "openrouter/openai/gpt-3.5-turbo": {
- "max_tokens": 4095,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/openai/gpt-3.5-turbo-16k": {
- "max_tokens": 16383,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 4e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/openai/gpt-4": {
- "max_tokens": 8192,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 6e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-instant-v1": {
- "max_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.63e-06,
- "output_cost_per_token": 5.51e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-2": {
- "max_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.102e-05,
- "output_cost_per_token": 3.268e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/anthropic/claude-3-opus": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 395,
- "supports_tool_choice": true
- },
- "openrouter/google/palm-2-chat-bison": {
- "max_tokens": 25804,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/google/palm-2-codechat-bison": {
- "max_tokens": 20070,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/meta-llama/llama-2-13b-chat": {
- "max_tokens": 4096,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/meta-llama/llama-2-70b-chat": {
- "max_tokens": 4096,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/meta-llama/codellama-34b-instruct": {
- "max_tokens": 8192,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/nousresearch/nous-hermes-llama2-13b": {
- "max_tokens": 4096,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/mancer/weaver": {
- "max_tokens": 8000,
- "input_cost_per_token": 5.625e-06,
- "output_cost_per_token": 5.625e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/gryphe/mythomax-l2-13b": {
- "max_tokens": 8192,
- "input_cost_per_token": 1.875e-06,
- "output_cost_per_token": 1.875e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/jondurbin/airoboros-l2-70b-2.1": {
- "max_tokens": 4096,
- "input_cost_per_token": 1.3875e-05,
- "output_cost_per_token": 1.3875e-05,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/undi95/remm-slerp-l2-13b": {
- "max_tokens": 6144,
- "input_cost_per_token": 1.875e-06,
- "output_cost_per_token": 1.875e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/pygmalionai/mythalion-13b": {
- "max_tokens": 4096,
- "input_cost_per_token": 1.875e-06,
- "output_cost_per_token": 1.875e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/mistralai/mistral-7b-instruct": {
- "max_tokens": 8192,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 1.3e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/mistralai/mistral-7b-instruct:free": {
- "max_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/qwen/qwen-2.5-coder-32b-instruct": {
- "max_tokens": 33792,
- "max_input_tokens": 33792,
- "max_output_tokens": 33792,
- "input_cost_per_token": 1.8e-07,
- "output_cost_per_token": 1.8e-07,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/qwen/qwen-vl-plus": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 2048,
- "input_cost_per_token": 0.21e-06,
- "output_cost_per_token": 0.63e-06,
- "litellm_provider": "openrouter",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/qwen/qwen3-coder": {
- "max_tokens": 1000000,
- "max_input_tokens": 1000000,
- "max_output_tokens": 1000000,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "openrouter",
- "source": "https://openrouter.ai/qwen/qwen3-coder",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "openrouter/switchpoint/router": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 8.5e-07,
- "output_cost_per_token": 3.4e-06,
- "litellm_provider": "openrouter",
- "source": "https://openrouter.ai/switchpoint/router",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "j2-ultra": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "ai21",
- "mode": "completion"
- },
- "jamba-1.5-mini@001": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "jamba-1.5-large@001": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "jamba-1.5": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "jamba-1.5-mini": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "jamba-1.5-large": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "jamba-large-1.6": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "jamba-large-1.7": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "jamba-mini-1.6": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "jamba-mini-1.7": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "ai21",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "j2-mid": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1e-05,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "ai21",
- "mode": "completion"
- },
- "j2-light": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "ai21",
- "mode": "completion"
- },
- "dolphin": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "nlp_cloud",
- "mode": "completion"
- },
- "chatdolphin": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "nlp_cloud",
- "mode": "chat"
- },
- "luminous-base": {
- "max_tokens": 2048,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 3.3e-05,
- "litellm_provider": "aleph_alpha",
- "mode": "completion"
- },
- "luminous-base-control": {
- "max_tokens": 2048,
- "input_cost_per_token": 3.75e-05,
- "output_cost_per_token": 4.125e-05,
- "litellm_provider": "aleph_alpha",
- "mode": "chat"
- },
- "luminous-extended": {
- "max_tokens": 2048,
- "input_cost_per_token": 4.5e-05,
- "output_cost_per_token": 4.95e-05,
- "litellm_provider": "aleph_alpha",
- "mode": "completion"
- },
- "luminous-extended-control": {
- "max_tokens": 2048,
- "input_cost_per_token": 5.625e-05,
- "output_cost_per_token": 6.1875e-05,
- "litellm_provider": "aleph_alpha",
- "mode": "chat"
- },
- "luminous-supreme": {
- "max_tokens": 2048,
- "input_cost_per_token": 0.000175,
- "output_cost_per_token": 0.0001925,
- "litellm_provider": "aleph_alpha",
- "mode": "completion"
- },
- "luminous-supreme-control": {
- "max_tokens": 2048,
- "input_cost_per_token": 0.00021875,
- "output_cost_per_token": 0.000240625,
- "litellm_provider": "aleph_alpha",
- "mode": "chat"
- },
- "ai21.j2-mid-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.25e-05,
- "output_cost_per_token": 1.25e-05,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "ai21.j2-ultra-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.88e-05,
- "output_cost_per_token": 1.88e-05,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "ai21.jamba-instruct-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 70000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_system_messages": true
- },
- "ai21.jamba-1-5-large-v1:0": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "ai21.jamba-1-5-mini-v1:0": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 256000,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "amazon.rerank-v1:0": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "max_output_tokens": 32000,
- "max_query_tokens": 32000,
- "max_document_chunks_per_query": 100,
- "max_tokens_per_document_chunk": 512,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.001,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "rerank"
- },
- "amazon.titan-text-lite-v1": {
- "max_tokens": 4000,
- "max_input_tokens": 42000,
- "max_output_tokens": 4000,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "amazon.titan-text-express-v1": {
- "max_tokens": 8000,
- "max_input_tokens": 42000,
- "max_output_tokens": 8000,
- "input_cost_per_token": 1.3e-06,
- "output_cost_per_token": 1.7e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "amazon.titan-text-premier-v1:0": {
- "max_tokens": 32000,
- "max_input_tokens": 42000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "amazon.titan-embed-text-v1": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "output_vector_size": 1536,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "embedding"
- },
- "amazon.titan-embed-text-v2:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "output_vector_size": 1024,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "embedding"
- },
- "amazon.titan-embed-image-v1": {
- "max_tokens": 128,
- "max_input_tokens": 128,
- "output_vector_size": 1024,
- "input_cost_per_token": 8e-07,
- "input_cost_per_image": 6e-05,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "supports_image_input": true,
- "supports_embedding_image_input": true,
- "mode": "embedding",
- "source": "https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/providers?model=amazon.titan-image-generator-v1",
- "metadata": {
- "notes": "'supports_image_input' is a deprecated field. Use 'supports_embedding_image_input' instead."
- }
- },
- "mistral.mistral-7b-instruct-v0:2": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "mistral.mixtral-8x7b-instruct-v0:1": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 4.5e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "mistral.mistral-large-2402-v1:0": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true
- },
- "mistral.mistral-large-2407-v1:0": {
- "max_tokens": 8191,
- "max_input_tokens": 128000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 9e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true
- },
- "mistral.mistral-small-2402-v1:0": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true
- },
- "eu.mistral.pixtral-large-2502-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "us.mistral.pixtral-large-2502-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 6e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "bedrock/us-west-2/mistral.mixtral-8x7b-instruct-v0:1": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 4.5e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/mistral.mixtral-8x7b-instruct-v0:1": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 4.5e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-west-3/mistral.mixtral-8x7b-instruct-v0:1": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 5.9e-07,
- "output_cost_per_token": 9.1e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/mistral.mistral-7b-instruct-v0:2": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/mistral.mistral-7b-instruct-v0:2": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-west-3/mistral.mistral-7b-instruct-v0:2": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2.6e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/mistral.mistral-large-2402-v1:0": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true
- },
- "bedrock/us-west-2/mistral.mistral-large-2402-v1:0": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true
- },
- "bedrock/eu-west-3/mistral.mistral-large-2402-v1:0": {
- "max_tokens": 8191,
- "max_input_tokens": 32000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.04e-05,
- "output_cost_per_token": 3.12e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true
- },
- "amazon.nova-micro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 128000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 3.5e-08,
- "output_cost_per_token": 1.4e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "us.amazon.nova-micro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 128000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 3.5e-08,
- "output_cost_per_token": 1.4e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "eu.amazon.nova-micro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 128000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 4.6e-08,
- "output_cost_per_token": 1.84e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "amazon.nova-lite-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 6e-08,
- "output_cost_per_token": 2.4e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "us.amazon.nova-lite-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 6e-08,
- "output_cost_per_token": 2.4e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "eu.amazon.nova-lite-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 7.8e-08,
- "output_cost_per_token": 3.12e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "amazon.nova-pro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 3.2e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "us.amazon.nova-pro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 3.2e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "1024-x-1024/50-steps/bedrock/amazon.nova-canvas-v1:0": {
- "max_input_tokens": 2600,
+ "gemini/imagen-4.0-generate-001": {
+ "litellm_provider": "gemini",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.04,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "gemini/imagen-4.0-ultra-generate-001": {
+ "litellm_provider": "gemini",
+ "mode": "image_generation",
"output_cost_per_image": 0.06,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "eu.amazon.nova-pro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 1.05e-06,
- "output_cost_per_token": 4.2e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "source": "https://aws.amazon.com/bedrock/pricing/"
- },
- "apac.amazon.nova-micro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 128000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 3.7e-08,
- "output_cost_per_token": 1.48e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "apac.amazon.nova-lite-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 6.3e-08,
- "output_cost_per_token": 2.52e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "apac.amazon.nova-pro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 8.4e-07,
- "output_cost_per_token": 3.36e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "us.amazon.nova-premier-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 1000000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 2.5e-06,
- "output_cost_per_token": 1.25e-05,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": false,
- "supports_response_schema": true
- },
- "anthropic.claude-3-sonnet-20240229-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "bedrock/invoke/anthropic.claude-3-5-sonnet-20240620-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true,
- "metadata": {
- "notes": "Anthropic via Invoke route does not currently support pdf input."
- }
- },
- "anthropic.claude-3-5-sonnet-20240620-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "anthropic.claude-opus-4-20250514-v1:0": {
- "max_tokens": 32000,
- "max_input_tokens": 200000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "anthropic.claude-sonnet-4-20250514-v1:0": {
- "max_tokens": 64000,
- "max_input_tokens": 200000,
- "max_output_tokens": 64000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "anthropic.claude-3-7-sonnet-20250219-v1:0": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_pdf_input": true,
- "supports_reasoning": true,
- "supports_tool_choice": true
- },
- "anthropic.claude-3-5-sonnet-20241022-v2:0": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "anthropic.claude-3-haiku-20240307-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "anthropic.claude-3-5-haiku-20241022-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 4e-06,
- "cache_creation_input_token_cost": 1e-06,
- "cache_read_input_token_cost": 8e-08,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_prompt_caching": true,
- "supports_tool_choice": true
- },
- "anthropic.claude-3-opus-20240229-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "us.anthropic.claude-3-sonnet-20240229-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "us.anthropic.claude-3-5-sonnet-20240620-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "us.anthropic.claude-3-5-sonnet-20241022-v2:0": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "us.anthropic.claude-3-7-sonnet-20250219-v1:0": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true,
- "supports_reasoning": true
- },
- "us.anthropic.claude-opus-4-20250514-v1:0": {
- "max_tokens": 32000,
- "max_input_tokens": 200000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "us.anthropic.claude-sonnet-4-20250514-v1:0": {
- "max_tokens": 64000,
- "max_input_tokens": 200000,
- "max_output_tokens": 64000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "us.anthropic.claude-3-haiku-20240307-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "us.anthropic.claude-3-5-haiku-20241022-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 4e-06,
- "cache_creation_input_token_cost": 1e-06,
- "cache_read_input_token_cost": 8e-08,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_function_calling": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "us.anthropic.claude-3-opus-20240229-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "eu.anthropic.claude-3-sonnet-20240229-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "eu.anthropic.claude-3-5-sonnet-20240620-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "eu.anthropic.claude-3-5-sonnet-20241022-v2:0": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "eu.anthropic.claude-3-7-sonnet-20250219-v1:0": {
- "supports_computer_use": true,
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true,
- "supports_reasoning": true
- },
- "eu.anthropic.claude-3-haiku-20240307-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "eu.anthropic.claude-opus-4-20250514-v1:0": {
- "max_tokens": 32000,
- "max_input_tokens": 200000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 1.875e-05,
- "cache_read_input_token_cost": 1.5e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "eu.anthropic.claude-sonnet-4-20250514-v1:0": {
- "max_tokens": 64000,
- "max_input_tokens": 200000,
- "max_output_tokens": 64000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "apac.anthropic.claude-3-haiku-20240307-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "apac.anthropic.claude-3-sonnet-20240229-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "apac.anthropic.claude-3-5-sonnet-20240620-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "apac.anthropic.claude-3-5-sonnet-20241022-v2:0": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_assistant_prefill": true,
- "supports_computer_use": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "apac.anthropic.claude-sonnet-4-20250514-v1:0": {
- "max_tokens": 64000,
- "max_input_tokens": 200000,
- "max_output_tokens": 64000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.01,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.01
- },
- "cache_creation_input_token_cost": 3.75e-06,
- "cache_read_input_token_cost": 3e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "tool_use_system_prompt_tokens": 159,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "supports_computer_use": true
- },
- "eu.anthropic.claude-3-5-haiku-20241022-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 1.25e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_assistant_prefill": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true,
- "supports_tool_choice": true
- },
- "eu.anthropic.claude-3-opus-20240229-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-05,
- "output_cost_per_token": 7.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_tool_choice": true
- },
- "anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-east-1/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/1-month-commitment/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0455,
- "output_cost_per_second": 0.0455,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/ap-northeast-1/6-month-commitment/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.02527,
- "output_cost_per_second": 0.02527,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/eu-central-1/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/eu-central-1/1-month-commitment/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0415,
- "output_cost_per_second": 0.0415,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/eu-central-1/6-month-commitment/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.02305,
- "output_cost_per_second": 0.02305,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-east-1/1-month-commitment/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0175,
- "output_cost_per_second": 0.0175,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-east-1/6-month-commitment/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.00972,
- "output_cost_per_second": 0.00972,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-west-2/1-month-commitment/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0175,
- "output_cost_per_second": 0.0175,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-west-2/6-month-commitment/anthropic.claude-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.00972,
- "output_cost_per_second": 0.00972,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/1-month-commitment/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0455,
- "output_cost_per_second": 0.0455,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/6-month-commitment/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.02527,
- "output_cost_per_second": 0.02527,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/1-month-commitment/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0415,
- "output_cost_per_second": 0.0415,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/6-month-commitment/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.02305,
- "output_cost_per_second": 0.02305,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/1-month-commitment/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0175,
- "output_cost_per_second": 0.0175,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/6-month-commitment/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.00972,
- "output_cost_per_second": 0.00972,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/1-month-commitment/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0175,
- "output_cost_per_second": 0.0175,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/6-month-commitment/anthropic.claude-v2": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.00972,
- "output_cost_per_second": 0.00972,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/1-month-commitment/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0455,
- "output_cost_per_second": 0.0455,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/6-month-commitment/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.02527,
- "output_cost_per_second": 0.02527,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-06,
- "output_cost_per_token": 2.4e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/1-month-commitment/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0415,
- "output_cost_per_second": 0.0415,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/6-month-commitment/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.02305,
- "output_cost_per_second": 0.02305,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/1-month-commitment/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0175,
- "output_cost_per_second": 0.0175,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/6-month-commitment/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.00972,
- "output_cost_per_second": 0.00972,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/1-month-commitment/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.0175,
- "output_cost_per_second": 0.0175,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/6-month-commitment/anthropic.claude-v2:1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.00972,
- "output_cost_per_second": 0.00972,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 2.4e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 2.4e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/1-month-commitment/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.011,
- "output_cost_per_second": 0.011,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-east-1/6-month-commitment/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.00611,
- "output_cost_per_second": 0.00611,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/1-month-commitment/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.011,
- "output_cost_per_second": 0.011,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/6-month-commitment/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.00611,
- "output_cost_per_second": 0.00611,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/us-west-2/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 2.4e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2.23e-06,
- "output_cost_per_token": 7.55e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/1-month-commitment/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.01475,
- "output_cost_per_second": 0.01475,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/ap-northeast-1/6-month-commitment/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.008194,
- "output_cost_per_second": 0.008194,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2.48e-06,
- "output_cost_per_token": 8.38e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/1-month-commitment/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.01635,
- "output_cost_per_second": 0.01635,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/eu-central-1/6-month-commitment/anthropic.claude-instant-v1": {
- "max_tokens": 8191,
- "max_input_tokens": 100000,
- "max_output_tokens": 8191,
- "input_cost_per_second": 0.009083,
- "output_cost_per_second": 0.009083,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "cohere.rerank-v3-5:0": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "max_output_tokens": 32000,
- "max_query_tokens": 32000,
- "max_document_chunks_per_query": 100,
- "max_tokens_per_document_chunk": 512,
- "input_cost_per_token": 0.0,
- "input_cost_per_query": 0.002,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "rerank"
- },
- "cohere.command-text-v14": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/*/1-month-commitment/cohere.command-text-v14": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_second": 0.011,
- "output_cost_per_second": 0.011,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/*/6-month-commitment/cohere.command-text-v14": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_second": 0.0066027,
- "output_cost_per_second": 0.0066027,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "cohere.command-light-text-v14": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/*/1-month-commitment/cohere.command-light-text-v14": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_second": 0.001902,
- "output_cost_per_second": 0.001902,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "bedrock/*/6-month-commitment/cohere.command-light-text-v14": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_second": 0.0011416,
- "output_cost_per_second": 0.0011416,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "cohere.command-r-plus-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "cohere.command-r-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "cohere.embed-english-v3": {
- "max_tokens": 512,
- "max_input_tokens": 512,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "embedding",
- "supports_embedding_image_input": true
- },
- "cohere.embed-multilingual-v3": {
- "max_tokens": 512,
- "max_input_tokens": 512,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "embedding",
- "supports_embedding_image_input": true
- },
- "us.deepseek.r1-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.35e-06,
- "output_cost_per_token": 5.4e-06,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_reasoning": true,
- "supports_function_calling": false,
- "supports_tool_choice": false
- },
- "meta.llama3-3-70b-instruct-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7.2e-07,
- "output_cost_per_token": 7.2e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "meta.llama2-13b-chat-v1": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7.5e-07,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "meta.llama2-70b-chat-v1": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.95e-06,
- "output_cost_per_token": 2.56e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-east-1/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-west-1/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/ap-south-1/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.6e-07,
- "output_cost_per_token": 7.2e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/ca-central-1/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.5e-07,
- "output_cost_per_token": 6.9e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/eu-west-1/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.2e-07,
- "output_cost_per_token": 6.5e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/eu-west-2/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.9e-07,
- "output_cost_per_token": 7.8e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/sa-east-1/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.01e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2.65e-06,
- "output_cost_per_token": 3.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-east-1/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2.65e-06,
- "output_cost_per_token": 3.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-west-1/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2.65e-06,
- "output_cost_per_token": 3.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/ap-south-1/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.18e-06,
- "output_cost_per_token": 4.2e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/ca-central-1/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.05e-06,
- "output_cost_per_token": 4.03e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/eu-west-1/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2.86e-06,
- "output_cost_per_token": 3.78e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/eu-west-2/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.45e-06,
- "output_cost_per_token": 4.55e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/sa-east-1/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 4.45e-06,
- "output_cost_per_token": 5.88e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "meta.llama3-1-8b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 2.2e-07,
- "output_cost_per_token": 2.2e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "us.meta.llama3-1-8b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 2.2e-07,
- "output_cost_per_token": 2.2e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "meta.llama3-1-70b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 9.9e-07,
- "output_cost_per_token": 9.9e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "us.meta.llama3-1-70b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 9.9e-07,
- "output_cost_per_token": 9.9e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "meta.llama3-1-405b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5.32e-06,
- "output_cost_per_token": 1.6e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "us.meta.llama3-1-405b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5.32e-06,
- "output_cost_per_token": 1.6e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "meta.llama3-2-1b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "us.meta.llama3-2-1b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "eu.meta.llama3-2-1b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 1.3e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "meta.llama3-2-3b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "us.meta.llama3-2-3b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "eu.meta.llama3-2-3b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.9e-07,
- "output_cost_per_token": 1.9e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "meta.llama3-2-11b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3.5e-07,
- "output_cost_per_token": 3.5e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false,
- "supports_vision": true
- },
- "us.meta.llama3-2-11b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3.5e-07,
- "output_cost_per_token": 3.5e-07,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false,
- "supports_vision": true
- },
- "meta.llama3-2-90b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false,
- "supports_vision": true
- },
- "us.meta.llama3-2-90b-instruct-v1:0": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false,
- "supports_vision": true
- },
- "us.meta.llama3-3-70b-instruct-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7.2e-07,
- "output_cost_per_token": 7.2e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false
- },
- "meta.llama4-maverick-17b-instruct-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.4e-07,
- "input_cost_per_token_batches": 1.2e-07,
- "output_cost_per_token": 9.7e-07,
- "output_cost_per_token_batches": 4.85e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text",
- "code"
- ]
- },
- "us.meta.llama4-maverick-17b-instruct-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.4e-07,
- "input_cost_per_token_batches": 1.2e-07,
- "output_cost_per_token": 9.7e-07,
- "output_cost_per_token_batches": 4.85e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text",
- "code"
- ]
- },
- "meta.llama4-scout-17b-instruct-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.7e-07,
- "input_cost_per_token_batches": 8.5e-08,
- "output_cost_per_token": 6.6e-07,
- "output_cost_per_token_batches": 3.3e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text",
- "code"
- ]
- },
- "us.meta.llama4-scout-17b-instruct-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 128000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.7e-07,
- "input_cost_per_token_batches": 8.5e-08,
- "output_cost_per_token": 6.6e-07,
- "output_cost_per_token_batches": 3.3e-07,
- "litellm_provider": "bedrock_converse",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": false,
- "supported_modalities": [
- "text",
- "image"
- ],
- "supported_output_modalities": [
- "text",
- "code"
- ]
- },
- "512-x-512/50-steps/stability.stable-diffusion-xl-v0": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.018,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "512-x-512/max-steps/stability.stable-diffusion-xl-v0": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.036,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "max-x-max/50-steps/stability.stable-diffusion-xl-v0": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.036,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "max-x-max/max-steps/stability.stable-diffusion-xl-v0": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.072,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "1024-x-1024/50-steps/stability.stable-diffusion-xl-v1": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.04,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "1024-x-1024/max-steps/stability.stable-diffusion-xl-v1": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.08,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "stability.sd3-large-v1:0": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.08,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "stability.sd3-5-large-v1:0": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.08,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "stability.stable-image-core-v1:0": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.04,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "stability.stable-image-core-v1:1": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.04,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "stability.stable-image-ultra-v1:0": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.14,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "stability.stable-image-ultra-v1:1": {
- "max_tokens": 77,
- "max_input_tokens": 77,
- "output_cost_per_image": 0.14,
- "litellm_provider": "bedrock",
- "mode": "image_generation"
- },
- "sagemaker/meta-textgeneration-llama-2-7b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "sagemaker",
- "mode": "completion"
- },
- "sagemaker/meta-textgeneration-llama-2-7b-f": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "sagemaker",
- "mode": "chat"
- },
- "sagemaker/meta-textgeneration-llama-2-13b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "sagemaker",
- "mode": "completion"
- },
- "sagemaker/meta-textgeneration-llama-2-13b-f": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "sagemaker",
- "mode": "chat"
- },
- "sagemaker/meta-textgeneration-llama-2-70b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "sagemaker",
- "mode": "completion"
- },
- "sagemaker/meta-textgeneration-llama-2-70b-b-f": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "sagemaker",
- "mode": "chat"
- },
- "together-ai-up-to-4b": {
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "together_ai",
- "mode": "chat"
- },
- "together-ai-4.1b-8b": {
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "together_ai",
- "mode": "chat"
- },
- "together-ai-8.1b-21b": {
- "max_tokens": 1000,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "together_ai",
- "mode": "chat"
- },
- "together-ai-21.1b-41b": {
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 8e-07,
- "litellm_provider": "together_ai",
- "mode": "chat"
- },
- "together-ai-41.1b-80b": {
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "together_ai",
- "mode": "chat"
- },
- "together-ai-81.1b-110b": {
- "input_cost_per_token": 1.8e-06,
- "output_cost_per_token": 1.8e-06,
- "litellm_provider": "together_ai",
- "mode": "chat"
- },
- "together-ai-embedding-up-to-150m": {
- "input_cost_per_token": 8e-09,
- "output_cost_per_token": 0.0,
- "litellm_provider": "together_ai",
- "mode": "embedding"
- },
- "together-ai-embedding-151m-to-350m": {
- "input_cost_per_token": 1.6e-08,
- "output_cost_per_token": 0.0,
- "litellm_provider": "together_ai",
- "mode": "embedding"
- },
- "together_ai/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo": {
- "input_cost_per_token": 1.8e-07,
- "output_cost_per_token": 1.8e-07,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo": {
- "input_cost_per_token": 8.8e-07,
- "output_cost_per_token": 8.8e-07,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo": {
- "input_cost_per_token": 3.5e-06,
- "output_cost_per_token": 3.5e-06,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo": {
- "input_cost_per_token": 8.8e-07,
- "output_cost_per_token": 8.8e-07,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo-Free": {
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "gemini/learnlm-1.5-pro-experimental": {
+ "input_cost_per_audio_per_second": 0,
+ "input_cost_per_audio_per_second_above_128k_tokens": 0,
+ "input_cost_per_character": 0,
+ "input_cost_per_character_above_128k_tokens": 0,
+ "input_cost_per_image": 0,
+ "input_cost_per_image_above_128k_tokens": 0,
"input_cost_per_token": 0,
+ "input_cost_per_token_above_128k_tokens": 0,
+ "input_cost_per_video_per_second": 0,
+ "input_cost_per_video_per_second_above_128k_tokens": 0,
+ "litellm_provider": "gemini",
+ "max_input_tokens": 32767,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_character": 0,
+ "output_cost_per_character_above_128k_tokens": 0,
"output_cost_per_token": 0,
- "litellm_provider": "together_ai",
+ "output_cost_per_token_above_128k_tokens": 0,
+ "source": "https://aistudio.google.com",
+ "supports_audio_output": false,
"supports_function_calling": true,
- "supports_parallel_function_calling": true,
"supports_response_schema": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1": {
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/mistralai/Mistral-7B-Instruct-v0.1": {
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_response_schema": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/togethercomputer/CodeLlama-34b-Instruct": {
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8": {
- "input_cost_per_token": 2.7e-07,
- "output_cost_per_token": 8.5e-07,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/meta-llama/Llama-4-Scout-17B-16E-Instruct": {
- "input_cost_per_token": 1.8e-07,
- "output_cost_per_token": 5.9e-07,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/meta-llama/Llama-3.2-3B-Instruct-Turbo": {
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/Qwen/Qwen2.5-7B-Instruct-Turbo": {
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/Qwen/Qwen2.5-72B-Instruct-Turbo": {
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/deepseek-ai/DeepSeek-V3": {
- "input_cost_per_token": 1.25e-06,
- "output_cost_per_token": 1.25e-06,
- "max_tokens": 8192,
- "max_input_tokens": 65536,
- "max_output_tokens": 8192,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/deepseek-ai/DeepSeek-R1": {
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 7e-06,
- "max_tokens": 20480,
- "max_input_tokens": 128000,
- "max_output_tokens": 20480,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/mistralai/Mistral-Small-24B-Instruct-2501": {
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "mode": "chat",
- "supports_tool_choice": true
- },
- "together_ai/moonshotai/Kimi-K2-Instruct": {
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "together_ai",
- "supports_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gemini/veo-2.0-generate-001": {
+ "litellm_provider": "gemini",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "video_generation",
+ "output_cost_per_second": 0.35,
+ "source": "https://ai.google.dev/gemini-api/docs/video",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "video"
+ ]
+ },
+ "gemini/veo-3.0-fast-generate-preview": {
+ "litellm_provider": "gemini",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "video_generation",
+ "output_cost_per_second": 0.4,
+ "source": "https://ai.google.dev/gemini-api/docs/video",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "video"
+ ]
+ },
+ "gemini/veo-3.0-generate-preview": {
+ "litellm_provider": "gemini",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "video_generation",
+ "output_cost_per_second": 0.75,
+ "source": "https://ai.google.dev/gemini-api/docs/video",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "video"
+ ]
+ },
+ "global.anthropic.claude-sonnet-4-5-20250929-v1:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 346
+ },
+ "global.anthropic.claude-sonnet-4-20250514-v1:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "gpt-3.5-turbo": {
+ "input_cost_per_token": 0.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 4097,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-3.5-turbo-0125": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 16385,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_function_calling": true,
"supports_parallel_function_calling": true,
- "mode": "chat",
- "source": "https://www.together.ai/models/kimi-k2-instruct"
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
},
- "ollama/codegemma": {
- "max_tokens": 8192,
+ "gpt-3.5-turbo-0301": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 4097,
+ "max_output_tokens": 4096,
+ "max_tokens": 4097,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-3.5-turbo-0613": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 4097,
+ "max_output_tokens": 4096,
+ "max_tokens": 4097,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-3.5-turbo-1106": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 16385,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-3.5-turbo-16k": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 16385,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-3.5-turbo-16k-0613": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 16385,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-3.5-turbo-instruct": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "text-completion-openai",
"max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "completion"
- },
- "ollama/codegeex4": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat",
- "supports_function_calling": false
- },
- "ollama/deepseek-coder-v2-instruct": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat",
- "supports_function_calling": true
- },
- "ollama/deepseek-coder-v2-base": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
"mode": "completion",
- "supports_function_calling": true
+ "output_cost_per_token": 2e-06
},
- "ollama/deepseek-coder-v2-lite-instruct": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat",
- "supports_function_calling": true
- },
- "ollama/deepseek-coder-v2-lite-base": {
- "max_tokens": 8192,
+ "gpt-3.5-turbo-instruct-0914": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "text-completion-openai",
"max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
+ "max_output_tokens": 4097,
+ "max_tokens": 4097,
"mode": "completion",
- "supports_function_calling": true
+ "output_cost_per_token": 2e-06
},
- "ollama/internlm2_5-20b-chat": {
- "max_tokens": 32768,
+ "gpt-4": {
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4-0125-preview": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4-0314": {
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4-0613": {
+ "deprecation_date": "2025-06-06",
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4-1106-preview": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4-1106-vision-preview": {
+ "deprecation_date": "2024-12-06",
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4-32k": {
+ "input_cost_per_token": 6e-05,
+ "litellm_provider": "openai",
"max_input_tokens": 32768,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
"mode": "chat",
- "supports_function_calling": true
+ "output_cost_per_token": 0.00012,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
},
- "ollama/llama2": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat"
- },
- "ollama/llama2:7b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat"
- },
- "ollama/llama2:13b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat"
- },
- "ollama/llama2:70b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat"
- },
- "ollama/llama2-uncensored": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "completion"
- },
- "ollama/llama3": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat"
- },
- "ollama/llama3:8b": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat"
- },
- "ollama/llama3:70b": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat"
- },
- "ollama/llama3.1": {
- "max_tokens": 32768,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat",
- "supports_function_calling": true
- },
- "ollama/mistral-large-instruct-2407": {
- "max_tokens": 65536,
- "max_input_tokens": 65536,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat",
- "supports_function_calling": true
- },
- "ollama/mistral": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "completion",
- "supports_function_calling": true
- },
- "ollama/mistral-7B-Instruct-v0.1": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat",
- "supports_function_calling": true
- },
- "ollama/mistral-7B-Instruct-v0.2": {
- "max_tokens": 32768,
+ "gpt-4-32k-0314": {
+ "input_cost_per_token": 6e-05,
+ "litellm_provider": "openai",
"max_input_tokens": 32768,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.00012,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4-32k-0613": {
+ "input_cost_per_token": 6e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.00012,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4-turbo": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4-turbo-2024-04-09": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4-turbo-preview": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4-vision-preview": {
+ "deprecation_date": "2024-12-06",
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4.1": {
+ "cache_read_input_token_cost": 5e-07,
+ "cache_read_input_token_cost_priority": 8.75e-07,
+ "input_cost_per_token": 2e-06,
+ "input_cost_per_token_batches": 1e-06,
+ "input_cost_per_token_priority": 3.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 1047576,
"max_output_tokens": 32768,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "chat",
- "supports_function_calling": true
- },
- "ollama/mixtral-8x7B-Instruct-v0.1": {
"max_tokens": 32768,
- "max_input_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "output_cost_per_token_batches": 4e-06,
+ "output_cost_per_token_priority": 1.4e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4.1-2025-04-14": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "input_cost_per_token_batches": 1e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 1047576,
"max_output_tokens": 32768,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
+ "max_tokens": 32768,
"mode": "chat",
- "supports_function_calling": true
+ "output_cost_per_token": 8e-06,
+ "output_cost_per_token_batches": 4e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "ollama/mixtral-8x22B-Instruct-v0.1": {
- "max_tokens": 65536,
- "max_input_tokens": 65536,
- "max_output_tokens": 65536,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
+ "gpt-4.1-mini": {
+ "cache_read_input_token_cost": 1e-07,
+ "cache_read_input_token_cost_priority": 1.75e-07,
+ "input_cost_per_token": 4e-07,
+ "input_cost_per_token_batches": 2e-07,
+ "input_cost_per_token_priority": 7e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
"mode": "chat",
- "supports_function_calling": true
+ "output_cost_per_token": 1.6e-06,
+ "output_cost_per_token_batches": 8e-07,
+ "output_cost_per_token_priority": 2.8e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "ollama/codellama": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "completion"
- },
- "ollama/orca-mini": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "completion"
- },
- "ollama/vicuna": {
- "max_tokens": 2048,
- "max_input_tokens": 2048,
- "max_output_tokens": 2048,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "ollama",
- "mode": "completion"
- },
- "deepinfra/lizpreciatior/lzlv_70b_fp16_hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "deepinfra",
+ "gpt-4.1-mini-2025-04-14": {
+ "cache_read_input_token_cost": 1e-07,
+ "input_cost_per_token": 4e-07,
+ "input_cost_per_token_batches": 2e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
"mode": "chat",
+ "output_cost_per_token": 1.6e-06,
+ "output_cost_per_token_batches": 8e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4.1-nano": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_priority": 5e-08,
+ "input_cost_per_token": 1e-07,
+ "input_cost_per_token_batches": 5e-08,
+ "input_cost_per_token_priority": 2e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "output_cost_per_token_batches": 2e-07,
+ "output_cost_per_token_priority": 8e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4.1-nano-2025-04-14": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "input_cost_per_token_batches": 5e-08,
+ "litellm_provider": "openai",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "output_cost_per_token_batches": 2e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4.5-preview": {
+ "cache_read_input_token_cost": 3.75e-05,
+ "input_cost_per_token": 7.5e-05,
+ "input_cost_per_token_batches": 3.75e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 0.00015,
+ "output_cost_per_token_batches": 7.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4.5-preview-2025-02-27": {
+ "cache_read_input_token_cost": 3.75e-05,
+ "deprecation_date": "2025-07-14",
+ "input_cost_per_token": 7.5e-05,
+ "input_cost_per_token_batches": 3.75e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 0.00015,
+ "output_cost_per_token_batches": 7.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "cache_read_input_token_cost_priority": 2.125e-06,
+ "input_cost_per_token": 2.5e-06,
+ "input_cost_per_token_batches": 1.25e-06,
+ "input_cost_per_token_priority": 4.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_batches": 5e-06,
+ "output_cost_per_token_priority": 1.7e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o-2024-05-13": {
+ "input_cost_per_token": 5e-06,
+ "input_cost_per_token_batches": 2.5e-06,
+ "input_cost_per_token_priority": 8.75e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "output_cost_per_token_batches": 7.5e-06,
+ "output_cost_per_token_priority": 2.625e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o-2024-08-06": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
+ "input_cost_per_token_batches": 1.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_batches": 5e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o-2024-11-20": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
+ "input_cost_per_token_batches": 1.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_batches": 5e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o-audio-preview": {
+ "input_cost_per_audio_token": 0.0001,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_audio_token": 0.0002,
+ "output_cost_per_token": 1e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "deepinfra/Gryphe/MythoMax-L2-13b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.2e-07,
- "output_cost_per_token": 2.2e-07,
- "litellm_provider": "deepinfra",
+ "gpt-4o-audio-preview-2024-10-01": {
+ "input_cost_per_audio_token": 0.0001,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_audio_token": 0.0002,
+ "output_cost_per_token": 1e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "deepinfra/mistralai/Mistral-7B-Instruct-v0.1": {
- "max_tokens": 8191,
- "max_input_tokens": 32768,
- "max_output_tokens": 8191,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 1.3e-07,
- "litellm_provider": "deepinfra",
+ "gpt-4o-audio-preview-2024-12-17": {
+ "input_cost_per_audio_token": 4e-05,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 1e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "deepinfra/meta-llama/Llama-2-70b-chat-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "deepinfra",
+ "gpt-4o-audio-preview-2025-06-03": {
+ "input_cost_per_audio_token": 4e-05,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 1e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "deepinfra/cognitivecomputations/dolphin-2.6-mixtral-8x7b": {
- "max_tokens": 8191,
- "max_input_tokens": 32768,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2.7e-07,
- "output_cost_per_token": 2.7e-07,
- "litellm_provider": "deepinfra",
+ "gpt-4o-mini": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "cache_read_input_token_cost_priority": 1.25e-07,
+ "input_cost_per_token": 1.5e-07,
+ "input_cost_per_token_batches": 7.5e-08,
+ "input_cost_per_token_priority": 2.5e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/codellama/CodeLlama-34b-Instruct-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-07,
"output_cost_per_token": 6e-07,
- "litellm_provider": "deepinfra",
+ "output_cost_per_token_batches": 3e-07,
+ "output_cost_per_token_priority": 1e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o-mini-2024-07-18": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_token": 1.5e-07,
+ "input_cost_per_token_batches": 7.5e-08,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
"mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "output_cost_per_token_batches": 3e-07,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.03,
+ "search_context_size_low": 0.025,
+ "search_context_size_medium": 0.0275
+ },
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o-mini-audio-preview": {
+ "input_cost_per_audio_token": 1e-05,
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_audio_token": 2e-05,
+ "output_cost_per_token": 6e-07,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "deepinfra/deepinfra/mixtral": {
+ "gpt-4o-mini-audio-preview-2024-12-17": {
+ "input_cost_per_audio_token": 1e-05,
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_audio_token": 2e-05,
+ "output_cost_per_token": 6e-07,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4o-mini-realtime-preview": {
+ "cache_creation_input_audio_token_cost": 3e-07,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_audio_token": 1e-05,
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
"max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 2e-05,
+ "output_cost_per_token": 2.4e-06,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4o-mini-realtime-preview-2024-12-17": {
+ "cache_creation_input_audio_token_cost": 3e-07,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_audio_token": 1e-05,
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 2e-05,
+ "output_cost_per_token": 2.4e-06,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4o-mini-search-preview": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_token": 1.5e-07,
+ "input_cost_per_token_batches": 7.5e-08,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "output_cost_per_token_batches": 3e-07,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.03,
+ "search_context_size_low": 0.025,
+ "search_context_size_medium": 0.0275
+ },
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gpt-4o-mini-search-preview-2025-03-11": {
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_token": 1.5e-07,
+ "input_cost_per_token_batches": 7.5e-08,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "output_cost_per_token_batches": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o-mini-transcribe": {
+ "input_cost_per_audio_token": 3e-06,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16000,
+ "max_output_tokens": 2000,
+ "mode": "audio_transcription",
+ "output_cost_per_token": 5e-06,
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "gpt-4o-mini-tts": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "openai",
+ "mode": "audio_speech",
+ "output_cost_per_audio_token": 1.2e-05,
+ "output_cost_per_second": 0.00025,
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/audio/speech"
+ ],
+ "supported_modalities": [
+ "text",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "audio"
+ ]
+ },
+ "gpt-4o-realtime-preview": {
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_audio_token": 4e-05,
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 2e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4o-realtime-preview-2024-10-01": {
+ "cache_creation_input_audio_token_cost": 2e-05,
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_audio_token": 0.0001,
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 0.0002,
+ "output_cost_per_token": 2e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4o-realtime-preview-2024-12-17": {
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_audio_token": 4e-05,
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 2e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4o-realtime-preview-2025-06-03": {
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_audio_token": 4e-05,
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_audio_token": 8e-05,
+ "output_cost_per_token": 2e-05,
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "gpt-4o-search-preview": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
+ "input_cost_per_token_batches": 1.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_batches": 5e-06,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.05,
+ "search_context_size_low": 0.03,
+ "search_context_size_medium": 0.035
+ },
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gpt-4o-search-preview-2025-03-11": {
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
+ "input_cost_per_token_batches": 1.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_batches": 5e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-4o-transcribe": {
+ "input_cost_per_audio_token": 6e-06,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 16000,
+ "max_output_tokens": 2000,
+ "mode": "audio_transcription",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "gpt-5": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_flex": 6.25e-08,
+ "cache_read_input_token_cost_priority": 2.5e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_flex": 6.25e-07,
+ "input_cost_per_token_priority": 2.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_flex": 5e-06,
+ "output_cost_per_token_priority": 2e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-5-pro": {
+ "input_cost_per_token": 1.5e-05,
+ "input_cost_per_token_batches": 7.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 400000,
+ "max_output_tokens": 272000,
+ "max_tokens": 272000,
+ "mode": "responses",
+ "output_cost_per_token": 1.2e-04,
+ "output_cost_per_token_batches": 6e-05,
+ "supported_endpoints": [
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": false,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gpt-5-pro-2025-10-06": {
+ "input_cost_per_token": 1.5e-05,
+ "input_cost_per_token_batches": 7.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 400000,
+ "max_output_tokens": 272000,
+ "max_tokens": 272000,
+ "mode": "responses",
+ "output_cost_per_token": 1.2e-04,
+ "output_cost_per_token_batches": 6e-05,
+ "supported_endpoints": [
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": false,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "gpt-5-2025-08-07": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "cache_read_input_token_cost_flex": 6.25e-08,
+ "cache_read_input_token_cost_priority": 2.5e-07,
+ "input_cost_per_token": 1.25e-06,
+ "input_cost_per_token_flex": 6.25e-07,
+ "input_cost_per_token_priority": 2.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "output_cost_per_token_flex": 5e-06,
+ "output_cost_per_token_priority": 2e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_service_tier": true,
+ "supports_vision": true
+ },
+ "gpt-5-chat": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": false,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": false,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "gpt-5-chat-latest": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": false,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": false,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "gpt-5-codex": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "responses",
+ "output_cost_per_token": 1e-05,
+ "supported_endpoints": [
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": false,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": false,
+ "supports_response_schema": true,
+ "supports_system_messages": false,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-5-mini": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_flex": 1.25e-08,
+ "cache_read_input_token_cost_priority": 4.5e-08,
+ "input_cost_per_token": 2.5e-07,
+ "input_cost_per_token_flex": 1.25e-07,
+ "input_cost_per_token_priority": 4.5e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "output_cost_per_token_flex": 1e-06,
+ "output_cost_per_token_priority": 3.6e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-5-mini-2025-08-07": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_flex": 1.25e-08,
+ "cache_read_input_token_cost_priority": 4.5e-08,
+ "input_cost_per_token": 2.5e-07,
+ "input_cost_per_token_flex": 1.25e-07,
+ "input_cost_per_token_priority": 4.5e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "output_cost_per_token_flex": 1e-06,
+ "output_cost_per_token_priority": 3.6e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-5-nano": {
+ "cache_read_input_token_cost": 5e-09,
+ "cache_read_input_token_cost_flex": 2.5e-09,
+ "input_cost_per_token": 5e-08,
+ "input_cost_per_token_flex": 2.5e-08,
+ "input_cost_per_token_priority": 2.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "output_cost_per_token_flex": 2e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-5-nano-2025-08-07": {
+ "cache_read_input_token_cost": 5e-09,
+ "cache_read_input_token_cost_flex": 2.5e-09,
+ "input_cost_per_token": 5e-08,
+ "input_cost_per_token_flex": 2.5e-08,
+ "litellm_provider": "openai",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "output_cost_per_token_flex": 2e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "gpt-image-1": {
+ "input_cost_per_pixel": 4.0054321e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "gpt-image-1-mini": {
+ "cache_read_input_image_token_cost": 2.5e-07,
+ "cache_read_input_token_cost": 2e-07,
+ "input_cost_per_image_token": 2.5e-06,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "openai",
+ "mode": "chat",
+ "output_cost_per_image_token": 8e-06,
+ "supported_endpoints": [
+ "/v1/images/generations",
+ "/v1/images/edits"
+ ]
+ },
+ "gpt-realtime": {
+ "cache_creation_input_audio_token_cost": 4e-07,
+ "cache_read_input_token_cost": 4e-07,
+ "input_cost_per_audio_token": 3.2e-05,
+ "input_cost_per_image": 5e-06,
+ "input_cost_per_token": 4e-06,
+ "litellm_provider": "openai",
"max_input_tokens": 32000,
"max_output_tokens": 4096,
- "input_cost_per_token": 2.7e-07,
- "output_cost_per_token": 2.7e-07,
- "litellm_provider": "deepinfra",
- "mode": "completion"
- },
- "deepinfra/Phind/Phind-CodeLlama-34B-v2": {
"max_tokens": 4096,
- "max_input_tokens": 16384,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/mistralai/Mixtral-8x7B-Instruct-v0.1": {
- "max_tokens": 8191,
- "max_input_tokens": 32768,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2.7e-07,
- "output_cost_per_token": 2.7e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/deepinfra/airoboros-70b": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/01-ai/Yi-34B-Chat": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/01-ai/Yi-6B-200K": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 1.3e-07,
- "litellm_provider": "deepinfra",
- "mode": "completion"
- },
- "deepinfra/jondurbin/airoboros-l2-70b-gpt4-1.4.1": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/meta-llama/Llama-2-13b-chat-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.2e-07,
- "output_cost_per_token": 2.2e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/amazon/MistralLite": {
- "max_tokens": 8191,
- "max_input_tokens": 32768,
- "max_output_tokens": 8191,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/meta-llama/Llama-2-7b-chat-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 1.3e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/meta-llama/Meta-Llama-3-8B-Instruct": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "max_output_tokens": 4096,
- "input_cost_per_token": 8e-08,
- "output_cost_per_token": 8e-08,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/meta-llama/Meta-Llama-3-70B-Instruct": {
- "max_tokens": 8191,
- "max_input_tokens": 8191,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5.9e-07,
- "output_cost_per_token": 7.9e-07,
- "litellm_provider": "deepinfra",
- "mode": "chat",
- "supports_tool_choice": true
- },
- "deepinfra/meta-llama/Meta-Llama-3.1-405B-Instruct": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "deepinfra",
"mode": "chat",
+ "output_cost_per_audio_token": 6.4e-05,
+ "output_cost_per_token": 1.6e-05,
+ "supported_endpoints": [
+ "/v1/realtime"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "deepinfra/01-ai/Yi-34B-200K": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
+ "gpt-realtime-mini": {
+ "cache_creation_input_audio_token_cost": 3e-07,
+ "cache_read_input_audio_token_cost": 3e-07,
+ "input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 6e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "deepinfra",
- "mode": "completion"
- },
- "deepinfra/openchat/openchat_3.5": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
"max_output_tokens": 4096,
- "input_cost_per_token": 1.3e-07,
- "output_cost_per_token": 1.3e-07,
- "litellm_provider": "deepinfra",
+ "max_tokens": 4096,
"mode": "chat",
+ "output_cost_per_audio_token": 2e-05,
+ "output_cost_per_token": 2.4e-06,
+ "supported_endpoints": [
+ "/v1/realtime"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "perplexity/codellama-34b-instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 3.5e-07,
- "output_cost_per_token": 1.4e-06,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/codellama-70b-instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 2.8e-06,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/llama-3.1-70b-instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/llama-3.1-8b-instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/llama-3.1-sonar-huge-128k-online": {
- "max_tokens": 127072,
- "max_input_tokens": 127072,
- "max_output_tokens": 127072,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "deprecation_date": "2025-02-22"
- },
- "perplexity/llama-3.1-sonar-large-128k-online": {
- "max_tokens": 127072,
- "max_input_tokens": 127072,
- "max_output_tokens": 127072,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "deprecation_date": "2025-02-22"
- },
- "perplexity/llama-3.1-sonar-large-128k-chat": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "deprecation_date": "2025-02-22"
- },
- "perplexity/llama-3.1-sonar-small-128k-chat": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "deprecation_date": "2025-02-22"
- },
- "perplexity/llama-3.1-sonar-small-128k-online": {
- "max_tokens": 127072,
- "max_input_tokens": 127072,
- "max_output_tokens": 127072,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "deprecation_date": "2025-02-22"
- },
- "perplexity/pplx-7b-chat": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 7e-08,
- "output_cost_per_token": 2.8e-07,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/pplx-70b-chat": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
+ "gpt-realtime-2025-08-28": {
+ "cache_creation_input_audio_token_cost": 4e-07,
+ "cache_read_input_token_cost": 4e-07,
+ "input_cost_per_audio_token": 3.2e-05,
+ "input_cost_per_image": 5e-06,
+ "input_cost_per_token": 4e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 32000,
"max_output_tokens": 4096,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 2.8e-06,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/pplx-7b-online": {
"max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 2.8e-07,
- "input_cost_per_request": 0.005,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/pplx-70b-online": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 2.8e-06,
- "input_cost_per_request": 0.005,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/llama-2-70b-chat": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7e-07,
- "output_cost_per_token": 2.8e-06,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/mistral-7b-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7e-08,
- "output_cost_per_token": 2.8e-07,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/mixtral-8x7b-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 7e-08,
- "output_cost_per_token": 2.8e-07,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/sonar-small-chat": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 7e-08,
- "output_cost_per_token": 2.8e-07,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/sonar-small-online": {
- "max_tokens": 12000,
- "max_input_tokens": 12000,
- "max_output_tokens": 12000,
- "input_cost_per_token": 0,
- "output_cost_per_token": 2.8e-07,
- "input_cost_per_request": 0.005,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/sonar-medium-chat": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 1.8e-06,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/sonar-medium-online": {
- "max_tokens": 12000,
- "max_input_tokens": 12000,
- "max_output_tokens": 12000,
- "input_cost_per_token": 0,
- "output_cost_per_token": 1.8e-06,
- "input_cost_per_request": 0.005,
- "litellm_provider": "perplexity",
- "mode": "chat"
- },
- "perplexity/sonar": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "search_context_cost_per_query": {
- "search_context_size_low": 0.005,
- "search_context_size_medium": 0.008,
- "search_context_size_high": 0.012
- },
- "supports_web_search": true
- },
- "perplexity/sonar-pro": {
- "max_tokens": 8000,
- "max_input_tokens": 200000,
- "max_output_tokens": 8000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "search_context_cost_per_query": {
- "search_context_size_low": 0.006,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.014
- },
- "supports_web_search": true
- },
- "perplexity/sonar-reasoning": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "search_context_cost_per_query": {
- "search_context_size_low": 0.005,
- "search_context_size_medium": 0.008,
- "search_context_size_high": 0.014
- },
- "supports_web_search": true,
- "supports_reasoning": true
- },
- "perplexity/sonar-reasoning-pro": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "perplexity",
- "mode": "chat",
- "search_context_cost_per_query": {
- "search_context_size_low": 0.006,
- "search_context_size_medium": 0.01,
- "search_context_size_high": 0.014
- },
- "supports_web_search": true,
- "supports_reasoning": true
- },
- "perplexity/sonar-deep-research": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 8e-06,
- "output_cost_per_reasoning_token": 3e-06,
- "citation_cost_per_token": 2e-06,
- "search_context_cost_per_query": {
- "search_context_size_low": 0.005,
- "search_context_size_medium": 0.005,
- "search_context_size_high": 0.005
- },
- "litellm_provider": "perplexity",
- "mode": "chat",
- "supports_reasoning": true,
- "supports_web_search": true
- },
- "fireworks_ai/accounts/fireworks/models/llama-v3p2-1b-instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/llama-v3p2-3b-instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/llama-v3p1-8b-instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/llama-v3p2-11b-vision-instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_vision": true,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/llama-v3p2-90b-vision-instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_tool_choice": false,
- "supports_vision": true,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing"
- },
- "fireworks_ai/accounts/fireworks/models/firefunction-v2": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "fireworks_ai",
"mode": "chat",
+ "output_cost_per_audio_token": 6.4e-05,
+ "output_cost_per_token": 1.6e-05,
+ "supported_endpoints": [
+ "/v1/realtime"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "audio"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "audio"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
"supports_function_calling": true,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "fireworks_ai/accounts/fireworks/models/mixtral-8x22b-instruct-hf": {
- "max_tokens": 65536,
- "max_input_tokens": 65536,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.2e-06,
- "output_cost_per_token": 1.2e-06,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": true
- },
- "fireworks_ai/accounts/fireworks/models/qwen2-72b-instruct": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/qwen2p5-coder-32b-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/yi-large": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/deepseek-coder-v2-instruct": {
- "max_tokens": 65536,
- "max_input_tokens": 65536,
- "max_output_tokens": 65536,
- "input_cost_per_token": 1.2e-06,
- "output_cost_per_token": 1.2e-06,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/deepseek-v3": {
- "max_tokens": 8192,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/deepseek-r1": {
- "max_tokens": 20480,
- "max_input_tokens": 128000,
- "max_output_tokens": 20480,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/deepseek-r1-basic": {
- "max_tokens": 20480,
- "max_input_tokens": 128000,
- "max_output_tokens": 20480,
- "input_cost_per_token": 5.5e-07,
- "output_cost_per_token": 2.19e-06,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/deepseek-r1-0528": {
- "max_tokens": 160000,
- "max_input_tokens": 160000,
- "max_output_tokens": 160000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 8e-06,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false,
- "supports_response_schema": true
- },
- "fireworks_ai/accounts/fireworks/models/kimi-k2-instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 16384,
- "input_cost_per_token": 0.6e-06,
- "output_cost_per_token": 2.5e-06,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "source": "https://fireworks.ai/models/fireworks/kimi-k2-instruct"
- },
- "fireworks_ai/accounts/fireworks/models/llama-v3p1-405b-instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 128000,
- "max_output_tokens": 16384,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": true,
- "supports_function_calling": true
- },
- "fireworks_ai/accounts/fireworks/models/llama4-maverick-instruct-basic": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2.2e-07,
- "output_cost_per_token": 8.8e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/accounts/fireworks/models/llama4-scout-instruct-basic": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "fireworks_ai",
- "mode": "chat",
- "supports_response_schema": true,
- "source": "https://fireworks.ai/pricing",
- "supports_tool_choice": false
- },
- "fireworks_ai/nomic-ai/nomic-embed-text-v1.5": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "input_cost_per_token": 8e-09,
- "output_cost_per_token": 0.0,
- "litellm_provider": "fireworks_ai-embedding-models",
- "mode": "embedding",
- "source": "https://fireworks.ai/pricing"
- },
- "fireworks_ai/nomic-ai/nomic-embed-text-v1": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "input_cost_per_token": 8e-09,
- "output_cost_per_token": 0.0,
- "litellm_provider": "fireworks_ai-embedding-models",
- "mode": "embedding",
- "source": "https://fireworks.ai/pricing"
- },
- "fireworks_ai/WhereIsAI/UAE-Large-V1": {
- "max_tokens": 512,
- "max_input_tokens": 512,
- "input_cost_per_token": 1.6e-08,
- "output_cost_per_token": 0.0,
- "litellm_provider": "fireworks_ai-embedding-models",
- "mode": "embedding",
- "source": "https://fireworks.ai/pricing"
- },
- "fireworks_ai/thenlper/gte-large": {
- "max_tokens": 512,
- "max_input_tokens": 512,
- "input_cost_per_token": 1.6e-08,
- "output_cost_per_token": 0.0,
- "litellm_provider": "fireworks_ai-embedding-models",
- "mode": "embedding",
- "source": "https://fireworks.ai/pricing"
- },
- "fireworks_ai/thenlper/gte-base": {
- "max_tokens": 512,
- "max_input_tokens": 512,
- "input_cost_per_token": 8e-09,
- "output_cost_per_token": 0.0,
- "litellm_provider": "fireworks_ai-embedding-models",
- "mode": "embedding",
- "source": "https://fireworks.ai/pricing"
- },
- "fireworks-ai-up-to-4b": {
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "fireworks_ai"
- },
- "fireworks-ai-4.1b-to-16b": {
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "fireworks_ai"
- },
- "fireworks-ai-above-16b": {
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "fireworks_ai"
- },
- "fireworks-ai-moe-up-to-56b": {
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 5e-07,
- "litellm_provider": "fireworks_ai"
- },
- "fireworks-ai-56b-to-176b": {
- "input_cost_per_token": 1.2e-06,
- "output_cost_per_token": 1.2e-06,
- "litellm_provider": "fireworks_ai"
- },
- "fireworks-ai-default": {
- "input_cost_per_token": 0.0,
- "output_cost_per_token": 0.0,
- "litellm_provider": "fireworks_ai"
- },
- "fireworks-ai-embedding-up-to-150m": {
- "input_cost_per_token": 8e-09,
- "output_cost_per_token": 0.0,
- "litellm_provider": "fireworks_ai-embedding-models"
- },
- "fireworks-ai-embedding-150m-to-350m": {
- "input_cost_per_token": 1.6e-08,
- "output_cost_per_token": 0.0,
- "litellm_provider": "fireworks_ai-embedding-models"
- },
- "anyscale/mistralai/Mistral-7B-Instruct-v0.1": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "anyscale",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/mistralai-Mistral-7B-Instruct-v0.1"
- },
- "anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "anyscale",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/mistralai-Mixtral-8x7B-Instruct-v0.1"
- },
- "anyscale/mistralai/Mixtral-8x22B-Instruct-v0.1": {
- "max_tokens": 65536,
- "max_input_tokens": 65536,
- "max_output_tokens": 65536,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 9e-07,
- "litellm_provider": "anyscale",
- "mode": "chat",
- "supports_function_calling": true,
- "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/mistralai-Mixtral-8x22B-Instruct-v0.1"
- },
- "anyscale/HuggingFaceH4/zephyr-7b-beta": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "anyscale",
- "mode": "chat"
- },
- "anyscale/google/gemma-7b-it": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "anyscale",
- "mode": "chat",
- "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/google-gemma-7b-it"
- },
- "anyscale/meta-llama/Llama-2-7b-chat-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "anyscale",
- "mode": "chat"
- },
- "anyscale/meta-llama/Llama-2-13b-chat-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "anyscale",
- "mode": "chat"
- },
- "anyscale/meta-llama/Llama-2-70b-chat-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "anyscale",
- "mode": "chat"
- },
- "anyscale/codellama/CodeLlama-34b-Instruct-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "anyscale",
- "mode": "chat"
- },
- "anyscale/codellama/CodeLlama-70b-Instruct-hf": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "anyscale",
- "mode": "chat",
- "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/codellama-CodeLlama-70b-Instruct-hf"
- },
- "anyscale/meta-llama/Meta-Llama-3-8B-Instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "anyscale",
- "mode": "chat",
- "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/meta-llama-Meta-Llama-3-8B-Instruct"
- },
- "anyscale/meta-llama/Meta-Llama-3-70B-Instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "anyscale",
- "mode": "chat",
- "source": "https://docs.anyscale.com/preview/endpoints/text-generation/supported-models/meta-llama-Meta-Llama-3-70B-Instruct"
- },
- "cloudflare/@cf/meta/llama-2-7b-chat-fp16": {
- "max_tokens": 3072,
- "max_input_tokens": 3072,
- "max_output_tokens": 3072,
- "input_cost_per_token": 1.923e-06,
- "output_cost_per_token": 1.923e-06,
- "litellm_provider": "cloudflare",
- "mode": "chat"
- },
- "cloudflare/@cf/meta/llama-2-7b-chat-int8": {
+ "gradient_ai/alibaba-qwen3-32b": {
+ "litellm_provider": "gradient_ai",
"max_tokens": 2048,
- "max_input_tokens": 2048,
- "max_output_tokens": 2048,
- "input_cost_per_token": 1.923e-06,
- "output_cost_per_token": 1.923e-06,
- "litellm_provider": "cloudflare",
- "mode": "chat"
- },
- "cloudflare/@cf/mistral/mistral-7b-instruct-v0.1": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 1.923e-06,
- "output_cost_per_token": 1.923e-06,
- "litellm_provider": "cloudflare",
- "mode": "chat"
- },
- "cloudflare/@hf/thebloke/codellama-7b-instruct-awq": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 1.923e-06,
- "output_cost_per_token": 1.923e-06,
- "litellm_provider": "cloudflare",
- "mode": "chat"
- },
- "v0/v0-1.0-md": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "v0",
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "v0/v0-1.5-md": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 3e-06,
- "output_cost_per_token": 1.5e-05,
- "litellm_provider": "v0",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "v0/v0-1.5-lg": {
- "max_tokens": 512000,
- "max_input_tokens": 512000,
- "max_output_tokens": 512000,
+ "gradient_ai/anthropic-claude-3-opus": {
"input_cost_per_token": 1.5e-05,
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 1024,
+ "mode": "chat",
"output_cost_per_token": 7.5e-05,
- "litellm_provider": "v0",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "lambda_ai/deepseek-llama3.3-70b": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_reasoning": true
- },
- "lambda_ai/deepseek-r1-0528": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_reasoning": true
- },
- "lambda_ai/deepseek-r1-671b": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
+ "gradient_ai/anthropic-claude-3.5-haiku": {
"input_cost_per_token": 8e-07,
- "output_cost_per_token": 8e-07,
- "litellm_provider": "lambda_ai",
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 1024,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_reasoning": true
+ "output_cost_per_token": 4e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "lambda_ai/deepseek-v3-0324": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
+ "gradient_ai/anthropic-claude-3.5-sonnet": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 1024,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
+ },
+ "gradient_ai/anthropic-claude-3.7-sonnet": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 1024,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
+ },
+ "gradient_ai/deepseek-r1-distill-llama-70b": {
+ "input_cost_per_token": 9.9e-07,
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 8000,
+ "mode": "chat",
+ "output_cost_per_token": 9.9e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
+ },
+ "gradient_ai/llama3-8b-instruct": {
"input_cost_per_token": 2e-07,
- "output_cost_per_token": 6e-07,
- "litellm_provider": "lambda_ai",
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 512,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/hermes3-405b": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 8e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/hermes3-70b": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/hermes3-8b": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2.5e-08,
- "output_cost_per_token": 4e-08,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/lfm-40b": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1e-07,
"output_cost_per_token": 2e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "lambda_ai/lfm-7b": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2.5e-08,
- "output_cost_per_token": 4e-08,
- "litellm_provider": "lambda_ai",
+ "gradient_ai/llama3.3-70b-instruct": {
+ "input_cost_per_token": 6.5e-07,
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 2048,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "output_cost_per_token": 6.5e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "lambda_ai/llama-4-maverick-17b-128e-instruct-fp8": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "lambda_ai",
+ "gradient_ai/mistral-nemo-instruct-2407": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 512,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "output_cost_per_token": 3e-07,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "lambda_ai/llama-4-scout-17b-16e-instruct": {
+ "gradient_ai/openai-gpt-4o": {
+ "litellm_provider": "gradient_ai",
"max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "lambda_ai",
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "lambda_ai/llama3.1-405b-instruct-fp8": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 8e-07,
- "output_cost_per_token": 8e-07,
- "litellm_provider": "lambda_ai",
+ "gradient_ai/openai-gpt-4o-mini": {
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 16384,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "lambda_ai/llama3.1-70b-instruct-fp8": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/llama3.1-8b-instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2.5e-08,
- "output_cost_per_token": 4e-08,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/llama3.1-nemotron-70b-instruct-fp8": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/llama3.2-11b-vision-instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1.5e-08,
- "output_cost_per_token": 2.5e-08,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_vision": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/llama3.2-3b-instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1.5e-08,
- "output_cost_per_token": 2.5e-08,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/llama3.3-70b-instruct-fp8": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/qwen25-coder-32b-instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "lambda_ai/qwen3-32b-fp8": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 5e-08,
- "output_cost_per_token": 1e-07,
- "litellm_provider": "lambda_ai",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true,
- "supports_reasoning": true
- },
- "hyperbolic/moonshotai/Kimi-K2-Instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
+ "gradient_ai/openai-o3": {
"input_cost_per_token": 2e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "hyperbolic",
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 100000,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "output_cost_per_token": 8e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "hyperbolic/deepseek-ai/DeepSeek-R1-0528": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2.5e-07,
- "output_cost_per_token": 2.5e-07,
- "litellm_provider": "hyperbolic",
+ "gradient_ai/openai-o3-mini": {
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "gradient_ai",
+ "max_tokens": 100000,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
+ "output_cost_per_token": 4.4e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions"
+ ],
+ "supported_modalities": [
+ "text"
+ ],
+ "supports_tool_choice": false
},
- "hyperbolic/Qwen/Qwen3-235B-A22B": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "hyperbolic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "hyperbolic/deepseek-ai/DeepSeek-V3-0324": {
+ "lemonade/Qwen3-Coder-30B-A3B-Instruct-GGUF": {
+ "input_cost_per_token": 0,
+ "litellm_provider": "lemonade",
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "hyperbolic",
"mode": "chat",
+ "output_cost_per_token": 0,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/deepseek-r1-distill-llama-70b": {
+ "input_cost_per_token": 7.5e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 9.9e-07,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/distil-whisper-large-v3-en": {
+ "input_cost_per_second": 5.56e-06,
+ "litellm_provider": "groq",
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0
+ },
+ "groq/gemma-7b-it": {
+ "deprecation_date": "2024-12-18",
+ "input_cost_per_token": 7e-08,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 7e-08,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/gemma2-9b-it": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "groq/llama-3.1-405b-reasoning": {
+ "input_cost_per_token": 5.9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 7.9e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama-3.1-70b-versatile": {
+ "deprecation_date": "2025-01-24",
+ "input_cost_per_token": 5.9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 7.9e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama-3.1-8b-instant": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "groq",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 8e-08,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama-3.2-11b-text-preview": {
+ "deprecation_date": "2024-10-28",
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama-3.2-11b-vision-preview": {
+ "deprecation_date": "2025-04-14",
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "groq/llama-3.2-1b-preview": {
+ "deprecation_date": "2025-04-14",
+ "input_cost_per_token": 4e-08,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4e-08,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama-3.2-3b-preview": {
+ "deprecation_date": "2025-04-14",
+ "input_cost_per_token": 6e-08,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6e-08,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama-3.2-90b-text-preview": {
+ "deprecation_date": "2024-11-25",
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama-3.2-90b-vision-preview": {
+ "deprecation_date": "2025-04-14",
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "groq/llama-3.3-70b-specdec": {
+ "deprecation_date": "2025-04-14",
+ "input_cost_per_token": 5.9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 9.9e-07,
+ "supports_tool_choice": true
+ },
+ "groq/llama-3.3-70b-versatile": {
+ "input_cost_per_token": 5.9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 7.9e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama-guard-3-8b": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07
+ },
+ "groq/llama2-70b-4096": {
+ "input_cost_per_token": 7e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 8e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama3-groq-70b-8192-tool-use-preview": {
+ "deprecation_date": "2025-01-06",
+ "input_cost_per_token": 8.9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 8.9e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/llama3-groq-8b-8192-tool-use-preview": {
+ "deprecation_date": "2025-01-06",
+ "input_cost_per_token": 1.9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.9e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/meta-llama/llama-4-maverick-17b-128e-instruct": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/meta-llama/llama-4-scout-17b-16e-instruct": {
+ "input_cost_per_token": 1.1e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 3.4e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/mistral-saba-24b": {
+ "input_cost_per_token": 7.9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.9e-07
+ },
+ "groq/mixtral-8x7b-32768": {
+ "deprecation_date": "2025-03-20",
+ "input_cost_per_token": 2.4e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-07,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/moonshotai/kimi-k2-instruct": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "groq",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/moonshotai/kimi-k2-instruct-0905": {
+ "input_cost_per_token": 1e-06,
+ "output_cost_per_token": 3e-06,
+ "cache_read_input_token_cost": 0.5e-06,
+ "litellm_provider": "groq",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 16384,
+ "max_tokens": 278528,
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/openai/gpt-oss-120b": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 32766,
+ "max_tokens": 32766,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "groq/openai/gpt-oss-20b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "groq/playai-tts": {
+ "input_cost_per_character": 5e-05,
+ "litellm_provider": "groq",
+ "max_input_tokens": 10000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "audio_speech"
+ },
+ "groq/qwen/qwen3-32b": {
+ "input_cost_per_token": 2.9e-07,
+ "litellm_provider": "groq",
+ "max_input_tokens": 131000,
+ "max_output_tokens": 131000,
+ "max_tokens": 131000,
+ "mode": "chat",
+ "output_cost_per_token": 5.9e-07,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "groq/whisper-large-v3": {
+ "input_cost_per_second": 3.083e-05,
+ "litellm_provider": "groq",
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0
+ },
+ "groq/whisper-large-v3-turbo": {
+ "input_cost_per_second": 1.111e-05,
+ "litellm_provider": "groq",
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0
+ },
+ "hd/1024-x-1024/dall-e-3": {
+ "input_cost_per_pixel": 7.629e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
+ },
+ "hd/1024-x-1792/dall-e-3": {
+ "input_cost_per_pixel": 6.539e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
+ },
+ "hd/1792-x-1024/dall-e-3": {
+ "input_cost_per_pixel": 6.539e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
+ },
+ "heroku/claude-3-5-haiku": {
+ "litellm_provider": "heroku",
+ "max_tokens": 4096,
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "heroku/claude-3-5-sonnet-latest": {
+ "litellm_provider": "heroku",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "heroku/claude-3-7-sonnet": {
+ "litellm_provider": "heroku",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "heroku/claude-4-sonnet": {
+ "litellm_provider": "heroku",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "high/1024-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 1.59263611e-07,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "high/1024-x-1536/gpt-image-1": {
+ "input_cost_per_pixel": 1.58945719e-07,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "high/1536-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 1.58945719e-07,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "hyperbolic/NousResearch/Hermes-3-Llama-3.1-70B": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"hyperbolic/Qwen/QwQ-32B": {
- "max_tokens": 131072,
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "hyperbolic",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "hyperbolic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "hyperbolic/deepseek-ai/DeepSeek-R1": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "hyperbolic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "hyperbolic/deepseek-ai/DeepSeek-V3": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "hyperbolic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "hyperbolic/meta-llama/Llama-3.3-70B-Instruct": {
"max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "hyperbolic/Qwen/Qwen2.5-Coder-32B-Instruct": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "hyperbolic/meta-llama/Llama-3.2-3B-Instruct": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
"mode": "chat",
+ "output_cost_per_token": 2e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"hyperbolic/Qwen/Qwen2.5-72B-Instruct": {
- "max_tokens": 131072,
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "hyperbolic",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
+ "max_tokens": 131072,
"mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "hyperbolic/Qwen/Qwen2.5-Coder-32B-Instruct": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "hyperbolic/Qwen/Qwen3-235B-A22B": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "hyperbolic/deepseek-ai/DeepSeek-R1": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "hyperbolic/deepseek-ai/DeepSeek-R1-0528": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "hyperbolic/deepseek-ai/DeepSeek-V3": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "hyperbolic/deepseek-ai/DeepSeek-V3-0324": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "hyperbolic/meta-llama/Llama-3.2-3B-Instruct": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "hyperbolic/meta-llama/Llama-3.3-70B-Instruct": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"hyperbolic/meta-llama/Meta-Llama-3-70B-Instruct": {
- "max_tokens": 131072,
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "hyperbolic",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
+ "max_tokens": 131072,
"mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "hyperbolic/NousResearch/Hermes-3-Llama-3.1-70B": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.2e-07,
"output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
- "mode": "chat",
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"hyperbolic/meta-llama/Meta-Llama-3.1-405B-Instruct": {
- "max_tokens": 32768,
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "hyperbolic",
"max_input_tokens": 32768,
"max_output_tokens": 32768,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_parallel_function_calling": true,
- "supports_system_messages": true,
- "supports_tool_choice": true
- },
- "hyperbolic/meta-llama/Meta-Llama-3.1-8B-Instruct": {
"max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
"mode": "chat",
+ "output_cost_per_token": 3e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"hyperbolic/meta-llama/Meta-Llama-3.1-70B-Instruct": {
- "max_tokens": 32768,
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "hyperbolic",
"max_input_tokens": 32768,
"max_output_tokens": 32768,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 3e-07,
- "litellm_provider": "hyperbolic",
+ "max_tokens": 32768,
"mode": "chat",
+ "output_cost_per_token": 3e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
- "voyage/voyage-lite-01": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-large-2": {
- "max_tokens": 16000,
- "max_input_tokens": 16000,
+ "hyperbolic/meta-llama/Meta-Llama-3.1-8B-Instruct": {
"input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-finance-2": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-lite-02-instruct": {
- "max_tokens": 4000,
- "max_input_tokens": 4000,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-law-2": {
- "max_tokens": 16000,
- "max_input_tokens": 16000,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-code-2": {
- "max_tokens": 16000,
- "max_input_tokens": 16000,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-2": {
- "max_tokens": 4000,
- "max_input_tokens": 4000,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-3-large": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "input_cost_per_token": 1.8e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-3": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "input_cost_per_token": 6e-08,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-3-lite": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "input_cost_per_token": 2e-08,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-code-3": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "input_cost_per_token": 1.8e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/voyage-multimodal-3": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "input_cost_per_token": 1.2e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "embedding"
- },
- "voyage/rerank-2": {
- "max_tokens": 16000,
- "max_input_tokens": 16000,
- "max_output_tokens": 16000,
- "max_query_tokens": 16000,
- "input_cost_per_token": 5e-08,
- "input_cost_per_query": 5e-08,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "rerank"
- },
- "voyage/rerank-2-lite": {
- "max_tokens": 8000,
- "max_input_tokens": 8000,
- "max_output_tokens": 8000,
- "max_query_tokens": 8000,
- "input_cost_per_token": 2e-08,
- "input_cost_per_query": 2e-08,
- "output_cost_per_token": 0.0,
- "litellm_provider": "voyage",
- "mode": "rerank"
- },
- "databricks/databricks-claude-3-7-sonnet": {
- "max_tokens": 200000,
- "max_input_tokens": 200000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 2.5e-06,
- "input_dbu_cost_per_token": 3.571e-05,
- "output_cost_per_token": 1.7857e-05,
- "output_db_cost_per_token": 0.000214286,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Claude 3.7 conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
- "supports_assistant_prefill": true,
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true
- },
- "databricks/databricks-meta-llama-3-1-405b-instruct": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 5e-06,
- "input_dbu_cost_per_token": 7.1429e-05,
- "output_cost_per_token": 1.500002e-05,
- "output_db_cost_per_token": 0.000214286,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
- "supports_tool_choice": true
- },
- "databricks/databricks-meta-llama-3-1-70b-instruct": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 1.00002e-06,
- "input_dbu_cost_per_token": 1.4286e-05,
- "output_cost_per_token": 2.99999e-06,
- "output_dbu_cost_per_token": 4.2857e-05,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
- "supports_tool_choice": true
- },
- "databricks/databricks-meta-llama-3-3-70b-instruct": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 1.00002e-06,
- "input_dbu_cost_per_token": 1.4286e-05,
- "output_cost_per_token": 2.99999e-06,
- "output_dbu_cost_per_token": 4.2857e-05,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
- "supports_tool_choice": true
- },
- "databricks/databricks-llama-4-maverick": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 128000,
- "input_cost_per_token": 5e-06,
- "input_dbu_cost_per_token": 7.143e-05,
- "output_cost_per_token": 1.5e-05,
- "output_dbu_cost_per_token": 0.00021429,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Databricks documentation now provides both DBU costs (_dbu_cost_per_token) and dollar costs(_cost_per_token)."
- },
- "supports_tool_choice": true
- },
- "databricks/databricks-dbrx-instruct": {
- "max_tokens": 32768,
+ "litellm_provider": "hyperbolic",
"max_input_tokens": 32768,
"max_output_tokens": 32768,
- "input_cost_per_token": 7.4998e-07,
- "input_dbu_cost_per_token": 1.0714e-05,
- "output_cost_per_token": 2.24901e-06,
- "output_dbu_cost_per_token": 3.2143e-05,
- "litellm_provider": "databricks",
+ "max_tokens": 32768,
"mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
"supports_tool_choice": true
},
- "databricks/databricks-meta-llama-3-70b-instruct": {
+ "hyperbolic/moonshotai/Kimi-K2-Instruct": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "hyperbolic",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "j2-light": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "completion",
+ "output_cost_per_token": 3e-06
+ },
+ "j2-mid": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "completion",
+ "output_cost_per_token": 1e-05
+ },
+ "j2-ultra": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "completion",
+ "output_cost_per_token": 1.5e-05
+ },
+ "jamba-1.5": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_tool_choice": true
+ },
+ "jamba-1.5-large": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supports_tool_choice": true
+ },
+ "jamba-1.5-large@001": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supports_tool_choice": true
+ },
+ "jamba-1.5-mini": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_tool_choice": true
+ },
+ "jamba-1.5-mini@001": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_tool_choice": true
+ },
+ "jamba-large-1.6": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supports_tool_choice": true
+ },
+ "jamba-large-1.7": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supports_tool_choice": true
+ },
+ "jamba-mini-1.6": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_tool_choice": true
+ },
+ "jamba-mini-1.7": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "ai21",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_tool_choice": true
+ },
+ "jina-reranker-v2-base-multilingual": {
+ "input_cost_per_token": 1.8e-08,
+ "litellm_provider": "jina_ai",
+ "max_document_chunks_per_query": 2048,
+ "max_input_tokens": 1024,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "rerank",
+ "output_cost_per_token": 1.8e-08
+ },
+ "jp.anthropic.claude-sonnet-4-5-20250929-v1:0": {
+ "cache_creation_input_token_cost": 4.125e-06,
+ "cache_read_input_token_cost": 3.3e-07,
+ "input_cost_per_token": 3.3e-06,
+ "input_cost_per_token_above_200k_tokens": 6.6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.475e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.65e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 346
+ },
+ "lambda_ai/deepseek-llama3.3-70b": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/deepseek-r1-0528": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/deepseek-r1-671b": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 8e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/deepseek-v3-0324": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/hermes3-405b": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 8e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/hermes3-70b": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/hermes3-8b": {
+ "input_cost_per_token": 2.5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 4e-08,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/lfm-40b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/lfm-7b": {
+ "input_cost_per_token": 2.5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 4e-08,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/llama-4-maverick-17b-128e-instruct-fp8": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8192,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/llama-4-scout-17b-16e-instruct": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 8192,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/llama3.1-405b-instruct-fp8": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 8e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/llama3.1-70b-instruct-fp8": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/llama3.1-8b-instruct": {
+ "input_cost_per_token": 2.5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 4e-08,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/llama3.1-nemotron-70b-instruct-fp8": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/llama3.2-11b-vision-instruct": {
+ "input_cost_per_token": 1.5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-08,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "lambda_ai/llama3.2-3b-instruct": {
+ "input_cost_per_token": 1.5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-08,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/llama3.3-70b-instruct-fp8": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/qwen25-coder-32b-instruct": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "lambda_ai/qwen3-32b-fp8": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "lambda_ai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true
+ },
+ "low/1024-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 1.0490417e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "low/1024-x-1536/gpt-image-1": {
+ "input_cost_per_pixel": 1.0172526e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "low/1536-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 1.0172526e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "luminous-base": {
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "aleph_alpha",
+ "max_tokens": 2048,
+ "mode": "completion",
+ "output_cost_per_token": 3.3e-05
+ },
+ "luminous-base-control": {
+ "input_cost_per_token": 3.75e-05,
+ "litellm_provider": "aleph_alpha",
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 4.125e-05
+ },
+ "luminous-extended": {
+ "input_cost_per_token": 4.5e-05,
+ "litellm_provider": "aleph_alpha",
+ "max_tokens": 2048,
+ "mode": "completion",
+ "output_cost_per_token": 4.95e-05
+ },
+ "luminous-extended-control": {
+ "input_cost_per_token": 5.625e-05,
+ "litellm_provider": "aleph_alpha",
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 6.1875e-05
+ },
+ "luminous-supreme": {
+ "input_cost_per_token": 0.000175,
+ "litellm_provider": "aleph_alpha",
+ "max_tokens": 2048,
+ "mode": "completion",
+ "output_cost_per_token": 0.0001925
+ },
+ "luminous-supreme-control": {
+ "input_cost_per_token": 0.00021875,
+ "litellm_provider": "aleph_alpha",
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 0.000240625
+ },
+ "max-x-max/50-steps/stability.stable-diffusion-xl-v0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.036
+ },
+ "max-x-max/max-steps/stability.stable-diffusion-xl-v0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.072
+ },
+ "medium/1024-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 4.0054321e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "medium/1024-x-1536/gpt-image-1": {
+ "input_cost_per_pixel": 4.0054321e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "medium/1536-x-1024/gpt-image-1": {
+ "input_cost_per_pixel": 4.0054321e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "low/1024-x-1024/gpt-image-1-mini": {
+ "input_cost_per_image": 0.005,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "low/1024-x-1536/gpt-image-1-mini": {
+ "input_cost_per_image": 0.006,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "low/1536-x-1024/gpt-image-1-mini": {
+ "input_cost_per_image": 0.006,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "medium/1024-x-1024/gpt-image-1-mini": {
+ "input_cost_per_image": 0.011,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "medium/1024-x-1536/gpt-image-1-mini": {
+ "input_cost_per_image": 0.015,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "medium/1536-x-1024/gpt-image-1-mini": {
+ "input_cost_per_image": 0.015,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "medlm-large": {
+ "input_cost_per_character": 5e-06,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "chat",
+ "output_cost_per_character": 1.5e-05,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_tool_choice": true
+ },
+ "medlm-medium": {
+ "input_cost_per_character": 5e-07,
+ "litellm_provider": "vertex_ai-language-models",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_character": 1e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models",
+ "supports_tool_choice": true
+ },
+ "meta.llama2-13b-chat-v1": {
+ "input_cost_per_token": 7.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06
+ },
+ "meta.llama2-70b-chat-v1": {
+ "input_cost_per_token": 1.95e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.56e-06
+ },
+ "meta.llama3-1-405b-instruct-v1:0": {
+ "input_cost_per_token": 5.32e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
"max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "meta.llama3-1-70b-instruct-v1:0": {
+ "input_cost_per_token": 9.9e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 9.9e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "meta.llama3-1-8b-instruct-v1:0": {
+ "input_cost_per_token": 2.2e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2.2e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "meta.llama3-2-11b-instruct-v1:0": {
+ "input_cost_per_token": 3.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3.5e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "meta.llama3-2-1b-instruct-v1:0": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "meta.llama3-2-3b-instruct-v1:0": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "meta.llama3-2-90b-instruct-v1:0": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "meta.llama3-3-70b-instruct-v1:0": {
+ "input_cost_per_token": 7.2e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.2e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "meta.llama3-70b-instruct-v1:0": {
+ "input_cost_per_token": 2.65e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 3.5e-06
+ },
+ "meta.llama3-8b-instruct-v1:0": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07
+ },
+ "meta.llama4-maverick-17b-instruct-v1:0": {
+ "input_cost_per_token": 2.4e-07,
+ "input_cost_per_token_batches": 1.2e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 9.7e-07,
+ "output_cost_per_token_batches": 4.85e-07,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "code"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "meta.llama4-scout-17b-instruct-v1:0": {
+ "input_cost_per_token": 1.7e-07,
+ "input_cost_per_token_batches": 8.5e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6.6e-07,
+ "output_cost_per_token_batches": 3.3e-07,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "code"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "meta_llama/Llama-3.3-70B-Instruct": {
+ "litellm_provider": "meta_llama",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4028,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "source": "https://llama.developer.meta.com/docs/models",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "meta_llama/Llama-3.3-8B-Instruct": {
+ "litellm_provider": "meta_llama",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4028,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "source": "https://llama.developer.meta.com/docs/models",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "meta_llama/Llama-4-Maverick-17B-128E-Instruct-FP8": {
+ "litellm_provider": "meta_llama",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 4028,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "source": "https://llama.developer.meta.com/docs/models",
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "meta_llama/Llama-4-Scout-17B-16E-Instruct-FP8": {
+ "litellm_provider": "meta_llama",
+ "max_input_tokens": 10000000,
+ "max_output_tokens": 4028,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "source": "https://llama.developer.meta.com/docs/models",
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "mistral.mistral-7b-instruct-v0:2": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "supports_tool_choice": true
+ },
+ "mistral.mistral-large-2402-v1:0": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_function_calling": true
+ },
+ "mistral.mistral-large-2407-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 9e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "mistral.mistral-small-2402-v1:0": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_function_calling": true
+ },
+ "mistral.mixtral-8x7b-instruct-v0:1": {
+ "input_cost_per_token": 4.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
+ "supports_tool_choice": true
+ },
+ "mistral/codestral-2405": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_assistant_prefill": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/codestral-latest": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_assistant_prefill": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/codestral-mamba-latest": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "source": "https://mistral.ai/technology/",
+ "supports_assistant_prefill": true,
+ "supports_tool_choice": true
+ },
+ "mistral/devstral-medium-2507": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "mistral",
"max_input_tokens": 128000,
"max_output_tokens": 128000,
- "input_cost_per_token": 1.00002e-06,
- "input_dbu_cost_per_token": 1.4286e-05,
- "output_cost_per_token": 2.99999e-06,
- "output_dbu_cost_per_token": 4.2857e-05,
- "litellm_provider": "databricks",
+ "max_tokens": 128000,
"mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
+ "output_cost_per_token": 2e-06,
+ "source": "https://mistral.ai/news/devstral",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
"supports_tool_choice": true
},
- "databricks/databricks-llama-2-70b-chat": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5.0001e-07,
- "input_dbu_cost_per_token": 7.143e-06,
- "output_cost_per_token": 1.5e-06,
- "output_dbu_cost_per_token": 2.1429e-05,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
- "supports_tool_choice": true
- },
- "databricks/databricks-mixtral-8x7b-instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 5.0001e-07,
- "input_dbu_cost_per_token": 7.143e-06,
- "output_cost_per_token": 9.9902e-07,
- "output_dbu_cost_per_token": 1.4286e-05,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
- "supports_tool_choice": true
- },
- "databricks/databricks-mpt-30b-instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 9.9902e-07,
- "input_dbu_cost_per_token": 1.4286e-05,
- "output_cost_per_token": 9.9902e-07,
- "output_dbu_cost_per_token": 1.4286e-05,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
- "supports_tool_choice": true
- },
- "databricks/databricks-mpt-7b-instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 5.0001e-07,
- "input_dbu_cost_per_token": 7.143e-06,
- "output_cost_per_token": 0.0,
- "output_dbu_cost_per_token": 0.0,
- "litellm_provider": "databricks",
- "mode": "chat",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- },
- "supports_tool_choice": true
- },
- "databricks/databricks-bge-large-en": {
- "max_tokens": 512,
- "max_input_tokens": 512,
- "output_vector_size": 1024,
- "input_cost_per_token": 1.0003e-07,
- "input_dbu_cost_per_token": 1.429e-06,
- "output_cost_per_token": 0.0,
- "output_dbu_cost_per_token": 0.0,
- "litellm_provider": "databricks",
- "mode": "embedding",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- }
- },
- "databricks/databricks-gte-large-en": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "output_vector_size": 1024,
- "input_cost_per_token": 1.2999e-07,
- "input_dbu_cost_per_token": 1.857e-06,
- "output_cost_per_token": 0.0,
- "output_dbu_cost_per_token": 0.0,
- "litellm_provider": "databricks",
- "mode": "embedding",
- "source": "https://www.databricks.com/product/pricing/foundation-model-serving",
- "metadata": {
- "notes": "Input/output cost per token is dbu cost * $0.070, based on databricks Llama 3.1 70B conversion. Number provided for reference, '*_dbu_cost_per_token' used in actual calculation."
- }
- },
- "sambanova/Meta-Llama-3.1-8B-Instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
+ "mistral/devstral-small-2505": {
"input_cost_per_token": 1e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "sambanova",
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
"mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.sambanova.ai/plans/pricing"
- },
- "sambanova/Meta-Llama-3.1-405B-Instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 1e-05,
- "litellm_provider": "sambanova",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.sambanova.ai/plans/pricing"
- },
- "sambanova/Meta-Llama-3.2-1B-Instruct": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 4e-08,
- "output_cost_per_token": 8e-08,
- "litellm_provider": "sambanova",
- "mode": "chat",
- "source": "https://cloud.sambanova.ai/plans/pricing"
- },
- "sambanova/Meta-Llama-3.2-3B-Instruct": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 4096,
- "input_cost_per_token": 8e-08,
- "output_cost_per_token": 1.6e-07,
- "litellm_provider": "sambanova",
- "mode": "chat",
- "source": "https://cloud.sambanova.ai/plans/pricing"
- },
- "sambanova/Llama-4-Maverick-17B-128E-Instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 6.3e-07,
- "output_cost_per_token": 1.8e-06,
- "litellm_provider": "sambanova",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "source": "https://cloud.sambanova.ai/plans/pricing",
- "metadata": {
- "notes": "For vision models, images are converted to 6432 input tokens and are billed at that amount"
- }
- },
- "sambanova/Llama-4-Scout-17B-16E-Instruct": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 4e-07,
- "output_cost_per_token": 7e-07,
- "litellm_provider": "sambanova",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_response_schema": true,
- "source": "https://cloud.sambanova.ai/plans/pricing",
- "metadata": {
- "notes": "For vision models, images are converted to 6432 input tokens and are billed at that amount"
- }
- },
- "sambanova/Meta-Llama-3.3-70B-Instruct": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 6e-07,
- "output_cost_per_token": 1.2e-06,
- "litellm_provider": "sambanova",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_tool_choice": true,
- "source": "https://cloud.sambanova.ai/plans/pricing"
- },
- "sambanova/Meta-Llama-Guard-3-8B": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 3e-07,
"output_cost_per_token": 3e-07,
- "litellm_provider": "sambanova",
- "mode": "chat",
- "source": "https://cloud.sambanova.ai/plans/pricing"
+ "source": "https://mistral.ai/news/devstral",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
},
- "sambanova/Qwen3-32B": {
- "max_tokens": 8192,
+ "mistral/devstral-small-2507": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "source": "https://mistral.ai/news/devstral",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/magistral-medium-2506": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 40000,
+ "max_output_tokens": 40000,
+ "max_tokens": 40000,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://mistral.ai/news/magistral",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/magistral-medium-latest": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 40000,
+ "max_output_tokens": 40000,
+ "max_tokens": 40000,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://mistral.ai/news/magistral",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/magistral-small-2506": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 40000,
+ "max_output_tokens": 40000,
+ "max_tokens": 40000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://mistral.ai/pricing#api-pricing",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/magistral-small-latest": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 40000,
+ "max_output_tokens": 40000,
+ "max_tokens": 40000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://mistral.ai/pricing#api-pricing",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-embed": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "mistral",
"max_input_tokens": 8192,
- "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding"
+ },
+ "mistral/mistral-large-2402": {
+ "input_cost_per_token": 4e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-large-2407": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 9e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-large-2411": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-large-latest": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-medium": {
+ "input_cost_per_token": 2.7e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 8.1e-06,
+ "supports_assistant_prefill": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-medium-2312": {
+ "input_cost_per_token": 2.7e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 8.1e-06,
+ "supports_assistant_prefill": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-medium-2505": {
"input_cost_per_token": 4e-07,
- "output_cost_per_token": 8e-07,
- "litellm_provider": "sambanova",
+ "litellm_provider": "mistral",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-medium-latest": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-small": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-small-latest": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/mistral-tiny": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_assistant_prefill": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/open-codestral-mamba": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "source": "https://mistral.ai/technology/",
+ "supports_assistant_prefill": true,
+ "supports_tool_choice": true
+ },
+ "mistral/open-mistral-7b": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_assistant_prefill": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/open-mistral-nemo": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "source": "https://mistral.ai/technology/",
+ "supports_assistant_prefill": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/open-mistral-nemo-2407": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "source": "https://mistral.ai/technology/",
+ "supports_assistant_prefill": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/open-mixtral-8x22b": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 65336,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/open-mixtral-8x7b": {
+ "input_cost_per_token": 7e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "mistral/pixtral-12b-2409": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "mistral/pixtral-large-2411": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "mistral/pixtral-large-latest": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "mistral",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "moonshot/kimi-k2-0711-preview": {
+ "cache_read_input_token_cost": 1.5e-07,
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing/chat#generation-model-kimi-k2",
"supports_function_calling": true,
"supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "moonshot/kimi-latest": {
+ "cache_read_input_token_cost": 1.5e-07,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "moonshot/kimi-latest-128k": {
+ "cache_read_input_token_cost": 1.5e-07,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "moonshot/kimi-latest-32k": {
+ "cache_read_input_token_cost": 1.5e-07,
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "moonshot/kimi-latest-8k": {
+ "cache_read_input_token_cost": 1.5e-07,
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "moonshot/kimi-thinking-preview": {
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_vision": true
+ },
+ "moonshot/moonshot-v1-128k": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "moonshot/moonshot-v1-128k-0430": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "moonshot/moonshot-v1-128k-vision-preview": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "moonshot/moonshot-v1-32k": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "moonshot/moonshot-v1-32k-0430": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "moonshot/moonshot-v1-32k-vision-preview": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "moonshot/moonshot-v1-8k": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "moonshot/moonshot-v1-8k-0430": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "moonshot/moonshot-v1-8k-vision-preview": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "moonshot/moonshot-v1-auto": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "moonshot",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://platform.moonshot.ai/docs/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "morph/morph-v3-fast": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "morph",
+ "max_input_tokens": 16000,
+ "max_output_tokens": 16000,
+ "max_tokens": 16000,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_system_messages": true,
+ "supports_tool_choice": false,
+ "supports_vision": false
+ },
+ "morph/morph-v3-large": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "morph",
+ "max_input_tokens": 16000,
+ "max_output_tokens": 16000,
+ "max_tokens": 16000,
+ "mode": "chat",
+ "output_cost_per_token": 1.9e-06,
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_system_messages": true,
+ "supports_tool_choice": false,
+ "supports_vision": false
+ },
+ "multimodalembedding": {
+ "input_cost_per_character": 2e-07,
+ "input_cost_per_image": 0.0001,
+ "input_cost_per_token": 8e-07,
+ "input_cost_per_video_per_second": 0.0005,
+ "input_cost_per_video_per_second_above_15s_interval": 0.002,
+ "input_cost_per_video_per_second_above_8s_interval": 0.001,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models",
+ "supported_endpoints": [
+ "/v1/embeddings"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "video"
+ ]
+ },
+ "multimodalembedding@001": {
+ "input_cost_per_character": 2e-07,
+ "input_cost_per_image": 0.0001,
+ "input_cost_per_token": 8e-07,
+ "input_cost_per_video_per_second": 0.0005,
+ "input_cost_per_video_per_second_above_15s_interval": 0.002,
+ "input_cost_per_video_per_second_above_8s_interval": 0.001,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models",
+ "supported_endpoints": [
+ "/v1/embeddings"
+ ],
+ "supported_modalities": [
+ "text",
+ "image",
+ "video"
+ ]
+ },
+ "nscale/Qwen/QwQ-32B": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "nscale",
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/Qwen/Qwen2.5-Coder-32B-Instruct": {
+ "input_cost_per_token": 6e-08,
+ "litellm_provider": "nscale",
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/Qwen/Qwen2.5-Coder-3B-Instruct": {
+ "input_cost_per_token": 1e-08,
+ "litellm_provider": "nscale",
+ "mode": "chat",
+ "output_cost_per_token": 3e-08,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/Qwen/Qwen2.5-Coder-7B-Instruct": {
+ "input_cost_per_token": 1e-08,
+ "litellm_provider": "nscale",
+ "mode": "chat",
+ "output_cost_per_token": 3e-08,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/black-forest-labs/FLUX.1-schnell": {
+ "input_cost_per_pixel": 1.3e-09,
+ "litellm_provider": "nscale",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#image-models",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "nscale/deepseek-ai/DeepSeek-R1-Distill-Llama-70B": {
+ "input_cost_per_token": 3.75e-07,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $0.75/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 3.75e-07,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/deepseek-ai/DeepSeek-R1-Distill-Llama-8B": {
+ "input_cost_per_token": 2.5e-08,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $0.05/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-08,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B": {
+ "input_cost_per_token": 9e-08,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $0.18/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 9e-08,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B": {
+ "input_cost_per_token": 7e-08,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $0.14/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 7e-08,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $0.30/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $0.40/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/meta-llama/Llama-3.1-8B-Instruct": {
+ "input_cost_per_token": 3e-08,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $0.06/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 3e-08,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/meta-llama/Llama-3.3-70B-Instruct": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $0.40/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/meta-llama/Llama-4-Scout-17B-16E-Instruct": {
+ "input_cost_per_token": 9e-08,
+ "litellm_provider": "nscale",
+ "mode": "chat",
+ "output_cost_per_token": 2.9e-07,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/mistralai/mixtral-8x22b-instruct-v0.1": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "nscale",
+ "metadata": {
+ "notes": "Pricing listed as $1.20/1M tokens total. Assumed 50/50 split for input/output."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ },
+ "nscale/stabilityai/stable-diffusion-xl-base-1.0": {
+ "input_cost_per_pixel": 3e-09,
+ "litellm_provider": "nscale",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0,
+ "source": "https://docs.nscale.com/docs/inference/serverless-models/current#image-models",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "o1": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
"supports_reasoning": true,
- "mode": "chat",
- "source": "https://cloud.sambanova.ai/plans/pricing"
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "sambanova/QwQ-32B": {
- "max_tokens": 16384,
- "max_input_tokens": 16384,
- "max_output_tokens": 16384,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1e-06,
- "litellm_provider": "sambanova",
+ "o1-2024-12-17": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
"mode": "chat",
- "source": "https://cloud.sambanova.ai/plans/pricing"
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
},
- "sambanova/Qwen2-Audio-7B-Instruct": {
- "max_tokens": 4096,
+ "o1-mini": {
+ "cache_read_input_token_cost": 5.5e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_vision": true
+ },
+ "o1-mini-2024-09-12": {
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-05,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_vision": true
+ },
+ "o1-preview": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_vision": true
+ },
+ "o1-preview-2024-09-12": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_vision": true
+ },
+ "o1-pro": {
+ "input_cost_per_token": 0.00015,
+ "input_cost_per_token_batches": 7.5e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 0.0006,
+ "output_cost_per_token_batches": 0.0003,
+ "supported_endpoints": [
+ "/v1/responses",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": false,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o1-pro-2025-03-19": {
+ "input_cost_per_token": 0.00015,
+ "input_cost_per_token_batches": 7.5e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 0.0006,
+ "output_cost_per_token_batches": 0.0003,
+ "supported_endpoints": [
+ "/v1/responses",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": false,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o3": {
+ "cache_read_input_token_cost": 5e-07,
+ "cache_read_input_token_cost_flex": 2.5e-07,
+ "cache_read_input_token_cost_priority": 8.75e-07,
+ "input_cost_per_token": 2e-06,
+ "input_cost_per_token_flex": 1e-06,
+ "input_cost_per_token_priority": 3.5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "output_cost_per_token_flex": 4e-06,
+ "output_cost_per_token_priority": 1.4e-05,
+ "supported_endpoints": [
+ "/v1/responses",
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o3-2025-04-16": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supported_endpoints": [
+ "/v1/responses",
+ "/v1/chat/completions",
+ "/v1/completions",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o3-deep-research": {
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_token": 1e-05,
+ "input_cost_per_token_batches": 5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 4e-05,
+ "output_cost_per_token_batches": 2e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o3-deep-research-2025-06-26": {
+ "cache_read_input_token_cost": 2.5e-06,
+ "input_cost_per_token": 1e-05,
+ "input_cost_per_token_batches": 5e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 4e-05,
+ "output_cost_per_token_batches": 2e-05,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o3-mini": {
+ "cache_read_input_token_cost": 5.5e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "o3-mini-2025-01-31": {
+ "cache_read_input_token_cost": 5.5e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "o3-pro": {
+ "input_cost_per_token": 2e-05,
+ "input_cost_per_token_batches": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 8e-05,
+ "output_cost_per_token_batches": 4e-05,
+ "supported_endpoints": [
+ "/v1/responses",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o3-pro-2025-06-10": {
+ "input_cost_per_token": 2e-05,
+ "input_cost_per_token_batches": 1e-05,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 8e-05,
+ "output_cost_per_token_batches": 4e-05,
+ "supported_endpoints": [
+ "/v1/responses",
+ "/v1/batch"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o4-mini": {
+ "cache_read_input_token_cost": 2.75e-07,
+ "cache_read_input_token_cost_flex": 1.375e-07,
+ "cache_read_input_token_cost_priority": 5e-07,
+ "input_cost_per_token": 1.1e-06,
+ "input_cost_per_token_flex": 5.5e-07,
+ "input_cost_per_token_priority": 2e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "output_cost_per_token_flex": 2.2e-06,
+ "output_cost_per_token_priority": 8e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o4-mini-2025-04-16": {
+ "cache_read_input_token_cost": 2.75e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o4-mini-deep-research": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "input_cost_per_token_batches": 1e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 8e-06,
+ "output_cost_per_token_batches": 4e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "o4-mini-deep-research-2025-06-26": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "input_cost_per_token_batches": 1e-06,
+ "litellm_provider": "openai",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "responses",
+ "output_cost_per_token": 8e-06,
+ "output_cost_per_token_batches": 4e-06,
+ "supported_endpoints": [
+ "/v1/chat/completions",
+ "/v1/batch",
+ "/v1/responses"
+ ],
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_function_calling": true,
+ "supports_native_streaming": true,
+ "supports_parallel_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "oci/meta.llama-3.1-405b-instruct": {
+ "input_cost_per_token": 1.068e-05,
+ "litellm_provider": "oci",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.068e-05,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/meta.llama-3.2-90b-vision-instruct": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "oci",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/meta.llama-3.3-70b-instruct": {
+ "input_cost_per_token": 7.2e-07,
+ "litellm_provider": "oci",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 7.2e-07,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/meta.llama-4-maverick-17b-128e-instruct-fp8": {
+ "input_cost_per_token": 7.2e-07,
+ "litellm_provider": "oci",
+ "max_input_tokens": 512000,
+ "max_output_tokens": 4000,
+ "max_tokens": 512000,
+ "mode": "chat",
+ "output_cost_per_token": 7.2e-07,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/meta.llama-4-scout-17b-16e-instruct": {
+ "input_cost_per_token": 7.2e-07,
+ "litellm_provider": "oci",
+ "max_input_tokens": 192000,
+ "max_output_tokens": 4000,
+ "max_tokens": 192000,
+ "mode": "chat",
+ "output_cost_per_token": 7.2e-07,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/xai.grok-3": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "oci",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/xai.grok-3-fast": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "oci",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-05,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/xai.grok-3-mini": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "oci",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/xai.grok-3-mini-fast": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "oci",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/xai.grok-4": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "oci",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/cohere.command-latest": {
+ "input_cost_per_token": 1.56e-06,
+ "litellm_provider": "oci",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.56e-06,
+ "source": "https://www.oracle.com/cloud/ai/generative-ai/pricing/",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/cohere.command-a-03-2025": {
+ "input_cost_per_token": 1.56e-06,
+ "litellm_provider": "oci",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 4000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.56e-06,
+ "source": "https://www.oracle.com/cloud/ai/generative-ai/pricing/",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "oci/cohere.command-plus-latest": {
+ "input_cost_per_token": 1.56e-06,
+ "litellm_provider": "oci",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.56e-06,
+ "source": "https://www.oracle.com/cloud/ai/generative-ai/pricing/",
+ "supports_function_calling": true,
+ "supports_response_schema": false
+ },
+ "ollama/codegeex4": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 8192,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": false
+ },
+ "ollama/codegemma": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "completion",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/codellama": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
"max_input_tokens": 4096,
"max_output_tokens": 4096,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 0.0001,
- "litellm_provider": "sambanova",
+ "max_tokens": 4096,
+ "mode": "completion",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/deepseek-coder-v2-base": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "completion",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/deepseek-coder-v2-instruct": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 8192,
+ "max_tokens": 32768,
"mode": "chat",
- "supports_audio_input": true,
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/deepseek-coder-v2-lite-base": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "completion",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/deepseek-coder-v2-lite-instruct": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 8192,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/deepseek-v3.1:671b-cloud" : {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "max_tokens": 163840,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/gpt-oss:120b-cloud" : {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/gpt-oss:20b-cloud" : {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/internlm2_5-20b-chat": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 8192,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/llama2": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/llama2-uncensored": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "completion",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/llama2:13b": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/llama2:70b": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/llama2:7b": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/llama3": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/llama3.1": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/llama3:70b": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/llama3:8b": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/mistral": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "completion",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/mistral-7B-Instruct-v0.1": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/mistral-7B-Instruct-v0.2": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/mistral-large-instruct-2407": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 8192,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/mixtral-8x22B-Instruct-v0.1": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/mixtral-8x7B-Instruct-v0.1": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/orca-mini": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "completion",
+ "output_cost_per_token": 0.0
+ },
+ "ollama/qwen3-coder:480b-cloud": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_function_calling": true
+ },
+ "ollama/vicuna": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "ollama",
+ "max_input_tokens": 2048,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "completion",
+ "output_cost_per_token": 0.0
+ },
+ "omni-moderation-2024-09-26": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "openai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 0,
+ "max_tokens": 32768,
+ "mode": "moderation",
+ "output_cost_per_token": 0.0
+ },
+ "omni-moderation-latest": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "openai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 0,
+ "max_tokens": 32768,
+ "mode": "moderation",
+ "output_cost_per_token": 0.0
+ },
+ "omni-moderation-latest-intents": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "openai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 0,
+ "max_tokens": 32768,
+ "mode": "moderation",
+ "output_cost_per_token": 0.0
+ },
+ "openai.gpt-oss-120b-1:0": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "openai.gpt-oss-20b-1:0": {
+ "input_cost_per_token": 7e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/anthropic/claude-2": {
+ "input_cost_per_token": 1.102e-05,
+ "litellm_provider": "openrouter",
+ "max_output_tokens": 8191,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 3.268e-05,
+ "supports_tool_choice": true
+ },
+ "openrouter/anthropic/claude-3-5-haiku": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/anthropic/claude-3-5-haiku-20241022": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "tool_use_system_prompt_tokens": 264
+ },
+ "openrouter/anthropic/claude-3-haiku": {
+ "input_cost_per_image": 0.0004,
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/anthropic/claude-3-haiku-20240307": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 264
+ },
+ "openrouter/anthropic/claude-3-opus": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 395
+ },
+ "openrouter/anthropic/claude-3-sonnet": {
+ "input_cost_per_image": 0.0048,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/anthropic/claude-3.5-sonnet": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "openrouter/anthropic/claude-3.5-sonnet:beta": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "openrouter/anthropic/claude-3.7-sonnet": {
+ "input_cost_per_image": 0.0048,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "openrouter/anthropic/claude-3.7-sonnet:beta": {
+ "input_cost_per_image": 0.0048,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "openrouter/anthropic/claude-instant-v1": {
+ "input_cost_per_token": 1.63e-06,
+ "litellm_provider": "openrouter",
+ "max_output_tokens": 8191,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 5.51e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/anthropic/claude-opus-4": {
+ "input_cost_per_image": 0.0048,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "openrouter/anthropic/claude-opus-4.1": {
+ "input_cost_per_image": 0.0048,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "openrouter/anthropic/claude-sonnet-4": {
+ "input_cost_per_image": 0.0048,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "openrouter/anthropic/claude-sonnet-4.5": {
+ "input_cost_per_image": 0.0048,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 1000000,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "openrouter/bytedance/ui-tars-1.5-7b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://openrouter.ai/api/v1/models/bytedance/ui-tars-1.5-7b",
+ "supports_tool_choice": true
+ },
+ "openrouter/cognitivecomputations/dolphin-mixtral-8x7b": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 32769,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/cohere/command-r-plus": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_tool_choice": true
+ },
+ "openrouter/databricks/dbrx-instruct": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/deepseek/deepseek-chat": {
+ "input_cost_per_token": 1.4e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07,
+ "supports_prompt_caching": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/deepseek/deepseek-chat-v3-0324": {
+ "input_cost_per_token": 1.4e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07,
+ "supports_prompt_caching": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/deepseek/deepseek-chat-v3.1": {
+ "input_cost_per_token": 2e-07,
+ "input_cost_per_token_cache_hit": 2e-08,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 163840,
+ "max_output_tokens": 163840,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 8e-07,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/deepseek/deepseek-coder": {
+ "input_cost_per_token": 1.4e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 66000,
+ "max_output_tokens": 4096,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07,
+ "supports_prompt_caching": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/deepseek/deepseek-r1": {
+ "input_cost_per_token": 5.5e-07,
+ "input_cost_per_token_cache_hit": 1.4e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 65336,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.19e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/deepseek/deepseek-r1-0528": {
+ "input_cost_per_token": 5e-07,
+ "input_cost_per_token_cache_hit": 1.4e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 65336,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.15e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/fireworks/firellava-13b": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/google/gemini-2.0-flash-001": {
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "openrouter",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/google/gemini-2.5-flash": {
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "openrouter",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-06,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/google/gemini-2.5-pro": {
+ "input_cost_per_audio_token": 7e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "openrouter",
+ "max_audio_length_hours": 8.4,
+ "max_audio_per_prompt": 1,
+ "max_images_per_prompt": 3000,
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_pdf_size_mb": 30,
+ "max_tokens": 8192,
+ "max_video_length": 1,
+ "max_videos_per_prompt": 10,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/google/gemini-pro-1.5": {
+ "input_cost_per_image": 0.00265,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/google/gemini-pro-vision": {
+ "input_cost_per_image": 0.0025,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 45875,
+ "mode": "chat",
+ "output_cost_per_token": 3.75e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/google/palm-2-chat-bison": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 25804,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/google/palm-2-codechat-bison": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 20070,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/gryphe/mythomax-l2-13b": {
+ "input_cost_per_token": 1.875e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.875e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/jondurbin/airoboros-l2-70b-2.1": {
+ "input_cost_per_token": 1.3875e-05,
+ "litellm_provider": "openrouter",
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.3875e-05,
+ "supports_tool_choice": true
+ },
+ "openrouter/mancer/weaver": {
+ "input_cost_per_token": 5.625e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8000,
+ "mode": "chat",
+ "output_cost_per_token": 5.625e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/meta-llama/codellama-34b-instruct": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/meta-llama/llama-2-13b-chat": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/meta-llama/llama-2-70b-chat": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/meta-llama/llama-3-70b-instruct": {
+ "input_cost_per_token": 5.9e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 7.9e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/meta-llama/llama-3-70b-instruct:nitro": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/meta-llama/llama-3-8b-instruct:extended": {
+ "input_cost_per_token": 2.25e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 2.25e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/meta-llama/llama-3-8b-instruct:free": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_tool_choice": true
+ },
+ "openrouter/microsoft/wizardlm-2-8x22b:nitro": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/mistralai/mistral-7b-instruct": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.3e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/mistralai/mistral-7b-instruct:free": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "supports_tool_choice": true
+ },
+ "openrouter/mistralai/mistral-large": {
+ "input_cost_per_token": 8e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-05,
+ "supports_tool_choice": true
+ },
+ "openrouter/mistralai/mistral-small-3.1-24b-instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/mistralai/mistral-small-3.2-24b-instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/mistralai/mixtral-8x22b-instruct": {
+ "input_cost_per_token": 6.5e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 6.5e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/nousresearch/nous-hermes-llama2-13b": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "openrouter",
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-3.5-turbo": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 4095,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-3.5-turbo-16k": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 16383,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-4": {
+ "input_cost_per_token": 3e-05,
+ "litellm_provider": "openrouter",
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-4-vision-preview": {
+ "input_cost_per_image": 0.01445,
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "openrouter",
+ "max_tokens": 130000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-4.1": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-4.1-2025-04-14": {
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-4.1-mini": {
+ "cache_read_input_token_cost": 1e-07,
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-4.1-mini-2025-04-14": {
+ "cache_read_input_token_cost": 1e-07,
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-4.1-nano": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-4.1-nano-2025-04-14": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-4o": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-4o-2024-05-13": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/gpt-5-chat": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-5-codex": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-5": {
+ "cache_read_input_token_cost": 1.25e-07,
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-5-mini": {
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-5-nano": {
+ "cache_read_input_token_cost": 5e-09,
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 272000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text"
+ ],
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-oss-120b": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 8e-07,
+ "source": "https://openrouter.ai/openai/gpt-oss-120b",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/gpt-oss-20b": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 8e-07,
+ "source": "https://openrouter.ai/openai/gpt-oss-20b",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "openrouter/openai/o1": {
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 100000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "openrouter/openai/o1-mini": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "openrouter/openai/o1-mini-2024-09-12": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "openrouter/openai/o1-preview": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "openrouter/openai/o1-preview-2024-09-12": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "openrouter/openai/o3-mini": {
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "openrouter/openai/o3-mini-high": {
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 65536,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "openrouter/pygmalionai/mythalion-13b": {
+ "input_cost_per_token": 1.875e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.875e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/qwen/qwen-2.5-coder-32b-instruct": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 33792,
+ "max_output_tokens": 33792,
+ "max_tokens": 33792,
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/qwen/qwen-vl-plus": {
+ "input_cost_per_token": 2.1e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 2048,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 6.3e-07,
+ "supports_tool_choice": true
+ },
+ "openrouter/qwen/qwen3-coder": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 1000000,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "source": "https://openrouter.ai/qwen/qwen3-coder",
+ "supports_tool_choice": true
+ },
+ "openrouter/switchpoint/router": {
+ "input_cost_per_token": 8.5e-07,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3.4e-06,
+ "source": "https://openrouter.ai/switchpoint/router",
+ "supports_tool_choice": true
+ },
+ "openrouter/undi95/remm-slerp-l2-13b": {
+ "input_cost_per_token": 1.875e-06,
+ "litellm_provider": "openrouter",
+ "max_tokens": 6144,
+ "mode": "chat",
+ "output_cost_per_token": 1.875e-06,
+ "supports_tool_choice": true
+ },
+ "openrouter/x-ai/grok-4": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://openrouter.ai/x-ai/grok-4",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "openrouter/x-ai/grok-4-fast:free": {
+ "input_cost_per_token": 0,
+ "litellm_provider": "openrouter",
+ "max_input_tokens": 2000000,
+ "max_output_tokens": 30000,
+ "max_tokens": 2000000,
+ "mode": "chat",
+ "output_cost_per_token": 0,
+ "source": "https://openrouter.ai/x-ai/grok-4-fast:free",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_web_search": false
+ },
+ "ovhcloud/DeepSeek-R1-Distill-Llama-70B": {
+ "input_cost_per_token": 6.7e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 131000,
+ "max_output_tokens": 131000,
+ "max_tokens": 131000,
+ "mode": "chat",
+ "output_cost_per_token": 6.7e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/deepseek-r1-distill-llama-70b",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "ovhcloud/Llama-3.1-8B-Instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 131000,
+ "max_output_tokens": 131000,
+ "max_tokens": 131000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/llama-3-1-8b-instruct",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "ovhcloud/Meta-Llama-3_1-70B-Instruct": {
+ "input_cost_per_token": 6.7e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 131000,
+ "max_output_tokens": 131000,
+ "max_tokens": 131000,
+ "mode": "chat",
+ "output_cost_per_token": 6.7e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/meta-llama-3-1-70b-instruct",
+ "supports_function_calling": false,
+ "supports_response_schema": false,
+ "supports_tool_choice": false
+ },
+ "ovhcloud/Meta-Llama-3_3-70B-Instruct": {
+ "input_cost_per_token": 6.7e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 131000,
+ "max_output_tokens": 131000,
+ "max_tokens": 131000,
+ "mode": "chat",
+ "output_cost_per_token": 6.7e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/meta-llama-3-3-70b-instruct",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "ovhcloud/Mistral-7B-Instruct-v0.3": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 127000,
+ "max_output_tokens": 127000,
+ "max_tokens": 127000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/mistral-7b-instruct-v0-3",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "ovhcloud/Mistral-Nemo-Instruct-2407": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 118000,
+ "max_output_tokens": 118000,
+ "max_tokens": 118000,
+ "mode": "chat",
+ "output_cost_per_token": 1.3e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/mistral-nemo-instruct-2407",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "ovhcloud/Mistral-Small-3.2-24B-Instruct-2506": {
+ "input_cost_per_token": 9e-08,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/mistral-small-3-2-24b-instruct-2506",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "ovhcloud/Mixtral-8x7B-Instruct-v0.1": {
+ "input_cost_per_token": 6.3e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 6.3e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/mixtral-8x7b-instruct-v0-1",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "ovhcloud/Qwen2.5-Coder-32B-Instruct": {
+ "input_cost_per_token": 8.7e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 8.7e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/qwen2-5-coder-32b-instruct",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "ovhcloud/Qwen2.5-VL-72B-Instruct": {
+ "input_cost_per_token": 9.1e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 9.1e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/qwen2-5-vl-72b-instruct",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "ovhcloud/Qwen3-32B": {
+ "input_cost_per_token": 8e-08,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 2.3e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/qwen3-32b",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "ovhcloud/gpt-oss-120b": {
+ "input_cost_per_token": 8e-08,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 131000,
+ "max_output_tokens": 131000,
+ "max_tokens": 131000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/gpt-oss-120b",
+ "supports_function_calling": false,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "ovhcloud/gpt-oss-20b": {
+ "input_cost_per_token": 4e-08,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 131000,
+ "max_output_tokens": 131000,
+ "max_tokens": 131000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/gpt-oss-20b",
+ "supports_function_calling": false,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "ovhcloud/llava-v1.6-mistral-7b-hf": {
+ "input_cost_per_token": 2.9e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 2.9e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/llava-next-mistral-7b",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "ovhcloud/mamba-codestral-7B-v0.1": {
+ "input_cost_per_token": 1.9e-07,
+ "litellm_provider": "ovhcloud",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.9e-07,
+ "source": "https://endpoints.ai.cloud.ovh.net/models/mamba-codestral-7b-v0-1",
+ "supports_function_calling": false,
+ "supports_response_schema": true,
+ "supports_tool_choice": false
+ },
+ "palm/chat-bison": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "palm",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "palm/chat-bison-001": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "palm",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "palm/text-bison": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "palm",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "palm/text-bison-001": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "palm",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "palm/text-bison-safety-off": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "palm",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "palm/text-bison-safety-recitation-off": {
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "palm",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "perplexity/codellama-34b-instruct": {
+ "input_cost_per_token": 3.5e-07,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.4e-06
+ },
+ "perplexity/codellama-70b-instruct": {
+ "input_cost_per_token": 7e-07,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-06
+ },
+ "perplexity/llama-2-70b-chat": {
+ "input_cost_per_token": 7e-07,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-06
+ },
+ "perplexity/llama-3.1-70b-instruct": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06
+ },
+ "perplexity/llama-3.1-8b-instruct": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07
+ },
+ "perplexity/llama-3.1-sonar-huge-128k-online": {
+ "deprecation_date": "2025-02-22",
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 127072,
+ "max_output_tokens": 127072,
+ "max_tokens": 127072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06
+ },
+ "perplexity/llama-3.1-sonar-large-128k-chat": {
+ "deprecation_date": "2025-02-22",
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06
+ },
+ "perplexity/llama-3.1-sonar-large-128k-online": {
+ "deprecation_date": "2025-02-22",
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 127072,
+ "max_output_tokens": 127072,
+ "max_tokens": 127072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06
+ },
+ "perplexity/llama-3.1-sonar-small-128k-chat": {
+ "deprecation_date": "2025-02-22",
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07
+ },
+ "perplexity/llama-3.1-sonar-small-128k-online": {
+ "deprecation_date": "2025-02-22",
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 127072,
+ "max_output_tokens": 127072,
+ "max_tokens": 127072,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07
+ },
+ "perplexity/mistral-7b-instruct": {
+ "input_cost_per_token": 7e-08,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07
+ },
+ "perplexity/mixtral-8x7b-instruct": {
+ "input_cost_per_token": 7e-08,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07
+ },
+ "perplexity/pplx-70b-chat": {
+ "input_cost_per_token": 7e-07,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-06
+ },
+ "perplexity/pplx-70b-online": {
+ "input_cost_per_request": 0.005,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-06
+ },
+ "perplexity/pplx-7b-chat": {
+ "input_cost_per_token": 7e-08,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07
+ },
+ "perplexity/pplx-7b-online": {
+ "input_cost_per_request": 0.005,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07
+ },
+ "perplexity/sonar": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.012,
+ "search_context_size_low": 0.005,
+ "search_context_size_medium": 0.008
+ },
+ "supports_web_search": true
+ },
+ "perplexity/sonar-deep-research": {
+ "citation_cost_per_token": 2e-06,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_reasoning_token": 3e-06,
+ "output_cost_per_token": 8e-06,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.005,
+ "search_context_size_low": 0.005,
+ "search_context_size_medium": 0.005
+ },
+ "supports_reasoning": true,
+ "supports_web_search": true
+ },
+ "perplexity/sonar-medium-chat": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-06
+ },
+ "perplexity/sonar-medium-online": {
+ "input_cost_per_request": 0.005,
+ "input_cost_per_token": 0,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 12000,
+ "max_output_tokens": 12000,
+ "max_tokens": 12000,
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-06
+ },
+ "perplexity/sonar-pro": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8000,
+ "max_tokens": 8000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.014,
+ "search_context_size_low": 0.006,
+ "search_context_size_medium": 0.01
+ },
+ "supports_web_search": true
+ },
+ "perplexity/sonar-reasoning": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.014,
+ "search_context_size_low": 0.005,
+ "search_context_size_medium": 0.008
+ },
+ "supports_reasoning": true,
+ "supports_web_search": true
+ },
+ "perplexity/sonar-reasoning-pro": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.014,
+ "search_context_size_low": 0.006,
+ "search_context_size_medium": 0.01
+ },
+ "supports_reasoning": true,
+ "supports_web_search": true
+ },
+ "perplexity/sonar-small-chat": {
+ "input_cost_per_token": 7e-08,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07
+ },
+ "perplexity/sonar-small-online": {
+ "input_cost_per_request": 0.005,
+ "input_cost_per_token": 0,
+ "litellm_provider": "perplexity",
+ "max_input_tokens": 12000,
+ "max_output_tokens": 12000,
+ "max_tokens": 12000,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07
+ },
+ "qwen.qwen3-coder-480b-a35b-v1:0": {
+ "input_cost_per_token": 2.2e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 262000,
+ "max_output_tokens": 65536,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-06,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "qwen.qwen3-235b-a22b-2507-v1:0": {
+ "input_cost_per_token": 2.2e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 131072,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 8.8e-07,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "qwen.qwen3-coder-30b-a3b-v1:0": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 131072,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 6.0e-07,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "qwen.qwen3-32b-v1:0": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 6.0e-07,
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "recraft/recraftv2": {
+ "litellm_provider": "recraft",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.022,
+ "source": "https://www.recraft.ai/docs#pricing",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "recraft/recraftv3": {
+ "litellm_provider": "recraft",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.04,
+ "source": "https://www.recraft.ai/docs#pricing",
+ "supported_endpoints": [
+ "/v1/images/generations"
+ ]
+ },
+ "replicate/meta/llama-2-13b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-2-13b-chat": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-2-70b": {
+ "input_cost_per_token": 6.5e-07,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.75e-06,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-2-70b-chat": {
+ "input_cost_per_token": 6.5e-07,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.75e-06,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-2-7b": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-2-7b-chat": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-3-70b": {
+ "input_cost_per_token": 6.5e-07,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.75e-06,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-3-70b-instruct": {
+ "input_cost_per_token": 6.5e-07,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2.75e-06,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-3-8b": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 8086,
+ "max_output_tokens": 8086,
+ "max_tokens": 8086,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_tool_choice": true
+ },
+ "replicate/meta/llama-3-8b-instruct": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 8086,
+ "max_output_tokens": 8086,
+ "max_tokens": 8086,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_tool_choice": true
+ },
+ "replicate/mistralai/mistral-7b-instruct-v0.2": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_tool_choice": true
+ },
+ "replicate/mistralai/mistral-7b-v0.1": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-07,
+ "supports_tool_choice": true
+ },
+ "replicate/mistralai/mixtral-8x7b-instruct-v0.1": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "replicate",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06,
+ "supports_tool_choice": true
+ },
+ "rerank-english-v2.0": {
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_query_tokens": 2048,
+ "max_tokens": 4096,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "rerank-english-v3.0": {
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_query_tokens": 2048,
+ "max_tokens": 4096,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "rerank-multilingual-v2.0": {
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_query_tokens": 2048,
+ "max_tokens": 4096,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "rerank-multilingual-v3.0": {
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_query_tokens": 2048,
+ "max_tokens": 4096,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "rerank-v3.5": {
+ "input_cost_per_query": 0.002,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "cohere",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_query_tokens": 2048,
+ "max_tokens": 4096,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "nvidia_nim/nvidia/nv-rerankqa-mistral-4b-v3": {
+ "input_cost_per_query": 0.0,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "nvidia_nim",
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "nvidia_nim/nvidia/llama-3_2-nv-rerankqa-1b-v2": {
+ "input_cost_per_query": 0.0,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "nvidia_nim",
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "sagemaker/meta-textgeneration-llama-2-13b": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "sagemaker",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "completion",
+ "output_cost_per_token": 0.0
+ },
+ "sagemaker/meta-textgeneration-llama-2-13b-f": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "sagemaker",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "sagemaker/meta-textgeneration-llama-2-70b": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "sagemaker",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "completion",
+ "output_cost_per_token": 0.0
+ },
+ "sagemaker/meta-textgeneration-llama-2-70b-b-f": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "sagemaker",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "sagemaker/meta-textgeneration-llama-2-7b": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "sagemaker",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "completion",
+ "output_cost_per_token": 0.0
+ },
+ "sagemaker/meta-textgeneration-llama-2-7b-f": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "sagemaker",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "sambanova/DeepSeek-R1": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 7e-06,
"source": "https://cloud.sambanova.ai/plans/pricing"
},
"sambanova/DeepSeek-R1-Distill-Llama-70B": {
- "max_tokens": 131072,
+ "input_cost_per_token": 7e-07,
+ "litellm_provider": "sambanova",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
- "input_cost_per_token": 7e-07,
+ "max_tokens": 131072,
+ "mode": "chat",
"output_cost_per_token": 1.4e-06,
- "litellm_provider": "sambanova",
- "mode": "chat",
- "source": "https://cloud.sambanova.ai/plans/pricing"
- },
- "sambanova/DeepSeek-R1": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 5e-06,
- "output_cost_per_token": 7e-06,
- "litellm_provider": "sambanova",
- "mode": "chat",
"source": "https://cloud.sambanova.ai/plans/pricing"
},
"sambanova/DeepSeek-V3-0324": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4.5e-06,
+ "source": "https://cloud.sambanova.ai/plans/pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "sambanova/Llama-4-Maverick-17B-128E-Instruct": {
+ "input_cost_per_token": 6.3e-07,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "metadata": {
+ "notes": "For vision models, images are converted to 6432 input tokens and are billed at that amount"
+ },
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-06,
+ "source": "https://cloud.sambanova.ai/plans/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "sambanova/Llama-4-Scout-17B-16E-Instruct": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "metadata": {
+ "notes": "For vision models, images are converted to 6432 input tokens and are billed at that amount"
+ },
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
+ "source": "https://cloud.sambanova.ai/plans/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "sambanova/Meta-Llama-3.1-405B-Instruct": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "source": "https://cloud.sambanova.ai/plans/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "sambanova/Meta-Llama-3.1-8B-Instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://cloud.sambanova.ai/plans/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "sambanova/Meta-Llama-3.2-1B-Instruct": {
+ "input_cost_per_token": 4e-08,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 8e-08,
+ "source": "https://cloud.sambanova.ai/plans/pricing"
+ },
+ "sambanova/Meta-Llama-3.2-3B-Instruct": {
+ "input_cost_per_token": 8e-08,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-07,
+ "source": "https://cloud.sambanova.ai/plans/pricing"
+ },
+ "sambanova/Meta-Llama-3.3-70B-Instruct": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://cloud.sambanova.ai/plans/pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "sambanova/Meta-Llama-Guard-3-8B": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "source": "https://cloud.sambanova.ai/plans/pricing"
+ },
+ "sambanova/QwQ-32B": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 16384,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06,
+ "source": "https://cloud.sambanova.ai/plans/pricing"
+ },
+ "sambanova/Qwen2-Audio-7B-Instruct": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 0.0001,
+ "source": "https://cloud.sambanova.ai/plans/pricing",
+ "supports_audio_input": true
+ },
+ "sambanova/Qwen3-32B": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "sambanova",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 8e-07,
+ "source": "https://cloud.sambanova.ai/plans/pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "sambanova/DeepSeek-V3.1": {
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
@@ -16045,1391 +19093,4166 @@
"supports_reasoning": true,
"source": "https://cloud.sambanova.ai/plans/pricing"
},
- "assemblyai/nano": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00010278,
- "output_cost_per_second": 0.0,
- "litellm_provider": "assemblyai"
- },
- "assemblyai/best": {
- "mode": "audio_transcription",
- "input_cost_per_second": 3.333e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "assemblyai"
- },
- "jina-reranker-v2-base-multilingual": {
- "max_tokens": 1024,
- "max_input_tokens": 1024,
- "max_output_tokens": 1024,
- "max_document_chunks_per_query": 2048,
- "input_cost_per_token": 1.8e-08,
- "output_cost_per_token": 1.8e-08,
- "litellm_provider": "jina_ai",
- "mode": "rerank"
- },
- "snowflake/deepseek-r1": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "sambanova/gpt-oss-120b": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 3e-06,
+ "output_cost_per_token": 4.5e-06,
+ "litellm_provider": "sambanova",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
"supports_reasoning": true,
- "mode": "chat"
+ "source": "https://cloud.sambanova.ai/plans/pricing"
},
- "snowflake/snowflake-arctic": {
- "max_tokens": 4096,
- "max_input_tokens": 4096,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
+ "sample_spec": {
+ "code_interpreter_cost_per_session": 0.0,
+ "computer_use_input_cost_per_1k_tokens": 0.0,
+ "computer_use_output_cost_per_1k_tokens": 0.0,
+ "deprecation_date": "date when the model becomes deprecated in the format YYYY-MM-DD",
+ "file_search_cost_per_1k_calls": 0.0,
+ "file_search_cost_per_gb_per_day": 0.0,
+ "input_cost_per_audio_token": 0.0,
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "one of https://docs.litellm.ai/docs/providers",
+ "max_input_tokens": "max input tokens, if the provider specifies it. if not default to max_tokens",
+ "max_output_tokens": "max output tokens, if the provider specifies it. if not default to max_tokens",
+ "max_tokens": "LEGACY parameter. set to max_output_tokens if provider specifies it. IF not set to max_input_tokens, if provider specifies it.",
+ "mode": "one of: chat, embedding, completion, image_generation, audio_transcription, audio_speech, image_generation, moderation, rerank",
+ "output_cost_per_reasoning_token": 0.0,
+ "output_cost_per_token": 0.0,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.0,
+ "search_context_size_low": 0.0,
+ "search_context_size_medium": 0.0
+ },
+ "supported_regions": [
+ "global",
+ "us-west-2",
+ "eu-west-1",
+ "ap-southeast-1",
+ "ap-northeast-1"
+ ],
+ "supports_audio_input": true,
+ "supports_audio_output": true,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_vision": true,
+ "supports_web_search": true,
+ "vector_store_cost_per_gb_per_day": 0.0
},
"snowflake/claude-3-5-sonnet": {
- "supports_computer_use": true,
- "max_tokens": 18000,
+ "litellm_provider": "snowflake",
"max_input_tokens": 18000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
+ "max_tokens": 18000,
+ "mode": "chat",
+ "supports_computer_use": true
},
- "snowflake/mistral-large": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "max_output_tokens": 8192,
+ "snowflake/deepseek-r1": {
"litellm_provider": "snowflake",
- "mode": "chat"
+ "max_input_tokens": 32768,
+ "max_output_tokens": 8192,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "supports_reasoning": true
},
- "snowflake/mistral-large2": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
+ "snowflake/gemma-7b": {
"litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/reka-flash": {
- "max_tokens": 100000,
- "max_input_tokens": 100000,
+ "max_input_tokens": 8000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/reka-core": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/jamba-instruct": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/jamba-1.5-mini": {
- "max_tokens": 256000,
- "max_input_tokens": 256000,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "max_tokens": 8000,
"mode": "chat"
},
"snowflake/jamba-1.5-large": {
- "max_tokens": 256000,
+ "litellm_provider": "snowflake",
"max_input_tokens": 256000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "max_tokens": 256000,
"mode": "chat"
},
- "snowflake/mixtral-8x7b": {
- "max_tokens": 32000,
- "max_input_tokens": 32000,
- "max_output_tokens": 8192,
+ "snowflake/jamba-1.5-mini": {
"litellm_provider": "snowflake",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 8192,
+ "max_tokens": 256000,
+ "mode": "chat"
+ },
+ "snowflake/jamba-instruct": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 8192,
+ "max_tokens": 256000,
"mode": "chat"
},
"snowflake/llama2-70b-chat": {
- "max_tokens": 4096,
+ "litellm_provider": "snowflake",
"max_input_tokens": 4096,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/llama3-8b": {
- "max_tokens": 8000,
- "max_input_tokens": 8000,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "max_tokens": 4096,
"mode": "chat"
},
"snowflake/llama3-70b": {
- "max_tokens": 8000,
+ "litellm_provider": "snowflake",
"max_input_tokens": 8000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/llama3.1-8b": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/llama3.1-70b": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/llama3.3-70b": {
- "max_tokens": 128000,
- "max_input_tokens": 128000,
- "max_output_tokens": 8192,
- "litellm_provider": "snowflake",
- "mode": "chat"
- },
- "snowflake/snowflake-llama-3.3-70b": {
"max_tokens": 8000,
+ "mode": "chat"
+ },
+ "snowflake/llama3-8b": {
+ "litellm_provider": "snowflake",
"max_input_tokens": 8000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "max_tokens": 8000,
"mode": "chat"
},
"snowflake/llama3.1-405b": {
- "max_tokens": 128000,
+ "litellm_provider": "snowflake",
"max_input_tokens": 128000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "max_tokens": 128000,
"mode": "chat"
},
- "snowflake/snowflake-llama-3.1-405b": {
- "max_tokens": 8000,
- "max_input_tokens": 8000,
- "max_output_tokens": 8192,
+ "snowflake/llama3.1-70b": {
"litellm_provider": "snowflake",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat"
+ },
+ "snowflake/llama3.1-8b": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
"mode": "chat"
},
"snowflake/llama3.2-1b": {
- "max_tokens": 128000,
+ "litellm_provider": "snowflake",
"max_input_tokens": 128000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "max_tokens": 128000,
"mode": "chat"
},
"snowflake/llama3.2-3b": {
- "max_tokens": 128000,
+ "litellm_provider": "snowflake",
"max_input_tokens": 128000,
"max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat"
+ },
+ "snowflake/llama3.3-70b": {
"litellm_provider": "snowflake",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
"mode": "chat"
},
"snowflake/mistral-7b": {
- "max_tokens": 32000,
+ "litellm_provider": "snowflake",
"max_input_tokens": 32000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "max_tokens": 32000,
"mode": "chat"
},
- "snowflake/gemma-7b": {
- "max_tokens": 8000,
+ "snowflake/mistral-large": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8192,
+ "max_tokens": 32000,
+ "mode": "chat"
+ },
+ "snowflake/mistral-large2": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat"
+ },
+ "snowflake/mixtral-8x7b": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8192,
+ "max_tokens": 32000,
+ "mode": "chat"
+ },
+ "snowflake/reka-core": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8192,
+ "max_tokens": 32000,
+ "mode": "chat"
+ },
+ "snowflake/reka-flash": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 100000,
+ "max_output_tokens": 8192,
+ "max_tokens": 100000,
+ "mode": "chat"
+ },
+ "snowflake/snowflake-arctic": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 4096,
+ "max_output_tokens": 8192,
+ "max_tokens": 4096,
+ "mode": "chat"
+ },
+ "snowflake/snowflake-llama-3.1-405b": {
+ "litellm_provider": "snowflake",
"max_input_tokens": 8000,
"max_output_tokens": 8192,
- "litellm_provider": "snowflake",
+ "max_tokens": 8000,
"mode": "chat"
},
- "nscale/meta-llama/Llama-4-Scout-17B-16E-Instruct": {
- "input_cost_per_token": 9e-08,
- "output_cost_per_token": 2.9e-07,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ "snowflake/snowflake-llama-3.3-70b": {
+ "litellm_provider": "snowflake",
+ "max_input_tokens": 8000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8000,
+ "mode": "chat"
},
- "nscale/Qwen/Qwen2.5-Coder-3B-Instruct": {
- "input_cost_per_token": 1e-08,
- "output_cost_per_token": 3e-08,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ "stability.sd3-5-large-v1:0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.08
},
- "nscale/Qwen/Qwen2.5-Coder-7B-Instruct": {
- "input_cost_per_token": 1e-08,
- "output_cost_per_token": 3e-08,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ "stability.sd3-large-v1:0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.08
},
- "nscale/Qwen/Qwen2.5-Coder-32B-Instruct": {
- "input_cost_per_token": 6e-08,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ "stability.stable-image-core-v1:0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.04
},
- "nscale/Qwen/QwQ-32B": {
- "input_cost_per_token": 1.8e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models"
+ "stability.stable-image-core-v1:1": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.04
},
- "nscale/deepseek-ai/DeepSeek-R1-Distill-Llama-70B": {
- "input_cost_per_token": 3.75e-07,
- "output_cost_per_token": 3.75e-07,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $0.75/1M tokens total. Assumed 50/50 split for input/output."
- }
+ "stability.stable-image-ultra-v1:0": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.14
},
- "nscale/deepseek-ai/DeepSeek-R1-Distill-Llama-8B": {
- "input_cost_per_token": 2.5e-08,
- "output_cost_per_token": 2.5e-08,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $0.05/1M tokens total. Assumed 50/50 split for input/output."
- }
+ "stability.stable-image-ultra-v1:1": {
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 77,
+ "max_tokens": 77,
+ "mode": "image_generation",
+ "output_cost_per_image": 0.14
},
- "nscale/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B": {
- "input_cost_per_token": 9e-08,
- "output_cost_per_token": 9e-08,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $0.18/1M tokens total. Assumed 50/50 split for input/output."
- }
+ "standard/1024-x-1024/dall-e-3": {
+ "input_cost_per_pixel": 3.81469e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
},
- "nscale/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B": {
+ "standard/1024-x-1792/dall-e-3": {
+ "input_cost_per_pixel": 4.359e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
+ },
+ "standard/1792-x-1024/dall-e-3": {
+ "input_cost_per_pixel": 4.359e-08,
+ "litellm_provider": "openai",
+ "mode": "image_generation",
+ "output_cost_per_pixel": 0.0
+ },
+ "text-bison": {
+ "input_cost_per_character": 2.5e-07,
+ "litellm_provider": "vertex_ai-text-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "completion",
+ "output_cost_per_character": 5e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "text-bison32k": {
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-text-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "text-bison32k@002": {
+ "input_cost_per_character": 2.5e-07,
+ "input_cost_per_token": 1.25e-07,
+ "litellm_provider": "vertex_ai-text-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_character": 5e-07,
+ "output_cost_per_token": 1.25e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "text-bison@001": {
+ "input_cost_per_character": 2.5e-07,
+ "litellm_provider": "vertex_ai-text-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_character": 5e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "text-bison@002": {
+ "input_cost_per_character": 2.5e-07,
+ "litellm_provider": "vertex_ai-text-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_character": 5e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "text-completion-codestral/codestral-2405": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "text-completion-codestral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "completion",
+ "output_cost_per_token": 0.0,
+ "source": "https://docs.mistral.ai/capabilities/code_generation/"
+ },
+ "text-completion-codestral/codestral-latest": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "text-completion-codestral",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "completion",
+ "output_cost_per_token": 0.0,
+ "source": "https://docs.mistral.ai/capabilities/code_generation/"
+ },
+ "text-embedding-004": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
+ },
+ "text-embedding-005": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
+ },
+ "text-embedding-3-large": {
+ "input_cost_per_token": 1.3e-07,
+ "input_cost_per_token_batches": 6.5e-08,
+ "litellm_provider": "openai",
+ "max_input_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_cost_per_token_batches": 0.0,
+ "output_vector_size": 3072
+ },
+ "text-embedding-3-small": {
+ "input_cost_per_token": 2e-08,
+ "input_cost_per_token_batches": 1e-08,
+ "litellm_provider": "openai",
+ "max_input_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_cost_per_token_batches": 0.0,
+ "output_vector_size": 1536
+ },
+ "text-embedding-ada-002": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "openai",
+ "max_input_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 1536
+ },
+ "text-embedding-ada-002-v2": {
+ "input_cost_per_token": 1e-07,
+ "input_cost_per_token_batches": 5e-08,
+ "litellm_provider": "openai",
+ "max_input_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_cost_per_token_batches": 0.0
+ },
+ "text-embedding-large-exp-03-07": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 3072,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
+ },
+ "text-embedding-preview-0409": {
+ "input_cost_per_token": 6.25e-09,
+ "input_cost_per_token_batch_requests": 5e-09,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 3072,
+ "max_tokens": 3072,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "text-moderation-007": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "openai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 0,
+ "max_tokens": 32768,
+ "mode": "moderation",
+ "output_cost_per_token": 0.0
+ },
+ "text-moderation-latest": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "openai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 0,
+ "max_tokens": 32768,
+ "mode": "moderation",
+ "output_cost_per_token": 0.0
+ },
+ "text-moderation-stable": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "openai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 0,
+ "max_tokens": 32768,
+ "mode": "moderation",
+ "output_cost_per_token": 0.0
+ },
+ "text-multilingual-embedding-002": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 2048,
+ "max_tokens": 2048,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models"
+ },
+ "text-multilingual-embedding-preview-0409": {
+ "input_cost_per_token": 6.25e-09,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 3072,
+ "max_tokens": 3072,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "text-unicorn": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "vertex_ai-text-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_token": 2.8e-05,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "text-unicorn@001": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "vertex_ai-text-models",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "completion",
+ "output_cost_per_token": 2.8e-05,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "textembedding-gecko": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 3072,
+ "max_tokens": 3072,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "textembedding-gecko-multilingual": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 3072,
+ "max_tokens": 3072,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "textembedding-gecko-multilingual@001": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 3072,
+ "max_tokens": 3072,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "textembedding-gecko@001": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 3072,
+ "max_tokens": 3072,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "textembedding-gecko@003": {
+ "input_cost_per_character": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vertex_ai-embedding-models",
+ "max_input_tokens": 3072,
+ "max_tokens": 3072,
+ "mode": "embedding",
+ "output_cost_per_token": 0,
+ "output_vector_size": 768,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#foundation_models"
+ },
+ "together-ai-21.1b-41b": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 8e-07
+ },
+ "together-ai-4.1b-8b": {
"input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "nscale",
+ "litellm_provider": "together_ai",
"mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $0.40/1M tokens total. Assumed 50/50 split for input/output."
- }
+ "output_cost_per_token": 2e-07
},
- "nscale/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B": {
- "input_cost_per_token": 7e-08,
- "output_cost_per_token": 7e-08,
- "litellm_provider": "nscale",
+ "together-ai-41.1b-80b": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "together_ai",
"mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $0.14/1M tokens total. Assumed 50/50 split for input/output."
- }
+ "output_cost_per_token": 9e-07
},
- "nscale/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B": {
- "input_cost_per_token": 1.5e-07,
- "output_cost_per_token": 1.5e-07,
- "litellm_provider": "nscale",
+ "together-ai-8.1b-21b": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "together_ai",
+ "max_tokens": 1000,
"mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $0.30/1M tokens total. Assumed 50/50 split for input/output."
- }
+ "output_cost_per_token": 3e-07
},
- "nscale/mistralai/mixtral-8x22b-instruct-v0.1": {
- "input_cost_per_token": 6e-07,
+ "together-ai-81.1b-110b": {
+ "input_cost_per_token": 1.8e-06,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-06
+ },
+ "together-ai-embedding-151m-to-350m": {
+ "input_cost_per_token": 1.6e-08,
+ "litellm_provider": "together_ai",
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "together-ai-embedding-up-to-150m": {
+ "input_cost_per_token": 8e-09,
+ "litellm_provider": "together_ai",
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "together_ai/baai/bge-base-en-v1.5": {
+ "input_cost_per_token": 8e-09,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 768
+ },
+ "together_ai/BAAI/bge-base-en-v1.5": {
+ "input_cost_per_token": 8e-09,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 512,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0,
+ "output_vector_size": 768
+ },
+ "together-ai-up-to-4b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 1e-07
+ },
+ "together_ai/Qwen/Qwen2.5-72B-Instruct-Turbo": {
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/Qwen/Qwen2.5-7B-Instruct-Turbo": {
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/Qwen/Qwen3-235B-A22B-Instruct-2507-tput": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 262000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "source": "https://www.together.ai/models/qwen3-235b-a22b-instruct-2507-fp8",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/Qwen/Qwen3-235B-A22B-Thinking-2507": {
+ "input_cost_per_token": 6.5e-07,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "source": "https://www.together.ai/models/qwen3-235b-a22b-thinking-2507",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/Qwen/Qwen3-235B-A22B-fp8-tput": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 40000,
+ "mode": "chat",
"output_cost_per_token": 6e-07,
- "litellm_provider": "nscale",
+ "source": "https://www.together.ai/models/qwen3-235b-a22b-fp8-tput",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_tool_choice": false
+ },
+ "together_ai/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 256000,
"mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $1.20/1M tokens total. Assumed 50/50 split for input/output."
- }
- },
- "nscale/meta-llama/Llama-3.1-8B-Instruct": {
- "input_cost_per_token": 3e-08,
- "output_cost_per_token": 3e-08,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $0.06/1M tokens total. Assumed 50/50 split for input/output."
- }
- },
- "nscale/meta-llama/Llama-3.3-70B-Instruct": {
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-07,
- "litellm_provider": "nscale",
- "mode": "chat",
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#chat-models",
- "metadata": {
- "notes": "Pricing listed as $0.40/1M tokens total. Assumed 50/50 split for input/output."
- }
- },
- "nscale/black-forest-labs/FLUX.1-schnell": {
- "mode": "image_generation",
- "input_cost_per_pixel": 1.3e-09,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "nscale",
- "supported_endpoints": [
- "/v1/images/generations"
- ],
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#image-models"
- },
- "nscale/stabilityai/stable-diffusion-xl-base-1.0": {
- "mode": "image_generation",
- "input_cost_per_pixel": 3e-09,
- "output_cost_per_pixel": 0.0,
- "litellm_provider": "nscale",
- "supported_endpoints": [
- "/v1/images/generations"
- ],
- "source": "https://docs.nscale.com/docs/inference/serverless-models/current#image-models"
- },
- "featherless_ai/featherless-ai/Qwerky-72B": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 4096,
- "litellm_provider": "featherless_ai",
- "mode": "chat"
- },
- "featherless_ai/featherless-ai/Qwerky-QwQ-32B": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 4096,
- "litellm_provider": "featherless_ai",
- "mode": "chat"
- },
- "deepgram/nova-3": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-3-general": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-3-medical": {
- "mode": "audio_transcription",
- "input_cost_per_second": 8.667e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0052,
- "calculation": "$0.0052/60 seconds = $0.00008667 per second (multilingual)"
- }
- },
- "deepgram/nova-2": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-general": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-meeting": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-phonecall": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-voicemail": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-finance": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-conversationalai": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-video": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-drivethru": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-automotive": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-2-atc": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-general": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/nova-phonecall": {
- "mode": "audio_transcription",
- "input_cost_per_second": 7.167e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0043,
- "calculation": "$0.0043/60 seconds = $0.00007167 per second"
- }
- },
- "deepgram/enhanced": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00024167,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0145,
- "calculation": "$0.0145/60 seconds = $0.00024167 per second"
- }
- },
- "deepgram/enhanced-general": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00024167,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0145,
- "calculation": "$0.0145/60 seconds = $0.00024167 per second"
- }
- },
- "deepgram/enhanced-meeting": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00024167,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0145,
- "calculation": "$0.0145/60 seconds = $0.00024167 per second"
- }
- },
- "deepgram/enhanced-phonecall": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00024167,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0145,
- "calculation": "$0.0145/60 seconds = $0.00024167 per second"
- }
- },
- "deepgram/enhanced-finance": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00024167,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0145,
- "calculation": "$0.0145/60 seconds = $0.00024167 per second"
- }
- },
- "deepgram/base": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00020833,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0125,
- "calculation": "$0.0125/60 seconds = $0.00020833 per second"
- }
- },
- "deepgram/base-general": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00020833,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0125,
- "calculation": "$0.0125/60 seconds = $0.00020833 per second"
- }
- },
- "deepgram/base-meeting": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00020833,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0125,
- "calculation": "$0.0125/60 seconds = $0.00020833 per second"
- }
- },
- "deepgram/base-phonecall": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00020833,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0125,
- "calculation": "$0.0125/60 seconds = $0.00020833 per second"
- }
- },
- "deepgram/base-voicemail": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00020833,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0125,
- "calculation": "$0.0125/60 seconds = $0.00020833 per second"
- }
- },
- "deepgram/base-finance": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00020833,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0125,
- "calculation": "$0.0125/60 seconds = $0.00020833 per second"
- }
- },
- "deepgram/base-conversationalai": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00020833,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0125,
- "calculation": "$0.0125/60 seconds = $0.00020833 per second"
- }
- },
- "deepgram/base-video": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.00020833,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "original_pricing_per_minute": 0.0125,
- "calculation": "$0.0125/60 seconds = $0.00020833 per second"
- }
- },
- "deepgram/whisper": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.0001,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
- }
- },
- "deepgram/whisper-tiny": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.0001,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
- }
- },
- "deepgram/whisper-base": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.0001,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
- }
- },
- "deepgram/whisper-small": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.0001,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
- }
- },
- "deepgram/whisper-medium": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.0001,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
- }
- },
- "deepgram/whisper-large": {
- "mode": "audio_transcription",
- "input_cost_per_second": 0.0001,
- "output_cost_per_second": 0.0,
- "litellm_provider": "deepgram",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://deepgram.com/pricing",
- "metadata": {
- "notes": "Deepgram's hosted OpenAI Whisper models - pricing may differ from native Deepgram models"
- }
- },
- "elevenlabs/scribe_v1": {
- "mode": "audio_transcription",
- "input_cost_per_second": 6.11e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "elevenlabs",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://elevenlabs.io/pricing",
- "metadata": {
- "original_pricing_per_hour": 0.22,
- "calculation": "$0.22/hour = $0.00366/minute = $0.0000611 per second (enterprise pricing)",
- "notes": "ElevenLabs Scribe v1 - state-of-the-art speech recognition model with 99 language support"
- }
- },
- "elevenlabs/scribe_v1_experimental": {
- "mode": "audio_transcription",
- "input_cost_per_second": 6.11e-05,
- "output_cost_per_second": 0.0,
- "litellm_provider": "elevenlabs",
- "supported_endpoints": [
- "/v1/audio/transcriptions"
- ],
- "source": "https://elevenlabs.io/pricing",
- "metadata": {
- "original_pricing_per_hour": 0.22,
- "calculation": "$0.22/hour = $0.00366/minute = $0.0000611 per second (enterprise pricing)",
- "notes": "ElevenLabs Scribe v1 experimental - enhanced version of the main Scribe model"
- }
- },
- "bedrock/us-gov-east-1/amazon.titan-embed-text-v1": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "output_vector_size": 1536,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "embedding"
- },
- "bedrock/us-gov-east-1/amazon.titan-embed-text-v2:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "output_vector_size": 1024,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "embedding"
- },
- "bedrock/us-gov-east-1/amazon.titan-text-express-v1": {
- "max_tokens": 8000,
- "max_input_tokens": 42000,
- "max_output_tokens": 8000,
- "input_cost_per_token": 1.3e-06,
- "output_cost_per_token": 1.7e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-gov-east-1/amazon.titan-text-lite-v1": {
- "max_tokens": 4000,
- "max_input_tokens": 42000,
- "max_output_tokens": 4000,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-gov-east-1/amazon.titan-text-premier-v1:0": {
- "max_tokens": 32000,
- "max_input_tokens": 42000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-gov-east-1/anthropic.claude-3-5-sonnet-20240620-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.6e-06,
- "output_cost_per_token": 1.8e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "bedrock/us-gov-east-1/anthropic.claude-3-haiku-20240307-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "bedrock/us-gov-east-1/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 2048,
- "max_input_tokens": 8000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 2.65e-06,
- "output_cost_per_token": 3.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_pdf_input": true
- },
- "bedrock/us-gov-east-1/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 2048,
- "max_input_tokens": 8000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 2.65e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_pdf_input": true
- },
- "bedrock/us-gov-west-1/amazon.titan-embed-text-v1": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "output_vector_size": 1536,
- "input_cost_per_token": 1e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "embedding"
- },
- "bedrock/us-gov-west-1/amazon.titan-embed-text-v2:0": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "output_vector_size": 1024,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 0.0,
- "litellm_provider": "bedrock",
- "mode": "embedding"
- },
- "bedrock/us-gov-west-1/amazon.titan-text-express-v1": {
- "max_tokens": 8000,
- "max_input_tokens": 42000,
- "max_output_tokens": 8000,
- "input_cost_per_token": 1.3e-06,
- "output_cost_per_token": 1.7e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-gov-west-1/amazon.titan-text-lite-v1": {
- "max_tokens": 4000,
- "max_input_tokens": 42000,
- "max_output_tokens": 4000,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 4e-07,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-gov-west-1/amazon.titan-text-premier-v1:0": {
- "max_tokens": 32000,
- "max_input_tokens": 42000,
- "max_output_tokens": 32000,
- "input_cost_per_token": 5e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat"
- },
- "bedrock/us-gov-west-1/anthropic.claude-3-5-sonnet-20240620-v1:0": {
- "max_tokens": 8192,
- "max_input_tokens": 200000,
- "max_output_tokens": 8192,
- "input_cost_per_token": 3.6e-06,
- "output_cost_per_token": 1.8e-05,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "bedrock/us-gov-west-1/anthropic.claude-3-haiku-20240307-v1:0": {
- "max_tokens": 4096,
- "max_input_tokens": 200000,
- "max_output_tokens": 4096,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 1.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_response_schema": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_tool_choice": true
- },
- "bedrock/us-gov-west-1/meta.llama3-70b-instruct-v1:0": {
- "max_tokens": 2048,
- "max_input_tokens": 8000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 2.65e-06,
- "output_cost_per_token": 3.5e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_pdf_input": true
- },
- "bedrock/us-gov-west-1/meta.llama3-8b-instruct-v1:0": {
- "max_tokens": 2048,
- "max_input_tokens": 8000,
- "max_output_tokens": 2048,
- "input_cost_per_token": 3e-07,
- "output_cost_per_token": 2.65e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_pdf_input": true
- },
- "bedrock/us-gov-east-1/amazon.nova-pro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 9.6e-07,
- "output_cost_per_token": 3.84e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "bedrock/us-gov-west-1/amazon.nova-pro-v1:0": {
- "max_tokens": 10000,
- "max_input_tokens": 300000,
- "max_output_tokens": 10000,
- "input_cost_per_token": 9.6e-07,
- "output_cost_per_token": 3.84e-06,
- "litellm_provider": "bedrock",
- "mode": "chat",
- "supports_function_calling": true,
- "supports_vision": true,
- "supports_pdf_input": true,
- "supports_prompt_caching": true,
- "supports_response_schema": true
- },
- "dashscope/qwen-max": {
- "max_tokens": 32768,
- "max_input_tokens": 30720,
- "max_output_tokens": 8192,
- "litellm_provider": "dashscope",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "mode": "chat",
- "source": "https://bailian.console.alibabacloud.com/?spm=a2c63.p38356.0.0.4a615d7bjSUCb4&tab=doc#/doc/?type=model&url=https%3A%2F%2Fwww.alibabacloud.com%2Fhelp%2Fen%2Fdoc-detail%2F2840914.html"
- },
- "dashscope/qwen-plus-latest": {
- "max_tokens": 131072,
- "max_input_tokens": 129024,
- "max_output_tokens": 16384,
- "litellm_provider": "dashscope",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "mode": "chat",
- "source": "https://bailian.console.alibabacloud.com/?spm=a2c63.p38356.0.0.4a615d7bjSUCb4&tab=doc#/doc/?type=model&url=https%3A%2F%2Fwww.alibabacloud.com%2Fhelp%2Fen%2Fdoc-detail%2F2840914.html"
- },
- "dashscope/qwen-turbo-latest": {
- "max_tokens": 131072,
- "max_input_tokens": 129024,
- "max_output_tokens": 16384,
- "litellm_provider": "dashscope",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "mode": "chat",
- "source": "https://bailian.console.alibabacloud.com/?spm=a2c63.p38356.0.0.4a615d7bjSUCb4&tab=doc#/doc/?type=model&url=https%3A%2F%2Fwww.alibabacloud.com%2Fhelp%2Fen%2Fdoc-detail%2F2840914.html"
- },
- "dashscope/qwen3-30b-a3b": {
- "max_tokens": 131072,
- "max_input_tokens": 129024,
- "max_output_tokens": 16384,
- "litellm_provider": "dashscope",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_reasoning": true,
- "mode": "chat",
- "source": "https://bailian.console.alibabacloud.com/?spm=a2c63.p38356.0.0.4a615d7bjSUCb4&tab=doc#/doc/?type=model&url=https%3A%2F%2Fwww.alibabacloud.com%2Fhelp%2Fen%2Fdoc-detail%2F2840914.html"
- },
- "moonshot/moonshot-v1-8k": {
- "max_tokens": 8192,
- "max_input_tokens": 8192,
- "max_output_tokens": 8192,
- "input_cost_per_token": 2e-07,
"output_cost_per_token": 2e-06,
- "litellm_provider": "moonshot",
+ "source": "https://www.together.ai/models/qwen3-coder-480b-a35b-instruct",
"supports_function_calling": true,
- "supports_tool_choice": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
},
- "moonshot/moonshot-v1-32k": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "moonshot",
+ "together_ai/deepseek-ai/DeepSeek-R1": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 20480,
+ "max_tokens": 20480,
+ "mode": "chat",
+ "output_cost_per_token": 7e-06,
"supports_function_calling": true,
- "supports_tool_choice": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
},
- "moonshot/moonshot-v1-128k": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "moonshot",
+ "together_ai/deepseek-ai/DeepSeek-R1-0528-tput": {
+ "input_cost_per_token": 5.5e-07,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2.19e-06,
+ "source": "https://www.together.ai/models/deepseek-r1-0528-throughput",
"supports_function_calling": true,
- "supports_tool_choice": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
},
- "moonshot/moonshot-v1-auto": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "moonshot",
+ "together_ai/deepseek-ai/DeepSeek-V3": {
+ "input_cost_per_token": 1.25e-06,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
"supports_function_calling": true,
- "supports_tool_choice": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
},
- "moonshot/kimi-k2-0711-preview": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
+ "together_ai/deepseek-ai/DeepSeek-V3.1": {
"input_cost_per_token": 6e-07,
- "output_cost_per_token": 2.5e-06,
- "cache_read_input_token_cost": 1.5e-07,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_web_search": true,
+ "litellm_provider": "together_ai",
+ "max_tokens": 128000,
"mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing/chat#generation-model-kimi-k2"
+ "output_cost_per_token": 1.7e-06,
+ "source": "https://www.together.ai/models/deepseek-v3-1",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
},
- "moonshot/moonshot-v1-32k-0430": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
+ "together_ai/meta-llama/Llama-3.2-3B-Instruct-Turbo": {
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo": {
+ "input_cost_per_token": 8.8e-07,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 8.8e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo-Free": {
+ "input_cost_per_token": 0,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 0,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8": {
+ "input_cost_per_token": 2.7e-07,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 8.5e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/meta-llama/Llama-4-Scout-17B-16E-Instruct": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 5.9e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo": {
+ "input_cost_per_token": 3.5e-06,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 3.5e-06,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo": {
+ "input_cost_per_token": 8.8e-07,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 8.8e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 1.8e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/mistralai/Mistral-7B-Instruct-v0.1": {
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/mistralai/Mistral-Small-24B-Instruct-2501": {
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/moonshotai/Kimi-K2-Instruct": {
"input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
+ "litellm_provider": "together_ai",
"mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "output_cost_per_token": 3e-06,
+ "source": "https://www.together.ai/models/kimi-k2-instruct",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
},
- "moonshot/moonshot-v1-128k-0430": {
- "max_tokens": 131072,
+ "together_ai/openai/gpt-oss-120b": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://www.together.ai/models/gpt-oss-120b",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/openai/gpt-oss-20b": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07,
+ "source": "https://www.together.ai/models/gpt-oss-20b",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/togethercomputer/CodeLlama-34b-Instruct": {
+ "litellm_provider": "together_ai",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/zai-org/GLM-4.5-Air-FP8": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.1e-06,
+ "source": "https://www.together.ai/models/glm-4-5-air",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/moonshotai/Kimi-K2-Instruct-0905": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "source": "https://www.together.ai/models/kimi-k2-0905",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/Qwen/Qwen3-Next-80B-A3B-Instruct": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://www.together.ai/models/qwen3-next-80b-a3b-instruct",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "together_ai/Qwen/Qwen3-Next-80B-A3B-Thinking": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "together_ai",
+ "max_input_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://www.together.ai/models/qwen3-next-80b-a3b-thinking",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "tts-1": {
+ "input_cost_per_character": 1.5e-05,
+ "litellm_provider": "openai",
+ "mode": "audio_speech",
+ "supported_endpoints": [
+ "/v1/audio/speech"
+ ]
+ },
+ "tts-1-hd": {
+ "input_cost_per_character": 3e-05,
+ "litellm_provider": "openai",
+ "mode": "audio_speech",
+ "supported_endpoints": [
+ "/v1/audio/speech"
+ ]
+ },
+ "us.amazon.nova-lite-v1:0": {
+ "input_cost_per_token": 6e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-07,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "us.amazon.nova-micro-v1:0": {
+ "input_cost_per_token": 3.5e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 1.4e-07,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true
+ },
+ "us.amazon.nova-premier-v1:0": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": false,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "us.amazon.nova-pro-v1:0": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 10000,
+ "max_tokens": 10000,
+ "mode": "chat",
+ "output_cost_per_token": 3.2e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_vision": true
+ },
+ "us.anthropic.claude-3-5-haiku-20241022-v1:0": {
+ "cache_creation_input_token_cost": 1e-06,
+ "cache_read_input_token_cost": 8e-08,
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true
+ },
+ "us.anthropic.claude-3-5-sonnet-20240620-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "us.anthropic.claude-3-5-sonnet-20241022-v2:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "us.anthropic.claude-3-7-sonnet-20250219-v1:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "us.anthropic.claude-3-haiku-20240307-v1:0": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "us.anthropic.claude-3-opus-20240229-v1:0": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_function_calling": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "us.anthropic.claude-3-sonnet-20240229-v1:0": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "us.anthropic.claude-opus-4-1-20250805-v1:0": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "us.anthropic.claude-sonnet-4-5-20250929-v1:0": {
+ "cache_creation_input_token_cost": 4.125e-06,
+ "cache_read_input_token_cost": 3.3e-07,
+ "input_cost_per_token": 3.3e-06,
+ "input_cost_per_token_above_200k_tokens": 6.6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.475e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6.6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.65e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 346
+ },
+ "us.anthropic.claude-opus-4-20250514-v1:0": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "us.anthropic.claude-sonnet-4-20250514-v1:0": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "us.deepseek.r1-v1:0": {
+ "input_cost_per_token": 1.35e-06,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 5.4e-06,
+ "supports_function_calling": false,
+ "supports_reasoning": true,
+ "supports_tool_choice": false
+ },
+ "us.meta.llama3-1-405b-instruct-v1:0": {
+ "input_cost_per_token": 5.32e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "us.meta.llama3-1-70b-instruct-v1:0": {
+ "input_cost_per_token": 9.9e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 9.9e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "us.meta.llama3-1-8b-instruct-v1:0": {
+ "input_cost_per_token": 2.2e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2.2e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "us.meta.llama3-2-11b-instruct-v1:0": {
+ "input_cost_per_token": 3.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3.5e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "us.meta.llama3-2-1b-instruct-v1:0": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "us.meta.llama3-2-3b-instruct-v1:0": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "us.meta.llama3-2-90b-instruct-v1:0": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "bedrock",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": false,
+ "supports_vision": true
+ },
+ "us.meta.llama3-3-70b-instruct-v1:0": {
+ "input_cost_per_token": 7.2e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.2e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "us.meta.llama4-maverick-17b-instruct-v1:0": {
+ "input_cost_per_token": 2.4e-07,
+ "input_cost_per_token_batches": 1.2e-07,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 9.7e-07,
+ "output_cost_per_token_batches": 4.85e-07,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "code"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "us.meta.llama4-scout-17b-instruct-v1:0": {
+ "input_cost_per_token": 1.7e-07,
+ "input_cost_per_token_batches": 8.5e-08,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 6.6e-07,
+ "output_cost_per_token_batches": 3.3e-07,
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "code"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "us.mistral.pixtral-large-2502-v1:0": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "bedrock_converse",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": false
+ },
+ "v0/v0-1.0-md": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "v0",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "v0/v0-1.5-lg": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "v0",
+ "max_input_tokens": 512000,
+ "max_output_tokens": 512000,
+ "max_tokens": 512000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "v0/v0-1.5-md": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "v0",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vercel_ai_gateway/alibaba/qwen-3-14b": {
+ "input_cost_per_token": 8e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 40960,
+ "max_output_tokens": 16384,
+ "max_tokens": 40960,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-07
+ },
+ "vercel_ai_gateway/alibaba/qwen-3-235b": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 40960,
+ "max_output_tokens": 16384,
+ "max_tokens": 40960,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07
+ },
+ "vercel_ai_gateway/alibaba/qwen-3-30b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 40960,
+ "max_output_tokens": 16384,
+ "max_tokens": 40960,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07
+ },
+ "vercel_ai_gateway/alibaba/qwen-3-32b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 40960,
+ "max_output_tokens": 16384,
+ "max_tokens": 40960,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07
+ },
+ "vercel_ai_gateway/alibaba/qwen3-coder": {
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 66536,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-06
+ },
+ "vercel_ai_gateway/amazon/nova-lite": {
+ "input_cost_per_token": 6e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 8192,
+ "max_tokens": 300000,
+ "mode": "chat",
+ "output_cost_per_token": 2.4e-07
+ },
+ "vercel_ai_gateway/amazon/nova-micro": {
+ "input_cost_per_token": 3.5e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.4e-07
+ },
+ "vercel_ai_gateway/amazon/nova-pro": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 300000,
+ "max_output_tokens": 8192,
+ "max_tokens": 300000,
+ "mode": "chat",
+ "output_cost_per_token": 3.2e-06
+ },
+ "vercel_ai_gateway/amazon/titan-embed-text-v2": {
+ "input_cost_per_token": 2e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/anthropic/claude-3-haiku": {
+ "cache_creation_input_token_cost": 3e-07,
+ "cache_read_input_token_cost": 3e-08,
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06
+ },
+ "vercel_ai_gateway/anthropic/claude-3-opus": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05
+ },
+ "vercel_ai_gateway/anthropic/claude-3.5-haiku": {
+ "cache_creation_input_token_cost": 1e-06,
+ "cache_read_input_token_cost": 8e-08,
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06
+ },
+ "vercel_ai_gateway/anthropic/claude-3.5-sonnet": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05
+ },
+ "vercel_ai_gateway/anthropic/claude-3.7-sonnet": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05
+ },
+ "vercel_ai_gateway/anthropic/claude-4-opus": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05
+ },
+ "vercel_ai_gateway/anthropic/claude-4-sonnet": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05
+ },
+ "vercel_ai_gateway/cohere/command-a": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 8000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05
+ },
+ "vercel_ai_gateway/cohere/command-r": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07
+ },
+ "vercel_ai_gateway/cohere/command-r-plus": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05
+ },
+ "vercel_ai_gateway/cohere/embed-v4.0": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/deepseek/deepseek-r1": {
+ "input_cost_per_token": 5.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2.19e-06
+ },
+ "vercel_ai_gateway/deepseek/deepseek-r1-distill-llama-70b": {
+ "input_cost_per_token": 7.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 5e-06,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
+ "max_tokens": 131072,
"mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "output_cost_per_token": 9.9e-07
},
- "moonshot/moonshot-v1-8k-0430": {
- "max_tokens": 8192,
+ "vercel_ai_gateway/deepseek/deepseek-v3": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07
+ },
+ "vercel_ai_gateway/google/gemini-2.0-flash": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_tokens": 1048576,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07
+ },
+ "vercel_ai_gateway/google/gemini-2.0-flash-lite": {
+ "input_cost_per_token": 7.5e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 8192,
+ "max_tokens": 1048576,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07
+ },
+ "vercel_ai_gateway/google/gemini-2.5-flash": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 65536,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-06
+ },
+ "vercel_ai_gateway/google/gemini-2.5-pro": {
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 1048576,
+ "max_output_tokens": 65536,
+ "max_tokens": 1048576,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05
+ },
+ "vercel_ai_gateway/google/gemini-embedding-001": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/google/gemma-2-9b": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vercel_ai_gateway",
"max_input_tokens": 8192,
"max_output_tokens": 8192,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
- },
- "moonshot/kimi-latest": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 5e-06,
- "cache_read_input_token_cost": 1.5e-07,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_vision": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
- },
- "moonshot/kimi-latest-8k": {
"max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2e-07
+ },
+ "vercel_ai_gateway/google/text-embedding-005": {
+ "input_cost_per_token": 2.5e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/google/text-multilingual-embedding-002": {
+ "input_cost_per_token": 2.5e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/inception/mercury-coder-small": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 16384,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06
+ },
+ "vercel_ai_gateway/meta/llama-3-70b": {
+ "input_cost_per_token": 5.9e-07,
+ "litellm_provider": "vercel_ai_gateway",
"max_input_tokens": 8192,
"max_output_tokens": 8192,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-06,
- "cache_read_input_token_cost": 1.5e-07,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_vision": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
- },
- "moonshot/kimi-latest-32k": {
- "max_tokens": 32768,
- "max_input_tokens": 32768,
- "max_output_tokens": 32768,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "cache_read_input_token_cost": 1.5e-07,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_vision": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
- },
- "moonshot/kimi-latest-128k": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
- "output_cost_per_token": 5e-06,
- "cache_read_input_token_cost": 1.5e-07,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_vision": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
- },
- "moonshot/kimi-thinking-preview": {
- "max_tokens": 131072,
- "max_input_tokens": 131072,
- "max_output_tokens": 131072,
- "input_cost_per_token": 3e-05,
- "output_cost_per_token": 3e-05,
- "litellm_provider": "moonshot",
- "supports_vision": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
- },
- "moonshot/moonshot-v1-8k-vision-preview": {
"max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 7.9e-07
+ },
+ "vercel_ai_gateway/meta/llama-3-8b": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "vercel_ai_gateway",
"max_input_tokens": 8192,
"max_output_tokens": 8192,
- "input_cost_per_token": 2e-07,
- "output_cost_per_token": 2e-06,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_vision": true,
+ "max_tokens": 8192,
"mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "output_cost_per_token": 8e-08
},
- "moonshot/moonshot-v1-32k-vision-preview": {
- "max_tokens": 32768,
+ "vercel_ai_gateway/meta/llama-3.1-70b": {
+ "input_cost_per_token": 7.2e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 7.2e-07
+ },
+ "vercel_ai_gateway/meta/llama-3.1-8b": {
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131000,
+ "max_output_tokens": 131072,
+ "max_tokens": 131000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-08
+ },
+ "vercel_ai_gateway/meta/llama-3.2-11b": {
+ "input_cost_per_token": 1.6e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-07
+ },
+ "vercel_ai_gateway/meta/llama-3.2-1b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07
+ },
+ "vercel_ai_gateway/meta/llama-3.2-3b": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07
+ },
+ "vercel_ai_gateway/meta/llama-3.2-90b": {
+ "input_cost_per_token": 7.2e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 7.2e-07
+ },
+ "vercel_ai_gateway/meta/llama-3.3-70b": {
+ "input_cost_per_token": 7.2e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8192,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 7.2e-07
+ },
+ "vercel_ai_gateway/meta/llama-4-maverick": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8192,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07
+ },
+ "vercel_ai_gateway/meta/llama-4-scout": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 8192,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07
+ },
+ "vercel_ai_gateway/mistral/codestral": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 4000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 9e-07
+ },
+ "vercel_ai_gateway/mistral/codestral-embed": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/mistral/devstral-small": {
+ "input_cost_per_token": 7e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 2.8e-07
+ },
+ "vercel_ai_gateway/mistral/magistral-medium": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 64000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06
+ },
+ "vercel_ai_gateway/mistral/magistral-small": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 64000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06
+ },
+ "vercel_ai_gateway/mistral/ministral-3b": {
+ "input_cost_per_token": 4e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-08
+ },
+ "vercel_ai_gateway/mistral/ministral-8b": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-07
+ },
+ "vercel_ai_gateway/mistral/mistral-embed": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "chat",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/mistral/mistral-large": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 4000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06
+ },
+ "vercel_ai_gateway/mistral/mistral-saba-24b": {
+ "input_cost_per_token": 7.9e-07,
+ "litellm_provider": "vercel_ai_gateway",
"max_input_tokens": 32768,
"max_output_tokens": 32768,
- "input_cost_per_token": 1e-06,
- "output_cost_per_token": 3e-06,
- "litellm_provider": "moonshot",
- "supports_function_calling": true,
- "supports_tool_choice": true,
- "supports_vision": true,
+ "max_tokens": 32768,
"mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "output_cost_per_token": 7.9e-07
},
- "moonshot/moonshot-v1-128k-vision-preview": {
+ "vercel_ai_gateway/mistral/mistral-small": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 4000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07
+ },
+ "vercel_ai_gateway/mistral/mixtral-8x22b-instruct": {
+ "input_cost_per_token": 1.2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 65536,
+ "max_output_tokens": 2048,
+ "max_tokens": 65536,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06
+ },
+ "vercel_ai_gateway/mistral/pixtral-12b": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07
+ },
+ "vercel_ai_gateway/mistral/pixtral-large": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06
+ },
+ "vercel_ai_gateway/moonshotai/kimi-k2": {
+ "input_cost_per_token": 5.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 16384,
"max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.2e-06
+ },
+ "vercel_ai_gateway/morph/morph-v3-fast": {
+ "input_cost_per_token": 8e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 16384,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06
+ },
+ "vercel_ai_gateway/morph/morph-v3-large": {
+ "input_cost_per_token": 9e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 16384,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1.9e-06
+ },
+ "vercel_ai_gateway/openai/gpt-3.5-turbo": {
+ "input_cost_per_token": 5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 16385,
+ "max_output_tokens": 4096,
+ "max_tokens": 16385,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06
+ },
+ "vercel_ai_gateway/openai/gpt-3.5-turbo-instruct": {
+ "input_cost_per_token": 1.5e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 4096,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 2e-06
+ },
+ "vercel_ai_gateway/openai/gpt-4-turbo": {
+ "input_cost_per_token": 1e-05,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 4096,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-05
+ },
+ "vercel_ai_gateway/openai/gpt-4.1": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 1047576,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06
+ },
+ "vercel_ai_gateway/openai/gpt-4.1-mini": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 1e-07,
+ "input_cost_per_token": 4e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 1047576,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-06
+ },
+ "vercel_ai_gateway/openai/gpt-4.1-nano": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 2.5e-08,
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 1047576,
+ "max_output_tokens": 32768,
+ "max_tokens": 1047576,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07
+ },
+ "vercel_ai_gateway/openai/gpt-4o": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 1.25e-06,
+ "input_cost_per_token": 2.5e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05
+ },
+ "vercel_ai_gateway/openai/gpt-4o-mini": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 7.5e-08,
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 16384,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07
+ },
+ "vercel_ai_gateway/openai/o1": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 7.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-05
+ },
+ "vercel_ai_gateway/openai/o3": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 5e-07,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06
+ },
+ "vercel_ai_gateway/openai/o3-mini": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 5.5e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06
+ },
+ "vercel_ai_gateway/openai/o4-mini": {
+ "cache_creation_input_token_cost": 0.0,
+ "cache_read_input_token_cost": 2.75e-07,
+ "input_cost_per_token": 1.1e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 100000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 4.4e-06
+ },
+ "vercel_ai_gateway/openai/text-embedding-3-large": {
+ "input_cost_per_token": 1.3e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/openai/text-embedding-3-small": {
+ "input_cost_per_token": 2e-08,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/openai/text-embedding-ada-002": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 0,
+ "max_output_tokens": 0,
+ "max_tokens": 0,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "vercel_ai_gateway/perplexity/sonar": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 127000,
+ "max_output_tokens": 8000,
+ "max_tokens": 127000,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06
+ },
+ "vercel_ai_gateway/perplexity/sonar-pro": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05
+ },
+ "vercel_ai_gateway/perplexity/sonar-reasoning": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 127000,
+ "max_output_tokens": 8000,
+ "max_tokens": 127000,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06
+ },
+ "vercel_ai_gateway/perplexity/sonar-reasoning-pro": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 127000,
+ "max_output_tokens": 8000,
+ "max_tokens": 127000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06
+ },
+ "vercel_ai_gateway/vercel/v0-1.0-md": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05
+ },
+ "vercel_ai_gateway/vercel/v0-1.5-md": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 32768,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05
+ },
+ "vercel_ai_gateway/xai/grok-2": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 4000,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05
+ },
+ "vercel_ai_gateway/xai/grok-2-vision": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05
+ },
+ "vercel_ai_gateway/xai/grok-3": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vercel_ai_gateway",
"max_input_tokens": 131072,
"max_output_tokens": 131072,
- "input_cost_per_token": 2e-06,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05
+ },
+ "vercel_ai_gateway/xai/grok-3-fast": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-05
+ },
+ "vercel_ai_gateway/xai/grok-3-mini": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07
+ },
+ "vercel_ai_gateway/xai/grok-3-mini-fast": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06
+ },
+ "vercel_ai_gateway/xai/grok-4": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05
+ },
+ "vercel_ai_gateway/zai/glm-4.5": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.2e-06
+ },
+ "vercel_ai_gateway/zai/glm-4.5-air": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vercel_ai_gateway",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 96000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.1e-06
+ },
+ "vertex_ai/claude-3-5-haiku": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
"output_cost_per_token": 5e-06,
- "litellm_provider": "moonshot",
+ "supports_assistant_prefill": true,
"supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/claude-3-5-haiku@20241022": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 5e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/claude-3-5-sonnet": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-5-sonnet-v2": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-5-sonnet-v2@20241022": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-5-sonnet@20240620": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-7-sonnet@20250219": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "deprecation_date": "2025-06-01",
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
- "mode": "chat",
- "source": "https://platform.moonshot.ai/docs/pricing"
+ "tool_use_system_prompt_tokens": 159
},
- "recraft/recraftv3": {
+ "vertex_ai/claude-3-haiku": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-haiku@20240307": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.25e-06,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-opus": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-opus@20240229": {
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-sonnet": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-3-sonnet@20240229": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-opus-4": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "vertex_ai/claude-opus-4-1": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "input_cost_per_token_batches": 7.5e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "output_cost_per_token_batches": 3.75e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-opus-4-1@20250805": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "input_cost_per_token_batches": 7.5e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "output_cost_per_token_batches": 3.75e-05,
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-sonnet-4-5": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "input_cost_per_token_batches": 1.5e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "output_cost_per_token_batches": 7.5e-06,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-sonnet-4-5@20250929": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "input_cost_per_token_batches": 1.5e-06,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 64000,
+ "max_tokens": 200000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "output_cost_per_token_batches": 7.5e-06,
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/claude-opus-4@20250514": {
+ "cache_creation_input_token_cost": 1.875e-05,
+ "cache_read_input_token_cost": 1.5e-06,
+ "input_cost_per_token": 1.5e-05,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 200000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 7.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "vertex_ai/claude-sonnet-4": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "vertex_ai/claude-sonnet-4@20250514": {
+ "cache_creation_input_token_cost": 3.75e-06,
+ "cache_read_input_token_cost": 3e-07,
+ "input_cost_per_token": 3e-06,
+ "input_cost_per_token_above_200k_tokens": 6e-06,
+ "output_cost_per_token_above_200k_tokens": 2.25e-05,
+ "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
+ "cache_read_input_token_cost_above_200k_tokens": 6e-07,
+ "litellm_provider": "vertex_ai-anthropic_models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 64000,
+ "max_tokens": 64000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "search_context_cost_per_query": {
+ "search_context_size_high": 0.01,
+ "search_context_size_low": 0.01,
+ "search_context_size_medium": 0.01
+ },
+ "supports_assistant_prefill": true,
+ "supports_computer_use": true,
+ "supports_function_calling": true,
+ "supports_pdf_input": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_response_schema": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "tool_use_system_prompt_tokens": 159
+ },
+ "vertex_ai/codestral-2501": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/codestral@2405": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/codestral@latest": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/deepseek-ai/deepseek-v3.1-maas": {
+ "input_cost_per_token": 1.35e-06,
+ "litellm_provider": "vertex_ai-deepseek_models",
+ "max_input_tokens": 163840,
+ "max_output_tokens": 32768,
+ "max_tokens": 163840,
+ "mode": "chat",
+ "output_cost_per_token": 5.4e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supported_regions": [
+ "us-west2"
+ ],
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/deepseek-ai/deepseek-r1-0528-maas": {
+ "input_cost_per_token": 1.35e-06,
+ "litellm_provider": "vertex_ai-deepseek_models",
+ "max_input_tokens": 65336,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 5.4e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supports_assistant_prefill": true,
+ "supports_function_calling": true,
+ "supports_prompt_caching": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/imagegeneration@006": {
+ "litellm_provider": "vertex_ai-image-models",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.02,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "vertex_ai/imagen-3.0-fast-generate-001": {
+ "litellm_provider": "vertex_ai-image-models",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.02,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "vertex_ai/imagen-3.0-generate-001": {
+ "litellm_provider": "vertex_ai-image-models",
"mode": "image_generation",
"output_cost_per_image": 0.04,
- "litellm_provider": "recraft",
- "supported_endpoints": [
- "/v1/images/generations"
- ],
- "source": "https://www.recraft.ai/docs#pricing"
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
},
- "recraft/recraftv2": {
+ "vertex_ai/imagen-3.0-generate-002": {
+ "litellm_provider": "vertex_ai-image-models",
"mode": "image_generation",
- "output_cost_per_image": 0.022,
- "litellm_provider": "recraft",
- "supported_endpoints": [
- "/v1/images/generations"
+ "output_cost_per_image": 0.04,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "vertex_ai/imagen-4.0-fast-generate-001": {
+ "litellm_provider": "vertex_ai-image-models",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.02,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "vertex_ai/imagen-4.0-generate-001": {
+ "litellm_provider": "vertex_ai-image-models",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.04,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "vertex_ai/imagen-4.0-ultra-generate-001": {
+ "litellm_provider": "vertex_ai-image-models",
+ "mode": "image_generation",
+ "output_cost_per_image": 0.06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
+ },
+ "vertex_ai/jamba-1.5": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vertex_ai-ai21_models",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/jamba-1.5-large": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vertex_ai-ai21_models",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/jamba-1.5-large@001": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vertex_ai-ai21_models",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 8e-06,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/jamba-1.5-mini": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vertex_ai-ai21_models",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/jamba-1.5-mini@001": {
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "vertex_ai-ai21_models",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 4e-07,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/meta/llama-3.1-405b-instruct-maas": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.6e-05,
+ "source": "https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-3.2-90b-vision-instruct-maas",
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/meta/llama-3.1-70b-instruct-maas": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "source": "https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-3.2-90b-vision-instruct-maas",
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/meta/llama-3.1-8b-instruct-maas": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 128000,
+ "metadata": {
+ "notes": "VertexAI states that The Llama 3.1 API service for llama-3.1-70b-instruct-maas and llama-3.1-8b-instruct-maas are in public preview and at no cost."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "source": "https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-3.2-90b-vision-instruct-maas",
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/meta/llama-3.2-90b-vision-instruct-maas": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 2048,
+ "max_tokens": 128000,
+ "metadata": {
+ "notes": "VertexAI states that The Llama 3.2 API service is at no cost during public preview, and will be priced as per dollar-per-1M-tokens at GA."
+ },
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "source": "https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-3.2-90b-vision-instruct-maas",
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/meta/llama-4-maverick-17b-128e-instruct-maas": {
+ "input_cost_per_token": 3.5e-07,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 1000000,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_token": 1.15e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supported_modalities": [
+ "text",
+ "image"
],
- "source": "https://www.recraft.ai/docs#pricing"
+ "supported_output_modalities": [
+ "text",
+ "code"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": true
},
- "morph/morph-v3-fast": {
- "max_tokens": 16000,
- "max_input_tokens": 16000,
- "max_output_tokens": 16000,
- "input_cost_per_token": 8e-07,
+ "vertex_ai/meta/llama-4-maverick-17b-16e-instruct-maas": {
+ "input_cost_per_token": 3.5e-07,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 1000000,
+ "max_output_tokens": 1000000,
+ "max_tokens": 1000000,
+ "mode": "chat",
+ "output_cost_per_token": 1.15e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "code"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/meta/llama-4-scout-17b-128e-instruct-maas": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 10000000,
+ "max_output_tokens": 10000000,
+ "max_tokens": 10000000,
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "code"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/meta/llama-4-scout-17b-16e-instruct-maas": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 10000000,
+ "max_output_tokens": 10000000,
+ "max_tokens": 10000000,
+ "mode": "chat",
+ "output_cost_per_token": 7e-07,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supported_modalities": [
+ "text",
+ "image"
+ ],
+ "supported_output_modalities": [
+ "text",
+ "code"
+ ],
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/meta/llama3-405b-instruct-maas": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supports_tool_choice": true
+ },
+ "vertex_ai/meta/llama3-70b-instruct-maas": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supports_tool_choice": true
+ },
+ "vertex_ai/meta/llama3-8b-instruct-maas": {
+ "input_cost_per_token": 0.0,
+ "litellm_provider": "vertex_ai-llama_models",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "chat",
+ "output_cost_per_token": 0.0,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#partner-models",
+ "supports_tool_choice": true
+ },
+ "vertex_ai/mistral-large-2411": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/mistral-large@2407": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/mistral-large@2411-001": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/mistral-large@latest": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 6e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/mistral-nemo@2407": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/mistral-nemo@latest": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-07,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/mistral-small-2503": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "max_tokens": 128000,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true
+ },
+ "vertex_ai/mistral-small-2503@001": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "vertex_ai-mistral_models",
+ "max_input_tokens": 32000,
+ "max_output_tokens": 8191,
+ "max_tokens": 8191,
+ "mode": "chat",
+ "output_cost_per_token": 3e-06,
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/openai/gpt-oss-120b-maas": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-openai_models",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 6e-07,
+ "source": "https://console.cloud.google.com/vertex-ai/publishers/openai/model-garden/gpt-oss-120b-maas",
+ "supports_reasoning": true
+ },
+ "vertex_ai/openai/gpt-oss-20b-maas": {
+ "input_cost_per_token": 7.5e-08,
+ "litellm_provider": "vertex_ai-openai_models",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 3e-07,
+ "source": "https://console.cloud.google.com/vertex-ai/publishers/openai/model-garden/gpt-oss-120b-maas",
+ "supports_reasoning": true
+ },
+ "vertex_ai/qwen/qwen3-235b-a22b-instruct-2507-maas": {
+ "input_cost_per_token": 2.5e-07,
+ "litellm_provider": "vertex_ai-qwen_models",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 16384,
+ "max_tokens": 16384,
+ "mode": "chat",
+ "output_cost_per_token": 1e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/qwen/qwen3-coder-480b-a35b-instruct-maas": {
+ "input_cost_per_token": 1e-06,
+ "litellm_provider": "vertex_ai-qwen_models",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/qwen/qwen3-next-80b-a3b-instruct-maas": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-qwen_models",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "max_tokens": 262144,
+ "mode": "chat",
"output_cost_per_token": 1.2e-06,
- "litellm_provider": "morph",
- "mode": "chat",
- "supports_function_calling": false,
- "supports_parallel_function_calling": false,
- "supports_vision": false,
- "supports_system_messages": true,
- "supports_tool_choice": false
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
},
- "morph/morph-v3-large": {
- "max_tokens": 16000,
+ "vertex_ai/qwen/qwen3-next-80b-a3b-thinking-maas": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-qwen_models",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/veo-2.0-generate-001": {
+ "litellm_provider": "vertex_ai-video-models",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "video_generation",
+ "output_cost_per_second": 0.35,
+ "source": "https://ai.google.dev/gemini-api/docs/video",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "video"
+ ]
+ },
+ "vertex_ai/veo-3.0-fast-generate-preview": {
+ "litellm_provider": "vertex_ai-video-models",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "video_generation",
+ "output_cost_per_second": 0.4,
+ "source": "https://ai.google.dev/gemini-api/docs/video",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "video"
+ ]
+ },
+ "vertex_ai/veo-3.0-generate-preview": {
+ "litellm_provider": "vertex_ai-video-models",
+ "max_input_tokens": 1024,
+ "max_tokens": 1024,
+ "mode": "video_generation",
+ "output_cost_per_second": 0.75,
+ "source": "https://ai.google.dev/gemini-api/docs/video",
+ "supported_modalities": [
+ "text"
+ ],
+ "supported_output_modalities": [
+ "video"
+ ]
+ },
+ "voyage/rerank-2": {
+ "input_cost_per_query": 5e-08,
+ "input_cost_per_token": 5e-08,
+ "litellm_provider": "voyage",
"max_input_tokens": 16000,
"max_output_tokens": 16000,
- "input_cost_per_token": 9e-07,
- "output_cost_per_token": 1.9e-06,
- "litellm_provider": "morph",
+ "max_query_tokens": 16000,
+ "max_tokens": 16000,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/rerank-2-lite": {
+ "input_cost_per_query": 2e-08,
+ "input_cost_per_token": 2e-08,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 8000,
+ "max_output_tokens": 8000,
+ "max_query_tokens": 8000,
+ "max_tokens": 8000,
+ "mode": "rerank",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-2": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 4000,
+ "max_tokens": 4000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-3": {
+ "input_cost_per_token": 6e-08,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-3-large": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-3-lite": {
+ "input_cost_per_token": 2e-08,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-code-2": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 16000,
+ "max_tokens": 16000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-code-3": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-context-3": {
+ "input_cost_per_token": 1.8e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 120000,
+ "max_tokens": 120000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-finance-2": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-large-2": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 16000,
+ "max_tokens": 16000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-law-2": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 16000,
+ "max_tokens": 16000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-lite-01": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 4096,
+ "max_tokens": 4096,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-lite-02-instruct": {
+ "input_cost_per_token": 1e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 4000,
+ "max_tokens": 4000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "voyage/voyage-multimodal-3": {
+ "input_cost_per_token": 1.2e-07,
+ "litellm_provider": "voyage",
+ "max_input_tokens": 32000,
+ "max_tokens": 32000,
+ "mode": "embedding",
+ "output_cost_per_token": 0.0
+ },
+ "wandb/openai/gpt-oss-120b": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 0.015,
+ "output_cost_per_token": 0.06,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/openai/gpt-oss-20b": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 0.005,
+ "output_cost_per_token": 0.02,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/zai-org/GLM-4.5": {
+ "max_tokens": 131072,
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "input_cost_per_token": 0.055,
+ "output_cost_per_token": 0.2,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/Qwen/Qwen3-235B-A22B-Instruct-2507": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 0.01,
+ "output_cost_per_token": 0.01,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/Qwen/Qwen3-Coder-480B-A35B-Instruct": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 0.1,
+ "output_cost_per_token": 0.15,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/Qwen/Qwen3-235B-A22B-Thinking-2507": {
+ "max_tokens": 262144,
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "input_cost_per_token": 0.01,
+ "output_cost_per_token": 0.01,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/moonshotai/Kimi-K2-Instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.135,
+ "output_cost_per_token": 0.4,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/meta-llama/Llama-3.1-8B-Instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.022,
+ "output_cost_per_token": 0.022,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/deepseek-ai/DeepSeek-V3.1": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.055,
+ "output_cost_per_token": 0.165,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/deepseek-ai/DeepSeek-R1-0528": {
+ "max_tokens": 161000,
+ "max_input_tokens": 161000,
+ "max_output_tokens": 161000,
+ "input_cost_per_token": 0.135,
+ "output_cost_per_token": 0.54,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/deepseek-ai/DeepSeek-V3-0324": {
+ "max_tokens": 161000,
+ "max_input_tokens": 161000,
+ "max_output_tokens": 161000,
+ "input_cost_per_token": 0.114,
+ "output_cost_per_token": 0.275,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/meta-llama/Llama-3.3-70B-Instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.071,
+ "output_cost_per_token": 0.071,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/meta-llama/Llama-4-Scout-17B-16E-Instruct": {
+ "max_tokens": 64000,
+ "max_input_tokens": 64000,
+ "max_output_tokens": 64000,
+ "input_cost_per_token": 0.017,
+ "output_cost_per_token": 0.066,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "wandb/microsoft/Phi-4-mini-instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.008,
+ "output_cost_per_token": 0.035,
+ "litellm_provider": "wandb",
+ "mode": "chat"
+ },
+ "watsonx/ibm/granite-3-8b-instruct": {
+ "input_cost_per_token": 0.0002,
+ "litellm_provider": "watsonx",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 1024,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 0.0002,
+ "supports_audio_input": false,
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "watsonx/mistralai/mistral-large": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "watsonx",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 16384,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_audio_input": false,
+ "supports_audio_output": false,
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": false,
+ "supports_prompt_caching": true,
+ "supports_response_schema": true,
+ "supports_system_messages": true,
+ "supports_tool_choice": true,
+ "supports_vision": false
+ },
+ "watsonx/bigscience/mt0-xxl-13b": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.0005,
+ "output_cost_per_token": 0.002,
+ "litellm_provider": "watsonx",
"mode": "chat",
"supports_function_calling": false,
"supports_parallel_function_calling": false,
- "supports_vision": false,
- "supports_system_messages": true,
- "supports_tool_choice": false
+ "supports_vision": false
+ },
+ "watsonx/core42/jais-13b-chat": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.0005,
+ "output_cost_per_token": 0.002,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/google/flan-t5-xl-3b": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.0001,
+ "output_cost_per_token": 0.00025,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-13b-chat-v2": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.0005,
+ "output_cost_per_token": 0.002,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-13b-instruct-v2": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.0005,
+ "output_cost_per_token": 0.002,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-3-3-8b-instruct": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.00025,
+ "output_cost_per_token": 0.001,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-4-h-small": {
+ "max_tokens": 20480,
+ "max_input_tokens": 20480,
+ "max_output_tokens": 20480,
+ "input_cost_per_token": 0.000625,
+ "output_cost_per_token": 0.0025,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-guardian-3-2-2b": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.00015,
+ "output_cost_per_token": 0.0006,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-guardian-3-3-8b": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.00025,
+ "output_cost_per_token": 0.001,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-ttm-1024-96-r2": {
+ "max_tokens": 512,
+ "max_input_tokens": 512,
+ "max_output_tokens": 512,
+ "input_cost_per_token": 0.000625,
+ "output_cost_per_token": 0.000625,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-ttm-1536-96-r2": {
+ "max_tokens": 512,
+ "max_input_tokens": 512,
+ "max_output_tokens": 512,
+ "input_cost_per_token": 0.000625,
+ "output_cost_per_token": 0.000625,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-ttm-512-96-r2": {
+ "max_tokens": 512,
+ "max_input_tokens": 512,
+ "max_output_tokens": 512,
+ "input_cost_per_token": 0.000625,
+ "output_cost_per_token": 0.000625,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/ibm/granite-vision-3-2-2b": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.00015,
+ "output_cost_per_token": 0.0006,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": true
+ },
+ "watsonx/meta-llama/llama-3-2-11b-vision-instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.00025,
+ "output_cost_per_token": 0.001,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": true
+ },
+ "watsonx/meta-llama/llama-3-2-1b-instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.0001,
+ "output_cost_per_token": 0.0002,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": false
+ },
+ "watsonx/meta-llama/llama-3-2-3b-instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.00015,
+ "output_cost_per_token": 0.0006,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": false
+ },
+ "watsonx/meta-llama/llama-3-2-90b-vision-instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.002,
+ "output_cost_per_token": 0.008,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": true
+ },
+ "watsonx/meta-llama/llama-3-3-70b-instruct": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.002,
+ "output_cost_per_token": 0.006,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": false
+ },
+ "watsonx/meta-llama/llama-4-maverick-17b": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.0005,
+ "output_cost_per_token": 0.002,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": false
+ },
+ "watsonx/meta-llama/llama-guard-3-11b-vision": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.00025,
+ "output_cost_per_token": 0.001,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": true
+ },
+ "watsonx/mistralai/mistral-medium-2505": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.00225,
+ "output_cost_per_token": 0.00675,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": false
+ },
+ "watsonx/mistralai/mistral-small-2503": {
+ "max_tokens": 32000,
+ "max_input_tokens": 32000,
+ "max_output_tokens": 32000,
+ "input_cost_per_token": 0.0002,
+ "output_cost_per_token": 0.0006,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": true,
+ "supports_parallel_function_calling": true,
+ "supports_vision": false
+ },
+ "watsonx/mistralai/pixtral-12b-2409": {
+ "max_tokens": 128000,
+ "max_input_tokens": 128000,
+ "max_output_tokens": 128000,
+ "input_cost_per_token": 0.00015,
+ "output_cost_per_token": 0.00015,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": true
+ },
+ "watsonx/openai/gpt-oss-120b": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.004,
+ "output_cost_per_token": 0.016,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+ "watsonx/sdaia/allam-1-13b-instruct": {
+ "max_tokens": 8192,
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "input_cost_per_token": 0.0005,
+ "output_cost_per_token": 0.002,
+ "litellm_provider": "watsonx",
+ "mode": "chat",
+ "supports_function_calling": false,
+ "supports_parallel_function_calling": false,
+ "supports_vision": false
+ },
+
+ "whisper-1": {
+ "input_cost_per_second": 0.0001,
+ "litellm_provider": "openai",
+ "mode": "audio_transcription",
+ "output_cost_per_second": 0.0001,
+ "supported_endpoints": [
+ "/v1/audio/transcriptions"
+ ]
+ },
+ "vertex_ai/qwen/qwen3-next-80b-a3b-instruct-maas": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-qwen_models",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "vertex_ai/qwen/qwen3-next-80b-a3b-thinking-maas": {
+ "input_cost_per_token": 1.5e-07,
+ "litellm_provider": "vertex_ai-qwen_models",
+ "max_input_tokens": 262144,
+ "max_output_tokens": 262144,
+ "max_tokens": 262144,
+ "mode": "chat",
+ "output_cost_per_token": 1.2e-06,
+ "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing",
+ "supports_function_calling": true,
+ "supports_tool_choice": true
+ },
+ "xai/grok-2": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-2-1212": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-2-latest": {
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-2-vision": {
+ "input_cost_per_image": 2e-06,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "xai/grok-2-vision-1212": {
+ "input_cost_per_image": 2e-06,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "xai/grok-2-vision-latest": {
+ "input_cost_per_image": 2e-06,
+ "input_cost_per_token": 2e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 32768,
+ "max_output_tokens": 32768,
+ "max_tokens": 32768,
+ "mode": "chat",
+ "output_cost_per_token": 1e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-beta": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-fast-beta": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-05,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-fast-latest": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 2.5e-05,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-latest": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-mini": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-mini-beta": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-mini-fast": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-mini-fast-beta": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-mini-fast-latest": {
+ "input_cost_per_token": 6e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 4e-06,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-3-mini-latest": {
+ "input_cost_per_token": 3e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 5e-07,
+ "source": "https://x.ai/api#pricing",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_response_schema": false,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-4": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://docs.x.ai/docs/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-4-fast-reasoning": {
+ "litellm_provider": "xai",
+ "max_input_tokens": 2e6,
+ "max_output_tokens": 2e6,
+ "max_tokens": 2e6,
+ "mode": "chat",
+ "input_cost_per_token": 0.2e-06,
+ "output_cost_per_token": 0.5e-06,
+ "cache_read_input_token_cost": 0.05e-06,
+ "source": "https://docs.x.ai/docs/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-4-fast-non-reasoning": {
+ "litellm_provider": "xai",
+ "max_input_tokens": 2e6,
+ "max_output_tokens": 2e6,
+ "cache_read_input_token_cost": 0.05e-06,
+ "max_tokens": 2e6,
+ "mode": "chat",
+ "input_cost_per_token": 0.2e-06,
+ "output_cost_per_token": 0.5e-06,
+ "source": "https://docs.x.ai/docs/models",
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-4-0709": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://docs.x.ai/docs/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-4-latest": {
+ "input_cost_per_token": 3e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "source": "https://docs.x.ai/docs/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true,
+ "supports_web_search": true
+ },
+ "xai/grok-beta": {
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 131072,
+ "max_output_tokens": 131072,
+ "max_tokens": 131072,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
+ },
+ "xai/grok-code-fast": {
+ "cache_read_input_token_cost": 2e-08,
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://docs.x.ai/docs/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "xai/grok-code-fast-1": {
+ "cache_read_input_token_cost": 2e-08,
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://docs.x.ai/docs/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "xai/grok-code-fast-1-0825": {
+ "cache_read_input_token_cost": 2e-08,
+ "input_cost_per_token": 2e-07,
+ "litellm_provider": "xai",
+ "max_input_tokens": 256000,
+ "max_output_tokens": 256000,
+ "max_tokens": 256000,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-06,
+ "source": "https://docs.x.ai/docs/models",
+ "supports_function_calling": true,
+ "supports_reasoning": true,
+ "supports_tool_choice": true
+ },
+ "xai/grok-vision-beta": {
+ "input_cost_per_image": 5e-06,
+ "input_cost_per_token": 5e-06,
+ "litellm_provider": "xai",
+ "max_input_tokens": 8192,
+ "max_output_tokens": 8192,
+ "max_tokens": 8192,
+ "mode": "chat",
+ "output_cost_per_token": 1.5e-05,
+ "supports_function_calling": true,
+ "supports_tool_choice": true,
+ "supports_vision": true,
+ "supports_web_search": true
}
}
diff --git a/litellm/mypy.ini b/litellm/mypy.ini
index c084de7c563..4702b591124 100644
--- a/litellm/mypy.ini
+++ b/litellm/mypy.ini
@@ -5,10 +5,15 @@ mypy_path = litellm/stubs
namespace_packages = True
disable_error_code =
valid-type,
- annotation-unchecked
+ annotation-unchecked,
+ import-untyped
[mypy-google.*]
ignore_missing_imports = True
[mypy-cryptography.hazmat.bindings._rust.x509]
+ignore_errors = True
+
+[mypy-fastuuid.*]
+ignore_missing_imports = True
ignore_errors = True
\ No newline at end of file
diff --git a/litellm/passthrough/main.py b/litellm/passthrough/main.py
index 59fab1b3369..b4a76822022 100644
--- a/litellm/passthrough/main.py
+++ b/litellm/passthrough/main.py
@@ -24,6 +24,7 @@ import litellm
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
+from litellm.passthrough.utils import CommonUtils
from litellm.utils import client
base_llm_http_handler = BaseLLMHTTPHandler()
@@ -241,6 +242,14 @@ def llm_passthrough_route(
request_query_params=request_query_params,
litellm_params=litellm_params_dict,
)
+
+ # [TODO: Refactor to bedrockpassthroughconfig] need to encode the id of application-inference-profile for bedrock
+ if custom_llm_provider == "bedrock" and "application-inference-profile" in endpoint:
+ encoded_url_str = CommonUtils.encode_bedrock_runtime_modelid_arn(
+ str(updated_url)
+ )
+ updated_url = httpx.URL(encoded_url_str)
+
# Add or update query parameters
provider_api_key = provider_config.get_api_key(api_key)
diff --git a/litellm/passthrough/utils.py b/litellm/passthrough/utils.py
index c52d0e3688d..4bf66d49881 100644
--- a/litellm/passthrough/utils.py
+++ b/litellm/passthrough/utils.py
@@ -37,3 +37,56 @@ class BasePassthroughUtils:
# Combine request headers with custom headers
headers = {**request_headers, **headers}
return headers
+
+class CommonUtils:
+ @staticmethod
+ def encode_bedrock_runtime_modelid_arn(endpoint: str) -> str:
+ """
+ Encodes any "/" found in the modelId of an AWS Bedrock Runtime Endpoint when arns are passed in.
+ - modelID value can be an ARN which contains slashes that SHOULD NOT be treated as path separators.
+ e.g endpoint: /model//invoke
+ containing arns with slashes need to be encoded from
+ arn:aws:bedrock:ap-southeast-1:123456789012:application-inference-profile/abdefg12334 =>
+ arn:aws:bedrock:ap-southeast-1:123456789012:application-inference-profile%2Fabdefg12334
+ so that it is treated as one part of the path.
+ Otherwise, the encoded endpoint will return 500 error when passed to Bedrock endpoint.
+
+ See the apis in https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Amazon_Bedrock_Runtime.html
+ for more details on the regex patterns of modelId which we use in the regex logic below.
+
+ Args:
+ endpoint (str): The original endpoint string which may contain ARNs that contain slashes.
+
+ Returns:
+ str: The endpoint with properly encoded ARN slashes
+ """
+ import re
+
+ # Early exit: if no ARN detected, return unchanged
+ if 'arn:aws:' not in endpoint:
+ return endpoint
+
+ # Handle all patterns in one go - more efficient and cleaner
+ patterns = [
+ # Custom model with 2 slashes (order matters - do this first)
+ (r'(custom-model)/([a-z0-9.-]+)/([a-z0-9]+)', r'\1%2F\2%2F\3'),
+
+ # All other resource types with 1 slash
+ (r'(:application-inference-profile)/', r'\1%2F'),
+ (r'(:inference-profile)/', r'\1%2F'),
+ (r'(:foundation-model)/', r'\1%2F'),
+ (r'(:imported-model)/', r'\1%2F'),
+ (r'(:provisioned-model)/', r'\1%2F'),
+ (r'(:prompt)/', r'\1%2F'),
+ (r'(:endpoint)/', r'\1%2F'),
+ (r'(:prompt-router)/', r'\1%2F'),
+ (r'(:default-prompt-router)/', r'\1%2F'),
+ ]
+
+ for pattern, replacement in patterns:
+ # Check if pattern exists before applying regex (early exit optimization)
+ if re.search(pattern, endpoint):
+ endpoint = re.sub(pattern, replacement, endpoint)
+ break # Exit after first match since each ARN has only one resource type
+
+ return endpoint
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/mcp_server/auth/litellm_auth_handler.py b/litellm/proxy/_experimental/mcp_server/auth/litellm_auth_handler.py
index cb4aab4bb1d..081d83dd1c8 100644
--- a/litellm/proxy/_experimental/mcp_server/auth/litellm_auth_handler.py
+++ b/litellm/proxy/_experimental/mcp_server/auth/litellm_auth_handler.py
@@ -1,4 +1,4 @@
-from typing import List, Optional, Dict
+from typing import Dict, List, Optional
from mcp.server.auth.middleware.bearer_auth import AuthenticatedUser
@@ -8,16 +8,30 @@ from litellm.proxy._types import UserAPIKeyAuth
class MCPAuthenticatedUser(AuthenticatedUser):
"""
Wrapper class to make LiteLLM's authentication and configuration compatible with MCP's AuthenticatedUser.
-
+
This class handles:
1. User API key authentication information
2. MCP authentication header (deprecated)
3. MCP server configuration (can include access groups)
4. Server-specific authentication headers
+ 5. OAuth2 headers
+ 6. Raw headers - allows forwarding specific headers to the MCP server, specified by the admin.
"""
- def __init__(self, user_api_key_auth: UserAPIKeyAuth, mcp_auth_header: Optional[str] = None, mcp_servers: Optional[List[str]] = None, mcp_server_auth_headers: Optional[Dict[str, str]] = None):
+ def __init__(
+ self,
+ user_api_key_auth: UserAPIKeyAuth,
+ mcp_auth_header: Optional[str] = None,
+ mcp_servers: Optional[List[str]] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
+ oauth2_headers: Optional[Dict[str, str]] = None,
+ mcp_protocol_version: Optional[str] = None,
+ raw_headers: Optional[Dict[str, str]] = None,
+ ):
self.user_api_key_auth = user_api_key_auth
self.mcp_auth_header = mcp_auth_header
self.mcp_servers = mcp_servers
self.mcp_server_auth_headers = mcp_server_auth_headers or {}
+ self.mcp_protocol_version = mcp_protocol_version
+ self.oauth2_headers = oauth2_headers
+ self.raw_headers = raw_headers
diff --git a/litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py b/litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
index 0f4acce21f2..e77ad11fae4 100644
--- a/litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
+++ b/litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py
@@ -1,4 +1,4 @@
-from typing import List, Optional, Tuple, Dict
+from typing import Dict, List, Optional, Set, Tuple
from starlette.datastructures import Headers
from starlette.requests import Request
@@ -29,14 +29,28 @@ class MCPRequestHandler:
LITELLM_MCP_ACCESS_GROUPS_HEADER_NAME = SpecialHeaders.mcp_access_groups.value
+ # MCP Protocol Version header
+ MCP_PROTOCOL_VERSION_HEADER_NAME = "MCP-Protocol-Version"
+
@staticmethod
- async def process_mcp_request(scope: Scope) -> Tuple[UserAPIKeyAuth, Optional[str], Optional[List[str]], Optional[Dict[str, str]]]:
+ async def process_mcp_request(
+ scope: Scope,
+ ) -> Tuple[
+ UserAPIKeyAuth,
+ Optional[str],
+ Optional[List[str]],
+ Optional[Dict[str, Dict[str, str]]],
+ Optional[Dict[str, str]],
+ Optional[Dict[str, str]],
+ ]:
"""
Process and validate MCP request headers from the ASGI scope.
This includes:
1. Extracting and validating authentication headers
2. Processing MCP server configuration
3. Handling MCP-specific headers
+ 4. Handling oauth2 headers
+ 5. Raw headers - allows forwarding specific headers to the MCP server, specified by the admin.
Args:
scope: ASGI scope containing request information
@@ -46,7 +60,8 @@ class MCPRequestHandler:
mcp_auth_header: Optional[str] MCP auth header to be passed to the MCP server (deprecated)
mcp_servers: Optional[List[str]] List of MCP servers and access groups to use
mcp_server_auth_headers: Optional[Dict[str, str]] Server-specific auth headers in format {server_alias: auth_value}
-
+ oauth2_headers: Optional[Dict[str, str]] OAuth2 headers
+ raw_headers: Optional[Dict[str, str]] Raw headers to be forwarded to the MCP server
Raises:
HTTPException: If headers are invalid or missing required headers
"""
@@ -54,36 +69,69 @@ class MCPRequestHandler:
litellm_api_key = (
MCPRequestHandler.get_litellm_api_key_from_headers(headers) or ""
)
-
+
# Get the old mcp_auth_header for backward compatibility
mcp_auth_header = MCPRequestHandler._get_mcp_auth_header_from_headers(headers)
-
+
# Get the new server-specific auth headers
- mcp_server_auth_headers = MCPRequestHandler._get_mcp_server_auth_headers_from_headers(headers)
+ mcp_server_auth_headers = (
+ MCPRequestHandler._get_mcp_server_auth_headers_from_headers(headers)
+ )
+
+ # Get the oauth2 headers
+ oauth2_headers = MCPRequestHandler._get_oauth2_headers_from_headers(headers)
# Parse MCP servers from header
- mcp_servers_header = headers.get(MCPRequestHandler.LITELLM_MCP_SERVERS_HEADER_NAME)
+ mcp_servers_header = headers.get(
+ MCPRequestHandler.LITELLM_MCP_SERVERS_HEADER_NAME
+ )
verbose_logger.debug(f"Raw MCP servers header: {mcp_servers_header}")
mcp_servers = None
if mcp_servers_header is not None:
try:
- mcp_servers = [s.strip() for s in mcp_servers_header.split(",") if s.strip()]
+ mcp_servers = [
+ s.strip() for s in mcp_servers_header.split(",") if s.strip()
+ ]
verbose_logger.debug(f"Parsed MCP servers: {mcp_servers}")
except Exception as e:
verbose_logger.debug(f"Error parsing mcp_servers header: {e}")
mcp_servers = None
- if mcp_servers_header == "" or (mcp_servers is not None and len(mcp_servers) == 0):
+ if mcp_servers_header == "" or (
+ mcp_servers is not None and len(mcp_servers) == 0
+ ):
mcp_servers = []
# Create a proper Request object with mock body method to avoid ASGI receive channel issues
request = Request(scope=scope)
+
async def mock_body():
return b"{}"
+
request.body = mock_body # type: ignore
- validated_user_api_key_auth = await user_api_key_auth(
- api_key=litellm_api_key, request=request
+ if ".well-known" in str(request.url): # public routes
+ validated_user_api_key_auth = UserAPIKeyAuth()
+ # elif litellm_api_key == "":
+ # from fastapi import HTTPException
+
+ # raise HTTPException(
+ # status_code=401,
+ # detail="LiteLLM API key is missing. Please add it or use OAuth authentication.",
+ # headers={
+ # "WWW-Authenticate": f'Bearer resource_metadata=f"{request.base_url}/.well-known/oauth-protected-resource"',
+ # },
+ # )
+ else:
+ validated_user_api_key_auth = await user_api_key_auth(
+ api_key=litellm_api_key, request=request
+ )
+
+ return (
+ validated_user_api_key_auth,
+ mcp_auth_header,
+ mcp_servers,
+ mcp_server_auth_headers,
+ oauth2_headers,
+ dict(headers),
)
- return validated_user_api_key_auth, mcp_auth_header, mcp_servers, mcp_server_auth_headers
-
@staticmethod
def _get_mcp_auth_header_from_headers(headers: Headers) -> Optional[str]:
@@ -97,10 +145,12 @@ class MCPRequestHandler:
Support this auth: https://docs.litellm.ai/docs/mcp#using-your-mcp-with-client-side-credentials
If you want to use a different header name, you can set the `LITELLM_MCP_CLIENT_SIDE_AUTH_HEADER_NAME` in the secret manager or `mcp_client_side_auth_header_name` in the general settings.
-
+
DEPRECATED: This method is deprecated in favor of server-specific auth headers using the format x-mcp-{{server_alias}}-{{header_name}} instead.
"""
- mcp_client_side_auth_header_name: str = MCPRequestHandler._get_mcp_client_side_auth_header_name()
+ mcp_client_side_auth_header_name: str = (
+ MCPRequestHandler._get_mcp_client_side_auth_header_name()
+ )
auth_header = headers.get(mcp_client_side_auth_header_name)
if auth_header:
verbose_logger.warning(
@@ -108,42 +158,73 @@ class MCPRequestHandler:
f"Please use server-specific auth headers in the format 'x-mcp-{{server_alias}}-{{header_name}}' instead."
)
return auth_header
-
+
@staticmethod
- def _get_mcp_server_auth_headers_from_headers(headers: Headers) -> Dict[str, str]:
+ def _get_mcp_server_auth_headers_from_headers(
+ headers: Headers,
+ ) -> Dict[str, Dict[str, str]]:
"""
Parse server-specific MCP auth headers from the request headers.
-
+
Looks for headers in the format: x-mcp-{server_alias}-{header_name}
Examples:
- x-mcp-github-authorization: Bearer token123
- x-mcp-zapier-x-api-key: api_key_456
- x-mcp-deepwiki-authorization: Basic base64_encoded_creds
-
+
Returns:
- Dict[str, str]: Mapping of server alias to auth value
+ Dict[str, Dict[str, str]]: Mapping of server alias to header dict
"""
- server_auth_headers = {}
+ server_auth_headers: Dict[str, Dict[str, str]] = {}
prefix = "x-mcp-"
-
+
for header_name, header_value in headers.items():
if header_name.lower().startswith(prefix):
# Skip the access groups header as it's not a server auth header
- if header_name.lower() == MCPRequestHandler.LITELLM_MCP_ACCESS_GROUPS_HEADER_NAME.lower() or header_name.lower() == MCPRequestHandler.LITELLM_MCP_SERVERS_HEADER_NAME.lower():
+ if (
+ header_name.lower()
+ == MCPRequestHandler.LITELLM_MCP_ACCESS_GROUPS_HEADER_NAME.lower()
+ or header_name.lower()
+ == MCPRequestHandler.LITELLM_MCP_SERVERS_HEADER_NAME.lower()
+ ):
continue
-
+
# Extract server_alias and header_name from x-mcp-{server_alias}-{header_name}
- remaining = header_name[len(prefix):].lower()
- if '-' in remaining:
- # Split on the last dash to separate server_alias from header_name
- parts = remaining.rsplit('-', 1)
+ remaining = header_name[len(prefix) :].lower()
+ if "-" in remaining:
+ # Split on the first dash to separate server_alias from header_name
+ parts = remaining.split("-", 1)
if len(parts) == 2:
server_alias, auth_header_name = parts
- server_auth_headers[server_alias] = header_value
- verbose_logger.debug(f"Found server auth header: {server_alias} -> {auth_header_name}: {header_value[:10]}...")
-
+
+ # Convert common header names to proper case
+ if auth_header_name == "authorization":
+ auth_header_name = "Authorization"
+
+ # Initialize server dict if not exists
+ if server_alias not in server_auth_headers:
+ server_auth_headers[server_alias] = {}
+
+ server_auth_headers[server_alias][
+ auth_header_name
+ ] = header_value
+ verbose_logger.debug(
+ f"Found server auth header: {server_alias} -> {auth_header_name}: {header_value[:10]}..."
+ )
+
return server_auth_headers
-
+
+ @staticmethod
+ def _get_oauth2_headers_from_headers(headers: Headers) -> Dict[str, str]:
+ """
+ Get the oauth2 headers from the request headers.
+ """
+ oauth2_headers = {}
+ for header_name, header_value in headers.items():
+ if header_name.lower().startswith("authorization"):
+ oauth2_headers["Authorization"] = header_value
+ return oauth2_headers
+
@staticmethod
def _get_mcp_client_side_auth_header_name() -> str:
"""
@@ -155,13 +236,21 @@ class MCPRequestHandler:
"""
from litellm.proxy.proxy_server import general_settings
from litellm.secret_managers.main import get_secret_str
- MCP_CLIENT_SIDE_AUTH_HEADER_NAME: str = MCPRequestHandler.LITELLM_MCP_AUTH_HEADER_NAME
- if get_secret_str("LITELLM_MCP_CLIENT_SIDE_AUTH_HEADER_NAME") is not None:
- MCP_CLIENT_SIDE_AUTH_HEADER_NAME = get_secret_str("LITELLM_MCP_CLIENT_SIDE_AUTH_HEADER_NAME") or MCP_CLIENT_SIDE_AUTH_HEADER_NAME
- elif general_settings.get("mcp_client_side_auth_header_name") is not None:
- MCP_CLIENT_SIDE_AUTH_HEADER_NAME = general_settings.get("mcp_client_side_auth_header_name") or MCP_CLIENT_SIDE_AUTH_HEADER_NAME
- return MCP_CLIENT_SIDE_AUTH_HEADER_NAME
+ MCP_CLIENT_SIDE_AUTH_HEADER_NAME: str = (
+ MCPRequestHandler.LITELLM_MCP_AUTH_HEADER_NAME
+ )
+ if get_secret_str("LITELLM_MCP_CLIENT_SIDE_AUTH_HEADER_NAME") is not None:
+ MCP_CLIENT_SIDE_AUTH_HEADER_NAME = (
+ get_secret_str("LITELLM_MCP_CLIENT_SIDE_AUTH_HEADER_NAME")
+ or MCP_CLIENT_SIDE_AUTH_HEADER_NAME
+ )
+ elif general_settings.get("mcp_client_side_auth_header_name") is not None:
+ MCP_CLIENT_SIDE_AUTH_HEADER_NAME = (
+ general_settings.get("mcp_client_side_auth_header_name")
+ or MCP_CLIENT_SIDE_AUTH_HEADER_NAME
+ )
+ return MCP_CLIENT_SIDE_AUTH_HEADER_NAME
@staticmethod
def get_litellm_api_key_from_headers(headers: Headers) -> Optional[str]:
@@ -216,34 +305,235 @@ class MCPRequestHandler:
) -> List[str]:
"""
Get list of allowed MCP servers for the given user/key based on permissions
+
+ Returns:
+ List[str]: List of allowed MCP servers by server id
"""
from typing import List
- allowed_mcp_servers: List[str] = []
- allowed_mcp_servers_for_key = (
- await MCPRequestHandler._get_allowed_mcp_servers_for_key(user_api_key_auth)
- )
- allowed_mcp_servers_for_team = (
- await MCPRequestHandler._get_allowed_mcp_servers_for_team(user_api_key_auth)
+ try:
+ allowed_mcp_servers: List[str] = []
+ allowed_mcp_servers_for_key = (
+ await MCPRequestHandler._get_allowed_mcp_servers_for_key(
+ user_api_key_auth
+ )
+ )
+ allowed_mcp_servers_for_team = (
+ await MCPRequestHandler._get_allowed_mcp_servers_for_team(
+ user_api_key_auth
+ )
+ )
+
+ #########################################################
+ # If team has mcp_servers, handle inheritance and intersection logic
+ #########################################################
+ if len(allowed_mcp_servers_for_team) > 0:
+ if len(allowed_mcp_servers_for_key) > 0:
+ # Key has its own MCP permissions - use intersection with team permissions
+ for _mcp_server in allowed_mcp_servers_for_key:
+ if _mcp_server in allowed_mcp_servers_for_team:
+ allowed_mcp_servers.append(_mcp_server)
+ else:
+ # Key has no MCP permissions - inherit from team
+ allowed_mcp_servers = allowed_mcp_servers_for_team
+ else:
+ allowed_mcp_servers = allowed_mcp_servers_for_key
+
+ return list(set(allowed_mcp_servers))
+ except Exception as e:
+ verbose_logger.warning(f"Failed to get allowed MCP servers: {str(e)}")
+ return []
+
+ @staticmethod
+ async def _get_key_object_permission(
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ ):
+ """Helper to get key object_permission from cache or DB."""
+ from litellm.proxy.auth.auth_checks import get_object_permission
+ from litellm.proxy.proxy_server import (
+ prisma_client,
+ proxy_logging_obj,
+ user_api_key_cache,
)
- #########################################################
- # If team has mcp_servers, then key must have a subset of the team's mcp_servers
- #########################################################
- if len(allowed_mcp_servers_for_team) > 0:
- for _mcp_server in allowed_mcp_servers_for_key:
- if _mcp_server in allowed_mcp_servers_for_team:
- allowed_mcp_servers.append(_mcp_server)
- else:
- allowed_mcp_servers = allowed_mcp_servers_for_key
+ if not user_api_key_auth:
+ return None
- return list(set(allowed_mcp_servers))
+ # Already loaded
+ if user_api_key_auth.object_permission:
+ return user_api_key_auth.object_permission
+
+ # Need to fetch from DB
+ if user_api_key_auth.object_permission_id and prisma_client:
+ return await get_object_permission(
+ object_permission_id=user_api_key_auth.object_permission_id,
+ prisma_client=prisma_client,
+ user_api_key_cache=user_api_key_cache,
+ parent_otel_span=user_api_key_auth.parent_otel_span,
+ proxy_logging_obj=proxy_logging_obj,
+ )
+
+ return None
+
+ @staticmethod
+ async def _get_team_object_permission(
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ ):
+ """Helper to get team object_permission from cache or DB."""
+ from litellm.proxy.auth.auth_checks import (
+ get_object_permission,
+ get_team_object,
+ )
+ from litellm.proxy.proxy_server import (
+ prisma_client,
+ proxy_logging_obj,
+ user_api_key_cache,
+ )
+
+ if not user_api_key_auth or not user_api_key_auth.team_id or not prisma_client:
+ return None
+
+ # First get the team object (which may have object_permission already loaded)
+ team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
+ team_id=user_api_key_auth.team_id,
+ prisma_client=prisma_client,
+ user_api_key_cache=user_api_key_cache,
+ parent_otel_span=user_api_key_auth.parent_otel_span,
+ proxy_logging_obj=proxy_logging_obj,
+ )
+
+ if not team_obj:
+ return None
+
+ # Already loaded
+ if team_obj.object_permission:
+ return team_obj.object_permission
+
+ # Need to fetch from DB using object_permission_id
+ if team_obj.object_permission_id:
+ return await get_object_permission(
+ object_permission_id=team_obj.object_permission_id,
+ prisma_client=prisma_client,
+ user_api_key_cache=user_api_key_cache,
+ parent_otel_span=user_api_key_auth.parent_otel_span,
+ proxy_logging_obj=proxy_logging_obj,
+ )
+
+ return None
+
+ @staticmethod
+ async def get_allowed_tools_for_server(
+ server_id: str,
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ ) -> Optional[List[str]]:
+ """
+ Get list of allowed tool names for a specific server based on key/team permissions.
+ Follows same inheritance logic as get_allowed_mcp_servers.
+
+ Args:
+ server_id: Server ID to check permissions for
+ user_api_key_auth: User auth
+
+ Returns:
+ List[str] if restrictions exist, None if no restrictions (allow all)
+ """
+ if not user_api_key_auth:
+ return None
+
+ try:
+ # Get key and team object permissions
+ key_obj_perm = await MCPRequestHandler._get_key_object_permission(
+ user_api_key_auth
+ )
+ team_obj_perm = await MCPRequestHandler._get_team_object_permission(
+ user_api_key_auth
+ )
+
+ # Extract tool permissions for this server
+ key_tools = (
+ key_obj_perm.mcp_tool_permissions.get(server_id)
+ if key_obj_perm and key_obj_perm.mcp_tool_permissions
+ else None
+ )
+ team_tools = (
+ team_obj_perm.mcp_tool_permissions.get(server_id)
+ if team_obj_perm and team_obj_perm.mcp_tool_permissions
+ else None
+ )
+
+ # Apply same inheritance logic as get_allowed_mcp_servers
+ if team_tools:
+ if key_tools:
+ # Both have restrictions → intersection
+ return list(set(team_tools) & set(key_tools))
+ else:
+ # Only team has restrictions → inherit from team
+ return team_tools
+ else:
+ # No team restrictions → use key restrictions
+ return key_tools
+
+ except Exception as e:
+ verbose_logger.warning(f"Failed to get allowed tools for server: {str(e)}")
+ return None
+
+ @staticmethod
+ async def is_tool_allowed_for_server(
+ tool_name: str,
+ server_id: str,
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ ) -> bool:
+ """
+ Check if a specific tool is allowed for a server based on key/team permissions.
+
+ Args:
+ tool_name: Name of the tool to check
+ server_id: Server ID
+ user_api_key_auth: User auth
+
+ Returns:
+ True if allowed, False if blocked
+ """
+ allowed_tools = await MCPRequestHandler.get_allowed_tools_for_server(
+ server_id=server_id,
+ user_api_key_auth=user_api_key_auth,
+ )
+
+ # None means no restrictions (allow all)
+ if allowed_tools is None:
+ return True
+
+ # Empty list means no tools allowed
+ if not allowed_tools:
+ return False
+
+ # Check if tool is in allowed list
+ return tool_name in allowed_tools
+
+ @staticmethod
+ def is_tool_allowed(
+ allowed_mcp_servers: List[str],
+ server_name: str,
+ ) -> bool:
+ """
+ Check if the tool is allowed for the given user/key based on permissions
+ """
+ if len(allowed_mcp_servers) == 0:
+ return True
+ elif server_name in allowed_mcp_servers:
+ return True
+ return False
@staticmethod
async def _get_allowed_mcp_servers_for_key(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
) -> List[str]:
- from litellm.proxy.proxy_server import prisma_client
+ from litellm.proxy.auth.auth_checks import get_object_permission
+ from litellm.proxy.proxy_server import (
+ prisma_client,
+ proxy_logging_obj,
+ user_api_key_cache,
+ )
if user_api_key_auth is None:
return []
@@ -255,79 +545,89 @@ class MCPRequestHandler:
verbose_logger.debug("prisma_client is None")
return []
- key_object_permission = (
- await prisma_client.db.litellm_objectpermissiontable.find_unique(
- where={"object_permission_id": user_api_key_auth.object_permission_id},
+ try:
+ key_object_permission = await get_object_permission(
+ object_permission_id=user_api_key_auth.object_permission_id,
+ prisma_client=prisma_client,
+ user_api_key_cache=user_api_key_cache,
+ parent_otel_span=user_api_key_auth.parent_otel_span,
+ proxy_logging_obj=proxy_logging_obj,
)
- )
- if key_object_permission is None:
- return []
+ if key_object_permission is None:
+ return []
- # Get direct MCP servers
- direct_mcp_servers = key_object_permission.mcp_servers or []
-
- # Get MCP servers from access groups
- access_group_servers = await MCPRequestHandler._get_mcp_servers_from_access_groups(
- key_object_permission.mcp_access_groups or []
- )
-
- # Combine both lists
- all_servers = direct_mcp_servers + access_group_servers
- return list(set(all_servers))
+ # Get direct MCP servers
+ direct_mcp_servers = key_object_permission.mcp_servers or []
+
+ # Get MCP servers from access groups
+ access_group_servers = (
+ await MCPRequestHandler._get_mcp_servers_from_access_groups(
+ key_object_permission.mcp_access_groups or []
+ )
+ )
+
+ # Combine both lists
+ all_servers = direct_mcp_servers + access_group_servers
+ return list(set(all_servers))
+ except Exception as e:
+ verbose_logger.warning(
+ f"Failed to get allowed MCP servers for key: {str(e)}"
+ )
+ return []
@staticmethod
async def _get_allowed_mcp_servers_for_team(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
) -> List[str]:
"""
- The `object_permission` for a team is not stored on the user_api_key_auth object
+ Get allowed MCP servers for a team.
- first we check if the team has a object_permission_id attached
- - if it does then we look up the object_permission for the team
+ Uses the helper _get_team_object_permission which:
+ 1. First checks if object_permission is already loaded on the team
+ 2. If not, fetches from DB using object_permission_id if it exists
"""
- from litellm.proxy.proxy_server import prisma_client
-
if user_api_key_auth is None:
return []
if user_api_key_auth.team_id is None:
return []
- if prisma_client is None:
- verbose_logger.debug("prisma_client is None")
- return []
-
- team_obj: Optional[LiteLLM_TeamTable] = (
- await prisma_client.db.litellm_teamtable.find_unique(
- where={"team_id": user_api_key_auth.team_id},
+ try:
+ # Use the helper method that properly handles fetching from DB if needed
+ object_permissions = await MCPRequestHandler._get_team_object_permission(
+ user_api_key_auth
)
- )
- if team_obj is None:
- verbose_logger.debug("team_obj is None")
- return []
- object_permissions = team_obj.object_permission
- if object_permissions is None:
- return []
+ if object_permissions is None:
+ return []
- # Get direct MCP servers
- direct_mcp_servers = object_permissions.mcp_servers or []
-
- # Get MCP servers from access groups
- access_group_servers = await MCPRequestHandler._get_mcp_servers_from_access_groups(
- object_permissions.mcp_access_groups or []
- )
-
- # Combine both lists
- all_servers = direct_mcp_servers + access_group_servers
- return list(set(all_servers))
+ # Get direct MCP servers
+ direct_mcp_servers = object_permissions.mcp_servers or []
+
+ # Get MCP servers from access groups
+ access_group_servers = (
+ await MCPRequestHandler._get_mcp_servers_from_access_groups(
+ object_permissions.mcp_access_groups or []
+ )
+ )
+
+ # Combine both lists
+ all_servers = direct_mcp_servers + access_group_servers
+ return list(set(all_servers))
+ except Exception as e:
+ verbose_logger.warning(
+ f"Failed to get allowed MCP servers for team: {str(e)}"
+ )
+ return []
@staticmethod
- def _get_config_server_ids_for_access_groups(config_mcp_servers, access_groups: List[str]) -> set:
+ def _get_config_server_ids_for_access_groups(
+ config_mcp_servers, access_groups: List[str]
+ ) -> Set[str]:
"""
Helper to get server_ids from config-loaded servers that match any of the given access groups.
"""
- server_ids = set()
+ server_ids: Set[str] = set()
for server_id, server in config_mcp_servers.items():
if server.access_groups:
if any(group in server.access_groups for group in access_groups):
@@ -335,48 +635,60 @@ class MCPRequestHandler:
return server_ids
@staticmethod
- async def _get_db_server_ids_for_access_groups(prisma_client, access_groups: List[str]) -> set:
+ async def _get_db_server_ids_for_access_groups(
+ prisma_client, access_groups: List[str]
+ ) -> Set[str]:
"""
Helper to get server_ids from DB servers that match any of the given access groups.
"""
- server_ids = set()
+ server_ids: Set[str] = set()
if access_groups and prisma_client is not None:
try:
mcp_servers = await prisma_client.db.litellm_mcpservertable.find_many(
- where={
- "mcp_access_groups": {
- "hasSome": access_groups
- }
- }
+ where={"mcp_access_groups": {"hasSome": access_groups}}
)
for server in mcp_servers:
server_ids.add(server.server_id)
except Exception as e:
- verbose_logger.debug(f"Error getting MCP servers from access groups: {e}")
+ verbose_logger.debug(
+ f"Error getting MCP servers from access groups: {e}"
+ )
return server_ids
@staticmethod
async def _get_mcp_servers_from_access_groups(
- access_groups: List[str]
+ access_groups: List[str],
) -> List[str]:
"""
Resolve MCP access groups to server IDs by querying BOTH the MCP server table (DB) AND config-loaded servers
"""
from litellm.proxy.proxy_server import prisma_client
- from litellm.proxy._experimental.mcp_server.mcp_server_manager import global_mcp_server_manager
- # Use the new helper for config-loaded servers
- server_ids = MCPRequestHandler._get_config_server_ids_for_access_groups(
- global_mcp_server_manager.config_mcp_servers, access_groups
- )
+ try:
+ # Import here to avoid circular import
+ from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
+ global_mcp_server_manager,
+ )
- # Use the new helper for DB servers
- db_server_ids = await MCPRequestHandler._get_db_server_ids_for_access_groups(
- prisma_client, access_groups
- )
- server_ids.update(db_server_ids)
+ # Use the new helper for config-loaded servers
+ server_ids = MCPRequestHandler._get_config_server_ids_for_access_groups(
+ global_mcp_server_manager.config_mcp_servers, access_groups
+ )
- return list(server_ids)
+ # Use the new helper for DB servers
+ db_server_ids = (
+ await MCPRequestHandler._get_db_server_ids_for_access_groups(
+ prisma_client, access_groups
+ )
+ )
+ server_ids.update(db_server_ids)
+
+ return list(server_ids)
+ except Exception as e:
+ verbose_logger.warning(
+ f"Failed to get MCP servers from access groups: {str(e)}"
+ )
+ return []
@staticmethod
async def get_mcp_access_groups(
@@ -388,8 +700,8 @@ class MCPRequestHandler:
from typing import List
access_groups: List[str] = []
- access_groups_for_key = (
- await MCPRequestHandler._get_mcp_access_groups_for_key(user_api_key_auth)
+ access_groups_for_key = await MCPRequestHandler._get_mcp_access_groups_for_key(
+ user_api_key_auth
)
access_groups_for_team = (
await MCPRequestHandler._get_mcp_access_groups_for_team(user_api_key_auth)
@@ -411,7 +723,12 @@ class MCPRequestHandler:
async def _get_mcp_access_groups_for_key(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
) -> List[str]:
- from litellm.proxy.proxy_server import prisma_client
+ from litellm.proxy.auth.auth_checks import get_object_permission
+ from litellm.proxy.proxy_server import (
+ prisma_client,
+ proxy_logging_obj,
+ user_api_key_cache,
+ )
if user_api_key_auth is None:
return []
@@ -423,15 +740,21 @@ class MCPRequestHandler:
verbose_logger.debug("prisma_client is None")
return []
- key_object_permission = (
- await prisma_client.db.litellm_objectpermissiontable.find_unique(
- where={"object_permission_id": user_api_key_auth.object_permission_id},
+ try:
+ key_object_permission = await get_object_permission(
+ object_permission_id=user_api_key_auth.object_permission_id,
+ prisma_client=prisma_client,
+ user_api_key_cache=user_api_key_cache,
+ parent_otel_span=user_api_key_auth.parent_otel_span,
+ proxy_logging_obj=proxy_logging_obj,
)
- )
- if key_object_permission is None:
- return []
+ if key_object_permission is None:
+ return []
- return key_object_permission.mcp_access_groups or []
+ return key_object_permission.mcp_access_groups or []
+ except Exception as e:
+ verbose_logger.warning(f"Failed to get MCP access groups for key: {str(e)}")
+ return []
@staticmethod
async def _get_mcp_access_groups_for_team(
@@ -440,7 +763,12 @@ class MCPRequestHandler:
"""
Get MCP access groups for the team
"""
- from litellm.proxy.proxy_server import prisma_client
+ from litellm.proxy.auth.auth_checks import get_team_object
+ from litellm.proxy.proxy_server import (
+ prisma_client,
+ proxy_logging_obj,
+ user_api_key_cache,
+ )
if user_api_key_auth is None:
return []
@@ -452,30 +780,42 @@ class MCPRequestHandler:
verbose_logger.debug("prisma_client is None")
return []
- team_obj: Optional[LiteLLM_TeamTable] = (
- await prisma_client.db.litellm_teamtable.find_unique(
- where={"team_id": user_api_key_auth.team_id},
+ try:
+ team_obj: Optional[LiteLLM_TeamTable] = await get_team_object(
+ team_id=user_api_key_auth.team_id,
+ prisma_client=prisma_client,
+ user_api_key_cache=user_api_key_cache,
+ parent_otel_span=user_api_key_auth.parent_otel_span,
+ proxy_logging_obj=proxy_logging_obj,
)
- )
- if team_obj is None:
- verbose_logger.debug("team_obj is None")
- return []
+ if team_obj is None:
+ verbose_logger.debug("team_obj is None")
+ return []
- object_permissions = team_obj.object_permission
- if object_permissions is None:
- return []
+ object_permissions = team_obj.object_permission
+ if object_permissions is None:
+ return []
- return object_permissions.mcp_access_groups or []
+ return object_permissions.mcp_access_groups or []
+ except Exception as e:
+ verbose_logger.warning(
+ f"Failed to get MCP access groups for team: {str(e)}"
+ )
+ return []
@staticmethod
def get_mcp_access_groups_from_headers(headers: Headers) -> Optional[List[str]]:
"""
Extract and parse the x-mcp-access-groups header as a list of strings.
"""
- mcp_access_groups_header = headers.get(MCPRequestHandler.LITELLM_MCP_ACCESS_GROUPS_HEADER_NAME)
+ mcp_access_groups_header = headers.get(
+ MCPRequestHandler.LITELLM_MCP_ACCESS_GROUPS_HEADER_NAME
+ )
if mcp_access_groups_header is not None:
try:
- return [s.strip() for s in mcp_access_groups_header.split(",") if s.strip()]
+ return [
+ s.strip() for s in mcp_access_groups_header.split(",") if s.strip()
+ ]
except Exception:
return None
return None
@@ -486,4 +826,4 @@ class MCPRequestHandler:
Extract and parse the x-mcp-access-groups header from an ASGI scope.
"""
headers = MCPRequestHandler._safe_get_headers_from_scope(scope)
- return MCPRequestHandler.get_mcp_access_groups_from_headers(headers)
\ No newline at end of file
+ return MCPRequestHandler.get_mcp_access_groups_from_headers(headers)
diff --git a/litellm/proxy/_experimental/mcp_server/cost_calculator.py b/litellm/proxy/_experimental/mcp_server/cost_calculator.py
index eea10924a11..b8fdba23d92 100644
--- a/litellm/proxy/_experimental/mcp_server/cost_calculator.py
+++ b/litellm/proxy/_experimental/mcp_server/cost_calculator.py
@@ -1,6 +1,7 @@
"""
Cost calculator for MCP tools.
"""
+
from typing import TYPE_CHECKING, Any, Optional, cast
from litellm.types.mcp import MCPServerCostInfo
@@ -13,11 +14,12 @@ if TYPE_CHECKING:
else:
LitellmLoggingObject = Any
+
class MCPCostCalculator:
@staticmethod
def calculate_mcp_tool_call_cost(
litellm_logging_obj: Optional[LitellmLoggingObject],
- ) -> float:
+ ) -> float:
"""
Calculate the cost of an MCP tool call.
@@ -25,28 +27,43 @@ class MCPCostCalculator:
"""
if litellm_logging_obj is None:
return 0.0
-
+
#########################################################
# Get the response cost from logging object model_call_details
# This is set when a user modifies the response in a post_mcp_tool_call_hook
#########################################################
- response_cost = litellm_logging_obj.model_call_details.get("response_cost", None)
+ response_cost = litellm_logging_obj.model_call_details.get(
+ "response_cost", None
+ )
if response_cost is not None:
return response_cost
-
+
#########################################################
# Unpack the mcp_tool_call_metadata
#########################################################
- mcp_tool_call_metadata: StandardLoggingMCPToolCall = cast(StandardLoggingMCPToolCall, litellm_logging_obj.model_call_details.get("mcp_tool_call_metadata", {})) or {}
- mcp_server_cost_info: MCPServerCostInfo = mcp_tool_call_metadata.get("mcp_server_cost_info", {}) or {}
+ mcp_tool_call_metadata: StandardLoggingMCPToolCall = (
+ cast(
+ StandardLoggingMCPToolCall,
+ litellm_logging_obj.model_call_details.get(
+ "mcp_tool_call_metadata", {}
+ ),
+ )
+ or {}
+ )
+ mcp_server_cost_info: MCPServerCostInfo = (
+ mcp_tool_call_metadata.get("mcp_server_cost_info") or MCPServerCostInfo()
+ )
#########################################################
# User defined cost per query
#########################################################
- default_cost_per_query = mcp_server_cost_info.get("default_cost_per_query", None)
- tool_name_to_cost_per_query: dict = mcp_server_cost_info.get("tool_name_to_cost_per_query", {}) or {}
+ default_cost_per_query = mcp_server_cost_info.get(
+ "default_cost_per_query", None
+ )
+ tool_name_to_cost_per_query: dict = (
+ mcp_server_cost_info.get("tool_name_to_cost_per_query", {}) or {}
+ )
tool_name = mcp_tool_call_metadata.get("name", "")
-
#########################################################
# 1. If tool_name is in tool_name_to_cost_per_query, use the cost per query
# 2. If tool_name is not in tool_name_to_cost_per_query, use the default cost per query
diff --git a/litellm/proxy/_experimental/mcp_server/db.py b/litellm/proxy/_experimental/mcp_server/db.py
index 414f8094c32..22695485741 100644
--- a/litellm/proxy/_experimental/mcp_server/db.py
+++ b/litellm/proxy/_experimental/mcp_server/db.py
@@ -1,6 +1,7 @@
-import uuid
from typing import Any, Dict, Iterable, List, Optional, Set, Union
+from litellm._logging import verbose_proxy_logger
+from litellm._uuid import uuid
from litellm.proxy._types import (
LiteLLM_MCPServerTable,
LiteLLM_ObjectPermissionTable,
@@ -29,7 +30,7 @@ def _prepare_mcp_server_data(
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
# Convert model to dict
- data_dict = data.model_dump()
+ data_dict = data.model_dump(exclude_none=True)
# Ensure alias is always present in the dict (even if None)
if "alias" not in data_dict:
data_dict["alias"] = getattr(data, "alias", None)
@@ -53,11 +54,20 @@ async def get_all_mcp_servers(
"""
Returns all of the mcp servers from the db
"""
- mcp_servers = await prisma_client.db.litellm_mcpservertable.find_many()
+ try:
+ mcp_servers = await prisma_client.db.litellm_mcpservertable.find_many()
- return [
- LiteLLM_MCPServerTable(**mcp_server.model_dump()) for mcp_server in mcp_servers
- ]
+ return [
+ LiteLLM_MCPServerTable(**mcp_server.model_dump())
+ for mcp_server in mcp_servers
+ ]
+ except Exception as e:
+ verbose_proxy_logger.debug(
+ "litellm.proxy._experimental.mcp_server.db.py::get_all_mcp_servers - {}".format(
+ str(e)
+ )
+ )
+ return []
async def get_mcp_server(
@@ -82,14 +92,18 @@ async def get_mcp_servers(
"""
Returns the matching mcp servers from the db with the server_ids
"""
- mcp_servers: List[LiteLLM_MCPServerTable] = (
+ _mcp_servers: List[LiteLLM_MCPServerTable] = (
await prisma_client.db.litellm_mcpservertable.find_many(
where={
"server_id": {"in": server_ids},
}
)
)
- return mcp_servers
+ final_mcp_servers: List[LiteLLM_MCPServerTable] = []
+ for _mcp_server in _mcp_servers:
+ final_mcp_servers.append(LiteLLM_MCPServerTable(**_mcp_server.model_dump()))
+
+ return final_mcp_servers
async def get_mcp_servers_by_verificationtoken(
diff --git a/litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py b/litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
new file mode 100644
index 00000000000..5e5099426a0
--- /dev/null
+++ b/litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
@@ -0,0 +1,252 @@
+import json
+from typing import Optional, Tuple
+from urllib.parse import urlencode, urlparse, urlunparse
+
+from fastapi import APIRouter, Form, HTTPException, Request
+from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse
+
+from litellm.llms.custom_httpx.http_handler import (
+ get_async_httpx_client,
+ httpxSpecialProvider,
+)
+from litellm.proxy.common_utils.encrypt_decrypt_utils import (
+ decrypt_value_helper,
+ encrypt_value_helper,
+)
+
+router = APIRouter(
+ tags=["mcp"],
+)
+
+
+def encode_state_with_base_url(base_url: str, original_state: str) -> str:
+ """
+ Encode the base_url and original state using encryption.
+
+ Args:
+ base_url: The base URL to encode
+ original_state: The original state parameter
+
+ Returns:
+ An encrypted string that encodes both values
+ """
+ state_data = {"base_url": base_url, "original_state": original_state}
+ state_json = json.dumps(state_data, sort_keys=True)
+ encrypted_state = encrypt_value_helper(state_json)
+ return encrypted_state
+
+
+def decode_state_hash(encrypted_state: str) -> Tuple[str, str]:
+ """
+ Decode an encrypted state to retrieve the base_url and original state.
+
+ Args:
+ encrypted_state: The encrypted string to decode
+
+ Returns:
+ A tuple of (base_url, original_state)
+
+ Raises:
+ Exception: If decryption fails or data is malformed
+ """
+ decrypted_json = decrypt_value_helper(encrypted_state, "oauth_state")
+ if decrypted_json is None:
+ raise ValueError("Failed to decrypt state parameter")
+
+ state_data = json.loads(decrypted_json)
+ return state_data["base_url"], state_data["original_state"]
+
+
+@router.get("/{mcp_server_name}/authorize")
+@router.get("/authorize")
+async def authorize(
+ request: Request,
+ client_id: str,
+ redirect_uri: str,
+ state: str = "",
+ mcp_server_name: Optional[str] = None,
+):
+ # Redirect to real GitHub OAuth
+ from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
+ global_mcp_server_manager,
+ )
+
+ mcp_server = global_mcp_server_manager.get_mcp_server_by_name(client_id)
+ if mcp_server is None:
+ raise HTTPException(status_code=404, detail="MCP server not found")
+ if mcp_server.auth_type != "oauth2":
+ raise HTTPException(status_code=400, detail="MCP server is not OAuth2")
+ if mcp_server.client_id is None:
+ raise HTTPException(status_code=400, detail="MCP server client id is not set")
+ if mcp_server.authorization_url is None:
+ raise HTTPException(
+ status_code=400, detail="MCP server authorization url is not set"
+ )
+ if mcp_server.scopes is None:
+ raise HTTPException(status_code=400, detail="MCP server scopes is not set")
+
+ # Parse it to remove any existing query
+ parsed = urlparse(redirect_uri)
+ base_url = urlunparse(parsed._replace(query=""))
+ request_base_url = str(request.base_url).rstrip("/")
+
+ # Encode the base_url and original state in a unique hash
+ encoded_state = encode_state_with_base_url(base_url, state)
+
+ params = {
+ "client_id": mcp_server.client_id,
+ "redirect_uri": f"{request_base_url}/callback",
+ "scope": " ".join(mcp_server.scopes),
+ "state": encoded_state,
+ }
+ return RedirectResponse(f"{mcp_server.authorization_url}?{urlencode(params)}")
+
+
+@router.post("/token")
+async def token_endpoint(
+ request: Request,
+ grant_type: str = Form(...),
+ code: str = Form(None),
+ redirect_uri: str = Form(None),
+ client_id: str = Form(...),
+ client_secret: str = Form(...),
+):
+ """
+ Accept the authorization code from Claude and exchange it for GitHub token.
+ Forward the GitHub token back to Claude in standard OAuth format.
+
+ 1. Call the token endpoint
+ 2. Store the user's PAT in the db - and generate a LiteLLM virtual key
+ 2. Return the token
+ 3. Return a virtual key in this response
+ """
+ from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
+ global_mcp_server_manager,
+ )
+
+ mcp_server = global_mcp_server_manager.get_mcp_server_by_name(client_id)
+ if mcp_server is None:
+ raise HTTPException(status_code=404, detail="MCP server not found")
+
+ if grant_type != "authorization_code":
+ raise HTTPException(status_code=400, detail="Unsupported grant_type")
+
+ if mcp_server.token_url is None:
+ raise HTTPException(status_code=400, detail="MCP server token url is not set")
+
+ proxy_base_url = str(request.base_url).rstrip("/")
+
+ # Exchange code for real GitHub token
+ async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.Oauth2Check)
+ response = await async_client.post(
+ mcp_server.token_url,
+ headers={"Accept": "application/json"},
+ data={
+ "client_id": mcp_server.client_id,
+ "client_secret": mcp_server.client_secret,
+ "code": code,
+ "redirect_uri": f"{proxy_base_url}/callback",
+ },
+ )
+
+ response.raise_for_status()
+ github_token = response.json()["access_token"]
+
+ # Return to Claude in expected OAuth 2 format
+
+ ### return a virtual key in this response
+
+ return JSONResponse(
+ {"access_token": github_token, "token_type": "Bearer", "expires_in": 3600}
+ )
+
+
+@router.get("/callback")
+async def callback(code: str, state: str):
+ try:
+ # Decode the state hash to get base_url and original state
+ base_url, original_state = decode_state_hash(state)
+
+ # Exchange code for token with GitHub
+ params = {"code": code, "state": original_state}
+
+ # Forward token to Claude ephemeral endpoint
+ complete_returned_url = f"{base_url}?{urlencode(params)}"
+ return RedirectResponse(url=complete_returned_url, status_code=302)
+
+ except Exception:
+ # fallback if state hash not found
+ return HTMLResponse(
+ "Authentication incomplete. You can close this window."
+ )
+
+
+# ------------------------------
+# Optional .well-known endpoints for MCP + OAuth discovery
+# ------------------------------
+@router.get("/.well-known/oauth-protected-resource/{mcp_server_name}/mcp")
+@router.get("/.well-known/oauth-protected-resource")
+async def oauth_protected_resource_mcp(
+ request: Request, mcp_server_name: Optional[str] = None
+):
+ request_base_url = str(request.base_url).rstrip("/")
+ return {
+ "authorization_servers": [
+ (
+ f"{request_base_url}/{mcp_server_name}"
+ if mcp_server_name
+ else f"{request_base_url}"
+ )
+ ],
+ "resource": (
+ f"{request_base_url}/{mcp_server_name}/mcp"
+ if mcp_server_name
+ else f"{request_base_url}/mcp"
+ ), # this is what Claude will call
+ }
+
+
+@router.get("/.well-known/oauth-authorization-server/{mcp_server_name}")
+@router.get("/.well-known/oauth-authorization-server")
+async def oauth_authorization_server_mcp(
+ request: Request, mcp_server_name: Optional[str] = None
+):
+ request_base_url = str(request.base_url).rstrip("/")
+ return {
+ "issuer": request_base_url, # point to your proxy
+ "authorization_endpoint": f"{request_base_url}/authorize",
+ "token_endpoint": f"{request_base_url}/token",
+ "response_types_supported": ["code"],
+ "grant_types_supported": ["authorization_code"],
+ "code_challenge_methods_supported": ["S256"],
+ "token_endpoint_auth_methods_supported": ["client_secret_post"],
+ # Claude expects a registration endpoint, even if we just fake it
+ "registration_endpoint": f"{request_base_url}/{mcp_server_name}/register",
+ }
+
+
+# Alias for standard OpenID discovery
+@router.get("/.well-known/openid-configuration")
+async def openid_configuration(request: Request):
+ return await oauth_authorization_server_mcp(request)
+
+
+@router.get("/.well-known/oauth-authorization-server/{mcp_server_name}/mcp")
+@router.get("/.well-known/oauth-authorization-server")
+async def oauth_authorization_server_root(
+ request: Request, mcp_server_name: Optional[str] = None
+):
+ return await oauth_authorization_server_mcp(request, mcp_server_name)
+
+
+@router.post("/{mcp_server_name}/register")
+@router.post("/register")
+async def register_client(request: Request, mcp_server_name: Optional[str] = None):
+ request_base_url = str(request.base_url).rstrip("/")
+
+ # return fixed GitHub client credentials
+ return {
+ "client_id": mcp_server_name or "dummy_client",
+ "client_secret": "dummy",
+ "redirect_uris": [f"{request_base_url}/mcp/callback"],
+ }
diff --git a/litellm/proxy/_experimental/mcp_server/mcp_server_manager.py b/litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
index 93ac08b4f0f..f313a673827 100644
--- a/litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
+++ b/litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
@@ -7,15 +7,18 @@ This is a Proxy
"""
import asyncio
+import datetime
import hashlib
import json
-from typing import Any, Dict, List, Optional, cast
+from typing import Any, Dict, List, Optional, Set, Union, cast
+from fastapi import HTTPException
from mcp.types import CallToolRequestParams as MCPCallToolRequestParams
from mcp.types import CallToolResult
from mcp.types import Tool as MCPTool
from litellm._logging import verbose_logger
+from litellm.exceptions import BlockedPiiEntityError, GuardrailRaisedException
from litellm.experimental_mcp_client.client import MCPClient
from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
MCPRequestHandler,
@@ -23,21 +26,20 @@ from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
from litellm.proxy._experimental.mcp_server.utils import (
add_server_prefix_to_tool_name,
get_server_name_prefix_tool_mcp,
+ get_server_prefix,
is_tool_name_prefixed,
normalize_server_name,
validate_mcp_server_name,
- get_server_prefix,
)
from litellm.proxy._types import (
LiteLLM_MCPServerTable,
MCPAuthType,
- MCPSpecVersion,
- MCPSpecVersionType,
MCPTransport,
MCPTransportType,
UserAPIKeyAuth,
)
-from litellm.types.mcp import MCPStdioConfig
+from litellm.proxy.utils import ProxyLogging
+from litellm.types.mcp import MCPAuth, MCPStdioConfig
from litellm.types.mcp_server.mcp_server_manager import MCPInfo, MCPServer
@@ -45,16 +47,16 @@ def _deserialize_env_dict(env_data: Any) -> Optional[Dict[str, str]]:
"""
Helper function to deserialize environment dictionary from database storage.
Handles both JSON string and dictionary formats.
-
+
Args:
env_data: The environment data from database (could be JSON string or dict)
-
+
Returns:
Dict[str, str] or None: Deserialized environment dictionary
"""
if not env_data:
return None
-
+
if isinstance(env_data, str):
try:
return json.loads(env_data)
@@ -77,8 +79,7 @@ class MCPServerManager:
"name": "zapier_mcp_server",
"url": "https://actions.zapier.com/mcp/sk-ak-2ew3bofIeQIkNoeKIdXrF1Hhhp/sse"
"transport": "sse",
- "auth_type": "api_key",
- "spec_version": "2025-03-26"
+ "auth_type": "api_key"
},
"uuid-2": {
"name": "google_drive_mcp_server",
@@ -100,70 +101,84 @@ class MCPServerManager:
"""
return self.config_mcp_servers | self.registry
- def load_servers_from_config(self, mcp_servers_config: Dict[str, Any], mcp_aliases: Optional[Dict[str, str]] = None):
+ def load_servers_from_config(
+ self,
+ mcp_servers_config: Dict[str, Any],
+ mcp_aliases: Optional[Dict[str, str]] = None,
+ ):
"""
Load the MCP Servers from the config
-
+
Args:
mcp_servers_config: Dictionary of MCP server configurations
mcp_aliases: Optional dictionary mapping aliases to server names from litellm_settings
"""
verbose_logger.debug("Loading MCP Servers from config-----")
-
+
# Track which aliases have been used to ensure only first occurrence is used
used_aliases = set()
-
+
for server_name, server_config in mcp_servers_config.items():
validate_mcp_server_name(server_name)
_mcp_info: Dict[str, Any] = server_config.get("mcp_info", None) or {}
- # Convert Dict[str, Any] to MCPInfo properly
- mcp_info: MCPInfo = {
- "server_name": _mcp_info.get("server_name", server_name),
- "description": _mcp_info.get("description", server_config.get("description", None)),
- "logo_url": _mcp_info.get("logo_url", None),
- "mcp_server_cost_info": _mcp_info.get("mcp_server_cost_info", None),
- }
+ # Preserve all custom fields from config while setting defaults for core fields
+ mcp_info: MCPInfo = _mcp_info.copy()
+ # Set default values for core fields if not present
+ if "server_name" not in mcp_info:
+ mcp_info["server_name"] = server_name
+ if "description" not in mcp_info and server_config.get("description"):
+ mcp_info["description"] = server_config.get("description")
# Use alias for name if present, else server_name
alias = server_config.get("alias", None)
-
+
# Apply mcp_aliases mapping if provided
if mcp_aliases and alias is None:
# Check if this server_name has an alias in mcp_aliases
for alias_name, target_server_name in mcp_aliases.items():
- if target_server_name == server_name and alias_name not in used_aliases:
+ if (
+ target_server_name == server_name
+ and alias_name not in used_aliases
+ ):
alias = alias_name
used_aliases.add(alias_name)
- verbose_logger.debug(f"Mapped alias '{alias_name}' to server '{server_name}'")
+ verbose_logger.debug(
+ f"Mapped alias '{alias_name}' to server '{server_name}'"
+ )
break
-
+
# Create a temporary server object to use with get_server_prefix utility
- temp_server = type('TempServer', (), {
- 'alias': alias,
- 'server_name': server_name,
- 'server_id': None
- })()
+ temp_server = type(
+ "TempServer",
+ (),
+ {"alias": alias, "server_name": server_name, "server_id": None},
+ )()
name_for_prefix = get_server_prefix(temp_server)
# Use alias for name if present, else server_name
alias = server_config.get("alias", None)
-
+
# Apply mcp_aliases mapping if provided
if mcp_aliases and alias is None:
# Check if this server_name has an alias in mcp_aliases
for alias_name, target_server_name in mcp_aliases.items():
- if target_server_name == server_name and alias_name not in used_aliases:
+ if (
+ target_server_name == server_name
+ and alias_name not in used_aliases
+ ):
alias = alias_name
used_aliases.add(alias_name)
- verbose_logger.debug(f"Mapped alias '{alias_name}' to server '{server_name}'")
+ verbose_logger.debug(
+ f"Mapped alias '{alias_name}' to server '{server_name}'"
+ )
break
-
+
# Create a temporary server object to use with get_server_prefix utility
- temp_server = type('TempServer', (), {
- 'alias': alias,
- 'server_name': server_name,
- 'server_id': None
- })()
+ temp_server = type(
+ "TempServer",
+ (),
+ {"alias": alias, "server_name": server_name, "server_id": None},
+ )()
name_for_prefix = get_server_prefix(temp_server)
# Generate stable server ID based on parameters
@@ -171,7 +186,6 @@ class MCPServerManager:
server_name=server_name,
url=server_config.get("url", None) or "",
transport=server_config.get("transport", MCPTransport.http),
- spec_version=server_config.get("spec_version", MCPSpecVersion.mar_2025),
auth_type=server_config.get("auth_type", None),
alias=alias,
)
@@ -181,24 +195,188 @@ class MCPServerManager:
name=name_for_prefix,
alias=alias,
server_name=server_name,
+ spec_path=server_config.get("spec_path", None),
url=server_config.get("url", None) or "",
command=server_config.get("command", None) or "",
args=server_config.get("args", None) or [],
env=server_config.get("env", None) or {},
+ # oauth specific fields
+ client_id=server_config.get("client_id", None),
+ client_secret=server_config.get("client_secret", None),
+ scopes=server_config.get("scopes", None),
+ authorization_url=server_config.get("authorization_url", None),
+ token_url=server_config.get("token_url", None),
# TODO: utility fn the default values
transport=server_config.get("transport", MCPTransport.http),
- spec_version=server_config.get("spec_version", MCPSpecVersion.mar_2025),
auth_type=server_config.get("auth_type", None),
+ authentication_token=server_config.get(
+ "authentication_token", server_config.get("auth_value", None)
+ ),
mcp_info=mcp_info,
+ extra_headers=server_config.get("extra_headers", None),
+ allowed_tools=server_config.get("allowed_tools", None),
+ disallowed_tools=server_config.get("disallowed_tools", None),
+ allowed_params=server_config.get("allowed_params", None),
access_groups=server_config.get("access_groups", None),
)
self.config_mcp_servers[server_id] = new_server
+
+ # Check if this is an OpenAPI-based server
+ spec_path = server_config.get("spec_path", None)
+ if spec_path:
+ verbose_logger.info(
+ f"Loading OpenAPI spec from {spec_path} for server {server_name}"
+ )
+ self._register_openapi_tools(
+ spec_path=spec_path,
+ server=new_server,
+ base_url=server_config.get("url", ""),
+ )
+
verbose_logger.debug(
f"Loaded MCP Servers: {json.dumps(self.config_mcp_servers, indent=4, default=str)}"
)
self.initialize_tool_name_to_mcp_server_name_mapping()
+ def _register_openapi_tools(self, spec_path: str, server: MCPServer, base_url: str):
+ """
+ Register tools from an OpenAPI specification for a given server.
+
+ This creates "virtual" MCP tools from OpenAPI endpoints that are:
+ 1. Registered in the global tool registry with server prefix
+ 2. Mapped to the server for routing
+ 3. Executed via the local tool handler
+
+ Args:
+ spec_path: Path to the OpenAPI specification file
+ server: The MCPServer instance to register tools for
+ base_url: Base URL for API calls
+ """
+ from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
+ build_input_schema,
+ create_tool_function,
+ )
+ from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
+ get_base_url as get_openapi_base_url,
+ )
+ from litellm.proxy._experimental.mcp_server.openapi_to_mcp_generator import (
+ load_openapi_spec,
+ )
+ from litellm.proxy._experimental.mcp_server.tool_registry import (
+ global_mcp_tool_registry,
+ )
+
+ try:
+ # Load OpenAPI spec
+ spec = load_openapi_spec(spec_path)
+
+ # Use base_url from config if provided, otherwise extract from spec
+ if not base_url:
+ base_url = get_openapi_base_url(spec)
+
+ verbose_logger.info(
+ f"Registering OpenAPI tools for server {server.name} with base URL: {base_url}"
+ )
+
+ # Get server prefix for tool naming
+ server_prefix = get_server_prefix(server)
+
+ # Build headers from server configuration
+ headers = {}
+
+ # Add authentication headers if configured
+ if server.authentication_token:
+ from litellm.types.mcp import MCPAuth
+
+ if server.auth_type == MCPAuth.bearer_token:
+ headers["Authorization"] = f"Bearer {server.authentication_token}"
+ elif server.auth_type == MCPAuth.api_key:
+ headers["Authorization"] = f"ApiKey {server.authentication_token}"
+ elif server.auth_type == MCPAuth.basic:
+ headers["Authorization"] = f"Basic {server.authentication_token}"
+
+ # Add any extra headers from server config
+ # Note: extra_headers is a List[str] of header names to forward, not a dict
+ # For OpenAPI tools, we'll just use the authentication headers
+ # If extra_headers were needed, they would be processed separately
+
+ verbose_logger.debug(
+ f"Using headers for OpenAPI tools (excluding sensitive values): "
+ f"{list(headers.keys())}"
+ )
+
+ # Extract and register tools from OpenAPI paths
+ paths = spec.get("paths", {})
+ registered_count = 0
+
+ verbose_logger.debug(f"Processing {len(paths)} paths from OpenAPI spec")
+
+ for path, path_item in paths.items():
+ for method in ["get", "post", "put", "delete", "patch"]:
+ if method not in path_item:
+ continue
+
+ operation = path_item[method]
+
+ # Generate tool name (without prefix initially)
+ operation_id = operation.get(
+ "operationId", f"{method}_{path.replace('/', '_')}"
+ )
+ base_tool_name = operation_id.replace(" ", "_").lower()
+
+ # Add server prefix to tool name
+ prefixed_tool_name = add_server_prefix_to_tool_name(
+ base_tool_name, server_prefix
+ )
+
+ # Get description
+ description = operation.get(
+ "summary",
+ operation.get("description", f"{method.upper()} {path}"),
+ )
+
+ # Build input schema using imported function
+ input_schema = build_input_schema(operation)
+
+ # Create tool function with headers using imported function
+ tool_func = create_tool_function(
+ path, method, operation, base_url, headers=headers
+ )
+ tool_func.__name__ = prefixed_tool_name
+ tool_func.__doc__ = description
+
+ # Register tool with prefixed name in global registry
+ global_mcp_tool_registry.register_tool(
+ name=prefixed_tool_name,
+ description=description,
+ input_schema=input_schema,
+ handler=tool_func,
+ )
+
+ # Update tool name to server name mapping (for both prefixed and base names)
+ self.tool_name_to_mcp_server_name_mapping[base_tool_name] = (
+ server_prefix
+ )
+ self.tool_name_to_mcp_server_name_mapping[prefixed_tool_name] = (
+ server_prefix
+ )
+
+ registered_count += 1
+ verbose_logger.debug(
+ f"Registered OpenAPI tool: {prefixed_tool_name} for server {server.name}"
+ )
+
+ verbose_logger.info(
+ f"Successfully registered {registered_count} OpenAPI tools for server {server.name}"
+ )
+
+ except Exception as e:
+ verbose_logger.error(
+ f"Failed to register OpenAPI tools for server {server.name}: {str(e)}"
+ )
+ raise e
+
def remove_server(self, mcp_server: LiteLLM_MCPServerTable):
"""
Remove a server from the registry
@@ -215,37 +393,64 @@ class MCPServerManager:
)
def add_update_server(self, mcp_server: LiteLLM_MCPServerTable):
- if mcp_server.server_id not in self.get_registry():
- _mcp_info: MCPInfo = mcp_server.mcp_info or {}
- # Use helper to deserialize environment dictionary
- # Safely access env field which may not exist on Prisma model objects
- env_data = getattr(mcp_server, 'env', None)
- env_dict = _deserialize_env_dict(env_data)
- # Use alias for name if present, else server_name
- name_for_prefix = mcp_server.alias or mcp_server.server_name or mcp_server.server_id
- new_server = MCPServer(
- server_id=mcp_server.server_id,
- name=name_for_prefix,
- alias=getattr(mcp_server, 'alias', None),
- server_name=getattr(mcp_server, 'server_name', None),
- url=mcp_server.url,
- transport=cast(MCPTransportType, mcp_server.transport),
- spec_version=cast(MCPSpecVersionType, mcp_server.spec_version),
- auth_type=cast(MCPAuthType, mcp_server.auth_type),
- mcp_info=MCPInfo(
- server_name=mcp_server.server_name or mcp_server.server_id,
- description=mcp_server.description,
- mcp_server_cost_info=_mcp_info.get("mcp_server_cost_info", None),
- ),
- # Stdio-specific fields
- command=getattr(mcp_server, 'command', None),
- args=getattr(mcp_server, 'args', None) or [],
- env=env_dict,
- )
- self.registry[mcp_server.server_id] = new_server
- verbose_logger.debug(
- f"Added MCP Server: {name_for_prefix}"
- )
+ try:
+ if mcp_server.server_id not in self.get_registry():
+ _mcp_info: MCPInfo = mcp_server.mcp_info or {}
+ # Use helper to deserialize environment dictionary
+ # Safely access env field which may not exist on Prisma model objects
+ env_data = getattr(mcp_server, "env", None)
+ env_dict = _deserialize_env_dict(env_data)
+ # Use alias for name if present, else server_name
+ name_for_prefix = (
+ mcp_server.alias or mcp_server.server_name or mcp_server.server_id
+ )
+ # Preserve all custom fields from database while setting defaults for core fields
+ mcp_info: MCPInfo = _mcp_info.copy()
+ # Set default values for core fields if not present
+ if "server_name" not in mcp_info:
+ mcp_info["server_name"] = (
+ mcp_server.server_name or mcp_server.server_id
+ )
+ if "description" not in mcp_info and mcp_server.description:
+ mcp_info["description"] = mcp_server.description
+
+ new_server = MCPServer(
+ server_id=mcp_server.server_id,
+ name=name_for_prefix,
+ alias=getattr(mcp_server, "alias", None),
+ server_name=getattr(mcp_server, "server_name", None),
+ url=mcp_server.url,
+ transport=cast(MCPTransportType, mcp_server.transport),
+ auth_type=cast(MCPAuthType, mcp_server.auth_type),
+ mcp_info=mcp_info,
+ extra_headers=getattr(mcp_server, "extra_headers", None),
+ # oauth specific fields
+ client_id=getattr(mcp_server, "client_id", None),
+ client_secret=getattr(mcp_server, "client_secret", None),
+ scopes=getattr(mcp_server, "scopes", None),
+ authorization_url=getattr(mcp_server, "authorization_url", None),
+ token_url=getattr(mcp_server, "token_url", None),
+ # Stdio-specific fields
+ command=getattr(mcp_server, "command", None),
+ args=getattr(mcp_server, "args", None) or [],
+ env=env_dict,
+ access_groups=getattr(mcp_server, "mcp_access_groups", None),
+ allowed_tools=getattr(mcp_server, "allowed_tools", None),
+ disallowed_tools=getattr(mcp_server, "disallowed_tools", None),
+ )
+ self.registry[mcp_server.server_id] = new_server
+ verbose_logger.debug(f"Added MCP Server: {name_for_prefix}")
+
+ except Exception as e:
+ verbose_logger.debug(f"Failed to add MCP server: {str(e)}")
+ raise e
+
+ def get_all_mcp_server_ids(self) -> Set[str]:
+ """
+ Get all MCP server IDs
+ """
+ all_servers = list(self.get_registry().values())
+ return {server.server_id for server in all_servers}
async def get_allowed_mcp_servers(
self, user_api_key_auth: Optional[UserAPIKeyAuth] = None
@@ -253,36 +458,47 @@ class MCPServerManager:
"""
Get the allowed MCP Servers for the user
"""
- allowed_mcp_servers = await MCPRequestHandler.get_allowed_mcp_servers(
- user_api_key_auth
- )
- verbose_logger.debug(
- f"Allowed MCP Servers for user api key auth: {allowed_mcp_servers}"
- )
- if len(allowed_mcp_servers) > 0:
- return allowed_mcp_servers
- else:
+ try:
+ allowed_mcp_servers = await MCPRequestHandler.get_allowed_mcp_servers(
+ user_api_key_auth
+ )
verbose_logger.debug(
- "No allowed MCP Servers found for user api key auth, returning default registry servers"
+ f"Allowed MCP Servers for user api key auth: {allowed_mcp_servers}"
+ )
+ if len(allowed_mcp_servers) > 0:
+ return allowed_mcp_servers
+ else:
+ verbose_logger.debug(
+ "No allowed MCP Servers found for user api key auth, returning default registry servers"
+ )
+ return list(self.get_registry().keys())
+ except Exception as e:
+ verbose_logger.warning(
+ f"Failed to get allowed MCP servers: {str(e)}. Returning default registry servers."
)
return list(self.get_registry().keys())
-
async def get_tools_for_server(self, server_id: str) -> List[MCPTool]:
"""
Get the tools for a given server
"""
- server = self.get_mcp_server_by_id(server_id)
- if server is None:
+ try:
+ server = self.get_mcp_server_by_id(server_id)
+ if server is None:
+ verbose_logger.warning(f"MCP Server {server_id} not found")
+ return []
+ return await self._get_tools_from_server(server)
+ except Exception as e:
+ verbose_logger.warning(
+ f"Failed to get tools from server {server_id}: {str(e)}"
+ )
return []
- return await self._get_tools_from_server(server)
-
async def list_tools(
- self,
+ self,
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
mcp_auth_header: Optional[str] = None,
- mcp_server_auth_headers: Optional[Dict[str, str]] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Union[str, Dict[str, str]]]] = None,
) -> List[MCPTool]:
"""
List all tools available across all MCP Servers.
@@ -291,6 +507,7 @@ class MCPServerManager:
user_api_key_auth: User authentication
mcp_auth_header: MCP auth header (deprecated)
mcp_server_auth_headers: Optional dict of server-specific auth headers {server_alias: auth_value}
+ mcp_protocol_version: Optional MCP protocol version from request header
Returns:
List[MCPTool]: Combined list of tools from all servers
@@ -305,38 +522,47 @@ class MCPServerManager:
if server is None:
verbose_logger.warning(f"MCP Server {server_id} not found")
continue
-
+
# Get server-specific auth header if available
server_auth_header = None
if mcp_server_auth_headers and server.alias:
server_auth_header = mcp_server_auth_headers.get(server.alias)
elif mcp_server_auth_headers and server.server_name:
server_auth_header = mcp_server_auth_headers.get(server.server_name)
-
+
# Fall back to deprecated mcp_auth_header if no server-specific header found
if server_auth_header is None:
server_auth_header = mcp_auth_header
-
+
try:
tools = await self._get_tools_from_server(
server=server,
mcp_auth_header=server_auth_header,
)
list_tools_result.extend(tools)
- verbose_logger.info(f"Successfully fetched {len(tools)} tools from server {server.name}")
+ verbose_logger.info(
+ f"Successfully fetched {len(tools)} tools from server {server.name}"
+ )
except Exception as e:
verbose_logger.warning(
f"Failed to list tools from server {server.name}: {str(e)}. Continuing with other servers."
)
# Continue with other servers instead of failing completely
- verbose_logger.info(f"Successfully fetched {len(list_tools_result)} tools total from all servers")
+ verbose_logger.info(
+ f"Successfully fetched {len(list_tools_result)} tools total from all servers"
+ )
return list_tools_result
#########################################################
# Methods that call the upstream MCP servers
#########################################################
- def _create_mcp_client(self, server: MCPServer, mcp_auth_header: Optional[str] = None) -> MCPClient:
+ def _create_mcp_client(
+ self,
+ server: MCPServer,
+ mcp_auth_header: Optional[Union[str, Dict[str, str]]] = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ ) -> MCPClient:
"""
Create an MCPClient instance for the given server.
@@ -348,18 +574,16 @@ class MCPServerManager:
MCPClient: Configured MCP client instance
"""
transport = server.transport or MCPTransport.sse
-
+
# Handle stdio transport
if transport == MCPTransport.stdio:
# For stdio, we need to get the stdio config from the server
stdio_config: Optional[MCPStdioConfig] = None
if server.command and server.args is not None:
stdio_config = MCPStdioConfig(
- command=server.command,
- args=server.args,
- env=server.env or {}
+ command=server.command, args=server.args, env=server.env or {}
)
-
+
return MCPClient(
server_url="", # Not used for stdio
transport_type=transport,
@@ -367,6 +591,7 @@ class MCPServerManager:
auth_value=mcp_auth_header or server.authentication_token,
timeout=60.0,
stdio_config=stdio_config,
+ extra_headers=extra_headers,
)
else:
# For HTTP/SSE transports
@@ -377,9 +602,16 @@ class MCPServerManager:
auth_type=server.auth_type,
auth_value=mcp_auth_header or server.authentication_token,
timeout=60.0,
+ extra_headers=extra_headers,
)
- async def _get_tools_from_server(self, server: MCPServer, mcp_auth_header: Optional[str] = None) -> List[MCPTool]:
+ async def _get_tools_from_server(
+ self,
+ server: MCPServer,
+ mcp_auth_header: Optional[Union[str, Dict[str, str]]] = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ add_prefix: bool = True,
+ ) -> List[MCPTool]:
"""
Helper method to get tools from a single MCP server with prefixed names.
@@ -390,74 +622,42 @@ class MCPServerManager:
Returns:
List[MCPTool]: List of tools available on the server with prefixed names
"""
+ from litellm.proxy._experimental.mcp_server.tool_registry import (
+ global_mcp_tool_registry,
+ )
+
verbose_logger.debug(f"Connecting to url: {server.url}")
verbose_logger.info(f"_get_tools_from_server for {server.name}...")
client = None
+
try:
client = self._create_mcp_client(
server=server,
mcp_auth_header=mcp_auth_header,
+ extra_headers=extra_headers,
)
- # Create a task for the client operations to ensure proper cancellation handling
- async def _list_tools_task():
- try:
- async with client:
- tools = await client.list_tools()
- verbose_logger.debug(f"Tools from {server.name}: {tools}")
- return tools
- except asyncio.CancelledError:
- verbose_logger.warning(f"Client operation cancelled for {server.name}")
- return []
- except Exception as e:
- verbose_logger.warning(f"Client operation failed for {server.name}: {str(e)}")
- return []
+ ## HANDLE OPENAPI TOOLS
+ if server.spec_path:
+ _tools = global_mcp_tool_registry.list_tools(tool_prefix=server.name)
+ tools = global_mcp_tool_registry.convert_tools_to_mcp_sdk_tool_type(
+ _tools
+ )
+ else:
+ tools = await self._fetch_tools_with_timeout(client, server.name)
- try:
- # Add timeout to prevent hanging
- tools = await asyncio.wait_for(_list_tools_task(), timeout=30.0)
+ prefixed_or_original_tools = self._create_prefixed_tools(
+ tools, server, add_prefix=add_prefix
+ )
- # Create new tools with prefixed names
- prefixed_tools = []
- for tool in tools:
- # Always use alias for prefixing if present
- prefix = get_server_prefix(server)
- prefixed_name = add_server_prefix_to_tool_name(tool.name, prefix)
+ return prefixed_or_original_tools
- # Create new tool with prefixed name
- prefixed_tool = MCPTool(
- name=prefixed_name,
- description=tool.description,
- inputSchema=tool.inputSchema
- )
- prefixed_tools.append(prefixed_tool)
-
- # Update tool to server mapping with both original and prefixed names
- self.tool_name_to_mcp_server_name_mapping[tool.name] = prefix
- self.tool_name_to_mcp_server_name_mapping[prefixed_name] = prefix
-
- verbose_logger.info(f"Successfully fetched {len(prefixed_tools)} tools from server {server.name}")
- return prefixed_tools
- except asyncio.TimeoutError:
- verbose_logger.warning(f"Timeout while listing tools from {server.name}")
- # Don't re-raise the exception, just return empty list
- return []
- except asyncio.CancelledError:
- verbose_logger.warning(f"Task cancelled while listing tools from {server.name}")
- # Don't re-raise cancellation, just return empty list
- return []
- except ConnectionError as e:
- verbose_logger.warning(f"Connection error while listing tools from {server.name}: {str(e)}")
- # Don't re-raise the exception, just return empty list
- return []
- except Exception as e:
- verbose_logger.warning(f"Error listing tools from {server.name}: {str(e)}")
- # Don't re-raise the exception, just return empty list
- return []
except Exception as e:
- verbose_logger.warning(f"Failed to get tools from server {server.name}: {str(e)}")
- return [] # Return empty list on failure
+ verbose_logger.warning(
+ f"Failed to get tools from server {server.name}: {str(e)}"
+ )
+ return []
finally:
if client:
try:
@@ -465,13 +665,506 @@ class MCPServerManager:
except Exception:
pass
+ async def _fetch_tools_with_timeout(
+ self, client: MCPClient, server_name: str
+ ) -> List[MCPTool]:
+ """
+ Fetch tools from MCP client with timeout and error handling.
+
+ Args:
+ client: MCP client instance
+ server_name: Name of the server for logging
+
+ Returns:
+ List of tools from the server
+ """
+
+ async def _list_tools_task():
+ try:
+ await client.connect()
+
+ tools = await client.list_tools()
+ verbose_logger.debug(f"Tools from {server_name}: {tools}")
+ return tools
+ except asyncio.CancelledError:
+ verbose_logger.warning(f"Client operation cancelled for {server_name}")
+ return []
+ except Exception as e:
+ verbose_logger.warning(
+ f"Client operation failed for {server_name}: {str(e)}"
+ )
+ return []
+ finally:
+ try:
+ await client.disconnect()
+ except Exception:
+ pass
+
+ try:
+ return await asyncio.wait_for(_list_tools_task(), timeout=30.0)
+ except asyncio.TimeoutError:
+ verbose_logger.warning(f"Timeout while listing tools from {server_name}")
+ return []
+ except asyncio.CancelledError:
+ verbose_logger.warning(
+ f"Task cancelled while listing tools from {server_name}"
+ )
+ return []
+ except ConnectionError as e:
+ verbose_logger.warning(
+ f"Connection error while listing tools from {server_name}: {str(e)}"
+ )
+ return []
+ except Exception as e:
+ verbose_logger.warning(f"Error listing tools from {server_name}: {str(e)}")
+ return []
+
+ def _create_prefixed_tools(
+ self, tools: List[MCPTool], server: MCPServer, add_prefix: bool = True
+ ) -> List[MCPTool]:
+ """
+ Create prefixed tools and update tool mapping.
+
+ Args:
+ tools: List of original tools from server
+ server: Server instance
+
+ Returns:
+ List of tools with prefixed names
+ """
+ prefixed_tools = []
+ prefix = get_server_prefix(server)
+
+ for tool in tools:
+ prefixed_name = add_server_prefix_to_tool_name(tool.name, prefix)
+
+ name_to_use = prefixed_name if add_prefix else tool.name
+
+ tool_obj = MCPTool(
+ name=name_to_use,
+ description=tool.description,
+ inputSchema=tool.inputSchema,
+ )
+ prefixed_tools.append(tool_obj)
+
+ # Update tool to server mapping for resolution (support both forms)
+ self.tool_name_to_mcp_server_name_mapping[tool.name] = prefix
+ self.tool_name_to_mcp_server_name_mapping[prefixed_name] = prefix
+
+ verbose_logger.info(
+ f"Successfully fetched {len(prefixed_tools)} tools from server {server.name}"
+ )
+ return prefixed_tools
+
+ def check_allowed_or_banned_tools(self, tool_name: str, server: MCPServer) -> bool:
+ """
+ Check if the tool is allowed or banned for the given server
+ """
+ if server.allowed_tools:
+ return (
+ tool_name in server.allowed_tools
+ or f"{server.name}-{tool_name}" in server.allowed_tools
+ )
+ if server.disallowed_tools:
+ return (
+ tool_name not in server.disallowed_tools
+ and f"{server.name}-{tool_name}" not in server.disallowed_tools
+ )
+ return True
+
+ def validate_allowed_params(
+ self, tool_name: str, arguments: Dict[str, Any], server: MCPServer
+ ) -> None:
+ """
+ Filter arguments to only include allowed parameters for the given tool.
+
+ Args:
+ tool_name: Name of the tool (with or without prefix)
+ arguments: Dictionary of arguments to filter
+ server: MCPServer configuration
+
+ Returns:
+ Filtered dictionary containing only allowed parameters
+
+ Raises:
+ HTTPException: If allowed_params is configured for this tool but arguments contain disallowed params
+ """
+ from litellm.proxy._experimental.mcp_server.utils import (
+ get_server_name_prefix_tool_mcp,
+ )
+
+ # If no allowed_params configured, return all arguments
+ if not server.allowed_params:
+ return
+
+ # Get the unprefixed tool name to match against config
+ unprefixed_tool_name, _ = get_server_name_prefix_tool_mcp(tool_name)
+
+ # Check both prefixed and unprefixed tool names
+ allowed_params_list = server.allowed_params.get(
+ tool_name
+ ) or server.allowed_params.get(unprefixed_tool_name)
+
+ # If this tool doesn't have allowed_params specified, allow all params
+ if allowed_params_list is None:
+ return None
+
+ # Filter arguments to only include allowed parameters
+ disallowed_params = [
+ param for param in arguments.keys() if param not in allowed_params_list
+ ]
+
+ if disallowed_params:
+ raise HTTPException(
+ status_code=403,
+ detail={
+ "error": f"Parameters {disallowed_params} are not allowed for tool {tool_name}. "
+ f"Allowed parameters: {allowed_params_list}. "
+ f"Contact proxy admin to allow these parameters."
+ },
+ )
+
+ async def check_tool_permission_for_key_team(
+ self,
+ tool_name: str,
+ server: MCPServer,
+ user_api_key_auth: Optional[UserAPIKeyAuth],
+ ) -> None:
+ """
+ Check if a tool is allowed based on key/team object_permission.mcp_tool_permissions.
+ Uses MCPRequestHandler.is_tool_allowed_for_server for consistent inheritance logic.
+ Raises HTTPException if tool is not allowed.
+
+ Args:
+ tool_name: Name of the tool to check
+ server: MCPServer object
+ user_api_key_auth: User authentication
+
+ Raises:
+ HTTPException: If tool is not allowed for this key/team
+ """
+ from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
+ MCPRequestHandler,
+ )
+
+ if not user_api_key_auth:
+ return
+
+ # Check if tool is allowed
+ is_allowed = await MCPRequestHandler.is_tool_allowed_for_server(
+ tool_name=tool_name,
+ server_id=server.server_id,
+ user_api_key_auth=user_api_key_auth,
+ )
+
+ if not is_allowed:
+ raise HTTPException(
+ status_code=403,
+ detail={
+ "error": f"Tool '{tool_name}' is not allowed for your key/team on server '{server.name}'. Contact proxy admin for access."
+ },
+ )
+
+ async def _call_openapi_tool_handler(
+ self,
+ server: MCPServer,
+ tool_name: str,
+ arguments: Dict[str, Any],
+ ) -> CallToolResult:
+ """
+ Call an OpenAPI tool handler directly.
+
+ For OpenAPI servers, instead of using MCP protocol, we call the tool handler
+ that was registered during OpenAPI spec parsing. This handler makes direct
+ HTTP requests to the API.
+
+ Args:
+ tool_name: The full tool name (with prefix) to call
+ arguments: Tool arguments to pass to the handler
+
+ Returns:
+ CallToolResult with the response from the API
+ """
+ from mcp.types import TextContent
+
+ from litellm.proxy._experimental.mcp_server.tool_registry import (
+ global_mcp_tool_registry,
+ )
+
+ # Get the tool from the registry
+ tool = global_mcp_tool_registry.get_tool(f"{server.name}-{tool_name}")
+ if tool is None:
+ # Tool not found in registry
+ error_msg = f"OpenAPI tool {tool_name} not found in registry"
+ verbose_logger.error(error_msg)
+ return CallToolResult(
+ content=[TextContent(type="text", text=error_msg)],
+ isError=True,
+ )
+
+ try:
+ # Call the tool handler with the arguments
+ # The handler is an async function that makes the HTTP request
+ handler_result = await tool.handler(**arguments)
+
+ # Convert the handler result (string response) to CallToolResult format
+ result = CallToolResult(
+ content=[TextContent(type="text", text=str(handler_result))],
+ isError=False,
+ )
+
+ return result
+
+ except Exception as e:
+ error_msg = f"Error calling OpenAPI tool {tool_name}: {str(e)}"
+ verbose_logger.error(error_msg)
+ return CallToolResult(
+ content=[TextContent(type="text", text=error_msg)],
+ isError=True,
+ )
+
+ async def pre_call_tool_check(
+ self,
+ name: str,
+ arguments: Dict[str, Any],
+ server_name_from_prefix: str,
+ user_api_key_auth: Optional[UserAPIKeyAuth],
+ proxy_logging_obj: ProxyLogging,
+ server: MCPServer,
+ ):
+ ## check if the tool is allowed or banned for the given server
+ if not self.check_allowed_or_banned_tools(name, server):
+ raise HTTPException(
+ status_code=403,
+ detail={
+ "error": f"Tool {name} is not allowed for server {server.name}. Contact proxy admin to allow this tool."
+ },
+ )
+
+ ## check tool-level permissions from object_permission
+ await self.check_tool_permission_for_key_team(
+ tool_name=name,
+ server=server,
+ user_api_key_auth=user_api_key_auth,
+ )
+
+ ## filter parameters based on allowed_params configuration
+ self.validate_allowed_params(
+ tool_name=name,
+ arguments=arguments,
+ server=server,
+ )
+
+ pre_hook_kwargs = {
+ "name": name,
+ "arguments": arguments,
+ "server_name": server_name_from_prefix,
+ "user_api_key_auth": user_api_key_auth,
+ "user_api_key_user_id": (
+ getattr(user_api_key_auth, "user_id", None)
+ if user_api_key_auth
+ else None
+ ),
+ "user_api_key_team_id": (
+ getattr(user_api_key_auth, "team_id", None)
+ if user_api_key_auth
+ else None
+ ),
+ "user_api_key_end_user_id": (
+ getattr(user_api_key_auth, "end_user_id", None)
+ if user_api_key_auth
+ else None
+ ),
+ "user_api_key_hash": (
+ getattr(user_api_key_auth, "api_key_hash", None)
+ if user_api_key_auth
+ else None
+ ),
+ }
+
+ # Create MCP request object for processing
+ mcp_request_obj = proxy_logging_obj._create_mcp_request_object_from_kwargs(
+ pre_hook_kwargs
+ )
+
+ # Convert to LLM format for existing guardrail compatibility
+ synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
+ mcp_request_obj, pre_hook_kwargs
+ )
+
+ try:
+ # Use standard pre_call_hook with call_type="mcp_call"
+ modified_data = await proxy_logging_obj.pre_call_hook(
+ user_api_key_dict=user_api_key_auth, # type: ignore
+ data=synthetic_llm_data,
+ call_type="mcp_call", # type: ignore
+ )
+ if modified_data:
+ # Convert response back to MCP format and apply modifications
+ modified_kwargs = (
+ proxy_logging_obj._convert_mcp_hook_response_to_kwargs(
+ modified_data, pre_hook_kwargs
+ )
+ )
+ if modified_kwargs.get("arguments") != arguments:
+ arguments = modified_kwargs["arguments"]
+
+ except (
+ BlockedPiiEntityError,
+ GuardrailRaisedException,
+ HTTPException,
+ ) as e:
+ # Re-raise guardrail exceptions to properly fail the MCP call
+ verbose_logger.error(f"Guardrail blocked MCP tool call pre call: {str(e)}")
+ raise e
+
+ def _create_during_hook_task(
+ self,
+ name: str,
+ arguments: Dict[str, Any],
+ server_name_from_prefix: Optional[str],
+ user_api_key_auth: Optional[UserAPIKeyAuth],
+ proxy_logging_obj: ProxyLogging,
+ start_time: datetime.datetime,
+ ):
+ """Create and return a during hook task for MCP tool calls."""
+ from litellm.types.llms.base import HiddenParams
+ from litellm.types.mcp import MCPDuringCallRequestObject
+
+ request_obj = MCPDuringCallRequestObject(
+ tool_name=name,
+ arguments=arguments,
+ server_name=server_name_from_prefix,
+ start_time=start_time.timestamp() if start_time else None,
+ hidden_params=HiddenParams(),
+ )
+
+ during_hook_kwargs = {
+ "name": name,
+ "arguments": arguments,
+ "server_name": server_name_from_prefix,
+ "user_api_key_auth": user_api_key_auth,
+ }
+
+ synthetic_llm_data = proxy_logging_obj._convert_mcp_to_llm_format(
+ request_obj, during_hook_kwargs
+ )
+
+ return asyncio.create_task(
+ proxy_logging_obj.during_call_hook(
+ user_api_key_dict=user_api_key_auth,
+ data=synthetic_llm_data,
+ call_type="mcp_call", # type: ignore
+ )
+ )
+
+ async def _call_regular_mcp_tool(
+ self,
+ mcp_server: MCPServer,
+ original_tool_name: str,
+ arguments: Dict[str, Any],
+ tasks: List,
+ mcp_auth_header: Optional[str],
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]],
+ oauth2_headers: Optional[Dict[str, str]],
+ raw_headers: Optional[Dict[str, str]],
+ proxy_logging_obj: Optional[ProxyLogging],
+ ) -> CallToolResult:
+ """
+ Call a regular MCP tool using the MCP client.
+
+ Args:
+ mcp_server: The MCP server configuration
+ original_tool_name: The original tool name (without prefix)
+ arguments: Tool arguments
+ tasks: List of async tasks to append to (for during hooks)
+ mcp_auth_header: MCP auth header (deprecated)
+ mcp_server_auth_headers: Optional dict of server-specific auth headers
+ oauth2_headers: Optional OAuth2 headers
+ raw_headers: Optional raw headers from the request
+ proxy_logging_obj: Optional ProxyLogging object for hook integration
+
+ Returns:
+ CallToolResult from the MCP server
+
+ Raises:
+ BlockedPiiEntityError: If PII is blocked by guardrails
+ GuardrailRaisedException: If guardrails block the call
+ HTTPException: If an HTTP error occurs
+ """
+ # Get server-specific auth header if available
+ server_auth_header: Optional[Union[Dict[str, str], str]] = None
+ if mcp_server_auth_headers and mcp_server.alias:
+ server_auth_header = mcp_server_auth_headers.get(mcp_server.alias)
+ elif mcp_server_auth_headers and mcp_server.server_name:
+ server_auth_header = mcp_server_auth_headers.get(mcp_server.server_name)
+
+ # Fall back to deprecated mcp_auth_header if no server-specific header found
+ if server_auth_header is None:
+ server_auth_header = mcp_auth_header
+
+ # oauth2 headers
+ extra_headers: Optional[Dict[str, str]] = None
+ if mcp_server.auth_type == MCPAuth.oauth2:
+ extra_headers = oauth2_headers
+
+ if mcp_server.extra_headers and raw_headers:
+ if extra_headers is None:
+ extra_headers = {}
+ for header in mcp_server.extra_headers:
+ if header in raw_headers:
+ extra_headers[header] = raw_headers[header]
+
+ client = self._create_mcp_client(
+ server=mcp_server,
+ mcp_auth_header=server_auth_header,
+ extra_headers=extra_headers,
+ )
+
+ call_tool_params = MCPCallToolRequestParams(
+ name=original_tool_name,
+ arguments=arguments,
+ )
+
+ async def _call_tool_via_client(client, params):
+ async with client:
+ return await client.call_tool(params)
+
+ tasks.append(
+ asyncio.create_task(_call_tool_via_client(client, call_tool_params))
+ )
+
+ # IMPORTANT: Must await tasks INSIDE the context manager to keep connection alive
+ try:
+ mcp_responses = await asyncio.gather(*tasks)
+ except (
+ BlockedPiiEntityError,
+ GuardrailRaisedException,
+ HTTPException,
+ ) as e:
+ # Re-raise guardrail exceptions to properly fail the MCP call
+ verbose_logger.error(
+ f"Guardrail blocked MCP tool call during result check: {str(e)}"
+ )
+ raise e
+
+ # If proxy_logging_obj is None, the tool call result is at index 0
+ # If proxy_logging_obj is not None, the tool call result is at index 1 (after the during hook task)
+ result_index = 1 if proxy_logging_obj else 0
+ result = mcp_responses[result_index]
+
+ return cast(CallToolResult, result)
+
async def call_tool(
- self,
- name: str,
- arguments: Dict[str, Any],
- user_api_key_auth: Optional[UserAPIKeyAuth] = None,
- mcp_auth_header: Optional[str] = None,
- mcp_server_auth_headers: Optional[Dict[str, str]] = None,
+ self,
+ name: str,
+ arguments: Dict[str, Any],
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ mcp_auth_header: Optional[str] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
+ proxy_logging_obj: Optional[ProxyLogging] = None,
+ oauth2_headers: Optional[Dict[str, str]] = None,
+ raw_headers: Optional[Dict[str, str]] = None,
) -> CallToolResult:
"""
Call a tool with the given name and arguments (handles prefixed tool names)
@@ -482,12 +1175,18 @@ class MCPServerManager:
user_api_key_auth: User authentication
mcp_auth_header: MCP auth header (deprecated)
mcp_server_auth_headers: Optional dict of server-specific auth headers {server_alias: auth_value}
+ proxy_logging_obj: Optional ProxyLogging object for hook integration
+
Returns:
CallToolResult from the MCP server
"""
+ start_time = datetime.datetime.now()
+
# Remove prefix if present to get the original tool name
- original_tool_name, server_name_from_prefix = get_server_name_prefix_tool_mcp(name)
+ original_tool_name, server_name_from_prefix = get_server_name_prefix_tool_mcp(
+ name
+ )
# Get the MCP server
mcp_server = self._get_mcp_server_from_tool_name(name)
@@ -497,38 +1196,90 @@ class MCPServerManager:
# Validate that the server from prefix matches the actual server (if prefix was used)
if server_name_from_prefix:
expected_prefix = get_server_prefix(mcp_server)
- if normalize_server_name(server_name_from_prefix) != normalize_server_name(expected_prefix):
+ if normalize_server_name(server_name_from_prefix) != normalize_server_name(
+ expected_prefix
+ ):
raise ValueError(
- f"Tool {name} server prefix mismatch: expected {expected_prefix}, got {server_name_from_prefix}")
+ f"Tool {name} server prefix mismatch: expected {expected_prefix}, got {server_name_from_prefix}"
+ )
- # Get server-specific auth header if available
- server_auth_header = None
- if mcp_server_auth_headers and mcp_server.alias:
- server_auth_header = mcp_server_auth_headers.get(mcp_server.alias)
- elif mcp_server_auth_headers and mcp_server.server_name:
- server_auth_header = mcp_server_auth_headers.get(mcp_server.server_name)
-
- # Fall back to deprecated mcp_auth_header if no server-specific header found
- if server_auth_header is None:
- server_auth_header = mcp_auth_header
-
- client = self._create_mcp_client(
- server=mcp_server,
- mcp_auth_header=server_auth_header,
- )
- async with client:
- # Use the original tool name (without prefix) for the actual call
- call_tool_params = MCPCallToolRequestParams(
+ #########################################################
+ # Pre MCP Tool Call Hook
+ # Allow validation and modification of tool calls before execution
+ # Using standard pre_call_hook with call_type="mcp_call"
+ #########################################################
+ if proxy_logging_obj:
+ await self.pre_call_tool_check(
name=original_tool_name,
arguments=arguments,
+ server_name_from_prefix=server_name_from_prefix,
+ user_api_key_auth=user_api_key_auth,
+ proxy_logging_obj=proxy_logging_obj,
+ server=mcp_server,
)
- return await client.call_tool(call_tool_params)
+
+ # Prepare tasks for during hooks
+ tasks = []
+ if proxy_logging_obj:
+ during_hook_task = self._create_during_hook_task(
+ name=name,
+ arguments=arguments,
+ server_name_from_prefix=server_name_from_prefix,
+ user_api_key_auth=user_api_key_auth,
+ proxy_logging_obj=proxy_logging_obj,
+ start_time=start_time,
+ )
+ tasks.append(during_hook_task)
+
+ # For OpenAPI servers, call the tool handler directly instead of via MCP client
+ if mcp_server.spec_path:
+ verbose_logger.debug(
+ f"Calling OpenAPI tool {name} directly via HTTP handler"
+ )
+ tasks.append(
+ asyncio.create_task(
+ self._call_openapi_tool_handler(mcp_server, name, arguments)
+ )
+ )
+ else:
+ # For regular MCP servers, use the MCP client
+ return await self._call_regular_mcp_tool(
+ mcp_server=mcp_server,
+ original_tool_name=original_tool_name,
+ arguments=arguments,
+ tasks=tasks,
+ mcp_auth_header=mcp_auth_header,
+ mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
+ proxy_logging_obj=proxy_logging_obj,
+ )
+
+ # For OpenAPI tools, await outside the client context
+ try:
+ mcp_responses = await asyncio.gather(*tasks)
+
+ # If proxy_logging_obj is None, the tool call result is at index 0
+ # If proxy_logging_obj is not None, the tool call result is at index 1 (after the during hook task)
+ result_index = 1 if proxy_logging_obj else 0
+ result = mcp_responses[result_index]
+
+ return cast(CallToolResult, result)
+ except (
+ BlockedPiiEntityError,
+ GuardrailRaisedException,
+ HTTPException,
+ ) as e:
+ # Re-raise guardrail exceptions to properly fail the MCP call
+ verbose_logger.error(
+ f"Guardrail blocked MCP tool call during result check: {str(e)}"
+ )
+ raise e
#########################################################
# End of Methods that call the upstream MCP servers
#########################################################
-
def initialize_tool_name_to_mcp_server_name_mapping(self):
"""
On startup, initialize the tool name to MCP server name mapping
@@ -571,14 +1322,18 @@ class MCPServerManager:
if tool_name in self.tool_name_to_mcp_server_name_mapping:
server_name = self.tool_name_to_mcp_server_name_mapping[tool_name]
for server in self.get_registry().values():
- if normalize_server_name(server.name) == normalize_server_name(server_name):
+ if normalize_server_name(server.name) == normalize_server_name(
+ server_name
+ ):
return server
# If not found and tool name is prefixed, try extracting server name from prefix
if is_tool_name_prefixed(tool_name):
_, server_name_from_prefix = get_server_name_prefix_tool_mcp(tool_name)
for server in self.get_registry().values():
- if normalize_server_name(server.name) == normalize_server_name(server_name_from_prefix):
+ if normalize_server_name(server.name) == normalize_server_name(
+ server_name_from_prefix
+ ):
return server
return None
@@ -589,30 +1344,59 @@ class MCPServerManager:
get_prisma_client_or_throw,
)
+ verbose_logger.debug("Loading MCP servers from database into registry...")
+
# perform authz check to filter the mcp servers user has access to
prisma_client = get_prisma_client_or_throw(
"Database not connected. Connect a database to your proxy"
)
db_mcp_servers = await get_all_mcp_servers(prisma_client)
+ verbose_logger.info(f"Found {len(db_mcp_servers)} MCP servers in database")
+
# ensure the global_mcp_server_manager is up to date with the db
for server in db_mcp_servers:
+ verbose_logger.debug(
+ f"Adding server to registry: {server.server_id} ({server.server_name})"
+ )
self.add_update_server(server)
+ verbose_logger.debug(
+ f"Registry now contains {len(self.get_registry())} servers"
+ )
+
def get_mcp_server_by_id(self, server_id: str) -> Optional[MCPServer]:
"""
Get the MCP Server from the server id
"""
- for server in self.get_registry().values():
+ registry = self.get_registry()
+ for server in registry.values():
if server.server_id == server_id:
return server
return None
+ def get_mcp_server_names_from_ids(self, server_ids: List[str]) -> List[str]:
+ server_names = []
+ registry = self.get_registry()
+ for server in registry.values():
+ if server.server_id in server_ids:
+ server_names.append(server.name)
+ return server_names
+
+ def get_mcp_server_by_name(self, server_name: str) -> Optional[MCPServer]:
+ """
+ Get the MCP Server from the server name
+ """
+ registry = self.get_registry()
+ for server in registry.values():
+ if server.server_name == server_name:
+ return server
+ return None
+
def _generate_stable_server_id(
self,
server_name: str,
url: str,
transport: str,
- spec_version: str,
auth_type: Optional[str] = None,
alias: Optional[str] = None,
) -> str:
@@ -628,7 +1412,6 @@ class MCPServerManager:
server_name: Name of the server
url: Server URL
transport: Transport type (sse, http, etc.)
- spec_version: MCP spec version
auth_type: Authentication type (optional)
alias: Server alias (optional)
@@ -637,7 +1420,7 @@ class MCPServerManager:
"""
# Create a string from all the identifying parameters
params_string = (
- f"{server_name}|{url}|{transport}|{spec_version}|{auth_type or ''}|{alias or ''}"
+ f"{server_name}|{url}|{transport}|{auth_type or ''}|{alias or ''}"
)
# Generate SHA-256 hash
@@ -647,5 +1430,228 @@ class MCPServerManager:
# Take first 32 characters and format as UUID-like string
return hash_hex[:32]
+ async def health_check_server(
+ self, server_id: str, mcp_auth_header: Optional[str] = None
+ ) -> Dict[str, Any]:
+ """
+ Perform a health check on a specific MCP server.
+
+ Args:
+ server_id: The ID of the server to health check
+ mcp_auth_header: Optional authentication header for the MCP server
+
+ Returns:
+ Dict containing health check results
+ """
+ import time
+ from datetime import datetime
+
+ server = self.get_mcp_server_by_id(server_id)
+ if not server:
+ return {
+ "server_id": server_id,
+ "server_name": None,
+ "status": "unknown",
+ "error": "Server not found",
+ "last_health_check": datetime.now().isoformat(),
+ "response_time_ms": None,
+ }
+
+ start_time = time.time()
+ try:
+ # Try to get tools from the server as a health check
+ tools = await self._get_tools_from_server(server, mcp_auth_header)
+ response_time = (time.time() - start_time) * 1000
+
+ return {
+ "server_id": server_id,
+ "server_name": server.name,
+ "status": "healthy",
+ "tools_count": len(tools),
+ "last_health_check": datetime.now().isoformat(),
+ "response_time_ms": round(response_time, 2),
+ "error": None,
+ }
+ except Exception as e:
+ response_time = (time.time() - start_time) * 1000
+ error_message = str(e)
+
+ return {
+ "server_id": server_id,
+ "server_name": server.name,
+ "status": "unhealthy",
+ "last_health_check": datetime.now().isoformat(),
+ "response_time_ms": round(response_time, 2),
+ "error": error_message,
+ }
+
+ async def health_check_all_servers(
+ self, mcp_auth_header: Optional[str] = None
+ ) -> Dict[str, Any]:
+ """
+ Perform health checks on all MCP servers.
+
+ Args:
+ mcp_auth_header: Optional authentication header for the MCP servers
+
+ Returns:
+ Dict containing health check results for all servers
+ """
+ all_servers = self.get_registry()
+ results = {}
+
+ for server_id, server in all_servers.items():
+ results[server_id] = await self.health_check_server(
+ server_id, mcp_auth_header
+ )
+
+ return results
+
+ async def health_check_allowed_servers(
+ self,
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ mcp_auth_header: Optional[str] = None,
+ ) -> Dict[str, Any]:
+ """
+ Perform health checks on all MCP servers that the user has access to.
+
+ Args:
+ user_api_key_auth: User authentication info for access control
+ mcp_auth_header: Optional authentication header for the MCP servers
+
+ Returns:
+ Dict containing health check results for accessible servers
+ """
+ # Get allowed servers for the user
+ allowed_server_ids = await self.get_allowed_mcp_servers(user_api_key_auth)
+
+ # Perform health checks on allowed servers
+ results = {}
+ for server_id in allowed_server_ids:
+ results[server_id] = await self.health_check_server(
+ server_id, mcp_auth_header
+ )
+
+ return results
+
+ async def get_all_mcp_servers_with_health_and_teams(
+ self,
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ include_health: bool = True,
+ ) -> List[LiteLLM_MCPServerTable]:
+ """
+ Get all MCP servers that the user has access to, with health status and team information.
+
+ Args:
+ user_api_key_auth: User authentication info for access control
+ include_health: Whether to include health check information
+
+ Returns:
+ List of MCP server objects with health and team data
+ """
+ from litellm.proxy._experimental.mcp_server.db import (
+ get_all_mcp_servers,
+ get_mcp_servers,
+ )
+ from litellm.proxy.management_endpoints.common_utils import _user_has_admin_view
+ from litellm.proxy.proxy_server import prisma_client
+
+ # Get allowed server IDs
+ allowed_server_ids = await self.get_allowed_mcp_servers(user_api_key_auth)
+
+ # Get servers from database
+ list_mcp_servers: List[LiteLLM_MCPServerTable] = []
+ if prisma_client is not None:
+ list_mcp_servers = await get_mcp_servers(prisma_client, allowed_server_ids)
+
+ # If admin, also get all servers from database
+ if user_api_key_auth and _user_has_admin_view(user_api_key_auth):
+ all_mcp_servers = await get_all_mcp_servers(prisma_client)
+ for server in all_mcp_servers:
+ if server.server_id not in allowed_server_ids:
+ list_mcp_servers.append(server)
+
+ # Add config.yaml servers
+ for _server_id, _server_config in self.config_mcp_servers.items():
+ if _server_id in allowed_server_ids:
+ list_mcp_servers.append(
+ LiteLLM_MCPServerTable(
+ **{
+ **_server_config.model_dump(),
+ "created_at": datetime.datetime.now(),
+ "updated_at": datetime.datetime.now(),
+ "description": (
+ _server_config.mcp_info.get("description")
+ if _server_config.mcp_info
+ else None
+ ),
+ "allowed_tools": _server_config.allowed_tools or [],
+ "mcp_info": _server_config.mcp_info,
+ "mcp_access_groups": _server_config.access_groups or [],
+ "extra_headers": _server_config.extra_headers or [],
+ "command": getattr(_server_config, "command", None),
+ "args": getattr(_server_config, "args", None) or [],
+ "env": getattr(_server_config, "env", None) or {},
+ }
+ )
+ )
+
+ # Get team information for non-admin users
+ server_to_teams_map: Dict[str, List[Dict[str, str]]] = {}
+ if (
+ user_api_key_auth
+ and not _user_has_admin_view(user_api_key_auth)
+ and prisma_client is not None
+ ):
+ teams = await prisma_client.db.litellm_teamtable.find_many(
+ include={"object_permission": True}
+ )
+
+ user_teams = []
+ for team in teams:
+ if team.members_with_roles:
+ for member in team.members_with_roles:
+ if (
+ "user_id" in member
+ and member["user_id"] is not None
+ and member["user_id"] == user_api_key_auth.user_id
+ ):
+ user_teams.append(team)
+
+ # Create a mapping of server_id to teams that have access to it
+ for team in user_teams:
+ if team.object_permission and team.object_permission.mcp_servers:
+ for server_id in team.object_permission.mcp_servers:
+ if server_id not in server_to_teams_map:
+ server_to_teams_map[server_id] = []
+ server_to_teams_map[server_id].append(
+ {
+ "team_id": team.team_id,
+ "team_alias": team.team_alias,
+ "organization_id": team.organization_id,
+ }
+ )
+
+ ## mark invalid servers w/ reason for being invalid
+ valid_server_ids = self.get_all_mcp_server_ids()
+ for server in list_mcp_servers:
+ if server.server_id not in valid_server_ids:
+ server.status = "unhealthy"
+ ## try adding server to registry to get error
+ try:
+ self.add_update_server(server)
+ except Exception as e:
+ server.health_check_error = str(e)
+ server.health_check_error = "Server is not in in memory registry yet. This could be a temporary sync issue."
+
+ return list_mcp_servers
+
+ async def reload_servers_from_database(self):
+ """
+ Public method to reload all MCP servers from database into registry.
+ This can be called from management endpoints to ensure registry is up to date.
+ """
+ await self._add_mcp_servers_from_db_to_in_memory_registry()
+
global_mcp_server_manager: MCPServerManager = MCPServerManager()
diff --git a/litellm/proxy/_experimental/mcp_server/openapi_to_mcp_generator.py b/litellm/proxy/_experimental/mcp_server/openapi_to_mcp_generator.py
new file mode 100644
index 00000000000..72288f8e673
--- /dev/null
+++ b/litellm/proxy/_experimental/mcp_server/openapi_to_mcp_generator.py
@@ -0,0 +1,236 @@
+"""
+This module is used to generate MCP tools from OpenAPI specs.
+"""
+
+import json
+from typing import Any, Dict, Optional
+
+import httpx
+
+from litellm._logging import verbose_logger
+from litellm.proxy._experimental.mcp_server.tool_registry import (
+ global_mcp_tool_registry,
+)
+
+# Store the base URL and headers globally
+BASE_URL = ""
+HEADERS: Dict[str, str] = {}
+
+
+def load_openapi_spec(filepath: str) -> Dict[str, Any]:
+ """Load OpenAPI specification from JSON file."""
+ with open(filepath, "r") as f:
+ return json.load(f)
+
+
+def get_base_url(spec: Dict[str, Any]) -> str:
+ """Extract base URL from OpenAPI spec."""
+ # OpenAPI 3.x
+ if "servers" in spec and spec["servers"]:
+ return spec["servers"][0]["url"]
+ # OpenAPI 2.x (Swagger)
+ elif "host" in spec:
+ scheme = spec.get("schemes", ["https"])[0]
+ base_path = spec.get("basePath", "")
+ return f"{scheme}://{spec['host']}{base_path}"
+ return ""
+
+
+def extract_parameters(operation: Dict[str, Any]) -> tuple:
+ """Extract parameter names from OpenAPI operation."""
+ path_params = []
+ query_params = []
+ body_params = []
+
+ # OpenAPI 3.x and 2.x parameters
+ if "parameters" in operation:
+ for param in operation["parameters"]:
+ param_name = param["name"]
+ if param.get("in") == "path":
+ path_params.append(param_name)
+ elif param.get("in") == "query":
+ query_params.append(param_name)
+ elif param.get("in") == "body":
+ body_params.append(param_name)
+
+ # OpenAPI 3.x requestBody
+ if "requestBody" in operation:
+ body_params.append("body")
+
+ return path_params, query_params, body_params
+
+
+def build_input_schema(operation: Dict[str, Any]) -> Dict[str, Any]:
+ """Build MCP input schema from OpenAPI operation."""
+ properties = {}
+ required = []
+
+ # Process parameters
+ if "parameters" in operation:
+ for param in operation["parameters"]:
+ param_name = param["name"]
+ param_schema = param.get("schema", {})
+ param_type = param_schema.get("type", "string")
+
+ properties[param_name] = {
+ "type": param_type,
+ "description": param.get("description", ""),
+ }
+
+ if param.get("required", False):
+ required.append(param_name)
+
+ # Process requestBody (OpenAPI 3.x)
+ if "requestBody" in operation:
+ request_body = operation["requestBody"]
+ content = request_body.get("content", {})
+
+ # Try to get JSON schema
+ if "application/json" in content:
+ schema = content["application/json"].get("schema", {})
+ properties["body"] = {
+ "type": "object",
+ "description": request_body.get("description", "Request body"),
+ "properties": schema.get("properties", {}),
+ }
+ if request_body.get("required", False):
+ required.append("body")
+
+ return {
+ "type": "object",
+ "properties": properties,
+ "required": required if required else [],
+ }
+
+
+def create_tool_function(
+ path: str,
+ method: str,
+ operation: Dict[str, Any],
+ base_url: str,
+ headers: Optional[Dict[str, str]] = None,
+):
+ """Create a tool function for an OpenAPI operation.
+
+ Args:
+ path: API endpoint path
+ method: HTTP method (get, post, put, delete, patch)
+ operation: OpenAPI operation object
+ base_url: Base URL for the API
+ headers: Optional headers to include in requests (e.g., authentication)
+ """
+ if headers is None:
+ headers = {}
+
+ path_params, query_params, body_params = extract_parameters(operation)
+ all_params = path_params + query_params + body_params
+
+ # Build function signature dynamically
+ if all_params:
+ params_str = ", ".join(f"{p}: str = ''" for p in all_params)
+ else:
+ params_str = ""
+
+ # Create the function code as a string
+ func_code = f'''
+async def tool_function({params_str}) -> str:
+ """Dynamically generated tool function."""
+ url = base_url + path
+
+ # Replace path parameters
+ path_param_names = {path_params}
+ for param_name in path_param_names:
+ param_value = locals().get(param_name, "")
+ if param_value:
+ url = url.replace("{{" + param_name + "}}", str(param_value))
+
+ # Build query params
+ query_param_names = {query_params}
+ params = {{}}
+ for param_name in query_param_names:
+ param_value = locals().get(param_name, "")
+ if param_value:
+ params[param_name] = param_value
+
+ # Build request body
+ body_param_names = {body_params}
+ json_body = None
+ if body_param_names:
+ body_value = locals().get("body", {{}})
+ if isinstance(body_value, dict):
+ json_body = body_value
+ elif body_value:
+ # If it's a string, try to parse as JSON
+ import json as json_module
+ try:
+ json_body = json_module.loads(body_value) if isinstance(body_value, str) else {{"data": body_value}}
+ except:
+ json_body = {{"data": body_value}}
+
+ # Make HTTP request
+ async with httpx.AsyncClient() as client:
+ if "{method.lower()}" == "get":
+ response = await client.get(url, params=params, headers=headers)
+ elif "{method.lower()}" == "post":
+ response = await client.post(url, params=params, json=json_body, headers=headers)
+ elif "{method.lower()}" == "put":
+ response = await client.put(url, params=params, json=json_body, headers=headers)
+ elif "{method.lower()}" == "delete":
+ response = await client.delete(url, params=params, headers=headers)
+ elif "{method.lower()}" == "patch":
+ response = await client.patch(url, params=params, json=json_body, headers=headers)
+ else:
+ return "Unsupported HTTP method: {method}"
+
+ return response.text
+'''
+
+ # Execute the function code to create the actual function
+ local_vars = {
+ "httpx": httpx,
+ "headers": headers,
+ "base_url": base_url,
+ "path": path,
+ "method": method,
+ }
+ exec(func_code, local_vars)
+
+ return local_vars["tool_function"]
+
+
+def register_tools_from_openapi(spec: Dict[str, Any], base_url: str):
+ """Register MCP tools from OpenAPI specification."""
+ paths = spec.get("paths", {})
+
+ for path, path_item in paths.items():
+ for method in ["get", "post", "put", "delete", "patch"]:
+ if method in path_item:
+ operation = path_item[method]
+
+ # Generate tool name
+ operation_id = operation.get(
+ "operationId", f"{method}_{path.replace('/', '_')}"
+ )
+ tool_name = operation_id.replace(" ", "_").lower()
+
+ # Get description
+ description = operation.get(
+ "summary", operation.get("description", f"{method.upper()} {path}")
+ )
+
+ # Build input schema
+ input_schema = build_input_schema(operation)
+
+ # Create tool function
+ tool_func = create_tool_function(path, method, operation, base_url)
+ tool_func.__name__ = tool_name
+ tool_func.__doc__ = description
+
+ # Register tool with local registry
+ global_mcp_tool_registry.register_tool(
+ name=tool_name,
+ description=description,
+ input_schema=input_schema,
+ handler=tool_func,
+ )
+ verbose_logger.debug(f"Registered tool: {tool_name}")
diff --git a/litellm/proxy/_experimental/mcp_server/rest_endpoints.py b/litellm/proxy/_experimental/mcp_server/rest_endpoints.py
index 86cb13746ae..6a9c425a81b 100644
--- a/litellm/proxy/_experimental/mcp_server/rest_endpoints.py
+++ b/litellm/proxy/_experimental/mcp_server/rest_endpoints.py
@@ -1,5 +1,5 @@
import importlib
-from typing import Optional
+from typing import Dict, List, Optional, Union
from fastapi import APIRouter, Depends, Query, Request
@@ -21,19 +21,73 @@ router = APIRouter(
)
if MCP_AVAILABLE:
+ from litellm.experimental_mcp_client.client import MCPTool
from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
global_mcp_server_manager,
)
from litellm.proxy._experimental.mcp_server.server import (
ListMCPToolsRestAPIResponseObject,
call_mcp_tool,
+ filter_tools_by_allowed_tools,
)
########################################################
############ MCP Server REST API Routes #################
+ def _get_server_auth_header(
+ server,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]],
+ mcp_auth_header: Optional[str],
+ ) -> Optional[Union[Dict[str, str], str]]:
+ """Helper function to get server-specific auth header with case-insensitive matching."""
+ if mcp_server_auth_headers and server.alias:
+ normalized_server_alias = server.alias.lower()
+ normalized_headers = {
+ k.lower(): v for k, v in mcp_server_auth_headers.items()
+ }
+ server_auth = normalized_headers.get(normalized_server_alias)
+ if server_auth is not None:
+ return server_auth
+ elif mcp_server_auth_headers and server.server_name:
+ normalized_server_name = server.server_name.lower()
+ normalized_headers = {
+ k.lower(): v for k, v in mcp_server_auth_headers.items()
+ }
+ server_auth = normalized_headers.get(normalized_server_name)
+ if server_auth is not None:
+ return server_auth
+ return mcp_auth_header
+
+ def _create_tool_response_objects(tools, server_mcp_info):
+ """Helper function to create tool response objects."""
+ return [
+ ListMCPToolsRestAPIResponseObject(
+ name=tool.name,
+ description=tool.description,
+ inputSchema=tool.inputSchema,
+ mcp_info=server_mcp_info,
+ )
+ for tool in tools
+ ]
+
+ async def _get_tools_for_single_server(server, server_auth_header):
+ """Helper function to get tools for a single server."""
+ tools = await global_mcp_server_manager._get_tools_from_server(
+ server=server,
+ mcp_auth_header=server_auth_header,
+ add_prefix=False,
+ )
+
+ # Filter tools based on allowed_tools configuration
+ # Only filter if allowed_tools is explicitly configured (not None and not empty)
+ if server.allowed_tools is not None and len(server.allowed_tools) > 0:
+ tools = filter_tools_by_allowed_tools(tools, server)
+
+ return _create_tool_response_objects(tools, server.mcp_info)
+
########################################################
@router.get("/tools/list", dependencies=[Depends(user_api_key_auth)])
async def list_tool_rest_api(
+ request: Request,
server_id: Optional[str] = Query(
None, description="The server id to list tools for"
),
@@ -59,10 +113,23 @@ if MCP_AVAILABLE:
"message": "Successfully retrieved tools"
}
"""
+ from litellm.proxy._experimental.mcp_server.auth.user_api_key_auth_mcp import (
+ MCPRequestHandler,
+ )
+
try:
+ # Extract auth headers from request
+ headers = request.headers
+ mcp_auth_header = MCPRequestHandler._get_mcp_auth_header_from_headers(
+ headers
+ )
+ mcp_server_auth_headers = (
+ MCPRequestHandler._get_mcp_server_auth_headers_from_headers(headers)
+ )
+
list_tools_result = []
error_message = None
-
+
# If server_id is specified, only query that specific server
if server_id:
server = global_mcp_server_manager.get_mcp_server_by_id(server_id)
@@ -70,65 +137,67 @@ if MCP_AVAILABLE:
return {
"tools": [],
"error": "server_not_found",
- "message": f"Server with id {server_id} not found"
+ "message": f"Server with id {server_id} not found",
}
+
+ server_auth_header = _get_server_auth_header(
+ server, mcp_server_auth_headers, mcp_auth_header
+ )
+
try:
- tools = await global_mcp_server_manager._get_tools_from_server(
- server=server,
+ list_tools_result = await _get_tools_for_single_server(
+ server, server_auth_header
)
- for tool in tools:
- list_tools_result.append(
- ListMCPToolsRestAPIResponseObject(
- name=tool.name,
- description=tool.description,
- inputSchema=tool.inputSchema,
- mcp_info=server.mcp_info,
- )
- )
except Exception as e:
- verbose_logger.exception(f"Error getting tools from {server.name}: {e}")
+ verbose_logger.exception(
+ f"Error getting tools from {server.name}: {e}"
+ )
return {
"tools": [],
"error": "server_error",
- "message": f"Failed to get tools from server {server.name}: {str(e)}"
+ "message": f"Failed to get tools from server {server.name}: {str(e)}",
}
else:
# Query all servers
errors = []
for server in global_mcp_server_manager.get_registry().values():
+ server_auth_header = _get_server_auth_header(
+ server, mcp_server_auth_headers, mcp_auth_header
+ )
+
try:
- tools = await global_mcp_server_manager._get_tools_from_server(
- server=server,
+ tools_result = await _get_tools_for_single_server(
+ server, server_auth_header
)
- for tool in tools:
- list_tools_result.append(
- ListMCPToolsRestAPIResponseObject(
- name=tool.name,
- description=tool.description,
- inputSchema=tool.inputSchema,
- mcp_info=server.mcp_info,
- )
- )
+ list_tools_result.extend(tools_result)
except Exception as e:
- verbose_logger.exception(f"Error getting tools from {server.name}: {e}")
+ verbose_logger.exception(
+ f"Error getting tools from {server.name}: {e}"
+ )
errors.append(f"{server.name}: {str(e)}")
continue
-
+
if errors and not list_tools_result:
- error_message = "Failed to get tools from servers: " + "; ".join(errors)
-
+ error_message = "Failed to get tools from servers: " + "; ".join(
+ errors
+ )
+
return {
"tools": list_tools_result,
"error": "partial_failure" if error_message else None,
- "message": error_message if error_message else "Successfully retrieved tools"
+ "message": (
+ error_message if error_message else "Successfully retrieved tools"
+ ),
}
-
+
except Exception as e:
- verbose_logger.exception("Unexpected error in list_tool_rest_api: %s", str(e))
+ verbose_logger.exception(
+ "Unexpected error in list_tool_rest_api: %s", str(e)
+ )
return {
"tools": [],
"error": "unexpected_error",
- "message": f"An unexpected error occurred: {str(e)}"
+ "message": f"An unexpected error occurred: {str(e)}",
}
@router.post("/tools/call", dependencies=[Depends(user_api_key_auth)])
@@ -139,17 +208,55 @@ if MCP_AVAILABLE:
"""
REST API to call a specific MCP tool with the provided arguments
"""
+ from fastapi import HTTPException
+
+ from litellm.exceptions import BlockedPiiEntityError, GuardrailRaisedException
from litellm.proxy.proxy_server import add_litellm_data_to_request, proxy_config
- data = await request.json()
- data = await add_litellm_data_to_request(
- data=data,
- request=request,
- user_api_key_dict=user_api_key_dict,
- proxy_config=proxy_config,
- )
- return await call_mcp_tool(**data)
-
+ try:
+ data = await request.json()
+ data = await add_litellm_data_to_request(
+ data=data,
+ request=request,
+ user_api_key_dict=user_api_key_dict,
+ proxy_config=proxy_config,
+ )
+ return await call_mcp_tool(**data)
+ except BlockedPiiEntityError as e:
+ verbose_logger.error(f"BlockedPiiEntityError in MCP tool call: {str(e)}")
+ raise HTTPException(
+ status_code=400,
+ detail={
+ "error": "blocked_pii_entity",
+ "message": str(e),
+ "entity_type": getattr(e, "entity_type", None),
+ "guardrail_name": getattr(e, "guardrail_name", None),
+ },
+ )
+ except GuardrailRaisedException as e:
+ verbose_logger.error(f"GuardrailRaisedException in MCP tool call: {str(e)}")
+ raise HTTPException(
+ status_code=400,
+ detail={
+ "error": "guardrail_violation",
+ "message": str(e),
+ "guardrail_name": getattr(e, "guardrail_name", None),
+ },
+ )
+ except HTTPException as e:
+ # Re-raise HTTPException as-is to preserve status code and detail
+ verbose_logger.error(f"HTTPException in MCP tool call: {str(e)}")
+ raise e
+ except Exception as e:
+ verbose_logger.exception(f"Unexpected error in MCP tool call: {str(e)}")
+ raise HTTPException(
+ status_code=500,
+ detail={
+ "error": "internal_server_error",
+ "message": f"An unexpected error occurred: {str(e)}",
+ },
+ )
+
########################################################
# MCP Connection testing routes
# /health -> Test if we can connect to the MCP server
@@ -160,13 +267,19 @@ if MCP_AVAILABLE:
from litellm.proxy.management_endpoints.mcp_management_endpoints import (
NewMCPServerRequest,
)
- @router.post("/test/connection")
- async def test_connection(
- request: NewMCPServerRequest,
- ):
+
+ async def _execute_with_mcp_client(request: NewMCPServerRequest, operation):
"""
- Test if we can connect to the provided MCP server before adding it
+ Common helper to create MCP client, execute operation, and ensure proper cleanup.
+
+ Args:
+ request: MCP server configuration
+ operation: Async function that takes a client and returns the operation result
+
+ Returns:
+ Operation result or error response
"""
+ client = None
try:
client = global_mcp_server_manager._create_mcp_client(
server=MCPServer(
@@ -174,20 +287,39 @@ if MCP_AVAILABLE:
name=request.alias or request.server_name or "",
url=request.url,
transport=request.transport,
- spec_version=request.spec_version,
auth_type=request.auth_type,
mcp_info=request.mcp_info,
),
mcp_auth_header=None,
)
- await client.connect()
+ return await operation(client)
+
except Exception as e:
- verbose_logger.error(f"Error in test_connection: {e}", exc_info=True)
+ verbose_logger.error(f"Error in MCP operation: {e}", exc_info=True)
return {"status": "error", "message": "An internal error has occurred."}
- return {"status": "ok"}
-
-
+ finally:
+ # Ensure client is properly disconnected before response is sent
+ if client is not None:
+ try:
+ await client.disconnect()
+ except Exception as e:
+ verbose_logger.warning(f"Error disconnecting MCP client: {e}")
+
+ @router.post("/test/connection")
+ async def test_connection(
+ request: NewMCPServerRequest,
+ ):
+ """
+ Test if we can connect to the provided MCP server before adding it
+ """
+
+ async def _test_connection_operation(client):
+ await client.connect()
+ return {"status": "ok"}
+
+ return await _execute_with_mcp_client(request, _test_connection_operation)
+
@router.post("/test/tools/list")
async def test_tools_list(
request: NewMCPServerRequest,
@@ -196,25 +328,16 @@ if MCP_AVAILABLE:
"""
Preview tools available from MCP server before adding it
"""
- try:
- client = global_mcp_server_manager._create_mcp_client(
- server=MCPServer(
- server_id=request.server_id or "",
- name=request.alias or request.server_name or "",
- url=request.url,
- transport=request.transport,
- spec_version=request.spec_version,
- auth_type=request.auth_type,
- mcp_info=request.mcp_info,
- ),
- mcp_auth_header=None,
- )
- list_tools_result = await client.list_tools()
- except Exception as e:
- verbose_logger.error(f"Error in test_tools_list: {e}", exc_info=True)
- return {"status": "error", "message": "An internal error has occurred."}
- return {
- "tools": list_tools_result,
- "error": None,
- "message": "Successfully retrieved tools"
- }
+
+ async def _list_tools_operation(client):
+ list_tools_result: List[MCPTool] = await client.list_tools()
+ model_dumped_tools: List[dict] = [
+ tool.model_dump() for tool in list_tools_result
+ ]
+ return {
+ "tools": model_dumped_tools,
+ "error": None,
+ "message": "Successfully retrieved tools",
+ }
+
+ return await _execute_with_mcp_client(request, _list_tools_operation)
diff --git a/litellm/proxy/_experimental/mcp_server/server.py b/litellm/proxy/_experimental/mcp_server/server.py
index 8250b3e769a..77d6abfed62 100644
--- a/litellm/proxy/_experimental/mcp_server/server.py
+++ b/litellm/proxy/_experimental/mcp_server/server.py
@@ -22,6 +22,7 @@ from litellm.proxy._experimental.mcp_server.utils import (
LITELLM_MCP_SERVER_VERSION,
)
from litellm.proxy._types import UserAPIKeyAuth
+from litellm.types.mcp import MCPAuth
from litellm.types.mcp_server.mcp_server_manager import MCPInfo, MCPServer
from litellm.types.utils import StandardLoggingMCPToolCall
from litellm.utils import client
@@ -40,6 +41,7 @@ except ImportError as e:
# Global variables to track initialization
_SESSION_MANAGERS_INITIALIZED = False
+_INITIALIZATION_LOCK = asyncio.Lock()
if MCP_AVAILABLE:
from mcp.server import Server
@@ -113,23 +115,25 @@ if MCP_AVAILABLE:
"""Initialize the session managers. Can be called from main app lifespan."""
global _SESSION_MANAGERS_INITIALIZED, _session_manager_cm, _sse_session_manager_cm
- if _SESSION_MANAGERS_INITIALIZED:
- return
+ # Use async lock to prevent concurrent initialization
+ async with _INITIALIZATION_LOCK:
+ if _SESSION_MANAGERS_INITIALIZED:
+ return
- verbose_logger.info("Initializing MCP session managers...")
+ verbose_logger.info("Initializing MCP session managers...")
- # Start the session managers with context managers
- _session_manager_cm = session_manager.run()
- _sse_session_manager_cm = sse_session_manager.run()
+ # Start the session managers with context managers
+ _session_manager_cm = session_manager.run()
+ _sse_session_manager_cm = sse_session_manager.run()
- # Enter the context managers
- await _session_manager_cm.__aenter__()
- await _sse_session_manager_cm.__aenter__()
+ # Enter the context managers
+ await _session_manager_cm.__aenter__()
+ await _sse_session_manager_cm.__aenter__()
- _SESSION_MANAGERS_INITIALIZED = True
- verbose_logger.info(
- "MCP Server started with StreamableHTTP and SSE session managers!"
- )
+ _SESSION_MANAGERS_INITIALIZED = True
+ verbose_logger.info(
+ "MCP Server started with StreamableHTTP and SSE session managers!"
+ )
async def shutdown_session_managers():
"""Shutdown the session managers."""
@@ -168,24 +172,44 @@ if MCP_AVAILABLE:
"""
List all available tools
"""
- # Get user authentication from context variable
- user_api_key_auth, mcp_auth_header, mcp_servers, mcp_server_auth_headers = get_auth_context()
- verbose_logger.debug(
- f"MCP list_tools - User API Key Auth from context: {user_api_key_auth}"
- )
- verbose_logger.debug(
- f"MCP list_tools - MCP servers from context: {mcp_servers}"
- )
- verbose_logger.debug(
- f"MCP list_tools - MCP server auth headers: {list(mcp_server_auth_headers.keys()) if mcp_server_auth_headers else None}"
- )
- # Get mcp_servers from context variable
- return await _list_mcp_tools(
- user_api_key_auth=user_api_key_auth,
- mcp_auth_header=mcp_auth_header,
- mcp_servers=mcp_servers,
- mcp_server_auth_headers=mcp_server_auth_headers,
- )
+ try:
+ # Get user authentication from context variable
+ (
+ user_api_key_auth,
+ mcp_auth_header,
+ mcp_servers,
+ mcp_server_auth_headers,
+ oauth2_headers,
+ raw_headers,
+ ) = get_auth_context()
+ verbose_logger.debug(
+ f"MCP list_tools - User API Key Auth from context: {user_api_key_auth}"
+ )
+ verbose_logger.debug(
+ f"MCP list_tools - MCP servers from context: {mcp_servers}"
+ )
+ verbose_logger.debug(
+ f"MCP list_tools - MCP server auth headers: {list(mcp_server_auth_headers.keys()) if mcp_server_auth_headers else None}"
+ )
+ # Get mcp_servers from context variable
+ verbose_logger.debug("MCP list_tools - Calling _list_mcp_tools")
+ tools = await _list_mcp_tools(
+ user_api_key_auth=user_api_key_auth,
+ mcp_auth_header=mcp_auth_header,
+ mcp_servers=mcp_servers,
+ mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
+ )
+ verbose_logger.info(
+ f"MCP list_tools - Successfully returned {len(tools)} tools"
+ )
+ return tools
+ except Exception as e:
+ verbose_logger.exception(f"Error in list_tools endpoint: {str(e)}")
+ # Return empty list instead of failing completely
+ # This prevents the HTTP stream from failing and allows the client to get a response
+ return []
@server.call_tool()
async def mcp_server_tool_call(
@@ -206,11 +230,19 @@ if MCP_AVAILABLE:
"""
from fastapi import Request
+ from litellm.exceptions import BlockedPiiEntityError, GuardrailRaisedException
from litellm.proxy.litellm_pre_call_utils import add_litellm_data_to_request
from litellm.proxy.proxy_server import proxy_config
# Validate arguments
- user_api_key_auth, mcp_auth_header, _, mcp_server_auth_headers = get_auth_context()
+ (
+ user_api_key_auth,
+ mcp_auth_header,
+ _,
+ mcp_server_auth_headers,
+ oauth2_headers,
+ raw_headers,
+ ) = get_auth_context()
verbose_logger.debug(
f"MCP mcp_server_tool_call - User API Key Auth from context: {user_api_key_auth}"
@@ -241,11 +273,32 @@ if MCP_AVAILABLE:
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
**data, # for logging
)
+ except BlockedPiiEntityError as e:
+ verbose_logger.error(f"BlockedPiiEntityError in MCP tool call: {str(e)}")
+ # Return error as text content for MCP protocol
+ return [
+ TextContent(
+ text=f"Error: Blocked PII entity detected - {str(e)}", type="text"
+ )
+ ]
+ except GuardrailRaisedException as e:
+ verbose_logger.error(f"GuardrailRaisedException in MCP tool call: {str(e)}")
+ # Return error as text content for MCP protocol
+ return [
+ TextContent(text=f"Error: Guardrail violation - {str(e)}", type="text")
+ ]
+ except HTTPException as e:
+ verbose_logger.error(f"HTTPException in MCP tool call: {str(e)}")
+ # Return error as text content for MCP protocol
+ return [TextContent(text=f"Error: {str(e.detail)}", type="text")]
except Exception as e:
verbose_logger.exception(f"MCP mcp_server_tool_call - error: {e}")
- raise e
+ # Return error as text content for MCP protocol
+ return [TextContent(text=f"Error: {str(e)}", type="text")]
return response
@@ -257,11 +310,128 @@ if MCP_AVAILABLE:
############ Helper Functions ##########################
########################################################
+ async def _get_allowed_mcp_servers_from_mcp_server_names(
+ mcp_servers: Optional[List[str]],
+ allowed_mcp_servers: List[str],
+ ) -> List[str]:
+ """
+ Get the filtered MCP servers from the MCP server names
+ """
+ from typing import Set
+
+ filtered_server_ids: Set[str] = set()
+ # Filter servers based on mcp_servers parameter if provided
+ if mcp_servers is not None:
+ for server_or_group in mcp_servers:
+ server_name_matched = False
+
+ for server_id in allowed_mcp_servers:
+ server = global_mcp_server_manager.get_mcp_server_by_id(server_id)
+
+ if server:
+ match_list = [
+ s.lower()
+ for s in [server.alias, server.server_name, server_id]
+ if s is not None
+ ]
+
+ if server_or_group.lower() in match_list:
+ filtered_server_ids.add(server_id)
+ server_name_matched = True
+ break
+
+ if not server_name_matched:
+ try:
+ access_group_server_ids = (
+ await MCPRequestHandler._get_mcp_servers_from_access_groups(
+ [server_or_group]
+ )
+ )
+ # Only include servers that the user has access to
+ for server_id in access_group_server_ids:
+ if server_id in allowed_mcp_servers:
+ filtered_server_ids.add(server_id)
+ except Exception as e:
+ verbose_logger.debug(
+ f"Could not resolve '{server_or_group}' as access group: {e}"
+ )
+
+ if filtered_server_ids:
+ allowed_mcp_servers = list(filtered_server_ids)
+
+ return allowed_mcp_servers
+
+ def _tool_name_matches(tool_name: str, filter_list: List[str]) -> bool:
+ """
+ Check if a tool name matches any name in the filter list.
+
+ Checks both the full tool name and unprefixed version (without server prefix).
+ This allows users to configure simple tool names regardless of prefixing.
+
+ Args:
+ tool_name: The tool name to check (may be prefixed like "server-tool_name")
+ filter_list: List of tool names to match against
+
+ Returns:
+ True if the tool name (prefixed or unprefixed) is in the filter list
+ """
+ from litellm.proxy._experimental.mcp_server.utils import (
+ get_server_name_prefix_tool_mcp,
+ )
+
+ # Check if the full name is in the list
+ if tool_name in filter_list:
+ return True
+
+ # Check if the unprefixed name is in the list
+ unprefixed_name, _ = get_server_name_prefix_tool_mcp(tool_name)
+ return unprefixed_name in filter_list
+
+ def filter_tools_by_allowed_tools(
+ tools: List[MCPTool],
+ mcp_server: MCPServer,
+ ) -> List[MCPTool]:
+ """
+ Filter tools by allowed/disallowed tools configuration.
+
+ If allowed_tools is set, only tools in that list are returned.
+ If disallowed_tools is set, tools in that list are excluded.
+ Tool names are matched with and without server prefixes for flexibility.
+
+ Args:
+ tools: List of tools to filter
+ mcp_server: Server configuration with allowed_tools/disallowed_tools
+
+ Returns:
+ Filtered list of tools
+ """
+ tools_to_return = tools
+
+ # Filter by allowed_tools (whitelist)
+ if mcp_server.allowed_tools:
+ tools_to_return = [
+ tool
+ for tool in tools
+ if _tool_name_matches(tool.name, mcp_server.allowed_tools)
+ ]
+
+ # Filter by disallowed_tools (blacklist)
+ if mcp_server.disallowed_tools:
+ tools_to_return = [
+ tool
+ for tool in tools_to_return
+ if not _tool_name_matches(tool.name, mcp_server.disallowed_tools)
+ ]
+
+ return tools_to_return
+
async def _get_tools_from_mcp_servers(
user_api_key_auth: Optional[UserAPIKeyAuth],
mcp_auth_header: Optional[str],
mcp_servers: Optional[List[str]],
- mcp_server_auth_headers: Optional[Dict[str, str]] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
+ oauth2_headers: Optional[Dict[str, str]] = None,
+ raw_headers: Optional[Dict[str, str]] = None,
) -> List[MCPTool]:
"""
Helper method to fetch tools from MCP servers based on server filtering criteria.
@@ -270,7 +440,8 @@ if MCP_AVAILABLE:
user_api_key_auth: User authentication info for access control
mcp_auth_header: Optional auth header for MCP server (deprecated)
mcp_servers: Optional list of server names/aliases to filter by
- mcp_server_auth_headers: Optional dict of server-specific auth headers {server_alias: auth_value}
+ mcp_server_auth_headers: Optional dict of server-specific auth headers
+ oauth2_headers: Optional dict of oauth2 headers
Returns:
List[MCPTool]: Combined list of tools from filtered servers
@@ -283,25 +454,14 @@ if MCP_AVAILABLE:
user_api_key_auth
)
- # Filter servers based on mcp_servers parameter if provided
if mcp_servers is not None:
- # Convert to lowercase for case-insensitive comparison
- mcp_servers_lower = [s.lower() for s in mcp_servers]
- allowed_mcp_servers = [
- server_id
- for server_id in allowed_mcp_servers
- if any(
- server_alias.lower() in mcp_servers_lower
- for server in [global_mcp_server_manager.get_mcp_server_by_id(server_id)]
- if server is not None
- for server_alias in [
- server.alias,
- server.server_name,
- server_id,
- ]
- if server_alias is not None
- )
- ]
+ allowed_mcp_servers = await _get_allowed_mcp_servers_from_mcp_server_names(
+ mcp_servers=mcp_servers,
+ allowed_mcp_servers=allowed_mcp_servers,
+ )
+
+ # Decide whether to add prefix based on number of allowed servers
+ add_prefix = not (len(allowed_mcp_servers) == 1)
# Get tools from each allowed server
all_tools = []
@@ -311,12 +471,23 @@ if MCP_AVAILABLE:
continue
# Get server-specific auth header if available
- server_auth_header = None
+ server_auth_header: Optional[Union[Dict[str, str], str]] = None
if mcp_server_auth_headers and server.alias is not None:
server_auth_header = mcp_server_auth_headers.get(server.alias)
elif mcp_server_auth_headers and server.server_name is not None:
server_auth_header = mcp_server_auth_headers.get(server.server_name)
-
+
+ extra_headers: Optional[Dict[str, str]] = None
+ if server.auth_type == MCPAuth.oauth2:
+ extra_headers = oauth2_headers
+
+ if server.extra_headers and raw_headers:
+ if extra_headers is None:
+ extra_headers = {}
+ for header in server.extra_headers:
+ if header in raw_headers:
+ extra_headers[header] = raw_headers[header]
+
# Fall back to deprecated mcp_auth_header if no server-specific header found
if server_auth_header is None:
server_auth_header = mcp_auth_header
@@ -325,23 +496,74 @@ if MCP_AVAILABLE:
tools = await global_mcp_server_manager._get_tools_from_server(
server=server,
mcp_auth_header=server_auth_header,
+ extra_headers=extra_headers,
+ add_prefix=add_prefix,
+ )
+
+ filtered_tools = filter_tools_by_allowed_tools(tools, server)
+
+ filtered_tools = await filter_tools_by_key_team_permissions(
+ tools=filtered_tools,
+ server_id=server_id,
+ user_api_key_auth=user_api_key_auth,
+ )
+
+ all_tools.extend(filtered_tools)
+
+ verbose_logger.debug(
+ f"Successfully fetched {len(tools)} tools from server {server.name}, {len(filtered_tools)} after filtering"
)
- all_tools.extend(tools)
- verbose_logger.debug(f"Successfully fetched {len(tools)} tools from server {server.name}")
except Exception as e:
verbose_logger.exception(
f"Error getting tools from server {server.name}: {str(e)}"
)
# Continue with other servers instead of failing completely
- verbose_logger.info(f"Successfully fetched {len(all_tools)} tools total from all MCP servers")
+ verbose_logger.info(
+ f"Successfully fetched {len(all_tools)} tools total from all MCP servers"
+ )
+
return all_tools
+ async def filter_tools_by_key_team_permissions(
+ tools: List[MCPTool],
+ server_id: str,
+ user_api_key_auth: Optional[UserAPIKeyAuth],
+ ) -> List[MCPTool]:
+ """
+ Filter tools based on key/team mcp_tool_permissions.
+
+ Note: Tool names in the DB are stored without server prefixes,
+ but tool names from MCP servers are prefixed. We need to strip
+ the prefix before comparing.
+ """
+ # Filter by key/team tool-level permissions
+ allowed_tool_names = await MCPRequestHandler.get_allowed_tools_for_server(
+ server_id=server_id,
+ user_api_key_auth=user_api_key_auth,
+ )
+ if allowed_tool_names is not None:
+ # Strip prefix from tool names before comparing
+ # Tools are stored in DB without prefix, but come from MCP server with prefix
+ filtered_tools = []
+ for t in tools:
+ # Get tool name without server prefix
+ unprefixed_tool_name, _ = get_server_name_prefix_tool_mcp(t.name)
+ if unprefixed_tool_name in allowed_tool_names:
+ filtered_tools.append(t)
+ else:
+ # No restrictions, return all tools
+ filtered_tools = tools
+
+ return filtered_tools
+
async def _list_mcp_tools(
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
mcp_auth_header: Optional[str] = None,
mcp_servers: Optional[List[str]] = None,
- mcp_server_auth_headers: Optional[Dict[str, str]] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
+ oauth2_headers: Optional[Dict[str, str]] = None,
+ raw_headers: Optional[Dict[str, str]] = None,
) -> List[MCPTool]:
"""
List all available MCP tools.
@@ -357,42 +579,38 @@ if MCP_AVAILABLE:
"""
if not MCP_AVAILABLE:
return []
-
- # Get tools from managed MCP servers
- managed_tools = await _get_tools_from_mcp_servers(
- user_api_key_auth=user_api_key_auth,
- mcp_auth_header=mcp_auth_header,
- mcp_servers=mcp_servers,
- mcp_server_auth_headers=mcp_server_auth_headers,
- )
-
- # Get tools from local registry
- local_tools_raw = global_mcp_tool_registry.list_tools()
-
- # Convert local tools to MCPTool format
- local_tools = []
- for tool in local_tools_raw:
- # Convert from litellm.types.mcp_server.tool_registry.MCPTool to mcp.types.Tool
- mcp_tool = MCPTool(
- name=tool.name,
- description=tool.description,
- inputSchema=tool.input_schema
+ # Get tools from managed MCP servers with error handling
+ managed_tools = []
+ try:
+ managed_tools = await _get_tools_from_mcp_servers(
+ user_api_key_auth=user_api_key_auth,
+ mcp_auth_header=mcp_auth_header,
+ mcp_servers=mcp_servers,
+ mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
)
- local_tools.append(mcp_tool)
+ verbose_logger.debug(
+ f"Successfully fetched {len(managed_tools)} tools from managed MCP servers"
+ )
+ except Exception as e:
+ verbose_logger.exception(
+ f"Error getting tools from managed MCP servers: {str(e)}"
+ )
+ # Continue with empty managed tools list instead of failing completely
- # Combine all tools
- all_tools = managed_tools + local_tools
-
- return all_tools
+ return managed_tools
@client
async def call_mcp_tool(
- name: str,
- arguments: Optional[Dict[str, Any]] = None,
- user_api_key_auth: Optional[UserAPIKeyAuth] = None,
- mcp_auth_header: Optional[str] = None,
- mcp_server_auth_headers: Optional[Dict[str, str]] = None,
- **kwargs: Any
+ name: str,
+ arguments: Optional[Dict[str, Any]] = None,
+ user_api_key_auth: Optional[UserAPIKeyAuth] = None,
+ mcp_auth_header: Optional[str] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
+ oauth2_headers: Optional[Dict[str, str]] = None,
+ raw_headers: Optional[Dict[str, str]] = None,
+ **kwargs: Any,
) -> List[Union[TextContent, ImageContent, EmbeddedResource]]:
"""
Call a specific tool with the provided arguments (handles prefixed tool names)
@@ -408,6 +626,25 @@ if MCP_AVAILABLE:
name
)
+ ## CHECK IF USER IS ALLOWED TO CALL THIS TOOL
+ allowed_mcp_server_ids = await MCPRequestHandler.get_allowed_mcp_servers(
+ user_api_key_auth=user_api_key_auth,
+ )
+
+ allowed_mcp_servers = global_mcp_server_manager.get_mcp_server_names_from_ids(
+ allowed_mcp_server_ids
+ )
+
+ if not MCPRequestHandler.is_tool_allowed(
+ allowed_mcp_servers=allowed_mcp_servers,
+ server_name=server_name_from_prefix,
+ ):
+
+ raise HTTPException(
+ status_code=403,
+ detail=f"User not allowed to call this tool. Allowed MCP servers: {allowed_mcp_servers}",
+ )
+
standard_logging_mcp_tool_call: StandardLoggingMCPToolCall = (
_get_standard_logging_mcp_tool_call(
name=original_tool_name, # Use original name for logging
@@ -423,31 +660,43 @@ if MCP_AVAILABLE:
standard_logging_mcp_tool_call
)
litellm_logging_obj.model = f"MCP: {name}"
- # Try managed server tool first (pass the full prefixed name)
- # Primary and recommended way to use MCP servers
+ # Check if tool exists in local registry first (for OpenAPI-based tools)
+ # These tools are registered with their prefixed names
#########################################################
- mcp_server: Optional[MCPServer] = (
- global_mcp_server_manager._get_mcp_server_from_tool_name(name)
- )
- if mcp_server:
- standard_logging_mcp_tool_call["mcp_server_cost_info"] = (
- mcp_server.mcp_info or {}
- ).get("mcp_server_cost_info")
- response = await _handle_managed_mcp_tool(
- name=name, # Pass the full name (potentially prefixed)
- arguments=arguments,
- user_api_key_auth=user_api_key_auth,
- mcp_auth_header=mcp_auth_header,
- mcp_server_auth_headers=mcp_server_auth_headers,
- )
+ local_tool = global_mcp_tool_registry.get_tool(name)
+ if local_tool:
+ verbose_logger.debug(f"Executing local registry tool: {name}")
+ response = await _handle_local_mcp_tool(name, arguments)
- # Fall back to local tool registry (use original name)
- #########################################################
- # Deprecated: Local MCP Server Tool
+ # Try managed MCP server tool (pass the full prefixed name)
+ # Primary and recommended way to use external MCP servers
#########################################################
else:
- response = await _handle_local_mcp_tool(original_tool_name, arguments)
-
+ mcp_server: Optional[MCPServer] = (
+ global_mcp_server_manager._get_mcp_server_from_tool_name(name)
+ )
+ if mcp_server:
+ standard_logging_mcp_tool_call["mcp_server_cost_info"] = (
+ mcp_server.mcp_info or {}
+ ).get("mcp_server_cost_info")
+ response = await _handle_managed_mcp_tool(
+ name=name, # Pass the full name (potentially prefixed)
+ arguments=arguments,
+ user_api_key_auth=user_api_key_auth,
+ mcp_auth_header=mcp_auth_header,
+ mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
+ litellm_logging_obj=litellm_logging_obj,
+ )
+
+ # Fall back to local tool registry with original name (legacy support)
+ #########################################################
+ # Deprecated: Local MCP Server Tool
+ #########################################################
+ else:
+ response = await _handle_local_mcp_tool(original_tool_name, arguments)
+
#########################################################
# Post MCP Tool Call Hook
# Allow modifying the MCP tool call response before it is returned to the user
@@ -489,15 +738,24 @@ if MCP_AVAILABLE:
arguments: Dict[str, Any],
user_api_key_auth: Optional[UserAPIKeyAuth] = None,
mcp_auth_header: Optional[str] = None,
- mcp_server_auth_headers: Optional[Dict[str, str]] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
+ oauth2_headers: Optional[Dict[str, str]] = None,
+ raw_headers: Optional[Dict[str, str]] = None,
+ litellm_logging_obj: Optional[Any] = None,
) -> List[Union[TextContent, ImageContent, EmbeddedResource]]:
"""Handle tool execution for managed server tools"""
+ # Import here to avoid circular import
+ from litellm.proxy.proxy_server import proxy_logging_obj
+
call_tool_result = await global_mcp_server_manager.call_tool(
name=name,
arguments=arguments,
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
+ proxy_logging_obj=proxy_logging_obj,
)
verbose_logger.debug("CALL TOOL RESULT: %s", call_tool_result)
return call_tool_result.content # type: ignore[return-value]
@@ -509,39 +767,102 @@ if MCP_AVAILABLE:
Handle tool execution for local registry tools
Note: Local tools don't use prefixes, so we use the original name
"""
+ import inspect
+
tool = global_mcp_tool_registry.get_tool(name)
if not tool:
raise HTTPException(status_code=404, detail=f"Tool '{name}' not found")
try:
- result = tool.handler(**arguments)
+ # Check if handler is async or sync
+ if inspect.iscoroutinefunction(tool.handler):
+ result = await tool.handler(**arguments)
+ else:
+ result = tool.handler(**arguments)
return [TextContent(text=str(result), type="text")]
except Exception as e:
+ verbose_logger.exception(f"Error executing local tool {name}: {str(e)}")
return [TextContent(text=f"Error: {str(e)}", type="text")]
+ def _get_mcp_servers_in_path(path: str) -> Optional[List[str]]:
+ """
+ Get the MCP servers from the path
+ """
+ import re
+
+ mcp_servers_from_path: Optional[List[str]] = None
+ # Match /mcp/
+ # Where servers can be comma-separated list of server names
+ # Server names can contain slashes (e.g., "custom_solutions/user_123")
+ mcp_path_match = re.match(r"^/mcp/([^?#]+)(?:\?.*)?(?:#.*)?$", path)
+ if mcp_path_match:
+ servers_and_path = mcp_path_match.group(1)
+
+ if servers_and_path:
+ # Check if it contains commas (comma-separated servers)
+ if "," in servers_and_path:
+ # For comma-separated, look for a path at the end
+ # Common patterns: /tools, /chat/completions, etc.
+ path_match = re.search(r"/([^/,]+(?:/[^/,]+)*)$", servers_and_path)
+ if path_match:
+ # Path found at the end, remove it from servers
+ path_part = "/" + path_match.group(1)
+ servers_part = servers_and_path[: -len(path_part)]
+ mcp_servers_from_path = [
+ s.strip() for s in servers_part.split(",") if s.strip()
+ ]
+ else:
+ # No path, just comma-separated servers
+ mcp_servers_from_path = [
+ s.strip() for s in servers_and_path.split(",") if s.strip()
+ ]
+ else:
+ # Single server case - use regex approach for server/path separation
+ # This handles cases like "custom_solutions/user_123/chat/completions"
+ # where we want to extract "custom_solutions/user_123" as the server name
+ single_server_match = re.match(
+ r"^([^/]+(?:/[^/]+)?)(?:/.*)?$", servers_and_path
+ )
+ if single_server_match:
+ server_name = single_server_match.group(1)
+ mcp_servers_from_path = [server_name]
+ else:
+ mcp_servers_from_path = [servers_and_path]
+ return mcp_servers_from_path
+
async def extract_mcp_auth_context(scope, path):
"""
Extracts mcp_servers from the path and processes the MCP request for auth context.
Returns: (user_api_key_auth, mcp_auth_header, mcp_servers, mcp_server_auth_headers)
"""
- import re
- mcp_servers_from_path = None
- mcp_path_match = re.match(r"^/mcp/([^/]+)(/.*)?$", path)
- if mcp_path_match:
- mcp_servers_str = mcp_path_match.group(1)
- if mcp_servers_str:
- mcp_servers_from_path = [s.strip() for s in mcp_servers_str.split(",") if s.strip()]
-
+ mcp_servers_from_path = _get_mcp_servers_in_path(path)
if mcp_servers_from_path is not None:
- user_api_key_auth, mcp_auth_header, _, mcp_server_auth_headers = (
- await MCPRequestHandler.process_mcp_request(scope)
- )
+ (
+ user_api_key_auth,
+ mcp_auth_header,
+ _,
+ mcp_server_auth_headers,
+ oauth2_headers,
+ raw_headers,
+ ) = await MCPRequestHandler.process_mcp_request(scope)
mcp_servers = mcp_servers_from_path
else:
- user_api_key_auth, mcp_auth_header, mcp_servers, mcp_server_auth_headers = (
- await MCPRequestHandler.process_mcp_request(scope)
- )
- return user_api_key_auth, mcp_auth_header, mcp_servers, mcp_server_auth_headers
+ (
+ user_api_key_auth,
+ mcp_auth_header,
+ mcp_servers,
+ mcp_server_auth_headers,
+ oauth2_headers,
+ raw_headers,
+ ) = await MCPRequestHandler.process_mcp_request(scope)
+ return (
+ user_api_key_auth,
+ mcp_auth_header,
+ mcp_servers,
+ mcp_server_auth_headers,
+ oauth2_headers,
+ raw_headers,
+ )
async def handle_streamable_http_mcp(
scope: Scope, receive: Receive, send: Send
@@ -549,15 +870,28 @@ if MCP_AVAILABLE:
"""Handle MCP requests through StreamableHTTP."""
try:
path = scope.get("path", "")
- user_api_key_auth, mcp_auth_header, mcp_servers, mcp_server_auth_headers = await extract_mcp_auth_context(scope, path)
- verbose_logger.debug(f"MCP request mcp_servers (header/path): {mcp_servers}")
- verbose_logger.debug(f"MCP server auth headers: {list(mcp_server_auth_headers.keys()) if mcp_server_auth_headers else None}")
+ (
+ user_api_key_auth,
+ mcp_auth_header,
+ mcp_servers,
+ mcp_server_auth_headers,
+ oauth2_headers,
+ raw_headers,
+ ) = await extract_mcp_auth_context(scope, path)
+ verbose_logger.debug(
+ f"MCP request mcp_servers (header/path): {mcp_servers}"
+ )
+ verbose_logger.debug(
+ f"MCP server auth headers: {list(mcp_server_auth_headers.keys()) if mcp_server_auth_headers else None}"
+ )
# Set the auth context variable for easy access in MCP functions
set_auth_context(
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_servers=mcp_servers,
mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
)
# Ensure session managers are initialized
@@ -568,21 +902,51 @@ if MCP_AVAILABLE:
await session_manager.handle_request(scope, receive, send)
except Exception as e:
- verbose_logger.exception(f"Error handling MCP request: {e}")
raise e
+ verbose_logger.exception(f"Error handling MCP request: {e}")
+ # Instead of re-raising, try to send a graceful error response
+ try:
+ # Send a proper HTTP error response instead of letting the exception bubble up
+ from starlette.responses import JSONResponse
+ from starlette.status import HTTP_500_INTERNAL_SERVER_ERROR
+
+ error_response = JSONResponse(
+ status_code=HTTP_500_INTERNAL_SERVER_ERROR,
+ content={"error": "MCP request failed", "details": str(e)},
+ )
+ await error_response(scope, receive, send)
+ except Exception as response_error:
+ verbose_logger.exception(
+ f"Failed to send error response: {response_error}"
+ )
+ # If we can't send a proper response, re-raise the original error
+ raise e
async def handle_sse_mcp(scope: Scope, receive: Receive, send: Send) -> None:
"""Handle MCP requests through SSE."""
try:
path = scope.get("path", "")
- user_api_key_auth, mcp_auth_header, mcp_servers, mcp_server_auth_headers = await extract_mcp_auth_context(scope, path)
- verbose_logger.debug(f"MCP request mcp_servers (header/path): {mcp_servers}")
- verbose_logger.debug(f"MCP server auth headers: {list(mcp_server_auth_headers.keys()) if mcp_server_auth_headers else None}")
+ (
+ user_api_key_auth,
+ mcp_auth_header,
+ mcp_servers,
+ mcp_server_auth_headers,
+ oauth2_headers,
+ raw_headers,
+ ) = await extract_mcp_auth_context(scope, path)
+ verbose_logger.debug(
+ f"MCP request mcp_servers (header/path): {mcp_servers}"
+ )
+ verbose_logger.debug(
+ f"MCP server auth headers: {list(mcp_server_auth_headers.keys()) if mcp_server_auth_headers else None}"
+ )
set_auth_context(
user_api_key_auth=user_api_key_auth,
mcp_auth_header=mcp_auth_header,
mcp_servers=mcp_servers,
mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
)
if not _SESSION_MANAGERS_INITIALIZED:
@@ -592,7 +956,23 @@ if MCP_AVAILABLE:
await sse_session_manager.handle_request(scope, receive, send)
except Exception as e:
verbose_logger.exception(f"Error handling MCP request: {e}")
- raise e
+ # Instead of re-raising, try to send a graceful error response
+ try:
+ # Send a proper HTTP error response instead of letting the exception bubble up
+ from starlette.responses import JSONResponse
+ from starlette.status import HTTP_500_INTERNAL_SERVER_ERROR
+
+ error_response = JSONResponse(
+ status_code=HTTP_500_INTERNAL_SERVER_ERROR,
+ content={"error": "MCP request failed", "details": str(e)},
+ )
+ await error_response(scope, receive, send)
+ except Exception as response_error:
+ verbose_logger.exception(
+ f"Failed to send error response: {response_error}"
+ )
+ # If we can't send a proper response, re-raise the original error
+ raise e
app = FastAPI(
title=LITELLM_MCP_SERVER_NAME,
@@ -614,6 +994,8 @@ if MCP_AVAILABLE:
# Mount the MCP handlers
app.mount("/", handle_streamable_http_mcp)
+ app.mount("/mcp", handle_streamable_http_mcp)
+ app.mount("/{mcp_server_name}/mcp", handle_streamable_http_mcp)
app.mount("/sse", handle_sse_mcp)
app.add_middleware(AuthContextMiddleware)
@@ -625,7 +1007,9 @@ if MCP_AVAILABLE:
user_api_key_auth: UserAPIKeyAuth,
mcp_auth_header: Optional[str] = None,
mcp_servers: Optional[List[str]] = None,
- mcp_server_auth_headers: Optional[Dict[str, str]] = None,
+ mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]] = None,
+ oauth2_headers: Optional[Dict[str, str]] = None,
+ raw_headers: Optional[Dict[str, str]] = None,
) -> None:
"""
Set the UserAPIKeyAuth in the auth context variable.
@@ -641,17 +1025,24 @@ if MCP_AVAILABLE:
mcp_auth_header=mcp_auth_header,
mcp_servers=mcp_servers,
mcp_server_auth_headers=mcp_server_auth_headers,
+ oauth2_headers=oauth2_headers,
+ raw_headers=raw_headers,
)
auth_context_var.set(auth_user)
- def get_auth_context() -> (
- Tuple[Optional[UserAPIKeyAuth], Optional[str], Optional[List[str]], Optional[Dict[str, str]]]
- ):
+ def get_auth_context() -> Tuple[
+ Optional[UserAPIKeyAuth],
+ Optional[str],
+ Optional[List[str]],
+ Optional[Dict[str, Dict[str, str]]],
+ Optional[Dict[str, str]],
+ Optional[Dict[str, str]],
+ ]:
"""
Get the UserAPIKeyAuth from the auth context variable.
Returns:
- Tuple[Optional[UserAPIKeyAuth], Optional[str], Optional[List[str]], Optional[Dict[str, str]]]:
+ Tuple[Optional[UserAPIKeyAuth], Optional[str], Optional[List[str]], Optional[Dict[str, str]]]:
UserAPIKeyAuth object, MCP auth header (deprecated), MCP servers (can include access groups), and server-specific auth headers
"""
auth_user = auth_context_var.get()
@@ -661,8 +1052,10 @@ if MCP_AVAILABLE:
auth_user.mcp_auth_header,
auth_user.mcp_servers,
auth_user.mcp_server_auth_headers,
+ auth_user.oauth2_headers,
+ auth_user.raw_headers,
)
- return None, None, None, None
+ return None, None, None, None, None, None
########################################################
############ End of Auth Context Functions #############
diff --git a/litellm/proxy/_experimental/mcp_server/tool_registry.py b/litellm/proxy/_experimental/mcp_server/tool_registry.py
index c08b7979683..58570aafadf 100644
--- a/litellm/proxy/_experimental/mcp_server/tool_registry.py
+++ b/litellm/proxy/_experimental/mcp_server/tool_registry.py
@@ -1,10 +1,18 @@
import json
-from typing import Any, Callable, Dict, List, Optional
+from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional
from litellm._logging import verbose_logger
from litellm.proxy.types_utils.utils import get_instance_fn
from litellm.types.mcp_server.tool_registry import MCPTool
+if TYPE_CHECKING:
+ from mcp.types import Tool as MCPToolSDKTool
+else:
+ try:
+ from mcp.types import Tool as MCPToolSDKTool
+ except ImportError:
+ MCPToolSDKTool = None # type: ignore
+
class MCPToolRegistry:
"""
@@ -39,12 +47,34 @@ class MCPToolRegistry:
"""
return self.tools.get(name)
- def list_tools(self) -> List[MCPTool]:
+ def list_tools(self, tool_prefix: Optional[str] = None) -> List[MCPTool]:
"""
List all registered tools
"""
+ if tool_prefix:
+ return [
+ tool
+ for tool in self.tools.values()
+ if tool.name.startswith(tool_prefix)
+ ]
return list(self.tools.values())
+ def convert_tools_to_mcp_sdk_tool_type(
+ self, tools: List[MCPTool]
+ ) -> List["MCPToolSDKTool"]:
+ if MCPToolSDKTool is None:
+ raise ImportError(
+ "MCP SDK is not installed. Please install it with: pip install 'litellm[proxy]'"
+ )
+ return [
+ MCPToolSDKTool(
+ name=tool.name,
+ description=tool.description,
+ inputSchema=tool.input_schema,
+ )
+ for tool in tools
+ ]
+
def load_tools_from_config(
self, mcp_tools_config: Optional[Dict[str, Any]] = None
) -> None:
diff --git a/litellm/proxy/_experimental/out/_next/static/N-wLM4VjZ1bVvg2NAx9GX/_buildManifest.js b/litellm/proxy/_experimental/out/_next/static/N-wLM4VjZ1bVvg2NAx9GX/_buildManifest.js
deleted file mode 100644
index 96ded068de5..00000000000
--- a/litellm/proxy/_experimental/out/_next/static/N-wLM4VjZ1bVvg2NAx9GX/_buildManifest.js
+++ /dev/null
@@ -1 +0,0 @@
-self.__BUILD_MANIFEST={__rewrites:{afterFiles:[],beforeFiles:[],fallback:[]},"/_error":["static/chunks/pages/_error-28b803cb2479b966.js"],sortedPages:["/_app","/_error"]},self.__BUILD_MANIFEST_CB&&self.__BUILD_MANIFEST_CB();
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/ZAqshlHWdpwZy_2QG1xUf/_buildManifest.js b/litellm/proxy/_experimental/out/_next/static/ZAqshlHWdpwZy_2QG1xUf/_buildManifest.js
new file mode 100644
index 00000000000..1b732be87b0
--- /dev/null
+++ b/litellm/proxy/_experimental/out/_next/static/ZAqshlHWdpwZy_2QG1xUf/_buildManifest.js
@@ -0,0 +1 @@
+self.__BUILD_MANIFEST={__rewrites:{afterFiles:[],beforeFiles:[],fallback:[]},"/_error":["static/chunks/pages/_error-cf5ca766ac8f493f.js"],sortedPages:["/_app","/_error"]},self.__BUILD_MANIFEST_CB&&self.__BUILD_MANIFEST_CB();
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/N-wLM4VjZ1bVvg2NAx9GX/_ssgManifest.js b/litellm/proxy/_experimental/out/_next/static/ZAqshlHWdpwZy_2QG1xUf/_ssgManifest.js
similarity index 100%
rename from litellm/proxy/_experimental/out/_next/static/N-wLM4VjZ1bVvg2NAx9GX/_ssgManifest.js
rename to litellm/proxy/_experimental/out/_next/static/ZAqshlHWdpwZy_2QG1xUf/_ssgManifest.js
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/1052-6c4e848aed27b319.js b/litellm/proxy/_experimental/out/_next/static/chunks/1052-6c4e848aed27b319.js
new file mode 100644
index 00000000000..8ab8f6ab8aa
--- /dev/null
+++ b/litellm/proxy/_experimental/out/_next/static/chunks/1052-6c4e848aed27b319.js
@@ -0,0 +1 @@
+"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[1052],{19046:function(e,t,s){s.d(t,{Dx:function(){return r.Z},Zb:function(){return a.Z},oi:function(){return l.Z},xv:function(){return n.Z},zx:function(){return o.Z}});var o=s(20831),a=s(12514),n=s(84264),l=s(49566),r=s(96761)},31052:function(e,t,s){s.d(t,{Z:function(){return eA}});var o=s(57437),a=s(2265),n=s(243),l=s(19046),r=s(93837),i=s(64482),c=s(65319),d=s(93192),m=s(52787),g=s(89970),p=s(87908),u=s(82680),x=s(73002),h=s(26433),f=s(19250);async function b(e,t,s,o,a,n,l,r,i,c,d,m,g,p){console.log=function(){},console.log("isLocal:",!1);let u=(0,f.getProxyBaseUrl)(),x={};a&&a.length>0&&(x["x-litellm-tags"]=a.join(","));let b=new h.ZP.OpenAI({apiKey:o,baseURL:u,dangerouslyAllowBrowser:!0,defaultHeaders:x});try{let a;let x=Date.now(),h=!1,f=g&&g.length>0?[{type:"mcp",server_label:"litellm",server_url:"".concat(u,"/mcp"),require_approval:"never",allowed_tools:g,headers:{"x-litellm-api-key":"Bearer ".concat(o)}}]:void 0;for await(let o of(await b.chat.completions.create({model:s,stream:!0,stream_options:{include_usage:!0},litellm_trace_id:c,messages:e,...d?{vector_store_ids:d}:{},...m?{guardrails:m}:{},...f?{tools:f,tool_choice:"auto"}:{}},{signal:n}))){var v,y,j,w,S,N,k,P;console.log("Stream chunk:",o);let e=null===(v=o.choices[0])||void 0===v?void 0:v.delta;if(console.log("Delta content:",null===(j=o.choices[0])||void 0===j?void 0:null===(y=j.delta)||void 0===y?void 0:y.content),console.log("Delta reasoning content:",null==e?void 0:e.reasoning_content),!h&&((null===(S=o.choices[0])||void 0===S?void 0:null===(w=S.delta)||void 0===w?void 0:w.content)||e&&e.reasoning_content)&&(h=!0,a=Date.now()-x,console.log("First token received! Time:",a,"ms"),r?(console.log("Calling onTimingData with:",a),r(a)):console.log("onTimingData callback is not defined!")),null===(k=o.choices[0])||void 0===k?void 0:null===(N=k.delta)||void 0===N?void 0:N.content){let e=o.choices[0].delta.content;t(e,o.model)}if(e&&e.image&&p&&(console.log("Image generated:",e.image),p(e.image.url,o.model)),e&&e.reasoning_content){let t=e.reasoning_content;l&&l(t)}if(o.usage&&i){console.log("Usage data found:",o.usage);let e={completionTokens:o.usage.completion_tokens,promptTokens:o.usage.prompt_tokens,totalTokens:o.usage.total_tokens};(null===(P=o.usage.completion_tokens_details)||void 0===P?void 0:P.reasoning_tokens)&&(e.reasoningTokens=o.usage.completion_tokens_details.reasoning_tokens),i(e)}}}catch(e){throw(null==n?void 0:n.aborted)&&console.log("Chat completion request was cancelled"),e}}var v=s(9114);async function y(e,t,s,o,a,n){console.log=function(){},console.log("isLocal:",!1);let l=(0,f.getProxyBaseUrl)(),r=new h.ZP.OpenAI({apiKey:o,baseURL:l,dangerouslyAllowBrowser:!0,defaultHeaders:a&&a.length>0?{"x-litellm-tags":a.join(",")}:void 0});try{let o=await r.images.generate({model:s,prompt:e},{signal:n});if(console.log(o.data),o.data&&o.data[0]){if(o.data[0].url)t(o.data[0].url,s);else if(o.data[0].b64_json){let e=o.data[0].b64_json;t("data:image/png;base64,".concat(e),s)}else throw Error("No image data found in response")}else throw Error("Invalid response format")}catch(e){throw(null==n?void 0:n.aborted)?console.log("Image generation request was cancelled"):v.Z.fromBackend("Error occurred while generating image. Please try again. Error: ".concat(e)),e}}async function j(e,t,s,o,a,n,l){console.log=function(){},console.log("isLocal:",!1);let r=(0,f.getProxyBaseUrl)(),i=new h.ZP.OpenAI({apiKey:a,baseURL:r,dangerouslyAllowBrowser:!0,defaultHeaders:n&&n.length>0?{"x-litellm-tags":n.join(",")}:void 0});try{let a=Array.isArray(e)?e:[e],n=[];for(let e=0;e1&&v.Z.success("Successfully processed ".concat(n.length," images"))}catch(e){if(console.error("Error making image edit request:",e),null==l?void 0:l.aborted)console.log("Image edits request was cancelled");else{var c;let t="Failed to edit image(s)";(null==e?void 0:null===(c=e.error)||void 0===c?void 0:c.message)?t=e.error.message:(null==e?void 0:e.message)&&(t=e.message),v.Z.fromBackend("Image edit failed: ".concat(t))}throw e}}async function w(e,t,s,o){let a=arguments.length>4&&void 0!==arguments[4]?arguments[4]:[],n=arguments.length>5?arguments[5]:void 0,l=arguments.length>6?arguments[6]:void 0,r=arguments.length>7?arguments[7]:void 0,i=arguments.length>8?arguments[8]:void 0,c=arguments.length>9?arguments[9]:void 0,d=arguments.length>10?arguments[10]:void 0,m=arguments.length>11?arguments[11]:void 0,g=arguments.length>12?arguments[12]:void 0,p=arguments.length>13?arguments[13]:void 0,u=arguments.length>14?arguments[14]:void 0,x=arguments.length>15?arguments[15]:void 0;if(!o)throw Error("API key is required");console.log=function(){};let b=(0,f.getProxyBaseUrl)(),y={};a&&a.length>0&&(y["x-litellm-tags"]=a.join(","));let j=new h.ZP.OpenAI({apiKey:o,baseURL:b,dangerouslyAllowBrowser:!0,defaultHeaders:y});try{let o=Date.now(),a=!1,h=e.map(e=>(Array.isArray(e.content),{role:e.role,content:e.content,type:"message"})),f=g&&g.length>0?[{type:"mcp",server_label:"litellm",server_url:"litellm_proxy/mcp",require_approval:"never",allowed_tools:g}]:void 0,b=await j.responses.create({model:s,input:h,stream:!0,litellm_trace_id:c,...p?{previous_response_id:p}:{},...d?{vector_store_ids:d}:{},...m?{guardrails:m}:{},...f?{tools:f,tool_choice:"required"}:{}},{signal:n}),v="";for await(let e of b)if(console.log("Response event:",e),"object"==typeof e&&null!==e){var w,S,N,k,P,C,I;if(((null===(w=e.type)||void 0===w?void 0:w.startsWith("response.mcp_"))||"response.output_item.done"===e.type&&((null===(S=e.item)||void 0===S?void 0:S.type)==="mcp_list_tools"||(null===(N=e.item)||void 0===N?void 0:N.type)==="mcp_call"))&&(console.log("MCP event received:",e),x)){let t={type:e.type,sequence_number:e.sequence_number,output_index:e.output_index,item_id:e.item_id||(null===(C=e.item)||void 0===C?void 0:C.id),item:e.item,delta:e.delta,arguments:e.arguments,timestamp:Date.now()};x(t)}if("response.output_item.done"===e.type&&(null===(k=e.item)||void 0===k?void 0:k.type)==="mcp_call"&&(null===(P=e.item)||void 0===P?void 0:P.name)&&(v=e.item.name,console.log("MCP tool used:",v)),"response.role.delta"===e.type)continue;if("response.output_text.delta"===e.type&&"string"==typeof e.delta){let n=e.delta;if(console.log("Text delta",n),n.trim().length>0&&(t("assistant",n,s),!a)){a=!0;let e=Date.now()-o;console.log("First token received! Time:",e,"ms"),r&&r(e)}}if("response.reasoning.delta"===e.type&&"delta"in e){let t=e.delta;"string"==typeof t&&l&&l(t)}if("response.completed"===e.type&&"response"in e){let t=e.response,s=t.usage;if(console.log("Usage data:",s),console.log("Response completed event:",t),t.id&&u&&(console.log("Response ID for session management:",t.id),u(t.id)),s&&i){console.log("Usage data:",s);let e={completionTokens:s.output_tokens,promptTokens:s.input_tokens,totalTokens:s.total_tokens};(null===(I=s.completion_tokens_details)||void 0===I?void 0:I.reasoning_tokens)&&(e.reasoningTokens=s.completion_tokens_details.reasoning_tokens),i(e,v)}}}return b}catch(e){throw(null==n?void 0:n.aborted)?console.log("Responses API request was cancelled"):v.Z.fromBackend("Error occurred while generating model response. Please try again. Error: ".concat(e)),e}}var S=s(85498);async function N(e,t,s,o){let a=arguments.length>4&&void 0!==arguments[4]?arguments[4]:[],n=arguments.length>5?arguments[5]:void 0,l=arguments.length>6?arguments[6]:void 0,r=arguments.length>7?arguments[7]:void 0,i=arguments.length>8?arguments[8]:void 0,c=arguments.length>9?arguments[9]:void 0,d=arguments.length>10?arguments[10]:void 0,m=arguments.length>11?arguments[11]:void 0,g=arguments.length>12?arguments[12]:void 0;if(!o)throw Error("API key is required");console.log=function(){};let p=(0,f.getProxyBaseUrl)(),u={};a&&a.length>0&&(u["x-litellm-tags"]=a.join(","));let x=new S.ZP({apiKey:o,baseURL:p,dangerouslyAllowBrowser:!0,defaultHeaders:u});try{let a=Date.now(),u=!1,h=g&&g.length>0?[{type:"mcp",server_label:"litellm",server_url:"".concat(p,"/mcp"),require_approval:"never",allowed_tools:g,headers:{"x-litellm-api-key":"Bearer ".concat(o)}}]:void 0,f={model:s,messages:e.map(e=>({role:e.role,content:e.content})),stream:!0,max_tokens:1024,litellm_trace_id:c};for await(let e of(d&&(f.vector_store_ids=d),m&&(f.guardrails=m),h&&(f.tools=h,f.tool_choice="auto"),x.messages.stream(f,{signal:n}))){if(console.log("Stream event:",e),"content_block_delta"===e.type){let o=e.delta;if(!u){u=!0;let e=Date.now()-a;console.log("First token received! Time:",e,"ms"),r&&r(e)}"text_delta"===o.type?t("assistant",o.text,s):"reasoning_delta"===o.type&&l&&l(o.text)}if("message_delta"===e.type&&e.usage&&i){let t=e.usage;console.log("Usage data found:",t);let s={completionTokens:t.output_tokens,promptTokens:t.input_tokens,totalTokens:t.input_tokens+t.output_tokens};i(s)}}}catch(e){throw(null==n?void 0:n.aborted)?console.log("Anthropic messages request was cancelled"):v.Z.fromBackend("Error occurred while generating model response. Please try again. Error: ".concat(e)),e}}var k=s(51601);async function P(e){try{return(await (0,f.mcpToolsCall)(e)).tools||[]}catch(e){return console.error("Error fetching MCP tools:",e),[]}}var C=s(49817),I=s(17906),_=s(57365),T=s(92280),Z=e=>{let{endpointType:t,onEndpointChange:s,className:a}=e,n=[{value:C.KP.CHAT,label:"/v1/chat/completions"},{value:C.KP.RESPONSES,label:"/v1/responses"},{value:C.KP.ANTHROPIC_MESSAGES,label:"/v1/messages"},{value:C.KP.IMAGE,label:"/v1/images/generations"},{value:C.KP.IMAGE_EDITS,label:"/v1/images/edits"}];return(0,o.jsxs)("div",{className:a,children:[(0,o.jsx)(T.x,{children:"Endpoint Type:"}),(0,o.jsx)(m.default,{showSearch:!0,value:t,style:{width:"100%"},onChange:s,options:n,className:"rounded-md"})]})},E=e=>{let{onChange:t,value:s,className:n,accessToken:l}=e,[r,i]=(0,a.useState)([]),[c,d]=(0,a.useState)(!1);return(0,a.useEffect)(()=>{(async()=>{if(l)try{let e=await (0,f.tagListCall)(l);console.log("List tags response:",e),i(Object.values(e))}catch(e){console.error("Error fetching tags:",e)}})()},[]),(0,o.jsx)(m.default,{mode:"multiple",placeholder:"Select tags",onChange:t,value:s,loading:c,className:n,options:r.map(e=>({label:e.name,value:e.name,title:e.description||e.name})),optionFilterProp:"label",showSearch:!0,style:{width:"100%"}})},A=s(97415),U=s(67479),R=s(88658),L=s(83322),M=s(70464),D=s(77565),K=e=>{let{reasoningContent:t}=e,[s,l]=(0,a.useState)(!0);return t?(0,o.jsxs)("div",{className:"reasoning-content mt-1 mb-2",children:[(0,o.jsxs)(x.ZP,{type:"text",className:"flex items-center text-xs text-gray-500 hover:text-gray-700",onClick:()=>l(!s),icon:(0,o.jsx)(L.Z,{}),children:[s?"Hide reasoning":"Show reasoning",s?(0,o.jsx)(M.Z,{className:"ml-1"}):(0,o.jsx)(D.Z,{className:"ml-1"})]}),s&&(0,o.jsx)("div",{className:"mt-2 p-3 bg-gray-50 border border-gray-200 rounded-md text-sm text-gray-700",children:(0,o.jsx)(n.U,{components:{code(e){let{node:t,inline:s,className:a,children:n,...l}=e,r=/language-(\w+)/.exec(a||"");return!s&&r?(0,o.jsx)(I.Z,{style:_.Z,language:r[1],PreTag:"div",className:"rounded-md my-2",...l,children:String(n).replace(/\n$/,"")}):(0,o.jsx)("code",{className:"".concat(a," px-1.5 py-0.5 rounded bg-gray-100 text-sm font-mono"),...l,children:n})}},children:t})})]}):null},z=s(5540),O=s(71282),F=s(11741),H=s(16601),B=s(58630),G=e=>{let{timeToFirstToken:t,usage:s,toolName:a}=e;return t||s?(0,o.jsxs)("div",{className:"response-metrics mt-2 pt-2 border-t border-gray-100 text-xs text-gray-500 flex flex-wrap gap-3",children:[void 0!==t&&(0,o.jsx)(g.Z,{title:"Time to first token",children:(0,o.jsxs)("div",{className:"flex items-center",children:[(0,o.jsx)(z.Z,{className:"mr-1"}),(0,o.jsxs)("span",{children:[(t/1e3).toFixed(2),"s"]})]})}),(null==s?void 0:s.promptTokens)!==void 0&&(0,o.jsx)(g.Z,{title:"Prompt tokens",children:(0,o.jsxs)("div",{className:"flex items-center",children:[(0,o.jsx)(O.Z,{className:"mr-1"}),(0,o.jsxs)("span",{children:["In: ",s.promptTokens]})]})}),(null==s?void 0:s.completionTokens)!==void 0&&(0,o.jsx)(g.Z,{title:"Completion tokens",children:(0,o.jsxs)("div",{className:"flex items-center",children:[(0,o.jsx)(F.Z,{className:"mr-1"}),(0,o.jsxs)("span",{children:["Out: ",s.completionTokens]})]})}),(null==s?void 0:s.reasoningTokens)!==void 0&&(0,o.jsx)(g.Z,{title:"Reasoning tokens",children:(0,o.jsxs)("div",{className:"flex items-center",children:[(0,o.jsx)(L.Z,{className:"mr-1"}),(0,o.jsxs)("span",{children:["Reasoning: ",s.reasoningTokens]})]})}),(null==s?void 0:s.totalTokens)!==void 0&&(0,o.jsx)(g.Z,{title:"Total tokens",children:(0,o.jsxs)("div",{className:"flex items-center",children:[(0,o.jsx)(H.Z,{className:"mr-1"}),(0,o.jsxs)("span",{children:["Total: ",s.totalTokens]})]})}),a&&(0,o.jsx)(g.Z,{title:"Tool used",children:(0,o.jsxs)("div",{className:"flex items-center",children:[(0,o.jsx)(B.Z,{className:"mr-1"}),(0,o.jsxs)("span",{children:["Tool: ",a]})]})})]}):null},q=s(53508);let{Dragger:J}=c.default;var W=e=>{let{responsesUploadedImage:t,responsesImagePreviewUrl:s,onImageUpload:a,onRemoveImage:n}=e;return(0,o.jsx)(o.Fragment,{children:!t&&(0,o.jsx)(J,{beforeUpload:a,accept:"image/*,.pdf",showUploadList:!1,className:"inline-block",style:{padding:0,border:"none",background:"none"},children:(0,o.jsx)(g.Z,{title:"Attach image or PDF",children:(0,o.jsx)("button",{type:"button",className:"flex items-center justify-center w-8 h-8 text-gray-400 hover:text-gray-600 hover:bg-gray-100 rounded-md transition-colors",children:(0,o.jsx)(q.Z,{style:{fontSize:"16px"}})})})})})};let V=e=>new Promise((t,s)=>{let o=new FileReader;o.onload=()=>{t(o.result.split(",")[1])},o.onerror=s,o.readAsDataURL(e)}),Y=async(e,t)=>{let s=await V(t),o=t.type||(t.name.toLowerCase().endsWith(".pdf")?"application/pdf":"image/jpeg");return{role:"user",content:[{type:"input_text",text:e},{type:"input_image",image_url:"data:".concat(o,";base64,").concat(s)}]}},X=(e,t,s,o)=>{let a="";t&&o&&(a=o.toLowerCase().endsWith(".pdf")?"[PDF attached]":"[Image attached]");let n={role:"user",content:t?"".concat(e," ").concat(a):e};return t&&s&&(n.imagePreviewUrl=s),n},$=e=>"user"===e.role&&"string"==typeof e.content&&(e.content.includes("[Image attached]")||e.content.includes("[PDF attached]"))&&!!e.imagePreviewUrl;var Q=s(50010),ee=e=>{let{message:t}=e;if(!$(t))return null;let s="string"==typeof t.content&&t.content.includes("[PDF attached]");return(0,o.jsx)("div",{className:"mb-2",children:s?(0,o.jsx)("div",{className:"w-64 h-32 rounded-md border border-gray-200 bg-red-50 flex items-center justify-center",children:(0,o.jsx)(Q.Z,{style:{fontSize:"48px",color:"#dc2626"}})}):(0,o.jsx)("img",{src:t.imagePreviewUrl,alt:"User uploaded image",className:"max-w-64 rounded-md border border-gray-200 shadow-sm",style:{maxHeight:"200px"}})})};let{Dragger:et}=c.default;var es=e=>{let{chatUploadedImage:t,chatImagePreviewUrl:s,onImageUpload:a,onRemoveImage:n}=e;return(0,o.jsx)(o.Fragment,{children:!t&&(0,o.jsx)(et,{beforeUpload:a,accept:"image/*,.pdf",showUploadList:!1,className:"inline-block",style:{padding:0,border:"none",background:"none"},children:(0,o.jsx)(g.Z,{title:"Attach image or PDF",children:(0,o.jsx)("button",{type:"button",className:"flex items-center justify-center w-8 h-8 text-gray-400 hover:text-gray-600 hover:bg-gray-100 rounded-md transition-colors",children:(0,o.jsx)(q.Z,{style:{fontSize:"16px"}})})})})})};let eo=e=>new Promise((t,s)=>{let o=new FileReader;o.onload=()=>{t(o.result)},o.onerror=s,o.readAsDataURL(e)}),ea=async(e,t)=>({role:"user",content:[{type:"text",text:e},{type:"image_url",image_url:{url:await eo(t)}}]}),en=(e,t,s,o)=>{let a="";t&&o&&(a=o.toLowerCase().endsWith(".pdf")?"[PDF attached]":"[Image attached]");let n={role:"user",content:t?"".concat(e," ").concat(a):e};return t&&s&&(n.imagePreviewUrl=s),n},el=e=>"user"===e.role&&"string"==typeof e.content&&(e.content.includes("[Image attached]")||e.content.includes("[PDF attached]"))&&!!e.imagePreviewUrl;var er=e=>{let{message:t}=e;if(!el(t))return null;let s="string"==typeof t.content&&t.content.includes("[PDF attached]");return(0,o.jsx)("div",{className:"mb-2",children:s?(0,o.jsx)("div",{className:"w-64 h-32 rounded-md border border-gray-200 bg-red-50 flex items-center justify-center",children:(0,o.jsx)(Q.Z,{style:{fontSize:"48px",color:"#dc2626"}})}):(0,o.jsx)("img",{src:t.imagePreviewUrl,alt:"User uploaded image",className:"max-w-64 rounded-md border border-gray-200 shadow-sm",style:{maxHeight:"200px"}})})},ei=s(63709),ec=s(15424),ed=s(23639),em=e=>{let{endpointType:t,responsesSessionId:s,useApiSessionManagement:a,onToggleSessionManagement:n}=e;return t!==C.KP.RESPONSES?null:(0,o.jsxs)("div",{className:"mb-4",children:[(0,o.jsxs)("div",{className:"flex items-center justify-between mb-2",children:[(0,o.jsxs)("div",{className:"flex items-center gap-2",children:[(0,o.jsx)("span",{className:"text-sm font-medium text-gray-700",children:"Session Management"}),(0,o.jsx)(g.Z,{title:"Choose between LiteLLM API session management (using previous_response_id) or UI-based session management (using chat history)",children:(0,o.jsx)(ec.Z,{className:"text-gray-400",style:{fontSize:"12px"}})})]}),(0,o.jsx)(ei.Z,{checked:a,onChange:n,checkedChildren:"API",unCheckedChildren:"UI",size:"small"})]}),(0,o.jsxs)("div",{className:"text-xs p-2 rounded-md ".concat(s?"bg-green-50 text-green-700 border border-green-200":"bg-blue-50 text-blue-700 border border-blue-200"),children:[(0,o.jsxs)("div",{className:"flex items-center justify-between",children:[(0,o.jsxs)("div",{className:"flex items-center gap-1",children:[(0,o.jsx)(ec.Z,{style:{fontSize:"12px"}}),(()=>{if(!s)return a?"API Session: Ready":"UI Session: Ready";let e=a?"Response ID":"UI Session",t=s.slice(0,10);return"".concat(e,": ").concat(t,"...")})()]}),s&&(0,o.jsx)(g.Z,{title:(0,o.jsxs)("div",{className:"text-xs",children:[(0,o.jsx)("div",{className:"mb-1",children:"Copy response ID to continue session:"}),(0,o.jsx)("div",{className:"bg-gray-800 text-gray-100 p-2 rounded font-mono text-xs whitespace-pre-wrap",children:'curl -X POST "your-proxy-url/v1/responses" \\\n -H "Authorization: Bearer your-api-key" \\\n -H "Content-Type: application/json" \\\n -d \'{\n "model": "your-model",\n "input": [{"role": "user", "content": "your message", "type": "message"}],\n "previous_response_id": "'.concat(s,'",\n "stream": true\n }\'')})]}),overlayStyle:{maxWidth:"500px"},children:(0,o.jsx)("button",{onClick:()=>{s&&(navigator.clipboard.writeText(s),v.Z.success("Response ID copied to clipboard!"))},className:"ml-2 p-1 hover:bg-green-100 rounded transition-colors",children:(0,o.jsx)(ed.Z,{style:{fontSize:"12px"}})})})]}),(0,o.jsx)("div",{className:"text-xs opacity-75 mt-1",children:s?a?"LiteLLM API session active - context maintained server-side":"UI session active - context maintained client-side":a?"LiteLLM will manage session using previous_response_id":"UI will manage session using chat history"})]})]})},eg=s(29),ep=s.n(eg),eu=s(44851);let{Text:ex}=d.default,{Panel:eh}=eu.default;var ef=e=>{var t,s;let{events:a,className:n}=e;if(console.log("MCPEventsDisplay: Received events:",a),!a||0===a.length)return console.log("MCPEventsDisplay: No events, returning null"),null;let l=a.find(e=>{var t;return"response.output_item.done"===e.type&&(null===(t=e.item)||void 0===t?void 0:t.type)==="mcp_list_tools"&&e.item.tools&&e.item.tools.length>0}),r=a.filter(e=>{var t;return"response.output_item.done"===e.type&&(null===(t=e.item)||void 0===t?void 0:t.type)==="mcp_call"});return(console.log("MCPEventsDisplay: toolsEvent:",l),console.log("MCPEventsDisplay: mcpCallEvents:",r),l||0!==r.length)?(0,o.jsxs)("div",{className:"jsx-32b14b04f420f3ac "+"mcp-events-display ".concat(n||""),children:[(0,o.jsx)(ep(),{id:"32b14b04f420f3ac",children:'.openai-mcp-tools.jsx-32b14b04f420f3ac{position:relative;margin:0;padding:0}.openai-mcp-tools.jsx-32b14b04f420f3ac .ant-collapse.jsx-32b14b04f420f3ac{background:transparent!important;border:none!important}.openai-mcp-tools.jsx-32b14b04f420f3ac .ant-collapse-item.jsx-32b14b04f420f3ac{border:none!important;background:transparent!important}.openai-mcp-tools.jsx-32b14b04f420f3ac .ant-collapse-header.jsx-32b14b04f420f3ac{padding:0 0 0 20px!important;background:transparent!important;border:none!important;font-size:14px!important;color:#9ca3af!important;font-weight:400!important;line-height:20px!important;min-height:20px!important}.openai-mcp-tools.jsx-32b14b04f420f3ac .ant-collapse-header.jsx-32b14b04f420f3ac:hover{background:transparent!important;color:#6b7280!important}.openai-mcp-tools.jsx-32b14b04f420f3ac .ant-collapse-content.jsx-32b14b04f420f3ac{border:none!important;background:transparent!important}.openai-mcp-tools.jsx-32b14b04f420f3ac .ant-collapse-content-box.jsx-32b14b04f420f3ac{padding:4px 0 0 20px!important}.openai-mcp-tools.jsx-32b14b04f420f3ac .ant-collapse-expand-icon.jsx-32b14b04f420f3ac{position:absolute!important;left:2px!important;top:2px!important;color:#9ca3af!important;font-size:10px!important;width:16px!important;height:16px!important;display:-webkit-box!important;display:-webkit-flex!important;display:-moz-box!important;display:-ms-flexbox!important;display:flex!important;-webkit-box-align:center!important;-webkit-align-items:center!important;-moz-box-align:center!important;-ms-flex-align:center!important;align-items:center!important;-webkit-box-pack:center!important;-webkit-justify-content:center!important;-moz-box-pack:center!important;-ms-flex-pack:center!important;justify-content:center!important}.openai-mcp-tools.jsx-32b14b04f420f3ac .ant-collapse-expand-icon.jsx-32b14b04f420f3ac:hover{color:#6b7280!important}.openai-vertical-line.jsx-32b14b04f420f3ac{position:absolute;left:9px;top:18px;bottom:0;width:.5px;background-color:#f3f4f6;opacity:.8}.tool-item.jsx-32b14b04f420f3ac{font-family:ui-monospace,SFMono-Regular,"SF Mono",Monaco,Consolas,"Liberation Mono","Courier New",monospace;font-size:13px;color:#4b5563;line-height:18px;padding:0;margin:0;background:white;position:relative;z-index:1}.mcp-section.jsx-32b14b04f420f3ac{margin-bottom:12px;background:white;position:relative;z-index:1}.mcp-section.jsx-32b14b04f420f3ac:last-child{margin-bottom:0}.mcp-section-header.jsx-32b14b04f420f3ac{font-size:13px;color:#6b7280;font-weight:500;margin-bottom:4px}.mcp-code-block.jsx-32b14b04f420f3ac{background:#f9fafb;border:1px solid#f3f4f6;-webkit-border-radius:6px;-moz-border-radius:6px;border-radius:6px;padding:8px;font-size:12px}.mcp-json.jsx-32b14b04f420f3ac{font-family:ui-monospace,SFMono-Regular,"SF Mono",Monaco,Consolas,"Liberation Mono","Courier New",monospace;color:#374151;margin:0;white-space:pre-wrap;word-wrap:break-word}.mcp-approved.jsx-32b14b04f420f3ac{display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;font-size:13px;color:#6b7280}.mcp-checkmark.jsx-32b14b04f420f3ac{color:#10b981;margin-right:6px;font-weight:bold}.mcp-response-content.jsx-32b14b04f420f3ac{font-size:13px;color:#374151;line-height:1.5;white-space:pre-wrap;font-family:ui-monospace,SFMono-Regular,"SF Mono",Monaco,Consolas,"Liberation Mono","Courier New",monospace}'}),(0,o.jsxs)("div",{className:"jsx-32b14b04f420f3ac openai-mcp-tools",children:[(0,o.jsx)("div",{className:"jsx-32b14b04f420f3ac openai-vertical-line"}),(0,o.jsxs)(eu.default,{ghost:!0,size:"small",expandIconPosition:"start",defaultActiveKey:l?["list-tools"]:r.map((e,t)=>"mcp-call-".concat(t)),children:[l&&(0,o.jsx)(eh,{header:"List tools",children:(0,o.jsx)("div",{className:"jsx-32b14b04f420f3ac",children:null===(s=l.item)||void 0===s?void 0:null===(t=s.tools)||void 0===t?void 0:t.map((e,t)=>(0,o.jsx)("div",{className:"jsx-32b14b04f420f3ac tool-item",children:e.name},t))})},"list-tools"),r.map((e,t)=>{var s,a,n;return(0,o.jsx)(eh,{header:(null===(s=e.item)||void 0===s?void 0:s.name)||"Tool call",children:(0,o.jsxs)("div",{className:"jsx-32b14b04f420f3ac",children:[(0,o.jsxs)("div",{className:"jsx-32b14b04f420f3ac mcp-section",children:[(0,o.jsx)("div",{className:"jsx-32b14b04f420f3ac mcp-section-header",children:"Request"}),(0,o.jsx)("div",{className:"jsx-32b14b04f420f3ac mcp-code-block",children:(null===(a=e.item)||void 0===a?void 0:a.arguments)&&(0,o.jsx)("pre",{className:"jsx-32b14b04f420f3ac mcp-json",children:(()=>{try{return JSON.stringify(JSON.parse(e.item.arguments),null,2)}catch(t){return e.item.arguments}})()})})]}),(0,o.jsx)("div",{className:"jsx-32b14b04f420f3ac mcp-section",children:(0,o.jsxs)("div",{className:"jsx-32b14b04f420f3ac mcp-approved",children:[(0,o.jsx)("span",{className:"jsx-32b14b04f420f3ac mcp-checkmark",children:"✓"})," Approved"]})}),(null===(n=e.item)||void 0===n?void 0:n.output)&&(0,o.jsxs)("div",{className:"jsx-32b14b04f420f3ac mcp-section",children:[(0,o.jsx)("div",{className:"jsx-32b14b04f420f3ac mcp-section-header",children:"Response"}),(0,o.jsx)("div",{className:"jsx-32b14b04f420f3ac mcp-response-content",children:e.item.output})]})]})},"mcp-call-".concat(t))})]})]})]}):(console.log("MCPEventsDisplay: No valid events found, returning null"),null)},eb=s(61935),ev=s(92403),ey=s(69993),ej=s(12660),ew=s(71891),eS=s(44625),eN=s(57400),ek=s(26430),eP=s(11894),eC=s(15883),eI=s(99890),e_=s(26349),eT=s(79276);let{TextArea:eZ}=i.default,{Dragger:eE}=c.default;var eA=e=>{let{accessToken:t,token:s,userRole:i,userID:c,disabledPersonalKeyCreation:h}=e,[f,S]=(0,a.useState)(!1),[T,L]=(0,a.useState)([]),[M,D]=(0,a.useState)(()=>{let e=sessionStorage.getItem("selectedMCPTools");try{let t=e?JSON.parse(e):[];return Array.isArray(t)?t:t?[t]:[]}catch(e){return console.error("Error parsing selectedMCPTools from sessionStorage",e),[]}}),[z,O]=(0,a.useState)(!1),[F,H]=(0,a.useState)(()=>{let e=sessionStorage.getItem("apiKeySource");if(e)try{return JSON.parse(e)}catch(e){console.error("Error parsing apiKeySource from sessionStorage",e)}return h?"custom":"session"}),[q,J]=(0,a.useState)(()=>sessionStorage.getItem("apiKey")||""),[V,$]=(0,a.useState)(""),[et,eo]=(0,a.useState)(()=>{try{let e=sessionStorage.getItem("chatHistory");return e?JSON.parse(e):[]}catch(e){return console.error("Error parsing chatHistory from sessionStorage",e),[]}}),[el,ei]=(0,a.useState)(()=>sessionStorage.getItem("selectedModel")||void 0),[ed,eg]=(0,a.useState)(!1),[ep,eu]=(0,a.useState)([]),ex=(0,a.useRef)(null),[eh,eA]=(0,a.useState)(()=>sessionStorage.getItem("endpointType")||C.KP.CHAT),[eU,eR]=(0,a.useState)(!1),eL=(0,a.useRef)(null),[eM,eD]=(0,a.useState)(()=>{let e=sessionStorage.getItem("selectedTags");try{return e?JSON.parse(e):[]}catch(e){return console.error("Error parsing selectedTags from sessionStorage",e),[]}}),[eK,ez]=(0,a.useState)(()=>{let e=sessionStorage.getItem("selectedVectorStores");try{return e?JSON.parse(e):[]}catch(e){return console.error("Error parsing selectedVectorStores from sessionStorage",e),[]}}),[eO,eF]=(0,a.useState)(()=>{let e=sessionStorage.getItem("selectedGuardrails");try{return e?JSON.parse(e):[]}catch(e){return console.error("Error parsing selectedGuardrails from sessionStorage",e),[]}}),[eH,eB]=(0,a.useState)(()=>sessionStorage.getItem("messageTraceId")||null),[eG,eq]=(0,a.useState)(()=>sessionStorage.getItem("responsesSessionId")||null),[eJ,eW]=(0,a.useState)(()=>{let e=sessionStorage.getItem("useApiSessionManagement");return!e||JSON.parse(e)}),[eV,eY]=(0,a.useState)([]),[eX,e$]=(0,a.useState)([]),[eQ,e0]=(0,a.useState)(null),[e1,e4]=(0,a.useState)(null),[e2,e3]=(0,a.useState)(null),[e6,e5]=(0,a.useState)(null),[e8,e7]=(0,a.useState)(!1),[e9,te]=(0,a.useState)(""),[tt,ts]=(0,a.useState)("openai"),[to,ta]=(0,a.useState)([]),tn=(0,a.useRef)(null),tl=async()=>{let e="session"===F?t:q;if(e){O(!0);try{let t=await P(e);L(t)}catch(e){console.error("Error fetching MCP tools:",e)}finally{O(!1)}}};(0,a.useEffect)(()=>{f&&tl()},[f,t,q,F]),(0,a.useEffect)(()=>{e8&&te((0,R.L)({apiKeySource:F,accessToken:t,apiKey:q,inputMessage:V,chatHistory:et,selectedTags:eM,selectedVectorStores:eK,selectedGuardrails:eO,selectedMCPTools:M,endpointType:eh,selectedModel:el,selectedSdk:tt}))},[e8,tt,F,t,q,V,et,eM,eK,eO,M,eh,el]),(0,a.useEffect)(()=>{let e=setTimeout(()=>{sessionStorage.setItem("chatHistory",JSON.stringify(et))},500);return()=>{clearTimeout(e)}},[et]),(0,a.useEffect)(()=>{sessionStorage.setItem("apiKeySource",JSON.stringify(F)),sessionStorage.setItem("apiKey",q),sessionStorage.setItem("endpointType",eh),sessionStorage.setItem("selectedTags",JSON.stringify(eM)),sessionStorage.setItem("selectedVectorStores",JSON.stringify(eK)),sessionStorage.setItem("selectedGuardrails",JSON.stringify(eO)),sessionStorage.setItem("selectedMCPTools",JSON.stringify(M)),el?sessionStorage.setItem("selectedModel",el):sessionStorage.removeItem("selectedModel"),eH?sessionStorage.setItem("messageTraceId",eH):sessionStorage.removeItem("messageTraceId"),eG?sessionStorage.setItem("responsesSessionId",eG):sessionStorage.removeItem("responsesSessionId"),sessionStorage.setItem("useApiSessionManagement",JSON.stringify(eJ))},[F,q,el,eh,eM,eK,eO,eH,eG,eJ,M]),(0,a.useEffect)(()=>{let e="session"===F?t:q;if(!e||!s||!i||!c){console.log("userApiKey or token or userRole or userID is missing = ",e,s,i,c);return}(async()=>{try{if(!e){console.log("userApiKey is missing");return}let t=await (0,k.p)(e);console.log("Fetched models:",t),eu(t);let s=t.some(e=>e.model_group===el);t.length?s||ei(t[0].model_group):ei(void 0)}catch(e){console.error("Error fetching model info:",e)}})(),tl()},[t,c,i,F,q,s]),(0,a.useEffect)(()=>{tn.current&&setTimeout(()=>{var e;null===(e=tn.current)||void 0===e||e.scrollIntoView({behavior:"smooth",block:"end"})},100)},[et]);let tr=(e,t,s)=>{console.log("updateTextUI called with:",e,t,s),eo(o=>{let a=o[o.length-1];if(!a||a.role!==e||a.isImage)return[...o,{role:e,content:t,model:s}];{var n;let e={...a,content:a.content+t,model:null!==(n=a.model)&&void 0!==n?n:s};return[...o.slice(0,-1),e]}})},ti=e=>{eo(t=>{let s=t[t.length-1];return s&&"assistant"===s.role&&!s.isImage?[...t.slice(0,t.length-1),{...s,reasoningContent:(s.reasoningContent||"")+e}]:t.length>0&&"user"===t[t.length-1].role?[...t,{role:"assistant",content:"",reasoningContent:e}]:t})},tc=e=>{console.log("updateTimingData called with:",e),eo(t=>{let s=t[t.length-1];if(console.log("Current last message:",s),s&&"assistant"===s.role){console.log("Updating assistant message with timeToFirstToken:",e);let o=[...t.slice(0,t.length-1),{...s,timeToFirstToken:e}];return console.log("Updated chat history:",o),o}return s&&"user"===s.role?(console.log("Creating new assistant message with timeToFirstToken:",e),[...t,{role:"assistant",content:"",timeToFirstToken:e}]):(console.log("No appropriate message found to update timing"),t)})},td=(e,t)=>{console.log("Received usage data:",e),eo(s=>{let o=s[s.length-1];if(o&&"assistant"===o.role){console.log("Updating message with usage data:",e);let a={...o,usage:e,toolName:t};return console.log("Updated message:",a),[...s.slice(0,s.length-1),a]}return s})},tm=e=>{console.log("Received response ID for session management:",e),eJ&&eq(e)},tg=e=>{console.log("ChatUI: Received MCP event:",e),ta(t=>{if(t.some(t=>t.item_id===e.item_id&&t.type===e.type&&t.sequence_number===e.sequence_number))return console.log("ChatUI: Duplicate MCP event, skipping"),t;let s=[...t,e];return console.log("ChatUI: Updated MCP events:",s),s})},tp=(e,t)=>{eo(s=>[...s,{role:"assistant",content:e,model:t,isImage:!0}])},tu=(e,t)=>{eo(s=>{let o=s[s.length-1];if(!o||"assistant"!==o.role||o.isImage)return[...s,{role:"assistant",content:"",model:t,image:{url:e,detail:"auto"}}];{var a;let n={...o,image:{url:e,detail:"auto"},model:null!==(a=o.model)&&void 0!==a?a:t};return[...s.slice(0,-1),n]}})},tx=e=>{eY(t=>[...t,e]);let t=URL.createObjectURL(e);return e$(e=>[...e,t]),!1},th=e=>{eX[e]&&URL.revokeObjectURL(eX[e]),eY(t=>t.filter((t,s)=>s!==e)),e$(t=>t.filter((t,s)=>s!==e))},tf=()=>{eX.forEach(e=>{URL.revokeObjectURL(e)}),eY([]),e$([])},tb=()=>{e1&&URL.revokeObjectURL(e1),e0(null),e4(null)},tv=()=>{e6&&URL.revokeObjectURL(e6),e3(null),e5(null)},ty=async()=>{let e;if(""===V.trim())return;if(eh===C.KP.IMAGE_EDITS&&0===eV.length){v.Z.fromBackend("Please upload at least one image for editing");return}if(!s||!i||!c)return;let o="session"===F?t:q;if(!o){v.Z.fromBackend("Please provide an API key or select Current UI Session");return}eL.current=new AbortController;let a=eL.current.signal;if(eh===C.KP.RESPONSES&&eQ)try{e=await Y(V,eQ)}catch(e){v.Z.fromBackend("Failed to process image. Please try again.");return}else if(eh===C.KP.CHAT&&e2)try{e=await ea(V,e2)}catch(e){v.Z.fromBackend("Failed to process image. Please try again.");return}else e={role:"user",content:V};let n=eH||(0,r.Z)();eH||eB(n),eo([...et,eh===C.KP.RESPONSES&&eQ?X(V,!0,e1||void 0,eQ.name):eh===C.KP.CHAT&&e2?en(V,!0,e6||void 0,e2.name):X(V,!1)]),ta([]),eR(!0);try{if(el){if(eh===C.KP.CHAT){let t=[...et.filter(e=>!e.isImage).map(e=>{let{role:t,content:s}=e;return{role:t,content:"string"==typeof s?s:""}}),e];await b(t,(e,t)=>tr("assistant",e,t),el,o,eM,a,ti,tc,td,n,eK.length>0?eK:void 0,eO.length>0?eO:void 0,M,tu)}else if(eh===C.KP.IMAGE)await y(V,(e,t)=>tp(e,t),el,o,eM,a);else if(eh===C.KP.IMAGE_EDITS)eV.length>0&&await j(1===eV.length?eV[0]:eV,V,(e,t)=>tp(e,t),el,o,eM,a);else if(eh===C.KP.RESPONSES){let t;t=eJ&&eG?[e]:[...et.filter(e=>!e.isImage).map(e=>{let{role:t,content:s}=e;return{role:t,content:s}}),e],await w(t,(e,t,s)=>tr(e,t,s),el,o,eM,a,ti,tc,td,n,eK.length>0?eK:void 0,eO.length>0?eO:void 0,M,eJ?eG:null,tm,tg)}else if(eh===C.KP.ANTHROPIC_MESSAGES){let t=[...et.filter(e=>!e.isImage).map(e=>{let{role:t,content:s}=e;return{role:t,content:s}}),e];await N(t,(e,t,s)=>tr(e,t,s),el,o,eM,a,ti,tc,td,n,eK.length>0?eK:void 0,eO.length>0?eO:void 0,M)}}}catch(e){a.aborted?console.log("Request was cancelled"):(console.error("Error fetching response",e),tr("assistant","Error fetching response:"+e))}finally{eR(!1),eL.current=null,eh===C.KP.IMAGE_EDITS&&tf(),eh===C.KP.RESPONSES&&eQ&&tb(),eh===C.KP.CHAT&&e2&&tv()}$("")};if(i&&"Admin Viewer"===i){let{Title:e,Paragraph:t}=d.default;return(0,o.jsxs)("div",{children:[(0,o.jsx)(e,{level:1,children:"Access Denied"}),(0,o.jsx)(t,{children:"Ask your proxy admin for access to test models"})]})}let tj=(0,o.jsx)(eb.Z,{style:{fontSize:24},spin:!0});return(0,o.jsxs)("div",{className:"w-full h-screen p-4 bg-white",children:[(0,o.jsx)(l.Zb,{className:"w-full rounded-xl shadow-md overflow-hidden",children:(0,o.jsxs)("div",{className:"flex h-[80vh] w-full gap-4",children:[(0,o.jsxs)("div",{className:"w-1/4 p-4 bg-gray-50 overflow-y-auto",children:[(0,o.jsx)(l.Dx,{className:"text-xl font-semibold mb-6 mt-2",children:"Configurations"}),(0,o.jsxs)("div",{className:"space-y-4",children:[(0,o.jsxs)("div",{children:[(0,o.jsxs)(l.xv,{className:"font-medium block mb-2 text-gray-700 flex items-center",children:[(0,o.jsx)(ev.Z,{className:"mr-2"})," API Key Source"]}),(0,o.jsx)(m.default,{disabled:h,value:F,style:{width:"100%"},onChange:e=>{H(e)},options:[{value:"session",label:"Current UI Session"},{value:"custom",label:"Virtual Key"}],className:"rounded-md"}),"custom"===F&&(0,o.jsx)(l.oi,{className:"mt-2",placeholder:"Enter custom API key",type:"password",onValueChange:J,value:q,icon:ev.Z})]}),(0,o.jsxs)("div",{children:[(0,o.jsxs)(l.xv,{className:"font-medium block mb-2 text-gray-700 flex items-center",children:[(0,o.jsx)(ey.Z,{className:"mr-2"})," Select Model"]}),(0,o.jsx)(m.default,{value:el,placeholder:"Select a Model",onChange:e=>{console.log("selected ".concat(e)),ei(e),eg("custom"===e)},options:[...Array.from(new Set(ep.map(e=>e.model_group))).map((e,t)=>({value:e,label:e,key:t})),{value:"custom",label:"Enter custom model",key:"custom"}],style:{width:"100%"},showSearch:!0,className:"rounded-md"}),ed&&(0,o.jsx)(l.oi,{className:"mt-2",placeholder:"Enter custom model name",onValueChange:e=>{ex.current&&clearTimeout(ex.current),ex.current=setTimeout(()=>{ei(e)},500)}})]}),(0,o.jsxs)("div",{children:[(0,o.jsxs)(l.xv,{className:"font-medium block mb-2 text-gray-700 flex items-center",children:[(0,o.jsx)(ej.Z,{className:"mr-2"})," Endpoint Type"]}),(0,o.jsx)(Z,{endpointType:eh,onEndpointChange:e=>{eA(e)},className:"mb-4"}),(0,o.jsx)(em,{endpointType:eh,responsesSessionId:eG,useApiSessionManagement:eJ,onToggleSessionManagement:e=>{eW(e),e||eq(null)}})]}),(0,o.jsxs)("div",{children:[(0,o.jsxs)(l.xv,{className:"font-medium block mb-2 text-gray-700 flex items-center",children:[(0,o.jsx)(ew.Z,{className:"mr-2"})," Tags"]}),(0,o.jsx)(E,{value:eM,onChange:eD,className:"mb-4",accessToken:t||""})]}),(0,o.jsxs)("div",{children:[(0,o.jsxs)(l.xv,{className:"font-medium block mb-2 text-gray-700 flex items-center",children:[(0,o.jsx)(B.Z,{className:"mr-2"})," MCP Tool",(0,o.jsx)(g.Z,{className:"ml-1",title:"Select MCP tools to use in your conversation, only available for /v1/responses endpoint",children:(0,o.jsx)(ec.Z,{})})]}),(0,o.jsx)(m.default,{mode:"multiple",style:{width:"100%"},placeholder:"Select MCP tools",value:M,onChange:e=>D(e),loading:z,className:"mb-4",allowClear:!0,optionLabelProp:"label",disabled:eh!==C.KP.RESPONSES,maxTagCount:"responsive",children:Array.isArray(T)&&T.map(e=>(0,o.jsx)(m.default.Option,{value:e.name,label:(0,o.jsx)("div",{className:"font-medium",children:e.name}),children:(0,o.jsxs)("div",{className:"flex flex-col py-1",children:[(0,o.jsx)("span",{className:"font-medium",children:e.name}),(0,o.jsx)("span",{className:"text-xs text-gray-500 mt-1",children:e.description})]})},e.name))})]}),(0,o.jsxs)("div",{children:[(0,o.jsxs)(l.xv,{className:"font-medium block mb-2 text-gray-700 flex items-center",children:[(0,o.jsx)(eS.Z,{className:"mr-2"})," Vector Store",(0,o.jsx)(g.Z,{className:"ml-1",title:(0,o.jsxs)("span",{children:["Select vector store(s) to use for this LLM API call. You can set up your vector store"," ",(0,o.jsx)("a",{href:"?page=vector-stores",style:{color:"#1890ff"},children:"here"}),"."]}),children:(0,o.jsx)(ec.Z,{})})]}),(0,o.jsx)(A.Z,{value:eK,onChange:ez,className:"mb-4",accessToken:t||""})]}),(0,o.jsxs)("div",{children:[(0,o.jsxs)(l.xv,{className:"font-medium block mb-2 text-gray-700 flex items-center",children:[(0,o.jsx)(eN.Z,{className:"mr-2"})," Guardrails",(0,o.jsx)(g.Z,{className:"ml-1",title:(0,o.jsxs)("span",{children:["Select guardrail(s) to use for this LLM API call. You can set up your guardrails"," ",(0,o.jsx)("a",{href:"?page=guardrails",style:{color:"#1890ff"},children:"here"}),"."]}),children:(0,o.jsx)(ec.Z,{})})]}),(0,o.jsx)(U.Z,{value:eO,onChange:eF,className:"mb-4",accessToken:t||""})]})]})]}),(0,o.jsxs)("div",{className:"w-3/4 flex flex-col bg-white",children:[(0,o.jsxs)("div",{className:"p-4 border-b border-gray-200 flex justify-between items-center",children:[(0,o.jsx)(l.Dx,{className:"text-xl font-semibold mb-0",children:"Test Key"}),(0,o.jsxs)("div",{className:"flex gap-2",children:[(0,o.jsx)(l.zx,{onClick:()=>{eo([]),eB(null),eq(null),ta([]),tf(),tb(),tv(),sessionStorage.removeItem("chatHistory"),sessionStorage.removeItem("messageTraceId"),sessionStorage.removeItem("responsesSessionId"),v.Z.success("Chat history cleared.")},className:"bg-gray-100 hover:bg-gray-200 text-gray-700 border-gray-300",icon:ek.Z,children:"Clear Chat"}),(0,o.jsx)(l.zx,{onClick:()=>e7(!0),className:"bg-gray-100 hover:bg-gray-200 text-gray-700 border-gray-300",icon:eP.Z,children:"Get Code"})]})]}),(0,o.jsxs)("div",{className:"flex-1 overflow-auto p-4 pb-0",children:[0===et.length&&(0,o.jsxs)("div",{className:"h-full flex flex-col items-center justify-center text-gray-400",children:[(0,o.jsx)(ey.Z,{style:{fontSize:"48px",marginBottom:"16px"}}),(0,o.jsx)(l.xv,{children:"Start a conversation or generate an image"})]}),et.map((e,t)=>(0,o.jsx)("div",{children:(0,o.jsx)("div",{className:"mb-4 ".concat("user"===e.role?"text-right":"text-left"),children:(0,o.jsxs)("div",{className:"inline-block max-w-[80%] rounded-lg shadow-sm p-3.5 px-4",style:{backgroundColor:"user"===e.role?"#f0f8ff":"#ffffff",border:"user"===e.role?"1px solid #e6f0fa":"1px solid #f0f0f0",textAlign:"left"},children:[(0,o.jsxs)("div",{className:"flex items-center gap-2 mb-1.5",children:[(0,o.jsx)("div",{className:"flex items-center justify-center w-6 h-6 rounded-full mr-1",style:{backgroundColor:"user"===e.role?"#e6f0fa":"#f5f5f5"},children:"user"===e.role?(0,o.jsx)(eC.Z,{style:{fontSize:"12px",color:"#2563eb"}}):(0,o.jsx)(ey.Z,{style:{fontSize:"12px",color:"#4b5563"}})}),(0,o.jsx)("strong",{className:"text-sm capitalize",children:e.role}),"assistant"===e.role&&e.model&&(0,o.jsx)("span",{className:"text-xs px-2 py-0.5 rounded bg-gray-100 text-gray-600 font-normal",children:e.model})]}),e.reasoningContent&&(0,o.jsx)(K,{reasoningContent:e.reasoningContent}),"assistant"===e.role&&t===et.length-1&&to.length>0&&eh===C.KP.RESPONSES&&(0,o.jsx)("div",{className:"mb-3",children:(0,o.jsx)(ef,{events:to})}),(0,o.jsxs)("div",{className:"whitespace-pre-wrap break-words max-w-full message-content",style:{wordWrap:"break-word",overflowWrap:"break-word",wordBreak:"break-word",hyphens:"auto"},children:[e.isImage?(0,o.jsx)("img",{src:"string"==typeof e.content?e.content:"",alt:"Generated image",className:"max-w-full rounded-md border border-gray-200 shadow-sm",style:{maxHeight:"500px"}}):(0,o.jsxs)(o.Fragment,{children:[eh===C.KP.RESPONSES&&(0,o.jsx)(ee,{message:e}),eh===C.KP.CHAT&&(0,o.jsx)(er,{message:e}),(0,o.jsx)(n.U,{components:{code(e){let{node:t,inline:s,className:a,children:n,...l}=e,r=/language-(\w+)/.exec(a||"");return!s&&r?(0,o.jsx)(I.Z,{style:_.Z,language:r[1],PreTag:"div",className:"rounded-md my-2",wrapLines:!0,wrapLongLines:!0,...l,children:String(n).replace(/\n$/,"")}):(0,o.jsx)("code",{className:"".concat(a," px-1.5 py-0.5 rounded bg-gray-100 text-sm font-mono"),style:{wordBreak:"break-word"},...l,children:n})},pre:e=>{let{node:t,...s}=e;return(0,o.jsx)("pre",{style:{overflowX:"auto",maxWidth:"100%"},...s})}},children:"string"==typeof e.content?e.content:""}),e.image&&(0,o.jsx)("div",{className:"mt-3",children:(0,o.jsx)("img",{src:e.image.url,alt:"Generated image",className:"max-w-full rounded-md border border-gray-200 shadow-sm",style:{maxHeight:"500px"}})})]}),"assistant"===e.role&&(e.timeToFirstToken||e.usage)&&(0,o.jsx)(G,{timeToFirstToken:e.timeToFirstToken,usage:e.usage,toolName:e.toolName})]})]})})},t)),eU&&to.length>0&&eh===C.KP.RESPONSES&&et.length>0&&"user"===et[et.length-1].role&&(0,o.jsx)("div",{className:"text-left mb-4",children:(0,o.jsxs)("div",{className:"inline-block max-w-[80%] rounded-lg shadow-sm p-3.5 px-4",style:{backgroundColor:"#ffffff",border:"1px solid #f0f0f0",textAlign:"left"},children:[(0,o.jsxs)("div",{className:"flex items-center gap-2 mb-1.5",children:[(0,o.jsx)("div",{className:"flex items-center justify-center w-6 h-6 rounded-full mr-1",style:{backgroundColor:"#f5f5f5"},children:(0,o.jsx)(ey.Z,{style:{fontSize:"12px",color:"#4b5563"}})}),(0,o.jsx)("strong",{className:"text-sm capitalize",children:"Assistant"})]}),(0,o.jsx)(ef,{events:to})]})}),eU&&(0,o.jsx)("div",{className:"flex justify-center items-center my-4",children:(0,o.jsx)(p.Z,{indicator:tj})}),(0,o.jsx)("div",{ref:tn,style:{height:"1px"}})]}),(0,o.jsxs)("div",{className:"p-4 border-t border-gray-200 bg-white",children:[eh===C.KP.IMAGE_EDITS&&(0,o.jsx)("div",{className:"mb-4",children:0===eV.length?(0,o.jsxs)(eE,{beforeUpload:tx,accept:"image/*",showUploadList:!1,className:"border-dashed border-2 border-gray-300 rounded-lg p-4",children:[(0,o.jsx)("p",{className:"ant-upload-drag-icon",children:(0,o.jsx)(eI.Z,{style:{fontSize:"24px",color:"#666"}})}),(0,o.jsx)("p",{className:"ant-upload-text text-sm",children:"Click or drag images to upload"}),(0,o.jsx)("p",{className:"ant-upload-hint text-xs text-gray-500",children:"Support for PNG, JPG, JPEG formats. Multiple images supported."})]}):(0,o.jsxs)("div",{className:"flex flex-wrap gap-2",children:[eV.map((e,t)=>(0,o.jsxs)("div",{className:"relative inline-block",children:[(0,o.jsx)("img",{src:eX[t]||"",alt:"Upload preview ".concat(t+1),className:"max-w-32 max-h-32 rounded-md border border-gray-200 object-cover"}),(0,o.jsx)("button",{className:"absolute top-1 right-1 bg-white shadow-sm border border-gray-200 rounded px-1 py-1 text-red-500 hover:bg-red-50 text-xs",onClick:()=>th(t),children:(0,o.jsx)(e_.Z,{})})]},t)),(0,o.jsxs)("div",{className:"flex items-center justify-center w-32 h-32 border-2 border-dashed border-gray-300 rounded-md hover:border-gray-400 cursor-pointer",onClick:()=>{var e;return null===(e=document.getElementById("additional-image-upload"))||void 0===e?void 0:e.click()},children:[(0,o.jsxs)("div",{className:"text-center",children:[(0,o.jsx)(eI.Z,{style:{fontSize:"24px",color:"#666"}}),(0,o.jsx)("p",{className:"text-xs text-gray-500 mt-1",children:"Add more"})]}),(0,o.jsx)("input",{id:"additional-image-upload",type:"file",accept:"image/*",multiple:!0,style:{display:"none"},onChange:e=>{Array.from(e.target.files||[]).forEach(e=>tx(e))}})]})]})}),eh===C.KP.RESPONSES&&eQ&&(0,o.jsx)("div",{className:"mb-2",children:(0,o.jsxs)("div",{className:"flex items-center gap-3 p-3 bg-gray-50 rounded-lg border border-gray-200",children:[(0,o.jsx)("div",{className:"relative inline-block",children:eQ.name.toLowerCase().endsWith(".pdf")?(0,o.jsx)("div",{className:"w-10 h-10 rounded-md bg-red-500 flex items-center justify-center",children:(0,o.jsx)(Q.Z,{style:{fontSize:"16px",color:"white"}})}):(0,o.jsx)("img",{src:e1||"",alt:"Upload preview",className:"w-10 h-10 rounded-md border border-gray-200 object-cover"})}),(0,o.jsxs)("div",{className:"flex-1 min-w-0",children:[(0,o.jsx)("div",{className:"text-sm font-medium text-gray-900 truncate",children:eQ.name}),(0,o.jsx)("div",{className:"text-xs text-gray-500",children:eQ.name.toLowerCase().endsWith(".pdf")?"PDF":"Image"})]}),(0,o.jsx)("button",{className:"flex items-center justify-center w-6 h-6 text-gray-400 hover:text-gray-600 hover:bg-gray-200 rounded-full transition-colors",onClick:tb,children:(0,o.jsx)(e_.Z,{style:{fontSize:"12px"}})})]})}),eh===C.KP.CHAT&&e2&&(0,o.jsx)("div",{className:"mb-2",children:(0,o.jsxs)("div",{className:"flex items-center gap-3 p-3 bg-gray-50 rounded-lg border border-gray-200",children:[(0,o.jsx)("div",{className:"relative inline-block",children:e2.name.toLowerCase().endsWith(".pdf")?(0,o.jsx)("div",{className:"w-10 h-10 rounded-md bg-red-500 flex items-center justify-center",children:(0,o.jsx)(Q.Z,{style:{fontSize:"16px",color:"white"}})}):(0,o.jsx)("img",{src:e6||"",alt:"Upload preview",className:"w-10 h-10 rounded-md border border-gray-200 object-cover"})}),(0,o.jsxs)("div",{className:"flex-1 min-w-0",children:[(0,o.jsx)("div",{className:"text-sm font-medium text-gray-900 truncate",children:e2.name}),(0,o.jsx)("div",{className:"text-xs text-gray-500",children:e2.name.toLowerCase().endsWith(".pdf")?"PDF":"Image"})]}),(0,o.jsx)("button",{className:"flex items-center justify-center w-6 h-6 text-gray-400 hover:text-gray-600 hover:bg-gray-200 rounded-full transition-colors",onClick:tv,children:(0,o.jsx)(e_.Z,{style:{fontSize:"12px"}})})]})}),(0,o.jsxs)("div",{className:"flex items-center gap-2",children:[(0,o.jsxs)("div",{className:"flex items-center flex-1 bg-white border border-gray-300 rounded-xl px-3 py-1 min-h-[44px]",children:[(0,o.jsxs)("div",{className:"flex-shrink-0 mr-2",children:[eh===C.KP.RESPONSES&&!eQ&&(0,o.jsx)(W,{responsesUploadedImage:eQ,responsesImagePreviewUrl:e1,onImageUpload:e=>(e0(e),e4(URL.createObjectURL(e)),!1),onRemoveImage:tb}),eh===C.KP.CHAT&&!e2&&(0,o.jsx)(es,{chatUploadedImage:e2,chatImagePreviewUrl:e6,onImageUpload:e=>(e3(e),e5(URL.createObjectURL(e)),!1),onRemoveImage:tv})]}),(0,o.jsx)(eZ,{value:V,onChange:e=>$(e.target.value),onKeyDown:e=>{"Enter"!==e.key||e.shiftKey||(e.preventDefault(),ty())},placeholder:eh===C.KP.CHAT||eh===C.KP.RESPONSES||eh===C.KP.ANTHROPIC_MESSAGES?"Type your message... (Shift+Enter for new line)":eh===C.KP.IMAGE_EDITS?"Describe how you want to edit the image...":"Describe the image you want to generate...",disabled:eU,className:"flex-1",autoSize:{minRows:1,maxRows:4},style:{resize:"none",border:"none",boxShadow:"none",background:"transparent",padding:"4px 0",fontSize:"14px",lineHeight:"20px"}}),(0,o.jsx)(l.zx,{onClick:ty,disabled:eU||!V.trim(),className:"flex-shrink-0 ml-2 !w-8 !h-8 !min-w-8 !p-0 !rounded-full !bg-blue-600 hover:!bg-blue-700 disabled:!bg-gray-300 !border-none !text-white disabled:!text-gray-500 !flex !items-center !justify-center",children:(0,o.jsx)(eT.Z,{style:{fontSize:"14px"}})})]}),eU&&(0,o.jsx)(l.zx,{onClick:()=>{eL.current&&(eL.current.abort(),eL.current=null,eR(!1),v.Z.info("Request cancelled"))},className:"bg-red-50 hover:bg-red-100 text-red-600 border-red-200",icon:e_.Z,children:"Cancel"})]})]})]})]})}),(0,o.jsxs)(u.Z,{title:"Generated Code",visible:e8,onCancel:()=>e7(!1),footer:null,width:800,children:[(0,o.jsxs)("div",{className:"flex justify-between items-end my-4",children:[(0,o.jsxs)("div",{children:[(0,o.jsx)(l.xv,{className:"font-medium block mb-1 text-gray-700",children:"SDK Type"}),(0,o.jsx)(m.default,{value:tt,onChange:e=>ts(e),style:{width:150},options:[{value:"openai",label:"OpenAI SDK"},{value:"azure",label:"Azure SDK"}]})]}),(0,o.jsx)(x.ZP,{onClick:()=>{navigator.clipboard.writeText(e9),v.Z.success("Copied to clipboard!")},children:"Copy to Clipboard"})]}),(0,o.jsx)(I.Z,{language:"python",style:_.Z,wrapLines:!0,wrapLongLines:!0,className:"rounded-md",customStyle:{maxHeight:"60vh",overflowY:"auto"},children:e9})]}),"custom"===F&&(0,o.jsx)(u.Z,{title:"Select MCP Tool",visible:f,onCancel:()=>S(!1),onOk:()=>{S(!1),v.Z.success("MCP tool selection updated")},width:800,children:z?(0,o.jsx)("div",{className:"flex justify-center items-center py-8",children:(0,o.jsx)(p.Z,{indicator:(0,o.jsx)(eb.Z,{style:{fontSize:24},spin:!0})})}):(0,o.jsxs)("div",{className:"space-y-4",children:[(0,o.jsx)(l.xv,{className:"text-gray-600 block mb-4",children:"Select the MCP tools you want to use in your conversation."}),(0,o.jsx)(m.default,{mode:"multiple",style:{width:"100%"},placeholder:"Select MCP tools",value:M,onChange:e=>D(e),optionLabelProp:"label",allowClear:!0,maxTagCount:"responsive",children:T.map(e=>(0,o.jsx)(m.default.Option,{value:e.name,label:(0,o.jsx)("div",{className:"font-medium",children:e.name}),children:(0,o.jsxs)("div",{className:"flex flex-col py-1",children:[(0,o.jsx)("span",{className:"font-medium",children:e.name}),(0,o.jsx)("span",{className:"text-xs text-gray-500 mt-1",children:e.description})]})},e.name))})]})})]})}}}]);
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/1160-3efb81c958413447.js b/litellm/proxy/_experimental/out/_next/static/chunks/1160-3efb81c958413447.js
new file mode 100644
index 00000000000..2a26d9fe08b
--- /dev/null
+++ b/litellm/proxy/_experimental/out/_next/static/chunks/1160-3efb81c958413447.js
@@ -0,0 +1 @@
+(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[1160],{69993:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),i={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M300 328a60 60 0 10120 0 60 60 0 10-120 0zM852 64H172c-17.7 0-32 14.3-32 32v660c0 17.7 14.3 32 32 32h680c17.7 0 32-14.3 32-32V96c0-17.7-14.3-32-32-32zm-32 660H204V128h616v596zM604 328a60 60 0 10120 0 60 60 0 10-120 0zm250.2 556H169.8c-16.5 0-29.8 14.3-29.8 32v36c0 4.4 3.3 8 7.4 8h729.1c4.1 0 7.4-3.6 7.4-8v-36c.1-17.7-13.2-32-29.7-32zM664 508H360c-4.4 0-8 3.6-8 8v60c0 4.4 3.6 8 8 8h304c4.4 0 8-3.6 8-8v-60c0-4.4-3.6-8-8-8z"}}]},name:"robot",theme:"outlined"},a=n(55015),c=o.forwardRef(function(e,t){return o.createElement(a.Z,(0,r.Z)({},e,{ref:t,icon:i}))})},14042:function(e,t,n){"use strict";n.d(t,{Z:function(){return eM}});var r=n(5853),o=n(7084),i=n(26898),a=n(97324),c=n(1153),l=n(2265),s=n(60474),u=n(47625),p=n(93765),f=n(86757),d=n.n(f),y=n(9841),m=n(81889),h=n(87602),v=n(82944),b=["points","className","baseLinePoints","connectNulls"];function g(){return(g=Object.assign?Object.assign.bind():function(e){for(var t=1;te.length)&&(t=e.length);for(var n=0,r=Array(t);n0&&void 0!==arguments[0]?arguments[0]:[],t=[[]];return e.forEach(function(e){x(e)?t[t.length-1].push(e):t[t.length-1].length>0&&t.push([])}),x(e[0])&&t[t.length-1].push(e[0]),t[t.length-1].length<=0&&(t=t.slice(0,-1)),t},j=function(e,t){var n=k(e);t&&(n=[n.reduce(function(e,t){return[].concat(A(e),A(t))},[])]);var r=n.map(function(e){return e.reduce(function(e,t,n){return"".concat(e).concat(0===n?"M":"L").concat(t.x,",").concat(t.y)},"")}).join("");return 1===n.length?"".concat(r,"Z"):r},w=function(e,t,n){var r=j(e,n);return"".concat("Z"===r.slice(-1)?r.slice(0,-1):r,"L").concat(j(t.reverse(),n).slice(1))},P=function(e){var t=e.points,n=e.className,r=e.baseLinePoints,o=e.connectNulls,i=function(e,t){if(null==e)return{};var n,r,o=function(e,t){if(null==e)return{};var n,r,o={},i=Object.keys(e);for(r=0;r=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0)&&Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}(e,b);if(!t||!t.length)return null;var a=(0,h.Z)("recharts-polygon",n);if(r&&r.length){var c=i.stroke&&"none"!==i.stroke,s=w(t,r,o);return l.createElement("g",{className:a},l.createElement("path",g({},(0,v.L6)(i,!0),{fill:"Z"===s.slice(-1)?i.fill:"none",stroke:"none",d:s})),c?l.createElement("path",g({},(0,v.L6)(i,!0),{fill:"none",d:j(t,o)})):null,c?l.createElement("path",g({},(0,v.L6)(i,!0),{fill:"none",d:j(r,o)})):null)}var u=j(t,o);return l.createElement("path",g({},(0,v.L6)(i,!0),{fill:"Z"===u.slice(-1)?i.fill:"none",className:a,d:u}))},E=n(58811),S=n(41637),T=n(39206);function R(e){return(R="function"==typeof Symbol&&"symbol"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&"function"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?"symbol":typeof e})(e)}function L(){return(L=Object.assign?Object.assign.bind():function(e){for(var t=1;t1e-5?"outer"===t?"start":"end":n<-.00001?"outer"===t?"end":"start":"middle"}},{key:"renderAxisLine",value:function(){var e=this.props,t=e.cx,n=e.cy,r=e.radius,o=e.axisLine,i=e.axisLineType,a=I(I({},(0,v.L6)(this.props,!1)),{},{fill:"none"},(0,v.L6)(o,!1));if("circle"===i)return l.createElement(m.o,L({className:"recharts-polar-angle-axis-line"},a,{cx:t,cy:n,r:r}));var c=this.props.ticks.map(function(e){return(0,T.op)(t,n,r,e.coordinate)});return l.createElement(P,L({className:"recharts-polar-angle-axis-line"},a,{points:c}))}},{key:"renderTicks",value:function(){var e=this,t=this.props,n=t.ticks,r=t.tick,o=t.tickLine,a=t.tickFormatter,c=t.stroke,s=(0,v.L6)(this.props,!1),u=(0,v.L6)(r,!1),p=I(I({},s),{},{fill:"none"},(0,v.L6)(o,!1)),f=n.map(function(t,n){var f=e.getTickLineCoord(t),d=I(I(I({textAnchor:e.getTickTextAnchor(t)},s),{},{stroke:"none",fill:c},u),{},{index:n,payload:t,x:f.x2,y:f.y2});return l.createElement(y.m,L({className:"recharts-polar-angle-axis-tick",key:"tick-".concat(t.coordinate)},(0,S.bw)(e.props,t,n)),o&&l.createElement("line",L({className:"recharts-polar-angle-axis-tick-line"},p,f)),r&&i.renderTickItem(r,d,a?a(t.value,n):t.value))});return l.createElement(y.m,{className:"recharts-polar-angle-axis-ticks"},f)}},{key:"render",value:function(){var e=this.props,t=e.ticks,n=e.radius,r=e.axisLine;return!(n<=0)&&t&&t.length?l.createElement(y.m,{className:"recharts-polar-angle-axis"},r&&this.renderAxisLine(),this.renderTicks()):null}}],r=[{key:"renderTickItem",value:function(e,t,n){return l.isValidElement(e)?l.cloneElement(e,t):d()(e)?e(t):l.createElement(E.x,L({},t,{className:"recharts-polar-angle-axis-tick-value"}),n)}}],n&&C(i.prototype,n),r&&C(i,r),Object.defineProperty(i,"prototype",{writable:!1}),i}(l.PureComponent);_(B,"displayName","PolarAngleAxis"),_(B,"axisType","angleAxis"),_(B,"defaultProps",{type:"category",angleAxisId:0,scale:"auto",cx:0,cy:0,orientation:"outer",axisLine:!0,tickLine:!0,tickSize:8,tick:!0,hide:!1,allowDuplicatedCategory:!0});var K=n(35802),V=n.n(K),z=n(37891),$=n.n(z),H=n(26680),q=["cx","cy","angle","ticks","axisLine"],G=["ticks","tick","angle","tickFormatter","stroke"];function Y(e){return(Y="function"==typeof Symbol&&"symbol"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&"function"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?"symbol":typeof e})(e)}function U(){return(U=Object.assign?Object.assign.bind():function(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0)&&Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}function Q(e,t){for(var n=0;n0?ec()(e,"paddingAngle",0):0;if(n){var c=(0,eh.k4)(n.endAngle-n.startAngle,e.endAngle-e.startAngle),l=ek(ek({},e),{},{startAngle:i+a,endAngle:i+c(r)+a});o.push(l),i=l.endAngle}else{var s=e.endAngle,p=e.startAngle,f=(0,eh.k4)(0,s-p)(r),d=ek(ek({},e),{},{startAngle:i+a,endAngle:i+f+a});o.push(d),i=d.endAngle}}),l.createElement(y.m,null,e.renderSectorsStatically(o))})}},{key:"attachKeyboardHandlers",value:function(e){var t=this;e.onkeydown=function(e){if(!e.altKey)switch(e.key){case"ArrowLeft":var n=++t.state.sectorToFocus%t.sectorRefs.length;t.sectorRefs[n].focus(),t.setState({sectorToFocus:n});break;case"ArrowRight":var r=--t.state.sectorToFocus<0?t.sectorRefs.length-1:t.state.sectorToFocus%t.sectorRefs.length;t.sectorRefs[r].focus(),t.setState({sectorToFocus:r});break;case"Escape":t.sectorRefs[t.state.sectorToFocus].blur(),t.setState({sectorToFocus:0})}}}},{key:"renderSectors",value:function(){var e=this.props,t=e.sectors,n=e.isAnimationActive,r=this.state.prevSectors;return n&&t&&t.length&&(!r||!es()(r,t))?this.renderSectorsWithAnimation():this.renderSectorsStatically(t)}},{key:"componentDidMount",value:function(){this.pieRef&&this.attachKeyboardHandlers(this.pieRef)}},{key:"render",value:function(){var e=this,t=this.props,n=t.hide,r=t.sectors,o=t.className,i=t.label,a=t.cx,c=t.cy,s=t.innerRadius,u=t.outerRadius,p=t.isAnimationActive,f=this.state.isAnimationFinished;if(n||!r||!r.length||!(0,eh.hj)(a)||!(0,eh.hj)(c)||!(0,eh.hj)(s)||!(0,eh.hj)(u))return null;var d=(0,h.Z)("recharts-pie",o);return l.createElement(y.m,{tabIndex:this.props.rootTabIndex,className:d,ref:function(t){e.pieRef=t}},this.renderSectors(),i&&this.renderLabels(r),H._.renderCallByParent(this.props,null,!1),(!p||f)&&ed.e.renderCallByParent(this.props,r,!1))}}],r=[{key:"getDerivedStateFromProps",value:function(e,t){return t.prevIsAnimationActive!==e.isAnimationActive?{prevIsAnimationActive:e.isAnimationActive,prevAnimationId:e.animationId,curSectors:e.sectors,prevSectors:[],isAnimationFinished:!0}:e.isAnimationActive&&e.animationId!==t.prevAnimationId?{prevAnimationId:e.animationId,curSectors:e.sectors,prevSectors:t.curSectors,isAnimationFinished:!0}:e.sectors!==t.curSectors?{curSectors:e.sectors,isAnimationFinished:!0}:null}},{key:"getTextAnchor",value:function(e,t){return e>t?"start":e=360?A:A-1)*u,x=i.reduce(function(e,t){var n=(0,ev.F$)(t,g,0);return e+((0,eh.hj)(n)?n:0)},0);return x>0&&(t=i.map(function(e,t){var r,o=(0,ev.F$)(e,g,0),i=(0,ev.F$)(e,f,t),a=((0,eh.hj)(o)?o:0)/x,s=(r=t?n.endAngle+(0,eh.uY)(v)*u*(0!==o?1:0):l)+(0,eh.uY)(v)*((0!==o?m:0)+a*O),p=(r+s)/2,d=(h.innerRadius+h.outerRadius)/2,b=[{name:i,value:o,payload:e,dataKey:g,type:y}],A=(0,T.op)(h.cx,h.cy,d,p);return n=ek(ek(ek({percent:a,cornerRadius:c,name:i,tooltipPayload:b,midAngle:p,middleRadius:d,tooltipPosition:A},e),h),{},{value:(0,ev.F$)(e,g),startAngle:r,endAngle:s,payload:e,paddingAngle:(0,eh.uY)(v)*u})})),ek(ek({},h),{},{sectors:t,data:i})});var eL=(0,p.z)({chartName:"PieChart",GraphicalChild:eR,validateTooltipEventTypes:["item"],defaultTooltipEventType:"item",legendContent:"children",axisComponents:[{axisType:"angleAxis",AxisComp:B},{axisType:"radiusAxis",AxisComp:eo}],formatAxisMap:T.t9,defaultProps:{layout:"centric",startAngle:0,endAngle:360,cx:"50%",cy:"50%",innerRadius:0,outerRadius:"80%"}}),eN=n(8147),eI=n(69448),eC=n(98593);let eD=e=>{let{active:t,payload:n,valueFormatter:r}=e;if(t&&(null==n?void 0:n[0])){let e=null==n?void 0:n[0];return l.createElement(eC.$B,null,l.createElement("div",{className:(0,a.q)("px-4 py-2")},l.createElement(eC.zX,{value:r(e.value),name:e.name,color:e.payload.color})))}return null},eF=(e,t)=>e.map((e,n)=>{let r=ne||t((0,c.vP)(n.map(e=>e[r]))),eZ=e=>{let{cx:t,cy:n,innerRadius:r,outerRadius:o,startAngle:i,endAngle:a,className:c}=e;return l.createElement("g",null,l.createElement(s.L,{cx:t,cy:n,innerRadius:r,outerRadius:o,startAngle:i,endAngle:a,className:c,fill:"",opacity:.3,style:{outline:"none"}}))},eM=l.forwardRef((e,t)=>{let{data:n=[],category:s="value",index:p="name",colors:f=i.s,variant:d="donut",valueFormatter:y=c.Cj,label:m,showLabel:h=!0,animationDuration:v=900,showAnimation:b=!1,showTooltip:g=!0,noDataText:A,onValueChange:O,customTooltip:x,className:k}=e,j=(0,r._T)(e,["data","category","index","colors","variant","valueFormatter","label","showLabel","animationDuration","showAnimation","showTooltip","noDataText","onValueChange","customTooltip","className"]),w="donut"==d,P=e_(m,y,n,s),[E,S]=l.useState(void 0),T=!!O;return(0,l.useEffect)(()=>{let e=document.querySelectorAll(".recharts-pie-sector");e&&e.forEach(e=>{e.setAttribute("style","outline: none")})},[E]),l.createElement("div",Object.assign({ref:t,className:(0,a.q)("w-full h-40",k)},j),l.createElement(u.h,{className:"h-full w-full"},(null==n?void 0:n.length)?l.createElement(eL,{onClick:T&&E?()=>{S(void 0),null==O||O(null)}:void 0,margin:{top:0,left:0,right:0,bottom:0}},h&&w?l.createElement("text",{className:(0,a.q)("fill-tremor-content-emphasis","dark:fill-dark-tremor-content-emphasis"),x:"50%",y:"50%",textAnchor:"middle",dominantBaseline:"middle"},P):null,l.createElement(eR,{className:(0,a.q)("stroke-tremor-background dark:stroke-dark-tremor-background",O?"cursor-pointer":"cursor-default"),data:eF(n,f),cx:"50%",cy:"50%",startAngle:90,endAngle:-270,innerRadius:w?"75%":"0%",outerRadius:"100%",stroke:"",strokeLinejoin:"round",dataKey:s,nameKey:p,isAnimationActive:b,animationDuration:v,onClick:function(e,t,n){n.stopPropagation(),T&&(E===t?(S(void 0),null==O||O(null)):(S(t),null==O||O(Object.assign({eventType:"slice"},e.payload.payload))))},activeIndex:E,inactiveShape:eZ,style:{outline:"none"}}),l.createElement(eN.u,{wrapperStyle:{outline:"none"},isAnimationActive:!1,content:g?e=>{var t;let{active:n,payload:r}=e;return x?l.createElement(x,{payload:null==r?void 0:r.map(e=>{var t,n,i;return Object.assign(Object.assign({},e),{color:null!==(i=null===(n=null===(t=null==r?void 0:r[0])||void 0===t?void 0:t.payload)||void 0===n?void 0:n.color)&&void 0!==i?i:o.fr.Gray})}),active:n,label:null===(t=null==r?void 0:r[0])||void 0===t?void 0:t.name}):l.createElement(eD,{active:n,payload:r,valueFormatter:y})}:l.createElement(l.Fragment,null)})):l.createElement(eI.Z,{noDataText:A})))});eM.displayName="DonutChart"},7366:function(e,t,n){"use strict";n.d(t,{Z:function(){return s}});var r=n(41154),o=n(25721),i=n(55463),a=n(99735),c=n(7656),l=n(47869);function s(e,t){if((0,c.Z)(2,arguments),!t||"object"!==(0,r.Z)(t))return new Date(NaN);var n=t.years?(0,l.Z)(t.years):0,s=t.months?(0,l.Z)(t.months):0,u=t.weeks?(0,l.Z)(t.weeks):0,p=t.days?(0,l.Z)(t.days):0,f=t.hours?(0,l.Z)(t.hours):0,d=t.minutes?(0,l.Z)(t.minutes):0,y=t.seconds?(0,l.Z)(t.seconds):0,m=(0,a.Z)(e),h=s||n?(0,i.Z)(m,s+12*n):m;return new Date((p||u?(0,o.Z)(h,p+7*u):h).getTime()+1e3*(y+60*(d+60*f)))}},35802:function(e,t,n){var r=n(67646),o=n(58905),i=n(88157);e.exports=function(e,t){return e&&e.length?r(e,i(t,2),o):void 0}},37891:function(e,t,n){var r=n(67646),o=n(88157),i=n(20121);e.exports=function(e,t){return e&&e.length?r(e,o(t,2),i):void 0}},58710:function(e,t,n){"use strict";var r=n(2265);let o=r.forwardRef(function(e,t){return r.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",fill:"none",viewBox:"0 0 24 24",strokeWidth:2,stroke:"currentColor","aria-hidden":"true",ref:t},e),r.createElement("path",{strokeLinecap:"round",strokeLinejoin:"round",d:"M12 8v4l3 3m6-3a9 9 0 11-18 0 9 9 0 0118 0z"}))});t.Z=o}}]);
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/117-a0da667066d322b6.js b/litellm/proxy/_experimental/out/_next/static/chunks/117-a0da667066d322b6.js
deleted file mode 100644
index 21cf536ca55..00000000000
--- a/litellm/proxy/_experimental/out/_next/static/chunks/117-a0da667066d322b6.js
+++ /dev/null
@@ -1,2 +0,0 @@
-(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[117],{65157:function(e,t){"use strict";function n(){return""}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"getDeploymentIdQueryOrEmptyString",{enumerable:!0,get:function(){return n}})},91572:function(){"trimStart"in String.prototype||(String.prototype.trimStart=String.prototype.trimLeft),"trimEnd"in String.prototype||(String.prototype.trimEnd=String.prototype.trimRight),"description"in Symbol.prototype||Object.defineProperty(Symbol.prototype,"description",{configurable:!0,get:function(){var e=/\((.*)\)/.exec(this.toString());return e?e[1]:void 0}}),Array.prototype.flat||(Array.prototype.flat=function(e,t){return t=this.concat.apply([],this),e>1&&t.some(Array.isArray)?t.flat(e-1):t},Array.prototype.flatMap=function(e,t){return this.map(e,t).flat()}),Promise.prototype.finally||(Promise.prototype.finally=function(e){if("function"!=typeof e)return this.then(e,e);var t=this.constructor||Promise;return this.then(function(n){return t.resolve(e()).then(function(){return n})},function(n){return t.resolve(e()).then(function(){throw n})})}),Object.fromEntries||(Object.fromEntries=function(e){return Array.from(e).reduce(function(e,t){return e[t[0]]=t[1],e},{})}),Array.prototype.at||(Array.prototype.at=function(e){var t=Math.trunc(e)||0;if(t<0&&(t+=this.length),!(t<0||t>=this.length))return this[t]}),Object.hasOwn||(Object.hasOwn=function(e,t){if(null==e)throw TypeError("Cannot convert undefined or null to object");return Object.prototype.hasOwnProperty.call(Object(e),t)}),"canParse"in URL||(URL.canParse=function(e,t){try{return new URL(e,t),!0}catch(e){return!1}})},1634:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"addBasePath",{enumerable:!0,get:function(){return u}});let r=n(68498),o=n(33068);function u(e,t){return(0,o.normalizePathTrailingSlash)((0,r.addPathPrefix)(e,""))}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},75266:function(e,t){"use strict";function n(e){var t,n;t=self.__next_s,n=()=>{e()},t&&t.length?t.reduce((e,t)=>{let[n,r]=t;return e.then(()=>new Promise((e,t)=>{let o=document.createElement("script");if(r)for(let e in r)"children"!==e&&o.setAttribute(e,r[e]);n?(o.src=n,o.onload=()=>e(),o.onerror=t):r&&(o.innerHTML=r.children,setTimeout(e)),document.head.appendChild(o)}))},Promise.resolve()).catch(e=>{console.error(e)}).then(()=>{n()}):n()}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"appBootstrap",{enumerable:!0,get:function(){return n}}),window.next={version:"14.2.30",appDir:!0},("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},83079:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"callServer",{enumerable:!0,get:function(){return o}});let r=n(12846);async function o(e,t){let n=(0,r.getServerActionDispatcher)();if(!n)throw Error("Invariant: missing action dispatcher.");return new Promise((r,o)=>{n({actionId:e,actionArgs:t,resolve:r,reject:o})})}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},92304:function(e,t,n){"use strict";let r,o;Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"hydrate",{enumerable:!0,get:function(){return x}});let u=n(47043),l=n(53099),a=n(57437);n(91572);let i=u._(n(34040)),c=l._(n(2265)),s=n(6671),f=n(48701),d=u._(n(61404)),p=n(83079),h=n(89721),y=n(2103);n(70647);let _=window.console.error;window.console.error=function(){for(var e=arguments.length,t=Array(e),n=0;n{if((0,h.isNextRouterError)(e.error)){e.preventDefault();return}});let v=document,b=new TextEncoder,g=!1,m=!1,R=null;function P(e){if(0===e[0])r=[];else if(1===e[0]){if(!r)throw Error("Unexpected server data: missing bootstrap script.");o?o.enqueue(b.encode(e[1])):r.push(e[1])}else 2===e[0]&&(R=e[1])}let j=function(){o&&!m&&(o.close(),m=!0,r=void 0),g=!0};"loading"===document.readyState?document.addEventListener("DOMContentLoaded",j,!1):j();let O=self.__next_f=self.__next_f||[];O.forEach(P),O.push=P;let S=new ReadableStream({start(e){r&&(r.forEach(t=>{e.enqueue(b.encode(t))}),g&&!m&&(e.close(),m=!0,r=void 0)),o=e}}),E=(0,s.createFromReadableStream)(S,{callServer:p.callServer});function w(){return(0,c.use)(E)}let T=c.default.StrictMode;function M(e){let{children:t}=e;return t}function x(){let e=(0,y.createMutableActionQueue)(),t=(0,a.jsx)(T,{children:(0,a.jsx)(f.HeadManagerContext.Provider,{value:{appDir:!0},children:(0,a.jsx)(y.ActionQueueContext.Provider,{value:e,children:(0,a.jsx)(M,{children:(0,a.jsx)(w,{})})})})}),n=window.__next_root_layout_missing_tags,r=!!(null==n?void 0:n.length),o={onRecoverableError:d.default};"__next_error__"===document.documentElement.id||r?i.default.createRoot(v,o).render(t):c.default.startTransition(()=>i.default.hydrateRoot(v,t,{...o,formState:R}))}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},54278:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),n(19506),(0,n(75266).appBootstrap)(()=>{let{hydrate:e}=n(92304);n(12846),n(4707),e()}),("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},19506:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),n(65157);{let e=n.u;n.u=function(){for(var t=arguments.length,n=Array(t),r=0;r(l(function(){var e;let t=document.getElementsByName(u)[0];if(null==t?void 0:null==(e=t.shadowRoot)?void 0:e.childNodes[0])return t.shadowRoot.childNodes[0];{let e=document.createElement(u);e.style.cssText="position:absolute";let t=document.createElement("div");return t.ariaLive="assertive",t.id="__next-route-announcer__",t.role="alert",t.style.cssText="position:absolute;border:0;height:1px;margin:-1px;padding:0;width:1px;clip:rect(0 0 0 0);overflow:hidden;white-space:nowrap;word-wrap:normal",e.attachShadow({mode:"open"}).appendChild(t),document.body.appendChild(e),t}}()),()=>{let e=document.getElementsByTagName(u)[0];(null==e?void 0:e.isConnected)&&document.body.removeChild(e)}),[]);let[a,i]=(0,r.useState)(""),c=(0,r.useRef)();return(0,r.useEffect)(()=>{let e="";if(document.title)e=document.title;else{let t=document.querySelector("h1");t&&(e=t.innerText||t.textContent||"")}void 0!==c.current&&c.current!==e&&i(e),c.current=e},[t]),n?(0,o.createPortal)(a,n):null}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},6866:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{ACTION:function(){return r},FLIGHT_PARAMETERS:function(){return i},NEXT_DID_POSTPONE_HEADER:function(){return s},NEXT_ROUTER_PREFETCH_HEADER:function(){return u},NEXT_ROUTER_STATE_TREE:function(){return o},NEXT_RSC_UNION_QUERY:function(){return c},NEXT_URL:function(){return l},RSC_CONTENT_TYPE_HEADER:function(){return a},RSC_HEADER:function(){return n}});let n="RSC",r="Next-Action",o="Next-Router-State-Tree",u="Next-Router-Prefetch",l="Next-Url",a="text/x-component",i=[[n],[o],[u]],c="_rsc",s="x-nextjs-postponed";("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},12846:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{createEmptyCacheNode:function(){return C},default:function(){return I},getServerActionDispatcher:function(){return E},urlToUrlWithoutFlightMarker:function(){return T}});let r=n(53099),o=n(57437),u=r._(n(2265)),l=n(61956),a=n(24673),i=n(33456),c=n(79060),s=n(47744),f=n(61060),d=n(82952),p=n(86146),h=n(1634),y=n(6495),_=n(4123),v=n(39320),b=n(38137),g=n(6866),m=n(35076),R=n(11283),P=n(84541),j="undefined"==typeof window,O=j?null:new Map,S=null;function E(){return S}let w={};function T(e){let t=new URL(e,location.origin);if(t.searchParams.delete(g.NEXT_RSC_UNION_QUERY),t.pathname.endsWith(".txt")){let{pathname:e}=t,n=e.endsWith("/index.txt")?10:4;t.pathname=e.slice(0,-n)}return t}function M(e){return e.origin!==window.location.origin}function x(e){let{appRouterState:t,sync:n}=e;return(0,u.useInsertionEffect)(()=>{let{tree:e,pushRef:r,canonicalUrl:o}=t,u={...r.preserveCustomHistoryState?window.history.state:{},__NA:!0,__PRIVATE_NEXTJS_INTERNALS_TREE:e};r.pendingPush&&(0,i.createHrefFromUrl)(new URL(window.location.href))!==o?(r.pendingPush=!1,window.history.pushState(u,"",o)):window.history.replaceState(u,"",o),n(t)},[t,n]),null}function C(){return{lazyData:null,rsc:null,prefetchRsc:null,head:null,prefetchHead:null,parallelRoutes:new Map,lazyDataResolved:!1,loading:null}}function A(e){null==e&&(e={});let t=window.history.state,n=null==t?void 0:t.__NA;n&&(e.__NA=n);let r=null==t?void 0:t.__PRIVATE_NEXTJS_INTERNALS_TREE;return r&&(e.__PRIVATE_NEXTJS_INTERNALS_TREE=r),e}function N(e){let{headCacheNode:t}=e,n=null!==t?t.head:null,r=null!==t?t.prefetchHead:null,o=null!==r?r:n;return(0,u.useDeferredValue)(n,o)}function D(e){let t,{buildId:n,initialHead:r,initialTree:i,urlParts:f,initialSeedData:g,couldBeIntercepted:E,assetPrefix:T,missingSlots:C}=e,D=(0,u.useMemo)(()=>(0,d.createInitialRouterState)({buildId:n,initialSeedData:g,urlParts:f,initialTree:i,initialParallelRoutes:O,location:j?null:window.location,initialHead:r,couldBeIntercepted:E}),[n,g,f,i,r,E]),[I,U,k]=(0,s.useReducerWithReduxDevtools)(D);(0,u.useEffect)(()=>{O=null},[]);let{canonicalUrl:F}=(0,s.useUnwrapState)(I),{searchParams:L,pathname:H}=(0,u.useMemo)(()=>{let e=new URL(F,"undefined"==typeof window?"http://n":window.location.href);return{searchParams:e.searchParams,pathname:(0,R.hasBasePath)(e.pathname)?(0,m.removeBasePath)(e.pathname):e.pathname}},[F]),$=(0,u.useCallback)(e=>{let{previousTree:t,serverResponse:n}=e;(0,u.startTransition)(()=>{U({type:a.ACTION_SERVER_PATCH,previousTree:t,serverResponse:n})})},[U]),G=(0,u.useCallback)((e,t,n)=>{let r=new URL((0,h.addBasePath)(e),location.href);return U({type:a.ACTION_NAVIGATE,url:r,isExternalUrl:M(r),locationSearch:location.search,shouldScroll:null==n||n,navigateType:t})},[U]);S=(0,u.useCallback)(e=>{(0,u.startTransition)(()=>{U({...e,type:a.ACTION_SERVER_ACTION})})},[U]);let z=(0,u.useMemo)(()=>({back:()=>window.history.back(),forward:()=>window.history.forward(),prefetch:(e,t)=>{let n;if(!(0,p.isBot)(window.navigator.userAgent)){try{n=new URL((0,h.addBasePath)(e),window.location.href)}catch(t){throw Error("Cannot prefetch '"+e+"' because it cannot be converted to a URL.")}M(n)||(0,u.startTransition)(()=>{var e;U({type:a.ACTION_PREFETCH,url:n,kind:null!=(e=null==t?void 0:t.kind)?e:a.PrefetchKind.FULL})})}},replace:(e,t)=>{void 0===t&&(t={}),(0,u.startTransition)(()=>{var n;G(e,"replace",null==(n=t.scroll)||n)})},push:(e,t)=>{void 0===t&&(t={}),(0,u.startTransition)(()=>{var n;G(e,"push",null==(n=t.scroll)||n)})},refresh:()=>{(0,u.startTransition)(()=>{U({type:a.ACTION_REFRESH,origin:window.location.origin})})},fastRefresh:()=>{throw Error("fastRefresh can only be used in development mode. Please use refresh instead.")}}),[U,G]);(0,u.useEffect)(()=>{window.next&&(window.next.router=z)},[z]),(0,u.useEffect)(()=>{function e(e){var t;e.persisted&&(null==(t=window.history.state)?void 0:t.__PRIVATE_NEXTJS_INTERNALS_TREE)&&(w.pendingMpaPath=void 0,U({type:a.ACTION_RESTORE,url:new URL(window.location.href),tree:window.history.state.__PRIVATE_NEXTJS_INTERNALS_TREE}))}return window.addEventListener("pageshow",e),()=>{window.removeEventListener("pageshow",e)}},[U]);let{pushRef:B}=(0,s.useUnwrapState)(I);if(B.mpaNavigation){if(w.pendingMpaPath!==F){let e=window.location;B.pendingPush?e.assign(F):e.replace(F),w.pendingMpaPath=F}(0,u.use)(b.unresolvedThenable)}(0,u.useEffect)(()=>{let e=window.history.pushState.bind(window.history),t=window.history.replaceState.bind(window.history),n=e=>{var t;let n=window.location.href,r=null==(t=window.history.state)?void 0:t.__PRIVATE_NEXTJS_INTERNALS_TREE;(0,u.startTransition)(()=>{U({type:a.ACTION_RESTORE,url:new URL(null!=e?e:n,n),tree:r})})};window.history.pushState=function(t,r,o){return(null==t?void 0:t.__NA)||(null==t?void 0:t._N)||(t=A(t),o&&n(o)),e(t,r,o)},window.history.replaceState=function(e,r,o){return(null==e?void 0:e.__NA)||(null==e?void 0:e._N)||(e=A(e),o&&n(o)),t(e,r,o)};let r=e=>{let{state:t}=e;if(t){if(!t.__NA){window.location.reload();return}(0,u.startTransition)(()=>{U({type:a.ACTION_RESTORE,url:new URL(window.location.href),tree:t.__PRIVATE_NEXTJS_INTERNALS_TREE})})}};return window.addEventListener("popstate",r),()=>{window.history.pushState=e,window.history.replaceState=t,window.removeEventListener("popstate",r)}},[U]);let{cache:W,tree:K,nextUrl:V,focusAndScrollRef:Y}=(0,s.useUnwrapState)(I),X=(0,u.useMemo)(()=>(0,v.findHeadInCache)(W,K[1]),[W,K]),q=(0,u.useMemo)(()=>(function e(t,n){for(let r of(void 0===n&&(n={}),Object.values(t[1]))){let t=r[0],o=Array.isArray(t),u=o?t[1]:t;!u||u.startsWith(P.PAGE_SEGMENT_KEY)||(o&&("c"===t[2]||"oc"===t[2])?n[t[0]]=t[1].split("/"):o&&(n[t[0]]=t[1]),n=e(r,n))}return n})(K),[K]);if(null!==X){let[e,n]=X;t=(0,o.jsx)(N,{headCacheNode:e},n)}else t=null;let J=(0,o.jsxs)(_.RedirectBoundary,{children:[t,W.rsc,(0,o.jsx)(y.AppRouterAnnouncer,{tree:K})]});return(0,o.jsxs)(o.Fragment,{children:[(0,o.jsx)(x,{appRouterState:(0,s.useUnwrapState)(I),sync:k}),(0,o.jsx)(c.PathParamsContext.Provider,{value:q,children:(0,o.jsx)(c.PathnameContext.Provider,{value:H,children:(0,o.jsx)(c.SearchParamsContext.Provider,{value:L,children:(0,o.jsx)(l.GlobalLayoutRouterContext.Provider,{value:{buildId:n,changeByServerResponse:$,tree:K,focusAndScrollRef:Y,nextUrl:V},children:(0,o.jsx)(l.AppRouterContext.Provider,{value:z,children:(0,o.jsx)(l.LayoutRouterContext.Provider,{value:{childNodes:W.parallelRoutes,tree:K,url:F,loading:W.loading},children:J})})})})})})]})}function I(e){let{globalErrorComponent:t,...n}=e;return(0,o.jsx)(f.ErrorBoundary,{errorComponent:t,children:(0,o.jsx)(D,{...n})})}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},96149:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"bailoutToClientRendering",{enumerable:!0,get:function(){return u}});let r=n(18993),o=n(51845);function u(e){let t=o.staticGenerationAsyncStorage.getStore();if((null==t||!t.forceStatic)&&(null==t?void 0:t.isStaticGeneration))throw new r.BailoutToCSRError(e)}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},19107:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"ClientPageRoot",{enumerable:!0,get:function(){return u}});let r=n(57437),o=n(54535);function u(e){let{Component:t,props:n}=e;return n.searchParams=(0,o.createDynamicallyTrackedSearchParams)(n.searchParams||{}),(0,r.jsx)(t,{...n})}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},61060:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{ErrorBoundary:function(){return h},ErrorBoundaryHandler:function(){return f},GlobalError:function(){return d},default:function(){return p}});let r=n(47043),o=n(57437),u=r._(n(2265)),l=n(35475),a=n(89721),i=n(51845),c={error:{fontFamily:'system-ui,"Segoe UI",Roboto,Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji"',height:"100vh",textAlign:"center",display:"flex",flexDirection:"column",alignItems:"center",justifyContent:"center"},text:{fontSize:"14px",fontWeight:400,lineHeight:"28px",margin:"0 8px"}};function s(e){let{error:t}=e,n=i.staticGenerationAsyncStorage.getStore();if((null==n?void 0:n.isRevalidate)||(null==n?void 0:n.isStaticGeneration))throw console.error(t),t;return null}class f extends u.default.Component{static getDerivedStateFromError(e){if((0,a.isNextRouterError)(e))throw e;return{error:e}}static getDerivedStateFromProps(e,t){return e.pathname!==t.previousPathname&&t.error?{error:null,previousPathname:e.pathname}:{error:t.error,previousPathname:e.pathname}}render(){return this.state.error?(0,o.jsxs)(o.Fragment,{children:[(0,o.jsx)(s,{error:this.state.error}),this.props.errorStyles,this.props.errorScripts,(0,o.jsx)(this.props.errorComponent,{error:this.state.error,reset:this.reset})]}):this.props.children}constructor(e){super(e),this.reset=()=>{this.setState({error:null})},this.state={error:null,previousPathname:this.props.pathname}}}function d(e){let{error:t}=e,n=null==t?void 0:t.digest;return(0,o.jsxs)("html",{id:"__next_error__",children:[(0,o.jsx)("head",{}),(0,o.jsxs)("body",{children:[(0,o.jsx)(s,{error:t}),(0,o.jsx)("div",{style:c.error,children:(0,o.jsxs)("div",{children:[(0,o.jsx)("h2",{style:c.text,children:"Application error: a "+(n?"server":"client")+"-side exception has occurred (see the "+(n?"server logs":"browser console")+" for more information)."}),n?(0,o.jsx)("p",{style:c.text,children:"Digest: "+n}):null]})})]})]})}let p=d;function h(e){let{errorComponent:t,errorStyles:n,errorScripts:r,children:u}=e,a=(0,l.usePathname)();return t?(0,o.jsx)(f,{pathname:a,errorComponent:t,errorStyles:n,errorScripts:r,children:u}):(0,o.jsx)(o.Fragment,{children:u})}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},46177:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{DynamicServerError:function(){return r},isDynamicServerError:function(){return o}});let n="DYNAMIC_SERVER_USAGE";class r extends Error{constructor(e){super("Dynamic server usage: "+e),this.description=e,this.digest=n}}function o(e){return"object"==typeof e&&null!==e&&"digest"in e&&"string"==typeof e.digest&&e.digest===n}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},89721:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"isNextRouterError",{enumerable:!0,get:function(){return u}});let r=n(98200),o=n(88968);function u(e){return e&&e.digest&&((0,o.isRedirectError)(e)||(0,r.isNotFoundError)(e))}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},4707:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"default",{enumerable:!0,get:function(){return S}});let r=n(47043),o=n(53099),u=n(57437),l=o._(n(2265)),a=r._(n(54887)),i=n(61956),c=n(44848),s=n(38137),f=n(61060),d=n(76015),p=n(7092),h=n(4123),y=n(80),_=n(73171),v=n(78505),b=n(28077),g=["bottom","height","left","right","top","width","x","y"];function m(e,t){let n=e.getBoundingClientRect();return n.top>=0&&n.top<=t}class R extends l.default.Component{componentDidMount(){this.handlePotentialScroll()}componentDidUpdate(){this.props.focusAndScrollRef.apply&&this.handlePotentialScroll()}render(){return this.props.children}constructor(...e){super(...e),this.handlePotentialScroll=()=>{let{focusAndScrollRef:e,segmentPath:t}=this.props;if(e.apply){var n;if(0!==e.segmentPaths.length&&!e.segmentPaths.some(e=>t.every((t,n)=>(0,d.matchSegment)(t,e[n]))))return;let r=null,o=e.hashFragment;if(o&&(r="top"===o?document.body:null!=(n=document.getElementById(o))?n:document.getElementsByName(o)[0]),r||(r="undefined"==typeof window?null:a.default.findDOMNode(this)),!(r instanceof Element))return;for(;!(r instanceof HTMLElement)||function(e){if(["sticky","fixed"].includes(getComputedStyle(e).position))return!0;let t=e.getBoundingClientRect();return g.every(e=>0===t[e])}(r);){if(null===r.nextElementSibling)return;r=r.nextElementSibling}e.apply=!1,e.hashFragment=null,e.segmentPaths=[],(0,p.handleSmoothScroll)(()=>{if(o){r.scrollIntoView();return}let e=document.documentElement,t=e.clientHeight;!m(r,t)&&(e.scrollTop=0,m(r,t)||r.scrollIntoView())},{dontForceLayout:!0,onlyHashChange:e.onlyHashChange}),e.onlyHashChange=!1,r.focus()}}}}function P(e){let{segmentPath:t,children:n}=e,r=(0,l.useContext)(i.GlobalLayoutRouterContext);if(!r)throw Error("invariant global layout router not mounted");return(0,u.jsx)(R,{segmentPath:t,focusAndScrollRef:r.focusAndScrollRef,children:n})}function j(e){let{parallelRouterKey:t,url:n,childNodes:r,segmentPath:o,tree:a,cacheKey:f}=e,p=(0,l.useContext)(i.GlobalLayoutRouterContext);if(!p)throw Error("invariant global layout router not mounted");let{buildId:h,changeByServerResponse:y,tree:_}=p,v=r.get(f);if(void 0===v){let e={lazyData:null,rsc:null,prefetchRsc:null,head:null,prefetchHead:null,parallelRoutes:new Map,lazyDataResolved:!1,loading:null};v=e,r.set(f,e)}let g=null!==v.prefetchRsc?v.prefetchRsc:v.rsc,m=(0,l.useDeferredValue)(v.rsc,g),R="object"==typeof m&&null!==m&&"function"==typeof m.then?(0,l.use)(m):m;if(!R){let e=v.lazyData;if(null===e){let t=function e(t,n){if(t){let[r,o]=t,u=2===t.length;if((0,d.matchSegment)(n[0],r)&&n[1].hasOwnProperty(o)){if(u){let t=e(void 0,n[1][o]);return[n[0],{...n[1],[o]:[t[0],t[1],t[2],"refetch"]}]}return[n[0],{...n[1],[o]:e(t.slice(2),n[1][o])}]}}return n}(["",...o],_),r=(0,b.hasInterceptionRouteInCurrentTree)(_);v.lazyData=e=(0,c.fetchServerResponse)(new URL(n,location.origin),t,r?p.nextUrl:null,h),v.lazyDataResolved=!1}let t=(0,l.use)(e);v.lazyDataResolved||(setTimeout(()=>{(0,l.startTransition)(()=>{y({previousTree:_,serverResponse:t})})}),v.lazyDataResolved=!0),(0,l.use)(s.unresolvedThenable)}return(0,u.jsx)(i.LayoutRouterContext.Provider,{value:{tree:a[1][t],childNodes:v.parallelRoutes,url:n,loading:v.loading},children:R})}function O(e){let{children:t,hasLoading:n,loading:r,loadingStyles:o,loadingScripts:a}=e;return n?(0,u.jsx)(l.Suspense,{fallback:(0,u.jsxs)(u.Fragment,{children:[o,a,r]}),children:t}):(0,u.jsx)(u.Fragment,{children:t})}function S(e){let{parallelRouterKey:t,segmentPath:n,error:r,errorStyles:o,errorScripts:a,templateStyles:c,templateScripts:s,template:d,notFound:p,notFoundStyles:b}=e,g=(0,l.useContext)(i.LayoutRouterContext);if(!g)throw Error("invariant expected layout router to be mounted");let{childNodes:m,tree:R,url:S,loading:E}=g,w=m.get(t);w||(w=new Map,m.set(t,w));let T=R[1][t][0],M=(0,_.getSegmentValue)(T),x=[T];return(0,u.jsx)(u.Fragment,{children:x.map(e=>{let l=(0,_.getSegmentValue)(e),g=(0,v.createRouterCacheKey)(e);return(0,u.jsxs)(i.TemplateContext.Provider,{value:(0,u.jsx)(P,{segmentPath:n,children:(0,u.jsx)(f.ErrorBoundary,{errorComponent:r,errorStyles:o,errorScripts:a,children:(0,u.jsx)(O,{hasLoading:!!E,loading:null==E?void 0:E[0],loadingStyles:null==E?void 0:E[1],loadingScripts:null==E?void 0:E[2],children:(0,u.jsx)(y.NotFoundBoundary,{notFound:p,notFoundStyles:b,children:(0,u.jsx)(h.RedirectBoundary,{children:(0,u.jsx)(j,{parallelRouterKey:t,url:S,tree:R,childNodes:w,segmentPath:n,cacheKey:g,isActive:M===l})})})})})}),children:[c,s,d]},(0,v.createRouterCacheKey)(e,!0))})})}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},76015:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{canSegmentBeOverridden:function(){return u},matchSegment:function(){return o}});let r=n(87417),o=(e,t)=>"string"==typeof e?"string"==typeof t&&e===t:"string"!=typeof t&&e[0]===t[0]&&e[1]===t[1],u=(e,t)=>{var n;return!Array.isArray(e)&&!!Array.isArray(t)&&(null==(n=(0,r.getSegmentParam)(e))?void 0:n.param)===t[0]};("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},35475:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{ReadonlyURLSearchParams:function(){return i.ReadonlyURLSearchParams},RedirectType:function(){return i.RedirectType},ServerInsertedHTMLContext:function(){return c.ServerInsertedHTMLContext},notFound:function(){return i.notFound},permanentRedirect:function(){return i.permanentRedirect},redirect:function(){return i.redirect},useParams:function(){return p},usePathname:function(){return f},useRouter:function(){return d},useSearchParams:function(){return s},useSelectedLayoutSegment:function(){return y},useSelectedLayoutSegments:function(){return h},useServerInsertedHTML:function(){return c.useServerInsertedHTML}});let r=n(2265),o=n(61956),u=n(79060),l=n(73171),a=n(84541),i=n(52646),c=n(55501);function s(){let e=(0,r.useContext)(u.SearchParamsContext),t=(0,r.useMemo)(()=>e?new i.ReadonlyURLSearchParams(e):null,[e]);if("undefined"==typeof window){let{bailoutToClientRendering:e}=n(96149);e("useSearchParams()")}return t}function f(){return(0,r.useContext)(u.PathnameContext)}function d(){let e=(0,r.useContext)(o.AppRouterContext);if(null===e)throw Error("invariant expected app router to be mounted");return e}function p(){return(0,r.useContext)(u.PathParamsContext)}function h(e){void 0===e&&(e="children");let t=(0,r.useContext)(o.LayoutRouterContext);return t?function e(t,n,r,o){let u;if(void 0===r&&(r=!0),void 0===o&&(o=[]),r)u=t[1][n];else{var i;let e=t[1];u=null!=(i=e.children)?i:Object.values(e)[0]}if(!u)return o;let c=u[0],s=(0,l.getSegmentValue)(c);return!s||s.startsWith(a.PAGE_SEGMENT_KEY)?o:(o.push(s),e(u,n,!1,o))}(t.tree,e):null}function y(e){void 0===e&&(e="children");let t=h(e);if(!t||0===t.length)return null;let n="children"===e?t[0]:t[t.length-1];return n===a.DEFAULT_SEGMENT_KEY?null:n}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},52646:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{ReadonlyURLSearchParams:function(){return l},RedirectType:function(){return r.RedirectType},notFound:function(){return o.notFound},permanentRedirect:function(){return r.permanentRedirect},redirect:function(){return r.redirect}});let r=n(88968),o=n(98200);class u extends Error{constructor(){super("Method unavailable on `ReadonlyURLSearchParams`. Read more: https://nextjs.org/docs/app/api-reference/functions/use-search-params#updating-searchparams")}}class l extends URLSearchParams{append(){throw new u}delete(){throw new u}set(){throw new u}sort(){throw new u}}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},80:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"NotFoundBoundary",{enumerable:!0,get:function(){return s}});let r=n(53099),o=n(57437),u=r._(n(2265)),l=n(35475),a=n(98200);n(31765);let i=n(61956);class c extends u.default.Component{componentDidCatch(){}static getDerivedStateFromError(e){if((0,a.isNotFoundError)(e))return{notFoundTriggered:!0};throw e}static getDerivedStateFromProps(e,t){return e.pathname!==t.previousPathname&&t.notFoundTriggered?{notFoundTriggered:!1,previousPathname:e.pathname}:{notFoundTriggered:t.notFoundTriggered,previousPathname:e.pathname}}render(){return this.state.notFoundTriggered?(0,o.jsxs)(o.Fragment,{children:[(0,o.jsx)("meta",{name:"robots",content:"noindex"}),!1,this.props.notFoundStyles,this.props.notFound]}):this.props.children}constructor(e){super(e),this.state={notFoundTriggered:!!e.asNotFound,previousPathname:e.pathname}}}function s(e){let{notFound:t,notFoundStyles:n,asNotFound:r,children:a}=e,s=(0,l.usePathname)(),f=(0,u.useContext)(i.MissingSlotContext);return t?(0,o.jsx)(c,{pathname:s,notFound:t,notFoundStyles:n,asNotFound:r,missingSlots:f,children:a}):(0,o.jsx)(o.Fragment,{children:a})}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},98200:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{isNotFoundError:function(){return o},notFound:function(){return r}});let n="NEXT_NOT_FOUND";function r(){let e=Error(n);throw e.digest=n,e}function o(e){return"object"==typeof e&&null!==e&&"digest"in e&&e.digest===n}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},29744:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"PromiseQueue",{enumerable:!0,get:function(){return c}});let r=n(2522),o=n(90675);var u=o._("_maxConcurrency"),l=o._("_runningCount"),a=o._("_queue"),i=o._("_processNext");class c{enqueue(e){let t,n;let o=new Promise((e,r)=>{t=e,n=r}),u=async()=>{try{r._(this,l)[l]++;let n=await e();t(n)}catch(e){n(e)}finally{r._(this,l)[l]--,r._(this,i)[i]()}};return r._(this,a)[a].push({promiseFn:o,task:u}),r._(this,i)[i](),o}bump(e){let t=r._(this,a)[a].findIndex(t=>t.promiseFn===e);if(t>-1){let e=r._(this,a)[a].splice(t,1)[0];r._(this,a)[a].unshift(e),r._(this,i)[i](!0)}}constructor(e=5){Object.defineProperty(this,i,{value:s}),Object.defineProperty(this,u,{writable:!0,value:void 0}),Object.defineProperty(this,l,{writable:!0,value:void 0}),Object.defineProperty(this,a,{writable:!0,value:void 0}),r._(this,u)[u]=e,r._(this,l)[l]=0,r._(this,a)[a]=[]}}function s(e){if(void 0===e&&(e=!1),(r._(this,l)[l]0){var t;null==(t=r._(this,a)[a].shift())||t.task()}}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},4123:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{RedirectBoundary:function(){return s},RedirectErrorBoundary:function(){return c}});let r=n(53099),o=n(57437),u=r._(n(2265)),l=n(35475),a=n(88968);function i(e){let{redirect:t,reset:n,redirectType:r}=e,o=(0,l.useRouter)();return(0,u.useEffect)(()=>{u.default.startTransition(()=>{r===a.RedirectType.push?o.push(t,{}):o.replace(t,{}),n()})},[t,r,n,o]),null}class c extends u.default.Component{static getDerivedStateFromError(e){if((0,a.isRedirectError)(e))return{redirect:(0,a.getURLFromRedirectError)(e),redirectType:(0,a.getRedirectTypeFromError)(e)};throw e}render(){let{redirect:e,redirectType:t}=this.state;return null!==e&&null!==t?(0,o.jsx)(i,{redirect:e,redirectType:t,reset:()=>this.setState({redirect:null})}):this.props.children}constructor(e){super(e),this.state={redirect:null,redirectType:null}}}function s(e){let{children:t}=e,n=(0,l.useRouter)();return(0,o.jsx)(c,{router:n,children:t})}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},5001:function(e,t){"use strict";var n,r;Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"RedirectStatusCode",{enumerable:!0,get:function(){return n}}),(r=n||(n={}))[r.SeeOther=303]="SeeOther",r[r.TemporaryRedirect=307]="TemporaryRedirect",r[r.PermanentRedirect=308]="PermanentRedirect",("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},88968:function(e,t,n){"use strict";var r,o;Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{RedirectType:function(){return r},getRedirectError:function(){return c},getRedirectStatusCodeFromError:function(){return y},getRedirectTypeFromError:function(){return h},getURLFromRedirectError:function(){return p},isRedirectError:function(){return d},permanentRedirect:function(){return f},redirect:function(){return s}});let u=n(20544),l=n(90295),a=n(5001),i="NEXT_REDIRECT";function c(e,t,n){void 0===n&&(n=a.RedirectStatusCode.TemporaryRedirect);let r=Error(i);r.digest=i+";"+t+";"+e+";"+n+";";let o=u.requestAsyncStorage.getStore();return o&&(r.mutableCookies=o.mutableCookies),r}function s(e,t){void 0===t&&(t="replace");let n=l.actionAsyncStorage.getStore();throw c(e,t,(null==n?void 0:n.isAction)?a.RedirectStatusCode.SeeOther:a.RedirectStatusCode.TemporaryRedirect)}function f(e,t){void 0===t&&(t="replace");let n=l.actionAsyncStorage.getStore();throw c(e,t,(null==n?void 0:n.isAction)?a.RedirectStatusCode.SeeOther:a.RedirectStatusCode.PermanentRedirect)}function d(e){if("object"!=typeof e||null===e||!("digest"in e)||"string"!=typeof e.digest)return!1;let[t,n,r,o]=e.digest.split(";",4),u=Number(o);return t===i&&("replace"===n||"push"===n)&&"string"==typeof r&&!isNaN(u)&&u in a.RedirectStatusCode}function p(e){return d(e)?e.digest.split(";",3)[2]:null}function h(e){if(!d(e))throw Error("Not a redirect error");return e.digest.split(";",2)[1]}function y(e){if(!d(e))throw Error("Not a redirect error");return Number(e.digest.split(";",4)[3])}(o=r||(r={})).push="push",o.replace="replace",("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},36423:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"default",{enumerable:!0,get:function(){return a}});let r=n(53099),o=n(57437),u=r._(n(2265)),l=n(61956);function a(){let e=(0,u.useContext)(l.TemplateContext);return(0,o.jsx)(o.Fragment,{children:e})}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},20544:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{getExpectedRequestStore:function(){return o},requestAsyncStorage:function(){return r.requestAsyncStorage}});let r=n(25575);function o(e){let t=r.requestAsyncStorage.getStore();if(t)return t;throw Error("`"+e+"` was called outside a request scope. Read more: https://nextjs.org/docs/messages/next-dynamic-api-wrong-context")}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},22356:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"applyFlightData",{enumerable:!0,get:function(){return u}});let r=n(27420),o=n(92576);function u(e,t,n,u){let[l,a,i]=n.slice(-3);if(null===a)return!1;if(3===n.length){let n=a[2],o=a[3];t.loading=o,t.rsc=n,t.prefetchRsc=null,(0,r.fillLazyItemsTillLeafWithHead)(t,e,l,a,i,u)}else t.rsc=e.rsc,t.prefetchRsc=e.prefetchRsc,t.parallelRoutes=new Map(e.parallelRoutes),t.loading=e.loading,(0,o.fillCacheWithNewSubTreeData)(t,e,n,u);return!0}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},81935:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"applyRouterStatePatchToTree",{enumerable:!0,get:function(){return function e(t,n,r,a){let i;let[c,s,f,d,p]=n;if(1===t.length){let e=l(n,r,t);return(0,u.addRefreshMarkerToActiveParallelSegments)(e,a),e}let[h,y]=t;if(!(0,o.matchSegment)(h,c))return null;if(2===t.length)i=l(s[y],r,t);else if(null===(i=e(t.slice(2),s[y],r,a)))return null;let _=[t[0],{...s,[y]:i},f,d];return p&&(_[4]=!0),(0,u.addRefreshMarkerToActiveParallelSegments)(_,a),_}}});let r=n(84541),o=n(76015),u=n(50232);function l(e,t,n){let[u,a]=e,[i,c]=t;if(i===r.DEFAULT_SEGMENT_KEY&&u!==r.DEFAULT_SEGMENT_KEY)return e;if((0,o.matchSegment)(u,i)){let t={};for(let e in a)void 0!==c[e]?t[e]=l(a[e],c[e],n):t[e]=a[e];for(let e in c)t[e]||(t[e]=c[e]);let r=[u,t];return e[2]&&(r[2]=e[2]),e[3]&&(r[3]=e[3]),e[4]&&(r[4]=e[4]),r}return t}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},65556:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"clearCacheNodeDataForSegmentPath",{enumerable:!0,get:function(){return function e(t,n,o){let u=o.length<=2,[l,a]=o,i=(0,r.createRouterCacheKey)(a),c=n.parallelRoutes.get(l),s=t.parallelRoutes.get(l);s&&s!==c||(s=new Map(c),t.parallelRoutes.set(l,s));let f=null==c?void 0:c.get(i),d=s.get(i);if(u){d&&d.lazyData&&d!==f||s.set(i,{lazyData:null,rsc:null,prefetchRsc:null,head:null,prefetchHead:null,parallelRoutes:new Map,lazyDataResolved:!1,loading:null});return}if(!d||!f){d||s.set(i,{lazyData:null,rsc:null,prefetchRsc:null,head:null,prefetchHead:null,parallelRoutes:new Map,lazyDataResolved:!1,loading:null});return}return d===f&&(d={lazyData:d.lazyData,rsc:d.rsc,prefetchRsc:d.prefetchRsc,head:d.head,prefetchHead:d.prefetchHead,parallelRoutes:new Map(d.parallelRoutes),lazyDataResolved:d.lazyDataResolved,loading:d.loading},s.set(i,d)),e(d,f,o.slice(2))}}});let r=n(78505);("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},5410:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{computeChangedPath:function(){return s},extractPathFromFlightRouterState:function(){return c}});let r=n(91182),o=n(84541),u=n(76015),l=e=>"/"===e[0]?e.slice(1):e,a=e=>"string"==typeof e?"children"===e?"":e:e[1];function i(e){return e.reduce((e,t)=>""===(t=l(t))||(0,o.isGroupSegment)(t)?e:e+"/"+t,"")||"/"}function c(e){var t;let n=Array.isArray(e[0])?e[0][1]:e[0];if(n===o.DEFAULT_SEGMENT_KEY||r.INTERCEPTION_ROUTE_MARKERS.some(e=>n.startsWith(e)))return;if(n.startsWith(o.PAGE_SEGMENT_KEY))return"";let u=[a(n)],l=null!=(t=e[1])?t:{},s=l.children?c(l.children):void 0;if(void 0!==s)u.push(s);else for(let[e,t]of Object.entries(l)){if("children"===e)continue;let n=c(t);void 0!==n&&u.push(n)}return i(u)}function s(e,t){let n=function e(t,n){let[o,l]=t,[i,s]=n,f=a(o),d=a(i);if(r.INTERCEPTION_ROUTE_MARKERS.some(e=>f.startsWith(e)||d.startsWith(e)))return"";if(!(0,u.matchSegment)(o,i)){var p;return null!=(p=c(n))?p:""}for(let t in l)if(s[t]){let n=e(l[t],s[t]);if(null!==n)return a(i)+"/"+n}return null}(e,t);return null==n||"/"===n?n:i(n.split("/"))}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},33456:function(e,t){"use strict";function n(e,t){return void 0===t&&(t=!0),e.pathname+e.search+(t?e.hash:"")}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"createHrefFromUrl",{enumerable:!0,get:function(){return n}}),("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},82952:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"createInitialRouterState",{enumerable:!0,get:function(){return c}});let r=n(33456),o=n(27420),u=n(5410),l=n(60305),a=n(24673),i=n(50232);function c(e){var t;let{buildId:n,initialTree:c,initialSeedData:s,urlParts:f,initialParallelRoutes:d,location:p,initialHead:h,couldBeIntercepted:y}=e,_=f.join("/"),v=!p,b={lazyData:null,rsc:s[2],prefetchRsc:null,head:null,prefetchHead:null,parallelRoutes:v?new Map:d,lazyDataResolved:!1,loading:s[3]},g=p?(0,r.createHrefFromUrl)(p):_;(0,i.addRefreshMarkerToActiveParallelSegments)(c,g);let m=new Map;(null===d||0===d.size)&&(0,o.fillLazyItemsTillLeafWithHead)(b,void 0,c,s,h);let R={buildId:n,tree:c,cache:b,prefetchCache:m,pushRef:{pendingPush:!1,mpaNavigation:!1,preserveCustomHistoryState:!0},focusAndScrollRef:{apply:!1,onlyHashChange:!1,hashFragment:null,segmentPaths:[]},canonicalUrl:g,nextUrl:null!=(t=(0,u.extractPathFromFlightRouterState)(c)||(null==p?void 0:p.pathname))?t:null};if(p){let e=new URL(""+p.pathname+p.search,p.origin),t=[["",c,null,null]];(0,l.createPrefetchCacheEntryForInitialLoad)({url:e,kind:a.PrefetchKind.AUTO,data:[t,void 0,!1,y],tree:R.tree,prefetchCache:R.prefetchCache,nextUrl:R.nextUrl})}return R}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},78505:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"createRouterCacheKey",{enumerable:!0,get:function(){return o}});let r=n(84541);function o(e,t){return(void 0===t&&(t=!1),Array.isArray(e))?e[0]+"|"+e[1]+"|"+e[2]:t&&e.startsWith(r.PAGE_SEGMENT_KEY)?r.PAGE_SEGMENT_KEY:e}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},44848:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"fetchServerResponse",{enumerable:!0,get:function(){return s}});let r=n(6866),o=n(12846),u=n(83079),l=n(24673),a=n(37207),{createFromFetch:i}=n(6671);function c(e){return[(0,o.urlToUrlWithoutFlightMarker)(e).toString(),void 0,!1,!1]}async function s(e,t,n,s,f){let d={[r.RSC_HEADER]:"1",[r.NEXT_ROUTER_STATE_TREE]:encodeURIComponent(JSON.stringify(t))};f===l.PrefetchKind.AUTO&&(d[r.NEXT_ROUTER_PREFETCH_HEADER]="1"),n&&(d[r.NEXT_URL]=n);let p=(0,a.hexHash)([d[r.NEXT_ROUTER_PREFETCH_HEADER]||"0",d[r.NEXT_ROUTER_STATE_TREE],d[r.NEXT_URL]].join(","));try{var h;let t=new URL(e);t.pathname.endsWith("/")?t.pathname+="index.txt":t.pathname+=".txt",t.searchParams.set(r.NEXT_RSC_UNION_QUERY,p);let n=await fetch(t,{credentials:"same-origin",headers:d}),l=(0,o.urlToUrlWithoutFlightMarker)(n.url),a=n.redirected?l:void 0,f=n.headers.get("content-type")||"",y=!!n.headers.get(r.NEXT_DID_POSTPONE_HEADER),_=!!(null==(h=n.headers.get("vary"))?void 0:h.includes(r.NEXT_URL)),v=f===r.RSC_CONTENT_TYPE_HEADER;if(v||(v=f.startsWith("text/plain")),!v||!n.ok)return e.hash&&(l.hash=e.hash),c(l.toString());let[b,g]=await i(Promise.resolve(n),{callServer:u.callServer});if(s!==b)return c(n.url);return[g,a,y,_]}catch(t){return console.error("Failed to fetch RSC payload for "+e+". Falling back to browser navigation.",t),[e.toString(),void 0,!1,!1]}}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},92576:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"fillCacheWithNewSubTreeData",{enumerable:!0,get:function(){return function e(t,n,l,a){let i=l.length<=5,[c,s]=l,f=(0,u.createRouterCacheKey)(s),d=n.parallelRoutes.get(c);if(!d)return;let p=t.parallelRoutes.get(c);p&&p!==d||(p=new Map(d),t.parallelRoutes.set(c,p));let h=d.get(f),y=p.get(f);if(i){if(!y||!y.lazyData||y===h){let e=l[3];y={lazyData:null,rsc:e[2],prefetchRsc:null,head:null,prefetchHead:null,loading:e[3],parallelRoutes:h?new Map(h.parallelRoutes):new Map,lazyDataResolved:!1},h&&(0,r.invalidateCacheByRouterState)(y,h,l[2]),(0,o.fillLazyItemsTillLeafWithHead)(y,h,l[2],e,l[4],a),p.set(f,y)}return}y&&h&&(y===h&&(y={lazyData:y.lazyData,rsc:y.rsc,prefetchRsc:y.prefetchRsc,head:y.head,prefetchHead:y.prefetchHead,parallelRoutes:new Map(y.parallelRoutes),lazyDataResolved:!1,loading:y.loading},p.set(f,y)),e(y,h,l.slice(2),a))}}});let r=n(94377),o=n(27420),u=n(78505);("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},27420:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"fillLazyItemsTillLeafWithHead",{enumerable:!0,get:function(){return function e(t,n,u,l,a,i){if(0===Object.keys(u[1]).length){t.head=a;return}for(let c in u[1]){let s;let f=u[1][c],d=f[0],p=(0,r.createRouterCacheKey)(d),h=null!==l&&void 0!==l[1][c]?l[1][c]:null;if(n){let r=n.parallelRoutes.get(c);if(r){let n;let u=(null==i?void 0:i.kind)==="auto"&&i.status===o.PrefetchCacheEntryStatus.reusable,l=new Map(r),s=l.get(p);n=null!==h?{lazyData:null,rsc:h[2],prefetchRsc:null,head:null,prefetchHead:null,loading:h[3],parallelRoutes:new Map(null==s?void 0:s.parallelRoutes),lazyDataResolved:!1}:u&&s?{lazyData:s.lazyData,rsc:s.rsc,prefetchRsc:s.prefetchRsc,head:s.head,prefetchHead:s.prefetchHead,parallelRoutes:new Map(s.parallelRoutes),lazyDataResolved:s.lazyDataResolved,loading:s.loading}:{lazyData:null,rsc:null,prefetchRsc:null,head:null,prefetchHead:null,parallelRoutes:new Map(null==s?void 0:s.parallelRoutes),lazyDataResolved:!1,loading:null},l.set(p,n),e(n,s,f,h||null,a,i),t.parallelRoutes.set(c,l);continue}}if(null!==h){let e=h[2],t=h[3];s={lazyData:null,rsc:e,prefetchRsc:null,head:null,prefetchHead:null,parallelRoutes:new Map,lazyDataResolved:!1,loading:t}}else s={lazyData:null,rsc:null,prefetchRsc:null,head:null,prefetchHead:null,parallelRoutes:new Map,lazyDataResolved:!1,loading:null};let y=t.parallelRoutes.get(c);y?y.set(p,s):t.parallelRoutes.set(c,new Map([[p,s]])),e(s,void 0,f,h,a,i)}}}});let r=n(78505),o=n(24673);("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},44510:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"handleMutable",{enumerable:!0,get:function(){return u}});let r=n(5410);function o(e){return void 0!==e}function u(e,t){var n,u,l;let a=null==(u=t.shouldScroll)||u,i=e.nextUrl;if(o(t.patchedTree)){let n=(0,r.computeChangedPath)(e.tree,t.patchedTree);n?i=n:i||(i=e.canonicalUrl)}return{buildId:e.buildId,canonicalUrl:o(t.canonicalUrl)?t.canonicalUrl===e.canonicalUrl?e.canonicalUrl:t.canonicalUrl:e.canonicalUrl,pushRef:{pendingPush:o(t.pendingPush)?t.pendingPush:e.pushRef.pendingPush,mpaNavigation:o(t.mpaNavigation)?t.mpaNavigation:e.pushRef.mpaNavigation,preserveCustomHistoryState:o(t.preserveCustomHistoryState)?t.preserveCustomHistoryState:e.pushRef.preserveCustomHistoryState},focusAndScrollRef:{apply:!!a&&(!!o(null==t?void 0:t.scrollableSegments)||e.focusAndScrollRef.apply),onlyHashChange:!!t.hashFragment&&e.canonicalUrl.split("#",1)[0]===(null==(n=t.canonicalUrl)?void 0:n.split("#",1)[0]),hashFragment:a?t.hashFragment&&""!==t.hashFragment?decodeURIComponent(t.hashFragment.slice(1)):e.focusAndScrollRef.hashFragment:null,segmentPaths:a?null!=(l=null==t?void 0:t.scrollableSegments)?l:e.focusAndScrollRef.segmentPaths:[]},cache:t.cache?t.cache:e.cache,prefetchCache:t.prefetchCache?t.prefetchCache:e.prefetchCache,tree:o(t.patchedTree)?t.patchedTree:e.tree,nextUrl:i}}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},77831:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"handleSegmentMismatch",{enumerable:!0,get:function(){return o}});let r=n(95967);function o(e,t,n){return(0,r.handleExternalUrl)(e,{},e.canonicalUrl,!0)}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},77058:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"invalidateCacheBelowFlightSegmentPath",{enumerable:!0,get:function(){return function e(t,n,o){let u=o.length<=2,[l,a]=o,i=(0,r.createRouterCacheKey)(a),c=n.parallelRoutes.get(l);if(!c)return;let s=t.parallelRoutes.get(l);if(s&&s!==c||(s=new Map(c),t.parallelRoutes.set(l,s)),u){s.delete(i);return}let f=c.get(i),d=s.get(i);d&&f&&(d===f&&(d={lazyData:d.lazyData,rsc:d.rsc,prefetchRsc:d.prefetchRsc,head:d.head,prefetchHead:d.prefetchHead,parallelRoutes:new Map(d.parallelRoutes),lazyDataResolved:d.lazyDataResolved},s.set(i,d)),e(d,f,o.slice(2)))}}});let r=n(78505);("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},94377:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"invalidateCacheByRouterState",{enumerable:!0,get:function(){return o}});let r=n(78505);function o(e,t,n){for(let o in n[1]){let u=n[1][o][0],l=(0,r.createRouterCacheKey)(u),a=t.parallelRoutes.get(o);if(a){let t=new Map(a);t.delete(l),e.parallelRoutes.set(o,t)}}}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},63237:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"isNavigatingToNewRootLayout",{enumerable:!0,get:function(){return function e(t,n){let r=t[0],o=n[0];if(Array.isArray(r)&&Array.isArray(o)){if(r[0]!==o[0]||r[2]!==o[2])return!0}else if(r!==o)return!0;if(t[4])return!n[4];if(n[4])return!0;let u=Object.values(t[1])[0],l=Object.values(n[1])[0];return!u||!l||e(u,l)}}}),("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},56118:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{abortTask:function(){return c},listenForDynamicRequest:function(){return a},updateCacheNodeOnNavigation:function(){return function e(t,n,a,c,s){let f=n[1],d=a[1],p=c[1],h=t.parallelRoutes,y=new Map(h),_={},v=null;for(let t in d){let n;let a=d[t],c=f[t],b=h.get(t),g=p[t],m=a[0],R=(0,u.createRouterCacheKey)(m),P=void 0!==c?c[0]:void 0,j=void 0!==b?b.get(R):void 0;if(null!==(n=m===r.PAGE_SEGMENT_KEY?l(a,void 0!==g?g:null,s):m===r.DEFAULT_SEGMENT_KEY?void 0!==c?{route:c,node:null,children:null}:l(a,void 0!==g?g:null,s):void 0!==P&&(0,o.matchSegment)(m,P)&&void 0!==j&&void 0!==c?null!=g?e(j,c,a,g,s):function(e){let t=i(e,null,null);return{route:e,node:t,children:null}}(a):l(a,void 0!==g?g:null,s))){null===v&&(v=new Map),v.set(t,n);let e=n.node;if(null!==e){let n=new Map(b);n.set(R,e),y.set(t,n)}_[t]=n.route}else _[t]=a}if(null===v)return null;let b={lazyData:null,rsc:t.rsc,prefetchRsc:t.prefetchRsc,head:t.head,prefetchHead:t.prefetchHead,loading:t.loading,parallelRoutes:y,lazyDataResolved:!1};return{route:function(e,t){let n=[e[0],t];return 2 in e&&(n[2]=e[2]),3 in e&&(n[3]=e[3]),4 in e&&(n[4]=e[4]),n}(a,_),node:b,children:v}}},updateCacheNodeOnPopstateRestoration:function(){return function e(t,n){let r=n[1],o=t.parallelRoutes,l=new Map(o);for(let t in r){let n=r[t],a=n[0],i=(0,u.createRouterCacheKey)(a),c=o.get(t);if(void 0!==c){let r=c.get(i);if(void 0!==r){let o=e(r,n),u=new Map(c);u.set(i,o),l.set(t,u)}}}let a=t.rsc,i=d(a)&&"pending"===a.status;return{lazyData:null,rsc:a,head:t.head,prefetchHead:i?t.prefetchHead:null,prefetchRsc:i?t.prefetchRsc:null,loading:i?t.loading:null,parallelRoutes:l,lazyDataResolved:!1}}}});let r=n(84541),o=n(76015),u=n(78505);function l(e,t,n){let r=i(e,t,n);return{route:e,node:r,children:null}}function a(e,t){t.then(t=>{for(let n of t[0]){let t=n.slice(0,-3),r=n[n.length-3],l=n[n.length-2],a=n[n.length-1];"string"!=typeof t&&function(e,t,n,r,l){let a=e;for(let e=0;e{c(e,t)})}function i(e,t,n){let r=e[1],o=null!==t?t[1]:null,l=new Map;for(let e in r){let t=r[e],a=null!==o?o[e]:null,c=t[0],s=(0,u.createRouterCacheKey)(c),f=i(t,void 0===a?null:a,n),d=new Map;d.set(s,f),l.set(e,d)}let a=0===l.size,c=null!==t?t[2]:null,s=null!==t?t[3]:null;return{lazyData:null,parallelRoutes:l,prefetchRsc:void 0!==c?c:null,prefetchHead:a?n:null,loading:void 0!==s?s:null,rsc:p(),head:a?p():null,lazyDataResolved:!1}}function c(e,t){let n=e.node;if(null===n)return;let r=e.children;if(null===r)s(e.route,n,t);else for(let e of r.values())c(e,t);e.node=null}function s(e,t,n){let r=e[1],o=t.parallelRoutes;for(let e in r){let t=r[e],l=o.get(e);if(void 0===l)continue;let a=t[0],i=(0,u.createRouterCacheKey)(a),c=l.get(i);void 0!==c&&s(t,c,n)}let l=t.rsc;d(l)&&(null===n?l.resolve(null):l.reject(n));let a=t.head;d(a)&&a.resolve(null)}let f=Symbol();function d(e){return e&&e.tag===f}function p(){let e,t;let n=new Promise((n,r)=>{e=n,t=r});return n.status="pending",n.resolve=t=>{"pending"===n.status&&(n.status="fulfilled",n.value=t,e(t))},n.reject=e=>{"pending"===n.status&&(n.status="rejected",n.reason=e,t(e))},n.tag=f,n}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},60305:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{createPrefetchCacheEntryForInitialLoad:function(){return c},getOrCreatePrefetchCacheEntry:function(){return i},prunePrefetchCache:function(){return f}});let r=n(33456),o=n(44848),u=n(24673),l=n(24819);function a(e,t){let n=(0,r.createHrefFromUrl)(e,!1);return t?t+"%"+n:n}function i(e){let t,{url:n,nextUrl:r,tree:o,buildId:l,prefetchCache:i,kind:c}=e,f=a(n,r),d=i.get(f);if(d)t=d;else{let e=a(n),r=i.get(e);r&&(t=r)}return t?(t.status=h(t),t.kind!==u.PrefetchKind.FULL&&c===u.PrefetchKind.FULL)?s({tree:o,url:n,buildId:l,nextUrl:r,prefetchCache:i,kind:null!=c?c:u.PrefetchKind.TEMPORARY}):(c&&t.kind===u.PrefetchKind.TEMPORARY&&(t.kind=c),t):s({tree:o,url:n,buildId:l,nextUrl:r,prefetchCache:i,kind:c||u.PrefetchKind.TEMPORARY})}function c(e){let{nextUrl:t,tree:n,prefetchCache:r,url:o,kind:l,data:i}=e,[,,,c]=i,s=c?a(o,t):a(o),f={treeAtTimeOfPrefetch:n,data:Promise.resolve(i),kind:l,prefetchTime:Date.now(),lastUsedTime:Date.now(),key:s,status:u.PrefetchCacheEntryStatus.fresh};return r.set(s,f),f}function s(e){let{url:t,kind:n,tree:r,nextUrl:i,buildId:c,prefetchCache:s}=e,f=a(t),d=l.prefetchQueue.enqueue(()=>(0,o.fetchServerResponse)(t,r,i,c,n).then(e=>{let[,,,n]=e;return n&&function(e){let{url:t,nextUrl:n,prefetchCache:r}=e,o=a(t),u=r.get(o);if(!u)return;let l=a(t,n);r.set(l,u),r.delete(o)}({url:t,nextUrl:i,prefetchCache:s}),e})),p={treeAtTimeOfPrefetch:r,data:d,kind:n,prefetchTime:Date.now(),lastUsedTime:null,key:f,status:u.PrefetchCacheEntryStatus.fresh};return s.set(f,p),p}function f(e){for(let[t,n]of e)h(n)===u.PrefetchCacheEntryStatus.expired&&e.delete(t)}let d=1e3*Number("30"),p=1e3*Number("300");function h(e){let{kind:t,prefetchTime:n,lastUsedTime:r}=e;return Date.now()<(null!=r?r:n)+d?r?u.PrefetchCacheEntryStatus.reusable:u.PrefetchCacheEntryStatus.fresh:"auto"===t&&Date.now(){let[n,f]=t,h=!1;if(S.lastUsedTime||(S.lastUsedTime=Date.now(),h=!0),"string"==typeof n)return _(e,R,n,O);if(document.getElementById("__next-page-redirect"))return _(e,R,j,O);let b=e.tree,g=e.cache,w=[];for(let t of n){let n=t.slice(0,-4),r=t.slice(-3)[0],c=["",...n],f=(0,u.applyRouterStatePatchToTree)(c,b,r,j);if(null===f&&(f=(0,u.applyRouterStatePatchToTree)(c,E,r,j)),null!==f){if((0,a.isNavigatingToNewRootLayout)(b,f))return _(e,R,j,O);let u=(0,d.createEmptyCacheNode)(),m=!1;for(let e of(S.status!==i.PrefetchCacheEntryStatus.stale||h?m=(0,s.applyFlightData)(g,u,t,S):(m=function(e,t,n,r){let o=!1;for(let u of(e.rsc=t.rsc,e.prefetchRsc=t.prefetchRsc,e.loading=t.loading,e.parallelRoutes=new Map(t.parallelRoutes),v(r).map(e=>[...n,...e])))(0,y.clearCacheNodeDataForSegmentPath)(e,t,u),o=!0;return o}(u,g,n,r),S.lastUsedTime=Date.now()),(0,l.shouldHardNavigate)(c,b)?(u.rsc=g.rsc,u.prefetchRsc=g.prefetchRsc,(0,o.invalidateCacheBelowFlightSegmentPath)(u,g,n),R.cache=u):m&&(R.cache=u,g=u),b=f,v(r))){let t=[...n,...e];t[t.length-1]!==p.DEFAULT_SEGMENT_KEY&&w.push(t)}}}return R.patchedTree=b,R.canonicalUrl=f?(0,r.createHrefFromUrl)(f):j,R.pendingPush=O,R.scrollableSegments=w,R.hashFragment=P,R.shouldScroll=m,(0,c.handleMutable)(e,R)},()=>e)};("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},24819:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{prefetchQueue:function(){return l},prefetchReducer:function(){return a}});let r=n(6866),o=n(29744),u=n(60305),l=new o.PromiseQueue(5);function a(e,t){(0,u.prunePrefetchCache)(e.prefetchCache);let{url:n}=t;return n.searchParams.delete(r.NEXT_RSC_UNION_QUERY),(0,u.getOrCreatePrefetchCacheEntry)({url:n,nextUrl:e.nextUrl,prefetchCache:e.prefetchCache,kind:t.kind,tree:e.tree,buildId:e.buildId}),e}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},99601:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"refreshReducer",{enumerable:!0,get:function(){return h}});let r=n(44848),o=n(33456),u=n(81935),l=n(63237),a=n(95967),i=n(44510),c=n(27420),s=n(12846),f=n(77831),d=n(28077),p=n(50232);function h(e,t){let{origin:n}=t,h={},y=e.canonicalUrl,_=e.tree;h.preserveCustomHistoryState=!1;let v=(0,s.createEmptyCacheNode)(),b=(0,d.hasInterceptionRouteInCurrentTree)(e.tree);return v.lazyData=(0,r.fetchServerResponse)(new URL(y,n),[_[0],_[1],_[2],"refetch"],b?e.nextUrl:null,e.buildId),v.lazyData.then(async n=>{let[r,s]=n;if("string"==typeof r)return(0,a.handleExternalUrl)(e,h,r,e.pushRef.pendingPush);for(let n of(v.lazyData=null,r)){if(3!==n.length)return console.log("REFRESH FAILED"),e;let[r]=n,i=(0,u.applyRouterStatePatchToTree)([""],_,r,e.canonicalUrl);if(null===i)return(0,f.handleSegmentMismatch)(e,t,r);if((0,l.isNavigatingToNewRootLayout)(_,i))return(0,a.handleExternalUrl)(e,h,y,e.pushRef.pendingPush);let d=s?(0,o.createHrefFromUrl)(s):void 0;s&&(h.canonicalUrl=d);let[g,m]=n.slice(-2);if(null!==g){let e=g[2];v.rsc=e,v.prefetchRsc=null,(0,c.fillLazyItemsTillLeafWithHead)(v,void 0,r,g,m),h.prefetchCache=new Map}await (0,p.refreshInactiveParallelSegments)({state:e,updatedTree:i,updatedCache:v,includeNextUrl:b,canonicalUrl:h.canonicalUrl||e.canonicalUrl}),h.cache=v,h.patchedTree=i,h.canonicalUrl=y,_=i}return(0,i.handleMutable)(e,h)},()=>e)}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},77784:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"restoreReducer",{enumerable:!0,get:function(){return u}});let r=n(33456),o=n(5410);function u(e,t){var n;let{url:u,tree:l}=t,a=(0,r.createHrefFromUrl)(u),i=l||e.tree,c=e.cache;return{buildId:e.buildId,canonicalUrl:a,pushRef:{pendingPush:!1,mpaNavigation:!1,preserveCustomHistoryState:!0},focusAndScrollRef:e.focusAndScrollRef,cache:c,prefetchCache:e.prefetchCache,tree:i,nextUrl:null!=(n=(0,o.extractPathFromFlightRouterState)(i))?n:u.pathname}}n(56118),("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},13722:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"serverActionReducer",{enumerable:!0,get:function(){return g}});let r=n(83079),o=n(6866),u=n(1634),l=n(33456),a=n(95967),i=n(81935),c=n(63237),s=n(44510),f=n(27420),d=n(12846),p=n(28077),h=n(77831),y=n(50232),{createFromFetch:_,encodeReply:v}=n(6671);async function b(e,t,n){let l,{actionId:a,actionArgs:i}=n,c=await v(i),s=await fetch("",{method:"POST",headers:{Accept:o.RSC_CONTENT_TYPE_HEADER,[o.ACTION]:a,[o.NEXT_ROUTER_STATE_TREE]:encodeURIComponent(JSON.stringify(e.tree)),...t?{[o.NEXT_URL]:t}:{}},body:c}),f=s.headers.get("x-action-redirect");try{let e=JSON.parse(s.headers.get("x-action-revalidated")||"[[],0,0]");l={paths:e[0]||[],tag:!!e[1],cookie:e[2]}}catch(e){l={paths:[],tag:!1,cookie:!1}}let d=f?new URL((0,u.addBasePath)(f),new URL(e.canonicalUrl,window.location.href)):void 0;if(s.headers.get("content-type")===o.RSC_CONTENT_TYPE_HEADER){let e=await _(Promise.resolve(s),{callServer:r.callServer});if(f){let[,t]=null!=e?e:[];return{actionFlightData:t,redirectLocation:d,revalidatedParts:l}}let[t,[,n]]=null!=e?e:[];return{actionResult:t,actionFlightData:n,redirectLocation:d,revalidatedParts:l}}return{redirectLocation:d,revalidatedParts:l}}function g(e,t){let{resolve:n,reject:r}=t,o={},u=e.canonicalUrl,_=e.tree;o.preserveCustomHistoryState=!1;let v=e.nextUrl&&(0,p.hasInterceptionRouteInCurrentTree)(e.tree)?e.nextUrl:null;return o.inFlightServerAction=b(e,v,t),o.inFlightServerAction.then(async r=>{let{actionResult:p,actionFlightData:b,redirectLocation:g}=r;if(g&&(e.pushRef.pendingPush=!0,o.pendingPush=!0),!b)return(n(p),g)?(0,a.handleExternalUrl)(e,o,g.href,e.pushRef.pendingPush):e;if("string"==typeof b)return(0,a.handleExternalUrl)(e,o,b,e.pushRef.pendingPush);if(o.inFlightServerAction=null,g){let e=(0,l.createHrefFromUrl)(g,!1);o.canonicalUrl=e}for(let n of b){if(3!==n.length)return console.log("SERVER ACTION APPLY FAILED"),e;let[r]=n,s=(0,i.applyRouterStatePatchToTree)([""],_,r,g?(0,l.createHrefFromUrl)(g):e.canonicalUrl);if(null===s)return(0,h.handleSegmentMismatch)(e,t,r);if((0,c.isNavigatingToNewRootLayout)(_,s))return(0,a.handleExternalUrl)(e,o,u,e.pushRef.pendingPush);let[p,b]=n.slice(-2),m=null!==p?p[2]:null;if(null!==m){let t=(0,d.createEmptyCacheNode)();t.rsc=m,t.prefetchRsc=null,(0,f.fillLazyItemsTillLeafWithHead)(t,void 0,r,p,b),await (0,y.refreshInactiveParallelSegments)({state:e,updatedTree:s,updatedCache:t,includeNextUrl:!!v,canonicalUrl:o.canonicalUrl||e.canonicalUrl}),o.cache=t,o.prefetchCache=new Map}o.patchedTree=s,_=s}return n(p),(0,s.handleMutable)(e,o)},t=>(r(t),e))}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},68448:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"serverPatchReducer",{enumerable:!0,get:function(){return f}});let r=n(33456),o=n(81935),u=n(63237),l=n(95967),a=n(22356),i=n(44510),c=n(12846),s=n(77831);function f(e,t){let{serverResponse:n}=t,[f,d]=n,p={};if(p.preserveCustomHistoryState=!1,"string"==typeof f)return(0,l.handleExternalUrl)(e,p,f,e.pushRef.pendingPush);let h=e.tree,y=e.cache;for(let n of f){let i=n.slice(0,-4),[f]=n.slice(-3,-2),_=(0,o.applyRouterStatePatchToTree)(["",...i],h,f,e.canonicalUrl);if(null===_)return(0,s.handleSegmentMismatch)(e,t,f);if((0,u.isNavigatingToNewRootLayout)(h,_))return(0,l.handleExternalUrl)(e,p,e.canonicalUrl,e.pushRef.pendingPush);let v=d?(0,r.createHrefFromUrl)(d):void 0;v&&(p.canonicalUrl=v);let b=(0,c.createEmptyCacheNode)();(0,a.applyFlightData)(y,b,n),p.patchedTree=_,p.cache=b,y=b,h=_}return(0,i.handleMutable)(e,p)}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},50232:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{addRefreshMarkerToActiveParallelSegments:function(){return function e(t,n){let[r,o,,l]=t;for(let a in r.includes(u.PAGE_SEGMENT_KEY)&&"refresh"!==l&&(t[2]=n,t[3]="refresh"),o)e(o[a],n)}},refreshInactiveParallelSegments:function(){return l}});let r=n(22356),o=n(44848),u=n(84541);async function l(e){let t=new Set;await a({...e,rootTree:e.updatedTree,fetchedSegments:t})}async function a(e){let{state:t,updatedTree:n,updatedCache:u,includeNextUrl:l,fetchedSegments:i,rootTree:c=n,canonicalUrl:s}=e,[,f,d,p]=n,h=[];if(d&&d!==s&&"refresh"===p&&!i.has(d)){i.add(d);let e=(0,o.fetchServerResponse)(new URL(d,location.origin),[c[0],c[1],c[2],"refetch"],l?t.nextUrl:null,t.buildId).then(e=>{let t=e[0];if("string"!=typeof t)for(let e of t)(0,r.applyFlightData)(u,u,e)});h.push(e)}for(let e in f){let n=a({state:t,updatedTree:f[e],updatedCache:u,includeNextUrl:l,fetchedSegments:i,rootTree:c,canonicalUrl:s});h.push(n)}await Promise.all(h)}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},24673:function(e,t){"use strict";var n,r,o,u;Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{ACTION_FAST_REFRESH:function(){return f},ACTION_NAVIGATE:function(){return a},ACTION_PREFETCH:function(){return s},ACTION_REFRESH:function(){return l},ACTION_RESTORE:function(){return i},ACTION_SERVER_ACTION:function(){return d},ACTION_SERVER_PATCH:function(){return c},PrefetchCacheEntryStatus:function(){return r},PrefetchKind:function(){return n},isThenable:function(){return p}});let l="refresh",a="navigate",i="restore",c="server-patch",s="prefetch",f="fast-refresh",d="server-action";function p(e){return e&&("object"==typeof e||"function"==typeof e)&&"function"==typeof e.then}(o=n||(n={})).AUTO="auto",o.FULL="full",o.TEMPORARY="temporary",(u=r||(r={})).fresh="fresh",u.reusable="reusable",u.expired="expired",u.stale="stale",("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},91450:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"reducer",{enumerable:!0,get:function(){return f}});let r=n(24673),o=n(95967),u=n(68448),l=n(77784),a=n(99601),i=n(24819),c=n(44529),s=n(13722),f="undefined"==typeof window?function(e,t){return e}:function(e,t){switch(t.type){case r.ACTION_NAVIGATE:return(0,o.navigateReducer)(e,t);case r.ACTION_SERVER_PATCH:return(0,u.serverPatchReducer)(e,t);case r.ACTION_RESTORE:return(0,l.restoreReducer)(e,t);case r.ACTION_REFRESH:return(0,a.refreshReducer)(e,t);case r.ACTION_FAST_REFRESH:return(0,c.fastRefreshReducer)(e,t);case r.ACTION_PREFETCH:return(0,i.prefetchReducer)(e,t);case r.ACTION_SERVER_ACTION:return(0,s.serverActionReducer)(e,t);default:throw Error("Unknown action")}};("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},53728:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"shouldHardNavigate",{enumerable:!0,get:function(){return function e(t,n){let[o,u]=n,[l,a]=t;return(0,r.matchSegment)(l,o)?!(t.length<=2)&&e(t.slice(2),u[a]):!!Array.isArray(l)}}});let r=n(76015);("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},54535:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{createDynamicallyTrackedSearchParams:function(){return a},createUntrackedSearchParams:function(){return l}});let r=n(51845),o=n(86999),u=n(30650);function l(e){let t=r.staticGenerationAsyncStorage.getStore();return t&&t.forceStatic?{}:e}function a(e){let t=r.staticGenerationAsyncStorage.getStore();return t?t.forceStatic?{}:t.isStaticGeneration||t.dynamicShouldError?new Proxy({},{get:(e,n,r)=>("string"==typeof n&&(0,o.trackDynamicDataAccessed)(t,"searchParams."+n),u.ReflectAdapter.get(e,n,r)),has:(e,n)=>("string"==typeof n&&(0,o.trackDynamicDataAccessed)(t,"searchParams."+n),Reflect.has(e,n)),ownKeys:e=>((0,o.trackDynamicDataAccessed)(t,"searchParams"),Reflect.ownKeys(e))}):e:e}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},51845:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"staticGenerationAsyncStorage",{enumerable:!0,get:function(){return r.staticGenerationAsyncStorage}});let r=n(20030);("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},36864:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{StaticGenBailoutError:function(){return r},isStaticGenBailoutError:function(){return o}});let n="NEXT_STATIC_GEN_BAILOUT";class r extends Error{constructor(...e){super(...e),this.code=n}}function o(e){return"object"==typeof e&&null!==e&&"code"in e&&e.code===n}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},38137:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"unresolvedThenable",{enumerable:!0,get:function(){return n}});let n={then:()=>{}};("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},47744:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{useReducerWithReduxDevtools:function(){return i},useUnwrapState:function(){return a}});let r=n(53099)._(n(2265)),o=n(24673),u=n(2103);function l(e){if(e instanceof Map){let t={};for(let[n,r]of e.entries()){if("function"==typeof r){t[n]="fn()";continue}if("object"==typeof r&&null!==r){if(r.$$typeof){t[n]=r.$$typeof.toString();continue}if(r._bundlerConfig){t[n]="FlightData";continue}}t[n]=l(r)}return t}if("object"==typeof e&&null!==e){let t={};for(let n in e){let r=e[n];if("function"==typeof r){t[n]="fn()";continue}if("object"==typeof r&&null!==r){if(r.$$typeof){t[n]=r.$$typeof.toString();continue}if(r.hasOwnProperty("_bundlerConfig")){t[n]="FlightData";continue}}t[n]=l(r)}return t}return Array.isArray(e)?e.map(l):e}function a(e){return(0,o.isThenable)(e)?(0,r.use)(e):e}let i="undefined"!=typeof window?function(e){let[t,n]=r.default.useState(e),o=(0,r.useContext)(u.ActionQueueContext);if(!o)throw Error("Invariant: Missing ActionQueueContext");let a=(0,r.useRef)(),i=(0,r.useRef)();return(0,r.useEffect)(()=>{if(!a.current&&!1!==i.current){if(void 0===i.current&&void 0===window.__REDUX_DEVTOOLS_EXTENSION__){i.current=!1;return}return a.current=window.__REDUX_DEVTOOLS_EXTENSION__.connect({instanceId:8e3,name:"next-router"}),a.current&&(a.current.init(l(e)),o&&(o.devToolsInstance=a.current)),()=>{a.current=void 0}}},[e,o]),[t,(0,r.useCallback)(t=>{o.state||(o.state=e),o.dispatch(t,n)},[o,e]),(0,r.useCallback)(e=>{a.current&&a.current.send({type:"RENDER_SYNC"},l(e))},[])]}:function(e){return[e,()=>{},()=>{}]};("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},11283:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"hasBasePath",{enumerable:!0,get:function(){return o}});let r=n(10580);function o(e){return(0,r.pathHasPrefix)(e,"")}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},33068:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"normalizePathTrailingSlash",{enumerable:!0,get:function(){return u}});let r=n(26674),o=n(63381),u=e=>{if(!e.startsWith("/"))return e;let{pathname:t,query:n,hash:u}=(0,o.parsePath)(e);return""+(0,r.removeTrailingSlash)(t)+n+u};("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},61404:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"default",{enumerable:!0,get:function(){return o}});let r=n(18993);function o(e){let t="function"==typeof reportError?reportError:e=>{window.console.error(e)};(0,r.isBailoutToCSRError)(e)||t(e)}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},35076:function(e,t,n){"use strict";function r(e){return e}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"removeBasePath",{enumerable:!0,get:function(){return r}}),n(11283),("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},12010:function(e,t){"use strict";function n(e,t){var n=e.length;for(e.push(t);0>>1,o=e[r];if(0>>1;ru(i,n))cu(s,i)?(e[r]=s,e[c]=n,r=c):(e[r]=i,e[a]=n,r=a);else if(cu(s,n))e[r]=s,e[c]=n,r=c;else break}}return t}function u(e,t){var n=e.sortIndex-t.sortIndex;return 0!==n?n:e.id-t.id}if(t.unstable_now=void 0,"object"==typeof performance&&"function"==typeof performance.now){var l,a=performance;t.unstable_now=function(){return a.now()}}else{var i=Date,c=i.now();t.unstable_now=function(){return i.now()-c}}var s=[],f=[],d=1,p=null,h=3,y=!1,_=!1,v=!1,b="function"==typeof setTimeout?setTimeout:null,g="function"==typeof clearTimeout?clearTimeout:null,m="undefined"!=typeof setImmediate?setImmediate:null;function R(e){for(var t=r(f);null!==t;){if(null===t.callback)o(f);else if(t.startTime<=e)o(f),t.sortIndex=t.expirationTime,n(s,t);else break;t=r(f)}}function P(e){if(v=!1,R(e),!_){if(null!==r(s))_=!0,C();else{var t=r(f);null!==t&&A(P,t.startTime-e)}}}"undefined"!=typeof navigator&&void 0!==navigator.scheduling&&void 0!==navigator.scheduling.isInputPending&&navigator.scheduling.isInputPending.bind(navigator.scheduling);var j=!1,O=-1,S=5,E=-1;function w(){return!(t.unstable_now()-Ee&&w());){var a=p.callback;if("function"==typeof a){p.callback=null,h=p.priorityLevel;var i=a(p.expirationTime<=e);if(e=t.unstable_now(),"function"==typeof i){p.callback=i,R(e),n=!0;break t}p===r(s)&&o(s),R(e)}else o(s);p=r(s)}if(null!==p)n=!0;else{var c=r(f);null!==c&&A(P,c.startTime-e),n=!1}}break e}finally{p=null,h=u,y=!1}n=void 0}}finally{n?l():j=!1}}}if("function"==typeof m)l=function(){m(T)};else if("undefined"!=typeof MessageChannel){var M=new MessageChannel,x=M.port2;M.port1.onmessage=T,l=function(){x.postMessage(null)}}else l=function(){b(T,0)};function C(){j||(j=!0,l())}function A(e,n){O=b(function(){e(t.unstable_now())},n)}t.unstable_IdlePriority=5,t.unstable_ImmediatePriority=1,t.unstable_LowPriority=4,t.unstable_NormalPriority=3,t.unstable_Profiling=null,t.unstable_UserBlockingPriority=2,t.unstable_cancelCallback=function(e){e.callback=null},t.unstable_continueExecution=function(){_||y||(_=!0,C())},t.unstable_forceFrameRate=function(e){0>e||125l?(e.sortIndex=u,n(f,e),null===r(s)&&e===r(f)&&(v?(g(O),O=-1):v=!0,A(P,u-l))):(e.sortIndex=a,n(s,e),_||y||(_=!0,C())),e},t.unstable_shouldYield=w,t.unstable_wrapCallback=function(e){var t=h;return function(){var n=h;h=t;try{return e.apply(this,arguments)}finally{h=n}}}},71767:function(e,t,n){"use strict";e.exports=n(12010)},60934:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{getPathname:function(){return r},isFullStringUrl:function(){return o},parseUrl:function(){return u}});let n="http://n";function r(e){return new URL(e,n).pathname}function o(e){return/https?:\/\//.test(e)}function u(e){let t;try{t=new URL(e,n)}catch{}return t}},86999:function(e,t,n){"use strict";var r;Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{Postpone:function(){return d},createPostponedAbortSignal:function(){return b},createPrerenderState:function(){return c},formatDynamicAPIAccesses:function(){return _},markCurrentScopeAsDynamic:function(){return s},trackDynamicDataAccessed:function(){return f},trackDynamicFetch:function(){return p},usedDynamicAPIs:function(){return y}});let o=(r=n(2265))&&r.__esModule?r:{default:r},u=n(46177),l=n(36864),a=n(60934),i="function"==typeof o.default.unstable_postpone;function c(e){return{isDebugSkeleton:e,dynamicAccesses:[]}}function s(e,t){let n=(0,a.getPathname)(e.urlPathname);if(!e.isUnstableCacheCallback){if(e.dynamicShouldError)throw new l.StaticGenBailoutError(`Route ${n} with \`dynamic = "error"\` couldn't be rendered statically because it used \`${t}\`. See more info here: https://nextjs.org/docs/app/building-your-application/rendering/static-and-dynamic#dynamic-rendering`);if(e.prerenderState)h(e.prerenderState,t,n);else if(e.revalidate=0,e.isStaticGeneration){let r=new u.DynamicServerError(`Route ${n} couldn't be rendered statically because it used ${t}. See more info here: https://nextjs.org/docs/messages/dynamic-server-error`);throw e.dynamicUsageDescription=t,e.dynamicUsageStack=r.stack,r}}}function f(e,t){let n=(0,a.getPathname)(e.urlPathname);if(e.isUnstableCacheCallback)throw Error(`Route ${n} used "${t}" inside a function cached with "unstable_cache(...)". Accessing Dynamic data sources inside a cache scope is not supported. If you need this data inside a cached function use "${t}" outside of the cached function and pass the required dynamic data in as an argument. See more info here: https://nextjs.org/docs/app/api-reference/functions/unstable_cache`);if(e.dynamicShouldError)throw new l.StaticGenBailoutError(`Route ${n} with \`dynamic = "error"\` couldn't be rendered statically because it used \`${t}\`. See more info here: https://nextjs.org/docs/app/building-your-application/rendering/static-and-dynamic#dynamic-rendering`);if(e.prerenderState)h(e.prerenderState,t,n);else if(e.revalidate=0,e.isStaticGeneration){let r=new u.DynamicServerError(`Route ${n} couldn't be rendered statically because it used \`${t}\`. See more info here: https://nextjs.org/docs/messages/dynamic-server-error`);throw e.dynamicUsageDescription=t,e.dynamicUsageStack=r.stack,r}}function d({reason:e,prerenderState:t,pathname:n}){h(t,e,n)}function p(e,t){e.prerenderState&&h(e.prerenderState,t,e.urlPathname)}function h(e,t,n){v();let r=`Route ${n} needs to bail out of prerendering at this point because it used ${t}. React throws this special object to indicate where. It should not be caught by your own try/catch. Learn more: https://nextjs.org/docs/messages/ppr-caught-error`;e.dynamicAccesses.push({stack:e.isDebugSkeleton?Error().stack:void 0,expression:t}),o.default.unstable_postpone(r)}function y(e){return e.dynamicAccesses.length>0}function _(e){return e.dynamicAccesses.filter(e=>"string"==typeof e.stack&&e.stack.length>0).map(({expression:e,stack:t})=>(t=t.split("\n").slice(4).filter(e=>!(e.includes("node_modules/next/")||e.includes(" ()")||e.includes(" (node:"))).join("\n"),`Dynamic API Usage Debug - ${e}:
-${t}`))}function v(){if(!i)throw Error("Invariant: React.unstable_postpone is not defined. This suggests the wrong version of React was loaded. This is a bug in Next.js")}function b(e){v();let t=new AbortController;try{o.default.unstable_postpone(e)}catch(e){t.abort(e)}return t.signal}},87417:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"getSegmentParam",{enumerable:!0,get:function(){return o}});let r=n(91182);function o(e){let t=r.INTERCEPTION_ROUTE_MARKERS.find(t=>e.startsWith(t));return(t&&(e=e.slice(t.length)),e.startsWith("[[...")&&e.endsWith("]]"))?{type:"optional-catchall",param:e.slice(5,-2)}:e.startsWith("[...")&&e.endsWith("]")?{type:t?"catchall-intercepted":"catchall",param:e.slice(4,-1)}:e.startsWith("[")&&e.endsWith("]")?{type:t?"dynamic-intercepted":"dynamic",param:e.slice(1,-1)}:null}},70647:function(e,t){"use strict";var n,r;Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"HMR_ACTIONS_SENT_TO_BROWSER",{enumerable:!0,get:function(){return n}}),(r=n||(n={})).ADDED_PAGE="addedPage",r.REMOVED_PAGE="removedPage",r.RELOAD_PAGE="reloadPage",r.SERVER_COMPONENT_CHANGES="serverComponentChanges",r.MIDDLEWARE_CHANGES="middlewareChanges",r.CLIENT_CHANGES="clientChanges",r.SERVER_ONLY_CHANGES="serverOnlyChanges",r.SYNC="sync",r.BUILT="built",r.BUILDING="building",r.DEV_PAGES_MANIFEST_UPDATE="devPagesManifestUpdate",r.TURBOPACK_MESSAGE="turbopack-message",r.SERVER_ERROR="serverError",r.TURBOPACK_CONNECTED="turbopack-connected"},91182:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{INTERCEPTION_ROUTE_MARKERS:function(){return o},extractInterceptionRouteInformation:function(){return l},isInterceptionRouteAppPath:function(){return u}});let r=n(20926),o=["(..)(..)","(.)","(..)","(...)"];function u(e){return void 0!==e.split("/").find(e=>o.find(t=>e.startsWith(t)))}function l(e){let t,n,u;for(let r of e.split("/"))if(n=o.find(e=>r.startsWith(e))){[t,u]=e.split(n,2);break}if(!t||!n||!u)throw Error(`Invalid interception route: ${e}. Must be in the format //(..|...|..)(..)/`);switch(t=(0,r.normalizeAppPath)(t),n){case"(.)":u="/"===t?`/${u}`:t+"/"+u;break;case"(..)":if("/"===t)throw Error(`Invalid interception route: ${e}. Cannot use (..) marker at the root level, use (.) instead.`);u=t.split("/").slice(0,-1).concat(u).join("/");break;case"(...)":u="/"+u;break;case"(..)(..)":let l=t.split("/");if(l.length<=2)throw Error(`Invalid interception route: ${e}. Cannot use (..)(..) marker at the root level or one level up.`);u=l.slice(0,-2).concat(u).join("/");break;default:throw Error("Invariant: unexpected marker")}return{interceptingRoute:t,interceptedRoute:u}}},30650:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"ReflectAdapter",{enumerable:!0,get:function(){return n}});class n{static get(e,t,n){let r=Reflect.get(e,t,n);return"function"==typeof r?r.bind(e):r}static set(e,t,n,r){return Reflect.set(e,t,n,r)}static has(e,t){return Reflect.has(e,t)}static deleteProperty(e,t){return Reflect.deleteProperty(e,t)}}},61956:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{AppRouterContext:function(){return o},GlobalLayoutRouterContext:function(){return l},LayoutRouterContext:function(){return u},MissingSlotContext:function(){return i},TemplateContext:function(){return a}});let r=n(47043)._(n(2265)),o=r.default.createContext(null),u=r.default.createContext(null),l=r.default.createContext(null),a=r.default.createContext(null),i=r.default.createContext(new Set)},37207:function(e,t){"use strict";function n(e){let t=5381;for(let n=0;n>>0}function r(e){return n(e).toString(36).slice(0,5)}Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{djb2Hash:function(){return n},hexHash:function(){return r}})},48701:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"HeadManagerContext",{enumerable:!0,get:function(){return r}});let r=n(47043)._(n(2265)).default.createContext({})},79060:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{PathParamsContext:function(){return l},PathnameContext:function(){return u},SearchParamsContext:function(){return o}});let r=n(2265),o=(0,r.createContext)(null),u=(0,r.createContext)(null),l=(0,r.createContext)(null)},18993:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{BailoutToCSRError:function(){return r},isBailoutToCSRError:function(){return o}});let n="BAILOUT_TO_CLIENT_SIDE_RENDERING";class r extends Error{constructor(e){super("Bail out to client-side rendering: "+e),this.reason=e,this.digest=n}}function o(e){return"object"==typeof e&&null!==e&&"digest"in e&&e.digest===n}},78162:function(e,t){"use strict";function n(e){return e.startsWith("/")?e:"/"+e}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"ensureLeadingSlash",{enumerable:!0,get:function(){return n}})},2103:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{ActionQueueContext:function(){return a},createMutableActionQueue:function(){return s}});let r=n(53099),o=n(24673),u=n(91450),l=r._(n(2265)),a=l.default.createContext(null);function i(e,t){null!==e.pending&&(e.pending=e.pending.next,null!==e.pending?c({actionQueue:e,action:e.pending,setState:t}):e.needsRefresh&&(e.needsRefresh=!1,e.dispatch({type:o.ACTION_REFRESH,origin:window.location.origin},t)))}async function c(e){let{actionQueue:t,action:n,setState:r}=e,u=t.state;if(!u)throw Error("Invariant: Router state not initialized");t.pending=n;let l=n.payload,a=t.action(u,l);function c(e){n.discarded||(t.state=e,t.devToolsInstance&&t.devToolsInstance.send(l,e),i(t,r),n.resolve(e))}(0,o.isThenable)(a)?a.then(c,e=>{i(t,r),n.reject(e)}):c(a)}function s(){let e={state:null,dispatch:(t,n)=>(function(e,t,n){let r={resolve:n,reject:()=>{}};if(t.type!==o.ACTION_RESTORE){let e=new Promise((e,t)=>{r={resolve:e,reject:t}});(0,l.startTransition)(()=>{n(e)})}let u={payload:t,next:null,resolve:r.resolve,reject:r.reject};null===e.pending?(e.last=u,c({actionQueue:e,action:u,setState:n})):t.type===o.ACTION_NAVIGATE||t.type===o.ACTION_RESTORE?(e.pending.discarded=!0,e.last=u,e.pending.payload.type===o.ACTION_SERVER_ACTION&&(e.needsRefresh=!0),c({actionQueue:e,action:u,setState:n})):(null!==e.last&&(e.last.next=u),e.last=u)})(e,t,n),action:async(e,t)=>{if(null===e)throw Error("Invariant: Router state not initialized");return(0,u.reducer)(e,t)},pending:null,last:null};return e}},68498:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"addPathPrefix",{enumerable:!0,get:function(){return o}});let r=n(63381);function o(e,t){if(!e.startsWith("/")||!t)return e;let{pathname:n,query:o,hash:u}=(0,r.parsePath)(e);return""+t+n+o+u}},20926:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{normalizeAppPath:function(){return u},normalizeRscURL:function(){return l}});let r=n(78162),o=n(84541);function u(e){return(0,r.ensureLeadingSlash)(e.split("/").reduce((e,t,n,r)=>!t||(0,o.isGroupSegment)(t)||"@"===t[0]||("page"===t||"route"===t)&&n===r.length-1?e:e+"/"+t,""))}function l(e){return e.replace(/\.rsc($|\?)/,"$1")}},7092:function(e,t){"use strict";function n(e,t){if(void 0===t&&(t={}),t.onlyHashChange){e();return}let n=document.documentElement,r=n.style.scrollBehavior;n.style.scrollBehavior="auto",t.dontForceLayout||n.getClientRects(),e(),n.style.scrollBehavior=r}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"handleSmoothScroll",{enumerable:!0,get:function(){return n}})},86146:function(e,t){"use strict";function n(e){return/Googlebot|Mediapartners-Google|AdsBot-Google|googleweblight|Storebot-Google|Google-PageRenderer|Bingbot|BingPreview|Slurp|DuckDuckBot|baiduspider|yandex|sogou|LinkedInBot|bitlybot|tumblr|vkShare|quora link preview|facebookexternalhit|facebookcatalog|Twitterbot|applebot|redditbot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|ia_archiver/i.test(e)}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"isBot",{enumerable:!0,get:function(){return n}})},63381:function(e,t){"use strict";function n(e){let t=e.indexOf("#"),n=e.indexOf("?"),r=n>-1&&(t<0||n-1?{pathname:e.substring(0,r?n:t),query:r?e.substring(n,t>-1?t:void 0):"",hash:t>-1?e.slice(t):""}:{pathname:e,query:"",hash:""}}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"parsePath",{enumerable:!0,get:function(){return n}})},10580:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"pathHasPrefix",{enumerable:!0,get:function(){return o}});let r=n(63381);function o(e,t){if("string"!=typeof e)return!1;let{pathname:n}=(0,r.parsePath)(e);return n===t||n.startsWith(t+"/")}},26674:function(e,t){"use strict";function n(e){return e.replace(/\/$/,"")||"/"}Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"removeTrailingSlash",{enumerable:!0,get:function(){return n}})},84541:function(e,t){"use strict";function n(e){return"("===e[0]&&e.endsWith(")")}Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{DEFAULT_SEGMENT_KEY:function(){return o},PAGE_SEGMENT_KEY:function(){return r},isGroupSegment:function(){return n}});let r="__PAGE__",o="__DEFAULT__"},55501:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),function(e,t){for(var n in t)Object.defineProperty(e,n,{enumerable:!0,get:t[n]})}(t,{ServerInsertedHTMLContext:function(){return o},useServerInsertedHTML:function(){return u}});let r=n(53099)._(n(2265)),o=r.default.createContext(null);function u(e){let t=(0,r.useContext)(o);t&&t(e)}},31765:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"warnOnce",{enumerable:!0,get:function(){return n}});let n=e=>{}},47149:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"actionAsyncStorage",{enumerable:!0,get:function(){return r}});let r=(0,n(54832).createAsyncLocalStorage)();("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},54832:function(e,t){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"createAsyncLocalStorage",{enumerable:!0,get:function(){return u}});let n=Error("Invariant: AsyncLocalStorage accessed in runtime where it is not available");class r{disable(){throw n}getStore(){}run(){throw n}exit(){throw n}enterWith(){throw n}}let o=globalThis.AsyncLocalStorage;function u(){return o?new o:new r}("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},25575:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"requestAsyncStorage",{enumerable:!0,get:function(){return r}});let r=(0,n(54832).createAsyncLocalStorage)();("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},20030:function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),Object.defineProperty(t,"staticGenerationAsyncStorage",{enumerable:!0,get:function(){return r}});let r=(0,n(54832).createAsyncLocalStorage)();("function"==typeof t.default||"object"==typeof t.default&&null!==t.default)&&void 0===t.default.__esModule&&(Object.defineProperty(t.default,"__esModule",{value:!0}),Object.assign(t.default,t),e.exports=t.default)},34040:function(e,t,n){"use strict";var r=n(54887);t.createRoot=r.createRoot,t.hydrateRoot=r.hydrateRoot},54887:function(e,t,n){"use strict";!function e(){if("undefined"!=typeof __REACT_DEVTOOLS_GLOBAL_HOOK__&&"function"==typeof __REACT_DEVTOOLS_GLOBAL_HOOK__.checkDCE)try{__REACT_DEVTOOLS_GLOBAL_HOOK__.checkDCE(e)}catch(e){console.error(e)}}(),e.exports=n(84417)},97950:function(e,t,n){"use strict";var r=n(54887),o={stream:!0},u=new Map;function l(e){var t=n(e);return"function"!=typeof t.then||"fulfilled"===t.status?null:(t.then(function(e){t.status="fulfilled",t.value=e},function(e){t.status="rejected",t.reason=e}),t)}function a(){}var i=new Map,c=n.u;n.u=function(e){var t=i.get(e);return void 0!==t?t:c(e)};var s=r.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED.Dispatcher,f=Symbol.for("react.element"),d=Symbol.for("react.lazy"),p=Symbol.iterator,h=Array.isArray,y=Object.getPrototypeOf,_=Object.prototype,v=new WeakMap;function b(e,t,n,r){this.status=e,this.value=t,this.reason=n,this._response=r}function g(e){switch(e.status){case"resolved_model":E(e);break;case"resolved_module":w(e)}switch(e.status){case"fulfilled":return e.value;case"pending":case"blocked":case"cyclic":throw e;default:throw e.reason}}function m(e,t){for(var n=0;nh?(_=h,h=3,p++):(_=0,h=3);continue;case 2:44===(m=d[p++])?h=4:v=v<<4|(96d.length&&(m=-1)}var O=d.byteOffset+p;if(-1{let{defaultValue:n,value:d,onValueChange:v,placeholder:b="Select...",disabled:g=!1,icon:h,enableClear:x=!0,children:O,className:I}=e,y=(0,o._T)(e,["defaultValue","value","onValueChange","placeholder","disabled","icon","enableClear","children","className"]),[S,R]=(0,r.useState)(""),[T,E]=(0,i.Z)(n,d),{reactElementChildren:C,valueToNameMapping:k}=(0,r.useMemo)(()=>{let e=r.Children.toArray(O).filter(r.isValidElement);return{reactElementChildren:e,valueToNameMapping:(0,f.sl)(e)}},[O]),M=(0,r.useMemo)(()=>(0,f.n0)(S,C),[S,C]);return r.createElement(l.h,Object.assign({as:"div",ref:t,defaultValue:T,value:T,onChange:e=>{null==v||v(e),E(e)},disabled:g,className:(0,c.q)("w-full min-w-[10rem] relative text-tremor-default",I)},y),e=>{let{value:t}=e;return r.createElement(r.Fragment,null,r.createElement(l.h.Button,{className:"w-full"},h&&r.createElement("span",{className:(0,c.q)("absolute inset-y-0 left-0 flex items-center ml-px pl-2.5")},r.createElement(h,{className:(0,c.q)(p("Icon"),"flex-none h-5 w-5","text-tremor-content-subtle","dark:text-dark-tremor-content-subtle")})),r.createElement(l.h.Input,{className:(0,c.q)("w-full outline-none text-left whitespace-nowrap truncate rounded-tremor-default focus:ring-2 transition duration-100 text-tremor-default pr-14 border py-2","border-tremor-border shadow-tremor-input focus:border-tremor-brand-subtle focus:ring-tremor-brand-muted","dark:border-dark-tremor-border dark:shadow-dark-tremor-input dark:focus:border-dark-tremor-brand-subtle dark:focus:ring-dark-tremor-brand-muted",h?"pl-10":"pl-3",g?"placeholder:text-tremor-content-subtle dark:placeholder:text-tremor-content-subtle":"placeholder:text-tremor-content dark:placeholder:text-tremor-content",(0,f.um)((0,f.Uh)(t),g)),placeholder:b,onChange:e=>R(e.target.value),displayValue:e=>{var t;return null!==(t=k.get(e))&&void 0!==t?t:""}}),r.createElement("div",{className:(0,c.q)("absolute inset-y-0 right-0 flex items-center pr-2.5")},r.createElement(u.Z,{className:(0,c.q)(p("arrowDownIcon"),"flex-none h-5 w-5","text-tremor-content-subtle","dark:text-dark-tremor-content-subtle")}))),x&&T?r.createElement("button",{type:"button",className:(0,c.q)("absolute inset-y-0 right-0 flex items-center mr-8"),onClick:e=>{e.preventDefault(),E(""),R(""),null==v||v("")}},r.createElement(s.Z,{className:(0,c.q)(m("clearIcon"),"flex-none h-4 w-4","text-tremor-content-subtle","dark:text-dark-tremor-content-subtle")})):null,M.length>0&&r.createElement(a.u,{className:"absolute z-10 w-full",enter:"transition ease duration-100 transform",enterFrom:"opacity-0 -translate-y-4",enterTo:"opacity-100 translate-y-0",leave:"transition ease duration-100 transform",leaveFrom:"opacity-100 translate-y-0",leaveTo:"opacity-0 -translate-y-4"},r.createElement(l.h.Options,{className:(0,c.q)("divide-y overflow-y-auto outline-none rounded-tremor-default text-tremor-default max-h-[228px] left-0 border my-1","bg-tremor-background border-tremor-border divide-tremor-border shadow-tremor-dropdown","dark:bg-dark-tremor-background dark:border-dark-tremor-border dark:divide-dark-tremor-border dark:shadow-dark-tremor-dropdown")},M)))})});v.displayName="SearchSelect"},70450:function(e,t,n){n.d(t,{Z:function(){return s}});var o=n(5853),r=n(2265),i=n(97324),l=n(1153),a=n(34237);let u=(0,l.fn)("SearchSelectItem"),s=r.forwardRef((e,t)=>{let{value:n,icon:l,className:s,children:c}=e,d=(0,o._T)(e,["value","icon","className","children"]);return r.createElement(a.h.Option,Object.assign({className:(0,i.q)(u("root"),"flex justify-start items-center cursor-default text-tremor-default p-2.5","ui-active:bg-tremor-background-muted ui-active:text-tremor-content-strong ui-selected:text-tremor-content-strong ui-selected:bg-tremor-background-muted text-tremor-content-emphasis","dark:ui-active:bg-dark-tremor-background-muted dark:ui-active:text-dark-tremor-content-strong dark:ui-selected:text-dark-tremor-content-strong dark:ui-selected:bg-dark-tremor-background-muted dark:text-dark-tremor-content-emphasis",s),ref:t,key:n,value:n},d),l&&r.createElement(l,{className:(0,i.q)(u("icon"),"flex-none h-5 w-5 mr-3","text-tremor-content-subtle","dark:text-dark-tremor-content-subtle")}),r.createElement("span",{className:"whitespace-nowrap truncate"},null!=c?c:n))});s.displayName="SearchSelectItem"},34237:function(e,t,n){let o,r,i,l;function a(){return(a=Object.assign?Object.assign.bind():function(e){for(var t=1;t=0;i--){var l=e[i];if(!n.has(l.lane)){var a=r.get(l.lane);if(null==a||l.end>a.end?r.set(l.lane,l):l.end0?Math.min.apply(Math,o.pendingMeasuredCacheIndexes):0;o.pendingMeasuredCacheIndexes=[];for(var u=o.measurementsCache.slice(0,a),s=a;s0&&t>0?function(e){for(var t=e.measurements,n=e.outerSize,o=e.scrollOffset,r=t.length-1,i=O(0,r,function(e){return t[e].start},o),l=i;l=o.scrollOffset+n?"end":"start"),"start"===t||("end"===t?e-=n:"center"===t&&(e-=n/2));var r=o.options.horizontal?"scrollWidth":"scrollHeight";return Math.max(Math.min((o.scrollElement?"document"in o.scrollElement?o.scrollElement.document.documentElement[r]:o.scrollElement[r]:0)-o.getSize(),e),0)},this.getOffsetForIndex=function(e,t){void 0===t&&(t="auto"),e=Math.max(0,Math.min(e,o.options.count-1));var n=f(o.getMeasurements()[e]);if("auto"===t){if(n.end>=o.scrollOffset+o.getSize()-o.options.scrollPaddingEnd)t="end";else{if(!(n.start<=o.scrollOffset+o.options.scrollPaddingStart))return[o.scrollOffset,t];t="start"}}var r="end"===t?n.end+o.options.scrollPaddingEnd:n.start-o.options.scrollPaddingStart;return[o.getOffsetForAlignment(r,t),t]},this.isDynamicMode=function(){return o.measureElementCache.size>0},this.cancelScrollToIndex=function(){null!==o.scrollToIndexTimeoutId&&(clearTimeout(o.scrollToIndexTimeoutId),o.scrollToIndexTimeoutId=null)},this.scrollToOffset=function(e,t){var n=void 0===t?{}:t,r=n.align,i=n.behavior;o.cancelScrollToIndex(),"smooth"===i&&o.isDynamicMode()&&console.warn("The `smooth` scroll behavior is not fully supported with dynamic size."),o._scrollToOffset(o.getOffsetForAlignment(e,void 0===r?"start":r),{adjustments:void 0,behavior:i})},this.scrollToIndex=function(e,t){var n=void 0===t?{}:t,r=n.align,i=n.behavior;e=Math.max(0,Math.min(e,o.options.count-1)),o.cancelScrollToIndex(),"smooth"===i&&o.isDynamicMode()&&console.warn("The `smooth` scroll behavior is not fully supported with dynamic size.");var l=o.getOffsetForIndex(e,void 0===r?"auto":r),a=l[0],u=l[1];o._scrollToOffset(a,{adjustments:void 0,behavior:i}),"smooth"!==i&&o.isDynamicMode()&&(o.scrollToIndexTimeoutId=setTimeout(function(){o.scrollToIndexTimeoutId=null,o.measureElementCache.has(o.options.getItemKey(e))&&1>Math.abs(o.getOffsetForIndex(e,u)[0]-o.scrollOffset)||o.scrollToIndex(e,{align:u,behavior:i})}))},this.scrollBy=function(e,t){var n=(void 0===t?{}:t).behavior;o.cancelScrollToIndex(),"smooth"===n&&o.isDynamicMode()&&console.warn("The `smooth` scroll behavior is not fully supported with dynamic size."),o._scrollToOffset(o.scrollOffset+e,{adjustments:void 0,behavior:n})},this.getTotalSize=function(){var e;return((null==(e=o.getMeasurements()[o.options.count-1])?void 0:e.end)||o.options.paddingStart)-o.options.scrollMargin+o.options.paddingEnd},this._scrollToOffset=function(e,t){var n=t.adjustments,r=t.behavior;o.options.scrollToFn(e,{behavior:r,adjustments:n},o)},this.measure=function(){o.itemSizeCache=new Map,o.notify(!1)},this.setOptions(e),this.scrollRect=this.options.initialRect,this.scrollOffset=this.options.initialOffset,this.measurementsCache=this.options.initialMeasurementsCache,this.measurementsCache.forEach(function(e){o.itemSizeCache.set(e.key,e.size)}),this.maybeNotify()},O=function(e,t,n,o){for(;e<=t;){var r=(e+t)/2|0,i=n(r);if(io))return r;t=r-1}}return e>0?e-1:0},I="undefined"!=typeof document?u.useLayoutEffect:u.useEffect,y=n(78138),S=n(62963),R=n(90945),T=n(13323),E=n(17684),C=n(64518),k=n(31948),M=n(32539),w=n(40048),z=n(80004),P=n(93689),F=n(15518),D=n(40293);function N(e,t){let n=(0,u.useRef)([]),o=(0,T.z)(e);(0,u.useEffect)(()=>{let e=[...n.current];for(let[r,i]of t.entries())if(n.current[r]!==i){let r=o(t,e);return n.current=t,r}},[o,...t])}var A=n(38198),_=n(37863);let L=[];!function(e){function t(){"loading"!==document.readyState&&(e(),document.removeEventListener("DOMContentLoaded",t))}"undefined"!=typeof window&&"undefined"!=typeof document&&(document.addEventListener("DOMContentLoaded",t),t())}(()=>{function e(e){e.target instanceof HTMLElement&&e.target!==document.body&&L[0]!==e.target&&(L.unshift(e.target),(L=L.filter(e=>null!=e&&e.isConnected)).splice(10))}window.addEventListener("click",e,{capture:!0}),window.addEventListener("mousedown",e,{capture:!0}),window.addEventListener("focus",e,{capture:!0}),document.body.addEventListener("click",e,{capture:!0}),document.body.addEventListener("mousedown",e,{capture:!0}),document.body.addEventListener("focus",e,{capture:!0})});var V=n(47634),j=n(34778),q=n(16015),B=n(37105),K=n(56314),U=n(24536),Z=n(52108),Y=n(27847),G=n(37388),H=n(40257),W=((o=W||{})[o.Open=0]="Open",o[o.Closed=1]="Closed",o),J=((r=J||{})[r.Single=0]="Single",r[r.Multi=1]="Multi",r),Q=((i=Q||{})[i.Pointer=0]="Pointer",i[i.Focus=1]="Focus",i[i.Other=2]="Other",i),X=((l=X||{})[l.OpenCombobox=0]="OpenCombobox",l[l.CloseCombobox=1]="CloseCombobox",l[l.GoToOption=2]="GoToOption",l[l.RegisterOption=3]="RegisterOption",l[l.UnregisterOption=4]="UnregisterOption",l[l.RegisterLabel=5]="RegisterLabel",l[l.SetActivationTrigger=6]="SetActivationTrigger",l[l.UpdateVirtualOptions=7]="UpdateVirtualOptions",l);function $(e){let t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:e=>e,n=null!==e.activeOptionIndex?e.options[e.activeOptionIndex]:null,o=t(e.options.slice()),r=o.length>0&&null!==o[0].dataRef.current.order?o.sort((e,t)=>e.dataRef.current.order-t.dataRef.current.order):(0,B.z2)(o,e=>e.dataRef.current.domRef.current),i=n?r.indexOf(n):null;return -1===i&&(i=null),{options:r,activeOptionIndex:i}}let ee={1(e){var t;return null!=(t=e.dataRef.current)&&t.disabled||1===e.comboboxState?e:{...e,activeOptionIndex:null,comboboxState:1}},0(e){var t,n;if(null!=(t=e.dataRef.current)&&t.disabled||0===e.comboboxState)return e;if(null!=(n=e.dataRef.current)&&n.value){let t=e.dataRef.current.calculateIndex(e.dataRef.current.value);if(-1!==t)return{...e,activeOptionIndex:t,comboboxState:0}}return{...e,comboboxState:0}},2(e,t){var n,o,r,i,l;if(null!=(n=e.dataRef.current)&&n.disabled||null!=(o=e.dataRef.current)&&o.optionsRef.current&&!(null!=(r=e.dataRef.current)&&r.optionsPropsRef.current.static)&&1===e.comboboxState)return e;if(e.virtual){let n=t.focus===j.T.Specific?t.idx:(0,j.d)(t,{resolveItems:()=>e.virtual.options,resolveActiveIndex:()=>{var t,n;return null!=(n=null!=(t=e.activeOptionIndex)?t:e.virtual.options.findIndex(t=>!e.virtual.disabled(t)))?n:null},resolveDisabled:e.virtual.disabled,resolveId(){throw Error("Function not implemented.")}}),o=null!=(i=t.trigger)?i:2;return e.activeOptionIndex===n&&e.activationTrigger===o?e:{...e,activeOptionIndex:n,activationTrigger:o}}let a=$(e);if(null===a.activeOptionIndex){let e=a.options.findIndex(e=>!e.dataRef.current.disabled);-1!==e&&(a.activeOptionIndex=e)}let u=t.focus===j.T.Specific?t.idx:(0,j.d)(t,{resolveItems:()=>a.options,resolveActiveIndex:()=>a.activeOptionIndex,resolveId:e=>e.id,resolveDisabled:e=>e.dataRef.current.disabled}),s=null!=(l=t.trigger)?l:2;return e.activeOptionIndex===u&&e.activationTrigger===s?e:{...e,...a,activeOptionIndex:u,activationTrigger:s}},3:(e,t)=>{var n,o,r;if(null!=(n=e.dataRef.current)&&n.virtual)return{...e,options:[...e.options,t.payload]};let i=t.payload,l=$(e,e=>(e.push(i),e));null===e.activeOptionIndex&&null!=(o=e.dataRef.current)&&o.isSelected(t.payload.dataRef.current.value)&&(l.activeOptionIndex=l.options.indexOf(i));let a={...e,...l,activationTrigger:2};return null!=(r=e.dataRef.current)&&r.__demoMode&&void 0===e.dataRef.current.value&&(a.activeOptionIndex=0),a},4:(e,t)=>{var n;if(null!=(n=e.dataRef.current)&&n.virtual)return{...e,options:e.options.filter(e=>e.id!==t.id)};let o=$(e,e=>{let n=e.findIndex(e=>e.id===t.id);return -1!==n&&e.splice(n,1),e});return{...e,...o,activationTrigger:2}},5:(e,t)=>e.labelId===t.id?e:{...e,labelId:t.id},6:(e,t)=>e.activationTrigger===t.trigger?e:{...e,activationTrigger:t.trigger},7:(e,t)=>{var n;if((null==(n=e.virtual)?void 0:n.options)===t.options)return e;let o=e.activeOptionIndex;if(null!==e.activeOptionIndex){let n=t.options.indexOf(e.virtual.options[e.activeOptionIndex]);o=-1!==n?n:null}return{...e,activeOptionIndex:o,virtual:Object.assign({},e.virtual,{options:t.options})}}},et=(0,u.createContext)(null);function en(e){let t=(0,u.useContext)(et);if(null===t){let t=Error("<".concat(e," /> is missing a parent component."));throw Error.captureStackTrace&&Error.captureStackTrace(t,en),t}return t}et.displayName="ComboboxActionsContext";let eo=(0,u.createContext)(null);function er(e){var t,n,o,r,i,l;let c=el("VirtualProvider"),[d,f]=(0,u.useMemo)(()=>{let e=c.optionsRef.current;if(!e)return[0,0];let t=window.getComputedStyle(e);return[parseFloat(t.paddingBlockStart||t.paddingTop),parseFloat(t.paddingBlockEnd||t.paddingBottom)]},[c.optionsRef.current]),p=(n={scrollPaddingStart:d,scrollPaddingEnd:f,count:c.virtual.options.length,estimateSize:()=>40,getScrollElement(){var e;return null!=(e=c.optionsRef.current)?e:null},overscan:12},o=a({observeElementRect:v,observeElementOffset:b,scrollToFn:h},n),r=u.useReducer(function(){return{}},{})[1],i=a({},o,{onChange:function(e,t){t?(0,s.flushSync)(r):r(),null==o.onChange||o.onChange(e,t)}}),(l=u.useState(function(){return new x(i)})[0]).setOptions(i),u.useEffect(function(){return l._didMount()},[]),I(function(){return l._willUpdate()}),l),[m,g]=(0,u.useState)(0);return(0,C.e)(()=>{g(e=>e+1)},[null==(t=c.virtual)?void 0:t.options]),u.createElement(eo.Provider,{value:p},u.createElement("div",{style:{position:"relative",width:"100%",height:"".concat(p.getTotalSize(),"px")},ref:e=>{e&&(void 0===H||void 0===H.env.JEST_WORKER_ID)&&0!==c.activationTrigger&&null!==c.activeOptionIndex&&c.virtual.options.length>c.activeOptionIndex&&p.scrollToIndex(c.activeOptionIndex)}},p.getVirtualItems().map(t=>{var n;return u.createElement(u.Fragment,{key:t.key},u.cloneElement(null==(n=e.children)?void 0:n.call(e,{option:c.virtual.options[t.index],open:0===c.comboboxState}),{key:"".concat(m,"-").concat(t.key),"data-index":t.index,"aria-setsize":c.virtual.options.length,"aria-posinset":t.index+1,style:{position:"absolute",top:0,left:0,transform:"translateY(".concat(t.start,"px)"),overflowAnchor:"none"}}))})))}let ei=(0,u.createContext)(null);function el(e){let t=(0,u.useContext)(ei);if(null===t){let t=Error("<".concat(e," /> is missing a parent component."));throw Error.captureStackTrace&&Error.captureStackTrace(t,el),t}return t}function ea(e,t){return(0,U.E)(t.type,ee,e,t)}ei.displayName="ComboboxDataContext";let eu=u.Fragment,es=Y.AN.RenderStrategy|Y.AN.Static,ec=(0,Y.yV)(function(e,t){let{value:n,defaultValue:o,onChange:r,form:i,name:l,by:a=null,disabled:s=!1,__demoMode:c=!1,nullable:d=!1,multiple:f=!1,immediate:p=!1,virtual:m=null,...v}=e,[b=f?[]:void 0,g]=(0,S.q)(n,r,o),[h,x]=(0,u.useReducer)(ea,{dataRef:(0,u.createRef)(),comboboxState:c?0:1,options:[],virtual:null,activeOptionIndex:null,activationTrigger:2,labelId:null}),O=(0,u.useRef)(!1),I=(0,u.useRef)({static:!1,hold:!1}),y=(0,u.useRef)(null),E=(0,u.useRef)(null),k=(0,u.useRef)(null),w=(0,u.useRef)(null),z=(0,T.z)("string"==typeof a?(e,t)=>(null==e?void 0:e[a])===(null==t?void 0:t[a]):null!=a?a:(e,t)=>e===t),P=(0,T.z)(e=>h.options.findIndex(t=>z(t.dataRef.current.value,e))),F=(0,u.useCallback)(e=>(0,U.E)(N.mode,{1:()=>b.some(t=>z(t,e)),0:()=>z(b,e)}),[b]),D=(0,T.z)(e=>h.activeOptionIndex===P(e)),N=(0,u.useMemo)(()=>({...h,immediate:!1,optionsPropsRef:I,labelRef:y,inputRef:E,buttonRef:k,optionsRef:w,value:b,defaultValue:o,disabled:s,mode:f?1:0,virtual:h.virtual,get activeOptionIndex(){if(O.current&&null===h.activeOptionIndex&&h.options.length>0){let e=h.options.findIndex(e=>!e.dataRef.current.disabled);if(-1!==e)return e}return h.activeOptionIndex},calculateIndex:P,compare:z,isSelected:F,isActive:D,nullable:d,__demoMode:c}),[b,o,s,f,d,c,h,null]);(0,C.e)(()=>{},[null,void 0]),(0,C.e)(()=>{h.dataRef.current=N},[N]),(0,M.O)([N.buttonRef,N.inputRef,N.optionsRef],()=>Q.closeCombobox(),0===N.comboboxState);let L=(0,u.useMemo)(()=>{var e,t,n;return{open:0===N.comboboxState,disabled:s,activeIndex:N.activeOptionIndex,activeOption:null===N.activeOptionIndex?null:N.virtual?N.virtual.options[null!=(e=N.activeOptionIndex)?e:0]:null!=(n=null==(t=N.options[N.activeOptionIndex])?void 0:t.dataRef.current.value)?n:null,value:b}},[N,s,b]),V=(0,T.z)(()=>{if(null!==N.activeOptionIndex){if(N.virtual)W(N.virtual.options[N.activeOptionIndex]);else{let{dataRef:e}=N.options[N.activeOptionIndex];W(e.current.value)}Q.goToOption(j.T.Specific,N.activeOptionIndex)}}),q=(0,T.z)(()=>{x({type:0}),O.current=!0}),B=(0,T.z)(()=>{x({type:1}),O.current=!1}),Z=(0,T.z)((e,t,n)=>(O.current=!1,e===j.T.Specific?x({type:2,focus:j.T.Specific,idx:t,trigger:n}):x({type:2,focus:e,trigger:n}))),G=(0,T.z)((e,t)=>(x({type:3,payload:{id:e,dataRef:t}}),()=>{N.isActive(t.current.value)&&(O.current=!0),x({type:4,id:e})})),H=(0,T.z)(e=>(x({type:5,id:e}),()=>x({type:5,id:null}))),W=(0,T.z)(e=>(0,U.E)(N.mode,{0:()=>null==g?void 0:g(e),1(){let t=N.value.slice(),n=t.findIndex(t=>z(t,e));return -1===n?t.push(e):t.splice(n,1),null==g?void 0:g(t)}})),J=(0,T.z)(e=>{x({type:6,trigger:e})}),Q=(0,u.useMemo)(()=>({onChange:W,registerOption:G,registerLabel:H,goToOption:Z,closeCombobox:B,openCombobox:q,setActivationTrigger:J,selectActiveOption:V}),[]),X=(0,u.useRef)(null),$=(0,R.G)();return(0,u.useEffect)(()=>{X.current&&void 0!==o&&$.addEventListener(X.current,"reset",()=>{null==g||g(o)})},[X,g]),u.createElement(et.Provider,{value:Q},u.createElement(ei.Provider,{value:N},u.createElement(_.up,{value:(0,U.E)(N.comboboxState,{0:_.ZM.Open,1:_.ZM.Closed})},null!=l&&null!=b&&(0,K.t)({[l]:b}).map((e,t)=>{let[n,o]=e;return u.createElement(A._,{features:A.A.Hidden,ref:0===t?e=>{var t;X.current=null!=(t=null==e?void 0:e.closest("form"))?t:null}:void 0,...(0,Y.oA)({key:n,as:"input",type:"hidden",hidden:!0,readOnly:!0,form:i,name:n,value:o})})}),(0,Y.sY)({ourProps:null===t?{}:{ref:t},theirProps:v,slot:L,defaultTag:eu,name:"Combobox"}))))}),ed=(0,Y.yV)(function(e,t){var n;let o=el("Combobox.Button"),r=en("Combobox.Button"),i=(0,P.T)(o.buttonRef,t),l=(0,E.M)(),{id:a="headlessui-combobox-button-".concat(l),...s}=e,c=(0,R.G)(),d=(0,T.z)(e=>{switch(e.key){case G.R.ArrowDown:return e.preventDefault(),e.stopPropagation(),1===o.comboboxState&&r.openCombobox(),c.nextFrame(()=>{var e;return null==(e=o.inputRef.current)?void 0:e.focus({preventScroll:!0})});case G.R.ArrowUp:return e.preventDefault(),e.stopPropagation(),1===o.comboboxState&&(r.openCombobox(),c.nextFrame(()=>{o.value||r.goToOption(j.T.Last)})),c.nextFrame(()=>{var e;return null==(e=o.inputRef.current)?void 0:e.focus({preventScroll:!0})});case G.R.Escape:return 0!==o.comboboxState?void 0:(e.preventDefault(),o.optionsRef.current&&!o.optionsPropsRef.current.static&&e.stopPropagation(),r.closeCombobox(),c.nextFrame(()=>{var e;return null==(e=o.inputRef.current)?void 0:e.focus({preventScroll:!0})}));default:return}}),f=(0,T.z)(e=>{if((0,V.P)(e.currentTarget))return e.preventDefault();0===o.comboboxState?r.closeCombobox():(e.preventDefault(),r.openCombobox()),c.nextFrame(()=>{var e;return null==(e=o.inputRef.current)?void 0:e.focus({preventScroll:!0})})}),p=(0,y.v)(()=>{if(o.labelId)return[o.labelId,a].join(" ")},[o.labelId,a]),m=(0,u.useMemo)(()=>({open:0===o.comboboxState,disabled:o.disabled,value:o.value}),[o]),v={ref:i,id:a,type:(0,z.f)(e,o.buttonRef),tabIndex:-1,"aria-haspopup":"listbox","aria-controls":null==(n=o.optionsRef.current)?void 0:n.id,"aria-expanded":0===o.comboboxState,"aria-labelledby":p,disabled:o.disabled,onClick:f,onKeyDown:d};return(0,Y.sY)({ourProps:v,theirProps:s,slot:m,defaultTag:"button",name:"Combobox.Button"})}),ef=Object.assign(ec,{Input:(0,Y.yV)(function(e,t){var n,o,r,i,l,a;let s=(0,E.M)(),{id:c="headlessui-combobox-input-".concat(s),onChange:d,displayValue:f,type:p="text",...m}=e,v=el("Combobox.Input"),b=en("Combobox.Input"),g=(0,P.T)(v.inputRef,t),h=(0,w.i)(v.inputRef),x=(0,u.useRef)(!1),O=(0,R.G)(),I=(0,T.z)(()=>{b.onChange(null),v.optionsRef.current&&(v.optionsRef.current.scrollTop=0),b.goToOption(j.T.Nothing)});N((e,t)=>{let[n,o]=e,[r,i]=t;if(x.current)return;let l=v.inputRef.current;l&&((0===i&&1===o||n!==r)&&(l.value=n),requestAnimationFrame(()=>{if(x.current||!l||(null==h?void 0:h.activeElement)!==l)return;let{selectionStart:e,selectionEnd:t}=l;0===Math.abs((null!=t?t:0)-(null!=e?e:0))&&0===e&&l.setSelectionRange(l.value.length,l.value.length)}))},["function"==typeof f&&void 0!==v.value?null!=(a=f(v.value))?a:"":"string"==typeof v.value?v.value:"",v.comboboxState,h]),N((e,t)=>{let[n]=e,[o]=t;if(0===n&&1===o){if(x.current)return;let e=v.inputRef.current;if(!e)return;let t=e.value,{selectionStart:n,selectionEnd:o,selectionDirection:r}=e;e.value="",e.value=t,null!==r?e.setSelectionRange(n,o,r):e.setSelectionRange(n,o)}},[v.comboboxState]);let S=(0,u.useRef)(!1),C=(0,T.z)(()=>{S.current=!0}),k=(0,T.z)(()=>{O.nextFrame(()=>{S.current=!1})}),M=(0,T.z)(e=>{switch(x.current=!0,e.key){case G.R.Enter:if(x.current=!1,0!==v.comboboxState||S.current)return;if(e.preventDefault(),e.stopPropagation(),null===v.activeOptionIndex){b.closeCombobox();return}b.selectActiveOption(),0===v.mode&&b.closeCombobox();break;case G.R.ArrowDown:return x.current=!1,e.preventDefault(),e.stopPropagation(),(0,U.E)(v.comboboxState,{0:()=>b.goToOption(j.T.Next),1:()=>b.openCombobox()});case G.R.ArrowUp:return x.current=!1,e.preventDefault(),e.stopPropagation(),(0,U.E)(v.comboboxState,{0:()=>b.goToOption(j.T.Previous),1:()=>{b.openCombobox(),O.nextFrame(()=>{v.value||b.goToOption(j.T.Last)})}});case G.R.Home:if(e.shiftKey)break;return x.current=!1,e.preventDefault(),e.stopPropagation(),b.goToOption(j.T.First);case G.R.PageUp:return x.current=!1,e.preventDefault(),e.stopPropagation(),b.goToOption(j.T.First);case G.R.End:if(e.shiftKey)break;return x.current=!1,e.preventDefault(),e.stopPropagation(),b.goToOption(j.T.Last);case G.R.PageDown:return x.current=!1,e.preventDefault(),e.stopPropagation(),b.goToOption(j.T.Last);case G.R.Escape:return x.current=!1,0!==v.comboboxState?void 0:(e.preventDefault(),v.optionsRef.current&&!v.optionsPropsRef.current.static&&e.stopPropagation(),v.nullable&&0===v.mode&&null===v.value&&I(),b.closeCombobox());case G.R.Tab:if(x.current=!1,0!==v.comboboxState)return;0===v.mode&&1!==v.activationTrigger&&b.selectActiveOption(),b.closeCombobox()}}),z=(0,T.z)(e=>{null==d||d(e),v.nullable&&0===v.mode&&""===e.target.value&&I(),b.openCombobox()}),F=(0,T.z)(e=>{var t,n,o;let r=null!=(t=e.relatedTarget)?t:L.find(t=>t!==e.currentTarget);if(x.current=!1,!(null!=(n=v.optionsRef.current)&&n.contains(r))&&!(null!=(o=v.buttonRef.current)&&o.contains(r))&&0===v.comboboxState)return e.preventDefault(),0===v.mode&&(v.nullable&&null===v.value?I():1!==v.activationTrigger&&b.selectActiveOption()),b.closeCombobox()}),D=(0,T.z)(e=>{var t,n,o;let r=null!=(t=e.relatedTarget)?t:L.find(t=>t!==e.currentTarget);null!=(n=v.buttonRef.current)&&n.contains(r)||null!=(o=v.optionsRef.current)&&o.contains(r)||v.disabled||v.immediate&&0!==v.comboboxState&&(b.openCombobox(),O.nextFrame(()=>{b.setActivationTrigger(1)}))}),A=(0,y.v)(()=>{if(v.labelId)return[v.labelId].join(" ")},[v.labelId]),_=(0,u.useMemo)(()=>({open:0===v.comboboxState,disabled:v.disabled}),[v]),V={ref:g,id:c,role:"combobox",type:p,"aria-controls":null==(n=v.optionsRef.current)?void 0:n.id,"aria-expanded":0===v.comboboxState,"aria-activedescendant":null===v.activeOptionIndex?void 0:v.virtual?null==(o=v.options.find(e=>{var t;return!(null!=(t=v.virtual)&&t.disabled(e.dataRef.current.value))&&v.compare(e.dataRef.current.value,v.virtual.options[v.activeOptionIndex])}))?void 0:o.id:null==(r=v.options[v.activeOptionIndex])?void 0:r.id,"aria-labelledby":A,"aria-autocomplete":"list",defaultValue:null!=(l=null!=(i=e.defaultValue)?i:void 0!==v.defaultValue?null==f?void 0:f(v.defaultValue):null)?l:v.defaultValue,disabled:v.disabled,onCompositionStart:C,onCompositionEnd:k,onKeyDown:M,onChange:z,onFocus:D,onBlur:F};return(0,Y.sY)({ourProps:V,theirProps:m,slot:_,defaultTag:"input",name:"Combobox.Input"})}),Button:ed,Label:(0,Y.yV)(function(e,t){let n=(0,E.M)(),{id:o="headlessui-combobox-label-".concat(n),...r}=e,i=el("Combobox.Label"),l=en("Combobox.Label"),a=(0,P.T)(i.labelRef,t);(0,C.e)(()=>l.registerLabel(o),[o]);let s=(0,T.z)(()=>{var e;return null==(e=i.inputRef.current)?void 0:e.focus({preventScroll:!0})}),c=(0,u.useMemo)(()=>({open:0===i.comboboxState,disabled:i.disabled}),[i]);return(0,Y.sY)({ourProps:{ref:a,id:o,onClick:s},theirProps:r,slot:c,defaultTag:"label",name:"Combobox.Label"})}),Options:(0,Y.yV)(function(e,t){let n=(0,E.M)(),{id:o="headlessui-combobox-options-".concat(n),hold:r=!1,...i}=e,l=el("Combobox.Options"),a=(0,P.T)(l.optionsRef,t),s=(0,_.oJ)(),c=null!==s?(s&_.ZM.Open)===_.ZM.Open:0===l.comboboxState;(0,C.e)(()=>{var t;l.optionsPropsRef.current.static=null!=(t=e.static)&&t},[l.optionsPropsRef,e.static]),(0,C.e)(()=>{l.optionsPropsRef.current.hold=r},[l.optionsPropsRef,r]),function(e){let{container:t,accept:n,walk:o,enabled:r=!0}=e,i=(0,u.useRef)(n),l=(0,u.useRef)(o);(0,u.useEffect)(()=>{i.current=n,l.current=o},[n,o]),(0,C.e)(()=>{if(!t||!r)return;let e=(0,D.r)(t);if(!e)return;let n=i.current,o=l.current,a=Object.assign(e=>n(e),{acceptNode:n}),u=e.createTreeWalker(t,NodeFilter.SHOW_ELEMENT,a,!1);for(;u.nextNode();)o(u.currentNode)},[t,r,i,l])}({container:l.optionsRef.current,enabled:0===l.comboboxState,accept:e=>"option"===e.getAttribute("role")?NodeFilter.FILTER_REJECT:e.hasAttribute("role")?NodeFilter.FILTER_SKIP:NodeFilter.FILTER_ACCEPT,walk(e){e.setAttribute("role","none")}});let d=(0,y.v)(()=>{var e,t;return null!=(t=l.labelId)?t:null==(e=l.buttonRef.current)?void 0:e.id},[l.labelId,l.buttonRef.current]),f=(0,u.useMemo)(()=>({open:0===l.comboboxState,option:void 0}),[l]),p={"aria-labelledby":d,role:"listbox","aria-multiselectable":1===l.mode||void 0,id:o,ref:a};return l.virtual&&0===l.comboboxState&&Object.assign(i,{children:u.createElement(er,null,i.children)}),(0,Y.sY)({ourProps:p,theirProps:i,slot:f,defaultTag:"ul",features:es,visible:c,name:"Combobox.Options"})}),Option:(0,Y.yV)(function(e,t){var n;let o=(0,E.M)(),{id:r="headlessui-combobox-option-".concat(o),disabled:i=!1,value:l,order:a=null,...s}=e,c=el("Combobox.Option"),d=en("Combobox.Option"),f=c.virtual?c.activeOptionIndex===c.calculateIndex(l):null!==c.activeOptionIndex&&(null==(n=c.options[c.activeOptionIndex])?void 0:n.id)===r,p=c.isSelected(l),m=(0,u.useRef)(null),v=(0,k.E)({disabled:i,value:l,domRef:m,order:a}),b=(0,u.useContext)(eo),g=(0,P.T)(t,m,b?b.measureElement:null),h=(0,T.z)(()=>d.onChange(l));(0,C.e)(()=>d.registerOption(r,v),[v,r]);let x=(0,u.useRef)(!(c.virtual||c.__demoMode));(0,C.e)(()=>{if(!c.virtual||!c.__demoMode)return;let e=(0,q.k)();return e.requestAnimationFrame(()=>{x.current=!0}),e.dispose},[c.virtual,c.__demoMode]),(0,C.e)(()=>{if(!x.current||0!==c.comboboxState||!f||0===c.activationTrigger)return;let e=(0,q.k)();return e.requestAnimationFrame(()=>{var e,t;null==(t=null==(e=m.current)?void 0:e.scrollIntoView)||t.call(e,{block:"nearest"})}),e.dispose},[m,f,c.comboboxState,c.activationTrigger,c.activeOptionIndex]);let O=(0,T.z)(e=>{var t;if(i||null!=(t=c.virtual)&&t.disabled(l))return e.preventDefault();h(),(0,Z.tq)()||requestAnimationFrame(()=>{var e;return null==(e=c.inputRef.current)?void 0:e.focus({preventScroll:!0})}),0===c.mode&&requestAnimationFrame(()=>d.closeCombobox())}),I=(0,T.z)(()=>{var e;if(i||null!=(e=c.virtual)&&e.disabled(l))return d.goToOption(j.T.Nothing);let t=c.calculateIndex(l);d.goToOption(j.T.Specific,t)}),y=(0,F.g)(),S=(0,T.z)(e=>y.update(e)),R=(0,T.z)(e=>{var t;if(!y.wasMoved(e)||i||null!=(t=c.virtual)&&t.disabled(l)||f)return;let n=c.calculateIndex(l);d.goToOption(j.T.Specific,n,0)}),M=(0,T.z)(e=>{var t;y.wasMoved(e)&&(i||null!=(t=c.virtual)&&t.disabled(l)||f&&(c.optionsPropsRef.current.hold||d.goToOption(j.T.Nothing)))}),w=(0,u.useMemo)(()=>({active:f,selected:p,disabled:i}),[f,p,i]);return(0,Y.sY)({ourProps:{id:r,ref:g,role:"option",tabIndex:!0===i?void 0:-1,"aria-disabled":!0===i||void 0,"aria-selected":p,disabled:void 0,onClick:O,onFocus:I,onPointerEnter:S,onMouseEnter:S,onPointerMove:R,onMouseMove:R,onPointerLeave:M,onMouseLeave:M},theirProps:s,slot:w,defaultTag:"li",name:"Combobox.Option"})})})}}]);
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/1264-2979d95e0b56a75c.js b/litellm/proxy/_experimental/out/_next/static/chunks/1264-2979d95e0b56a75c.js
new file mode 100644
index 00000000000..fd249f9de97
--- /dev/null
+++ b/litellm/proxy/_experimental/out/_next/static/chunks/1264-2979d95e0b56a75c.js
@@ -0,0 +1 @@
+"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[1264],{87045:function(t,e,s){s.d(e,{j:function(){return n}});var i=s(24112),r=s(45345),n=new class extends i.l{#t;#e;#s;constructor(){super(),this.#s=t=>{if(!r.sk&&window.addEventListener){let e=()=>t();return window.addEventListener("visibilitychange",e,!1),()=>{window.removeEventListener("visibilitychange",e)}}}}onSubscribe(){this.#e||this.setEventListener(this.#s)}onUnsubscribe(){this.hasListeners()||(this.#e?.(),this.#e=void 0)}setEventListener(t){this.#s=t,this.#e?.(),this.#e=t(t=>{"boolean"==typeof t?this.setFocused(t):this.onFocus()})}setFocused(t){this.#t!==t&&(this.#t=t,this.onFocus())}onFocus(){let t=this.isFocused();this.listeners.forEach(e=>{e(t)})}isFocused(){return"boolean"==typeof this.#t?this.#t:globalThis.document?.visibilityState!=="hidden"}}},2894:function(t,e,s){s.d(e,{R:function(){return u},m:function(){return a}});var i=s(18238),r=s(7989),n=s(11255),a=class extends r.F{#i;#r;#n;constructor(t){super(),this.mutationId=t.mutationId,this.#r=t.mutationCache,this.#i=[],this.state=t.state||u(),this.setOptions(t.options),this.scheduleGc()}setOptions(t){this.options=t,this.updateGcTime(this.options.gcTime)}get meta(){return this.options.meta}addObserver(t){this.#i.includes(t)||(this.#i.push(t),this.clearGcTimeout(),this.#r.notify({type:"observerAdded",mutation:this,observer:t}))}removeObserver(t){this.#i=this.#i.filter(e=>e!==t),this.scheduleGc(),this.#r.notify({type:"observerRemoved",mutation:this,observer:t})}optionalRemove(){this.#i.length||("pending"===this.state.status?this.scheduleGc():this.#r.remove(this))}continue(){return this.#n?.continue()??this.execute(this.state.variables)}async execute(t){this.#n=(0,n.Mz)({fn:()=>this.options.mutationFn?this.options.mutationFn(t):Promise.reject(Error("No mutationFn found")),onFail:(t,e)=>{this.#a({type:"failed",failureCount:t,error:e})},onPause:()=>{this.#a({type:"pause"})},onContinue:()=>{this.#a({type:"continue"})},retry:this.options.retry??0,retryDelay:this.options.retryDelay,networkMode:this.options.networkMode,canRun:()=>this.#r.canRun(this)});let e="pending"===this.state.status,s=!this.#n.canStart();try{if(!e){this.#a({type:"pending",variables:t,isPaused:s}),await this.#r.config.onMutate?.(t,this);let e=await this.options.onMutate?.(t);e!==this.state.context&&this.#a({type:"pending",context:e,variables:t,isPaused:s})}let i=await this.#n.start();return await this.#r.config.onSuccess?.(i,t,this.state.context,this),await this.options.onSuccess?.(i,t,this.state.context),await this.#r.config.onSettled?.(i,null,this.state.variables,this.state.context,this),await this.options.onSettled?.(i,null,t,this.state.context),this.#a({type:"success",data:i}),i}catch(e){try{throw await this.#r.config.onError?.(e,t,this.state.context,this),await this.options.onError?.(e,t,this.state.context),await this.#r.config.onSettled?.(void 0,e,this.state.variables,this.state.context,this),await this.options.onSettled?.(void 0,e,t,this.state.context),e}finally{this.#a({type:"error",error:e})}}finally{this.#r.runNext(this)}}#a(t){this.state=(e=>{switch(t.type){case"failed":return{...e,failureCount:t.failureCount,failureReason:t.error};case"pause":return{...e,isPaused:!0};case"continue":return{...e,isPaused:!1};case"pending":return{...e,context:t.context,data:void 0,failureCount:0,failureReason:null,error:null,isPaused:t.isPaused,status:"pending",variables:t.variables,submittedAt:Date.now()};case"success":return{...e,data:t.data,failureCount:0,failureReason:null,error:null,status:"success",isPaused:!1};case"error":return{...e,data:void 0,error:t.error,failureCount:e.failureCount+1,failureReason:t.error,isPaused:!1,status:"error"}}})(this.state),i.V.batch(()=>{this.#i.forEach(e=>{e.onMutationUpdate(t)}),this.#r.notify({mutation:this,type:"updated",action:t})})}};function u(){return{context:void 0,data:void 0,error:null,failureCount:0,failureReason:null,isPaused:!1,status:"idle",variables:void 0,submittedAt:0}}},18238:function(t,e,s){s.d(e,{V:function(){return i}});var i=function(){let t=[],e=0,s=t=>{t()},i=t=>{t()},r=t=>setTimeout(t,0),n=i=>{e?t.push(i):r(()=>{s(i)})},a=()=>{let e=t;t=[],e.length&&r(()=>{i(()=>{e.forEach(t=>{s(t)})})})};return{batch:t=>{let s;e++;try{s=t()}finally{--e||a()}return s},batchCalls:t=>(...e)=>{n(()=>{t(...e)})},schedule:n,setNotifyFunction:t=>{s=t},setBatchNotifyFunction:t=>{i=t},setScheduler:t=>{r=t}}}()},57853:function(t,e,s){s.d(e,{N:function(){return n}});var i=s(24112),r=s(45345),n=new class extends i.l{#u=!0;#e;#s;constructor(){super(),this.#s=t=>{if(!r.sk&&window.addEventListener){let e=()=>t(!0),s=()=>t(!1);return window.addEventListener("online",e,!1),window.addEventListener("offline",s,!1),()=>{window.removeEventListener("online",e),window.removeEventListener("offline",s)}}}}onSubscribe(){this.#e||this.setEventListener(this.#s)}onUnsubscribe(){this.hasListeners()||(this.#e?.(),this.#e=void 0)}setEventListener(t){this.#s=t,this.#e?.(),this.#e=t(this.setOnline.bind(this))}setOnline(t){this.#u!==t&&(this.#u=t,this.listeners.forEach(e=>{e(t)}))}isOnline(){return this.#u}}},21733:function(t,e,s){s.d(e,{A:function(){return u},z:function(){return o}});var i=s(45345),r=s(18238),n=s(11255),a=s(7989),u=class extends a.F{#o;#h;#c;#n;#l;#d;constructor(t){super(),this.#d=!1,this.#l=t.defaultOptions,this.setOptions(t.options),this.observers=[],this.#c=t.cache,this.queryKey=t.queryKey,this.queryHash=t.queryHash,this.#o=function(t){let e="function"==typeof t.initialData?t.initialData():t.initialData,s=void 0!==e,i=s?"function"==typeof t.initialDataUpdatedAt?t.initialDataUpdatedAt():t.initialDataUpdatedAt:0;return{data:e,dataUpdateCount:0,dataUpdatedAt:s?i??Date.now():0,error:null,errorUpdateCount:0,errorUpdatedAt:0,fetchFailureCount:0,fetchFailureReason:null,fetchMeta:null,isInvalidated:!1,status:s?"success":"pending",fetchStatus:"idle"}}(this.options),this.state=t.state??this.#o,this.scheduleGc()}get meta(){return this.options.meta}get promise(){return this.#n?.promise}setOptions(t){this.options={...this.#l,...t},this.updateGcTime(this.options.gcTime)}optionalRemove(){this.observers.length||"idle"!==this.state.fetchStatus||this.#c.remove(this)}setData(t,e){let s=(0,i.oE)(this.state.data,t,this.options);return this.#a({data:s,type:"success",dataUpdatedAt:e?.updatedAt,manual:e?.manual}),s}setState(t,e){this.#a({type:"setState",state:t,setStateOptions:e})}cancel(t){let e=this.#n?.promise;return this.#n?.cancel(t),e?e.then(i.ZT).catch(i.ZT):Promise.resolve()}destroy(){super.destroy(),this.cancel({silent:!0})}reset(){this.destroy(),this.setState(this.#o)}isActive(){return this.observers.some(t=>!1!==(0,i.Nc)(t.options.enabled,this))}isDisabled(){return this.getObserversCount()>0?!this.isActive():this.options.queryFn===i.CN||this.state.dataUpdateCount+this.state.errorUpdateCount===0}isStale(){return!!this.state.isInvalidated||(this.getObserversCount()>0?this.observers.some(t=>t.getCurrentResult().isStale):void 0===this.state.data)}isStaleByTime(t=0){return this.state.isInvalidated||void 0===this.state.data||!(0,i.Kp)(this.state.dataUpdatedAt,t)}onFocus(){let t=this.observers.find(t=>t.shouldFetchOnWindowFocus());t?.refetch({cancelRefetch:!1}),this.#n?.continue()}onOnline(){let t=this.observers.find(t=>t.shouldFetchOnReconnect());t?.refetch({cancelRefetch:!1}),this.#n?.continue()}addObserver(t){this.observers.includes(t)||(this.observers.push(t),this.clearGcTimeout(),this.#c.notify({type:"observerAdded",query:this,observer:t}))}removeObserver(t){this.observers.includes(t)&&(this.observers=this.observers.filter(e=>e!==t),this.observers.length||(this.#n&&(this.#d?this.#n.cancel({revert:!0}):this.#n.cancelRetry()),this.scheduleGc()),this.#c.notify({type:"observerRemoved",query:this,observer:t}))}getObserversCount(){return this.observers.length}invalidate(){this.state.isInvalidated||this.#a({type:"invalidate"})}fetch(t,e){if("idle"!==this.state.fetchStatus){if(void 0!==this.state.data&&e?.cancelRefetch)this.cancel({silent:!0});else if(this.#n)return this.#n.continueRetry(),this.#n.promise}if(t&&this.setOptions(t),!this.options.queryFn){let t=this.observers.find(t=>t.options.queryFn);t&&this.setOptions(t.options)}let s=new AbortController,r=t=>{Object.defineProperty(t,"signal",{enumerable:!0,get:()=>(this.#d=!0,s.signal)})},a={fetchOptions:e,options:this.options,queryKey:this.queryKey,state:this.state,fetchFn:()=>{let t=(0,i.cG)(this.options,e),s={queryKey:this.queryKey,meta:this.meta};return(r(s),this.#d=!1,this.options.persister)?this.options.persister(t,s,this):t(s)}};r(a),this.options.behavior?.onFetch(a,this),this.#h=this.state,("idle"===this.state.fetchStatus||this.state.fetchMeta!==a.fetchOptions?.meta)&&this.#a({type:"fetch",meta:a.fetchOptions?.meta});let u=t=>{(0,n.DV)(t)&&t.silent||this.#a({type:"error",error:t}),(0,n.DV)(t)||(this.#c.config.onError?.(t,this),this.#c.config.onSettled?.(this.state.data,t,this)),this.scheduleGc()};return this.#n=(0,n.Mz)({initialPromise:e?.initialPromise,fn:a.fetchFn,abort:s.abort.bind(s),onSuccess:t=>{if(void 0===t){u(Error(`${this.queryHash} data is undefined`));return}try{this.setData(t)}catch(t){u(t);return}this.#c.config.onSuccess?.(t,this),this.#c.config.onSettled?.(t,this.state.error,this),this.scheduleGc()},onError:u,onFail:(t,e)=>{this.#a({type:"failed",failureCount:t,error:e})},onPause:()=>{this.#a({type:"pause"})},onContinue:()=>{this.#a({type:"continue"})},retry:a.options.retry,retryDelay:a.options.retryDelay,networkMode:a.options.networkMode,canRun:()=>!0}),this.#n.start()}#a(t){this.state=(e=>{switch(t.type){case"failed":return{...e,fetchFailureCount:t.failureCount,fetchFailureReason:t.error};case"pause":return{...e,fetchStatus:"paused"};case"continue":return{...e,fetchStatus:"fetching"};case"fetch":return{...e,...o(e.data,this.options),fetchMeta:t.meta??null};case"success":return{...e,data:t.data,dataUpdateCount:e.dataUpdateCount+1,dataUpdatedAt:t.dataUpdatedAt??Date.now(),error:null,isInvalidated:!1,status:"success",...!t.manual&&{fetchStatus:"idle",fetchFailureCount:0,fetchFailureReason:null}};case"error":let s=t.error;if((0,n.DV)(s)&&s.revert&&this.#h)return{...this.#h,fetchStatus:"idle"};return{...e,error:s,errorUpdateCount:e.errorUpdateCount+1,errorUpdatedAt:Date.now(),fetchFailureCount:e.fetchFailureCount+1,fetchFailureReason:s,fetchStatus:"idle",status:"error"};case"invalidate":return{...e,isInvalidated:!0};case"setState":return{...e,...t.state}}})(this.state),r.V.batch(()=>{this.observers.forEach(t=>{t.onQueryUpdate()}),this.#c.notify({query:this,type:"updated",action:t})})}};function o(t,e){return{fetchFailureCount:0,fetchFailureReason:null,fetchStatus:(0,n.Kw)(e.networkMode)?"fetching":"paused",...void 0===t&&{error:null,status:"pending"}}}},21623:function(t,e,s){s.d(e,{S:function(){return y}});var i=s(45345),r=s(21733),n=s(18238),a=s(24112),u=class extends a.l{constructor(t={}){super(),this.config=t,this.#f=new Map}#f;build(t,e,s){let n=e.queryKey,a=e.queryHash??(0,i.Rm)(n,e),u=this.get(a);return u||(u=new r.A({cache:this,queryKey:n,queryHash:a,options:t.defaultQueryOptions(e),state:s,defaultOptions:t.getQueryDefaults(n)}),this.add(u)),u}add(t){this.#f.has(t.queryHash)||(this.#f.set(t.queryHash,t),this.notify({type:"added",query:t}))}remove(t){let e=this.#f.get(t.queryHash);e&&(t.destroy(),e===t&&this.#f.delete(t.queryHash),this.notify({type:"removed",query:t}))}clear(){n.V.batch(()=>{this.getAll().forEach(t=>{this.remove(t)})})}get(t){return this.#f.get(t)}getAll(){return[...this.#f.values()]}find(t){let e={exact:!0,...t};return this.getAll().find(t=>(0,i._x)(e,t))}findAll(t={}){let e=this.getAll();return Object.keys(t).length>0?e.filter(e=>(0,i._x)(t,e)):e}notify(t){n.V.batch(()=>{this.listeners.forEach(e=>{e(t)})})}onFocus(){n.V.batch(()=>{this.getAll().forEach(t=>{t.onFocus()})})}onOnline(){n.V.batch(()=>{this.getAll().forEach(t=>{t.onOnline()})})}},o=s(2894),h=class extends a.l{constructor(t={}){super(),this.config=t,this.#p=new Set,this.#y=new Map,this.#m=0}#p;#y;#m;build(t,e,s){let i=new o.m({mutationCache:this,mutationId:++this.#m,options:t.defaultMutationOptions(e),state:s});return this.add(i),i}add(t){this.#p.add(t);let e=c(t);if("string"==typeof e){let s=this.#y.get(e);s?s.push(t):this.#y.set(e,[t])}this.notify({type:"added",mutation:t})}remove(t){if(this.#p.delete(t)){let e=c(t);if("string"==typeof e){let s=this.#y.get(e);if(s){if(s.length>1){let e=s.indexOf(t);-1!==e&&s.splice(e,1)}else s[0]===t&&this.#y.delete(e)}}}this.notify({type:"removed",mutation:t})}canRun(t){let e=c(t);if("string"!=typeof e)return!0;{let s=this.#y.get(e),i=s?.find(t=>"pending"===t.state.status);return!i||i===t}}runNext(t){let e=c(t);if("string"!=typeof e)return Promise.resolve();{let s=this.#y.get(e)?.find(e=>e!==t&&e.state.isPaused);return s?.continue()??Promise.resolve()}}clear(){n.V.batch(()=>{this.#p.forEach(t=>{this.notify({type:"removed",mutation:t})}),this.#p.clear(),this.#y.clear()})}getAll(){return Array.from(this.#p)}find(t){let e={exact:!0,...t};return this.getAll().find(t=>(0,i.X7)(e,t))}findAll(t={}){return this.getAll().filter(e=>(0,i.X7)(t,e))}notify(t){n.V.batch(()=>{this.listeners.forEach(e=>{e(t)})})}resumePausedMutations(){let t=this.getAll().filter(t=>t.state.isPaused);return n.V.batch(()=>Promise.all(t.map(t=>t.continue().catch(i.ZT))))}};function c(t){return t.options.scope?.id}var l=s(87045),d=s(57853);function f(t){return{onFetch:(e,s)=>{let r=e.options,n=e.fetchOptions?.meta?.fetchMore?.direction,a=e.state.data?.pages||[],u=e.state.data?.pageParams||[],o={pages:[],pageParams:[]},h=0,c=async()=>{let s=!1,c=t=>{Object.defineProperty(t,"signal",{enumerable:!0,get:()=>(e.signal.aborted?s=!0:e.signal.addEventListener("abort",()=>{s=!0}),e.signal)})},l=(0,i.cG)(e.options,e.fetchOptions),d=async(t,r,n)=>{if(s)return Promise.reject();if(null==r&&t.pages.length)return Promise.resolve(t);let a={queryKey:e.queryKey,pageParam:r,direction:n?"backward":"forward",meta:e.options.meta};c(a);let u=await l(a),{maxPages:o}=e.options,h=n?i.Ht:i.VX;return{pages:h(t.pages,u,o),pageParams:h(t.pageParams,r,o)}};if(n&&a.length){let t="backward"===n,e={pages:a,pageParams:u},s=(t?function(t,{pages:e,pageParams:s}){return e.length>0?t.getPreviousPageParam?.(e[0],e,s[0],s):void 0}:p)(r,e);o=await d(e,s,t)}else{let e=t??a.length;do{let t=0===h?u[0]??r.initialPageParam:p(r,o);if(h>0&&null==t)break;o=await d(o,t),h++}while(he.options.persister?.(c,{queryKey:e.queryKey,meta:e.options.meta,signal:e.signal},s):e.fetchFn=c}}}function p(t,{pages:e,pageParams:s}){let i=e.length-1;return e.length>0?t.getNextPageParam(e[i],e,s[i],s):void 0}var y=class{#v;#r;#l;#b;#g;#C;#O;#R;constructor(t={}){this.#v=t.queryCache||new u,this.#r=t.mutationCache||new h,this.#l=t.defaultOptions||{},this.#b=new Map,this.#g=new Map,this.#C=0}mount(){this.#C++,1===this.#C&&(this.#O=l.j.subscribe(async t=>{t&&(await this.resumePausedMutations(),this.#v.onFocus())}),this.#R=d.N.subscribe(async t=>{t&&(await this.resumePausedMutations(),this.#v.onOnline())}))}unmount(){this.#C--,0===this.#C&&(this.#O?.(),this.#O=void 0,this.#R?.(),this.#R=void 0)}isFetching(t){return this.#v.findAll({...t,fetchStatus:"fetching"}).length}isMutating(t){return this.#r.findAll({...t,status:"pending"}).length}getQueryData(t){let e=this.defaultQueryOptions({queryKey:t});return this.#v.get(e.queryHash)?.state.data}ensureQueryData(t){let e=this.defaultQueryOptions(t),s=this.#v.build(this,e),r=s.state.data;return void 0===r?this.fetchQuery(t):(t.revalidateIfStale&&s.isStaleByTime((0,i.KC)(e.staleTime,s))&&this.prefetchQuery(e),Promise.resolve(r))}getQueriesData(t){return this.#v.findAll(t).map(({queryKey:t,state:e})=>[t,e.data])}setQueryData(t,e,s){let r=this.defaultQueryOptions({queryKey:t}),n=this.#v.get(r.queryHash),a=n?.state.data,u=(0,i.SE)(e,a);if(void 0!==u)return this.#v.build(this,r).setData(u,{...s,manual:!0})}setQueriesData(t,e,s){return n.V.batch(()=>this.#v.findAll(t).map(({queryKey:t})=>[t,this.setQueryData(t,e,s)]))}getQueryState(t){let e=this.defaultQueryOptions({queryKey:t});return this.#v.get(e.queryHash)?.state}removeQueries(t){let e=this.#v;n.V.batch(()=>{e.findAll(t).forEach(t=>{e.remove(t)})})}resetQueries(t,e){let s=this.#v,i={type:"active",...t};return n.V.batch(()=>(s.findAll(t).forEach(t=>{t.reset()}),this.refetchQueries(i,e)))}cancelQueries(t,e={}){let s={revert:!0,...e};return Promise.all(n.V.batch(()=>this.#v.findAll(t).map(t=>t.cancel(s)))).then(i.ZT).catch(i.ZT)}invalidateQueries(t,e={}){return n.V.batch(()=>{if(this.#v.findAll(t).forEach(t=>{t.invalidate()}),t?.refetchType==="none")return Promise.resolve();let s={...t,type:t?.refetchType??t?.type??"active"};return this.refetchQueries(s,e)})}refetchQueries(t,e={}){let s={...e,cancelRefetch:e.cancelRefetch??!0};return Promise.all(n.V.batch(()=>this.#v.findAll(t).filter(t=>!t.isDisabled()).map(t=>{let e=t.fetch(void 0,s);return s.throwOnError||(e=e.catch(i.ZT)),"paused"===t.state.fetchStatus?Promise.resolve():e}))).then(i.ZT)}fetchQuery(t){let e=this.defaultQueryOptions(t);void 0===e.retry&&(e.retry=!1);let s=this.#v.build(this,e);return s.isStaleByTime((0,i.KC)(e.staleTime,s))?s.fetch(e):Promise.resolve(s.state.data)}prefetchQuery(t){return this.fetchQuery(t).then(i.ZT).catch(i.ZT)}fetchInfiniteQuery(t){return t.behavior=f(t.pages),this.fetchQuery(t)}prefetchInfiniteQuery(t){return this.fetchInfiniteQuery(t).then(i.ZT).catch(i.ZT)}ensureInfiniteQueryData(t){return t.behavior=f(t.pages),this.ensureQueryData(t)}resumePausedMutations(){return d.N.isOnline()?this.#r.resumePausedMutations():Promise.resolve()}getQueryCache(){return this.#v}getMutationCache(){return this.#r}getDefaultOptions(){return this.#l}setDefaultOptions(t){this.#l=t}setQueryDefaults(t,e){this.#b.set((0,i.Ym)(t),{queryKey:t,defaultOptions:e})}getQueryDefaults(t){let e=[...this.#b.values()],s={};return e.forEach(e=>{(0,i.to)(t,e.queryKey)&&Object.assign(s,e.defaultOptions)}),s}setMutationDefaults(t,e){this.#g.set((0,i.Ym)(t),{mutationKey:t,defaultOptions:e})}getMutationDefaults(t){let e=[...this.#g.values()],s={};return e.forEach(e=>{(0,i.to)(t,e.mutationKey)&&(s={...s,...e.defaultOptions})}),s}defaultQueryOptions(t){if(t._defaulted)return t;let e={...this.#l.queries,...this.getQueryDefaults(t.queryKey),...t,_defaulted:!0};return e.queryHash||(e.queryHash=(0,i.Rm)(e.queryKey,e)),void 0===e.refetchOnReconnect&&(e.refetchOnReconnect="always"!==e.networkMode),void 0===e.throwOnError&&(e.throwOnError=!!e.suspense),!e.networkMode&&e.persister&&(e.networkMode="offlineFirst"),e.queryFn===i.CN&&(e.enabled=!1),e}defaultMutationOptions(t){return t?._defaulted?t:{...this.#l.mutations,...t?.mutationKey&&this.getMutationDefaults(t.mutationKey),...t,_defaulted:!0}}clear(){this.#v.clear(),this.#r.clear()}}},7989:function(t,e,s){s.d(e,{F:function(){return r}});var i=s(45345),r=class{#w;destroy(){this.clearGcTimeout()}scheduleGc(){this.clearGcTimeout(),(0,i.PN)(this.gcTime)&&(this.#w=setTimeout(()=>{this.optionalRemove()},this.gcTime))}updateGcTime(t){this.gcTime=Math.max(this.gcTime||0,t??(i.sk?1/0:3e5))}clearGcTimeout(){this.#w&&(clearTimeout(this.#w),this.#w=void 0)}}},11255:function(t,e,s){s.d(e,{DV:function(){return c},Kw:function(){return o},Mz:function(){return l}});var i=s(87045),r=s(57853),n=s(16803),a=s(45345);function u(t){return Math.min(1e3*2**t,3e4)}function o(t){return(t??"online")!=="online"||r.N.isOnline()}var h=class extends Error{constructor(t){super("CancelledError"),this.revert=t?.revert,this.silent=t?.silent}};function c(t){return t instanceof h}function l(t){let e,s=!1,c=0,l=!1,d=(0,n.O)(),f=()=>i.j.isFocused()&&("always"===t.networkMode||r.N.isOnline())&&t.canRun(),p=()=>o(t.networkMode)&&t.canRun(),y=s=>{l||(l=!0,t.onSuccess?.(s),e?.(),d.resolve(s))},m=s=>{l||(l=!0,t.onError?.(s),e?.(),d.reject(s))},v=()=>new Promise(s=>{e=t=>{(l||f())&&s(t)},t.onPause?.()}).then(()=>{e=void 0,l||t.onContinue?.()}),b=()=>{let e;if(l)return;let i=0===c?t.initialPromise:void 0;try{e=i??t.fn()}catch(t){e=Promise.reject(t)}Promise.resolve(e).then(y).catch(e=>{if(l)return;let i=t.retry??(a.sk?0:3),r=t.retryDelay??u,n="function"==typeof r?r(c,e):r,o=!0===i||"number"==typeof i&&cf()?void 0:v()).then(()=>{s?m(e):b()})})};return{promise:d,cancel:e=>{l||(m(new h(e)),t.abort?.())},continue:()=>(e?.(),d),cancelRetry:()=>{s=!0},continueRetry:()=>{s=!1},canStart:p,start:()=>(p()?b():v().then(b),d)}}},24112:function(t,e,s){s.d(e,{l:function(){return i}});var i=class{constructor(){this.listeners=new Set,this.subscribe=this.subscribe.bind(this)}subscribe(t){return this.listeners.add(t),this.onSubscribe(),()=>{this.listeners.delete(t),this.onUnsubscribe()}}hasListeners(){return this.listeners.size>0}onSubscribe(){}onUnsubscribe(){}}},16803:function(t,e,s){s.d(e,{O:function(){return i}});function i(){let t,e;let s=new Promise((s,i)=>{t=s,e=i});function i(t){Object.assign(s,t),delete s.resolve,delete s.reject}return s.status="pending",s.catch(()=>{}),s.resolve=e=>{i({status:"fulfilled",value:e}),t(e)},s.reject=t=>{i({status:"rejected",reason:t}),e(t)},s}},45345:function(t,e,s){s.d(e,{CN:function(){return w},Ht:function(){return R},KC:function(){return o},Kp:function(){return u},Nc:function(){return h},PN:function(){return a},Rm:function(){return d},SE:function(){return n},VS:function(){return y},VX:function(){return O},X7:function(){return l},Ym:function(){return f},ZT:function(){return r},_v:function(){return g},_x:function(){return c},cG:function(){return S},oE:function(){return C},sk:function(){return i},to:function(){return p}});var i="undefined"==typeof window||"Deno"in globalThis;function r(){}function n(t,e){return"function"==typeof t?t(e):t}function a(t){return"number"==typeof t&&t>=0&&t!==1/0}function u(t,e){return Math.max(t+(e||0)-Date.now(),0)}function o(t,e){return"function"==typeof t?t(e):t}function h(t,e){return"function"==typeof t?t(e):t}function c(t,e){let{type:s="all",exact:i,fetchStatus:r,predicate:n,queryKey:a,stale:u}=t;if(a){if(i){if(e.queryHash!==d(a,e.options))return!1}else if(!p(e.queryKey,a))return!1}if("all"!==s){let t=e.isActive();if("active"===s&&!t||"inactive"===s&&t)return!1}return("boolean"!=typeof u||e.isStale()===u)&&(!r||r===e.state.fetchStatus)&&(!n||!!n(e))}function l(t,e){let{exact:s,status:i,predicate:r,mutationKey:n}=t;if(n){if(!e.options.mutationKey)return!1;if(s){if(f(e.options.mutationKey)!==f(n))return!1}else if(!p(e.options.mutationKey,n))return!1}return(!i||e.state.status===i)&&(!r||!!r(e))}function d(t,e){return(e?.queryKeyHashFn||f)(t)}function f(t){return JSON.stringify(t,(t,e)=>v(e)?Object.keys(e).sort().reduce((t,s)=>(t[s]=e[s],t),{}):e)}function p(t,e){return t===e||typeof t==typeof e&&!!t&&!!e&&"object"==typeof t&&"object"==typeof e&&!Object.keys(e).some(s=>!p(t[s],e[s]))}function y(t,e){if(!e||Object.keys(t).length!==Object.keys(e).length)return!1;for(let s in t)if(t[s]!==e[s])return!1;return!0}function m(t){return Array.isArray(t)&&t.length===Object.keys(t).length}function v(t){if(!b(t))return!1;let e=t.constructor;if(void 0===e)return!0;let s=e.prototype;return!!(b(s)&&s.hasOwnProperty("isPrototypeOf"))&&Object.getPrototypeOf(t)===Object.prototype}function b(t){return"[object Object]"===Object.prototype.toString.call(t)}function g(t){return new Promise(e=>{setTimeout(e,t)})}function C(t,e,s){return"function"==typeof s.structuralSharing?s.structuralSharing(t,e):!1!==s.structuralSharing?function t(e,s){if(e===s)return e;let i=m(e)&&m(s);if(i||v(e)&&v(s)){let r=i?e:Object.keys(e),n=r.length,a=i?s:Object.keys(s),u=a.length,o=i?[]:{},h=0;for(let n=0;ns?i.slice(1):i}function R(t,e,s=0){let i=[e,...t];return s&&i.length>s?i.slice(0,-1):i}var w=Symbol();function S(t,e){return!t.queryFn&&e?.initialPromise?()=>e.initialPromise:t.queryFn&&t.queryFn!==w?t.queryFn:()=>Promise.reject(Error(`Missing queryFn: '${t.queryHash}'`))}},16593:function(t,e,s){let i;s.d(e,{a:function(){return E}});var r=s(87045),n=s(18238),a=s(21733),u=s(24112),o=s(16803),h=s(45345),c=class extends u.l{constructor(t,e){super(),this.options=e,this.#S=t,this.#Q=null,this.#q=(0,o.O)(),this.options.experimental_prefetchInRender||this.#q.reject(Error("experimental_prefetchInRender feature flag is not enabled")),this.bindMethods(),this.setOptions(e)}#S;#F=void 0;#P=void 0;#E=void 0;#T;#D;#q;#Q;#I;#x;#A;#M;#k;#U;#j=new Set;bindMethods(){this.refetch=this.refetch.bind(this)}onSubscribe(){1===this.listeners.size&&(this.#F.addObserver(this),l(this.#F,this.options)?this.#K():this.updateResult(),this.#N())}onUnsubscribe(){this.hasListeners()||this.destroy()}shouldFetchOnReconnect(){return d(this.#F,this.options,this.options.refetchOnReconnect)}shouldFetchOnWindowFocus(){return d(this.#F,this.options,this.options.refetchOnWindowFocus)}destroy(){this.listeners=new Set,this.#V(),this.#L(),this.#F.removeObserver(this)}setOptions(t,e){let s=this.options,i=this.#F;if(this.options=this.#S.defaultQueryOptions(t),void 0!==this.options.enabled&&"boolean"!=typeof this.options.enabled&&"function"!=typeof this.options.enabled&&"boolean"!=typeof(0,h.Nc)(this.options.enabled,this.#F))throw Error("Expected enabled to be a boolean or a callback that returns a boolean");this.#_(),this.#F.setOptions(this.options),s._defaulted&&!(0,h.VS)(this.options,s)&&this.#S.getQueryCache().notify({type:"observerOptionsUpdated",query:this.#F,observer:this});let r=this.hasListeners();r&&f(this.#F,i,this.options,s)&&this.#K(),this.updateResult(e),r&&(this.#F!==i||(0,h.Nc)(this.options.enabled,this.#F)!==(0,h.Nc)(s.enabled,this.#F)||(0,h.KC)(this.options.staleTime,this.#F)!==(0,h.KC)(s.staleTime,this.#F))&&this.#H();let n=this.#G();r&&(this.#F!==i||(0,h.Nc)(this.options.enabled,this.#F)!==(0,h.Nc)(s.enabled,this.#F)||n!==this.#U)&&this.#Z(n)}getOptimisticResult(t){let e=this.#S.getQueryCache().build(this.#S,t),s=this.createResult(e,t);return(0,h.VS)(this.getCurrentResult(),s)||(this.#E=s,this.#D=this.options,this.#T=this.#F.state),s}getCurrentResult(){return this.#E}trackResult(t,e){let s={};return Object.keys(t).forEach(i=>{Object.defineProperty(s,i,{configurable:!1,enumerable:!0,get:()=>(this.trackProp(i),e?.(i),t[i])})}),s}trackProp(t){this.#j.add(t)}getCurrentQuery(){return this.#F}refetch({...t}={}){return this.fetch({...t})}fetchOptimistic(t){let e=this.#S.defaultQueryOptions(t),s=this.#S.getQueryCache().build(this.#S,e);return s.fetch().then(()=>this.createResult(s,e))}fetch(t){return this.#K({...t,cancelRefetch:t.cancelRefetch??!0}).then(()=>(this.updateResult(),this.#E))}#K(t){this.#_();let e=this.#F.fetch(this.options,t);return t?.throwOnError||(e=e.catch(h.ZT)),e}#H(){this.#V();let t=(0,h.KC)(this.options.staleTime,this.#F);if(h.sk||this.#E.isStale||!(0,h.PN)(t))return;let e=(0,h.Kp)(this.#E.dataUpdatedAt,t);this.#M=setTimeout(()=>{this.#E.isStale||this.updateResult()},e+1)}#G(){return("function"==typeof this.options.refetchInterval?this.options.refetchInterval(this.#F):this.options.refetchInterval)??!1}#Z(t){this.#L(),this.#U=t,!h.sk&&!1!==(0,h.Nc)(this.options.enabled,this.#F)&&(0,h.PN)(this.#U)&&0!==this.#U&&(this.#k=setInterval(()=>{(this.options.refetchIntervalInBackground||r.j.isFocused())&&this.#K()},this.#U))}#N(){this.#H(),this.#Z(this.#G())}#V(){this.#M&&(clearTimeout(this.#M),this.#M=void 0)}#L(){this.#k&&(clearInterval(this.#k),this.#k=void 0)}createResult(t,e){let s;let i=this.#F,r=this.options,n=this.#E,u=this.#T,c=this.#D,d=t!==i?t.state:this.#P,{state:y}=t,m={...y},v=!1;if(e._optimisticResults){let s=this.hasListeners(),n=!s&&l(t,e),u=s&&f(t,i,e,r);(n||u)&&(m={...m,...(0,a.z)(y.data,t.options)}),"isRestoring"===e._optimisticResults&&(m.fetchStatus="idle")}let{error:b,errorUpdatedAt:g,status:C}=m;if(e.select&&void 0!==m.data){if(n&&m.data===u?.data&&e.select===this.#I)s=this.#x;else try{this.#I=e.select,s=e.select(m.data),s=(0,h.oE)(n?.data,s,e),this.#x=s,this.#Q=null}catch(t){this.#Q=t}}else s=m.data;if(void 0!==e.placeholderData&&void 0===s&&"pending"===C){let t;if(n?.isPlaceholderData&&e.placeholderData===c?.placeholderData)t=n.data;else if(t="function"==typeof e.placeholderData?e.placeholderData(this.#A?.state.data,this.#A):e.placeholderData,e.select&&void 0!==t)try{t=e.select(t),this.#Q=null}catch(t){this.#Q=t}void 0!==t&&(C="success",s=(0,h.oE)(n?.data,t,e),v=!0)}this.#Q&&(b=this.#Q,s=this.#x,g=Date.now(),C="error");let O="fetching"===m.fetchStatus,R="pending"===C,w="error"===C,S=R&&O,Q=void 0!==s,q={status:C,fetchStatus:m.fetchStatus,isPending:R,isSuccess:"success"===C,isError:w,isInitialLoading:S,isLoading:S,data:s,dataUpdatedAt:m.dataUpdatedAt,error:b,errorUpdatedAt:g,failureCount:m.fetchFailureCount,failureReason:m.fetchFailureReason,errorUpdateCount:m.errorUpdateCount,isFetched:m.dataUpdateCount>0||m.errorUpdateCount>0,isFetchedAfterMount:m.dataUpdateCount>d.dataUpdateCount||m.errorUpdateCount>d.errorUpdateCount,isFetching:O,isRefetching:O&&!R,isLoadingError:w&&!Q,isPaused:"paused"===m.fetchStatus,isPlaceholderData:v,isRefetchError:w&&Q,isStale:p(t,e),refetch:this.refetch,promise:this.#q};if(this.options.experimental_prefetchInRender){let e=t=>{"error"===q.status?t.reject(q.error):void 0!==q.data&&t.resolve(q.data)},s=()=>{e(this.#q=q.promise=(0,o.O)())},r=this.#q;switch(r.status){case"pending":t.queryHash===i.queryHash&&e(r);break;case"fulfilled":("error"===q.status||q.data!==r.value)&&s();break;case"rejected":("error"!==q.status||q.error!==r.reason)&&s()}}return q}updateResult(t){let e=this.#E,s=this.createResult(this.#F,this.options);if(this.#T=this.#F.state,this.#D=this.options,void 0!==this.#T.data&&(this.#A=this.#F),(0,h.VS)(s,e))return;this.#E=s;let i={};t?.listeners!==!1&&(()=>{if(!e)return!0;let{notifyOnChangeProps:t}=this.options,s="function"==typeof t?t():t;if("all"===s||!s&&!this.#j.size)return!0;let i=new Set(s??this.#j);return this.options.throwOnError&&i.add("error"),Object.keys(this.#E).some(t=>this.#E[t]!==e[t]&&i.has(t))})()&&(i.listeners=!0),this.#z({...i,...t})}#_(){let t=this.#S.getQueryCache().build(this.#S,this.options);if(t===this.#F)return;let e=this.#F;this.#F=t,this.#P=t.state,this.hasListeners()&&(e?.removeObserver(this),t.addObserver(this))}onQueryUpdate(){this.updateResult(),this.hasListeners()&&this.#N()}#z(t){n.V.batch(()=>{t.listeners&&this.listeners.forEach(t=>{t(this.#E)}),this.#S.getQueryCache().notify({query:this.#F,type:"observerResultsUpdated"})})}};function l(t,e){return!1!==(0,h.Nc)(e.enabled,t)&&void 0===t.state.data&&!("error"===t.state.status&&!1===e.retryOnMount)||void 0!==t.state.data&&d(t,e,e.refetchOnMount)}function d(t,e,s){if(!1!==(0,h.Nc)(e.enabled,t)){let i="function"==typeof s?s(t):s;return"always"===i||!1!==i&&p(t,e)}return!1}function f(t,e,s,i){return(t!==e||!1===(0,h.Nc)(i.enabled,t))&&(!s.suspense||"error"!==t.state.status)&&p(t,s)}function p(t,e){return!1!==(0,h.Nc)(e.enabled,t)&&t.isStaleByTime((0,h.KC)(e.staleTime,t))}var y=s(2265),m=s(29827);s(57437);var v=y.createContext((i=!1,{clearReset:()=>{i=!1},reset:()=>{i=!0},isReset:()=>i})),b=()=>y.useContext(v),g=s(51172),C=(t,e)=>{(t.suspense||t.throwOnError||t.experimental_prefetchInRender)&&!e.isReset()&&(t.retryOnMount=!1)},O=t=>{y.useEffect(()=>{t.clearReset()},[t])},R=t=>{let{result:e,errorResetBoundary:s,throwOnError:i,query:r}=t;return e.isError&&!s.isReset()&&!e.isFetching&&r&&(0,g.L)(i,[e.error,r])},w=y.createContext(!1),S=()=>y.useContext(w);w.Provider;var Q=t=>{let e=t.staleTime;t.suspense&&(t.staleTime="function"==typeof e?(...t)=>Math.max(e(...t),1e3):Math.max(e??1e3,1e3),"number"==typeof t.gcTime&&(t.gcTime=Math.max(t.gcTime,1e3)))},q=(t,e)=>t.isLoading&&t.isFetching&&!e,F=(t,e)=>t?.suspense&&e.isPending,P=(t,e,s)=>e.fetchOptimistic(t).catch(()=>{s.clearReset()});function E(t,e){return function(t,e,s){var i,r,a,u,o;let c=(0,m.NL)(s),l=S(),d=b(),f=c.defaultQueryOptions(t);null===(r=c.getDefaultOptions().queries)||void 0===r||null===(i=r._experimental_beforeQuery)||void 0===i||i.call(r,f),f._optimisticResults=l?"isRestoring":"optimistic",Q(f),C(f,d),O(d);let p=!c.getQueryCache().get(f.queryHash),[v]=y.useState(()=>new e(c,f)),w=v.getOptimisticResult(f),E=!l&&!1!==t.subscribed;if(y.useSyncExternalStore(y.useCallback(t=>{let e=E?v.subscribe(n.V.batchCalls(t)):g.Z;return v.updateResult(),e},[v,E]),()=>v.getCurrentResult(),()=>v.getCurrentResult()),y.useEffect(()=>{v.setOptions(f,{listeners:!1})},[f,v]),F(f,w))throw P(f,v,d);if(R({result:w,errorResetBoundary:d,throwOnError:f.throwOnError,query:c.getQueryCache().get(f.queryHash)}))throw w.error;if(null===(u=c.getDefaultOptions().queries)||void 0===u||null===(a=u._experimental_afterQuery)||void 0===a||a.call(u,f,w),f.experimental_prefetchInRender&&!h.sk&&q(w,l)){let t=p?P(f,v,d):null===(o=c.getQueryCache().get(f.queryHash))||void 0===o?void 0:o.promise;null==t||t.catch(g.Z).finally(()=>{v.updateResult()})}return f.notifyOnChangeProps?w:v.trackResult(w)}(t,c,e)}},51172:function(t,e,s){function i(t,e){return"function"==typeof t?t(...e):!!t}function r(){}s.d(e,{L:function(){return i},Z:function(){return r}})}}]);
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/1307-6127ab4e0e743a22.js b/litellm/proxy/_experimental/out/_next/static/chunks/1307-6127ab4e0e743a22.js
new file mode 100644
index 00000000000..6b6de2bd74d
--- /dev/null
+++ b/litellm/proxy/_experimental/out/_next/static/chunks/1307-6127ab4e0e743a22.js
@@ -0,0 +1 @@
+"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[1307],{21307:function(e,s,r){r.d(s,{d:function(){return eR},o:function(){return eY}});var l=r(57437),t=r(2265),a=r(16593),n=r(52787),i=r(82680),o=r(89970),c=r(20831),d=r(12485),m=r(18135),x=r(35242),u=r(29706),h=r(77991),p=r(49804),j=r(67101),g=r(84264),v=r(96761),f=r(60493),b=r(47323),y=r(53410),N=r(74998);let _=e=>{try{let s=e.indexOf("/mcp/");if(-1===s)return{token:null,baseUrl:e};let r=e.split("/mcp/");if(2!==r.length)return{token:null,baseUrl:e};let l=r[0]+"/mcp/",t=r[1];if(!t)return{token:null,baseUrl:e};return{token:t,baseUrl:l}}catch(s){return console.error("Error parsing MCP URL:",s),{token:null,baseUrl:e}}},Z=e=>{let{token:s,baseUrl:r}=_(e);return s?r+"...":e},w=e=>{let{token:s}=_(e);return{maskedUrl:Z(e),hasToken:!!s}},C=e=>e?/^https?:\/\/[^\s/$.?#].[^\s]*$/i.test(e)?Promise.resolve():Promise.reject("Please enter a valid URL (e.g., http://service-name.domain:1234/path or https://example.com)"):Promise.resolve(),S=e=>e&&e.includes("-")?Promise.reject("Server name cannot contain '-' (hyphen). Please use '_' (underscore) instead."):Promise.resolve(),k=(e,s,r,t)=>[{accessorKey:"server_id",header:"Server ID",cell:e=>{let{row:r}=e;return(0,l.jsxs)("button",{onClick:()=>s(r.original.server_id),className:"font-mono text-blue-500 bg-blue-50 hover:bg-blue-100 text-xs font-normal px-2 py-0.5 text-left w-full truncate whitespace-nowrap cursor-pointer max-w-[15ch]",children:[r.original.server_id.slice(0,7),"..."]})}},{accessorKey:"server_name",header:"Name"},{accessorKey:"alias",header:"Alias"},{id:"url",header:"URL",cell:e=>{let{row:s}=e,{maskedUrl:r}=w(s.original.url);return(0,l.jsx)("span",{className:"font-mono text-sm",children:r})}},{accessorKey:"transport",header:"Transport",cell:e=>{let{getValue:s}=e;return(0,l.jsx)("span",{children:(s()||"http").toUpperCase()})}},{accessorKey:"auth_type",header:"Auth Type",cell:e=>{let{getValue:s}=e;return(0,l.jsx)("span",{children:s()||"none"})}},{id:"health_status",header:"Health Status",cell:e=>{let{row:s}=e,r=s.original,t=r.status||"unknown",a=r.last_health_check,n=r.health_check_error,i=(0,l.jsxs)("div",{className:"max-w-xs",children:[(0,l.jsxs)("div",{className:"font-semibold mb-1",children:["Health Status: ",t]}),a&&(0,l.jsxs)("div",{className:"text-xs mb-1",children:["Last Check: ",new Date(a).toLocaleString()]}),n&&(0,l.jsxs)("div",{className:"text-xs",children:[(0,l.jsx)("div",{className:"font-medium text-red-400 mb-1",children:"Error:"}),(0,l.jsx)("div",{className:"break-words",children:n})]}),!a&&!n&&(0,l.jsx)("div",{className:"text-xs text-gray-400",children:"No health check data available"})]});return(0,l.jsx)(o.Z,{title:i,placement:"top",children:(0,l.jsxs)("button",{className:"font-mono text-xs font-normal px-2 py-0.5 text-left w-full truncate whitespace-nowrap cursor-pointer max-w-[10ch] ".concat((e=>{switch(e){case"healthy":return"text-green-500 bg-green-50 hover:bg-green-100";case"unhealthy":return"text-red-500 bg-red-50 hover:bg-red-100";default:return"text-gray-500 bg-gray-50 hover:bg-gray-100"}})(t)),children:[(0,l.jsx)("span",{className:"mr-1",children:"●"}),t.charAt(0).toUpperCase()+t.slice(1)]})})}},{id:"mcp_access_groups",header:"Access Groups",cell:e=>{let{row:s}=e,r=s.original.mcp_access_groups;if(Array.isArray(r)&&r.length>0&&"string"==typeof r[0]){let e=r.join(", ");return(0,l.jsx)(o.Z,{title:e,children:(0,l.jsx)("span",{className:"max-w-[200px] truncate block",children:e.length>30?"".concat(e.slice(0,30),"..."):e})})}return(0,l.jsx)("span",{className:"text-gray-400 italic",children:"None"})}},{header:"Created At",accessorKey:"created_at",sortingFn:"datetime",cell:e=>{let{row:s}=e,r=s.original;return(0,l.jsx)("span",{className:"text-xs",children:r.created_at?new Date(r.created_at).toLocaleDateString():"-"})}},{header:"Updated At",accessorKey:"updated_at",sortingFn:"datetime",cell:e=>{let{row:s}=e,r=s.original;return(0,l.jsx)("span",{className:"text-xs",children:r.updated_at?new Date(r.updated_at).toLocaleDateString():"-"})}},{id:"actions",header:"Actions",cell:e=>{let{row:s}=e;return(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)(b.Z,{icon:y.Z,size:"sm",onClick:()=>r(s.original.server_id),className:"cursor-pointer"}),(0,l.jsx)(b.Z,{icon:N.Z,size:"sm",onClick:()=>t(s.original.server_id),className:"cursor-pointer"})]})}}];var P=r(19250),A=r(20347),L=r(77331),M=r(82376),T=r(71437),I=r(12514);let E={NONE:"none",API_KEY:"api_key",BEARER_TOKEN:"bearer_token",BASIC:"basic"},z={SSE:"sse"},q=e=>(console.log(e),null==e)?z.SSE:e,O=e=>null==e?E.NONE:e,R=e=>O(e)!==E.NONE;var U=r(13634),F=r(73002),B=r(49566),K=r(20577),V=r(44851),D=r(33866),H=r(62670),G=r(15424),Y=r(58630),J=e=>{let{value:s={},onChange:r,tools:t=[],disabled:a=!1}=e,n=(e,l)=>{let t={...s,tool_name_to_cost_per_query:{...s.tool_name_to_cost_per_query,[e]:l}};null==r||r(t)};return(0,l.jsx)(I.Z,{children:(0,l.jsxs)("div",{className:"space-y-6",children:[(0,l.jsxs)("div",{className:"flex items-center gap-2 mb-4",children:[(0,l.jsx)(H.Z,{className:"text-green-600"}),(0,l.jsx)(v.Z,{children:"Cost Configuration"}),(0,l.jsx)(o.Z,{title:"Configure costs for this MCP server's tool calls. Set a default rate and per-tool overrides.",children:(0,l.jsx)(G.Z,{className:"text-gray-400"})})]}),(0,l.jsxs)("div",{className:"space-y-4",children:[(0,l.jsxs)("div",{children:[(0,l.jsxs)("label",{className:"block text-sm font-medium text-gray-700 mb-2",children:["Default Cost per Query ($)",(0,l.jsx)(o.Z,{title:"Default cost charged for each tool call to this server.",children:(0,l.jsx)(G.Z,{className:"ml-1 text-gray-400"})})]}),(0,l.jsx)(K.Z,{min:0,step:1e-4,precision:4,placeholder:"0.0000",value:s.default_cost_per_query,onChange:e=>{let l={...s,default_cost_per_query:e};null==r||r(l)},disabled:a,style:{width:"200px"},addonBefore:"$"}),(0,l.jsx)(g.Z,{className:"block mt-1 text-gray-500 text-sm",children:"Set a default cost for all tool calls to this server"})]}),t.length>0&&(0,l.jsxs)("div",{className:"space-y-4",children:[(0,l.jsxs)("label",{className:"block text-sm font-medium text-gray-700",children:["Tool-Specific Costs ($)",(0,l.jsx)(o.Z,{title:"Override the default cost for specific tools. Leave blank to use the default rate.",children:(0,l.jsx)(G.Z,{className:"ml-1 text-gray-400"})})]}),(0,l.jsx)(V.default,{items:[{key:"1",label:(0,l.jsxs)("div",{className:"flex items-center",children:[(0,l.jsx)(Y.Z,{className:"mr-2 text-blue-500"}),(0,l.jsx)("span",{className:"font-medium",children:"Available Tools"}),(0,l.jsx)(D.Z,{count:t.length,style:{backgroundColor:"#52c41a",marginLeft:"8px"}})]}),children:(0,l.jsx)("div",{className:"space-y-3 max-h-64 overflow-y-auto",children:t.map((e,r)=>{var t;return(0,l.jsxs)("div",{className:"flex items-center justify-between p-3 bg-gray-50 rounded-lg",children:[(0,l.jsxs)("div",{className:"flex-1",children:[(0,l.jsx)(g.Z,{className:"font-medium text-gray-900",children:e.name}),e.description&&(0,l.jsx)(g.Z,{className:"text-gray-500 text-sm block mt-1",children:e.description})]}),(0,l.jsx)("div",{className:"ml-4",children:(0,l.jsx)(K.Z,{min:0,step:1e-4,precision:4,placeholder:"Use default",value:null===(t=s.tool_name_to_cost_per_query)||void 0===t?void 0:t[e.name],onChange:s=>n(e.name,s),disabled:a,style:{width:"120px"},addonBefore:"$"})})]},r)})})}]})]})]}),(s.default_cost_per_query||s.tool_name_to_cost_per_query&&Object.keys(s.tool_name_to_cost_per_query).length>0)&&(0,l.jsxs)("div",{className:"mt-6 p-4 bg-blue-50 border border-blue-200 rounded-lg",children:[(0,l.jsx)(g.Z,{className:"text-blue-800 font-medium",children:"Cost Summary:"}),(0,l.jsxs)("div",{className:"mt-2 space-y-1",children:[s.default_cost_per_query&&(0,l.jsxs)(g.Z,{className:"text-blue-700",children:["• Default cost: $",s.default_cost_per_query.toFixed(4)," per query"]}),s.tool_name_to_cost_per_query&&Object.entries(s.tool_name_to_cost_per_query).map(e=>{let[s,r]=e;return null!=r&&(0,l.jsxs)(g.Z,{className:"text-blue-700",children:["• ",s,": $",r.toFixed(4)," per query"]},s)})]})]})]})})};let{Panel:$}=V.default;var W=e=>{let{availableAccessGroups:s,mcpServer:r,searchValue:a,setSearchValue:i,getAccessGroupOptions:c}=e,d=U.Z.useFormInstance();return(0,t.useEffect)(()=>{r&&r.extra_headers&&d.setFieldValue("extra_headers",r.extra_headers)},[r,d]),(0,l.jsx)(V.default,{className:"bg-gray-50 border border-gray-200 rounded-lg",expandIconPosition:"end",ghost:!1,children:(0,l.jsx)($,{header:(0,l.jsxs)("div",{className:"flex items-center",children:[(0,l.jsxs)("div",{className:"flex items-center space-x-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-blue-500 rounded-full"}),(0,l.jsx)("h3",{className:"text-lg font-semibold text-gray-900",children:"Permission Management / Access Control"})]}),(0,l.jsx)("p",{className:"text-sm text-gray-600 ml-4",children:"Configure access permissions and security settings (Optional)"})]}),className:"border-0",children:(0,l.jsxs)("div",{className:"space-y-6 pt-4",children:[(0,l.jsx)(U.Z.Item,{label:(0,l.jsxs)("span",{className:"text-sm font-medium text-gray-700 flex items-center",children:["MCP Access Groups",(0,l.jsx)(o.Z,{title:"Specify access groups for this MCP server. Users must be in at least one of these groups to access the server.",children:(0,l.jsx)(G.Z,{className:"ml-2 text-blue-400 hover:text-blue-600 cursor-help"})})]}),name:"mcp_access_groups",className:"mb-4",children:(0,l.jsx)(n.default,{mode:"tags",showSearch:!0,placeholder:"Select existing groups or type to create new ones",optionFilterProp:"value",filterOption:(e,s)=>{var r;return(null!==(r=null==s?void 0:s.value)&&void 0!==r?r:"").toLowerCase().includes(e.toLowerCase())},onSearch:e=>i(e),tokenSeparators:[","],options:c(),maxTagCount:"responsive",allowClear:!0})}),(0,l.jsx)(U.Z.Item,{label:(0,l.jsxs)("span",{className:"text-sm font-medium text-gray-700 flex items-center",children:["Extra Headers",(0,l.jsx)(o.Z,{title:"Forward custom headers from incoming requests to this MCP server (e.g., Authorization, X-Custom-Header, User-Agent)",children:(0,l.jsx)(G.Z,{className:"ml-2 text-blue-400 hover:text-blue-600 cursor-help"})}),(null==r?void 0:r.extra_headers)&&r.extra_headers.length>0&&(0,l.jsxs)("span",{className:"ml-2 text-xs bg-blue-100 text-blue-700 px-2 py-1 rounded-full",children:[r.extra_headers.length," configured"]})]}),name:"extra_headers",children:(0,l.jsx)(n.default,{mode:"tags",placeholder:(null==r?void 0:r.extra_headers)&&r.extra_headers.length>0?"Currently: ".concat(r.extra_headers.join(", ")):"Enter header names (e.g., Authorization, X-Custom-Header)",className:"rounded-lg",size:"large",tokenSeparators:[","],allowClear:!0})})]})},"permissions")})},Q=r(83669),X=r(87908),ee=r(61994);let es=e=>{let{accessToken:s,formValues:r,enabled:l=!0}=e,[a,n]=(0,t.useState)([]),[i,o]=(0,t.useState)(!1),[c,d]=(0,t.useState)(null),[m,x]=(0,t.useState)(!1),u=!!(r.url&&r.transport&&r.auth_type&&s),h=async()=>{if(s&&r.url){o(!0),d(null);try{let e={server_id:r.server_id||"",server_name:r.server_name||"",url:r.url,transport:r.transport,auth_type:r.auth_type,mcp_info:r.mcp_info},l=await (0,P.testMCPToolsListRequest)(s,e);if(l.tools&&!l.error)n(l.tools),d(null),l.tools.length>0&&!m&&x(!0);else{let e=l.message||"Failed to retrieve tools list";d(e),n([]),x(!1)}}catch(e){console.error("Tools fetch error:",e),d(e instanceof Error?e.message:String(e)),n([]),x(!1)}finally{o(!1)}}},p=()=>{n([]),d(null),x(!1)};return(0,t.useEffect)(()=>{l&&(u?h():p())},[r.url,r.transport,r.auth_type,s,l,u]),{tools:a,isLoadingTools:i,toolsError:c,hasShownSuccessMessage:m,canFetchTools:u,fetchTools:h,clearTools:p}};var er=e=>{let{accessToken:s,formValues:r,allowedTools:a,existingAllowedTools:n,onAllowedToolsChange:i}=e,o=(0,t.useRef)(0),{tools:c,isLoadingTools:d,toolsError:m,canFetchTools:x}=es({accessToken:s,formValues:r,enabled:!0});(0,t.useEffect)(()=>{if(c.length>0&&c.length!==o.current&&0===a.length){if(n&&n.length>0){let e=c.map(e=>e.name);i(n.filter(s=>e.includes(s)))}else i(c.map(e=>e.name))}o.current=c.length},[c,a.length,n,i]);let u=e=>{a.includes(e)?i(a.filter(s=>s!==e)):i([...a,e])};return x||r.url?(0,l.jsx)(I.Z,{children:(0,l.jsxs)("div",{className:"space-y-4",children:[(0,l.jsx)("div",{className:"flex items-center justify-between",children:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)(Y.Z,{className:"text-blue-600"}),(0,l.jsx)(v.Z,{children:"Tool Configuration"}),c.length>0&&(0,l.jsx)(D.Z,{count:c.length,style:{backgroundColor:"#52c41a"}})]})}),(0,l.jsx)("div",{className:"bg-blue-50 border border-blue-200 rounded-lg p-3",children:(0,l.jsxs)(g.Z,{className:"text-blue-800 text-sm",children:[(0,l.jsx)("strong",{children:"Select which tools users can call:"})," Only checked tools will be available for users to invoke. Unchecked tools will be blocked from execution."]})}),d&&(0,l.jsxs)("div",{className:"flex items-center justify-center py-6",children:[(0,l.jsx)(X.Z,{size:"large"}),(0,l.jsx)(g.Z,{className:"ml-3",children:"Loading tools..."})]}),m&&!d&&(0,l.jsxs)("div",{className:"text-center py-6 text-red-500 border rounded-lg border-dashed border-red-300 bg-red-50",children:[(0,l.jsx)(Y.Z,{className:"text-2xl mb-2"}),(0,l.jsx)(g.Z,{className:"text-red-600 font-medium",children:"Unable to load tools"}),(0,l.jsx)("br",{}),(0,l.jsx)(g.Z,{className:"text-sm text-red-500",children:m})]}),!d&&!m&&0===c.length&&x&&(0,l.jsxs)("div",{className:"text-center py-6 text-gray-400 border rounded-lg border-dashed",children:[(0,l.jsx)(Y.Z,{className:"text-2xl mb-2"}),(0,l.jsx)(g.Z,{children:"No tools available for configuration"}),(0,l.jsx)("br",{}),(0,l.jsx)(g.Z,{className:"text-sm",children:"Connect to an MCP server with tools to configure them"})]}),!x&&r.url&&(0,l.jsxs)("div",{className:"text-center py-6 text-gray-400 border rounded-lg border-dashed",children:[(0,l.jsx)(Y.Z,{className:"text-2xl mb-2"}),(0,l.jsx)(g.Z,{children:"Complete required fields to configure tools"}),(0,l.jsx)("br",{}),(0,l.jsx)(g.Z,{className:"text-sm",children:"Fill in URL, Transport, and Authentication to load available tools"})]}),!d&&!m&&c.length>0&&(0,l.jsxs)("div",{className:"space-y-3",children:[(0,l.jsxs)("div",{className:"flex items-center justify-between",children:[(0,l.jsxs)("div",{className:"flex items-center gap-2 p-3 bg-green-50 rounded-lg border border-green-200 flex-1",children:[(0,l.jsx)(Q.Z,{className:"text-green-600"}),(0,l.jsxs)(g.Z,{className:"text-green-700 font-medium",children:[a.length," of ",c.length," ",1===c.length?"tool":"tools"," enabled for user access"]})]}),(0,l.jsxs)("div",{className:"flex gap-2 ml-3",children:[(0,l.jsx)("button",{type:"button",onClick:()=>{i(c.map(e=>e.name))},className:"px-3 py-1.5 text-sm text-blue-600 hover:text-blue-700 hover:bg-blue-50 rounded-md transition-colors",children:"Enable All"}),(0,l.jsx)("button",{type:"button",onClick:()=>{i([])},className:"px-3 py-1.5 text-sm text-gray-600 hover:text-gray-700 hover:bg-gray-100 rounded-md transition-colors",children:"Disable All"})]})]}),(0,l.jsx)("div",{className:"space-y-2",children:c.map((e,s)=>(0,l.jsx)("div",{className:"p-4 rounded-lg border transition-colors cursor-pointer ".concat(a.includes(e.name)?"bg-blue-50 border-blue-300 hover:border-blue-400":"bg-gray-50 border-gray-200 hover:border-gray-300"),onClick:()=>u(e.name),children:(0,l.jsxs)("div",{className:"flex items-start gap-3",children:[(0,l.jsx)(ee.Z,{checked:a.includes(e.name),onChange:()=>u(e.name)}),(0,l.jsxs)("div",{className:"flex-1",children:[(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)(g.Z,{className:"font-medium text-gray-900",children:e.name}),(0,l.jsx)("span",{className:"px-2 py-0.5 text-xs rounded-full font-medium ".concat(a.includes(e.name)?"bg-green-100 text-green-800":"bg-red-100 text-red-800"),children:a.includes(e.name)?"Enabled":"Disabled"})]}),e.description&&(0,l.jsx)(g.Z,{className:"text-gray-500 text-sm block mt-1",children:e.description}),(0,l.jsx)(g.Z,{className:"text-gray-400 text-xs block mt-1",children:a.includes(e.name)?"✓ Users can call this tool":"✗ Users cannot call this tool"})]})]})},s))})]})]})}):null},el=r(9114),et=e=>{let{mcpServer:s,accessToken:r,onCancel:a,onSuccess:i,availableAccessGroups:o}=e,[p]=U.Z.useForm(),[j,g]=(0,t.useState)({}),[v,f]=(0,t.useState)([]),[b,y]=(0,t.useState)(!1),[N,_]=(0,t.useState)(""),[Z,w]=(0,t.useState)(!1),[k,A]=(0,t.useState)([]);(0,t.useEffect)(()=>{var e;(null===(e=s.mcp_info)||void 0===e?void 0:e.mcp_server_cost_info)&&g(s.mcp_info.mcp_server_cost_info)},[s]),(0,t.useEffect)(()=>{s.allowed_tools&&A(s.allowed_tools)},[s]),(0,t.useEffect)(()=>{if(s.mcp_access_groups){let e=s.mcp_access_groups.map(e=>"string"==typeof e?e:e.name||String(e));p.setFieldValue("mcp_access_groups",e)}},[s]),(0,t.useEffect)(()=>{L()},[s,r]);let L=async()=>{if(r&&s.url){y(!0);try{let e={server_id:s.server_id,server_name:s.server_name,url:s.url,transport:s.transport,auth_type:s.auth_type,mcp_info:s.mcp_info},l=await (0,P.testMCPToolsListRequest)(r,e);l.tools&&!l.error?f(l.tools):(console.error("Failed to fetch tools:",l.message),f([]))}catch(e){console.error("Tools fetch error:",e),f([])}finally{y(!1)}}},M=async e=>{if(r)try{let l=(e.mcp_access_groups||[]).map(e=>"string"==typeof e?e:e.name||String(e)),t={...e,server_id:s.server_id,mcp_info:{server_name:e.server_name||e.url,description:e.description,mcp_server_cost_info:Object.keys(j).length>0?j:null},mcp_access_groups:l,alias:e.alias,extra_headers:e.extra_headers||[],allowed_tools:k.length>0?k:null,disallowed_tools:e.disallowed_tools||[]},a=await (0,P.updateMCPServer)(r,t);el.Z.success("MCP Server updated successfully"),i(a)}catch(e){el.Z.fromBackend("Failed to update MCP Server"+((null==e?void 0:e.message)?": ".concat(e.message):""))}};return(0,l.jsxs)(m.Z,{children:[(0,l.jsxs)(x.Z,{className:"grid w-full grid-cols-2",children:[(0,l.jsx)(d.Z,{children:"Server Configuration"}),(0,l.jsx)(d.Z,{children:"Cost Configuration"})]}),(0,l.jsxs)(h.Z,{className:"mt-6",children:[(0,l.jsx)(u.Z,{children:(0,l.jsxs)(U.Z,{form:p,onFinish:M,initialValues:s,layout:"vertical",children:[(0,l.jsx)(U.Z.Item,{label:"MCP Server Name",name:"server_name",rules:[{validator:(e,s)=>S(s)}],children:(0,l.jsx)(B.Z,{})}),(0,l.jsx)(U.Z.Item,{label:"Alias",name:"alias",rules:[{validator:(e,s)=>S(s)}],children:(0,l.jsx)(B.Z,{onChange:()=>w(!0)})}),(0,l.jsx)(U.Z.Item,{label:"Description",name:"description",children:(0,l.jsx)(B.Z,{})}),(0,l.jsx)(U.Z.Item,{label:"MCP Server URL",name:"url",rules:[{required:!0,message:"Please enter a server URL"},{validator:(e,s)=>C(s)}],children:(0,l.jsx)(B.Z,{})}),(0,l.jsx)(U.Z.Item,{label:"Transport Type",name:"transport",rules:[{required:!0}],children:(0,l.jsxs)(n.default,{children:[(0,l.jsx)(n.default.Option,{value:"sse",children:"Server-Sent Events (SSE)"}),(0,l.jsx)(n.default.Option,{value:"http",children:"HTTP"})]})}),(0,l.jsx)(U.Z.Item,{label:"Authentication",name:"auth_type",rules:[{required:!0}],children:(0,l.jsxs)(n.default,{children:[(0,l.jsx)(n.default.Option,{value:"none",children:"None"}),(0,l.jsx)(n.default.Option,{value:"api_key",children:"API Key"}),(0,l.jsx)(n.default.Option,{value:"bearer_token",children:"Bearer Token"}),(0,l.jsx)(n.default.Option,{value:"basic",children:"Basic Auth"})]})}),(0,l.jsx)("div",{className:"mt-6",children:(0,l.jsx)(W,{availableAccessGroups:o,mcpServer:s,searchValue:N,setSearchValue:_,getAccessGroupOptions:()=>{let e=o.map(e=>({value:e,label:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-green-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:e})]})}));return N&&!o.some(e=>e.toLowerCase().includes(N.toLowerCase()))&&e.push({value:N,label:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-blue-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:N}),(0,l.jsx)("span",{className:"text-gray-400 text-xs ml-1",children:"create new group"})]})}),e}})}),(0,l.jsx)("div",{className:"mt-6",children:(0,l.jsx)(er,{accessToken:r,formValues:{server_id:s.server_id,server_name:s.server_name,url:s.url,transport:s.transport,auth_type:s.auth_type,mcp_info:s.mcp_info},allowedTools:k,existingAllowedTools:s.allowed_tools||null,onAllowedToolsChange:A})}),(0,l.jsxs)("div",{className:"flex justify-end gap-2",children:[(0,l.jsx)(F.ZP,{onClick:a,children:"Cancel"}),(0,l.jsx)(c.Z,{type:"submit",children:"Save Changes"})]})]})}),(0,l.jsx)(u.Z,{children:(0,l.jsxs)("div",{className:"space-y-6",children:[(0,l.jsx)(J,{value:j,onChange:g,tools:v,disabled:b}),(0,l.jsxs)("div",{className:"flex justify-end gap-2",children:[(0,l.jsx)(F.ZP,{onClick:a,children:"Cancel"}),(0,l.jsx)(c.Z,{onClick:()=>p.submit(),children:"Save Changes"})]})]})})]})]})},ea=r(92280),en=e=>{let{costConfig:s}=e,r=(null==s?void 0:s.default_cost_per_query)!==void 0&&(null==s?void 0:s.default_cost_per_query)!==null,t=(null==s?void 0:s.tool_name_to_cost_per_query)&&Object.keys(s.tool_name_to_cost_per_query).length>0;return r||t?(0,l.jsx)("div",{className:"mt-6 pt-6 border-t border-gray-200",children:(0,l.jsxs)("div",{className:"space-y-4",children:[r&&(null==s?void 0:s.default_cost_per_query)!==void 0&&(null==s?void 0:s.default_cost_per_query)!==null&&(0,l.jsxs)("div",{children:[(0,l.jsx)(ea.x,{className:"font-medium",children:"Default Cost per Query"}),(0,l.jsxs)("div",{className:"text-green-600 font-mono",children:["$",s.default_cost_per_query.toFixed(4)]})]}),t&&(null==s?void 0:s.tool_name_to_cost_per_query)&&(0,l.jsxs)("div",{children:[(0,l.jsx)(ea.x,{className:"font-medium",children:"Tool-Specific Costs"}),(0,l.jsx)("div",{className:"mt-2 space-y-2",children:Object.entries(s.tool_name_to_cost_per_query).map(e=>{let[s,r]=e;return null!=r&&(0,l.jsxs)("div",{className:"flex justify-between items-center p-3 bg-gray-50 rounded-lg",children:[(0,l.jsx)(ea.x,{className:"font-medium",children:s}),(0,l.jsxs)(ea.x,{className:"text-green-600 font-mono",children:["$",r.toFixed(4)," per query"]})]},s)})})]}),(0,l.jsxs)("div",{className:"mt-4 p-4 bg-blue-50 border border-blue-200 rounded-lg",children:[(0,l.jsx)(ea.x,{className:"text-blue-800 font-medium",children:"Cost Summary:"}),(0,l.jsxs)("div",{className:"mt-2 space-y-1",children:[r&&(null==s?void 0:s.default_cost_per_query)!==void 0&&(null==s?void 0:s.default_cost_per_query)!==null&&(0,l.jsxs)(ea.x,{className:"text-blue-700",children:["• Default cost: $",s.default_cost_per_query.toFixed(4)," per query"]}),t&&(null==s?void 0:s.tool_name_to_cost_per_query)&&(0,l.jsxs)(ea.x,{className:"text-blue-700",children:["• ",Object.keys(s.tool_name_to_cost_per_query).length," tool(s) with custom pricing"]})]})]})]})}):(0,l.jsx)("div",{className:"mt-6 pt-6 border-t border-gray-200",children:(0,l.jsx)("div",{className:"space-y-4",children:(0,l.jsx)("div",{className:"p-4 bg-gray-50 border border-gray-200 rounded-lg",children:(0,l.jsx)(ea.x,{className:"text-gray-600",children:"No cost configuration set for this server. Tool calls will be charged at $0.00 per tool call."})})})})},ei=r(59872),eo=r(30401),ec=r(78867);let ed=e=>{var s,r,a,n,i;let{mcpServer:o,onBack:p,isEditing:f,isProxyAdmin:y,accessToken:N,userRole:_,userID:Z,availableAccessGroups:C}=e,[S,k]=(0,t.useState)(f),[P,A]=(0,t.useState)(!1),[E,z]=(0,t.useState)({}),{maskedUrl:R,hasToken:U}=w(o.url),B=(e,s)=>U?s?e:R:e,K=async(e,s)=>{await (0,ei.vQ)(e)&&(z(e=>({...e,[s]:!0})),setTimeout(()=>{z(e=>({...e,[s]:!1}))},2e3))};return(0,l.jsxs)("div",{className:"p-4 max-w-full",children:[(0,l.jsx)("div",{className:"flex justify-between items-center mb-6",children:(0,l.jsxs)("div",{children:[(0,l.jsx)(c.Z,{icon:L.Z,variant:"light",className:"mb-4",onClick:p,children:"Back to All Servers"}),(0,l.jsxs)("div",{className:"flex items-center cursor-pointer",children:[(0,l.jsx)(v.Z,{children:o.server_name}),(0,l.jsx)(F.ZP,{type:"text",size:"small",icon:E["mcp-server_name"]?(0,l.jsx)(eo.Z,{size:12}):(0,l.jsx)(ec.Z,{size:12}),onClick:()=>K(o.server_name,"mcp-server_name"),className:"left-2 z-10 transition-all duration-200 ".concat(E["mcp-server_name"]?"text-green-600 bg-green-50 border-green-200":"text-gray-500 hover:text-gray-700 hover:bg-gray-100")}),o.alias&&(0,l.jsxs)(l.Fragment,{children:[(0,l.jsx)("span",{className:"ml-4 text-gray-500",children:"Alias:"}),(0,l.jsx)("span",{className:"ml-1 font-mono text-blue-600",children:o.alias}),(0,l.jsx)(F.ZP,{type:"text",size:"small",icon:E["mcp-alias"]?(0,l.jsx)(eo.Z,{size:12}):(0,l.jsx)(ec.Z,{size:12}),onClick:()=>K(o.alias,"mcp-alias"),className:"left-2 z-10 transition-all duration-200 ".concat(E["mcp-alias"]?"text-green-600 bg-green-50 border-green-200":"text-gray-500 hover:text-gray-700 hover:bg-gray-100")})]})]}),(0,l.jsxs)("div",{className:"flex items-center cursor-pointer",children:[(0,l.jsx)(g.Z,{className:"text-gray-500 font-mono",children:o.server_id}),(0,l.jsx)(F.ZP,{type:"text",size:"small",icon:E["mcp-server-id"]?(0,l.jsx)(eo.Z,{size:12}):(0,l.jsx)(ec.Z,{size:12}),onClick:()=>K(o.server_id,"mcp-server-id"),className:"left-2 z-10 transition-all duration-200 ".concat(E["mcp-server-id"]?"text-green-600 bg-green-50 border-green-200":"text-gray-500 hover:text-gray-700 hover:bg-gray-100")})]})]})}),(0,l.jsxs)(m.Z,{defaultIndex:S?2:0,children:[(0,l.jsx)(x.Z,{className:"mb-4",children:[(0,l.jsx)(d.Z,{children:"Overview"},"overview"),(0,l.jsx)(d.Z,{children:"MCP Tools"},"tools"),...y?[(0,l.jsx)(d.Z,{children:"Settings"},"settings")]:[]]}),(0,l.jsxs)(h.Z,{children:[(0,l.jsxs)(u.Z,{children:[(0,l.jsxs)(j.Z,{numItems:1,numItemsSm:2,numItemsLg:3,className:"gap-6",children:[(0,l.jsxs)(I.Z,{children:[(0,l.jsx)(g.Z,{children:"Transport"}),(0,l.jsx)("div",{className:"mt-2",children:(0,l.jsx)(v.Z,{children:q(null!==(n=o.transport)&&void 0!==n?n:void 0)})})]}),(0,l.jsxs)(I.Z,{children:[(0,l.jsx)(g.Z,{children:"Auth Type"}),(0,l.jsx)("div",{className:"mt-2",children:(0,l.jsx)(g.Z,{children:O(null!==(i=o.auth_type)&&void 0!==i?i:void 0)})})]}),(0,l.jsxs)(I.Z,{children:[(0,l.jsx)(g.Z,{children:"Host Url"}),(0,l.jsxs)("div",{className:"mt-2 flex items-center gap-2",children:[(0,l.jsx)(g.Z,{className:"break-all overflow-wrap-anywhere",children:B(o.url,P)}),U&&(0,l.jsx)("button",{onClick:()=>A(!P),className:"p-1 hover:bg-gray-100 rounded",children:(0,l.jsx)(b.Z,{icon:P?M.Z:T.Z,size:"sm",className:"text-gray-500"})})]})]})]}),(0,l.jsxs)(I.Z,{className:"mt-2",children:[(0,l.jsx)(v.Z,{children:"Cost Configuration"}),(0,l.jsx)(en,{costConfig:null===(s=o.mcp_info)||void 0===s?void 0:s.mcp_server_cost_info})]})]}),(0,l.jsx)(u.Z,{children:(0,l.jsx)(eY,{serverId:o.server_id,accessToken:N,auth_type:o.auth_type,userRole:_,userID:Z,serverAlias:o.alias})}),(0,l.jsx)(u.Z,{children:(0,l.jsxs)(I.Z,{children:[(0,l.jsxs)("div",{className:"flex justify-between items-center mb-4",children:[(0,l.jsx)(v.Z,{children:"MCP Server Settings"}),S?null:(0,l.jsx)(c.Z,{variant:"light",onClick:()=>k(!0),children:"Edit Settings"})]}),S?(0,l.jsx)(et,{mcpServer:o,accessToken:N,onCancel:()=>k(!1),onSuccess:e=>{k(!1),p()},availableAccessGroups:C}):(0,l.jsxs)("div",{className:"space-y-4",children:[(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Server Name"}),(0,l.jsx)("div",{children:o.server_name})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Alias"}),(0,l.jsx)("div",{children:o.alias})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Description"}),(0,l.jsx)("div",{children:o.description})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"URL"}),(0,l.jsxs)("div",{className:"font-mono break-all overflow-wrap-anywhere max-w-full flex items-center gap-2",children:[B(o.url,P),U&&(0,l.jsx)("button",{onClick:()=>A(!P),className:"p-1 hover:bg-gray-100 rounded",children:(0,l.jsx)(b.Z,{icon:P?M.Z:T.Z,size:"sm",className:"text-gray-500"})})]})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Transport"}),(0,l.jsx)("div",{children:q(o.transport)})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Extra Headers"}),(0,l.jsx)("div",{children:null===(r=o.extra_headers)||void 0===r?void 0:r.join(", ")})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Auth Type"}),(0,l.jsx)("div",{children:O(o.auth_type)})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Access Groups"}),(0,l.jsx)("div",{children:o.mcp_access_groups&&o.mcp_access_groups.length>0?(0,l.jsx)("div",{className:"flex flex-wrap gap-2",children:o.mcp_access_groups.map((e,s)=>{var r;return(0,l.jsx)("span",{className:"px-2 py-1 bg-gray-100 rounded-md text-sm",children:"string"==typeof e?e:null!==(r=null==e?void 0:e.name)&&void 0!==r?r:""},s)})}):(0,l.jsx)(g.Z,{className:"text-gray-500",children:"No access groups defined"})})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Allowed Tools"}),(0,l.jsx)("div",{children:o.allowed_tools&&o.allowed_tools.length>0?(0,l.jsx)("div",{className:"flex flex-wrap gap-2",children:o.allowed_tools.map((e,s)=>(0,l.jsx)("span",{className:"px-2 py-1 bg-blue-50 border border-blue-200 rounded-md text-sm",children:e},s))}):(0,l.jsx)(g.Z,{className:"text-gray-500",children:"All tools enabled"})})]}),(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"font-medium",children:"Cost Configuration"}),(0,l.jsx)(en,{costConfig:null===(a=o.mcp_info)||void 0===a?void 0:a.mcp_server_cost_info})]})]})]})})]})]})]})};var em=r(64504),ex=r(61778),eu=r(29271),eh=r(89245),ep=e=>{let{accessToken:s,formValues:r,onToolsLoaded:a}=e,{tools:n,isLoadingTools:i,toolsError:o,canFetchTools:c,fetchTools:d}=es({accessToken:s,formValues:r,enabled:!0});return((0,t.useEffect)(()=>{null==a||a(n)},[n,a]),c||r.url)?(0,l.jsx)(I.Z,{children:(0,l.jsxs)("div",{className:"space-y-4",children:[(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)(Q.Z,{className:"text-blue-600"}),(0,l.jsx)(v.Z,{children:"Connection Status"})]}),!c&&r.url&&(0,l.jsxs)("div",{className:"text-center py-6 text-gray-400 border rounded-lg border-dashed",children:[(0,l.jsx)(Y.Z,{className:"text-2xl mb-2"}),(0,l.jsx)(g.Z,{children:"Complete required fields to test connection"}),(0,l.jsx)("br",{}),(0,l.jsx)(g.Z,{className:"text-sm",children:"Fill in URL, Transport, and Authentication to test MCP server connection"})]}),c&&(0,l.jsxs)("div",{children:[(0,l.jsxs)("div",{className:"flex items-center justify-between mb-4",children:[(0,l.jsxs)("div",{children:[(0,l.jsx)(g.Z,{className:"text-gray-700 font-medium",children:i?"Testing connection to MCP server...":n.length>0?"Connection successful":o?"Connection failed":"Ready to test connection"}),(0,l.jsx)("br",{}),(0,l.jsxs)(g.Z,{className:"text-gray-500 text-sm",children:["Server: ",r.url]})]}),i&&(0,l.jsxs)("div",{className:"flex items-center text-blue-600",children:[(0,l.jsx)(X.Z,{size:"small",className:"mr-2"}),(0,l.jsx)(g.Z,{className:"text-blue-600",children:"Connecting..."})]}),!i&&!o&&n.length>0&&(0,l.jsxs)("div",{className:"flex items-center text-green-600",children:[(0,l.jsx)(Q.Z,{className:"mr-1"}),(0,l.jsx)(g.Z,{className:"text-green-600 font-medium",children:"Connected"})]}),o&&(0,l.jsxs)("div",{className:"flex items-center text-red-600",children:[(0,l.jsx)(eu.Z,{className:"mr-1"}),(0,l.jsx)(g.Z,{className:"text-red-600 font-medium",children:"Failed"})]})]}),i&&(0,l.jsxs)("div",{className:"flex items-center justify-center py-6",children:[(0,l.jsx)(X.Z,{size:"large"}),(0,l.jsx)(g.Z,{className:"ml-3",children:"Testing connection and loading tools..."})]}),o&&(0,l.jsx)(ex.Z,{message:"Connection Failed",description:o,type:"error",showIcon:!0,action:(0,l.jsx)(F.ZP,{icon:(0,l.jsx)(eh.Z,{}),onClick:d,size:"small",children:"Retry"})}),!i&&0===n.length&&!o&&(0,l.jsxs)("div",{className:"text-center py-6 text-gray-500 border rounded-lg border-dashed",children:[(0,l.jsx)(Q.Z,{className:"text-2xl mb-2 text-green-500"}),(0,l.jsx)(g.Z,{className:"text-green-600 font-medium",children:"Connection successful!"}),(0,l.jsx)("br",{}),(0,l.jsx)(g.Z,{className:"text-gray-500",children:"No tools found for this MCP server"})]})]})]})}):null},ej=r(64482),eg=e=>{let{isVisible:s}=e;return s?(0,l.jsx)(U.Z.Item,{label:(0,l.jsxs)("span",{className:"text-sm font-medium text-gray-700 flex items-center",children:["Stdio Configuration (JSON)",(0,l.jsx)(o.Z,{title:"Paste your stdio MCP server configuration in JSON format. You can use the full mcpServers structure from config.yaml or just the inner server configuration.",children:(0,l.jsx)(G.Z,{className:"ml-2 text-blue-400 hover:text-blue-600 cursor-help"})})]}),name:"stdio_config",rules:[{required:!0,message:"Please enter stdio configuration"},{validator:(e,s)=>{if(!s)return Promise.resolve();try{return JSON.parse(s),Promise.resolve()}catch(e){return Promise.reject("Please enter valid JSON")}}}],children:(0,l.jsx)(ej.default.TextArea,{placeholder:'{\n "mcpServers": {\n "circleci-mcp-server": {\n "command": "npx",\n "args": ["-y", "@circleci/mcp-server-circleci"],\n "env": {\n "CIRCLECI_TOKEN": "your-circleci-token",\n "CIRCLECI_BASE_URL": "https://circleci.com"\n }\n }\n }\n}',rows:12,className:"rounded-lg border-gray-300 focus:border-blue-500 focus:ring-blue-500 font-mono text-sm"})}):null};let ev="".concat("../ui/assets/logos/","mcp_logo.png");var ef=e=>{let{userRole:s,accessToken:r,onCreateSuccess:a,isModalVisible:c,setModalVisible:d,availableAccessGroups:m}=e,[x]=U.Z.useForm(),[u,h]=(0,t.useState)(!1),[p,j]=(0,t.useState)({}),[g,v]=(0,t.useState)({}),[f,b]=(0,t.useState)(!1),[y,N]=(0,t.useState)([]),[_,Z]=(0,t.useState)([]),[w,k]=(0,t.useState)(""),[L,M]=(0,t.useState)(""),[T,I]=(0,t.useState)(""),E=(e,s)=>{if(!e){I("");return}"sse"!==s||e.endsWith("/sse")?"http"!==s||e.endsWith("/mcp")?I(""):I("Typically MCP HTTP URLs end with /mcp. You can add this url but this is a warning."):I("Typically MCP SSE URLs end with /sse. You can add this url but this is a warning.")},z=async e=>{h(!0);try{let s=e.mcp_access_groups,l={};if(e.stdio_config&&"stdio"===w)try{let s=JSON.parse(e.stdio_config),r=s;if(s.mcpServers&&"object"==typeof s.mcpServers){let l=Object.keys(s.mcpServers);if(l.length>0){let t=l[0];r=s.mcpServers[t],e.server_name||(e.server_name=t.replace(/-/g,"_"))}}l={command:r.command,args:r.args,env:r.env},console.log("Parsed stdio config:",l)}catch(e){el.Z.fromBackend("Invalid JSON in stdio configuration");return}let t={...e,...l,stdio_config:void 0,mcp_info:{server_name:e.server_name||e.url,description:e.description,mcp_server_cost_info:Object.keys(p).length>0?p:null},mcp_access_groups:s,alias:e.alias,allowed_tools:_.length>0?_:null};if(console.log("Payload: ".concat(JSON.stringify(t))),null!=r){let e=await (0,P.createMCPServer)(r,t);el.Z.success("MCP Server created successfully"),x.resetFields(),j({}),N([]),Z([]),I(""),b(!1),d(!1),a(e)}}catch(e){el.Z.fromBackend("Error creating MCP Server: "+e)}finally{h(!1)}},q=()=>{x.resetFields(),j({}),N([]),Z([]),I(""),b(!1),d(!1)};return(t.useEffect(()=>{if(!f&&g.server_name){let e=g.server_name.replace(/\s+/g,"_");x.setFieldsValue({alias:e}),v(s=>({...s,alias:e}))}},[g.server_name]),t.useEffect(()=>{c||v({})},[c]),(0,A.tY)(s))?(0,l.jsx)(i.Z,{title:(0,l.jsxs)("div",{className:"flex items-center space-x-3 pb-4 border-b border-gray-100",children:[(0,l.jsx)("img",{src:ev,alt:"MCP Logo",className:"w-8 h-8 object-contain",style:{height:"20px",width:"20px",marginRight:"8px",objectFit:"contain"}}),(0,l.jsx)("h2",{className:"text-xl font-semibold text-gray-900",children:"Add New MCP Server"})]}),open:c,width:1e3,onCancel:q,footer:null,className:"top-8",styles:{body:{padding:"24px"},header:{padding:"24px 24px 0 24px",border:"none"}},children:(0,l.jsx)("div",{className:"mt-6",children:(0,l.jsxs)(U.Z,{form:x,onFinish:z,onValuesChange:(e,s)=>v(s),layout:"vertical",className:"space-y-6",children:[(0,l.jsxs)("div",{className:"grid grid-cols-1 gap-6",children:[(0,l.jsx)(U.Z.Item,{label:(0,l.jsxs)("span",{className:"text-sm font-medium text-gray-700 flex items-center",children:["MCP Server Name",(0,l.jsx)(o.Z,{title:"Best practice: Use a descriptive name that indicates the server's purpose (e.g., 'GitHub_MCP', 'Email_Service'). Hyphens '-' are not allowed; use underscores '_' instead.",children:(0,l.jsx)(G.Z,{className:"ml-2 text-blue-400 hover:text-blue-600 cursor-help"})})]}),name:"server_name",rules:[{required:!1,message:"Please enter a server name"},{validator:(e,s)=>S(s)}],children:(0,l.jsx)(em.o,{placeholder:"e.g., GitHub_MCP, Zapier_MCP, etc.",className:"rounded-lg border-gray-300 focus:border-blue-500 focus:ring-blue-500"})}),(0,l.jsx)(U.Z.Item,{label:(0,l.jsxs)("span",{className:"text-sm font-medium text-gray-700 flex items-center",children:["Alias",(0,l.jsx)(o.Z,{title:"A short, unique identifier for this server. Defaults to the server name with spaces replaced by underscores.",children:(0,l.jsx)(G.Z,{className:"ml-2 text-blue-400 hover:text-blue-600 cursor-help"})})]}),name:"alias",rules:[{required:!1},{validator:(e,s)=>s&&s.includes("-")?Promise.reject("Alias cannot contain '-' (hyphen). Please use '_' (underscore) instead."):Promise.resolve()}],children:(0,l.jsx)(em.o,{placeholder:"e.g., GitHub_MCP, Zapier_MCP, etc.",className:"rounded-lg border-gray-300 focus:border-blue-500 focus:ring-blue-500",onChange:()=>b(!0)})}),(0,l.jsx)(U.Z.Item,{label:(0,l.jsx)("span",{className:"text-sm font-medium text-gray-700",children:"Description"}),name:"description",rules:[{required:!1,message:"Please enter a server description!!!!!!!!!"}],children:(0,l.jsx)(em.o,{placeholder:"Brief description of what this server does",className:"rounded-lg border-gray-300 focus:border-blue-500 focus:ring-blue-500"})}),(0,l.jsx)(U.Z.Item,{label:(0,l.jsx)("span",{className:"text-sm font-medium text-gray-700",children:"Transport Type"}),name:"transport",rules:[{required:!0,message:"Please select a transport type"}],children:(0,l.jsxs)(n.default,{placeholder:"Select transport",className:"rounded-lg",size:"large",onChange:e=>{if(k(e),"stdio"===e)x.setFieldsValue({url:void 0,auth_type:void 0}),I("");else{x.setFieldsValue({command:void 0,args:void 0,env:void 0});let s=x.getFieldValue("url");s&&E(s,e)}},value:w,children:[(0,l.jsx)(n.default.Option,{value:"http",children:"HTTP"}),(0,l.jsx)(n.default.Option,{value:"sse",children:"Server-Sent Events (SSE)"}),(0,l.jsx)(n.default.Option,{value:"stdio",children:"Standard Input/Output (stdio)"})]})}),"stdio"!==w&&(0,l.jsx)(U.Z.Item,{label:(0,l.jsx)("span",{className:"text-sm font-medium text-gray-700",children:"MCP Server URL"}),name:"url",rules:[{required:!0,message:"Please enter a server URL"},{validator:(e,s)=>C(s)}],children:(0,l.jsxs)("div",{children:[(0,l.jsx)(em.o,{placeholder:"https://your-mcp-server.com",className:"rounded-lg border-gray-300 focus:border-blue-500 focus:ring-blue-500",onChange:e=>E(e.target.value,w)}),T&&(0,l.jsx)("div",{className:"mt-1 text-red-500 text-sm font-medium",children:T})]})}),"stdio"!==w&&(0,l.jsx)(U.Z.Item,{label:(0,l.jsx)("span",{className:"text-sm font-medium text-gray-700",children:"Authentication"}),name:"auth_type",rules:[{required:!0,message:"Please select an auth type"}],children:(0,l.jsxs)(n.default,{placeholder:"Select auth type",className:"rounded-lg",size:"large",children:[(0,l.jsx)(n.default.Option,{value:"none",children:"None"}),(0,l.jsx)(n.default.Option,{value:"api_key",children:"API Key"}),(0,l.jsx)(n.default.Option,{value:"bearer_token",children:"Bearer Token"}),(0,l.jsx)(n.default.Option,{value:"basic",children:"Basic Auth"})]})}),(0,l.jsx)(eg,{isVisible:"stdio"===w})]}),(0,l.jsx)("div",{className:"mt-8",children:(0,l.jsx)(W,{availableAccessGroups:m,mcpServer:null,searchValue:L,setSearchValue:M,getAccessGroupOptions:()=>{let e=m.map(e=>({value:e,label:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-green-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:e})]})}));return L&&!m.some(e=>e.toLowerCase().includes(L.toLowerCase()))&&e.push({value:L,label:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-blue-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:L}),(0,l.jsx)("span",{className:"text-gray-400 text-xs ml-1",children:"create new group"})]})}),e}})}),(0,l.jsx)("div",{className:"mt-8 pt-6 border-t border-gray-200",children:(0,l.jsx)(ep,{accessToken:r,formValues:g,onToolsLoaded:N})}),(0,l.jsx)("div",{className:"mt-6",children:(0,l.jsx)(er,{accessToken:r,formValues:g,allowedTools:_,existingAllowedTools:null,onAllowedToolsChange:Z})}),(0,l.jsx)("div",{className:"mt-6",children:(0,l.jsx)(J,{value:p,onChange:j,tools:y.filter(e=>_.includes(e.name)),disabled:!1})}),(0,l.jsxs)("div",{className:"flex items-center justify-end space-x-3 pt-6 border-t border-gray-100",children:[(0,l.jsx)(em.z,{variant:"secondary",onClick:q,children:"Cancel"}),(0,l.jsx)(em.z,{variant:"primary",loading:u,children:u?"Creating...":"Add MCP Server"})]})]})})}):null},eb=r(93192),ey=r(67960),eN=r(63709),e_=r(93142),eZ=r(64935),ew=r(11239),eC=r(54001),eS=r(96137),ek=r(96362),eP=r(80221),eA=r(29202);let{Title:eL,Text:eM}=eb.default,{Panel:eT}=V.default,eI=e=>{let{icon:s,title:r,description:a,children:n,serverName:i,accessGroups:o=["dev"]}=e,[c,d]=(0,t.useState)(!1),m=()=>{let e={"x-litellm-api-key":"Bearer YOUR_LITELLM_API_KEY"};if(c&&i){let s=[i.replace(/\s+/g,"_"),...o].join(",");e["x-mcp-servers"]=[s]}return e};return(0,l.jsxs)(ey.Z,{className:"border border-gray-200",children:[(0,l.jsxs)("div",{className:"flex items-center gap-3 mb-3",children:[(0,l.jsx)("span",{className:"p-2 rounded-lg bg-gray-50",children:s}),(0,l.jsxs)("div",{children:[(0,l.jsx)(eL,{level:5,className:"mb-0",children:r}),(0,l.jsx)(eM,{className:"text-gray-600",children:a})]})]}),i&&("Implementation Example"===r||"Configuration"===r)&&(0,l.jsxs)(U.Z.Item,{className:"mb-4",children:[(0,l.jsxs)("div",{className:"flex items-center gap-2 mb-2",children:[(0,l.jsx)(eN.Z,{size:"small",checked:c,onChange:d}),(0,l.jsxs)(eM,{className:"text-sm",children:["Limit tools to specific MCP servers or MCP groups by passing the ",(0,l.jsx)("code",{children:"x-mcp-servers"})," header"]})]}),c&&(0,l.jsx)(ex.Z,{className:"mt-2",type:"info",showIcon:!0,message:"Two Options",description:(0,l.jsxs)("div",{children:[(0,l.jsxs)("p",{children:[(0,l.jsx)("strong",{children:"Option 1:"})," Get a specific server: ",(0,l.jsxs)("code",{children:['["',i.replace(/\s+/g,"_"),'"]']})]}),(0,l.jsxs)("p",{children:[(0,l.jsx)("strong",{children:"Option 2:"})," Get a group of MCPs: ",(0,l.jsx)("code",{children:'["dev-group"]'})]}),(0,l.jsxs)("p",{className:"mt-2 text-sm text-gray-600",children:["You can also mix both: ",(0,l.jsx)("code",{children:'["Server1,dev-group"]'})]})]})})]}),t.Children.map(n,e=>{if(t.isValidElement(e)&&e.props.hasOwnProperty("code")&&e.props.hasOwnProperty("copyKey")){let s=e.props.code;if(s&&s.includes('"headers":'))return t.cloneElement(e,{code:s.replace(/"headers":\s*{[^}]*}/,'"headers": '.concat(JSON.stringify(m(),null,8)))})}return e})]})};var eE=e=>{let{currentServerAccessGroups:s=[]}=e,r=(0,P.getProxyBaseUrl)(),[a,n]=(0,t.useState)({}),[i,o]=(0,t.useState)({openai:[],litellm:[],cursor:[],http:[]}),[c]=(0,t.useState)("Zapier_MCP"),p=async(e,s)=>{await (0,ei.vQ)(e)&&(n(e=>({...e,[s]:!0})),setTimeout(()=>{n(e=>({...e,[s]:!1}))},2e3))},j=e=>{let{code:s,copyKey:r,title:t,className:n=""}=e;return(0,l.jsxs)("div",{className:"relative group",children:[t&&(0,l.jsxs)("div",{className:"flex items-center gap-2 mb-2",children:[(0,l.jsx)(eZ.Z,{size:16,className:"text-blue-600"}),(0,l.jsx)(eM,{strong:!0,className:"text-gray-700",children:t})]}),(0,l.jsxs)(ey.Z,{className:"bg-gray-50 border border-gray-200 relative ".concat(n),children:[(0,l.jsx)(F.ZP,{type:"text",size:"small",icon:a[r]?(0,l.jsx)(eo.Z,{size:12}):(0,l.jsx)(ec.Z,{size:12}),onClick:()=>p(s,r),className:"absolute top-2 right-2 z-10 transition-all duration-200 ".concat(a[r]?"text-green-600 bg-green-50 border-green-200":"text-gray-500 hover:text-gray-700 hover:bg-gray-100")}),(0,l.jsx)("pre",{className:"text-sm overflow-x-auto pr-10 text-gray-800 font-mono leading-relaxed",children:s})]})]})},f=e=>{let{step:s,title:r,children:t}=e;return(0,l.jsxs)("div",{className:"flex gap-4",children:[(0,l.jsx)("div",{className:"flex-shrink-0",children:(0,l.jsx)("div",{className:"w-8 h-8 bg-blue-600 text-white rounded-full flex items-center justify-center text-sm font-semibold",children:s})}),(0,l.jsxs)("div",{className:"flex-1",children:[(0,l.jsx)(eM,{strong:!0,className:"text-gray-800 block mb-2",children:r}),t]})]})};return(0,l.jsx)("div",{children:(0,l.jsxs)(e_.Z,{direction:"vertical",size:"large",className:"w-full",children:[(0,l.jsxs)("div",{children:[(0,l.jsx)(v.Z,{className:"text-3xl font-bold text-gray-900 mb-3",children:"Connect to your MCP client"}),(0,l.jsx)(g.Z,{className:"text-lg text-gray-600",children:"Use tools directly from any MCP client with LiteLLM MCP. Enable your AI assistant to perform real-world tasks through a simple, secure connection."})]}),(0,l.jsxs)(m.Z,{className:"w-full",children:[(0,l.jsx)(x.Z,{className:"flex justify-start mt-8 mb-6",children:(0,l.jsxs)("div",{className:"flex bg-gray-100 p-1 rounded-lg",children:[(0,l.jsx)(d.Z,{className:"px-6 py-3 rounded-md transition-all duration-200",children:(0,l.jsxs)("span",{className:"flex items-center gap-2 font-medium",children:[(0,l.jsx)(eZ.Z,{size:18}),"OpenAI API"]})}),(0,l.jsx)(d.Z,{className:"px-6 py-3 rounded-md transition-all duration-200",children:(0,l.jsxs)("span",{className:"flex items-center gap-2 font-medium",children:[(0,l.jsx)(ew.Z,{size:18}),"LiteLLM Proxy"]})}),(0,l.jsx)(d.Z,{className:"px-6 py-3 rounded-md transition-all duration-200",children:(0,l.jsxs)("span",{className:"flex items-center gap-2 font-medium",children:[(0,l.jsx)(eP.Z,{size:18}),"Cursor"]})}),(0,l.jsx)(d.Z,{className:"px-6 py-3 rounded-md transition-all duration-200",children:(0,l.jsxs)("span",{className:"flex items-center gap-2 font-medium",children:[(0,l.jsx)(eA.Z,{size:18}),"Streamable HTTP"]})})]})}),(0,l.jsxs)(h.Z,{children:[(0,l.jsx)(u.Z,{className:"mt-6",children:(0,l.jsx)(()=>(0,l.jsxs)(e_.Z,{direction:"vertical",size:"large",className:"w-full",children:[(0,l.jsxs)("div",{className:"bg-gradient-to-r from-blue-50 to-indigo-50 p-6 rounded-lg border border-blue-100",children:[(0,l.jsxs)("div",{className:"flex items-center gap-3 mb-3",children:[(0,l.jsx)(eZ.Z,{className:"text-blue-600",size:24}),(0,l.jsx)(eL,{level:4,className:"mb-0 text-blue-900",children:"OpenAI Responses API Integration"})]}),(0,l.jsx)(eM,{className:"text-blue-700",children:"Connect OpenAI Responses API to your LiteLLM MCP server for seamless tool integration"})]}),(0,l.jsxs)(e_.Z,{direction:"vertical",size:"large",className:"w-full",children:[(0,l.jsx)(eI,{icon:(0,l.jsx)(eC.Z,{className:"text-blue-600",size:16}),title:"API Key Setup",description:"Configure your OpenAI API key for authentication",children:(0,l.jsxs)(e_.Z,{direction:"vertical",size:"middle",className:"w-full",children:[(0,l.jsx)("div",{children:(0,l.jsxs)(eM,{children:["Get your API key from the"," ",(0,l.jsxs)("a",{href:"https://platform.openai.com/api-keys",target:"_blank",rel:"noopener noreferrer",className:"text-blue-600 hover:text-blue-700 inline-flex items-center gap-1",children:["OpenAI platform ",(0,l.jsx)(ek.Z,{size:12})]})]})}),(0,l.jsx)(j,{title:"Environment Variable",code:'export OPENAI_API_KEY="sk-..."',copyKey:"openai-env"})]})}),(0,l.jsx)(eI,{icon:(0,l.jsx)(eS.Z,{className:"text-blue-600",size:16}),title:"MCP Server Information",description:"Connection details for your LiteLLM MCP server",children:(0,l.jsx)(j,{title:"Server URL",code:"".concat(r,"/mcp"),copyKey:"openai-server-url"})}),(0,l.jsx)(eI,{icon:(0,l.jsx)(eZ.Z,{className:"text-blue-600",size:16}),title:"Implementation Example",description:"Complete cURL example for using the Responses API",serverName:"Zapier Gmail",accessGroups:["dev"],children:(0,l.jsx)(j,{code:'curl --location \'https://api.openai.com/v1/responses\' \\\n--header \'Content-Type: application/json\' \\\n--header "Authorization: Bearer $OPENAI_API_KEY" \\\n--data \'{\n "model": "gpt-4.1",\n "tools": [\n {\n "type": "mcp",\n "server_label": "litellm",\n "server_url": "'.concat(r,'/mcp",\n "require_approval": "never",\n "headers": {\n "x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY",\n "x-mcp-servers": ["Zapier_MCP,dev"]\n }\n }\n ],\n "input": "Run available tools",\n "tool_choice": "required"\n}\''),copyKey:"openai-curl",className:"text-xs"})})]})]}),{})}),(0,l.jsx)(u.Z,{className:"mt-6",children:(0,l.jsx)(()=>(0,l.jsxs)(e_.Z,{direction:"vertical",size:"large",className:"w-full",children:[(0,l.jsxs)("div",{className:"bg-gradient-to-r from-emerald-50 to-green-50 p-6 rounded-lg border border-emerald-100",children:[(0,l.jsxs)("div",{className:"flex items-center gap-3 mb-3",children:[(0,l.jsx)(ew.Z,{className:"text-emerald-600",size:24}),(0,l.jsx)(eL,{level:4,className:"mb-0 text-emerald-900",children:"LiteLLM Proxy API Integration"})]}),(0,l.jsx)(eM,{className:"text-emerald-700",children:"Connect to LiteLLM Proxy Responses API for seamless tool integration with multiple model providers"})]}),(0,l.jsxs)(e_.Z,{direction:"vertical",size:"large",className:"w-full",children:[(0,l.jsx)(eI,{icon:(0,l.jsx)(eC.Z,{className:"text-emerald-600",size:16}),title:"API Key Setup",description:"Configure your LiteLLM Proxy API key for authentication",children:(0,l.jsxs)(e_.Z,{direction:"vertical",size:"middle",className:"w-full",children:[(0,l.jsx)("div",{children:(0,l.jsx)(eM,{children:"Get your API key from your LiteLLM Proxy dashboard or contact your administrator"})}),(0,l.jsx)(j,{title:"Environment Variable",code:'export LITELLM_API_KEY="sk-..."',copyKey:"litellm-env"})]})}),(0,l.jsx)(eI,{icon:(0,l.jsx)(eS.Z,{className:"text-emerald-600",size:16}),title:"MCP Server Information",description:"Connection details for your LiteLLM MCP server",children:(0,l.jsx)(j,{title:"Server URL",code:"".concat(r,"/mcp"),copyKey:"litellm-server-url"})}),(0,l.jsx)(eI,{icon:(0,l.jsx)(eZ.Z,{className:"text-emerald-600",size:16}),title:"Implementation Example",description:"Complete cURL example for using the LiteLLM Proxy Responses API",serverName:c,accessGroups:["dev"],children:(0,l.jsx)(j,{code:"curl --location '".concat(r,'/v1/responses\' \\\n--header \'Content-Type: application/json\' \\\n--header "Authorization: Bearer $LITELLM_API_KEY" \\\n--data \'{\n "model": "gpt-4",\n "tools": [\n {\n "type": "mcp",\n "server_label": "litellm",\n "server_url": "').concat(r,'/mcp",\n "require_approval": "never",\n "headers": {\n "x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY",\n "x-mcp-servers": ["Zapier_MCP,dev"]\n }\n }\n ],\n "input": "Run available tools",\n "tool_choice": "required"\n}\''),copyKey:"litellm-curl",className:"text-xs"})})]})]}),{})}),(0,l.jsx)(u.Z,{className:"mt-6",children:(0,l.jsx)(()=>(0,l.jsxs)(e_.Z,{direction:"vertical",size:"large",className:"w-full",children:[(0,l.jsxs)("div",{className:"bg-gradient-to-r from-purple-50 to-blue-50 p-6 rounded-lg border border-purple-100",children:[(0,l.jsxs)("div",{className:"flex items-center gap-3 mb-3",children:[(0,l.jsx)(eP.Z,{className:"text-purple-600",size:24}),(0,l.jsx)(eL,{level:4,className:"mb-0 text-purple-900",children:"Cursor IDE Integration"})]}),(0,l.jsx)(eM,{className:"text-purple-700",children:"Use tools directly from Cursor IDE with LiteLLM MCP. Enable your AI assistant to perform real-world tasks without leaving your coding environment."})]}),(0,l.jsxs)(ey.Z,{className:"border border-gray-200",children:[(0,l.jsx)(eL,{level:5,className:"mb-4 text-gray-800",children:"Setup Instructions"}),(0,l.jsxs)(e_.Z,{direction:"vertical",size:"large",className:"w-full",children:[(0,l.jsx)(f,{step:1,title:"Open Cursor Settings",children:(0,l.jsxs)(eM,{className:"text-gray-600",children:["Use the keyboard shortcut ",(0,l.jsx)("code",{className:"bg-gray-100 px-2 py-1 rounded",children:"⇧+⌘+J"})," (Mac) or"," ",(0,l.jsx)("code",{className:"bg-gray-100 px-2 py-1 rounded",children:"Ctrl+Shift+J"})," (Windows/Linux)"]})}),(0,l.jsx)(f,{step:2,title:"Navigate to MCP Tools",children:(0,l.jsx)(eM,{className:"text-gray-600",children:'Go to the "MCP Tools" tab and click "New MCP Server"'})}),(0,l.jsxs)(f,{step:3,title:"Add Configuration",children:[(0,l.jsxs)(eM,{className:"text-gray-600 mb-3",children:["Copy the JSON configuration below and paste it into Cursor, then save with"," ",(0,l.jsx)("code",{className:"bg-gray-100 px-2 py-1 rounded",children:"Cmd+S"})," or"," ",(0,l.jsx)("code",{className:"bg-gray-100 px-2 py-1 rounded",children:"Ctrl+S"})]}),(0,l.jsx)(eI,{icon:(0,l.jsx)(eZ.Z,{className:"text-purple-600",size:16}),title:"Configuration",description:"Cursor MCP configuration",serverName:"Zapier Gmail",accessGroups:["dev"],children:(0,l.jsx)(j,{code:'{\n "mcpServers": {\n "Zapier_MCP": {\n "server_url": "'.concat(r,'/mcp",\n "headers": {\n "x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY",\n "x-mcp-servers": ["Zapier_MCP,dev"]\n }\n }\n }\n}'),copyKey:"cursor-config",className:"text-xs"})})]})]})]})]}),{})}),(0,l.jsx)(u.Z,{className:"mt-6",children:(0,l.jsx)(()=>(0,l.jsxs)(e_.Z,{direction:"vertical",size:"large",className:"w-full",children:[(0,l.jsxs)("div",{className:"bg-gradient-to-r from-green-50 to-teal-50 p-6 rounded-lg border border-green-100",children:[(0,l.jsxs)("div",{className:"flex items-center gap-3 mb-3",children:[(0,l.jsx)(eA.Z,{className:"text-green-600",size:24}),(0,l.jsx)(eL,{level:4,className:"mb-0 text-green-900",children:"Streamable HTTP Transport"})]}),(0,l.jsx)(eM,{className:"text-green-700",children:"Connect to LiteLLM MCP using HTTP transport. Compatible with any MCP client that supports HTTP streaming."})]}),(0,l.jsx)(eI,{icon:(0,l.jsx)(eA.Z,{className:"text-green-600",size:16}),title:"Universal MCP Connection",description:"Use this URL with any MCP client that supports HTTP transport",children:(0,l.jsxs)(e_.Z,{direction:"vertical",size:"middle",className:"w-full",children:[(0,l.jsx)("div",{children:(0,l.jsx)(eM,{children:"Each MCP client supports different transports. Refer to your client documentation to determine the appropriate transport method."})}),(0,l.jsx)(j,{title:"Server URL",code:"".concat(r,"/mcp"),copyKey:"http-server-url"}),(0,l.jsx)(j,{title:"Headers Configuration",code:JSON.stringify({"x-litellm-api-key":"Bearer YOUR_LITELLM_API_KEY"},null,2),copyKey:"http-headers"}),(0,l.jsx)("div",{className:"mt-4",children:(0,l.jsx)(F.ZP,{type:"link",className:"p-0 h-auto text-blue-600 hover:text-blue-700",href:"https://modelcontextprotocol.io/docs/concepts/transports",icon:(0,l.jsx)(ek.Z,{size:14}),children:"Learn more about MCP transports"})})]})})]}),{})})]})]})]})})},ez=r(67187);let{Option:eq}=n.default,eO=e=>{let{isModalOpen:s,title:r,confirmDelete:t,cancelDelete:a}=e;return s?(0,l.jsx)(i.Z,{open:s,onOk:t,okType:"danger",onCancel:a,children:(0,l.jsxs)(j.Z,{numItems:1,className:"gap-2 w-full",children:[(0,l.jsx)(v.Z,{children:r}),(0,l.jsx)(p.Z,{numColSpan:1,children:(0,l.jsx)("p",{children:"Are you sure you want to delete this MCP Server?"})})]})}):null};var eR=e=>{let{accessToken:s,userRole:r,userID:i}=e,{data:p,isLoading:j,refetch:b,dataUpdatedAt:y}=(0,a.a)({queryKey:["mcpServers"],queryFn:()=>{if(!s)throw Error("Access Token required");return(0,P.fetchMCPServers)(s)},enabled:!!s});t.useEffect(()=>{p&&(console.log("MCP Servers fetched:",p),p.forEach(e=>{console.log("Server: ".concat(e.server_name||e.server_id)),console.log(" allowed_tools:",e.allowed_tools)}))},[p]);let[N,_]=(0,t.useState)(null),[Z,w]=(0,t.useState)(!1),[C,S]=(0,t.useState)(null),[L,M]=(0,t.useState)(!1),[T,I]=(0,t.useState)("all"),[E,z]=(0,t.useState)("all"),[q,O]=(0,t.useState)([]),[R,U]=(0,t.useState)(!1),F="Internal User"===r,B=t.useMemo(()=>{if(!p)return[];let e=new Set,s=[];return p.forEach(r=>{r.teams&&r.teams.forEach(r=>{let l=r.team_id;e.has(l)||(e.add(l),s.push(r))})}),s},[p]),K=t.useMemo(()=>p?Array.from(new Set(p.flatMap(e=>e.mcp_access_groups).filter(e=>null!=e))):[],[p]),V=e=>{I(e),H(e,E)},D=e=>{z(e),H(T,e)},H=(e,s)=>{if(!p)return O([]);let r=p;if("personal"===e){O([]);return}"all"!==e&&(r=r.filter(s=>{var r;return null===(r=s.teams)||void 0===r?void 0:r.some(s=>s.team_id===e)})),"all"!==s&&(r=r.filter(e=>{var r;return null===(r=e.mcp_access_groups)||void 0===r?void 0:r.some(e=>"string"==typeof e?e===s:e&&e.name===s)})),O(r)};(0,t.useEffect)(()=>{H(T,E)},[y]);let G=t.useMemo(()=>k(null!=r?r:"",e=>{S(e),M(!1)},e=>{S(e),M(!0)},Y),[r]);function Y(e){_(e),w(!0)}let J=async()=>{if(null!=N&&null!=s){try{await (0,P.deleteMCPServer)(s,N),el.Z.success("Deleted MCP Server successfully"),b()}catch(e){console.error("Error deleting the mcp server:",e)}w(!1),_(null)}};return s&&r&&i?(0,l.jsxs)("div",{className:"w-full h-full p-6",children:[(0,l.jsx)(eO,{isModalOpen:Z,title:"Delete MCP Server",confirmDelete:J,cancelDelete:()=>{w(!1),_(null)}}),(0,l.jsx)(ef,{userRole:r,accessToken:s,onCreateSuccess:e=>{O(s=>[...s,e]),U(!1)},isModalVisible:R,setModalVisible:U,availableAccessGroups:K}),(0,l.jsx)(v.Z,{children:"MCP Servers"}),(0,l.jsx)(g.Z,{className:"text-tremor-content mt-2",children:"Configure and manage your MCP servers"}),(0,A.tY)(r)&&(0,l.jsx)(c.Z,{className:"mt-4 mb-4",onClick:()=>U(!0),children:"+ Add New MCP Server"}),(0,l.jsxs)(m.Z,{className:"w-full h-full",children:[(0,l.jsx)(x.Z,{className:"flex justify-between mt-2 w-full items-center",children:(0,l.jsxs)("div",{className:"flex",children:[(0,l.jsx)(d.Z,{children:"All Servers"}),(0,l.jsx)(d.Z,{children:"Connect"})]})}),(0,l.jsxs)(h.Z,{children:[(0,l.jsx)(u.Z,{children:(0,l.jsx)(()=>C?(0,l.jsx)(ed,{mcpServer:q.find(e=>e.server_id===C)||{server_id:"",server_name:"",alias:"",url:"",transport:"",auth_type:"",created_at:"",created_by:"",updated_at:"",updated_by:""},onBack:()=>{M(!1),S(null),b()},isProxyAdmin:(0,A.tY)(r),isEditing:L,accessToken:s,userID:i,userRole:r,availableAccessGroups:K}):(0,l.jsxs)("div",{className:"w-full h-full",children:[(0,l.jsx)("div",{className:"w-full px-6",children:(0,l.jsx)("div",{className:"flex flex-col space-y-4",children:(0,l.jsx)("div",{className:"flex items-center justify-between bg-gray-50 rounded-lg p-4 border-2 border-gray-200",children:(0,l.jsxs)("div",{className:"flex items-center gap-4",children:[(0,l.jsx)(g.Z,{className:"text-lg font-semibold text-gray-900",children:"Current Team:"}),(0,l.jsxs)(n.default,{value:T,onChange:V,style:{width:300},children:[(0,l.jsx)(eq,{value:"all",children:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-blue-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:F?"All Available Servers":"All Servers"})]})}),(0,l.jsx)(eq,{value:"personal",children:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-green-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:"Personal"})]})}),B.map(e=>(0,l.jsx)(eq,{value:e.team_id,children:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-green-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:e.team_alias||e.team_id})]})},e.team_id))]}),(0,l.jsxs)(g.Z,{className:"text-lg font-semibold text-gray-900 ml-6",children:["Access Group:",(0,l.jsx)(o.Z,{title:"An MCP Access Group is a set of users or teams that have permission to access specific MCP servers. Use access groups to control and organize who can connect to which servers.",children:(0,l.jsx)(ez.Z,{style:{marginLeft:4,color:"#888"}})})]}),(0,l.jsxs)(n.default,{value:E,onChange:D,style:{width:300},children:[(0,l.jsx)(eq,{value:"all",children:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-blue-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:"All Access Groups"})]})}),K.map(e=>(0,l.jsx)(eq,{value:e,children:(0,l.jsxs)("div",{className:"flex items-center gap-2",children:[(0,l.jsx)("div",{className:"w-2 h-2 bg-green-500 rounded-full"}),(0,l.jsx)("span",{className:"font-medium",children:e})]})},e))]})]})})})}),(0,l.jsx)("div",{className:"w-full px-6 mt-6",children:(0,l.jsx)(f.w,{data:q,columns:G,renderSubComponent:()=>(0,l.jsx)("div",{}),getRowCanExpand:()=>!1,isLoading:j,noDataMessage:"No MCP servers configured"})})]}),{})}),(0,l.jsx)(u.Z,{children:(0,l.jsx)(eE,{})})]})]})]}):(console.log("Missing required authentication parameters",{accessToken:s,userRole:r,userID:i}),(0,l.jsx)("div",{className:"p-6 text-center text-gray-500",children:"Missing required authentication parameters."}))},eU=r(21770);function eF(e){let{tool:s,needsAuth:r,authValue:a,onSubmit:n,isLoading:i,result:c,error:d,onClose:m}=e,[x]=U.Z.useForm(),[u,h]=t.useState("formatted"),[p,j]=t.useState(null),[g,v]=t.useState(null),f=t.useMemo(()=>"string"==typeof s.inputSchema?{type:"object",properties:{input:{type:"string",description:"Input for this tool"}},required:["input"]}:s.inputSchema,[s.inputSchema]),b=t.useMemo(()=>f.properties&&f.properties.params&&"object"===f.properties.params.type&&f.properties.params.properties?{type:"object",properties:f.properties.params.properties,required:f.properties.params.required||[]}:f,[f]);t.useEffect(()=>{p&&(c||d)&&v(Date.now()-p)},[c,d,p]);let y=async e=>{try{if(navigator.clipboard&&window.isSecureContext)return await navigator.clipboard.writeText(e),!0;{let s=document.createElement("textarea");s.value=e,s.style.position="fixed",s.style.opacity="0",document.body.appendChild(s),s.focus(),s.select();let r=document.execCommand("copy");if(document.body.removeChild(s),!r)throw Error("execCommand failed");return!0}}catch(e){return console.error("Copy failed:",e),!1}},N=async()=>{await y(JSON.stringify(c,null,2))?el.Z.success("Result copied to clipboard"):el.Z.fromBackend("Failed to copy result")},_=async()=>{await y(s.name)?el.Z.success("Tool name copied to clipboard"):el.Z.fromBackend("Failed to copy tool name")};return(0,l.jsxs)("div",{className:"space-y-4 h-full",children:[(0,l.jsxs)("div",{className:"flex items-center justify-between pb-3 border-b border-gray-200",children:[(0,l.jsxs)("div",{className:"flex items-center space-x-3",children:[s.mcp_info.logo_url&&(0,l.jsx)("img",{src:s.mcp_info.logo_url,alt:"".concat(s.mcp_info.server_name," logo"),className:"w-6 h-6 object-contain"}),(0,l.jsxs)("div",{className:"flex-1 min-w-0",children:[(0,l.jsxs)("div",{className:"flex items-center space-x-2 mb-1",children:[(0,l.jsx)("h2",{className:"text-lg font-semibold text-gray-900",children:"Test Tool:"}),(0,l.jsxs)("div",{className:"group inline-flex items-center space-x-1 bg-slate-50 hover:bg-slate-100 px-3 py-1 rounded-md cursor-pointer transition-colors border border-slate-200",onClick:_,title:"Click to copy tool name",children:[(0,l.jsx)("span",{className:"font-mono text-slate-700 font-medium text-sm",children:s.name}),(0,l.jsx)("svg",{className:"w-3 h-3 text-slate-400 group-hover:text-slate-600 transition-colors",fill:"none",stroke:"currentColor",viewBox:"0 0 24 24",children:(0,l.jsx)("path",{strokeLinecap:"round",strokeLinejoin:"round",strokeWidth:2,d:"M8 16H6a2 2 0 01-2-2V6a2 2 0 012-2h8a2 2 0 012 2v2m-6 12h8a2 2 0 002-2v-8a2 2 0 00-2-2h-8a2 2 0 00-2 2v8a2 2 0 002 2z"})})]})]}),(0,l.jsx)("p",{className:"text-xs text-gray-600",children:s.description}),(0,l.jsxs)("p",{className:"text-xs text-gray-500",children:["Provider: ",s.mcp_info.server_name]})]})]}),(0,l.jsx)(em.z,{onClick:m,variant:"light",size:"sm",className:"text-gray-500 hover:text-gray-700",children:(0,l.jsx)("svg",{className:"w-4 h-4",fill:"none",stroke:"currentColor",viewBox:"0 0 24 24",children:(0,l.jsx)("path",{strokeLinecap:"round",strokeLinejoin:"round",strokeWidth:2,d:"M6 18L18 6M6 6l12 12"})})})]}),(0,l.jsxs)("div",{className:"grid grid-cols-2 gap-4 h-full",children:[(0,l.jsxs)("div",{className:"bg-white border border-gray-200 rounded-lg",children:[(0,l.jsx)("div",{className:"border-b border-gray-100 px-4 py-2",children:(0,l.jsxs)("div",{className:"flex items-center justify-between",children:[(0,l.jsx)("h3",{className:"text-sm font-semibold text-gray-900",children:"Input Parameters"}),(0,l.jsx)(o.Z,{title:"Configure the input parameters for this tool call",children:(0,l.jsx)(G.Z,{className:"text-gray-400 hover:text-gray-600"})})]})}),(0,l.jsx)("div",{className:"p-4",children:(0,l.jsxs)(U.Z,{form:x,onFinish:e=>{j(Date.now()),v(null);let s={};Object.entries(e).forEach(e=>{var r;let[l,t]=e,a=null===(r=b.properties)||void 0===r?void 0:r[l];if(a&&null!=t&&""!==t)switch(a.type){case"boolean":s[l]="true"===t||!0===t;break;case"number":s[l]=Number(t);break;case"string":s[l]=String(t);break;default:s[l]=t}else null!=t&&""!==t&&(s[l]=t)}),n(f.properties&&f.properties.params&&"object"===f.properties.params.type&&f.properties.params.properties?{params:s}:s)},layout:"vertical",className:"space-y-3",children:["string"==typeof s.inputSchema?(0,l.jsx)("div",{className:"space-y-3",children:(0,l.jsx)(U.Z.Item,{label:(0,l.jsxs)("span",{className:"text-sm font-medium text-gray-700",children:["Input ",(0,l.jsx)("span",{className:"text-red-500",children:"*"})]}),name:"input",rules:[{required:!0,message:"Please enter input for this tool"}],className:"mb-3",children:(0,l.jsx)(em.o,{placeholder:"Enter input for this tool",className:"rounded-lg border-gray-300 focus:border-blue-500 focus:ring-blue-500"})})}):void 0===b.properties?(0,l.jsx)("div",{className:"text-center py-6 bg-gray-50 rounded-lg border border-gray-200",children:(0,l.jsxs)("div",{className:"max-w-sm mx-auto",children:[(0,l.jsx)("h4",{className:"text-sm font-medium text-gray-900 mb-1",children:"No Parameters Required"}),(0,l.jsx)("p",{className:"text-xs text-gray-500",children:"This tool can be called without any input parameters."})]})}):(0,l.jsx)("div",{className:"space-y-3",children:Object.entries(b.properties).map(e=>{var s,r,t,a,n;let[i,c]=e;return(0,l.jsxs)(U.Z.Item,{label:(0,l.jsxs)("span",{className:"text-sm font-medium text-gray-700 flex items-center",children:[i," ",(null===(s=b.required)||void 0===s?void 0:s.includes(i))&&(0,l.jsx)("span",{className:"text-red-500",children:"*"}),c.description&&(0,l.jsx)(o.Z,{title:c.description,children:(0,l.jsx)(G.Z,{className:"ml-2 text-gray-400 hover:text-gray-600"})})]}),name:i,rules:[{required:null===(r=b.required)||void 0===r?void 0:r.includes(i),message:"Please enter ".concat(i)}],className:"mb-3",children:["string"===c.type&&c.enum&&(0,l.jsxs)("select",{className:"w-full px-3 py-2 border border-gray-300 rounded-lg shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500 text-sm transition-colors",defaultValue:c.default,children:[!(null===(t=b.required)||void 0===t?void 0:t.includes(i))&&(0,l.jsxs)("option",{value:"",children:["Select ",i]}),c.enum.map(e=>(0,l.jsx)("option",{value:e,children:e},e))]}),"string"===c.type&&!c.enum&&(0,l.jsx)(em.o,{placeholder:c.description||"Enter ".concat(i),className:"rounded-lg border-gray-300 focus:border-blue-500 focus:ring-blue-500"}),"number"===c.type&&(0,l.jsx)("input",{type:"number",placeholder:c.description||"Enter ".concat(i),className:"w-full px-3 py-2 border border-gray-300 rounded-lg shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500 text-sm transition-colors"}),"boolean"===c.type&&(0,l.jsxs)("select",{className:"w-full px-3 py-2 border border-gray-300 rounded-lg shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500 text-sm transition-colors",defaultValue:(null===(a=c.default)||void 0===a?void 0:a.toString())||"",children:[!(null===(n=b.required)||void 0===n?void 0:n.includes(i))&&(0,l.jsxs)("option",{value:"",children:["Select ",i]}),(0,l.jsx)("option",{value:"true",children:"True"}),(0,l.jsx)("option",{value:"false",children:"False"})]})]},i)})}),(0,l.jsx)("div",{className:"pt-3 border-t border-gray-100",children:(0,l.jsx)(em.z,{onClick:()=>x.submit(),disabled:i,variant:"primary",className:"w-full",loading:i,children:i?"Calling Tool...":c||d?"Call Again":"Call Tool"})})]})})]}),(0,l.jsxs)("div",{className:"bg-white border border-gray-200 rounded-lg",children:[(0,l.jsx)("div",{className:"border-b border-gray-100 px-4 py-2",children:(0,l.jsx)("h3",{className:"text-sm font-semibold text-gray-900",children:"Tool Result"})}),(0,l.jsx)("div",{className:"p-4",children:c||d||i?(0,l.jsxs)("div",{className:"space-y-3",children:[c&&!i&&!d&&(0,l.jsx)("div",{className:"p-2 bg-green-50 border border-green-200 rounded-lg",children:(0,l.jsxs)("div",{className:"flex items-center justify-between",children:[(0,l.jsxs)("div",{className:"flex items-center space-x-2",children:[(0,l.jsx)("svg",{className:"h-4 w-4 text-green-500",fill:"none",viewBox:"0 0 24 24",stroke:"currentColor",children:(0,l.jsx)("path",{strokeLinecap:"round",strokeLinejoin:"round",strokeWidth:2,d:"M9 12l2 2 4-4m6 2a9 9 0 11-18 0 9 9 0 0118 0z"})}),(0,l.jsx)("h4",{className:"text-xs font-medium text-green-900",children:"Tool executed successfully"}),null!==g&&(0,l.jsxs)("span",{className:"text-xs text-green-600 ml-1",children:["• ",(g/1e3).toFixed(2),"s"]})]}),(0,l.jsxs)("div",{className:"flex items-center space-x-1",children:[(0,l.jsxs)("div",{className:"flex bg-white rounded border border-green-300 p-0.5",children:[(0,l.jsx)("button",{onClick:()=>h("formatted"),className:"px-2 py-1 text-xs font-medium rounded transition-colors ".concat("formatted"===u?"bg-green-100 text-green-800":"text-green-600 hover:text-green-800"),children:"Formatted"}),(0,l.jsx)("button",{onClick:()=>h("json"),className:"px-2 py-1 text-xs font-medium rounded transition-colors ".concat("json"===u?"bg-green-100 text-green-800":"text-green-600 hover:text-green-800"),children:"JSON"})]}),(0,l.jsx)("button",{onClick:N,className:"p-1 hover:bg-green-100 rounded text-green-700",title:"Copy response",children:(0,l.jsxs)("svg",{xmlns:"http://www.w3.org/2000/svg",width:"14",height:"14",viewBox:"0 0 24 24",fill:"none",stroke:"currentColor",strokeWidth:"2",strokeLinecap:"round",strokeLinejoin:"round",children:[(0,l.jsx)("rect",{x:"9",y:"9",width:"13",height:"13",rx:"2",ry:"2"}),(0,l.jsx)("path",{d:"M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"})]})})]})]})}),(0,l.jsxs)("div",{className:"max-h-96 overflow-y-auto",children:[i&&(0,l.jsxs)("div",{className:"flex flex-col justify-center items-center h-48 text-gray-500",children:[(0,l.jsxs)("div",{className:"relative",children:[(0,l.jsx)("div",{className:"animate-spin rounded-full h-8 w-8 border-2 border-gray-200"}),(0,l.jsx)("div",{className:"animate-spin rounded-full h-8 w-8 border-2 border-blue-600 border-t-transparent absolute top-0"})]}),(0,l.jsx)("p",{className:"text-sm font-medium mt-3",children:"Calling tool..."}),(0,l.jsx)("p",{className:"text-xs text-gray-400 mt-1",children:"Please wait while we process your request"})]}),d&&(0,l.jsx)("div",{className:"bg-red-50 border border-red-200 rounded-lg p-3",children:(0,l.jsxs)("div",{className:"flex items-start space-x-2",children:[(0,l.jsx)("div",{className:"flex-shrink-0",children:(0,l.jsx)("svg",{className:"h-4 w-4 text-red-400",fill:"none",viewBox:"0 0 24 24",stroke:"currentColor",children:(0,l.jsx)("path",{strokeLinecap:"round",strokeLinejoin:"round",strokeWidth:2,d:"M12 8v4m0 4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z"})})}),(0,l.jsxs)("div",{className:"flex-1",children:[(0,l.jsxs)("div",{className:"flex items-center space-x-2 mb-1",children:[(0,l.jsx)("h4",{className:"text-xs font-medium text-red-900",children:"Tool Call Failed"}),null!==g&&(0,l.jsxs)("span",{className:"text-xs text-red-600",children:["• ",(g/1e3).toFixed(2),"s"]})]}),(0,l.jsx)("div",{className:"bg-white border border-red-200 rounded p-2 max-h-48 overflow-y-auto",children:(0,l.jsx)("pre",{className:"text-xs whitespace-pre-wrap text-red-700 font-mono",children:d.message})})]})]})}),c&&!i&&!d&&(0,l.jsx)("div",{className:"space-y-3",children:"formatted"===u?c.map((e,s)=>(0,l.jsxs)("div",{className:"border border-gray-200 rounded-lg overflow-hidden",children:["text"===e.type&&(0,l.jsxs)("div",{children:[(0,l.jsx)("div",{className:"bg-gray-50 px-3 py-1 border-b border-gray-200",children:(0,l.jsx)("span",{className:"text-xs font-medium text-gray-700 uppercase tracking-wide",children:"Text Response"})}),(0,l.jsx)("div",{className:"p-3",children:(0,l.jsx)("div",{className:"bg-white rounded border border-gray-200 max-h-64 overflow-y-auto",children:(0,l.jsx)("div",{className:"p-3 space-y-2",children:e.text.split("\n\n").map((e,s)=>{if(""===e.trim())return null;if(e.startsWith("##")){let r=e.replace(/^#+\s/,"");return(0,l.jsx)("div",{className:"border-b border-gray-200 pb-1 mb-2",children:(0,l.jsx)("h3",{className:"text-sm font-semibold text-gray-900",children:r})},s)}let r=/(https?:\/\/[^\s\)]+)/g;if(r.test(e)){let t=e.split(r);return(0,l.jsx)("div",{className:"bg-blue-50 border border-blue-200 rounded p-2",children:(0,l.jsx)("div",{className:"text-xs text-gray-700 leading-relaxed whitespace-pre-wrap",children:t.map((e,s)=>r.test(e)?(0,l.jsx)("a",{href:e,target:"_blank",rel:"noopener noreferrer",className:"text-blue-600 hover:text-blue-800 underline break-all",children:e},s):e)})},s)}return e.includes("Score:")?(0,l.jsx)("div",{className:"bg-green-50 border-l-4 border-green-400 p-2 rounded-r",children:(0,l.jsx)("p",{className:"text-xs text-green-800 font-medium whitespace-pre-wrap",children:e})},s):(0,l.jsx)("div",{className:"bg-gray-50 rounded p-2 border border-gray-200",children:(0,l.jsx)("div",{className:"text-xs text-gray-700 leading-relaxed whitespace-pre-wrap font-mono",children:e})},s)}).filter(Boolean)})})})]}),"image"===e.type&&e.url&&(0,l.jsxs)("div",{children:[(0,l.jsx)("div",{className:"bg-gray-50 px-3 py-1 border-b border-gray-200",children:(0,l.jsx)("span",{className:"text-xs font-medium text-gray-700 uppercase tracking-wide",children:"Image Response"})}),(0,l.jsx)("div",{className:"p-3",children:(0,l.jsx)("div",{className:"bg-gray-50 rounded p-3 border border-gray-200",children:(0,l.jsx)("img",{src:e.url,alt:"Tool result",className:"max-w-full h-auto rounded shadow-sm"})})})]}),"embedded_resource"===e.type&&(0,l.jsxs)("div",{children:[(0,l.jsx)("div",{className:"bg-gray-50 px-3 py-1 border-b border-gray-200",children:(0,l.jsx)("span",{className:"text-xs font-medium text-gray-700 uppercase tracking-wide",children:"Embedded Resource"})}),(0,l.jsx)("div",{className:"p-3",children:(0,l.jsxs)("div",{className:"flex items-center space-x-2 p-3 bg-blue-50 border border-blue-200 rounded",children:[(0,l.jsx)("div",{className:"flex-shrink-0",children:(0,l.jsx)("svg",{className:"h-5 w-5 text-blue-500",fill:"none",viewBox:"0 0 24 24",stroke:"currentColor",children:(0,l.jsx)("path",{strokeLinecap:"round",strokeLinejoin:"round",strokeWidth:2,d:"M9 12h6m-6 4h6m2 5H7a2 2 0 01-2-2V5a2 2 0 012-2h5.586a1 1 0 01.707.293l5.414 5.414a1 1 0 01.293.707V19a2 2 0 01-2 2z"})})}),(0,l.jsxs)("div",{className:"flex-1",children:[(0,l.jsxs)("p",{className:"text-xs font-medium text-blue-900",children:["Resource Type: ",e.resource_type]}),e.url&&(0,l.jsxs)("a",{href:e.url,target:"_blank",rel:"noopener noreferrer",className:"inline-flex items-center text-xs text-blue-600 hover:text-blue-800 hover:underline mt-1 transition-colors",children:["View Resource",(0,l.jsxs)("svg",{className:"ml-1 h-3 w-3",fill:"currentColor",viewBox:"0 0 20 20",children:[(0,l.jsx)("path",{d:"M11 3a1 1 0 100 2h2.586l-6.293 6.293a1 1 0 101.414 1.414L15 6.414V9a1 1 0 102 0V4a1 1 0 00-1-1h-5z"}),(0,l.jsx)("path",{d:"M5 5a2 2 0 00-2 2v8a2 2 0 002 2h8a2 2 0 002-2v-3a1 1 0 10-2 0v3H5V7h3a1 1 0 000-2H5z"})]})]})]})]})})]})]},s)):(0,l.jsx)("div",{className:"bg-white rounded border border-gray-200",children:(0,l.jsx)("div",{className:"p-3 overflow-auto max-h-80 bg-gray-50",children:(0,l.jsx)("pre",{className:"text-xs font-mono whitespace-pre-wrap break-all text-gray-800",children:JSON.stringify(c,null,2)})})})})]})]}):(0,l.jsx)("div",{className:"flex flex-col justify-center items-center h-48 text-gray-500",children:(0,l.jsxs)("div",{className:"text-center max-w-sm",children:[(0,l.jsx)("div",{className:"mb-3",children:(0,l.jsx)("svg",{className:"mx-auto h-12 w-12 text-gray-300",fill:"none",viewBox:"0 0 24 24",stroke:"currentColor",children:(0,l.jsx)("path",{strokeLinecap:"round",strokeLinejoin:"round",strokeWidth:1,d:"M13 10V3L4 14h7v7l9-11h-7z"})})}),(0,l.jsx)("h4",{className:"text-sm font-medium text-gray-900 mb-1",children:"Ready to Call Tool"}),(0,l.jsx)("p",{className:"text-xs text-gray-500 leading-relaxed",children:'Configure the input parameters and click "Call Tool" to see the results here.'})]})})})]})]})]})}var eB=r(29488),eK=r(36724),eV=r(57400),eD=r(69993);let eH=e=>{let s,{visible:r,onOk:t,onCancel:a,authType:n}=e,[o]=U.Z.useForm();if(n===E.API_KEY||n===E.BEARER_TOKEN){let e=n===E.API_KEY?"API Key":"Bearer Token";s=(0,l.jsx)(U.Z.Item,{name:"authValue",label:e,rules:[{required:!0,message:"Please input your ".concat(e)}],children:(0,l.jsx)(ej.default.Password,{})})}else n===E.BASIC&&(s=(0,l.jsxs)(l.Fragment,{children:[(0,l.jsx)(U.Z.Item,{name:"username",label:"Username",rules:[{required:!0,message:"Please input your username"}],children:(0,l.jsx)(ej.default,{})}),(0,l.jsx)(U.Z.Item,{name:"password",label:"Password",rules:[{required:!0,message:"Please input your password"}],children:(0,l.jsx)(ej.default.Password,{})})]}));return(0,l.jsx)(i.Z,{open:r,title:"Authentication",onOk:()=>{o.validateFields().then(e=>{n===E.BASIC?t("".concat(e.username.trim(),":").concat(e.password.trim())):t(e.authValue.trim())})},onCancel:a,destroyOnClose:!0,children:(0,l.jsx)(U.Z,{form:o,layout:"vertical",children:s})})},eG=e=>{let{authType:s,onAuthSubmit:r,onClearAuth:a,hasAuth:n}=e,[i,o]=(0,t.useState)(!1);return(0,l.jsxs)("div",{className:"space-y-3",children:[(0,l.jsxs)("div",{className:"flex items-center justify-between",children:[(0,l.jsxs)(eK.xv,{className:"text-sm font-medium text-gray-700",children:["Authentication ",n?"✓":""]}),(0,l.jsxs)("div",{className:"flex gap-2",children:[n&&(0,l.jsx)(eK.zx,{onClick:()=>{a()},size:"sm",variant:"secondary",className:"text-xs text-red-600 hover:text-red-700",children:"Clear"}),(0,l.jsx)(eK.zx,{onClick:()=>o(!0),size:"sm",variant:"secondary",className:"text-xs",children:n?"Update":"Add Auth"})]})]}),(0,l.jsx)(eK.xv,{className:"text-xs text-gray-500",children:n?"Authentication configured and saved locally":"Some tools may require authentication"}),(0,l.jsx)(eH,{visible:i,onOk:e=>{r(e),o(!1)},onCancel:()=>o(!1),authType:s})]})};var eY=e=>{let{serverId:s,accessToken:r,auth_type:n,userRole:i,userID:o,serverAlias:c}=e,[d,m]=(0,t.useState)(""),[x,u]=(0,t.useState)(null),[h,p]=(0,t.useState)(null),[j,g]=(0,t.useState)(null);(0,t.useEffect)(()=>{if(R(n)){let e=(0,eB.Ui)(s,c||void 0);e&&m(e)}},[s,c,n]);let v=e=>{m(e),e&&R(n)&&((0,eB.Hc)(s,e,n||"none",c||void 0),el.Z.success("Authentication token saved locally"))},f=()=>{m(""),(0,eB.e4)(s),el.Z.info("Authentication token cleared")},{data:b,isLoading:y,error:N}=(0,a.a)({queryKey:["mcpTools",s,d,c],queryFn:()=>{if(!r)throw Error("Access Token required");return(0,P.listMCPTools)(r,s,d,c||void 0)},enabled:!!r,staleTime:3e4}),{mutate:_,isPending:Z}=(0,eU.D)({mutationFn:async e=>{if(!r)throw Error("Access Token required");try{return await (0,P.callMCPTool)(r,e.tool.name,e.arguments,e.authValue,c||void 0)}catch(e){throw e}},onSuccess:e=>{p(e),g(null)},onError:e=>{g(e),p(null)}}),w=(null==b?void 0:b.tools)||[],C=""!==d;return(0,l.jsx)("div",{className:"w-full h-screen p-4 bg-white",children:(0,l.jsx)(eK.Zb,{className:"w-full rounded-xl shadow-md overflow-hidden",children:(0,l.jsxs)("div",{className:"flex h-auto w-full gap-4",children:[(0,l.jsxs)("div",{className:"w-1/4 p-4 bg-gray-50 flex flex-col",children:[(0,l.jsx)(eK.Dx,{className:"text-xl font-semibold mb-6 mt-2",children:"MCP Tools"}),(0,l.jsxs)("div",{className:"flex flex-col flex-1",children:[(0,l.jsxs)("div",{className:"flex flex-col flex-1 min-h-0",children:[(0,l.jsxs)(eK.xv,{className:"font-medium block mb-3 text-gray-700 flex items-center",children:[(0,l.jsx)(Y.Z,{className:"mr-2"})," Available Tools",w.length>0&&(0,l.jsx)("span",{className:"ml-2 bg-blue-100 text-blue-800 text-xs font-medium px-2 py-0.5 rounded-full",children:w.length})]}),y&&(0,l.jsxs)("div",{className:"flex flex-col items-center justify-center py-8 bg-white border border-gray-200 rounded-lg",children:[(0,l.jsxs)("div",{className:"relative mb-3",children:[(0,l.jsx)("div",{className:"animate-spin rounded-full h-6 w-6 border-2 border-gray-200"}),(0,l.jsx)("div",{className:"animate-spin rounded-full h-6 w-6 border-2 border-blue-600 border-t-transparent absolute top-0"})]}),(0,l.jsx)("p",{className:"text-xs font-medium text-gray-700",children:"Loading tools..."})]}),(null==b?void 0:b.error)&&!y&&!w.length&&(0,l.jsx)("div",{className:"p-3 text-xs text-red-800 rounded-lg bg-red-50 border border-red-200",children:(0,l.jsxs)("p",{className:"font-medium",children:["Error: ",b.message]})}),!y&&!(null==b?void 0:b.error)&&(!w||0===w.length)&&(0,l.jsxs)("div",{className:"p-4 text-center bg-white border border-gray-200 rounded-lg",children:[(0,l.jsx)("div",{className:"mx-auto w-8 h-8 bg-gray-200 rounded-full flex items-center justify-center mb-2",children:(0,l.jsx)("svg",{className:"w-4 h-4 text-gray-400",fill:"none",stroke:"currentColor",viewBox:"0 0 24 24",children:(0,l.jsx)("path",{strokeLinecap:"round",strokeLinejoin:"round",strokeWidth:2,d:"M19.428 15.428a2 2 0 00-1.022-.547l-2.387-.477a6 6 0 00-3.86.517l-.318.158a6 6 0 01-3.86.517L6.05 15.21a2 2 0 00-1.806.547M8 4h8l-1 1v5.172a2 2 0 00.586 1.414l5 5c1.26 1.26.367 3.414-1.415 3.414H4.828c-1.782 0-2.674-2.154-1.414-3.414l5-5A2 2 0 009 8.172V5L8 4z"})})}),(0,l.jsx)("p",{className:"text-xs font-medium text-gray-700 mb-1",children:"No tools available"}),(0,l.jsx)("p",{className:"text-xs text-gray-500",children:"No tools found for this server"})]}),!y&&!(null==b?void 0:b.error)&&w.length>0&&(0,l.jsx)("div",{className:"space-y-2 flex-1 overflow-y-auto min-h-0 mcp-tools-scrollable",style:{maxHeight:"400px",scrollbarWidth:"auto",scrollbarColor:"#cbd5e0 #f7fafc"},children:w.map(e=>(0,l.jsxs)("div",{className:"border rounded-lg p-3 cursor-pointer transition-all hover:shadow-sm ".concat((null==x?void 0:x.name)===e.name?"border-blue-500 bg-blue-50 ring-1 ring-blue-200":"border-gray-200 bg-white hover:border-gray-300"),onClick:()=>{u(e),p(null),g(null)},children:[(0,l.jsxs)("div",{className:"flex items-start space-x-2",children:[e.mcp_info.logo_url&&(0,l.jsx)("img",{src:e.mcp_info.logo_url,alt:"".concat(e.mcp_info.server_name," logo"),className:"w-4 h-4 object-contain flex-shrink-0 mt-0.5"}),(0,l.jsxs)("div",{className:"flex-1 min-w-0",children:[(0,l.jsx)("h4",{className:"font-mono text-xs font-medium text-gray-900 truncate",children:e.name}),(0,l.jsx)("p",{className:"text-xs text-gray-500 truncate",children:e.mcp_info.server_name}),(0,l.jsx)("p",{className:"text-xs text-gray-600 mt-1 line-clamp-2 leading-relaxed",children:e.description})]})]}),(null==x?void 0:x.name)===e.name&&(0,l.jsx)("div",{className:"mt-2 pt-2 border-t border-blue-200",children:(0,l.jsxs)("div",{className:"flex items-center text-xs font-medium text-blue-700",children:[(0,l.jsx)("svg",{className:"w-3 h-3 mr-1",fill:"currentColor",viewBox:"0 0 20 20",children:(0,l.jsx)("path",{fillRule:"evenodd",d:"M16.707 5.293a1 1 0 010 1.414l-8 8a1 1 0 01-1.414 0l-4-4a1 1 0 011.414-1.414L8 12.586l7.293-7.293a1 1 0 011.414 0z",clipRule:"evenodd"})}),"Selected"]})})]},e.name))})]}),R(n)&&(0,l.jsx)("div",{className:"pt-4 border-t border-gray-200 flex-shrink-0 mt-6",children:C?(0,l.jsxs)(l.Fragment,{children:[(0,l.jsxs)(eK.xv,{className:"font-medium block mb-3 text-gray-700 flex items-center",children:[(0,l.jsx)(eV.Z,{className:"mr-2"})," Authentication"]}),(0,l.jsx)(eG,{authType:n,onAuthSubmit:v,onClearAuth:f,hasAuth:C})]}):(0,l.jsxs)("div",{className:"p-4 bg-gradient-to-r from-orange-50 to-red-50 border border-orange-200 rounded-lg",children:[(0,l.jsxs)("div",{className:"flex items-center mb-3",children:[(0,l.jsx)(eV.Z,{className:"mr-2 text-orange-600 text-lg"}),(0,l.jsx)(eK.xv,{className:"font-semibold text-orange-800",children:"Authentication Required"})]}),(0,l.jsx)(eK.xv,{className:"text-sm text-orange-700 mb-4",children:"This MCP server requires authentication. You must add your credentials below to access the tools."}),(0,l.jsx)(eG,{authType:n,onAuthSubmit:v,onClearAuth:f,hasAuth:C})]})})]})]}),(0,l.jsxs)("div",{className:"w-3/4 flex flex-col bg-white",children:[(0,l.jsx)("div",{className:"p-4 border-b border-gray-200 flex justify-between items-center",children:(0,l.jsx)(eK.Dx,{className:"text-xl font-semibold mb-0",children:"Tool Testing Playground"})}),(0,l.jsx)("div",{className:"flex-1 overflow-auto p-4",children:x?(0,l.jsx)("div",{className:"h-full",children:(0,l.jsx)(eF,{tool:x,needsAuth:R(n),authValue:d,onSubmit:e=>{_({tool:x,arguments:e,authValue:d})},result:h,error:j,isLoading:Z,onClose:()=>u(null)})}):(0,l.jsxs)("div",{className:"h-full flex flex-col items-center justify-center text-gray-400",children:[(0,l.jsx)(eD.Z,{style:{fontSize:"48px",marginBottom:"16px"}}),(0,l.jsx)(eK.xv,{className:"text-lg font-medium text-gray-600 mb-2",children:"Select a Tool to Test"}),(0,l.jsx)(eK.xv,{className:"text-center text-gray-500 max-w-md",children:"Choose a tool from the left sidebar to start testing its functionality with custom inputs."})]})})]})]})})})}}}]);
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/13b76428-e1bf383848c17260.js b/litellm/proxy/_experimental/out/_next/static/chunks/13b76428-e1bf383848c17260.js
new file mode 100644
index 00000000000..76cea35732a
--- /dev/null
+++ b/litellm/proxy/_experimental/out/_next/static/chunks/13b76428-e1bf383848c17260.js
@@ -0,0 +1 @@
+(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[6990],{77398:function(e,t,n){var s;e=n.nmd(e),s=function(){"use strict";function t(){return V.apply(null,arguments)}function n(e){return e instanceof Array||"[object Array]"===Object.prototype.toString.call(e)}function s(e){return null!=e&&"[object Object]"===Object.prototype.toString.call(e)}function i(e,t){return Object.prototype.hasOwnProperty.call(e,t)}function r(e){var t;if(Object.getOwnPropertyNames)return 0===Object.getOwnPropertyNames(e).length;for(t in e)if(i(e,t))return!1;return!0}function a(e){return void 0===e}function o(e){return"number"==typeof e||"[object Number]"===Object.prototype.toString.call(e)}function u(e){return e instanceof Date||"[object Date]"===Object.prototype.toString.call(e)}function l(e,t){var n,s=[],i=e.length;for(n=0;n>>0;for(t=0;t0)for(n=0;n=0?n?"+":"":"-")+Math.pow(10,Math.max(0,t-s.length)).toString().substr(1)+s}t.suppressDeprecationWarnings=!1,t.deprecationHandler=null,A=Object.keys?Object.keys:function(e){var t,n=[];for(t in e)i(e,t)&&n.push(t);return n};var N=/(\[[^\[]*\])|(\\)?([Hh]mm(ss)?|Mo|MM?M?M?|Do|DDDo|DD?D?D?|ddd?d?|do?|w[o|w]?|W[o|W]?|Qo?|N{1,5}|YYYYYY|YYYYY|YYYY|YY|y{2,4}|yo?|gg(ggg?)?|GG(GGG?)?|e|E|a|A|hh?|HH?|kk?|mm?|ss?|S{1,9}|x|X|zz?|ZZ?|.)/g,W=/(\[[^\[]*\])|(\\)?(LTS|LT|LL?L?L?|l{1,4})/g,P={},R={};function C(e,t,n,s){var i=s;"string"==typeof s&&(i=function(){return this[s]()}),e&&(R[e]=i),t&&(R[t[0]]=function(){return x(i.apply(this,arguments),t[1],t[2])}),n&&(R[n]=function(){return this.localeData().ordinal(i.apply(this,arguments),e)})}function U(e,t){return e.isValid()?(P[t=H(t,e.localeData())]=P[t]||function(e){var t,n,s,i=e.match(N);for(n=0,s=i.length;n=0&&W.test(e);)e=e.replace(W,s),W.lastIndex=0,n-=1;return e}var F={D:"date",dates:"date",date:"date",d:"day",days:"day",day:"day",e:"weekday",weekdays:"weekday",weekday:"weekday",E:"isoWeekday",isoweekdays:"isoWeekday",isoweekday:"isoWeekday",DDD:"dayOfYear",dayofyears:"dayOfYear",dayofyear:"dayOfYear",h:"hour",hours:"hour",hour:"hour",ms:"millisecond",milliseconds:"millisecond",millisecond:"millisecond",m:"minute",minutes:"minute",minute:"minute",M:"month",months:"month",month:"month",Q:"quarter",quarters:"quarter",quarter:"quarter",s:"second",seconds:"second",second:"second",gg:"weekYear",weekyears:"weekYear",weekyear:"weekYear",GG:"isoWeekYear",isoweekyears:"isoWeekYear",isoweekyear:"isoWeekYear",w:"week",weeks:"week",week:"week",W:"isoWeek",isoweeks:"isoWeek",isoweek:"isoWeek",y:"year",years:"year",year:"year"};function L(e){return"string"==typeof e?F[e]||F[e.toLowerCase()]:void 0}function E(e){var t,n,s={};for(n in e)i(e,n)&&(t=L(n))&&(s[t]=e[n]);return s}var V,G,A,I,j={date:9,day:11,weekday:11,isoWeekday:11,dayOfYear:4,hour:13,millisecond:16,minute:14,month:8,quarter:7,second:15,weekYear:1,isoWeekYear:1,week:5,isoWeek:5,year:1},Z=/\d/,z=/\d\d/,$=/\d{3}/,q=/\d{4}/,B=/[+-]?\d{6}/,J=/\d\d?/,Q=/\d\d\d\d?/,X=/\d\d\d\d\d\d?/,K=/\d{1,3}/,ee=/\d{1,4}/,et=/[+-]?\d{1,6}/,en=/\d+/,es=/[+-]?\d+/,ei=/Z|[+-]\d\d:?\d\d/gi,er=/Z|[+-]\d\d(?::?\d\d)?/gi,ea=/[0-9]{0,256}['a-z\u00A0-\u05FF\u0700-\uD7FF\uF900-\uFDCF\uFDF0-\uFF07\uFF10-\uFFEF]{1,256}|[\u0600-\u06FF\/]{1,256}(\s*?[\u0600-\u06FF]{1,256}){1,2}/i,eo=/^[1-9]\d?/,eu=/^([1-9]\d|\d)/;function el(e,t,n){I[e]=O(t)?t:function(e,s){return e&&n?n:t}}function eh(e){return e.replace(/[-\/\\^$*+?.()|[\]{}]/g,"\\$&")}function ed(e){return e<0?Math.ceil(e)||0:Math.floor(e)}function ec(e){var t=+e,n=0;return 0!==t&&isFinite(t)&&(n=ed(t)),n}I={};var ef={};function em(e,t){var n,s,i=t;for("string"==typeof e&&(e=[e]),o(t)&&(i=function(e,n){n[t]=ec(e)}),s=e.length,n=0;n68?1900:2e3)};var ew=ep("FullYear",!0);function ep(e,n){return function(s){return null!=s?(ek(this,e,s),t.updateOffset(this,n),this):ev(this,e)}}function ev(e,t){if(!e.isValid())return NaN;var n=e._d,s=e._isUTC;switch(t){case"Milliseconds":return s?n.getUTCMilliseconds():n.getMilliseconds();case"Seconds":return s?n.getUTCSeconds():n.getSeconds();case"Minutes":return s?n.getUTCMinutes():n.getMinutes();case"Hours":return s?n.getUTCHours():n.getHours();case"Date":return s?n.getUTCDate():n.getDate();case"Day":return s?n.getUTCDay():n.getDay();case"Month":return s?n.getUTCMonth():n.getMonth();case"FullYear":return s?n.getUTCFullYear():n.getFullYear();default:return NaN}}function ek(e,t,n){var s,i,r,a;if(!(!e.isValid()||isNaN(n))){switch(s=e._d,i=e._isUTC,t){case"Milliseconds":return void(i?s.setUTCMilliseconds(n):s.setMilliseconds(n));case"Seconds":return void(i?s.setUTCSeconds(n):s.setSeconds(n));case"Minutes":return void(i?s.setUTCMinutes(n):s.setMinutes(n));case"Hours":return void(i?s.setUTCHours(n):s.setHours(n));case"Date":return void(i?s.setUTCDate(n):s.setDate(n));case"FullYear":break;default:return}r=e.month(),a=29!==(a=e.date())||1!==r||ey(n)?a:28,i?s.setUTCFullYear(n,r,a):s.setFullYear(n,r,a)}}function eM(e,t){if(isNaN(e)||isNaN(t))return NaN;var n=(t%12+12)%12;return e+=(t-n)/12,1===n?ey(e)?29:28:31-n%7%2}eA=Array.prototype.indexOf?Array.prototype.indexOf:function(e){var t;for(t=0;t=0?isFinite((o=new Date(e+400,t,n,s,i,r,a)).getFullYear())&&o.setFullYear(e):o=new Date(e,t,n,s,i,r,a),o}function eN(e){var t,n;return e<100&&e>=0?(n=Array.prototype.slice.call(arguments),n[0]=e+400,isFinite((t=new Date(Date.UTC.apply(null,n))).getUTCFullYear())&&t.setUTCFullYear(e)):t=new Date(Date.UTC.apply(null,arguments)),t}function eW(e,t,n){var s=7+t-n;return-((7+eN(e,0,s).getUTCDay()-t)%7)+s-1}function eP(e,t,n,s,i){var r,a,o=1+7*(t-1)+(7+n-s)%7+eW(e,s,i);return o<=0?a=eg(r=e-1)+o:o>eg(e)?(r=e+1,a=o-eg(e)):(r=e,a=o),{year:r,dayOfYear:a}}function eR(e,t,n){var s,i,r=eW(e.year(),t,n),a=Math.floor((e.dayOfYear()-r-1)/7)+1;return a<1?s=a+eC(i=e.year()-1,t,n):a>eC(e.year(),t,n)?(s=a-eC(e.year(),t,n),i=e.year()+1):(i=e.year(),s=a),{week:s,year:i}}function eC(e,t,n){var s=eW(e,t,n),i=eW(e+1,t,n);return(eg(e)-s+i)/7}function eU(e,t){return e.slice(t,7).concat(e.slice(0,t))}C("w",["ww",2],"wo","week"),C("W",["WW",2],"Wo","isoWeek"),el("w",J,eo),el("ww",J,z),el("W",J,eo),el("WW",J,z),e_(["w","ww","W","WW"],function(e,t,n,s){t[s.substr(0,1)]=ec(e)}),C("d",0,"do","day"),C("dd",0,0,function(e){return this.localeData().weekdaysMin(this,e)}),C("ddd",0,0,function(e){return this.localeData().weekdaysShort(this,e)}),C("dddd",0,0,function(e){return this.localeData().weekdays(this,e)}),C("e",0,0,"weekday"),C("E",0,0,"isoWeekday"),el("d",J),el("e",J),el("E",J),el("dd",function(e,t){return t.weekdaysMinRegex(e)}),el("ddd",function(e,t){return t.weekdaysShortRegex(e)}),el("dddd",function(e,t){return t.weekdaysRegex(e)}),e_(["dd","ddd","dddd"],function(e,t,n,s){var i=n._locale.weekdaysParse(e,s,n._strict);null!=i?t.d=i:c(n).invalidWeekday=e}),e_(["d","e","E"],function(e,t,n,s){t[s]=ec(e)});var eH="Sun_Mon_Tue_Wed_Thu_Fri_Sat".split("_");function eF(e,t,n){var s,i,r,a=e.toLocaleLowerCase();if(!this._weekdaysParse)for(s=0,this._weekdaysParse=[],this._shortWeekdaysParse=[],this._minWeekdaysParse=[];s<7;++s)r=d([2e3,1]).day(s),this._minWeekdaysParse[s]=this.weekdaysMin(r,"").toLocaleLowerCase(),this._shortWeekdaysParse[s]=this.weekdaysShort(r,"").toLocaleLowerCase(),this._weekdaysParse[s]=this.weekdays(r,"").toLocaleLowerCase();return n?"dddd"===t?-1!==(i=eA.call(this._weekdaysParse,a))?i:null:"ddd"===t?-1!==(i=eA.call(this._shortWeekdaysParse,a))?i:null:-1!==(i=eA.call(this._minWeekdaysParse,a))?i:null:"dddd"===t?-1!==(i=eA.call(this._weekdaysParse,a))||-1!==(i=eA.call(this._shortWeekdaysParse,a))?i:-1!==(i=eA.call(this._minWeekdaysParse,a))?i:null:"ddd"===t?-1!==(i=eA.call(this._shortWeekdaysParse,a))||-1!==(i=eA.call(this._weekdaysParse,a))?i:-1!==(i=eA.call(this._minWeekdaysParse,a))?i:null:-1!==(i=eA.call(this._minWeekdaysParse,a))||-1!==(i=eA.call(this._weekdaysParse,a))?i:-1!==(i=eA.call(this._shortWeekdaysParse,a))?i:null}function eL(){function e(e,t){return t.length-e.length}var t,n,s,i,r,a=[],o=[],u=[],l=[];for(t=0;t<7;t++)n=d([2e3,1]).day(t),s=eh(this.weekdaysMin(n,"")),i=eh(this.weekdaysShort(n,"")),r=eh(this.weekdays(n,"")),a.push(s),o.push(i),u.push(r),l.push(s),l.push(i),l.push(r);a.sort(e),o.sort(e),u.sort(e),l.sort(e),this._weekdaysRegex=RegExp("^("+l.join("|")+")","i"),this._weekdaysShortRegex=this._weekdaysRegex,this._weekdaysMinRegex=this._weekdaysRegex,this._weekdaysStrictRegex=RegExp("^("+u.join("|")+")","i"),this._weekdaysShortStrictRegex=RegExp("^("+o.join("|")+")","i"),this._weekdaysMinStrictRegex=RegExp("^("+a.join("|")+")","i")}function eE(){return this.hours()%12||12}function eV(e,t){C(e,0,0,function(){return this.localeData().meridiem(this.hours(),this.minutes(),t)})}function eG(e,t){return t._meridiemParse}C("H",["HH",2],0,"hour"),C("h",["hh",2],0,eE),C("k",["kk",2],0,function(){return this.hours()||24}),C("hmm",0,0,function(){return""+eE.apply(this)+x(this.minutes(),2)}),C("hmmss",0,0,function(){return""+eE.apply(this)+x(this.minutes(),2)+x(this.seconds(),2)}),C("Hmm",0,0,function(){return""+this.hours()+x(this.minutes(),2)}),C("Hmmss",0,0,function(){return""+this.hours()+x(this.minutes(),2)+x(this.seconds(),2)}),eV("a",!0),eV("A",!1),el("a",eG),el("A",eG),el("H",J,eu),el("h",J,eo),el("k",J,eo),el("HH",J,z),el("hh",J,z),el("kk",J,z),el("hmm",Q),el("hmmss",X),el("Hmm",Q),el("Hmmss",X),em(["H","HH"],3),em(["k","kk"],function(e,t,n){var s=ec(e);t[3]=24===s?0:s}),em(["a","A"],function(e,t,n){n._isPm=n._locale.isPM(e),n._meridiem=e}),em(["h","hh"],function(e,t,n){t[3]=ec(e),c(n).bigHour=!0}),em("hmm",function(e,t,n){var s=e.length-2;t[3]=ec(e.substr(0,s)),t[4]=ec(e.substr(s)),c(n).bigHour=!0}),em("hmmss",function(e,t,n){var s=e.length-4,i=e.length-2;t[3]=ec(e.substr(0,s)),t[4]=ec(e.substr(s,2)),t[5]=ec(e.substr(i)),c(n).bigHour=!0}),em("Hmm",function(e,t,n){var s=e.length-2;t[3]=ec(e.substr(0,s)),t[4]=ec(e.substr(s))}),em("Hmmss",function(e,t,n){var s=e.length-4,i=e.length-2;t[3]=ec(e.substr(0,s)),t[4]=ec(e.substr(s,2)),t[5]=ec(e.substr(i))});var eA,eI,ej=ep("Hours",!0),eZ={calendar:{sameDay:"[Today at] LT",nextDay:"[Tomorrow at] LT",nextWeek:"dddd [at] LT",lastDay:"[Yesterday at] LT",lastWeek:"[Last] dddd [at] LT",sameElse:"L"},longDateFormat:{LTS:"h:mm:ss A",LT:"h:mm A",L:"MM/DD/YYYY",LL:"MMMM D, YYYY",LLL:"MMMM D, YYYY h:mm A",LLLL:"dddd, MMMM D, YYYY h:mm A"},invalidDate:"Invalid date",ordinal:"%d",dayOfMonthOrdinalParse:/\d{1,2}/,relativeTime:{future:"in %s",past:"%s ago",s:"a few seconds",ss:"%d seconds",m:"a minute",mm:"%d minutes",h:"an hour",hh:"%d hours",d:"a day",dd:"%d days",w:"a week",ww:"%d weeks",M:"a month",MM:"%d months",y:"a year",yy:"%d years"},months:"January_February_March_April_May_June_July_August_September_October_November_December".split("_"),monthsShort:eD,week:{dow:0,doy:6},weekdays:"Sunday_Monday_Tuesday_Wednesday_Thursday_Friday_Saturday".split("_"),weekdaysMin:"Su_Mo_Tu_We_Th_Fr_Sa".split("_"),weekdaysShort:eH,meridiemParse:/[ap]\.?m?\.?/i},ez={},e$={};function eq(e){return e?e.toLowerCase().replace("_","-"):e}function eB(t){var n=null;if(void 0===ez[t]&&e&&e.exports&&t&&t.match("^[^/\\\\]*$"))try{n=eI._abbr,function(){var e=Error("Cannot find module 'undefined'");throw e.code="MODULE_NOT_FOUND",e}(),eJ(n)}catch(e){ez[t]=null}return ez[t]}function eJ(e,t){var n;return e&&((n=a(t)?eX(e):eQ(e,t))?eI=n:"undefined"!=typeof console&&console.warn&&console.warn("Locale "+e+" not found. Did you forget to load it?")),eI._abbr}function eQ(e,t){if(null===t)return delete ez[e],null;var n,s=eZ;if(t.abbr=e,null!=ez[e])S("defineLocaleOverride","use moment.updateLocale(localeName, config) to change an existing locale. moment.defineLocale(localeName, config) should only be used for creating a new locale See http://momentjs.com/guides/#/warnings/define-locale/ for more info."),s=ez[e]._config;else if(null!=t.parentLocale){if(null!=ez[t.parentLocale])s=ez[t.parentLocale]._config;else{if(null==(n=eB(t.parentLocale)))return e$[t.parentLocale]||(e$[t.parentLocale]=[]),e$[t.parentLocale].push({name:e,config:t}),null;s=n._config}}return ez[e]=new T(b(s,t)),e$[e]&&e$[e].forEach(function(e){eQ(e.name,e.config)}),eJ(e),ez[e]}function eX(e){var t;if(e&&e._locale&&e._locale._abbr&&(e=e._locale._abbr),!e)return eI;if(!n(e)){if(t=eB(e))return t;e=[e]}return function(e){for(var t,n,s,i,r=0;r0;){if(s=eB(i.slice(0,t).join("-")))return s;if(n&&n.length>=t&&function(e,t){var n,s=Math.min(e.length,t.length);for(n=0;n=t-1)break;t--}r++}return eI}(e)}function eK(e){var t,n=e._a;return n&&-2===c(e).overflow&&(t=n[1]<0||n[1]>11?1:n[2]<1||n[2]>eM(n[0],n[1])?2:n[3]<0||n[3]>24||24===n[3]&&(0!==n[4]||0!==n[5]||0!==n[6])?3:n[4]<0||n[4]>59?4:n[5]<0||n[5]>59?5:n[6]<0||n[6]>999?6:-1,c(e)._overflowDayOfYear&&(t<0||t>2)&&(t=2),c(e)._overflowWeeks&&-1===t&&(t=7),c(e)._overflowWeekday&&-1===t&&(t=8),c(e).overflow=t),e}var e0=/^\s*((?:[+-]\d{6}|\d{4})-(?:\d\d-\d\d|W\d\d-\d|W\d\d|\d\d\d|\d\d))(?:(T| )(\d\d(?::\d\d(?::\d\d(?:[.,]\d+)?)?)?)([+-]\d\d(?::?\d\d)?|\s*Z)?)?$/,e1=/^\s*((?:[+-]\d{6}|\d{4})(?:\d\d\d\d|W\d\d\d|W\d\d|\d\d\d|\d\d|))(?:(T| )(\d\d(?:\d\d(?:\d\d(?:[.,]\d+)?)?)?)([+-]\d\d(?::?\d\d)?|\s*Z)?)?$/,e2=/Z|[+-]\d\d(?::?\d\d)?/,e4=[["YYYYYY-MM-DD",/[+-]\d{6}-\d\d-\d\d/],["YYYY-MM-DD",/\d{4}-\d\d-\d\d/],["GGGG-[W]WW-E",/\d{4}-W\d\d-\d/],["GGGG-[W]WW",/\d{4}-W\d\d/,!1],["YYYY-DDD",/\d{4}-\d{3}/],["YYYY-MM",/\d{4}-\d\d/,!1],["YYYYYYMMDD",/[+-]\d{10}/],["YYYYMMDD",/\d{8}/],["GGGG[W]WWE",/\d{4}W\d{3}/],["GGGG[W]WW",/\d{4}W\d{2}/,!1],["YYYYDDD",/\d{7}/],["YYYYMM",/\d{6}/,!1],["YYYY",/\d{4}/,!1]],e6=[["HH:mm:ss.SSSS",/\d\d:\d\d:\d\d\.\d+/],["HH:mm:ss,SSSS",/\d\d:\d\d:\d\d,\d+/],["HH:mm:ss",/\d\d:\d\d:\d\d/],["HH:mm",/\d\d:\d\d/],["HHmmss.SSSS",/\d\d\d\d\d\d\.\d+/],["HHmmss,SSSS",/\d\d\d\d\d\d,\d+/],["HHmmss",/\d\d\d\d\d\d/],["HHmm",/\d\d\d\d/],["HH",/\d\d/]],e3=/^\/?Date\((-?\d+)/i,e5=/^(?:(Mon|Tue|Wed|Thu|Fri|Sat|Sun),?\s)?(\d{1,2})\s(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s(\d{2,4})\s(\d\d):(\d\d)(?::(\d\d))?\s(?:(UT|GMT|[ECMP][SD]T)|([Zz])|([+-]\d{4}))$/,e7={UT:0,GMT:0,EDT:-240,EST:-300,CDT:-300,CST:-360,MDT:-360,MST:-420,PDT:-420,PST:-480};function e9(e){var t,n,s,i,r,a,o=e._i,u=e0.exec(o)||e1.exec(o),l=e4.length,h=e6.length;if(u){for(t=0,c(e).iso=!0,n=l;t7)&&(l=!0)):(a=e._locale._week.dow,o=e._locale._week.doy,h=eR(tr(),a,o),s=te(n.gg,e._a[0],h.year),i=te(n.w,h.week),null!=n.d?((r=n.d)<0||r>6)&&(l=!0):null!=n.e?(r=n.e+a,(n.e<0||n.e>6)&&(l=!0)):r=a),i<1||i>eC(s,a,o)?c(e)._overflowWeeks=!0:null!=l?c(e)._overflowWeekday=!0:(u=eP(s,i,r,a,o),e._a[0]=u.year,e._dayOfYear=u.dayOfYear)),null!=e._dayOfYear&&(g=te(e._a[0],_[0]),(e._dayOfYear>eg(g)||0===e._dayOfYear)&&(c(e)._overflowDayOfYear=!0),m=eN(g,0,e._dayOfYear),e._a[1]=m.getUTCMonth(),e._a[2]=m.getUTCDate()),f=0;f<3&&null==e._a[f];++f)e._a[f]=w[f]=_[f];for(;f<7;f++)e._a[f]=w[f]=null==e._a[f]?2===f?1:0:e._a[f];24===e._a[3]&&0===e._a[4]&&0===e._a[5]&&0===e._a[6]&&(e._nextDay=!0,e._a[3]=0),e._d=(e._useUTC?eN:ex).apply(null,w),y=e._useUTC?e._d.getUTCDay():e._d.getDay(),null!=e._tzm&&e._d.setUTCMinutes(e._d.getUTCMinutes()-e._tzm),e._nextDay&&(e._a[3]=24),e._w&&void 0!==e._w.d&&e._w.d!==y&&(c(e).weekdayMismatch=!0)}}function tn(e){if(e._f===t.ISO_8601){e9(e);return}if(e._f===t.RFC_2822){e8(e);return}e._a=[],c(e).empty=!0;var n,s,r,a,o,u,l,h,d,f,m,_=""+e._i,y=_.length,g=0;for(o=0,m=(l=H(e._f,e._locale).match(N)||[]).length;o0&&c(e).unusedInput.push(d),_=_.slice(_.indexOf(u)+u.length),g+=u.length),R[h])?(u?c(e).empty=!1:c(e).unusedTokens.push(h),null!=u&&i(ef,h)&&ef[h](u,e._a,e,h)):e._strict&&!u&&c(e).unusedTokens.push(h);c(e).charsLeftOver=y-g,_.length>0&&c(e).unusedInput.push(_),e._a[3]<=12&&!0===c(e).bigHour&&e._a[3]>0&&(c(e).bigHour=void 0),c(e).parsedDateParts=e._a.slice(0),c(e).meridiem=e._meridiem,e._a[3]=(n=e._locale,s=e._a[3],null==(r=e._meridiem)?s:null!=n.meridiemHour?n.meridiemHour(s,r):(null!=n.isPM&&((a=n.isPM(r))&&s<12&&(s+=12),a||12!==s||(s=0)),s)),null!==(f=c(e).era)&&(e._a[0]=e._locale.erasConvertYear(f,e._a[0])),tt(e),eK(e)}function ts(e){var i,r=e._i,d=e._f;return(e._locale=e._locale||eX(e._l),null===r||void 0===d&&""===r)?m({nullInput:!0}):("string"==typeof r&&(e._i=r=e._locale.preparse(r)),k(r))?new v(eK(r)):(u(r)?e._d=r:n(d)?function(e){var t,n,s,i,r,a,o=!1,u=e._f.length;if(0===u){c(e).invalidFormat=!0,e._d=new Date(NaN);return}for(i=0;ithis?this:e:m()});function tu(e,t){var s,i;if(1===t.length&&n(t[0])&&(t=t[0]),!t.length)return tr();for(i=1,s=t[0];i=0?new Date(e+400,t,n)-126227808e5:new Date(e,t,n).valueOf()}function tC(e,t,n){return e<100&&e>=0?Date.UTC(e+400,t,n)-126227808e5:Date.UTC(e,t,n)}function tU(e,t){return t.erasAbbrRegex(e)}function tH(){var e,t,n,s,i,r=[],a=[],o=[],u=[],l=this.eras();for(e=0,t=l.length;e(r=eC(e,s,i))&&(t=r),tE.call(this,e,t,n,s,i))}function tE(e,t,n,s,i){var r=eP(e,t,n,s,i),a=eN(r.year,0,r.dayOfYear);return this.year(a.getUTCFullYear()),this.month(a.getUTCMonth()),this.date(a.getUTCDate()),this}C("N",0,0,"eraAbbr"),C("NN",0,0,"eraAbbr"),C("NNN",0,0,"eraAbbr"),C("NNNN",0,0,"eraName"),C("NNNNN",0,0,"eraNarrow"),C("y",["y",1],"yo","eraYear"),C("y",["yy",2],0,"eraYear"),C("y",["yyy",3],0,"eraYear"),C("y",["yyyy",4],0,"eraYear"),el("N",tU),el("NN",tU),el("NNN",tU),el("NNNN",function(e,t){return t.erasNameRegex(e)}),el("NNNNN",function(e,t){return t.erasNarrowRegex(e)}),em(["N","NN","NNN","NNNN","NNNNN"],function(e,t,n,s){var i=n._locale.erasParse(e,s,n._strict);i?c(n).era=i:c(n).invalidEra=e}),el("y",en),el("yy",en),el("yyy",en),el("yyyy",en),el("yo",function(e,t){return t._eraYearOrdinalRegex||en}),em(["y","yy","yyy","yyyy"],0),em(["yo"],function(e,t,n,s){var i;n._locale._eraYearOrdinalRegex&&(i=e.match(n._locale._eraYearOrdinalRegex)),n._locale.eraYearOrdinalParse?t[0]=n._locale.eraYearOrdinalParse(e,i):t[0]=parseInt(e,10)}),C(0,["gg",2],0,function(){return this.weekYear()%100}),C(0,["GG",2],0,function(){return this.isoWeekYear()%100}),tF("gggg","weekYear"),tF("ggggg","weekYear"),tF("GGGG","isoWeekYear"),tF("GGGGG","isoWeekYear"),el("G",es),el("g",es),el("GG",J,z),el("gg",J,z),el("GGGG",ee,q),el("gggg",ee,q),el("GGGGG",et,B),el("ggggg",et,B),e_(["gggg","ggggg","GGGG","GGGGG"],function(e,t,n,s){t[s.substr(0,2)]=ec(e)}),e_(["gg","GG"],function(e,n,s,i){n[i]=t.parseTwoDigitYear(e)}),C("Q",0,"Qo","quarter"),el("Q",Z),em("Q",function(e,t){t[1]=(ec(e)-1)*3}),C("D",["DD",2],"Do","date"),el("D",J,eo),el("DD",J,z),el("Do",function(e,t){return e?t._dayOfMonthOrdinalParse||t._ordinalParse:t._dayOfMonthOrdinalParseLenient}),em(["D","DD"],2),em("Do",function(e,t){t[2]=ec(e.match(J)[0])});var tV=ep("Date",!0);C("DDD",["DDDD",3],"DDDo","dayOfYear"),el("DDD",K),el("DDDD",$),em(["DDD","DDDD"],function(e,t,n){n._dayOfYear=ec(e)}),C("m",["mm",2],0,"minute"),el("m",J,eu),el("mm",J,z),em(["m","mm"],4);var tG=ep("Minutes",!1);C("s",["ss",2],0,"second"),el("s",J,eu),el("ss",J,z),em(["s","ss"],5);var tA=ep("Seconds",!1);for(C("S",0,0,function(){return~~(this.millisecond()/100)}),C(0,["SS",2],0,function(){return~~(this.millisecond()/10)}),C(0,["SSS",3],0,"millisecond"),C(0,["SSSS",4],0,function(){return 10*this.millisecond()}),C(0,["SSSSS",5],0,function(){return 100*this.millisecond()}),C(0,["SSSSSS",6],0,function(){return 1e3*this.millisecond()}),C(0,["SSSSSSS",7],0,function(){return 1e4*this.millisecond()}),C(0,["SSSSSSSS",8],0,function(){return 1e5*this.millisecond()}),C(0,["SSSSSSSSS",9],0,function(){return 1e6*this.millisecond()}),el("S",K,Z),el("SS",K,z),el("SSS",K,$),_="SSSS";_.length<=9;_+="S")el(_,en);function tI(e,t){t[6]=ec(("0."+e)*1e3)}for(_="S";_.length<=9;_+="S")em(_,tI);y=ep("Milliseconds",!1),C("z",0,0,"zoneAbbr"),C("zz",0,0,"zoneName");var tj=v.prototype;function tZ(e){return e}tj.add=tO,tj.calendar=function(e,a){if(1==arguments.length){if(arguments[0]){var l,h,d;(l=arguments[0],k(l)||u(l)||tT(l)||o(l)||(h=n(l),d=!1,h&&(d=0===l.filter(function(e){return!o(e)&&tT(l)}).length),h&&d)||function(e){var t,n,a=s(e)&&!r(e),o=!1,u=["years","year","y","months","month","M","days","day","d","dates","date","D","hours","hour","h","minutes","minute","m","seconds","second","s","milliseconds","millisecond","ms"],l=u.length;for(t=0;tn.valueOf():n.valueOf()n.year()||n.year()>9999?U(n,t?"YYYYYY-MM-DD[T]HH:mm:ss.SSS[Z]":"YYYYYY-MM-DD[T]HH:mm:ss.SSSZ"):O(Date.prototype.toISOString)?t?this.toDate().toISOString():new Date(this.valueOf()+6e4*this.utcOffset()).toISOString().replace("Z",U(n,"Z")):U(n,t?"YYYY-MM-DD[T]HH:mm:ss.SSS[Z]":"YYYY-MM-DD[T]HH:mm:ss.SSSZ")},tj.inspect=function(){if(!this.isValid())return"moment.invalid(/* "+this._i+" */)";var e,t,n,s,i="moment",r="";return this.isLocal()||(i=0===this.utcOffset()?"moment.utc":"moment.parseZone",r="Z"),e="["+i+'("]',t=0<=this.year()&&9999>=this.year()?"YYYY":"YYYYYY",n="-MM-DD[T]HH:mm:ss.SSS",s=r+'[")]',this.format(e+t+n+s)},"undefined"!=typeof Symbol&&null!=Symbol.for&&(tj[Symbol.for("nodejs.util.inspect.custom")]=function(){return"Moment<"+this.format()+">"}),tj.toJSON=function(){return this.isValid()?this.toISOString():null},tj.toString=function(){return this.clone().locale("en").format("ddd MMM DD YYYY HH:mm:ss [GMT]ZZ")},tj.unix=function(){return Math.floor(this.valueOf()/1e3)},tj.valueOf=function(){return this._d.valueOf()-6e4*(this._offset||0)},tj.creationData=function(){return{input:this._i,format:this._f,locale:this._locale,isUTC:this._isUTC,strict:this._strict}},tj.eraName=function(){var e,t,n,s=this.localeData().eras();for(e=0,t=s.length;eMath.abs(e)&&!s&&(e*=60);return!this._isUTC&&n&&(i=tg(this)),this._offset=e,this._isUTC=!0,null!=i&&this.add(i,"m"),r===e||(!n||this._changeInProgress?tS(this,tk(e-r,"m"),1,!1):this._changeInProgress||(this._changeInProgress=!0,t.updateOffset(this,!0),this._changeInProgress=null)),this},tj.utc=function(e){return this.utcOffset(0,e)},tj.local=function(e){return this._isUTC&&(this.utcOffset(0,e),this._isUTC=!1,e&&this.subtract(tg(this),"m")),this},tj.parseZone=function(){if(null!=this._tzm)this.utcOffset(this._tzm,!1,!0);else if("string"==typeof this._i){var e=t_(ei,this._i);null!=e?this.utcOffset(e):this.utcOffset(0,!0)}return this},tj.hasAlignedHourOffset=function(e){return!!this.isValid()&&(e=e?tr(e).utcOffset():0,(this.utcOffset()-e)%60==0)},tj.isDST=function(){return this.utcOffset()>this.clone().month(0).utcOffset()||this.utcOffset()>this.clone().month(5).utcOffset()},tj.isLocal=function(){return!!this.isValid()&&!this._isUTC},tj.isUtcOffset=function(){return!!this.isValid()&&this._isUTC},tj.isUtc=tw,tj.isUTC=tw,tj.zoneAbbr=function(){return this._isUTC?"UTC":""},tj.zoneName=function(){return this._isUTC?"Coordinated Universal Time":""},tj.dates=D("dates accessor is deprecated. Use date instead.",tV),tj.months=D("months accessor is deprecated. Use month instead",eb),tj.years=D("years accessor is deprecated. Use year instead",ew),tj.zone=D("moment().zone is deprecated, use moment().utcOffset instead. http://momentjs.com/guides/#/warnings/zone/",function(e,t){return null!=e?("string"!=typeof e&&(e=-e),this.utcOffset(e,t),this):-this.utcOffset()}),tj.isDSTShifted=D("isDSTShifted is deprecated. See http://momentjs.com/guides/#/warnings/dst-shifted/ for more information",function(){if(!a(this._isDSTShifted))return this._isDSTShifted;var e,t={};return p(t,this),(t=ts(t))._a?(e=t._isUTC?d(t._a):tr(t._a),this._isDSTShifted=this.isValid()&&function(e,t,n){var s,i=Math.min(e.length,t.length),r=Math.abs(e.length-t.length),a=0;for(s=0;s0):this._isDSTShifted=!1,this._isDSTShifted});var tz=T.prototype;function t$(e,t,n,s){var i=eX(),r=d().set(s,t);return i[n](r,e)}function tq(e,t,n){if(o(e)&&(t=e,e=void 0),e=e||"",null!=t)return t$(e,t,n,"month");var s,i=[];for(s=0;s<12;s++)i[s]=t$(e,s,n,"month");return i}function tB(e,t,n,s){"boolean"==typeof e||(n=t=e,e=!1),o(t)&&(n=t,t=void 0),t=t||"";var i,r=eX(),a=e?r._week.dow:0,u=[];if(null!=n)return t$(t,(n+a)%7,s,"day");for(i=0;i<7;i++)u[i]=t$(t,(i+a)%7,s,"day");return u}tz.calendar=function(e,t,n){var s=this._calendar[e]||this._calendar.sameElse;return O(s)?s.call(t,n):s},tz.longDateFormat=function(e){var t=this._longDateFormat[e],n=this._longDateFormat[e.toUpperCase()];return t||!n?t:(this._longDateFormat[e]=n.match(N).map(function(e){return"MMMM"===e||"MM"===e||"DD"===e||"dddd"===e?e.slice(1):e}).join(""),this._longDateFormat[e])},tz.invalidDate=function(){return this._invalidDate},tz.ordinal=function(e){return this._ordinal.replace("%d",e)},tz.preparse=tZ,tz.postformat=tZ,tz.relativeTime=function(e,t,n,s){var i=this._relativeTime[n];return O(i)?i(e,t,n,s):i.replace(/%d/i,e)},tz.pastFuture=function(e,t){var n=this._relativeTime[e>0?"future":"past"];return O(n)?n(t):n.replace(/%s/i,t)},tz.set=function(e){var t,n;for(n in e)i(e,n)&&(O(t=e[n])?this[n]=t:this["_"+n]=t);this._config=e,this._dayOfMonthOrdinalParseLenient=RegExp((this._dayOfMonthOrdinalParse.source||this._ordinalParse.source)+"|"+/\d{1,2}/.source)},tz.eras=function(e,n){var s,i,r,a=this._eras||eX("en")._eras;for(s=0,i=a.length;s=0)return u[s]},tz.erasConvertYear=function(e,n){var s=e.since<=e.until?1:-1;return void 0===n?t(e.since).year():t(e.since).year()+(n-e.offset)*s},tz.erasAbbrRegex=function(e){return i(this,"_erasAbbrRegex")||tH.call(this),e?this._erasAbbrRegex:this._erasRegex},tz.erasNameRegex=function(e){return i(this,"_erasNameRegex")||tH.call(this),e?this._erasNameRegex:this._erasRegex},tz.erasNarrowRegex=function(e){return i(this,"_erasNarrowRegex")||tH.call(this),e?this._erasNarrowRegex:this._erasRegex},tz.months=function(e,t){return e?n(this._months)?this._months[e.month()]:this._months[(this._months.isFormat||eY).test(t)?"format":"standalone"][e.month()]:n(this._months)?this._months:this._months.standalone},tz.monthsShort=function(e,t){return e?n(this._monthsShort)?this._monthsShort[e.month()]:this._monthsShort[eY.test(t)?"format":"standalone"][e.month()]:n(this._monthsShort)?this._monthsShort:this._monthsShort.standalone},tz.monthsParse=function(e,t,n){var s,i,r;if(this._monthsParseExact)return eS.call(this,e,t,n);for(this._monthsParse||(this._monthsParse=[],this._longMonthsParse=[],this._shortMonthsParse=[]),s=0;s<12;s++)if(i=d([2e3,s]),n&&!this._longMonthsParse[s]&&(this._longMonthsParse[s]=RegExp("^"+this.months(i,"").replace(".","")+"$","i"),this._shortMonthsParse[s]=RegExp("^"+this.monthsShort(i,"").replace(".","")+"$","i")),n||this._monthsParse[s]||(r="^"+this.months(i,"")+"|^"+this.monthsShort(i,""),this._monthsParse[s]=RegExp(r.replace(".",""),"i")),n&&"MMMM"===t&&this._longMonthsParse[s].test(e)||n&&"MMM"===t&&this._shortMonthsParse[s].test(e)||!n&&this._monthsParse[s].test(e))return s},tz.monthsRegex=function(e){return this._monthsParseExact?(i(this,"_monthsRegex")||eT.call(this),e)?this._monthsStrictRegex:this._monthsRegex:(i(this,"_monthsRegex")||(this._monthsRegex=ea),this._monthsStrictRegex&&e?this._monthsStrictRegex:this._monthsRegex)},tz.monthsShortRegex=function(e){return this._monthsParseExact?(i(this,"_monthsRegex")||eT.call(this),e)?this._monthsShortStrictRegex:this._monthsShortRegex:(i(this,"_monthsShortRegex")||(this._monthsShortRegex=ea),this._monthsShortStrictRegex&&e?this._monthsShortStrictRegex:this._monthsShortRegex)},tz.week=function(e){return eR(e,this._week.dow,this._week.doy).week},tz.firstDayOfYear=function(){return this._week.doy},tz.firstDayOfWeek=function(){return this._week.dow},tz.weekdays=function(e,t){var s=n(this._weekdays)?this._weekdays:this._weekdays[e&&!0!==e&&this._weekdays.isFormat.test(t)?"format":"standalone"];return!0===e?eU(s,this._week.dow):e?s[e.day()]:s},tz.weekdaysMin=function(e){return!0===e?eU(this._weekdaysMin,this._week.dow):e?this._weekdaysMin[e.day()]:this._weekdaysMin},tz.weekdaysShort=function(e){return!0===e?eU(this._weekdaysShort,this._week.dow):e?this._weekdaysShort[e.day()]:this._weekdaysShort},tz.weekdaysParse=function(e,t,n){var s,i,r;if(this._weekdaysParseExact)return eF.call(this,e,t,n);for(this._weekdaysParse||(this._weekdaysParse=[],this._minWeekdaysParse=[],this._shortWeekdaysParse=[],this._fullWeekdaysParse=[]),s=0;s<7;s++){if(i=d([2e3,1]).day(s),n&&!this._fullWeekdaysParse[s]&&(this._fullWeekdaysParse[s]=RegExp("^"+this.weekdays(i,"").replace(".","\\.?")+"$","i"),this._shortWeekdaysParse[s]=RegExp("^"+this.weekdaysShort(i,"").replace(".","\\.?")+"$","i"),this._minWeekdaysParse[s]=RegExp("^"+this.weekdaysMin(i,"").replace(".","\\.?")+"$","i")),this._weekdaysParse[s]||(r="^"+this.weekdays(i,"")+"|^"+this.weekdaysShort(i,"")+"|^"+this.weekdaysMin(i,""),this._weekdaysParse[s]=RegExp(r.replace(".",""),"i")),n&&"dddd"===t&&this._fullWeekdaysParse[s].test(e)||n&&"ddd"===t&&this._shortWeekdaysParse[s].test(e))return s;if(n&&"dd"===t&&this._minWeekdaysParse[s].test(e))return s;if(!n&&this._weekdaysParse[s].test(e))return s}},tz.weekdaysRegex=function(e){return this._weekdaysParseExact?(i(this,"_weekdaysRegex")||eL.call(this),e)?this._weekdaysStrictRegex:this._weekdaysRegex:(i(this,"_weekdaysRegex")||(this._weekdaysRegex=ea),this._weekdaysStrictRegex&&e?this._weekdaysStrictRegex:this._weekdaysRegex)},tz.weekdaysShortRegex=function(e){return this._weekdaysParseExact?(i(this,"_weekdaysRegex")||eL.call(this),e)?this._weekdaysShortStrictRegex:this._weekdaysShortRegex:(i(this,"_weekdaysShortRegex")||(this._weekdaysShortRegex=ea),this._weekdaysShortStrictRegex&&e?this._weekdaysShortStrictRegex:this._weekdaysShortRegex)},tz.weekdaysMinRegex=function(e){return this._weekdaysParseExact?(i(this,"_weekdaysRegex")||eL.call(this),e)?this._weekdaysMinStrictRegex:this._weekdaysMinRegex:(i(this,"_weekdaysMinRegex")||(this._weekdaysMinRegex=ea),this._weekdaysMinStrictRegex&&e?this._weekdaysMinStrictRegex:this._weekdaysMinRegex)},tz.isPM=function(e){return"p"===(e+"").toLowerCase().charAt(0)},tz.meridiem=function(e,t,n){return e>11?n?"pm":"PM":n?"am":"AM"},eJ("en",{eras:[{since:"0001-01-01",until:Infinity,offset:1,name:"Anno Domini",narrow:"AD",abbr:"AD"},{since:"0000-12-31",until:-1/0,offset:1,name:"Before Christ",narrow:"BC",abbr:"BC"}],dayOfMonthOrdinalParse:/\d{1,2}(th|st|nd|rd)/,ordinal:function(e){var t=e%10,n=1===ec(e%100/10)?"th":1===t?"st":2===t?"nd":3===t?"rd":"th";return e+n}}),t.lang=D("moment.lang is deprecated. Use moment.locale instead.",eJ),t.langData=D("moment.langData is deprecated. Use moment.localeData instead.",eX);var tJ=Math.abs;function tQ(e,t,n,s){var i=tk(t,n);return e._milliseconds+=s*i._milliseconds,e._days+=s*i._days,e._months+=s*i._months,e._bubble()}function tX(e){return e<0?Math.floor(e):Math.ceil(e)}function tK(e){return 4800*e/146097}function t0(e){return 146097*e/4800}function t1(e){return function(){return this.as(e)}}var t2=t1("ms"),t4=t1("s"),t6=t1("m"),t3=t1("h"),t5=t1("d"),t7=t1("w"),t9=t1("M"),t8=t1("Q"),ne=t1("y");function nt(e){return function(){return this.isValid()?this._data[e]:NaN}}var nn=nt("milliseconds"),ns=nt("seconds"),ni=nt("minutes"),nr=nt("hours"),na=nt("days"),no=nt("months"),nu=nt("years"),nl=Math.round,nh={ss:44,s:45,m:45,h:22,d:26,w:null,M:11};function nd(e,t,n,s,i){return i.relativeTime(t||1,!!n,e,s)}var nc=Math.abs;function nf(e){return(e>0)-(e<0)||+e}function nm(){if(!this.isValid())return this.localeData().invalidDate();var e,t,n,s,i,r,a,o,u=nc(this._milliseconds)/1e3,l=nc(this._days),h=nc(this._months),d=this.asSeconds();return d?(e=ed(u/60),t=ed(e/60),u%=60,e%=60,n=ed(h/12),h%=12,s=u?u.toFixed(3).replace(/\.?0+$/,""):"",i=d<0?"-":"",r=nf(this._months)!==nf(d)?"-":"",a=nf(this._days)!==nf(d)?"-":"",o=nf(this._milliseconds)!==nf(d)?"-":"",i+"P"+(n?r+n+"Y":"")+(h?r+h+"M":"")+(l?a+l+"D":"")+(t||e||u?"T":"")+(t?o+t+"H":"")+(e?o+e+"M":"")+(u?o+s+"S":"")):"P0D"}var n_=th.prototype;return n_.isValid=function(){return this._isValid},n_.abs=function(){var e=this._data;return this._milliseconds=tJ(this._milliseconds),this._days=tJ(this._days),this._months=tJ(this._months),e.milliseconds=tJ(e.milliseconds),e.seconds=tJ(e.seconds),e.minutes=tJ(e.minutes),e.hours=tJ(e.hours),e.months=tJ(e.months),e.years=tJ(e.years),this},n_.add=function(e,t){return tQ(this,e,t,1)},n_.subtract=function(e,t){return tQ(this,e,t,-1)},n_.as=function(e){if(!this.isValid())return NaN;var t,n,s=this._milliseconds;if("month"===(e=L(e))||"quarter"===e||"year"===e)switch(t=this._days+s/864e5,n=this._months+tK(t),e){case"month":return n;case"quarter":return n/3;case"year":return n/12}else switch(t=this._days+Math.round(t0(this._months)),e){case"week":return t/7+s/6048e5;case"day":return t+s/864e5;case"hour":return 24*t+s/36e5;case"minute":return 1440*t+s/6e4;case"second":return 86400*t+s/1e3;case"millisecond":return Math.floor(864e5*t)+s;default:throw Error("Unknown unit "+e)}},n_.asMilliseconds=t2,n_.asSeconds=t4,n_.asMinutes=t6,n_.asHours=t3,n_.asDays=t5,n_.asWeeks=t7,n_.asMonths=t9,n_.asQuarters=t8,n_.asYears=ne,n_.valueOf=t2,n_._bubble=function(){var e,t,n,s,i,r=this._milliseconds,a=this._days,o=this._months,u=this._data;return r>=0&&a>=0&&o>=0||r<=0&&a<=0&&o<=0||(r+=864e5*tX(t0(o)+a),a=0,o=0),u.milliseconds=r%1e3,e=ed(r/1e3),u.seconds=e%60,t=ed(e/60),u.minutes=t%60,n=ed(t/60),u.hours=n%24,a+=ed(n/24),o+=i=ed(tK(a)),a-=tX(t0(i)),s=ed(o/12),o%=12,u.days=a,u.months=o,u.years=s,this},n_.clone=function(){return tk(this)},n_.get=function(e){return e=L(e),this.isValid()?this[e+"s"]():NaN},n_.milliseconds=nn,n_.seconds=ns,n_.minutes=ni,n_.hours=nr,n_.days=na,n_.weeks=function(){return ed(this.days()/7)},n_.months=no,n_.years=nu,n_.humanize=function(e,t){if(!this.isValid())return this.localeData().invalidDate();var n,s,i,r,a,o,u,l,h,d,c,f,m,_=!1,y=nh;return"object"==typeof e&&(t=e,e=!1),"boolean"==typeof e&&(_=e),"object"==typeof t&&(y=Object.assign({},nh,t),null!=t.s&&null==t.ss&&(y.ss=t.s-1)),f=this.localeData(),n=!_,s=y,i=tk(this).abs(),r=nl(i.as("s")),a=nl(i.as("m")),o=nl(i.as("h")),u=nl(i.as("d")),l=nl(i.as("M")),h=nl(i.as("w")),d=nl(i.as("y")),c=r<=s.ss&&["s",r]||r0,c[4]=f,m=nd.apply(null,c),_&&(m=f.pastFuture(+this,m)),f.postformat(m)},n_.toISOString=nm,n_.toString=nm,n_.toJSON=nm,n_.locale=tN,n_.localeData=tP,n_.toIsoString=D("toIsoString() is deprecated. Please use toISOString() instead (notice the capitals)",nm),n_.lang=tW,C("X",0,0,"unix"),C("x",0,0,"valueOf"),el("x",es),el("X",/[+-]?\d+(\.\d{1,3})?/),em("X",function(e,t,n){n._d=new Date(1e3*parseFloat(e))}),em("x",function(e,t,n){n._d=new Date(ec(e))}),t.version="2.30.1",V=tr,t.fn=tj,t.min=function(){var e=[].slice.call(arguments,0);return tu("isBefore",e)},t.max=function(){var e=[].slice.call(arguments,0);return tu("isAfter",e)},t.now=function(){return Date.now?Date.now():+new Date},t.utc=d,t.unix=function(e){return tr(1e3*e)},t.months=function(e,t){return tq(e,t,"months")},t.isDate=u,t.locale=eJ,t.invalid=m,t.duration=tk,t.isMoment=k,t.weekdays=function(e,t,n){return tB(e,t,n,"weekdays")},t.parseZone=function(){return tr.apply(null,arguments).parseZone()},t.localeData=eX,t.isDuration=td,t.monthsShort=function(e,t){return tq(e,t,"monthsShort")},t.weekdaysMin=function(e,t,n){return tB(e,t,n,"weekdaysMin")},t.defineLocale=eQ,t.updateLocale=function(e,t){if(null!=t){var n,s,i=eZ;null!=ez[e]&&null!=ez[e].parentLocale?ez[e].set(b(ez[e]._config,t)):(null!=(s=eB(e))&&(i=s._config),t=b(i,t),null==s&&(t.abbr=e),(n=new T(t)).parentLocale=ez[e],ez[e]=n),eJ(e)}else null!=ez[e]&&(null!=ez[e].parentLocale?(ez[e]=ez[e].parentLocale,e===eJ()&&eJ(e)):null!=ez[e]&&delete ez[e]);return ez[e]},t.locales=function(){return A(ez)},t.weekdaysShort=function(e,t,n){return tB(e,t,n,"weekdaysShort")},t.normalizeUnits=L,t.relativeTimeRounding=function(e){return void 0===e?nl:"function"==typeof e&&(nl=e,!0)},t.relativeTimeThreshold=function(e,t){return void 0!==nh[e]&&(void 0===t?nh[e]:(nh[e]=t,"s"===e&&(nh.ss=t-1),!0))},t.calendarFormat=function(e,t){var n=e.diff(t,"days",!0);return n<-6?"sameElse":n<-1?"lastWeek":n<0?"lastDay":n<1?"sameDay":n<2?"nextDay":n<7?"nextWeek":"sameElse"},t.prototype=tj,t.HTML5_FMT={DATETIME_LOCAL:"YYYY-MM-DDTHH:mm",DATETIME_LOCAL_SECONDS:"YYYY-MM-DDTHH:mm:ss",DATETIME_LOCAL_MS:"YYYY-MM-DDTHH:mm:ss.SSS",DATE:"YYYY-MM-DD",TIME:"HH:mm",TIME_SECONDS:"HH:mm:ss",TIME_MS:"HH:mm:ss.SSS",WEEK:"GGGG-[W]WW",MONTH:"YYYY-MM"},t},e.exports=s()}}]);
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/13b76428-ebdf3012af0e4489.js b/litellm/proxy/_experimental/out/_next/static/chunks/13b76428-ebdf3012af0e4489.js
deleted file mode 100644
index 307379053b6..00000000000
--- a/litellm/proxy/_experimental/out/_next/static/chunks/13b76428-ebdf3012af0e4489.js
+++ /dev/null
@@ -1 +0,0 @@
-(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[990],{77398:function(e,t,n){var s;e=n.nmd(e),s=function(){"use strict";function t(){return V.apply(null,arguments)}function n(e){return e instanceof Array||"[object Array]"===Object.prototype.toString.call(e)}function s(e){return null!=e&&"[object Object]"===Object.prototype.toString.call(e)}function i(e,t){return Object.prototype.hasOwnProperty.call(e,t)}function r(e){var t;if(Object.getOwnPropertyNames)return 0===Object.getOwnPropertyNames(e).length;for(t in e)if(i(e,t))return!1;return!0}function a(e){return void 0===e}function o(e){return"number"==typeof e||"[object Number]"===Object.prototype.toString.call(e)}function u(e){return e instanceof Date||"[object Date]"===Object.prototype.toString.call(e)}function l(e,t){var n,s=[],i=e.length;for(n=0;n>>0;for(t=0;t0)for(n=0;n=0?n?"+":"":"-")+Math.pow(10,Math.max(0,t-s.length)).toString().substr(1)+s}t.suppressDeprecationWarnings=!1,t.deprecationHandler=null,A=Object.keys?Object.keys:function(e){var t,n=[];for(t in e)i(e,t)&&n.push(t);return n};var N=/(\[[^\[]*\])|(\\)?([Hh]mm(ss)?|Mo|MM?M?M?|Do|DDDo|DD?D?D?|ddd?d?|do?|w[o|w]?|W[o|W]?|Qo?|N{1,5}|YYYYYY|YYYYY|YYYY|YY|y{2,4}|yo?|gg(ggg?)?|GG(GGG?)?|e|E|a|A|hh?|HH?|kk?|mm?|ss?|S{1,9}|x|X|zz?|ZZ?|.)/g,W=/(\[[^\[]*\])|(\\)?(LTS|LT|LL?L?L?|l{1,4})/g,P={},R={};function C(e,t,n,s){var i=s;"string"==typeof s&&(i=function(){return this[s]()}),e&&(R[e]=i),t&&(R[t[0]]=function(){return x(i.apply(this,arguments),t[1],t[2])}),n&&(R[n]=function(){return this.localeData().ordinal(i.apply(this,arguments),e)})}function U(e,t){return e.isValid()?(P[t=H(t,e.localeData())]=P[t]||function(e){var t,n,s,i=e.match(N);for(n=0,s=i.length;n=0&&W.test(e);)e=e.replace(W,s),W.lastIndex=0,n-=1;return e}var F={D:"date",dates:"date",date:"date",d:"day",days:"day",day:"day",e:"weekday",weekdays:"weekday",weekday:"weekday",E:"isoWeekday",isoweekdays:"isoWeekday",isoweekday:"isoWeekday",DDD:"dayOfYear",dayofyears:"dayOfYear",dayofyear:"dayOfYear",h:"hour",hours:"hour",hour:"hour",ms:"millisecond",milliseconds:"millisecond",millisecond:"millisecond",m:"minute",minutes:"minute",minute:"minute",M:"month",months:"month",month:"month",Q:"quarter",quarters:"quarter",quarter:"quarter",s:"second",seconds:"second",second:"second",gg:"weekYear",weekyears:"weekYear",weekyear:"weekYear",GG:"isoWeekYear",isoweekyears:"isoWeekYear",isoweekyear:"isoWeekYear",w:"week",weeks:"week",week:"week",W:"isoWeek",isoweeks:"isoWeek",isoweek:"isoWeek",y:"year",years:"year",year:"year"};function L(e){return"string"==typeof e?F[e]||F[e.toLowerCase()]:void 0}function E(e){var t,n,s={};for(n in e)i(e,n)&&(t=L(n))&&(s[t]=e[n]);return s}var V,G,A,I,j={date:9,day:11,weekday:11,isoWeekday:11,dayOfYear:4,hour:13,millisecond:16,minute:14,month:8,quarter:7,second:15,weekYear:1,isoWeekYear:1,week:5,isoWeek:5,year:1},Z=/\d/,z=/\d\d/,$=/\d{3}/,q=/\d{4}/,B=/[+-]?\d{6}/,J=/\d\d?/,Q=/\d\d\d\d?/,X=/\d\d\d\d\d\d?/,K=/\d{1,3}/,ee=/\d{1,4}/,et=/[+-]?\d{1,6}/,en=/\d+/,es=/[+-]?\d+/,ei=/Z|[+-]\d\d:?\d\d/gi,er=/Z|[+-]\d\d(?::?\d\d)?/gi,ea=/[0-9]{0,256}['a-z\u00A0-\u05FF\u0700-\uD7FF\uF900-\uFDCF\uFDF0-\uFF07\uFF10-\uFFEF]{1,256}|[\u0600-\u06FF\/]{1,256}(\s*?[\u0600-\u06FF]{1,256}){1,2}/i,eo=/^[1-9]\d?/,eu=/^([1-9]\d|\d)/;function el(e,t,n){I[e]=O(t)?t:function(e,s){return e&&n?n:t}}function eh(e){return e.replace(/[-\/\\^$*+?.()|[\]{}]/g,"\\$&")}function ed(e){return e<0?Math.ceil(e)||0:Math.floor(e)}function ec(e){var t=+e,n=0;return 0!==t&&isFinite(t)&&(n=ed(t)),n}I={};var ef={};function em(e,t){var n,s,i=t;for("string"==typeof e&&(e=[e]),o(t)&&(i=function(e,n){n[t]=ec(e)}),s=e.length,n=0;n68?1900:2e3)};var ew=ep("FullYear",!0);function ep(e,n){return function(s){return null!=s?(ek(this,e,s),t.updateOffset(this,n),this):ev(this,e)}}function ev(e,t){if(!e.isValid())return NaN;var n=e._d,s=e._isUTC;switch(t){case"Milliseconds":return s?n.getUTCMilliseconds():n.getMilliseconds();case"Seconds":return s?n.getUTCSeconds():n.getSeconds();case"Minutes":return s?n.getUTCMinutes():n.getMinutes();case"Hours":return s?n.getUTCHours():n.getHours();case"Date":return s?n.getUTCDate():n.getDate();case"Day":return s?n.getUTCDay():n.getDay();case"Month":return s?n.getUTCMonth():n.getMonth();case"FullYear":return s?n.getUTCFullYear():n.getFullYear();default:return NaN}}function ek(e,t,n){var s,i,r,a;if(!(!e.isValid()||isNaN(n))){switch(s=e._d,i=e._isUTC,t){case"Milliseconds":return void(i?s.setUTCMilliseconds(n):s.setMilliseconds(n));case"Seconds":return void(i?s.setUTCSeconds(n):s.setSeconds(n));case"Minutes":return void(i?s.setUTCMinutes(n):s.setMinutes(n));case"Hours":return void(i?s.setUTCHours(n):s.setHours(n));case"Date":return void(i?s.setUTCDate(n):s.setDate(n));case"FullYear":break;default:return}r=e.month(),a=29!==(a=e.date())||1!==r||ey(n)?a:28,i?s.setUTCFullYear(n,r,a):s.setFullYear(n,r,a)}}function eM(e,t){if(isNaN(e)||isNaN(t))return NaN;var n=(t%12+12)%12;return e+=(t-n)/12,1===n?ey(e)?29:28:31-n%7%2}eA=Array.prototype.indexOf?Array.prototype.indexOf:function(e){var t;for(t=0;t=0?isFinite((o=new Date(e+400,t,n,s,i,r,a)).getFullYear())&&o.setFullYear(e):o=new Date(e,t,n,s,i,r,a),o}function eN(e){var t,n;return e<100&&e>=0?(n=Array.prototype.slice.call(arguments),n[0]=e+400,isFinite((t=new Date(Date.UTC.apply(null,n))).getUTCFullYear())&&t.setUTCFullYear(e)):t=new Date(Date.UTC.apply(null,arguments)),t}function eW(e,t,n){var s=7+t-n;return-((7+eN(e,0,s).getUTCDay()-t)%7)+s-1}function eP(e,t,n,s,i){var r,a,o=1+7*(t-1)+(7+n-s)%7+eW(e,s,i);return o<=0?a=eg(r=e-1)+o:o>eg(e)?(r=e+1,a=o-eg(e)):(r=e,a=o),{year:r,dayOfYear:a}}function eR(e,t,n){var s,i,r=eW(e.year(),t,n),a=Math.floor((e.dayOfYear()-r-1)/7)+1;return a<1?s=a+eC(i=e.year()-1,t,n):a>eC(e.year(),t,n)?(s=a-eC(e.year(),t,n),i=e.year()+1):(i=e.year(),s=a),{week:s,year:i}}function eC(e,t,n){var s=eW(e,t,n),i=eW(e+1,t,n);return(eg(e)-s+i)/7}function eU(e,t){return e.slice(t,7).concat(e.slice(0,t))}C("w",["ww",2],"wo","week"),C("W",["WW",2],"Wo","isoWeek"),el("w",J,eo),el("ww",J,z),el("W",J,eo),el("WW",J,z),e_(["w","ww","W","WW"],function(e,t,n,s){t[s.substr(0,1)]=ec(e)}),C("d",0,"do","day"),C("dd",0,0,function(e){return this.localeData().weekdaysMin(this,e)}),C("ddd",0,0,function(e){return this.localeData().weekdaysShort(this,e)}),C("dddd",0,0,function(e){return this.localeData().weekdays(this,e)}),C("e",0,0,"weekday"),C("E",0,0,"isoWeekday"),el("d",J),el("e",J),el("E",J),el("dd",function(e,t){return t.weekdaysMinRegex(e)}),el("ddd",function(e,t){return t.weekdaysShortRegex(e)}),el("dddd",function(e,t){return t.weekdaysRegex(e)}),e_(["dd","ddd","dddd"],function(e,t,n,s){var i=n._locale.weekdaysParse(e,s,n._strict);null!=i?t.d=i:c(n).invalidWeekday=e}),e_(["d","e","E"],function(e,t,n,s){t[s]=ec(e)});var eH="Sun_Mon_Tue_Wed_Thu_Fri_Sat".split("_");function eF(e,t,n){var s,i,r,a=e.toLocaleLowerCase();if(!this._weekdaysParse)for(s=0,this._weekdaysParse=[],this._shortWeekdaysParse=[],this._minWeekdaysParse=[];s<7;++s)r=d([2e3,1]).day(s),this._minWeekdaysParse[s]=this.weekdaysMin(r,"").toLocaleLowerCase(),this._shortWeekdaysParse[s]=this.weekdaysShort(r,"").toLocaleLowerCase(),this._weekdaysParse[s]=this.weekdays(r,"").toLocaleLowerCase();return n?"dddd"===t?-1!==(i=eA.call(this._weekdaysParse,a))?i:null:"ddd"===t?-1!==(i=eA.call(this._shortWeekdaysParse,a))?i:null:-1!==(i=eA.call(this._minWeekdaysParse,a))?i:null:"dddd"===t?-1!==(i=eA.call(this._weekdaysParse,a))||-1!==(i=eA.call(this._shortWeekdaysParse,a))?i:-1!==(i=eA.call(this._minWeekdaysParse,a))?i:null:"ddd"===t?-1!==(i=eA.call(this._shortWeekdaysParse,a))||-1!==(i=eA.call(this._weekdaysParse,a))?i:-1!==(i=eA.call(this._minWeekdaysParse,a))?i:null:-1!==(i=eA.call(this._minWeekdaysParse,a))||-1!==(i=eA.call(this._weekdaysParse,a))?i:-1!==(i=eA.call(this._shortWeekdaysParse,a))?i:null}function eL(){function e(e,t){return t.length-e.length}var t,n,s,i,r,a=[],o=[],u=[],l=[];for(t=0;t<7;t++)n=d([2e3,1]).day(t),s=eh(this.weekdaysMin(n,"")),i=eh(this.weekdaysShort(n,"")),r=eh(this.weekdays(n,"")),a.push(s),o.push(i),u.push(r),l.push(s),l.push(i),l.push(r);a.sort(e),o.sort(e),u.sort(e),l.sort(e),this._weekdaysRegex=RegExp("^("+l.join("|")+")","i"),this._weekdaysShortRegex=this._weekdaysRegex,this._weekdaysMinRegex=this._weekdaysRegex,this._weekdaysStrictRegex=RegExp("^("+u.join("|")+")","i"),this._weekdaysShortStrictRegex=RegExp("^("+o.join("|")+")","i"),this._weekdaysMinStrictRegex=RegExp("^("+a.join("|")+")","i")}function eE(){return this.hours()%12||12}function eV(e,t){C(e,0,0,function(){return this.localeData().meridiem(this.hours(),this.minutes(),t)})}function eG(e,t){return t._meridiemParse}C("H",["HH",2],0,"hour"),C("h",["hh",2],0,eE),C("k",["kk",2],0,function(){return this.hours()||24}),C("hmm",0,0,function(){return""+eE.apply(this)+x(this.minutes(),2)}),C("hmmss",0,0,function(){return""+eE.apply(this)+x(this.minutes(),2)+x(this.seconds(),2)}),C("Hmm",0,0,function(){return""+this.hours()+x(this.minutes(),2)}),C("Hmmss",0,0,function(){return""+this.hours()+x(this.minutes(),2)+x(this.seconds(),2)}),eV("a",!0),eV("A",!1),el("a",eG),el("A",eG),el("H",J,eu),el("h",J,eo),el("k",J,eo),el("HH",J,z),el("hh",J,z),el("kk",J,z),el("hmm",Q),el("hmmss",X),el("Hmm",Q),el("Hmmss",X),em(["H","HH"],3),em(["k","kk"],function(e,t,n){var s=ec(e);t[3]=24===s?0:s}),em(["a","A"],function(e,t,n){n._isPm=n._locale.isPM(e),n._meridiem=e}),em(["h","hh"],function(e,t,n){t[3]=ec(e),c(n).bigHour=!0}),em("hmm",function(e,t,n){var s=e.length-2;t[3]=ec(e.substr(0,s)),t[4]=ec(e.substr(s)),c(n).bigHour=!0}),em("hmmss",function(e,t,n){var s=e.length-4,i=e.length-2;t[3]=ec(e.substr(0,s)),t[4]=ec(e.substr(s,2)),t[5]=ec(e.substr(i)),c(n).bigHour=!0}),em("Hmm",function(e,t,n){var s=e.length-2;t[3]=ec(e.substr(0,s)),t[4]=ec(e.substr(s))}),em("Hmmss",function(e,t,n){var s=e.length-4,i=e.length-2;t[3]=ec(e.substr(0,s)),t[4]=ec(e.substr(s,2)),t[5]=ec(e.substr(i))});var eA,eI,ej=ep("Hours",!0),eZ={calendar:{sameDay:"[Today at] LT",nextDay:"[Tomorrow at] LT",nextWeek:"dddd [at] LT",lastDay:"[Yesterday at] LT",lastWeek:"[Last] dddd [at] LT",sameElse:"L"},longDateFormat:{LTS:"h:mm:ss A",LT:"h:mm A",L:"MM/DD/YYYY",LL:"MMMM D, YYYY",LLL:"MMMM D, YYYY h:mm A",LLLL:"dddd, MMMM D, YYYY h:mm A"},invalidDate:"Invalid date",ordinal:"%d",dayOfMonthOrdinalParse:/\d{1,2}/,relativeTime:{future:"in %s",past:"%s ago",s:"a few seconds",ss:"%d seconds",m:"a minute",mm:"%d minutes",h:"an hour",hh:"%d hours",d:"a day",dd:"%d days",w:"a week",ww:"%d weeks",M:"a month",MM:"%d months",y:"a year",yy:"%d years"},months:"January_February_March_April_May_June_July_August_September_October_November_December".split("_"),monthsShort:eD,week:{dow:0,doy:6},weekdays:"Sunday_Monday_Tuesday_Wednesday_Thursday_Friday_Saturday".split("_"),weekdaysMin:"Su_Mo_Tu_We_Th_Fr_Sa".split("_"),weekdaysShort:eH,meridiemParse:/[ap]\.?m?\.?/i},ez={},e$={};function eq(e){return e?e.toLowerCase().replace("_","-"):e}function eB(t){var n=null;if(void 0===ez[t]&&e&&e.exports&&t&&t.match("^[^/\\\\]*$"))try{n=eI._abbr,function(){var e=Error("Cannot find module 'undefined'");throw e.code="MODULE_NOT_FOUND",e}(),eJ(n)}catch(e){ez[t]=null}return ez[t]}function eJ(e,t){var n;return e&&((n=a(t)?eX(e):eQ(e,t))?eI=n:"undefined"!=typeof console&&console.warn&&console.warn("Locale "+e+" not found. Did you forget to load it?")),eI._abbr}function eQ(e,t){if(null===t)return delete ez[e],null;var n,s=eZ;if(t.abbr=e,null!=ez[e])S("defineLocaleOverride","use moment.updateLocale(localeName, config) to change an existing locale. moment.defineLocale(localeName, config) should only be used for creating a new locale See http://momentjs.com/guides/#/warnings/define-locale/ for more info."),s=ez[e]._config;else if(null!=t.parentLocale){if(null!=ez[t.parentLocale])s=ez[t.parentLocale]._config;else{if(null==(n=eB(t.parentLocale)))return e$[t.parentLocale]||(e$[t.parentLocale]=[]),e$[t.parentLocale].push({name:e,config:t}),null;s=n._config}}return ez[e]=new T(b(s,t)),e$[e]&&e$[e].forEach(function(e){eQ(e.name,e.config)}),eJ(e),ez[e]}function eX(e){var t;if(e&&e._locale&&e._locale._abbr&&(e=e._locale._abbr),!e)return eI;if(!n(e)){if(t=eB(e))return t;e=[e]}return function(e){for(var t,n,s,i,r=0;r0;){if(s=eB(i.slice(0,t).join("-")))return s;if(n&&n.length>=t&&function(e,t){var n,s=Math.min(e.length,t.length);for(n=0;n=t-1)break;t--}r++}return eI}(e)}function eK(e){var t,n=e._a;return n&&-2===c(e).overflow&&(t=n[1]<0||n[1]>11?1:n[2]<1||n[2]>eM(n[0],n[1])?2:n[3]<0||n[3]>24||24===n[3]&&(0!==n[4]||0!==n[5]||0!==n[6])?3:n[4]<0||n[4]>59?4:n[5]<0||n[5]>59?5:n[6]<0||n[6]>999?6:-1,c(e)._overflowDayOfYear&&(t<0||t>2)&&(t=2),c(e)._overflowWeeks&&-1===t&&(t=7),c(e)._overflowWeekday&&-1===t&&(t=8),c(e).overflow=t),e}var e0=/^\s*((?:[+-]\d{6}|\d{4})-(?:\d\d-\d\d|W\d\d-\d|W\d\d|\d\d\d|\d\d))(?:(T| )(\d\d(?::\d\d(?::\d\d(?:[.,]\d+)?)?)?)([+-]\d\d(?::?\d\d)?|\s*Z)?)?$/,e1=/^\s*((?:[+-]\d{6}|\d{4})(?:\d\d\d\d|W\d\d\d|W\d\d|\d\d\d|\d\d|))(?:(T| )(\d\d(?:\d\d(?:\d\d(?:[.,]\d+)?)?)?)([+-]\d\d(?::?\d\d)?|\s*Z)?)?$/,e2=/Z|[+-]\d\d(?::?\d\d)?/,e4=[["YYYYYY-MM-DD",/[+-]\d{6}-\d\d-\d\d/],["YYYY-MM-DD",/\d{4}-\d\d-\d\d/],["GGGG-[W]WW-E",/\d{4}-W\d\d-\d/],["GGGG-[W]WW",/\d{4}-W\d\d/,!1],["YYYY-DDD",/\d{4}-\d{3}/],["YYYY-MM",/\d{4}-\d\d/,!1],["YYYYYYMMDD",/[+-]\d{10}/],["YYYYMMDD",/\d{8}/],["GGGG[W]WWE",/\d{4}W\d{3}/],["GGGG[W]WW",/\d{4}W\d{2}/,!1],["YYYYDDD",/\d{7}/],["YYYYMM",/\d{6}/,!1],["YYYY",/\d{4}/,!1]],e3=[["HH:mm:ss.SSSS",/\d\d:\d\d:\d\d\.\d+/],["HH:mm:ss,SSSS",/\d\d:\d\d:\d\d,\d+/],["HH:mm:ss",/\d\d:\d\d:\d\d/],["HH:mm",/\d\d:\d\d/],["HHmmss.SSSS",/\d\d\d\d\d\d\.\d+/],["HHmmss,SSSS",/\d\d\d\d\d\d,\d+/],["HHmmss",/\d\d\d\d\d\d/],["HHmm",/\d\d\d\d/],["HH",/\d\d/]],e6=/^\/?Date\((-?\d+)/i,e5=/^(?:(Mon|Tue|Wed|Thu|Fri|Sat|Sun),?\s)?(\d{1,2})\s(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s(\d{2,4})\s(\d\d):(\d\d)(?::(\d\d))?\s(?:(UT|GMT|[ECMP][SD]T)|([Zz])|([+-]\d{4}))$/,e7={UT:0,GMT:0,EDT:-240,EST:-300,CDT:-300,CST:-360,MDT:-360,MST:-420,PDT:-420,PST:-480};function e9(e){var t,n,s,i,r,a,o=e._i,u=e0.exec(o)||e1.exec(o),l=e4.length,h=e3.length;if(u){for(t=0,c(e).iso=!0,n=l;t7)&&(l=!0)):(a=e._locale._week.dow,o=e._locale._week.doy,h=eR(tr(),a,o),s=te(n.gg,e._a[0],h.year),i=te(n.w,h.week),null!=n.d?((r=n.d)<0||r>6)&&(l=!0):null!=n.e?(r=n.e+a,(n.e<0||n.e>6)&&(l=!0)):r=a),i<1||i>eC(s,a,o)?c(e)._overflowWeeks=!0:null!=l?c(e)._overflowWeekday=!0:(u=eP(s,i,r,a,o),e._a[0]=u.year,e._dayOfYear=u.dayOfYear)),null!=e._dayOfYear&&(g=te(e._a[0],_[0]),(e._dayOfYear>eg(g)||0===e._dayOfYear)&&(c(e)._overflowDayOfYear=!0),m=eN(g,0,e._dayOfYear),e._a[1]=m.getUTCMonth(),e._a[2]=m.getUTCDate()),f=0;f<3&&null==e._a[f];++f)e._a[f]=w[f]=_[f];for(;f<7;f++)e._a[f]=w[f]=null==e._a[f]?2===f?1:0:e._a[f];24===e._a[3]&&0===e._a[4]&&0===e._a[5]&&0===e._a[6]&&(e._nextDay=!0,e._a[3]=0),e._d=(e._useUTC?eN:ex).apply(null,w),y=e._useUTC?e._d.getUTCDay():e._d.getDay(),null!=e._tzm&&e._d.setUTCMinutes(e._d.getUTCMinutes()-e._tzm),e._nextDay&&(e._a[3]=24),e._w&&void 0!==e._w.d&&e._w.d!==y&&(c(e).weekdayMismatch=!0)}}function tn(e){if(e._f===t.ISO_8601){e9(e);return}if(e._f===t.RFC_2822){e8(e);return}e._a=[],c(e).empty=!0;var n,s,r,a,o,u,l,h,d,f,m,_=""+e._i,y=_.length,g=0;for(o=0,m=(l=H(e._f,e._locale).match(N)||[]).length;o0&&c(e).unusedInput.push(d),_=_.slice(_.indexOf(u)+u.length),g+=u.length),R[h])?(u?c(e).empty=!1:c(e).unusedTokens.push(h),null!=u&&i(ef,h)&&ef[h](u,e._a,e,h)):e._strict&&!u&&c(e).unusedTokens.push(h);c(e).charsLeftOver=y-g,_.length>0&&c(e).unusedInput.push(_),e._a[3]<=12&&!0===c(e).bigHour&&e._a[3]>0&&(c(e).bigHour=void 0),c(e).parsedDateParts=e._a.slice(0),c(e).meridiem=e._meridiem,e._a[3]=(n=e._locale,s=e._a[3],null==(r=e._meridiem)?s:null!=n.meridiemHour?n.meridiemHour(s,r):(null!=n.isPM&&((a=n.isPM(r))&&s<12&&(s+=12),a||12!==s||(s=0)),s)),null!==(f=c(e).era)&&(e._a[0]=e._locale.erasConvertYear(f,e._a[0])),tt(e),eK(e)}function ts(e){var i,r=e._i,d=e._f;return(e._locale=e._locale||eX(e._l),null===r||void 0===d&&""===r)?m({nullInput:!0}):("string"==typeof r&&(e._i=r=e._locale.preparse(r)),k(r))?new v(eK(r)):(u(r)?e._d=r:n(d)?function(e){var t,n,s,i,r,a,o=!1,u=e._f.length;if(0===u){c(e).invalidFormat=!0,e._d=new Date(NaN);return}for(i=0;ithis?this:e:m()});function tu(e,t){var s,i;if(1===t.length&&n(t[0])&&(t=t[0]),!t.length)return tr();for(i=1,s=t[0];i=0?new Date(e+400,t,n)-126227808e5:new Date(e,t,n).valueOf()}function tC(e,t,n){return e<100&&e>=0?Date.UTC(e+400,t,n)-126227808e5:Date.UTC(e,t,n)}function tU(e,t){return t.erasAbbrRegex(e)}function tH(){var e,t,n,s,i,r=[],a=[],o=[],u=[],l=this.eras();for(e=0,t=l.length;e(r=eC(e,s,i))&&(t=r),tE.call(this,e,t,n,s,i))}function tE(e,t,n,s,i){var r=eP(e,t,n,s,i),a=eN(r.year,0,r.dayOfYear);return this.year(a.getUTCFullYear()),this.month(a.getUTCMonth()),this.date(a.getUTCDate()),this}C("N",0,0,"eraAbbr"),C("NN",0,0,"eraAbbr"),C("NNN",0,0,"eraAbbr"),C("NNNN",0,0,"eraName"),C("NNNNN",0,0,"eraNarrow"),C("y",["y",1],"yo","eraYear"),C("y",["yy",2],0,"eraYear"),C("y",["yyy",3],0,"eraYear"),C("y",["yyyy",4],0,"eraYear"),el("N",tU),el("NN",tU),el("NNN",tU),el("NNNN",function(e,t){return t.erasNameRegex(e)}),el("NNNNN",function(e,t){return t.erasNarrowRegex(e)}),em(["N","NN","NNN","NNNN","NNNNN"],function(e,t,n,s){var i=n._locale.erasParse(e,s,n._strict);i?c(n).era=i:c(n).invalidEra=e}),el("y",en),el("yy",en),el("yyy",en),el("yyyy",en),el("yo",function(e,t){return t._eraYearOrdinalRegex||en}),em(["y","yy","yyy","yyyy"],0),em(["yo"],function(e,t,n,s){var i;n._locale._eraYearOrdinalRegex&&(i=e.match(n._locale._eraYearOrdinalRegex)),n._locale.eraYearOrdinalParse?t[0]=n._locale.eraYearOrdinalParse(e,i):t[0]=parseInt(e,10)}),C(0,["gg",2],0,function(){return this.weekYear()%100}),C(0,["GG",2],0,function(){return this.isoWeekYear()%100}),tF("gggg","weekYear"),tF("ggggg","weekYear"),tF("GGGG","isoWeekYear"),tF("GGGGG","isoWeekYear"),el("G",es),el("g",es),el("GG",J,z),el("gg",J,z),el("GGGG",ee,q),el("gggg",ee,q),el("GGGGG",et,B),el("ggggg",et,B),e_(["gggg","ggggg","GGGG","GGGGG"],function(e,t,n,s){t[s.substr(0,2)]=ec(e)}),e_(["gg","GG"],function(e,n,s,i){n[i]=t.parseTwoDigitYear(e)}),C("Q",0,"Qo","quarter"),el("Q",Z),em("Q",function(e,t){t[1]=(ec(e)-1)*3}),C("D",["DD",2],"Do","date"),el("D",J,eo),el("DD",J,z),el("Do",function(e,t){return e?t._dayOfMonthOrdinalParse||t._ordinalParse:t._dayOfMonthOrdinalParseLenient}),em(["D","DD"],2),em("Do",function(e,t){t[2]=ec(e.match(J)[0])});var tV=ep("Date",!0);C("DDD",["DDDD",3],"DDDo","dayOfYear"),el("DDD",K),el("DDDD",$),em(["DDD","DDDD"],function(e,t,n){n._dayOfYear=ec(e)}),C("m",["mm",2],0,"minute"),el("m",J,eu),el("mm",J,z),em(["m","mm"],4);var tG=ep("Minutes",!1);C("s",["ss",2],0,"second"),el("s",J,eu),el("ss",J,z),em(["s","ss"],5);var tA=ep("Seconds",!1);for(C("S",0,0,function(){return~~(this.millisecond()/100)}),C(0,["SS",2],0,function(){return~~(this.millisecond()/10)}),C(0,["SSS",3],0,"millisecond"),C(0,["SSSS",4],0,function(){return 10*this.millisecond()}),C(0,["SSSSS",5],0,function(){return 100*this.millisecond()}),C(0,["SSSSSS",6],0,function(){return 1e3*this.millisecond()}),C(0,["SSSSSSS",7],0,function(){return 1e4*this.millisecond()}),C(0,["SSSSSSSS",8],0,function(){return 1e5*this.millisecond()}),C(0,["SSSSSSSSS",9],0,function(){return 1e6*this.millisecond()}),el("S",K,Z),el("SS",K,z),el("SSS",K,$),_="SSSS";_.length<=9;_+="S")el(_,en);function tI(e,t){t[6]=ec(("0."+e)*1e3)}for(_="S";_.length<=9;_+="S")em(_,tI);y=ep("Milliseconds",!1),C("z",0,0,"zoneAbbr"),C("zz",0,0,"zoneName");var tj=v.prototype;function tZ(e){return e}tj.add=tO,tj.calendar=function(e,a){if(1==arguments.length){if(arguments[0]){var l,h,d;(l=arguments[0],k(l)||u(l)||tT(l)||o(l)||(h=n(l),d=!1,h&&(d=0===l.filter(function(e){return!o(e)&&tT(l)}).length),h&&d)||function(e){var t,n,a=s(e)&&!r(e),o=!1,u=["years","year","y","months","month","M","days","day","d","dates","date","D","hours","hour","h","minutes","minute","m","seconds","second","s","milliseconds","millisecond","ms"],l=u.length;for(t=0;tn.valueOf():n.valueOf()n.year()||n.year()>9999?U(n,t?"YYYYYY-MM-DD[T]HH:mm:ss.SSS[Z]":"YYYYYY-MM-DD[T]HH:mm:ss.SSSZ"):O(Date.prototype.toISOString)?t?this.toDate().toISOString():new Date(this.valueOf()+6e4*this.utcOffset()).toISOString().replace("Z",U(n,"Z")):U(n,t?"YYYY-MM-DD[T]HH:mm:ss.SSS[Z]":"YYYY-MM-DD[T]HH:mm:ss.SSSZ")},tj.inspect=function(){if(!this.isValid())return"moment.invalid(/* "+this._i+" */)";var e,t,n,s,i="moment",r="";return this.isLocal()||(i=0===this.utcOffset()?"moment.utc":"moment.parseZone",r="Z"),e="["+i+'("]',t=0<=this.year()&&9999>=this.year()?"YYYY":"YYYYYY",n="-MM-DD[T]HH:mm:ss.SSS",s=r+'[")]',this.format(e+t+n+s)},"undefined"!=typeof Symbol&&null!=Symbol.for&&(tj[Symbol.for("nodejs.util.inspect.custom")]=function(){return"Moment<"+this.format()+">"}),tj.toJSON=function(){return this.isValid()?this.toISOString():null},tj.toString=function(){return this.clone().locale("en").format("ddd MMM DD YYYY HH:mm:ss [GMT]ZZ")},tj.unix=function(){return Math.floor(this.valueOf()/1e3)},tj.valueOf=function(){return this._d.valueOf()-6e4*(this._offset||0)},tj.creationData=function(){return{input:this._i,format:this._f,locale:this._locale,isUTC:this._isUTC,strict:this._strict}},tj.eraName=function(){var e,t,n,s=this.localeData().eras();for(e=0,t=s.length;eMath.abs(e)&&!s&&(e*=60);return!this._isUTC&&n&&(i=tg(this)),this._offset=e,this._isUTC=!0,null!=i&&this.add(i,"m"),r===e||(!n||this._changeInProgress?tS(this,tk(e-r,"m"),1,!1):this._changeInProgress||(this._changeInProgress=!0,t.updateOffset(this,!0),this._changeInProgress=null)),this},tj.utc=function(e){return this.utcOffset(0,e)},tj.local=function(e){return this._isUTC&&(this.utcOffset(0,e),this._isUTC=!1,e&&this.subtract(tg(this),"m")),this},tj.parseZone=function(){if(null!=this._tzm)this.utcOffset(this._tzm,!1,!0);else if("string"==typeof this._i){var e=t_(ei,this._i);null!=e?this.utcOffset(e):this.utcOffset(0,!0)}return this},tj.hasAlignedHourOffset=function(e){return!!this.isValid()&&(e=e?tr(e).utcOffset():0,(this.utcOffset()-e)%60==0)},tj.isDST=function(){return this.utcOffset()>this.clone().month(0).utcOffset()||this.utcOffset()>this.clone().month(5).utcOffset()},tj.isLocal=function(){return!!this.isValid()&&!this._isUTC},tj.isUtcOffset=function(){return!!this.isValid()&&this._isUTC},tj.isUtc=tw,tj.isUTC=tw,tj.zoneAbbr=function(){return this._isUTC?"UTC":""},tj.zoneName=function(){return this._isUTC?"Coordinated Universal Time":""},tj.dates=D("dates accessor is deprecated. Use date instead.",tV),tj.months=D("months accessor is deprecated. Use month instead",eb),tj.years=D("years accessor is deprecated. Use year instead",ew),tj.zone=D("moment().zone is deprecated, use moment().utcOffset instead. http://momentjs.com/guides/#/warnings/zone/",function(e,t){return null!=e?("string"!=typeof e&&(e=-e),this.utcOffset(e,t),this):-this.utcOffset()}),tj.isDSTShifted=D("isDSTShifted is deprecated. See http://momentjs.com/guides/#/warnings/dst-shifted/ for more information",function(){if(!a(this._isDSTShifted))return this._isDSTShifted;var e,t={};return p(t,this),(t=ts(t))._a?(e=t._isUTC?d(t._a):tr(t._a),this._isDSTShifted=this.isValid()&&function(e,t,n){var s,i=Math.min(e.length,t.length),r=Math.abs(e.length-t.length),a=0;for(s=0;s0):this._isDSTShifted=!1,this._isDSTShifted});var tz=T.prototype;function t$(e,t,n,s){var i=eX(),r=d().set(s,t);return i[n](r,e)}function tq(e,t,n){if(o(e)&&(t=e,e=void 0),e=e||"",null!=t)return t$(e,t,n,"month");var s,i=[];for(s=0;s<12;s++)i[s]=t$(e,s,n,"month");return i}function tB(e,t,n,s){"boolean"==typeof e||(n=t=e,e=!1),o(t)&&(n=t,t=void 0),t=t||"";var i,r=eX(),a=e?r._week.dow:0,u=[];if(null!=n)return t$(t,(n+a)%7,s,"day");for(i=0;i<7;i++)u[i]=t$(t,(i+a)%7,s,"day");return u}tz.calendar=function(e,t,n){var s=this._calendar[e]||this._calendar.sameElse;return O(s)?s.call(t,n):s},tz.longDateFormat=function(e){var t=this._longDateFormat[e],n=this._longDateFormat[e.toUpperCase()];return t||!n?t:(this._longDateFormat[e]=n.match(N).map(function(e){return"MMMM"===e||"MM"===e||"DD"===e||"dddd"===e?e.slice(1):e}).join(""),this._longDateFormat[e])},tz.invalidDate=function(){return this._invalidDate},tz.ordinal=function(e){return this._ordinal.replace("%d",e)},tz.preparse=tZ,tz.postformat=tZ,tz.relativeTime=function(e,t,n,s){var i=this._relativeTime[n];return O(i)?i(e,t,n,s):i.replace(/%d/i,e)},tz.pastFuture=function(e,t){var n=this._relativeTime[e>0?"future":"past"];return O(n)?n(t):n.replace(/%s/i,t)},tz.set=function(e){var t,n;for(n in e)i(e,n)&&(O(t=e[n])?this[n]=t:this["_"+n]=t);this._config=e,this._dayOfMonthOrdinalParseLenient=RegExp((this._dayOfMonthOrdinalParse.source||this._ordinalParse.source)+"|"+/\d{1,2}/.source)},tz.eras=function(e,n){var s,i,r,a=this._eras||eX("en")._eras;for(s=0,i=a.length;s=0)return u[s]},tz.erasConvertYear=function(e,n){var s=e.since<=e.until?1:-1;return void 0===n?t(e.since).year():t(e.since).year()+(n-e.offset)*s},tz.erasAbbrRegex=function(e){return i(this,"_erasAbbrRegex")||tH.call(this),e?this._erasAbbrRegex:this._erasRegex},tz.erasNameRegex=function(e){return i(this,"_erasNameRegex")||tH.call(this),e?this._erasNameRegex:this._erasRegex},tz.erasNarrowRegex=function(e){return i(this,"_erasNarrowRegex")||tH.call(this),e?this._erasNarrowRegex:this._erasRegex},tz.months=function(e,t){return e?n(this._months)?this._months[e.month()]:this._months[(this._months.isFormat||eY).test(t)?"format":"standalone"][e.month()]:n(this._months)?this._months:this._months.standalone},tz.monthsShort=function(e,t){return e?n(this._monthsShort)?this._monthsShort[e.month()]:this._monthsShort[eY.test(t)?"format":"standalone"][e.month()]:n(this._monthsShort)?this._monthsShort:this._monthsShort.standalone},tz.monthsParse=function(e,t,n){var s,i,r;if(this._monthsParseExact)return eS.call(this,e,t,n);for(this._monthsParse||(this._monthsParse=[],this._longMonthsParse=[],this._shortMonthsParse=[]),s=0;s<12;s++)if(i=d([2e3,s]),n&&!this._longMonthsParse[s]&&(this._longMonthsParse[s]=RegExp("^"+this.months(i,"").replace(".","")+"$","i"),this._shortMonthsParse[s]=RegExp("^"+this.monthsShort(i,"").replace(".","")+"$","i")),n||this._monthsParse[s]||(r="^"+this.months(i,"")+"|^"+this.monthsShort(i,""),this._monthsParse[s]=RegExp(r.replace(".",""),"i")),n&&"MMMM"===t&&this._longMonthsParse[s].test(e)||n&&"MMM"===t&&this._shortMonthsParse[s].test(e)||!n&&this._monthsParse[s].test(e))return s},tz.monthsRegex=function(e){return this._monthsParseExact?(i(this,"_monthsRegex")||eT.call(this),e)?this._monthsStrictRegex:this._monthsRegex:(i(this,"_monthsRegex")||(this._monthsRegex=ea),this._monthsStrictRegex&&e?this._monthsStrictRegex:this._monthsRegex)},tz.monthsShortRegex=function(e){return this._monthsParseExact?(i(this,"_monthsRegex")||eT.call(this),e)?this._monthsShortStrictRegex:this._monthsShortRegex:(i(this,"_monthsShortRegex")||(this._monthsShortRegex=ea),this._monthsShortStrictRegex&&e?this._monthsShortStrictRegex:this._monthsShortRegex)},tz.week=function(e){return eR(e,this._week.dow,this._week.doy).week},tz.firstDayOfYear=function(){return this._week.doy},tz.firstDayOfWeek=function(){return this._week.dow},tz.weekdays=function(e,t){var s=n(this._weekdays)?this._weekdays:this._weekdays[e&&!0!==e&&this._weekdays.isFormat.test(t)?"format":"standalone"];return!0===e?eU(s,this._week.dow):e?s[e.day()]:s},tz.weekdaysMin=function(e){return!0===e?eU(this._weekdaysMin,this._week.dow):e?this._weekdaysMin[e.day()]:this._weekdaysMin},tz.weekdaysShort=function(e){return!0===e?eU(this._weekdaysShort,this._week.dow):e?this._weekdaysShort[e.day()]:this._weekdaysShort},tz.weekdaysParse=function(e,t,n){var s,i,r;if(this._weekdaysParseExact)return eF.call(this,e,t,n);for(this._weekdaysParse||(this._weekdaysParse=[],this._minWeekdaysParse=[],this._shortWeekdaysParse=[],this._fullWeekdaysParse=[]),s=0;s<7;s++){if(i=d([2e3,1]).day(s),n&&!this._fullWeekdaysParse[s]&&(this._fullWeekdaysParse[s]=RegExp("^"+this.weekdays(i,"").replace(".","\\.?")+"$","i"),this._shortWeekdaysParse[s]=RegExp("^"+this.weekdaysShort(i,"").replace(".","\\.?")+"$","i"),this._minWeekdaysParse[s]=RegExp("^"+this.weekdaysMin(i,"").replace(".","\\.?")+"$","i")),this._weekdaysParse[s]||(r="^"+this.weekdays(i,"")+"|^"+this.weekdaysShort(i,"")+"|^"+this.weekdaysMin(i,""),this._weekdaysParse[s]=RegExp(r.replace(".",""),"i")),n&&"dddd"===t&&this._fullWeekdaysParse[s].test(e)||n&&"ddd"===t&&this._shortWeekdaysParse[s].test(e))return s;if(n&&"dd"===t&&this._minWeekdaysParse[s].test(e))return s;if(!n&&this._weekdaysParse[s].test(e))return s}},tz.weekdaysRegex=function(e){return this._weekdaysParseExact?(i(this,"_weekdaysRegex")||eL.call(this),e)?this._weekdaysStrictRegex:this._weekdaysRegex:(i(this,"_weekdaysRegex")||(this._weekdaysRegex=ea),this._weekdaysStrictRegex&&e?this._weekdaysStrictRegex:this._weekdaysRegex)},tz.weekdaysShortRegex=function(e){return this._weekdaysParseExact?(i(this,"_weekdaysRegex")||eL.call(this),e)?this._weekdaysShortStrictRegex:this._weekdaysShortRegex:(i(this,"_weekdaysShortRegex")||(this._weekdaysShortRegex=ea),this._weekdaysShortStrictRegex&&e?this._weekdaysShortStrictRegex:this._weekdaysShortRegex)},tz.weekdaysMinRegex=function(e){return this._weekdaysParseExact?(i(this,"_weekdaysRegex")||eL.call(this),e)?this._weekdaysMinStrictRegex:this._weekdaysMinRegex:(i(this,"_weekdaysMinRegex")||(this._weekdaysMinRegex=ea),this._weekdaysMinStrictRegex&&e?this._weekdaysMinStrictRegex:this._weekdaysMinRegex)},tz.isPM=function(e){return"p"===(e+"").toLowerCase().charAt(0)},tz.meridiem=function(e,t,n){return e>11?n?"pm":"PM":n?"am":"AM"},eJ("en",{eras:[{since:"0001-01-01",until:Infinity,offset:1,name:"Anno Domini",narrow:"AD",abbr:"AD"},{since:"0000-12-31",until:-1/0,offset:1,name:"Before Christ",narrow:"BC",abbr:"BC"}],dayOfMonthOrdinalParse:/\d{1,2}(th|st|nd|rd)/,ordinal:function(e){var t=e%10,n=1===ec(e%100/10)?"th":1===t?"st":2===t?"nd":3===t?"rd":"th";return e+n}}),t.lang=D("moment.lang is deprecated. Use moment.locale instead.",eJ),t.langData=D("moment.langData is deprecated. Use moment.localeData instead.",eX);var tJ=Math.abs;function tQ(e,t,n,s){var i=tk(t,n);return e._milliseconds+=s*i._milliseconds,e._days+=s*i._days,e._months+=s*i._months,e._bubble()}function tX(e){return e<0?Math.floor(e):Math.ceil(e)}function tK(e){return 4800*e/146097}function t0(e){return 146097*e/4800}function t1(e){return function(){return this.as(e)}}var t2=t1("ms"),t4=t1("s"),t3=t1("m"),t6=t1("h"),t5=t1("d"),t7=t1("w"),t9=t1("M"),t8=t1("Q"),ne=t1("y");function nt(e){return function(){return this.isValid()?this._data[e]:NaN}}var nn=nt("milliseconds"),ns=nt("seconds"),ni=nt("minutes"),nr=nt("hours"),na=nt("days"),no=nt("months"),nu=nt("years"),nl=Math.round,nh={ss:44,s:45,m:45,h:22,d:26,w:null,M:11};function nd(e,t,n,s,i){return i.relativeTime(t||1,!!n,e,s)}var nc=Math.abs;function nf(e){return(e>0)-(e<0)||+e}function nm(){if(!this.isValid())return this.localeData().invalidDate();var e,t,n,s,i,r,a,o,u=nc(this._milliseconds)/1e3,l=nc(this._days),h=nc(this._months),d=this.asSeconds();return d?(e=ed(u/60),t=ed(e/60),u%=60,e%=60,n=ed(h/12),h%=12,s=u?u.toFixed(3).replace(/\.?0+$/,""):"",i=d<0?"-":"",r=nf(this._months)!==nf(d)?"-":"",a=nf(this._days)!==nf(d)?"-":"",o=nf(this._milliseconds)!==nf(d)?"-":"",i+"P"+(n?r+n+"Y":"")+(h?r+h+"M":"")+(l?a+l+"D":"")+(t||e||u?"T":"")+(t?o+t+"H":"")+(e?o+e+"M":"")+(u?o+s+"S":"")):"P0D"}var n_=th.prototype;return n_.isValid=function(){return this._isValid},n_.abs=function(){var e=this._data;return this._milliseconds=tJ(this._milliseconds),this._days=tJ(this._days),this._months=tJ(this._months),e.milliseconds=tJ(e.milliseconds),e.seconds=tJ(e.seconds),e.minutes=tJ(e.minutes),e.hours=tJ(e.hours),e.months=tJ(e.months),e.years=tJ(e.years),this},n_.add=function(e,t){return tQ(this,e,t,1)},n_.subtract=function(e,t){return tQ(this,e,t,-1)},n_.as=function(e){if(!this.isValid())return NaN;var t,n,s=this._milliseconds;if("month"===(e=L(e))||"quarter"===e||"year"===e)switch(t=this._days+s/864e5,n=this._months+tK(t),e){case"month":return n;case"quarter":return n/3;case"year":return n/12}else switch(t=this._days+Math.round(t0(this._months)),e){case"week":return t/7+s/6048e5;case"day":return t+s/864e5;case"hour":return 24*t+s/36e5;case"minute":return 1440*t+s/6e4;case"second":return 86400*t+s/1e3;case"millisecond":return Math.floor(864e5*t)+s;default:throw Error("Unknown unit "+e)}},n_.asMilliseconds=t2,n_.asSeconds=t4,n_.asMinutes=t3,n_.asHours=t6,n_.asDays=t5,n_.asWeeks=t7,n_.asMonths=t9,n_.asQuarters=t8,n_.asYears=ne,n_.valueOf=t2,n_._bubble=function(){var e,t,n,s,i,r=this._milliseconds,a=this._days,o=this._months,u=this._data;return r>=0&&a>=0&&o>=0||r<=0&&a<=0&&o<=0||(r+=864e5*tX(t0(o)+a),a=0,o=0),u.milliseconds=r%1e3,e=ed(r/1e3),u.seconds=e%60,t=ed(e/60),u.minutes=t%60,n=ed(t/60),u.hours=n%24,a+=ed(n/24),o+=i=ed(tK(a)),a-=tX(t0(i)),s=ed(o/12),o%=12,u.days=a,u.months=o,u.years=s,this},n_.clone=function(){return tk(this)},n_.get=function(e){return e=L(e),this.isValid()?this[e+"s"]():NaN},n_.milliseconds=nn,n_.seconds=ns,n_.minutes=ni,n_.hours=nr,n_.days=na,n_.weeks=function(){return ed(this.days()/7)},n_.months=no,n_.years=nu,n_.humanize=function(e,t){if(!this.isValid())return this.localeData().invalidDate();var n,s,i,r,a,o,u,l,h,d,c,f,m,_=!1,y=nh;return"object"==typeof e&&(t=e,e=!1),"boolean"==typeof e&&(_=e),"object"==typeof t&&(y=Object.assign({},nh,t),null!=t.s&&null==t.ss&&(y.ss=t.s-1)),f=this.localeData(),n=!_,s=y,i=tk(this).abs(),r=nl(i.as("s")),a=nl(i.as("m")),o=nl(i.as("h")),u=nl(i.as("d")),l=nl(i.as("M")),h=nl(i.as("w")),d=nl(i.as("y")),c=r<=s.ss&&["s",r]||r0,c[4]=f,m=nd.apply(null,c),_&&(m=f.pastFuture(+this,m)),f.postformat(m)},n_.toISOString=nm,n_.toString=nm,n_.toJSON=nm,n_.locale=tN,n_.localeData=tP,n_.toIsoString=D("toIsoString() is deprecated. Please use toISOString() instead (notice the capitals)",nm),n_.lang=tW,C("X",0,0,"unix"),C("x",0,0,"valueOf"),el("x",es),el("X",/[+-]?\d+(\.\d{1,3})?/),em("X",function(e,t,n){n._d=new Date(1e3*parseFloat(e))}),em("x",function(e,t,n){n._d=new Date(ec(e))}),t.version="2.30.1",V=tr,t.fn=tj,t.min=function(){var e=[].slice.call(arguments,0);return tu("isBefore",e)},t.max=function(){var e=[].slice.call(arguments,0);return tu("isAfter",e)},t.now=function(){return Date.now?Date.now():+new Date},t.utc=d,t.unix=function(e){return tr(1e3*e)},t.months=function(e,t){return tq(e,t,"months")},t.isDate=u,t.locale=eJ,t.invalid=m,t.duration=tk,t.isMoment=k,t.weekdays=function(e,t,n){return tB(e,t,n,"weekdays")},t.parseZone=function(){return tr.apply(null,arguments).parseZone()},t.localeData=eX,t.isDuration=td,t.monthsShort=function(e,t){return tq(e,t,"monthsShort")},t.weekdaysMin=function(e,t,n){return tB(e,t,n,"weekdaysMin")},t.defineLocale=eQ,t.updateLocale=function(e,t){if(null!=t){var n,s,i=eZ;null!=ez[e]&&null!=ez[e].parentLocale?ez[e].set(b(ez[e]._config,t)):(null!=(s=eB(e))&&(i=s._config),t=b(i,t),null==s&&(t.abbr=e),(n=new T(t)).parentLocale=ez[e],ez[e]=n),eJ(e)}else null!=ez[e]&&(null!=ez[e].parentLocale?(ez[e]=ez[e].parentLocale,e===eJ()&&eJ(e)):null!=ez[e]&&delete ez[e]);return ez[e]},t.locales=function(){return A(ez)},t.weekdaysShort=function(e,t,n){return tB(e,t,n,"weekdaysShort")},t.normalizeUnits=L,t.relativeTimeRounding=function(e){return void 0===e?nl:"function"==typeof e&&(nl=e,!0)},t.relativeTimeThreshold=function(e,t){return void 0!==nh[e]&&(void 0===t?nh[e]:(nh[e]=t,"s"===e&&(nh.ss=t-1),!0))},t.calendarFormat=function(e,t){var n=e.diff(t,"days",!0);return n<-6?"sameElse":n<-1?"lastWeek":n<0?"lastDay":n<1?"sameDay":n<2?"nextDay":n<7?"nextWeek":"sameElse"},t.prototype=tj,t.HTML5_FMT={DATETIME_LOCAL:"YYYY-MM-DDTHH:mm",DATETIME_LOCAL_SECONDS:"YYYY-MM-DDTHH:mm:ss",DATETIME_LOCAL_MS:"YYYY-MM-DDTHH:mm:ss.SSS",DATE:"YYYY-MM-DD",TIME:"HH:mm",TIME_SECONDS:"HH:mm:ss",TIME_MS:"HH:mm:ss.SSS",WEEK:"GGGG-[W]WW",MONTH:"YYYY-MM"},t},e.exports=s()}}]);
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/1487-2f4bad651391939b.js b/litellm/proxy/_experimental/out/_next/static/chunks/1487-2f4bad651391939b.js
new file mode 100644
index 00000000000..eee3c8a3eb7
--- /dev/null
+++ b/litellm/proxy/_experimental/out/_next/static/chunks/1487-2f4bad651391939b.js
@@ -0,0 +1 @@
+"use strict";(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[1487],{21487:function(e,t,n){let r,o,a;n.d(t,{Z:function(){return nU}});var i,l,u,s,d=n(5853),c=n(2265),f=n(54887),m=n(13323),v=n(64518),h=n(96822),p=n(40048),g=n(72238),b=n(93689);let y=(0,c.createContext)(!1);var w=n(61424),x=n(27847);let k=c.Fragment,M=c.Fragment,C=(0,c.createContext)(null),D=(0,c.createContext)(null);Object.assign((0,x.yV)(function(e,t){var n;let r,o,a=(0,c.useRef)(null),i=(0,b.T)((0,b.h)(e=>{a.current=e}),t),l=(0,p.i)(a),u=function(e){let t=(0,c.useContext)(y),n=(0,c.useContext)(C),r=(0,p.i)(e),[o,a]=(0,c.useState)(()=>{if(!t&&null!==n||w.O.isServer)return null;let e=null==r?void 0:r.getElementById("headlessui-portal-root");if(e)return e;if(null===r)return null;let o=r.createElement("div");return o.setAttribute("id","headlessui-portal-root"),r.body.appendChild(o)});return(0,c.useEffect)(()=>{null!==o&&(null!=r&&r.body.contains(o)||null==r||r.body.appendChild(o))},[o,r]),(0,c.useEffect)(()=>{t||null!==n&&a(n.current)},[n,a,t]),o}(a),[s]=(0,c.useState)(()=>{var e;return w.O.isServer?null:null!=(e=null==l?void 0:l.createElement("div"))?e:null}),d=(0,c.useContext)(D),M=(0,g.H)();return(0,v.e)(()=>{!u||!s||u.contains(s)||(s.setAttribute("data-headlessui-portal",""),u.appendChild(s))},[u,s]),(0,v.e)(()=>{if(s&&d)return d.register(s)},[d,s]),n=()=>{var e;u&&s&&(s instanceof Node&&u.contains(s)&&u.removeChild(s),u.childNodes.length<=0&&(null==(e=u.parentElement)||e.removeChild(u)))},r=(0,m.z)(n),o=(0,c.useRef)(!1),(0,c.useEffect)(()=>(o.current=!1,()=>{o.current=!0,(0,h.Y)(()=>{o.current&&r()})}),[r]),M&&u&&s?(0,f.createPortal)((0,x.sY)({ourProps:{ref:i},theirProps:e,defaultTag:k,name:"Portal"}),s):null}),{Group:(0,x.yV)(function(e,t){let{target:n,...r}=e,o={ref:(0,b.T)(t)};return c.createElement(C.Provider,{value:n},(0,x.sY)({ourProps:o,theirProps:r,defaultTag:M,name:"Popover.Group"}))})});var T=n(31948),N=n(17684),P=n(32539),E=n(80004),S=n(38198),_=n(3141),j=((r=j||{})[r.Forwards=0]="Forwards",r[r.Backwards=1]="Backwards",r);function O(){let e=(0,c.useRef)(0);return(0,_.s)("keydown",t=>{"Tab"===t.key&&(e.current=t.shiftKey?1:0)},!0),e}var Z=n(37863),L=n(47634),Y=n(37105),F=n(24536),W=n(40293),R=n(37388),I=((o=I||{})[o.Open=0]="Open",o[o.Closed=1]="Closed",o),U=((a=U||{})[a.TogglePopover=0]="TogglePopover",a[a.ClosePopover=1]="ClosePopover",a[a.SetButton=2]="SetButton",a[a.SetButtonId=3]="SetButtonId",a[a.SetPanel=4]="SetPanel",a[a.SetPanelId=5]="SetPanelId",a);let H={0:e=>{let t={...e,popoverState:(0,F.E)(e.popoverState,{0:1,1:0})};return 0===t.popoverState&&(t.__demoMode=!1),t},1:e=>1===e.popoverState?e:{...e,popoverState:1},2:(e,t)=>e.button===t.button?e:{...e,button:t.button},3:(e,t)=>e.buttonId===t.buttonId?e:{...e,buttonId:t.buttonId},4:(e,t)=>e.panel===t.panel?e:{...e,panel:t.panel},5:(e,t)=>e.panelId===t.panelId?e:{...e,panelId:t.panelId}},B=(0,c.createContext)(null);function z(e){let t=(0,c.useContext)(B);if(null===t){let t=Error("<".concat(e," /> is missing a parent component."));throw Error.captureStackTrace&&Error.captureStackTrace(t,z),t}return t}B.displayName="PopoverContext";let A=(0,c.createContext)(null);function q(e){let t=(0,c.useContext)(A);if(null===t){let t=Error("<".concat(e," /> is missing a parent component."));throw Error.captureStackTrace&&Error.captureStackTrace(t,q),t}return t}A.displayName="PopoverAPIContext";let V=(0,c.createContext)(null);function G(){return(0,c.useContext)(V)}V.displayName="PopoverGroupContext";let X=(0,c.createContext)(null);function K(e,t){return(0,F.E)(t.type,H,e,t)}X.displayName="PopoverPanelContext";let Q=x.AN.RenderStrategy|x.AN.Static,J=x.AN.RenderStrategy|x.AN.Static,$=Object.assign((0,x.yV)(function(e,t){var n,r,o,a;let i,l,u,s,d,f;let{__demoMode:v=!1,...h}=e,g=(0,c.useRef)(null),y=(0,b.T)(t,(0,b.h)(e=>{g.current=e})),w=(0,c.useRef)([]),k=(0,c.useReducer)(K,{__demoMode:v,popoverState:v?0:1,buttons:w,button:null,buttonId:null,panel:null,panelId:null,beforePanelSentinel:(0,c.createRef)(),afterPanelSentinel:(0,c.createRef)()}),[{popoverState:M,button:C,buttonId:N,panel:E,panelId:_,beforePanelSentinel:j,afterPanelSentinel:O},L]=k,W=(0,p.i)(null!=(n=g.current)?n:C),R=(0,c.useMemo)(()=>{if(!C||!E)return!1;for(let e of document.querySelectorAll("body > *"))if(Number(null==e?void 0:e.contains(C))^Number(null==e?void 0:e.contains(E)))return!0;let e=(0,Y.GO)(),t=e.indexOf(C),n=(t+e.length-1)%e.length,r=(t+1)%e.length,o=e[n],a=e[r];return!E.contains(o)&&!E.contains(a)},[C,E]),I=(0,T.E)(N),U=(0,T.E)(_),H=(0,c.useMemo)(()=>({buttonId:I,panelId:U,close:()=>L({type:1})}),[I,U,L]),z=G(),q=null==z?void 0:z.registerPopover,V=(0,m.z)(()=>{var e;return null!=(e=null==z?void 0:z.isFocusWithinPopoverGroup())?e:(null==W?void 0:W.activeElement)&&((null==C?void 0:C.contains(W.activeElement))||(null==E?void 0:E.contains(W.activeElement)))});(0,c.useEffect)(()=>null==q?void 0:q(H),[q,H]);let[Q,J]=(i=(0,c.useContext)(D),l=(0,c.useRef)([]),u=(0,m.z)(e=>(l.current.push(e),i&&i.register(e),()=>s(e))),s=(0,m.z)(e=>{let t=l.current.indexOf(e);-1!==t&&l.current.splice(t,1),i&&i.unregister(e)}),d=(0,c.useMemo)(()=>({register:u,unregister:s,portals:l}),[u,s,l]),[l,(0,c.useMemo)(()=>function(e){let{children:t}=e;return c.createElement(D.Provider,{value:d},t)},[d])]),$=function(){var e;let{defaultContainers:t=[],portals:n,mainTreeNodeRef:r}=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{},o=(0,c.useRef)(null!=(e=null==r?void 0:r.current)?e:null),a=(0,p.i)(o),i=(0,m.z)(()=>{var e,r,i;let l=[];for(let e of t)null!==e&&(e instanceof HTMLElement?l.push(e):"current"in e&&e.current instanceof HTMLElement&&l.push(e.current));if(null!=n&&n.current)for(let e of n.current)l.push(e);for(let t of null!=(e=null==a?void 0:a.querySelectorAll("html > *, body > *"))?e:[])t!==document.body&&t!==document.head&&t instanceof HTMLElement&&"headlessui-portal-root"!==t.id&&(t.contains(o.current)||t.contains(null==(i=null==(r=o.current)?void 0:r.getRootNode())?void 0:i.host)||l.some(e=>t.contains(e))||l.push(t));return l});return{resolveContainers:i,contains:(0,m.z)(e=>i().some(t=>t.contains(e))),mainTreeNodeRef:o,MainTreeNode:(0,c.useMemo)(()=>function(){return null!=r?null:c.createElement(S._,{features:S.A.Hidden,ref:o})},[o,r])}}({mainTreeNodeRef:null==z?void 0:z.mainTreeNodeRef,portals:Q,defaultContainers:[C,E]});r=null==W?void 0:W.defaultView,o="focus",a=e=>{var t,n,r,o;e.target!==window&&e.target instanceof HTMLElement&&0===M&&(V()||C&&E&&($.contains(e.target)||null!=(n=null==(t=j.current)?void 0:t.contains)&&n.call(t,e.target)||null!=(o=null==(r=O.current)?void 0:r.contains)&&o.call(r,e.target)||L({type:1})))},f=(0,T.E)(a),(0,c.useEffect)(()=>{function e(e){f.current(e)}return(r=null!=r?r:window).addEventListener(o,e,!0),()=>r.removeEventListener(o,e,!0)},[r,o,!0]),(0,P.O)($.resolveContainers,(e,t)=>{L({type:1}),(0,Y.sP)(t,Y.tJ.Loose)||(e.preventDefault(),null==C||C.focus())},0===M);let ee=(0,m.z)(e=>{L({type:1});let t=e?e instanceof HTMLElement?e:"current"in e&&e.current instanceof HTMLElement?e.current:C:C;null==t||t.focus()}),et=(0,c.useMemo)(()=>({close:ee,isPortalled:R}),[ee,R]),en=(0,c.useMemo)(()=>({open:0===M,close:ee}),[M,ee]);return c.createElement(X.Provider,{value:null},c.createElement(B.Provider,{value:k},c.createElement(A.Provider,{value:et},c.createElement(Z.up,{value:(0,F.E)(M,{0:Z.ZM.Open,1:Z.ZM.Closed})},c.createElement(J,null,(0,x.sY)({ourProps:{ref:y},theirProps:h,slot:en,defaultTag:"div",name:"Popover"}),c.createElement($.MainTreeNode,null))))))}),{Button:(0,x.yV)(function(e,t){let n=(0,N.M)(),{id:r="headlessui-popover-button-".concat(n),...o}=e,[a,i]=z("Popover.Button"),{isPortalled:l}=q("Popover.Button"),u=(0,c.useRef)(null),s="headlessui-focus-sentinel-".concat((0,N.M)()),d=G(),f=null==d?void 0:d.closeOthers,v=null!==(0,c.useContext)(X);(0,c.useEffect)(()=>{if(!v)return i({type:3,buttonId:r}),()=>{i({type:3,buttonId:null})}},[v,r,i]);let[h]=(0,c.useState)(()=>Symbol()),g=(0,b.T)(u,t,v?null:e=>{if(e)a.buttons.current.push(h);else{let e=a.buttons.current.indexOf(h);-1!==e&&a.buttons.current.splice(e,1)}a.buttons.current.length>1&&console.warn("You are already using a but only 1 is supported."),e&&i({type:2,button:e})}),y=(0,b.T)(u,t),w=(0,p.i)(u),k=(0,m.z)(e=>{var t,n,r;if(v){if(1===a.popoverState)return;switch(e.key){case R.R.Space:case R.R.Enter:e.preventDefault(),null==(n=(t=e.target).click)||n.call(t),i({type:1}),null==(r=a.button)||r.focus()}}else switch(e.key){case R.R.Space:case R.R.Enter:e.preventDefault(),e.stopPropagation(),1===a.popoverState&&(null==f||f(a.buttonId)),i({type:0});break;case R.R.Escape:if(0!==a.popoverState)return null==f?void 0:f(a.buttonId);if(!u.current||null!=w&&w.activeElement&&!u.current.contains(w.activeElement))return;e.preventDefault(),e.stopPropagation(),i({type:1})}}),M=(0,m.z)(e=>{v||e.key===R.R.Space&&e.preventDefault()}),C=(0,m.z)(t=>{var n,r;(0,L.P)(t.currentTarget)||e.disabled||(v?(i({type:1}),null==(n=a.button)||n.focus()):(t.preventDefault(),t.stopPropagation(),1===a.popoverState&&(null==f||f(a.buttonId)),i({type:0}),null==(r=a.button)||r.focus()))}),D=(0,m.z)(e=>{e.preventDefault(),e.stopPropagation()}),T=0===a.popoverState,P=(0,c.useMemo)(()=>({open:T}),[T]),_=(0,E.f)(e,u),Z=v?{ref:y,type:_,onKeyDown:k,onClick:C}:{ref:g,id:a.buttonId,type:_,"aria-expanded":0===a.popoverState,"aria-controls":a.panel?a.panelId:void 0,onKeyDown:k,onKeyUp:M,onClick:C,onMouseDown:D},W=O(),I=(0,m.z)(()=>{let e=a.panel;e&&(0,F.E)(W.current,{[j.Forwards]:()=>(0,Y.jA)(e,Y.TO.First),[j.Backwards]:()=>(0,Y.jA)(e,Y.TO.Last)})===Y.fE.Error&&(0,Y.jA)((0,Y.GO)().filter(e=>"true"!==e.dataset.headlessuiFocusGuard),(0,F.E)(W.current,{[j.Forwards]:Y.TO.Next,[j.Backwards]:Y.TO.Previous}),{relativeTo:a.button})});return c.createElement(c.Fragment,null,(0,x.sY)({ourProps:Z,theirProps:o,slot:P,defaultTag:"button",name:"Popover.Button"}),T&&!v&&l&&c.createElement(S._,{id:s,features:S.A.Focusable,"data-headlessui-focus-guard":!0,as:"button",type:"button",onFocus:I}))}),Overlay:(0,x.yV)(function(e,t){let n=(0,N.M)(),{id:r="headlessui-popover-overlay-".concat(n),...o}=e,[{popoverState:a},i]=z("Popover.Overlay"),l=(0,b.T)(t),u=(0,Z.oJ)(),s=null!==u?(u&Z.ZM.Open)===Z.ZM.Open:0===a,d=(0,m.z)(e=>{if((0,L.P)(e.currentTarget))return e.preventDefault();i({type:1})}),f=(0,c.useMemo)(()=>({open:0===a}),[a]);return(0,x.sY)({ourProps:{ref:l,id:r,"aria-hidden":!0,onClick:d},theirProps:o,slot:f,defaultTag:"div",features:Q,visible:s,name:"Popover.Overlay"})}),Panel:(0,x.yV)(function(e,t){let n=(0,N.M)(),{id:r="headlessui-popover-panel-".concat(n),focus:o=!1,...a}=e,[i,l]=z("Popover.Panel"),{close:u,isPortalled:s}=q("Popover.Panel"),d="headlessui-focus-sentinel-before-".concat((0,N.M)()),f="headlessui-focus-sentinel-after-".concat((0,N.M)()),h=(0,c.useRef)(null),g=(0,b.T)(h,t,e=>{l({type:4,panel:e})}),y=(0,p.i)(h),w=(0,x.Y2)();(0,v.e)(()=>(l({type:5,panelId:r}),()=>{l({type:5,panelId:null})}),[r,l]);let k=(0,Z.oJ)(),M=null!==k?(k&Z.ZM.Open)===Z.ZM.Open:0===i.popoverState,C=(0,m.z)(e=>{var t;if(e.key===R.R.Escape){if(0!==i.popoverState||!h.current||null!=y&&y.activeElement&&!h.current.contains(y.activeElement))return;e.preventDefault(),e.stopPropagation(),l({type:1}),null==(t=i.button)||t.focus()}});(0,c.useEffect)(()=>{var t;e.static||1===i.popoverState&&(null==(t=e.unmount)||t)&&l({type:4,panel:null})},[i.popoverState,e.unmount,e.static,l]),(0,c.useEffect)(()=>{if(i.__demoMode||!o||0!==i.popoverState||!h.current)return;let e=null==y?void 0:y.activeElement;h.current.contains(e)||(0,Y.jA)(h.current,Y.TO.First)},[i.__demoMode,o,h,i.popoverState]);let D=(0,c.useMemo)(()=>({open:0===i.popoverState,close:u}),[i,u]),T={ref:g,id:r,onKeyDown:C,onBlur:o&&0===i.popoverState?e=>{var t,n,r,o,a;let u=e.relatedTarget;u&&h.current&&(null!=(t=h.current)&&t.contains(u)||(l({type:1}),(null!=(r=null==(n=i.beforePanelSentinel.current)?void 0:n.contains)&&r.call(n,u)||null!=(a=null==(o=i.afterPanelSentinel.current)?void 0:o.contains)&&a.call(o,u))&&u.focus({preventScroll:!0})))}:void 0,tabIndex:-1},P=O(),E=(0,m.z)(()=>{let e=h.current;e&&(0,F.E)(P.current,{[j.Forwards]:()=>{var t;(0,Y.jA)(e,Y.TO.First)===Y.fE.Error&&(null==(t=i.afterPanelSentinel.current)||t.focus())},[j.Backwards]:()=>{var e;null==(e=i.button)||e.focus({preventScroll:!0})}})}),_=(0,m.z)(()=>{let e=h.current;e&&(0,F.E)(P.current,{[j.Forwards]:()=>{var e;if(!i.button)return;let t=(0,Y.GO)(),n=t.indexOf(i.button),r=t.slice(0,n+1),o=[...t.slice(n+1),...r];for(let t of o.slice())if("true"===t.dataset.headlessuiFocusGuard||null!=(e=i.panel)&&e.contains(t)){let e=o.indexOf(t);-1!==e&&o.splice(e,1)}(0,Y.jA)(o,Y.TO.First,{sorted:!1})},[j.Backwards]:()=>{var t;(0,Y.jA)(e,Y.TO.Previous)===Y.fE.Error&&(null==(t=i.button)||t.focus())}})});return c.createElement(X.Provider,{value:r},M&&s&&c.createElement(S._,{id:d,ref:i.beforePanelSentinel,features:S.A.Focusable,"data-headlessui-focus-guard":!0,as:"button",type:"button",onFocus:E}),(0,x.sY)({mergeRefs:w,ourProps:T,theirProps:a,slot:D,defaultTag:"div",features:J,visible:M,name:"Popover.Panel"}),M&&s&&c.createElement(S._,{id:f,ref:i.afterPanelSentinel,features:S.A.Focusable,"data-headlessui-focus-guard":!0,as:"button",type:"button",onFocus:_}))}),Group:(0,x.yV)(function(e,t){let n;let r=(0,c.useRef)(null),o=(0,b.T)(r,t),[a,i]=(0,c.useState)([]),l={mainTreeNodeRef:n=(0,c.useRef)(null),MainTreeNode:(0,c.useMemo)(()=>function(){return c.createElement(S._,{features:S.A.Hidden,ref:n})},[n])},u=(0,m.z)(e=>{i(t=>{let n=t.indexOf(e);if(-1!==n){let e=t.slice();return e.splice(n,1),e}return t})}),s=(0,m.z)(e=>(i(t=>[...t,e]),()=>u(e))),d=(0,m.z)(()=>{var e;let t=(0,W.r)(r);if(!t)return!1;let n=t.activeElement;return!!(null!=(e=r.current)&&e.contains(n))||a.some(e=>{var r,o;return(null==(r=t.getElementById(e.buttonId.current))?void 0:r.contains(n))||(null==(o=t.getElementById(e.panelId.current))?void 0:o.contains(n))})}),f=(0,m.z)(e=>{for(let t of a)t.buttonId.current!==e&&t.close()}),v=(0,c.useMemo)(()=>({registerPopover:s,unregisterPopover:u,isFocusWithinPopoverGroup:d,closeOthers:f,mainTreeNodeRef:l.mainTreeNodeRef}),[s,u,d,f,l.mainTreeNodeRef]),h=(0,c.useMemo)(()=>({}),[]);return c.createElement(V.Provider,{value:v},(0,x.sY)({ourProps:{ref:o},theirProps:e,slot:h,defaultTag:"div",name:"Popover.Group"}),c.createElement(l.MainTreeNode,null))})});var ee=n(33044),et=n(9528);let en=e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({},t,{xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 20 20",fill:"currentColor"}),c.createElement("path",{fillRule:"evenodd",d:"M6 2a1 1 0 00-1 1v1H4a2 2 0 00-2 2v10a2 2 0 002 2h12a2 2 0 002-2V6a2 2 0 00-2-2h-1V3a1 1 0 10-2 0v1H7V3a1 1 0 00-1-1zm0 5a1 1 0 000 2h8a1 1 0 100-2H6z",clipRule:"evenodd"}))};var er=n(4537),eo=n(99735),ea=n(7656);function ei(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e);return t.setHours(0,0,0,0),t}function el(){return ei(Date.now())}function eu(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e);return t.setDate(1),t.setHours(0,0,0,0),t}var es=n(97324),ed=n(96398),ec=n(41154);function ef(e){var t,n;if((0,ea.Z)(1,arguments),e&&"function"==typeof e.forEach)t=e;else{if("object"!==(0,ec.Z)(e)||null===e)return new Date(NaN);t=Array.prototype.slice.call(e)}return t.forEach(function(e){var t=(0,eo.Z)(e);(void 0===n||nt||isNaN(t.getDate()))&&(n=t)}),n||new Date(NaN)}var ev=n(25721),eh=n(47869);function ep(e,t){(0,ea.Z)(2,arguments);var n=(0,eh.Z)(t);return(0,ev.Z)(e,-n)}var eg=n(55463);function eb(e,t){if((0,ea.Z)(2,arguments),!t||"object"!==(0,ec.Z)(t))return new Date(NaN);var n=t.years?(0,eh.Z)(t.years):0,r=t.months?(0,eh.Z)(t.months):0,o=t.weeks?(0,eh.Z)(t.weeks):0,a=t.days?(0,eh.Z)(t.days):0,i=t.hours?(0,eh.Z)(t.hours):0,l=t.minutes?(0,eh.Z)(t.minutes):0,u=t.seconds?(0,eh.Z)(t.seconds):0;return new Date(ep(function(e,t){(0,ea.Z)(2,arguments);var n=(0,eh.Z)(t);return(0,eg.Z)(e,-n)}(e,r+12*n),a+7*o).getTime()-1e3*(u+60*(l+60*i)))}function ey(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e),n=new Date(0);return n.setFullYear(t.getFullYear(),0,1),n.setHours(0,0,0,0),n}function ew(e){return(0,ea.Z)(1,arguments),e instanceof Date||"object"===(0,ec.Z)(e)&&"[object Date]"===Object.prototype.toString.call(e)}function ex(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e),n=t.getUTCDay();return t.setUTCDate(t.getUTCDate()-((n<1?7:0)+n-1)),t.setUTCHours(0,0,0,0),t}function ek(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e),n=t.getUTCFullYear(),r=new Date(0);r.setUTCFullYear(n+1,0,4),r.setUTCHours(0,0,0,0);var o=ex(r),a=new Date(0);a.setUTCFullYear(n,0,4),a.setUTCHours(0,0,0,0);var i=ex(a);return t.getTime()>=o.getTime()?n+1:t.getTime()>=i.getTime()?n:n-1}var eM={};function eC(e,t){(0,ea.Z)(1,arguments);var n,r,o,a,i,l,u,s,d=(0,eh.Z)(null!==(n=null!==(r=null!==(o=null!==(a=null==t?void 0:t.weekStartsOn)&&void 0!==a?a:null==t?void 0:null===(i=t.locale)||void 0===i?void 0:null===(l=i.options)||void 0===l?void 0:l.weekStartsOn)&&void 0!==o?o:eM.weekStartsOn)&&void 0!==r?r:null===(u=eM.locale)||void 0===u?void 0:null===(s=u.options)||void 0===s?void 0:s.weekStartsOn)&&void 0!==n?n:0);if(!(d>=0&&d<=6))throw RangeError("weekStartsOn must be between 0 and 6 inclusively");var c=(0,eo.Z)(e),f=c.getUTCDay();return c.setUTCDate(c.getUTCDate()-((f=1&&f<=7))throw RangeError("firstWeekContainsDate must be between 1 and 7 inclusively");var m=new Date(0);m.setUTCFullYear(c+1,0,f),m.setUTCHours(0,0,0,0);var v=eC(m,t),h=new Date(0);h.setUTCFullYear(c,0,f),h.setUTCHours(0,0,0,0);var p=eC(h,t);return d.getTime()>=v.getTime()?c+1:d.getTime()>=p.getTime()?c:c-1}function eT(e,t){for(var n=Math.abs(e).toString();n.length0?n:1-n;return eT("yy"===t?r%100:r,t.length)},M:function(e,t){var n=e.getUTCMonth();return"M"===t?String(n+1):eT(n+1,2)},d:function(e,t){return eT(e.getUTCDate(),t.length)},h:function(e,t){return eT(e.getUTCHours()%12||12,t.length)},H:function(e,t){return eT(e.getUTCHours(),t.length)},m:function(e,t){return eT(e.getUTCMinutes(),t.length)},s:function(e,t){return eT(e.getUTCSeconds(),t.length)},S:function(e,t){var n=t.length;return eT(Math.floor(e.getUTCMilliseconds()*Math.pow(10,n-3)),t.length)}},eP={midnight:"midnight",noon:"noon",morning:"morning",afternoon:"afternoon",evening:"evening",night:"night"};function eE(e,t){var n=e>0?"-":"+",r=Math.abs(e),o=Math.floor(r/60),a=r%60;return 0===a?n+String(o):n+String(o)+(t||"")+eT(a,2)}function eS(e,t){return e%60==0?(e>0?"-":"+")+eT(Math.abs(e)/60,2):e_(e,t)}function e_(e,t){var n=Math.abs(e);return(e>0?"-":"+")+eT(Math.floor(n/60),2)+(t||"")+eT(n%60,2)}var ej={G:function(e,t,n){var r=e.getUTCFullYear()>0?1:0;switch(t){case"G":case"GG":case"GGG":return n.era(r,{width:"abbreviated"});case"GGGGG":return n.era(r,{width:"narrow"});default:return n.era(r,{width:"wide"})}},y:function(e,t,n){if("yo"===t){var r=e.getUTCFullYear();return n.ordinalNumber(r>0?r:1-r,{unit:"year"})}return eN.y(e,t)},Y:function(e,t,n,r){var o=eD(e,r),a=o>0?o:1-o;return"YY"===t?eT(a%100,2):"Yo"===t?n.ordinalNumber(a,{unit:"year"}):eT(a,t.length)},R:function(e,t){return eT(ek(e),t.length)},u:function(e,t){return eT(e.getUTCFullYear(),t.length)},Q:function(e,t,n){var r=Math.ceil((e.getUTCMonth()+1)/3);switch(t){case"Q":return String(r);case"QQ":return eT(r,2);case"Qo":return n.ordinalNumber(r,{unit:"quarter"});case"QQQ":return n.quarter(r,{width:"abbreviated",context:"formatting"});case"QQQQQ":return n.quarter(r,{width:"narrow",context:"formatting"});default:return n.quarter(r,{width:"wide",context:"formatting"})}},q:function(e,t,n){var r=Math.ceil((e.getUTCMonth()+1)/3);switch(t){case"q":return String(r);case"qq":return eT(r,2);case"qo":return n.ordinalNumber(r,{unit:"quarter"});case"qqq":return n.quarter(r,{width:"abbreviated",context:"standalone"});case"qqqqq":return n.quarter(r,{width:"narrow",context:"standalone"});default:return n.quarter(r,{width:"wide",context:"standalone"})}},M:function(e,t,n){var r=e.getUTCMonth();switch(t){case"M":case"MM":return eN.M(e,t);case"Mo":return n.ordinalNumber(r+1,{unit:"month"});case"MMM":return n.month(r,{width:"abbreviated",context:"formatting"});case"MMMMM":return n.month(r,{width:"narrow",context:"formatting"});default:return n.month(r,{width:"wide",context:"formatting"})}},L:function(e,t,n){var r=e.getUTCMonth();switch(t){case"L":return String(r+1);case"LL":return eT(r+1,2);case"Lo":return n.ordinalNumber(r+1,{unit:"month"});case"LLL":return n.month(r,{width:"abbreviated",context:"standalone"});case"LLLLL":return n.month(r,{width:"narrow",context:"standalone"});default:return n.month(r,{width:"wide",context:"standalone"})}},w:function(e,t,n,r){var o=function(e,t){(0,ea.Z)(1,arguments);var n=(0,eo.Z)(e);return Math.round((eC(n,t).getTime()-(function(e,t){(0,ea.Z)(1,arguments);var n,r,o,a,i,l,u,s,d=(0,eh.Z)(null!==(n=null!==(r=null!==(o=null!==(a=null==t?void 0:t.firstWeekContainsDate)&&void 0!==a?a:null==t?void 0:null===(i=t.locale)||void 0===i?void 0:null===(l=i.options)||void 0===l?void 0:l.firstWeekContainsDate)&&void 0!==o?o:eM.firstWeekContainsDate)&&void 0!==r?r:null===(u=eM.locale)||void 0===u?void 0:null===(s=u.options)||void 0===s?void 0:s.firstWeekContainsDate)&&void 0!==n?n:1),c=eD(e,t),f=new Date(0);return f.setUTCFullYear(c,0,d),f.setUTCHours(0,0,0,0),eC(f,t)})(n,t).getTime())/6048e5)+1}(e,r);return"wo"===t?n.ordinalNumber(o,{unit:"week"}):eT(o,t.length)},I:function(e,t,n){var r=function(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e);return Math.round((ex(t).getTime()-(function(e){(0,ea.Z)(1,arguments);var t=ek(e),n=new Date(0);return n.setUTCFullYear(t,0,4),n.setUTCHours(0,0,0,0),ex(n)})(t).getTime())/6048e5)+1}(e);return"Io"===t?n.ordinalNumber(r,{unit:"week"}):eT(r,t.length)},d:function(e,t,n){return"do"===t?n.ordinalNumber(e.getUTCDate(),{unit:"date"}):eN.d(e,t)},D:function(e,t,n){var r=function(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e),n=t.getTime();return t.setUTCMonth(0,1),t.setUTCHours(0,0,0,0),Math.floor((n-t.getTime())/864e5)+1}(e);return"Do"===t?n.ordinalNumber(r,{unit:"dayOfYear"}):eT(r,t.length)},E:function(e,t,n){var r=e.getUTCDay();switch(t){case"E":case"EE":case"EEE":return n.day(r,{width:"abbreviated",context:"formatting"});case"EEEEE":return n.day(r,{width:"narrow",context:"formatting"});case"EEEEEE":return n.day(r,{width:"short",context:"formatting"});default:return n.day(r,{width:"wide",context:"formatting"})}},e:function(e,t,n,r){var o=e.getUTCDay(),a=(o-r.weekStartsOn+8)%7||7;switch(t){case"e":return String(a);case"ee":return eT(a,2);case"eo":return n.ordinalNumber(a,{unit:"day"});case"eee":return n.day(o,{width:"abbreviated",context:"formatting"});case"eeeee":return n.day(o,{width:"narrow",context:"formatting"});case"eeeeee":return n.day(o,{width:"short",context:"formatting"});default:return n.day(o,{width:"wide",context:"formatting"})}},c:function(e,t,n,r){var o=e.getUTCDay(),a=(o-r.weekStartsOn+8)%7||7;switch(t){case"c":return String(a);case"cc":return eT(a,t.length);case"co":return n.ordinalNumber(a,{unit:"day"});case"ccc":return n.day(o,{width:"abbreviated",context:"standalone"});case"ccccc":return n.day(o,{width:"narrow",context:"standalone"});case"cccccc":return n.day(o,{width:"short",context:"standalone"});default:return n.day(o,{width:"wide",context:"standalone"})}},i:function(e,t,n){var r=e.getUTCDay(),o=0===r?7:r;switch(t){case"i":return String(o);case"ii":return eT(o,t.length);case"io":return n.ordinalNumber(o,{unit:"day"});case"iii":return n.day(r,{width:"abbreviated",context:"formatting"});case"iiiii":return n.day(r,{width:"narrow",context:"formatting"});case"iiiiii":return n.day(r,{width:"short",context:"formatting"});default:return n.day(r,{width:"wide",context:"formatting"})}},a:function(e,t,n){var r=e.getUTCHours()/12>=1?"pm":"am";switch(t){case"a":case"aa":return n.dayPeriod(r,{width:"abbreviated",context:"formatting"});case"aaa":return n.dayPeriod(r,{width:"abbreviated",context:"formatting"}).toLowerCase();case"aaaaa":return n.dayPeriod(r,{width:"narrow",context:"formatting"});default:return n.dayPeriod(r,{width:"wide",context:"formatting"})}},b:function(e,t,n){var r,o=e.getUTCHours();switch(r=12===o?eP.noon:0===o?eP.midnight:o/12>=1?"pm":"am",t){case"b":case"bb":return n.dayPeriod(r,{width:"abbreviated",context:"formatting"});case"bbb":return n.dayPeriod(r,{width:"abbreviated",context:"formatting"}).toLowerCase();case"bbbbb":return n.dayPeriod(r,{width:"narrow",context:"formatting"});default:return n.dayPeriod(r,{width:"wide",context:"formatting"})}},B:function(e,t,n){var r,o=e.getUTCHours();switch(r=o>=17?eP.evening:o>=12?eP.afternoon:o>=4?eP.morning:eP.night,t){case"B":case"BB":case"BBB":return n.dayPeriod(r,{width:"abbreviated",context:"formatting"});case"BBBBB":return n.dayPeriod(r,{width:"narrow",context:"formatting"});default:return n.dayPeriod(r,{width:"wide",context:"formatting"})}},h:function(e,t,n){if("ho"===t){var r=e.getUTCHours()%12;return 0===r&&(r=12),n.ordinalNumber(r,{unit:"hour"})}return eN.h(e,t)},H:function(e,t,n){return"Ho"===t?n.ordinalNumber(e.getUTCHours(),{unit:"hour"}):eN.H(e,t)},K:function(e,t,n){var r=e.getUTCHours()%12;return"Ko"===t?n.ordinalNumber(r,{unit:"hour"}):eT(r,t.length)},k:function(e,t,n){var r=e.getUTCHours();return(0===r&&(r=24),"ko"===t)?n.ordinalNumber(r,{unit:"hour"}):eT(r,t.length)},m:function(e,t,n){return"mo"===t?n.ordinalNumber(e.getUTCMinutes(),{unit:"minute"}):eN.m(e,t)},s:function(e,t,n){return"so"===t?n.ordinalNumber(e.getUTCSeconds(),{unit:"second"}):eN.s(e,t)},S:function(e,t){return eN.S(e,t)},X:function(e,t,n,r){var o=(r._originalDate||e).getTimezoneOffset();if(0===o)return"Z";switch(t){case"X":return eS(o);case"XXXX":case"XX":return e_(o);default:return e_(o,":")}},x:function(e,t,n,r){var o=(r._originalDate||e).getTimezoneOffset();switch(t){case"x":return eS(o);case"xxxx":case"xx":return e_(o);default:return e_(o,":")}},O:function(e,t,n,r){var o=(r._originalDate||e).getTimezoneOffset();switch(t){case"O":case"OO":case"OOO":return"GMT"+eE(o,":");default:return"GMT"+e_(o,":")}},z:function(e,t,n,r){var o=(r._originalDate||e).getTimezoneOffset();switch(t){case"z":case"zz":case"zzz":return"GMT"+eE(o,":");default:return"GMT"+e_(o,":")}},t:function(e,t,n,r){return eT(Math.floor((r._originalDate||e).getTime()/1e3),t.length)},T:function(e,t,n,r){return eT((r._originalDate||e).getTime(),t.length)}},eO=function(e,t){switch(e){case"P":return t.date({width:"short"});case"PP":return t.date({width:"medium"});case"PPP":return t.date({width:"long"});default:return t.date({width:"full"})}},eZ=function(e,t){switch(e){case"p":return t.time({width:"short"});case"pp":return t.time({width:"medium"});case"ppp":return t.time({width:"long"});default:return t.time({width:"full"})}},eL={p:eZ,P:function(e,t){var n,r=e.match(/(P+)(p+)?/)||[],o=r[1],a=r[2];if(!a)return eO(e,t);switch(o){case"P":n=t.dateTime({width:"short"});break;case"PP":n=t.dateTime({width:"medium"});break;case"PPP":n=t.dateTime({width:"long"});break;default:n=t.dateTime({width:"full"})}return n.replace("{{date}}",eO(o,t)).replace("{{time}}",eZ(a,t))}};function eY(e){var t=new Date(Date.UTC(e.getFullYear(),e.getMonth(),e.getDate(),e.getHours(),e.getMinutes(),e.getSeconds(),e.getMilliseconds()));return t.setUTCFullYear(e.getFullYear()),e.getTime()-t.getTime()}var eF=["D","DD"],eW=["YY","YYYY"];function eR(e,t,n){if("YYYY"===e)throw RangeError("Use `yyyy` instead of `YYYY` (in `".concat(t,"`) for formatting years to the input `").concat(n,"`; see: https://github.com/date-fns/date-fns/blob/master/docs/unicodeTokens.md"));if("YY"===e)throw RangeError("Use `yy` instead of `YY` (in `".concat(t,"`) for formatting years to the input `").concat(n,"`; see: https://github.com/date-fns/date-fns/blob/master/docs/unicodeTokens.md"));if("D"===e)throw RangeError("Use `d` instead of `D` (in `".concat(t,"`) for formatting days of the month to the input `").concat(n,"`; see: https://github.com/date-fns/date-fns/blob/master/docs/unicodeTokens.md"));if("DD"===e)throw RangeError("Use `dd` instead of `DD` (in `".concat(t,"`) for formatting days of the month to the input `").concat(n,"`; see: https://github.com/date-fns/date-fns/blob/master/docs/unicodeTokens.md"))}var eI={lessThanXSeconds:{one:"less than a second",other:"less than {{count}} seconds"},xSeconds:{one:"1 second",other:"{{count}} seconds"},halfAMinute:"half a minute",lessThanXMinutes:{one:"less than a minute",other:"less than {{count}} minutes"},xMinutes:{one:"1 minute",other:"{{count}} minutes"},aboutXHours:{one:"about 1 hour",other:"about {{count}} hours"},xHours:{one:"1 hour",other:"{{count}} hours"},xDays:{one:"1 day",other:"{{count}} days"},aboutXWeeks:{one:"about 1 week",other:"about {{count}} weeks"},xWeeks:{one:"1 week",other:"{{count}} weeks"},aboutXMonths:{one:"about 1 month",other:"about {{count}} months"},xMonths:{one:"1 month",other:"{{count}} months"},aboutXYears:{one:"about 1 year",other:"about {{count}} years"},xYears:{one:"1 year",other:"{{count}} years"},overXYears:{one:"over 1 year",other:"over {{count}} years"},almostXYears:{one:"almost 1 year",other:"almost {{count}} years"}};function eU(e){return function(){var t=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{},n=t.width?String(t.width):e.defaultWidth;return e.formats[n]||e.formats[e.defaultWidth]}}var eH={date:eU({formats:{full:"EEEE, MMMM do, y",long:"MMMM do, y",medium:"MMM d, y",short:"MM/dd/yyyy"},defaultWidth:"full"}),time:eU({formats:{full:"h:mm:ss a zzzz",long:"h:mm:ss a z",medium:"h:mm:ss a",short:"h:mm a"},defaultWidth:"full"}),dateTime:eU({formats:{full:"{{date}} 'at' {{time}}",long:"{{date}} 'at' {{time}}",medium:"{{date}}, {{time}}",short:"{{date}}, {{time}}"},defaultWidth:"full"})},eB={lastWeek:"'last' eeee 'at' p",yesterday:"'yesterday at' p",today:"'today at' p",tomorrow:"'tomorrow at' p",nextWeek:"eeee 'at' p",other:"P"};function ez(e){return function(t,n){var r;if("formatting"===(null!=n&&n.context?String(n.context):"standalone")&&e.formattingValues){var o=e.defaultFormattingWidth||e.defaultWidth,a=null!=n&&n.width?String(n.width):o;r=e.formattingValues[a]||e.formattingValues[o]}else{var i=e.defaultWidth,l=null!=n&&n.width?String(n.width):e.defaultWidth;r=e.values[l]||e.values[i]}return r[e.argumentCallback?e.argumentCallback(t):t]}}function eA(e){return function(t){var n,r=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},o=r.width,a=o&&e.matchPatterns[o]||e.matchPatterns[e.defaultMatchWidth],i=t.match(a);if(!i)return null;var l=i[0],u=o&&e.parsePatterns[o]||e.parsePatterns[e.defaultParseWidth],s=Array.isArray(u)?function(e,t){for(var n=0;n0?"in "+r:r+" ago":r},formatLong:eH,formatRelative:function(e,t,n,r){return eB[e]},localize:{ordinalNumber:function(e,t){var n=Number(e),r=n%100;if(r>20||r<10)switch(r%10){case 1:return n+"st";case 2:return n+"nd";case 3:return n+"rd"}return n+"th"},era:ez({values:{narrow:["B","A"],abbreviated:["BC","AD"],wide:["Before Christ","Anno Domini"]},defaultWidth:"wide"}),quarter:ez({values:{narrow:["1","2","3","4"],abbreviated:["Q1","Q2","Q3","Q4"],wide:["1st quarter","2nd quarter","3rd quarter","4th quarter"]},defaultWidth:"wide",argumentCallback:function(e){return e-1}}),month:ez({values:{narrow:["J","F","M","A","M","J","J","A","S","O","N","D"],abbreviated:["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"],wide:["January","February","March","April","May","June","July","August","September","October","November","December"]},defaultWidth:"wide"}),day:ez({values:{narrow:["S","M","T","W","T","F","S"],short:["Su","Mo","Tu","We","Th","Fr","Sa"],abbreviated:["Sun","Mon","Tue","Wed","Thu","Fri","Sat"],wide:["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"]},defaultWidth:"wide"}),dayPeriod:ez({values:{narrow:{am:"a",pm:"p",midnight:"mi",noon:"n",morning:"morning",afternoon:"afternoon",evening:"evening",night:"night"},abbreviated:{am:"AM",pm:"PM",midnight:"midnight",noon:"noon",morning:"morning",afternoon:"afternoon",evening:"evening",night:"night"},wide:{am:"a.m.",pm:"p.m.",midnight:"midnight",noon:"noon",morning:"morning",afternoon:"afternoon",evening:"evening",night:"night"}},defaultWidth:"wide",formattingValues:{narrow:{am:"a",pm:"p",midnight:"mi",noon:"n",morning:"in the morning",afternoon:"in the afternoon",evening:"in the evening",night:"at night"},abbreviated:{am:"AM",pm:"PM",midnight:"midnight",noon:"noon",morning:"in the morning",afternoon:"in the afternoon",evening:"in the evening",night:"at night"},wide:{am:"a.m.",pm:"p.m.",midnight:"midnight",noon:"noon",morning:"in the morning",afternoon:"in the afternoon",evening:"in the evening",night:"at night"}},defaultFormattingWidth:"wide"})},match:{ordinalNumber:(i={matchPattern:/^(\d+)(th|st|nd|rd)?/i,parsePattern:/\d+/i,valueCallback:function(e){return parseInt(e,10)}},function(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},n=e.match(i.matchPattern);if(!n)return null;var r=n[0],o=e.match(i.parsePattern);if(!o)return null;var a=i.valueCallback?i.valueCallback(o[0]):o[0];return{value:a=t.valueCallback?t.valueCallback(a):a,rest:e.slice(r.length)}}),era:eA({matchPatterns:{narrow:/^(b|a)/i,abbreviated:/^(b\.?\s?c\.?|b\.?\s?c\.?\s?e\.?|a\.?\s?d\.?|c\.?\s?e\.?)/i,wide:/^(before christ|before common era|anno domini|common era)/i},defaultMatchWidth:"wide",parsePatterns:{any:[/^b/i,/^(a|c)/i]},defaultParseWidth:"any"}),quarter:eA({matchPatterns:{narrow:/^[1234]/i,abbreviated:/^q[1234]/i,wide:/^[1234](th|st|nd|rd)? quarter/i},defaultMatchWidth:"wide",parsePatterns:{any:[/1/i,/2/i,/3/i,/4/i]},defaultParseWidth:"any",valueCallback:function(e){return e+1}}),month:eA({matchPatterns:{narrow:/^[jfmasond]/i,abbreviated:/^(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)/i,wide:/^(january|february|march|april|may|june|july|august|september|october|november|december)/i},defaultMatchWidth:"wide",parsePatterns:{narrow:[/^j/i,/^f/i,/^m/i,/^a/i,/^m/i,/^j/i,/^j/i,/^a/i,/^s/i,/^o/i,/^n/i,/^d/i],any:[/^ja/i,/^f/i,/^mar/i,/^ap/i,/^may/i,/^jun/i,/^jul/i,/^au/i,/^s/i,/^o/i,/^n/i,/^d/i]},defaultParseWidth:"any"}),day:eA({matchPatterns:{narrow:/^[smtwf]/i,short:/^(su|mo|tu|we|th|fr|sa)/i,abbreviated:/^(sun|mon|tue|wed|thu|fri|sat)/i,wide:/^(sunday|monday|tuesday|wednesday|thursday|friday|saturday)/i},defaultMatchWidth:"wide",parsePatterns:{narrow:[/^s/i,/^m/i,/^t/i,/^w/i,/^t/i,/^f/i,/^s/i],any:[/^su/i,/^m/i,/^tu/i,/^w/i,/^th/i,/^f/i,/^sa/i]},defaultParseWidth:"any"}),dayPeriod:eA({matchPatterns:{narrow:/^(a|p|mi|n|(in the|at) (morning|afternoon|evening|night))/i,any:/^([ap]\.?\s?m\.?|midnight|noon|(in the|at) (morning|afternoon|evening|night))/i},defaultMatchWidth:"any",parsePatterns:{any:{am:/^a/i,pm:/^p/i,midnight:/^mi/i,noon:/^no/i,morning:/morning/i,afternoon:/afternoon/i,evening:/evening/i,night:/night/i}},defaultParseWidth:"any"})},options:{weekStartsOn:0,firstWeekContainsDate:1}},eV=/[yYQqMLwIdDecihHKkms]o|(\w)\1*|''|'(''|[^'])+('|$)|./g,eG=/P+p+|P+|p+|''|'(''|[^'])+('|$)|./g,eX=/^'([^]*?)'?$/,eK=/''/g,eQ=/[a-zA-Z]/;function eJ(e,t,n){(0,ea.Z)(2,arguments);var r,o,a,i,l,u,s,d,c,f,m,v,h,p,g,b,y,w,x=String(t),k=null!==(r=null!==(o=null==n?void 0:n.locale)&&void 0!==o?o:eM.locale)&&void 0!==r?r:eq,M=(0,eh.Z)(null!==(a=null!==(i=null!==(l=null!==(u=null==n?void 0:n.firstWeekContainsDate)&&void 0!==u?u:null==n?void 0:null===(s=n.locale)||void 0===s?void 0:null===(d=s.options)||void 0===d?void 0:d.firstWeekContainsDate)&&void 0!==l?l:eM.firstWeekContainsDate)&&void 0!==i?i:null===(c=eM.locale)||void 0===c?void 0:null===(f=c.options)||void 0===f?void 0:f.firstWeekContainsDate)&&void 0!==a?a:1);if(!(M>=1&&M<=7))throw RangeError("firstWeekContainsDate must be between 1 and 7 inclusively");var C=(0,eh.Z)(null!==(m=null!==(v=null!==(h=null!==(p=null==n?void 0:n.weekStartsOn)&&void 0!==p?p:null==n?void 0:null===(g=n.locale)||void 0===g?void 0:null===(b=g.options)||void 0===b?void 0:b.weekStartsOn)&&void 0!==h?h:eM.weekStartsOn)&&void 0!==v?v:null===(y=eM.locale)||void 0===y?void 0:null===(w=y.options)||void 0===w?void 0:w.weekStartsOn)&&void 0!==m?m:0);if(!(C>=0&&C<=6))throw RangeError("weekStartsOn must be between 0 and 6 inclusively");if(!k.localize)throw RangeError("locale must contain localize property");if(!k.formatLong)throw RangeError("locale must contain formatLong property");var D=(0,eo.Z)(e);if(!function(e){return(0,ea.Z)(1,arguments),(!!ew(e)||"number"==typeof e)&&!isNaN(Number((0,eo.Z)(e)))}(D))throw RangeError("Invalid time value");var T=eY(D),N=function(e,t){return(0,ea.Z)(2,arguments),function(e,t){return(0,ea.Z)(2,arguments),new Date((0,eo.Z)(e).getTime()+(0,eh.Z)(t))}(e,-(0,eh.Z)(t))}(D,T),P={firstWeekContainsDate:M,weekStartsOn:C,locale:k,_originalDate:D};return x.match(eG).map(function(e){var t=e[0];return"p"===t||"P"===t?(0,eL[t])(e,k.formatLong):e}).join("").match(eV).map(function(r){if("''"===r)return"'";var o,a=r[0];if("'"===a)return(o=r.match(eX))?o[1].replace(eK,"'"):r;var i=ej[a];if(i)return null!=n&&n.useAdditionalWeekYearTokens||-1===eW.indexOf(r)||eR(r,t,String(e)),null!=n&&n.useAdditionalDayOfYearTokens||-1===eF.indexOf(r)||eR(r,t,String(e)),i(N,r,k.localize,P);if(a.match(eQ))throw RangeError("Format string contains an unescaped latin alphabet character `"+a+"`");return r}).join("")}var e$=n(1153);let e0=(0,e$.fn)("DateRangePicker"),e1=(e,t,n,r)=>{var o;if(n&&(e=null===(o=r.get(n))||void 0===o?void 0:o.from),e)return ei(e&&!t?e:ef([e,t]))},e2=(e,t,n,r)=>{var o,a;if(n&&(e=ei(null!==(a=null===(o=r.get(n))||void 0===o?void 0:o.to)&&void 0!==a?a:el())),e)return ei(e&&!t?e:em([e,t]))},e4=[{value:"tdy",text:"Today",from:el()},{value:"w",text:"Last 7 days",from:eb(el(),{days:7})},{value:"t",text:"Last 30 days",from:eb(el(),{days:30})},{value:"m",text:"Month to Date",from:eu(el())},{value:"y",text:"Year to Date",from:ey(el())}],e3=(e,t,n,r)=>{let o=(null==n?void 0:n.code)||"en-US";if(!e&&!t)return"";if(e&&!t)return r?eJ(e,r):e.toLocaleDateString(o,{year:"numeric",month:"short",day:"numeric"});if(e&&t){if(function(e,t){(0,ea.Z)(2,arguments);var n=(0,eo.Z)(e),r=(0,eo.Z)(t);return n.getTime()===r.getTime()}(e,t))return r?eJ(e,r):e.toLocaleDateString(o,{year:"numeric",month:"short",day:"numeric"});if(e.getMonth()===t.getMonth()&&e.getFullYear()===t.getFullYear())return r?"".concat(eJ(e,r)," - ").concat(eJ(t,r)):"".concat(e.toLocaleDateString(o,{month:"short",day:"numeric"})," - \n ").concat(t.getDate(),", ").concat(t.getFullYear());{if(r)return"".concat(eJ(e,r)," - ").concat(eJ(t,r));let n={year:"numeric",month:"short",day:"numeric"};return"".concat(e.toLocaleDateString(o,n)," - \n ").concat(t.toLocaleDateString(o,n))}}return""};function e8(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e),n=t.getMonth();return t.setFullYear(t.getFullYear(),n+1,0),t.setHours(23,59,59,999),t}function e6(e,t){(0,ea.Z)(2,arguments);var n=(0,eo.Z)(e),r=(0,eh.Z)(t),o=n.getFullYear(),a=n.getDate(),i=new Date(0);i.setFullYear(o,r,15),i.setHours(0,0,0,0);var l=function(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e),n=t.getFullYear(),r=t.getMonth(),o=new Date(0);return o.setFullYear(n,r+1,0),o.setHours(0,0,0,0),o.getDate()}(i);return n.setMonth(r,Math.min(a,l)),n}function e5(e,t){(0,ea.Z)(2,arguments);var n=(0,eo.Z)(e),r=(0,eh.Z)(t);return isNaN(n.getTime())?new Date(NaN):(n.setFullYear(r),n)}function e7(e,t){(0,ea.Z)(2,arguments);var n=(0,eo.Z)(e),r=(0,eo.Z)(t);return 12*(n.getFullYear()-r.getFullYear())+(n.getMonth()-r.getMonth())}function e9(e,t){(0,ea.Z)(2,arguments);var n=(0,eo.Z)(e),r=(0,eo.Z)(t);return n.getFullYear()===r.getFullYear()&&n.getMonth()===r.getMonth()}function te(e,t){(0,ea.Z)(2,arguments);var n=(0,eo.Z)(e),r=(0,eo.Z)(t);return n.getTime()=0&&d<=6))throw RangeError("weekStartsOn must be between 0 and 6 inclusively");var c=(0,eo.Z)(e),f=c.getDay();return c.setDate(c.getDate()-((fr.getTime()}function ta(e,t){(0,ea.Z)(2,arguments);var n=ei(e),r=ei(t);return Math.round((n.getTime()-eY(n)-(r.getTime()-eY(r)))/864e5)}function ti(e,t){(0,ea.Z)(2,arguments);var n=(0,eh.Z)(t);return(0,ev.Z)(e,7*n)}function tl(e,t){(0,ea.Z)(2,arguments);var n=(0,eh.Z)(t);return(0,eg.Z)(e,12*n)}function tu(e,t){(0,ea.Z)(1,arguments);var n,r,o,a,i,l,u,s,d=(0,eh.Z)(null!==(n=null!==(r=null!==(o=null!==(a=null==t?void 0:t.weekStartsOn)&&void 0!==a?a:null==t?void 0:null===(i=t.locale)||void 0===i?void 0:null===(l=i.options)||void 0===l?void 0:l.weekStartsOn)&&void 0!==o?o:eM.weekStartsOn)&&void 0!==r?r:null===(u=eM.locale)||void 0===u?void 0:null===(s=u.options)||void 0===s?void 0:s.weekStartsOn)&&void 0!==n?n:0);if(!(d>=0&&d<=6))throw RangeError("weekStartsOn must be between 0 and 6 inclusively");var c=(0,eo.Z)(e),f=c.getDay();return c.setDate(c.getDate()+((fe7(l,i)&&(i=(0,eg.Z)(l,-1*((void 0===s?1:s)-1))),u&&0>e7(i,u)&&(i=u),d=eu(i),f=t.month,v=(m=(0,c.useState)(d))[0],h=[void 0===f?v:f,m[1]])[0],g=h[1],[p,function(e){if(!t.disableNavigation){var n,r=eu(e);g(r),null===(n=t.onMonthChange)||void 0===n||n.call(t,r)}}]),w=y[0],x=y[1],k=function(e,t){for(var n=t.reverseMonths,r=t.numberOfMonths,o=eu(e),a=e7(eu((0,eg.Z)(o,r)),o),i=[],l=0;l=e7(a,n)))return(0,eg.Z)(a,-(r?void 0===o?1:o:1))}}(w,b),D=function(e){return k.some(function(t){return e9(e,t)})};return tv.jsx(tE.Provider,{value:{currentMonth:w,displayMonths:k,goToMonth:x,goToDate:function(e,t){D(e)||(t&&te(e,t)?x((0,eg.Z)(e,1+-1*b.numberOfMonths)):x(e))},previousMonth:C,nextMonth:M,isDateDisplayed:D},children:e.children})}function t_(){var e=(0,c.useContext)(tE);if(!e)throw Error("useNavigation must be used within a NavigationProvider");return e}function tj(e){var t,n=tM(),r=n.classNames,o=n.styles,a=n.components,i=t_().goToMonth,l=function(t){i((0,eg.Z)(t,e.displayIndex?-e.displayIndex:0))},u=null!==(t=null==a?void 0:a.CaptionLabel)&&void 0!==t?t:tC,s=tv.jsx(u,{id:e.id,displayMonth:e.displayMonth});return tv.jsxs("div",{className:r.caption_dropdowns,style:o.caption_dropdowns,children:[tv.jsx("div",{className:r.vhidden,children:s}),tv.jsx(tN,{onChange:l,displayMonth:e.displayMonth}),tv.jsx(tP,{onChange:l,displayMonth:e.displayMonth})]})}function tO(e){return tv.jsx("svg",td({width:"16px",height:"16px",viewBox:"0 0 120 120"},e,{children:tv.jsx("path",{d:"M69.490332,3.34314575 C72.6145263,0.218951416 77.6798462,0.218951416 80.8040405,3.34314575 C83.8617626,6.40086786 83.9268205,11.3179931 80.9992143,14.4548388 L80.8040405,14.6568542 L35.461,60 L80.8040405,105.343146 C83.8617626,108.400868 83.9268205,113.317993 80.9992143,116.454839 L80.8040405,116.656854 C77.7463184,119.714576 72.8291931,119.779634 69.6923475,116.852028 L69.490332,116.656854 L18.490332,65.6568542 C15.4326099,62.5991321 15.367552,57.6820069 18.2951583,54.5451612 L18.490332,54.3431458 L69.490332,3.34314575 Z",fill:"currentColor",fillRule:"nonzero"})}))}function tZ(e){return tv.jsx("svg",td({width:"16px",height:"16px",viewBox:"0 0 120 120"},e,{children:tv.jsx("path",{d:"M49.8040405,3.34314575 C46.6798462,0.218951416 41.6145263,0.218951416 38.490332,3.34314575 C35.4326099,6.40086786 35.367552,11.3179931 38.2951583,14.4548388 L38.490332,14.6568542 L83.8333725,60 L38.490332,105.343146 C35.4326099,108.400868 35.367552,113.317993 38.2951583,116.454839 L38.490332,116.656854 C41.5480541,119.714576 46.4651794,119.779634 49.602025,116.852028 L49.8040405,116.656854 L100.804041,65.6568542 C103.861763,62.5991321 103.926821,57.6820069 100.999214,54.5451612 L100.804041,54.3431458 L49.8040405,3.34314575 Z",fill:"currentColor"})}))}var tL=(0,c.forwardRef)(function(e,t){var n=tM(),r=n.classNames,o=n.styles,a=[r.button_reset,r.button];e.className&&a.push(e.className);var i=a.join(" "),l=td(td({},o.button_reset),o.button);return e.style&&Object.assign(l,e.style),tv.jsx("button",td({},e,{ref:t,type:"button",className:i,style:l}))});function tY(e){var t,n,r=tM(),o=r.dir,a=r.locale,i=r.classNames,l=r.styles,u=r.labels,s=u.labelPrevious,d=u.labelNext,c=r.components;if(!e.nextMonth&&!e.previousMonth)return tv.jsx(tv.Fragment,{});var f=s(e.previousMonth,{locale:a}),m=[i.nav_button,i.nav_button_previous].join(" "),v=d(e.nextMonth,{locale:a}),h=[i.nav_button,i.nav_button_next].join(" "),p=null!==(t=null==c?void 0:c.IconRight)&&void 0!==t?t:tZ,g=null!==(n=null==c?void 0:c.IconLeft)&&void 0!==n?n:tO;return tv.jsxs("div",{className:i.nav,style:l.nav,children:[!e.hidePrevious&&tv.jsx(tL,{name:"previous-month","aria-label":f,className:m,style:l.nav_button_previous,disabled:!e.previousMonth,onClick:e.onPreviousClick,children:"rtl"===o?tv.jsx(p,{className:i.nav_icon,style:l.nav_icon}):tv.jsx(g,{className:i.nav_icon,style:l.nav_icon})}),!e.hideNext&&tv.jsx(tL,{name:"next-month","aria-label":v,className:h,style:l.nav_button_next,disabled:!e.nextMonth,onClick:e.onNextClick,children:"rtl"===o?tv.jsx(g,{className:i.nav_icon,style:l.nav_icon}):tv.jsx(p,{className:i.nav_icon,style:l.nav_icon})})]})}function tF(e){var t=tM().numberOfMonths,n=t_(),r=n.previousMonth,o=n.nextMonth,a=n.goToMonth,i=n.displayMonths,l=i.findIndex(function(t){return e9(e.displayMonth,t)}),u=0===l,s=l===i.length-1;return tv.jsx(tY,{displayMonth:e.displayMonth,hideNext:t>1&&(u||!s),hidePrevious:t>1&&(s||!u),nextMonth:o,previousMonth:r,onPreviousClick:function(){r&&a(r)},onNextClick:function(){o&&a(o)}})}function tW(e){var t,n,r=tM(),o=r.classNames,a=r.disableNavigation,i=r.styles,l=r.captionLayout,u=r.components,s=null!==(t=null==u?void 0:u.CaptionLabel)&&void 0!==t?t:tC;return n=a?tv.jsx(s,{id:e.id,displayMonth:e.displayMonth}):"dropdown"===l?tv.jsx(tj,{displayMonth:e.displayMonth,id:e.id}):"dropdown-buttons"===l?tv.jsxs(tv.Fragment,{children:[tv.jsx(tj,{displayMonth:e.displayMonth,displayIndex:e.displayIndex,id:e.id}),tv.jsx(tF,{displayMonth:e.displayMonth,displayIndex:e.displayIndex,id:e.id})]}):tv.jsxs(tv.Fragment,{children:[tv.jsx(s,{id:e.id,displayMonth:e.displayMonth,displayIndex:e.displayIndex}),tv.jsx(tF,{displayMonth:e.displayMonth,id:e.id})]}),tv.jsx("div",{className:o.caption,style:i.caption,children:n})}function tR(e){var t=tM(),n=t.footer,r=t.styles,o=t.classNames.tfoot;return n?tv.jsx("tfoot",{className:o,style:r.tfoot,children:tv.jsx("tr",{children:tv.jsx("td",{colSpan:8,children:n})})}):tv.jsx(tv.Fragment,{})}function tI(){var e=tM(),t=e.classNames,n=e.styles,r=e.showWeekNumber,o=e.locale,a=e.weekStartsOn,i=e.ISOWeek,l=e.formatters.formatWeekdayName,u=e.labels.labelWeekday,s=function(e,t,n){for(var r=n?tn(new Date):tt(new Date,{locale:e,weekStartsOn:t}),o=[],a=0;a<7;a++){var i=(0,ev.Z)(r,a);o.push(i)}return o}(o,a,i);return tv.jsxs("tr",{style:n.head_row,className:t.head_row,children:[r&&tv.jsx("td",{style:n.head_cell,className:t.head_cell}),s.map(function(e,r){return tv.jsx("th",{scope:"col",className:t.head_cell,style:n.head_cell,"aria-label":u(e,{locale:o}),children:l(e,{locale:o})},r)})]})}function tU(){var e,t=tM(),n=t.classNames,r=t.styles,o=t.components,a=null!==(e=null==o?void 0:o.HeadRow)&&void 0!==e?e:tI;return tv.jsx("thead",{style:r.head,className:n.head,children:tv.jsx(a,{})})}function tH(e){var t=tM(),n=t.locale,r=t.formatters.formatDay;return tv.jsx(tv.Fragment,{children:r(e.date,{locale:n})})}var tB=(0,c.createContext)(void 0);function tz(e){return th(e.initialProps)?tv.jsx(tA,{initialProps:e.initialProps,children:e.children}):tv.jsx(tB.Provider,{value:{selected:void 0,modifiers:{disabled:[]}},children:e.children})}function tA(e){var t=e.initialProps,n=e.children,r=t.selected,o=t.min,a=t.max,i={disabled:[]};return r&&i.disabled.push(function(e){var t=a&&r.length>a-1,n=r.some(function(t){return tr(t,e)});return!!(t&&!n)}),tv.jsx(tB.Provider,{value:{selected:r,onDayClick:function(e,n,i){if(null===(l=t.onDayClick)||void 0===l||l.call(t,e,n,i),(!n.selected||!o||(null==r?void 0:r.length)!==o)&&(n.selected||!a||(null==r?void 0:r.length)!==a)){var l,u,s=r?tc([],r,!0):[];if(n.selected){var d=s.findIndex(function(t){return tr(e,t)});s.splice(d,1)}else s.push(e);null===(u=t.onSelect)||void 0===u||u.call(t,s,e,n,i)}},modifiers:i},children:n})}function tq(){var e=(0,c.useContext)(tB);if(!e)throw Error("useSelectMultiple must be used within a SelectMultipleProvider");return e}var tV=(0,c.createContext)(void 0);function tG(e){return tp(e.initialProps)?tv.jsx(tX,{initialProps:e.initialProps,children:e.children}):tv.jsx(tV.Provider,{value:{selected:void 0,modifiers:{range_start:[],range_end:[],range_middle:[],disabled:[]}},children:e.children})}function tX(e){var t=e.initialProps,n=e.children,r=t.selected,o=r||{},a=o.from,i=o.to,l=t.min,u=t.max,s={range_start:[],range_end:[],range_middle:[],disabled:[]};if(a?(s.range_start=[a],i?(s.range_end=[i],tr(a,i)||(s.range_middle=[{after:a,before:i}])):s.range_end=[a]):i&&(s.range_start=[i],s.range_end=[i]),l&&(a&&!i&&s.disabled.push({after:ep(a,l-1),before:(0,ev.Z)(a,l-1)}),a&&i&&s.disabled.push({after:a,before:(0,ev.Z)(a,l-1)}),!a&&i&&s.disabled.push({after:ep(i,l-1),before:(0,ev.Z)(i,l-1)})),u){if(a&&!i&&(s.disabled.push({before:(0,ev.Z)(a,-u+1)}),s.disabled.push({after:(0,ev.Z)(a,u-1)})),a&&i){var d=u-(ta(i,a)+1);s.disabled.push({before:ep(a,d)}),s.disabled.push({after:(0,ev.Z)(i,d)})}!a&&i&&(s.disabled.push({before:(0,ev.Z)(i,-u+1)}),s.disabled.push({after:(0,ev.Z)(i,u-1)}))}return tv.jsx(tV.Provider,{value:{selected:r,onDayClick:function(e,n,o){null===(u=t.onDayClick)||void 0===u||u.call(t,e,n,o);var a,i,l,u,s,d=(i=(a=r||{}).from,l=a.to,i&&l?tr(l,e)&&tr(i,e)?void 0:tr(l,e)?{from:l,to:void 0}:tr(i,e)?void 0:to(i,e)?{from:e,to:l}:{from:i,to:e}:l?to(e,l)?{from:l,to:e}:{from:e,to:l}:i?te(e,i)?{from:e,to:i}:{from:i,to:e}:{from:e,to:void 0});null===(s=t.onSelect)||void 0===s||s.call(t,d,e,n,o)},modifiers:s},children:n})}function tK(){var e=(0,c.useContext)(tV);if(!e)throw Error("useSelectRange must be used within a SelectRangeProvider");return e}function tQ(e){return Array.isArray(e)?tc([],e,!0):void 0!==e?[e]:[]}(l=s||(s={})).Outside="outside",l.Disabled="disabled",l.Selected="selected",l.Hidden="hidden",l.Today="today",l.RangeStart="range_start",l.RangeEnd="range_end",l.RangeMiddle="range_middle";var tJ=s.Selected,t$=s.Disabled,t0=s.Hidden,t1=s.Today,t2=s.RangeEnd,t4=s.RangeMiddle,t3=s.RangeStart,t8=s.Outside,t6=(0,c.createContext)(void 0);function t5(e){var t,n,r,o=tM(),a=tq(),i=tK(),l=((t={})[tJ]=tQ(o.selected),t[t$]=tQ(o.disabled),t[t0]=tQ(o.hidden),t[t1]=[o.today],t[t2]=[],t[t4]=[],t[t3]=[],t[t8]=[],o.fromDate&&t[t$].push({before:o.fromDate}),o.toDate&&t[t$].push({after:o.toDate}),th(o)?t[t$]=t[t$].concat(a.modifiers[t$]):tp(o)&&(t[t$]=t[t$].concat(i.modifiers[t$]),t[t3]=i.modifiers[t3],t[t4]=i.modifiers[t4],t[t2]=i.modifiers[t2]),t),u=(n=o.modifiers,r={},Object.entries(n).forEach(function(e){var t=e[0],n=e[1];r[t]=tQ(n)}),r),s=td(td({},l),u);return tv.jsx(t6.Provider,{value:s,children:e.children})}function t7(){var e=(0,c.useContext)(t6);if(!e)throw Error("useModifiers must be used within a ModifiersProvider");return e}function t9(e,t,n){var r=Object.keys(t).reduce(function(n,r){return t[r].some(function(t){if("boolean"==typeof t)return t;if(ew(t))return tr(e,t);if(Array.isArray(t)&&t.every(ew))return t.includes(e);if(t&&"object"==typeof t&&"from"in t)return r=t.from,o=t.to,r&&o?(0>ta(o,r)&&(r=(n=[o,r])[0],o=n[1]),ta(e,r)>=0&&ta(o,e)>=0):o?tr(o,e):!!r&&tr(r,e);if(t&&"object"==typeof t&&"dayOfWeek"in t)return t.dayOfWeek.includes(e.getDay());if(t&&"object"==typeof t&&"before"in t&&"after"in t){var n,r,o,a=ta(t.before,e),i=ta(t.after,e),l=a>0,u=i<0;return to(t.before,t.after)?u&&l:l||u}return t&&"object"==typeof t&&"after"in t?ta(e,t.after)>0:t&&"object"==typeof t&&"before"in t?ta(t.before,e)>0:"function"==typeof t&&t(e)})&&n.push(r),n},[]),o={};return r.forEach(function(e){return o[e]=!0}),n&&!e9(e,n)&&(o.outside=!0),o}var ne=(0,c.createContext)(void 0);function nt(e){var t=t_(),n=t7(),r=(0,c.useState)(),o=r[0],a=r[1],i=(0,c.useState)(),l=i[0],u=i[1],s=function(e,t){for(var n,r,o=eu(e[0]),a=e8(e[e.length-1]),i=o;i<=a;){var l=t9(i,t);if(!(!l.disabled&&!l.hidden)){i=(0,ev.Z)(i,1);continue}if(l.selected)return i;l.today&&!r&&(r=i),n||(n=i),i=(0,ev.Z)(i,1)}return r||n}(t.displayMonths,n),d=(null!=o?o:l&&t.isDateDisplayed(l))?l:s,f=function(e){a(e)},m=tM(),v=function(e,r){if(o){var a=function e(t,n){var r=n.moveBy,o=n.direction,a=n.context,i=n.modifiers,l=n.retry,u=void 0===l?{count:0,lastFocused:t}:l,s=a.weekStartsOn,d=a.fromDate,c=a.toDate,f=a.locale,m=({day:ev.Z,week:ti,month:eg.Z,year:tl,startOfWeek:function(e){return a.ISOWeek?tn(e):tt(e,{locale:f,weekStartsOn:s})},endOfWeek:function(e){return a.ISOWeek?ts(e):tu(e,{locale:f,weekStartsOn:s})}})[r](t,"after"===o?1:-1);"before"===o&&d?m=ef([d,m]):"after"===o&&c&&(m=em([c,m]));var v=!0;if(i){var h=t9(m,i);v=!h.disabled&&!h.hidden}return v?m:u.count>365?u.lastFocused:e(m,{moveBy:r,direction:o,context:a,modifiers:i,retry:td(td({},u),{count:u.count+1})})}(o,{moveBy:e,direction:r,context:m,modifiers:n});tr(o,a)||(t.goToDate(a,o),f(a))}};return tv.jsx(ne.Provider,{value:{focusedDay:o,focusTarget:d,blur:function(){u(o),a(void 0)},focus:f,focusDayAfter:function(){return v("day","after")},focusDayBefore:function(){return v("day","before")},focusWeekAfter:function(){return v("week","after")},focusWeekBefore:function(){return v("week","before")},focusMonthBefore:function(){return v("month","before")},focusMonthAfter:function(){return v("month","after")},focusYearBefore:function(){return v("year","before")},focusYearAfter:function(){return v("year","after")},focusStartOfWeek:function(){return v("startOfWeek","before")},focusEndOfWeek:function(){return v("endOfWeek","after")}},children:e.children})}function nn(){var e=(0,c.useContext)(ne);if(!e)throw Error("useFocusContext must be used within a FocusProvider");return e}var nr=(0,c.createContext)(void 0);function no(e){return tg(e.initialProps)?tv.jsx(na,{initialProps:e.initialProps,children:e.children}):tv.jsx(nr.Provider,{value:{selected:void 0},children:e.children})}function na(e){var t=e.initialProps,n=e.children,r={selected:t.selected,onDayClick:function(e,n,r){var o,a,i;if(null===(o=t.onDayClick)||void 0===o||o.call(t,e,n,r),n.selected&&!t.required){null===(a=t.onSelect)||void 0===a||a.call(t,void 0,e,n,r);return}null===(i=t.onSelect)||void 0===i||i.call(t,e,e,n,r)}};return tv.jsx(nr.Provider,{value:r,children:n})}function ni(){var e=(0,c.useContext)(nr);if(!e)throw Error("useSelectSingle must be used within a SelectSingleProvider");return e}function nl(e){var t,n,r,o,a,i,l,u,d,f,m,v,h,p,g,b,y,w,x,k,M,C,D,T,N,P,E,S,_,j,O,Z,L,Y,F,W,R,I,U,H,B,z,A=(0,c.useRef)(null),q=(t=e.date,n=e.displayMonth,i=tM(),l=nn(),u=t9(t,t7(),n),d=tM(),f=ni(),m=tq(),v=tK(),p=(h=nn()).focusDayAfter,g=h.focusDayBefore,b=h.focusWeekAfter,y=h.focusWeekBefore,w=h.blur,x=h.focus,k=h.focusMonthBefore,M=h.focusMonthAfter,C=h.focusYearBefore,D=h.focusYearAfter,T=h.focusStartOfWeek,N=h.focusEndOfWeek,P={onClick:function(e){var n,r,o,a;tg(d)?null===(n=f.onDayClick)||void 0===n||n.call(f,t,u,e):th(d)?null===(r=m.onDayClick)||void 0===r||r.call(m,t,u,e):tp(d)?null===(o=v.onDayClick)||void 0===o||o.call(v,t,u,e):null===(a=d.onDayClick)||void 0===a||a.call(d,t,u,e)},onFocus:function(e){var n;x(t),null===(n=d.onDayFocus)||void 0===n||n.call(d,t,u,e)},onBlur:function(e){var n;w(),null===(n=d.onDayBlur)||void 0===n||n.call(d,t,u,e)},onKeyDown:function(e){var n;switch(e.key){case"ArrowLeft":e.preventDefault(),e.stopPropagation(),"rtl"===d.dir?p():g();break;case"ArrowRight":e.preventDefault(),e.stopPropagation(),"rtl"===d.dir?g():p();break;case"ArrowDown":e.preventDefault(),e.stopPropagation(),b();break;case"ArrowUp":e.preventDefault(),e.stopPropagation(),y();break;case"PageUp":e.preventDefault(),e.stopPropagation(),e.shiftKey?C():k();break;case"PageDown":e.preventDefault(),e.stopPropagation(),e.shiftKey?D():M();break;case"Home":e.preventDefault(),e.stopPropagation(),T();break;case"End":e.preventDefault(),e.stopPropagation(),N()}null===(n=d.onDayKeyDown)||void 0===n||n.call(d,t,u,e)},onKeyUp:function(e){var n;null===(n=d.onDayKeyUp)||void 0===n||n.call(d,t,u,e)},onMouseEnter:function(e){var n;null===(n=d.onDayMouseEnter)||void 0===n||n.call(d,t,u,e)},onMouseLeave:function(e){var n;null===(n=d.onDayMouseLeave)||void 0===n||n.call(d,t,u,e)},onPointerEnter:function(e){var n;null===(n=d.onDayPointerEnter)||void 0===n||n.call(d,t,u,e)},onPointerLeave:function(e){var n;null===(n=d.onDayPointerLeave)||void 0===n||n.call(d,t,u,e)},onTouchCancel:function(e){var n;null===(n=d.onDayTouchCancel)||void 0===n||n.call(d,t,u,e)},onTouchEnd:function(e){var n;null===(n=d.onDayTouchEnd)||void 0===n||n.call(d,t,u,e)},onTouchMove:function(e){var n;null===(n=d.onDayTouchMove)||void 0===n||n.call(d,t,u,e)},onTouchStart:function(e){var n;null===(n=d.onDayTouchStart)||void 0===n||n.call(d,t,u,e)}},E=tM(),S=ni(),_=tq(),j=tK(),O=tg(E)?S.selected:th(E)?_.selected:tp(E)?j.selected:void 0,Z=!!(i.onDayClick||"default"!==i.mode),(0,c.useEffect)(function(){var e;!u.outside&&l.focusedDay&&Z&&tr(l.focusedDay,t)&&(null===(e=A.current)||void 0===e||e.focus())},[l.focusedDay,t,A,Z,u.outside]),Y=(L=[i.classNames.day],Object.keys(u).forEach(function(e){var t=i.modifiersClassNames[e];if(t)L.push(t);else if(Object.values(s).includes(e)){var n=i.classNames["day_".concat(e)];n&&L.push(n)}}),L).join(" "),F=td({},i.styles.day),Object.keys(u).forEach(function(e){var t;F=td(td({},F),null===(t=i.modifiersStyles)||void 0===t?void 0:t[e])}),W=F,R=!!(u.outside&&!i.showOutsideDays||u.hidden),I=null!==(a=null===(o=i.components)||void 0===o?void 0:o.DayContent)&&void 0!==a?a:tH,U={style:W,className:Y,children:tv.jsx(I,{date:t,displayMonth:n,activeModifiers:u}),role:"gridcell"},H=l.focusTarget&&tr(l.focusTarget,t)&&!u.outside,B=l.focusedDay&&tr(l.focusedDay,t),z=td(td(td({},U),((r={disabled:u.disabled,role:"gridcell"})["aria-selected"]=u.selected,r.tabIndex=B||H?0:-1,r)),P),{isButton:Z,isHidden:R,activeModifiers:u,selectedDays:O,buttonProps:z,divProps:U});return q.isHidden?tv.jsx("div",{role:"gridcell"}):q.isButton?tv.jsx(tL,td({name:"day",ref:A},q.buttonProps)):tv.jsx("div",td({},q.divProps))}function nu(e){var t=e.number,n=e.dates,r=tM(),o=r.onWeekNumberClick,a=r.styles,i=r.classNames,l=r.locale,u=r.labels.labelWeekNumber,s=(0,r.formatters.formatWeekNumber)(Number(t),{locale:l});if(!o)return tv.jsx("span",{className:i.weeknumber,style:a.weeknumber,children:s});var d=u(Number(t),{locale:l});return tv.jsx(tL,{name:"week-number","aria-label":d,className:i.weeknumber,style:a.weeknumber,onClick:function(e){o(t,n,e)},children:s})}function ns(e){var t,n,r,o=tM(),a=o.styles,i=o.classNames,l=o.showWeekNumber,u=o.components,s=null!==(t=null==u?void 0:u.Day)&&void 0!==t?t:nl,d=null!==(n=null==u?void 0:u.WeekNumber)&&void 0!==n?n:nu;return l&&(r=tv.jsx("td",{className:i.cell,style:a.cell,children:tv.jsx(d,{number:e.weekNumber,dates:e.dates})})),tv.jsxs("tr",{className:i.row,style:a.row,children:[r,e.dates.map(function(t){return tv.jsx("td",{className:i.cell,style:a.cell,role:"presentation",children:tv.jsx(s,{displayMonth:e.displayMonth,date:t})},function(e){return(0,ea.Z)(1,arguments),Math.floor(function(e){return(0,ea.Z)(1,arguments),(0,eo.Z)(e).getTime()}(e)/1e3)}(t))})]})}function nd(e,t,n){for(var r=(null==n?void 0:n.ISOWeek)?ts(t):tu(t,n),o=(null==n?void 0:n.ISOWeek)?tn(e):tt(e,n),a=ta(r,o),i=[],l=0;l<=a;l++)i.push((0,ev.Z)(o,l));return i.reduce(function(e,t){var r=(null==n?void 0:n.ISOWeek)?function(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e);return Math.round((tn(t).getTime()-(function(e){(0,ea.Z)(1,arguments);var t=function(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e),n=t.getFullYear(),r=new Date(0);r.setFullYear(n+1,0,4),r.setHours(0,0,0,0);var o=tn(r),a=new Date(0);a.setFullYear(n,0,4),a.setHours(0,0,0,0);var i=tn(a);return t.getTime()>=o.getTime()?n+1:t.getTime()>=i.getTime()?n:n-1}(e),n=new Date(0);return n.setFullYear(t,0,4),n.setHours(0,0,0,0),tn(n)})(t).getTime())/6048e5)+1}(t):function(e,t){(0,ea.Z)(1,arguments);var n=(0,eo.Z)(e);return Math.round((tt(n,t).getTime()-(function(e,t){(0,ea.Z)(1,arguments);var n,r,o,a,i,l,u,s,d=(0,eh.Z)(null!==(n=null!==(r=null!==(o=null!==(a=null==t?void 0:t.firstWeekContainsDate)&&void 0!==a?a:null==t?void 0:null===(i=t.locale)||void 0===i?void 0:null===(l=i.options)||void 0===l?void 0:l.firstWeekContainsDate)&&void 0!==o?o:eM.firstWeekContainsDate)&&void 0!==r?r:null===(u=eM.locale)||void 0===u?void 0:null===(s=u.options)||void 0===s?void 0:s.firstWeekContainsDate)&&void 0!==n?n:1),c=function(e,t){(0,ea.Z)(1,arguments);var n,r,o,a,i,l,u,s,d=(0,eo.Z)(e),c=d.getFullYear(),f=(0,eh.Z)(null!==(n=null!==(r=null!==(o=null!==(a=null==t?void 0:t.firstWeekContainsDate)&&void 0!==a?a:null==t?void 0:null===(i=t.locale)||void 0===i?void 0:null===(l=i.options)||void 0===l?void 0:l.firstWeekContainsDate)&&void 0!==o?o:eM.firstWeekContainsDate)&&void 0!==r?r:null===(u=eM.locale)||void 0===u?void 0:null===(s=u.options)||void 0===s?void 0:s.firstWeekContainsDate)&&void 0!==n?n:1);if(!(f>=1&&f<=7))throw RangeError("firstWeekContainsDate must be between 1 and 7 inclusively");var m=new Date(0);m.setFullYear(c+1,0,f),m.setHours(0,0,0,0);var v=tt(m,t),h=new Date(0);h.setFullYear(c,0,f),h.setHours(0,0,0,0);var p=tt(h,t);return d.getTime()>=v.getTime()?c+1:d.getTime()>=p.getTime()?c:c-1}(e,t),f=new Date(0);return f.setFullYear(c,0,d),f.setHours(0,0,0,0),tt(f,t)})(n,t).getTime())/6048e5)+1}(t,n),o=e.find(function(e){return e.weekNumber===r});return o?o.dates.push(t):e.push({weekNumber:r,dates:[t]}),e},[])}function nc(e){var t,n,r,o=tM(),a=o.locale,i=o.classNames,l=o.styles,u=o.hideHead,s=o.fixedWeeks,d=o.components,c=o.weekStartsOn,f=o.firstWeekContainsDate,m=o.ISOWeek,v=function(e,t){var n=nd(eu(e),e8(e),t);if(null==t?void 0:t.useFixedWeeks){var r=function(e,t){return(0,ea.Z)(1,arguments),function(e,t,n){(0,ea.Z)(2,arguments);var r=tt(e,n),o=tt(t,n);return Math.round((r.getTime()-eY(r)-(o.getTime()-eY(o)))/6048e5)}(function(e){(0,ea.Z)(1,arguments);var t=(0,eo.Z)(e),n=t.getMonth();return t.setFullYear(t.getFullYear(),n+1,0),t.setHours(0,0,0,0),t}(e),eu(e),t)+1}(e,t);if(r<6){var o=n[n.length-1],a=o.dates[o.dates.length-1],i=ti(a,6-r),l=nd(ti(a,1),i,t);n.push.apply(n,l)}}return n}(e.displayMonth,{useFixedWeeks:!!s,ISOWeek:m,locale:a,weekStartsOn:c,firstWeekContainsDate:f}),h=null!==(t=null==d?void 0:d.Head)&&void 0!==t?t:tU,p=null!==(n=null==d?void 0:d.Row)&&void 0!==n?n:ns,g=null!==(r=null==d?void 0:d.Footer)&&void 0!==r?r:tR;return tv.jsxs("table",{id:e.id,className:i.table,style:l.table,role:"grid","aria-labelledby":e["aria-labelledby"],children:[!u&&tv.jsx(h,{}),tv.jsx("tbody",{className:i.tbody,style:l.tbody,children:v.map(function(t){return tv.jsx(p,{displayMonth:e.displayMonth,dates:t.dates,weekNumber:t.weekNumber},t.weekNumber)})}),tv.jsx(g,{displayMonth:e.displayMonth})]})}var nf="undefined"!=typeof window&&window.document&&window.document.createElement?c.useLayoutEffect:c.useEffect,nm=!1,nv=0;function nh(){return"react-day-picker-".concat(++nv)}function np(e){var t,n,r,o,a,i,l,u,s=tM(),d=s.dir,f=s.classNames,m=s.styles,v=s.components,h=t_().displayMonths,p=(r=null!=(t=s.id?"".concat(s.id,"-").concat(e.displayIndex):void 0)?t:nm?nh():null,a=(o=(0,c.useState)(r))[0],i=o[1],nf(function(){null===a&&i(nh())},[]),(0,c.useEffect)(function(){!1===nm&&(nm=!0)},[]),null!==(n=null!=t?t:a)&&void 0!==n?n:void 0),g=s.id?"".concat(s.id,"-grid-").concat(e.displayIndex):void 0,b=[f.month],y=m.month,w=0===e.displayIndex,x=e.displayIndex===h.length-1,k=!w&&!x;"rtl"===d&&(x=(l=[w,x])[0],w=l[1]),w&&(b.push(f.caption_start),y=td(td({},y),m.caption_start)),x&&(b.push(f.caption_end),y=td(td({},y),m.caption_end)),k&&(b.push(f.caption_between),y=td(td({},y),m.caption_between));var M=null!==(u=null==v?void 0:v.Caption)&&void 0!==u?u:tW;return tv.jsxs("div",{className:b.join(" "),style:y,children:[tv.jsx(M,{id:p,displayMonth:e.displayMonth,displayIndex:e.displayIndex}),tv.jsx(nc,{id:g,"aria-labelledby":p,displayMonth:e.displayMonth})]},e.displayIndex)}function ng(e){var t=tM(),n=t.classNames,r=t.styles;return tv.jsx("div",{className:n.months,style:r.months,children:e.children})}function nb(e){var t,n,r=e.initialProps,o=tM(),a=nn(),i=t_(),l=(0,c.useState)(!1),u=l[0],s=l[1];(0,c.useEffect)(function(){o.initialFocus&&a.focusTarget&&(u||(a.focus(a.focusTarget),s(!0)))},[o.initialFocus,u,a.focus,a.focusTarget,a]);var d=[o.classNames.root,o.className];o.numberOfMonths>1&&d.push(o.classNames.multiple_months),o.showWeekNumber&&d.push(o.classNames.with_weeknumber);var f=td(td({},o.styles.root),o.style),m=Object.keys(r).filter(function(e){return e.startsWith("data-")}).reduce(function(e,t){var n;return td(td({},e),((n={})[t]=r[t],n))},{}),v=null!==(n=null===(t=r.components)||void 0===t?void 0:t.Months)&&void 0!==n?n:ng;return tv.jsx("div",td({className:d.join(" "),style:f,dir:o.dir,id:o.id,nonce:r.nonce,title:r.title,lang:r.lang},m,{children:tv.jsx(v,{children:i.displayMonths.map(function(e,t){return tv.jsx(np,{displayIndex:t,displayMonth:e},t)})})}))}function ny(e){var t=e.children,n=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n}(e,["children"]);return tv.jsx(tk,{initialProps:n,children:tv.jsx(tS,{children:tv.jsx(no,{initialProps:n,children:tv.jsx(tz,{initialProps:n,children:tv.jsx(tG,{initialProps:n,children:tv.jsx(t5,{children:tv.jsx(nt,{children:t})})})})})})})}function nw(e){return tv.jsx(ny,td({},e,{children:tv.jsx(nb,{initialProps:e})}))}let nx=e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M10.8284 12.0007L15.7782 16.9504L14.364 18.3646L8 12.0007L14.364 5.63672L15.7782 7.05093L10.8284 12.0007Z"}))},nk=e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M13.1717 12.0007L8.22192 7.05093L9.63614 5.63672L16.0001 12.0007L9.63614 18.3646L8.22192 16.9504L13.1717 12.0007Z"}))},nM=e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M4.83582 12L11.0429 18.2071L12.4571 16.7929L7.66424 12L12.4571 7.20712L11.0429 5.79291L4.83582 12ZM10.4857 12L16.6928 18.2071L18.107 16.7929L13.3141 12L18.107 7.20712L16.6928 5.79291L10.4857 12Z"}))},nC=e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M19.1642 12L12.9571 5.79291L11.5429 7.20712L16.3358 12L11.5429 16.7929L12.9571 18.2071L19.1642 12ZM13.5143 12L7.30722 5.79291L5.89301 7.20712L10.6859 12L5.89301 16.7929L7.30722 18.2071L13.5143 12Z"}))};var nD=n(84264);n(41649);var nT=n(1526),nN=n(7084),nP=n(26898);let nE={xs:{paddingX:"px-2",paddingY:"py-0.5",fontSize:"text-xs"},sm:{paddingX:"px-2.5",paddingY:"py-1",fontSize:"text-sm"},md:{paddingX:"px-3",paddingY:"py-1.5",fontSize:"text-md"},lg:{paddingX:"px-3.5",paddingY:"py-1.5",fontSize:"text-lg"},xl:{paddingX:"px-3.5",paddingY:"py-1.5",fontSize:"text-xl"}},nS={xs:{paddingX:"px-2",paddingY:"py-0.5",fontSize:"text-xs"},sm:{paddingX:"px-2.5",paddingY:"py-0.5",fontSize:"text-sm"},md:{paddingX:"px-3",paddingY:"py-0.5",fontSize:"text-md"},lg:{paddingX:"px-3.5",paddingY:"py-0.5",fontSize:"text-lg"},xl:{paddingX:"px-4",paddingY:"py-1",fontSize:"text-xl"}},n_={xs:{height:"h-4",width:"w-4"},sm:{height:"h-4",width:"w-4"},md:{height:"h-4",width:"w-4"},lg:{height:"h-5",width:"w-5"},xl:{height:"h-6",width:"w-6"}},nj={[nN.wu.Increase]:{bgColor:(0,e$.bM)(nN.fr.Emerald,nP.K.background).bgColor,textColor:(0,e$.bM)(nN.fr.Emerald,nP.K.text).textColor},[nN.wu.ModerateIncrease]:{bgColor:(0,e$.bM)(nN.fr.Emerald,nP.K.background).bgColor,textColor:(0,e$.bM)(nN.fr.Emerald,nP.K.text).textColor},[nN.wu.Decrease]:{bgColor:(0,e$.bM)(nN.fr.Rose,nP.K.background).bgColor,textColor:(0,e$.bM)(nN.fr.Rose,nP.K.text).textColor},[nN.wu.ModerateDecrease]:{bgColor:(0,e$.bM)(nN.fr.Rose,nP.K.background).bgColor,textColor:(0,e$.bM)(nN.fr.Rose,nP.K.text).textColor},[nN.wu.Unchanged]:{bgColor:(0,e$.bM)(nN.fr.Orange,nP.K.background).bgColor,textColor:(0,e$.bM)(nN.fr.Orange,nP.K.text).textColor}},nO={[nN.wu.Increase]:e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M13.0001 7.82843V20H11.0001V7.82843L5.63614 13.1924L4.22192 11.7782L12.0001 4L19.7783 11.7782L18.3641 13.1924L13.0001 7.82843Z"}))},[nN.wu.ModerateIncrease]:e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M16.0037 9.41421L7.39712 18.0208L5.98291 16.6066L14.5895 8H7.00373V6H18.0037V17H16.0037V9.41421Z"}))},[nN.wu.Decrease]:e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M13.0001 16.1716L18.3641 10.8076L19.7783 12.2218L12.0001 20L4.22192 12.2218L5.63614 10.8076L11.0001 16.1716V4H13.0001V16.1716Z"}))},[nN.wu.ModerateDecrease]:e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M14.5895 16.0032L5.98291 7.39664L7.39712 5.98242L16.0037 14.589V7.00324H18.0037V18.0032H7.00373V16.0032H14.5895Z"}))},[nN.wu.Unchanged]:e=>{var t=(0,d._T)(e,[]);return c.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),c.createElement("path",{d:"M16.1716 10.9999L10.8076 5.63589L12.2218 4.22168L20 11.9999L12.2218 19.778L10.8076 18.3638L16.1716 12.9999H4V10.9999H16.1716Z"}))}},nZ=(0,e$.fn)("BadgeDelta");c.forwardRef((e,t)=>{let{deltaType:n=nN.wu.Increase,isIncreasePositive:r=!0,size:o=nN.u8.SM,tooltip:a,children:i,className:l}=e,u=(0,d._T)(e,["deltaType","isIncreasePositive","size","tooltip","children","className"]),s=nO[n],f=(0,e$.Fo)(n,r),m=i?nS:nE,{tooltipProps:v,getReferenceProps:h}=(0,nT.l)();return c.createElement("span",Object.assign({ref:(0,e$.lq)([t,v.refs.setReference]),className:(0,es.q)(nZ("root"),"w-max flex-shrink-0 inline-flex justify-center items-center cursor-default rounded-tremor-full bg-opacity-20 dark:bg-opacity-25",nj[f].bgColor,nj[f].textColor,m[o].paddingX,m[o].paddingY,m[o].fontSize,l)},h,u),c.createElement(nT.Z,Object.assign({text:a},v)),c.createElement(s,{className:(0,es.q)(nZ("icon"),"shrink-0",i?(0,es.q)("-ml-1 mr-1.5"):n_[o].height,n_[o].width)}),i?c.createElement("p",{className:(0,es.q)(nZ("text"),"text-sm whitespace-nowrap")},i):null)}).displayName="BadgeDelta";var nL=n(47323);let nY=e=>{var{onClick:t,icon:n}=e,r=(0,d._T)(e,["onClick","icon"]);return c.createElement("button",Object.assign({type:"button",className:(0,es.q)("flex items-center justify-center p-1 h-7 w-7 outline-none focus:ring-2 transition duration-100 border border-tremor-border dark:border-dark-tremor-border hover:bg-tremor-background-muted dark:hover:bg-dark-tremor-background-muted rounded-tremor-small focus:border-tremor-brand-subtle select-none dark:focus:border-dark-tremor-brand-subtle focus:ring-tremor-brand-muted dark:focus:ring-dark-tremor-brand-muted text-tremor-content-subtle dark:text-dark-tremor-content-subtle hover:text-tremor-content dark:hover:text-dark-tremor-content")},r),c.createElement(nL.Z,{onClick:t,icon:n,variant:"simple",color:"slate",size:"sm"}))};function nF(e){var{mode:t,defaultMonth:n,selected:r,onSelect:o,locale:a,disabled:i,enableYearNavigation:l,classNames:u,weekStartsOn:s=0}=e,f=(0,d._T)(e,["mode","defaultMonth","selected","onSelect","locale","disabled","enableYearNavigation","classNames","weekStartsOn"]);return c.createElement(nw,Object.assign({showOutsideDays:!0,mode:t,defaultMonth:n,selected:r,onSelect:o,locale:a,disabled:i,weekStartsOn:s,classNames:Object.assign({months:"flex flex-col sm:flex-row space-y-4 sm:space-x-4 sm:space-y-0",month:"space-y-4",caption:"flex justify-center pt-2 relative items-center",caption_label:"text-tremor-default text-tremor-content-emphasis dark:text-dark-tremor-content-emphasis font-medium",nav:"space-x-1 flex items-center",nav_button:"flex items-center justify-center p-1 h-7 w-7 outline-none focus:ring-2 transition duration-100 border border-tremor-border dark:border-dark-tremor-border hover:bg-tremor-background-muted dark:hover:bg-dark-tremor-background-muted rounded-tremor-small focus:border-tremor-brand-subtle dark:focus:border-dark-tremor-brand-subtle focus:ring-tremor-brand-muted dark:focus:ring-dark-tremor-brand-muted text-tremor-content-subtle dark:text-dark-tremor-content-subtle hover:text-tremor-content dark:hover:text-dark-tremor-content",nav_button_previous:"absolute left-1",nav_button_next:"absolute right-1",table:"w-full border-collapse space-y-1",head_row:"flex",head_cell:"w-9 font-normal text-center text-tremor-content-subtle dark:text-dark-tremor-content-subtle",row:"flex w-full mt-0.5",cell:"text-center p-0 relative focus-within:relative text-tremor-default text-tremor-content-emphasis dark:text-dark-tremor-content-emphasis",day:"h-9 w-9 p-0 hover:bg-tremor-background-subtle dark:hover:bg-dark-tremor-background-subtle outline-tremor-brand dark:outline-dark-tremor-brand rounded-tremor-default",day_today:"font-bold",day_selected:"aria-selected:bg-tremor-background-emphasis aria-selected:text-tremor-content-inverted dark:aria-selected:bg-dark-tremor-background-emphasis dark:aria-selected:text-dark-tremor-content-inverted ",day_disabled:"text-tremor-content-subtle dark:text-dark-tremor-content-subtle disabled:hover:bg-transparent",day_outside:"text-tremor-content-subtle dark:text-dark-tremor-content-subtle"},u),components:{IconLeft:e=>{var t=(0,d._T)(e,[]);return c.createElement(nx,Object.assign({className:"h-4 w-4"},t))},IconRight:e=>{var t=(0,d._T)(e,[]);return c.createElement(nk,Object.assign({className:"h-4 w-4"},t))},Caption:e=>{var t=(0,d._T)(e,[]);let{goToMonth:n,nextMonth:r,previousMonth:o,currentMonth:i}=t_();return c.createElement("div",{className:"flex justify-between items-center"},c.createElement("div",{className:"flex items-center space-x-1"},l&&c.createElement(nY,{onClick:()=>i&&n(tl(i,-1)),icon:nM}),c.createElement(nY,{onClick:()=>o&&n(o),icon:nx})),c.createElement(nD.Z,{className:"text-tremor-default tabular-nums capitalize text-tremor-content-emphasis dark:text-dark-tremor-content-emphasis font-medium"},eJ(t.displayMonth,"LLLL yyy",{locale:a})),c.createElement("div",{className:"flex items-center space-x-1"},c.createElement(nY,{onClick:()=>r&&n(r),icon:nk}),l&&c.createElement(nY,{onClick:()=>i&&n(tl(i,1)),icon:nC})))}}},f))}nF.displayName="DateRangePicker",n(27281);var nW=n(43227),nR=n(44140);let nI=el(),nU=c.forwardRef((e,t)=>{var n,r;let{value:o,defaultValue:a,onValueChange:i,enableSelect:l=!0,minDate:u,maxDate:s,placeholder:f="Select range",selectPlaceholder:m="Select range",disabled:v=!1,locale:h=eq,enableClear:p=!0,displayFormat:g,children:b,className:y,enableYearNavigation:w=!1,weekStartsOn:x=0,disabledDates:k}=e,M=(0,d._T)(e,["value","defaultValue","onValueChange","enableSelect","minDate","maxDate","placeholder","selectPlaceholder","disabled","locale","enableClear","displayFormat","children","className","enableYearNavigation","weekStartsOn","disabledDates"]),[C,D]=(0,nR.Z)(a,o),[T,N]=(0,c.useState)(!1),[P,E]=(0,c.useState)(!1),S=(0,c.useMemo)(()=>{let e=[];return u&&e.push({before:u}),s&&e.push({after:s}),[...e,...null!=k?k:[]]},[u,s,k]),_=(0,c.useMemo)(()=>{let e=new Map;return b?c.Children.forEach(b,t=>{var n;e.set(t.props.value,{text:null!==(n=(0,ed.qg)(t))&&void 0!==n?n:t.props.value,from:t.props.from,to:t.props.to})}):e4.forEach(t=>{e.set(t.value,{text:t.text,from:t.from,to:nI})}),e},[b]),j=(0,c.useMemo)(()=>{if(b)return(0,ed.sl)(b);let e=new Map;return e4.forEach(t=>e.set(t.value,t.text)),e},[b]),O=(null==C?void 0:C.selectValue)||"",Z=e1(null==C?void 0:C.from,u,O,_),L=e2(null==C?void 0:C.to,s,O,_),Y=Z||L?e3(Z,L,h,g):f,F=eu(null!==(r=null!==(n=null!=L?L:Z)&&void 0!==n?n:s)&&void 0!==r?r:nI),W=p&&!v;return c.createElement("div",Object.assign({ref:t,className:(0,es.q)("w-full min-w-[10rem] relative flex justify-between text-tremor-default max-w-sm shadow-tremor-input dark:shadow-dark-tremor-input rounded-tremor-default",y)},M),c.createElement($,{as:"div",className:(0,es.q)("w-full",l?"rounded-l-tremor-default":"rounded-tremor-default",T&&"ring-2 ring-tremor-brand-muted dark:ring-dark-tremor-brand-muted z-10")},c.createElement("div",{className:"relative w-full"},c.createElement($.Button,{onFocus:()=>N(!0),onBlur:()=>N(!1),disabled:v,className:(0,es.q)("w-full outline-none text-left whitespace-nowrap truncate focus:ring-2 transition duration-100 rounded-l-tremor-default flex flex-nowrap border pl-3 py-2","rounded-l-tremor-default border-tremor-border text-tremor-content-emphasis focus:border-tremor-brand-subtle focus:ring-tremor-brand-muted","dark:border-dark-tremor-border dark:text-dark-tremor-content-emphasis dark:focus:border-dark-tremor-brand-subtle dark:focus:ring-dark-tremor-brand-muted",l?"rounded-l-tremor-default":"rounded-tremor-default",W?"pr-8":"pr-4",(0,ed.um)((0,ed.Uh)(Z||L),v))},c.createElement(en,{className:(0,es.q)(e0("calendarIcon"),"flex-none shrink-0 h-5 w-5 -ml-0.5 mr-2","text-tremor-content-subtle","dark:text-dark-tremor-content-subtle"),"aria-hidden":"true"}),c.createElement("p",{className:"truncate"},Y)),W&&Z?c.createElement("button",{type:"button",className:(0,es.q)("absolute outline-none inset-y-0 right-0 flex items-center transition duration-100 mr-4"),onClick:e=>{e.preventDefault(),null==i||i({}),D({})}},c.createElement(er.Z,{className:(0,es.q)(e0("clearIcon"),"flex-none h-4 w-4","text-tremor-content-subtle","dark:text-dark-tremor-content-subtle")})):null),c.createElement(ee.u,{className:"absolute z-10 min-w-min left-0",enter:"transition ease duration-100 transform",enterFrom:"opacity-0 -translate-y-4",enterTo:"opacity-100 translate-y-0",leave:"transition ease duration-100 transform",leaveFrom:"opacity-100 translate-y-0",leaveTo:"opacity-0 -translate-y-4"},c.createElement($.Panel,{focus:!0,className:(0,es.q)("divide-y overflow-y-auto outline-none rounded-tremor-default p-3 border my-1","bg-tremor-background border-tremor-border divide-tremor-border shadow-tremor-dropdown","dark:bg-dark-tremor-background dark:border-dark-tremor-border dark:divide-dark-tremor-border dark:shadow-dark-tremor-dropdown")},c.createElement(nF,Object.assign({mode:"range",showOutsideDays:!0,defaultMonth:F,selected:{from:Z,to:L},onSelect:e=>{null==i||i({from:null==e?void 0:e.from,to:null==e?void 0:e.to}),D({from:null==e?void 0:e.from,to:null==e?void 0:e.to})},locale:h,disabled:S,enableYearNavigation:w,classNames:{day_range_middle:(0,es.q)("!rounded-none aria-selected:!bg-tremor-background-subtle aria-selected:dark:!bg-dark-tremor-background-subtle aria-selected:!text-tremor-content aria-selected:dark:!bg-dark-tremor-background-subtle"),day_range_start:"rounded-r-none rounded-l-tremor-small aria-selected:text-tremor-brand-inverted dark:aria-selected:text-dark-tremor-brand-inverted",day_range_end:"rounded-l-none rounded-r-tremor-small aria-selected:text-tremor-brand-inverted dark:aria-selected:text-dark-tremor-brand-inverted"},weekStartsOn:x},e))))),l&&c.createElement(et.R,{as:"div",className:(0,es.q)("w-48 -ml-px rounded-r-tremor-default",P&&"ring-2 ring-tremor-brand-muted dark:ring-dark-tremor-brand-muted z-10"),value:O,onChange:e=>{let{from:t,to:n}=_.get(e),r=null!=n?n:nI;null==i||i({from:t,to:r,selectValue:e}),D({from:t,to:r,selectValue:e})},disabled:v},e=>{var t;let{value:n}=e;return c.createElement(c.Fragment,null,c.createElement(et.R.Button,{onFocus:()=>E(!0),onBlur:()=>E(!1),className:(0,es.q)("w-full outline-none text-left whitespace-nowrap truncate rounded-r-tremor-default transition duration-100 border px-4 py-2","border-tremor-border shadow-tremor-input text-tremor-content-emphasis focus:border-tremor-brand-subtle","dark:border-dark-tremor-border dark:shadow-dark-tremor-input dark:text-dark-tremor-content-emphasis dark:focus:border-dark-tremor-brand-subtle",(0,ed.um)((0,ed.Uh)(n),v))},n&&null!==(t=j.get(n))&&void 0!==t?t:m),c.createElement(ee.u,{className:"absolute z-10 w-full inset-x-0 right-0",enter:"transition ease duration-100 transform",enterFrom:"opacity-0 -translate-y-4",enterTo:"opacity-100 translate-y-0",leave:"transition ease duration-100 transform",leaveFrom:"opacity-100 translate-y-0",leaveTo:"opacity-0 -translate-y-4"},c.createElement(et.R.Options,{className:(0,es.q)("divide-y overflow-y-auto outline-none border my-1","shadow-tremor-dropdown bg-tremor-background border-tremor-border divide-tremor-border rounded-tremor-default","dark:shadow-dark-tremor-dropdown dark:bg-dark-tremor-background dark:border-dark-tremor-border dark:divide-dark-tremor-border")},null!=b?b:e4.map(e=>c.createElement(nW.Z,{key:e.value,value:e.value},e.text)))))}))});nU.displayName="DateRangePicker"},40048:function(e,t,n){n.d(t,{i:function(){return a}});var r=n(2265),o=n(40293);function a(){for(var e=arguments.length,t=Array(e),n=0;n(0,o.r)(...t),[...t])}}}]);
\ No newline at end of file
diff --git a/litellm/proxy/_experimental/out/_next/static/chunks/1491-8280340b5391aa11.js b/litellm/proxy/_experimental/out/_next/static/chunks/1491-8280340b5391aa11.js
new file mode 100644
index 00000000000..f2f6a5f9bde
--- /dev/null
+++ b/litellm/proxy/_experimental/out/_next/static/chunks/1491-8280340b5391aa11.js
@@ -0,0 +1 @@
+(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[1491],{31373:function(e,t,n){"use strict";n.d(t,{iN:function(){return h},R_:function(){return d},EV:function(){return g},ez:function(){return f}});var r=n(82082),o=n(96021),a=[{index:7,opacity:.15},{index:6,opacity:.25},{index:5,opacity:.3},{index:5,opacity:.45},{index:5,opacity:.65},{index:5,opacity:.85},{index:4,opacity:.9},{index:3,opacity:.95},{index:2,opacity:.97},{index:1,opacity:.98}];function i(e){var t=e.r,n=e.g,o=e.b,a=(0,r.py)(t,n,o);return{h:360*a.h,s:a.s,v:a.v}}function c(e){var t=e.r,n=e.g,o=e.b;return"#".concat((0,r.vq)(t,n,o,!1))}function l(e,t,n){var r;return(r=Math.round(e.h)>=60&&240>=Math.round(e.h)?n?Math.round(e.h)-2*t:Math.round(e.h)+2*t:n?Math.round(e.h)+2*t:Math.round(e.h)-2*t)<0?r+=360:r>=360&&(r-=360),r}function s(e,t,n){var r;return 0===e.h&&0===e.s?e.s:((r=n?e.s-.16*t:4===t?e.s+.16:e.s+.05*t)>1&&(r=1),n&&5===t&&r>.1&&(r=.1),r<.06&&(r=.06),Number(r.toFixed(2)))}function u(e,t,n){var r;return(r=n?e.v+.05*t:e.v-.15*t)>1&&(r=1),Number(r.toFixed(2))}function d(e){for(var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},n=[],r=(0,o.uA)(e),d=5;d>0;d-=1){var f=i(r),p=c((0,o.uA)({h:l(f,d,!0),s:s(f,d,!0),v:u(f,d,!0)}));n.push(p)}n.push(c(r));for(var m=1;m<=4;m+=1){var g=i(r),h=c((0,o.uA)({h:l(g,m),s:s(g,m),v:u(g,m)}));n.push(h)}return"dark"===t.theme?a.map(function(e){var r,a,i,l=e.index,s=e.opacity;return c((r=(0,o.uA)(t.backgroundColor||"#141414"),a=(0,o.uA)(n[l]),i=100*s/100,{r:(a.r-r.r)*i+r.r,g:(a.g-r.g)*i+r.g,b:(a.b-r.b)*i+r.b}))}):n}var f={red:"#F5222D",volcano:"#FA541C",orange:"#FA8C16",gold:"#FAAD14",yellow:"#FADB14",lime:"#A0D911",green:"#52C41A",cyan:"#13C2C2",blue:"#1677FF",geekblue:"#2F54EB",purple:"#722ED1",magenta:"#EB2F96",grey:"#666666"},p={},m={};Object.keys(f).forEach(function(e){p[e]=d(f[e]),p[e].primary=p[e][5],m[e]=d(f[e],{theme:"dark",backgroundColor:"#141414"}),m[e].primary=m[e][5]}),p.red,p.volcano;var g=p.gold;p.orange,p.yellow,p.lime,p.green,p.cyan;var h=p.blue;p.geekblue,p.purple,p.magenta,p.grey,p.grey},352:function(e,t,n){"use strict";n.d(t,{E4:function(){return eL},jG:function(){return M},ks:function(){return H},bf:function(){return z},CI:function(){return eA},fp:function(){return Y},xy:function(){return eF}});var r,o,a=n(11993),i=n(26365),c=n(83145),l=n(31686),s=function(e){for(var t,n=0,r=0,o=e.length;o>=4;++r,o-=4)t=(65535&(t=255&e.charCodeAt(r)|(255&e.charCodeAt(++r))<<8|(255&e.charCodeAt(++r))<<16|(255&e.charCodeAt(++r))<<24))*1540483477+((t>>>16)*59797<<16),t^=t>>>24,n=(65535&t)*1540483477+((t>>>16)*59797<<16)^(65535&n)*1540483477+((n>>>16)*59797<<16);switch(o){case 3:n^=(255&e.charCodeAt(r+2))<<16;case 2:n^=(255&e.charCodeAt(r+1))<<8;case 1:n^=255&e.charCodeAt(r),n=(65535&n)*1540483477+((n>>>16)*59797<<16)}return n^=n>>>13,(((n=(65535&n)*1540483477+((n>>>16)*59797<<16))^n>>>15)>>>0).toString(36)},u=n(21717),d=n(2265),f=n.t(d,2);n(6397),n(16671);var p=n(76405),m=n(25049);function g(e){return e.join("%")}var h=function(){function e(t){(0,p.Z)(this,e),(0,a.Z)(this,"instanceId",void 0),(0,a.Z)(this,"cache",new Map),this.instanceId=t}return(0,m.Z)(e,[{key:"get",value:function(e){return this.opGet(g(e))}},{key:"opGet",value:function(e){return this.cache.get(e)||null}},{key:"update",value:function(e,t){return this.opUpdate(g(e),t)}},{key:"opUpdate",value:function(e,t){var n=t(this.cache.get(e));null===n?this.cache.delete(e):this.cache.set(e,n)}}]),e}(),v="data-token-hash",b="data-css-hash",y="__cssinjs_instance__",w=d.createContext({hashPriority:"low",cache:function(){var e=Math.random().toString(12).slice(2);if("undefined"!=typeof document&&document.head&&document.body){var t=document.body.querySelectorAll("style[".concat(b,"]"))||[],n=document.head.firstChild;Array.from(t).forEach(function(t){t[y]=t[y]||e,t[y]===e&&document.head.insertBefore(t,n)});var r={};Array.from(document.querySelectorAll("style[".concat(b,"]"))).forEach(function(t){var n,o=t.getAttribute(b);r[o]?t[y]===e&&(null===(n=t.parentNode)||void 0===n||n.removeChild(t)):r[o]=!0})}return new h(e)}(),defaultCache:!0}),x=n(41154),E=n(94981),S=function(){function e(){(0,p.Z)(this,e),(0,a.Z)(this,"cache",void 0),(0,a.Z)(this,"keys",void 0),(0,a.Z)(this,"cacheCallTimes",void 0),this.cache=new Map,this.keys=[],this.cacheCallTimes=0}return(0,m.Z)(e,[{key:"size",value:function(){return this.keys.length}},{key:"internalGet",value:function(e){var t,n,r=arguments.length>1&&void 0!==arguments[1]&&arguments[1],o={map:this.cache};return e.forEach(function(e){if(o){var t;o=null===(t=o)||void 0===t||null===(t=t.map)||void 0===t?void 0:t.get(e)}else o=void 0}),null!==(t=o)&&void 0!==t&&t.value&&r&&(o.value[1]=this.cacheCallTimes++),null===(n=o)||void 0===n?void 0:n.value}},{key:"get",value:function(e){var t;return null===(t=this.internalGet(e,!0))||void 0===t?void 0:t[0]}},{key:"has",value:function(e){return!!this.internalGet(e)}},{key:"set",value:function(t,n){var r=this;if(!this.has(t)){if(this.size()+1>e.MAX_CACHE_SIZE+e.MAX_CACHE_OFFSET){var o=this.keys.reduce(function(e,t){var n=(0,i.Z)(e,2)[1];return r.internalGet(t)[1]0,"[Ant Design CSS-in-JS] Theme should have at least one derivative function."),Z+=1}return(0,m.Z)(e,[{key:"getDerivativeToken",value:function(e){return this.derivatives.reduce(function(t,n){return n(e,t)},void 0)}}]),e}(),k=new S;function M(e){var t=Array.isArray(e)?e:[e];return k.has(t)||k.set(t,new O(t)),k.get(t)}var j=new WeakMap,I={},R=new WeakMap;function N(e){var t=R.get(e)||"";return t||(Object.keys(e).forEach(function(n){var r=e[n];t+=n,r instanceof O?t+=r.id:r&&"object"===(0,x.Z)(r)?t+=N(r):t+=r}),R.set(e,t)),t}function P(e,t){return s("".concat(t,"_").concat(N(e)))}var F="random-".concat(Date.now(),"-").concat(Math.random()).replace(/\./g,""),T="_bAmBoO_",A=void 0,L=(0,E.Z)();function z(e){return"number"==typeof e?"".concat(e,"px"):e}function _(e,t,n){var r,o=arguments.length>3&&void 0!==arguments[3]?arguments[3]:{},i=arguments.length>4&&void 0!==arguments[4]&&arguments[4];if(i)return e;var c=(0,l.Z)((0,l.Z)({},o),{},(r={},(0,a.Z)(r,v,t),(0,a.Z)(r,b,n),r)),s=Object.keys(c).map(function(e){var t=c[e];return t?"".concat(e,'="').concat(t,'"'):null}).filter(function(e){return e}).join(" ");return"")}var H=function(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:"";return"--".concat(t?"".concat(t,"-"):"").concat(e).replace(/([a-z0-9])([A-Z])/g,"$1-$2").replace(/([A-Z]+)([A-Z][a-z0-9]+)/g,"$1-$2").replace(/([a-z])([A-Z0-9])/g,"$1-$2").toLowerCase()},B=function(e,t,n){var r,o={},a={};return Object.entries(e).forEach(function(e){var t=(0,i.Z)(e,2),r=t[0],c=t[1];if(null!=n&&null!==(l=n.preserve)&&void 0!==l&&l[r])a[r]=c;else if(("string"==typeof c||"number"==typeof c)&&!(null!=n&&null!==(s=n.ignore)&&void 0!==s&&s[r])){var l,s,u,d=H(r,null==n?void 0:n.prefix);o[d]="number"!=typeof c||null!=n&&null!==(u=n.unitless)&&void 0!==u&&u[r]?String(c):"".concat(c,"px"),a[r]="var(".concat(d,")")}}),[a,(r={scope:null==n?void 0:n.scope},Object.keys(o).length?".".concat(t).concat(null!=r&&r.scope?".".concat(r.scope):"","{").concat(Object.entries(o).map(function(e){var t=(0,i.Z)(e,2),n=t[0],r=t[1];return"".concat(n,":").concat(r,";")}).join(""),"}"):"")]},D=n(27380),W=(0,l.Z)({},f).useInsertionEffect,V=W?function(e,t,n){return W(function(){return e(),t()},n)}:function(e,t,n){d.useMemo(e,n),(0,D.Z)(function(){return t(!0)},n)},q=void 0!==(0,l.Z)({},f).useInsertionEffect?function(e){var t=[],n=!1;return d.useEffect(function(){return n=!1,function(){n=!0,t.length&&t.forEach(function(e){return e()})}},e),function(e){n||t.push(e)}}:function(){return function(e){e()}};function G(e,t,n,r,o){var a=d.useContext(w).cache,l=g([e].concat((0,c.Z)(t))),s=q([l]),u=function(e){a.opUpdate(l,function(t){var r=(0,i.Z)(t||[void 0,void 0],2),o=r[0],a=[void 0===o?0:o,r[1]||n()];return e?e(a):a})};d.useMemo(function(){u()},[l]);var f=a.opGet(l)[1];return V(function(){null==o||o(f)},function(e){return u(function(t){var n=(0,i.Z)(t,2),r=n[0],a=n[1];return e&&0===r&&(null==o||o(f)),[r+1,a]}),function(){a.opUpdate(l,function(t){var n=(0,i.Z)(t||[],2),o=n[0],c=void 0===o?0:o,u=n[1];return 0==c-1?(s(function(){(e||!a.opGet(l))&&(null==r||r(u,!1))}),null):[c-1,u]})}},[l]),f}var X={},U=new Map,$=function(e,t,n,r){var o=n.getDerivativeToken(e),a=(0,l.Z)((0,l.Z)({},o),t);return r&&(a=r(a)),a},K="token";function Y(e,t){var n=arguments.length>2&&void 0!==arguments[2]?arguments[2]:{},r=(0,d.useContext)(w),o=r.cache.instanceId,a=r.container,f=n.salt,p=void 0===f?"":f,m=n.override,g=void 0===m?X:m,h=n.formatToken,x=n.getComputedToken,E=n.cssVar,S=function(e,t){for(var n=j,r=0;r=(U.get(e)||0)}),n.length-r.length>0&&r.forEach(function(e){"undefined"!=typeof document&&document.querySelectorAll("style[".concat(v,'="').concat(e,'"]')).forEach(function(e){if(e[y]===o){var t;null===(t=e.parentNode)||void 0===t||t.removeChild(e)}}),U.delete(e)})},function(e){var t=(0,i.Z)(e,4),n=t[0],r=t[3];if(E&&r){var c=(0,u.hq)(r,s("css-variables-".concat(n._themeKey)),{mark:b,prepend:"queue",attachTo:a,priority:-999});c[y]=o,c.setAttribute(v,n._themeKey)}})}var Q=n(1119),J={animationIterationCount:1,borderImageOutset:1,borderImageSlice:1,borderImageWidth:1,boxFlex:1,boxFlexGroup:1,boxOrdinalGroup:1,columnCount:1,columns:1,flex:1,flexGrow:1,flexPositive:1,flexShrink:1,flexNegative:1,flexOrder:1,gridRow:1,gridRowEnd:1,gridRowSpan:1,gridRowStart:1,gridColumn:1,gridColumnEnd:1,gridColumnSpan:1,gridColumnStart:1,msGridRow:1,msGridRowSpan:1,msGridColumn:1,msGridColumnSpan:1,fontWeight:1,lineHeight:1,opacity:1,order:1,orphans:1,tabSize:1,widows:1,zIndex:1,zoom:1,WebkitLineClamp:1,fillOpacity:1,floodOpacity:1,stopOpacity:1,strokeDasharray:1,strokeDashoffset:1,strokeMiterlimit:1,strokeOpacity:1,strokeWidth:1},ee="comm",et="rule",en="decl",er=Math.abs,eo=String.fromCharCode;function ea(e,t,n){return e.replace(t,n)}function ei(e,t){return 0|e.charCodeAt(t)}function ec(e,t,n){return e.slice(t,n)}function el(e){return e.length}function es(e,t){return t.push(e),e}function eu(e,t){for(var n="",r=0;r0?p[b]+" "+y:ea(y,/&\f/g,p[b])).trim())&&(l[v++]=w);return eb(e,t,n,0===o?et:c,l,s,u,d)}function eC(e,t,n,r,o){return eb(e,t,n,en,ec(e,0,r),ec(e,r+1,-1),r,o)}var eZ="data-ant-cssinjs-cache-path",eO="_FILE_STYLE__",ek=!0,eM="_multi_value_";function ej(e){var t,n,r;return eu((r=function e(t,n,r,o,a,i,c,l,s){for(var u,d,f,p=0,m=0,g=c,h=0,v=0,b=0,y=1,w=1,x=1,E=0,S="",C=a,Z=i,O=o,k=S;w;)switch(b=E,E=ey()){case 40:if(108!=b&&58==ei(k,g-1)){-1!=(d=k+=ea(eE(E),"&","&\f"),f=er(p?l[p-1]:0),d.indexOf("&\f",f))&&(x=-1);break}case 34:case 39:case 91:k+=eE(E);break;case 9:case 10:case 13:case 32:k+=function(e){for(;eh=ew();)if(eh<33)ey();else break;return ex(e)>2||ex(eh)>3?"":" "}(b);break;case 92:k+=function(e,t){for(var n;--t&&ey()&&!(eh<48)&&!(eh>102)&&(!(eh>57)||!(eh<65))&&(!(eh>70)||!(eh<97)););return n=eg+(t<6&&32==ew()&&32==ey()),ec(ev,e,n)}(eg-1,7);continue;case 47:switch(ew()){case 42:case 47:es(eb(u=function(e,t){for(;ey();)if(e+eh===57)break;else if(e+eh===84&&47===ew())break;return"/*"+ec(ev,t,eg-1)+"*"+eo(47===e?e:ey())}(ey(),eg),n,r,ee,eo(eh),ec(u,2,-2),0,s),s),(5==ex(b||1)||5==ex(ew()||1))&&el(k)&&" "!==ec(k,-1,void 0)&&(k+=" ");break;default:k+="/"}break;case 123*y:l[p++]=el(k)*x;case 125*y:case 59:case 0:switch(E){case 0:case 125:w=0;case 59+m:-1==x&&(k=ea(k,/\f/g,"")),v>0&&(el(k)-g||0===y&&47===b)&&es(v>32?eC(k+";",o,r,g-1,s):eC(ea(k," ","")+";",o,r,g-2,s),s);break;case 59:k+=";";default:if(es(O=eS(k,n,r,p,m,a,l,S,C=[],Z=[],g,i),i),123===E){if(0===m)e(k,n,O,O,C,i,g,l,Z);else{switch(h){case 99:if(110===ei(k,3))break;case 108:if(97===ei(k,2))break;default:m=0;case 100:case 109:case 115:}m?e(t,O,O,o&&es(eS(t,O,O,0,0,a,l,S,a,C=[],g,Z),Z),a,Z,g,l,o?C:Z):e(k,O,O,O,[""],Z,0,l,Z)}}}p=m=v=0,y=x=1,S=k="",g=c;break;case 58:g=1+el(k),v=b;default:if(y<1){if(123==E)--y;else if(125==E&&0==y++&&125==(eh=eg>0?ei(ev,--eg):0,ep--,10===eh&&(ep=1,ef--),eh))continue}switch(k+=eo(E),E*y){case 38:x=m>0?1:(k+="\f",-1);break;case 44:l[p++]=(el(k)-1)*x,x=1;break;case 64:45===ew()&&(k+=eE(ey())),h=ew(),m=g=el(S=k+=function(e){for(;!ex(ew());)ey();return ec(ev,e,eg)}(eg)),E++;break;case 45:45===b&&2==el(k)&&(y=0)}}return i}("",null,null,null,[""],(n=t=e,ef=ep=1,em=el(ev=n),eg=0,t=[]),0,[0],t),ev="",r),ed).replace(/\{%%%\:[^;];}/g,";")}var eI=function e(t){var n=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},r=arguments.length>2&&void 0!==arguments[2]?arguments[2]:{root:!0,parentSelectors:[]},o=r.root,a=r.injectHash,s=r.parentSelectors,d=n.hashId,f=n.layer,p=(n.path,n.hashPriority),m=n.transformers,g=void 0===m?[]:m;n.linters;var h="",v={};function b(t){var r=t.getName(d);if(!v[r]){var o=e(t.style,n,{root:!1,parentSelectors:s}),a=(0,i.Z)(o,1)[0];v[r]="@keyframes ".concat(t.getName(d)).concat(a)}}if((function e(t){var n=arguments.length>1&&void 0!==arguments[1]?arguments[1]:[];return t.forEach(function(t){Array.isArray(t)?e(t,n):t&&n.push(t)}),n})(Array.isArray(t)?t:[t]).forEach(function(t){var r="string"!=typeof t||o?t:{};if("string"==typeof r)h+="".concat(r,"\n");else if(r._keyframe)b(r);else{var u=g.reduce(function(e,t){var n;return(null==t||null===(n=t.visit)||void 0===n?void 0:n.call(t,e))||e},r);Object.keys(u).forEach(function(t){var r=u[t];if("object"!==(0,x.Z)(r)||!r||"animationName"===t&&r._keyframe||"object"===(0,x.Z)(r)&&r&&("_skip_check_"in r||eM in r)){function f(e,t){var n=e.replace(/[A-Z]/g,function(e){return"-".concat(e.toLowerCase())}),r=t;J[e]||"number"!=typeof r||0===r||(r="".concat(r,"px")),"animationName"===e&&null!=t&&t._keyframe&&(b(t),r=t.getName(d)),h+="".concat(n,":").concat(r,";")}var m,g=null!==(m=null==r?void 0:r.value)&&void 0!==m?m:r;"object"===(0,x.Z)(r)&&null!=r&&r[eM]&&Array.isArray(g)?g.forEach(function(e){f(t,e)}):f(t,g)}else{var y=!1,w=t.trim(),E=!1;(o||a)&&d?w.startsWith("@")?y=!0:w=function(e,t,n){if(!t)return e;var r=".".concat(t),o="low"===n?":where(".concat(r,")"):r;return e.split(",").map(function(e){var t,n=e.trim().split(/\s+/),r=n[0]||"",a=(null===(t=r.match(/^\w+/))||void 0===t?void 0:t[0])||"";return[r="".concat(a).concat(o).concat(r.slice(a.length))].concat((0,c.Z)(n.slice(1))).join(" ")}).join(",")}(t,d,p):o&&!d&&("&"===w||""===w)&&(w="",E=!0);var S=e(r,n,{root:E,injectHash:y,parentSelectors:[].concat((0,c.Z)(s),[w])}),C=(0,i.Z)(S,2),Z=C[0],O=C[1];v=(0,l.Z)((0,l.Z)({},v),O),h+="".concat(w).concat(Z)}})}}),o){if(f&&(void 0===A&&(A=function(e,t,n){if((0,E.Z)()){(0,u.hq)(e,F);var r,o,a=document.createElement("div");a.style.position="fixed",a.style.left="0",a.style.top="0",null==t||t(a),document.body.appendChild(a);var i=null===(r=getComputedStyle(a).content)||void 0===r?void 0:r.includes(T);return null===(o=a.parentNode)||void 0===o||o.removeChild(a),(0,u.jL)(F),i}return!1}("@layer ".concat(F," { .").concat(F,' { content: "').concat(T,'"!important; } }'),function(e){e.className=F})),A)){var y=f.split(","),w=y[y.length-1].trim();h="@layer ".concat(w," {").concat(h,"}"),y.length>1&&(h="@layer ".concat(f,"{%%%:%}").concat(h))}}else h="{".concat(h,"}");return[h,v]};function eR(e,t){return s("".concat(e.join("%")).concat(t))}function eN(){return null}var eP="style";function eF(e,t){var n=e.token,o=e.path,l=e.hashId,s=e.layer,f=e.nonce,p=e.clientOnly,m=e.order,g=void 0===m?0:m,h=d.useContext(w),x=h.autoClear,S=(h.mock,h.defaultCache),C=h.hashPriority,Z=h.container,O=h.ssrInline,k=h.transformers,M=h.linters,j=h.cache,I=n._tokenKey,R=[I].concat((0,c.Z)(o)),N=G(eP,R,function(){var e=R.join("|");if(!function(){if(!r&&(r={},(0,E.Z)())){var e,t=document.createElement("div");t.className=eZ,t.style.position="fixed",t.style.visibility="hidden",t.style.top="-9999px",document.body.appendChild(t);var n=getComputedStyle(t).content||"";(n=n.replace(/^"/,"").replace(/"$/,"")).split(";").forEach(function(e){var t=e.split(":"),n=(0,i.Z)(t,2),o=n[0],a=n[1];r[o]=a});var o=document.querySelector("style[".concat(eZ,"]"));o&&(ek=!1,null===(e=o.parentNode)||void 0===e||e.removeChild(o)),document.body.removeChild(t)}}(),r[e]){var n=function(e){var t=r[e],n=null;if(t&&(0,E.Z)()){if(ek)n=eO;else{var o=document.querySelector("style[".concat(b,'="').concat(r[e],'"]'));o?n=o.innerHTML:delete r[e]}}return[n,t]}(e),a=(0,i.Z)(n,2),c=a[0],u=a[1];if(c)return[c,I,u,{},p,g]}var d=eI(t(),{hashId:l,hashPriority:C,layer:s,path:o.join("-"),transformers:k,linters:M}),f=(0,i.Z)(d,2),m=f[0],h=f[1],v=ej(m),y=eR(R,v);return[v,I,y,h,p,g]},function(e,t){var n=(0,i.Z)(e,3)[2];(t||x)&&L&&(0,u.jL)(n,{mark:b})},function(e){var t=(0,i.Z)(e,4),n=t[0],r=(t[1],t[2]),o=t[3];if(L&&n!==eO){var a={mark:b,prepend:"queue",attachTo:Z,priority:g},c="function"==typeof f?f():f;c&&(a.csp={nonce:c});var l=(0,u.hq)(n,r,a);l[y]=j.instanceId,l.setAttribute(v,I),Object.keys(o).forEach(function(e){(0,u.hq)(ej(o[e]),"_effect-".concat(e),a)})}}),P=(0,i.Z)(N,3),F=P[0],T=P[1],A=P[2];return function(e){var t,n;return t=O&&!L&&S?d.createElement("style",(0,Q.Z)({},(n={},(0,a.Z)(n,v,T),(0,a.Z)(n,b,A),n),{dangerouslySetInnerHTML:{__html:F}})):d.createElement(eN,null),d.createElement(d.Fragment,null,t,e)}}var eT="cssVar",eA=function(e,t){var n=e.key,r=e.prefix,o=e.unitless,a=e.ignore,l=e.token,s=e.scope,f=void 0===s?"":s,p=(0,d.useContext)(w),m=p.cache.instanceId,g=p.container,h=l._tokenKey,x=[].concat((0,c.Z)(e.path),[n,f,h]);return G(eT,x,function(){var e=B(t(),n,{prefix:r,unitless:o,ignore:a,scope:f}),c=(0,i.Z)(e,2),l=c[0],s=c[1],u=eR(x,s);return[l,s,u,n]},function(e){var t=(0,i.Z)(e,3)[2];L&&(0,u.jL)(t,{mark:b})},function(e){var t=(0,i.Z)(e,3),r=t[1],o=t[2];if(r){var a=(0,u.hq)(r,o,{mark:b,prepend:"queue",attachTo:g,priority:-999});a[y]=m,a.setAttribute(v,n)}})};o={},(0,a.Z)(o,eP,function(e,t,n){var r=(0,i.Z)(e,6),o=r[0],a=r[1],c=r[2],l=r[3],s=r[4],u=r[5],d=(n||{}).plain;if(s)return null;var f=o,p={"data-rc-order":"prependQueue","data-rc-priority":"".concat(u)};return f=_(o,a,c,p,d),l&&Object.keys(l).forEach(function(e){if(!t[e]){t[e]=!0;var n=ej(l[e]);f+=_(n,a,"_effect-".concat(e),p,d)}}),[u,c,f]}),(0,a.Z)(o,K,function(e,t,n){var r=(0,i.Z)(e,5),o=r[2],a=r[3],c=r[4],l=(n||{}).plain;if(!a)return null;var s=o._tokenKey,u=_(a,c,s,{"data-rc-order":"prependQueue","data-rc-priority":"".concat(-999)},l);return[-999,s,u]}),(0,a.Z)(o,eT,function(e,t,n){var r=(0,i.Z)(e,4),o=r[1],a=r[2],c=r[3],l=(n||{}).plain;if(!o)return null;var s=_(o,c,a,{"data-rc-order":"prependQueue","data-rc-priority":"".concat(-999)},l);return[-999,a,s]});var eL=function(){function e(t,n){(0,p.Z)(this,e),(0,a.Z)(this,"name",void 0),(0,a.Z)(this,"style",void 0),(0,a.Z)(this,"_keyframe",!0),this.name=t,this.style=n}return(0,m.Z)(e,[{key:"getName",value:function(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:"";return e?"".concat(e,"-").concat(this.name):this.name}}]),e}();function ez(e){return e.notSplit=!0,e}ez(["borderTop","borderBottom"]),ez(["borderTop"]),ez(["borderBottom"]),ez(["borderLeft","borderRight"]),ez(["borderLeft"]),ez(["borderRight"])},55015:function(e,t,n){"use strict";n.d(t,{Z:function(){return M}});var r=n(1119),o=n(26365),a=n(11993),i=n(6989),c=n(2265),l=n(36760),s=n.n(l),u=n(31373),d=n(20902),f=n(31686),p=n(41154),m=n(21717),g=n(13211),h=n(32559);function v(e){return"object"===(0,p.Z)(e)&&"string"==typeof e.name&&"string"==typeof e.theme&&("object"===(0,p.Z)(e.icon)||"function"==typeof e.icon)}function b(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{};return Object.keys(e).reduce(function(t,n){var r=e[n];return"class"===n?(t.className=r,delete t.class):(delete t[n],t[n.replace(/-(.)/g,function(e,t){return t.toUpperCase()})]=r),t},{})}function y(e){return(0,u.R_)(e)[0]}function w(e){return e?Array.isArray(e)?e:[e]:[]}var x=function(e){var t=(0,c.useContext)(d.Z),n=t.csp,r=t.prefixCls,o="\n.anticon {\n display: inline-block;\n color: inherit;\n font-style: normal;\n line-height: 0;\n text-align: center;\n text-transform: none;\n vertical-align: -0.125em;\n text-rendering: optimizeLegibility;\n -webkit-font-smoothing: antialiased;\n -moz-osx-font-smoothing: grayscale;\n}\n\n.anticon > * {\n line-height: 1;\n}\n\n.anticon svg {\n display: inline-block;\n}\n\n.anticon::before {\n display: none;\n}\n\n.anticon .anticon-icon {\n display: block;\n}\n\n.anticon[tabindex] {\n cursor: pointer;\n}\n\n.anticon-spin::before,\n.anticon-spin {\n display: inline-block;\n -webkit-animation: loadingCircle 1s infinite linear;\n animation: loadingCircle 1s infinite linear;\n}\n\n@-webkit-keyframes loadingCircle {\n 100% {\n -webkit-transform: rotate(360deg);\n transform: rotate(360deg);\n }\n}\n\n@keyframes loadingCircle {\n 100% {\n -webkit-transform: rotate(360deg);\n transform: rotate(360deg);\n }\n}\n";r&&(o=o.replace(/anticon/g,r)),(0,c.useEffect)(function(){var t=e.current,r=(0,g.A)(t);(0,m.hq)(o,"@ant-design-icons",{prepend:!0,csp:n,attachTo:r})},[])},E=["icon","className","onClick","style","primaryColor","secondaryColor"],S={primaryColor:"#333",secondaryColor:"#E6E6E6",calculated:!1},C=function(e){var t,n,r=e.icon,o=e.className,a=e.onClick,l=e.style,s=e.primaryColor,u=e.secondaryColor,d=(0,i.Z)(e,E),p=c.useRef(),m=S;if(s&&(m={primaryColor:s,secondaryColor:u||y(s)}),x(p),t=v(r),n="icon should be icon definiton, but got ".concat(r),(0,h.ZP)(t,"[@ant-design/icons] ".concat(n)),!v(r))return null;var g=r;return g&&"function"==typeof g.icon&&(g=(0,f.Z)((0,f.Z)({},g),{},{icon:g.icon(m.primaryColor,m.secondaryColor)})),function e(t,n,r){return r?c.createElement(t.tag,(0,f.Z)((0,f.Z)({key:n},b(t.attrs)),r),(t.children||[]).map(function(r,o){return e(r,"".concat(n,"-").concat(t.tag,"-").concat(o))})):c.createElement(t.tag,(0,f.Z)({key:n},b(t.attrs)),(t.children||[]).map(function(r,o){return e(r,"".concat(n,"-").concat(t.tag,"-").concat(o))}))}(g.icon,"svg-".concat(g.name),(0,f.Z)((0,f.Z)({className:o,onClick:a,style:l,"data-icon":g.name,width:"1em",height:"1em",fill:"currentColor","aria-hidden":"true"},d),{},{ref:p}))};function Z(e){var t=w(e),n=(0,o.Z)(t,2),r=n[0],a=n[1];return C.setTwoToneColors({primaryColor:r,secondaryColor:a})}C.displayName="IconReact",C.getTwoToneColors=function(){return(0,f.Z)({},S)},C.setTwoToneColors=function(e){var t=e.primaryColor,n=e.secondaryColor;S.primaryColor=t,S.secondaryColor=n||y(t),S.calculated=!!n};var O=["className","icon","spin","rotate","tabIndex","onClick","twoToneColor"];Z(u.iN.primary);var k=c.forwardRef(function(e,t){var n,l=e.className,u=e.icon,f=e.spin,p=e.rotate,m=e.tabIndex,g=e.onClick,h=e.twoToneColor,v=(0,i.Z)(e,O),b=c.useContext(d.Z),y=b.prefixCls,x=void 0===y?"anticon":y,E=b.rootClassName,S=s()(E,x,(n={},(0,a.Z)(n,"".concat(x,"-").concat(u.name),!!u.name),(0,a.Z)(n,"".concat(x,"-spin"),!!f||"loading"===u.name),n),l),Z=m;void 0===Z&&g&&(Z=-1);var k=w(h),M=(0,o.Z)(k,2),j=M[0],I=M[1];return c.createElement("span",(0,r.Z)({role:"img","aria-label":u.name},v,{ref:t,tabIndex:Z,onClick:g,className:S}),c.createElement(C,{icon:u,primaryColor:j,secondaryColor:I,style:p?{msTransform:"rotate(".concat(p,"deg)"),transform:"rotate(".concat(p,"deg)")}:void 0}))});k.displayName="AntdIcon",k.getTwoToneColor=function(){var e=C.getTwoToneColors();return e.calculated?[e.primaryColor,e.secondaryColor]:e.primaryColor},k.setTwoToneColor=Z;var M=k},20902:function(e,t,n){"use strict";var r=(0,n(2265).createContext)({});t.Z=r},8900:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M512 64C264.6 64 64 264.6 64 512s200.6 448 448 448 448-200.6 448-448S759.4 64 512 64zm193.5 301.7l-210.6 292a31.8 31.8 0 01-51.7 0L318.5 484.9c-3.8-5.3 0-12.7 6.5-12.7h46.9c10.2 0 19.9 4.9 25.9 13.3l71.2 98.8 157.2-218c6-8.3 15.6-13.3 25.9-13.3H699c6.5 0 10.3 7.4 6.5 12.7z"}}]},name:"check-circle",theme:"filled"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},9738:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M912 190h-69.9c-9.8 0-19.1 4.5-25.1 12.2L404.7 724.5 207 474a32 32 0 00-25.1-12.2H112c-6.7 0-10.4 7.7-6.3 12.9l273.9 347c12.8 16.2 37.4 16.2 50.3 0l488.4-618.9c4.1-5.1.4-12.8-6.3-12.8z"}}]},name:"check",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},39725:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{"fill-rule":"evenodd",viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M512 64c247.4 0 448 200.6 448 448S759.4 960 512 960 64 759.4 64 512 264.6 64 512 64zm127.98 274.82h-.04l-.08.06L512 466.75 384.14 338.88c-.04-.05-.06-.06-.08-.06a.12.12 0 00-.07 0c-.03 0-.05.01-.09.05l-45.02 45.02a.2.2 0 00-.05.09.12.12 0 000 .07v.02a.27.27 0 00.06.06L466.75 512 338.88 639.86c-.05.04-.06.06-.06.08a.12.12 0 000 .07c0 .03.01.05.05.09l45.02 45.02a.2.2 0 00.09.05.12.12 0 00.07 0c.02 0 .04-.01.08-.05L512 557.25l127.86 127.87c.04.04.06.05.08.05a.12.12 0 00.07 0c.03 0 .05-.01.09-.05l45.02-45.02a.2.2 0 00.05-.09.12.12 0 000-.07v-.02a.27.27 0 00-.05-.06L557.25 512l127.87-127.86c.04-.04.05-.06.05-.08a.12.12 0 000-.07c0-.03-.01-.05-.05-.09l-45.02-45.02a.2.2 0 00-.09-.05.12.12 0 00-.07 0z"}}]},name:"close-circle",theme:"filled"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},49638:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{"fill-rule":"evenodd",viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M799.86 166.31c.02 0 .04.02.08.06l57.69 57.7c.04.03.05.05.06.08a.12.12 0 010 .06c0 .03-.02.05-.06.09L569.93 512l287.7 287.7c.04.04.05.06.06.09a.12.12 0 010 .07c0 .02-.02.04-.06.08l-57.7 57.69c-.03.04-.05.05-.07.06a.12.12 0 01-.07 0c-.03 0-.05-.02-.09-.06L512 569.93l-287.7 287.7c-.04.04-.06.05-.09.06a.12.12 0 01-.07 0c-.02 0-.04-.02-.08-.06l-57.69-57.7c-.04-.03-.05-.05-.06-.07a.12.12 0 010-.07c0-.03.02-.05.06-.09L454.07 512l-287.7-287.7c-.04-.04-.05-.06-.06-.09a.12.12 0 010-.07c0-.02.02-.04.06-.08l57.7-57.69c.03-.04.05-.05.07-.06a.12.12 0 01.07 0c.03 0 .05.02.09.06L512 454.07l287.7-287.7c.04-.04.06-.05.09-.06a.12.12 0 01.07 0z"}}]},name:"close",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},70464:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M884 256h-75c-5.1 0-9.9 2.5-12.9 6.6L512 654.2 227.9 262.6c-3-4.1-7.8-6.6-12.9-6.6h-75c-6.5 0-10.3 7.4-6.5 12.7l352.6 486.1c12.8 17.6 39 17.6 51.7 0l352.6-486.1c3.9-5.3.1-12.7-6.4-12.7z"}}]},name:"down",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},54537:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M512 64C264.6 64 64 264.6 64 512s200.6 448 448 448 448-200.6 448-448S759.4 64 512 64zm-32 232c0-4.4 3.6-8 8-8h48c4.4 0 8 3.6 8 8v272c0 4.4-3.6 8-8 8h-48c-4.4 0-8-3.6-8-8V296zm32 440a48.01 48.01 0 010-96 48.01 48.01 0 010 96z"}}]},name:"exclamation-circle",theme:"filled"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},97416:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M942.2 486.2Q889.47 375.11 816.7 305l-50.88 50.88C807.31 395.53 843.45 447.4 874.7 512 791.5 684.2 673.4 766 512 766q-72.67 0-133.87-22.38L323 798.75Q408 838 512 838q288.3 0 430.2-300.3a60.29 60.29 0 000-51.5zm-63.57-320.64L836 122.88a8 8 0 00-11.32 0L715.31 232.2Q624.86 186 512 186q-288.3 0-430.2 300.3a60.3 60.3 0 000 51.5q56.69 119.4 136.5 191.41L112.48 835a8 8 0 000 11.31L155.17 889a8 8 0 0011.31 0l712.15-712.12a8 8 0 000-11.32zM149.3 512C232.6 339.8 350.7 258 512 258c54.54 0 104.13 9.36 149.12 28.39l-70.3 70.3a176 176 0 00-238.13 238.13l-83.42 83.42C223.1 637.49 183.3 582.28 149.3 512zm246.7 0a112.11 112.11 0 01146.2-106.69L401.31 546.2A112 112 0 01396 512z"}},{tag:"path",attrs:{d:"M508 624c-3.46 0-6.87-.16-10.25-.47l-52.82 52.82a176.09 176.09 0 00227.42-227.42l-52.82 52.82c.31 3.38.47 6.79.47 10.25a111.94 111.94 0 01-112 112z"}}]},name:"eye-invisible",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},6520:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M942.2 486.2C847.4 286.5 704.1 186 512 186c-192.2 0-335.4 100.5-430.2 300.3a60.3 60.3 0 000 51.5C176.6 737.5 319.9 838 512 838c192.2 0 335.4-100.5 430.2-300.3 7.7-16.2 7.7-35 0-51.5zM512 766c-161.3 0-279.4-81.8-362.7-254C232.6 339.8 350.7 258 512 258c161.3 0 279.4 81.8 362.7 254C791.5 684.2 673.4 766 512 766zm-4-430c-97.2 0-176 78.8-176 176s78.8 176 176 176 176-78.8 176-176-78.8-176-176-176zm0 288c-61.9 0-112-50.1-112-112s50.1-112 112-112 112 50.1 112 112-50.1 112-112 112z"}}]},name:"eye",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},55726:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M512 64C264.6 64 64 264.6 64 512s200.6 448 448 448 448-200.6 448-448S759.4 64 512 64zm32 664c0 4.4-3.6 8-8 8h-48c-4.4 0-8-3.6-8-8V456c0-4.4 3.6-8 8-8h48c4.4 0 8 3.6 8 8v272zm-32-344a48.01 48.01 0 010-96 48.01 48.01 0 010 96z"}}]},name:"info-circle",theme:"filled"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},15424:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M512 64C264.6 64 64 264.6 64 512s200.6 448 448 448 448-200.6 448-448S759.4 64 512 64zm0 820c-205.4 0-372-166.6-372-372s166.6-372 372-372 372 166.6 372 372-166.6 372-372 372z"}},{tag:"path",attrs:{d:"M464 336a48 48 0 1096 0 48 48 0 10-96 0zm72 112h-48c-4.4 0-8 3.6-8 8v272c0 4.4 3.6 8 8 8h48c4.4 0 8-3.6 8-8V456c0-4.4-3.6-8-8-8z"}}]},name:"info-circle",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},61935:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"0 0 1024 1024",focusable:"false"},children:[{tag:"path",attrs:{d:"M988 548c-19.9 0-36-16.1-36-36 0-59.4-11.6-117-34.6-171.3a440.45 440.45 0 00-94.3-139.9 437.71 437.71 0 00-139.9-94.3C629 83.6 571.4 72 512 72c-19.9 0-36-16.1-36-36s16.1-36 36-36c69.1 0 136.2 13.5 199.3 40.3C772.3 66 827 103 874 150c47 47 83.9 101.8 109.7 162.7 26.7 63.1 40.2 130.2 40.2 199.3.1 19.9-16 36-35.9 36z"}}]},name:"loading",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},67187:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M512 64C264.6 64 64 264.6 64 512s200.6 448 448 448 448-200.6 448-448S759.4 64 512 64zm0 820c-205.4 0-372-166.6-372-372s166.6-372 372-372 372 166.6 372 372-166.6 372-372 372z"}},{tag:"path",attrs:{d:"M623.6 316.7C593.6 290.4 554 276 512 276s-81.6 14.5-111.6 40.7C369.2 344 352 380.7 352 420v7.6c0 4.4 3.6 8 8 8h48c4.4 0 8-3.6 8-8V420c0-44.1 43.1-80 96-80s96 35.9 96 80c0 31.1-22 59.6-56.1 72.7-21.2 8.1-39.2 22.3-52.1 40.9-13.1 19-19.9 41.8-19.9 64.9V620c0 4.4 3.6 8 8 8h48c4.4 0 8-3.6 8-8v-22.7a48.3 48.3 0 0130.9-44.8c59-22.7 97.1-74.7 97.1-132.5.1-39.3-17.1-76-48.3-103.3zM472 732a40 40 0 1080 0 40 40 0 10-80 0z"}}]},name:"question-circle",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},29436:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(1119),o=n(2265),a={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M909.6 854.5L649.9 594.8C690.2 542.7 712 479 712 412c0-80.2-31.3-155.4-87.9-212.1-56.6-56.7-132-87.9-212.1-87.9s-155.5 31.3-212.1 87.9C143.2 256.5 112 331.8 112 412c0 80.1 31.3 155.5 87.9 212.1C256.5 680.8 331.8 712 412 712c67 0 130.6-21.8 182.7-62l259.7 259.6a8.2 8.2 0 0011.6 0l43.6-43.5a8.2 8.2 0 000-11.6zM570.4 570.4C528 612.7 471.8 636 412 636s-116-23.3-158.4-65.6C211.3 528 188 471.8 188 412s23.3-116.1 65.6-158.4C296 211.3 352.2 188 412 188s116.1 23.2 158.4 65.6S636 352.2 636 412s-23.3 116.1-65.6 158.4z"}}]},name:"search",theme:"outlined"},i=n(55015),c=o.forwardRef(function(e,t){return o.createElement(i.Z,(0,r.Z)({},e,{ref:t,icon:a}))})},82082:function(e,t,n){"use strict";n.d(t,{T6:function(){return f},VD:function(){return p},WE:function(){return s},Yt:function(){return m},lC:function(){return a},py:function(){return l},rW:function(){return o},s:function(){return d},ve:function(){return c},vq:function(){return u}});var r=n(58317);function o(e,t,n){return{r:255*(0,r.sh)(e,255),g:255*(0,r.sh)(t,255),b:255*(0,r.sh)(n,255)}}function a(e,t,n){var o=Math.max(e=(0,r.sh)(e,255),t=(0,r.sh)(t,255),n=(0,r.sh)(n,255)),a=Math.min(e,t,n),i=0,c=0,l=(o+a)/2;if(o===a)c=0,i=0;else{var s=o-a;switch(c=l>.5?s/(2-o-a):s/(o+a),o){case e:i=(t-n)/s+(t1&&(n-=1),n<1/6)?e+6*n*(t-e):n<.5?t:n<2/3?e+(t-e)*(2/3-n)*6:e}function c(e,t,n){if(e=(0,r.sh)(e,360),t=(0,r.sh)(t,100),n=(0,r.sh)(n,100),0===t)a=n,c=n,o=n;else{var o,a,c,l=n<.5?n*(1+t):n+t-n*t,s=2*n-l;o=i(s,l,e+1/3),a=i(s,l,e),c=i(s,l,e-1/3)}return{r:255*o,g:255*a,b:255*c}}function l(e,t,n){var o=Math.max(e=(0,r.sh)(e,255),t=(0,r.sh)(t,255),n=(0,r.sh)(n,255)),a=Math.min(e,t,n),i=0,c=o-a;if(o===a)i=0;else{switch(o){case e:i=(t-n)/c+(t>16,g:(65280&e)>>8,b:255&e}}},28052:function(e,t,n){"use strict";n.d(t,{R:function(){return r}});var r={aliceblue:"#f0f8ff",antiquewhite:"#faebd7",aqua:"#00ffff",aquamarine:"#7fffd4",azure:"#f0ffff",beige:"#f5f5dc",bisque:"#ffe4c4",black:"#000000",blanchedalmond:"#ffebcd",blue:"#0000ff",blueviolet:"#8a2be2",brown:"#a52a2a",burlywood:"#deb887",cadetblue:"#5f9ea0",chartreuse:"#7fff00",chocolate:"#d2691e",coral:"#ff7f50",cornflowerblue:"#6495ed",cornsilk:"#fff8dc",crimson:"#dc143c",cyan:"#00ffff",darkblue:"#00008b",darkcyan:"#008b8b",darkgoldenrod:"#b8860b",darkgray:"#a9a9a9",darkgreen:"#006400",darkgrey:"#a9a9a9",darkkhaki:"#bdb76b",darkmagenta:"#8b008b",darkolivegreen:"#556b2f",darkorange:"#ff8c00",darkorchid:"#9932cc",darkred:"#8b0000",darksalmon:"#e9967a",darkseagreen:"#8fbc8f",darkslateblue:"#483d8b",darkslategray:"#2f4f4f",darkslategrey:"#2f4f4f",darkturquoise:"#00ced1",darkviolet:"#9400d3",deeppink:"#ff1493",deepskyblue:"#00bfff",dimgray:"#696969",dimgrey:"#696969",dodgerblue:"#1e90ff",firebrick:"#b22222",floralwhite:"#fffaf0",forestgreen:"#228b22",fuchsia:"#ff00ff",gainsboro:"#dcdcdc",ghostwhite:"#f8f8ff",goldenrod:"#daa520",gold:"#ffd700",gray:"#808080",green:"#008000",greenyellow:"#adff2f",grey:"#808080",honeydew:"#f0fff0",hotpink:"#ff69b4",indianred:"#cd5c5c",indigo:"#4b0082",ivory:"#fffff0",khaki:"#f0e68c",lavenderblush:"#fff0f5",lavender:"#e6e6fa",lawngreen:"#7cfc00",lemonchiffon:"#fffacd",lightblue:"#add8e6",lightcoral:"#f08080",lightcyan:"#e0ffff",lightgoldenrodyellow:"#fafad2",lightgray:"#d3d3d3",lightgreen:"#90ee90",lightgrey:"#d3d3d3",lightpink:"#ffb6c1",lightsalmon:"#ffa07a",lightseagreen:"#20b2aa",lightskyblue:"#87cefa",lightslategray:"#778899",lightslategrey:"#778899",lightsteelblue:"#b0c4de",lightyellow:"#ffffe0",lime:"#00ff00",limegreen:"#32cd32",linen:"#faf0e6",magenta:"#ff00ff",maroon:"#800000",mediumaquamarine:"#66cdaa",mediumblue:"#0000cd",mediumorchid:"#ba55d3",mediumpurple:"#9370db",mediumseagreen:"#3cb371",mediumslateblue:"#7b68ee",mediumspringgreen:"#00fa9a",mediumturquoise:"#48d1cc",mediumvioletred:"#c71585",midnightblue:"#191970",mintcream:"#f5fffa",mistyrose:"#ffe4e1",moccasin:"#ffe4b5",navajowhite:"#ffdead",navy:"#000080",oldlace:"#fdf5e6",olive:"#808000",olivedrab:"#6b8e23",orange:"#ffa500",orangered:"#ff4500",orchid:"#da70d6",palegoldenrod:"#eee8aa",palegreen:"#98fb98",paleturquoise:"#afeeee",palevioletred:"#db7093",papayawhip:"#ffefd5",peachpuff:"#ffdab9",peru:"#cd853f",pink:"#ffc0cb",plum:"#dda0dd",powderblue:"#b0e0e6",purple:"#800080",rebeccapurple:"#663399",red:"#ff0000",rosybrown:"#bc8f8f",royalblue:"#4169e1",saddlebrown:"#8b4513",salmon:"#fa8072",sandybrown:"#f4a460",seagreen:"#2e8b57",seashell:"#fff5ee",sienna:"#a0522d",silver:"#c0c0c0",skyblue:"#87ceeb",slateblue:"#6a5acd",slategray:"#708090",slategrey:"#708090",snow:"#fffafa",springgreen:"#00ff7f",steelblue:"#4682b4",tan:"#d2b48c",teal:"#008080",thistle:"#d8bfd8",tomato:"#ff6347",turquoise:"#40e0d0",violet:"#ee82ee",wheat:"#f5deb3",white:"#ffffff",whitesmoke:"#f5f5f5",yellow:"#ffff00",yellowgreen:"#9acd32"}},96021:function(e,t,n){"use strict";n.d(t,{uA:function(){return i}});var r=n(82082),o=n(28052),a=n(58317);function i(e){var t={r:0,g:0,b:0},n=1,i=null,c=null,l=null,s=!1,f=!1;return"string"==typeof e&&(e=function(e){if(0===(e=e.trim().toLowerCase()).length)return!1;var t=!1;if(o.R[e])e=o.R[e],t=!0;else if("transparent"===e)return{r:0,g:0,b:0,a:0,format:"name"};var n=u.rgb.exec(e);return n?{r:n[1],g:n[2],b:n[3]}:(n=u.rgba.exec(e))?{r:n[1],g:n[2],b:n[3],a:n[4]}:(n=u.hsl.exec(e))?{h:n[1],s:n[2],l:n[3]}:(n=u.hsla.exec(e))?{h:n[1],s:n[2],l:n[3],a:n[4]}:(n=u.hsv.exec(e))?{h:n[1],s:n[2],v:n[3]}:(n=u.hsva.exec(e))?{h:n[1],s:n[2],v:n[3],a:n[4]}:(n=u.hex8.exec(e))?{r:(0,r.VD)(n[1]),g:(0,r.VD)(n[2]),b:(0,r.VD)(n[3]),a:(0,r.T6)(n[4]),format:t?"name":"hex8"}:(n=u.hex6.exec(e))?{r:(0,r.VD)(n[1]),g:(0,r.VD)(n[2]),b:(0,r.VD)(n[3]),format:t?"name":"hex"}:(n=u.hex4.exec(e))?{r:(0,r.VD)(n[1]+n[1]),g:(0,r.VD)(n[2]+n[2]),b:(0,r.VD)(n[3]+n[3]),a:(0,r.T6)(n[4]+n[4]),format:t?"name":"hex8"}:!!(n=u.hex3.exec(e))&&{r:(0,r.VD)(n[1]+n[1]),g:(0,r.VD)(n[2]+n[2]),b:(0,r.VD)(n[3]+n[3]),format:t?"name":"hex"}}(e)),"object"==typeof e&&(d(e.r)&&d(e.g)&&d(e.b)?(t=(0,r.rW)(e.r,e.g,e.b),s=!0,f="%"===String(e.r).substr(-1)?"prgb":"rgb"):d(e.h)&&d(e.s)&&d(e.v)?(i=(0,a.JX)(e.s),c=(0,a.JX)(e.v),t=(0,r.WE)(e.h,i,c),s=!0,f="hsv"):d(e.h)&&d(e.s)&&d(e.l)&&(i=(0,a.JX)(e.s),l=(0,a.JX)(e.l),t=(0,r.ve)(e.h,i,l),s=!0,f="hsl"),Object.prototype.hasOwnProperty.call(e,"a")&&(n=e.a)),n=(0,a.Yq)(n),{ok:s,format:e.format||f,r:Math.min(255,Math.max(t.r,0)),g:Math.min(255,Math.max(t.g,0)),b:Math.min(255,Math.max(t.b,0)),a:n}}var c="(?:".concat("[-\\+]?\\d*\\.\\d+%?",")|(?:").concat("[-\\+]?\\d+%?",")"),l="[\\s|\\(]+(".concat(c,")[,|\\s]+(").concat(c,")[,|\\s]+(").concat(c,")\\s*\\)?"),s="[\\s|\\(]+(".concat(c,")[,|\\s]+(").concat(c,")[,|\\s]+(").concat(c,")[,|\\s]+(").concat(c,")\\s*\\)?"),u={CSS_UNIT:new RegExp(c),rgb:RegExp("rgb"+l),rgba:RegExp("rgba"+s),hsl:RegExp("hsl"+l),hsla:RegExp("hsla"+s),hsv:RegExp("hsv"+l),hsva:RegExp("hsva"+s),hex3:/^#?([0-9a-fA-F]{1})([0-9a-fA-F]{1})([0-9a-fA-F]{1})$/,hex6:/^#?([0-9a-fA-F]{2})([0-9a-fA-F]{2})([0-9a-fA-F]{2})$/,hex4:/^#?([0-9a-fA-F]{1})([0-9a-fA-F]{1})([0-9a-fA-F]{1})([0-9a-fA-F]{1})$/,hex8:/^#?([0-9a-fA-F]{2})([0-9a-fA-F]{2})([0-9a-fA-F]{2})([0-9a-fA-F]{2})$/};function d(e){return!!u.CSS_UNIT.exec(String(e))}},36360:function(e,t,n){"use strict";n.d(t,{C:function(){return c}});var r=n(82082),o=n(28052),a=n(96021),i=n(58317),c=function(){function e(t,n){if(void 0===t&&(t=""),void 0===n&&(n={}),t instanceof e)return t;"number"==typeof t&&(t=(0,r.Yt)(t)),this.originalInput=t;var o,i=(0,a.uA)(t);this.originalInput=t,this.r=i.r,this.g=i.g,this.b=i.b,this.a=i.a,this.roundA=Math.round(100*this.a)/100,this.format=null!==(o=n.format)&&void 0!==o?o:i.format,this.gradientType=n.gradientType,this.r<1&&(this.r=Math.round(this.r)),this.g<1&&(this.g=Math.round(this.g)),this.b<1&&(this.b=Math.round(this.b)),this.isValid=i.ok}return e.prototype.isDark=function(){return 128>this.getBrightness()},e.prototype.isLight=function(){return!this.isDark()},e.prototype.getBrightness=function(){var e=this.toRgb();return(299*e.r+587*e.g+114*e.b)/1e3},e.prototype.getLuminance=function(){var e=this.toRgb(),t=e.r/255,n=e.g/255,r=e.b/255;return .2126*(t<=.03928?t/12.92:Math.pow((t+.055)/1.055,2.4))+.7152*(n<=.03928?n/12.92:Math.pow((n+.055)/1.055,2.4))+.0722*(r<=.03928?r/12.92:Math.pow((r+.055)/1.055,2.4))},e.prototype.getAlpha=function(){return this.a},e.prototype.setAlpha=function(e){return this.a=(0,i.Yq)(e),this.roundA=Math.round(100*this.a)/100,this},e.prototype.isMonochrome=function(){return 0===this.toHsl().s},e.prototype.toHsv=function(){var e=(0,r.py)(this.r,this.g,this.b);return{h:360*e.h,s:e.s,v:e.v,a:this.a}},e.prototype.toHsvString=function(){var e=(0,r.py)(this.r,this.g,this.b),t=Math.round(360*e.h),n=Math.round(100*e.s),o=Math.round(100*e.v);return 1===this.a?"hsv(".concat(t,", ").concat(n,"%, ").concat(o,"%)"):"hsva(".concat(t,", ").concat(n,"%, ").concat(o,"%, ").concat(this.roundA,")")},e.prototype.toHsl=function(){var e=(0,r.lC)(this.r,this.g,this.b);return{h:360*e.h,s:e.s,l:e.l,a:this.a}},e.prototype.toHslString=function(){var e=(0,r.lC)(this.r,this.g,this.b),t=Math.round(360*e.h),n=Math.round(100*e.s),o=Math.round(100*e.l);return 1===this.a?"hsl(".concat(t,", ").concat(n,"%, ").concat(o,"%)"):"hsla(".concat(t,", ").concat(n,"%, ").concat(o,"%, ").concat(this.roundA,")")},e.prototype.toHex=function(e){return void 0===e&&(e=!1),(0,r.vq)(this.r,this.g,this.b,e)},e.prototype.toHexString=function(e){return void 0===e&&(e=!1),"#"+this.toHex(e)},e.prototype.toHex8=function(e){return void 0===e&&(e=!1),(0,r.s)(this.r,this.g,this.b,this.a,e)},e.prototype.toHex8String=function(e){return void 0===e&&(e=!1),"#"+this.toHex8(e)},e.prototype.toHexShortString=function(e){return void 0===e&&(e=!1),1===this.a?this.toHexString(e):this.toHex8String(e)},e.prototype.toRgb=function(){return{r:Math.round(this.r),g:Math.round(this.g),b:Math.round(this.b),a:this.a}},e.prototype.toRgbString=function(){var e=Math.round(this.r),t=Math.round(this.g),n=Math.round(this.b);return 1===this.a?"rgb(".concat(e,", ").concat(t,", ").concat(n,")"):"rgba(".concat(e,", ").concat(t,", ").concat(n,", ").concat(this.roundA,")")},e.prototype.toPercentageRgb=function(){var e=function(e){return"".concat(Math.round(100*(0,i.sh)(e,255)),"%")};return{r:e(this.r),g:e(this.g),b:e(this.b),a:this.a}},e.prototype.toPercentageRgbString=function(){var e=function(e){return Math.round(100*(0,i.sh)(e,255))};return 1===this.a?"rgb(".concat(e(this.r),"%, ").concat(e(this.g),"%, ").concat(e(this.b),"%)"):"rgba(".concat(e(this.r),"%, ").concat(e(this.g),"%, ").concat(e(this.b),"%, ").concat(this.roundA,")")},e.prototype.toName=function(){if(0===this.a)return"transparent";if(this.a<1)return!1;for(var e="#"+(0,r.vq)(this.r,this.g,this.b,!1),t=0,n=Object.entries(o.R);t=0;return!t&&r&&(e.startsWith("hex")||"name"===e)?"name"===e&&0===this.a?this.toName():this.toRgbString():("rgb"===e&&(n=this.toRgbString()),"prgb"===e&&(n=this.toPercentageRgbString()),("hex"===e||"hex6"===e)&&(n=this.toHexString()),"hex3"===e&&(n=this.toHexString(!0)),"hex4"===e&&(n=this.toHex8String(!0)),"hex8"===e&&(n=this.toHex8String()),"name"===e&&(n=this.toName()),"hsl"===e&&(n=this.toHslString()),"hsv"===e&&(n=this.toHsvString()),n||this.toHexString())},e.prototype.toNumber=function(){return(Math.round(this.r)<<16)+(Math.round(this.g)<<8)+Math.round(this.b)},e.prototype.clone=function(){return new e(this.toString())},e.prototype.lighten=function(t){void 0===t&&(t=10);var n=this.toHsl();return n.l+=t/100,n.l=(0,i.V2)(n.l),new e(n)},e.prototype.brighten=function(t){void 0===t&&(t=10);var n=this.toRgb();return n.r=Math.max(0,Math.min(255,n.r-Math.round(-(t/100*255)))),n.g=Math.max(0,Math.min(255,n.g-Math.round(-(t/100*255)))),n.b=Math.max(0,Math.min(255,n.b-Math.round(-(t/100*255)))),new e(n)},e.prototype.darken=function(t){void 0===t&&(t=10);var n=this.toHsl();return n.l-=t/100,n.l=(0,i.V2)(n.l),new e(n)},e.prototype.tint=function(e){return void 0===e&&(e=10),this.mix("white",e)},e.prototype.shade=function(e){return void 0===e&&(e=10),this.mix("black",e)},e.prototype.desaturate=function(t){void 0===t&&(t=10);var n=this.toHsl();return n.s-=t/100,n.s=(0,i.V2)(n.s),new e(n)},e.prototype.saturate=function(t){void 0===t&&(t=10);var n=this.toHsl();return n.s+=t/100,n.s=(0,i.V2)(n.s),new e(n)},e.prototype.greyscale=function(){return this.desaturate(100)},e.prototype.spin=function(t){var n=this.toHsl(),r=(n.h+t)%360;return n.h=r<0?360+r:r,new e(n)},e.prototype.mix=function(t,n){void 0===n&&(n=50);var r=this.toRgb(),o=new e(t).toRgb(),a=n/100;return new e({r:(o.r-r.r)*a+r.r,g:(o.g-r.g)*a+r.g,b:(o.b-r.b)*a+r.b,a:(o.a-r.a)*a+r.a})},e.prototype.analogous=function(t,n){void 0===t&&(t=6),void 0===n&&(n=30);var r=this.toHsl(),o=360/n,a=[this];for(r.h=(r.h-(o*t>>1)+720)%360;--t;)r.h=(r.h+o)%360,a.push(new e(r));return a},e.prototype.complement=function(){var t=this.toHsl();return t.h=(t.h+180)%360,new e(t)},e.prototype.monochromatic=function(t){void 0===t&&(t=6);for(var n=this.toHsv(),r=n.h,o=n.s,a=n.v,i=[],c=1/t;t--;)i.push(new e({h:r,s:o,v:a})),a=(a+c)%1;return i},e.prototype.splitcomplement=function(){var t=this.toHsl(),n=t.h;return[this,new e({h:(n+72)%360,s:t.s,l:t.l}),new e({h:(n+216)%360,s:t.s,l:t.l})]},e.prototype.onBackground=function(t){var n=this.toRgb(),r=new e(t).toRgb(),o=n.a+r.a*(1-n.a);return new e({r:(n.r*n.a+r.r*r.a*(1-n.a))/o,g:(n.g*n.a+r.g*r.a*(1-n.a))/o,b:(n.b*n.a+r.b*r.a*(1-n.a))/o,a:o})},e.prototype.triad=function(){return this.polyad(3)},e.prototype.tetrad=function(){return this.polyad(4)},e.prototype.polyad=function(t){for(var n=this.toHsl(),r=n.h,o=[this],a=360/t,i=1;iMath.abs(e-t))?1:e=360===t?(e<0?e%t+t:e%t)/parseFloat(String(t)):e%t/parseFloat(String(t))}function o(e){return Math.min(1,Math.max(0,e))}function a(e){return(isNaN(e=parseFloat(e))||e<0||e>1)&&(e=1),e}function i(e){return e<=1?"".concat(100*Number(e),"%"):e}function c(e){return 1===e.length?"0"+e:String(e)}n.d(t,{FZ:function(){return c},JX:function(){return i},V2:function(){return o},Yq:function(){return a},sh:function(){return r}})},28036:function(e,t,n){"use strict";n.d(t,{Z:function(){return v}});var r=n(26365),o=n(2265),a=n(54887),i=n(94981);n(32559);var c=n(28791),l=o.createContext(null),s=n(83145),u=n(27380),d=[],f=n(21717),p=n(3208),m="rc-util-locker-".concat(Date.now()),g=0,h=function(e){return!1!==e&&((0,i.Z)()&&e?"string"==typeof e?document.querySelector(e):"function"==typeof e?e():e:null)},v=o.forwardRef(function(e,t){var n,v,b,y=e.open,w=e.autoLock,x=e.getContainer,E=(e.debug,e.autoDestroy),S=void 0===E||E,C=e.children,Z=o.useState(y),O=(0,r.Z)(Z,2),k=O[0],M=O[1],j=k||y;o.useEffect(function(){(S||y)&&M(y)},[y,S]);var I=o.useState(function(){return h(x)}),R=(0,r.Z)(I,2),N=R[0],P=R[1];o.useEffect(function(){var e=h(x);P(null!=e?e:null)});var F=function(e,t){var n=o.useState(function(){return(0,i.Z)()?document.createElement("div"):null}),a=(0,r.Z)(n,1)[0],c=o.useRef(!1),f=o.useContext(l),p=o.useState(d),m=(0,r.Z)(p,2),g=m[0],h=m[1],v=f||(c.current?void 0:function(e){h(function(t){return[e].concat((0,s.Z)(t))})});function b(){a.parentElement||document.body.appendChild(a),c.current=!0}function y(){var e;null===(e=a.parentElement)||void 0===e||e.removeChild(a),c.current=!1}return(0,u.Z)(function(){return e?f?f(b):b():y(),y},[e]),(0,u.Z)(function(){g.length&&(g.forEach(function(e){return e()}),h(d))},[g]),[a,v]}(j&&!N,0),T=(0,r.Z)(F,2),A=T[0],L=T[1],z=null!=N?N:A;n=!!(w&&y&&(0,i.Z)()&&(z===A||z===document.body)),v=o.useState(function(){return g+=1,"".concat(m,"_").concat(g)}),b=(0,r.Z)(v,1)[0],(0,u.Z)(function(){if(n){var e=(0,p.o)(document.body).width,t=document.body.scrollHeight>(window.innerHeight||document.documentElement.clientHeight)&&window.innerWidth>document.body.offsetWidth;(0,f.hq)("\nhtml body {\n overflow-y: hidden;\n ".concat(t?"width: calc(100% - ".concat(e,"px);"):"","\n}"),b)}else(0,f.jL)(b);return function(){(0,f.jL)(b)}},[n,b]);var _=null;C&&(0,c.Yr)(C)&&t&&(_=C.ref);var H=(0,c.x1)(_,t);if(!j||!(0,i.Z)()||void 0===N)return null;var B=!1===z,D=C;return t&&(D=o.cloneElement(C,{ref:H})),o.createElement(l.Provider,{value:L},B?D:(0,a.createPortal)(D,z))})},97821:function(e,t,n){"use strict";n.d(t,{Z:function(){return D}});var r=n(31686),o=n(26365),a=n(6989),i=n(28036),c=n(36760),l=n.n(c),s=n(31474),u=n(2868),d=n(13211),f=n(58525),p=n(92491),m=n(27380),g=n(79267),h=n(2265),v=n(1119),b=n(47970),y=n(28791);function w(e){var t=e.prefixCls,n=e.align,r=e.arrow,o=e.arrowPos,a=r||{},i=a.className,c=a.content,s=o.x,u=o.y,d=h.useRef();if(!n||!n.points)return null;var f={position:"absolute"};if(!1!==n.autoArrow){var p=n.points[0],m=n.points[1],g=p[0],v=p[1],b=m[0],y=m[1];g!==b&&["t","b"].includes(g)?"t"===g?f.top=0:f.bottom=0:f.top=void 0===u?0:u,v!==y&&["l","r"].includes(v)?"l"===v?f.left=0:f.right=0:f.left=void 0===s?0:s}return h.createElement("div",{ref:d,className:l()("".concat(t,"-arrow"),i),style:f},c)}function x(e){var t=e.prefixCls,n=e.open,r=e.zIndex,o=e.mask,a=e.motion;return o?h.createElement(b.ZP,(0,v.Z)({},a,{motionAppear:!0,visible:n,removeOnLeave:!0}),function(e){var n=e.className;return h.createElement("div",{style:{zIndex:r},className:l()("".concat(t,"-mask"),n)})}):null}var E=h.memo(function(e){return e.children},function(e,t){return t.cache}),S=h.forwardRef(function(e,t){var n=e.popup,a=e.className,i=e.prefixCls,c=e.style,u=e.target,d=e.onVisibleChanged,f=e.open,p=e.keepDom,g=e.fresh,S=e.onClick,C=e.mask,Z=e.arrow,O=e.arrowPos,k=e.align,M=e.motion,j=e.maskMotion,I=e.forceRender,R=e.getPopupContainer,N=e.autoDestroy,P=e.portal,F=e.zIndex,T=e.onMouseEnter,A=e.onMouseLeave,L=e.onPointerEnter,z=e.ready,_=e.offsetX,H=e.offsetY,B=e.offsetR,D=e.offsetB,W=e.onAlign,V=e.onPrepare,q=e.stretch,G=e.targetWidth,X=e.targetHeight,U="function"==typeof n?n():n,$=f||p,K=(null==R?void 0:R.length)>0,Y=h.useState(!R||!K),Q=(0,o.Z)(Y,2),J=Q[0],ee=Q[1];if((0,m.Z)(function(){!J&&K&&u&&ee(!0)},[J,K,u]),!J)return null;var et="auto",en={left:"-1000vw",top:"-1000vh",right:et,bottom:et};if(z||!f){var er,eo=k.points,ea=k.dynamicInset||(null===(er=k._experimental)||void 0===er?void 0:er.dynamicInset),ei=ea&&"r"===eo[0][1],ec=ea&&"b"===eo[0][0];ei?(en.right=B,en.left=et):(en.left=_,en.right=et),ec?(en.bottom=D,en.top=et):(en.top=H,en.bottom=et)}var el={};return q&&(q.includes("height")&&X?el.height=X:q.includes("minHeight")&&X&&(el.minHeight=X),q.includes("width")&&G?el.width=G:q.includes("minWidth")&&G&&(el.minWidth=G)),f||(el.pointerEvents="none"),h.createElement(P,{open:I||$,getContainer:R&&function(){return R(u)},autoDestroy:N},h.createElement(x,{prefixCls:i,open:f,zIndex:F,mask:C,motion:j}),h.createElement(s.Z,{onResize:W,disabled:!f},function(e){return h.createElement(b.ZP,(0,v.Z)({motionAppear:!0,motionEnter:!0,motionLeave:!0,removeOnLeave:!1,forceRender:I,leavedClassName:"".concat(i,"-hidden")},M,{onAppearPrepare:V,onEnterPrepare:V,visible:f,onVisibleChanged:function(e){var t;null==M||null===(t=M.onVisibleChanged)||void 0===t||t.call(M,e),d(e)}}),function(n,o){var s=n.className,u=n.style,d=l()(i,s,a);return h.createElement("div",{ref:(0,y.sQ)(e,t,o),className:d,style:(0,r.Z)((0,r.Z)((0,r.Z)((0,r.Z)({"--arrow-x":"".concat(O.x||0,"px"),"--arrow-y":"".concat(O.y||0,"px")},en),el),u),{},{boxSizing:"border-box",zIndex:F},c),onMouseEnter:T,onMouseLeave:A,onPointerEnter:L,onClick:S},Z&&h.createElement(w,{prefixCls:i,arrow:Z,arrowPos:O,align:k}),h.createElement(E,{cache:!f&&!g},U))})}))}),C=h.forwardRef(function(e,t){var n=e.children,r=e.getTriggerDOMNode,o=(0,y.Yr)(n),a=h.useCallback(function(e){(0,y.mH)(t,r?r(e):e)},[r]),i=(0,y.x1)(a,n.ref);return o?h.cloneElement(n,{ref:i}):n}),Z=h.createContext(null);function O(e){return e?Array.isArray(e)?e:[e]:[]}var k=n(2857);function M(e,t,n,r){return t||(n?{motionName:"".concat(e,"-").concat(n)}:r?{motionName:r}:null)}function j(e){return e.ownerDocument.defaultView}function I(e){for(var t=[],n=null==e?void 0:e.parentElement,r=["hidden","scroll","clip","auto"];n;){var o=j(n).getComputedStyle(n);[o.overflowX,o.overflowY,o.overflow].some(function(e){return r.includes(e)})&&t.push(n),n=n.parentElement}return t}function R(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:1;return Number.isNaN(e)?t:e}function N(e){return R(parseFloat(e),0)}function P(e,t){var n=(0,r.Z)({},e);return(t||[]).forEach(function(e){if(!(e instanceof HTMLBodyElement||e instanceof HTMLHtmlElement)){var t=j(e).getComputedStyle(e),r=t.overflow,o=t.overflowClipMargin,a=t.borderTopWidth,i=t.borderBottomWidth,c=t.borderLeftWidth,l=t.borderRightWidth,s=e.getBoundingClientRect(),u=e.offsetHeight,d=e.clientHeight,f=e.offsetWidth,p=e.clientWidth,m=N(a),g=N(i),h=N(c),v=N(l),b=R(Math.round(s.width/f*1e3)/1e3),y=R(Math.round(s.height/u*1e3)/1e3),w=m*y,x=h*b,E=0,S=0;if("clip"===r){var C=N(o);E=C*b,S=C*y}var Z=s.x+x-E,O=s.y+w-S,k=Z+s.width+2*E-x-v*b-(f-p-h-v)*b,M=O+s.height+2*S-w-g*y-(u-d-m-g)*y;n.left=Math.max(n.left,Z),n.top=Math.max(n.top,O),n.right=Math.min(n.right,k),n.bottom=Math.min(n.bottom,M)}}),n}function F(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:0,n="".concat(t),r=n.match(/^(.*)\%$/);return r?parseFloat(r[1])/100*e:parseFloat(n)}function T(e,t){var n=(0,o.Z)(t||[],2),r=n[0],a=n[1];return[F(e.width,r),F(e.height,a)]}function A(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:"";return[e[0],e[1]]}function L(e,t){var n,r=t[0],o=t[1];return n="t"===r?e.y:"b"===r?e.y+e.height:e.y+e.height/2,{x:"l"===o?e.x:"r"===o?e.x+e.width:e.x+e.width/2,y:n}}function z(e,t){var n={t:"b",b:"t",l:"r",r:"l"};return e.map(function(e,r){return r===t?n[e]||"c":e}).join("")}var _=n(83145);n(32559);var H=n(53346),B=["prefixCls","children","action","showAction","hideAction","popupVisible","defaultPopupVisible","onPopupVisibleChange","afterPopupVisibleChange","mouseEnterDelay","mouseLeaveDelay","focusDelay","blurDelay","mask","maskClosable","getPopupContainer","forceRender","autoDestroy","destroyPopupOnHide","popup","popupClassName","popupStyle","popupPlacement","builtinPlacements","popupAlign","zIndex","stretch","getPopupClassNameFromAlign","fresh","alignPoint","onPopupClick","onPopupAlign","arrow","popupMotion","maskMotion","popupTransitionName","popupAnimation","maskTransitionName","maskAnimation","className","getTriggerDOMNode"],D=function(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:i.Z;return h.forwardRef(function(t,n){var i,c,v,b,y,w,x,E,N,F,D,W,V,q,G,X,U,$=t.prefixCls,K=void 0===$?"rc-trigger-popup":$,Y=t.children,Q=t.action,J=t.showAction,ee=t.hideAction,et=t.popupVisible,en=t.defaultPopupVisible,er=t.onPopupVisibleChange,eo=t.afterPopupVisibleChange,ea=t.mouseEnterDelay,ei=t.mouseLeaveDelay,ec=void 0===ei?.1:ei,el=t.focusDelay,es=t.blurDelay,eu=t.mask,ed=t.maskClosable,ef=t.getPopupContainer,ep=t.forceRender,em=t.autoDestroy,eg=t.destroyPopupOnHide,eh=t.popup,ev=t.popupClassName,eb=t.popupStyle,ey=t.popupPlacement,ew=t.builtinPlacements,ex=void 0===ew?{}:ew,eE=t.popupAlign,eS=t.zIndex,eC=t.stretch,eZ=t.getPopupClassNameFromAlign,eO=t.fresh,ek=t.alignPoint,eM=t.onPopupClick,ej=t.onPopupAlign,eI=t.arrow,eR=t.popupMotion,eN=t.maskMotion,eP=t.popupTransitionName,eF=t.popupAnimation,eT=t.maskTransitionName,eA=t.maskAnimation,eL=t.className,ez=t.getTriggerDOMNode,e_=(0,a.Z)(t,B),eH=h.useState(!1),eB=(0,o.Z)(eH,2),eD=eB[0],eW=eB[1];(0,m.Z)(function(){eW((0,g.Z)())},[]);var eV=h.useRef({}),eq=h.useContext(Z),eG=h.useMemo(function(){return{registerSubPopup:function(e,t){eV.current[e]=t,null==eq||eq.registerSubPopup(e,t)}}},[eq]),eX=(0,p.Z)(),eU=h.useState(null),e$=(0,o.Z)(eU,2),eK=e$[0],eY=e$[1],eQ=(0,f.Z)(function(e){(0,u.S)(e)&&eK!==e&&eY(e),null==eq||eq.registerSubPopup(eX,e)}),eJ=h.useState(null),e0=(0,o.Z)(eJ,2),e1=e0[0],e2=e0[1],e6=h.useRef(null),e5=(0,f.Z)(function(e){(0,u.S)(e)&&e1!==e&&(e2(e),e6.current=e)}),e4=h.Children.only(Y),e3=(null==e4?void 0:e4.props)||{},e8={},e9=(0,f.Z)(function(e){var t,n;return(null==e1?void 0:e1.contains(e))||(null===(t=(0,d.A)(e1))||void 0===t?void 0:t.host)===e||e===e1||(null==eK?void 0:eK.contains(e))||(null===(n=(0,d.A)(eK))||void 0===n?void 0:n.host)===e||e===eK||Object.values(eV.current).some(function(t){return(null==t?void 0:t.contains(e))||e===t})}),e7=M(K,eR,eF,eP),te=M(K,eN,eA,eT),tt=h.useState(en||!1),tn=(0,o.Z)(tt,2),tr=tn[0],to=tn[1],ta=null!=et?et:tr,ti=(0,f.Z)(function(e){void 0===et&&to(e)});(0,m.Z)(function(){to(et||!1)},[et]);var tc=h.useRef(ta);tc.current=ta;var tl=h.useRef([]);tl.current=[];var ts=(0,f.Z)(function(e){var t;ti(e),(null!==(t=tl.current[tl.current.length-1])&&void 0!==t?t:ta)!==e&&(tl.current.push(e),null==er||er(e))}),tu=h.useRef(),td=function(){clearTimeout(tu.current)},tf=function(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:0;td(),0===t?ts(e):tu.current=setTimeout(function(){ts(e)},1e3*t)};h.useEffect(function(){return td},[]);var tp=h.useState(!1),tm=(0,o.Z)(tp,2),tg=tm[0],th=tm[1];(0,m.Z)(function(e){(!e||ta)&&th(!0)},[ta]);var tv=h.useState(null),tb=(0,o.Z)(tv,2),ty=tb[0],tw=tb[1],tx=h.useState([0,0]),tE=(0,o.Z)(tx,2),tS=tE[0],tC=tE[1],tZ=function(e){tC([e.clientX,e.clientY])},tO=(i=ek?tS:e1,c=h.useState({ready:!1,offsetX:0,offsetY:0,offsetR:0,offsetB:0,arrowX:0,arrowY:0,scaleX:1,scaleY:1,align:ex[ey]||{}}),b=(v=(0,o.Z)(c,2))[0],y=v[1],w=h.useRef(0),x=h.useMemo(function(){return eK?I(eK):[]},[eK]),E=h.useRef({}),ta||(E.current={}),N=(0,f.Z)(function(){if(eK&&i&&ta){var e,t,n,a,c,l,s,d=eK.ownerDocument,f=j(eK).getComputedStyle(eK),p=f.width,m=f.height,g=f.position,h=eK.style.left,v=eK.style.top,b=eK.style.right,w=eK.style.bottom,S=eK.style.overflow,C=(0,r.Z)((0,r.Z)({},ex[ey]),eE),Z=d.createElement("div");if(null===(e=eK.parentElement)||void 0===e||e.appendChild(Z),Z.style.left="".concat(eK.offsetLeft,"px"),Z.style.top="".concat(eK.offsetTop,"px"),Z.style.position=g,Z.style.height="".concat(eK.offsetHeight,"px"),Z.style.width="".concat(eK.offsetWidth,"px"),eK.style.left="0",eK.style.top="0",eK.style.right="auto",eK.style.bottom="auto",eK.style.overflow="hidden",Array.isArray(i))n={x:i[0],y:i[1],width:0,height:0};else{var O=i.getBoundingClientRect();n={x:O.x,y:O.y,width:O.width,height:O.height}}var M=eK.getBoundingClientRect(),I=d.documentElement,N=I.clientWidth,F=I.clientHeight,_=I.scrollWidth,H=I.scrollHeight,B=I.scrollTop,D=I.scrollLeft,W=M.height,V=M.width,q=n.height,G=n.width,X=C.htmlRegion,U="visible",$="visibleFirst";"scroll"!==X&&X!==$&&(X=U);var K=X===$,Y=P({left:-D,top:-B,right:_-D,bottom:H-B},x),Q=P({left:0,top:0,right:N,bottom:F},x),J=X===U?Q:Y,ee=K?Q:J;eK.style.left="auto",eK.style.top="auto",eK.style.right="0",eK.style.bottom="0";var et=eK.getBoundingClientRect();eK.style.left=h,eK.style.top=v,eK.style.right=b,eK.style.bottom=w,eK.style.overflow=S,null===(t=eK.parentElement)||void 0===t||t.removeChild(Z);var en=R(Math.round(V/parseFloat(p)*1e3)/1e3),er=R(Math.round(W/parseFloat(m)*1e3)/1e3);if(!(0===en||0===er||(0,u.S)(i)&&!(0,k.Z)(i))){var eo=C.offset,ea=C.targetOffset,ei=T(M,eo),ec=(0,o.Z)(ei,2),el=ec[0],es=ec[1],eu=T(n,ea),ed=(0,o.Z)(eu,2),ef=ed[0],ep=ed[1];n.x-=ef,n.y-=ep;var em=C.points||[],eg=(0,o.Z)(em,2),eh=eg[0],ev=A(eg[1]),eb=A(eh),ew=L(n,ev),eS=L(M,eb),eC=(0,r.Z)({},C),eZ=ew.x-eS.x+el,eO=ew.y-eS.y+es,ek=tt(eZ,eO),eM=tt(eZ,eO,Q),eI=L(n,["t","l"]),eR=L(M,["t","l"]),eN=L(n,["b","r"]),eP=L(M,["b","r"]),eF=C.overflow||{},eT=eF.adjustX,eA=eF.adjustY,eL=eF.shiftX,ez=eF.shiftY,e_=function(e){return"boolean"==typeof e?e:e>=0};tn();var eH=e_(eA),eB=eb[0]===ev[0];if(eH&&"t"===eb[0]&&(c>ee.bottom||E.current.bt)){var eD=eO;eB?eD-=W-q:eD=eI.y-eP.y-es;var eW=tt(eZ,eD),eV=tt(eZ,eD,Q);eW>ek||eW===ek&&(!K||eV>=eM)?(E.current.bt=!0,eO=eD,es=-es,eC.points=[z(eb,0),z(ev,0)]):E.current.bt=!1}if(eH&&"b"===eb[0]&&(aek||eG===ek&&(!K||eX>=eM)?(E.current.tb=!0,eO=eq,es=-es,eC.points=[z(eb,0),z(ev,0)]):E.current.tb=!1}var eU=e_(eT),e$=eb[1]===ev[1];if(eU&&"l"===eb[1]&&(s>ee.right||E.current.rl)){var eY=eZ;e$?eY-=V-G:eY=eI.x-eP.x-el;var eQ=tt(eY,eO),eJ=tt(eY,eO,Q);eQ>ek||eQ===ek&&(!K||eJ>=eM)?(E.current.rl=!0,eZ=eY,el=-el,eC.points=[z(eb,1),z(ev,1)]):E.current.rl=!1}if(eU&&"r"===eb[1]&&(lek||e1===ek&&(!K||e2>=eM)?(E.current.lr=!0,eZ=e0,el=-el,eC.points=[z(eb,1),z(ev,1)]):E.current.lr=!1}tn();var e6=!0===eL?0:eL;"number"==typeof e6&&(lQ.right&&(eZ-=s-Q.right-el,n.x>Q.right-e6&&(eZ+=n.x-Q.right+e6)));var e5=!0===ez?0:ez;"number"==typeof e5&&(aQ.bottom&&(eO-=c-Q.bottom-es,n.y>Q.bottom-e5&&(eO+=n.y-Q.bottom+e5)));var e4=M.x+eZ,e3=M.y+eO,e8=n.x,e9=n.y;null==ej||ej(eK,eC);var e7=et.right-M.x-(eZ+M.width),te=et.bottom-M.y-(eO+M.height);y({ready:!0,offsetX:eZ/en,offsetY:eO/er,offsetR:e7/en,offsetB:te/er,arrowX:((Math.max(e4,e8)+Math.min(e4+V,e8+G))/2-e4)/en,arrowY:((Math.max(e3,e9)+Math.min(e3+W,e9+q))/2-e3)/er,scaleX:en,scaleY:er,align:eC})}function tt(e,t){var n=arguments.length>2&&void 0!==arguments[2]?arguments[2]:J,r=M.x+e,o=M.y+t,a=Math.max(r,n.left),i=Math.max(o,n.top);return Math.max(0,(Math.min(r+V,n.right)-a)*(Math.min(o+W,n.bottom)-i))}function tn(){c=(a=M.y+eO)+W,s=(l=M.x+eZ)+V}}}),F=function(){y(function(e){return(0,r.Z)((0,r.Z)({},e),{},{ready:!1})})},(0,m.Z)(F,[ey]),(0,m.Z)(function(){ta||F()},[ta]),[b.ready,b.offsetX,b.offsetY,b.offsetR,b.offsetB,b.arrowX,b.arrowY,b.scaleX,b.scaleY,b.align,function(){w.current+=1;var e=w.current;Promise.resolve().then(function(){w.current===e&&N()})}]),tk=(0,o.Z)(tO,11),tM=tk[0],tj=tk[1],tI=tk[2],tR=tk[3],tN=tk[4],tP=tk[5],tF=tk[6],tT=tk[7],tA=tk[8],tL=tk[9],tz=tk[10],t_=(D=void 0===Q?"hover":Q,h.useMemo(function(){var e=O(null!=J?J:D),t=O(null!=ee?ee:D),n=new Set(e),r=new Set(t);return eD&&(n.has("hover")&&(n.delete("hover"),n.add("click")),r.has("hover")&&(r.delete("hover"),r.add("click"))),[n,r]},[eD,D,J,ee])),tH=(0,o.Z)(t_,2),tB=tH[0],tD=tH[1],tW=tB.has("click"),tV=tD.has("click")||tD.has("contextMenu"),tq=(0,f.Z)(function(){tg||tz()});W=function(){tc.current&&ek&&tV&&tf(!1)},(0,m.Z)(function(){if(ta&&e1&&eK){var e=I(e1),t=I(eK),n=j(eK),r=new Set([n].concat((0,_.Z)(e),(0,_.Z)(t)));function o(){tq(),W()}return r.forEach(function(e){e.addEventListener("scroll",o,{passive:!0})}),n.addEventListener("resize",o,{passive:!0}),tq(),function(){r.forEach(function(e){e.removeEventListener("scroll",o),n.removeEventListener("resize",o)})}}},[ta,e1,eK]),(0,m.Z)(function(){tq()},[tS,ey]),(0,m.Z)(function(){ta&&!(null!=ex&&ex[ey])&&tq()},[JSON.stringify(eE)]);var tG=h.useMemo(function(){var e=function(e,t,n,r){for(var o=n.points,a=Object.keys(e),i=0;i0&&void 0!==arguments[0]?arguments[0]:[],t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:[],n=arguments.length>2?arguments[2]:void 0;return n?e[0]===t[0]:e[0]===t[0]&&e[1]===t[1]}(null===(c=e[l])||void 0===c?void 0:c.points,o,r))return"".concat(t,"-placement-").concat(l)}return""}(ex,K,tL,ek);return l()(e,null==eZ?void 0:eZ(tL))},[tL,eZ,ex,K,ek]);h.useImperativeHandle(n,function(){return{nativeElement:e6.current,forceAlign:tq}});var tX=h.useState(0),tU=(0,o.Z)(tX,2),t$=tU[0],tK=tU[1],tY=h.useState(0),tQ=(0,o.Z)(tY,2),tJ=tQ[0],t0=tQ[1],t1=function(){if(eC&&e1){var e=e1.getBoundingClientRect();tK(e.width),t0(e.height)}};function t2(e,t,n,r){e8[e]=function(o){var a;null==r||r(o),tf(t,n);for(var i=arguments.length,c=Array(i>1?i-1:0),l=1;l1?n-1:0),o=1;o1?n-1:0),o=1;o{var t=(0,r._T)(e,[]);return o.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),o.createElement("path",{d:"M12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22ZM11 15V17H13V15H11ZM11 7V13H13V7H11Z"}))},i=e=>{var t=(0,r._T)(e,[]);return o.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),o.createElement("path",{d:"M1.18164 12C2.12215 6.87976 6.60812 3 12.0003 3C17.3924 3 21.8784 6.87976 22.8189 12C21.8784 17.1202 17.3924 21 12.0003 21C6.60812 21 2.12215 17.1202 1.18164 12ZM12.0003 17C14.7617 17 17.0003 14.7614 17.0003 12C17.0003 9.23858 14.7617 7 12.0003 7C9.23884 7 7.00026 9.23858 7.00026 12C7.00026 14.7614 9.23884 17 12.0003 17ZM12.0003 15C10.3434 15 9.00026 13.6569 9.00026 12C9.00026 10.3431 10.3434 9 12.0003 9C13.6571 9 15.0003 10.3431 15.0003 12C15.0003 13.6569 13.6571 15 12.0003 15Z"}))},c=e=>{var t=(0,r._T)(e,[]);return o.createElement("svg",Object.assign({xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 24 24",fill:"currentColor"},t),o.createElement("path",{d:"M4.52047 5.93457L1.39366 2.80777L2.80788 1.39355L22.6069 21.1925L21.1927 22.6068L17.8827 19.2968C16.1814 20.3755 14.1638 21.0002 12.0003 21.0002C6.60812 21.0002 2.12215 17.1204 1.18164 12.0002C1.61832 9.62282 2.81932 7.5129 4.52047 5.93457ZM14.7577 16.1718L13.2937 14.7078C12.902 14.8952 12.4634 15.0002 12.0003 15.0002C10.3434 15.0002 9.00026 13.657 9.00026 12.0002C9.00026 11.537 9.10522 11.0984 9.29263 10.7067L7.82866 9.24277C7.30514 10.0332 7.00026 10.9811 7.00026 12.0002C7.00026 14.7616 9.23884 17.0002 12.0003 17.0002C13.0193 17.0002 13.9672 16.6953 14.7577 16.1718ZM7.97446 3.76015C9.22127 3.26959 10.5793 3.00016 12.0003 3.00016C17.3924 3.00016 21.8784 6.87992 22.8189 12.0002C22.5067 13.6998 21.8038 15.2628 20.8068 16.5925L16.947 12.7327C16.9821 12.4936 17.0003 12.249 17.0003 12.0002C17.0003 9.23873 14.7617 7.00016 12.0003 7.00016C11.7514 7.00016 11.5068 7.01833 11.2677 7.05343L7.97446 3.76015Z"}))};var l=n(96398),s=n(97324),u=n(1153);let d=o.forwardRef((e,t)=>{let{value:n,defaultValue:d,type:f,placeholder:p="Type...",icon:m,error:g=!1,errorMessage:h,disabled:v=!1,stepper:b,makeInputClassName:y,className:w,onChange:x,onValueChange:E,autoFocus:S}=e,C=(0,r._T)(e,["value","defaultValue","type","placeholder","icon","error","errorMessage","disabled","stepper","makeInputClassName","className","onChange","onValueChange","autoFocus"]),[Z,O]=(0,o.useState)(S||!1),[k,M]=(0,o.useState)(!1),j=(0,o.useCallback)(()=>M(!k),[k,M]),I=(0,o.useRef)(null),R=(0,l.Uh)(n||d);return o.useEffect(()=>{let e=()=>O(!0),t=()=>O(!1),n=I.current;return n&&(n.addEventListener("focus",e),n.addEventListener("blur",t),S&&n.focus()),()=>{n&&(n.removeEventListener("focus",e),n.removeEventListener("blur",t))}},[S]),o.createElement(o.Fragment,null,o.createElement("div",{className:(0,s.q)(y("root"),"relative w-full flex items-center min-w-[10rem] outline-none rounded-tremor-default transition duration-100 border","shadow-tremor-input","dark:shadow-dark-tremor-input",(0,l.um)(R,v,g),Z&&(0,s.q)("ring-2","border-tremor-brand-subtle ring-tremor-brand-muted","dark:border-dark-tremor-brand-subtle dark:ring-dark-tremor-brand-muted"),w)},m?o.createElement(m,{className:(0,s.q)(y("icon"),"shrink-0 h-5 w-5 ml-2.5","text-tremor-content-subtle","dark:text-dark-tremor-content-subtle")}):null,o.createElement("input",Object.assign({ref:(0,u.lq)([I,t]),defaultValue:d,value:n,type:k?"text":f,className:(0,s.q)(y("input"),"w-full focus:outline-none focus:ring-0 border-none bg-transparent text-tremor-default rounded-tremor-default transition duration-100 py-2","text-tremor-content-emphasis","dark:text-dark-tremor-content-emphasis","[appearance:textfield] [&::-webkit-outer-spin-button]:appearance-none [&::-webkit-inner-spin-button]:appearance-none",m?"pl-2":"pl-3",g?"pr-3":"pr-4",v?"placeholder:text-tremor-content-subtle dark:placeholder:text-dark-tremor-content-subtle":"placeholder:text-tremor-content dark:placeholder:text-dark-tremor-content"),placeholder:p,disabled:v,"data-testid":"base-input",onChange:e=>{null==x||x(e),null==E||E(e.target.value)}},C)),"password"!==f||v?null:o.createElement("button",{className:(0,s.q)(y("toggleButton"),"mr-2"),type:"button",onClick:()=>j(),"aria-label":k?"Hide password":"Show Password"},k?o.createElement(c,{className:(0,s.q)("flex-none h-5 w-5 transition","text-tremor-content-subtle hover:text-tremor-content","dark:text-dark-tremor-content-subtle hover:dark:text-dark-tremor-content"),"aria-hidden":!0}):o.createElement(i,{className:(0,s.q)("flex-none h-5 w-5 transition","text-tremor-content-subtle hover:text-tremor-content","dark:text-dark-tremor-content-subtle hover:dark:text-dark-tremor-content"),"aria-hidden":!0})),g?o.createElement(a,{className:(0,s.q)(y("errorIcon"),"text-red-500 shrink-0 w-5 h-5 mr-2.5")}):null,null!=b?b:null),g&&h?o.createElement("p",{className:(0,s.q)(y("errorMessage"),"text-sm text-red-500 mt-1")},h):null)});d.displayName="BaseInput"},49566:function(e,t,n){"use strict";n.d(t,{Z:function(){return l}});var r=n(5853),o=n(2265);n(97324);var a=n(1153),i=n(69262);let c=(0,a.fn)("TextInput"),l=o.forwardRef((e,t)=>{let{type:n="text"}=e,a=(0,r._T)(e,["type"]);return o.createElement(i.Z,Object.assign({ref:t,type:n,makeInputClassName:c},a))});l.displayName="TextInput"},96398:function(e,t,n){"use strict";n.d(t,{Uh:function(){return s},n0:function(){return c},qg:function(){return a},sl:function(){return i},um:function(){return l}});var r=n(97324),o=n(2265);let a=e=>["string","number"].includes(typeof e)?e:e instanceof Array?e.map(a).join(""):"object"==typeof e&&e?a(e.props.children):void 0;function i(e){let t=new Map;return o.Children.map(e,e=>{var n;t.set(e.props.value,null!==(n=a(e))&&void 0!==n?n:e.props.value)}),t}function c(e,t){return o.Children.map(t,t=>{var n;if((null!==(n=a(t))&&void 0!==n?n:t.props.value).toLowerCase().includes(e.toLowerCase()))return t})}let l=function(e,t){let n=arguments.length>2&&void 0!==arguments[2]&&arguments[2];return(0,r.q)(t?"bg-tremor-background-subtle dark:bg-dark-tremor-background-subtle":"bg-tremor-background dark:bg-dark-tremor-background",!t&&"hover:bg-tremor-background-muted dark:hover:bg-dark-tremor-background-muted",e?"text-tremor-content-emphasis dark:text-dark-tremor-content-emphasis":"text-tremor-content dark:text-dark-tremor-content",t&&"text-tremor-content-subtle dark:text-dark-tremor-content-subtle",n&&"text-red-500",n?"border-red-500":"border-tremor-border dark:border-dark-tremor-border")};function s(e){return null!=e&&""!==e}},93942:function(e,t,n){"use strict";n.d(t,{i:function(){return c}});var r=n(2265),o=n(50506),a=n(13959),i=n(71744);function c(e){return t=>r.createElement(a.ZP,{theme:{token:{motion:!1,zIndexPopupBase:0}}},r.createElement(e,Object.assign({},t)))}t.Z=(e,t,n,a)=>c(c=>{let{prefixCls:l,style:s}=c,u=r.useRef(null),[d,f]=r.useState(0),[p,m]=r.useState(0),[g,h]=(0,o.Z)(!1,{value:c.open}),{getPrefixCls:v}=r.useContext(i.E_),b=v(t||"select",l);r.useEffect(()=>{if(h(!0),"undefined"!=typeof ResizeObserver){let e=new ResizeObserver(e=>{let t=e[0].target;f(t.offsetHeight+8),m(t.offsetWidth)}),t=setInterval(()=>{var r;let o=n?".".concat(n(b)):".".concat(b,"-dropdown"),a=null===(r=u.current)||void 0===r?void 0:r.querySelector(o);a&&(clearInterval(t),e.observe(a))},10);return()=>{clearInterval(t),e.disconnect()}}},[]);let y=Object.assign(Object.assign({},c),{style:Object.assign(Object.assign({},s),{margin:0}),open:g,visible:g,getPopupContainer:()=>u.current});return a&&(y=a(y)),r.createElement("div",{ref:u,style:{paddingBottom:d,position:"relative",minWidth:p}},r.createElement(e,Object.assign({},y)))})},93350:function(e,t,n){"use strict";n.d(t,{o2:function(){return c},yT:function(){return l}});var r=n(83145),o=n(53454);let a=o.i.map(e=>"".concat(e,"-inverse")),i=["success","processing","error","default","warning"];function c(e){let t=!(arguments.length>1)||void 0===arguments[1]||arguments[1];return t?[].concat((0,r.Z)(a),(0,r.Z)(o.i)).includes(e):o.i.includes(e)}function l(e){return i.includes(e)}},62236:function(e,t,n){"use strict";n.d(t,{Cn:function(){return s},u6:function(){return i}});var r=n(2265),o=n(29961),a=n(95140);let i=1e3,c={Modal:100,Drawer:100,Popover:100,Popconfirm:100,Tooltip:100,Tour:100},l={SelectLike:50,Dropdown:50,DatePicker:50,Menu:50,ImagePreview:1};function s(e,t){let[,n]=(0,o.ZP)(),s=r.useContext(a.Z);if(void 0!==t)return[t,t];let u=null!=s?s:0;return e in c?(u+=(s?0:n.zIndexPopupBase)+c[e],u=Math.min(u,n.zIndexPopupBase+i)):u+=l[e],[void 0===s?t:u,u]}},68710:function(e,t,n){"use strict";n.d(t,{m:function(){return c}});let r=()=>({height:0,opacity:0}),o=e=>{let{scrollHeight:t}=e;return{height:t,opacity:1}},a=e=>({height:e?e.offsetHeight:0}),i=(e,t)=>(null==t?void 0:t.deadline)===!0||"height"===t.propertyName,c=(e,t,n)=>void 0!==n?n:"".concat(e,"-").concat(t);t.Z=function(){let e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:"ant";return{motionName:"".concat(e,"-motion-collapse"),onAppearStart:r,onEnterStart:r,onAppearActive:o,onEnterActive:o,onLeaveStart:a,onLeaveActive:r,onAppearEnd:i,onEnterEnd:i,onLeaveEnd:i,motionDeadline:500}}},92736:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(88260);let o={left:{points:["cr","cl"]},right:{points:["cl","cr"]},top:{points:["bc","tc"]},bottom:{points:["tc","bc"]},topLeft:{points:["bl","tl"]},leftTop:{points:["tr","tl"]},topRight:{points:["br","tr"]},rightTop:{points:["tl","tr"]},bottomRight:{points:["tr","br"]},rightBottom:{points:["bl","br"]},bottomLeft:{points:["tl","bl"]},leftBottom:{points:["br","bl"]}},a={topLeft:{points:["bl","tc"]},leftTop:{points:["tr","cl"]},topRight:{points:["br","tc"]},rightTop:{points:["tl","cr"]},bottomRight:{points:["tr","bc"]},rightBottom:{points:["bl","cr"]},bottomLeft:{points:["tl","bc"]},leftBottom:{points:["br","cl"]}},i=new Set(["topLeft","topRight","bottomLeft","bottomRight","leftTop","leftBottom","rightTop","rightBottom"]);function c(e){let{arrowWidth:t,autoAdjustOverflow:n,arrowPointAtCenter:c,offset:l,borderRadius:s,visibleFirst:u}=e,d=t/2,f={};return Object.keys(o).forEach(e=>{let p=Object.assign(Object.assign({},c&&a[e]||o[e]),{offset:[0,0],dynamicInset:!0});switch(f[e]=p,i.has(e)&&(p.autoArrow=!1),e){case"top":case"topLeft":case"topRight":p.offset[1]=-d-l;break;case"bottom":case"bottomLeft":case"bottomRight":p.offset[1]=d+l;break;case"left":case"leftTop":case"leftBottom":p.offset[0]=-d-l;break;case"right":case"rightTop":case"rightBottom":p.offset[0]=d+l}let m=(0,r.wZ)({contentRadius:s,limitVerticalRadius:!0});if(c)switch(e){case"topLeft":case"bottomLeft":p.offset[0]=-m.arrowOffsetHorizontal-d;break;case"topRight":case"bottomRight":p.offset[0]=m.arrowOffsetHorizontal+d;break;case"leftTop":case"rightTop":p.offset[1]=-m.arrowOffsetHorizontal-d;break;case"leftBottom":case"rightBottom":p.offset[1]=m.arrowOffsetHorizontal+d}p.overflow=function(e,t,n,r){if(!1===r)return{adjustX:!1,adjustY:!1};let o={};switch(e){case"top":case"bottom":o.shiftX=2*t.arrowOffsetHorizontal+n,o.shiftY=!0,o.adjustY=!0;break;case"left":case"right":o.shiftY=2*t.arrowOffsetVertical+n,o.shiftX=!0,o.adjustX=!0}let a=Object.assign(Object.assign({},o),r&&"object"==typeof r?r:{});return a.shiftX||(a.adjustX=!0),a.shiftY||(a.adjustY=!0),a}(e,m,t,n),u&&(p.htmlRegion="visibleFirst")}),f}},19722:function(e,t,n){"use strict";n.d(t,{M2:function(){return i},Tm:function(){return l},l$:function(){return a},wm:function(){return c}});var r,o=n(2265);let{isValidElement:a}=r||(r=n.t(o,2));function i(e){return e&&a(e)&&e.type===o.Fragment}function c(e,t,n){return a(e)?o.cloneElement(e,"function"==typeof n?n(e.props||{}):n):t}function l(e,t){return c(e,e,t)}},6543:function(e,t,n){"use strict";n.d(t,{ZP:function(){return l},c4:function(){return a}});var r=n(2265),o=n(29961);let a=["xxl","xl","lg","md","sm","xs"],i=e=>({xs:"(max-width: ".concat(e.screenXSMax,"px)"),sm:"(min-width: ".concat(e.screenSM,"px)"),md:"(min-width: ".concat(e.screenMD,"px)"),lg:"(min-width: ".concat(e.screenLG,"px)"),xl:"(min-width: ".concat(e.screenXL,"px)"),xxl:"(min-width: ".concat(e.screenXXL,"px)")}),c=e=>{let t=[].concat(a).reverse();return t.forEach((n,r)=>{let o=n.toUpperCase(),a="screen".concat(o,"Min"),i="screen".concat(o);if(!(e[a]<=e[i]))throw Error("".concat(a,"<=").concat(i," fails : !(").concat(e[a],"<=").concat(e[i],")"));if(r{let e=new Map,n=-1,r={};return{matchHandlers:{},dispatch:t=>(r=t,e.forEach(e=>e(r)),e.size>=1),subscribe(t){return e.size||this.register(),n+=1,e.set(n,t),t(r),n},unsubscribe(t){e.delete(t),e.size||this.unregister()},unregister(){Object.keys(t).forEach(e=>{let n=t[e],r=this.matchHandlers[n];null==r||r.mql.removeListener(null==r?void 0:r.listener)}),e.clear()},register(){Object.keys(t).forEach(e=>{let n=t[e],o=t=>{let{matches:n}=t;this.dispatch(Object.assign(Object.assign({},r),{[e]:n}))},a=window.matchMedia(n);a.addListener(o),this.matchHandlers[n]={mql:a,listener:o},o(a)})},responsiveMap:t}},[e])}},12757:function(e,t,n){"use strict";n.d(t,{F:function(){return i},Z:function(){return a}});var r=n(36760),o=n.n(r);function a(e,t,n){return o()({["".concat(e,"-status-success")]:"success"===t,["".concat(e,"-status-warning")]:"warning"===t,["".concat(e,"-status-error")]:"error"===t,["".concat(e,"-status-validating")]:"validating"===t,["".concat(e,"-has-feedback")]:n})}let i=(e,t)=>t||e},13613:function(e,t,n){"use strict";n.d(t,{G8:function(){return a},ln:function(){return i}});var r=n(2265);function o(){}n(32559);let a=r.createContext({}),i=()=>{let e=()=>{};return e.deprecated=o,e}},6694:function(e,t,n){"use strict";n.d(t,{Z:function(){return S}});var r=n(36760),o=n.n(r),a=n(28791),i=n(2857),c=n(2265),l=n(71744),s=n(19722),u=n(80669);let d=e=>{let{componentCls:t,colorPrimary:n}=e;return{[t]:{position:"absolute",background:"transparent",pointerEvents:"none",boxSizing:"border-box",color:"var(--wave-color, ".concat(n,")"),boxShadow:"0 0 0 0 currentcolor",opacity:.2,"&.wave-motion-appear":{transition:["box-shadow 0.4s ".concat(e.motionEaseOutCirc),"opacity 2s ".concat(e.motionEaseOutCirc)].join(","),"&-active":{boxShadow:"0 0 0 6px currentcolor",opacity:0},"&.wave-quick":{transition:["box-shadow 0.3s ".concat(e.motionEaseInOut),"opacity 0.35s ".concat(e.motionEaseInOut)].join(",")}}}}};var f=(0,u.ZP)("Wave",e=>[d(e)]),p=n(74126),m=n(53346),g=n(47970),h=n(18404);function v(e){return e&&"#fff"!==e&&"#ffffff"!==e&&"rgb(255, 255, 255)"!==e&&"rgba(255, 255, 255, 1)"!==e&&function(e){let t=(e||"").match(/rgba?\((\d*), (\d*), (\d*)(, [\d.]*)?\)/);return!t||!t[1]||!t[2]||!t[3]||!(t[1]===t[2]&&t[2]===t[3])}(e)&&!/rgba\((?:\d*, ){3}0\)/.test(e)&&"transparent"!==e}var b=n(34709);function y(e){return Number.isNaN(e)?0:e}let w=e=>{let{className:t,target:n,component:r}=e,a=c.useRef(null),[i,l]=c.useState(null),[s,u]=c.useState([]),[d,f]=c.useState(0),[p,w]=c.useState(0),[x,E]=c.useState(0),[S,C]=c.useState(0),[Z,O]=c.useState(!1),k={left:d,top:p,width:x,height:S,borderRadius:s.map(e=>"".concat(e,"px")).join(" ")};function M(){let e=getComputedStyle(n);l(function(e){let{borderTopColor:t,borderColor:n,backgroundColor:r}=getComputedStyle(e);return v(t)?t:v(n)?n:v(r)?r:null}(n));let t="static"===e.position,{borderLeftWidth:r,borderTopWidth:o}=e;f(t?n.offsetLeft:y(-parseFloat(r))),w(t?n.offsetTop:y(-parseFloat(o))),E(n.offsetWidth),C(n.offsetHeight);let{borderTopLeftRadius:a,borderTopRightRadius:i,borderBottomLeftRadius:c,borderBottomRightRadius:s}=e;u([a,i,s,c].map(e=>y(parseFloat(e))))}if(i&&(k["--wave-color"]=i),c.useEffect(()=>{if(n){let e;let t=(0,m.Z)(()=>{M(),O(!0)});return"undefined"!=typeof ResizeObserver&&(e=new ResizeObserver(M)).observe(n),()=>{m.Z.cancel(t),null==e||e.disconnect()}}},[]),!Z)return null;let j=("Checkbox"===r||"Radio"===r)&&(null==n?void 0:n.classList.contains(b.A));return c.createElement(g.ZP,{visible:!0,motionAppear:!0,motionName:"wave-motion",motionDeadline:5e3,onAppearEnd:(e,t)=>{var n;if(t.deadline||"opacity"===t.propertyName){let e=null===(n=a.current)||void 0===n?void 0:n.parentElement;(0,h.v)(e).then(()=>{null==e||e.remove()})}return!1}},e=>{let{className:n}=e;return c.createElement("div",{ref:a,className:o()(t,{"wave-quick":j},n),style:k})})};var x=(e,t)=>{var n;let{component:r}=t;if("Checkbox"===r&&!(null===(n=e.querySelector("input"))||void 0===n?void 0:n.checked))return;let o=document.createElement("div");o.style.position="absolute",o.style.left="0px",o.style.top="0px",null==e||e.insertBefore(o,null==e?void 0:e.firstChild),(0,h.s)(c.createElement(w,Object.assign({},t,{target:e})),o)},E=n(29961),S=e=>{let{children:t,disabled:n,component:r}=e,{getPrefixCls:u}=(0,c.useContext)(l.E_),d=(0,c.useRef)(null),g=u("wave"),[,h]=f(g),v=function(e,t,n){let{wave:r}=c.useContext(l.E_),[,o,a]=(0,E.ZP)(),i=(0,p.zX)(i=>{let c=e.current;if((null==r?void 0:r.disabled)||!c)return;let l=c.querySelector(".".concat(b.A))||c,{showEffect:s}=r||{};(s||x)(l,{className:t,token:o,component:n,event:i,hashId:a})}),s=c.useRef();return e=>{m.Z.cancel(s.current),s.current=(0,m.Z)(()=>{i(e)})}}(d,o()(g,h),r);if(c.useEffect(()=>{let e=d.current;if(!e||1!==e.nodeType||n)return;let t=t=>{!(0,i.Z)(t.target)||!e.getAttribute||e.getAttribute("disabled")||e.disabled||e.className.includes("disabled")||e.className.includes("-leave")||v(t)};return e.addEventListener("click",t,!0),()=>{e.removeEventListener("click",t,!0)}},[n]),!c.isValidElement(t))return null!=t?t:null;let y=(0,a.Yr)(t)?(0,a.sQ)(t.ref,d):d;return(0,s.Tm)(t,{ref:y})}},34709:function(e,t,n){"use strict";n.d(t,{A:function(){return r}});let r="ant-wave-target"},95140:function(e,t,n){"use strict";let r=n(2265).createContext(void 0);t.Z=r},52402:function(e,t,n){"use strict";n.d(t,{J:function(){return r}});let r=n(2265).createContext({})},51248:function(e,t,n){"use strict";n.d(t,{Te:function(){return s},aG:function(){return i},hU:function(){return u},nx:function(){return c}});var r=n(2265),o=n(19722);let a=/^[\u4e00-\u9fa5]{2}$/,i=a.test.bind(a);function c(e){return"danger"===e?{danger:!0}:{type:e}}function l(e){return"string"==typeof e}function s(e){return"text"===e||"link"===e}function u(e,t){let n=!1,a=[];return r.Children.forEach(e,e=>{let t=typeof e,r="string"===t||"number"===t;if(n&&r){let t=a.length-1,n=a[t];a[t]="".concat(n).concat(e)}else a.push(e);n=r}),r.Children.map(a,e=>(function(e,t){if(null==e)return;let n=t?" ":"";return"string"!=typeof e&&"number"!=typeof e&&l(e.type)&&i(e.props.children)?(0,o.Tm)(e,{children:e.props.children.split("").join(n)}):l(e)?i(e)?r.createElement("span",null,e.split("").join(n)):r.createElement("span",null,e):(0,o.M2)(e)?r.createElement("span",null,e):e})(e,t))}},73002:function(e,t,n){"use strict";n.d(t,{ZP:function(){return ea}});var r=n(2265),o=n(36760),a=n.n(o),i=n(18694),c=n(28791),l=n(6694),s=n(71744),u=n(86586),d=n(33759),f=n(65658),p=n(29961),m=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let g=r.createContext(void 0);var h=n(51248);let v=(0,r.forwardRef)((e,t)=>{let{className:n,style:o,children:i,prefixCls:c}=e,l=a()("".concat(c,"-icon"),n);return r.createElement("span",{ref:t,className:l,style:o},i)});var b=n(61935),y=n(47970);let w=(0,r.forwardRef)((e,t)=>{let{prefixCls:n,className:o,style:i,iconClassName:c}=e,l=a()("".concat(n,"-loading-icon"),o);return r.createElement(v,{prefixCls:n,className:l,style:i,ref:t},r.createElement(b.Z,{className:c}))}),x=()=>({width:0,opacity:0,transform:"scale(0)"}),E=e=>({width:e.scrollWidth,opacity:1,transform:"scale(1)"});var S=e=>{let{prefixCls:t,loading:n,existIcon:o,className:a,style:i}=e,c=!!n;return o?r.createElement(w,{prefixCls:t,className:a,style:i}):r.createElement(y.ZP,{visible:c,motionName:"".concat(t,"-loading-icon-motion"),motionLeave:c,removeOnLeave:!0,onAppearStart:x,onAppearActive:E,onEnterStart:x,onEnterActive:E,onLeaveStart:E,onLeaveActive:x},(e,n)=>{let{className:o,style:c}=e;return r.createElement(w,{prefixCls:t,className:a,style:Object.assign(Object.assign({},i),c),ref:n,iconClassName:o})})},C=n(352),Z=n(12918),O=n(3104),k=n(80669);let M=(e,t)=>({["> span, > ".concat(e)]:{"&:not(:last-child)":{["&, & > ".concat(e)]:{"&:not(:disabled)":{borderInlineEndColor:t}}},"&:not(:first-child)":{["&, & > ".concat(e)]:{"&:not(:disabled)":{borderInlineStartColor:t}}}}});var j=e=>{let{componentCls:t,fontSize:n,lineWidth:r,groupBorderColor:o,colorErrorHover:a}=e;return{["".concat(t,"-group")]:[{position:"relative",display:"inline-flex",["> span, > ".concat(t)]:{"&:not(:last-child)":{["&, & > ".concat(t)]:{borderStartEndRadius:0,borderEndEndRadius:0}},"&:not(:first-child)":{marginInlineStart:e.calc(r).mul(-1).equal(),["&, & > ".concat(t)]:{borderStartStartRadius:0,borderEndStartRadius:0}}},[t]:{position:"relative",zIndex:1,"&:hover,\n &:focus,\n &:active":{zIndex:2},"&[disabled]":{zIndex:0}},["".concat(t,"-icon-only")]:{fontSize:n}},M("".concat(t,"-primary"),o),M("".concat(t,"-danger"),a)]}},I=n(1319);let R=e=>{let{paddingInline:t,onlyIconSize:n,paddingBlock:r}=e;return(0,O.TS)(e,{buttonPaddingHorizontal:t,buttonPaddingVertical:r,buttonIconOnlyFontSize:n})},N=e=>{var t,n,r,o,a,i;let c=null!==(t=e.contentFontSize)&&void 0!==t?t:e.fontSize,l=null!==(n=e.contentFontSizeSM)&&void 0!==n?n:e.fontSize,s=null!==(r=e.contentFontSizeLG)&&void 0!==r?r:e.fontSizeLG,u=null!==(o=e.contentLineHeight)&&void 0!==o?o:(0,I.D)(c),d=null!==(a=e.contentLineHeightSM)&&void 0!==a?a:(0,I.D)(l),f=null!==(i=e.contentLineHeightLG)&&void 0!==i?i:(0,I.D)(s);return{fontWeight:400,defaultShadow:"0 ".concat(e.controlOutlineWidth,"px 0 ").concat(e.controlTmpOutline),primaryShadow:"0 ".concat(e.controlOutlineWidth,"px 0 ").concat(e.controlOutline),dangerShadow:"0 ".concat(e.controlOutlineWidth,"px 0 ").concat(e.colorErrorOutline),primaryColor:e.colorTextLightSolid,dangerColor:e.colorTextLightSolid,borderColorDisabled:e.colorBorder,defaultGhostColor:e.colorBgContainer,ghostBg:"transparent",defaultGhostBorderColor:e.colorBgContainer,paddingInline:e.paddingContentHorizontal-e.lineWidth,paddingInlineLG:e.paddingContentHorizontal-e.lineWidth,paddingInlineSM:8-e.lineWidth,onlyIconSize:e.fontSizeLG,onlyIconSizeSM:e.fontSizeLG-2,onlyIconSizeLG:e.fontSizeLG+2,groupBorderColor:e.colorPrimaryHover,linkHoverBg:"transparent",textHoverBg:e.colorBgTextHover,defaultColor:e.colorText,defaultBg:e.colorBgContainer,defaultBorderColor:e.colorBorder,defaultBorderColorDisabled:e.colorBorder,contentFontSize:c,contentFontSizeSM:l,contentFontSizeLG:s,contentLineHeight:u,contentLineHeightSM:d,contentLineHeightLG:f,paddingBlock:Math.max((e.controlHeight-c*u)/2-e.lineWidth,0),paddingBlockSM:Math.max((e.controlHeightSM-l*d)/2-e.lineWidth,0),paddingBlockLG:Math.max((e.controlHeightLG-s*f)/2-e.lineWidth,0)}},P=e=>{let{componentCls:t,iconCls:n,fontWeight:r}=e;return{[t]:{outline:"none",position:"relative",display:"inline-block",fontWeight:r,whiteSpace:"nowrap",textAlign:"center",backgroundImage:"none",background:"transparent",border:"".concat((0,C.bf)(e.lineWidth)," ").concat(e.lineType," transparent"),cursor:"pointer",transition:"all ".concat(e.motionDurationMid," ").concat(e.motionEaseInOut),userSelect:"none",touchAction:"manipulation",color:e.colorText,"&:disabled > *":{pointerEvents:"none"},"> span":{display:"inline-block"},["".concat(t,"-icon")]:{lineHeight:0},["> ".concat(n," + span, > span + ").concat(n)]:{marginInlineStart:e.marginXS},["&:not(".concat(t,"-icon-only) > ").concat(t,"-icon")]:{["&".concat(t,"-loading-icon, &:not(:last-child)")]:{marginInlineEnd:e.marginXS}},"> a":{color:"currentColor"},"&:not(:disabled)":Object.assign({},(0,Z.Qy)(e)),["&".concat(t,"-two-chinese-chars::first-letter")]:{letterSpacing:"0.34em"},["&".concat(t,"-two-chinese-chars > *:not(").concat(n,")")]:{marginInlineEnd:"-0.34em",letterSpacing:"0.34em"},["&-icon-only".concat(t,"-compact-item")]:{flex:"none"}}}},F=(e,t,n)=>({["&:not(:disabled):not(".concat(e,"-disabled)")]:{"&:hover":t,"&:active":n}}),T=e=>({minWidth:e.controlHeight,paddingInlineStart:0,paddingInlineEnd:0,borderRadius:"50%"}),A=e=>({borderRadius:e.controlHeight,paddingInlineStart:e.calc(e.controlHeight).div(2).equal(),paddingInlineEnd:e.calc(e.controlHeight).div(2).equal()}),L=e=>({cursor:"not-allowed",borderColor:e.borderColorDisabled,color:e.colorTextDisabled,background:e.colorBgContainerDisabled,boxShadow:"none"}),z=(e,t,n,r,o,a,i,c)=>({["&".concat(e,"-background-ghost")]:Object.assign(Object.assign({color:n||void 0,background:t,borderColor:r||void 0,boxShadow:"none"},F(e,Object.assign({background:t},i),Object.assign({background:t},c))),{"&:disabled":{cursor:"not-allowed",color:o||void 0,borderColor:a||void 0}})}),_=e=>({["&:disabled, &".concat(e.componentCls,"-disabled")]:Object.assign({},L(e))}),H=e=>Object.assign({},_(e)),B=e=>({["&:disabled, &".concat(e.componentCls,"-disabled")]:{cursor:"not-allowed",color:e.colorTextDisabled}}),D=e=>Object.assign(Object.assign(Object.assign(Object.assign(Object.assign({},H(e)),{background:e.defaultBg,borderColor:e.defaultBorderColor,color:e.defaultColor,boxShadow:e.defaultShadow}),F(e.componentCls,{color:e.colorPrimaryHover,borderColor:e.colorPrimaryHover},{color:e.colorPrimaryActive,borderColor:e.colorPrimaryActive})),z(e.componentCls,e.ghostBg,e.defaultGhostColor,e.defaultGhostBorderColor,e.colorTextDisabled,e.colorBorder)),{["&".concat(e.componentCls,"-dangerous")]:Object.assign(Object.assign(Object.assign({color:e.colorError,borderColor:e.colorError},F(e.componentCls,{color:e.colorErrorHover,borderColor:e.colorErrorBorderHover},{color:e.colorErrorActive,borderColor:e.colorErrorActive})),z(e.componentCls,e.ghostBg,e.colorError,e.colorError,e.colorTextDisabled,e.colorBorder)),_(e))}),W=e=>Object.assign(Object.assign(Object.assign(Object.assign(Object.assign({},H(e)),{color:e.primaryColor,background:e.colorPrimary,boxShadow:e.primaryShadow}),F(e.componentCls,{color:e.colorTextLightSolid,background:e.colorPrimaryHover},{color:e.colorTextLightSolid,background:e.colorPrimaryActive})),z(e.componentCls,e.ghostBg,e.colorPrimary,e.colorPrimary,e.colorTextDisabled,e.colorBorder,{color:e.colorPrimaryHover,borderColor:e.colorPrimaryHover},{color:e.colorPrimaryActive,borderColor:e.colorPrimaryActive})),{["&".concat(e.componentCls,"-dangerous")]:Object.assign(Object.assign(Object.assign({background:e.colorError,boxShadow:e.dangerShadow,color:e.dangerColor},F(e.componentCls,{background:e.colorErrorHover},{background:e.colorErrorActive})),z(e.componentCls,e.ghostBg,e.colorError,e.colorError,e.colorTextDisabled,e.colorBorder,{color:e.colorErrorHover,borderColor:e.colorErrorHover},{color:e.colorErrorActive,borderColor:e.colorErrorActive})),_(e))}),V=e=>Object.assign(Object.assign({},D(e)),{borderStyle:"dashed"}),q=e=>Object.assign(Object.assign(Object.assign({color:e.colorLink},F(e.componentCls,{color:e.colorLinkHover,background:e.linkHoverBg},{color:e.colorLinkActive})),B(e)),{["&".concat(e.componentCls,"-dangerous")]:Object.assign(Object.assign({color:e.colorError},F(e.componentCls,{color:e.colorErrorHover},{color:e.colorErrorActive})),B(e))}),G=e=>Object.assign(Object.assign(Object.assign({},F(e.componentCls,{color:e.colorText,background:e.textHoverBg},{color:e.colorText,background:e.colorBgTextActive})),B(e)),{["&".concat(e.componentCls,"-dangerous")]:Object.assign(Object.assign({color:e.colorError},B(e)),F(e.componentCls,{color:e.colorErrorHover,background:e.colorErrorBg},{color:e.colorErrorHover,background:e.colorErrorBg}))}),X=e=>{let{componentCls:t}=e;return{["".concat(t,"-default")]:D(e),["".concat(t,"-primary")]:W(e),["".concat(t,"-dashed")]:V(e),["".concat(t,"-link")]:q(e),["".concat(t,"-text")]:G(e),["".concat(t,"-ghost")]:z(e.componentCls,e.ghostBg,e.colorBgContainer,e.colorBgContainer,e.colorTextDisabled,e.colorBorder)}},U=function(e){let t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:"",{componentCls:n,controlHeight:r,fontSize:o,lineHeight:a,borderRadius:i,buttonPaddingHorizontal:c,iconCls:l,buttonPaddingVertical:s}=e,u="".concat(n,"-icon-only");return[{["".concat(n).concat(t)]:{fontSize:o,lineHeight:a,height:r,padding:"".concat((0,C.bf)(s)," ").concat((0,C.bf)(c)),borderRadius:i,["&".concat(u)]:{width:r,paddingInlineStart:0,paddingInlineEnd:0,["&".concat(n,"-round")]:{width:"auto"},[l]:{fontSize:e.buttonIconOnlyFontSize}},["&".concat(n,"-loading")]:{opacity:e.opacityLoading,cursor:"default"},["".concat(n,"-loading-icon")]:{transition:"width ".concat(e.motionDurationSlow," ").concat(e.motionEaseInOut,", opacity ").concat(e.motionDurationSlow," ").concat(e.motionEaseInOut)}}},{["".concat(n).concat(n,"-circle").concat(t)]:T(e)},{["".concat(n).concat(n,"-round").concat(t)]:A(e)}]},$=e=>U((0,O.TS)(e,{fontSize:e.contentFontSize,lineHeight:e.contentLineHeight})),K=e=>U((0,O.TS)(e,{controlHeight:e.controlHeightSM,fontSize:e.contentFontSizeSM,lineHeight:e.contentLineHeightSM,padding:e.paddingXS,buttonPaddingHorizontal:e.paddingInlineSM,buttonPaddingVertical:e.paddingBlockSM,borderRadius:e.borderRadiusSM,buttonIconOnlyFontSize:e.onlyIconSizeSM}),"".concat(e.componentCls,"-sm")),Y=e=>U((0,O.TS)(e,{controlHeight:e.controlHeightLG,fontSize:e.contentFontSizeLG,lineHeight:e.contentLineHeightLG,buttonPaddingHorizontal:e.paddingInlineLG,buttonPaddingVertical:e.paddingBlockLG,borderRadius:e.borderRadiusLG,buttonIconOnlyFontSize:e.onlyIconSizeLG}),"".concat(e.componentCls,"-lg")),Q=e=>{let{componentCls:t}=e;return{[t]:{["&".concat(t,"-block")]:{width:"100%"}}}};var J=(0,k.I$)("Button",e=>{let t=R(e);return[P(t),K(t),$(t),Y(t),Q(t),X(t),j(t)]},N,{unitless:{fontWeight:!0,contentLineHeight:!0,contentLineHeightSM:!0,contentLineHeightLG:!0}}),ee=n(17691);let et=e=>{let{componentCls:t,calc:n}=e;return{[t]:{["&-compact-item".concat(t,"-primary")]:{["&:not([disabled]) + ".concat(t,"-compact-item").concat(t,"-primary:not([disabled])")]:{position:"relative","&:before":{position:"absolute",top:n(e.lineWidth).mul(-1).equal(),insetInlineStart:n(e.lineWidth).mul(-1).equal(),display:"inline-block",width:e.lineWidth,height:"calc(100% + ".concat((0,C.bf)(e.lineWidth)," * 2)"),backgroundColor:e.colorPrimaryHover,content:'""'}}},"&-compact-vertical-item":{["&".concat(t,"-primary")]:{["&:not([disabled]) + ".concat(t,"-compact-vertical-item").concat(t,"-primary:not([disabled])")]:{position:"relative","&:before":{position:"absolute",top:n(e.lineWidth).mul(-1).equal(),insetInlineStart:n(e.lineWidth).mul(-1).equal(),display:"inline-block",width:"calc(100% + ".concat((0,C.bf)(e.lineWidth)," * 2)"),height:e.lineWidth,backgroundColor:e.colorPrimaryHover,content:'""'}}}}}}};var en=(0,k.bk)(["Button","compact"],e=>{let t=R(e);return[(0,ee.c)(t),function(e){var t;let n="".concat(e.componentCls,"-compact-vertical");return{[n]:Object.assign(Object.assign({},{["&-item:not(".concat(n,"-last-item)")]:{marginBottom:e.calc(e.lineWidth).mul(-1).equal()},"&-item":{"&:hover,&:focus,&:active":{zIndex:2},"&[disabled]":{zIndex:0}}}),(t=e.componentCls,{["&-item:not(".concat(n,"-first-item):not(").concat(n,"-last-item)")]:{borderRadius:0},["&-item".concat(n,"-first-item:not(").concat(n,"-last-item)")]:{["&, &".concat(t,"-sm, &").concat(t,"-lg")]:{borderEndEndRadius:0,borderEndStartRadius:0}},["&-item".concat(n,"-last-item:not(").concat(n,"-first-item)")]:{["&, &".concat(t,"-sm, &").concat(t,"-lg")]:{borderStartStartRadius:0,borderStartEndRadius:0}}}))}}(t),et(t)]},N),er=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let eo=(0,r.forwardRef)((e,t)=>{var n,o;let{loading:p=!1,prefixCls:m,type:b="default",danger:y,shape:w="default",size:x,styles:E,disabled:C,className:Z,rootClassName:O,children:k,icon:M,ghost:j=!1,block:I=!1,htmlType:R="button",classNames:N,style:P={}}=e,F=er(e,["loading","prefixCls","type","danger","shape","size","styles","disabled","className","rootClassName","children","icon","ghost","block","htmlType","classNames","style"]),{getPrefixCls:T,autoInsertSpaceInButton:A,direction:L,button:z}=(0,r.useContext)(s.E_),_=T("btn",m),[H,B,D]=J(_),W=(0,r.useContext)(u.Z),V=null!=C?C:W,q=(0,r.useContext)(g),G=(0,r.useMemo)(()=>(function(e){if("object"==typeof e&&e){let t=null==e?void 0:e.delay;return{loading:(t=Number.isNaN(t)||"number"!=typeof t?0:t)<=0,delay:t}}return{loading:!!e,delay:0}})(p),[p]),[X,U]=(0,r.useState)(G.loading),[$,K]=(0,r.useState)(!1),Y=(0,r.createRef)(),Q=(0,c.sQ)(t,Y),ee=1===r.Children.count(k)&&!M&&!(0,h.Te)(b);(0,r.useEffect)(()=>{let e=null;return G.delay>0?e=setTimeout(()=>{e=null,U(!0)},G.delay):U(G.loading),function(){e&&(clearTimeout(e),e=null)}},[G]),(0,r.useEffect)(()=>{if(!Q||!Q.current||!1===A)return;let e=Q.current.textContent;ee&&(0,h.aG)(e)?$||K(!0):$&&K(!1)},[Q]);let et=t=>{let{onClick:n}=e;if(X||V){t.preventDefault();return}null==n||n(t)},eo=!1!==A,{compactSize:ea,compactItemClassnames:ei}=(0,f.ri)(_,L),ec=(0,d.Z)(e=>{var t,n;return null!==(n=null!==(t=null!=x?x:ea)&&void 0!==t?t:q)&&void 0!==n?n:e}),el=ec&&({large:"lg",small:"sm",middle:void 0})[ec]||"",es=X?"loading":M,eu=(0,i.Z)(F,["navigate"]),ed=a()(_,B,D,{["".concat(_,"-").concat(w)]:"default"!==w&&w,["".concat(_,"-").concat(b)]:b,["".concat(_,"-").concat(el)]:el,["".concat(_,"-icon-only")]:!k&&0!==k&&!!es,["".concat(_,"-background-ghost")]:j&&!(0,h.Te)(b),["".concat(_,"-loading")]:X,["".concat(_,"-two-chinese-chars")]:$&&eo&&!X,["".concat(_,"-block")]:I,["".concat(_,"-dangerous")]:!!y,["".concat(_,"-rtl")]:"rtl"===L},ei,Z,O,null==z?void 0:z.className),ef=Object.assign(Object.assign({},null==z?void 0:z.style),P),ep=a()(null==N?void 0:N.icon,null===(n=null==z?void 0:z.classNames)||void 0===n?void 0:n.icon),em=Object.assign(Object.assign({},(null==E?void 0:E.icon)||{}),(null===(o=null==z?void 0:z.styles)||void 0===o?void 0:o.icon)||{}),eg=M&&!X?r.createElement(v,{prefixCls:_,className:ep,style:em},M):r.createElement(S,{existIcon:!!M,prefixCls:_,loading:!!X}),eh=k||0===k?(0,h.hU)(k,ee&&eo):null;if(void 0!==eu.href)return H(r.createElement("a",Object.assign({},eu,{className:a()(ed,{["".concat(_,"-disabled")]:V}),href:V?void 0:eu.href,style:ef,onClick:et,ref:Q,tabIndex:V?-1:0}),eg,eh));let ev=r.createElement("button",Object.assign({},F,{type:R,className:ed,style:ef,onClick:et,disabled:V,ref:Q}),eg,eh,!!ei&&r.createElement(en,{key:"compact",prefixCls:_}));return(0,h.Te)(b)||(ev=r.createElement(l.Z,{component:"Button",disabled:!!X},ev)),H(ev)});eo.Group=e=>{let{getPrefixCls:t,direction:n}=r.useContext(s.E_),{prefixCls:o,size:i,className:c}=e,l=m(e,["prefixCls","size","className"]),u=t("btn-group",o),[,,d]=(0,p.ZP)(),f="";switch(i){case"large":f="lg";break;case"small":f="sm"}let h=a()(u,{["".concat(u,"-").concat(f)]:f,["".concat(u,"-rtl")]:"rtl"===n},c,d);return r.createElement(g.Provider,{value:i},r.createElement("div",Object.assign({},l,{className:h})))},eo.__ANT_BUTTON=!0;var ea=eo},86586:function(e,t,n){"use strict";n.d(t,{n:function(){return a}});var r=n(2265);let o=r.createContext(!1),a=e=>{let{children:t,disabled:n}=e,a=r.useContext(o);return r.createElement(o.Provider,{value:null!=n?n:a},t)};t.Z=o},59189:function(e,t,n){"use strict";n.d(t,{q:function(){return a}});var r=n(2265);let o=r.createContext(void 0),a=e=>{let{children:t,size:n}=e,a=r.useContext(o);return r.createElement(o.Provider,{value:n||a},t)};t.Z=o},71744:function(e,t,n){"use strict";n.d(t,{E_:function(){return a},oR:function(){return o}});var r=n(2265);let o="anticon",a=r.createContext({getPrefixCls:(e,t)=>t||(e?"ant-".concat(e):"ant"),iconPrefixCls:o}),{Consumer:i}=a},91086:function(e,t,n){"use strict";var r=n(2265),o=n(71744),a=n(85180);t.Z=e=>{let{componentName:t}=e,{getPrefixCls:n}=(0,r.useContext)(o.E_),i=n("empty");switch(t){case"Table":case"List":return r.createElement(a.Z,{image:a.Z.PRESENTED_IMAGE_SIMPLE});case"Select":case"TreeSelect":case"Cascader":case"Transfer":case"Mentions":return r.createElement(a.Z,{image:a.Z.PRESENTED_IMAGE_SIMPLE,className:"".concat(i,"-small")});default:return r.createElement(a.Z,null)}}},64024:function(e,t,n){"use strict";var r=n(29961);t.Z=e=>{let[,,,,t]=(0,r.ZP)();return t?"".concat(e,"-css-var"):""}},33759:function(e,t,n){"use strict";var r=n(2265),o=n(59189);t.Z=e=>{let t=r.useContext(o.Z);return r.useMemo(()=>e?"string"==typeof e?null!=e?e:t:e instanceof Function?e(t):t:t,[e,t])}},13959:function(e,t,n){"use strict";let r,o,a,i;n.d(t,{ZP:function(){return V},w6:function(){return B}});var c=n(2265),l=n.t(c,2),s=n(352),u=n(20902),d=n(6397),f=n(23789),p=n(13613),m=n(77360),g=n(92246),h=n(91325),v=e=>{let{locale:t={},children:n,_ANT_MARK__:r}=e;c.useEffect(()=>(0,g.f)(t&&t.Modal),[t]);let o=c.useMemo(()=>Object.assign(Object.assign({},t),{exist:!0}),[t]);return c.createElement(h.Z.Provider,{value:o},n)},b=n(13823),y=n(37516),w=n(70774),x=n(71744),E=n(31373),S=n(36360),C=n(94981),Z=n(21717);let O="-ant-".concat(Date.now(),"-").concat(Math.random());var k=n(86586),M=n(59189),j=n(16671);let{useId:I}=Object.assign({},l);var R=void 0===I?()=>"":I,N=n(47970),P=n(29961);function F(e){let{children:t}=e,[,n]=(0,P.ZP)(),{motion:r}=n,o=c.useRef(!1);return(o.current=o.current||!1===r,o.current)?c.createElement(N.zt,{motion:r},t):t}var T=()=>null,A=n(36198),L=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let z=["getTargetContainer","getPopupContainer","renderEmpty","pageHeader","input","pagination","form","select","button"];function _(){return r||"ant"}function H(){return o||x.oR}let B=()=>({getPrefixCls:(e,t)=>t||(e?"".concat(_(),"-").concat(e):_()),getIconPrefixCls:H,getRootPrefixCls:()=>r||_(),getTheme:()=>a,holderRender:i}),D=e=>{let{children:t,csp:n,autoInsertSpaceInButton:r,alert:o,anchor:a,form:i,locale:l,componentSize:g,direction:h,space:E,virtual:S,dropdownMatchSelectWidth:C,popupMatchSelectWidth:Z,popupOverflow:O,legacyLocale:I,parentContext:N,iconPrefixCls:P,theme:_,componentDisabled:H,segmented:B,statistic:D,spin:W,calendar:V,carousel:q,cascader:G,collapse:X,typography:U,checkbox:$,descriptions:K,divider:Y,drawer:Q,skeleton:J,steps:ee,image:et,layout:en,list:er,mentions:eo,modal:ea,progress:ei,result:ec,slider:el,breadcrumb:es,menu:eu,pagination:ed,input:ef,empty:ep,badge:em,radio:eg,rate:eh,switch:ev,transfer:eb,avatar:ey,message:ew,tag:ex,table:eE,card:eS,tabs:eC,timeline:eZ,timePicker:eO,upload:ek,notification:eM,tree:ej,colorPicker:eI,datePicker:eR,rangePicker:eN,flex:eP,wave:eF,dropdown:eT,warning:eA}=e,eL=c.useCallback((t,n)=>{let{prefixCls:r}=e;if(n)return n;let o=r||N.getPrefixCls("");return t?"".concat(o,"-").concat(t):o},[N.getPrefixCls,e.prefixCls]),ez=P||N.iconPrefixCls||x.oR,e_=n||N.csp;(0,A.Z)(ez,e_);let eH=function(e,t){(0,p.ln)("ConfigProvider");let n=e||{},r=!1!==n.inherit&&t?t:y.u_,o=R();return(0,d.Z)(()=>{var a,i;if(!e)return t;let c=Object.assign({},r.components);Object.keys(e.components||{}).forEach(t=>{c[t]=Object.assign(Object.assign({},c[t]),e.components[t])});let l="css-var-".concat(o.replace(/:/g,"")),s=(null!==(a=n.cssVar)&&void 0!==a?a:r.cssVar)&&Object.assign(Object.assign(Object.assign({prefix:"ant"},"object"==typeof r.cssVar?r.cssVar:{}),"object"==typeof n.cssVar?n.cssVar:{}),{key:"object"==typeof n.cssVar&&(null===(i=n.cssVar)||void 0===i?void 0:i.key)||l});return Object.assign(Object.assign(Object.assign({},r),n),{token:Object.assign(Object.assign({},r.token),n.token),components:c,cssVar:s})},[n,r],(e,t)=>e.some((e,n)=>{let r=t[n];return!(0,j.Z)(e,r,!0)}))}(_,N.theme),eB={csp:e_,autoInsertSpaceInButton:r,alert:o,anchor:a,locale:l||I,direction:h,space:E,virtual:S,popupMatchSelectWidth:null!=Z?Z:C,popupOverflow:O,getPrefixCls:eL,iconPrefixCls:ez,theme:eH,segmented:B,statistic:D,spin:W,calendar:V,carousel:q,cascader:G,collapse:X,typography:U,checkbox:$,descriptions:K,divider:Y,drawer:Q,skeleton:J,steps:ee,image:et,input:ef,layout:en,list:er,mentions:eo,modal:ea,progress:ei,result:ec,slider:el,breadcrumb:es,menu:eu,pagination:ed,empty:ep,badge:em,radio:eg,rate:eh,switch:ev,transfer:eb,avatar:ey,message:ew,tag:ex,table:eE,card:eS,tabs:eC,timeline:eZ,timePicker:eO,upload:ek,notification:eM,tree:ej,colorPicker:eI,datePicker:eR,rangePicker:eN,flex:eP,wave:eF,dropdown:eT,warning:eA},eD=Object.assign({},N);Object.keys(eB).forEach(e=>{void 0!==eB[e]&&(eD[e]=eB[e])}),z.forEach(t=>{let n=e[t];n&&(eD[t]=n)});let eW=(0,d.Z)(()=>eD,eD,(e,t)=>{let n=Object.keys(e),r=Object.keys(t);return n.length!==r.length||n.some(n=>e[n]!==t[n])}),eV=c.useMemo(()=>({prefixCls:ez,csp:e_}),[ez,e_]),eq=c.createElement(c.Fragment,null,c.createElement(T,{dropdownMatchSelectWidth:C}),t),eG=c.useMemo(()=>{var e,t,n,r;return(0,f.T)((null===(e=b.Z.Form)||void 0===e?void 0:e.defaultValidateMessages)||{},(null===(n=null===(t=eW.locale)||void 0===t?void 0:t.Form)||void 0===n?void 0:n.defaultValidateMessages)||{},(null===(r=eW.form)||void 0===r?void 0:r.validateMessages)||{},(null==i?void 0:i.validateMessages)||{})},[eW,null==i?void 0:i.validateMessages]);Object.keys(eG).length>0&&(eq=c.createElement(m.Z.Provider,{value:eG},eq)),l&&(eq=c.createElement(v,{locale:l,_ANT_MARK__:"internalMark"},eq)),(ez||e_)&&(eq=c.createElement(u.Z.Provider,{value:eV},eq)),g&&(eq=c.createElement(M.q,{size:g},eq)),eq=c.createElement(F,null,eq);let eX=c.useMemo(()=>{let e=eH||{},{algorithm:t,token:n,components:r,cssVar:o}=e,a=L(e,["algorithm","token","components","cssVar"]),i=t&&(!Array.isArray(t)||t.length>0)?(0,s.jG)(t):y.uH,c={};Object.entries(r||{}).forEach(e=>{let[t,n]=e,r=Object.assign({},n);"algorithm"in r&&(!0===r.algorithm?r.theme=i:(Array.isArray(r.algorithm)||"function"==typeof r.algorithm)&&(r.theme=(0,s.jG)(r.algorithm)),delete r.algorithm),c[t]=r});let l=Object.assign(Object.assign({},w.Z),n);return Object.assign(Object.assign({},a),{theme:i,token:l,components:c,override:Object.assign({override:l},c),cssVar:o})},[eH]);return _&&(eq=c.createElement(y.Mj.Provider,{value:eX},eq)),eW.warning&&(eq=c.createElement(p.G8.Provider,{value:eW.warning},eq)),void 0!==H&&(eq=c.createElement(k.n,{disabled:H},eq)),c.createElement(x.E_.Provider,{value:eW},eq)},W=e=>{let t=c.useContext(x.E_),n=c.useContext(h.Z);return c.createElement(D,Object.assign({parentContext:t,legacyLocale:n},e))};W.ConfigContext=x.E_,W.SizeContext=M.Z,W.config=e=>{let{prefixCls:t,iconPrefixCls:n,theme:c,holderRender:l}=e;void 0!==t&&(r=t),void 0!==n&&(o=n),"holderRender"in e&&(i=l),c&&(Object.keys(c).some(e=>e.endsWith("Color"))?function(e,t){let n=function(e,t){let n={},r=(e,t)=>{let n=e.clone();return(n=(null==t?void 0:t(n))||n).toRgbString()},o=(e,t)=>{let o=new S.C(e),a=(0,E.R_)(o.toRgbString());n["".concat(t,"-color")]=r(o),n["".concat(t,"-color-disabled")]=a[1],n["".concat(t,"-color-hover")]=a[4],n["".concat(t,"-color-active")]=a[6],n["".concat(t,"-color-outline")]=o.clone().setAlpha(.2).toRgbString(),n["".concat(t,"-color-deprecated-bg")]=a[0],n["".concat(t,"-color-deprecated-border")]=a[2]};if(t.primaryColor){o(t.primaryColor,"primary");let e=new S.C(t.primaryColor),a=(0,E.R_)(e.toRgbString());a.forEach((e,t)=>{n["primary-".concat(t+1)]=e}),n["primary-color-deprecated-l-35"]=r(e,e=>e.lighten(35)),n["primary-color-deprecated-l-20"]=r(e,e=>e.lighten(20)),n["primary-color-deprecated-t-20"]=r(e,e=>e.tint(20)),n["primary-color-deprecated-t-50"]=r(e,e=>e.tint(50)),n["primary-color-deprecated-f-12"]=r(e,e=>e.setAlpha(.12*e.getAlpha()));let i=new S.C(a[0]);n["primary-color-active-deprecated-f-30"]=r(i,e=>e.setAlpha(.3*e.getAlpha())),n["primary-color-active-deprecated-d-02"]=r(i,e=>e.darken(2))}t.successColor&&o(t.successColor,"success"),t.warningColor&&o(t.warningColor,"warning"),t.errorColor&&o(t.errorColor,"error"),t.infoColor&&o(t.infoColor,"info");let a=Object.keys(n).map(t=>"--".concat(e,"-").concat(t,": ").concat(n[t],";"));return"\n :root {\n ".concat(a.join("\n"),"\n }\n ").trim()}(e,t);(0,C.Z)()&&(0,Z.hq)(n,"".concat(O,"-dynamic-theme"))}(_(),c):a=c)},W.useConfig=function(){return{componentDisabled:(0,c.useContext)(k.Z),componentSize:(0,c.useContext)(M.Z)}},Object.defineProperty(W,"SizeContext",{get:()=>M.Z});var V=W},85180:function(e,t,n){"use strict";n.d(t,{Z:function(){return b}});var r=n(36760),o=n.n(r),a=n(2265),i=n(71744),c=n(55274),l=n(36360),s=n(29961),u=n(80669),d=n(3104);let f=e=>{let{componentCls:t,margin:n,marginXS:r,marginXL:o,fontSize:a,lineHeight:i}=e;return{[t]:{marginInline:r,fontSize:a,lineHeight:i,textAlign:"center",["".concat(t,"-image")]:{height:e.emptyImgHeight,marginBottom:r,opacity:e.opacityImage,img:{height:"100%"},svg:{maxWidth:"100%",height:"100%",margin:"auto"}},["".concat(t,"-description")]:{color:e.colorText},["".concat(t,"-footer")]:{marginTop:n},"&-normal":{marginBlock:o,color:e.colorTextDisabled,["".concat(t,"-description")]:{color:e.colorTextDisabled},["".concat(t,"-image")]:{height:e.emptyImgHeightMD}},"&-small":{marginBlock:r,color:e.colorTextDisabled,["".concat(t,"-image")]:{height:e.emptyImgHeightSM}}}}};var p=(0,u.I$)("Empty",e=>{let{componentCls:t,controlHeightLG:n,calc:r}=e;return[f((0,d.TS)(e,{emptyImgCls:"".concat(t,"-img"),emptyImgHeight:r(n).mul(2.5).equal(),emptyImgHeightMD:n,emptyImgHeightSM:r(n).mul(.875).equal()}))]}),m=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let g=a.createElement(()=>{let[,e]=(0,s.ZP)(),t=new l.C(e.colorBgBase).toHsl().l<.5?{opacity:.65}:{};return a.createElement("svg",{style:t,width:"184",height:"152",viewBox:"0 0 184 152",xmlns:"http://www.w3.org/2000/svg"},a.createElement("g",{fill:"none",fillRule:"evenodd"},a.createElement("g",{transform:"translate(24 31.67)"},a.createElement("ellipse",{fillOpacity:".8",fill:"#F5F5F7",cx:"67.797",cy:"106.89",rx:"67.797",ry:"12.668"}),a.createElement("path",{d:"M122.034 69.674L98.109 40.229c-1.148-1.386-2.826-2.225-4.593-2.225h-51.44c-1.766 0-3.444.839-4.592 2.225L13.56 69.674v15.383h108.475V69.674z",fill:"#AEB8C2"}),a.createElement("path",{d:"M101.537 86.214L80.63 61.102c-1.001-1.207-2.507-1.867-4.048-1.867H31.724c-1.54 0-3.047.66-4.048 1.867L6.769 86.214v13.792h94.768V86.214z",fill:"url(#linearGradient-1)",transform:"translate(13.56)"}),a.createElement("path",{d:"M33.83 0h67.933a4 4 0 0 1 4 4v93.344a4 4 0 0 1-4 4H33.83a4 4 0 0 1-4-4V4a4 4 0 0 1 4-4z",fill:"#F5F5F7"}),a.createElement("path",{d:"M42.678 9.953h50.237a2 2 0 0 1 2 2V36.91a2 2 0 0 1-2 2H42.678a2 2 0 0 1-2-2V11.953a2 2 0 0 1 2-2zM42.94 49.767h49.713a2.262 2.262 0 1 1 0 4.524H42.94a2.262 2.262 0 0 1 0-4.524zM42.94 61.53h49.713a2.262 2.262 0 1 1 0 4.525H42.94a2.262 2.262 0 0 1 0-4.525zM121.813 105.032c-.775 3.071-3.497 5.36-6.735 5.36H20.515c-3.238 0-5.96-2.29-6.734-5.36a7.309 7.309 0 0 1-.222-1.79V69.675h26.318c2.907 0 5.25 2.448 5.25 5.42v.04c0 2.971 2.37 5.37 5.277 5.37h34.785c2.907 0 5.277-2.421 5.277-5.393V75.1c0-2.972 2.343-5.426 5.25-5.426h26.318v33.569c0 .617-.077 1.216-.221 1.789z",fill:"#DCE0E6"})),a.createElement("path",{d:"M149.121 33.292l-6.83 2.65a1 1 0 0 1-1.317-1.23l1.937-6.207c-2.589-2.944-4.109-6.534-4.109-10.408C138.802 8.102 148.92 0 161.402 0 173.881 0 184 8.102 184 18.097c0 9.995-10.118 18.097-22.599 18.097-4.528 0-8.744-1.066-12.28-2.902z",fill:"#DCE0E6"}),a.createElement("g",{transform:"translate(149.65 15.383)",fill:"#FFF"},a.createElement("ellipse",{cx:"20.654",cy:"3.167",rx:"2.849",ry:"2.815"}),a.createElement("path",{d:"M5.698 5.63H0L2.898.704zM9.259.704h4.985V5.63H9.259z"}))))},null),h=a.createElement(()=>{let[,e]=(0,s.ZP)(),{colorFill:t,colorFillTertiary:n,colorFillQuaternary:r,colorBgContainer:o}=e,{borderColor:i,shadowColor:c,contentColor:u}=(0,a.useMemo)(()=>({borderColor:new l.C(t).onBackground(o).toHexShortString(),shadowColor:new l.C(n).onBackground(o).toHexShortString(),contentColor:new l.C(r).onBackground(o).toHexShortString()}),[t,n,r,o]);return a.createElement("svg",{width:"64",height:"41",viewBox:"0 0 64 41",xmlns:"http://www.w3.org/2000/svg"},a.createElement("g",{transform:"translate(0 1)",fill:"none",fillRule:"evenodd"},a.createElement("ellipse",{fill:c,cx:"32",cy:"33",rx:"32",ry:"7"}),a.createElement("g",{fillRule:"nonzero",stroke:i},a.createElement("path",{d:"M55 12.76L44.854 1.258C44.367.474 43.656 0 42.907 0H21.093c-.749 0-1.46.474-1.947 1.257L9 12.761V22h46v-9.24z"}),a.createElement("path",{d:"M41.613 15.931c0-1.605.994-2.93 2.227-2.931H55v18.137C55 33.26 53.68 35 52.05 35h-40.1C10.32 35 9 33.259 9 31.137V13h11.16c1.233 0 2.227 1.323 2.227 2.928v.022c0 1.605 1.005 2.901 2.237 2.901h14.752c1.232 0 2.237-1.308 2.237-2.913v-.007z",fill:u}))))},null),v=e=>{var{className:t,rootClassName:n,prefixCls:r,image:l=g,description:s,children:u,imageStyle:d,style:f}=e,v=m(e,["className","rootClassName","prefixCls","image","description","children","imageStyle","style"]);let{getPrefixCls:b,direction:y,empty:w}=a.useContext(i.E_),x=b("empty",r),[E,S,C]=p(x),[Z]=(0,c.Z)("Empty"),O=void 0!==s?s:null==Z?void 0:Z.description,k=null;return k="string"==typeof l?a.createElement("img",{alt:"string"==typeof O?O:"empty",src:l}):l,E(a.createElement("div",Object.assign({className:o()(S,C,x,null==w?void 0:w.className,{["".concat(x,"-normal")]:l===h,["".concat(x,"-rtl")]:"rtl"===y},t,n),style:Object.assign(Object.assign({},null==w?void 0:w.style),f)},v),a.createElement("div",{className:"".concat(x,"-image"),style:d},k),O&&a.createElement("div",{className:"".concat(x,"-description")},O),u&&a.createElement("div",{className:"".concat(x,"-footer")},u)))};v.PRESENTED_IMAGE_DEFAULT=g,v.PRESENTED_IMAGE_SIMPLE=h;var b=v},14605:function(e,t,n){"use strict";var r=n(83145),o=n(36760),a=n.n(o),i=n(47970),c=n(2265),l=n(68710),s=n(39109),u=n(4064),d=n(47713),f=n(64024);let p=[];function m(e,t,n){let r=arguments.length>3&&void 0!==arguments[3]?arguments[3]:0;return{key:"string"==typeof e?e:"".concat(t,"-").concat(r),error:e,errorStatus:n}}t.Z=e=>{let{help:t,helpStatus:n,errors:o=p,warnings:g=p,className:h,fieldId:v,onVisibleChanged:b}=e,{prefixCls:y}=c.useContext(s.Rk),w="".concat(y,"-item-explain"),x=(0,f.Z)(y),[E,S,C]=(0,d.ZP)(y,x),Z=(0,c.useMemo)(()=>(0,l.Z)(y),[y]),O=(0,u.Z)(o),k=(0,u.Z)(g),M=c.useMemo(()=>null!=t?[m(t,"help",n)]:[].concat((0,r.Z)(O.map((e,t)=>m(e,"error","error",t))),(0,r.Z)(k.map((e,t)=>m(e,"warning","warning",t)))),[t,n,O,k]),j={};return v&&(j.id="".concat(v,"_help")),E(c.createElement(i.ZP,{motionDeadline:Z.motionDeadline,motionName:"".concat(y,"-show-help"),visible:!!M.length,onVisibleChanged:b},e=>{let{className:t,style:n}=e;return c.createElement("div",Object.assign({},j,{className:a()(w,t,C,x,h,S),style:n,role:"alert"}),c.createElement(i.V4,Object.assign({keys:M},(0,l.Z)(y),{motionName:"".concat(y,"-show-help-item"),component:!1}),e=>{let{key:t,error:n,errorStatus:r,className:o,style:i}=e;return c.createElement("div",{key:t,className:a()(o,{["".concat(w,"-").concat(r)]:r}),style:i},n)}))}))}},38994:function(e,t,n){"use strict";n.d(t,{Z:function(){return U}});var r=n(83145),o=n(2265),a=n(36760),i=n.n(a),c=n(64834),l=n(69819),s=n(28791),u=n(19722),d=n(13613),f=n(71744),p=n(64024),m=n(39109),g=n(45287);let h=()=>{let{status:e,errors:t=[],warnings:n=[]}=(0,o.useContext)(m.aM);return{status:e,errors:t,warnings:n}};h.Context=m.aM;var v=n(53346),b=n(47713),y=n(13861),w=n(2857),x=n(27380),E=n(18694),S=n(10295),C=n(54998),Z=n(14605),O=n(80669);let k=e=>{let{formItemCls:t}=e;return{"@media screen and (-ms-high-contrast: active), (-ms-high-contrast: none)":{["".concat(t,"-control")]:{display:"flex"}}}};var M=(0,O.bk)(["Form","item-item"],(e,t)=>{let{rootPrefixCls:n}=t;return[k((0,b.B4)(e,n))]}),j=e=>{let{prefixCls:t,status:n,wrapperCol:r,children:a,errors:c,warnings:l,_internalItemRender:s,extra:u,help:d,fieldId:f,marginBottom:p,onErrorVisibleChanged:g}=e,h="".concat(t,"-item"),v=o.useContext(m.q3),b=r||v.wrapperCol||{},y=i()("".concat(h,"-control"),b.className),w=o.useMemo(()=>Object.assign({},v),[v]);delete w.labelCol,delete w.wrapperCol;let x=o.createElement("div",{className:"".concat(h,"-control-input")},o.createElement("div",{className:"".concat(h,"-control-input-content")},a)),E=o.useMemo(()=>({prefixCls:t,status:n}),[t,n]),S=null!==p||c.length||l.length?o.createElement("div",{style:{display:"flex",flexWrap:"nowrap"}},o.createElement(m.Rk.Provider,{value:E},o.createElement(Z.Z,{fieldId:f,errors:c,warnings:l,help:d,helpStatus:n,className:"".concat(h,"-explain-connected"),onVisibleChanged:g})),!!p&&o.createElement("div",{style:{width:0,height:p}})):null,O={};f&&(O.id="".concat(f,"_extra"));let k=u?o.createElement("div",Object.assign({},O,{className:"".concat(h,"-extra")}),u):null,j=s&&"pro_table_render"===s.mark&&s.render?s.render(e,{input:x,errorList:S,extra:k}):o.createElement(o.Fragment,null,x,S,k);return o.createElement(m.q3.Provider,{value:w},o.createElement(C.Z,Object.assign({},b,{className:y}),j),o.createElement(M,{prefixCls:t}))},I=n(67187),R=n(13823),N=n(55274),P=n(89970),F=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n},T=e=>{var t;let{prefixCls:n,label:r,htmlFor:a,labelCol:c,labelAlign:l,colon:s,required:u,requiredMark:d,tooltip:f}=e,[p]=(0,N.Z)("Form"),{vertical:g,labelAlign:h,labelCol:v,labelWrap:b,colon:y}=o.useContext(m.q3);if(!r)return null;let w=c||v||{},x="".concat(n,"-item-label"),E=i()(x,"left"===(l||h)&&"".concat(x,"-left"),w.className,{["".concat(x,"-wrap")]:!!b}),S=r,Z=!0===s||!1!==y&&!1!==s;Z&&!g&&"string"==typeof r&&""!==r.trim()&&(S=r.replace(/[:|:]\s*$/,""));let O=f?"object"!=typeof f||o.isValidElement(f)?{title:f}:f:null;if(O){let{icon:e=o.createElement(I.Z,null)}=O,t=F(O,["icon"]),r=o.createElement(P.Z,Object.assign({},t),o.cloneElement(e,{className:"".concat(n,"-item-tooltip"),title:"",onClick:e=>{e.preventDefault()},tabIndex:null}));S=o.createElement(o.Fragment,null,S,r)}let k="optional"===d,M="function"==typeof d;M?S=d(S,{required:!!u}):k&&!u&&(S=o.createElement(o.Fragment,null,S,o.createElement("span",{className:"".concat(n,"-item-optional"),title:""},(null==p?void 0:p.optional)||(null===(t=R.Z.Form)||void 0===t?void 0:t.optional))));let j=i()({["".concat(n,"-item-required")]:u,["".concat(n,"-item-required-mark-optional")]:k||M,["".concat(n,"-item-no-colon")]:!Z});return o.createElement(C.Z,Object.assign({},w,{className:E}),o.createElement("label",{htmlFor:a,className:j,title:"string"==typeof r?r:""},S))},A=n(4064),L=n(8900),z=n(39725),_=n(54537),H=n(61935);let B={success:L.Z,warning:_.Z,error:z.Z,validating:H.Z};function D(e){let{children:t,errors:n,warnings:r,hasFeedback:a,validateStatus:c,prefixCls:l,meta:s,noStyle:u}=e,d="".concat(l,"-item"),{feedbackIcons:f}=o.useContext(m.q3),p=(0,y.lR)(n,r,s,null,!!a,c),{isFormItemInput:g,status:h,hasFeedback:v,feedbackIcon:b}=o.useContext(m.aM),w=o.useMemo(()=>{var e;let t;if(a){let c=!0!==a&&a.icons||f,l=p&&(null===(e=null==c?void 0:c({status:p,errors:n,warnings:r}))||void 0===e?void 0:e[p]),s=p&&B[p];t=!1!==l&&s?o.createElement("span",{className:i()("".concat(d,"-feedback-icon"),"".concat(d,"-feedback-icon-").concat(p))},l||o.createElement(s,null)):null}let c={status:p||"",errors:n,warnings:r,hasFeedback:!!a,feedbackIcon:t,isFormItemInput:!0};return u&&(c.status=(null!=p?p:h)||"",c.isFormItemInput=g,c.hasFeedback=!!(null!=a?a:v),c.feedbackIcon=void 0!==a?c.feedbackIcon:b),c},[p,a,u,g,h]);return o.createElement(m.aM.Provider,{value:w},t)}var W=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};function V(e){let{prefixCls:t,className:n,rootClassName:r,style:a,help:c,errors:l,warnings:s,validateStatus:u,meta:d,hasFeedback:f,hidden:p,children:g,fieldId:h,required:v,isRequired:b,onSubItemMetaChange:C}=e,Z=W(e,["prefixCls","className","rootClassName","style","help","errors","warnings","validateStatus","meta","hasFeedback","hidden","children","fieldId","required","isRequired","onSubItemMetaChange"]),O="".concat(t,"-item"),{requiredMark:k}=o.useContext(m.q3),M=o.useRef(null),I=(0,A.Z)(l),R=(0,A.Z)(s),N=null!=c,P=!!(N||l.length||s.length),F=!!M.current&&(0,w.Z)(M.current),[L,z]=o.useState(null);(0,x.Z)(()=>{P&&M.current&&z(parseInt(getComputedStyle(M.current).marginBottom,10))},[P,F]);let _=function(){let e=arguments.length>0&&void 0!==arguments[0]&&arguments[0],t=e?I:d.errors,n=e?R:d.warnings;return(0,y.lR)(t,n,d,"",!!f,u)}(),H=i()(O,n,r,{["".concat(O,"-with-help")]:N||I.length||R.length,["".concat(O,"-has-feedback")]:_&&f,["".concat(O,"-has-success")]:"success"===_,["".concat(O,"-has-warning")]:"warning"===_,["".concat(O,"-has-error")]:"error"===_,["".concat(O,"-is-validating")]:"validating"===_,["".concat(O,"-hidden")]:p});return o.createElement("div",{className:H,style:a,ref:M},o.createElement(S.Z,Object.assign({className:"".concat(O,"-row")},(0,E.Z)(Z,["_internalItemRender","colon","dependencies","extra","fieldKey","getValueFromEvent","getValueProps","htmlFor","id","initialValue","isListField","label","labelAlign","labelCol","labelWrap","messageVariables","name","normalize","noStyle","preserve","requiredMark","rules","shouldUpdate","trigger","tooltip","validateFirst","validateTrigger","valuePropName","wrapperCol","validateDebounce"])),o.createElement(T,Object.assign({htmlFor:h},e,{requiredMark:k,required:null!=v?v:b,prefixCls:t})),o.createElement(j,Object.assign({},e,d,{errors:I,warnings:R,prefixCls:t,status:_,help:c,marginBottom:L,onErrorVisibleChanged:e=>{e||z(null)}}),o.createElement(m.qI.Provider,{value:C},o.createElement(D,{prefixCls:t,meta:d,errors:d.errors,warnings:d.warnings,hasFeedback:f,validateStatus:_},g)))),!!L&&o.createElement("div",{className:"".concat(O,"-margin-offset"),style:{marginBottom:-L}}))}let q=o.memo(e=>{let{children:t}=e;return t},(e,t)=>(function(e,t){let n=Object.keys(e),r=Object.keys(t);return n.length===r.length&&n.every(n=>{let r=e[n],o=t[n];return r===o||"function"==typeof r||"function"==typeof o})})(e.control,t.control)&&e.update===t.update&&e.childProps.length===t.childProps.length&&e.childProps.every((e,n)=>e===t.childProps[n]));function G(){return{errors:[],warnings:[],touched:!1,validating:!1,name:[],validated:!1}}let X=function(e){let{name:t,noStyle:n,className:a,dependencies:h,prefixCls:w,shouldUpdate:x,rules:E,children:S,required:C,label:Z,messageVariables:O,trigger:k="onChange",validateTrigger:M,hidden:j,help:I}=e,{getPrefixCls:R}=o.useContext(f.E_),{name:N}=o.useContext(m.q3),P=function(e){if("function"==typeof e)return e;let t=(0,g.Z)(e);return t.length<=1?t[0]:t}(S),F="function"==typeof P,T=o.useContext(m.qI),{validateTrigger:A}=o.useContext(c.zb),L=void 0!==M?M:A,z=null!=t,_=R("form",w),H=(0,p.Z)(_),[B,W,X]=(0,b.ZP)(_,H);(0,d.ln)("Form.Item");let U=o.useContext(c.ZM),$=o.useRef(),[K,Y]=function(e){let[t,n]=o.useState(e),r=(0,o.useRef)(null),a=(0,o.useRef)([]),i=(0,o.useRef)(!1);return o.useEffect(()=>(i.current=!1,()=>{i.current=!0,v.Z.cancel(r.current),r.current=null}),[]),[t,function(e){i.current||(null===r.current&&(a.current=[],r.current=(0,v.Z)(()=>{r.current=null,n(e=>{let t=e;return a.current.forEach(e=>{t=e(t)}),t})})),a.current.push(e))}]}({}),[Q,J]=(0,l.Z)(()=>G()),ee=(e,t)=>{Y(n=>{let o=Object.assign({},n),a=[].concat((0,r.Z)(e.name.slice(0,-1)),(0,r.Z)(t)).join("__SPLIT__");return e.destroy?delete o[a]:o[a]=e,o})},[et,en]=o.useMemo(()=>{let e=(0,r.Z)(Q.errors),t=(0,r.Z)(Q.warnings);return Object.values(K).forEach(n=>{e.push.apply(e,(0,r.Z)(n.errors||[])),t.push.apply(t,(0,r.Z)(n.warnings||[]))}),[e,t]},[K,Q.errors,Q.warnings]),er=function(){let{itemRef:e}=o.useContext(m.q3),t=o.useRef({});return function(n,r){let o=r&&"object"==typeof r&&r.ref,a=n.join("_");return(t.current.name!==a||t.current.originRef!==o)&&(t.current.name=a,t.current.originRef=o,t.current.ref=(0,s.sQ)(e(n),o)),t.current.ref}}();function eo(t,r,c){return n&&!j?o.createElement(D,{prefixCls:_,hasFeedback:e.hasFeedback,validateStatus:e.validateStatus,meta:Q,errors:et,warnings:en,noStyle:!0},t):o.createElement(V,Object.assign({key:"row"},e,{className:i()(a,X,H,W),prefixCls:_,fieldId:r,isRequired:c,errors:et,warnings:en,meta:Q,onSubItemMetaChange:ee}),t)}if(!z&&!F&&!h)return B(eo(P));let ea={};return"string"==typeof Z?ea.label=Z:t&&(ea.label=String(t)),O&&(ea=Object.assign(Object.assign({},ea),O)),B(o.createElement(c.gN,Object.assign({},e,{messageVariables:ea,trigger:k,validateTrigger:L,onMetaChange:e=>{let t=null==U?void 0:U.getKey(e.name);if(J(e.destroy?G():e,!0),n&&!1!==I&&T){let n=e.name;if(e.destroy)n=$.current||n;else if(void 0!==t){let[e,o]=t;n=[e].concat((0,r.Z)(o)),$.current=n}T(e,n)}}}),(n,a,i)=>{let c=(0,y.qo)(t).length&&a?a.name:[],l=(0,y.dD)(c,N),d=void 0!==C?C:!!(E&&E.some(e=>{if(e&&"object"==typeof e&&e.required&&!e.warningOnly)return!0;if("function"==typeof e){let t=e(i);return t&&t.required&&!t.warningOnly}return!1})),f=Object.assign({},n),p=null;if(Array.isArray(P)&&z)p=P;else if(F&&(!(x||h)||z));else if(!h||F||z){if((0,u.l$)(P)){let t=Object.assign(Object.assign({},P.props),f);if(t.id||(t.id=l),I||et.length>0||en.length>0||e.extra){let n=[];(I||et.length>0)&&n.push("".concat(l,"_help")),e.extra&&n.push("".concat(l,"_extra")),t["aria-describedby"]=n.join(" ")}et.length>0&&(t["aria-invalid"]="true"),d&&(t["aria-required"]="true"),(0,s.Yr)(P)&&(t.ref=er(c,P)),new Set([].concat((0,r.Z)((0,y.qo)(k)),(0,r.Z)((0,y.qo)(L)))).forEach(e=>{t[e]=function(){for(var t,n,r,o=arguments.length,a=Array(o),i=0;i{}}),c=r.createContext(null),l=e=>{let t=(0,a.Z)(e,["prefixCls"]);return r.createElement(o.RV,Object.assign({},t))},s=r.createContext({prefixCls:""}),u=r.createContext({}),d=e=>{let{children:t,status:n,override:o}=e,a=(0,r.useContext)(u),i=(0,r.useMemo)(()=>{let e=Object.assign({},a);return o&&delete e.isFormItemInput,n&&(delete e.status,delete e.hasFeedback,delete e.feedbackIcon),e},[n,o,a]);return r.createElement(u.Provider,{value:i},t)},f=(0,r.createContext)(void 0)},4064:function(e,t,n){"use strict";n.d(t,{Z:function(){return o}});var r=n(2265);function o(e){let[t,n]=r.useState(e);return r.useEffect(()=>{let t=setTimeout(()=>{n(e)},e.length?0:10);return()=>{clearTimeout(t)}},[e]),t}},56250:function(e,t,n){"use strict";var r=n(2265),o=n(39109);let a=["outlined","borderless","filled"];t.Z=function(e){let t,n=arguments.length>1&&void 0!==arguments[1]?arguments[1]:void 0,i=(0,r.useContext)(o.pg);t=void 0!==e?e:!1===n?"borderless":null!=i?i:"outlined";let c=a.includes(t);return[t,c]}},13634:function(e,t,n){"use strict";n.d(t,{Z:function(){return I}});var r=n(14605),o=n(2265),a=n(36760),i=n.n(a),c=n(64834),l=n(71744),s=n(86586),u=n(64024),d=n(33759),f=n(59189),p=n(39109);let m=e=>"object"==typeof e&&null!=e&&1===e.nodeType,g=(e,t)=>(!t||"hidden"!==e)&&"visible"!==e&&"clip"!==e,h=(e,t)=>{if(e.clientHeight{let t=(e=>{if(!e.ownerDocument||!e.ownerDocument.defaultView)return null;try{return e.ownerDocument.defaultView.frameElement}catch(e){return null}})(e);return!!t&&(t.clientHeightat||a>e&&i=t&&c>=n?a-e-r:i>t&&cn?i-t+o:0,b=e=>{let t=e.parentElement;return null==t?e.getRootNode().host||null:t},y=(e,t)=>{var n,r,o,a;if("undefined"==typeof document)return[];let{scrollMode:i,block:c,inline:l,boundary:s,skipOverflowHiddenElements:u}=t,d="function"==typeof s?s:e=>e!==s;if(!m(e))throw TypeError("Invalid target");let f=document.scrollingElement||document.documentElement,p=[],g=e;for(;m(g)&&d(g);){if((g=b(g))===f){p.push(g);break}null!=g&&g===document.body&&h(g)&&!h(document.documentElement)||null!=g&&h(g,u)&&p.push(g)}let y=null!=(r=null==(n=window.visualViewport)?void 0:n.width)?r:innerWidth,w=null!=(a=null==(o=window.visualViewport)?void 0:o.height)?a:innerHeight,{scrollX:x,scrollY:E}=window,{height:S,width:C,top:Z,right:O,bottom:k,left:M}=e.getBoundingClientRect(),{top:j,right:I,bottom:R,left:N}=(e=>{let t=window.getComputedStyle(e);return{top:parseFloat(t.scrollMarginTop)||0,right:parseFloat(t.scrollMarginRight)||0,bottom:parseFloat(t.scrollMarginBottom)||0,left:parseFloat(t.scrollMarginLeft)||0}})(e),P="start"===c||"nearest"===c?Z-j:"end"===c?k+R:Z+S/2-j+R,F="center"===l?M+C/2-N+I:"end"===l?O+I:M-N,T=[];for(let e=0;e=0&&M>=0&&k<=w&&O<=y&&Z>=o&&k<=s&&M>=u&&O<=a)break;let d=getComputedStyle(t),m=parseInt(d.borderLeftWidth,10),g=parseInt(d.borderTopWidth,10),h=parseInt(d.borderRightWidth,10),b=parseInt(d.borderBottomWidth,10),j=0,I=0,R="offsetWidth"in t?t.offsetWidth-t.clientWidth-m-h:0,N="offsetHeight"in t?t.offsetHeight-t.clientHeight-g-b:0,A="offsetWidth"in t?0===t.offsetWidth?0:r/t.offsetWidth:0,L="offsetHeight"in t?0===t.offsetHeight?0:n/t.offsetHeight:0;if(f===t)j="start"===c?P:"end"===c?P-w:"nearest"===c?v(E,E+w,w,g,b,E+P,E+P+S,S):P-w/2,I="start"===l?F:"center"===l?F-y/2:"end"===l?F-y:v(x,x+y,y,m,h,x+F,x+F+C,C),j=Math.max(0,j+E),I=Math.max(0,I+x);else{j="start"===c?P-o-g:"end"===c?P-s+b+N:"nearest"===c?v(o,s,n,g,b+N,P,P+S,S):P-(o+n/2)+N/2,I="start"===l?F-u-m:"center"===l?F-(u+r/2)+R/2:"end"===l?F-a+h+R:v(u,a,r,m,h+R,F,F+C,C);let{scrollLeft:e,scrollTop:i}=t;j=0===L?0:Math.max(0,Math.min(i+j/L,t.scrollHeight-n/L+N)),I=0===A?0:Math.max(0,Math.min(e+I/A,t.scrollWidth-r/A+R)),P+=i-j,F+=e-I}T.push({el:t,top:j,left:I})}return T},w=e=>!1===e?{block:"end",inline:"nearest"}:e===Object(e)&&0!==Object.keys(e).length?e:{block:"start",inline:"nearest"};var x=n(13861);function E(e){return(0,x.qo)(e).join("_")}function S(e){let[t]=(0,c.cI)(),n=o.useRef({}),r=o.useMemo(()=>null!=e?e:Object.assign(Object.assign({},t),{__INTERNAL__:{itemRef:e=>t=>{let r=E(e);t?n.current[r]=t:delete n.current[r]}},scrollToField:function(e){let t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},n=(0,x.qo)(e),o=(0,x.dD)(n,r.__INTERNAL__.name),a=o?document.getElementById(o):null;a&&function(e,t){if(!e.isConnected||!(e=>{let t=e;for(;t&&t.parentNode;){if(t.parentNode===document)return!0;t=t.parentNode instanceof ShadowRoot?t.parentNode.host:t.parentNode}return!1})(e))return;let n=(e=>{let t=window.getComputedStyle(e);return{top:parseFloat(t.scrollMarginTop)||0,right:parseFloat(t.scrollMarginRight)||0,bottom:parseFloat(t.scrollMarginBottom)||0,left:parseFloat(t.scrollMarginLeft)||0}})(e);if("object"==typeof t&&"function"==typeof t.behavior)return t.behavior(y(e,t));let r="boolean"==typeof t||null==t?void 0:t.behavior;for(let{el:o,top:a,left:i}of y(e,w(t))){let e=a-n.top+n.bottom,t=i-n.left+n.right;o.scroll({top:e,left:t,behavior:r})}}(a,Object.assign({scrollMode:"if-needed",block:"nearest"},t))},getFieldInstance:e=>{let t=E(e);return n.current[t]}}),[e,t]);return[r]}var C=n(47713),Z=n(77360),O=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let k=o.forwardRef((e,t)=>{let n=o.useContext(s.Z),{getPrefixCls:r,direction:a,form:m}=o.useContext(l.E_),{prefixCls:g,className:h,rootClassName:v,size:b,disabled:y=n,form:w,colon:x,labelAlign:E,labelWrap:k,labelCol:M,wrapperCol:j,hideRequiredMark:I,layout:R="horizontal",scrollToFirstError:N,requiredMark:P,onFinishFailed:F,name:T,style:A,feedbackIcons:L,variant:z}=e,_=O(e,["prefixCls","className","rootClassName","size","disabled","form","colon","labelAlign","labelWrap","labelCol","wrapperCol","hideRequiredMark","layout","scrollToFirstError","requiredMark","onFinishFailed","name","style","feedbackIcons","variant"]),H=(0,d.Z)(b),B=o.useContext(Z.Z),D=(0,o.useMemo)(()=>void 0!==P?P:!I&&(!m||void 0===m.requiredMark||m.requiredMark),[I,P,m]),W=null!=x?x:null==m?void 0:m.colon,V=r("form",g),q=(0,u.Z)(V),[G,X,U]=(0,C.ZP)(V,q),$=i()(V,"".concat(V,"-").concat(R),{["".concat(V,"-hide-required-mark")]:!1===D,["".concat(V,"-rtl")]:"rtl"===a,["".concat(V,"-").concat(H)]:H},U,q,X,null==m?void 0:m.className,h,v),[K]=S(w),{__INTERNAL__:Y}=K;Y.name=T;let Q=(0,o.useMemo)(()=>({name:T,labelAlign:E,labelCol:M,labelWrap:k,wrapperCol:j,vertical:"vertical"===R,colon:W,requiredMark:D,itemRef:Y.itemRef,form:K,feedbackIcons:L}),[T,E,M,j,R,W,D,K,L]);o.useImperativeHandle(t,()=>K);let J=(e,t)=>{if(e){let n={block:"nearest"};"object"==typeof e&&(n=e),K.scrollToField(t,n)}};return G(o.createElement(p.pg.Provider,{value:z},o.createElement(s.n,{disabled:y},o.createElement(f.Z.Provider,{value:H},o.createElement(p.RV,{validateMessages:B},o.createElement(p.q3.Provider,{value:Q},o.createElement(c.ZP,Object.assign({id:T},_,{name:T,onFinishFailed:e=>{if(null==F||F(e),e.errorFields.length){let t=e.errorFields[0].name;if(void 0!==N){J(N,t);return}m&&void 0!==m.scrollToFirstError&&J(m.scrollToFirstError,t)}},form:K,style:Object.assign(Object.assign({},null==m?void 0:m.style),A),className:$}))))))))});var M=n(38994),j=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};k.Item=M.Z,k.List=e=>{var{prefixCls:t,children:n}=e,r=j(e,["prefixCls","children"]);let{getPrefixCls:a}=o.useContext(l.E_),i=a("form",t),s=o.useMemo(()=>({prefixCls:i,status:"error"}),[i]);return o.createElement(c.aV,Object.assign({},r),(e,t,r)=>o.createElement(p.Rk.Provider,{value:s},n(e.map(e=>Object.assign(Object.assign({},e),{fieldKey:e.key})),t,{errors:r.errors,warnings:r.warnings})))},k.ErrorList=r.Z,k.useForm=S,k.useFormInstance=function(){let{form:e}=(0,o.useContext)(p.q3);return e},k.useWatch=c.qo,k.Provider=p.RV,k.create=()=>{};var I=k},47713:function(e,t,n){"use strict";n.d(t,{ZP:function(){return w},B4:function(){return y}});var r=n(352),o=n(12918),a=n(691),i=n(63074),c=n(3104),l=n(80669),s=e=>{let{componentCls:t}=e,n="".concat(t,"-show-help"),r="".concat(t,"-show-help-item");return{[n]:{transition:"opacity ".concat(e.motionDurationSlow," ").concat(e.motionEaseInOut),"&-appear, &-enter":{opacity:0,"&-active":{opacity:1}},"&-leave":{opacity:1,"&-active":{opacity:0}},[r]:{overflow:"hidden",transition:"height ".concat(e.motionDurationSlow," ").concat(e.motionEaseInOut,",\n opacity ").concat(e.motionDurationSlow," ").concat(e.motionEaseInOut,",\n transform ").concat(e.motionDurationSlow," ").concat(e.motionEaseInOut," !important"),["&".concat(r,"-appear, &").concat(r,"-enter")]:{transform:"translateY(-5px)",opacity:0,"&-active":{transform:"translateY(0)",opacity:1}},["&".concat(r,"-leave-active")]:{transform:"translateY(-5px)"}}}}};let u=e=>({legend:{display:"block",width:"100%",marginBottom:e.marginLG,padding:0,color:e.colorTextDescription,fontSize:e.fontSizeLG,lineHeight:"inherit",border:0,borderBottom:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorBorder)},'input[type="search"]':{boxSizing:"border-box"},'input[type="radio"], input[type="checkbox"]':{lineHeight:"normal"},'input[type="file"]':{display:"block"},'input[type="range"]':{display:"block",width:"100%"},"select[multiple], select[size]":{height:"auto"},"input[type='file']:focus,\n input[type='radio']:focus,\n input[type='checkbox']:focus":{outline:0,boxShadow:"0 0 0 ".concat((0,r.bf)(e.controlOutlineWidth)," ").concat(e.controlOutline)},output:{display:"block",paddingTop:15,color:e.colorText,fontSize:e.fontSize,lineHeight:e.lineHeight}}),d=(e,t)=>{let{formItemCls:n}=e;return{[n]:{["".concat(n,"-label > label")]:{height:t},["".concat(n,"-control-input")]:{minHeight:t}}}},f=e=>{let{componentCls:t}=e;return{[e.componentCls]:Object.assign(Object.assign(Object.assign({},(0,o.Wf)(e)),u(e)),{["".concat(t,"-text")]:{display:"inline-block",paddingInlineEnd:e.paddingSM},"&-small":Object.assign({},d(e,e.controlHeightSM)),"&-large":Object.assign({},d(e,e.controlHeightLG))})}},p=e=>{let{formItemCls:t,iconCls:n,componentCls:r,rootPrefixCls:i,labelRequiredMarkColor:c,labelColor:l,labelFontSize:s,labelHeight:u,labelColonMarginInlineStart:d,labelColonMarginInlineEnd:f,itemMarginBottom:p}=e;return{[t]:Object.assign(Object.assign({},(0,o.Wf)(e)),{marginBottom:p,verticalAlign:"top","&-with-help":{transition:"none"},["&-hidden,\n &-hidden.".concat(i,"-row")]:{display:"none"},"&-has-warning":{["".concat(t,"-split")]:{color:e.colorError}},"&-has-error":{["".concat(t,"-split")]:{color:e.colorWarning}},["".concat(t,"-label")]:{flexGrow:0,overflow:"hidden",whiteSpace:"nowrap",textAlign:"end",verticalAlign:"middle","&-left":{textAlign:"start"},"&-wrap":{overflow:"unset",lineHeight:e.lineHeight,whiteSpace:"unset"},"> label":{position:"relative",display:"inline-flex",alignItems:"center",maxWidth:"100%",height:u,color:l,fontSize:s,["> ".concat(n)]:{fontSize:e.fontSize,verticalAlign:"top"},["&".concat(t,"-required:not(").concat(t,"-required-mark-optional)::before")]:{display:"inline-block",marginInlineEnd:e.marginXXS,color:c,fontSize:e.fontSize,fontFamily:"SimSun, sans-serif",lineHeight:1,content:'"*"',["".concat(r,"-hide-required-mark &")]:{display:"none"}},["".concat(t,"-optional")]:{display:"inline-block",marginInlineStart:e.marginXXS,color:e.colorTextDescription,["".concat(r,"-hide-required-mark &")]:{display:"none"}},["".concat(t,"-tooltip")]:{color:e.colorTextDescription,cursor:"help",writingMode:"horizontal-tb",marginInlineStart:e.marginXXS},"&::after":{content:'":"',position:"relative",marginBlock:0,marginInlineStart:d,marginInlineEnd:f},["&".concat(t,"-no-colon::after")]:{content:'"\\a0"'}}},["".concat(t,"-control")]:{"--ant-display":"flex",flexDirection:"column",flexGrow:1,["&:first-child:not([class^=\"'".concat(i,"-col-'\"]):not([class*=\"' ").concat(i,"-col-'\"])")]:{width:"100%"},"&-input":{position:"relative",display:"flex",alignItems:"center",minHeight:e.controlHeight,"&-content":{flex:"auto",maxWidth:"100%"}}},[t]:{"&-explain, &-extra":{clear:"both",color:e.colorTextDescription,fontSize:e.fontSize,lineHeight:e.lineHeight},"&-explain-connected":{width:"100%"},"&-extra":{minHeight:e.controlHeightSM,transition:"color ".concat(e.motionDurationMid," ").concat(e.motionEaseOut)},"&-explain":{"&-error":{color:e.colorError},"&-warning":{color:e.colorWarning}}},["&-with-help ".concat(t,"-explain")]:{height:"auto",opacity:1},["".concat(t,"-feedback-icon")]:{fontSize:e.fontSize,textAlign:"center",visibility:"visible",animationName:a.kr,animationDuration:e.motionDurationMid,animationTimingFunction:e.motionEaseOutBack,pointerEvents:"none","&-success":{color:e.colorSuccess},"&-error":{color:e.colorError},"&-warning":{color:e.colorWarning},"&-validating":{color:e.colorPrimary}}})}},m=e=>{let{componentCls:t,formItemCls:n}=e;return{["".concat(t,"-horizontal")]:{["".concat(n,"-label")]:{flexGrow:0},["".concat(n,"-control")]:{flex:"1 1 0",minWidth:0},["".concat(n,"-label[class$='-24'], ").concat(n,"-label[class*='-24 ']")]:{["& + ".concat(n,"-control")]:{minWidth:"unset"}}}}},g=e=>{let{componentCls:t,formItemCls:n}=e;return{["".concat(t,"-inline")]:{display:"flex",flexWrap:"wrap",[n]:{flex:"none",marginInlineEnd:e.margin,marginBottom:0,"&-row":{flexWrap:"nowrap"},["> ".concat(n,"-label,\n > ").concat(n,"-control")]:{display:"inline-block",verticalAlign:"top"},["> ".concat(n,"-label")]:{flex:"none"},["".concat(t,"-text")]:{display:"inline-block"},["".concat(n,"-has-feedback")]:{display:"inline-block"}}}}},h=e=>({padding:e.verticalLabelPadding,margin:e.verticalLabelMargin,whiteSpace:"initial",textAlign:"start","> label":{margin:0,"&::after":{visibility:"hidden"}}}),v=e=>{let{componentCls:t,formItemCls:n,rootPrefixCls:r}=e;return{["".concat(n," ").concat(n,"-label")]:h(e),["".concat(t,":not(").concat(t,"-inline)")]:{[n]:{flexWrap:"wrap",["".concat(n,"-label, ").concat(n,"-control")]:{['&:not([class*=" '.concat(r,'-col-xs"])')]:{flex:"0 0 100%",maxWidth:"100%"}}}}}},b=e=>{let{componentCls:t,formItemCls:n,rootPrefixCls:o}=e;return{["".concat(t,"-vertical")]:{[n]:{"&-row":{flexDirection:"column"},"&-label > label":{height:"auto"},["".concat(t,"-item-control")]:{width:"100%"}}},["".concat(t,"-vertical ").concat(n,"-label,\n .").concat(o,"-col-24").concat(n,"-label,\n .").concat(o,"-col-xl-24").concat(n,"-label")]:h(e),["@media (max-width: ".concat((0,r.bf)(e.screenXSMax),")")]:[v(e),{[t]:{[".".concat(o,"-col-xs-24").concat(n,"-label")]:h(e)}}],["@media (max-width: ".concat((0,r.bf)(e.screenSMMax),")")]:{[t]:{[".".concat(o,"-col-sm-24").concat(n,"-label")]:h(e)}},["@media (max-width: ".concat((0,r.bf)(e.screenMDMax),")")]:{[t]:{[".".concat(o,"-col-md-24").concat(n,"-label")]:h(e)}},["@media (max-width: ".concat((0,r.bf)(e.screenLGMax),")")]:{[t]:{[".".concat(o,"-col-lg-24").concat(n,"-label")]:h(e)}}}},y=(e,t)=>(0,c.TS)(e,{formItemCls:"".concat(e.componentCls,"-item"),rootPrefixCls:t});var w=(0,l.I$)("Form",(e,t)=>{let{rootPrefixCls:n}=t,r=y(e,n);return[f(r),p(r),s(r),m(r),g(r),b(r),(0,i.Z)(r),a.kr]},e=>({labelRequiredMarkColor:e.colorError,labelColor:e.colorTextHeading,labelFontSize:e.fontSize,labelHeight:e.controlHeight,labelColonMarginInlineStart:e.marginXXS/2,labelColonMarginInlineEnd:e.marginXS,itemMarginBottom:e.marginLG,verticalLabelPadding:"0 0 ".concat(e.paddingXS,"px"),verticalLabelMargin:0}),{order:-1e3})},13861:function(e,t,n){"use strict";n.d(t,{dD:function(){return a},lR:function(){return i},qo:function(){return o}});let r=["parentNode"];function o(e){return void 0===e||!1===e?[]:Array.isArray(e)?e:[e]}function a(e,t){if(!e.length)return;let n=e.join("_");return t?"".concat(t,"_").concat(n):r.includes(n)?"".concat("form_item","_").concat(n):n}function i(e,t,n,r,o,a){let i=r;return void 0!==a?i=a:n.validating?i="validating":e.length?i="error":t.length?i="warning":(n.touched||o&&n.validated)&&(i="success"),i}},77360:function(e,t,n){"use strict";var r=n(2265);t.Z=(0,r.createContext)(void 0)},62807:function(e,t,n){"use strict";let r=(0,n(2265).createContext)({});t.Z=r},54998:function(e,t,n){"use strict";var r=n(2265),o=n(36760),a=n.n(o),i=n(71744),c=n(62807),l=n(96776),s=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let u=["xs","sm","md","lg","xl","xxl"],d=r.forwardRef((e,t)=>{let{getPrefixCls:n,direction:o}=r.useContext(i.E_),{gutter:d,wrap:f}=r.useContext(c.Z),{prefixCls:p,span:m,order:g,offset:h,push:v,pull:b,className:y,children:w,flex:x,style:E}=e,S=s(e,["prefixCls","span","order","offset","push","pull","className","children","flex","style"]),C=n("col",p),[Z,O,k]=(0,l.cG)(C),M={};u.forEach(t=>{let n={},r=e[t];"number"==typeof r?n.span=r:"object"==typeof r&&(n=r||{}),delete S[t],M=Object.assign(Object.assign({},M),{["".concat(C,"-").concat(t,"-").concat(n.span)]:void 0!==n.span,["".concat(C,"-").concat(t,"-order-").concat(n.order)]:n.order||0===n.order,["".concat(C,"-").concat(t,"-offset-").concat(n.offset)]:n.offset||0===n.offset,["".concat(C,"-").concat(t,"-push-").concat(n.push)]:n.push||0===n.push,["".concat(C,"-").concat(t,"-pull-").concat(n.pull)]:n.pull||0===n.pull,["".concat(C,"-").concat(t,"-flex-").concat(n.flex)]:n.flex||"auto"===n.flex,["".concat(C,"-rtl")]:"rtl"===o})});let j=a()(C,{["".concat(C,"-").concat(m)]:void 0!==m,["".concat(C,"-order-").concat(g)]:g,["".concat(C,"-offset-").concat(h)]:h,["".concat(C,"-push-").concat(v)]:v,["".concat(C,"-pull-").concat(b)]:b},y,M,O,k),I={};if(d&&d[0]>0){let e=d[0]/2;I.paddingLeft=e,I.paddingRight=e}return x&&(I.flex="number"==typeof x?"".concat(x," ").concat(x," auto"):/^\d+(\.\d+)?(px|em|rem|%)$/.test(x)?"0 0 ".concat(x):x,!1!==f||I.minWidth||(I.minWidth=0)),Z(r.createElement("div",Object.assign({},S,{style:Object.assign(Object.assign({},I),E),className:j,ref:t}),w))});t.Z=d},10295:function(e,t,n){"use strict";var r=n(2265),o=n(36760),a=n.n(o),i=n(6543),c=n(71744),l=n(62807),s=n(96776),u=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};function d(e,t){let[n,o]=r.useState("string"==typeof e?e:""),a=()=>{if("string"==typeof e&&o(e),"object"==typeof e)for(let n=0;n{a()},[JSON.stringify(e),t]),n}let f=r.forwardRef((e,t)=>{let{prefixCls:n,justify:o,align:f,className:p,style:m,children:g,gutter:h=0,wrap:v}=e,b=u(e,["prefixCls","justify","align","className","style","children","gutter","wrap"]),{getPrefixCls:y,direction:w}=r.useContext(c.E_),[x,E]=r.useState({xs:!0,sm:!0,md:!0,lg:!0,xl:!0,xxl:!0}),[S,C]=r.useState({xs:!1,sm:!1,md:!1,lg:!1,xl:!1,xxl:!1}),Z=d(f,S),O=d(o,S),k=r.useRef(h),M=(0,i.ZP)();r.useEffect(()=>{let e=M.subscribe(e=>{C(e);let t=k.current||0;(!Array.isArray(t)&&"object"==typeof t||Array.isArray(t)&&("object"==typeof t[0]||"object"==typeof t[1]))&&E(e)});return()=>M.unsubscribe(e)},[]);let j=y("row",n),[I,R,N]=(0,s.VM)(j),P=(()=>{let e=[void 0,void 0];return(Array.isArray(h)?h:[h,void 0]).forEach((t,n)=>{if("object"==typeof t)for(let r=0;r0?-(P[0]/2):void 0;A&&(T.marginLeft=A,T.marginRight=A),[,T.rowGap]=P;let[L,z]=P,_=r.useMemo(()=>({gutter:[L,z],wrap:v}),[L,z,v]);return I(r.createElement(l.Z.Provider,{value:_},r.createElement("div",Object.assign({},b,{className:F,style:Object.assign(Object.assign({},T),m),ref:t}),g)))});t.Z=f},96776:function(e,t,n){"use strict";n.d(t,{VM:function(){return u},cG:function(){return d}});var r=n(352),o=n(80669),a=n(3104);let i=e=>{let{componentCls:t}=e;return{[t]:{position:"relative",maxWidth:"100%",minHeight:1}}},c=(e,t)=>{let{componentCls:n,gridColumns:r}=e,o={};for(let e=r;e>=0;e--)0===e?(o["".concat(n).concat(t,"-").concat(e)]={display:"none"},o["".concat(n,"-push-").concat(e)]={insetInlineStart:"auto"},o["".concat(n,"-pull-").concat(e)]={insetInlineEnd:"auto"},o["".concat(n).concat(t,"-push-").concat(e)]={insetInlineStart:"auto"},o["".concat(n).concat(t,"-pull-").concat(e)]={insetInlineEnd:"auto"},o["".concat(n).concat(t,"-offset-").concat(e)]={marginInlineStart:0},o["".concat(n).concat(t,"-order-").concat(e)]={order:0}):(o["".concat(n).concat(t,"-").concat(e)]=[{"--ant-display":"block",display:"block"},{display:"var(--ant-display)",flex:"0 0 ".concat(e/r*100,"%"),maxWidth:"".concat(e/r*100,"%")}],o["".concat(n).concat(t,"-push-").concat(e)]={insetInlineStart:"".concat(e/r*100,"%")},o["".concat(n).concat(t,"-pull-").concat(e)]={insetInlineEnd:"".concat(e/r*100,"%")},o["".concat(n).concat(t,"-offset-").concat(e)]={marginInlineStart:"".concat(e/r*100,"%")},o["".concat(n).concat(t,"-order-").concat(e)]={order:e});return o},l=(e,t)=>c(e,t),s=(e,t,n)=>({["@media (min-width: ".concat((0,r.bf)(t),")")]:Object.assign({},l(e,n))}),u=(0,o.I$)("Grid",e=>{let{componentCls:t}=e;return{[t]:{display:"flex",flexFlow:"row wrap",minWidth:0,"&::before, &::after":{display:"flex"},"&-no-wrap":{flexWrap:"nowrap"},"&-start":{justifyContent:"flex-start"},"&-center":{justifyContent:"center"},"&-end":{justifyContent:"flex-end"},"&-space-between":{justifyContent:"space-between"},"&-space-around":{justifyContent:"space-around"},"&-space-evenly":{justifyContent:"space-evenly"},"&-top":{alignItems:"flex-start"},"&-middle":{alignItems:"center"},"&-bottom":{alignItems:"flex-end"}}}},()=>({})),d=(0,o.I$)("Grid",e=>{let t=(0,a.TS)(e,{gridColumns:24}),n={"-sm":t.screenSMMin,"-md":t.screenMDMin,"-lg":t.screenLGMin,"-xl":t.screenXLMin,"-xxl":t.screenXXLMin};return[i(t),l(t,""),l(t,"-xs"),Object.keys(n).map(e=>s(t,n[e],e)).reduce((e,t)=>Object.assign(Object.assign({},e),t),{})]},()=>({}))},20577:function(e,t,n){"use strict";n.d(t,{Z:function(){return eg}});var r=n(2265),o=n(70464),a=n(1119),i={icon:{tag:"svg",attrs:{viewBox:"64 64 896 896",focusable:"false"},children:[{tag:"path",attrs:{d:"M890.5 755.3L537.9 269.2c-12.8-17.6-39-17.6-51.7 0L133.5 755.3A8 8 0 00140 768h75c5.1 0 9.9-2.5 12.9-6.6L512 369.8l284.1 391.6c3 4.1 7.8 6.6 12.9 6.6h75c6.5 0 10.3-7.4 6.5-12.7z"}}]},name:"up",theme:"outlined"},c=n(55015),l=r.forwardRef(function(e,t){return r.createElement(c.Z,(0,a.Z)({},e,{ref:t,icon:i}))}),s=n(36760),u=n.n(s),d=n(11993),f=n(41154),p=n(26365),m=n(6989),g=n(76405),h=n(25049);function v(){return"function"==typeof BigInt}function b(e){return!e&&0!==e&&!Number.isNaN(e)||!String(e).trim()}function y(e){var t=e.trim(),n=t.startsWith("-");n&&(t=t.slice(1)),(t=t.replace(/(\.\d*[^0])0*$/,"$1").replace(/\.0*$/,"").replace(/^0+/,"")).startsWith(".")&&(t="0".concat(t));var r=t||"0",o=r.split("."),a=o[0]||"0",i=o[1]||"0";"0"===a&&"0"===i&&(n=!1);var c=n?"-":"";return{negative:n,negativeStr:c,trimStr:r,integerStr:a,decimalStr:i,fullStr:"".concat(c).concat(r)}}function w(e){var t=String(e);return!Number.isNaN(Number(t))&&t.includes("e")}function x(e){var t=String(e);if(w(e)){var n=Number(t.slice(t.indexOf("e-")+2)),r=t.match(/\.(\d+)/);return null!=r&&r[1]&&(n+=r[1].length),n}return t.includes(".")&&S(t)?t.length-t.indexOf(".")-1:0}function E(e){var t=String(e);if(w(e)){if(e>Number.MAX_SAFE_INTEGER)return String(v()?BigInt(e).toString():Number.MAX_SAFE_INTEGER);if(e=this.add(e.negate().toString()).toNumber()}},{key:"toNumber",value:function(){return this.isNaN()?NaN:Number(this.toString())}},{key:"toString",value:function(){var e=!(arguments.length>0)||void 0===arguments[0]||arguments[0];return e?this.isInvalidate()?"":y("".concat(this.getMark()).concat(this.getIntegerStr(),".").concat(this.getDecimalStr())).fullStr:this.origin}}]),e}(),Z=function(){function e(t){if((0,g.Z)(this,e),(0,d.Z)(this,"origin",""),(0,d.Z)(this,"number",void 0),(0,d.Z)(this,"empty",void 0),b(t)){this.empty=!0;return}this.origin=String(t),this.number=Number(t)}return(0,h.Z)(e,[{key:"negate",value:function(){return new e(-this.toNumber())}},{key:"add",value:function(t){if(this.isInvalidate())return new e(t);var n=Number(t);if(Number.isNaN(n))return this;var r=this.number+n;if(r>Number.MAX_SAFE_INTEGER)return new e(Number.MAX_SAFE_INTEGER);if(rNumber.MAX_SAFE_INTEGER)return new e(Number.MAX_SAFE_INTEGER);if(r=this.add(e.negate().toString()).toNumber()}},{key:"toNumber",value:function(){return this.number}},{key:"toString",value:function(){var e=!(arguments.length>0)||void 0===arguments[0]||arguments[0];return e?this.isInvalidate()?"":E(this.number):this.origin}}]),e}();function O(e){return v()?new C(e):new Z(e)}function k(e,t,n){var r=arguments.length>3&&void 0!==arguments[3]&&arguments[3];if(""===e)return"";var o=y(e),a=o.negativeStr,i=o.integerStr,c=o.decimalStr,l="".concat(t).concat(c),s="".concat(a).concat(i);if(n>=0){var u=Number(c[n]);return u>=5&&!r?k(O(e).add("".concat(a,"0.").concat("0".repeat(n)).concat(10-u)).toString(),t,n,r):0===n?s:"".concat(s).concat(t).concat(c.padEnd(n,"0").slice(0,n))}return".0"===l?s:"".concat(s).concat(l)}var M=n(2027),j=n(27380),I=n(28791),R=n(32559),N=n(79267),P=function(){var e=(0,r.useState)(!1),t=(0,p.Z)(e,2),n=t[0],o=t[1];return(0,j.Z)(function(){o((0,N.Z)())},[]),n},F=n(53346);function T(e){var t=e.prefixCls,n=e.upNode,o=e.downNode,i=e.upDisabled,c=e.downDisabled,l=e.onStep,s=r.useRef(),f=r.useRef([]),p=r.useRef();p.current=l;var m=function(){clearTimeout(s.current)},g=function(e,t){e.preventDefault(),m(),p.current(t),s.current=setTimeout(function e(){p.current(t),s.current=setTimeout(e,200)},600)};if(r.useEffect(function(){return function(){m(),f.current.forEach(function(e){return F.Z.cancel(e)})}},[]),P())return null;var h="".concat(t,"-handler"),v=u()(h,"".concat(h,"-up"),(0,d.Z)({},"".concat(h,"-up-disabled"),i)),b=u()(h,"".concat(h,"-down"),(0,d.Z)({},"".concat(h,"-down-disabled"),c)),y=function(){return f.current.push((0,F.Z)(m))},w={unselectable:"on",role:"button",onMouseUp:y,onMouseLeave:y};return r.createElement("div",{className:"".concat(h,"-wrap")},r.createElement("span",(0,a.Z)({},w,{onMouseDown:function(e){g(e,!0)},"aria-label":"Increase Value","aria-disabled":i,className:v}),n||r.createElement("span",{unselectable:"on",className:"".concat(t,"-handler-up-inner")})),r.createElement("span",(0,a.Z)({},w,{onMouseDown:function(e){g(e,!1)},"aria-label":"Decrease Value","aria-disabled":c,className:b}),o||r.createElement("span",{unselectable:"on",className:"".concat(t,"-handler-down-inner")})))}function A(e){var t="number"==typeof e?E(e):y(e).fullStr;return t.includes(".")?y(t.replace(/(\d)\.(\d)/g,"$1$2.")).fullStr:e+"0"}var L=n(55041),z=function(){var e=(0,r.useRef)(0),t=function(){F.Z.cancel(e.current)};return(0,r.useEffect)(function(){return t},[]),function(n){t(),e.current=(0,F.Z)(function(){n()})}},_=["prefixCls","className","style","min","max","step","defaultValue","value","disabled","readOnly","upHandler","downHandler","keyboard","wheel","controls","classNames","stringMode","parser","formatter","precision","decimalSeparator","onChange","onInput","onPressEnter","onStep","changeOnBlur"],H=["disabled","style","prefixCls","value","prefix","suffix","addonBefore","addonAfter","className","classNames"],B=function(e,t){return e||t.isEmpty()?t.toString():t.toNumber()},D=function(e){var t=O(e);return t.isInvalidate()?null:t},W=r.forwardRef(function(e,t){var n,o,i,c=e.prefixCls,l=void 0===c?"rc-input-number":c,s=e.className,g=e.style,h=e.min,v=e.max,b=e.step,y=void 0===b?1:b,w=e.defaultValue,C=e.value,Z=e.disabled,M=e.readOnly,N=e.upHandler,P=e.downHandler,F=e.keyboard,L=e.wheel,H=e.controls,W=(e.classNames,e.stringMode),V=e.parser,q=e.formatter,G=e.precision,X=e.decimalSeparator,U=e.onChange,$=e.onInput,K=e.onPressEnter,Y=e.onStep,Q=e.changeOnBlur,J=void 0===Q||Q,ee=(0,m.Z)(e,_),et="".concat(l,"-input"),en=r.useRef(null),er=r.useState(!1),eo=(0,p.Z)(er,2),ea=eo[0],ei=eo[1],ec=r.useRef(!1),el=r.useRef(!1),es=r.useRef(!1),eu=r.useState(function(){return O(null!=C?C:w)}),ed=(0,p.Z)(eu,2),ef=ed[0],ep=ed[1],em=r.useCallback(function(e,t){return t?void 0:G>=0?G:Math.max(x(e),x(y))},[G,y]),eg=r.useCallback(function(e){var t=String(e);if(V)return V(t);var n=t;return X&&(n=n.replace(X,".")),n.replace(/[^\w.-]+/g,"")},[V,X]),eh=r.useRef(""),ev=r.useCallback(function(e,t){if(q)return q(e,{userTyping:t,input:String(eh.current)});var n="number"==typeof e?E(e):e;if(!t){var r=em(n,t);S(n)&&(X||r>=0)&&(n=k(n,X||".",r))}return n},[q,em,X]),eb=r.useState(function(){var e=null!=w?w:C;return ef.isInvalidate()&&["string","number"].includes((0,f.Z)(e))?Number.isNaN(e)?"":e:ev(ef.toString(),!1)}),ey=(0,p.Z)(eb,2),ew=ey[0],ex=ey[1];function eE(e,t){ex(ev(e.isInvalidate()?e.toString(!1):e.toString(!t),t))}eh.current=ew;var eS=r.useMemo(function(){return D(v)},[v,G]),eC=r.useMemo(function(){return D(h)},[h,G]),eZ=r.useMemo(function(){return!(!eS||!ef||ef.isInvalidate())&&eS.lessEquals(ef)},[eS,ef]),eO=r.useMemo(function(){return!(!eC||!ef||ef.isInvalidate())&&ef.lessEquals(eC)},[eC,ef]),ek=(n=en.current,o=(0,r.useRef)(null),[function(){try{var e=n.selectionStart,t=n.selectionEnd,r=n.value,a=r.substring(0,e),i=r.substring(t);o.current={start:e,end:t,value:r,beforeTxt:a,afterTxt:i}}catch(e){}},function(){if(n&&o.current&&ea)try{var e=n.value,t=o.current,r=t.beforeTxt,a=t.afterTxt,i=t.start,c=e.length;if(e.endsWith(a))c=e.length-o.current.afterTxt.length;else if(e.startsWith(r))c=r.length;else{var l=r[i-1],s=e.indexOf(l,i-1);-1!==s&&(c=s+1)}n.setSelectionRange(c,c)}catch(e){(0,R.ZP)(!1,"Something warning of cursor restore. Please fire issue about this: ".concat(e.message))}}]),eM=(0,p.Z)(ek,2),ej=eM[0],eI=eM[1],eR=function(e){return eS&&!e.lessEquals(eS)?eS:eC&&!eC.lessEquals(e)?eC:null},eN=function(e){return!eR(e)},eP=function(e,t){var n=e,r=eN(n)||n.isEmpty();if(n.isEmpty()||t||(n=eR(n)||n,r=!0),!M&&!Z&&r){var o,a=n.toString(),i=em(a,t);return i>=0&&!eN(n=O(k(a,".",i)))&&(n=O(k(a,".",i,!0))),n.equals(ef)||(o=n,void 0===C&&ep(o),null==U||U(n.isEmpty()?null:B(W,n)),void 0===C&&eE(n,t)),n}return ef},eF=z(),eT=function e(t){if(ej(),eh.current=t,ex(t),!el.current){var n=O(eg(t));n.isNaN()||eP(n,!0)}null==$||$(t),eF(function(){var n=t;V||(n=t.replace(/。/g,".")),n!==t&&e(n)})},eA=function(e){if((!e||!eZ)&&(e||!eO)){ec.current=!1;var t,n=O(es.current?A(y):y);e||(n=n.negate());var r=eP((ef||O(0)).add(n.toString()),!1);null==Y||Y(B(W,r),{offset:es.current?A(y):y,type:e?"up":"down"}),null===(t=en.current)||void 0===t||t.focus()}},eL=function(e){var t=O(eg(ew)),n=t;n=t.isNaN()?eP(ef,e):eP(t,e),void 0!==C?eE(ef,!1):n.isNaN()||eE(n,!1)};return r.useEffect(function(){var e=function(e){!1!==L&&(eA(e.deltaY<0),e.preventDefault())},t=en.current;if(t)return t.addEventListener("wheel",e),function(){return t.removeEventListener("wheel",e)}},[eA]),(0,j.o)(function(){ef.isInvalidate()||eE(ef,!1)},[G,q]),(0,j.o)(function(){var e=O(C);ep(e);var t=O(eg(ew));e.equals(t)&&ec.current&&!q||eE(e,ec.current)},[C]),(0,j.o)(function(){q&&eI()},[ew]),r.createElement("div",{className:u()(l,s,(i={},(0,d.Z)(i,"".concat(l,"-focused"),ea),(0,d.Z)(i,"".concat(l,"-disabled"),Z),(0,d.Z)(i,"".concat(l,"-readonly"),M),(0,d.Z)(i,"".concat(l,"-not-a-number"),ef.isNaN()),(0,d.Z)(i,"".concat(l,"-out-of-range"),!ef.isInvalidate()&&!eN(ef)),i)),style:g,onFocus:function(){ei(!0)},onBlur:function(){J&&eL(!1),ei(!1),ec.current=!1},onKeyDown:function(e){var t=e.key,n=e.shiftKey;ec.current=!0,es.current=n,"Enter"===t&&(el.current||(ec.current=!1),eL(!1),null==K||K(e)),!1!==F&&!el.current&&["Up","ArrowUp","Down","ArrowDown"].includes(t)&&(eA("Up"===t||"ArrowUp"===t),e.preventDefault())},onKeyUp:function(){ec.current=!1,es.current=!1},onCompositionStart:function(){el.current=!0},onCompositionEnd:function(){el.current=!1,eT(en.current.value)},onBeforeInput:function(){ec.current=!0}},(void 0===H||H)&&r.createElement(T,{prefixCls:l,upNode:N,downNode:P,upDisabled:eZ,downDisabled:eO,onStep:eA}),r.createElement("div",{className:"".concat(et,"-wrap")},r.createElement("input",(0,a.Z)({autoComplete:"off",role:"spinbutton","aria-valuemin":h,"aria-valuemax":v,"aria-valuenow":ef.isInvalidate()?null:ef.toString(),step:y},ee,{ref:(0,I.sQ)(en,t),className:et,value:ew,onChange:function(e){eT(e.target.value)},disabled:Z,readOnly:M}))))}),V=r.forwardRef(function(e,t){var n=e.disabled,o=e.style,i=e.prefixCls,c=e.value,l=e.prefix,s=e.suffix,u=e.addonBefore,d=e.addonAfter,f=e.className,p=e.classNames,g=(0,m.Z)(e,H),h=r.useRef(null);return r.createElement(M.Q,{className:f,triggerFocus:function(e){h.current&&(0,L.nH)(h.current,e)},prefixCls:i,value:c,disabled:n,style:o,prefix:l,suffix:s,addonAfter:d,addonBefore:u,classNames:p,components:{affixWrapper:"div",groupWrapper:"div",wrapper:"div",groupAddon:"div"}},r.createElement(W,(0,a.Z)({prefixCls:i,disabled:n,ref:(0,I.sQ)(h,t),className:null==p?void 0:p.input},g)))});V.displayName="InputNumber";var q=n(12757),G=n(71744),X=n(13959),U=n(86586),$=n(64024),K=n(33759),Y=n(39109),Q=n(56250),J=n(65658),ee=n(352),et=n(31282),en=n(37433),er=n(65265),eo=n(12918),ea=n(17691),ei=n(80669),ec=n(3104),el=n(36360);let es=(e,t)=>{let{componentCls:n,borderRadiusSM:r,borderRadiusLG:o}=e,a="lg"===t?o:r;return{["&-".concat(t)]:{["".concat(n,"-handler-wrap")]:{borderStartEndRadius:a,borderEndEndRadius:a},["".concat(n,"-handler-up")]:{borderStartEndRadius:a},["".concat(n,"-handler-down")]:{borderEndEndRadius:a}}}},eu=e=>{let{componentCls:t,lineWidth:n,lineType:r,borderRadius:o,fontSizeLG:a,controlHeightLG:i,controlHeightSM:c,colorError:l,paddingInlineSM:s,paddingBlockSM:u,paddingBlockLG:d,paddingInlineLG:f,colorTextDescription:p,motionDurationMid:m,handleHoverColor:g,paddingInline:h,paddingBlock:v,handleBg:b,handleActiveBg:y,colorTextDisabled:w,borderRadiusSM:x,borderRadiusLG:E,controlWidth:S,handleOpacity:C,handleBorderColor:Z,filledHandleBg:O,lineHeightLG:k,calc:M}=e;return[{[t]:Object.assign(Object.assign(Object.assign(Object.assign(Object.assign(Object.assign(Object.assign({},(0,eo.Wf)(e)),(0,et.ik)(e)),{display:"inline-block",width:S,margin:0,padding:0,borderRadius:o}),(0,er.qG)(e,{["".concat(t,"-handler-wrap")]:{background:b,["".concat(t,"-handler-down")]:{borderBlockStart:"".concat((0,ee.bf)(n)," ").concat(r," ").concat(Z)}}})),(0,er.H8)(e,{["".concat(t,"-handler-wrap")]:{background:O,["".concat(t,"-handler-down")]:{borderBlockStart:"".concat((0,ee.bf)(n)," ").concat(r," ").concat(Z)}},"&:focus-within":{["".concat(t,"-handler-wrap")]:{background:b}}})),(0,er.Mu)(e)),{"&-rtl":{direction:"rtl",["".concat(t,"-input")]:{direction:"rtl"}},"&-lg":{padding:0,fontSize:a,lineHeight:k,borderRadius:E,["input".concat(t,"-input")]:{height:M(i).sub(M(n).mul(2)).equal(),padding:"".concat((0,ee.bf)(d)," ").concat((0,ee.bf)(f))}},"&-sm":{padding:0,borderRadius:x,["input".concat(t,"-input")]:{height:M(c).sub(M(n).mul(2)).equal(),padding:"".concat((0,ee.bf)(u)," ").concat((0,ee.bf)(s))}},"&-out-of-range":{["".concat(t,"-input-wrap")]:{input:{color:l}}},"&-group":Object.assign(Object.assign(Object.assign({},(0,eo.Wf)(e)),(0,et.s7)(e)),{"&-wrapper":Object.assign(Object.assign(Object.assign({display:"inline-block",textAlign:"start",verticalAlign:"top",["".concat(t,"-affix-wrapper")]:{width:"100%"},"&-lg":{["".concat(t,"-group-addon")]:{borderRadius:E,fontSize:e.fontSizeLG}},"&-sm":{["".concat(t,"-group-addon")]:{borderRadius:x}}},(0,er.ir)(e)),(0,er.S5)(e)),{["&:not(".concat(t,"-compact-first-item):not(").concat(t,"-compact-last-item)").concat(t,"-compact-item")]:{["".concat(t,", ").concat(t,"-group-addon")]:{borderRadius:0}},["&:not(".concat(t,"-compact-last-item)").concat(t,"-compact-first-item")]:{["".concat(t,", ").concat(t,"-group-addon")]:{borderStartEndRadius:0,borderEndEndRadius:0}},["&:not(".concat(t,"-compact-first-item)").concat(t,"-compact-last-item")]:{["".concat(t,", ").concat(t,"-group-addon")]:{borderStartStartRadius:0,borderEndStartRadius:0}}})}),["&-disabled ".concat(t,"-input")]:{cursor:"not-allowed"},[t]:{"&-input":Object.assign(Object.assign(Object.assign(Object.assign({},(0,eo.Wf)(e)),{width:"100%",padding:"".concat((0,ee.bf)(v)," ").concat((0,ee.bf)(h)),textAlign:"start",backgroundColor:"transparent",border:0,borderRadius:o,outline:0,transition:"all ".concat(m," linear"),appearance:"textfield",fontSize:"inherit"}),(0,et.nz)(e.colorTextPlaceholder)),{'&[type="number"]::-webkit-inner-spin-button, &[type="number"]::-webkit-outer-spin-button':{margin:0,webkitAppearance:"none",appearance:"none"}})}})},{[t]:Object.assign(Object.assign(Object.assign({["&:hover ".concat(t,"-handler-wrap, &-focused ").concat(t,"-handler-wrap")]:{opacity:1},["".concat(t,"-handler-wrap")]:{position:"absolute",insetBlockStart:0,insetInlineEnd:0,width:e.handleWidth,height:"100%",borderStartStartRadius:0,borderStartEndRadius:o,borderEndEndRadius:o,borderEndStartRadius:0,opacity:C,display:"flex",flexDirection:"column",alignItems:"stretch",transition:"opacity ".concat(m," linear ").concat(m),["".concat(t,"-handler")]:{display:"flex",alignItems:"center",justifyContent:"center",flex:"auto",height:"40%",["\n ".concat(t,"-handler-up-inner,\n ").concat(t,"-handler-down-inner\n ")]:{marginInlineEnd:0,fontSize:e.handleFontSize}}},["".concat(t,"-handler")]:{height:"50%",overflow:"hidden",color:p,fontWeight:"bold",lineHeight:0,textAlign:"center",cursor:"pointer",borderInlineStart:"".concat((0,ee.bf)(n)," ").concat(r," ").concat(Z),transition:"all ".concat(m," linear"),"&:active":{background:y},"&:hover":{height:"60%",["\n ".concat(t,"-handler-up-inner,\n ").concat(t,"-handler-down-inner\n ")]:{color:g}},"&-up-inner, &-down-inner":Object.assign(Object.assign({},(0,eo.Ro)()),{color:p,transition:"all ".concat(m," linear"),userSelect:"none"})},["".concat(t,"-handler-up")]:{borderStartEndRadius:o},["".concat(t,"-handler-down")]:{borderEndEndRadius:o}},es(e,"lg")),es(e,"sm")),{"&-disabled, &-readonly":{["".concat(t,"-handler-wrap")]:{display:"none"},["".concat(t,"-input")]:{color:"inherit"}},["\n ".concat(t,"-handler-up-disabled,\n ").concat(t,"-handler-down-disabled\n ")]:{cursor:"not-allowed"},["\n ".concat(t,"-handler-up-disabled:hover &-handler-up-inner,\n ").concat(t,"-handler-down-disabled:hover &-handler-down-inner\n ")]:{color:w}})}]},ed=e=>{let{componentCls:t,paddingBlock:n,paddingInline:r,inputAffixPadding:o,controlWidth:a,borderRadiusLG:i,borderRadiusSM:c,paddingInlineLG:l,paddingInlineSM:s,paddingBlockLG:u,paddingBlockSM:d}=e;return{["".concat(t,"-affix-wrapper")]:Object.assign(Object.assign({["input".concat(t,"-input")]:{padding:"".concat((0,ee.bf)(n)," 0")}},(0,et.ik)(e)),{position:"relative",display:"inline-flex",width:a,padding:0,paddingInlineStart:r,"&-lg":{borderRadius:i,paddingInlineStart:l,["input".concat(t,"-input")]:{padding:"".concat((0,ee.bf)(u)," 0")}},"&-sm":{borderRadius:c,paddingInlineStart:s,["input".concat(t,"-input")]:{padding:"".concat((0,ee.bf)(d)," 0")}},["&:not(".concat(t,"-disabled):hover")]:{zIndex:1},"&-focused, &:focus":{zIndex:1},["&-disabled > ".concat(t,"-disabled")]:{background:"transparent"},["> div".concat(t)]:{width:"100%",border:"none",outline:"none",["&".concat(t,"-focused")]:{boxShadow:"none !important"}},"&::before":{display:"inline-block",width:0,visibility:"hidden",content:'"\\a0"'},["".concat(t,"-handler-wrap")]:{zIndex:2},[t]:{color:"inherit","&-prefix, &-suffix":{display:"flex",flex:"none",alignItems:"center",pointerEvents:"none"},"&-prefix":{marginInlineEnd:o},"&-suffix":{position:"absolute",insetBlockStart:0,insetInlineEnd:0,zIndex:1,height:"100%",marginInlineEnd:r,marginInlineStart:o}}})}};var ef=(0,ei.I$)("InputNumber",e=>{let t=(0,ec.TS)(e,(0,en.e)(e));return[eu(t),ed(t),(0,ea.c)(t)]},e=>{var t;let n=null!==(t=e.handleVisible)&&void 0!==t?t:"auto";return Object.assign(Object.assign({},(0,en.T)(e)),{controlWidth:90,handleWidth:e.controlHeightSM-2*e.lineWidth,handleFontSize:e.fontSize/2,handleVisible:n,handleActiveBg:e.colorFillAlter,handleBg:e.colorBgContainer,filledHandleBg:new el.C(e.colorFillSecondary).onBackground(e.colorBgContainer).toHexString(),handleHoverColor:e.colorPrimary,handleBorderColor:e.colorBorder,handleOpacity:!0===n?1:0})},{unitless:{handleOpacity:!0}}),ep=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let em=r.forwardRef((e,t)=>{let{getPrefixCls:n,direction:a}=r.useContext(G.E_),i=r.useRef(null);r.useImperativeHandle(t,()=>i.current);let{className:c,rootClassName:s,size:d,disabled:f,prefixCls:p,addonBefore:m,addonAfter:g,prefix:h,bordered:v,readOnly:b,status:y,controls:w,variant:x}=e,E=ep(e,["className","rootClassName","size","disabled","prefixCls","addonBefore","addonAfter","prefix","bordered","readOnly","status","controls","variant"]),S=n("input-number",p),C=(0,$.Z)(S),[Z,O,k]=ef(S,C),{compactSize:M,compactItemClassnames:j}=(0,J.ri)(S,a),I=r.createElement(l,{className:"".concat(S,"-handler-up-inner")}),R=r.createElement(o.Z,{className:"".concat(S,"-handler-down-inner")});"object"==typeof w&&(I=void 0===w.upIcon?I:r.createElement("span",{className:"".concat(S,"-handler-up-inner")},w.upIcon),R=void 0===w.downIcon?R:r.createElement("span",{className:"".concat(S,"-handler-down-inner")},w.downIcon));let{hasFeedback:N,status:P,isFormItemInput:F,feedbackIcon:T}=r.useContext(Y.aM),A=(0,q.F)(P,y),L=(0,K.Z)(e=>{var t;return null!==(t=null!=d?d:M)&&void 0!==t?t:e}),z=r.useContext(U.Z),[_,H]=(0,Q.Z)(x,v),B=N&&r.createElement(r.Fragment,null,T),D=u()({["".concat(S,"-lg")]:"large"===L,["".concat(S,"-sm")]:"small"===L,["".concat(S,"-rtl")]:"rtl"===a,["".concat(S,"-in-form-item")]:F},O),W="".concat(S,"-group");return Z(r.createElement(V,Object.assign({ref:i,disabled:null!=f?f:z,className:u()(k,C,c,s,j),upHandler:I,downHandler:R,prefixCls:S,readOnly:b,controls:"boolean"==typeof w?w:void 0,prefix:h,suffix:B,addonAfter:g&&r.createElement(J.BR,null,r.createElement(Y.Ux,{override:!0,status:!0},g)),addonBefore:m&&r.createElement(J.BR,null,r.createElement(Y.Ux,{override:!0,status:!0},m)),classNames:{input:D,variant:u()({["".concat(S,"-").concat(_)]:H},(0,q.Z)(S,A,N)),affixWrapper:u()({["".concat(S,"-affix-wrapper-sm")]:"small"===L,["".concat(S,"-affix-wrapper-lg")]:"large"===L,["".concat(S,"-affix-wrapper-rtl")]:"rtl"===a},O),wrapper:u()({["".concat(W,"-rtl")]:"rtl"===a},O),groupWrapper:u()({["".concat(S,"-group-wrapper-sm")]:"small"===L,["".concat(S,"-group-wrapper-lg")]:"large"===L,["".concat(S,"-group-wrapper-rtl")]:"rtl"===a,["".concat(S,"-group-wrapper-").concat(_)]:H},(0,q.Z)("".concat(S,"-group-wrapper"),A,N),O)}},E)))});em._InternalPanelDoNotUseOrYouWillBeFired=e=>r.createElement(X.ZP,{theme:{components:{InputNumber:{handleVisible:!0}}}},r.createElement(em,Object.assign({},e)));var eg=em},65863:function(e,t,n){"use strict";n.d(t,{Z:function(){return E},n:function(){return x}});var r=n(2265),o=n(36760),a=n.n(o),i=n(2027),c=n(28791),l=n(12757),s=n(71744),u=n(86586),d=n(33759),f=n(39109),p=n(65658),m=n(39164),g=n(31282),h=n(64024),v=n(56250),b=n(39725),y=e=>{let t;return"object"==typeof e&&(null==e?void 0:e.clearIcon)?t=e:e&&(t={clearIcon:r.createElement(b.Z,null)}),t},w=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};function x(e,t){if(!e)return;e.focus(t);let{cursor:n}=t||{};if(n){let t=e.value.length;switch(n){case"start":e.setSelectionRange(0,0);break;case"end":e.setSelectionRange(t,t);break;default:e.setSelectionRange(0,t)}}}var E=(0,r.forwardRef)((e,t)=>{var n;let{prefixCls:o,bordered:b=!0,status:x,size:E,disabled:S,onBlur:C,onFocus:Z,suffix:O,allowClear:k,addonAfter:M,addonBefore:j,className:I,style:R,styles:N,rootClassName:P,onChange:F,classNames:T,variant:A}=e,L=w(e,["prefixCls","bordered","status","size","disabled","onBlur","onFocus","suffix","allowClear","addonAfter","addonBefore","className","style","styles","rootClassName","onChange","classNames","variant"]),{getPrefixCls:z,direction:_,input:H}=r.useContext(s.E_),B=z("input",o),D=(0,r.useRef)(null),W=(0,h.Z)(B),[V,q,G]=(0,g.ZP)(B,W),{compactSize:X,compactItemClassnames:U}=(0,p.ri)(B,_),$=(0,d.Z)(e=>{var t;return null!==(t=null!=E?E:X)&&void 0!==t?t:e}),K=r.useContext(u.Z),{status:Y,hasFeedback:Q,feedbackIcon:J}=(0,r.useContext)(f.aM),ee=(0,l.F)(Y,x),et=!!(e.prefix||e.suffix||e.allowClear||e.showCount)||!!Q;(0,r.useRef)(et);let en=(0,m.Z)(D,!0),er=(Q||O)&&r.createElement(r.Fragment,null,O,Q&&J),eo=y(k),[ea,ei]=(0,v.Z)(A,b);return V(r.createElement(i.Z,Object.assign({ref:(0,c.sQ)(t,D),prefixCls:B,autoComplete:null==H?void 0:H.autoComplete},L,{disabled:null!=S?S:K,onBlur:e=>{en(),null==C||C(e)},onFocus:e=>{en(),null==Z||Z(e)},style:Object.assign(Object.assign({},null==H?void 0:H.style),R),styles:Object.assign(Object.assign({},null==H?void 0:H.styles),N),suffix:er,allowClear:eo,className:a()(I,P,G,W,U,null==H?void 0:H.className),onChange:e=>{en(),null==F||F(e)},addonAfter:M&&r.createElement(p.BR,null,r.createElement(f.Ux,{override:!0,status:!0},M)),addonBefore:j&&r.createElement(p.BR,null,r.createElement(f.Ux,{override:!0,status:!0},j)),classNames:Object.assign(Object.assign(Object.assign({},T),null==H?void 0:H.classNames),{input:a()({["".concat(B,"-sm")]:"small"===$,["".concat(B,"-lg")]:"large"===$,["".concat(B,"-rtl")]:"rtl"===_},null==T?void 0:T.input,null===(n=null==H?void 0:H.classNames)||void 0===n?void 0:n.input,q),variant:a()({["".concat(B,"-").concat(ea)]:ei},(0,l.Z)(B,ee)),affixWrapper:a()({["".concat(B,"-affix-wrapper-sm")]:"small"===$,["".concat(B,"-affix-wrapper-lg")]:"large"===$,["".concat(B,"-affix-wrapper-rtl")]:"rtl"===_},q),wrapper:a()({["".concat(B,"-group-rtl")]:"rtl"===_},q),groupWrapper:a()({["".concat(B,"-group-wrapper-sm")]:"small"===$,["".concat(B,"-group-wrapper-lg")]:"large"===$,["".concat(B,"-group-wrapper-rtl")]:"rtl"===_,["".concat(B,"-group-wrapper-").concat(ea)]:ei},(0,l.Z)("".concat(B,"-group-wrapper"),ee,Q),q)})})))})},90464:function(e,t,n){"use strict";n.d(t,{Z:function(){return z}});var r,o=n(2265),a=n(39725),i=n(36760),c=n.n(i),l=n(1119),s=n(11993),u=n(31686),d=n(83145),f=n(26365),p=n(6989),m=n(2027),g=n(96032),h=n(55041),v=n(50506),b=n(41154),y=n(31474),w=n(27380),x=n(53346),E=["letter-spacing","line-height","padding-top","padding-bottom","font-family","font-weight","font-size","font-variant","text-rendering","text-transform","width","text-indent","padding-left","padding-right","border-width","box-sizing","word-break","white-space"],S={},C=["prefixCls","onPressEnter","defaultValue","value","autoSize","onResize","className","style","disabled","onChange","onInternalAutoSize"],Z=o.forwardRef(function(e,t){var n=e.prefixCls,a=(e.onPressEnter,e.defaultValue),i=e.value,d=e.autoSize,m=e.onResize,g=e.className,h=e.style,Z=e.disabled,O=e.onChange,k=(e.onInternalAutoSize,(0,p.Z)(e,C)),M=(0,v.Z)(a,{value:i,postState:function(e){return null!=e?e:""}}),j=(0,f.Z)(M,2),I=j[0],R=j[1],N=o.useRef();o.useImperativeHandle(t,function(){return{textArea:N.current}});var P=o.useMemo(function(){return d&&"object"===(0,b.Z)(d)?[d.minRows,d.maxRows]:[]},[d]),F=(0,f.Z)(P,2),T=F[0],A=F[1],L=!!d,z=function(){try{if(document.activeElement===N.current){var e=N.current,t=e.selectionStart,n=e.selectionEnd,r=e.scrollTop;N.current.setSelectionRange(t,n),N.current.scrollTop=r}}catch(e){}},_=o.useState(2),H=(0,f.Z)(_,2),B=H[0],D=H[1],W=o.useState(),V=(0,f.Z)(W,2),q=V[0],G=V[1],X=function(){D(0)};(0,w.Z)(function(){L&&X()},[i,T,A,L]),(0,w.Z)(function(){if(0===B)D(1);else if(1===B){var e=function(e){var t,n=arguments.length>1&&void 0!==arguments[1]&&arguments[1],o=arguments.length>2&&void 0!==arguments[2]?arguments[2]:null,a=arguments.length>3&&void 0!==arguments[3]?arguments[3]:null;r||((r=document.createElement("textarea")).setAttribute("tab-index","-1"),r.setAttribute("aria-hidden","true"),document.body.appendChild(r)),e.getAttribute("wrap")?r.setAttribute("wrap",e.getAttribute("wrap")):r.removeAttribute("wrap");var i=function(e){var t=arguments.length>1&&void 0!==arguments[1]&&arguments[1],n=e.getAttribute("id")||e.getAttribute("data-reactid")||e.getAttribute("name");if(t&&S[n])return S[n];var r=window.getComputedStyle(e),o=r.getPropertyValue("box-sizing")||r.getPropertyValue("-moz-box-sizing")||r.getPropertyValue("-webkit-box-sizing"),a=parseFloat(r.getPropertyValue("padding-bottom"))+parseFloat(r.getPropertyValue("padding-top")),i=parseFloat(r.getPropertyValue("border-bottom-width"))+parseFloat(r.getPropertyValue("border-top-width")),c={sizingStyle:E.map(function(e){return"".concat(e,":").concat(r.getPropertyValue(e))}).join(";"),paddingSize:a,borderSize:i,boxSizing:o};return t&&n&&(S[n]=c),c}(e,n),c=i.paddingSize,l=i.borderSize,s=i.boxSizing,u=i.sizingStyle;r.setAttribute("style","".concat(u,";").concat("\n min-height:0 !important;\n max-height:none !important;\n height:0 !important;\n visibility:hidden !important;\n overflow:hidden !important;\n position:absolute !important;\n z-index:-1000 !important;\n top:0 !important;\n right:0 !important;\n pointer-events: none !important;\n")),r.value=e.value||e.placeholder||"";var d=void 0,f=void 0,p=r.scrollHeight;if("border-box"===s?p+=l:"content-box"===s&&(p-=c),null!==o||null!==a){r.value=" ";var m=r.scrollHeight-c;null!==o&&(d=m*o,"border-box"===s&&(d=d+c+l),p=Math.max(d,p)),null!==a&&(f=m*a,"border-box"===s&&(f=f+c+l),t=p>f?"":"hidden",p=Math.min(f,p))}var g={height:p,overflowY:t,resize:"none"};return d&&(g.minHeight=d),f&&(g.maxHeight=f),g}(N.current,!1,T,A);D(2),G(e)}else z()},[B]);var U=o.useRef(),$=function(){x.Z.cancel(U.current)};o.useEffect(function(){return $},[]);var K=(0,u.Z)((0,u.Z)({},h),L?q:null);return(0===B||1===B)&&(K.overflowY="hidden",K.overflowX="hidden"),o.createElement(y.Z,{onResize:function(e){2===B&&(null==m||m(e),d&&($(),U.current=(0,x.Z)(function(){X()})))},disabled:!(d||m)},o.createElement("textarea",(0,l.Z)({},k,{ref:N,style:K,className:c()(n,g,(0,s.Z)({},"".concat(n,"-disabled"),Z)),disabled:Z,value:I,onChange:function(e){R(e.target.value),null==O||O(e)}})))}),O=["defaultValue","value","onFocus","onBlur","onChange","allowClear","maxLength","onCompositionStart","onCompositionEnd","suffix","prefixCls","showCount","count","className","style","disabled","hidden","classNames","styles","onResize"],k=o.forwardRef(function(e,t){var n,r,a,i=e.defaultValue,b=e.value,y=e.onFocus,w=e.onBlur,x=e.onChange,E=e.allowClear,S=e.maxLength,C=e.onCompositionStart,k=e.onCompositionEnd,M=e.suffix,j=e.prefixCls,I=void 0===j?"rc-textarea":j,R=e.showCount,N=e.count,P=e.className,F=e.style,T=e.disabled,A=e.hidden,L=e.classNames,z=e.styles,_=e.onResize,H=(0,p.Z)(e,O),B=(0,v.Z)(i,{value:b,defaultValue:i}),D=(0,f.Z)(B,2),W=D[0],V=D[1],q=null==W?"":String(W),G=o.useState(!1),X=(0,f.Z)(G,2),U=X[0],$=X[1],K=o.useRef(!1),Y=o.useState(null),Q=(0,f.Z)(Y,2),J=Q[0],ee=Q[1],et=(0,o.useRef)(null),en=function(){var e;return null===(e=et.current)||void 0===e?void 0:e.textArea},er=function(){en().focus()};(0,o.useImperativeHandle)(t,function(){return{resizableTextArea:et.current,focus:er,blur:function(){en().blur()}}}),(0,o.useEffect)(function(){$(function(e){return!T&&e})},[T]);var eo=o.useState(null),ea=(0,f.Z)(eo,2),ei=ea[0],ec=ea[1];o.useEffect(function(){if(ei){var e;(e=en()).setSelectionRange.apply(e,(0,d.Z)(ei))}},[ei]);var el=(0,g.Z)(N,R),es=null!==(n=el.max)&&void 0!==n?n:S,eu=Number(es)>0,ed=el.strategy(q),ef=!!es&&ed>es,ep=function(e,t){var n=t;!K.current&&el.exceedFormatter&&el.max&&el.strategy(t)>el.max&&(n=el.exceedFormatter(t,{max:el.max}),t!==n&&ec([en().selectionStart||0,en().selectionEnd||0])),V(n),(0,h.rJ)(e.currentTarget,e,x,n)},em=M;el.show&&(a=el.showFormatter?el.showFormatter({value:q,count:ed,maxLength:es}):"".concat(ed).concat(eu?" / ".concat(es):""),em=o.createElement(o.Fragment,null,em,o.createElement("span",{className:c()("".concat(I,"-data-count"),null==L?void 0:L.count),style:null==z?void 0:z.count},a)));var eg=!H.autoSize&&!R&&!E;return o.createElement(m.Q,{value:q,allowClear:E,handleReset:function(e){V(""),er(),(0,h.rJ)(en(),e,x)},suffix:em,prefixCls:I,classNames:(0,u.Z)((0,u.Z)({},L),{},{affixWrapper:c()(null==L?void 0:L.affixWrapper,(r={},(0,s.Z)(r,"".concat(I,"-show-count"),R),(0,s.Z)(r,"".concat(I,"-textarea-allow-clear"),E),r))}),disabled:T,focused:U,className:c()(P,ef&&"".concat(I,"-out-of-range")),style:(0,u.Z)((0,u.Z)({},F),J&&!eg?{height:"auto"}:{}),dataAttrs:{affixWrapper:{"data-count":"string"==typeof a?a:void 0}},hidden:A},o.createElement(Z,(0,l.Z)({},H,{maxLength:S,onKeyDown:function(e){var t=H.onPressEnter,n=H.onKeyDown;"Enter"===e.key&&t&&t(e),null==n||n(e)},onChange:function(e){ep(e,e.target.value)},onFocus:function(e){$(!0),null==y||y(e)},onBlur:function(e){$(!1),null==w||w(e)},onCompositionStart:function(e){K.current=!0,null==C||C(e)},onCompositionEnd:function(e){K.current=!1,ep(e,e.currentTarget.value),null==k||k(e)},className:c()(null==L?void 0:L.textarea),style:(0,u.Z)((0,u.Z)({},null==z?void 0:z.textarea),{},{resize:null==F?void 0:F.resize}),disabled:T,prefixCls:I,onResize:function(e){var t;null==_||_(e),null!==(t=en())&&void 0!==t&&t.style.height&&ee(!0)},ref:et})))}),M=n(12757),j=n(71744),I=n(86586),R=n(33759),N=n(39109),P=n(65863),F=n(31282),T=n(64024),A=n(56250),L=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n},z=(0,o.forwardRef)((e,t)=>{var n;let r;let{prefixCls:i,bordered:l=!0,size:s,disabled:u,status:d,allowClear:f,classNames:p,rootClassName:m,className:g,variant:h}=e,v=L(e,["prefixCls","bordered","size","disabled","status","allowClear","classNames","rootClassName","className","variant"]),{getPrefixCls:b,direction:y}=o.useContext(j.E_),w=(0,R.Z)(s),x=o.useContext(I.Z),{status:E,hasFeedback:S,feedbackIcon:C}=o.useContext(N.aM),Z=(0,M.F)(E,d),O=o.useRef(null);o.useImperativeHandle(t,()=>{var e;return{resizableTextArea:null===(e=O.current)||void 0===e?void 0:e.resizableTextArea,focus:e=>{var t,n;(0,P.n)(null===(n=null===(t=O.current)||void 0===t?void 0:t.resizableTextArea)||void 0===n?void 0:n.textArea,e)},blur:()=>{var e;return null===(e=O.current)||void 0===e?void 0:e.blur()}}});let z=b("input",i);"object"==typeof f&&(null==f?void 0:f.clearIcon)?r=f:f&&(r={clearIcon:o.createElement(a.Z,null)});let _=(0,T.Z)(z),[H,B,D]=(0,F.ZP)(z,_),[W,V]=(0,A.Z)(h,l);return H(o.createElement(k,Object.assign({},v,{disabled:null!=u?u:x,allowClear:r,className:c()(D,_,g,m),classNames:Object.assign(Object.assign({},p),{textarea:c()({["".concat(z,"-sm")]:"small"===w,["".concat(z,"-lg")]:"large"===w},B,null==p?void 0:p.textarea),variant:c()({["".concat(z,"-").concat(W)]:V},(0,M.Z)(z,Z)),affixWrapper:c()("".concat(z,"-textarea-affix-wrapper"),{["".concat(z,"-affix-wrapper-rtl")]:"rtl"===y,["".concat(z,"-affix-wrapper-sm")]:"small"===w,["".concat(z,"-affix-wrapper-lg")]:"large"===w,["".concat(z,"-textarea-show-count")]:e.showCount||(null===(n=e.count)||void 0===n?void 0:n.show)},B)}),prefixCls:z,suffix:S&&o.createElement("span",{className:"".concat(z,"-textarea-suffix")},C),ref:O})))})},39164:function(e,t,n){"use strict";n.d(t,{Z:function(){return o}});var r=n(2265);function o(e,t){let n=(0,r.useRef)([]),o=()=>{n.current.push(setTimeout(()=>{var t,n,r,o;(null===(t=e.current)||void 0===t?void 0:t.input)&&(null===(n=e.current)||void 0===n?void 0:n.input.getAttribute("type"))==="password"&&(null===(r=e.current)||void 0===r?void 0:r.input.hasAttribute("value"))&&(null===(o=e.current)||void 0===o||o.input.removeAttribute("value"))}))};return(0,r.useEffect)(()=>(t&&o(),()=>n.current.forEach(e=>{e&&clearTimeout(e)})),[]),o}},64482:function(e,t,n){"use strict";n.d(t,{default:function(){return M}});var r=n(2265),o=n(36760),a=n.n(o),i=n(71744),c=n(39109),l=n(31282),s=n(65863),u=n(97416),d=n(6520),f=n(18694),p=n(28791),m=n(39164),g=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let h=e=>e?r.createElement(d.Z,null):r.createElement(u.Z,null),v={click:"onClick",hover:"onMouseOver"},b=r.forwardRef((e,t)=>{let{visibilityToggle:n=!0}=e,o="object"==typeof n&&void 0!==n.visible,[c,l]=(0,r.useState)(()=>!!o&&n.visible),u=(0,r.useRef)(null);r.useEffect(()=>{o&&l(n.visible)},[o,n]);let d=(0,m.Z)(u),b=()=>{let{disabled:t}=e;t||(c&&d(),l(e=>{var t;let r=!e;return"object"==typeof n&&(null===(t=n.onVisibleChange)||void 0===t||t.call(n,r)),r}))},{className:y,prefixCls:w,inputPrefixCls:x,size:E}=e,S=g(e,["className","prefixCls","inputPrefixCls","size"]),{getPrefixCls:C}=r.useContext(i.E_),Z=C("input",x),O=C("input-password",w),k=n&&(t=>{let{action:n="click",iconRender:o=h}=e,a=v[n]||"",i=o(c);return r.cloneElement(r.isValidElement(i)?i:r.createElement("span",null,i),{[a]:b,className:"".concat(t,"-icon"),key:"passwordIcon",onMouseDown:e=>{e.preventDefault()},onMouseUp:e=>{e.preventDefault()}})})(O),M=a()(O,y,{["".concat(O,"-").concat(E)]:!!E}),j=Object.assign(Object.assign({},(0,f.Z)(S,["suffix","iconRender","visibilityToggle"])),{type:c?"text":"password",className:M,prefixCls:Z,suffix:k});return E&&(j.size=E),r.createElement(s.Z,Object.assign({ref:(0,p.sQ)(t,u)},j))});var y=n(29436),w=n(19722),x=n(73002),E=n(33759),S=n(65658),C=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let Z=r.forwardRef((e,t)=>{let n;let{prefixCls:o,inputPrefixCls:c,className:l,size:u,suffix:d,enterButton:f=!1,addonAfter:m,loading:g,disabled:h,onSearch:v,onChange:b,onCompositionStart:Z,onCompositionEnd:O}=e,k=C(e,["prefixCls","inputPrefixCls","className","size","suffix","enterButton","addonAfter","loading","disabled","onSearch","onChange","onCompositionStart","onCompositionEnd"]),{getPrefixCls:M,direction:j}=r.useContext(i.E_),I=r.useRef(!1),R=M("input-search",o),N=M("input",c),{compactSize:P}=(0,S.ri)(R,j),F=(0,E.Z)(e=>{var t;return null!==(t=null!=u?u:P)&&void 0!==t?t:e}),T=r.useRef(null),A=e=>{var t;document.activeElement===(null===(t=T.current)||void 0===t?void 0:t.input)&&e.preventDefault()},L=e=>{var t,n;v&&v(null===(n=null===(t=T.current)||void 0===t?void 0:t.input)||void 0===n?void 0:n.value,e,{source:"input"})},z="boolean"==typeof f?r.createElement(y.Z,null):null,_="".concat(R,"-button"),H=f||{},B=H.type&&!0===H.type.__ANT_BUTTON;n=B||"button"===H.type?(0,w.Tm)(H,Object.assign({onMouseDown:A,onClick:e=>{var t,n;null===(n=null===(t=null==H?void 0:H.props)||void 0===t?void 0:t.onClick)||void 0===n||n.call(t,e),L(e)},key:"enterButton"},B?{className:_,size:F}:{})):r.createElement(x.ZP,{className:_,type:f?"primary":void 0,size:F,disabled:h,key:"enterButton",onMouseDown:A,onClick:L,loading:g,icon:z},f),m&&(n=[n,(0,w.Tm)(m,{key:"addonAfter"})]);let D=a()(R,{["".concat(R,"-rtl")]:"rtl"===j,["".concat(R,"-").concat(F)]:!!F,["".concat(R,"-with-button")]:!!f},l);return r.createElement(s.Z,Object.assign({ref:(0,p.sQ)(T,t),onPressEnter:e=>{I.current||g||L(e)}},k,{size:F,onCompositionStart:e=>{I.current=!0,null==Z||Z(e)},onCompositionEnd:e=>{I.current=!1,null==O||O(e)},prefixCls:N,addonAfter:n,suffix:d,onChange:e=>{e&&e.target&&"click"===e.type&&v&&v(e.target.value,e,{source:"clear"}),b&&b(e)},className:D,disabled:h}))});var O=n(90464);let k=s.Z;k.Group=e=>{let{getPrefixCls:t,direction:n}=(0,r.useContext)(i.E_),{prefixCls:o,className:s}=e,u=t("input-group",o),d=t("input"),[f,p]=(0,l.ZP)(d),m=a()(u,{["".concat(u,"-lg")]:"large"===e.size,["".concat(u,"-sm")]:"small"===e.size,["".concat(u,"-compact")]:e.compact,["".concat(u,"-rtl")]:"rtl"===n},p,s),g=(0,r.useContext)(c.aM),h=(0,r.useMemo)(()=>Object.assign(Object.assign({},g),{isFormItemInput:!1}),[g]);return f(r.createElement("span",{className:m,style:e.style,onMouseEnter:e.onMouseEnter,onMouseLeave:e.onMouseLeave,onFocus:e.onFocus,onBlur:e.onBlur},r.createElement(c.aM.Provider,{value:h},e.children)))},k.Search=Z,k.TextArea=O.Z,k.Password=b;var M=k},31282:function(e,t,n){"use strict";n.d(t,{ik:function(){return p},nz:function(){return u},s7:function(){return m},x0:function(){return f}});var r=n(352),o=n(12918),a=n(17691),i=n(80669),c=n(3104),l=n(37433),s=n(65265);let u=e=>({"&::-moz-placeholder":{opacity:1},"&::placeholder":{color:e,userSelect:"none"},"&:placeholder-shown":{textOverflow:"ellipsis"}}),d=e=>{let{paddingBlockLG:t,lineHeightLG:n,borderRadiusLG:o,paddingInlineLG:a}=e;return{padding:"".concat((0,r.bf)(t)," ").concat((0,r.bf)(a)),fontSize:e.inputFontSizeLG,lineHeight:n,borderRadius:o}},f=e=>({padding:"".concat((0,r.bf)(e.paddingBlockSM)," ").concat((0,r.bf)(e.paddingInlineSM)),fontSize:e.inputFontSizeSM,borderRadius:e.borderRadiusSM}),p=e=>Object.assign(Object.assign({position:"relative",display:"inline-block",width:"100%",minWidth:0,padding:"".concat((0,r.bf)(e.paddingBlock)," ").concat((0,r.bf)(e.paddingInline)),color:e.colorText,fontSize:e.inputFontSize,lineHeight:e.lineHeight,borderRadius:e.borderRadius,transition:"all ".concat(e.motionDurationMid)},u(e.colorTextPlaceholder)),{"textarea&":{maxWidth:"100%",height:"auto",minHeight:e.controlHeight,lineHeight:e.lineHeight,verticalAlign:"bottom",transition:"all ".concat(e.motionDurationSlow,", height 0s"),resize:"vertical"},"&-lg":Object.assign({},d(e)),"&-sm":Object.assign({},f(e)),"&-rtl":{direction:"rtl"},"&-textarea-rtl":{direction:"rtl"}}),m=e=>{let{componentCls:t,antCls:n}=e;return{position:"relative",display:"table",width:"100%",borderCollapse:"separate",borderSpacing:0,"&[class*='col-']":{paddingInlineEnd:e.paddingXS,"&:last-child":{paddingInlineEnd:0}},["&-lg ".concat(t,", &-lg > ").concat(t,"-group-addon")]:Object.assign({},d(e)),["&-sm ".concat(t,", &-sm > ").concat(t,"-group-addon")]:Object.assign({},f(e)),["&-lg ".concat(n,"-select-single ").concat(n,"-select-selector")]:{height:e.controlHeightLG},["&-sm ".concat(n,"-select-single ").concat(n,"-select-selector")]:{height:e.controlHeightSM},["> ".concat(t)]:{display:"table-cell","&:not(:first-child):not(:last-child)":{borderRadius:0}},["".concat(t,"-group")]:{"&-addon, &-wrap":{display:"table-cell",width:1,whiteSpace:"nowrap",verticalAlign:"middle","&:not(:first-child):not(:last-child)":{borderRadius:0}},"&-wrap > *":{display:"block !important"},"&-addon":{position:"relative",padding:"0 ".concat((0,r.bf)(e.paddingInline)),color:e.colorText,fontWeight:"normal",fontSize:e.inputFontSize,textAlign:"center",borderRadius:e.borderRadius,transition:"all ".concat(e.motionDurationSlow),lineHeight:1,["".concat(n,"-select")]:{margin:"".concat((0,r.bf)(e.calc(e.paddingBlock).add(1).mul(-1).equal())," ").concat((0,r.bf)(e.calc(e.paddingInline).mul(-1).equal())),["&".concat(n,"-select-single:not(").concat(n,"-select-customize-input):not(").concat(n,"-pagination-size-changer)")]:{["".concat(n,"-select-selector")]:{backgroundColor:"inherit",border:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," transparent"),boxShadow:"none"}},"&-open, &-focused":{["".concat(n,"-select-selector")]:{color:e.colorPrimary}}},["".concat(n,"-cascader-picker")]:{margin:"-9px ".concat((0,r.bf)(e.calc(e.paddingInline).mul(-1).equal())),backgroundColor:"transparent",["".concat(n,"-cascader-input")]:{textAlign:"start",border:0,boxShadow:"none"}}}},["".concat(t)]:{width:"100%",marginBottom:0,textAlign:"inherit","&:focus":{zIndex:1,borderInlineEndWidth:1},"&:hover":{zIndex:1,borderInlineEndWidth:1,["".concat(t,"-search-with-button &")]:{zIndex:0}}},["> ".concat(t,":first-child, ").concat(t,"-group-addon:first-child")]:{borderStartEndRadius:0,borderEndEndRadius:0,["".concat(n,"-select ").concat(n,"-select-selector")]:{borderStartEndRadius:0,borderEndEndRadius:0}},["> ".concat(t,"-affix-wrapper")]:{["&:not(:first-child) ".concat(t)]:{borderStartStartRadius:0,borderEndStartRadius:0},["&:not(:last-child) ".concat(t)]:{borderStartEndRadius:0,borderEndEndRadius:0}},["> ".concat(t,":last-child, ").concat(t,"-group-addon:last-child")]:{borderStartStartRadius:0,borderEndStartRadius:0,["".concat(n,"-select ").concat(n,"-select-selector")]:{borderStartStartRadius:0,borderEndStartRadius:0}},["".concat(t,"-affix-wrapper")]:{"&:not(:last-child)":{borderStartEndRadius:0,borderEndEndRadius:0,["".concat(t,"-search &")]:{borderStartStartRadius:e.borderRadius,borderEndStartRadius:e.borderRadius}},["&:not(:first-child), ".concat(t,"-search &:not(:first-child)")]:{borderStartStartRadius:0,borderEndStartRadius:0}},["&".concat(t,"-group-compact")]:Object.assign(Object.assign({display:"block"},(0,o.dF)()),{["".concat(t,"-group-addon, ").concat(t,"-group-wrap, > ").concat(t)]:{"&:not(:first-child):not(:last-child)":{borderInlineEndWidth:e.lineWidth,"&:hover":{zIndex:1},"&:focus":{zIndex:1}}},"& > *":{display:"inline-block",float:"none",verticalAlign:"top",borderRadius:0},["\n & > ".concat(t,"-affix-wrapper,\n & > ").concat(t,"-number-affix-wrapper,\n & > ").concat(n,"-picker-range\n ")]:{display:"inline-flex"},"& > *:not(:last-child)":{marginInlineEnd:e.calc(e.lineWidth).mul(-1).equal(),borderInlineEndWidth:e.lineWidth},["".concat(t)]:{float:"none"},["& > ".concat(n,"-select > ").concat(n,"-select-selector,\n & > ").concat(n,"-select-auto-complete ").concat(t,",\n & > ").concat(n,"-cascader-picker ").concat(t,",\n & > ").concat(t,"-group-wrapper ").concat(t)]:{borderInlineEndWidth:e.lineWidth,borderRadius:0,"&:hover":{zIndex:1},"&:focus":{zIndex:1}},["& > ".concat(n,"-select-focused")]:{zIndex:1},["& > ".concat(n,"-select > ").concat(n,"-select-arrow")]:{zIndex:1},["& > *:first-child,\n & > ".concat(n,"-select:first-child > ").concat(n,"-select-selector,\n & > ").concat(n,"-select-auto-complete:first-child ").concat(t,",\n & > ").concat(n,"-cascader-picker:first-child ").concat(t)]:{borderStartStartRadius:e.borderRadius,borderEndStartRadius:e.borderRadius},["& > *:last-child,\n & > ".concat(n,"-select:last-child > ").concat(n,"-select-selector,\n & > ").concat(n,"-cascader-picker:last-child ").concat(t,",\n & > ").concat(n,"-cascader-picker-focused:last-child ").concat(t)]:{borderInlineEndWidth:e.lineWidth,borderStartEndRadius:e.borderRadius,borderEndEndRadius:e.borderRadius},["& > ".concat(n,"-select-auto-complete ").concat(t)]:{verticalAlign:"top"},["".concat(t,"-group-wrapper + ").concat(t,"-group-wrapper")]:{marginInlineStart:e.calc(e.lineWidth).mul(-1).equal(),["".concat(t,"-affix-wrapper")]:{borderRadius:0}},["".concat(t,"-group-wrapper:not(:last-child)")]:{["&".concat(t,"-search > ").concat(t,"-group")]:{["& > ".concat(t,"-group-addon > ").concat(t,"-search-button")]:{borderRadius:0},["& > ".concat(t)]:{borderStartStartRadius:e.borderRadius,borderStartEndRadius:0,borderEndEndRadius:0,borderEndStartRadius:e.borderRadius}}}})}},g=e=>{let{componentCls:t,controlHeightSM:n,lineWidth:r,calc:a}=e,i=a(n).sub(a(r).mul(2)).sub(16).div(2).equal();return{[t]:Object.assign(Object.assign(Object.assign(Object.assign(Object.assign(Object.assign({},(0,o.Wf)(e)),p(e)),(0,s.qG)(e)),(0,s.H8)(e)),(0,s.Mu)(e)),{'&[type="color"]':{height:e.controlHeight,["&".concat(t,"-lg")]:{height:e.controlHeightLG},["&".concat(t,"-sm")]:{height:n,paddingTop:i,paddingBottom:i}},'&[type="search"]::-webkit-search-cancel-button, &[type="search"]::-webkit-search-decoration':{"-webkit-appearance":"none"}})}},h=e=>{let{componentCls:t}=e;return{["".concat(t,"-clear-icon")]:{margin:0,color:e.colorTextQuaternary,fontSize:e.fontSizeIcon,verticalAlign:-1,cursor:"pointer",transition:"color ".concat(e.motionDurationSlow),"&:hover":{color:e.colorTextTertiary},"&:active":{color:e.colorText},"&-hidden":{visibility:"hidden"},"&-has-suffix":{margin:"0 ".concat((0,r.bf)(e.inputAffixPadding))}}}},v=e=>{let{componentCls:t,inputAffixPadding:n,colorTextDescription:r,motionDurationSlow:o,colorIcon:a,colorIconHover:i,iconCls:c}=e;return{["".concat(t,"-affix-wrapper")]:Object.assign(Object.assign(Object.assign(Object.assign({},p(e)),{display:"inline-flex",["&:not(".concat(t,"-disabled):hover")]:{zIndex:1,["".concat(t,"-search-with-button &")]:{zIndex:0}},"&-focused, &:focus":{zIndex:1},["> input".concat(t)]:{padding:0,fontSize:"inherit",border:"none",borderRadius:0,outline:"none",background:"transparent",color:"inherit","&::-ms-reveal":{display:"none"},"&:focus":{boxShadow:"none !important"}},"&::before":{display:"inline-block",width:0,visibility:"hidden",content:'"\\a0"'},["".concat(t)]:{"&-prefix, &-suffix":{display:"flex",flex:"none",alignItems:"center","> *:not(:last-child)":{marginInlineEnd:e.paddingXS}},"&-show-count-suffix":{color:r},"&-show-count-has-suffix":{marginInlineEnd:e.paddingXXS},"&-prefix":{marginInlineEnd:n},"&-suffix":{marginInlineStart:n}}}),h(e)),{["".concat(c).concat(t,"-password-icon")]:{color:a,cursor:"pointer",transition:"all ".concat(o),"&:hover":{color:i}}})}},b=e=>{let{componentCls:t,borderRadiusLG:n,borderRadiusSM:r}=e;return{["".concat(t,"-group")]:Object.assign(Object.assign(Object.assign({},(0,o.Wf)(e)),m(e)),{"&-rtl":{direction:"rtl"},"&-wrapper":Object.assign(Object.assign(Object.assign({display:"inline-block",width:"100%",textAlign:"start",verticalAlign:"top","&-rtl":{direction:"rtl"},"&-lg":{["".concat(t,"-group-addon")]:{borderRadius:n,fontSize:e.inputFontSizeLG}},"&-sm":{["".concat(t,"-group-addon")]:{borderRadius:r}}},(0,s.ir)(e)),(0,s.S5)(e)),{["&:not(".concat(t,"-compact-first-item):not(").concat(t,"-compact-last-item)").concat(t,"-compact-item")]:{["".concat(t,", ").concat(t,"-group-addon")]:{borderRadius:0}},["&:not(".concat(t,"-compact-last-item)").concat(t,"-compact-first-item")]:{["".concat(t,", ").concat(t,"-group-addon")]:{borderStartEndRadius:0,borderEndEndRadius:0}},["&:not(".concat(t,"-compact-first-item)").concat(t,"-compact-last-item")]:{["".concat(t,", ").concat(t,"-group-addon")]:{borderStartStartRadius:0,borderEndStartRadius:0}}})})}},y=e=>{let{componentCls:t,antCls:n}=e,r="".concat(t,"-search");return{[r]:{["".concat(t)]:{"&:hover, &:focus":{borderColor:e.colorPrimaryHover,["+ ".concat(t,"-group-addon ").concat(r,"-button:not(").concat(n,"-btn-primary)")]:{borderInlineStartColor:e.colorPrimaryHover}}},["".concat(t,"-affix-wrapper")]:{borderRadius:0},["".concat(t,"-lg")]:{lineHeight:e.calc(e.lineHeightLG).sub(2e-4).equal({unit:!1})},["> ".concat(t,"-group")]:{["> ".concat(t,"-group-addon:last-child")]:{insetInlineStart:-1,padding:0,border:0,["".concat(r,"-button")]:{paddingTop:0,paddingBottom:0,borderStartStartRadius:0,borderStartEndRadius:e.borderRadius,borderEndEndRadius:e.borderRadius,borderEndStartRadius:0,boxShadow:"none"},["".concat(r,"-button:not(").concat(n,"-btn-primary)")]:{color:e.colorTextDescription,"&:hover":{color:e.colorPrimaryHover},"&:active":{color:e.colorPrimaryActive},["&".concat(n,"-btn-loading::before")]:{insetInlineStart:0,insetInlineEnd:0,insetBlockStart:0,insetBlockEnd:0}}}},["".concat(r,"-button")]:{height:e.controlHeight,"&:hover, &:focus":{zIndex:1}},["&-large ".concat(r,"-button")]:{height:e.controlHeightLG},["&-small ".concat(r,"-button")]:{height:e.controlHeightSM},"&-rtl":{direction:"rtl"},["&".concat(t,"-compact-item")]:{["&:not(".concat(t,"-compact-last-item)")]:{["".concat(t,"-group-addon")]:{["".concat(t,"-search-button")]:{marginInlineEnd:e.calc(e.lineWidth).mul(-1).equal(),borderRadius:0}}},["&:not(".concat(t,"-compact-first-item)")]:{["".concat(t,",").concat(t,"-affix-wrapper")]:{borderRadius:0}},["> ".concat(t,"-group-addon ").concat(t,"-search-button,\n > ").concat(t,",\n ").concat(t,"-affix-wrapper")]:{"&:hover,&:focus,&:active":{zIndex:2}},["> ".concat(t,"-affix-wrapper-focused")]:{zIndex:2}}}}},w=e=>{let{componentCls:t,paddingLG:n}=e,r="".concat(t,"-textarea");return{[r]:{position:"relative","&-show-count":{["> ".concat(t)]:{height:"100%"},["".concat(t,"-data-count")]:{position:"absolute",bottom:e.calc(e.fontSize).mul(e.lineHeight).mul(-1).equal(),insetInlineEnd:0,color:e.colorTextDescription,whiteSpace:"nowrap",pointerEvents:"none"}},"&-allow-clear":{["> ".concat(t)]:{paddingInlineEnd:n}},["&-affix-wrapper".concat(r,"-has-feedback")]:{["".concat(t)]:{paddingInlineEnd:n}},["&-affix-wrapper".concat(t,"-affix-wrapper")]:{padding:0,["> textarea".concat(t)]:{fontSize:"inherit",border:"none",outline:"none",background:"transparent","&:focus":{boxShadow:"none !important"}},["".concat(t,"-suffix")]:{margin:0,"> *:not(:last-child)":{marginInline:0},["".concat(t,"-clear-icon")]:{position:"absolute",insetInlineEnd:e.paddingXS,insetBlockStart:e.paddingXS},["".concat(r,"-suffix")]:{position:"absolute",top:0,insetInlineEnd:e.paddingInline,bottom:0,zIndex:1,display:"inline-flex",alignItems:"center",margin:"auto",pointerEvents:"none"}}}}}},x=e=>{let{componentCls:t}=e;return{["".concat(t,"-out-of-range")]:{["&, & input, & textarea, ".concat(t,"-show-count-suffix, ").concat(t,"-data-count")]:{color:e.colorError}}}};t.ZP=(0,i.I$)("Input",e=>{let t=(0,c.TS)(e,(0,l.e)(e));return[g(t),w(t),v(t),b(t),y(t),x(t),(0,a.c)(t)]},l.T)},37433:function(e,t,n){"use strict";n.d(t,{T:function(){return a},e:function(){return o}});var r=n(3104);function o(e){return(0,r.TS)(e,{inputAffixPadding:e.paddingXXS})}let a=e=>{let{controlHeight:t,fontSize:n,lineHeight:r,lineWidth:o,controlHeightSM:a,controlHeightLG:i,fontSizeLG:c,lineHeightLG:l,paddingSM:s,controlPaddingHorizontalSM:u,controlPaddingHorizontal:d,colorFillAlter:f,colorPrimaryHover:p,colorPrimary:m,controlOutlineWidth:g,controlOutline:h,colorErrorOutline:v,colorWarningOutline:b,colorBgContainer:y}=e;return{paddingBlock:Math.max(Math.round((t-n*r)/2*10)/10-o,0),paddingBlockSM:Math.max(Math.round((a-n*r)/2*10)/10-o,0),paddingBlockLG:Math.ceil((i-c*l)/2*10)/10-o,paddingInline:s-o,paddingInlineSM:u-o,paddingInlineLG:d-o,addonBg:f,activeBorderColor:m,hoverBorderColor:p,activeShadow:"0 0 0 ".concat(g,"px ").concat(h),errorActiveShadow:"0 0 0 ".concat(g,"px ").concat(v),warningActiveShadow:"0 0 0 ".concat(g,"px ").concat(b),hoverBg:y,activeBg:y,inputFontSize:n,inputFontSizeLG:c,inputFontSizeSM:n}}},65265:function(e,t,n){"use strict";n.d(t,{$U:function(){return c},H8:function(){return g},Mu:function(){return f},S5:function(){return v},Xy:function(){return i},ir:function(){return d},qG:function(){return s}});var r=n(352),o=n(3104);let a=e=>({borderColor:e.hoverBorderColor,backgroundColor:e.hoverBg}),i=e=>({color:e.colorTextDisabled,backgroundColor:e.colorBgContainerDisabled,borderColor:e.colorBorder,boxShadow:"none",cursor:"not-allowed",opacity:1,"&:hover:not([disabled])":Object.assign({},a((0,o.TS)(e,{hoverBorderColor:e.colorBorder,hoverBg:e.colorBgContainerDisabled})))}),c=(e,t)=>({background:e.colorBgContainer,borderWidth:e.lineWidth,borderStyle:e.lineType,borderColor:t.borderColor,"&:hover":{borderColor:t.hoverBorderColor,backgroundColor:e.hoverBg},"&:focus, &:focus-within":{borderColor:t.activeBorderColor,boxShadow:t.activeShadow,outline:0,backgroundColor:e.activeBg}}),l=(e,t)=>({["&".concat(e.componentCls,"-status-").concat(t.status,":not(").concat(e.componentCls,"-disabled)")]:Object.assign(Object.assign({},c(e,t)),{["".concat(e.componentCls,"-prefix, ").concat(e.componentCls,"-suffix")]:{color:t.affixColor}})}),s=(e,t)=>({"&-outlined":Object.assign(Object.assign(Object.assign(Object.assign(Object.assign({},c(e,{borderColor:e.colorBorder,hoverBorderColor:e.colorPrimaryHover,activeBorderColor:e.colorPrimary,activeShadow:e.activeShadow})),{["&".concat(e.componentCls,"-disabled, &[disabled]")]:Object.assign({},i(e))}),l(e,{status:"error",borderColor:e.colorError,hoverBorderColor:e.colorErrorBorderHover,activeBorderColor:e.colorError,activeShadow:e.errorActiveShadow,affixColor:e.colorError})),l(e,{status:"warning",borderColor:e.colorWarning,hoverBorderColor:e.colorWarningBorderHover,activeBorderColor:e.colorWarning,activeShadow:e.warningActiveShadow,affixColor:e.colorWarning})),t)}),u=(e,t)=>({["&".concat(e.componentCls,"-group-wrapper-status-").concat(t.status)]:{["".concat(e.componentCls,"-group-addon")]:{borderColor:t.addonBorderColor,color:t.addonColor}}}),d=e=>({"&-outlined":Object.assign(Object.assign(Object.assign({["".concat(e.componentCls,"-group")]:{"&-addon":{background:e.addonBg,border:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorBorder)},"&-addon:first-child":{borderInlineEnd:0},"&-addon:last-child":{borderInlineStart:0}}},u(e,{status:"error",addonBorderColor:e.colorError,addonColor:e.colorErrorText})),u(e,{status:"warning",addonBorderColor:e.colorWarning,addonColor:e.colorWarningText})),{["&".concat(e.componentCls,"-group-wrapper-disabled")]:{["".concat(e.componentCls,"-group-addon")]:Object.assign({},i(e))}})}),f=(e,t)=>({"&-borderless":Object.assign({background:"transparent",border:"none","&:focus, &:focus-within":{outline:"none"},["&".concat(e.componentCls,"-disabled, &[disabled]")]:{color:e.colorTextDisabled}},t)}),p=(e,t)=>({background:t.bg,borderWidth:e.lineWidth,borderStyle:e.lineType,borderColor:"transparent","input&, & input, textarea&, & textarea":{color:null==t?void 0:t.inputColor},"&:hover":{background:t.hoverBg},"&:focus, &:focus-within":{outline:0,borderColor:t.activeBorderColor,backgroundColor:e.activeBg}}),m=(e,t)=>({["&".concat(e.componentCls,"-status-").concat(t.status,":not(").concat(e.componentCls,"-disabled)")]:Object.assign(Object.assign({},p(e,t)),{["".concat(e.componentCls,"-prefix, ").concat(e.componentCls,"-suffix")]:{color:t.affixColor}})}),g=(e,t)=>({"&-filled":Object.assign(Object.assign(Object.assign(Object.assign(Object.assign({},p(e,{bg:e.colorFillTertiary,hoverBg:e.colorFillSecondary,activeBorderColor:e.colorPrimary})),{["&".concat(e.componentCls,"-disabled, &[disabled]")]:Object.assign({},i(e))}),m(e,{status:"error",bg:e.colorErrorBg,hoverBg:e.colorErrorBgHover,activeBorderColor:e.colorError,inputColor:e.colorErrorText,affixColor:e.colorError})),m(e,{status:"warning",bg:e.colorWarningBg,hoverBg:e.colorWarningBgHover,activeBorderColor:e.colorWarning,inputColor:e.colorWarningText,affixColor:e.colorWarning})),t)}),h=(e,t)=>({["&".concat(e.componentCls,"-group-wrapper-status-").concat(t.status)]:{["".concat(e.componentCls,"-group-addon")]:{background:t.addonBg,color:t.addonColor}}}),v=e=>({"&-filled":Object.assign(Object.assign(Object.assign({["".concat(e.componentCls,"-group")]:{"&-addon":{background:e.colorFillTertiary},["".concat(e.componentCls,"-filled:not(:focus):not(:focus-within)")]:{"&:not(:first-child)":{borderInlineStart:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorSplit)},"&:not(:last-child)":{borderInlineEnd:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorSplit)}}}},h(e,{status:"error",addonBg:e.colorErrorBg,addonColor:e.colorErrorText})),h(e,{status:"warning",addonBg:e.colorWarningBg,addonColor:e.colorWarningText})),{["&".concat(e.componentCls,"-group-wrapper-disabled")]:{["".concat(e.componentCls,"-group")]:{"&-addon":{background:e.colorFillTertiary,color:e.colorTextDisabled},"&-addon:first-child":{borderInlineStart:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorBorder),borderTop:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorBorder),borderBottom:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorBorder)},"&-addon:last-child":{borderInlineEnd:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorBorder),borderTop:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorBorder),borderBottom:"".concat((0,r.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorBorder)}}}})})},91325:function(e,t,n){"use strict";let r=(0,n(2265).createContext)(void 0);t.Z=r},13823:function(e,t,n){"use strict";n.d(t,{Z:function(){return c}});var r=n(96257),o={placeholder:"Select time",rangePlaceholder:["Start time","End time"]};let a={lang:Object.assign({placeholder:"Select date",yearPlaceholder:"Select year",quarterPlaceholder:"Select quarter",monthPlaceholder:"Select month",weekPlaceholder:"Select week",rangePlaceholder:["Start date","End date"],rangeYearPlaceholder:["Start year","End year"],rangeQuarterPlaceholder:["Start quarter","End quarter"],rangeMonthPlaceholder:["Start month","End month"],rangeWeekPlaceholder:["Start week","End week"]},{locale:"en_US",today:"Today",now:"Now",backToToday:"Back to today",ok:"OK",clear:"Clear",month:"Month",year:"Year",timeSelect:"select time",dateSelect:"select date",weekSelect:"Choose a week",monthSelect:"Choose a month",yearSelect:"Choose a year",decadeSelect:"Choose a decade",yearFormat:"YYYY",dateFormat:"M/D/YYYY",dayFormat:"D",dateTimeFormat:"M/D/YYYY HH:mm:ss",monthBeforeYear:!0,previousMonth:"Previous month (PageUp)",nextMonth:"Next month (PageDown)",previousYear:"Last year (Control + left)",nextYear:"Next year (Control + right)",previousDecade:"Last decade",nextDecade:"Next decade",previousCentury:"Last century",nextCentury:"Next century"}),timePickerLocale:Object.assign({},o)},i="${label} is not a valid ${type}";var c={locale:"en",Pagination:r.Z,DatePicker:a,TimePicker:o,Calendar:a,global:{placeholder:"Please select"},Table:{filterTitle:"Filter menu",filterConfirm:"OK",filterReset:"Reset",filterEmptyText:"No filters",filterCheckall:"Select all items",filterSearchPlaceholder:"Search in filters",emptyText:"No data",selectAll:"Select current page",selectInvert:"Invert current page",selectNone:"Clear all data",selectionAll:"Select all data",sortTitle:"Sort",expand:"Expand row",collapse:"Collapse row",triggerDesc:"Click to sort descending",triggerAsc:"Click to sort ascending",cancelSort:"Click to cancel sorting"},Tour:{Next:"Next",Previous:"Previous",Finish:"Finish"},Modal:{okText:"OK",cancelText:"Cancel",justOkText:"OK"},Popconfirm:{okText:"OK",cancelText:"Cancel"},Transfer:{titles:["",""],searchPlaceholder:"Search here",itemUnit:"item",itemsUnit:"items",remove:"Remove",selectCurrent:"Select current page",removeCurrent:"Remove current page",selectAll:"Select all data",removeAll:"Remove all data",selectInvert:"Invert current page"},Upload:{uploading:"Uploading...",removeFile:"Remove file",uploadError:"Upload error",previewFile:"Preview file",downloadFile:"Download file"},Empty:{description:"No data"},Icon:{icon:"icon"},Text:{edit:"Edit",copy:"Copy",copied:"Copied",expand:"Expand"},PageHeader:{back:"Back"},Form:{optional:"(optional)",defaultValidateMessages:{default:"Field validation error for ${label}",required:"Please enter ${label}",enum:"${label} must be one of [${enum}]",whitespace:"${label} cannot be a blank character",date:{format:"${label} date format is invalid",parse:"${label} cannot be converted to a date",invalid:"${label} is an invalid date"},types:{string:i,method:i,array:i,object:i,number:i,date:i,boolean:i,integer:i,float:i,regexp:i,email:i,url:i,hex:i},string:{len:"${label} must be ${len} characters",min:"${label} must be at least ${min} characters",max:"${label} must be up to ${max} characters",range:"${label} must be between ${min}-${max} characters"},number:{len:"${label} must be equal to ${len}",min:"${label} must be minimum ${min}",max:"${label} must be maximum ${max}",range:"${label} must be between ${min}-${max}"},array:{len:"Must be ${len} ${label}",min:"At least ${min} ${label}",max:"At most ${max} ${label}",range:"The amount of ${label} must be between ${min}-${max}"},pattern:{mismatch:"${label} does not match the pattern ${pattern}"}}},Image:{preview:"Preview"},QRCode:{expired:"QR code expired",refresh:"Refresh",scanned:"Scanned"},ColorPicker:{presetEmpty:"Empty"}}},55274:function(e,t,n){"use strict";var r=n(2265),o=n(91325),a=n(13823);t.Z=(e,t)=>{let n=r.useContext(o.Z);return[r.useMemo(()=>{var r;let o=t||a.Z[e],i=null!==(r=null==n?void 0:n[e])&&void 0!==r?r:{};return Object.assign(Object.assign({},"function"==typeof o?o():o),i||{})},[e,t,n]),r.useMemo(()=>{let e=null==n?void 0:n.locale;return(null==n?void 0:n.exist)&&!e?a.Z.locale:e},[n])]}},42264:function(e,t,n){"use strict";n.d(t,{ZP:function(){return G}});var r=n(83145),o=n(2265),a=n(18404),i=n(52402),c=n(71744),l=n(13959),s=n(8900),u=n(39725),d=n(54537),f=n(55726),p=n(61935),m=n(36760),g=n.n(m),h=n(49283),v=n(352),b=n(62236),y=n(12918),w=n(80669),x=n(3104);let E=e=>{let{componentCls:t,iconCls:n,boxShadow:r,colorText:o,colorSuccess:a,colorError:i,colorWarning:c,colorInfo:l,fontSizeLG:s,motionEaseInOutCirc:u,motionDurationSlow:d,marginXS:f,paddingXS:p,borderRadiusLG:m,zIndexPopup:g,contentPadding:h,contentBg:b}=e,w="".concat(t,"-notice"),x=new v.E4("MessageMoveIn",{"0%":{padding:0,transform:"translateY(-100%)",opacity:0},"100%":{padding:p,transform:"translateY(0)",opacity:1}}),E=new v.E4("MessageMoveOut",{"0%":{maxHeight:e.height,padding:p,opacity:1},"100%":{maxHeight:0,padding:0,opacity:0}}),S={padding:p,textAlign:"center",["".concat(t,"-custom-content > ").concat(n)]:{verticalAlign:"text-bottom",marginInlineEnd:f,fontSize:s},["".concat(w,"-content")]:{display:"inline-block",padding:h,background:b,borderRadius:m,boxShadow:r,pointerEvents:"all"},["".concat(t,"-success > ").concat(n)]:{color:a},["".concat(t,"-error > ").concat(n)]:{color:i},["".concat(t,"-warning > ").concat(n)]:{color:c},["".concat(t,"-info > ").concat(n,",\n ").concat(t,"-loading > ").concat(n)]:{color:l}};return[{[t]:Object.assign(Object.assign({},(0,y.Wf)(e)),{color:o,position:"fixed",top:f,width:"100%",pointerEvents:"none",zIndex:g,["".concat(t,"-move-up")]:{animationFillMode:"forwards"},["\n ".concat(t,"-move-up-appear,\n ").concat(t,"-move-up-enter\n ")]:{animationName:x,animationDuration:d,animationPlayState:"paused",animationTimingFunction:u},["\n ".concat(t,"-move-up-appear").concat(t,"-move-up-appear-active,\n ").concat(t,"-move-up-enter").concat(t,"-move-up-enter-active\n ")]:{animationPlayState:"running"},["".concat(t,"-move-up-leave")]:{animationName:E,animationDuration:d,animationPlayState:"paused",animationTimingFunction:u},["".concat(t,"-move-up-leave").concat(t,"-move-up-leave-active")]:{animationPlayState:"running"},"&-rtl":{direction:"rtl",span:{direction:"rtl"}}})},{[t]:{["".concat(w,"-wrapper")]:Object.assign({},S)}},{["".concat(t,"-notice-pure-panel")]:Object.assign(Object.assign({},S),{padding:0,textAlign:"start"})}]};var S=(0,w.I$)("Message",e=>[E((0,x.TS)(e,{height:150}))],e=>({zIndexPopup:e.zIndexPopupBase+b.u6+10,contentBg:e.colorBgElevated,contentPadding:"".concat((e.controlHeightLG-e.fontSize*e.lineHeight)/2,"px ").concat(e.paddingSM,"px")})),C=n(64024),Z=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let O={info:o.createElement(f.Z,null),success:o.createElement(s.Z,null),error:o.createElement(u.Z,null),warning:o.createElement(d.Z,null),loading:o.createElement(p.Z,null)},k=e=>{let{prefixCls:t,type:n,icon:r,children:a}=e;return o.createElement("div",{className:g()("".concat(t,"-custom-content"),"".concat(t,"-").concat(n))},r||O[n],o.createElement("span",null,a))};var M=n(49638),j=n(13613);function I(e){let t;let n=new Promise(n=>{t=e(()=>{n(!0)})}),r=()=>{null==t||t()};return r.then=(e,t)=>n.then(e,t),r.promise=n,r}var R=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let N=e=>{let{children:t,prefixCls:n}=e,r=(0,C.Z)(n),[a,i,c]=S(n,r);return a(o.createElement(h.JB,{classNames:{list:g()(i,c,r)}},t))},P=(e,t)=>{let{prefixCls:n,key:r}=t;return o.createElement(N,{prefixCls:n,key:r},e)},F=o.forwardRef((e,t)=>{let{top:n,prefixCls:r,getContainer:a,maxCount:i,duration:l=3,rtl:s,transitionName:u,onAllRemoved:d}=e,{getPrefixCls:f,getPopupContainer:p,message:m,direction:v}=o.useContext(c.E_),b=r||f("message"),y=o.createElement("span",{className:"".concat(b,"-close-x")},o.createElement(M.Z,{className:"".concat(b,"-close-icon")})),[w,x]=(0,h.lm)({prefixCls:b,style:()=>({left:"50%",transform:"translateX(-50%)",top:null!=n?n:8}),className:()=>g()({["".concat(b,"-rtl")]:null!=s?s:"rtl"===v}),motion:()=>({motionName:null!=u?u:"".concat(b,"-move-up")}),closable:!1,closeIcon:y,duration:l,getContainer:()=>(null==a?void 0:a())||(null==p?void 0:p())||document.body,maxCount:i,onAllRemoved:d,renderNotifications:P});return o.useImperativeHandle(t,()=>Object.assign(Object.assign({},w),{prefixCls:b,message:m})),x}),T=0;function A(e){let t=o.useRef(null);return(0,j.ln)("Message"),[o.useMemo(()=>{let e=e=>{var n;null===(n=t.current)||void 0===n||n.close(e)},n=n=>{if(!t.current){let e=()=>{};return e.then=()=>{},e}let{open:r,prefixCls:a,message:i}=t.current,c="".concat(a,"-notice"),{content:l,icon:s,type:u,key:d,className:f,style:p,onClose:m}=n,h=R(n,["content","icon","type","key","className","style","onClose"]),v=d;return null==v&&(T+=1,v="antd-message-".concat(T)),I(t=>(r(Object.assign(Object.assign({},h),{key:v,content:o.createElement(k,{prefixCls:a,type:u,icon:s},l),placement:"top",className:g()(u&&"".concat(c,"-").concat(u),f,null==i?void 0:i.className),style:Object.assign(Object.assign({},null==i?void 0:i.style),p),onClose:()=>{null==m||m(),t()}})),()=>{e(v)}))},r={open:n,destroy:n=>{var r;void 0!==n?e(n):null===(r=t.current)||void 0===r||r.destroy()}};return["info","success","warning","error","loading"].forEach(e=>{r[e]=(t,r,o)=>{let a,i,c;return a=t&&"object"==typeof t&&"content"in t?t:{content:t},"function"==typeof r?c=r:(i=r,c=o),n(Object.assign(Object.assign({onClose:c,duration:i},a),{type:e}))}}),r},[]),o.createElement(F,Object.assign({key:"message-holder"},e,{ref:t}))]}let L=null,z=e=>e(),_=[],H={};function B(){let{getContainer:e,duration:t,rtl:n,maxCount:r,top:o}=H,a=(null==e?void 0:e())||document.body;return{getContainer:()=>a,duration:t,rtl:n,maxCount:r,top:o}}let D=o.forwardRef((e,t)=>{let{messageConfig:n,sync:r}=e,{getPrefixCls:a}=(0,o.useContext)(c.E_),l=H.prefixCls||a("message"),s=(0,o.useContext)(i.J),[u,d]=A(Object.assign(Object.assign(Object.assign({},n),{prefixCls:l}),s.message));return o.useImperativeHandle(t,()=>{let e=Object.assign({},u);return Object.keys(e).forEach(t=>{e[t]=function(){return r(),u[t].apply(u,arguments)}}),{instance:e,sync:r}}),d}),W=o.forwardRef((e,t)=>{let[n,r]=o.useState(B),a=()=>{r(B)};o.useEffect(a,[]);let i=(0,l.w6)(),c=i.getRootPrefixCls(),s=i.getIconPrefixCls(),u=i.getTheme(),d=o.createElement(D,{ref:t,sync:a,messageConfig:n});return o.createElement(l.ZP,{prefixCls:c,iconPrefixCls:s,theme:u},i.holderRender?i.holderRender(d):d)});function V(){if(!L){let e=document.createDocumentFragment(),t={fragment:e};L=t,z(()=>{(0,a.s)(o.createElement(W,{ref:e=>{let{instance:n,sync:r}=e||{};Promise.resolve().then(()=>{!t.instance&&n&&(t.instance=n,t.sync=r,V())})}}),e)});return}L.instance&&(_.forEach(e=>{let{type:t,skipped:n}=e;if(!n)switch(t){case"open":z(()=>{let t=L.instance.open(Object.assign(Object.assign({},H),e.config));null==t||t.then(e.resolve),e.setCloseFn(t)});break;case"destroy":z(()=>{null==L||L.instance.destroy(e.key)});break;default:z(()=>{var n;let o=(n=L.instance)[t].apply(n,(0,r.Z)(e.args));null==o||o.then(e.resolve),e.setCloseFn(o)})}}),_=[])}let q={open:function(e){let t=I(t=>{let n;let r={type:"open",config:e,resolve:t,setCloseFn:e=>{n=e}};return _.push(r),()=>{n?z(()=>{n()}):r.skipped=!0}});return V(),t},destroy:function(e){_.push({type:"destroy",key:e}),V()},config:function(e){H=Object.assign(Object.assign({},H),e),z(()=>{var e;null===(e=null==L?void 0:L.sync)||void 0===e||e.call(L)})},useMessage:function(e){return A(e)},_InternalPanelDoNotUseOrYouWillBeFired:e=>{let{prefixCls:t,className:n,type:r,icon:a,content:i}=e,l=Z(e,["prefixCls","className","type","icon","content"]),{getPrefixCls:s}=o.useContext(c.E_),u=t||s("message"),d=(0,C.Z)(u),[f,p,m]=S(u,d);return f(o.createElement(h.qX,Object.assign({},l,{prefixCls:u,className:g()(n,p,"".concat(u,"-notice-pure-panel"),m,d),eventKey:"pure",duration:null,content:o.createElement(k,{prefixCls:u,type:r,icon:a},i)})))}};["success","info","warning","error","loading"].forEach(e=>{q[e]=function(){for(var t=arguments.length,n=Array(t),r=0;r{let r;let o={type:e,args:t,resolve:n,setCloseFn:e=>{r=e}};return _.push(o),()=>{r?z(()=>{r()}):o.skipped=!0}});return V(),n}(e,n)}});var G=q},92246:function(e,t,n){"use strict";n.d(t,{A:function(){return l},f:function(){return c}});var r=n(13823);let o=Object.assign({},r.Z.Modal),a=[],i=()=>a.reduce((e,t)=>Object.assign(Object.assign({},e),t),r.Z.Modal);function c(e){if(e){let t=Object.assign({},e);return a.push(t),o=i(),()=>{a=a.filter(e=>e!==t),o=i()}}o=Object.assign({},r.Z.Modal)}function l(){return o}},57271:function(e,t,n){"use strict";n.d(t,{ZP:function(){return er}});var r=n(2265),o=n(18404),a=n(52402),i=n(71744),c=n(13959),l=n(8900),s=n(39725),u=n(49638),d=n(54537),f=n(55726),p=n(61935),m=n(36760),g=n.n(m),h=n(49283),v=n(64024),b=n(352),y=n(62236),w=n(12918),x=n(3104),E=n(80669),S=e=>{let{componentCls:t,notificationMarginEdge:n,animationMaxHeight:r}=e,o="".concat(t,"-notice"),a=new b.E4("antNotificationFadeIn",{"0%":{transform:"translate3d(100%, 0, 0)",opacity:0},"100%":{transform:"translate3d(0, 0, 0)",opacity:1}}),i=new b.E4("antNotificationTopFadeIn",{"0%":{top:-r,opacity:0},"100%":{top:0,opacity:1}}),c=new b.E4("antNotificationBottomFadeIn",{"0%":{bottom:e.calc(r).mul(-1).equal(),opacity:0},"100%":{bottom:0,opacity:1}}),l=new b.E4("antNotificationLeftFadeIn",{"0%":{transform:"translate3d(-100%, 0, 0)",opacity:0},"100%":{transform:"translate3d(0, 0, 0)",opacity:1}});return{[t]:{["&".concat(t,"-top, &").concat(t,"-bottom")]:{marginInline:0,[o]:{marginInline:"auto auto"}},["&".concat(t,"-top")]:{["".concat(t,"-fade-enter").concat(t,"-fade-enter-active, ").concat(t,"-fade-appear").concat(t,"-fade-appear-active")]:{animationName:i}},["&".concat(t,"-bottom")]:{["".concat(t,"-fade-enter").concat(t,"-fade-enter-active, ").concat(t,"-fade-appear").concat(t,"-fade-appear-active")]:{animationName:c}},["&".concat(t,"-topRight, &").concat(t,"-bottomRight")]:{["".concat(t,"-fade-enter").concat(t,"-fade-enter-active, ").concat(t,"-fade-appear").concat(t,"-fade-appear-active")]:{animationName:a}},["&".concat(t,"-topLeft, &").concat(t,"-bottomLeft")]:{marginRight:{value:0,_skip_check_:!0},marginLeft:{value:n,_skip_check_:!0},[o]:{marginInlineEnd:"auto",marginInlineStart:0},["".concat(t,"-fade-enter").concat(t,"-fade-enter-active, ").concat(t,"-fade-appear").concat(t,"-fade-appear-active")]:{animationName:l}}}}};let C=["top","topLeft","topRight","bottom","bottomLeft","bottomRight"],Z={topLeft:"left",topRight:"right",bottomLeft:"left",bottomRight:"right",top:"left",bottom:"left"},O=(e,t)=>{let{componentCls:n}=e;return{["".concat(n,"-").concat(t)]:{["&".concat(n,"-stack > ").concat(n,"-notice-wrapper")]:{[t.startsWith("top")?"top":"bottom"]:0,[Z[t]]:{value:0,_skip_check_:!0}}}}},k=e=>{let t={};for(let n=1;n ".concat(e.componentCls,"-notice")]:{opacity:0,transition:"opacity ".concat(e.motionDurationMid)}};return Object.assign({["&:not(:nth-last-child(-n+".concat(e.notificationStackLayer,"))")]:{opacity:0,overflow:"hidden",color:"transparent",pointerEvents:"none"}},t)},M=e=>{let t={};for(let n=1;n{let{componentCls:t}=e;return Object.assign({["".concat(t,"-stack")]:{["& > ".concat(t,"-notice-wrapper")]:Object.assign({transition:"all ".concat(e.motionDurationSlow,", backdrop-filter 0s"),position:"absolute"},k(e))},["".concat(t,"-stack:not(").concat(t,"-stack-expanded)")]:{["& > ".concat(t,"-notice-wrapper")]:Object.assign({},M(e))},["".concat(t,"-stack").concat(t,"-stack-expanded")]:{["& > ".concat(t,"-notice-wrapper")]:{"&:not(:nth-last-child(-n + 1))":{opacity:1,overflow:"unset",color:"inherit",pointerEvents:"auto",["& > ".concat(e.componentCls,"-notice")]:{opacity:1}},"&:after":{content:'""',position:"absolute",height:e.margin,width:"100%",insetInline:0,bottom:e.calc(e.margin).mul(-1).equal(),background:"transparent",pointerEvents:"auto"}}}},C.map(t=>O(e,t)).reduce((e,t)=>Object.assign(Object.assign({},e),t),{}))};let I=e=>{let{iconCls:t,componentCls:n,boxShadow:r,fontSizeLG:o,notificationMarginBottom:a,borderRadiusLG:i,colorSuccess:c,colorInfo:l,colorWarning:s,colorError:u,colorTextHeading:d,notificationBg:f,notificationPadding:p,notificationMarginEdge:m,fontSize:g,lineHeight:h,width:v,notificationIconSize:y,colorText:w}=e,x="".concat(n,"-notice");return{position:"relative",marginBottom:a,marginInlineStart:"auto",background:f,borderRadius:i,boxShadow:r,[x]:{padding:p,width:v,maxWidth:"calc(100vw - ".concat((0,b.bf)(e.calc(m).mul(2).equal()),")"),overflow:"hidden",lineHeight:h,wordWrap:"break-word"},["".concat(n,"-close-icon")]:{fontSize:g,cursor:"pointer"},["".concat(x,"-message")]:{marginBottom:e.marginXS,color:d,fontSize:o,lineHeight:e.lineHeightLG},["".concat(x,"-description")]:{fontSize:g,color:w},["".concat(x,"-closable ").concat(x,"-message")]:{paddingInlineEnd:e.paddingLG},["".concat(x,"-with-icon ").concat(x,"-message")]:{marginBottom:e.marginXS,marginInlineStart:e.calc(e.marginSM).add(y).equal(),fontSize:o},["".concat(x,"-with-icon ").concat(x,"-description")]:{marginInlineStart:e.calc(e.marginSM).add(y).equal(),fontSize:g},["".concat(x,"-icon")]:{position:"absolute",fontSize:y,lineHeight:1,["&-success".concat(t)]:{color:c},["&-info".concat(t)]:{color:l},["&-warning".concat(t)]:{color:s},["&-error".concat(t)]:{color:u}},["".concat(x,"-close")]:{position:"absolute",top:e.notificationPaddingVertical,insetInlineEnd:e.notificationPaddingHorizontal,color:e.colorIcon,outline:"none",width:e.notificationCloseButtonSize,height:e.notificationCloseButtonSize,borderRadius:e.borderRadiusSM,transition:"background-color ".concat(e.motionDurationMid,", color ").concat(e.motionDurationMid),display:"flex",alignItems:"center",justifyContent:"center","&:hover":{color:e.colorIconHover,backgroundColor:e.closeBtnHoverBg}},["".concat(x,"-btn")]:{float:"right",marginTop:e.marginSM}}},R=e=>{let{componentCls:t,notificationMarginBottom:n,notificationMarginEdge:r,motionDurationMid:o,motionEaseInOut:a}=e,i="".concat(t,"-notice"),c=new b.E4("antNotificationFadeOut",{"0%":{maxHeight:e.animationMaxHeight,marginBottom:n},"100%":{maxHeight:0,marginBottom:0,paddingTop:0,paddingBottom:0,opacity:0}});return[{[t]:Object.assign(Object.assign({},(0,w.Wf)(e)),{position:"fixed",zIndex:e.zIndexPopup,marginRight:{value:r,_skip_check_:!0},["".concat(t,"-hook-holder")]:{position:"relative"},["".concat(t,"-fade-appear-prepare")]:{opacity:"0 !important"},["".concat(t,"-fade-enter, ").concat(t,"-fade-appear")]:{animationDuration:e.motionDurationMid,animationTimingFunction:a,animationFillMode:"both",opacity:0,animationPlayState:"paused"},["".concat(t,"-fade-leave")]:{animationTimingFunction:a,animationFillMode:"both",animationDuration:o,animationPlayState:"paused"},["".concat(t,"-fade-enter").concat(t,"-fade-enter-active, ").concat(t,"-fade-appear").concat(t,"-fade-appear-active")]:{animationPlayState:"running"},["".concat(t,"-fade-leave").concat(t,"-fade-leave-active")]:{animationName:c,animationPlayState:"running"},"&-rtl":{direction:"rtl",["".concat(i,"-btn")]:{float:"left"}}})},{[t]:{["".concat(i,"-wrapper")]:Object.assign({},I(e))}}]},N=e=>({zIndexPopup:e.zIndexPopupBase+y.u6+50,width:384,closeBtnHoverBg:e.wireframe?"transparent":e.colorFillContent}),P=e=>{let t=e.paddingMD,n=e.paddingLG;return(0,x.TS)(e,{notificationBg:e.colorBgElevated,notificationPaddingVertical:t,notificationPaddingHorizontal:n,notificationIconSize:e.calc(e.fontSizeLG).mul(e.lineHeightLG).equal(),notificationCloseButtonSize:e.calc(e.controlHeightLG).mul(.55).equal(),notificationMarginBottom:e.margin,notificationPadding:"".concat((0,b.bf)(e.paddingMD)," ").concat((0,b.bf)(e.paddingContentHorizontalLG)),notificationMarginEdge:e.marginLG,animationMaxHeight:150,notificationStackLayer:3})};var F=(0,E.I$)("Notification",e=>{let t=P(e);return[R(t),S(t),j(t)]},N),T=(0,E.bk)(["Notification","PurePanel"],e=>{let t="".concat(e.componentCls,"-notice"),n=P(e);return{["".concat(t,"-pure-panel")]:Object.assign(Object.assign({},I(n)),{width:n.width,maxWidth:"calc(100vw - ".concat((0,b.bf)(e.calc(n.notificationMarginEdge).mul(2).equal()),")"),margin:0})}},N),A=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};function L(e,t){return null===t||!1===t?null:t||r.createElement("span",{className:"".concat(e,"-close-x")},r.createElement(u.Z,{className:"".concat(e,"-close-icon")}))}f.Z,l.Z,s.Z,d.Z,p.Z;let z={success:l.Z,info:f.Z,error:s.Z,warning:d.Z},_=e=>{let{prefixCls:t,icon:n,type:o,message:a,description:i,btn:c,role:l="alert"}=e,s=null;return n?s=r.createElement("span",{className:"".concat(t,"-icon")},n):o&&(s=r.createElement(z[o]||null,{className:g()("".concat(t,"-icon"),"".concat(t,"-icon-").concat(o))})),r.createElement("div",{className:g()({["".concat(t,"-with-icon")]:s}),role:l},s,r.createElement("div",{className:"".concat(t,"-message")},a),r.createElement("div",{className:"".concat(t,"-description")},i),c&&r.createElement("div",{className:"".concat(t,"-btn")},c))};var H=n(13613),B=n(29961),D=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let W=e=>{let{children:t,prefixCls:n}=e,o=(0,v.Z)(n),[a,i,c]=F(n,o);return a(r.createElement(h.JB,{classNames:{list:g()(i,c,o)}},t))},V=(e,t)=>{let{prefixCls:n,key:o}=t;return r.createElement(W,{prefixCls:n,key:o},e)},q=r.forwardRef((e,t)=>{let{top:n,bottom:o,prefixCls:a,getContainer:c,maxCount:l,rtl:s,onAllRemoved:u,stack:d}=e,{getPrefixCls:f,getPopupContainer:p,notification:m,direction:v}=(0,r.useContext)(i.E_),[,b]=(0,B.ZP)(),y=a||f("notification"),[w,x]=(0,h.lm)({prefixCls:y,style:e=>(function(e,t,n){let r;switch(e){case"top":r={left:"50%",transform:"translateX(-50%)",right:"auto",top:t,bottom:"auto"};break;case"topLeft":r={left:0,top:t,bottom:"auto"};break;case"topRight":r={right:0,top:t,bottom:"auto"};break;case"bottom":r={left:"50%",transform:"translateX(-50%)",right:"auto",top:"auto",bottom:n};break;case"bottomLeft":r={left:0,top:"auto",bottom:n};break;default:r={right:0,top:"auto",bottom:n}}return r})(e,null!=n?n:24,null!=o?o:24),className:()=>g()({["".concat(y,"-rtl")]:null!=s?s:"rtl"===v}),motion:()=>({motionName:"".concat(y,"-fade")}),closable:!0,closeIcon:L(y),duration:4.5,getContainer:()=>(null==c?void 0:c())||(null==p?void 0:p())||document.body,maxCount:l,onAllRemoved:u,renderNotifications:V,stack:!1!==d&&{threshold:"object"==typeof d?null==d?void 0:d.threshold:void 0,offset:8,gap:b.margin}});return r.useImperativeHandle(t,()=>Object.assign(Object.assign({},w),{prefixCls:y,notification:m})),x});function G(e){let t=r.useRef(null);return(0,H.ln)("Notification"),[r.useMemo(()=>{let n=n=>{var o;if(!t.current)return;let{open:a,prefixCls:i,notification:c}=t.current,l="".concat(i,"-notice"),{message:s,description:u,icon:d,type:f,btn:p,className:m,style:h,role:v="alert",closeIcon:b}=n,y=D(n,["message","description","icon","type","btn","className","style","role","closeIcon"]),w=L(l,b);return a(Object.assign(Object.assign({placement:null!==(o=null==e?void 0:e.placement)&&void 0!==o?o:"topRight"},y),{content:r.createElement(_,{prefixCls:l,icon:d,type:f,message:s,description:u,btn:p,role:v}),className:g()(f&&"".concat(l,"-").concat(f),m,null==c?void 0:c.className),style:Object.assign(Object.assign({},null==c?void 0:c.style),h),closeIcon:w,closable:!!w}))},o={open:n,destroy:e=>{var n,r;void 0!==e?null===(n=t.current)||void 0===n||n.close(e):null===(r=t.current)||void 0===r||r.destroy()}};return["success","info","warning","error"].forEach(e=>{o[e]=t=>n(Object.assign(Object.assign({},t),{type:e}))}),o},[]),r.createElement(q,Object.assign({key:"notification-holder"},e,{ref:t}))]}let X=null,U=e=>e(),$=[],K={};function Y(){let{getContainer:e,rtl:t,maxCount:n,top:r,bottom:o}=K,a=(null==e?void 0:e())||document.body;return{getContainer:()=>a,rtl:t,maxCount:n,top:r,bottom:o}}let Q=r.forwardRef((e,t)=>{let{notificationConfig:n,sync:o}=e,{getPrefixCls:c}=(0,r.useContext)(i.E_),l=K.prefixCls||c("notification"),s=(0,r.useContext)(a.J),[u,d]=G(Object.assign(Object.assign(Object.assign({},n),{prefixCls:l}),s.notification));return r.useEffect(o,[]),r.useImperativeHandle(t,()=>{let e=Object.assign({},u);return Object.keys(e).forEach(t=>{e[t]=function(){return o(),u[t].apply(u,arguments)}}),{instance:e,sync:o}}),d}),J=r.forwardRef((e,t)=>{let[n,o]=r.useState(Y),a=()=>{o(Y)};r.useEffect(a,[]);let i=(0,c.w6)(),l=i.getRootPrefixCls(),s=i.getIconPrefixCls(),u=i.getTheme(),d=r.createElement(Q,{ref:t,sync:a,notificationConfig:n});return r.createElement(c.ZP,{prefixCls:l,iconPrefixCls:s,theme:u},i.holderRender?i.holderRender(d):d)});function ee(){if(!X){let e=document.createDocumentFragment(),t={fragment:e};X=t,U(()=>{(0,o.s)(r.createElement(J,{ref:e=>{let{instance:n,sync:r}=e||{};Promise.resolve().then(()=>{!t.instance&&n&&(t.instance=n,t.sync=r,ee())})}}),e)});return}X.instance&&($.forEach(e=>{switch(e.type){case"open":U(()=>{X.instance.open(Object.assign(Object.assign({},K),e.config))});break;case"destroy":U(()=>{null==X||X.instance.destroy(e.key)})}}),$=[])}function et(e){(0,c.w6)(),$.push({type:"open",config:e}),ee()}let en={open:et,destroy:function(e){$.push({type:"destroy",key:e}),ee()},config:function(e){K=Object.assign(Object.assign({},K),e),U(()=>{var e;null===(e=null==X?void 0:X.sync)||void 0===e||e.call(X)})},useNotification:function(e){return G(e)},_InternalPanelDoNotUseOrYouWillBeFired:e=>{let{prefixCls:t,className:n,icon:o,type:a,message:c,description:l,btn:s,closable:u=!0,closeIcon:d,className:f}=e,p=A(e,["prefixCls","className","icon","type","message","description","btn","closable","closeIcon","className"]),{getPrefixCls:m}=r.useContext(i.E_),b=t||m("notification"),y="".concat(b,"-notice"),w=(0,v.Z)(b),[x,E,S]=F(b,w);return x(r.createElement("div",{className:g()("".concat(y,"-pure-panel"),E,n,S,w)},r.createElement(T,{prefixCls:b}),r.createElement(h.qX,Object.assign({},p,{prefixCls:b,eventKey:"pure",duration:null,closable:u,className:g()({notificationClassName:f}),closeIcon:L(b,d),content:r.createElement(_,{prefixCls:y,icon:o,type:a,message:c,description:l,btn:s})}))))}};["success","info","warning","error"].forEach(e=>{en[e]=t=>et(Object.assign(Object.assign({},t),{type:e}))});var er=en},52787:function(e,t,n){"use strict";n.d(t,{default:function(){return tt}});var r=n(2265),o=n(36760),a=n.n(o),i=n(1119),c=n(83145),l=n(11993),s=n(31686),u=n(26365),d=n(6989),f=n(41154),p=n(50506),m=n(32559),g=n(27380),h=n(79267),v=n(95814),b=n(28791),y=function(e){var t=e.className,n=e.customizeIcon,o=e.customizeIconProps,i=e.children,c=e.onMouseDown,l=e.onClick,s="function"==typeof n?n(o):n;return r.createElement("span",{className:t,onMouseDown:function(e){e.preventDefault(),null==c||c(e)},style:{userSelect:"none",WebkitUserSelect:"none"},unselectable:"on",onClick:l,"aria-hidden":!0},void 0!==s?s:r.createElement("span",{className:a()(t.split(/\s+/).map(function(e){return"".concat(e,"-icon")}))},i))},w=function(e,t,n,o,a){var i=arguments.length>5&&void 0!==arguments[5]&&arguments[5],c=arguments.length>6?arguments[6]:void 0,l=arguments.length>7?arguments[7]:void 0,s=r.useMemo(function(){return"object"===(0,f.Z)(o)?o.clearIcon:a||void 0},[o,a]);return{allowClear:r.useMemo(function(){return!i&&!!o&&(!!n.length||!!c)&&!("combobox"===l&&""===c)},[o,i,n.length,c,l]),clearIcon:r.createElement(y,{className:"".concat(e,"-clear"),onMouseDown:t,customizeIcon:s},"\xd7")}},x=r.createContext(null);function E(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:250,t=r.useRef(null),n=r.useRef(null);return r.useEffect(function(){return function(){window.clearTimeout(n.current)}},[]),[function(){return t.current},function(r){(r||null===t.current)&&(t.current=r),window.clearTimeout(n.current),n.current=window.setTimeout(function(){t.current=null},e)}]}var S=n(18242),C=n(1699),Z=r.forwardRef(function(e,t){var n,o=e.prefixCls,i=e.id,c=e.inputElement,l=e.disabled,u=e.tabIndex,d=e.autoFocus,f=e.autoComplete,p=e.editable,g=e.activeDescendantId,h=e.value,v=e.maxLength,y=e.onKeyDown,w=e.onMouseDown,x=e.onChange,E=e.onPaste,S=e.onCompositionStart,C=e.onCompositionEnd,Z=e.open,O=e.attrs,k=c||r.createElement("input",null),M=k,j=M.ref,I=M.props,R=I.onKeyDown,N=I.onChange,P=I.onMouseDown,F=I.onCompositionStart,T=I.onCompositionEnd,A=I.style;return(0,m.Kp)(!("maxLength"in k.props),"Passing 'maxLength' to input element directly may not work because input in BaseSelect is controlled."),k=r.cloneElement(k,(0,s.Z)((0,s.Z)((0,s.Z)({type:"search"},I),{},{id:i,ref:(0,b.sQ)(t,j),disabled:l,tabIndex:u,autoComplete:f||"off",autoFocus:d,className:a()("".concat(o,"-selection-search-input"),null===(n=k)||void 0===n||null===(n=n.props)||void 0===n?void 0:n.className),role:"combobox","aria-expanded":Z||!1,"aria-haspopup":"listbox","aria-owns":"".concat(i,"_list"),"aria-autocomplete":"list","aria-controls":"".concat(i,"_list"),"aria-activedescendant":Z?g:void 0},O),{},{value:p?h:"",maxLength:v,readOnly:!p,unselectable:p?null:"on",style:(0,s.Z)((0,s.Z)({},A),{},{opacity:p?null:0}),onKeyDown:function(e){y(e),R&&R(e)},onMouseDown:function(e){w(e),P&&P(e)},onChange:function(e){x(e),N&&N(e)},onCompositionStart:function(e){S(e),F&&F(e)},onCompositionEnd:function(e){C(e),T&&T(e)},onPaste:E}))});function O(e){return Array.isArray(e)?e:void 0!==e?[e]:[]}var k="undefined"!=typeof window&&window.document&&window.document.documentElement;function M(e){return["string","number"].includes((0,f.Z)(e))}function j(e){var t=void 0;return e&&(M(e.title)?t=e.title.toString():M(e.label)&&(t=e.label.toString())),t}function I(e){var t;return null!==(t=e.key)&&void 0!==t?t:e.value}var R=function(e){e.preventDefault(),e.stopPropagation()},N=function(e){var t,n,o=e.id,i=e.prefixCls,c=e.values,s=e.open,d=e.searchValue,f=e.autoClearSearchValue,p=e.inputRef,m=e.placeholder,g=e.disabled,h=e.mode,v=e.showSearch,b=e.autoFocus,w=e.autoComplete,x=e.activeDescendantId,E=e.tabIndex,O=e.removeIcon,M=e.maxTagCount,N=e.maxTagTextLength,P=e.maxTagPlaceholder,F=void 0===P?function(e){return"+ ".concat(e.length," ...")}:P,T=e.tagRender,A=e.onToggleOpen,L=e.onRemove,z=e.onInputChange,_=e.onInputPaste,H=e.onInputKeyDown,B=e.onInputMouseDown,D=e.onInputCompositionStart,W=e.onInputCompositionEnd,V=r.useRef(null),q=(0,r.useState)(0),G=(0,u.Z)(q,2),X=G[0],U=G[1],$=(0,r.useState)(!1),K=(0,u.Z)($,2),Y=K[0],Q=K[1],J="".concat(i,"-selection"),ee=s||"multiple"===h&&!1===f||"tags"===h?d:"",et="tags"===h||"multiple"===h&&!1===f||v&&(s||Y);t=function(){U(V.current.scrollWidth)},n=[ee],k?r.useLayoutEffect(t,n):r.useEffect(t,n);var en=function(e,t,n,o,i){return r.createElement("span",{title:j(e),className:a()("".concat(J,"-item"),(0,l.Z)({},"".concat(J,"-item-disabled"),n))},r.createElement("span",{className:"".concat(J,"-item-content")},t),o&&r.createElement(y,{className:"".concat(J,"-item-remove"),onMouseDown:R,onClick:i,customizeIcon:O},"\xd7"))},er=r.createElement("div",{className:"".concat(J,"-search"),style:{width:X},onFocus:function(){Q(!0)},onBlur:function(){Q(!1)}},r.createElement(Z,{ref:p,open:s,prefixCls:i,id:o,inputElement:null,disabled:g,autoFocus:b,autoComplete:w,editable:et,activeDescendantId:x,value:ee,onKeyDown:H,onMouseDown:B,onChange:z,onPaste:_,onCompositionStart:D,onCompositionEnd:W,tabIndex:E,attrs:(0,S.Z)(e,!0)}),r.createElement("span",{ref:V,className:"".concat(J,"-search-mirror"),"aria-hidden":!0},ee,"\xa0")),eo=r.createElement(C.Z,{prefixCls:"".concat(J,"-overflow"),data:c,renderItem:function(e){var t,n=e.disabled,o=e.label,a=e.value,i=!g&&!n,c=o;if("number"==typeof N&&("string"==typeof o||"number"==typeof o)){var l=String(c);l.length>N&&(c="".concat(l.slice(0,N),"..."))}var u=function(t){t&&t.stopPropagation(),L(e)};return"function"==typeof T?(t=c,r.createElement("span",{onMouseDown:function(e){R(e),A(!s)}},T({label:t,value:a,disabled:n,closable:i,onClose:u}))):en(e,c,n,i,u)},renderRest:function(e){var t="function"==typeof F?F(e):F;return en({title:t},t,!1)},suffix:er,itemKey:I,maxCount:M});return r.createElement(r.Fragment,null,eo,!c.length&&!ee&&r.createElement("span",{className:"".concat(J,"-placeholder")},m))},P=function(e){var t=e.inputElement,n=e.prefixCls,o=e.id,a=e.inputRef,i=e.disabled,c=e.autoFocus,l=e.autoComplete,s=e.activeDescendantId,d=e.mode,f=e.open,p=e.values,m=e.placeholder,g=e.tabIndex,h=e.showSearch,v=e.searchValue,b=e.activeValue,y=e.maxLength,w=e.onInputKeyDown,x=e.onInputMouseDown,E=e.onInputChange,C=e.onInputPaste,O=e.onInputCompositionStart,k=e.onInputCompositionEnd,M=e.title,I=r.useState(!1),R=(0,u.Z)(I,2),N=R[0],P=R[1],F="combobox"===d,T=F||h,A=p[0],L=v||"";F&&b&&!N&&(L=b),r.useEffect(function(){F&&P(!1)},[F,b]);var z=("combobox"===d||!!f||!!h)&&!!L,_=void 0===M?j(A):M,H=r.useMemo(function(){return A?null:r.createElement("span",{className:"".concat(n,"-selection-placeholder"),style:z?{visibility:"hidden"}:void 0},m)},[A,z,m,n]);return r.createElement(r.Fragment,null,r.createElement("span",{className:"".concat(n,"-selection-search")},r.createElement(Z,{ref:a,prefixCls:n,id:o,open:f,inputElement:t,disabled:i,autoFocus:c,autoComplete:l,editable:T,activeDescendantId:s,value:L,onKeyDown:w,onMouseDown:x,onChange:function(e){P(!0),E(e)},onPaste:C,onCompositionStart:O,onCompositionEnd:k,tabIndex:g,attrs:(0,S.Z)(e,!0),maxLength:F?y:void 0})),!F&&A?r.createElement("span",{className:"".concat(n,"-selection-item"),title:_,style:z?{visibility:"hidden"}:void 0},A.label):null,H)},F=r.forwardRef(function(e,t){var n=(0,r.useRef)(null),o=(0,r.useRef)(!1),a=e.prefixCls,c=e.open,l=e.mode,s=e.showSearch,d=e.tokenWithEnter,f=e.autoClearSearchValue,p=e.onSearch,m=e.onSearchSubmit,g=e.onToggleOpen,h=e.onInputKeyDown,b=e.domRef;r.useImperativeHandle(t,function(){return{focus:function(){n.current.focus()},blur:function(){n.current.blur()}}});var y=E(0),w=(0,u.Z)(y,2),x=w[0],S=w[1],C=(0,r.useRef)(null),Z=function(e){!1!==p(e,!0,o.current)&&g(!0)},O={inputRef:n,onInputKeyDown:function(e){var t=e.which;(t===v.Z.UP||t===v.Z.DOWN)&&e.preventDefault(),h&&h(e),t!==v.Z.ENTER||"tags"!==l||o.current||c||null==m||m(e.target.value),[v.Z.ESC,v.Z.SHIFT,v.Z.BACKSPACE,v.Z.TAB,v.Z.WIN_KEY,v.Z.ALT,v.Z.META,v.Z.WIN_KEY_RIGHT,v.Z.CTRL,v.Z.SEMICOLON,v.Z.EQUALS,v.Z.CAPS_LOCK,v.Z.CONTEXT_MENU,v.Z.F1,v.Z.F2,v.Z.F3,v.Z.F4,v.Z.F5,v.Z.F6,v.Z.F7,v.Z.F8,v.Z.F9,v.Z.F10,v.Z.F11,v.Z.F12].includes(t)||g(!0)},onInputMouseDown:function(){S(!0)},onInputChange:function(e){var t=e.target.value;if(d&&C.current&&/[\r\n]/.test(C.current)){var n=C.current.replace(/[\r\n]+$/,"").replace(/\r\n/g," ").replace(/[\r\n]/g," ");t=t.replace(n,C.current)}C.current=null,Z(t)},onInputPaste:function(e){var t=e.clipboardData,n=null==t?void 0:t.getData("text");C.current=n||""},onInputCompositionStart:function(){o.current=!0},onInputCompositionEnd:function(e){o.current=!1,"combobox"!==l&&Z(e.target.value)}},k="multiple"===l||"tags"===l?r.createElement(N,(0,i.Z)({},e,O)):r.createElement(P,(0,i.Z)({},e,O));return r.createElement("div",{ref:b,className:"".concat(a,"-selector"),onClick:function(e){e.target!==n.current&&(void 0!==document.body.style.msTouchAction?setTimeout(function(){n.current.focus()}):n.current.focus())},onMouseDown:function(e){var t=x();e.target===n.current||t||"combobox"===l||e.preventDefault(),("combobox"===l||s&&t)&&c||(c&&!1!==f&&p("",!0,!1),g())}},k)}),T=n(97821),A=["prefixCls","disabled","visible","children","popupElement","animation","transitionName","dropdownStyle","dropdownClassName","direction","placement","builtinPlacements","dropdownMatchSelectWidth","dropdownRender","dropdownAlign","getPopupContainer","empty","getTriggerDOMNode","onPopupVisibleChange","onPopupMouseEnter"],L=function(e){var t=!0===e?0:1;return{bottomLeft:{points:["tl","bl"],offset:[0,4],overflow:{adjustX:t,adjustY:1},htmlRegion:"scroll"},bottomRight:{points:["tr","br"],offset:[0,4],overflow:{adjustX:t,adjustY:1},htmlRegion:"scroll"},topLeft:{points:["bl","tl"],offset:[0,-4],overflow:{adjustX:t,adjustY:1},htmlRegion:"scroll"},topRight:{points:["br","tr"],offset:[0,-4],overflow:{adjustX:t,adjustY:1},htmlRegion:"scroll"}}},z=r.forwardRef(function(e,t){var n=e.prefixCls,o=(e.disabled,e.visible),c=e.children,u=e.popupElement,f=e.animation,p=e.transitionName,m=e.dropdownStyle,g=e.dropdownClassName,h=e.direction,v=e.placement,b=e.builtinPlacements,y=e.dropdownMatchSelectWidth,w=e.dropdownRender,x=e.dropdownAlign,E=e.getPopupContainer,S=e.empty,C=e.getTriggerDOMNode,Z=e.onPopupVisibleChange,O=e.onPopupMouseEnter,k=(0,d.Z)(e,A),M="".concat(n,"-dropdown"),j=u;w&&(j=w(u));var I=r.useMemo(function(){return b||L(y)},[b,y]),R=f?"".concat(M,"-").concat(f):p,N="number"==typeof y,P=r.useMemo(function(){return N?null:!1===y?"minWidth":"width"},[y,N]),F=m;N&&(F=(0,s.Z)((0,s.Z)({},F),{},{width:y}));var z=r.useRef(null);return r.useImperativeHandle(t,function(){return{getPopupElement:function(){return z.current}}}),r.createElement(T.Z,(0,i.Z)({},k,{showAction:Z?["click"]:[],hideAction:Z?["click"]:[],popupPlacement:v||("rtl"===(void 0===h?"ltr":h)?"bottomRight":"bottomLeft"),builtinPlacements:I,prefixCls:M,popupTransitionName:R,popup:r.createElement("div",{ref:z,onMouseEnter:O},j),stretch:P,popupAlign:x,popupVisible:o,getPopupContainer:E,popupClassName:a()(g,(0,l.Z)({},"".concat(M,"-empty"),S)),popupStyle:F,getTriggerDOMNode:C,onPopupVisibleChange:Z}),c)}),_=n(87099);function H(e,t){var n,r=e.key;return("value"in e&&(n=e.value),null!=r)?r:void 0!==n?n:"rc-index-key-".concat(t)}function B(e,t){var n=e||{},r=n.label,o=n.value,a=n.options,i=n.groupLabel,c=r||(t?"children":"label");return{label:c,value:o||"value",options:a||"options",groupLabel:i||c}}function D(e){var t=(0,s.Z)({},e);return"props"in t||Object.defineProperty(t,"props",{get:function(){return(0,m.ZP)(!1,"Return type is option instead of Option instance. Please read value directly instead of reading from `props`."),t}}),t}var W=function(e,t,n){if(!t||!t.length)return null;var r=!1,o=function e(t,n){var o=(0,_.Z)(n),a=o[0],i=o.slice(1);if(!a)return[t];var l=t.split(a);return r=r||l.length>1,l.reduce(function(t,n){return[].concat((0,c.Z)(t),(0,c.Z)(e(n,i)))},[]).filter(Boolean)}(e,t);return r?void 0!==n?o.slice(0,n):o:null},V=r.createContext(null),q=["id","prefixCls","className","showSearch","tagRender","direction","omitDomProps","displayValues","onDisplayValuesChange","emptyOptions","notFoundContent","onClear","mode","disabled","loading","getInputElement","getRawInputElement","open","defaultOpen","onDropdownVisibleChange","activeValue","onActiveValueChange","activeDescendantId","searchValue","autoClearSearchValue","onSearch","onSearchSplit","tokenSeparators","allowClear","suffixIcon","clearIcon","OptionList","animation","transitionName","dropdownStyle","dropdownClassName","dropdownMatchSelectWidth","dropdownRender","dropdownAlign","placement","builtinPlacements","getPopupContainer","showAction","onFocus","onBlur","onKeyUp","onKeyDown","onMouseDown"],G=["value","onChange","removeIcon","placeholder","autoFocus","maxTagCount","maxTagTextLength","maxTagPlaceholder","choiceTransitionName","onInputKeyDown","onPopupScroll","tabIndex"],X=function(e){return"tags"===e||"multiple"===e},U=r.forwardRef(function(e,t){var n,o,m,S,C,Z,O,k,M=e.id,j=e.prefixCls,I=e.className,R=e.showSearch,N=e.tagRender,P=e.direction,T=e.omitDomProps,A=e.displayValues,L=e.onDisplayValuesChange,_=e.emptyOptions,H=e.notFoundContent,B=void 0===H?"Not Found":H,D=e.onClear,U=e.mode,$=e.disabled,K=e.loading,Y=e.getInputElement,Q=e.getRawInputElement,J=e.open,ee=e.defaultOpen,et=e.onDropdownVisibleChange,en=e.activeValue,er=e.onActiveValueChange,eo=e.activeDescendantId,ea=e.searchValue,ei=e.autoClearSearchValue,ec=e.onSearch,el=e.onSearchSplit,es=e.tokenSeparators,eu=e.allowClear,ed=e.suffixIcon,ef=e.clearIcon,ep=e.OptionList,em=e.animation,eg=e.transitionName,eh=e.dropdownStyle,ev=e.dropdownClassName,eb=e.dropdownMatchSelectWidth,ey=e.dropdownRender,ew=e.dropdownAlign,ex=e.placement,eE=e.builtinPlacements,eS=e.getPopupContainer,eC=e.showAction,eZ=void 0===eC?[]:eC,eO=e.onFocus,ek=e.onBlur,eM=e.onKeyUp,ej=e.onKeyDown,eI=e.onMouseDown,eR=(0,d.Z)(e,q),eN=X(U),eP=(void 0!==R?R:eN)||"combobox"===U,eF=(0,s.Z)({},eR);G.forEach(function(e){delete eF[e]}),null==T||T.forEach(function(e){delete eF[e]});var eT=r.useState(!1),eA=(0,u.Z)(eT,2),eL=eA[0],ez=eA[1];r.useEffect(function(){ez((0,h.Z)())},[]);var e_=r.useRef(null),eH=r.useRef(null),eB=r.useRef(null),eD=r.useRef(null),eW=r.useRef(null),eV=r.useRef(!1),eq=function(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:10,t=r.useState(!1),n=(0,u.Z)(t,2),o=n[0],a=n[1],i=r.useRef(null),c=function(){window.clearTimeout(i.current)};return r.useEffect(function(){return c},[]),[o,function(t,n){c(),i.current=window.setTimeout(function(){a(t),n&&n()},e)},c]}(),eG=(0,u.Z)(eq,3),eX=eG[0],eU=eG[1],e$=eG[2];r.useImperativeHandle(t,function(){var e,t;return{focus:null===(e=eD.current)||void 0===e?void 0:e.focus,blur:null===(t=eD.current)||void 0===t?void 0:t.blur,scrollTo:function(e){var t;return null===(t=eW.current)||void 0===t?void 0:t.scrollTo(e)}}});var eK=r.useMemo(function(){if("combobox"!==U)return ea;var e,t=null===(e=A[0])||void 0===e?void 0:e.value;return"string"==typeof t||"number"==typeof t?String(t):""},[ea,U,A]),eY="combobox"===U&&"function"==typeof Y&&Y()||null,eQ="function"==typeof Q&&Q(),eJ=(0,b.x1)(eH,null==eQ||null===(S=eQ.props)||void 0===S?void 0:S.ref),e0=r.useState(!1),e1=(0,u.Z)(e0,2),e2=e1[0],e6=e1[1];(0,g.Z)(function(){e6(!0)},[]);var e5=(0,p.Z)(!1,{defaultValue:ee,value:J}),e4=(0,u.Z)(e5,2),e3=e4[0],e8=e4[1],e9=!!e2&&e3,e7=!B&&_;($||e7&&e9&&"combobox"===U)&&(e9=!1);var te=!e7&&e9,tt=r.useCallback(function(e){var t=void 0!==e?e:!e9;$||(e8(t),e9!==t&&(null==et||et(t)))},[$,e9,e8,et]),tn=r.useMemo(function(){return(es||[]).some(function(e){return["\n","\r\n"].includes(e)})},[es]),tr=r.useContext(V)||{},to=tr.maxCount,ta=tr.rawValues,ti=function(e,t,n){if(!((null==ta?void 0:ta.size)>=to)){var r=!0,o=e;null==er||er(null);var a=W(e,es,to&&to-ta.size),i=n?null:a;return"combobox"!==U&&i&&(o="",null==el||el(i),tt(!1),r=!1),ec&&eK!==o&&ec(o,{source:t?"typing":"effect"}),r}};r.useEffect(function(){e9||eN||"combobox"===U||ti("",!1,!1)},[e9]),r.useEffect(function(){e3&&$&&e8(!1),$&&!eV.current&&eU(!1)},[$]);var tc=E(),tl=(0,u.Z)(tc,2),ts=tl[0],tu=tl[1],td=r.useRef(!1),tf=[];r.useEffect(function(){return function(){tf.forEach(function(e){return clearTimeout(e)}),tf.splice(0,tf.length)}},[]);var tp=r.useState({}),tm=(0,u.Z)(tp,2)[1];eQ&&(Z=function(e){tt(e)}),n=function(){var e;return[e_.current,null===(e=eB.current)||void 0===e?void 0:e.getPopupElement()]},o=!!eQ,(m=r.useRef(null)).current={open:te,triggerOpen:tt,customizedTrigger:o},r.useEffect(function(){function e(e){if(null===(t=m.current)||void 0===t||!t.customizedTrigger){var t,r=e.target;r.shadowRoot&&e.composed&&(r=e.composedPath()[0]||r),m.current.open&&n().filter(function(e){return e}).every(function(e){return!e.contains(r)&&e!==r})&&m.current.triggerOpen(!1)}}return window.addEventListener("mousedown",e),function(){return window.removeEventListener("mousedown",e)}},[]);var tg=r.useMemo(function(){return(0,s.Z)((0,s.Z)({},e),{},{notFoundContent:B,open:e9,triggerOpen:te,id:M,showSearch:eP,multiple:eN,toggleOpen:tt})},[e,B,te,e9,M,eP,eN,tt]),th=!!ed||K;th&&(O=r.createElement(y,{className:a()("".concat(j,"-arrow"),(0,l.Z)({},"".concat(j,"-arrow-loading"),K)),customizeIcon:ed,customizeIconProps:{loading:K,searchValue:eK,open:e9,focused:eX,showSearch:eP}}));var tv=w(j,function(){var e;null==D||D(),null===(e=eD.current)||void 0===e||e.focus(),L([],{type:"clear",values:A}),ti("",!1,!1)},A,eu,ef,$,eK,U),tb=tv.allowClear,ty=tv.clearIcon,tw=r.createElement(ep,{ref:eW}),tx=a()(j,I,(C={},(0,l.Z)(C,"".concat(j,"-focused"),eX),(0,l.Z)(C,"".concat(j,"-multiple"),eN),(0,l.Z)(C,"".concat(j,"-single"),!eN),(0,l.Z)(C,"".concat(j,"-allow-clear"),eu),(0,l.Z)(C,"".concat(j,"-show-arrow"),th),(0,l.Z)(C,"".concat(j,"-disabled"),$),(0,l.Z)(C,"".concat(j,"-loading"),K),(0,l.Z)(C,"".concat(j,"-open"),e9),(0,l.Z)(C,"".concat(j,"-customize-input"),eY),(0,l.Z)(C,"".concat(j,"-show-search"),eP),C)),tE=r.createElement(z,{ref:eB,disabled:$,prefixCls:j,visible:te,popupElement:tw,animation:em,transitionName:eg,dropdownStyle:eh,dropdownClassName:ev,direction:P,dropdownMatchSelectWidth:eb,dropdownRender:ey,dropdownAlign:ew,placement:ex,builtinPlacements:eE,getPopupContainer:eS,empty:_,getTriggerDOMNode:function(){return eH.current},onPopupVisibleChange:Z,onPopupMouseEnter:function(){tm({})}},eQ?r.cloneElement(eQ,{ref:eJ}):r.createElement(F,(0,i.Z)({},e,{domRef:eH,prefixCls:j,inputElement:eY,ref:eD,id:M,showSearch:eP,autoClearSearchValue:ei,mode:U,activeDescendantId:eo,tagRender:N,values:A,open:e9,onToggleOpen:tt,activeValue:en,searchValue:eK,onSearch:ti,onSearchSubmit:function(e){e&&e.trim()&&ec(e,{source:"submit"})},onRemove:function(e){L(A.filter(function(t){return t!==e}),{type:"remove",values:[e]})},tokenWithEnter:tn})));return k=eQ?tE:r.createElement("div",(0,i.Z)({className:tx},eF,{ref:e_,onMouseDown:function(e){var t,n=e.target,r=null===(t=eB.current)||void 0===t?void 0:t.getPopupElement();if(r&&r.contains(n)){var o=setTimeout(function(){var e,t=tf.indexOf(o);-1!==t&&tf.splice(t,1),e$(),eL||r.contains(document.activeElement)||null===(e=eD.current)||void 0===e||e.focus()});tf.push(o)}for(var a=arguments.length,i=Array(a>1?a-1:0),c=1;c=0;i-=1){var l=o[i];if(!l.disabled){o.splice(i,1),a=l;break}}a&&L(o,{type:"remove",values:[a]})}for(var s=arguments.length,u=Array(s>1?s-1:0),d=1;d1?n-1:0),o=1;o=C},[p,C,null==I?void 0:I.size]),B=function(e){e.preventDefault()},D=function(e){var t;null===(t=_.current)||void 0===t||t.scrollTo("number"==typeof e?{index:e}:e)},W=function(e){for(var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:1,n=z.length,r=0;r1&&void 0!==arguments[1]&&arguments[1];U(e);var n={source:t?"keyboard":"mouse"},r=z[e];if(!r){O(null,-1,n);return}O(r.value,e,n)};(0,r.useEffect)(function(){$(!1!==k?W(0):-1)},[z.length,g]);var K=r.useCallback(function(e){return I.has(e)&&"combobox"!==m},[m,(0,c.Z)(I).toString(),I.size]);(0,r.useEffect)(function(){var e,t=setTimeout(function(){if(!p&&f&&1===I.size){var e=Array.from(I)[0],t=z.findIndex(function(t){return t.data.value===e});-1!==t&&($(t),D(t))}});return f&&(null===(e=_.current)||void 0===e||e.scrollTo(void 0)),function(){return clearTimeout(t)}},[f,g]);var en=function(e){void 0!==e&&M(e,{selected:!I.has(e)}),p||h(!1)};if(r.useImperativeHandle(t,function(){return{onKeyDown:function(e){var t=e.which,n=e.ctrlKey;switch(t){case v.Z.N:case v.Z.P:case v.Z.UP:case v.Z.DOWN:var r=0;if(t===v.Z.UP?r=-1:t===v.Z.DOWN?r=1:/(mac\sos|macintosh)/i.test(navigator.appVersion)&&n&&(t===v.Z.N?r=1:t===v.Z.P&&(r=-1)),0!==r){var o=W(X+r,r);D(o),$(o,!0)}break;case v.Z.ENTER:var a,i=z[X];!i||null!=i&&null!==(a=i.data)&&void 0!==a&&a.disabled||H?en(void 0):en(i.value),f&&e.preventDefault();break;case v.Z.ESC:h(!1),f&&e.stopPropagation()}},onKeyUp:function(){},scrollTo:function(e){D(e)}}}),0===z.length)return r.createElement("div",{role:"listbox",id:"".concat(s,"_list"),className:"".concat(L,"-empty"),onMouseDown:B},b);var er=Object.keys(R).map(function(e){return R[e]}),eo=function(e){return e.label};function ea(e,t){return{role:e.group?"presentation":"option",id:"".concat(s,"_list_").concat(t)}}var ei=function(e){var t=z[e];if(!t)return null;var n=t.data||{},o=n.value,a=t.group,c=(0,S.Z)(n,!0),l=eo(t);return t?r.createElement("div",(0,i.Z)({"aria-label":"string"!=typeof l||a?null:l},c,{key:e},ea(t,e),{"aria-selected":K(o)}),o):null},ec={role:"listbox",id:"".concat(s,"_list")};return r.createElement(r.Fragment,null,N&&r.createElement("div",(0,i.Z)({},ec,{style:{height:0,width:0,overflow:"hidden"}}),ei(X-1),ei(X),ei(X+1)),r.createElement(J.Z,{itemKey:"key",ref:_,data:z,height:F,itemHeight:T,fullHeight:!1,onMouseDown:B,onScroll:w,virtual:N,direction:P,innerProps:N?null:ec},function(e,t){var n=e.group,o=e.groupOption,c=e.data,s=e.label,u=e.value,f=c.key;if(n){var p,m,g=null!==(m=c.title)&&void 0!==m?m:et(s)?s.toString():void 0;return r.createElement("div",{className:a()(L,"".concat(L,"-group")),title:g},void 0!==s?s:f)}var h=c.disabled,v=c.title,b=(c.children,c.style),w=c.className,x=(0,d.Z)(c,ee),E=(0,Q.Z)(x,er),C=K(u),Z=h||!C&&H,O="".concat(L,"-option"),k=a()(L,O,w,(p={},(0,l.Z)(p,"".concat(O,"-grouped"),o),(0,l.Z)(p,"".concat(O,"-active"),X===t&&!Z),(0,l.Z)(p,"".concat(O,"-disabled"),Z),(0,l.Z)(p,"".concat(O,"-selected"),C),p)),M=eo(e),I=!j||"function"==typeof j||C,R="number"==typeof M?M:M||u,P=et(R)?R.toString():void 0;return void 0!==v&&(P=v),r.createElement("div",(0,i.Z)({},(0,S.Z)(E),N?{}:ea(e,t),{"aria-selected":C,className:k,title:P,onMouseMove:function(){X===t||Z||$(t)},onClick:function(){Z||en(u)},style:b}),r.createElement("div",{className:"".concat(O,"-content")},"function"==typeof A?A(e,{index:t}):R),r.isValidElement(j)||C,I&&r.createElement(y,{className:"".concat(L,"-option-state"),customizeIcon:j,customizeIconProps:{value:u,disabled:Z,isSelected:C}},C?"✓":null))}))}),er=function(e,t){var n=r.useRef({values:new Map,options:new Map});return[r.useMemo(function(){var r=n.current,o=r.values,a=r.options,i=e.map(function(e){if(void 0===e.label){var t;return(0,s.Z)((0,s.Z)({},e),{},{label:null===(t=o.get(e.value))||void 0===t?void 0:t.label})}return e}),c=new Map,l=new Map;return i.forEach(function(e){c.set(e.value,e),l.set(e.value,t.get(e.value)||a.get(e.value))}),n.current.values=c,n.current.options=l,i},[e,t]),r.useCallback(function(e){return t.get(e)||n.current.options.get(e)},[t])]};function eo(e,t){return O(e).join("").toUpperCase().includes(t)}var ea=n(94981),ei=0,ec=(0,ea.Z)(),el=n(45287),es=["children","value"],eu=["children"];function ed(e){var t=r.useRef();return t.current=e,r.useCallback(function(){return t.current.apply(t,arguments)},[])}var ef=["id","mode","prefixCls","backfill","fieldNames","inputValue","searchValue","onSearch","autoClearSearchValue","onSelect","onDeselect","dropdownMatchSelectWidth","filterOption","filterSort","optionFilterProp","optionLabelProp","options","optionRender","children","defaultActiveFirstOption","menuItemSelectedIcon","virtual","direction","listHeight","listItemHeight","value","defaultValue","labelInValue","onChange","maxCount"],ep=["inputValue"],em=r.forwardRef(function(e,t){var n,o,a,m,g,h=e.id,v=e.mode,b=e.prefixCls,y=e.backfill,w=e.fieldNames,x=e.inputValue,E=e.searchValue,S=e.onSearch,C=e.autoClearSearchValue,Z=void 0===C||C,k=e.onSelect,M=e.onDeselect,j=e.dropdownMatchSelectWidth,I=void 0===j||j,R=e.filterOption,N=e.filterSort,P=e.optionFilterProp,F=e.optionLabelProp,T=e.options,A=e.optionRender,L=e.children,z=e.defaultActiveFirstOption,_=e.menuItemSelectedIcon,W=e.virtual,q=e.direction,G=e.listHeight,$=void 0===G?200:G,K=e.listItemHeight,Y=void 0===K?20:K,Q=e.value,J=e.defaultValue,ee=e.labelInValue,et=e.onChange,ea=e.maxCount,em=(0,d.Z)(e,ef),eg=(n=r.useState(),a=(o=(0,u.Z)(n,2))[0],m=o[1],r.useEffect(function(){var e;m("rc_select_".concat((ec?(e=ei,ei+=1):e="TEST_OR_SSR",e)))},[]),h||a),eh=X(v),ev=!!(!T&&L),eb=r.useMemo(function(){return(void 0!==R||"combobox"!==v)&&R},[R,v]),ey=r.useMemo(function(){return B(w,ev)},[JSON.stringify(w),ev]),ew=(0,p.Z)("",{value:void 0!==E?E:x,postState:function(e){return e||""}}),ex=(0,u.Z)(ew,2),eE=ex[0],eS=ex[1],eC=r.useMemo(function(){var e=T;T||(e=function e(t){var n=arguments.length>1&&void 0!==arguments[1]&&arguments[1];return(0,el.Z)(t).map(function(t,o){if(!r.isValidElement(t)||!t.type)return null;var a,i,c,l,u,f=t.type.isSelectOptGroup,p=t.key,m=t.props,g=m.children,h=(0,d.Z)(m,eu);return n||!f?(a=t.key,c=(i=t.props).children,l=i.value,u=(0,d.Z)(i,es),(0,s.Z)({key:a,value:void 0!==l?l:a,children:c},u)):(0,s.Z)((0,s.Z)({key:"__RC_SELECT_GRP__".concat(null===p?o:p,"__"),label:p},h),{},{options:e(g)})}).filter(function(e){return e})}(L));var t=new Map,n=new Map,o=function(e,t,n){n&&"string"==typeof n&&e.set(t[n],t)};return function e(r){for(var a=arguments.length>1&&void 0!==arguments[1]&&arguments[1],i=0;i1&&void 0!==arguments[1]?arguments[1]:{},n=t.fieldNames,r=t.childrenAsData,o=[],a=B(n,!1),i=a.label,c=a.value,l=a.options,s=a.groupLabel;return!function e(t,n){Array.isArray(t)&&t.forEach(function(t){if(!n&&l in t){var a=t[s];void 0===a&&r&&(a=t.label),o.push({key:H(t,o.length),group:!0,data:t,label:a}),e(t[l],!0)}else{var u=t[c];o.push({key:H(t,o.length),groupOption:n,data:t,label:t[i],value:u})}})}(e,!1),o}(eD,{fieldNames:ey,childrenAsData:ev})},[eD,ey,ev]),eV=function(e){var t=eM(e);if(eN(t),et&&(t.length!==eT.length||t.some(function(e,t){var n;return(null===(n=eT[t])||void 0===n?void 0:n.value)!==(null==e?void 0:e.value)}))){var n=ee?t:t.map(function(e){return e.value}),r=t.map(function(e){return D(eA(e.value))});et(eh?n:n[0],eh?r:r[0])}},eq=r.useState(null),eG=(0,u.Z)(eq,2),eX=eG[0],eU=eG[1],e$=r.useState(0),eK=(0,u.Z)(e$,2),eY=eK[0],eQ=eK[1],eJ=void 0!==z?z:"combobox"!==v,e0=r.useCallback(function(e,t){var n=arguments.length>2&&void 0!==arguments[2]?arguments[2]:{},r=n.source;eQ(t),y&&"combobox"===v&&null!==e&&"keyboard"===(void 0===r?"keyboard":r)&&eU(String(e))},[y,v]),e1=function(e,t,n){var r=function(){var t,n=eA(e);return[ee?{label:null==n?void 0:n[ey.label],value:e,key:null!==(t=null==n?void 0:n.key)&&void 0!==t?t:e}:e,D(n)]};if(t&&k){var o=r(),a=(0,u.Z)(o,2);k(a[0],a[1])}else if(!t&&M&&"clear"!==n){var i=r(),c=(0,u.Z)(i,2);M(c[0],c[1])}},e2=ed(function(e,t){var n=!eh||t.selected;eV(n?eh?[].concat((0,c.Z)(eT),[e]):[e]:eT.filter(function(t){return t.value!==e})),e1(e,n),"combobox"===v?eU(""):(!X||Z)&&(eS(""),eU(""))}),e6=r.useMemo(function(){var e=!1!==W&&!1!==I;return(0,s.Z)((0,s.Z)({},eC),{},{flattenOptions:eW,onActiveValue:e0,defaultActiveFirstOption:eJ,onSelect:e2,menuItemSelectedIcon:_,rawValues:ez,fieldNames:ey,virtual:e,direction:q,listHeight:$,listItemHeight:Y,childrenAsData:ev,maxCount:ea,optionRender:A})},[ea,eC,eW,e0,eJ,e2,_,ez,ey,W,I,q,$,Y,ev,A]);return r.createElement(V.Provider,{value:e6},r.createElement(U,(0,i.Z)({},em,{id:eg,prefixCls:void 0===b?"rc-select":b,ref:t,omitDomProps:ep,mode:v,displayValues:eL,onDisplayValuesChange:function(e,t){eV(e);var n=t.type,r=t.values;("remove"===n||"clear"===n)&&r.forEach(function(e){e1(e.value,!1,n)})},direction:q,searchValue:eE,onSearch:function(e,t){if(eS(e),eU(null),"submit"===t.source){var n=(e||"").trim();n&&(eV(Array.from(new Set([].concat((0,c.Z)(ez),[n])))),e1(n,!0),eS(""));return}"blur"!==t.source&&("combobox"===v&&eV(e),null==S||S(e))},autoClearSearchValue:Z,onSearchSplit:function(e){var t=e;"tags"!==v&&(t=e.map(function(e){var t=eO.get(e);return null==t?void 0:t.value}).filter(function(e){return void 0!==e}));var n=Array.from(new Set([].concat((0,c.Z)(ez),(0,c.Z)(t))));eV(n),n.forEach(function(e){e1(e,!0)})},dropdownMatchSelectWidth:I,OptionList:en,emptyOptions:!eW.length,activeValue:eX,activeDescendantId:"".concat(eg,"_list_").concat(eY)})))});em.Option=K,em.OptGroup=$;var eg=n(62236),eh=n(68710),ev=n(93942),eb=n(12757),ey=n(71744),ew=n(91086),ex=n(86586),eE=n(64024),eS=n(33759),eC=n(39109),eZ=n(56250),eO=n(65658),ek=n(29961);let eM=e=>{let t={overflow:{adjustX:!0,adjustY:!0,shiftY:!0},htmlRegion:"scroll"===e?"scroll":"visible",dynamicInset:!0};return{bottomLeft:Object.assign(Object.assign({},t),{points:["tl","bl"],offset:[0,4]}),bottomRight:Object.assign(Object.assign({},t),{points:["tr","br"],offset:[0,4]}),topLeft:Object.assign(Object.assign({},t),{points:["bl","tl"],offset:[0,-4]}),topRight:Object.assign(Object.assign({},t),{points:["br","tr"],offset:[0,-4]})}};var ej=n(12918),eI=n(17691),eR=n(80669),eN=n(3104),eP=n(18544),eF=n(29382);let eT=e=>{let{optionHeight:t,optionFontSize:n,optionLineHeight:r,optionPadding:o}=e;return{position:"relative",display:"block",minHeight:t,padding:o,color:e.colorText,fontWeight:"normal",fontSize:n,lineHeight:r,boxSizing:"border-box"}};var eA=e=>{let{antCls:t,componentCls:n}=e,r="".concat(n,"-item"),o="&".concat(t,"-slide-up-enter").concat(t,"-slide-up-enter-active"),a="&".concat(t,"-slide-up-appear").concat(t,"-slide-up-appear-active"),i="&".concat(t,"-slide-up-leave").concat(t,"-slide-up-leave-active"),c="".concat(n,"-dropdown-placement-");return[{["".concat(n,"-dropdown")]:Object.assign(Object.assign({},(0,ej.Wf)(e)),{position:"absolute",top:-9999,zIndex:e.zIndexPopup,boxSizing:"border-box",padding:e.paddingXXS,overflow:"hidden",fontSize:e.fontSize,fontVariant:"initial",backgroundColor:e.colorBgElevated,borderRadius:e.borderRadiusLG,outline:"none",boxShadow:e.boxShadowSecondary,["\n ".concat(o).concat(c,"bottomLeft,\n ").concat(a).concat(c,"bottomLeft\n ")]:{animationName:eP.fJ},["\n ".concat(o).concat(c,"topLeft,\n ").concat(a).concat(c,"topLeft,\n ").concat(o).concat(c,"topRight,\n ").concat(a).concat(c,"topRight\n ")]:{animationName:eP.Qt},["".concat(i).concat(c,"bottomLeft")]:{animationName:eP.Uw},["\n ".concat(i).concat(c,"topLeft,\n ").concat(i).concat(c,"topRight\n ")]:{animationName:eP.ly},"&-hidden":{display:"none"},["".concat(r)]:Object.assign(Object.assign({},eT(e)),{cursor:"pointer",transition:"background ".concat(e.motionDurationSlow," ease"),borderRadius:e.borderRadiusSM,"&-group":{color:e.colorTextDescription,fontSize:e.fontSizeSM,cursor:"default"},"&-option":{display:"flex","&-content":Object.assign({flex:"auto"},ej.vS),"&-state":{flex:"none",display:"flex",alignItems:"center"},["&-active:not(".concat(r,"-option-disabled)")]:{backgroundColor:e.optionActiveBg},["&-selected:not(".concat(r,"-option-disabled)")]:{color:e.optionSelectedColor,fontWeight:e.optionSelectedFontWeight,backgroundColor:e.optionSelectedBg,["".concat(r,"-option-state")]:{color:e.colorPrimary},["&:has(+ ".concat(r,"-option-selected:not(").concat(r,"-option-disabled))")]:{borderEndStartRadius:0,borderEndEndRadius:0,["& + ".concat(r,"-option-selected:not(").concat(r,"-option-disabled)")]:{borderStartStartRadius:0,borderStartEndRadius:0}}},"&-disabled":{["&".concat(r,"-option-selected")]:{backgroundColor:e.colorBgContainerDisabled},color:e.colorTextDisabled,cursor:"not-allowed"},"&-grouped":{paddingInlineStart:e.calc(e.controlPaddingHorizontal).mul(2).equal()}}}),"&-rtl":{direction:"rtl"}})},(0,eP.oN)(e,"slide-up"),(0,eP.oN)(e,"slide-down"),(0,eF.Fm)(e,"move-up"),(0,eF.Fm)(e,"move-down")]},eL=n(352);let ez=e=>{let{multipleSelectItemHeight:t,selectHeight:n,lineWidth:r}=e;return e.calc(n).sub(t).div(2).sub(r).equal()};function e_(e,t){let{componentCls:n,iconCls:r}=e,o="".concat(n,"-selection-overflow"),a=e.multipleSelectItemHeight,i=ez(e),c=t?"".concat(n,"-").concat(t):"";return{["".concat(n,"-multiple").concat(c)]:{fontSize:e.fontSize,[o]:{position:"relative",display:"flex",flex:"auto",flexWrap:"wrap",maxWidth:"100%","&-item":{flex:"none",alignSelf:"center",maxWidth:"100%",display:"inline-flex"}},["".concat(n,"-selector")]:{display:"flex",flexWrap:"wrap",alignItems:"center",height:"100%",paddingInline:e.calc(2).mul(2).equal(),paddingBlock:e.calc(i).sub(2).equal(),borderRadius:e.borderRadius,["".concat(n,"-show-search&")]:{cursor:"text"},["".concat(n,"-disabled&")]:{background:e.multipleSelectorBgDisabled,cursor:"not-allowed"},"&:after":{display:"inline-block",width:0,margin:"".concat((0,eL.bf)(2)," 0"),lineHeight:(0,eL.bf)(a),visibility:"hidden",content:'"\\a0"'}},["\n &".concat(n,"-show-arrow ").concat(n,"-selector,\n &").concat(n,"-allow-clear ").concat(n,"-selector\n ")]:{paddingInlineEnd:e.calc(e.fontSizeIcon).add(e.controlPaddingHorizontal).equal()},["".concat(n,"-selection-item")]:{display:"flex",alignSelf:"center",flex:"none",boxSizing:"border-box",maxWidth:"100%",height:a,marginTop:2,marginBottom:2,lineHeight:(0,eL.bf)(e.calc(a).sub(e.calc(e.lineWidth).mul(2)).equal()),borderRadius:e.borderRadiusSM,cursor:"default",transition:"font-size ".concat(e.motionDurationSlow,", line-height ").concat(e.motionDurationSlow,", height ").concat(e.motionDurationSlow),marginInlineEnd:e.calc(2).mul(2).equal(),paddingInlineStart:e.paddingXS,paddingInlineEnd:e.calc(e.paddingXS).div(2).equal(),["".concat(n,"-disabled&")]:{color:e.multipleItemColorDisabled,borderColor:e.multipleItemBorderColorDisabled,cursor:"not-allowed"},"&-content":{display:"inline-block",marginInlineEnd:e.calc(e.paddingXS).div(2).equal(),overflow:"hidden",whiteSpace:"pre",textOverflow:"ellipsis"},"&-remove":Object.assign(Object.assign({},(0,ej.Ro)()),{display:"inline-flex",alignItems:"center",color:e.colorIcon,fontWeight:"bold",fontSize:10,lineHeight:"inherit",cursor:"pointer",["> ".concat(r)]:{verticalAlign:"-0.2em"},"&:hover":{color:e.colorIconHover}})},["".concat(o,"-item + ").concat(o,"-item")]:{["".concat(n,"-selection-search")]:{marginInlineStart:0}},["".concat(o,"-item-suffix")]:{height:"100%"},["".concat(n,"-selection-search")]:{display:"inline-flex",position:"relative",maxWidth:"100%",marginInlineStart:e.calc(e.inputPaddingHorizontalBase).sub(i).equal(),"\n &-input,\n &-mirror\n ":{height:a,fontFamily:e.fontFamily,lineHeight:(0,eL.bf)(a),transition:"all ".concat(e.motionDurationSlow)},"&-input":{width:"100%",minWidth:4.1},"&-mirror":{position:"absolute",top:0,insetInlineStart:0,insetInlineEnd:"auto",zIndex:999,whiteSpace:"pre",visibility:"hidden"}},["".concat(n,"-selection-placeholder")]:{position:"absolute",top:"50%",insetInlineStart:e.inputPaddingHorizontalBase,insetInlineEnd:e.inputPaddingHorizontalBase,transform:"translateY(-50%)",transition:"all ".concat(e.motionDurationSlow)}}}}var eH=e=>{let{componentCls:t}=e,n=(0,eN.TS)(e,{selectHeight:e.controlHeightSM,multipleSelectItemHeight:e.controlHeightXS,borderRadius:e.borderRadiusSM,borderRadiusSM:e.borderRadiusXS}),r=(0,eN.TS)(e,{fontSize:e.fontSizeLG,selectHeight:e.controlHeightLG,multipleSelectItemHeight:e.multipleItemHeightLG,borderRadius:e.borderRadiusLG,borderRadiusSM:e.borderRadius});return[e_(e),e_(n,"sm"),{["".concat(t,"-multiple").concat(t,"-sm")]:{["".concat(t,"-selection-placeholder")]:{insetInline:e.calc(e.controlPaddingHorizontalSM).sub(e.lineWidth).equal()},["".concat(t,"-selection-search")]:{marginInlineStart:2}}},e_(r,"lg")]};function eB(e,t){let{componentCls:n,inputPaddingHorizontalBase:r,borderRadius:o}=e,a=e.calc(e.controlHeight).sub(e.calc(e.lineWidth).mul(2)).equal(),i=t?"".concat(n,"-").concat(t):"";return{["".concat(n,"-single").concat(i)]:{fontSize:e.fontSize,height:e.controlHeight,["".concat(n,"-selector")]:Object.assign(Object.assign({},(0,ej.Wf)(e,!0)),{display:"flex",borderRadius:o,["".concat(n,"-selection-search")]:{position:"absolute",top:0,insetInlineStart:r,insetInlineEnd:r,bottom:0,"&-input":{width:"100%",WebkitAppearance:"textfield"}},["\n ".concat(n,"-selection-item,\n ").concat(n,"-selection-placeholder\n ")]:{padding:0,lineHeight:(0,eL.bf)(a),transition:"all ".concat(e.motionDurationSlow,", visibility 0s"),alignSelf:"center"},["".concat(n,"-selection-placeholder")]:{transition:"none",pointerEvents:"none"},[["&:after","".concat(n,"-selection-item:empty:after"),"".concat(n,"-selection-placeholder:empty:after")].join(",")]:{display:"inline-block",width:0,visibility:"hidden",content:'"\\a0"'}}),["\n &".concat(n,"-show-arrow ").concat(n,"-selection-item,\n &").concat(n,"-show-arrow ").concat(n,"-selection-placeholder\n ")]:{paddingInlineEnd:e.showArrowPaddingInlineEnd},["&".concat(n,"-open ").concat(n,"-selection-item")]:{color:e.colorTextPlaceholder},["&:not(".concat(n,"-customize-input)")]:{["".concat(n,"-selector")]:{width:"100%",height:"100%",padding:"0 ".concat((0,eL.bf)(r)),["".concat(n,"-selection-search-input")]:{height:a},"&:after":{lineHeight:(0,eL.bf)(a)}}},["&".concat(n,"-customize-input")]:{["".concat(n,"-selector")]:{"&:after":{display:"none"},["".concat(n,"-selection-search")]:{position:"static",width:"100%"},["".concat(n,"-selection-placeholder")]:{position:"absolute",insetInlineStart:0,insetInlineEnd:0,padding:"0 ".concat((0,eL.bf)(r)),"&:after":{display:"none"}}}}}}}let eD=(e,t)=>{let{componentCls:n,antCls:r,controlOutlineWidth:o}=e;return{["&:not(".concat(n,"-customize-input) ").concat(n,"-selector")]:{border:"".concat((0,eL.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(t.borderColor),background:e.selectorBg},["&:not(".concat(n,"-disabled):not(").concat(n,"-customize-input):not(").concat(r,"-pagination-size-changer)")]:{["&:hover ".concat(n,"-selector")]:{borderColor:t.hoverBorderHover},["".concat(n,"-focused& ").concat(n,"-selector")]:{borderColor:t.activeBorderColor,boxShadow:"0 0 0 ".concat((0,eL.bf)(o)," ").concat(t.activeShadowColor),outline:0}}}},eW=(e,t)=>({["&".concat(e.componentCls,"-status-").concat(t.status)]:Object.assign({},eD(e,t))}),eV=e=>({"&-outlined":Object.assign(Object.assign(Object.assign(Object.assign({},eD(e,{borderColor:e.colorBorder,hoverBorderHover:e.colorPrimaryHover,activeBorderColor:e.colorPrimary,activeShadowColor:e.controlOutline})),eW(e,{status:"error",borderColor:e.colorError,hoverBorderHover:e.colorErrorHover,activeBorderColor:e.colorError,activeShadowColor:e.colorErrorOutline})),eW(e,{status:"warning",borderColor:e.colorWarning,hoverBorderHover:e.colorWarningHover,activeBorderColor:e.colorWarning,activeShadowColor:e.colorWarningOutline})),{["&".concat(e.componentCls,"-disabled")]:{["&:not(".concat(e.componentCls,"-customize-input) ").concat(e.componentCls,"-selector")]:{background:e.colorBgContainerDisabled,color:e.colorTextDisabled}},["&".concat(e.componentCls,"-multiple ").concat(e.componentCls,"-selection-item")]:{background:e.multipleItemBg,border:"".concat((0,eL.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.multipleItemBorderColor)}})}),eq=(e,t)=>{let{componentCls:n,antCls:r}=e;return{["&:not(".concat(n,"-customize-input) ").concat(n,"-selector")]:{background:t.bg,border:"".concat((0,eL.bf)(e.lineWidth)," ").concat(e.lineType," transparent"),color:t.color},["&:not(".concat(n,"-disabled):not(").concat(n,"-customize-input):not(").concat(r,"-pagination-size-changer)")]:{["&:hover ".concat(n,"-selector")]:{background:t.hoverBg},["".concat(n,"-focused& ").concat(n,"-selector")]:{background:e.selectorBg,borderColor:t.activeBorderColor,outline:0}}}},eG=(e,t)=>({["&".concat(e.componentCls,"-status-").concat(t.status)]:Object.assign({},eq(e,t))}),eX=e=>({"&-filled":Object.assign(Object.assign(Object.assign(Object.assign({},eq(e,{bg:e.colorFillTertiary,hoverBg:e.colorFillSecondary,activeBorderColor:e.colorPrimary,color:e.colorText})),eG(e,{status:"error",bg:e.colorErrorBg,hoverBg:e.colorErrorBgHover,activeBorderColor:e.colorError,color:e.colorError})),eG(e,{status:"warning",bg:e.colorWarningBg,hoverBg:e.colorWarningBgHover,activeBorderColor:e.colorWarning,color:e.colorWarning})),{["&".concat(e.componentCls,"-disabled")]:{["&:not(".concat(e.componentCls,"-customize-input) ").concat(e.componentCls,"-selector")]:{borderColor:e.colorBorder,background:e.colorBgContainerDisabled,color:e.colorTextDisabled}},["&".concat(e.componentCls,"-multiple ").concat(e.componentCls,"-selection-item")]:{background:e.colorBgContainer,border:"".concat((0,eL.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.colorSplit)}})}),eU=e=>({"&-borderless":{["".concat(e.componentCls,"-selector")]:{background:"transparent",borderColor:"transparent"},["&".concat(e.componentCls,"-disabled")]:{["&:not(".concat(e.componentCls,"-customize-input) ").concat(e.componentCls,"-selector")]:{color:e.colorTextDisabled}},["&".concat(e.componentCls,"-multiple ").concat(e.componentCls,"-selection-item")]:{background:e.multipleItemBg,border:"".concat((0,eL.bf)(e.lineWidth)," ").concat(e.lineType," ").concat(e.multipleItemBorderColor)}}});var e$=e=>({[e.componentCls]:Object.assign(Object.assign(Object.assign({},eV(e)),eX(e)),eU(e))});let eK=e=>{let{componentCls:t}=e;return{position:"relative",transition:"all ".concat(e.motionDurationMid," ").concat(e.motionEaseInOut),input:{cursor:"pointer"},["".concat(t,"-show-search&")]:{cursor:"text",input:{cursor:"auto",color:"inherit",height:"100%"}},["".concat(t,"-disabled&")]:{cursor:"not-allowed",input:{cursor:"not-allowed"}}}},eY=e=>{let{componentCls:t}=e;return{["".concat(t,"-selection-search-input")]:{margin:0,padding:0,background:"transparent",border:"none",outline:"none",appearance:"none",fontFamily:"inherit","&::-webkit-search-cancel-button":{display:"none","-webkit-appearance":"none"}}}},eQ=e=>{let{antCls:t,componentCls:n,inputPaddingHorizontalBase:r,iconCls:o}=e;return{[n]:Object.assign(Object.assign({},(0,ej.Wf)(e)),{position:"relative",display:"inline-block",cursor:"pointer",["&:not(".concat(n,"-customize-input) ").concat(n,"-selector")]:Object.assign(Object.assign({},eK(e)),eY(e)),["".concat(n,"-selection-item")]:Object.assign(Object.assign({flex:1,fontWeight:"normal",position:"relative",userSelect:"none"},ej.vS),{["> ".concat(t,"-typography")]:{display:"inline"}}),["".concat(n,"-selection-placeholder")]:Object.assign(Object.assign({},ej.vS),{flex:1,color:e.colorTextPlaceholder,pointerEvents:"none"}),["".concat(n,"-arrow")]:Object.assign(Object.assign({},(0,ej.Ro)()),{position:"absolute",top:"50%",insetInlineStart:"auto",insetInlineEnd:r,height:e.fontSizeIcon,marginTop:e.calc(e.fontSizeIcon).mul(-1).div(2).equal(),color:e.colorTextQuaternary,fontSize:e.fontSizeIcon,lineHeight:1,textAlign:"center",pointerEvents:"none",display:"flex",alignItems:"center",transition:"opacity ".concat(e.motionDurationSlow," ease"),[o]:{verticalAlign:"top",transition:"transform ".concat(e.motionDurationSlow),"> svg":{verticalAlign:"top"},["&:not(".concat(n,"-suffix)")]:{pointerEvents:"auto"}},["".concat(n,"-disabled &")]:{cursor:"not-allowed"},"> *:not(:last-child)":{marginInlineEnd:8}}),["".concat(n,"-clear")]:{position:"absolute",top:"50%",insetInlineStart:"auto",insetInlineEnd:r,zIndex:1,display:"inline-block",width:e.fontSizeIcon,height:e.fontSizeIcon,marginTop:e.calc(e.fontSizeIcon).mul(-1).div(2).equal(),color:e.colorTextQuaternary,fontSize:e.fontSizeIcon,fontStyle:"normal",lineHeight:1,textAlign:"center",textTransform:"none",cursor:"pointer",opacity:0,transition:"color ".concat(e.motionDurationMid," ease, opacity ").concat(e.motionDurationSlow," ease"),textRendering:"auto","&:before":{display:"block"},"&:hover":{color:e.colorTextTertiary}},"&:hover":{["".concat(n,"-clear")]:{opacity:1},["".concat(n,"-arrow:not(:last-child)")]:{opacity:0}}}),["".concat(n,"-has-feedback")]:{["".concat(n,"-clear")]:{insetInlineEnd:e.calc(r).add(e.fontSize).add(e.paddingXS).equal()}}}},eJ=e=>{let{componentCls:t}=e;return[{[t]:{["&".concat(t,"-in-form-item")]:{width:"100%"}}},eQ(e),function(e){let{componentCls:t}=e,n=e.calc(e.controlPaddingHorizontalSM).sub(e.lineWidth).equal();return[eB(e),eB((0,eN.TS)(e,{controlHeight:e.controlHeightSM,borderRadius:e.borderRadiusSM}),"sm"),{["".concat(t,"-single").concat(t,"-sm")]:{["&:not(".concat(t,"-customize-input)")]:{["".concat(t,"-selection-search")]:{insetInlineStart:n,insetInlineEnd:n},["".concat(t,"-selector")]:{padding:"0 ".concat((0,eL.bf)(n))},["&".concat(t,"-show-arrow ").concat(t,"-selection-search")]:{insetInlineEnd:e.calc(n).add(e.calc(e.fontSize).mul(1.5)).equal()},["\n &".concat(t,"-show-arrow ").concat(t,"-selection-item,\n &").concat(t,"-show-arrow ").concat(t,"-selection-placeholder\n ")]:{paddingInlineEnd:e.calc(e.fontSize).mul(1.5).equal()}}}},eB((0,eN.TS)(e,{controlHeight:e.singleItemHeightLG,fontSize:e.fontSizeLG,borderRadius:e.borderRadiusLG}),"lg")]}(e),eH(e),eA(e),{["".concat(t,"-rtl")]:{direction:"rtl"}},(0,eI.c)(e,{borderElCls:"".concat(t,"-selector"),focusElCls:"".concat(t,"-focused")})]};var e0=(0,eR.I$)("Select",(e,t)=>{let{rootPrefixCls:n}=t,r=(0,eN.TS)(e,{rootPrefixCls:n,inputPaddingHorizontalBase:e.calc(e.paddingSM).sub(1).equal(),multipleSelectItemHeight:e.multipleItemHeight,selectHeight:e.controlHeight});return[eJ(r),e$(r)]},e=>{let{fontSize:t,lineHeight:n,controlHeight:r,controlPaddingHorizontal:o,zIndexPopupBase:a,colorText:i,fontWeightStrong:c,controlItemBgActive:l,controlItemBgHover:s,colorBgContainer:u,colorFillSecondary:d,controlHeightLG:f,controlHeightSM:p,colorBgContainerDisabled:m,colorTextDisabled:g}=e;return{zIndexPopup:a+50,optionSelectedColor:i,optionSelectedFontWeight:c,optionSelectedBg:l,optionActiveBg:s,optionPadding:"".concat((r-t*n)/2,"px ").concat(o,"px"),optionFontSize:t,optionLineHeight:n,optionHeight:r,selectorBg:u,clearBg:u,singleItemHeightLG:f,multipleItemBg:d,multipleItemBorderColor:"transparent",multipleItemHeight:p,multipleItemHeightLG:r,multipleSelectorBgDisabled:m,multipleItemColorDisabled:g,multipleItemBorderColorDisabled:"transparent",showArrowPaddingInlineEnd:Math.ceil(1.25*e.fontSize)}},{unitless:{optionLineHeight:!0,optionSelectedFontWeight:!0}}),e1=n(9738),e2=n(39725),e6=n(49638),e5=n(70464),e4=n(61935),e3=n(29436),e8=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let e9="SECRET_COMBOBOX_MODE_DO_NOT_USE",e7=r.forwardRef((e,t)=>{var n,o,i;let c;let{prefixCls:l,bordered:s,className:u,rootClassName:d,getPopupContainer:f,popupClassName:p,dropdownClassName:m,listHeight:g=256,placement:h,listItemHeight:v,size:b,disabled:y,notFoundContent:w,status:x,builtinPlacements:E,dropdownMatchSelectWidth:S,popupMatchSelectWidth:C,direction:Z,style:O,allowClear:k,variant:M,dropdownStyle:j,transitionName:I,tagRender:R,maxCount:N}=e,P=e8(e,["prefixCls","bordered","className","rootClassName","getPopupContainer","popupClassName","dropdownClassName","listHeight","placement","listItemHeight","size","disabled","notFoundContent","status","builtinPlacements","dropdownMatchSelectWidth","popupMatchSelectWidth","direction","style","allowClear","variant","dropdownStyle","transitionName","tagRender","maxCount"]),{getPopupContainer:F,getPrefixCls:T,renderEmpty:A,direction:L,virtual:z,popupMatchSelectWidth:_,popupOverflow:H,select:B}=r.useContext(ey.E_),[,D]=(0,ek.ZP)(),W=null!=v?v:null==D?void 0:D.controlHeight,V=T("select",l),q=T(),G=null!=Z?Z:L,{compactSize:X,compactItemClassnames:U}=(0,eO.ri)(V,G),[$,K]=(0,eZ.Z)(M,s),Y=(0,eE.Z)(V),[J,ee,et]=e0(V,Y),en=r.useMemo(()=>{let{mode:t}=e;return"combobox"===t?void 0:t===e9?"combobox":t},[e.mode]),er="multiple"===en||"tags"===en,eo=(o=e.suffixIcon,void 0!==(i=e.showArrow)?i:null!==o),ea=null!==(n=null!=C?C:S)&&void 0!==n?n:_,{status:ei,hasFeedback:ec,isFormItemInput:el,feedbackIcon:es}=r.useContext(eC.aM),eu=(0,eb.F)(ei,x);c=void 0!==w?w:"combobox"===en?null:(null==A?void 0:A("Select"))||r.createElement(ew.Z,{componentName:"Select"});let{suffixIcon:ed,itemIcon:ef,removeIcon:ep,clearIcon:ev}=function(e){let{suffixIcon:t,clearIcon:n,menuItemSelectedIcon:o,removeIcon:a,loading:i,multiple:c,hasFeedback:l,prefixCls:s,showSuffixIcon:u,feedbackIcon:d,showArrow:f,componentName:p}=e,m=null!=n?n:r.createElement(e2.Z,null),g=e=>null!==t||l||f?r.createElement(r.Fragment,null,!1!==u&&e,l&&d):null,h=null;if(void 0!==t)h=g(t);else if(i)h=g(r.createElement(e4.Z,{spin:!0}));else{let e="".concat(s,"-suffix");h=t=>{let{open:n,showSearch:o}=t;return n&&o?g(r.createElement(e3.Z,{className:e})):g(r.createElement(e5.Z,{className:e}))}}let v=null;return v=void 0!==o?o:c?r.createElement(e1.Z,null):null,{clearIcon:m,suffixIcon:h,itemIcon:v,removeIcon:void 0!==a?a:r.createElement(e6.Z,null)}}(Object.assign(Object.assign({},P),{multiple:er,hasFeedback:ec,feedbackIcon:es,showSuffixIcon:eo,prefixCls:V,componentName:"Select"})),ej=(0,Q.Z)(P,["suffixIcon","itemIcon"]),eI=a()(p||m,{["".concat(V,"-dropdown-").concat(G)]:"rtl"===G},d,et,Y,ee),eR=(0,eS.Z)(e=>{var t;return null!==(t=null!=b?b:X)&&void 0!==t?t:e}),eN=r.useContext(ex.Z),eP=a()({["".concat(V,"-lg")]:"large"===eR,["".concat(V,"-sm")]:"small"===eR,["".concat(V,"-rtl")]:"rtl"===G,["".concat(V,"-").concat($)]:K,["".concat(V,"-in-form-item")]:el},(0,eb.Z)(V,eu,ec),U,null==B?void 0:B.className,u,d,et,Y,ee),eF=r.useMemo(()=>void 0!==h?h:"rtl"===G?"bottomRight":"bottomLeft",[h,G]),[eT]=(0,eg.Cn)("SelectLike",null==j?void 0:j.zIndex);return J(r.createElement(em,Object.assign({ref:t,virtual:z,showSearch:null==B?void 0:B.showSearch},ej,{style:Object.assign(Object.assign({},null==B?void 0:B.style),O),dropdownMatchSelectWidth:ea,transitionName:(0,eh.m)(q,"slide-up",I),builtinPlacements:E||eM(H),listHeight:g,listItemHeight:W,mode:en,prefixCls:V,placement:eF,direction:G,suffixIcon:ed,menuItemSelectedIcon:ef,removeIcon:ep,allowClear:!0===k?{clearIcon:ev}:k,notFoundContent:c,className:eP,getPopupContainer:f||F,dropdownClassName:eI,disabled:null!=y?y:eN,dropdownStyle:Object.assign(Object.assign({},j),{zIndex:eT}),maxCount:er?N:void 0,tagRender:er?R:void 0})))}),te=(0,ev.Z)(e7);e7.SECRET_COMBOBOX_MODE_DO_NOT_USE=e9,e7.Option=K,e7.OptGroup=$,e7._InternalPanelDoNotUseOrYouWillBeFired=te;var tt=e7},65658:function(e,t,n){"use strict";n.d(t,{BR:function(){return p},ri:function(){return f}});var r=n(36760),o=n.n(r),a=n(45287),i=n(2265),c=n(71744),l=n(33759),s=n(4924),u=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let d=i.createContext(null),f=(e,t)=>{let n=i.useContext(d),r=i.useMemo(()=>{if(!n)return"";let{compactDirection:r,isFirstItem:a,isLastItem:i}=n,c="vertical"===r?"-vertical-":"-";return o()("".concat(e,"-compact").concat(c,"item"),{["".concat(e,"-compact").concat(c,"first-item")]:a,["".concat(e,"-compact").concat(c,"last-item")]:i,["".concat(e,"-compact").concat(c,"item-rtl")]:"rtl"===t})},[e,t,n]);return{compactSize:null==n?void 0:n.compactSize,compactDirection:null==n?void 0:n.compactDirection,compactItemClassnames:r}},p=e=>{let{children:t}=e;return i.createElement(d.Provider,{value:null},t)},m=e=>{var{children:t}=e,n=u(e,["children"]);return i.createElement(d.Provider,{value:n},t)};t.ZP=e=>{let{getPrefixCls:t,direction:n}=i.useContext(c.E_),{size:r,direction:f,block:p,prefixCls:g,className:h,rootClassName:v,children:b}=e,y=u(e,["size","direction","block","prefixCls","className","rootClassName","children"]),w=(0,l.Z)(e=>null!=r?r:e),x=t("space-compact",g),[E,S]=(0,s.Z)(x),C=o()(x,S,{["".concat(x,"-rtl")]:"rtl"===n,["".concat(x,"-block")]:p,["".concat(x,"-vertical")]:"vertical"===f},h,v),Z=i.useContext(d),O=(0,a.Z)(b),k=i.useMemo(()=>O.map((e,t)=>{let n=e&&e.key||"".concat(x,"-item-").concat(t);return i.createElement(m,{key:n,compactSize:w,compactDirection:f,isFirstItem:0===t&&(!Z||(null==Z?void 0:Z.isFirstItem)),isLastItem:t===O.length-1&&(!Z||(null==Z?void 0:Z.isLastItem))},e)}),[r,O,Z]);return 0===O.length?null:E(i.createElement("div",Object.assign({className:C},y),k))}},4924:function(e,t,n){"use strict";n.d(t,{Z:function(){return l}});var r=n(80669),o=n(3104),a=e=>{let{componentCls:t}=e;return{[t]:{"&-block":{display:"flex",width:"100%"},"&-vertical":{flexDirection:"column"}}}};let i=e=>{let{componentCls:t}=e;return{[t]:{display:"inline-flex","&-rtl":{direction:"rtl"},"&-vertical":{flexDirection:"column"},"&-align":{flexDirection:"column","&-center":{alignItems:"center"},"&-start":{alignItems:"flex-start"},"&-end":{alignItems:"flex-end"},"&-baseline":{alignItems:"baseline"}},["".concat(t,"-item:empty")]:{display:"none"}}}},c=e=>{let{componentCls:t}=e;return{[t]:{"&-gap-row-small":{rowGap:e.spaceGapSmallSize},"&-gap-row-middle":{rowGap:e.spaceGapMiddleSize},"&-gap-row-large":{rowGap:e.spaceGapLargeSize},"&-gap-col-small":{columnGap:e.spaceGapSmallSize},"&-gap-col-middle":{columnGap:e.spaceGapMiddleSize},"&-gap-col-large":{columnGap:e.spaceGapLargeSize}}}};var l=(0,r.I$)("Space",e=>{let t=(0,o.TS)(e,{spaceGapSmallSize:e.paddingXS,spaceGapMiddleSize:e.padding,spaceGapLargeSize:e.paddingLG});return[i(t),c(t),a(t)]},()=>({}),{resetStyle:!1})},17691:function(e,t,n){"use strict";function r(e){let t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{focus:!0},{componentCls:n}=e,r="".concat(n,"-compact");return{[r]:Object.assign(Object.assign({},function(e,t,n){let{focusElCls:r,focus:o,borderElCls:a}=n,i=a?"> *":"",c=["hover",o?"focus":null,"active"].filter(Boolean).map(e=>"&:".concat(e," ").concat(i)).join(",");return{["&-item:not(".concat(t,"-last-item)")]:{marginInlineEnd:e.calc(e.lineWidth).mul(-1).equal()},"&-item":Object.assign(Object.assign({[c]:{zIndex:2}},r?{["&".concat(r)]:{zIndex:2}}:{}),{["&[disabled] ".concat(i)]:{zIndex:0}})}}(e,r,t)),function(e,t,n){let{borderElCls:r}=n,o=r?"> ".concat(r):"";return{["&-item:not(".concat(t,"-first-item):not(").concat(t,"-last-item) ").concat(o)]:{borderRadius:0},["&-item:not(".concat(t,"-last-item)").concat(t,"-first-item")]:{["& ".concat(o,", &").concat(e,"-sm ").concat(o,", &").concat(e,"-lg ").concat(o)]:{borderStartEndRadius:0,borderEndEndRadius:0}},["&-item:not(".concat(t,"-first-item)").concat(t,"-last-item")]:{["& ".concat(o,", &").concat(e,"-sm ").concat(o,", &").concat(e,"-lg ").concat(o)]:{borderStartStartRadius:0,borderEndStartRadius:0}}}}(n,r,t))}}n.d(t,{c:function(){return r}})},12918:function(e,t,n){"use strict";n.d(t,{Lx:function(){return l},Qy:function(){return d},Ro:function(){return i},Wf:function(){return a},dF:function(){return c},du:function(){return s},oN:function(){return u},vS:function(){return o}});var r=n(352);let o={overflow:"hidden",whiteSpace:"nowrap",textOverflow:"ellipsis"},a=function(e){let t=arguments.length>1&&void 0!==arguments[1]&&arguments[1];return{boxSizing:"border-box",margin:0,padding:0,color:e.colorText,fontSize:e.fontSize,lineHeight:e.lineHeight,listStyle:"none",fontFamily:t?"inherit":e.fontFamily}},i=()=>({display:"inline-flex",alignItems:"center",color:"inherit",fontStyle:"normal",lineHeight:0,textAlign:"center",textTransform:"none",verticalAlign:"-0.125em",textRendering:"optimizeLegibility","-webkit-font-smoothing":"antialiased","-moz-osx-font-smoothing":"grayscale","> *":{lineHeight:1},svg:{display:"inline-block"}}),c=()=>({"&::before":{display:"table",content:'""'},"&::after":{display:"table",clear:"both",content:'""'}}),l=e=>({a:{color:e.colorLink,textDecoration:e.linkDecoration,backgroundColor:"transparent",outline:"none",cursor:"pointer",transition:"color ".concat(e.motionDurationSlow),"-webkit-text-decoration-skip":"objects","&:hover":{color:e.colorLinkHover},"&:active":{color:e.colorLinkActive},"&:active,\n &:hover":{textDecoration:e.linkHoverDecoration,outline:0},"&:focus":{textDecoration:e.linkFocusDecoration,outline:0},"&[disabled]":{color:e.colorTextDisabled,cursor:"not-allowed"}}}),s=(e,t)=>{let{fontFamily:n,fontSize:r}=e,o='[class^="'.concat(t,'"], [class*=" ').concat(t,'"]');return{[o]:{fontFamily:n,fontSize:r,boxSizing:"border-box","&::before, &::after":{boxSizing:"border-box"},[o]:{boxSizing:"border-box","&::before, &::after":{boxSizing:"border-box"}}}}},u=e=>({outline:"".concat((0,r.bf)(e.lineWidthFocus)," solid ").concat(e.colorPrimaryBorder),outlineOffset:1,transition:"outline-offset 0s, outline 0s"}),d=e=>({"&:focus-visible":Object.assign({},u(e))})},63074:function(e,t){"use strict";t.Z=e=>({[e.componentCls]:{["".concat(e.antCls,"-motion-collapse-legacy")]:{overflow:"hidden","&-active":{transition:"height ".concat(e.motionDurationMid," ").concat(e.motionEaseInOut,",\n opacity ").concat(e.motionDurationMid," ").concat(e.motionEaseInOut," !important")}},["".concat(e.antCls,"-motion-collapse")]:{overflow:"hidden",transition:"height ".concat(e.motionDurationMid," ").concat(e.motionEaseInOut,",\n opacity ").concat(e.motionDurationMid," ").concat(e.motionEaseInOut," !important")}}})},37133:function(e,t,n){"use strict";n.d(t,{R:function(){return a}});let r=e=>({animationDuration:e,animationFillMode:"both"}),o=e=>({animationDuration:e,animationFillMode:"both"}),a=function(e,t,n,a){let i=arguments.length>4&&void 0!==arguments[4]&&arguments[4],c=i?"&":"";return{["\n ".concat(c).concat(e,"-enter,\n ").concat(c).concat(e,"-appear\n ")]:Object.assign(Object.assign({},r(a)),{animationPlayState:"paused"}),["".concat(c).concat(e,"-leave")]:Object.assign(Object.assign({},o(a)),{animationPlayState:"paused"}),["\n ".concat(c).concat(e,"-enter").concat(e,"-enter-active,\n ").concat(c).concat(e,"-appear").concat(e,"-appear-active\n ")]:{animationName:t,animationPlayState:"running"},["".concat(c).concat(e,"-leave").concat(e,"-leave-active")]:{animationName:n,animationPlayState:"running",pointerEvents:"none"}}}},29382:function(e,t,n){"use strict";n.d(t,{Fm:function(){return f}});var r=n(352),o=n(37133);let a=new r.E4("antMoveDownIn",{"0%":{transform:"translate3d(0, 100%, 0)",transformOrigin:"0 0",opacity:0},"100%":{transform:"translate3d(0, 0, 0)",transformOrigin:"0 0",opacity:1}}),i=new r.E4("antMoveDownOut",{"0%":{transform:"translate3d(0, 0, 0)",transformOrigin:"0 0",opacity:1},"100%":{transform:"translate3d(0, 100%, 0)",transformOrigin:"0 0",opacity:0}}),c=new r.E4("antMoveLeftIn",{"0%":{transform:"translate3d(-100%, 0, 0)",transformOrigin:"0 0",opacity:0},"100%":{transform:"translate3d(0, 0, 0)",transformOrigin:"0 0",opacity:1}}),l=new r.E4("antMoveLeftOut",{"0%":{transform:"translate3d(0, 0, 0)",transformOrigin:"0 0",opacity:1},"100%":{transform:"translate3d(-100%, 0, 0)",transformOrigin:"0 0",opacity:0}}),s=new r.E4("antMoveRightIn",{"0%":{transform:"translate3d(100%, 0, 0)",transformOrigin:"0 0",opacity:0},"100%":{transform:"translate3d(0, 0, 0)",transformOrigin:"0 0",opacity:1}}),u=new r.E4("antMoveRightOut",{"0%":{transform:"translate3d(0, 0, 0)",transformOrigin:"0 0",opacity:1},"100%":{transform:"translate3d(100%, 0, 0)",transformOrigin:"0 0",opacity:0}}),d={"move-up":{inKeyframes:new r.E4("antMoveUpIn",{"0%":{transform:"translate3d(0, -100%, 0)",transformOrigin:"0 0",opacity:0},"100%":{transform:"translate3d(0, 0, 0)",transformOrigin:"0 0",opacity:1}}),outKeyframes:new r.E4("antMoveUpOut",{"0%":{transform:"translate3d(0, 0, 0)",transformOrigin:"0 0",opacity:1},"100%":{transform:"translate3d(0, -100%, 0)",transformOrigin:"0 0",opacity:0}})},"move-down":{inKeyframes:a,outKeyframes:i},"move-left":{inKeyframes:c,outKeyframes:l},"move-right":{inKeyframes:s,outKeyframes:u}},f=(e,t)=>{let{antCls:n}=e,r="".concat(n,"-").concat(t),{inKeyframes:a,outKeyframes:i}=d[t];return[(0,o.R)(r,a,i,e.motionDurationMid),{["\n ".concat(r,"-enter,\n ").concat(r,"-appear\n ")]:{opacity:0,animationTimingFunction:e.motionEaseOutCirc},["".concat(r,"-leave")]:{animationTimingFunction:e.motionEaseInOutCirc}}]}},18544:function(e,t,n){"use strict";n.d(t,{Qt:function(){return c},Uw:function(){return i},fJ:function(){return a},ly:function(){return l},oN:function(){return d}});var r=n(352),o=n(37133);let a=new r.E4("antSlideUpIn",{"0%":{transform:"scaleY(0.8)",transformOrigin:"0% 0%",opacity:0},"100%":{transform:"scaleY(1)",transformOrigin:"0% 0%",opacity:1}}),i=new r.E4("antSlideUpOut",{"0%":{transform:"scaleY(1)",transformOrigin:"0% 0%",opacity:1},"100%":{transform:"scaleY(0.8)",transformOrigin:"0% 0%",opacity:0}}),c=new r.E4("antSlideDownIn",{"0%":{transform:"scaleY(0.8)",transformOrigin:"100% 100%",opacity:0},"100%":{transform:"scaleY(1)",transformOrigin:"100% 100%",opacity:1}}),l=new r.E4("antSlideDownOut",{"0%":{transform:"scaleY(1)",transformOrigin:"100% 100%",opacity:1},"100%":{transform:"scaleY(0.8)",transformOrigin:"100% 100%",opacity:0}}),s=new r.E4("antSlideLeftIn",{"0%":{transform:"scaleX(0.8)",transformOrigin:"0% 0%",opacity:0},"100%":{transform:"scaleX(1)",transformOrigin:"0% 0%",opacity:1}}),u={"slide-up":{inKeyframes:a,outKeyframes:i},"slide-down":{inKeyframes:c,outKeyframes:l},"slide-left":{inKeyframes:s,outKeyframes:new r.E4("antSlideLeftOut",{"0%":{transform:"scaleX(1)",transformOrigin:"0% 0%",opacity:1},"100%":{transform:"scaleX(0.8)",transformOrigin:"0% 0%",opacity:0}})},"slide-right":{inKeyframes:new r.E4("antSlideRightIn",{"0%":{transform:"scaleX(0.8)",transformOrigin:"100% 0%",opacity:0},"100%":{transform:"scaleX(1)",transformOrigin:"100% 0%",opacity:1}}),outKeyframes:new r.E4("antSlideRightOut",{"0%":{transform:"scaleX(1)",transformOrigin:"100% 0%",opacity:1},"100%":{transform:"scaleX(0.8)",transformOrigin:"100% 0%",opacity:0}})}},d=(e,t)=>{let{antCls:n}=e,r="".concat(n,"-").concat(t),{inKeyframes:a,outKeyframes:i}=u[t];return[(0,o.R)(r,a,i,e.motionDurationMid),{["\n ".concat(r,"-enter,\n ").concat(r,"-appear\n ")]:{transform:"scale(0)",transformOrigin:"0% 0%",opacity:0,animationTimingFunction:e.motionEaseOutQuint,"&-prepare":{transform:"scale(1)"}},["".concat(r,"-leave")]:{animationTimingFunction:e.motionEaseInQuint}}]}},691:function(e,t,n){"use strict";n.d(t,{_y:function(){return g},kr:function(){return a}});var r=n(352),o=n(37133);let a=new r.E4("antZoomIn",{"0%":{transform:"scale(0.2)",opacity:0},"100%":{transform:"scale(1)",opacity:1}}),i=new r.E4("antZoomOut",{"0%":{transform:"scale(1)"},"100%":{transform:"scale(0.2)",opacity:0}}),c=new r.E4("antZoomBigIn",{"0%":{transform:"scale(0.8)",opacity:0},"100%":{transform:"scale(1)",opacity:1}}),l=new r.E4("antZoomBigOut",{"0%":{transform:"scale(1)"},"100%":{transform:"scale(0.8)",opacity:0}}),s=new r.E4("antZoomUpIn",{"0%":{transform:"scale(0.8)",transformOrigin:"50% 0%",opacity:0},"100%":{transform:"scale(1)",transformOrigin:"50% 0%"}}),u=new r.E4("antZoomUpOut",{"0%":{transform:"scale(1)",transformOrigin:"50% 0%"},"100%":{transform:"scale(0.8)",transformOrigin:"50% 0%",opacity:0}}),d=new r.E4("antZoomLeftIn",{"0%":{transform:"scale(0.8)",transformOrigin:"0% 50%",opacity:0},"100%":{transform:"scale(1)",transformOrigin:"0% 50%"}}),f=new r.E4("antZoomLeftOut",{"0%":{transform:"scale(1)",transformOrigin:"0% 50%"},"100%":{transform:"scale(0.8)",transformOrigin:"0% 50%",opacity:0}}),p=new r.E4("antZoomRightIn",{"0%":{transform:"scale(0.8)",transformOrigin:"100% 50%",opacity:0},"100%":{transform:"scale(1)",transformOrigin:"100% 50%"}}),m={zoom:{inKeyframes:a,outKeyframes:i},"zoom-big":{inKeyframes:c,outKeyframes:l},"zoom-big-fast":{inKeyframes:c,outKeyframes:l},"zoom-left":{inKeyframes:d,outKeyframes:f},"zoom-right":{inKeyframes:p,outKeyframes:new r.E4("antZoomRightOut",{"0%":{transform:"scale(1)",transformOrigin:"100% 50%"},"100%":{transform:"scale(0.8)",transformOrigin:"100% 50%",opacity:0}})},"zoom-up":{inKeyframes:s,outKeyframes:u},"zoom-down":{inKeyframes:new r.E4("antZoomDownIn",{"0%":{transform:"scale(0.8)",transformOrigin:"50% 100%",opacity:0},"100%":{transform:"scale(1)",transformOrigin:"50% 100%"}}),outKeyframes:new r.E4("antZoomDownOut",{"0%":{transform:"scale(1)",transformOrigin:"50% 100%"},"100%":{transform:"scale(0.8)",transformOrigin:"50% 100%",opacity:0}})}},g=(e,t)=>{let{antCls:n}=e,r="".concat(n,"-").concat(t),{inKeyframes:a,outKeyframes:i}=m[t];return[(0,o.R)(r,a,i,"zoom-big-fast"===t?e.motionDurationFast:e.motionDurationMid),{["\n ".concat(r,"-enter,\n ").concat(r,"-appear\n ")]:{transform:"scale(0)",opacity:0,animationTimingFunction:e.motionEaseOutCirc,"&-prepare":{transform:"none"}},["".concat(r,"-leave")]:{animationTimingFunction:e.motionEaseInOutCirc}}]}},88260:function(e,t,n){"use strict";n.d(t,{ZP:function(){return i},qN:function(){return o},wZ:function(){return a}});var r=n(34442);let o=8;function a(e){let{contentRadius:t,limitVerticalRadius:n}=e,r=t>12?t+2:12;return{arrowOffsetHorizontal:r,arrowOffsetVertical:n?o:r}}function i(e,t,n){var o,a,i,c,l,s,u,d;let{componentCls:f,boxShadowPopoverArrow:p,arrowOffsetVertical:m,arrowOffsetHorizontal:g}=e,{arrowDistance:h=0,arrowPlacement:v={left:!0,right:!0,top:!0,bottom:!0}}=n||{};return{[f]:Object.assign(Object.assign(Object.assign(Object.assign({["".concat(f,"-arrow")]:[Object.assign(Object.assign({position:"absolute",zIndex:1,display:"block"},(0,r.W)(e,t,p)),{"&:before":{background:t}})]},(o=!!v.top,a={[["&-placement-top > ".concat(f,"-arrow"),"&-placement-topLeft > ".concat(f,"-arrow"),"&-placement-topRight > ".concat(f,"-arrow")].join(",")]:{bottom:h,transform:"translateY(100%) rotate(180deg)"},["&-placement-top > ".concat(f,"-arrow")]:{left:{_skip_check_:!0,value:"50%"},transform:"translateX(-50%) translateY(100%) rotate(180deg)"},["&-placement-topLeft > ".concat(f,"-arrow")]:{left:{_skip_check_:!0,value:g}},["&-placement-topRight > ".concat(f,"-arrow")]:{right:{_skip_check_:!0,value:g}}},o?a:{})),(i=!!v.bottom,c={[["&-placement-bottom > ".concat(f,"-arrow"),"&-placement-bottomLeft > ".concat(f,"-arrow"),"&-placement-bottomRight > ".concat(f,"-arrow")].join(",")]:{top:h,transform:"translateY(-100%)"},["&-placement-bottom > ".concat(f,"-arrow")]:{left:{_skip_check_:!0,value:"50%"},transform:"translateX(-50%) translateY(-100%)"},["&-placement-bottomLeft > ".concat(f,"-arrow")]:{left:{_skip_check_:!0,value:g}},["&-placement-bottomRight > ".concat(f,"-arrow")]:{right:{_skip_check_:!0,value:g}}},i?c:{})),(l=!!v.left,s={[["&-placement-left > ".concat(f,"-arrow"),"&-placement-leftTop > ".concat(f,"-arrow"),"&-placement-leftBottom > ".concat(f,"-arrow")].join(",")]:{right:{_skip_check_:!0,value:h},transform:"translateX(100%) rotate(90deg)"},["&-placement-left > ".concat(f,"-arrow")]:{top:{_skip_check_:!0,value:"50%"},transform:"translateY(-50%) translateX(100%) rotate(90deg)"},["&-placement-leftTop > ".concat(f,"-arrow")]:{top:m},["&-placement-leftBottom > ".concat(f,"-arrow")]:{bottom:m}},l?s:{})),(u=!!v.right,d={[["&-placement-right > ".concat(f,"-arrow"),"&-placement-rightTop > ".concat(f,"-arrow"),"&-placement-rightBottom > ".concat(f,"-arrow")].join(",")]:{left:{_skip_check_:!0,value:h},transform:"translateX(-100%) rotate(-90deg)"},["&-placement-right > ".concat(f,"-arrow")]:{top:{_skip_check_:!0,value:"50%"},transform:"translateY(-50%) translateX(-100%) rotate(-90deg)"},["&-placement-rightTop > ".concat(f,"-arrow")]:{top:m},["&-placement-rightBottom > ".concat(f,"-arrow")]:{bottom:m}},u?d:{}))}}},34442:function(e,t,n){"use strict";n.d(t,{W:function(){return a},w:function(){return o}});var r=n(352);function o(e){let{sizePopupArrow:t,borderRadiusXS:n,borderRadiusOuter:r}=e,o=t/2,a=1*r/Math.sqrt(2),i=o-r*(1-1/Math.sqrt(2)),c=o-1/Math.sqrt(2)*n,l=r*(Math.sqrt(2)-1)+1/Math.sqrt(2)*n,s=2*o-c,u=2*o-a,d=2*o-0,f=o*Math.sqrt(2)+r*(Math.sqrt(2)-2),p=r*(Math.sqrt(2)-1),m="polygon(".concat(p,"px 100%, 50% ").concat(p,"px, ").concat(2*o-p,"px 100%, ").concat(p,"px 100%)");return{arrowShadowWidth:f,arrowPath:"path('M ".concat(0," ").concat(o," A ").concat(r," ").concat(r," 0 0 0 ").concat(a," ").concat(i," L ").concat(c," ").concat(l," A ").concat(n," ").concat(n," 0 0 1 ").concat(s," ").concat(l," L ").concat(u," ").concat(i," A ").concat(r," ").concat(r," 0 0 0 ").concat(d," ").concat(o," Z')"),arrowPolygon:m}}let a=(e,t,n)=>{let{sizePopupArrow:o,arrowPolygon:a,arrowPath:i,arrowShadowWidth:c,borderRadiusXS:l,calc:s}=e;return{pointerEvents:"none",width:o,height:o,overflow:"hidden","&::before":{position:"absolute",bottom:0,insetInlineStart:0,width:o,height:s(o).div(2).equal(),background:t,clipPath:{_multi_value_:!0,value:[a,i]},content:'""'},"&::after":{content:'""',position:"absolute",width:c,height:c,bottom:0,insetInline:0,margin:"auto",borderRadius:{_skip_check_:!0,value:"0 0 ".concat((0,r.bf)(l)," 0")},transform:"translateY(50%) rotate(-135deg)",boxShadow:n,zIndex:0,background:"transparent"}}}},37516:function(e,t,n){"use strict";n.d(t,{Mj:function(){return b},u_:function(){return v},uH:function(){return h}});var r=n(2265),o=n(352),a=n(31373),i=e=>{let{controlHeight:t}=e;return{controlHeightSM:.75*t,controlHeightXS:.5*t,controlHeightLG:1.25*t}},c=n(70774),l=n(36360),s=e=>{let t=e,n=e,r=e,o=e;return e<6&&e>=5?t=e+1:e<16&&e>=6?t=e+2:e>=16&&(t=16),e<7&&e>=5?n=4:e<8&&e>=7?n=5:e<14&&e>=8?n=6:e<16&&e>=14?n=7:e>=16&&(n=8),e<6&&e>=2?r=1:e>=6&&(r=2),e>4&&e<8?o=4:e>=8&&(o=6),{borderRadius:e,borderRadiusXS:r,borderRadiusSM:n,borderRadiusLG:t,borderRadiusOuter:o}};let u=(e,t)=>new l.C(e).setAlpha(t).toRgbString(),d=(e,t)=>new l.C(e).darken(t).toHexString(),f=e=>{let t=(0,a.R_)(e);return{1:t[0],2:t[1],3:t[2],4:t[3],5:t[4],6:t[5],7:t[6],8:t[4],9:t[5],10:t[6]}},p=(e,t)=>{let n=e||"#fff",r=t||"#000";return{colorBgBase:n,colorTextBase:r,colorText:u(r,.88),colorTextSecondary:u(r,.65),colorTextTertiary:u(r,.45),colorTextQuaternary:u(r,.25),colorFill:u(r,.15),colorFillSecondary:u(r,.06),colorFillTertiary:u(r,.04),colorFillQuaternary:u(r,.02),colorBgLayout:d(n,4),colorBgContainer:d(n,0),colorBgElevated:d(n,0),colorBgSpotlight:u(r,.85),colorBgBlur:"transparent",colorBorder:d(n,15),colorBorderSecondary:d(n,6)}};var m=n(1319),g=e=>{let t=(0,m.Z)(e),n=t.map(e=>e.size),r=t.map(e=>e.lineHeight),o=n[1],a=n[0],i=n[2],c=r[1],l=r[0],s=r[2];return{fontSizeSM:a,fontSize:o,fontSizeLG:i,fontSizeXL:n[3],fontSizeHeading1:n[6],fontSizeHeading2:n[5],fontSizeHeading3:n[4],fontSizeHeading4:n[3],fontSizeHeading5:n[2],lineHeight:c,lineHeightLG:s,lineHeightSM:l,fontHeight:Math.round(c*o),fontHeightLG:Math.round(s*i),fontHeightSM:Math.round(l*a),lineHeightHeading1:r[6],lineHeightHeading2:r[5],lineHeightHeading3:r[4],lineHeightHeading4:r[3],lineHeightHeading5:r[2]}};let h=(0,o.jG)(function(e){let t=Object.keys(c.M).map(t=>{let n=(0,a.R_)(e[t]);return Array(10).fill(1).reduce((e,r,o)=>(e["".concat(t,"-").concat(o+1)]=n[o],e["".concat(t).concat(o+1)]=n[o],e),{})}).reduce((e,t)=>e=Object.assign(Object.assign({},e),t),{});return Object.assign(Object.assign(Object.assign(Object.assign(Object.assign(Object.assign(Object.assign({},e),t),function(e,t){let{generateColorPalettes:n,generateNeutralColorPalettes:r}=t,{colorSuccess:o,colorWarning:a,colorError:i,colorInfo:c,colorPrimary:s,colorBgBase:u,colorTextBase:d}=e,f=n(s),p=n(o),m=n(a),g=n(i),h=n(c),v=r(u,d),b=n(e.colorLink||e.colorInfo);return Object.assign(Object.assign({},v),{colorPrimaryBg:f[1],colorPrimaryBgHover:f[2],colorPrimaryBorder:f[3],colorPrimaryBorderHover:f[4],colorPrimaryHover:f[5],colorPrimary:f[6],colorPrimaryActive:f[7],colorPrimaryTextHover:f[8],colorPrimaryText:f[9],colorPrimaryTextActive:f[10],colorSuccessBg:p[1],colorSuccessBgHover:p[2],colorSuccessBorder:p[3],colorSuccessBorderHover:p[4],colorSuccessHover:p[4],colorSuccess:p[6],colorSuccessActive:p[7],colorSuccessTextHover:p[8],colorSuccessText:p[9],colorSuccessTextActive:p[10],colorErrorBg:g[1],colorErrorBgHover:g[2],colorErrorBorder:g[3],colorErrorBorderHover:g[4],colorErrorHover:g[5],colorError:g[6],colorErrorActive:g[7],colorErrorTextHover:g[8],colorErrorText:g[9],colorErrorTextActive:g[10],colorWarningBg:m[1],colorWarningBgHover:m[2],colorWarningBorder:m[3],colorWarningBorderHover:m[4],colorWarningHover:m[4],colorWarning:m[6],colorWarningActive:m[7],colorWarningTextHover:m[8],colorWarningText:m[9],colorWarningTextActive:m[10],colorInfoBg:h[1],colorInfoBgHover:h[2],colorInfoBorder:h[3],colorInfoBorderHover:h[4],colorInfoHover:h[4],colorInfo:h[6],colorInfoActive:h[7],colorInfoTextHover:h[8],colorInfoText:h[9],colorInfoTextActive:h[10],colorLinkHover:b[4],colorLink:b[6],colorLinkActive:b[7],colorBgMask:new l.C("#000").setAlpha(.45).toRgbString(),colorWhite:"#fff"})}(e,{generateColorPalettes:f,generateNeutralColorPalettes:p})),g(e.fontSize)),function(e){let{sizeUnit:t,sizeStep:n}=e;return{sizeXXL:t*(n+8),sizeXL:t*(n+4),sizeLG:t*(n+2),sizeMD:t*(n+1),sizeMS:t*n,size:t*n,sizeSM:t*(n-1),sizeXS:t*(n-2),sizeXXS:t*(n-3)}}(e)),i(e)),function(e){let{motionUnit:t,motionBase:n,borderRadius:r,lineWidth:o}=e;return Object.assign({motionDurationFast:"".concat((n+t).toFixed(1),"s"),motionDurationMid:"".concat((n+2*t).toFixed(1),"s"),motionDurationSlow:"".concat((n+3*t).toFixed(1),"s"),lineWidthBold:o+1},s(r))}(e))}),v={token:c.Z,override:{override:c.Z},hashed:!0},b=r.createContext(v)},53454:function(e,t,n){"use strict";n.d(t,{i:function(){return r}});let r=["blue","purple","cyan","green","magenta","pink","red","orange","yellow","volcano","geekblue","lime","gold"]},70774:function(e,t,n){"use strict";n.d(t,{M:function(){return r}});let r={blue:"#1677ff",purple:"#722ED1",cyan:"#13C2C2",green:"#52C41A",magenta:"#EB2F96",pink:"#eb2f96",red:"#F5222D",orange:"#FA8C16",yellow:"#FADB14",volcano:"#FA541C",geekblue:"#2F54EB",gold:"#FAAD14",lime:"#A0D911"},o=Object.assign(Object.assign({},r),{colorPrimary:"#1677ff",colorSuccess:"#52c41a",colorWarning:"#faad14",colorError:"#ff4d4f",colorInfo:"#1677ff",colorLink:"",colorTextBase:"",colorBgBase:"",fontFamily:"-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial,\n'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol',\n'Noto Color Emoji'",fontFamilyCode:"'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier, monospace",fontSize:14,lineWidth:1,lineType:"solid",motionUnit:.1,motionBase:0,motionEaseOutCirc:"cubic-bezier(0.08, 0.82, 0.17, 1)",motionEaseInOutCirc:"cubic-bezier(0.78, 0.14, 0.15, 0.86)",motionEaseOut:"cubic-bezier(0.215, 0.61, 0.355, 1)",motionEaseInOut:"cubic-bezier(0.645, 0.045, 0.355, 1)",motionEaseOutBack:"cubic-bezier(0.12, 0.4, 0.29, 1.46)",motionEaseInBack:"cubic-bezier(0.71, -0.46, 0.88, 0.6)",motionEaseInQuint:"cubic-bezier(0.755, 0.05, 0.855, 0.06)",motionEaseOutQuint:"cubic-bezier(0.23, 1, 0.32, 1)",borderRadius:6,sizeUnit:4,sizeStep:4,sizePopupArrow:16,controlHeight:32,zIndexBase:0,zIndexPopupBase:1e3,opacityImage:1,wireframe:!1,motion:!0});t.Z=o},1319:function(e,t,n){"use strict";function r(e){return(e+8)/e}function o(e){let t=Array(10).fill(null).map((t,n)=>{let r=e*Math.pow(2.71828,(n-1)/5);return 2*Math.floor((n>1?Math.floor(r):Math.ceil(r))/2)});return t[1]=e,t.map(e=>({size:e,lineHeight:r(e)}))}n.d(t,{D:function(){return r},Z:function(){return o}})},29961:function(e,t,n){"use strict";n.d(t,{ZP:function(){return v},ID:function(){return m},NJ:function(){return p}});var r=n(2265),o=n(352),a=n(37516),i=n(70774),c=n(36360);function l(e){return e>=0&&e<=255}var s=function(e,t){let{r:n,g:r,b:o,a:a}=new c.C(e).toRgb();if(a<1)return e;let{r:i,g:s,b:u}=new c.C(t).toRgb();for(let e=.01;e<=1;e+=.01){let t=Math.round((n-i*(1-e))/e),a=Math.round((r-s*(1-e))/e),d=Math.round((o-u*(1-e))/e);if(l(t)&&l(a)&&l(d))return new c.C({r:t,g:a,b:d,a:Math.round(100*e)/100}).toRgbString()}return new c.C({r:n,g:r,b:o,a:1}).toRgbString()},u=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};function d(e){let{override:t}=e,n=u(e,["override"]),r=Object.assign({},t);Object.keys(i.Z).forEach(e=>{delete r[e]});let o=Object.assign(Object.assign({},n),r);return!1===o.motion&&(o.motionDurationFast="0s",o.motionDurationMid="0s",o.motionDurationSlow="0s"),Object.assign(Object.assign(Object.assign({},o),{colorFillContent:o.colorFillSecondary,colorFillContentHover:o.colorFill,colorFillAlter:o.colorFillQuaternary,colorBgContainerDisabled:o.colorFillTertiary,colorBorderBg:o.colorBgContainer,colorSplit:s(o.colorBorderSecondary,o.colorBgContainer),colorTextPlaceholder:o.colorTextQuaternary,colorTextDisabled:o.colorTextQuaternary,colorTextHeading:o.colorText,colorTextLabel:o.colorTextSecondary,colorTextDescription:o.colorTextTertiary,colorTextLightSolid:o.colorWhite,colorHighlight:o.colorError,colorBgTextHover:o.colorFillSecondary,colorBgTextActive:o.colorFill,colorIcon:o.colorTextTertiary,colorIconHover:o.colorText,colorErrorOutline:s(o.colorErrorBg,o.colorBgContainer),colorWarningOutline:s(o.colorWarningBg,o.colorBgContainer),fontSizeIcon:o.fontSizeSM,lineWidthFocus:4*o.lineWidth,lineWidth:o.lineWidth,controlOutlineWidth:2*o.lineWidth,controlInteractiveSize:o.controlHeight/2,controlItemBgHover:o.colorFillTertiary,controlItemBgActive:o.colorPrimaryBg,controlItemBgActiveHover:o.colorPrimaryBgHover,controlItemBgActiveDisabled:o.colorFill,controlTmpOutline:o.colorFillQuaternary,controlOutline:s(o.colorPrimaryBg,o.colorBgContainer),lineType:o.lineType,borderRadius:o.borderRadius,borderRadiusXS:o.borderRadiusXS,borderRadiusSM:o.borderRadiusSM,borderRadiusLG:o.borderRadiusLG,fontWeightStrong:600,opacityLoading:.65,linkDecoration:"none",linkHoverDecoration:"none",linkFocusDecoration:"none",controlPaddingHorizontal:12,controlPaddingHorizontalSM:8,paddingXXS:o.sizeXXS,paddingXS:o.sizeXS,paddingSM:o.sizeSM,padding:o.size,paddingMD:o.sizeMD,paddingLG:o.sizeLG,paddingXL:o.sizeXL,paddingContentHorizontalLG:o.sizeLG,paddingContentVerticalLG:o.sizeMS,paddingContentHorizontal:o.sizeMS,paddingContentVertical:o.sizeSM,paddingContentHorizontalSM:o.size,paddingContentVerticalSM:o.sizeXS,marginXXS:o.sizeXXS,marginXS:o.sizeXS,marginSM:o.sizeSM,margin:o.size,marginMD:o.sizeMD,marginLG:o.sizeLG,marginXL:o.sizeXL,marginXXL:o.sizeXXL,boxShadow:"\n 0 6px 16px 0 rgba(0, 0, 0, 0.08),\n 0 3px 6px -4px rgba(0, 0, 0, 0.12),\n 0 9px 28px 8px rgba(0, 0, 0, 0.05)\n ",boxShadowSecondary:"\n 0 6px 16px 0 rgba(0, 0, 0, 0.08),\n 0 3px 6px -4px rgba(0, 0, 0, 0.12),\n 0 9px 28px 8px rgba(0, 0, 0, 0.05)\n ",boxShadowTertiary:"\n 0 1px 2px 0 rgba(0, 0, 0, 0.03),\n 0 1px 6px -1px rgba(0, 0, 0, 0.02),\n 0 2px 4px 0 rgba(0, 0, 0, 0.02)\n ",screenXS:480,screenXSMin:480,screenXSMax:575,screenSM:576,screenSMMin:576,screenSMMax:767,screenMD:768,screenMDMin:768,screenMDMax:991,screenLG:992,screenLGMin:992,screenLGMax:1199,screenXL:1200,screenXLMin:1200,screenXLMax:1599,screenXXL:1600,screenXXLMin:1600,boxShadowPopoverArrow:"2px 2px 5px rgba(0, 0, 0, 0.05)",boxShadowCard:"\n 0 1px 2px -2px ".concat(new c.C("rgba(0, 0, 0, 0.16)").toRgbString(),",\n 0 3px 6px 0 ").concat(new c.C("rgba(0, 0, 0, 0.12)").toRgbString(),",\n 0 5px 12px 4px ").concat(new c.C("rgba(0, 0, 0, 0.09)").toRgbString(),"\n "),boxShadowDrawerRight:"\n -6px 0 16px 0 rgba(0, 0, 0, 0.08),\n -3px 0 6px -4px rgba(0, 0, 0, 0.12),\n -9px 0 28px 8px rgba(0, 0, 0, 0.05)\n ",boxShadowDrawerLeft:"\n 6px 0 16px 0 rgba(0, 0, 0, 0.08),\n 3px 0 6px -4px rgba(0, 0, 0, 0.12),\n 9px 0 28px 8px rgba(0, 0, 0, 0.05)\n ",boxShadowDrawerUp:"\n 0 6px 16px 0 rgba(0, 0, 0, 0.08),\n 0 3px 6px -4px rgba(0, 0, 0, 0.12),\n 0 9px 28px 8px rgba(0, 0, 0, 0.05)\n ",boxShadowDrawerDown:"\n 0 -6px 16px 0 rgba(0, 0, 0, 0.08),\n 0 -3px 6px -4px rgba(0, 0, 0, 0.12),\n 0 -9px 28px 8px rgba(0, 0, 0, 0.05)\n ",boxShadowTabsOverflowLeft:"inset 10px 0 8px -8px rgba(0, 0, 0, 0.08)",boxShadowTabsOverflowRight:"inset -10px 0 8px -8px rgba(0, 0, 0, 0.08)",boxShadowTabsOverflowTop:"inset 0 10px 8px -8px rgba(0, 0, 0, 0.08)",boxShadowTabsOverflowBottom:"inset 0 -10px 8px -8px rgba(0, 0, 0, 0.08)"}),r)}var f=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let p={lineHeight:!0,lineHeightSM:!0,lineHeightLG:!0,lineHeightHeading1:!0,lineHeightHeading2:!0,lineHeightHeading3:!0,lineHeightHeading4:!0,lineHeightHeading5:!0,opacityLoading:!0,fontWeightStrong:!0,zIndexPopupBase:!0,zIndexBase:!0},m={size:!0,sizeSM:!0,sizeLG:!0,sizeMD:!0,sizeXS:!0,sizeXXS:!0,sizeMS:!0,sizeXL:!0,sizeXXL:!0,sizeUnit:!0,sizeStep:!0,motionBase:!0,motionUnit:!0},g={screenXS:!0,screenXSMin:!0,screenXSMax:!0,screenSM:!0,screenSMMin:!0,screenSMMax:!0,screenMD:!0,screenMDMin:!0,screenMDMax:!0,screenLG:!0,screenLGMin:!0,screenLGMax:!0,screenXL:!0,screenXLMin:!0,screenXLMax:!0,screenXXL:!0,screenXXLMin:!0},h=(e,t,n)=>{let r=n.getDerivativeToken(e),{override:o}=t,a=f(t,["override"]),i=Object.assign(Object.assign({},r),{override:o});return i=d(i),a&&Object.entries(a).forEach(e=>{let[t,n]=e,{theme:r}=n,o=f(n,["theme"]),a=o;r&&(a=h(Object.assign(Object.assign({},i),o),{override:o},r)),i[t]=a}),i};function v(){let{token:e,hashed:t,theme:n,override:c,cssVar:l}=r.useContext(a.Mj),s="".concat("5.13.2","-").concat(t||""),u=n||a.uH,[f,v,b]=(0,o.fp)(u,[i.Z,e],{salt:s,override:c,getComputedToken:h,formatToken:d,cssVar:l&&{prefix:l.prefix,key:l.key,unitless:p,ignore:m,preserve:g}});return[u,b,t?v:"",f,l]}},80669:function(e,t,n){"use strict";n.d(t,{ZP:function(){return Z},I$:function(){return M},bk:function(){return O}});var r=n(2265),o=n(352);n(74126);var a=n(71744),i=n(12918),c=n(29961),l=n(76405),s=n(25049),u=n(37977),d=n(63929),f=n(24995),p=n(15354);let m=(0,s.Z)(function e(){(0,l.Z)(this,e)}),g=function(e){function t(e){var n,r,o;return(0,l.Z)(this,t),r=t,r=(0,f.Z)(r),(n=(0,u.Z)(this,(0,d.Z)()?Reflect.construct(r,[],(0,f.Z)(this).constructor):r.apply(this,o))).result=0,e instanceof t?n.result=e.result:"number"==typeof e&&(n.result=e),n}return(0,p.Z)(t,e),(0,s.Z)(t,[{key:"add",value:function(e){return e instanceof t?this.result+=e.result:"number"==typeof e&&(this.result+=e),this}},{key:"sub",value:function(e){return e instanceof t?this.result-=e.result:"number"==typeof e&&(this.result-=e),this}},{key:"mul",value:function(e){return e instanceof t?this.result*=e.result:"number"==typeof e&&(this.result*=e),this}},{key:"div",value:function(e){return e instanceof t?this.result/=e.result:"number"==typeof e&&(this.result/=e),this}},{key:"equal",value:function(){return this.result}}]),t}(m),h="CALC_UNIT";function v(e){return"number"==typeof e?"".concat(e).concat(h):e}let b=function(e){function t(e){var n,r,o;return(0,l.Z)(this,t),r=t,r=(0,f.Z)(r),(n=(0,u.Z)(this,(0,d.Z)()?Reflect.construct(r,[],(0,f.Z)(this).constructor):r.apply(this,o))).result="",e instanceof t?n.result="(".concat(e.result,")"):"number"==typeof e?n.result=v(e):"string"==typeof e&&(n.result=e),n}return(0,p.Z)(t,e),(0,s.Z)(t,[{key:"add",value:function(e){return e instanceof t?this.result="".concat(this.result," + ").concat(e.getResult()):("number"==typeof e||"string"==typeof e)&&(this.result="".concat(this.result," + ").concat(v(e))),this.lowPriority=!0,this}},{key:"sub",value:function(e){return e instanceof t?this.result="".concat(this.result," - ").concat(e.getResult()):("number"==typeof e||"string"==typeof e)&&(this.result="".concat(this.result," - ").concat(v(e))),this.lowPriority=!0,this}},{key:"mul",value:function(e){return this.lowPriority&&(this.result="(".concat(this.result,")")),e instanceof t?this.result="".concat(this.result," * ").concat(e.getResult(!0)):("number"==typeof e||"string"==typeof e)&&(this.result="".concat(this.result," * ").concat(e)),this.lowPriority=!1,this}},{key:"div",value:function(e){return this.lowPriority&&(this.result="(".concat(this.result,")")),e instanceof t?this.result="".concat(this.result," / ").concat(e.getResult(!0)):("number"==typeof e||"string"==typeof e)&&(this.result="".concat(this.result," / ").concat(e)),this.lowPriority=!1,this}},{key:"getResult",value:function(e){return this.lowPriority||e?"(".concat(this.result,")"):this.result}},{key:"equal",value:function(e){let{unit:t=!0}=e||{},n=RegExp("".concat(h),"g");return(this.result=this.result.replace(n,t?"px":""),void 0!==this.lowPriority)?"calc(".concat(this.result,")"):this.result}}]),t}(m);var y=e=>{let t="css"===e?b:g;return e=>new t(e)},w=n(3104),x=n(36198);let E=(e,t,n)=>{var r;return"function"==typeof n?n((0,w.TS)(t,null!==(r=t[e])&&void 0!==r?r:{})):null!=n?n:{}},S=(e,t,n,r)=>{let o=Object.assign({},t[e]);if(null==r?void 0:r.deprecatedTokens){let{deprecatedTokens:e}=r;e.forEach(e=>{var t;let[n,r]=e;((null==o?void 0:o[n])||(null==o?void 0:o[r]))&&(null!==(t=o[r])&&void 0!==t||(o[r]=null==o?void 0:o[n]))})}let a=Object.assign(Object.assign({},n),o);return Object.keys(a).forEach(e=>{a[e]===t[e]&&delete a[e]}),a},C=(e,t)=>"".concat([t,e.replace(/([A-Z]+)([A-Z][a-z]+)/g,"$1-$2").replace(/([a-z])([A-Z])/g,"$1-$2")].filter(Boolean).join("-"));function Z(e,t,n){let l=arguments.length>3&&void 0!==arguments[3]?arguments[3]:{},s=Array.isArray(e)?e:[e,e],[u]=s,d=s.join("-");return e=>{let[s,f,p,m,g]=(0,c.ZP)(),{getPrefixCls:h,iconPrefixCls:v,csp:b}=(0,r.useContext)(a.E_),Z=h(),O=g?"css":"js",k=y(O),{max:M,min:j}="js"===O?{max:Math.max,min:Math.min}:{max:function(){for(var e=arguments.length,t=Array(e),n=0;n(0,o.bf)(e)).join(","),")")},min:function(){for(var e=arguments.length,t=Array(e),n=0;n(0,o.bf)(e)).join(","),")")}},I={theme:s,token:m,hashId:p,nonce:()=>null==b?void 0:b.nonce,clientOnly:l.clientOnly,order:l.order||-999};return(0,o.xy)(Object.assign(Object.assign({},I),{clientOnly:!1,path:["Shared",Z]}),()=>[{"&":(0,i.Lx)(m)}]),(0,x.Z)(v,b),[(0,o.xy)(Object.assign(Object.assign({},I),{path:[d,e,v]}),()=>{if(!1===l.injectStyle)return[];let{token:r,flush:a}=(0,w.ZP)(m),c=E(u,f,n),s=".".concat(e),d=S(u,f,c,{deprecatedTokens:l.deprecatedTokens});g&&Object.keys(c).forEach(e=>{c[e]="var(".concat((0,o.ks)(e,C(u,g.prefix)),")")});let h=(0,w.TS)(r,{componentCls:s,prefixCls:e,iconCls:".".concat(v),antCls:".".concat(Z),calc:k,max:M,min:j},g?c:d),b=t(h,{hashId:p,prefixCls:e,rootPrefixCls:Z,iconPrefixCls:v});return a(u,d),[!1===l.resetStyle?null:(0,i.du)(h,e),b]}),p]}}let O=(e,t,n,r)=>{let o=Z(e,t,n,Object.assign({resetStyle:!1,order:-998},r));return e=>{let{prefixCls:t}=e;return o(t),null}},k=(e,t,n)=>{function a(t){return"".concat(e).concat(t.slice(0,1).toUpperCase()).concat(t.slice(1))}let{unitless:i={},injectStyle:l=!0}=null!=n?n:{},s={[a("zIndexPopup")]:!0};Object.keys(i).forEach(e=>{s[a(e)]=i[e]});let u=r=>{let{rootCls:i,cssVar:l}=r,[,u]=(0,c.ZP)();return(0,o.CI)({path:[e],prefix:l.prefix,key:null==l?void 0:l.key,unitless:Object.assign(Object.assign({},c.NJ),s),ignore:c.ID,token:u,scope:i},()=>{let r=E(e,u,t),o=S(e,u,r,{deprecatedTokens:null==n?void 0:n.deprecatedTokens});return Object.keys(r).forEach(e=>{o[a(e)]=o[e],delete o[e]}),o}),null};return t=>{let[,,,,n]=(0,c.ZP)();return[o=>l&&n?r.createElement(r.Fragment,null,r.createElement(u,{rootCls:t,cssVar:n,component:e}),o):o,null==n?void 0:n.key]}},M=(e,t,n,r)=>{let o=Z(e,t,n,r),a=k(Array.isArray(e)?e[0]:e,n,r);return function(e){let t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:e,[,n]=o(e),[r,i]=a(t);return[r,n,i]}}},18536:function(e,t,n){"use strict";n.d(t,{Z:function(){return o}});var r=n(53454);function o(e,t){return r.i.reduce((n,r)=>{let o=e["".concat(r,"1")],a=e["".concat(r,"3")],i=e["".concat(r,"6")],c=e["".concat(r,"7")];return Object.assign(Object.assign({},n),t(r,{lightColor:o,lightBorderColor:a,darkColor:i,textColor:c}))},{})}},3104:function(e,t,n){"use strict";n.d(t,{TS:function(){return a}});let r="undefined"!=typeof CSSINJS_STATISTIC,o=!0;function a(){for(var e=arguments.length,t=Array(e),n=0;n{Object.keys(e).forEach(t=>{Object.defineProperty(a,t,{configurable:!0,enumerable:!0,get:()=>e[t]})})}),o=!0,a}let i={};function c(){}t.ZP=e=>{let t;let n=e,a=c;return r&&"undefined"!=typeof Proxy&&(t=new Set,n=new Proxy(e,{get:(e,n)=>(o&&t.add(n),e[n])}),a=(e,n)=>{var r;i[e]={global:Array.from(t),component:Object.assign(Object.assign({},null===(r=i[e])||void 0===r?void 0:r.component),n)}}),{token:n,keys:t,flush:a}}},36198:function(e,t,n){"use strict";var r=n(352),o=n(12918),a=n(29961);t.Z=(e,t)=>{let[n,i]=(0,a.ZP)();return(0,r.xy)({theme:n,token:i,hashId:"",path:["ant-design-icons",e],nonce:()=>null==t?void 0:t.nonce},()=>[{[".".concat(e)]:Object.assign(Object.assign({},(0,o.Ro)()),{[".".concat(e," .").concat(e,"-icon")]:{display:"block"}})}])}},89970:function(e,t,n){"use strict";n.d(t,{Z:function(){return N}});var r=n(2265),o=n(36760),a=n.n(o),i=n(5769),c=n(50506),l=n(62236),s=n(68710),u=n(92736),d=n(19722),f=n(13613),p=n(95140),m=n(71744),g=n(65658),h=n(29961),v=n(12918),b=n(691),y=n(88260),w=n(18536),x=n(3104),E=n(80669),S=n(352),C=n(34442);let Z=e=>{let{componentCls:t,tooltipMaxWidth:n,tooltipColor:r,tooltipBg:o,tooltipBorderRadius:a,zIndexPopup:i,controlHeight:c,boxShadowSecondary:l,paddingSM:s,paddingXS:u}=e;return[{[t]:Object.assign(Object.assign(Object.assign(Object.assign({},(0,v.Wf)(e)),{position:"absolute",zIndex:i,display:"block",width:"max-content",maxWidth:n,visibility:"visible",transformOrigin:"var(--arrow-x, 50%) var(--arrow-y, 50%)","&-hidden":{display:"none"},"--antd-arrow-background-color":o,["".concat(t,"-inner")]:{minWidth:c,minHeight:c,padding:"".concat((0,S.bf)(e.calc(s).div(2).equal())," ").concat((0,S.bf)(u)),color:r,textAlign:"start",textDecoration:"none",wordWrap:"break-word",backgroundColor:o,borderRadius:a,boxShadow:l,boxSizing:"border-box"},"&-placement-left,&-placement-leftTop,&-placement-leftBottom,&-placement-right,&-placement-rightTop,&-placement-rightBottom":{["".concat(t,"-inner")]:{borderRadius:e.min(a,y.qN)}},["".concat(t,"-content")]:{position:"relative"}}),(0,w.Z)(e,(e,n)=>{let{darkColor:r}=n;return{["&".concat(t,"-").concat(e)]:{["".concat(t,"-inner")]:{backgroundColor:r},["".concat(t,"-arrow")]:{"--antd-arrow-background-color":r}}}})),{"&-rtl":{direction:"rtl"}})},(0,y.ZP)(e,"var(--antd-arrow-background-color)"),{["".concat(t,"-pure")]:{position:"relative",maxWidth:"none",margin:e.sizePopupArrow}}]},O=e=>Object.assign(Object.assign({zIndexPopup:e.zIndexPopupBase+70},(0,y.wZ)({contentRadius:e.borderRadius,limitVerticalRadius:!0})),(0,C.w)((0,x.TS)(e,{borderRadiusOuter:Math.min(e.borderRadiusOuter,4)})));function k(e){let t=!(arguments.length>1)||void 0===arguments[1]||arguments[1];return(0,E.I$)("Tooltip",e=>{let{borderRadius:t,colorTextLightSolid:n,colorBgSpotlight:r}=e;return[Z((0,x.TS)(e,{tooltipMaxWidth:250,tooltipColor:n,tooltipBorderRadius:t,tooltipBg:r})),(0,b._y)(e,"zoom-big-fast")]},O,{resetStyle:!1,injectStyle:t})(e)}var M=n(93350);function j(e,t){let n=(0,M.o2)(t),r=a()({["".concat(e,"-").concat(t)]:t&&n}),o={},i={};return t&&!n&&(o.background=t,i["--antd-arrow-background-color"]=t),{className:r,overlayStyle:o,arrowStyle:i}}var I=function(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&0>t.indexOf(r)&&(n[r]=e[r]);if(null!=e&&"function"==typeof Object.getOwnPropertySymbols)for(var o=0,r=Object.getOwnPropertySymbols(e);ot.indexOf(r[o])&&Object.prototype.propertyIsEnumerable.call(e,r[o])&&(n[r[o]]=e[r[o]]);return n};let R=r.forwardRef((e,t)=>{var n,o;let{prefixCls:v,openClassName:b,getTooltipContainer:y,overlayClassName:w,color:x,overlayInnerStyle:E,children:S,afterOpenChange:C,afterVisibleChange:Z,destroyTooltipOnHide:O,arrow:M=!0,title:R,overlay:N,builtinPlacements:P,arrowPointAtCenter:F=!1,autoAdjustOverflow:T=!0}=e,A=!!M,[,L]=(0,h.ZP)(),{getPopupContainer:z,getPrefixCls:_,direction:H}=r.useContext(m.E_),B=(0,f.ln)("Tooltip"),D=r.useRef(null),W=()=>{var e;null===(e=D.current)||void 0===e||e.forceAlign()};r.useImperativeHandle(t,()=>({forceAlign:W,forcePopupAlign:()=>{B.deprecated(!1,"forcePopupAlign","forceAlign"),W()}}));let[V,q]=(0,c.Z)(!1,{value:null!==(n=e.open)&&void 0!==n?n:e.visible,defaultValue:null!==(o=e.defaultOpen)&&void 0!==o?o:e.defaultVisible}),G=!R&&!N&&0!==R,X=r.useMemo(()=>{var e,t;let n=F;return"object"==typeof M&&(n=null!==(t=null!==(e=M.pointAtCenter)&&void 0!==e?e:M.arrowPointAtCenter)&&void 0!==t?t:F),P||(0,u.Z)({arrowPointAtCenter:n,autoAdjustOverflow:T,arrowWidth:A?L.sizePopupArrow:0,borderRadius:L.borderRadius,offset:L.marginXXS,visibleFirst:!0})},[F,M,P,L]),U=r.useMemo(()=>0===R?R:N||R||"",[N,R]),$=r.createElement(g.BR,null,"function"==typeof U?U():U),{getPopupContainer:K,placement:Y="top",mouseEnterDelay:Q=.1,mouseLeaveDelay:J=.1,overlayStyle:ee,rootClassName:et}=e,en=I(e,["getPopupContainer","placement","mouseEnterDelay","mouseLeaveDelay","overlayStyle","rootClassName"]),er=_("tooltip",v),eo=_(),ea=e["data-popover-inject"],ei=V;"open"in e||"visible"in e||!G||(ei=!1);let ec=(0,d.l$)(S)&&!(0,d.M2)(S)?S:r.createElement("span",null,S),el=ec.props,es=el.className&&"string"!=typeof el.className?el.className:a()(el.className,b||"".concat(er,"-open")),[eu,ed,ef]=k(er,!ea),ep=j(er,x),em=ep.arrowStyle,eg=Object.assign(Object.assign({},E),ep.overlayStyle),eh=a()(w,{["".concat(er,"-rtl")]:"rtl"===H},ep.className,et,ed,ef),[ev,eb]=(0,l.Cn)("Tooltip",en.zIndex),ey=r.createElement(i.Z,Object.assign({},en,{zIndex:ev,showArrow:A,placement:Y,mouseEnterDelay:Q,mouseLeaveDelay:J,prefixCls:er,overlayClassName:eh,overlayStyle:Object.assign(Object.assign({},em),ee),getTooltipContainer:K||y||z,ref:D,builtinPlacements:X,overlay:$,visible:ei,onVisibleChange:t=>{var n,r;q(!G&&t),G||(null===(n=e.onOpenChange)||void 0===n||n.call(e,t),null===(r=e.onVisibleChange)||void 0===r||r.call(e,t))},afterVisibleChange:null!=C?C:Z,overlayInnerStyle:eg,arrowContent:r.createElement("span",{className:"".concat(er,"-arrow-content")}),motion:{motionName:(0,s.m)(eo,"zoom-big-fast",e.transitionName),motionDeadline:1e3},destroyTooltipOnHide:!!O}),ei?(0,d.Tm)(ec,{className:es}):ec);return eu(r.createElement(p.Z.Provider,{value:eb},ey))});R._InternalPanelDoNotUseOrYouWillBeFired=e=>{let{prefixCls:t,className:n,placement:o="top",title:c,color:l,overlayInnerStyle:s}=e,{getPrefixCls:u}=r.useContext(m.E_),d=u("tooltip",t),[f,p,g]=k(d),h=j(d,l),v=h.arrowStyle,b=Object.assign(Object.assign({},s),h.overlayStyle),y=a()(p,g,d,"".concat(d,"-pure"),"".concat(d,"-placement-").concat(o),n,h.className);return f(r.createElement("div",{className:y,style:v},r.createElement("div",{className:"".concat(d,"-arrow")}),r.createElement(i.G,Object.assign({},e,{className:p,prefixCls:d,overlayInnerStyle:b}),c)))};var N=R},99376:function(e,t,n){"use strict";var r=n(35475);n.o(r,"usePathname")&&n.d(t,{usePathname:function(){return r.usePathname}}),n.o(r,"useRouter")&&n.d(t,{useRouter:function(){return r.useRouter}}),n.o(r,"useSearchParams")&&n.d(t,{useSearchParams:function(){return r.useSearchParams}})},40257:function(e,t,n){"use strict";var r,o;e.exports=(null==(r=n.g.process)?void 0:r.env)&&"object"==typeof(null==(o=n.g.process)?void 0:o.env)?n.g.process:n(44227)},44227:function(e){!function(){var t={229:function(e){var t,n,r,o=e.exports={};function a(){throw Error("setTimeout has not been defined")}function i(){throw Error("clearTimeout has not been defined")}function c(e){if(t===setTimeout)return setTimeout(e,0);if((t===a||!t)&&setTimeout)return t=setTimeout,setTimeout(e,0);try{return t(e,0)}catch(n){try{return t.call(null,e,0)}catch(n){return t.call(this,e,0)}}}!function(){try{t="function"==typeof setTimeout?setTimeout:a}catch(e){t=a}try{n="function"==typeof clearTimeout?clearTimeout:i}catch(e){n=i}}();var l=[],s=!1,u=-1;function d(){s&&r&&(s=!1,r.length?l=r.concat(l):u=-1,l.length&&f())}function f(){if(!s){var e=c(d);s=!0;for(var t=l.length;t;){for(r=l,l=[];++u1)for(var n=1;n1?t-1:0),r=1;r=a)return e;switch(e){case"%s":return String(n[o++]);case"%d":return Number(n[o++]);case"%j":try{return JSON.stringify(n[o++])}catch(e){return"[Circular]"}break;default:return e}}):e}function F(e,t){return!!(null==e||"array"===t&&Array.isArray(e)&&!e.length)||("string"===t||"url"===t||"hex"===t||"email"===t||"date"===t||"pattern"===t)&&"string"==typeof e&&!e}function T(e,t,n){var r=0,o=e.length;!function a(i){if(i&&i.length){n(i);return}var c=r;r+=1,c()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-Z\-0-9\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF]+\.)+[a-zA-Z\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF]{2,}))$/,hex:/^#?([a-f0-9]{6}|[a-f0-9]{3})$/i},D={integer:function(e){return D.number(e)&&parseInt(e,10)===e},float:function(e){return D.number(e)&&!D.integer(e)},array:function(e){return Array.isArray(e)},regexp:function(e){if(e instanceof RegExp)return!0;try{return new RegExp(e),!0}catch(e){return!1}},date:function(e){return"function"==typeof e.getTime&&"function"==typeof e.getMonth&&"function"==typeof e.getYear&&!isNaN(e.getTime())},number:function(e){return!isNaN(e)&&"number"==typeof e},object:function(e){return"object"==typeof e&&!D.array(e)},method:function(e){return"function"==typeof e},email:function(e){return"string"==typeof e&&e.length<=320&&!!e.match(B.email)},url:function(e){return"string"==typeof e&&e.length<=2048&&!!e.match(H())},hex:function(e){return"string"==typeof e&&!!e.match(B.hex)}},W="enum",V={required:_,whitespace:function(e,t,n,r,o){(/^\s+$/.test(t)||""===t)&&r.push(P(o.messages.whitespace,e.fullField))},type:function(e,t,n,r,o){if(e.required&&void 0===t){_(e,t,n,r,o);return}var a=e.type;["integer","float","array","regexp","object","method","email","number","date","url","hex"].indexOf(a)>-1?D[a](t)||r.push(P(o.messages.types[a],e.fullField,e.type)):a&&typeof t!==e.type&&r.push(P(o.messages.types[a],e.fullField,e.type))},range:function(e,t,n,r,o){var a="number"==typeof e.len,i="number"==typeof e.min,c="number"==typeof e.max,l=t,s=null,u="number"==typeof t,d="string"==typeof t,f=Array.isArray(t);if(u?s="number":d?s="string":f&&(s="array"),!s)return!1;f&&(l=t.length),d&&(l=t.replace(/[\uD800-\uDBFF][\uDC00-\uDFFF]/g,"_").length),a?l!==e.len&&r.push(P(o.messages[s].len,e.fullField,e.len)):i&&!c&&le.max?r.push(P(o.messages[s].max,e.fullField,e.max)):i&&c&&(le.max)&&r.push(P(o.messages[s].range,e.fullField,e.min,e.max))},enum:function(e,t,n,r,o){e[W]=Array.isArray(e[W])?e[W]:[],-1===e[W].indexOf(t)&&r.push(P(o.messages[W],e.fullField,e[W].join(", ")))},pattern:function(e,t,n,r,o){!e.pattern||(e.pattern instanceof RegExp?(e.pattern.lastIndex=0,e.pattern.test(t)||r.push(P(o.messages.pattern.mismatch,e.fullField,t,e.pattern))):"string"!=typeof e.pattern||new RegExp(e.pattern).test(t)||r.push(P(o.messages.pattern.mismatch,e.fullField,t,e.pattern)))}},q=function(e,t,n,r,o){var a=e.type,i=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t,a)&&!e.required)return n();V.required(e,t,r,i,o,a),F(t,a)||V.type(e,t,r,i,o)}n(i)},G={string:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t,"string")&&!e.required)return n();V.required(e,t,r,a,o,"string"),F(t,"string")||(V.type(e,t,r,a,o),V.range(e,t,r,a,o),V.pattern(e,t,r,a,o),!0===e.whitespace&&V.whitespace(e,t,r,a,o))}n(a)},method:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t)&&!e.required)return n();V.required(e,t,r,a,o),void 0!==t&&V.type(e,t,r,a,o)}n(a)},number:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(""===t&&(t=void 0),F(t)&&!e.required)return n();V.required(e,t,r,a,o),void 0!==t&&(V.type(e,t,r,a,o),V.range(e,t,r,a,o))}n(a)},boolean:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t)&&!e.required)return n();V.required(e,t,r,a,o),void 0!==t&&V.type(e,t,r,a,o)}n(a)},regexp:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t)&&!e.required)return n();V.required(e,t,r,a,o),F(t)||V.type(e,t,r,a,o)}n(a)},integer:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t)&&!e.required)return n();V.required(e,t,r,a,o),void 0!==t&&(V.type(e,t,r,a,o),V.range(e,t,r,a,o))}n(a)},float:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t)&&!e.required)return n();V.required(e,t,r,a,o),void 0!==t&&(V.type(e,t,r,a,o),V.range(e,t,r,a,o))}n(a)},array:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(null==t&&!e.required)return n();V.required(e,t,r,a,o,"array"),null!=t&&(V.type(e,t,r,a,o),V.range(e,t,r,a,o))}n(a)},object:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t)&&!e.required)return n();V.required(e,t,r,a,o),void 0!==t&&V.type(e,t,r,a,o)}n(a)},enum:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t)&&!e.required)return n();V.required(e,t,r,a,o),void 0!==t&&V.enum(e,t,r,a,o)}n(a)},pattern:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t,"string")&&!e.required)return n();V.required(e,t,r,a,o),F(t,"string")||V.pattern(e,t,r,a,o)}n(a)},date:function(e,t,n,r,o){var a,i=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t,"date")&&!e.required)return n();V.required(e,t,r,i,o),!F(t,"date")&&(a=t instanceof Date?t:new Date(t),V.type(e,a,r,i,o),a&&V.range(e,a.getTime(),r,i,o))}n(i)},url:q,hex:q,email:q,required:function(e,t,n,r,o){var a=[],i=Array.isArray(t)?"array":typeof t;V.required(e,t,r,a,o,i),n(a)},any:function(e,t,n,r,o){var a=[];if(e.required||!e.required&&r.hasOwnProperty(e.field)){if(F(t)&&!e.required)return n();V.required(e,t,r,a,o)}n(a)}};function X(){return{default:"Validation error on field %s",required:"%s is required",enum:"%s must be one of %s",whitespace:"%s cannot be empty",date:{format:"%s date %s is invalid for format %s",parse:"%s date could not be parsed, %s is invalid ",invalid:"%s date %s is invalid"},types:{string:"%s is not a %s",method:"%s is not a %s (function)",array:"%s is not an %s",object:"%s is not an %s",number:"%s is not a %s",date:"%s is not a %s",boolean:"%s is not a %s",integer:"%s is not an %s",float:"%s is not a %s",regexp:"%s is not a valid %s",email:"%s is not a valid %s",url:"%s is not a valid %s",hex:"%s is not a valid %s"},string:{len:"%s must be exactly %s characters",min:"%s must be at least %s characters",max:"%s cannot be longer than %s characters",range:"%s must be between %s and %s characters"},number:{len:"%s must equal %s",min:"%s cannot be less than %s",max:"%s cannot be greater than %s",range:"%s must be between %s and %s"},array:{len:"%s must be exactly %s in length",min:"%s cannot be less than %s in length",max:"%s cannot be greater than %s in length",range:"%s must be between %s and %s in length"},pattern:{mismatch:"%s value %s does not match pattern %s"},clone:function(){var e=JSON.parse(JSON.stringify(this));return e.clone=this.clone,e}}}var U=X(),$=function(){function e(e){this.rules=null,this._messages=U,this.define(e)}var t=e.prototype;return t.define=function(e){var t=this;if(!e)throw Error("Cannot configure a schema with no rules");if("object"!=typeof e||Array.isArray(e))throw Error("Rules must be an object");this.rules={},Object.keys(e).forEach(function(n){var r=e[n];t.rules[n]=Array.isArray(r)?r:[r]})},t.messages=function(e){return e&&(this._messages=z(X(),e)),this._messages},t.validate=function(t,n,r){var o=this;void 0===n&&(n={}),void 0===r&&(r=function(){});var a=t,i=n,c=r;if("function"==typeof i&&(c=i,i={}),!this.rules||0===Object.keys(this.rules).length)return c&&c(null,a),Promise.resolve(a);if(i.messages){var l=this.messages();l===U&&(l=X()),z(l,i.messages),i.messages=l}else i.messages=this.messages();var s={};(i.keys||Object.keys(this.rules)).forEach(function(e){var n=o.rules[e],r=a[e];n.forEach(function(n){var i=n;"function"==typeof i.transform&&(a===t&&(a=O({},a)),r=a[e]=i.transform(r)),(i="function"==typeof i?{validator:i}:O({},i)).validator=o.getValidationMethod(i),i.validator&&(i.field=e,i.fullField=i.fullField||e,i.type=o.getType(i),s[e]=s[e]||[],s[e].push({rule:i,value:r,source:a,field:e}))})});var u={};return function(e,t,n,r,o){if(t.first){var a=new Promise(function(t,a){var i;T((i=[],Object.keys(e).forEach(function(t){i.push.apply(i,e[t]||[])}),i),n,function(e){return r(e),e.length?a(new A(e,N(e))):t(o)})});return a.catch(function(e){return e}),a}var i=!0===t.firstFields?Object.keys(e):t.firstFields||[],c=Object.keys(e),l=c.length,s=0,u=[],d=new Promise(function(t,a){var d=function(e){if(u.push.apply(u,e),++s===l)return r(u),u.length?a(new A(u,N(u))):t(o)};c.length||(r(u),t(o)),c.forEach(function(t){var r=e[t];-1!==i.indexOf(t)?T(r,n,d):function(e,t,n){var r=[],o=0,a=e.length;function i(e){r.push.apply(r,e||[]),++o===a&&n(r)}e.forEach(function(e){t(e,i)})}(r,n,d)})});return d.catch(function(e){return e}),d}(s,i,function(t,n){var r,o=t.rule,c=("object"===o.type||"array"===o.type)&&("object"==typeof o.fields||"object"==typeof o.defaultField);function l(e,t){return O({},t,{fullField:o.fullField+"."+e,fullFields:o.fullFields?[].concat(o.fullFields,[e]):[e]})}function s(r){void 0===r&&(r=[]);var s=Array.isArray(r)?r:[r];!i.suppressWarning&&s.length&&e.warning("async-validator:",s),s.length&&void 0!==o.message&&(s=[].concat(o.message));var d=s.map(L(o,a));if(i.first&&d.length)return u[o.field]=1,n(d);if(c){if(o.required&&!t.value)return void 0!==o.message?d=[].concat(o.message).map(L(o,a)):i.error&&(d=[i.error(o,P(i.messages.required,o.field))]),n(d);var f={};o.defaultField&&Object.keys(t.value).map(function(e){f[e]=o.defaultField});var p={};Object.keys(f=O({},f,t.rule.fields)).forEach(function(e){var t=f[e],n=Array.isArray(t)?t:[t];p[e]=n.map(l.bind(null,e))});var m=new e(p);m.messages(i.messages),t.rule.options&&(t.rule.options.messages=i.messages,t.rule.options.error=i.error),m.validate(t.value,t.rule.options||i,function(e){var t=[];d&&d.length&&t.push.apply(t,d),e&&e.length&&t.push.apply(t,e),n(t.length?t:null)})}else n(d)}if(c=c&&(o.required||!o.required&&t.value),o.field=t.field,o.asyncValidator)r=o.asyncValidator(o,t.value,s,t.source,i);else if(o.validator){try{r=o.validator(o,t.value,s,t.source,i)}catch(e){null==console.error||console.error(e),i.suppressValidatorError||setTimeout(function(){throw e},0),s(e.message)}!0===r?s():!1===r?s("function"==typeof o.message?o.message(o.fullField||o.field):o.message||(o.fullField||o.field)+" fails"):r instanceof Array?s(r):r instanceof Error&&s(r.message)}r&&r.then&&r.then(function(){return s()},function(e){return s(e)})},function(e){!function(e){for(var t=[],n={},r=0;r2&&void 0!==arguments[2]&&arguments[2];return e&&e.some(function(e){return es(t,e,n)})}function es(e,t){var n=arguments.length>2&&void 0!==arguments[2]&&arguments[2];return!!e&&!!t&&(!!n||e.length===t.length)&&t.every(function(t,n){return e[n]===t})}function eu(e){var t=arguments.length<=1?void 0:arguments[1];return t&&t.target&&"object"===(0,eo.Z)(t.target)&&e in t.target?t.target[e]:t}function ed(e,t,n){var r=e.length;if(t<0||t>=r||n<0||n>=r)return e;var o=e[t],a=t-n;return a>0?[].concat((0,u.Z)(e.slice(0,n)),[o],(0,u.Z)(e.slice(n,t)),(0,u.Z)(e.slice(t+1,r))):a<0?[].concat((0,u.Z)(e.slice(0,t)),(0,u.Z)(e.slice(t+1,n+1)),[o],(0,u.Z)(e.slice(n+1,r))):e}var ef=["name"],ep=[];function em(e,t,n,r,o,a){return"function"==typeof e?e(t,n,"source"in a?{source:a.source}:{}):r!==o}var eg=function(e){(0,m.Z)(n,e);var t=(0,g.Z)(n);function n(e){var r;return(0,d.Z)(this,n),r=t.call(this,e),(0,h.Z)((0,p.Z)(r),"state",{resetCount:0}),(0,h.Z)((0,p.Z)(r),"cancelRegisterFunc",null),(0,h.Z)((0,p.Z)(r),"mounted",!1),(0,h.Z)((0,p.Z)(r),"touched",!1),(0,h.Z)((0,p.Z)(r),"dirty",!1),(0,h.Z)((0,p.Z)(r),"validatePromise",void 0),(0,h.Z)((0,p.Z)(r),"prevValidating",void 0),(0,h.Z)((0,p.Z)(r),"errors",ep),(0,h.Z)((0,p.Z)(r),"warnings",ep),(0,h.Z)((0,p.Z)(r),"cancelRegister",function(){var e=r.props,t=e.preserve,n=e.isListField,o=e.name;r.cancelRegisterFunc&&r.cancelRegisterFunc(n,t,ei(o)),r.cancelRegisterFunc=null}),(0,h.Z)((0,p.Z)(r),"getNamePath",function(){var e=r.props,t=e.name,n=e.fieldContext.prefixName;return void 0!==t?[].concat((0,u.Z)(void 0===n?[]:n),(0,u.Z)(t)):[]}),(0,h.Z)((0,p.Z)(r),"getRules",function(){var e=r.props,t=e.rules,n=e.fieldContext;return(void 0===t?[]:t).map(function(e){return"function"==typeof e?e(n):e})}),(0,h.Z)((0,p.Z)(r),"refresh",function(){r.mounted&&r.setState(function(e){return{resetCount:e.resetCount+1}})}),(0,h.Z)((0,p.Z)(r),"metaCache",null),(0,h.Z)((0,p.Z)(r),"triggerMetaEvent",function(e){var t=r.props.onMetaChange;if(t){var n=(0,s.Z)((0,s.Z)({},r.getMeta()),{},{destroy:e});(0,b.Z)(r.metaCache,n)||t(n),r.metaCache=n}else r.metaCache=null}),(0,h.Z)((0,p.Z)(r),"onStoreChange",function(e,t,n){var o=r.props,a=o.shouldUpdate,i=o.dependencies,c=void 0===i?[]:i,l=o.onReset,s=n.store,u=r.getNamePath(),d=r.getValue(e),f=r.getValue(s),p=t&&el(t,u);switch("valueUpdate"===n.type&&"external"===n.source&&d!==f&&(r.touched=!0,r.dirty=!0,r.validatePromise=null,r.errors=ep,r.warnings=ep,r.triggerMetaEvent()),n.type){case"reset":if(!t||p){r.touched=!1,r.dirty=!1,r.validatePromise=void 0,r.errors=ep,r.warnings=ep,r.triggerMetaEvent(),null==l||l(),r.refresh();return}break;case"remove":if(a){r.reRender();return}break;case"setField":var m=n.data;if(p){"touched"in m&&(r.touched=m.touched),"validating"in m&&!("originRCField"in m)&&(r.validatePromise=m.validating?Promise.resolve([]):null),"errors"in m&&(r.errors=m.errors||ep),"warnings"in m&&(r.warnings=m.warnings||ep),r.dirty=!0,r.triggerMetaEvent(),r.reRender();return}if("value"in m&&el(t,u,!0)||a&&!u.length&&em(a,e,s,d,f,n)){r.reRender();return}break;case"dependenciesUpdate":if(c.map(ei).some(function(e){return el(n.relatedFields,e)})){r.reRender();return}break;default:if(p||(!c.length||u.length||a)&&em(a,e,s,d,f,n)){r.reRender();return}}!0===a&&r.reRender()}),(0,h.Z)((0,p.Z)(r),"validateRules",function(e){var t=r.getNamePath(),n=r.getValue(),o=e||{},a=o.triggerName,i=o.validateOnly,d=Promise.resolve().then((0,l.Z)((0,c.Z)().mark(function o(){var i,f,p,m,g,h,v;return(0,c.Z)().wrap(function(o){for(;;)switch(o.prev=o.next){case 0:if(r.mounted){o.next=2;break}return o.abrupt("return",[]);case 2:if(p=void 0!==(f=(i=r.props).validateFirst)&&f,m=i.messageVariables,g=i.validateDebounce,h=r.getRules(),a&&(h=h.filter(function(e){return e}).filter(function(e){var t=e.validateTrigger;return!t||C(t).includes(a)})),!(g&&a)){o.next=10;break}return o.next=8,new Promise(function(e){setTimeout(e,g)});case 8:if(!(r.validatePromise!==d)){o.next=10;break}return o.abrupt("return",[]);case 10:return(v=function(e,t,n,r,o,a){var i,u,d=e.join("."),f=n.map(function(e,t){var n=e.validator,r=(0,s.Z)((0,s.Z)({},e),{},{ruleIndex:t});return n&&(r.validator=function(e,t,r){var o=!1,a=n(e,t,function(){for(var e=arguments.length,t=Array(e),n=0;n0&&void 0!==arguments[0]?arguments[0]:ep;if(r.validatePromise===d){r.validatePromise=null;var t,n=[],o=[];null===(t=e.forEach)||void 0===t||t.call(e,function(e){var t=e.rule.warningOnly,r=e.errors,a=void 0===r?ep:r;t?o.push.apply(o,(0,u.Z)(a)):n.push.apply(n,(0,u.Z)(a))}),r.errors=n,r.warnings=o,r.triggerMetaEvent(),r.reRender()}}),o.abrupt("return",v);case 13:case"end":return o.stop()}},o)})));return void 0!==i&&i||(r.validatePromise=d,r.dirty=!0,r.errors=ep,r.warnings=ep,r.triggerMetaEvent(),r.reRender()),d}),(0,h.Z)((0,p.Z)(r),"isFieldValidating",function(){return!!r.validatePromise}),(0,h.Z)((0,p.Z)(r),"isFieldTouched",function(){return r.touched}),(0,h.Z)((0,p.Z)(r),"isFieldDirty",function(){return!!r.dirty||void 0!==r.props.initialValue||void 0!==(0,r.props.fieldContext.getInternalHooks(w).getInitialValue)(r.getNamePath())}),(0,h.Z)((0,p.Z)(r),"getErrors",function(){return r.errors}),(0,h.Z)((0,p.Z)(r),"getWarnings",function(){return r.warnings}),(0,h.Z)((0,p.Z)(r),"isListField",function(){return r.props.isListField}),(0,h.Z)((0,p.Z)(r),"isList",function(){return r.props.isList}),(0,h.Z)((0,p.Z)(r),"isPreserve",function(){return r.props.preserve}),(0,h.Z)((0,p.Z)(r),"getMeta",function(){return r.prevValidating=r.isFieldValidating(),{touched:r.isFieldTouched(),validating:r.prevValidating,errors:r.errors,warnings:r.warnings,name:r.getNamePath(),validated:null===r.validatePromise}}),(0,h.Z)((0,p.Z)(r),"getOnlyChild",function(e){if("function"==typeof e){var t=r.getMeta();return(0,s.Z)((0,s.Z)({},r.getOnlyChild(e(r.getControlled(),t,r.props.fieldContext))),{},{isFunction:!0})}var n=(0,v.Z)(e);return 1===n.length&&o.isValidElement(n[0])?{child:n[0],isFunction:!1}:{child:n,isFunction:!1}}),(0,h.Z)((0,p.Z)(r),"getValue",function(e){var t=r.props.fieldContext.getFieldsValue,n=r.getNamePath();return(0,ea.Z)(e||t(!0),n)}),(0,h.Z)((0,p.Z)(r),"getControlled",function(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{},t=r.props,n=t.trigger,o=t.validateTrigger,a=t.getValueFromEvent,i=t.normalize,c=t.valuePropName,l=t.getValueProps,u=t.fieldContext,d=void 0!==o?o:u.validateTrigger,f=r.getNamePath(),p=u.getInternalHooks,m=u.getFieldsValue,g=p(w).dispatch,v=r.getValue(),b=l||function(e){return(0,h.Z)({},c,e)},y=e[n],x=(0,s.Z)((0,s.Z)({},e),b(v));return x[n]=function(){r.touched=!0,r.dirty=!0,r.triggerMetaEvent();for(var e,t=arguments.length,n=Array(t),o=0;o=0&&t<=n.length?(f.keys=[].concat((0,u.Z)(f.keys.slice(0,t)),[f.id],(0,u.Z)(f.keys.slice(t))),o([].concat((0,u.Z)(n.slice(0,t)),[e],(0,u.Z)(n.slice(t))))):(f.keys=[].concat((0,u.Z)(f.keys),[f.id]),o([].concat((0,u.Z)(n),[e]))),f.id+=1},remove:function(e){var t=i(),n=new Set(Array.isArray(e)?e:[e]);n.size<=0||(f.keys=f.keys.filter(function(e,t){return!n.has(t)}),o(t.filter(function(e,t){return!n.has(t)})))},move:function(e,t){if(e!==t){var n=i();e<0||e>=n.length||t<0||t>=n.length||(f.keys=ed(f.keys,e,t),o(ed(n,e,t)))}}},t)})))},eb=n(26365),ey="__@field_split__";function ew(e){return e.map(function(e){return"".concat((0,eo.Z)(e),":").concat(e)}).join(ey)}var ex=function(){function e(){(0,d.Z)(this,e),(0,h.Z)(this,"kvs",new Map)}return(0,f.Z)(e,[{key:"set",value:function(e,t){this.kvs.set(ew(e),t)}},{key:"get",value:function(e){return this.kvs.get(ew(e))}},{key:"update",value:function(e,t){var n=t(this.get(e));n?this.set(e,n):this.delete(e)}},{key:"delete",value:function(e){this.kvs.delete(ew(e))}},{key:"map",value:function(e){return(0,u.Z)(this.kvs.entries()).map(function(t){var n=(0,eb.Z)(t,2),r=n[0],o=n[1];return e({key:r.split(ey).map(function(e){var t=e.match(/^([^:]*):(.*)$/),n=(0,eb.Z)(t,3),r=n[1],o=n[2];return"number"===r?Number(o):o}),value:o})})}},{key:"toJSON",value:function(){var e={};return this.map(function(t){var n=t.key,r=t.value;return e[n.join(".")]=r,null}),e}}]),e}(),eE=["name"],eS=(0,f.Z)(function e(t){var n=this;(0,d.Z)(this,e),(0,h.Z)(this,"formHooked",!1),(0,h.Z)(this,"forceRootUpdate",void 0),(0,h.Z)(this,"subscribable",!0),(0,h.Z)(this,"store",{}),(0,h.Z)(this,"fieldEntities",[]),(0,h.Z)(this,"initialValues",{}),(0,h.Z)(this,"callbacks",{}),(0,h.Z)(this,"validateMessages",null),(0,h.Z)(this,"preserve",null),(0,h.Z)(this,"lastValidatePromise",null),(0,h.Z)(this,"getForm",function(){return{getFieldValue:n.getFieldValue,getFieldsValue:n.getFieldsValue,getFieldError:n.getFieldError,getFieldWarning:n.getFieldWarning,getFieldsError:n.getFieldsError,isFieldsTouched:n.isFieldsTouched,isFieldTouched:n.isFieldTouched,isFieldValidating:n.isFieldValidating,isFieldsValidating:n.isFieldsValidating,resetFields:n.resetFields,setFields:n.setFields,setFieldValue:n.setFieldValue,setFieldsValue:n.setFieldsValue,validateFields:n.validateFields,submit:n.submit,_init:!0,getInternalHooks:n.getInternalHooks}}),(0,h.Z)(this,"getInternalHooks",function(e){return e===w?(n.formHooked=!0,{dispatch:n.dispatch,initEntityValue:n.initEntityValue,registerField:n.registerField,useSubscribe:n.useSubscribe,setInitialValues:n.setInitialValues,destroyForm:n.destroyForm,setCallbacks:n.setCallbacks,setValidateMessages:n.setValidateMessages,getFields:n.getFields,setPreserve:n.setPreserve,getInitialValue:n.getInitialValue,registerWatch:n.registerWatch}):((0,y.ZP)(!1,"`getInternalHooks` is internal usage. Should not call directly."),null)}),(0,h.Z)(this,"useSubscribe",function(e){n.subscribable=e}),(0,h.Z)(this,"prevWithoutPreserves",null),(0,h.Z)(this,"setInitialValues",function(e,t){if(n.initialValues=e||{},t){var r,o=(0,Q.T)(e,n.store);null===(r=n.prevWithoutPreserves)||void 0===r||r.map(function(t){var n=t.key;o=(0,Q.Z)(o,n,(0,ea.Z)(e,n))}),n.prevWithoutPreserves=null,n.updateStore(o)}}),(0,h.Z)(this,"destroyForm",function(){var e=new ex;n.getFieldEntities(!0).forEach(function(t){n.isMergedPreserve(t.isPreserve())||e.set(t.getNamePath(),!0)}),n.prevWithoutPreserves=e}),(0,h.Z)(this,"getInitialValue",function(e){var t=(0,ea.Z)(n.initialValues,e);return e.length?(0,Q.T)(t):t}),(0,h.Z)(this,"setCallbacks",function(e){n.callbacks=e}),(0,h.Z)(this,"setValidateMessages",function(e){n.validateMessages=e}),(0,h.Z)(this,"setPreserve",function(e){n.preserve=e}),(0,h.Z)(this,"watchList",[]),(0,h.Z)(this,"registerWatch",function(e){return n.watchList.push(e),function(){n.watchList=n.watchList.filter(function(t){return t!==e})}}),(0,h.Z)(this,"notifyWatch",function(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:[];if(n.watchList.length){var t=n.getFieldsValue(),r=n.getFieldsValue(!0);n.watchList.forEach(function(n){n(t,r,e)})}}),(0,h.Z)(this,"timeoutId",null),(0,h.Z)(this,"warningUnhooked",function(){}),(0,h.Z)(this,"updateStore",function(e){n.store=e}),(0,h.Z)(this,"getFieldEntities",function(){var e=arguments.length>0&&void 0!==arguments[0]&&arguments[0];return e?n.fieldEntities.filter(function(e){return e.getNamePath().length}):n.fieldEntities}),(0,h.Z)(this,"getFieldsMap",function(){var e=arguments.length>0&&void 0!==arguments[0]&&arguments[0],t=new ex;return n.getFieldEntities(e).forEach(function(e){var n=e.getNamePath();t.set(n,e)}),t}),(0,h.Z)(this,"getFieldEntitiesForNamePathList",function(e){if(!e)return n.getFieldEntities(!0);var t=n.getFieldsMap(!0);return e.map(function(e){var n=ei(e);return t.get(n)||{INVALIDATE_NAME_PATH:ei(e)}})}),(0,h.Z)(this,"getFieldsValue",function(e,t){if(n.warningUnhooked(),!0===e||Array.isArray(e)?(r=e,o=t):e&&"object"===(0,eo.Z)(e)&&(a=e.strict,o=e.filter),!0===r&&!o)return n.store;var r,o,a,i=n.getFieldEntitiesForNamePathList(Array.isArray(r)?r:null),c=[];return i.forEach(function(e){var t,n,i,l="INVALIDATE_NAME_PATH"in e?e.INVALIDATE_NAME_PATH:e.getNamePath();if(a){if(null!==(i=e.isList)&&void 0!==i&&i.call(e))return}else if(!r&&null!==(t=(n=e).isListField)&&void 0!==t&&t.call(n))return;if(o){var s="getMeta"in e?e.getMeta():null;o(s)&&c.push(l)}else c.push(l)}),ec(n.store,c.map(ei))}),(0,h.Z)(this,"getFieldValue",function(e){n.warningUnhooked();var t=ei(e);return(0,ea.Z)(n.store,t)}),(0,h.Z)(this,"getFieldsError",function(e){return n.warningUnhooked(),n.getFieldEntitiesForNamePathList(e).map(function(t,n){return!t||"INVALIDATE_NAME_PATH"in t?{name:ei(e[n]),errors:[],warnings:[]}:{name:t.getNamePath(),errors:t.getErrors(),warnings:t.getWarnings()}})}),(0,h.Z)(this,"getFieldError",function(e){n.warningUnhooked();var t=ei(e);return n.getFieldsError([t])[0].errors}),(0,h.Z)(this,"getFieldWarning",function(e){n.warningUnhooked();var t=ei(e);return n.getFieldsError([t])[0].warnings}),(0,h.Z)(this,"isFieldsTouched",function(){n.warningUnhooked();for(var e,t=arguments.length,r=Array(t),o=0;o0&&void 0!==arguments[0]?arguments[0]:{},r=new ex,o=n.getFieldEntities(!0);o.forEach(function(e){var t=e.props.initialValue,n=e.getNamePath();if(void 0!==t){var o=r.get(n)||new Set;o.add({entity:e,value:t}),r.set(n,o)}}),t.entities?e=t.entities:t.namePathList?(e=[],t.namePathList.forEach(function(t){var n,o=r.get(t);o&&(n=e).push.apply(n,(0,u.Z)((0,u.Z)(o).map(function(e){return e.entity})))})):e=o,function(e){e.forEach(function(e){if(void 0!==e.props.initialValue){var o=e.getNamePath();if(void 0!==n.getInitialValue(o))(0,y.ZP)(!1,"Form already set 'initialValues' with path '".concat(o.join("."),"'. Field can not overwrite it."));else{var a=r.get(o);if(a&&a.size>1)(0,y.ZP)(!1,"Multiple Field with path '".concat(o.join("."),"' set 'initialValue'. Can not decide which one to pick."));else if(a){var i=n.getFieldValue(o);e.isListField()||t.skipExist&&void 0!==i||n.updateStore((0,Q.Z)(n.store,o,(0,u.Z)(a)[0].value))}}}})}(e)}),(0,h.Z)(this,"resetFields",function(e){n.warningUnhooked();var t=n.store;if(!e){n.updateStore((0,Q.T)(n.initialValues)),n.resetWithFieldInitialValue(),n.notifyObservers(t,null,{type:"reset"}),n.notifyWatch();return}var r=e.map(ei);r.forEach(function(e){var t=n.getInitialValue(e);n.updateStore((0,Q.Z)(n.store,e,t))}),n.resetWithFieldInitialValue({namePathList:r}),n.notifyObservers(t,r,{type:"reset"}),n.notifyWatch(r)}),(0,h.Z)(this,"setFields",function(e){n.warningUnhooked();var t=n.store,r=[];e.forEach(function(e){var o=e.name,a=(0,i.Z)(e,eE),c=ei(o);r.push(c),"value"in a&&n.updateStore((0,Q.Z)(n.store,c,a.value)),n.notifyObservers(t,[c],{type:"setField",data:e})}),n.notifyWatch(r)}),(0,h.Z)(this,"getFields",function(){return n.getFieldEntities(!0).map(function(e){var t=e.getNamePath(),r=e.getMeta(),o=(0,s.Z)((0,s.Z)({},r),{},{name:t,value:n.getFieldValue(t)});return Object.defineProperty(o,"originRCField",{value:!0}),o})}),(0,h.Z)(this,"initEntityValue",function(e){var t=e.props.initialValue;if(void 0!==t){var r=e.getNamePath();void 0===(0,ea.Z)(n.store,r)&&n.updateStore((0,Q.Z)(n.store,r,t))}}),(0,h.Z)(this,"isMergedPreserve",function(e){var t=void 0!==e?e:n.preserve;return null==t||t}),(0,h.Z)(this,"registerField",function(e){n.fieldEntities.push(e);var t=e.getNamePath();if(n.notifyWatch([t]),void 0!==e.props.initialValue){var r=n.store;n.resetWithFieldInitialValue({entities:[e],skipExist:!0}),n.notifyObservers(r,[e.getNamePath()],{type:"valueUpdate",source:"internal"})}return function(r,o){var a=arguments.length>2&&void 0!==arguments[2]?arguments[2]:[];if(n.fieldEntities=n.fieldEntities.filter(function(t){return t!==e}),!n.isMergedPreserve(o)&&(!r||a.length>1)){var i=r?void 0:n.getInitialValue(t);if(t.length&&n.getFieldValue(t)!==i&&n.fieldEntities.every(function(e){return!es(e.getNamePath(),t)})){var c=n.store;n.updateStore((0,Q.Z)(c,t,i,!0)),n.notifyObservers(c,[t],{type:"remove"}),n.triggerDependenciesUpdate(c,t)}}n.notifyWatch([t])}}),(0,h.Z)(this,"dispatch",function(e){switch(e.type){case"updateValue":var t=e.namePath,r=e.value;n.updateValue(t,r);break;case"validateField":var o=e.namePath,a=e.triggerName;n.validateFields([o],{triggerName:a})}}),(0,h.Z)(this,"notifyObservers",function(e,t,r){if(n.subscribable){var o=(0,s.Z)((0,s.Z)({},r),{},{store:n.getFieldsValue(!0)});n.getFieldEntities().forEach(function(n){(0,n.onStoreChange)(e,t,o)})}else n.forceRootUpdate()}),(0,h.Z)(this,"triggerDependenciesUpdate",function(e,t){var r=n.getDependencyChildrenFields(t);return r.length&&n.validateFields(r),n.notifyObservers(e,r,{type:"dependenciesUpdate",relatedFields:[t].concat((0,u.Z)(r))}),r}),(0,h.Z)(this,"updateValue",function(e,t){var r=ei(e),o=n.store;n.updateStore((0,Q.Z)(n.store,r,t)),n.notifyObservers(o,[r],{type:"valueUpdate",source:"internal"}),n.notifyWatch([r]);var a=n.triggerDependenciesUpdate(o,r),i=n.callbacks.onValuesChange;i&&i(ec(n.store,[r]),n.getFieldsValue()),n.triggerOnFieldsChange([r].concat((0,u.Z)(a)))}),(0,h.Z)(this,"setFieldsValue",function(e){n.warningUnhooked();var t=n.store;if(e){var r=(0,Q.T)(n.store,e);n.updateStore(r)}n.notifyObservers(t,null,{type:"valueUpdate",source:"external"}),n.notifyWatch()}),(0,h.Z)(this,"setFieldValue",function(e,t){n.setFields([{name:e,value:t}])}),(0,h.Z)(this,"getDependencyChildrenFields",function(e){var t=new Set,r=[],o=new ex;return n.getFieldEntities().forEach(function(e){(e.props.dependencies||[]).forEach(function(t){var n=ei(t);o.update(n,function(){var t=arguments.length>0&&void 0!==arguments[0]?arguments[0]:new Set;return t.add(e),t})})}),function e(n){(o.get(n)||new Set).forEach(function(n){if(!t.has(n)){t.add(n);var o=n.getNamePath();n.isFieldDirty()&&o.length&&(r.push(o),e(o))}})}(e),r}),(0,h.Z)(this,"triggerOnFieldsChange",function(e,t){var r=n.callbacks.onFieldsChange;if(r){var o=n.getFields();if(t){var a=new ex;t.forEach(function(e){var t=e.name,n=e.errors;a.set(t,n)}),o.forEach(function(e){e.errors=a.get(e.name)||e.errors})}var i=o.filter(function(t){return el(e,t.name)});i.length&&r(i,o)}}),(0,h.Z)(this,"validateFields",function(e,t){n.warningUnhooked(),Array.isArray(e)||"string"==typeof e||"string"==typeof t?(i=e,c=t):c=e;var r,o,a,i,c,l=!!i,d=l?i.map(ei):[],f=[],p=String(Date.now()),m=new Set,g=c||{},h=g.recursive,v=g.dirty;n.getFieldEntities(!0).forEach(function(e){if(l||d.push(e.getNamePath()),e.props.rules&&e.props.rules.length&&(!v||e.isFieldDirty())){var t=e.getNamePath();if(m.add(t.join(p)),!l||el(d,t,h)){var r=e.validateRules((0,s.Z)({validateMessages:(0,s.Z)((0,s.Z)({},Y),n.validateMessages)},c));f.push(r.then(function(){return{name:t,errors:[],warnings:[]}}).catch(function(e){var n,r=[],o=[];return(null===(n=e.forEach)||void 0===n||n.call(e,function(e){var t=e.rule.warningOnly,n=e.errors;t?o.push.apply(o,(0,u.Z)(n)):r.push.apply(r,(0,u.Z)(n))}),r.length)?Promise.reject({name:t,errors:r,warnings:o}):{name:t,errors:r,warnings:o}}))}}});var b=(r=!1,o=f.length,a=[],f.length?new Promise(function(e,t){f.forEach(function(n,i){n.catch(function(e){return r=!0,e}).then(function(n){o-=1,a[i]=n,o>0||(r&&t(a),e(a))})})}):Promise.resolve([]));n.lastValidatePromise=b,b.catch(function(e){return e}).then(function(e){var t=e.map(function(e){return e.name});n.notifyObservers(n.store,t,{type:"validateFinish"}),n.triggerOnFieldsChange(t,e)});var y=b.then(function(){return n.lastValidatePromise===b?Promise.resolve(n.getFieldsValue(d)):Promise.reject([])}).catch(function(e){var t=e.filter(function(e){return e&&e.errors.length});return Promise.reject({values:n.getFieldsValue(d),errorFields:t,outOfDate:n.lastValidatePromise!==b})});y.catch(function(e){return e});var w=d.filter(function(e){return m.has(e.join(p))});return n.triggerOnFieldsChange(w),y}),(0,h.Z)(this,"submit",function(){n.warningUnhooked(),n.validateFields().then(function(e){var t=n.callbacks.onFinish;if(t)try{t(e)}catch(e){console.error(e)}}).catch(function(e){var t=n.callbacks.onFinishFailed;t&&t(e)})}),this.forceRootUpdate=t}),eC=function(e){var t=o.useRef(),n=o.useState({}),r=(0,eb.Z)(n,2)[1];if(!t.current){if(e)t.current=e;else{var a=new eS(function(){r({})});t.current=a.getForm()}}return[t.current]},eZ=o.createContext({triggerFormChange:function(){},triggerFormFinish:function(){},registerForm:function(){},unregisterForm:function(){}}),eO=function(e){var t=e.validateMessages,n=e.onFormChange,r=e.onFormFinish,a=e.children,i=o.useContext(eZ),c=o.useRef({});return o.createElement(eZ.Provider,{value:(0,s.Z)((0,s.Z)({},i),{},{validateMessages:(0,s.Z)((0,s.Z)({},i.validateMessages),t),triggerFormChange:function(e,t){n&&n(e,{changedFields:t,forms:c.current}),i.triggerFormChange(e,t)},triggerFormFinish:function(e,t){r&&r(e,{values:t,forms:c.current}),i.triggerFormFinish(e,t)},registerForm:function(e,t){e&&(c.current=(0,s.Z)((0,s.Z)({},c.current),{},(0,h.Z)({},e,t))),i.registerForm(e,t)},unregisterForm:function(e){var t=(0,s.Z)({},c.current);delete t[e],c.current=t,i.unregisterForm(e)}})},a)},ek=["name","initialValues","fields","form","preserve","children","component","validateMessages","validateTrigger","onValuesChange","onFieldsChange","onFinish","onFinishFailed"];function eM(e){try{return JSON.stringify(e)}catch(e){return Math.random()}}var ej=function(){},eI=function(){for(var e=arguments.length,t=Array(e),n=0;n1?t-1:0),o=1;oen;(0,s.useImperativeHandle)(t,function(){return{focus:q,blur:function(){var e;null===(e=V.current)||void 0===e||e.blur()},setSelectionRange:function(e,t,n){var r;null===(r=V.current)||void 0===r||r.setSelectionRange(e,t,n)},select:function(){var e;null===(e=V.current)||void 0===e||e.select()},input:V.current}}),(0,s.useEffect)(function(){D(function(e){return(!e||!Z)&&e})},[Z]);var ea=function(e,t,n){var r,o,a=t;if(!W.current&&et.exceedFormatter&&et.max&&et.strategy(t)>et.max)a=et.exceedFormatter(t,{max:et.max}),t!==a&&ee([(null===(r=V.current)||void 0===r?void 0:r.selectionStart)||0,(null===(o=V.current)||void 0===o?void 0:o.selectionEnd)||0]);else if("compositionEnd"===n.source)return;$(a),V.current&&(0,u.rJ)(V.current,e,c,a)};(0,s.useEffect)(function(){if(J){var e;null===(e=V.current)||void 0===e||e.setSelectionRange.apply(e,(0,f.Z)(J))}},[J]);var ei=eo&&"".concat(C,"-out-of-range");return s.createElement(d,(0,o.Z)({},z,{prefixCls:C,className:l()(k,ei),handleReset:function(e){$(""),q(),V.current&&(0,u.rJ)(V.current,e,c)},value:K,focused:B,triggerFocus:q,suffix:function(){var e=Number(en)>0;if(j||et.show){var t=et.showFormatter?et.showFormatter({value:K,count:er,maxLength:en}):"".concat(er).concat(e?" / ".concat(en):"");return s.createElement(s.Fragment,null,et.show&&s.createElement("span",{className:l()("".concat(C,"-show-count-suffix"),(0,a.Z)({},"".concat(C,"-show-count-has-suffix"),!!j),null==F?void 0:F.count),style:(0,r.Z)({},null==T?void 0:T.count)},t),j)}return null}(),disabled:Z,classes:P,classNames:F,styles:T}),(n=(0,h.Z)(e,["prefixCls","onPressEnter","addonBefore","addonAfter","prefix","suffix","allowClear","defaultValue","showCount","count","classes","htmlSize","styles","classNames"]),s.createElement("input",(0,o.Z)({autoComplete:i},n,{onChange:function(e){ea(e,e.target.value,{source:"change"})},onFocus:function(e){D(!0),null==y||y(e)},onBlur:function(e){D(!1),null==w||w(e)},onKeyDown:function(e){x&&"Enter"===e.key&&x(e),null==E||E(e)},className:l()(C,(0,a.Z)({},"".concat(C,"-disabled"),Z),null==F?void 0:F.input),style:null==T?void 0:T.input,ref:V,size:O,type:void 0===N?"text":N,onCompositionStart:function(e){W.current=!0,null==A||A(e)},onCompositionEnd:function(e){W.current=!1,ea(e,e.currentTarget.value,{source:"compositionEnd"}),null==L||L(e)}}))))})},55041:function(e,t,n){"use strict";function r(e){return!!(e.addonBefore||e.addonAfter)}function o(e){return!!(e.prefix||e.suffix||e.allowClear)}function a(e,t,n,r){if(n){var o=t;if("click"===t.type){var a=e.cloneNode(!0);o=Object.create(t,{target:{value:a},currentTarget:{value:a}}),a.value="",n(o);return}if("file"!==e.type&&void 0!==r){var i=e.cloneNode(!0);o=Object.create(t,{target:{value:i},currentTarget:{value:i}}),i.value=r,n(o);return}n(o)}}function i(e,t){if(e){e.focus(t);var n=(t||{}).cursor;if(n){var r=e.value.length;switch(n){case"start":e.setSelectionRange(0,0);break;case"end":e.setSelectionRange(r,r);break;default:e.setSelectionRange(0,r)}}}}n.d(t,{He:function(){return r},X3:function(){return o},nH:function(){return i},rJ:function(){return a}})},47970:function(e,t,n){"use strict";n.d(t,{V4:function(){return ep},zt:function(){return w},ZP:function(){return em}});var r,o,a,i,c,l=n(11993),s=n(31686),u=n(26365),d=n(41154),f=n(36760),p=n.n(f),m=n(2868),g=n(28791),h=n(2265),v=n(6989),b=["children"],y=h.createContext({});function w(e){var t=e.children,n=(0,v.Z)(e,b);return h.createElement(y.Provider,{value:n},t)}var x=n(76405),E=n(25049),S=n(15354),C=n(15900),Z=function(e){(0,S.Z)(n,e);var t=(0,C.Z)(n);function n(){return(0,x.Z)(this,n),t.apply(this,arguments)}return(0,E.Z)(n,[{key:"render",value:function(){return this.props.children}}]),n}(h.Component),O=n(69819),k="none",M="appear",j="enter",I="leave",R="none",N="prepare",P="start",F="active",T="prepared",A=n(94981);function L(e,t){var n={};return n[e.toLowerCase()]=t.toLowerCase(),n["Webkit".concat(e)]="webkit".concat(t),n["Moz".concat(e)]="moz".concat(t),n["ms".concat(e)]="MS".concat(t),n["O".concat(e)]="o".concat(t.toLowerCase()),n}var z=(r=(0,A.Z)(),o="undefined"!=typeof window?window:{},a={animationend:L("Animation","AnimationEnd"),transitionend:L("Transition","TransitionEnd")},!r||("AnimationEvent"in o||delete a.animationend.animation,"TransitionEvent"in o||delete a.transitionend.transition),a),_={};(0,A.Z)()&&(_=document.createElement("div").style);var H={};function B(e){if(H[e])return H[e];var t=z[e];if(t)for(var n=Object.keys(t),r=n.length,o=0;o1&&void 0!==arguments[1]?arguments[1]:2;t();var a=(0,K.Z)(function(){o<=1?r({isCanceled:function(){return a!==e.current}}):n(r,o-1)});e.current=a},t]},Q=[N,P,F,"end"],J=[N,T];function ee(e){return e===F||"end"===e}var et=function(e,t,n){var r=(0,O.Z)(R),o=(0,u.Z)(r,2),a=o[0],i=o[1],c=Y(),l=(0,u.Z)(c,2),s=l[0],d=l[1],f=t?J:Q;return $(function(){if(a!==R&&"end"!==a){var e=f.indexOf(a),t=f[e+1],r=n(a);!1===r?i(t,!0):t&&s(function(e){function n(){e.isCanceled()||i(t,!0)}!0===r?n():Promise.resolve(r).then(n)})}},[e,a]),h.useEffect(function(){return function(){d()}},[]),[function(){i(N,!0)},a]},en=(i=V,"object"===(0,d.Z)(V)&&(i=V.transitionSupport),(c=h.forwardRef(function(e,t){var n=e.visible,r=void 0===n||n,o=e.removeOnLeave,a=void 0===o||o,c=e.forceRender,d=e.children,f=e.motionName,v=e.leavedClassName,b=e.eventProps,w=h.useContext(y).motion,x=!!(e.motionName&&i&&!1!==w),E=(0,h.useRef)(),S=(0,h.useRef)(),C=function(e,t,n,r){var o=r.motionEnter,a=void 0===o||o,i=r.motionAppear,c=void 0===i||i,d=r.motionLeave,f=void 0===d||d,p=r.motionDeadline,m=r.motionLeaveImmediately,g=r.onAppearPrepare,v=r.onEnterPrepare,b=r.onLeavePrepare,y=r.onAppearStart,w=r.onEnterStart,x=r.onLeaveStart,E=r.onAppearActive,S=r.onEnterActive,C=r.onLeaveActive,Z=r.onAppearEnd,R=r.onEnterEnd,A=r.onLeaveEnd,L=r.onVisibleChanged,z=(0,O.Z)(),_=(0,u.Z)(z,2),H=_[0],B=_[1],D=(0,O.Z)(k),W=(0,u.Z)(D,2),V=W[0],q=W[1],G=(0,O.Z)(null),X=(0,u.Z)(G,2),K=X[0],Y=X[1],Q=(0,h.useRef)(!1),J=(0,h.useRef)(null),en=(0,h.useRef)(!1);function er(){q(k,!0),Y(null,!0)}function eo(e){var t,r=n();if(!e||e.deadline||e.target===r){var o=en.current;V===M&&o?t=null==Z?void 0:Z(r,e):V===j&&o?t=null==R?void 0:R(r,e):V===I&&o&&(t=null==A?void 0:A(r,e)),V!==k&&o&&!1!==t&&er()}}var ea=U(eo),ei=(0,u.Z)(ea,1)[0],ec=function(e){var t,n,r;switch(e){case M:return t={},(0,l.Z)(t,N,g),(0,l.Z)(t,P,y),(0,l.Z)(t,F,E),t;case j:return n={},(0,l.Z)(n,N,v),(0,l.Z)(n,P,w),(0,l.Z)(n,F,S),n;case I:return r={},(0,l.Z)(r,N,b),(0,l.Z)(r,P,x),(0,l.Z)(r,F,C),r;default:return{}}},el=h.useMemo(function(){return ec(V)},[V]),es=et(V,!e,function(e){if(e===N){var t,r=el[N];return!!r&&r(n())}return ef in el&&Y((null===(t=el[ef])||void 0===t?void 0:t.call(el,n(),null))||null),ef===F&&(ei(n()),p>0&&(clearTimeout(J.current),J.current=setTimeout(function(){eo({deadline:!0})},p))),ef===T&&er(),!0}),eu=(0,u.Z)(es,2),ed=eu[0],ef=eu[1],ep=ee(ef);en.current=ep,$(function(){B(t);var n,r=Q.current;Q.current=!0,!r&&t&&c&&(n=M),r&&t&&a&&(n=j),(r&&!t&&f||!r&&m&&!t&&f)&&(n=I);var o=ec(n);n&&(e||o[N])?(q(n),ed()):q(k)},[t]),(0,h.useEffect)(function(){(V!==M||c)&&(V!==j||a)&&(V!==I||f)||q(k)},[c,a,f]),(0,h.useEffect)(function(){return function(){Q.current=!1,clearTimeout(J.current)}},[]);var em=h.useRef(!1);(0,h.useEffect)(function(){H&&(em.current=!0),void 0!==H&&V===k&&((em.current||H)&&(null==L||L(H)),em.current=!0)},[H,V]);var eg=K;return el[N]&&ef===P&&(eg=(0,s.Z)({transition:"none"},eg)),[V,ef,eg,null!=H?H:t]}(x,r,function(){try{return E.current instanceof HTMLElement?E.current:(0,m.Z)(S.current)}catch(e){return null}},e),R=(0,u.Z)(C,4),A=R[0],L=R[1],z=R[2],_=R[3],H=h.useRef(_);_&&(H.current=!0);var B=h.useCallback(function(e){E.current=e,(0,g.mH)(t,e)},[t]),D=(0,s.Z)((0,s.Z)({},b),{},{visible:r});if(d){if(A===k)W=_?d((0,s.Z)({},D),B):!a&&H.current&&v?d((0,s.Z)((0,s.Z)({},D),{},{className:v}),B):!c&&(a||v)?null:d((0,s.Z)((0,s.Z)({},D),{},{style:{display:"none"}}),B);else{L===N?q="prepare":ee(L)?q="active":L===P&&(q="start");var W,V,q,G=X(f,"".concat(A,"-").concat(q));W=d((0,s.Z)((0,s.Z)({},D),{},{className:p()(X(f,A),(V={},(0,l.Z)(V,G,G&&q),(0,l.Z)(V,f,"string"==typeof f),V)),style:z}),B)}}else W=null;return h.isValidElement(W)&&(0,g.Yr)(W)&&!W.ref&&(W=h.cloneElement(W,{ref:B})),h.createElement(Z,{ref:S},W)})).displayName="CSSMotion",c),er=n(1119),eo=n(63496),ea="keep",ei="remove",ec="removed";function el(e){var t;return t=e&&"object"===(0,d.Z)(e)&&"key"in e?e:{key:e},(0,s.Z)((0,s.Z)({},t),{},{key:String(t.key)})}function es(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:[];return e.map(el)}var eu=["component","children","onVisibleChanged","onAllRemoved"],ed=["status"],ef=["eventProps","visible","children","motionName","motionAppear","motionEnter","motionLeave","motionLeaveImmediately","motionDeadline","removeOnLeave","leavedClassName","onAppearPrepare","onAppearStart","onAppearActive","onAppearEnd","onEnterStart","onEnterActive","onEnterEnd","onLeaveStart","onLeaveActive","onLeaveEnd"],ep=function(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:en,n=function(e){(0,S.Z)(r,e);var n=(0,C.Z)(r);function r(){var e;(0,x.Z)(this,r);for(var t=arguments.length,o=Array(t),a=0;a0&&void 0!==arguments[0]?arguments[0]:[],t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:[],n=[],r=0,o=t.length,a=es(e),i=es(t);a.forEach(function(e){for(var t=!1,a=r;a1}).forEach(function(e){(n=n.filter(function(t){var n=t.key,r=t.status;return n!==e||r!==ei})).forEach(function(t){t.key===e&&(t.status=ea)})}),n})(r,es(n)).filter(function(e){var t=r.find(function(t){var n=t.key;return e.key===n});return!t||t.status!==ec||e.status!==ei})}}}]),r}(h.Component);return(0,l.Z)(n,"defaultProps",{component:"div"}),n}(V),em=en},49283:function(e,t,n){"use strict";n.d(t,{qX:function(){return g},JB:function(){return v},lm:function(){return O}});var r=n(83145),o=n(26365),a=n(6989),i=n(2265),c=n(31686),l=n(54887),s=n(1119),u=n(11993),d=n(36760),f=n.n(d),p=n(47970),m=n(95814),g=i.forwardRef(function(e,t){var n=e.prefixCls,r=e.style,a=e.className,c=e.duration,l=void 0===c?4.5:c,d=e.eventKey,p=e.content,g=e.closable,h=e.closeIcon,v=e.props,b=e.onClick,y=e.onNoticeClose,w=e.times,x=e.hovering,E=i.useState(!1),S=(0,o.Z)(E,2),C=S[0],Z=S[1],O=x||C,k=function(){y(d)};i.useEffect(function(){if(!O&&l>0){var e=setTimeout(function(){k()},1e3*l);return function(){clearTimeout(e)}}},[l,O,w]);var M="".concat(n,"-notice");return i.createElement("div",(0,s.Z)({},v,{ref:t,className:f()(M,a,(0,u.Z)({},"".concat(M,"-closable"),g)),style:r,onMouseEnter:function(e){var t;Z(!0),null==v||null===(t=v.onMouseEnter)||void 0===t||t.call(v,e)},onMouseLeave:function(e){var t;Z(!1),null==v||null===(t=v.onMouseLeave)||void 0===t||t.call(v,e)},onClick:b}),i.createElement("div",{className:"".concat(M,"-content")},p),g&&i.createElement("a",{tabIndex:0,className:"".concat(M,"-close"),onKeyDown:function(e){("Enter"===e.key||"Enter"===e.code||e.keyCode===m.Z.ENTER)&&k()},onClick:function(e){e.preventDefault(),e.stopPropagation(),k()}},void 0===h?"x":h))}),h=i.createContext({}),v=function(e){var t=e.children,n=e.classNames;return i.createElement(h.Provider,{value:{classNames:n}},t)},b=n(41154),y=function(e){var t,n,r,o={offset:8,threshold:3,gap:16};return e&&"object"===(0,b.Z)(e)&&(o.offset=null!==(t=e.offset)&&void 0!==t?t:8,o.threshold=null!==(n=e.threshold)&&void 0!==n?n:3,o.gap=null!==(r=e.gap)&&void 0!==r?r:16),[!!e,o]},w=["className","style","classNames","styles"],x=function(e){var t,n=e.configList,l=e.placement,d=e.prefixCls,m=e.className,v=e.style,b=e.motion,x=e.onAllNoticeRemoved,E=e.onNoticeClose,S=e.stack,C=(0,i.useContext)(h).classNames,Z=(0,i.useRef)({}),O=(0,i.useState)(null),k=(0,o.Z)(O,2),M=k[0],j=k[1],I=(0,i.useState)([]),R=(0,o.Z)(I,2),N=R[0],P=R[1],F=n.map(function(e){return{config:e,key:String(e.key)}}),T=y(S),A=(0,o.Z)(T,2),L=A[0],z=A[1],_=z.offset,H=z.threshold,B=z.gap,D=L&&(N.length>0||F.length<=H),W="function"==typeof b?b(l):b;return(0,i.useEffect)(function(){L&&N.length>1&&P(function(e){return e.filter(function(e){return F.some(function(t){return e===t.key})})})},[N,F,L]),(0,i.useEffect)(function(){var e,t;L&&Z.current[null===(e=F[F.length-1])||void 0===e?void 0:e.key]&&j(Z.current[null===(t=F[F.length-1])||void 0===t?void 0:t.key])},[F,L]),i.createElement(p.V4,(0,s.Z)({key:l,className:f()(d,"".concat(d,"-").concat(l),null==C?void 0:C.list,m,(t={},(0,u.Z)(t,"".concat(d,"-stack"),!!L),(0,u.Z)(t,"".concat(d,"-stack-expanded"),D),t)),style:v,keys:F,motionAppear:!0},W,{onAllRemoved:function(){x(l)}}),function(e,t){var n=e.config,o=e.className,u=e.style,p=e.index,m=n.key,h=n.times,v=String(m),b=n.className,y=n.style,x=n.classNames,S=n.styles,O=(0,a.Z)(n,w),k=F.findIndex(function(e){return e.key===v}),j={};if(L){var I=F.length-1-(k>-1?k:p-1),R="top"===l||"bottom"===l?"-50%":"0";if(I>0){j.height=D?null===(T=Z.current[v])||void 0===T?void 0:T.offsetHeight:null==M?void 0:M.offsetHeight;for(var T,A,z,H,W=0,V=0;V-1?Z.current[v]=e:delete Z.current[v]},prefixCls:d,classNames:x,styles:S,className:f()(b,null==C?void 0:C.notice),style:y,times:h,key:m,eventKey:m,onNoticeClose:E,hovering:L&&N.length>0})))})},E=i.forwardRef(function(e,t){var n=e.prefixCls,a=void 0===n?"rc-notification":n,s=e.container,u=e.motion,d=e.maxCount,f=e.className,p=e.style,m=e.onAllRemoved,g=e.stack,h=e.renderNotifications,v=i.useState([]),b=(0,o.Z)(v,2),y=b[0],w=b[1],E=function(e){var t,n=y.find(function(t){return t.key===e});null==n||null===(t=n.onClose)||void 0===t||t.call(n),w(function(t){return t.filter(function(t){return t.key!==e})})};i.useImperativeHandle(t,function(){return{open:function(e){w(function(t){var n,o=(0,r.Z)(t),a=o.findIndex(function(t){return t.key===e.key}),i=(0,c.Z)({},e);return a>=0?(i.times=((null===(n=t[a])||void 0===n?void 0:n.times)||0)+1,o[a]=i):(i.times=0,o.push(i)),d>0&&o.length>d&&(o=o.slice(-d)),o})},close:function(e){E(e)},destroy:function(){w([])}}});var S=i.useState({}),C=(0,o.Z)(S,2),Z=C[0],O=C[1];i.useEffect(function(){var e={};y.forEach(function(t){var n=t.placement,r=void 0===n?"topRight":n;r&&(e[r]=e[r]||[],e[r].push(t))}),Object.keys(Z).forEach(function(t){e[t]=e[t]||[]}),O(e)},[y]);var k=function(e){O(function(t){var n=(0,c.Z)({},t);return(n[e]||[]).length||delete n[e],n})},M=i.useRef(!1);if(i.useEffect(function(){Object.keys(Z).length>0?M.current=!0:M.current&&(null==m||m(),M.current=!1)},[Z]),!s)return null;var j=Object.keys(Z);return(0,l.createPortal)(i.createElement(i.Fragment,null,j.map(function(e){var t=Z[e],n=i.createElement(x,{key:e,configList:t,placement:e,prefixCls:a,className:null==f?void 0:f(e),style:null==p?void 0:p(e),motion:u,onNoticeClose:E,onAllNoticeRemoved:k,stack:g});return h?h(n,{prefixCls:a,key:e}):n})),s)}),S=["getContainer","motion","prefixCls","maxCount","className","style","onAllRemoved","stack","renderNotifications"],C=function(){return document.body},Z=0;function O(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{},t=e.getContainer,n=void 0===t?C:t,c=e.motion,l=e.prefixCls,s=e.maxCount,u=e.className,d=e.style,f=e.onAllRemoved,p=e.stack,m=e.renderNotifications,g=(0,a.Z)(e,S),h=i.useState(),v=(0,o.Z)(h,2),b=v[0],y=v[1],w=i.useRef(),x=i.createElement(E,{container:b,ref:w,prefixCls:l,motion:c,maxCount:s,className:u,style:d,onAllRemoved:f,stack:p,renderNotifications:m}),O=i.useState([]),k=(0,o.Z)(O,2),M=k[0],j=k[1],I=i.useMemo(function(){return{open:function(e){var t=function(){for(var e={},t=arguments.length,n=Array(t),r=0;rP,ej=(0,c.useMemo)(function(){var e=w;return eO?e=null===q&&B?w:w.slice(0,Math.min(w.length,X/j)):"number"==typeof P&&(e=w.slice(0,P)),e},[w,j,q,P,eO]),eI=(0,c.useMemo)(function(){return eO?w.slice(eb+1):w.slice(ej.length)},[w,ej,eO,eb]),eR=(0,c.useCallback)(function(e,t){var n;return"function"==typeof S?S(e):null!==(n=S&&(null==e?void 0:e[S]))&&void 0!==n?n:t},[S]),eN=(0,c.useCallback)(x||function(e){return e},[x]);function eP(e,t,n){(eh!==e||void 0!==t&&t!==ef)&&(ev(e),n||(eE(eX){eP(r-1,e-o-el+eo);break}}A&&eT(0)+el>X&&ep(null)}},[X,K,eo,el,eR,ej]);var eA=ex&&!!eI.length,eL={};null!==ef&&eO&&(eL={position:"absolute",left:ef,top:0});var ez={prefixCls:eS,responsive:eO,component:z,invalidate:ek},e_=E?function(e,t){var n=eR(e,t);return c.createElement(y.Provider,{key:n,value:(0,o.Z)((0,o.Z)({},ez),{},{order:t,item:e,itemKey:n,registerSize:eF,display:t<=eb})},E(e,t))}:function(e,t){var n=eR(e,t);return c.createElement(m,(0,r.Z)({},ez,{order:t,key:n,item:e,renderItem:eN,itemKey:n,registerSize:eF,display:t<=eb}))},eH={order:eA?eb:Number.MAX_SAFE_INTEGER,className:"".concat(eS,"-rest"),registerSize:function(e,t){ea(t),et(eo)},display:eA};if(T)T&&(l=c.createElement(y.Provider,{value:(0,o.Z)((0,o.Z)({},ez),eH)},T(eI)));else{var eB=F||k;l=c.createElement(m,(0,r.Z)({},ez,eH),"function"==typeof eB?eB(eI):eB)}var eD=c.createElement(void 0===L?"div":L,(0,r.Z)({className:s()(!ek&&p,N),style:R,ref:t},H),ej.map(e_),eM?l:null,A&&c.createElement(m,(0,r.Z)({},ez,{responsive:eZ,responsiveDisabled:!eO,order:eb,className:"".concat(eS,"-suffix"),registerSize:function(e,t){es(t)},display:!0,style:eL}),A));return eZ&&(eD=c.createElement(u.Z,{onResize:function(e,t){G(t.clientWidth)},disabled:!eO},eD)),eD});M.displayName="Overflow",M.Item=S,M.RESPONSIVE=Z,M.INVALIDATE=O;var j=M},96257:function(e,t){"use strict";t.Z={items_per_page:"/ page",jump_to:"Go to",jump_to_confirm:"confirm",page:"Page",prev_page:"Previous Page",next_page:"Next Page",prev_5:"Previous 5 Pages",next_5:"Next 5 Pages",prev_3:"Previous 3 Pages",next_3:"Next 3 Pages",page_size:"Page Size"}},31474:function(e,t,n){"use strict";n.d(t,{Z:function(){return H}});var r=n(1119),o=n(2265),a=n(45287);n(32559);var i=n(31686),c=n(41154),l=n(2868),s=n(28791),u=o.createContext(null),d=function(){if("undefined"!=typeof Map)return Map;function e(e,t){var n=-1;return e.some(function(e,r){return e[0]===t&&(n=r,!0)}),n}return function(){function t(){this.__entries__=[]}return Object.defineProperty(t.prototype,"size",{get:function(){return this.__entries__.length},enumerable:!0,configurable:!0}),t.prototype.get=function(t){var n=e(this.__entries__,t),r=this.__entries__[n];return r&&r[1]},t.prototype.set=function(t,n){var r=e(this.__entries__,t);~r?this.__entries__[r][1]=n:this.__entries__.push([t,n])},t.prototype.delete=function(t){var n=this.__entries__,r=e(n,t);~r&&n.splice(r,1)},t.prototype.has=function(t){return!!~e(this.__entries__,t)},t.prototype.clear=function(){this.__entries__.splice(0)},t.prototype.forEach=function(e,t){void 0===t&&(t=null);for(var n=0,r=this.__entries__;n0},e.prototype.connect_=function(){f&&!this.connected_&&(document.addEventListener("transitionend",this.onTransitionEnd_),window.addEventListener("resize",this.refresh),h?(this.mutationsObserver_=new MutationObserver(this.refresh),this.mutationsObserver_.observe(document,{attributes:!0,childList:!0,characterData:!0,subtree:!0})):(document.addEventListener("DOMSubtreeModified",this.refresh),this.mutationEventsAdded_=!0),this.connected_=!0)},e.prototype.disconnect_=function(){f&&this.connected_&&(document.removeEventListener("transitionend",this.onTransitionEnd_),window.removeEventListener("resize",this.refresh),this.mutationsObserver_&&this.mutationsObserver_.disconnect(),this.mutationEventsAdded_&&document.removeEventListener("DOMSubtreeModified",this.refresh),this.mutationsObserver_=null,this.mutationEventsAdded_=!1,this.connected_=!1)},e.prototype.onTransitionEnd_=function(e){var t=e.propertyName,n=void 0===t?"":t;g.some(function(e){return!!~n.indexOf(e)})&&this.refresh()},e.getInstance=function(){return this.instance_||(this.instance_=new e),this.instance_},e.instance_=null,e}(),b=function(e,t){for(var n=0,r=Object.keys(t);n0},e}(),M="undefined"!=typeof WeakMap?new WeakMap:new d,j=function e(t){if(!(this instanceof e))throw TypeError("Cannot call a class as a function.");if(!arguments.length)throw TypeError("1 argument required, but only 0 present.");var n=new k(t,v.getInstance(),this);M.set(this,n)};["observe","unobserve","disconnect"].forEach(function(e){j.prototype[e]=function(){var t;return(t=M.get(this))[e].apply(t,arguments)}});var I=void 0!==p.ResizeObserver?p.ResizeObserver:j,R=new Map,N=new I(function(e){e.forEach(function(e){var t,n=e.target;null===(t=R.get(n))||void 0===t||t.forEach(function(e){return e(n)})})}),P=n(76405),F=n(25049),T=n(15354),A=n(15900),L=function(e){(0,T.Z)(n,e);var t=(0,A.Z)(n);function n(){return(0,P.Z)(this,n),t.apply(this,arguments)}return(0,F.Z)(n,[{key:"render",value:function(){return this.props.children}}]),n}(o.Component),z=o.forwardRef(function(e,t){var n=e.children,r=e.disabled,a=o.useRef(null),d=o.useRef(null),f=o.useContext(u),p="function"==typeof n,m=p?n(a):n,g=o.useRef({width:-1,height:-1,offsetWidth:-1,offsetHeight:-1}),h=!p&&o.isValidElement(m)&&(0,s.Yr)(m),v=h?m.ref:null,b=(0,s.x1)(v,a),y=function(){var e;return(0,l.Z)(a.current)||(a.current&&"object"===(0,c.Z)(a.current)?(0,l.Z)(null===(e=a.current)||void 0===e?void 0:e.nativeElement):null)||(0,l.Z)(d.current)};o.useImperativeHandle(t,function(){return y()});var w=o.useRef(e);w.current=e;var x=o.useCallback(function(e){var t=w.current,n=t.onResize,r=t.data,o=e.getBoundingClientRect(),a=o.width,c=o.height,l=e.offsetWidth,s=e.offsetHeight,u=Math.floor(a),d=Math.floor(c);if(g.current.width!==u||g.current.height!==d||g.current.offsetWidth!==l||g.current.offsetHeight!==s){var p={width:u,height:d,offsetWidth:l,offsetHeight:s};g.current=p;var m=(0,i.Z)((0,i.Z)({},p),{},{offsetWidth:l===Math.round(a)?a:l,offsetHeight:s===Math.round(c)?c:s});null==f||f(m,e,r),n&&Promise.resolve().then(function(){n(m,e)})}},[]);return o.useEffect(function(){var e=y();return e&&!r&&(R.has(e)||(R.set(e,new Set),N.observe(e)),R.get(e).add(x)),function(){R.has(e)&&(R.get(e).delete(x),R.get(e).size||(N.unobserve(e),R.delete(e)))}},[a.current,r]),o.createElement(L,{ref:d},h?o.cloneElement(m,{ref:b}):m)}),_=o.forwardRef(function(e,t){var n=e.children;return("function"==typeof n?[n]:(0,a.Z)(n)).map(function(n,a){var i=(null==n?void 0:n.key)||"".concat("rc-observer-key","-").concat(a);return o.createElement(z,(0,r.Z)({},e,{key:i,ref:0===a?t:void 0}),n)})});_.Collection=function(e){var t=e.children,n=e.onBatchResize,r=o.useRef(0),a=o.useRef([]),i=o.useContext(u),c=o.useCallback(function(e,t,o){r.current+=1;var c=r.current;a.current.push({size:e,element:t,data:o}),Promise.resolve().then(function(){c===r.current&&(null==n||n(a.current),a.current=[])}),null==i||i(e,t,o)},[n,i]);return o.createElement(u.Provider,{value:c},t)};var H=_},5769:function(e,t,n){"use strict";n.d(t,{G:function(){return i},Z:function(){return h}});var r=n(36760),o=n.n(r),a=n(2265);function i(e){var t=e.children,n=e.prefixCls,r=e.id,i=e.overlayInnerStyle,c=e.className,l=e.style;return a.createElement("div",{className:o()("".concat(n,"-content"),c),style:l},a.createElement("div",{className:"".concat(n,"-inner"),id:r,role:"tooltip",style:i},"function"==typeof t?t():t))}var c=n(1119),l=n(31686),s=n(6989),u=n(97821),d={shiftX:64,adjustY:1},f={adjustX:1,shiftY:!0},p=[0,0],m={left:{points:["cr","cl"],overflow:f,offset:[-4,0],targetOffset:p},right:{points:["cl","cr"],overflow:f,offset:[4,0],targetOffset:p},top:{points:["bc","tc"],overflow:d,offset:[0,-4],targetOffset:p},bottom:{points:["tc","bc"],overflow:d,offset:[0,4],targetOffset:p},topLeft:{points:["bl","tl"],overflow:d,offset:[0,-4],targetOffset:p},leftTop:{points:["tr","tl"],overflow:f,offset:[-4,0],targetOffset:p},topRight:{points:["br","tr"],overflow:d,offset:[0,-4],targetOffset:p},rightTop:{points:["tl","tr"],overflow:f,offset:[4,0],targetOffset:p},bottomRight:{points:["tr","br"],overflow:d,offset:[0,4],targetOffset:p},rightBottom:{points:["bl","br"],overflow:f,offset:[4,0],targetOffset:p},bottomLeft:{points:["tl","bl"],overflow:d,offset:[0,4],targetOffset:p},leftBottom:{points:["br","bl"],overflow:f,offset:[-4,0],targetOffset:p}},g=["overlayClassName","trigger","mouseEnterDelay","mouseLeaveDelay","overlayStyle","prefixCls","children","onVisibleChange","afterVisibleChange","transitionName","animation","motion","placement","align","destroyTooltipOnHide","defaultVisible","getTooltipContainer","overlayInnerStyle","arrowContent","overlay","id","showArrow"],h=(0,a.forwardRef)(function(e,t){var n=e.overlayClassName,r=e.trigger,o=e.mouseEnterDelay,d=e.mouseLeaveDelay,f=e.overlayStyle,p=e.prefixCls,h=void 0===p?"rc-tooltip":p,v=e.children,b=e.onVisibleChange,y=e.afterVisibleChange,w=e.transitionName,x=e.animation,E=e.motion,S=e.placement,C=e.align,Z=e.destroyTooltipOnHide,O=e.defaultVisible,k=e.getTooltipContainer,M=e.overlayInnerStyle,j=(e.arrowContent,e.overlay),I=e.id,R=e.showArrow,N=(0,s.Z)(e,g),P=(0,a.useRef)(null);(0,a.useImperativeHandle)(t,function(){return P.current});var F=(0,l.Z)({},N);return"visible"in e&&(F.popupVisible=e.visible),a.createElement(u.Z,(0,c.Z)({popupClassName:n,prefixCls:h,popup:function(){return a.createElement(i,{key:"content",prefixCls:h,id:I,overlayInnerStyle:M},j)},action:void 0===r?["hover"]:r,builtinPlacements:m,popupPlacement:void 0===S?"right":S,ref:P,popupAlign:void 0===C?{}:C,getPopupContainer:k,onPopupVisibleChange:b,afterPopupVisibleChange:y,popupTransitionName:w,popupAnimation:x,popupMotion:E,defaultPopupVisible:O,autoDestroy:void 0!==Z&&Z,mouseLeaveDelay:void 0===d?.1:d,popupStyle:f,mouseEnterDelay:void 0===o?0:o,arrow:void 0===R||R},F),v)})},45287:function(e,t,n){"use strict";n.d(t,{Z:function(){return function e(t){var n=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},a=[];return r.Children.forEach(t,function(t){(null!=t||n.keepEmpty)&&(Array.isArray(t)?a=a.concat(e(t)):(0,o.isFragment)(t)&&t.props?a=a.concat(e(t.props.children,n)):a.push(t))}),a}}});var r=n(2265),o=n(93754)},94981:function(e,t,n){"use strict";function r(){return!!("undefined"!=typeof window&&window.document&&window.document.createElement)}n.d(t,{Z:function(){return r}})},2161:function(e,t,n){"use strict";function r(e,t){if(!e)return!1;if(e.contains)return e.contains(t);for(var n=t;n;){if(n===e)return!0;n=n.parentNode}return!1}n.d(t,{Z:function(){return r}})},21717:function(e,t,n){"use strict";n.d(t,{hq:function(){return m},jL:function(){return p}});var r=n(94981),o=n(2161),a="data-rc-order",i="data-rc-priority",c=new Map;function l(){var e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{},t=e.mark;return t?t.startsWith("data-")?t:"data-".concat(t):"rc-util-key"}function s(e){return e.attachTo?e.attachTo:document.querySelector("head")||document.body}function u(e){return Array.from((c.get(e)||e).children).filter(function(e){return"STYLE"===e.tagName})}function d(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{};if(!(0,r.Z)())return null;var n=t.csp,o=t.prepend,c=t.priority,l=void 0===c?0:c,d="queue"===o?"prependQueue":o?"prepend":"append",f="prependQueue"===d,p=document.createElement("style");p.setAttribute(a,d),f&&l&&p.setAttribute(i,"".concat(l)),null!=n&&n.nonce&&(p.nonce=null==n?void 0:n.nonce),p.innerHTML=e;var m=s(t),g=m.firstChild;if(o){if(f){var h=u(m).filter(function(e){return!!["prepend","prependQueue"].includes(e.getAttribute(a))&&l>=Number(e.getAttribute(i)||0)});if(h.length)return m.insertBefore(p,h[h.length-1].nextSibling),p}m.insertBefore(p,g)}else m.appendChild(p);return p}function f(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{};return u(s(t)).find(function(n){return n.getAttribute(l(t))===e})}function p(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},n=f(e,t);n&&s(t).removeChild(n)}function m(e,t){var n,r,a,i=arguments.length>2&&void 0!==arguments[2]?arguments[2]:{};!function(e,t){var n=c.get(e);if(!n||!(0,o.Z)(document,n)){var r=d("",t),a=r.parentNode;c.set(e,a),e.removeChild(r)}}(s(i),i);var u=f(t,i);if(u)return null!==(n=i.csp)&&void 0!==n&&n.nonce&&u.nonce!==(null===(r=i.csp)||void 0===r?void 0:r.nonce)&&(u.nonce=null===(a=i.csp)||void 0===a?void 0:a.nonce),u.innerHTML!==e&&(u.innerHTML=e),u;var p=d(e,i);return p.setAttribute(l(i),t),p}},2868:function(e,t,n){"use strict";n.d(t,{S:function(){return a},Z:function(){return i}});var r=n(2265),o=n(54887);function a(e){return e instanceof HTMLElement||e instanceof SVGElement}function i(e){return a(e)?e:e instanceof r.Component?o.findDOMNode(e):null}},2857:function(e,t){"use strict";t.Z=function(e){if(!e)return!1;if(e instanceof Element){if(e.offsetParent)return!0;if(e.getBBox){var t=e.getBBox(),n=t.width,r=t.height;if(n||r)return!0}if(e.getBoundingClientRect){var o=e.getBoundingClientRect(),a=o.width,i=o.height;if(a||i)return!0}}return!1}},13211:function(e,t,n){"use strict";function r(e){var t;return null==e||null===(t=e.getRootNode)||void 0===t?void 0:t.call(e)}function o(e){return r(e) instanceof ShadowRoot?r(e):null}n.d(t,{A:function(){return o}})},95814:function(e,t){"use strict";var n={MAC_ENTER:3,BACKSPACE:8,TAB:9,NUM_CENTER:12,ENTER:13,SHIFT:16,CTRL:17,ALT:18,PAUSE:19,CAPS_LOCK:20,ESC:27,SPACE:32,PAGE_UP:33,PAGE_DOWN:34,END:35,HOME:36,LEFT:37,UP:38,RIGHT:39,DOWN:40,PRINT_SCREEN:44,INSERT:45,DELETE:46,ZERO:48,ONE:49,TWO:50,THREE:51,FOUR:52,FIVE:53,SIX:54,SEVEN:55,EIGHT:56,NINE:57,QUESTION_MARK:63,A:65,B:66,C:67,D:68,E:69,F:70,G:71,H:72,I:73,J:74,K:75,L:76,M:77,N:78,O:79,P:80,Q:81,R:82,S:83,T:84,U:85,V:86,W:87,X:88,Y:89,Z:90,META:91,WIN_KEY_RIGHT:92,CONTEXT_MENU:93,NUM_ZERO:96,NUM_ONE:97,NUM_TWO:98,NUM_THREE:99,NUM_FOUR:100,NUM_FIVE:101,NUM_SIX:102,NUM_SEVEN:103,NUM_EIGHT:104,NUM_NINE:105,NUM_MULTIPLY:106,NUM_PLUS:107,NUM_MINUS:109,NUM_PERIOD:110,NUM_DIVISION:111,F1:112,F2:113,F3:114,F4:115,F5:116,F6:117,F7:118,F8:119,F9:120,F10:121,F11:122,F12:123,NUMLOCK:144,SEMICOLON:186,DASH:189,EQUALS:187,COMMA:188,PERIOD:190,SLASH:191,APOSTROPHE:192,SINGLE_QUOTE:222,OPEN_SQUARE_BRACKET:219,BACKSLASH:220,CLOSE_SQUARE_BRACKET:221,WIN_KEY:224,MAC_FF_META:224,WIN_IME:229,isTextModifyingKeyEvent:function(e){var t=e.keyCode;if(e.altKey&&!e.ctrlKey||e.metaKey||t>=n.F1&&t<=n.F12)return!1;switch(t){case n.ALT:case n.CAPS_LOCK:case n.CONTEXT_MENU:case n.CTRL:case n.DOWN:case n.END:case n.ESC:case n.HOME:case n.INSERT:case n.LEFT:case n.MAC_FF_META:case n.META:case n.NUMLOCK:case n.NUM_CENTER:case n.PAGE_DOWN:case n.PAGE_UP:case n.PAUSE:case n.PRINT_SCREEN:case n.RIGHT:case n.SHIFT:case n.UP:case n.WIN_KEY:case n.WIN_KEY_RIGHT:return!1;default:return!0}},isCharacterKey:function(e){if(e>=n.ZERO&&e<=n.NINE||e>=n.NUM_ZERO&&e<=n.NUM_MULTIPLY||e>=n.A&&e<=n.Z||-1!==window.navigator.userAgent.indexOf("WebKit")&&0===e)return!0;switch(e){case n.SPACE:case n.QUESTION_MARK:case n.NUM_PLUS:case n.NUM_MINUS:case n.NUM_PERIOD:case n.NUM_DIVISION:case n.SEMICOLON:case n.DASH:case n.EQUALS:case n.COMMA:case n.PERIOD:case n.SLASH:case n.APOSTROPHE:case n.SINGLE_QUOTE:case n.OPEN_SQUARE_BRACKET:case n.BACKSLASH:case n.CLOSE_SQUARE_BRACKET:return!0;default:return!1}}};t.Z=n},18404:function(e,t,n){"use strict";n.d(t,{s:function(){return h},v:function(){return b}});var r,o,a=n(73129),i=n(54580),c=n(41154),l=n(31686),s=n(54887),u=(0,l.Z)({},r||(r=n.t(s,2))),d=u.version,f=u.render,p=u.unmountComponentAtNode;try{Number((d||"").split(".")[0])>=18&&(o=u.createRoot)}catch(e){}function m(e){var t=u.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED;t&&"object"===(0,c.Z)(t)&&(t.usingClientEntryPoint=e)}var g="__rc_react_root__";function h(e,t){if(o){var n;m(!0),n=t[g]||o(t),m(!1),n.render(e),t[g]=n;return}f(e,t)}function v(){return(v=(0,i.Z)((0,a.Z)().mark(function e(t){return(0,a.Z)().wrap(function(e){for(;;)switch(e.prev=e.next){case 0:return e.abrupt("return",Promise.resolve().then(function(){var e;null===(e=t[g])||void 0===e||e.unmount(),delete t[g]}));case 1:case"end":return e.stop()}},e)}))).apply(this,arguments)}function b(e){return y.apply(this,arguments)}function y(){return(y=(0,i.Z)((0,a.Z)().mark(function e(t){return(0,a.Z)().wrap(function(e){for(;;)switch(e.prev=e.next){case 0:if(!(void 0!==o)){e.next=2;break}return e.abrupt("return",function(e){return v.apply(this,arguments)}(t));case 2:p(t);case 3:case"end":return e.stop()}},e)}))).apply(this,arguments)}},3208:function(e,t,n){"use strict";var r;function o(e){if("undefined"==typeof document)return 0;if(e||void 0===r){var t=document.createElement("div");t.style.width="100%",t.style.height="200px";var n=document.createElement("div"),o=n.style;o.position="absolute",o.top="0",o.left="0",o.pointerEvents="none",o.visibility="hidden",o.width="200px",o.height="150px",o.overflow="hidden",n.appendChild(t),document.body.appendChild(n);var a=t.offsetWidth;n.style.overflow="scroll";var i=t.offsetWidth;a===i&&(i=n.clientWidth),document.body.removeChild(n),r=a-i}return r}function a(e){var t=e.match(/^(.*)px$/),n=Number(null==t?void 0:t[1]);return Number.isNaN(n)?o():n}function i(e){if("undefined"==typeof document||!e||!(e instanceof Element))return{width:0,height:0};var t=getComputedStyle(e,"::-webkit-scrollbar"),n=t.width,r=t.height;return{width:a(n),height:a(r)}}n.d(t,{Z:function(){return o},o:function(){return i}})},58525:function(e,t,n){"use strict";n.d(t,{Z:function(){return o}});var r=n(2265);function o(e){var t=r.useRef();return t.current=e,r.useCallback(function(){for(var e,n=arguments.length,r=Array(n),o=0;o2&&void 0!==arguments[2]&&arguments[2],a=new Set;return function e(t,i){var c=arguments.length>2&&void 0!==arguments[2]?arguments[2]:1,l=a.has(t);if((0,o.ZP)(!l,"Warning: There may be circular references"),l)return!1;if(t===i)return!0;if(n&&c>1)return!1;a.add(t);var s=c+1;if(Array.isArray(t)){if(!Array.isArray(i)||t.length!==i.length)return!1;for(var u=0;u1&&void 0!==arguments[1]&&arguments[1];t=!1===n?{aria:!0,data:!0,attr:!0}:!0===n?{aria:!0}:(0,r.Z)({},n);var i={};return Object.keys(e).forEach(function(n){(t.aria&&("role"===n||a(n,"aria-"))||t.data&&a(n,"data-")||t.attr&&o.includes(n))&&(i[n]=e[n])}),i}},53346:function(e,t){"use strict";var n=function(e){return+setTimeout(e,16)},r=function(e){return clearTimeout(e)};"undefined"!=typeof window&&"requestAnimationFrame"in window&&(n=function(e){return window.requestAnimationFrame(e)},r=function(e){return window.cancelAnimationFrame(e)});var o=0,a=new Map,i=function(e){var t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:1,r=o+=1;return!function t(o){if(0===o)a.delete(r),e();else{var i=n(function(){t(o-1)});a.set(r,i)}}(t),r};i.cancel=function(e){var t=a.get(e);return a.delete(e),r(t)},t.Z=i},28791:function(e,t,n){"use strict";n.d(t,{Yr:function(){return u},mH:function(){return c},sQ:function(){return l},t4:function(){return d},x1:function(){return s}});var r=n(41154),o=n(2265),a=n(93754),i=n(6397);function c(e,t){"function"==typeof e?e(t):"object"===(0,r.Z)(e)&&e&&"current"in e&&(e.current=t)}function l(){for(var e=arguments.length,t=Array(e),n=0;n3&&void 0!==arguments[3]&&arguments[3];return t.length&&r&&void 0===n&&!(0,c.Z)(e,t.slice(0,-1))?e:function e(t,n,r,c){if(!n.length)return r;var l,s=(0,i.Z)(n),u=s[0],d=s.slice(1);return l=t||"number"!=typeof u?Array.isArray(t)?(0,a.Z)(t):(0,o.Z)({},t):[],c&&void 0===r&&1===d.length?delete l[u][d[0]]:l[u]=e(l[u],d,r,c),l}(e,t,n,r)}function s(e){return Array.isArray(e)?[]:{}}var u="undefined"==typeof Reflect?Object.keys:Reflect.ownKeys;function d(){for(var e=arguments.length,t=Array(e),n=0;n