+ +## What is Apertis AI? + +Apertis AI is a unified API platform that lets developers: +- **Access 430+ AI Models**: All models through a single API +- **Save 50% on Costs**: Competitive pricing with significant discounts +- **Unified Billing**: Single bill for all model usage +- **Quick Setup**: Start with just $2 registration +- **GitHub Integration**: Link with your GitHub account + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["STIMA_API_KEY"] = "" # your Apertis AI API key +``` + +Get your Apertis AI API key from [api.stima.tech](https://api.stima.tech). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="Apertis AI Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["STIMA_API_KEY"] = "" # your Apertis AI API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# Apertis AI call +response = completion( + model="apertis/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="Apertis AI Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["STIMA_API_KEY"] = "" # your Apertis AI API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# Apertis AI call with streaming +response = completion( + model="apertis/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export STIMA_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: apertis-model + litellm_params: + model: apertis/model-name # Replace with actual model name + api_key: os.environ/STIMA_API_KEY +``` + +## Supported OpenAI Parameters + +Apertis AI supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID from 430+ available models | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | + +## Cost Benefits + +Apertis AI offers significant cost advantages: +- **50% Cost Savings**: Save money compared to direct provider costs +- **Unified Billing**: Single invoice for all your AI model usage +- **Low Entry**: Start with just $2 registration + +## Model Availability + +With access to 430+ AI models, Apertis AI provides: +- Multiple providers through one API +- Latest model releases +- Various model types (text, image, video) + +## Additional Resources + +- [Apertis AI Website](https://api.stima.tech) +- [Apertis AI Enterprise](https://api.stima.tech/enterprise) diff --git a/docs/my-website/docs/providers/bedrock.md b/docs/my-website/docs/providers/bedrock.md index 122554fe8a4..f1eed4b4d52 100644 --- a/docs/my-website/docs/providers/bedrock.md +++ b/docs/my-website/docs/providers/bedrock.md @@ -2208,6 +2208,53 @@ response = completion( | `aws_role_name` | `RoleArn` | The Amazon Resource Name (ARN) of the role to assume | [AssumeRole API](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role) | | `aws_session_name` | `RoleSessionName` | An identifier for the assumed role session | [AssumeRole API](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role) | +### IAM Roles Anywhere (On-Premise / External Workloads) + +[IAM Roles Anywhere](https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html) extends IAM roles to workloads **outside of AWS** (on-premise servers, edge devices, other clouds). It uses the same STS mechanism as regular IAM roles but authenticates via X.509 certificates instead of AWS credentials. + +**Setup**: Configure the [AWS Signing Helper](https://docs.aws.amazon.com/rolesanywhere/latest/userguide/credential-helper.html) as a credential process in `~/.aws/config`: + +```ini +[profile litellm-roles-anywhere] +credential_process = aws_signing_helper credential-process \ + --certificate /path/to/certificate.pem \ + --private-key /path/to/private-key.pem \ + --trust-anchor-arn arn:aws:rolesanywhere:us-east-1:123456789012:trust-anchor/abc123 \ + --profile-arn arn:aws:rolesanywhere:us-east-1:123456789012:profile/def456 \ + --role-arn arn:aws:iam::123456789012:role/MyBedrockRole +``` + +**Usage**: Reference the profile in LiteLLM: + +
+ +## What is Chutes? + +Chutes is a powerful AI deployment and serving platform that provides: +- **Pre-built Templates**: Ready-to-use configurations for vLLM, SGLang, diffusion models, and embeddings +- **OpenAI-Compatible APIs**: Use standard OpenAI SDKs and clients +- **Multi-GPU Scaling**: Support for large models across multiple GPUs +- **Streaming Responses**: Real-time model outputs +- **Custom Configurations**: Override any parameter for your specific needs +- **Performance Optimization**: Pre-configured optimization settings + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["CHUTES_API_KEY"] = "" # your Chutes API key +``` + +Get your Chutes API key from [chutes.ai](https://chutes.ai). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="Chutes Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["CHUTES_API_KEY"] = "" # your Chutes API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# Chutes call +response = completion( + model="chutes/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="Chutes Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["CHUTES_API_KEY"] = "" # your Chutes API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# Chutes call with streaming +response = completion( + model="chutes/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export CHUTES_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: chutes-model + litellm_params: + model: chutes/model-name # Replace with actual model name + api_key: os.environ/CHUTES_API_KEY +``` + +## Supported OpenAI Parameters + +Chutes supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID or HuggingFace model identifier | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | +| `response_format` | object | Optional. Response format specification | + +## Support Frameworks + +Chutes provides optimized templates for popular AI frameworks: + +### vLLM (High-Performance LLM Serving) +- OpenAI-compatible endpoints +- Multi-GPU scaling support +- Advanced optimization settings +- Best for production workloads + +### SGLang (Advanced LLM Serving) +- Structured generation capabilities +- Advanced features and controls +- Custom configuration options +- Best for complex use cases + +### Diffusion Models (Image Generation) +- Pre-configured image generation templates +- Optimized settings for best results +- Support for popular diffusion models + +### Embedding Models +- Text embedding templates +- Vector search optimization +- Support for popular embedding models + +## Authentication + +Chutes supports multiple authentication methods: +- API Key via `X-API-Key` header +- Bearer token via `Authorization` header + +Example for LiteLLM (uses environment variable): +```python +os.environ["CHUTES_API_KEY"] = "your-api-key" +``` + +## Performance Optimization + +Chutes offers hardware selection and optimization: +- **Small Models (7B-13B)**: 1 GPU with 24GB VRAM +- **Medium Models (30B-70B)**: 4 GPUs with 80GB VRAM each +- **Large Models (100B+)**: 8 GPUs with 140GB+ VRAM each + +Engine optimization parameters available for fine-tuning performance. + +## Deployment Options + +Chutes provides flexible deployment: +- **Quick Setup**: Use pre-built templates for instant deployment +- **Custom Images**: Deploy with custom Docker images +- **Scaling**: Configure max instances and auto-scaling thresholds +- **Hardware**: Choose specific GPU types and configurations + +## Additional Resources + +- [Chutes Documentation](https://chutes.ai/docs) +- [Chutes Getting Started](https://chutes.ai/docs/getting-started/running-a-chute) +- [Chutes API Reference](https://chutes.ai/docs/sdk-reference) diff --git a/docs/my-website/docs/providers/databricks.md b/docs/my-website/docs/providers/databricks.md index 921b06a17b7..2791d55dff1 100644 --- a/docs/my-website/docs/providers/databricks.md +++ b/docs/my-website/docs/providers/databricks.md @@ -11,6 +11,99 @@ LiteLLM supports all models on Databricks ::: +## Authentication + +LiteLLM supports multiple authentication methods for Databricks, listed in order of preference: + +### OAuth M2M (Recommended for Production) + +OAuth Machine-to-Machine authentication using Service Principal credentials is the **recommended method for production** deployments per Databricks Partner requirements. + +```python +import os +from litellm import completion + +# Set OAuth credentials (Service Principal) +os.environ["DATABRICKS_CLIENT_ID"] = "your-service-principal-application-id" +os.environ["DATABRICKS_CLIENT_SECRET"] = "your-service-principal-secret" +os.environ["DATABRICKS_API_BASE"] = "https://adb-xxx.azuredatabricks.net/serving-endpoints" + +response = completion( + model="databricks/databricks-dbrx-instruct", + messages=[{"role": "user", "content": "Hello!"}], +) +``` + +### Personal Access Token (PAT) + +PAT authentication is supported for development and testing scenarios. + +```python +import os +from litellm import completion + +os.environ["DATABRICKS_API_KEY"] = "dapi..." # Your Personal Access Token +os.environ["DATABRICKS_API_BASE"] = "https://adb-xxx.azuredatabricks.net/serving-endpoints" + +response = completion( + model="databricks/databricks-dbrx-instruct", + messages=[{"role": "user", "content": "Hello!"}], +) +``` + +### Databricks SDK Authentication (Automatic) + +If no credentials are provided, LiteLLM will use the Databricks SDK for automatic authentication. This supports OAuth, Azure AD, and other unified auth methods configured in your environment. + +```python +from litellm import completion + +# No environment variables needed - uses Databricks SDK unified auth +# Requires: pip install databricks-sdk +response = completion( + model="databricks/databricks-dbrx-instruct", + messages=[{"role": "user", "content": "Hello!"}], +) +``` + +## Custom User-Agent for Partner Attribution + +If you're building a product on top of LiteLLM that integrates with Databricks, you can pass your own partner identifier for proper attribution in Databricks telemetry. + +The partner name will be prefixed to the LiteLLM user agent: + +```python +# Via parameter +response = completion( + model="databricks/databricks-dbrx-instruct", + messages=[{"role": "user", "content": "Hello!"}], + user_agent="mycompany/1.0.0", +) +# Resulting User-Agent: mycompany_litellm/1.79.1 + +# Via environment variable +os.environ["DATABRICKS_USER_AGENT"] = "mycompany/1.0.0" +# Resulting User-Agent: mycompany_litellm/1.79.1 +``` + +| Input | Resulting User-Agent | +|-------|---------------------| +| (none) | `litellm/1.79.1` | +| `mycompany/1.0.0` | `mycompany_litellm/1.79.1` | +| `partner_product/2.5.0` | `partner_product_litellm/1.79.1` | +| `acme` | `acme_litellm/1.79.1` | + +**Note:** The version from your custom user agent is ignored; LiteLLM's version is always used. + +## Security + +LiteLLM automatically redacts sensitive information (tokens, secrets, API keys) from all debug logs to prevent credential leakage. This includes: + +- Authorization headers +- API keys and tokens +- Client secrets +- Personal access tokens (PATs) + ## Usage
+ +## What is NanoGPT? + +NanoGPT is a flexible AI API service that offers: +- **Pay-Per-Prompt Pricing**: No subscriptions, pay only for what you use +- **200+ AI Models**: Access to text, image, and video generation models +- **No Registration Required**: Get started instantly +- **OpenAI-Compatible API**: Easy integration with existing code +- **Streaming Support**: Real-time response streaming +- **Tool Calling**: Support for function calling + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key +``` + +Get your NanoGPT API key from [nano-gpt.com](https://nano-gpt.com). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="NanoGPT Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# NanoGPT call +response = completion( + model="nano-gpt/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="NanoGPT Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# NanoGPT call with streaming +response = completion( + model="nano-gpt/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +### Tool Calling + +```python showLineNumbers title="NanoGPT Tool Calling" +import os +import litellm + +os.environ["NANOGPT_API_KEY"] = "" + +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"} + } + } + } + } +] + +response = litellm.completion( + model="nano-gpt/model-name", + messages=[{"role": "user", "content": "What's the weather in Paris?"}], + tools=tools +) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export NANOGPT_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: nano-gpt-model + litellm_params: + model: nano-gpt/model-name # Replace with actual model name + api_key: os.environ/NANOGPT_API_KEY +``` + +## Supported OpenAI Parameters + +NanoGPT supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID from 200+ available models | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `n` | integer | Optional. Number of completions to generate | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | +| `response_format` | object | Optional. Response format specification | +| `user` | string | Optional. User identifier | + +## Model Categories + +NanoGPT provides access to multiple model categories: +- **Text Generation**: 200+ LLMs for chat, completion, and analysis +- **Image Generation**: AI models for creating images +- **Video Generation**: AI models for video creation +- **Embedding Models**: Text embedding models for vector search + +## Pricing Model + +NanoGPT offers a flexible pricing structure: +- **Pay-Per-Prompt**: No subscription required +- **No Registration**: Get started immediately +- **Transparent Pricing**: Pay only for what you use + +## API Documentation + +For detailed API documentation, visit [docs.nano-gpt.com](https://docs.nano-gpt.com). + +## Additional Resources + +- [NanoGPT Website](https://nano-gpt.com) +- [NanoGPT API Documentation](https://nano-gpt.com/api) +- [NanoGPT Model List](https://docs.nano-gpt.com/api-reference/endpoint/models) diff --git a/docs/my-website/docs/providers/openai.md b/docs/my-website/docs/providers/openai.md index 509a106d8a4..80645a51ac5 100644 --- a/docs/my-website/docs/providers/openai.md +++ b/docs/my-website/docs/providers/openai.md @@ -495,7 +495,7 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \ |-------|----------------------|------------------| | `gpt-5.1` | `none` | `none`, `low`, `medium`, `high` | | `gpt-5` | `medium` | `minimal`, `low`, `medium`, `high` | -| `gpt-5-mini` | `medium` | `none`, `minimal`, `low`, `medium`, `high` | +| `gpt-5-mini` | `medium` | `minimal`, `low`, `medium`, `high` | | `gpt-5-nano` | `none` | `none`, `low`, `medium`, `high` | | `gpt-5-codex` | `adaptive` | `low`, `medium`, `high` (no `minimal`) | | `gpt-5.1-codex` | `adaptive` | `low`, `medium`, `high` (no `minimal`) | diff --git a/docs/my-website/docs/providers/poe.md b/docs/my-website/docs/providers/poe.md new file mode 100644 index 00000000000..ba4089ae6a4 --- /dev/null +++ b/docs/my-website/docs/providers/poe.md @@ -0,0 +1,139 @@ +# Poe + +## Overview + +| Property | Details | +|-------|-------| +| Description | Poe is Quora's AI platform that provides access to more than 100 models across text, image, video, and voice modalities through a developer-friendly API. | +| Provider Route on LiteLLM | `poe/` | +| Link to Provider Doc | [Poe Website ↗](https://poe.com) | +| Base URL | `https://api.poe.com/v1` | +| Supported Operations | [`/chat/completions`](#sample-usage) | + +
+ +## What is Poe? + +Poe is Quora's comprehensive AI platform that offers: +- **100+ Models**: Access to a wide variety of AI models +- **Multiple Modalities**: Text, image, video, and voice AI +- **Popular Models**: Including OpenAI's GPT series and Anthropic's Claude +- **Developer API**: Easy integration for applications +- **Extensive Reach**: Benefits from Quora's 400M monthly unique visitors + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["POE_API_KEY"] = "" # your Poe API key +``` + +Get your Poe API key from the [Poe platform](https://poe.com). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="Poe Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["POE_API_KEY"] = "" # your Poe API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# Poe call +response = completion( + model="poe/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="Poe Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["POE_API_KEY"] = "" # your Poe API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# Poe call with streaming +response = completion( + model="poe/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export POE_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: poe-model + litellm_params: + model: poe/model-name # Replace with actual model name + api_key: os.environ/POE_API_KEY +``` + +## Supported OpenAI Parameters + +Poe supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID from 100+ available models | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | +| `response_format` | object | Optional. Response format specification | +| `user` | string | Optional. User identifier | + +## Available Model Categories + +Poe provides access to models across multiple providers: +- **OpenAI Models**: Including GPT-4, GPT-4 Turbo, GPT-3.5 Turbo +- **Anthropic Models**: Including Claude 3 Opus, Sonnet, Haiku +- **Other Popular Models**: Various provider models available +- **Multi-Modal**: Text, image, video, and voice models + +## Platform Benefits + +Using Poe through LiteLLM offers several advantages: +- **Unified Access**: Single API for many different models +- **Quora Integration**: Access to large user base and content ecosystem +- **Content Sharing**: Capabilities to share model outputs with followers +- **Content Distribution**: Best AI content distributed to all users +- **Model Discovery**: Efficient way to explore new AI models + +## Developer Resources + +Poe is actively building developer features and welcomes early access requests for API integration. + +## Additional Resources + +- [Poe Website](https://poe.com) +- [Poe AI Quora Space](https://poeai.quora.com) +- [Quora Blog Post about Poe](https://quorablog.quora.com/Poe) diff --git a/docs/my-website/docs/providers/synthetic.md b/docs/my-website/docs/providers/synthetic.md new file mode 100644 index 00000000000..b3ba3d0a9e7 --- /dev/null +++ b/docs/my-website/docs/providers/synthetic.md @@ -0,0 +1,119 @@ +# Synthetic + +## Overview + +| Property | Details | +|-------|-------| +| Description | Synthetic runs open-source AI models in secure datacenters within the US and EU, with a focus on privacy. They never train on your data and auto-delete API data within 14 days. | +| Provider Route on LiteLLM | `synthetic/` | +| Link to Provider Doc | [Synthetic Website ↗](https://synthetic.new) | +| Base URL | `https://api.synthetic.new/openai/v1` | +| Supported Operations | [`/chat/completions`](#sample-usage) | + +
+ +## What is Synthetic? + +Synthetic is a privacy-focused AI platform that provides access to open-source LLMs with the following guarantees: +- **Privacy-First**: Data never used for training +- **Secure Hosting**: Models run in secure datacenters in US and EU +- **Auto-Deletion**: API data automatically deleted within 14 days +- **Open Source**: Runs open-source AI models + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key +``` + +Get your Synthetic API key from [synthetic.new](https://synthetic.new). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="Synthetic Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# Synthetic call +response = completion( + model="synthetic/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="Synthetic Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# Synthetic call with streaming +response = completion( + model="synthetic/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export SYNTHETIC_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: synthetic-model + litellm_params: + model: synthetic/model-name # Replace with actual model name + api_key: os.environ/SYNTHETIC_API_KEY +``` + +## Supported OpenAI Parameters + +Synthetic supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | + +## Privacy & Security + +Synthetic provides enterprise-grade privacy protections: +- Data auto-deleted within 14 days +- No data used for model training +- Secure hosting in US and EU datacenters +- Compliance-friendly architecture + +## Additional Resources + +- [Synthetic Website](https://synthetic.new) diff --git a/docs/my-website/docs/providers/zai.md b/docs/my-website/docs/providers/zai.md index 5055d0c1cdd..937ccd67680 100644 --- a/docs/my-website/docs/providers/zai.md +++ b/docs/my-website/docs/providers/zai.md @@ -19,7 +19,7 @@ import os os.environ['ZAI_API_KEY'] = "" response = completion( - model="zai/glm-4.6", + model="zai/glm-4.7", messages=[ {"role": "user", "content": "hello from litellm"} ], @@ -34,7 +34,7 @@ import os os.environ['ZAI_API_KEY'] = "" response = completion( - model="zai/glm-4.6", + model="zai/glm-4.7", messages=[ {"role": "user", "content": "hello from litellm"} ], @@ -51,7 +51,8 @@ We support ALL Z.AI GLM models, just set `zai/` as a prefix when sending complet | Model Name | Function Call | Notes | |------------|---------------|-------| -| glm-4.6 | `completion(model="zai/glm-4.6", messages)` | Latest flagship model, 200K context | +| glm-4.7 | `completion(model="zai/glm-4.7", messages)` | **Latest flagship**, 200K context, **Reasoning** | +| glm-4.6 | `completion(model="zai/glm-4.6", messages)` | 200K context | | glm-4.5 | `completion(model="zai/glm-4.5", messages)` | 128K context | | glm-4.5v | `completion(model="zai/glm-4.5v", messages)` | Vision model | | glm-4.5-x | `completion(model="zai/glm-4.5-x", messages)` | Premium tier | @@ -62,16 +63,17 @@ We support ALL Z.AI GLM models, just set `zai/` as a prefix when sending complet ## Model Pricing -| Model | Input ($/1M tokens) | Output ($/1M tokens) | Context Window | -|-------|---------------------|----------------------|----------------| -| glm-4.6 | $0.60 | $2.20 | 200K | -| glm-4.5 | $0.60 | $2.20 | 128K | -| glm-4.5v | $0.60 | $1.80 | 128K | -| glm-4.5-x | $2.20 | $8.90 | 128K | -| glm-4.5-air | $0.20 | $1.10 | 128K | -| glm-4.5-airx | $1.10 | $4.50 | 128K | -| glm-4-32b-0414-128k | $0.10 | $0.10 | 128K | -| glm-4.5-flash | **FREE** | **FREE** | 128K | +| Model | Input ($/1M tokens) | Output ($/1M tokens) | Cached Input ($/1M tokens) | Context Window | +|-------|---------------------|----------------------|---------------------------|----------------| +| glm-4.7 | $0.60 | $2.20 | $0.11 | 200K | +| glm-4.6 | $0.60 | $2.20 | - | 200K | +| glm-4.5 | $0.60 | $2.20 | - | 128K | +| glm-4.5v | $0.60 | $1.80 | - | 128K | +| glm-4.5-x | $2.20 | $8.90 | - | 128K | +| glm-4.5-air | $0.20 | $1.10 | - | 128K | +| glm-4.5-airx | $1.10 | $4.50 | - | 128K | +| glm-4-32b-0414-128k | $0.10 | $0.10 | - | 128K | +| glm-4.5-flash | **FREE** | **FREE** | - | 128K | ## Using with LiteLLM Proxy @@ -84,7 +86,7 @@ import os os.environ['ZAI_API_KEY'] = "" response = completion( - model="zai/glm-4.6", + model="zai/glm-4.7", messages=[{"role": "user", "content": "Hello, how are you?"}], ) @@ -98,9 +100,9 @@ print(response.choices[0].message.content) ```yaml model_list: - - model_name: glm-4.6 + - model_name: glm-4.7 litellm_params: - model: zai/glm-4.6 + model: zai/glm-4.7 api_key: os.environ/ZAI_API_KEY - model_name: glm-4.5-flash # Free tier litellm_params: @@ -121,7 +123,7 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer sk-1234' \ -d '{ - "model": "glm-4.6", + "model": "glm-4.7", "messages": [ { "role": "user", diff --git a/docs/my-website/docs/proxy/alerting.md b/docs/my-website/docs/proxy/alerting.md index 4cbcd0cffce..38d6d47be44 100644 --- a/docs/my-website/docs/proxy/alerting.md +++ b/docs/my-website/docs/proxy/alerting.md @@ -215,16 +215,16 @@ general_settings: alerting: ["slack"] alerting_threshold: 0.0001 # (Seconds) set an artificially low threshold for testing alerting alert_to_webhook_url: { - "llm_exceptions": "https://hooks.slack.com/services/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", - "llm_too_slow": "https://hooks.slack.com/services/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", - "llm_requests_hanging": "https://hooks.slack.com/services/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", - "budget_alerts": "https://hooks.slack.com/services/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", - "db_exceptions": "https://hooks.slack.com/services/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", - "daily_reports": "https://hooks.slack.com/services/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", - "spend_reports": "https://hooks.slack.com/services/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", - "cooldown_deployment": "https://hooks.slack.com/services/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", - "new_model_added": "https://hooks.slack.com/services/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", - "outage_alerts": "https://hooks.slack.com/services/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", + "llm_exceptions": "example-slack-webhook-url", + "llm_too_slow": "example-slack-webhook-url", + "llm_requests_hanging": "example-slack-webhook-url", + "budget_alerts": "example-slack-webhook-url", + "db_exceptions": "example-slack-webhook-url", + "daily_reports": "example-slack-webhook-url", + "spend_reports": "example-slack-webhook-url", + "cooldown_deployment": "example-slack-webhook-url", + "new_model_added": "example-slack-webhook-url", + "outage_alerts": "example-slack-webhook-url", } litellm_settings: @@ -399,7 +399,7 @@ curl -X GET --location 'http://0.0.0.0:4000/health/services?service=webhook' \ { "spend": 1, # the spend for the 'event_group' "max_budget": 0, # the 'max_budget' set for the 'event_group' - "token": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b", + "token": "example-api-key-123", "user_id": "default_user_id", "team_id": null, "user_email": null, diff --git a/docs/my-website/docs/proxy/call_hooks.md b/docs/my-website/docs/proxy/call_hooks.md index fa420009cf1..fe865f67e09 100644 --- a/docs/my-website/docs/proxy/call_hooks.md +++ b/docs/my-website/docs/proxy/call_hooks.md @@ -17,6 +17,7 @@ import Image from '@theme/IdealImage'; | `async_pre_call_hook` | Modify incoming request before it's sent to model | Before the LLM API call is made | | `async_moderation_hook` | Run checks on input in parallel to LLM API call | In parallel with the LLM API call | | `async_post_call_success_hook` | Modify outgoing response (non-streaming) | After successful LLM API call, for non-streaming responses | +| `async_post_call_failure_hook` | Transform error responses sent to clients | After failed LLM API call | | `async_post_call_streaming_hook` | Modify outgoing response (streaming) | After successful LLM API call, for streaming responses | See a complete example with our [parallel request rate limiter](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/hooks/parallel_request_limiter.py) @@ -60,7 +61,21 @@ class MyCustomHandler(CustomLogger): # https://docs.litellm.ai/docs/observabilit original_exception: Exception, user_api_key_dict: UserAPIKeyAuth, traceback_str: Optional[str] = None, - ): + ) -> Optional[HTTPException]: + """ + Transform error responses sent to clients. + + Return an HTTPException to replace the original error with a user-friendly message. + Return None to use the original exception. + + Example: + if isinstance(original_exception, litellm.ContextWindowExceededError): + return HTTPException( + status_code=400, + detail="Your prompt is too long. Please reduce the length and try again." + ) + return None # Use original exception + """ pass async def async_post_call_success_hook( @@ -339,3 +354,38 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \ "usage": {} } ``` + +## Advanced - Transform Error Responses + +Transform technical API errors into user-friendly messages using `async_post_call_failure_hook`. Return an `HTTPException` to replace the original error, or `None` to use the original exception. + +```python +from litellm.integrations.custom_logger import CustomLogger +from fastapi import HTTPException +from typing import Optional +import litellm + +class MyErrorTransformer(CustomLogger): + async def async_post_call_failure_hook( + self, + request_data: dict, + original_exception: Exception, + user_api_key_dict: UserAPIKeyAuth, + traceback_str: Optional[str] = None, + ) -> Optional[HTTPException]: + if isinstance(original_exception, litellm.ContextWindowExceededError): + return HTTPException( + status_code=400, + detail="Your prompt is too long. Please reduce the length and try again." + ) + if isinstance(original_exception, litellm.RateLimitError): + return HTTPException( + status_code=429, + detail="Rate limit exceeded. Please try again in a moment." + ) + return None # Use original exception + +proxy_handler_instance = MyErrorTransformer() +``` + +**Result:** Clients receive `"Your prompt is too long..."` instead of `"ContextWindowExceededError: Prompt exceeds context window"`. diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md index 343cbd0e53f..f4359a86ba9 100644 --- a/docs/my-website/docs/proxy/config_settings.md +++ b/docs/my-website/docs/proxy/config_settings.md @@ -111,6 +111,7 @@ general_settings: master_key: string maximum_spend_logs_retention_period: 30d # The maximum time to retain spend logs before deletion. maximum_spend_logs_retention_interval: 1d # interval in which the spend log cleanup task should run in. + user_mcp_management_mode: restricted # or "view_all" # Database Settings database_url: string @@ -230,6 +231,7 @@ router_settings: | image_generation_model | str | The default model to use for image generation - ignores model set in request | | store_model_in_db | boolean | If true, enables storing model + credential information in the DB. | | supported_db_objects | List[str] | Fine-grained control over which object types to load from the database when `store_model_in_db` is True. Available types: `"models"`, `"mcp"`, `"guardrails"`, `"vector_stores"`, `"pass_through_endpoints"`, `"prompts"`, `"model_cost_map"`. If not set, all object types are loaded (default behavior). Example: `supported_db_objects: ["mcp"]` to only load MCP servers from DB. | +| user_mcp_management_mode | string | Controls what non-admins can see on the MCP dashboard. `restricted` (default) only lists MCP servers that the user’s teams are explicitly allowed to access. `view_all` lets every user see the full MCP server list. Tool list/call always respects per-key permissions, so users still cannot run MCP calls without access. | | store_prompts_in_spend_logs | boolean | If true, allows prompts and responses to be stored in the spend logs table. | | max_request_size_mb | int | The maximum size for requests in MB. Requests above this size will be rejected. | | max_response_size_mb | int | The maximum size for responses in MB. LLM Responses above this size will not be sent. | @@ -464,6 +466,9 @@ router_settings: | DATABASE_USER | Username for database connection | DATABASE_USERNAME | Alias for database user | DATABRICKS_API_BASE | Base URL for Databricks API +| DATABRICKS_CLIENT_ID | Client ID for Databricks OAuth M2M authentication (Service Principal application ID) +| DATABRICKS_CLIENT_SECRET | Client secret for Databricks OAuth M2M authentication +| DATABRICKS_USER_AGENT | Custom user agent string for Databricks API requests. Used for partner telemetry attribution | DAYS_IN_A_MONTH | Days in a month for calculation purposes. Default is 28 | DAYS_IN_A_WEEK | Days in a week for calculation purposes. Default is 7 | DAYS_IN_A_YEAR | Days in a year for calculation purposes. Default is 365 @@ -666,6 +671,7 @@ router_settings: | LANGSMITH_DEFAULT_RUN_NAME | Default name for Langsmith run | LANGSMITH_PROJECT | Project name for Langsmith integration | LANGSMITH_SAMPLING_RATE | Sampling rate for Langsmith logging +| LANGSMITH_TENANT_ID | Tenant ID for Langsmith multi-tenant deployments | LANGTRACE_API_KEY | API key for Langtrace service | LASSO_API_BASE | Base URL for Lasso API | LASSO_API_KEY | API key for Lasso service @@ -704,10 +710,12 @@ router_settings: | LITELLM_MODE | Operating mode for LiteLLM (e.g., production, development) | LITELLM_NON_ROOT | Flag to run LiteLLM in non-root mode for enhanced security in Docker containers | LITELLM_RATE_LIMIT_WINDOW_SIZE | Rate limit window size for LiteLLM. Default is 60 +| LITELLM_REASONING_AUTO_SUMMARY | If set to "true", automatically enables detailed reasoning summaries for reasoning models (e.g., o1, o3-mini, deepseek-reasoner). When enabled, adds `summary: "detailed"` to reasoning effort configurations. Default is "false" | LITELLM_SALT_KEY | Salt key for encryption in LiteLLM | LITELLM_SSL_CIPHERS | SSL/TLS cipher configuration for faster handshakes. Controls cipher suite preferences for OpenSSL connections. | LITELLM_SECRET_AWS_KMS_LITELLM_LICENSE | AWS KMS encrypted license for LiteLLM | LITELLM_TOKEN | Access token for LiteLLM integration +| LITELLM_USER_AGENT | Custom user agent string for LiteLLM API requests. Used for partner telemetry attribution | LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD | If true, prints the standard logging payload to the console - useful for debugging | LITELM_ENVIRONMENT | Environment for LiteLLM Instance. This is currently only logged to DeepEval to determine the environment for DeepEval integration. | LOGFIRE_TOKEN | Token for Logfire logging service @@ -770,6 +778,7 @@ router_settings: | OTEL_EXPORTER_OTLP_HEADERS | Headers for OpenTelemetry requests | OTEL_SERVICE_NAME | Service name identifier for OpenTelemetry | OTEL_TRACER_NAME | Tracer name for OpenTelemetry tracing +| OTEL_LOGS_EXPORTER | Exporter type for OpenTelemetry logs (e.g., console) | PAGERDUTY_API_KEY | API key for PagerDuty Alerting | PANW_PRISMA_AIRS_API_KEY | API key for PANW Prisma AIRS service | PANW_PRISMA_AIRS_API_BASE | Base URL for PANW Prisma AIRS service @@ -884,4 +893,4 @@ router_settings: | DEFAULT_SHARED_HEALTH_CHECK_LOCK_TTL | Time-to-live in seconds for health check lock in shared health check mode. Default is 60 (1 minute) | ZSCALER_AI_GUARD_API_KEY | API key for Zscaler AI Guard service | ZSCALER_AI_GUARD_POLICY_ID | Policy ID for Zscaler AI Guard guardrails -| ZSCALER_AI_GUARD_URL | Base URL for Zscaler AI Guard API. Default is https://api.us1.zseclipse.net/v1/detection/execute-policy \ No newline at end of file +| ZSCALER_AI_GUARD_URL | Base URL for Zscaler AI Guard API. Default is https://api.us1.zseclipse.net/v1/detection/execute-policy diff --git a/docs/my-website/docs/proxy/configs.md b/docs/my-website/docs/proxy/configs.md index ba4ca190aa9..bc2f6a13362 100644 --- a/docs/my-website/docs/proxy/configs.md +++ b/docs/my-website/docs/proxy/configs.md @@ -116,7 +116,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \ "role": "user", "content": "what llm are you" } - ], + ] } ' ``` diff --git a/docs/my-website/docs/proxy/cost_tracking.md b/docs/my-website/docs/proxy/cost_tracking.md index 019cd62c620..26a4920c093 100644 --- a/docs/my-website/docs/proxy/cost_tracking.md +++ b/docs/my-website/docs/proxy/cost_tracking.md @@ -722,7 +722,7 @@ curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end ```shell [ { - "api_key": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b", + "api_key": "example-api-key-123", "total_cost": 0.3201286305151999, "total_input_tokens": 36.0, "total_output_tokens": 1593.0, @@ -766,7 +766,7 @@ curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end ```shell [ { - "api_key": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b", + "api_key": "example-api-key-123", "total_cost": 0.00013132, "total_input_tokens": 105.0, "total_output_tokens": 872.0, @@ -1151,7 +1151,7 @@ curl -X GET "http://0.0.0.0:4000/spend/logs?request_id=
Loading...
;
- }
-
- if (!accessToken || !token || !userRole || !userID) {
- return Loading...
;
- }
- let all_models_on_proxy: any[] = [];
- let all_providers: string[] = [];
-
- // loop through model data and edit each row
- for (let i = 0; i < modelData.data.length; i++) {
- let curr_model = modelData.data[i];
- let litellm_model_name = curr_model?.litellm_params?.model;
- let custom_llm_provider = curr_model?.litellm_params?.custom_llm_provider;
- let model_info = curr_model?.model_info;
-
- let defaultProvider = "openai";
- let provider = "";
- let input_cost = "Undefined";
- let output_cost = "Undefined";
- let max_tokens = "Undefined";
- let max_input_tokens = "Undefined";
- let cleanedLitellmParams = {};
-
- const getProviderFromModel = (model: string) => {
- /**
- * Use model map
- * - check if model in model map
- * - return it's litellm_provider, if so
- */
- if (modelMap !== null && modelMap !== undefined) {
- if (typeof modelMap == "object" && model in modelMap) {
- return modelMap[model]["litellm_provider"];
- }
- }
- return "openai";
- };
-
- // Check if litellm_model_name is null or undefined
- if (litellm_model_name) {
- // Split litellm_model_name based on "/"
- let splitModel = litellm_model_name.split("/");
-
- // Get the first element in the split
- let firstElement = splitModel[0];
-
- // If there is only one element, default provider to openai
- provider = custom_llm_provider;
- if (!provider) {
- provider = splitModel.length === 1 ? getProviderFromModel(litellm_model_name) : firstElement;
- }
- } else {
- // litellm_model_name is null or undefined, default provider to openai
- provider = "-";
- }
-
- if (model_info) {
- input_cost = model_info?.input_cost_per_token;
- output_cost = model_info?.output_cost_per_token;
- max_tokens = model_info?.max_tokens;
- max_input_tokens = model_info?.max_input_tokens;
- }
-
- if (curr_model?.litellm_params) {
- cleanedLitellmParams = Object.fromEntries(
- Object.entries(curr_model?.litellm_params).filter(([key]) => key !== "model" && key !== "api_base"),
- );
- }
-
- modelData.data[i].provider = provider;
- modelData.data[i].input_cost = input_cost;
- modelData.data[i].output_cost = output_cost;
- modelData.data[i].litellm_model_name = litellm_model_name;
- all_providers.push(provider);
-
- // Convert Cost in terms of Cost per 1M tokens
- if (modelData.data[i].input_cost) {
- modelData.data[i].input_cost = (Number(modelData.data[i].input_cost) * 1000000).toFixed(2);
- }
-
- if (modelData.data[i].output_cost) {
- modelData.data[i].output_cost = (Number(modelData.data[i].output_cost) * 1000000).toFixed(2);
- }
-
- modelData.data[i].max_tokens = max_tokens;
- modelData.data[i].max_input_tokens = max_input_tokens;
- modelData.data[i].api_base = curr_model?.litellm_params?.api_base;
- modelData.data[i].cleanedLitellmParams = cleanedLitellmParams;
-
- all_models_on_proxy.push(curr_model.model_name);
- }
- // when users click request access show pop up to allow them to request access
+ const isLoading = isLoadingModels || isLoadingModelCostMap || isLoadingCredentials || isLoadingUISettings;
if (userRole && userRole == "Admin Viewer") {
const { Title, Paragraph } = Typography;
@@ -499,62 +224,20 @@ const ModelsAndEndpointsView: React.FC
- {date &&
- );
- };
-
- const handleOk = () => {
- addModelForm
- .validateFields()
- .then((values: any) => {
- handleAddModelSubmit(values, accessToken, addModelForm, handleRefreshClick);
- })
- .catch((error: any) => {
- const errorMessages =
- error.errorFields
- ?.map((field: any) => {
- return `${field.name.join(".")}: ${field.errors.join(", ")}`;
- })
- .join(" | ") || "Unknown validation error";
- NotificationsManager.fromBackend(`Please fill in the following required fields: ${errorMessages}`);
- });
};
Object.keys(Providers).find((key) => (Providers as { [index: string]: any })[key] === selectedProvider);
@@ -568,7 +251,7 @@ const ModelsAndEndpointsView: React.FCDate: {date}
} - {sortedPayload.map((category: any, idx: number) => { - const roundedValue = parseFloat(category.value.toFixed(5)); - const displayValue = roundedValue === 0 && category.value > 0 ? "<0.00001" : roundedValue.toFixed(5); - return ( -
-
- );
- })}
-
-
-
- {category.dataKey}
-{displayValue}
-
+
+ {selectedModelId && !isLoading ? (
+
+
+
+
+ Request Provider
+
+
+ Missing a provider?
++ The LiteLLM engineering team is constantly adding support for new LLM models, providers, endpoints. If you don't see the one you need, let us know and we'll prioritize it. +
+
-
- {filteredData.length > 0
- ? `Showing ${pagination.pageIndex * pagination.pageSize + 1} - ${Math.min(
- (pagination.pageIndex + 1) * pagination.pageSize,
- filteredData.length,
- )} of ${filteredData.length} results`
- : "Showing 0 results"}
-
+ {isLoading ? (
+
@@ -366,13 +393,14 @@ const AllModelsTab = ({
getDisplayModelName,
() => {},
() => {},
- setEditModel,
expandedRows,
setExpandedRows,
)}
- data={paginatedData}
+ data={filteredData}
isLoading={false}
- table={tableRef}
+ pagination={pagination}
+ onPaginationChange={setPagination}
+ enablePagination={true}
/>
diff --git a/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/PriceDataManagementTab.tsx b/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/PriceDataManagementTab.tsx
index 4076c19c665..d44d19879d5 100644
--- a/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/PriceDataManagementTab.tsx
+++ b/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/PriceDataManagementTab.tsx
@@ -1,15 +1,12 @@
import { TabPanel, Text, Title } from "@tremor/react";
import PriceDataReload from "@/components/price_data_reload";
-import { modelCostMap } from "@/components/networking";
import React from "react";
import useAuthorized from "@/app/(dashboard)/hooks/useAuthorized";
+import { useModelCostMap } from "../../hooks/models/useModelCostMap";
-interface PriceDataManagementPanelProps {
- setModelMap: (data: any) => void;
-}
-
-const PriceDataManagementTab = ({ setModelMap }: PriceDataManagementPanelProps) => {
+const PriceDataManagementTab = () => {
const { accessToken } = useAuthorized();
+ const { refetch: refetchModelCostMap } = useModelCostMap();
return (
+
+ {isLoading ? (
+
- )}
+ )}
+
+
+
+
+
+ The Admin UI has been disabled by the administrator. To re-enable it, please update the following
+ environment variable:
+
+
+
+
+ }
+ type="warning"
+ showIcon
+ />
+
+
+
+ );
+ }
+
return (
+ 🚅 LiteLLM
+
+
+ DISABLE_ADMIN_UI=False
+ Public Model Hub
,
}));
@@ -51,7 +51,12 @@ describe("ModelHubTable", () => {
const getUiConfigMock = vi.mocked(networking.getUiConfig);
const modelHubPublicModelsCallMock = vi.mocked(networking.modelHubPublicModelsCall);
- getUiConfigMock.mockResolvedValue({ server_root_path: "/", proxy_base_url: "http://localhost:4000" });
+ getUiConfigMock.mockResolvedValue({
+ server_root_path: "/",
+ proxy_base_url: "http://localhost:4000",
+ auto_redirect_to_sso: false,
+ admin_ui_disabled: false,
+ });
modelHubPublicModelsCallMock.mockResolvedValue([]);
render(
-
- ),
- width: 500,
- okText: "Close",
- maskClosable: true,
- keyboard: true,
- });
return true;
} catch (error) {
@@ -319,29 +293,41 @@ const UsefulLinksManagement: React.FC- Your useful links have been saved and are now visible on the public model hub. -
-
-
- View your updated model hub:
- - Open Public Model Hub → - -
+ Public Model Hub
+
- )}
+ ) : (
+
+
+
+
+ )}
+
diff --git a/ui/litellm-dashboard/src/components/AIHub/forms/MakeAgentPublicForm.test.tsx b/ui/litellm-dashboard/src/components/AIHub/forms/MakeAgentPublicForm.test.tsx
new file mode 100644
index 00000000000..67c6d7d6cc9
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/AIHub/forms/MakeAgentPublicForm.test.tsx
@@ -0,0 +1,505 @@
+import { render, screen, fireEvent, act, waitFor } from "@testing-library/react";
+import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
+import MakeAgentPublicForm from "./MakeAgentPublicForm";
+import { AgentHubData } from "@/components/AIHub/AgentHubTableColumns";
+
+// Mock the networking function
+vi.mock("../../networking", () => ({
+ makeAgentsPublicCall: vi.fn(),
+}));
+
+// Import the mocked function
+import { makeAgentsPublicCall } from "../../networking";
+const mockMakeAgentsPublicCall = vi.mocked(makeAgentsPublicCall);
+
+// Mock antd components
+vi.mock("antd", () => ({
+ Modal: ({ open, title, children, onCancel, footer }: any) =>
+ open ? (
+ ();
+ return {
+ ...actual,
+ Text: ({ children, className }: any) => {children},
+ Title: ({ children }: any) => = ({
+ marginConfig,
+ selectedProvider,
+ marginType,
+ percentageValue,
+ fixedAmountValue,
+ onProviderChange,
+ onMarginTypeChange,
+ onPercentageChange,
+ onFixedAmountChange,
+ onAddProvider,
+}) => {
+ return (
+ = ({
return (
- Configure cost discounts for different LLM providers. Changes are saved automatically.
+ Configure cost discounts and margins for different LLM providers. Changes are saved automatically.
-
+
+ ) : null,
+ Form: Object.assign(({ children, form }: any) => , {
+ useForm: () => [
+ {
+ resetFields: vi.fn(),
+ validateFields: vi.fn(),
+ getFieldsValue: vi.fn(),
+ setFieldsValue: vi.fn(),
+ },
+ vi.fn(),
+ ],
+ Item: ({ children }: any) => {title}
+ {children}
+ {footer}
+ {children}
,
+ }),
+ Steps: Object.assign(
+ ({ children, current, className }: any) => (
+
+ {children}
+
+ ),
+ {
+ Step: ({ title }: any) => {title}
,
+ },
+ ),
+ Button: ({ children, onClick, disabled, loading, ...props }: any) => (
+
+ ),
+ Checkbox: ({ checked, indeterminate, onChange, children, disabled }: any) => (
+
+ ),
+}));
+
+// Mock @tremor/react components
+vi.mock("@tremor/react", () => ({
+ Text: ({ children, className }: any) => {children},
+ Title: ({ children }: any) => {children}
, + Badge: ({ children, color, size }: any) => ( + + {children} + + ), +})); + +describe("MakeAgentPublicForm", () => { + const mockProps = { + visible: true, + onClose: vi.fn(), + accessToken: "test-token", + agentHubData: [ + { + agent_id: "agent-1", + name: "Test Agent 1", + description: "Description 1", + version: "1.0", + is_public: false, + skills: [ + { id: "skill-1", name: "Skill 1", description: "Skill desc" }, + { id: "skill-2", name: "Skill 2", description: "Skill desc" }, + ], + protocolVersion: "1.0", + }, + { + agent_id: "agent-2", + name: "Test Agent 2", + description: "Description 2", + version: "2.0", + is_public: true, + skills: [], + protocolVersion: "1.0", + }, + ] as AgentHubData[], + onSuccess: vi.fn(), + }; + + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(() => { + vi.resetAllMocks(); + }); + + it("should render the component", () => { + render(
+
+ ) : null,
+ Form: Object.assign(({ children, form }: any) => , {
+ useForm: () => [
+ {
+ resetFields: vi.fn(),
+ validateFields: vi.fn(),
+ getFieldsValue: vi.fn(),
+ setFieldsValue: vi.fn(),
+ },
+ vi.fn(),
+ ],
+ Item: ({ children }: any) => {title}
+ {children}
+ {footer}
+ {children}
,
+ }),
+ Steps: Object.assign(
+ ({ children, current, className }: any) => (
+
+ {children}
+
+ ),
+ {
+ Step: ({ title }: any) => {title}
,
+ },
+ ),
+ Button: ({ children, onClick, disabled, loading, ...props }: any) => (
+
+ ),
+ Checkbox: ({ checked, indeterminate, onChange, children, disabled }: any) => (
+
+ ),
+}));
+
+// Additional @tremor/react mocks (Button is already mocked globally)
+vi.mock("@tremor/react", async (importOriginal) => {
+ const actual = await importOriginal{children}
, + Badge: ({ children, color, size }: any) => ( + + {children} + + ), + }; +}); + +describe("MakeMCPPublicForm", () => { + const mockProps = { + visible: true, + onClose: vi.fn(), + accessToken: "test-token", + mcpHubData: [ + { + server_id: "server-1", + server_name: "Test Server 1", + description: "Description 1", + url: "http://example.com/server1", + transport: "http", + status: "active", + mcp_info: { is_public: false }, + allowed_tools: ["tool-1", "tool-2"], + auth_type: "bearer", + credentials: {}, + created_at: "2024-01-01T00:00:00Z", + created_by: "user1", + updated_at: "2024-01-01T00:00:00Z", + updated_by: "user1", + teams: [], + mcp_access_groups: [], + extra_headers: [], + static_headers: {}, + args: [], + env: {}, + }, + { + server_id: "server-2", + server_name: "Test Server 2", + description: "Description 2", + url: "http://example.com/server2", + transport: "websocket", + status: "inactive", + mcp_info: { is_public: true }, + allowed_tools: [], + auth_type: "none", + credentials: {}, + created_at: "2024-01-01T00:00:00Z", + created_by: "user2", + updated_at: "2024-01-01T00:00:00Z", + updated_by: "user2", + teams: [], + mcp_access_groups: [], + extra_headers: [], + static_headers: {}, + args: [], + env: {}, + }, + ] as MCPServerData[], + onSuccess: vi.fn(), + }; + + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(() => { + vi.resetAllMocks(); + }); + + it("should render the component", () => { + render(
+
+ ) : null,
+ Form: Object.assign(({ children, form }: any) => , {
+ useForm: () => [
+ {
+ resetFields: vi.fn(),
+ validateFields: vi.fn(),
+ getFieldsValue: vi.fn(),
+ setFieldsValue: vi.fn(),
+ },
+ vi.fn(),
+ ],
+ Item: ({ children }: any) => {title}
+ {children}
+ {footer}
+ {children}
,
+ }),
+ Steps: Object.assign(
+ ({ children, current, className }: any) => (
+
+ {children}
+
+ ),
+ {
+ Step: ({ title }: any) => {title}
,
+ },
+ ),
+ Button: ({ children, onClick, disabled, loading, ...props }: any) => (
+
+ ),
+ Checkbox: ({ checked, indeterminate, onChange, children, disabled }: any) => (
+
+ ),
+}));
+
+// Mock @tremor/react components
+vi.mock("@tremor/react", () => ({
+ Text: ({ children, className }: any) => {children},
+ Title: ({ children }: any) => {children}
, + Badge: ({ children, color, size }: any) => ( + + {children} + + ), +})); + +// Mock ModelFilters component +vi.mock("../../model_filters", () => ({ + default: ({ onFilteredDataChange, modelHubData }: any) => ( +
+
+
+ ),
+}));
+
+// Mock NotificationsManager
+vi.mock("../../molecules/notifications_manager", () => ({
+ default: {
+ fromBackend: vi.fn(),
+ success: vi.fn(),
+ },
+}));
+
+describe("MakeModelPublicForm", () => {
+ const mockProps = {
+ visible: true,
+ onClose: vi.fn(),
+ accessToken: "test-token",
+ modelHubData: [
+ {
+ model_group: "gpt-4",
+ providers: ["openai"],
+ max_input_tokens: 8192,
+ max_output_tokens: 4096,
+ input_cost_per_token: 0.03,
+ output_cost_per_token: 0.06,
+ mode: "chat",
+ tpm: 10000,
+ rpm: 200,
+ supports_parallel_function_calling: true,
+ supports_vision: false,
+ supports_function_calling: true,
+ supported_openai_params: ["temperature", "max_tokens"],
+ is_public_model_group: false,
+ },
+ {
+ model_group: "gpt-3.5-turbo",
+ providers: ["openai"],
+ max_input_tokens: 4096,
+ max_output_tokens: 2048,
+ input_cost_per_token: 0.0015,
+ output_cost_per_token: 0.002,
+ mode: "chat",
+ tpm: 60000,
+ rpm: 3500,
+ supports_parallel_function_calling: false,
+ supports_vision: false,
+ supports_function_calling: true,
+ supported_openai_params: ["temperature", "max_tokens"],
+ is_public_model_group: true,
+ },
+ ] as ModelGroupInfo[],
+ onSuccess: vi.fn(),
+ };
+
+ beforeEach(() => {
+ vi.clearAllMocks();
+ });
+
+ afterEach(() => {
+ vi.resetAllMocks();
+ });
+
+ it("should render the component", () => {
+ render(
+
+ Provider
+
+
+
+ }
+ rules={[{ required: true, message: "Please select a provider" }]}
+ >
+
+ String(option?.label ?? "").toLowerCase().includes(input.toLowerCase())
+ }
+ >
+
+
+ {Object.entries(Providers).map(([providerEnum, providerDisplayName]) => {
+ const providerValue = provider_map[providerEnum as keyof typeof provider_map];
+ // Only show providers that don't already have a margin configured
+ if (providerValue && marginConfig[providerValue]) {
+ return null;
+ }
+ return (
+
+
+ );
+ })}
+
+
+
+
+ Margin Type
+
+
+
+ }
+ rules={[{ required: true, message: "Please select a margin type" }]}
+ >
+ onMarginTypeChange(e.target.value)}
+ className="w-full"
+ >
+ Percentage-based
+ Fixed Amount
+
+
+
+ {marginType === "percentage" && (
+
+ Margin Percentage
+
+
+
+ }
+ rules={[
+ { required: true, message: "Please enter a margin percentage" },
+ {
+ validator: (_, value) => {
+ if (!value) {
+ return Promise.reject(new Error("Please enter a margin percentage"));
+ }
+ const numValue = parseFloat(value);
+ if (isNaN(numValue) || numValue < 0 || numValue > 1000) {
+ return Promise.reject(new Error("Percentage must be between 0 and 1000"));
+ }
+ return Promise.resolve();
+ },
+ },
+ ]}
+ >
+
+ )}
+
+ {marginType === "fixed" && (
+
+ Fixed Margin Amount
+
+
+
+ }
+ rules={[
+ { required: true, message: "Please enter a fixed amount" },
+ {
+ validator: (_, value) => {
+ if (!value) {
+ return Promise.reject(new Error("Please enter a fixed amount"));
+ }
+ const numValue = parseFloat(value);
+ if (isNaN(numValue) || numValue < 0) {
+ return Promise.reject(new Error("Fixed amount must be non-negative"));
+ }
+ return Promise.resolve();
+ },
+ },
+ ]}
+ >
+
+ )}
+
+
+ );
+};
+
+export default AddMarginForm;
+
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
index 2d530be71eb..7f79a6848bb 100644
--- a/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
@@ -2,7 +2,6 @@ import React from "react";
import { TextInput, Button } from "@tremor/react";
import { Select as AntdSelect, Form, Tooltip } from "antd";
import { InfoCircleOutlined } from "@ant-design/icons";
-import Image from "next/image";
import { Providers, provider_map, providerLogoMap } from "../provider_info_helpers";
import { DiscountConfig } from "./types";
import { handleImageError } from "./provider_display_helpers";
@@ -58,11 +57,9 @@ const AddProviderForm: React.FC
+ Global (All Providers)
+
+
+ 
handleImageError(e, providerDisplayName)}
+ />
+ {providerDisplayName}
+
+
+
+
+ $
+
+
+
+
+
- handleImageError(e, providerDisplayName)}
/>
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
index c356982f189..3b9ea30e128 100644
--- a/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
@@ -1,16 +1,18 @@
-import React, { useState, useEffect, useCallback } from "react";
-import { Title, Text, Button, TabGroup, TabList, Tab, TabPanels, TabPanel } from "@tremor/react";
+import React, { useState, useEffect } from "react";
+import { Title, Text, Button, Accordion, AccordionHeader, AccordionBody, TabGroup, TabList, Tab, TabPanels, TabPanel } from "@tremor/react";
import { Modal, Form } from "antd";
-import { getProxyBaseUrl } from "@/components/networking";
-import NotificationsManager from "../molecules/notifications_manager";
-import { Providers } from "../provider_info_helpers";
-import { CostTrackingSettingsProps, DiscountConfig } from "./types";
-import { getProviderBackendValue } from "./provider_display_helpers";
+import { CostTrackingSettingsProps } from "./types";
import ProviderDiscountTable from "./provider_discount_table";
import AddProviderForm from "./add_provider_form";
+import ProviderMarginTable from "./provider_margin_table";
+import AddMarginForm from "./add_margin_form";
+import PricingCalculator from "./pricing_calculator/index";
import { ExclamationCircleOutlined } from "@ant-design/icons";
import { DocsMenu } from "../HelpLink";
import HowItWorks from "./how_it_works";
+import { useDiscountConfig } from "./use_discount_config";
+import { useMarginConfig } from "./use_margin_config";
+import { fetchAvailableModels, ModelGroup } from "../playground/llm_calls/fetch_models";
const DOCS_LINKS = [
{ label: "Custom pricing for models", href: "https://docs.litellm.ai/docs/proxy/custom_pricing" },
@@ -22,118 +24,65 @@ const CostTrackingSettings: React.FC = ({
userRole,
accessToken
}) => {
- const [discountConfig, setDiscountConfig] = useState({});
const [selectedProvider, setSelectedProvider] = useState(undefined);
const [newDiscount, setNewDiscount] = useState("");
const [isFetching, setIsFetching] = useState(true);
const [isModalVisible, setIsModalVisible] = useState(false);
+ const [isMarginModalVisible, setIsMarginModalVisible] = useState(false);
+ const [selectedMarginProvider, setSelectedMarginProvider] = useState(undefined);
+ const [marginType, setMarginType] = useState<"percentage" | "fixed">("percentage");
+ const [percentageValue, setPercentageValue] = useState("");
+ const [fixedAmountValue, setFixedAmountValue] = useState("");
+ const [models, setModels] = useState([]);
const [form] = Form.useForm();
+ const [marginForm] = Form.useForm();
const [modal, contextHolder] = Modal.useModal();
+
+ const isProxyAdmin = userRole === "proxy_admin" || userRole === "Admin";
- const fetchDiscountConfig = useCallback(async () => {
- setIsFetching(true);
- try {
- const proxyBaseUrl = getProxyBaseUrl();
- const url = proxyBaseUrl
- ? `${proxyBaseUrl}/config/cost_discount_config`
- : "/config/cost_discount_config";
-
- const response = await fetch(url, {
- method: "GET",
- headers: {
- Authorization: `Bearer ${accessToken}`,
- "Content-Type": "application/json",
- },
- });
+ // Use custom hooks for discount and margin config
+ const {
+ discountConfig,
+ fetchDiscountConfig,
+ handleAddProvider: addProvider,
+ handleRemoveProvider: removeProvider,
+ handleDiscountChange,
+ } = useDiscountConfig({ accessToken });
- if (response.ok) {
- const data = await response.json();
- setDiscountConfig(data.values || {});
- } else {
- console.error("Failed to fetch discount config");
- }
- } catch (error) {
- console.error("Error fetching discount config:", error);
- NotificationsManager.fromBackend("Failed to fetch discount configuration");
- } finally {
- setIsFetching(false);
- }
- }, [accessToken]);
+ const {
+ marginConfig,
+ fetchMarginConfig,
+ handleAddMargin: addMargin,
+ handleRemoveMargin: removeMargin,
+ handleMarginChange,
+ } = useMarginConfig({ accessToken });
useEffect(() => {
if (accessToken) {
- fetchDiscountConfig();
- }
- }, [accessToken, fetchDiscountConfig]);
-
- const saveDiscountConfig = async (config: DiscountConfig) => {
- try {
- const proxyBaseUrl = getProxyBaseUrl();
- const url = proxyBaseUrl
- ? `${proxyBaseUrl}/config/cost_discount_config`
- : "/config/cost_discount_config";
-
- const response = await fetch(url, {
- method: "PATCH",
- headers: {
- Authorization: `Bearer ${accessToken}`,
- "Content-Type": "application/json",
- },
- body: JSON.stringify(config),
+ Promise.all([fetchDiscountConfig(), fetchMarginConfig()]).finally(() => {
+ setIsFetching(false);
});
-
- if (response.ok) {
- NotificationsManager.success("Discount configuration updated successfully");
- await fetchDiscountConfig();
- } else {
- const errorData = await response.json();
- const errorMessage = errorData.detail?.error || errorData.detail || "Failed to update settings";
- NotificationsManager.fromBackend(errorMessage);
- }
- } catch (error) {
- console.error("Error updating discount config:", error);
- NotificationsManager.fromBackend("Failed to update discount configuration");
+
+ // Fetch models for pricing calculator (available to all roles)
+ const loadModels = async () => {
+ try {
+ const modelGroups = await fetchAvailableModels(accessToken);
+ setModels(modelGroups.map((m: ModelGroup) => m.model_group));
+ } catch (error) {
+ console.error("Error fetching models:", error);
+ }
+ };
+ loadModels();
}
- };
+ }, [accessToken, fetchDiscountConfig, fetchMarginConfig]);
const handleAddProvider = async () => {
- if (!selectedProvider || !newDiscount) {
- NotificationsManager.fromBackend("Please select a provider and enter discount percentage");
- return;
+ const success = await addProvider(selectedProvider, newDiscount);
+ if (success) {
+ setSelectedProvider(undefined);
+ setNewDiscount("");
+ setIsModalVisible(false);
}
-
- const percentageValue = parseFloat(newDiscount);
- if (isNaN(percentageValue) || percentageValue < 0 || percentageValue > 100) {
- NotificationsManager.fromBackend("Discount must be between 0% and 100%");
- return;
- }
-
- const providerValue = getProviderBackendValue(selectedProvider);
-
- if (!providerValue) {
- NotificationsManager.fromBackend("Invalid provider selected");
- return;
- }
-
- if (discountConfig[providerValue]) {
- NotificationsManager.fromBackend(
- `Discount for ${Providers[selectedProvider as keyof typeof Providers]} already exists. Edit it in the table above.`
- );
- return;
- }
-
- // Convert percentage to decimal for storage
- const discountValue = percentageValue / 100;
- const updatedConfig = {
- ...discountConfig,
- [providerValue]: discountValue,
- };
-
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
- setSelectedProvider(undefined);
- setNewDiscount("");
- setIsModalVisible(false);
};
const handleModalCancel = () => {
@@ -143,7 +92,7 @@ const CostTrackingSettings: React.FC = ({
setNewDiscount("");
};
- const handleFormSubmit = (values: any) => {
+ const handleFormSubmit = () => {
handleAddProvider();
};
@@ -155,27 +104,47 @@ const CostTrackingSettings: React.FC = ({
okText: 'Remove',
okType: 'danger',
cancelText: 'Cancel',
- onOk: async () => {
- const updatedConfig = { ...discountConfig };
- delete updatedConfig[provider];
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
- },
+ onOk: () => removeProvider(provider),
});
};
- const handleDiscountChange = async (provider: string, value: string) => {
- const discountValue = parseFloat(value);
- if (!isNaN(discountValue) && discountValue >= 0 && discountValue <= 1) {
- const updatedConfig = {
- ...discountConfig,
- [provider]: discountValue,
- };
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
+ const handleAddMargin = async () => {
+ const success = await addMargin({
+ selectedProvider: selectedMarginProvider,
+ marginType,
+ percentageValue,
+ fixedAmountValue,
+ });
+ if (success) {
+ setSelectedMarginProvider(undefined);
+ setPercentageValue("");
+ setFixedAmountValue("");
+ setMarginType("percentage");
+ setIsMarginModalVisible(false);
}
};
+ const handleMarginModalCancel = () => {
+ setIsMarginModalVisible(false);
+ marginForm.resetFields();
+ setSelectedMarginProvider(undefined);
+ setPercentageValue("");
+ setFixedAmountValue("");
+ setMarginType("percentage");
+ };
+
+ const handleRemoveMargin = async (provider: string, providerDisplayName: string) => {
+ modal.confirm({
+ title: 'Remove Provider Margin',
+ icon: = ({
-
-
- Provider Discounts
- Test It
-
-
-
- {isFetching ? (
-
-
-
+
+ }
+ open={isMarginModalVisible}
+ width={1000}
+ onCancel={handleMarginModalCancel}
+ footer={null}
+ className="top-8"
+ styles={{
+ body: { padding: "24px" },
+ header: { padding: "24px 24px 0 24px", border: "none" },
+ }}
+ >
+
- Loading configuration...
-
- ) : Object.keys(discountConfig).length > 0 ? (
-
-
- ) : (
-
-
-
- No provider discounts configured
-
-
- Click "Add Provider Discount" to get started
-
-
- )}
-
- = ({
+
+
+ {/* Accordion 1: Provider Discounts - Only for proxy admins */}
+ {isProxyAdmin && (
+
+
+
+
+
+
+ Discounts
+ Test It
+
+
+
+
+
+
+
+
+
+
+ )}
+
+ {/* Accordion 2: Fee/Price Margin - Only for proxy admins */}
+ {isProxyAdmin && (
+
+
+
+
+
+
+ )}
+
+ {/* Accordion 3: Pricing Calculator - Available to all roles */}
+
+
+
+
+
+
+ Provider Discounts
+
+ Apply percentage-based discounts to reduce costs for specific providers
+
-
-
-
+
+
+
+
+
+ {isFetching ? (
+
+ Loading configuration...
+
+ ) : Object.keys(discountConfig).length > 0 ? (
+
+
+
+ No provider discounts configured
+
+
+ Click "Add Provider Discount" to get started
+
+
+ )}
+
+
+
+ Fee/Price Margin
+
+ Add fees or margins to LLM costs for internal billing and cost recovery
+
+
+
+
+
+
+
+ {isFetching ? (
+
+ Loading configuration...
+
+ ) : Object.keys(marginConfig).length > 0 ? (
+
+
+
+ No provider margins configured
+
+
+ Click "Add Provider Margin" to get started
+
+
+ )}
+
+ Pricing Calculator
+
+ Estimate LLM costs based on expected token usage and request volume
+
+
+
+
+ Add Provider Margin
+
+
+ Select a provider (or "Global" for all providers) and configure the margin. You can use percentage-based or fixed amount.
+
+
+
+
);
};
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/index.ts b/ui/litellm-dashboard/src/components/CostTrackingSettings/index.ts
index 11adc414664..feba943154b 100644
--- a/ui/litellm-dashboard/src/components/CostTrackingSettings/index.ts
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/index.ts
@@ -1,8 +1,12 @@
export { default as CostTrackingSettings } from "./cost_tracking_settings";
export { default as ProviderDiscountTable } from "./provider_discount_table";
export { default as AddProviderForm } from "./add_provider_form";
+export { default as ProviderMarginTable } from "./provider_margin_table";
+export { default as AddMarginForm } from "./add_margin_form";
export { default as HowItWorks } from "./how_it_works";
-export type { CostTrackingSettingsProps, DiscountConfig, CostDiscountResponse } from "./types";
+export type { CostTrackingSettingsProps, DiscountConfig, CostDiscountResponse, MarginConfig, CostMarginResponse } from "./types";
export type { ProviderDisplayInfo } from "./provider_display_helpers";
export * from "./provider_display_helpers";
+export { useDiscountConfig } from "./use_discount_config";
+export { useMarginConfig } from "./use_margin_config";
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/cost_results.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/cost_results.tsx
new file mode 100644
index 00000000000..03d5ca0e518
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/cost_results.tsx
@@ -0,0 +1,203 @@
+import React from "react";
+import { Text } from "@tremor/react";
+import { Card, Statistic, Row, Col, Divider, Spin } from "antd";
+import { DollarOutlined, LoadingOutlined } from "@ant-design/icons";
+import { CostEstimateResponse } from "../types";
+import { formatNumberWithCommas } from "@/utils/dataUtils";
+import ExportDropdown from "./export_dropdown";
+
+interface CostResultsProps {
+ result: CostEstimateResponse | null;
+ loading: boolean;
+}
+
+const formatCost = (value: number | null | undefined): string => {
+ if (value === null || value === undefined) return "-";
+ if (value === 0) return "$0";
+ if (value < 0.0001) return `$${value.toExponential(2)}`;
+ if (value < 1) return `$${value.toFixed(4)}`;
+ return `$${formatNumberWithCommas(value, 2, true)}`;
+};
+
+const formatRequests = (value: number | null | undefined): string => {
+ if (value === null || value === undefined) return "-";
+ return formatNumberWithCommas(value, 0, true);
+};
+
+const CostResults: React.FC
+
+ Select a model to see cost estimates
+
+
+ );
+ }
+
+ if (loading && !result) {
+ return (
+
+ Calculating costs...
+
+ );
+ }
+
+ if (!result) return null;
+
+ return (
+
+
+
+ 0 ? "#faad14" : undefined,
+ }}
+ />
+
+
+
+
+ {result.daily_cost !== null && (
+
+
+ 0 ? "#faad14" : undefined,
+ }}
+ />
+
+
+
+ )}
+
+ {result.monthly_cost !== null && (
+
+
+ 0 ? "#faad14" : undefined,
+ }}
+ />
+
+
+
+ )}
+
+ {(result.input_cost_per_token || result.output_cost_per_token) && (
+
+ );
+};
+
+export default CostResults;
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_dropdown.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_dropdown.tsx
new file mode 100644
index 00000000000..e8a681021d6
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_dropdown.tsx
@@ -0,0 +1,71 @@
+import React, { useState, useRef, useEffect } from "react";
+import { Button } from "@tremor/react";
+import { DownloadOutlined, FilePdfOutlined, FileExcelOutlined } from "@ant-design/icons";
+import { CostEstimateResponse } from "../types";
+import { exportToPDF, exportToCSV } from "./export_utils";
+
+interface ExportDropdownProps {
+ result: CostEstimateResponse;
+}
+
+const ExportDropdown: React.FC
+
+
+
+ Cost Estimate
+
+ Model: {result.model} {result.provider && `(${result.provider})`}
+
+
+
+ {loading &&
+
+ Token Pricing:
+ {result.input_cost_per_token && (
+ Input: ${formatNumberWithCommas(result.input_cost_per_token * 1_000_000, 2)}/1M tokens
+ )}
+ {result.input_cost_per_token && result.output_cost_per_token && " | "}
+ {result.output_cost_per_token && (
+ Output: ${formatNumberWithCommas(result.output_cost_per_token * 1_000_000, 2)}/1M tokens
+ )}
+
+ )}
+
+
+
+ {isOpen && (
+
+ );
+};
+
+export default ExportDropdown;
+
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_utils.ts b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_utils.ts
new file mode 100644
index 00000000000..a205efa3bf8
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_utils.ts
@@ -0,0 +1,276 @@
+import { CostEstimateResponse } from "../types";
+import { formatNumberWithCommas } from "@/utils/dataUtils";
+
+const formatCostForExport = (value: number | null | undefined): string => {
+ if (value === null || value === undefined) return "-";
+ if (value === 0) return "$0.00";
+ if (value < 0.01) return `$${value.toFixed(6)}`;
+ if (value < 1) return `$${value.toFixed(4)}`;
+ return `$${formatNumberWithCommas(value, 2)}`;
+};
+
+const formatRequestsForExport = (value: number | null | undefined): string => {
+ if (value === null || value === undefined) return "-";
+ return formatNumberWithCommas(value, 0);
+};
+
+export const exportToPDF = (result: CostEstimateResponse): void => {
+ const printWindow = window.open("", "_blank");
+ if (!printWindow) {
+ alert("Please allow popups to export PDF");
+ return;
+ }
+
+ const html = `
+
+
+
+
+
+
+
+ )}
+ 🚅 LiteLLM Cost Estimate Report
+ + + +Per-Request Cost Breakdown
+| Cost Type | +Amount | +
|---|---|
| Input Cost | +${formatCostForExport(result.input_cost_per_request)} | +
| Output Cost | +${formatCostForExport(result.output_cost_per_request)} | +
| Margin/Fee | +${formatCostForExport(result.margin_cost_per_request)} | +
| Total per Request | +${formatCostForExport(result.cost_per_request)} | +
Daily Cost Estimate (${formatRequestsForExport(result.num_requests_per_day)} requests/day)
+| Cost Type | +Amount | +
|---|---|
| Input Cost | +${formatCostForExport(result.daily_input_cost)} | +
| Output Cost | +${formatCostForExport(result.daily_output_cost)} | +
| Margin/Fee | +${formatCostForExport(result.daily_margin_cost)} | +
| Total Daily | +${formatCostForExport(result.daily_cost)} | +
Monthly Cost Estimate (${formatRequestsForExport(result.num_requests_per_month)} requests/month)
+| Cost Type | +Amount | +
|---|---|
| Input Cost | +${formatCostForExport(result.monthly_input_cost)} | +
| Output Cost | +${formatCostForExport(result.monthly_output_cost)} | +
| Margin/Fee | +${formatCostForExport(result.monthly_margin_cost)} | +
| Total Monthly | +${formatCostForExport(result.monthly_cost)} | +
Token Pricing
+| Token Type | +Price per 1M Tokens | +
|---|---|
| Input Tokens | +$${(result.input_cost_per_token * 1000000).toFixed(2)} | +
| Output Tokens | +$${(result.output_cost_per_token * 1000000).toFixed(2)} | +