+ +## What is LlamaGate? + +LlamaGate provides access to open-source LLMs through an OpenAI-compatible API: +- **26+ Open-Source Models**: Llama 3.1/3.2, Mistral, Qwen, DeepSeek R1, and more +- **OpenAI-Compatible API**: Drop-in replacement for OpenAI SDK +- **Vision Models**: Qwen VL, LLaVA, olmOCR, UI-TARS for multimodal tasks +- **Reasoning Models**: DeepSeek R1, OpenThinker for complex problem-solving +- **Code Models**: CodeLlama, DeepSeek Coder, Qwen Coder, StarCoder2 +- **Embedding Models**: Nomic, Qwen3 Embedding for RAG and search +- **Competitive Pricing**: $0.02-$0.55 per 1M tokens + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["LLAMAGATE_API_KEY"] = "" # your LlamaGate API key +``` + +Get your API key from [llamagate.dev](https://llamagate.dev). + +## Supported Models + +### General Purpose +| Model | Model ID | +|-------|----------| +| Llama 3.1 8B | `llamagate/llama-3.1-8b` | +| Llama 3.2 3B | `llamagate/llama-3.2-3b` | +| Mistral 7B v0.3 | `llamagate/mistral-7b-v0.3` | +| Qwen 3 8B | `llamagate/qwen3-8b` | +| Dolphin 3 8B | `llamagate/dolphin3-8b` | + +### Reasoning Models +| Model | Model ID | +|-------|----------| +| DeepSeek R1 8B | `llamagate/deepseek-r1-8b` | +| DeepSeek R1 Distill Qwen 7B | `llamagate/deepseek-r1-7b-qwen` | +| OpenThinker 7B | `llamagate/openthinker-7b` | + +### Code Models +| Model | Model ID | +|-------|----------| +| Qwen 2.5 Coder 7B | `llamagate/qwen2.5-coder-7b` | +| DeepSeek Coder 6.7B | `llamagate/deepseek-coder-6.7b` | +| CodeLlama 7B | `llamagate/codellama-7b` | +| CodeGemma 7B | `llamagate/codegemma-7b` | +| StarCoder2 7B | `llamagate/starcoder2-7b` | + +### Vision Models +| Model | Model ID | +|-------|----------| +| Qwen 3 VL 8B | `llamagate/qwen3-vl-8b` | +| LLaVA 1.5 7B | `llamagate/llava-7b` | +| Gemma 3 4B | `llamagate/gemma3-4b` | +| olmOCR 7B | `llamagate/olmocr-7b` | +| UI-TARS 1.5 7B | `llamagate/ui-tars-7b` | + +### Embedding Models +| Model | Model ID | +|-------|----------| +| Nomic Embed Text | `llamagate/nomic-embed-text` | +| Qwen 3 Embedding 8B | `llamagate/qwen3-embedding-8b` | +| EmbeddingGemma 300M | `llamagate/embeddinggemma-300m` | + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="LlamaGate Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["LLAMAGATE_API_KEY"] = "" # your LlamaGate API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# LlamaGate call +response = completion( + model="llamagate/llama-3.1-8b", + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="LlamaGate Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["LLAMAGATE_API_KEY"] = "" # your LlamaGate API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# LlamaGate call with streaming +response = completion( + model="llamagate/llama-3.1-8b", + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +### Vision + +```python showLineNumbers title="LlamaGate Vision Completion" +import os +import litellm +from litellm import completion + +os.environ["LLAMAGATE_API_KEY"] = "" # your LlamaGate API key + +messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's in this image?"}, + {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} + ] + } +] + +# LlamaGate vision call +response = completion( + model="llamagate/qwen3-vl-8b", + messages=messages +) + +print(response) +``` + +### Embeddings + +```python showLineNumbers title="LlamaGate Embeddings" +import os +import litellm +from litellm import embedding + +os.environ["LLAMAGATE_API_KEY"] = "" # your LlamaGate API key + +# LlamaGate embedding call +response = embedding( + model="llamagate/nomic-embed-text", + input=["Hello world", "How are you?"] +) + +print(response) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export LLAMAGATE_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: llama-3.1-8b + litellm_params: + model: llamagate/llama-3.1-8b + api_key: os.environ/LLAMAGATE_API_KEY + - model_name: deepseek-r1 + litellm_params: + model: llamagate/deepseek-r1-8b + api_key: os.environ/LLAMAGATE_API_KEY + - model_name: qwen-coder + litellm_params: + model: llamagate/qwen2.5-coder-7b + api_key: os.environ/LLAMAGATE_API_KEY +``` + +## Supported OpenAI Parameters + +LlamaGate supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature (0-2) | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | +| `response_format` | object | Optional. JSON mode or JSON schema | + +## Pricing + +LlamaGate offers competitive per-token pricing: + +| Model Category | Input (per 1M) | Output (per 1M) | +|----------------|----------------|-----------------| +| Embeddings | $0.02 | - | +| Small (3-4B) | $0.03-$0.04 | $0.08 | +| Medium (7-8B) | $0.03-$0.15 | $0.05-$0.55 | +| Code Models | $0.06-$0.10 | $0.12-$0.20 | +| Reasoning | $0.08-$0.10 | $0.15-$0.20 | + +## Additional Resources + +- [LlamaGate Documentation](https://llamagate.dev/docs) +- [LlamaGate Pricing](https://llamagate.dev/pricing) +- [LlamaGate API Reference](https://llamagate.dev/docs/api) diff --git a/docs/my-website/docs/providers/minimax.md b/docs/my-website/docs/providers/minimax.md new file mode 100644 index 00000000000..9505c26aade --- /dev/null +++ b/docs/my-website/docs/providers/minimax.md @@ -0,0 +1,639 @@ +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# MiniMax + +# MiniMax - v1/messages + +## Overview + +Litellm provides anthropic specs compatible support for minmax + +## Supported Models + +MiniMax offers three models through their Anthropic-compatible API: + +| Model | Description | Input Cost | Output Cost | Prompt Caching Read | Prompt Caching Write | +|-------|-------------|------------|-------------|---------------------|----------------------| +| **MiniMax-M2.1** | Powerful Multi-Language Programming with Enhanced Programming Experience (~60 tps) | $0.3/M tokens | $1.2/M tokens | $0.03/M tokens | $0.375/M tokens | +| **MiniMax-M2.1-lightning** | Faster and More Agile (~100 tps) | $0.3/M tokens | $2.4/M tokens | $0.03/M tokens | $0.375/M tokens | +| **MiniMax-M2** | Agentic capabilities, Advanced reasoning | $0.3/M tokens | $1.2/M tokens | $0.03/M tokens | $0.375/M tokens | + + +## Usage Examples + +### Basic Chat Completion + +```python +import litellm + +response = litellm.anthropic.messages.acreate( + model="minimax/MiniMax-M2.1", + messages=[{"role": "user", "content": "Hello, how are you?"}], + api_key="your-minimax-api-key", + api_base="https://api.minimax.io/anthropic/v1/messages", + max_tokens=1000 +) + +print(response.choices[0].message.content) +``` + +### Using Environment Variables + +```bash +export MINIMAX_API_KEY="your-minimax-api-key" +export MINIMAX_API_BASE="https://api.minimax.io/anthropic/v1/messages" +``` + +```python +import litellm + +response = litellm.anthropic.messages.acreate( + model="minimax/MiniMax-M2.1", + messages=[{"role": "user", "content": "Hello!"}], + max_tokens=1000 +) +``` + +### With Thinking (M2.1 Feature) + +```python +response = litellm.anthropic.messages.acreate( + model="minimax/MiniMax-M2.1", + messages=[{"role": "user", "content": "Solve: 2+2=?"}], + thinking={"type": "enabled", "budget_tokens": 1000}, + api_key="your-minimax-api-key" +) + +# Access thinking content +for block in response.choices[0].message.content: + if hasattr(block, 'type') and block.type == 'thinking': + print(f"Thinking: {block.thinking}") +``` + +### With Tool Calling + +```python +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"} + }, + "required": ["location"] + } + } + } +] + +response = litellm.anthropic.messages.acreate( + model="minimax/MiniMax-M2.1", + messages=[{"role": "user", "content": "What's the weather in SF?"}], + tools=tools, + api_key="your-minimax-api-key", + max_tokens=1000 +) +``` + + + +## Usage with LiteLLM Proxy + +You can use MiniMax models with the Anthropic SDK by routing through LiteLLM Proxy: + +| Step | Description | +|------|-------------| +| **1. Start LiteLLM Proxy** | Configure proxy with MiniMax models in `config.yaml` | +| **2. Set Environment Variables** | Point Anthropic SDK to proxy endpoint | +| **3. Use Anthropic SDK** | Call MiniMax models using native Anthropic SDK | + +### Step 1: Configure LiteLLM Proxy + +Create a `config.yaml`: + +```yaml +model_list: + - model_name: minimax/MiniMax-M2.1 + litellm_params: + model: minimax/MiniMax-M2.1 + api_key: os.environ/MINIMAX_API_KEY + api_base: https://api.minimax.io/anthropic/v1/messages +``` + +Start the proxy: + +```bash +litellm --config config.yaml +``` + +### Step 2: Use with Anthropic SDK + +```python +import os +os.environ["ANTHROPIC_BASE_URL"] = "http://localhost:4000" +os.environ["ANTHROPIC_API_KEY"] = "sk-1234" # Your LiteLLM proxy key + +import anthropic + +client = anthropic.Anthropic() + +message = client.messages.create( + model="minimax/MiniMax-M2.1", + max_tokens=1000, + system="You are a helpful assistant.", + messages=[ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Hi, how are you?" + } + ] + } + ] +) + +for block in message.content: + if block.type == "thinking": + print(f"Thinking:\n{block.thinking}\n") + elif block.type == "text": + print(f"Text:\n{block.text}\n") +``` + +# MiniMax - v1/chat/completions + +## Usage with LiteLLM SDK + +You can use MiniMax's OpenAI-compatible API directly with LiteLLM: + +### Basic Chat Completion + +```python +import litellm + +response = litellm.completion( + model="minimax/MiniMax-M2.1", + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello, how are you?"} + ], + api_key="your-minimax-api-key", + api_base="https://api.minimax.io/v1" +) + +print(response.choices[0].message.content) +``` + +### Using Environment Variables + +```bash +export MINIMAX_API_KEY="your-minimax-api-key" +export MINIMAX_API_BASE="https://api.minimax.io/v1" +``` + +```python +import litellm + +response = litellm.completion( + model="minimax/MiniMax-M2.1", + messages=[{"role": "user", "content": "Hello!"}] +) +``` + +### With Reasoning Split + +```python +response = litellm.completion( + model="minimax/MiniMax-M2.1", + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Solve: 2+2=?"} + ], + extra_body={"reasoning_split": True}, + api_key="your-minimax-api-key", + api_base="https://api.minimax.io/v1" +) + +# Access reasoning details if available +if hasattr(response.choices[0].message, 'reasoning_details'): + print(f"Thinking: {response.choices[0].message.reasoning_details}") +print(f"Response: {response.choices[0].message.content}") +``` + +### With Tool Calling + +```python +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"} + }, + "required": ["location"] + } + } + } +] + +response = litellm.completion( + model="minimax/MiniMax-M2.1", + messages=[{"role": "user", "content": "What's the weather in SF?"}], + tools=tools, + api_key="your-minimax-api-key", + api_base="https://api.minimax.io/v1" +) +``` + +### Streaming + +```python +response = litellm.completion( + model="minimax/MiniMax-M2.1", + messages=[{"role": "user", "content": "Tell me a story"}], + stream=True, + api_key="your-minimax-api-key", + api_base="https://api.minimax.io/v1" +) + +for chunk in response: + if chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="") +``` + + +## Usage with OpenAI SDK via LiteLLM Proxy + +You can also use MiniMax models with the OpenAI SDK by routing through LiteLLM Proxy: + +| Step | Description | +|------|-------------| +| **1. Start LiteLLM Proxy** | Configure proxy with MiniMax models in `config.yaml` | +| **2. Set Environment Variables** | Point OpenAI SDK to proxy endpoint | +| **3. Use OpenAI SDK** | Call MiniMax models using native OpenAI SDK | + +### Step 1: Configure LiteLLM Proxy + +Create a `config.yaml`: + +```yaml +model_list: + - model_name: minimax/MiniMax-M2.1 + litellm_params: + model: minimax/MiniMax-M2.1 + api_key: os.environ/MINIMAX_API_KEY + api_base: https://api.minimax.io/v1 +``` + +Start the proxy: + +```bash +litellm --config config.yaml +``` + +### Step 2: Use with OpenAI SDK + +```python +import os +os.environ["OPENAI_BASE_URL"] = "http://localhost:4000" +os.environ["OPENAI_API_KEY"] = "sk-1234" # Your LiteLLM proxy key + +from openai import OpenAI + +client = OpenAI() + +response = client.chat.completions.create( + model="minimax/MiniMax-M2.1", + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hi, how are you?"}, + ], + # Set reasoning_split=True to separate thinking content + extra_body={"reasoning_split": True}, +) + +# Access thinking and response +if hasattr(response.choices[0].message, 'reasoning_details'): + print(f"Thinking:\n{response.choices[0].message.reasoning_details[0]['text']}\n") +print(f"Text:\n{response.choices[0].message.content}\n") +``` + +### Streaming with OpenAI SDK + +```python +from openai import OpenAI + +client = OpenAI() + +stream = client.chat.completions.create( + model="minimax/MiniMax-M2.1", + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Tell me a story"}, + ], + extra_body={"reasoning_split": True}, + stream=True, +) + +reasoning_buffer = "" +text_buffer = "" + +for chunk in stream: + if hasattr(chunk.choices[0].delta, "reasoning_details") and chunk.choices[0].delta.reasoning_details: + for detail in chunk.choices[0].delta.reasoning_details: + if "text" in detail: + reasoning_text = detail["text"] + new_reasoning = reasoning_text[len(reasoning_buffer):] + if new_reasoning: + print(new_reasoning, end="", flush=True) + reasoning_buffer = reasoning_text + + if chunk.choices[0].delta.content: + content_text = chunk.choices[0].delta.content + new_text = content_text[len(text_buffer):] if text_buffer else content_text + if new_text: + print(new_text, end="", flush=True) + text_buffer = content_text +``` + +## Cost Calculation + +Cost calculation works automatically using the pricing information in `model_prices_and_context_window.json`. + +Example: +```python +response = litellm.completion( + model="minimax/MiniMax-M2.1", + messages=[{"role": "user", "content": "Hello!"}], + api_key="your-minimax-api-key" +) + +# Access cost information +print(f"Cost: ${response._hidden_params.get('response_cost', 0)}") +``` + +# MiniMax - Text-to-Speech + +## Quick Start + +## **LiteLLM Python SDK Usage** + +### Basic Usage + +```python +from pathlib import Path +from litellm import speech +import os + +os.environ["MINIMAX_API_KEY"] = "your-api-key" + +speech_file_path = Path(__file__).parent / "speech.mp3" +response = speech( + model="minimax/speech-2.6-hd", + voice="alloy", + input="The quick brown fox jumped over the lazy dogs", +) +response.stream_to_file(speech_file_path) +``` + +### Async Usage + +```python +from litellm import aspeech +from pathlib import Path +import os, asyncio + +os.environ["MINIMAX_API_KEY"] = "your-api-key" + +async def test_async_speech(): + speech_file_path = Path(__file__).parent / "speech.mp3" + response = await aspeech( + model="minimax/speech-2.6-hd", + voice="alloy", + input="The quick brown fox jumped over the lazy dogs", + ) + response.stream_to_file(speech_file_path) + +asyncio.run(test_async_speech()) +``` + +### Voice Selection + +MiniMax supports many voices. LiteLLM provides OpenAI-compatible voice names that map to MiniMax voices: + +```python +from litellm import speech + +# OpenAI-compatible voice names +voices = ["alloy", "echo", "fable", "onyx", "nova", "shimmer"] + +for voice in voices: + response = speech( + model="minimax/speech-2.6-hd", + voice=voice, + input=f"This is the {voice} voice", + ) + response.stream_to_file(f"speech_{voice}.mp3") +``` + +You can also use MiniMax-native voice IDs directly: + +```python +response = speech( + model="minimax/speech-2.6-hd", + voice="male-qn-qingse", # MiniMax native voice ID + input="Using native MiniMax voice ID", +) +``` + +### Custom Parameters + +MiniMax TTS supports additional parameters for fine-tuning audio output: + +```python +from litellm import speech + +response = speech( + model="minimax/speech-2.6-hd", + voice="alloy", + input="Custom audio parameters", + speed=1.5, # Speed: 0.5 to 2.0 + response_format="mp3", # Format: mp3, pcm, wav, flac + extra_body={ + "vol": 1.2, # Volume: 0.1 to 10 + "pitch": 2, # Pitch adjustment: -12 to 12 + "sample_rate": 32000, # 16000, 24000, or 32000 + "bitrate": 128000, # For MP3: 64000, 128000, 192000, 256000 + "channel": 1, # 1 for mono, 2 for stereo + } +) +response.stream_to_file("custom_speech.mp3") +``` + +### Response Formats + +```python +from litellm import speech + +# MP3 format (default) +response = speech( + model="minimax/speech-2.6-hd", + voice="alloy", + input="MP3 format audio", + response_format="mp3", +) + +# PCM format +response = speech( + model="minimax/speech-2.6-hd", + voice="alloy", + input="PCM format audio", + response_format="pcm", +) + +# WAV format +response = speech( + model="minimax/speech-2.6-hd", + voice="alloy", + input="WAV format audio", + response_format="wav", +) + +# FLAC format +response = speech( + model="minimax/speech-2.6-hd", + voice="alloy", + input="FLAC format audio", + response_format="flac", +) +``` + +## **LiteLLM Proxy Usage** + +LiteLLM provides an OpenAI-compatible `/audio/speech` endpoint for MiniMax TTS. + +### Setup + +Add MiniMax to your proxy configuration: + +```yaml +model_list: + - model_name: tts + litellm_params: + model: minimax/speech-2.6-hd + api_key: os.environ/MINIMAX_API_KEY + + - model_name: tts-turbo + litellm_params: + model: minimax/speech-2.6-turbo + api_key: os.environ/MINIMAX_API_KEY +``` + +Start the proxy: + +```bash +litellm --config /path/to/config.yaml + +# RUNNING on http://0.0.0.0:4000 +``` + +### Making Requests + +```bash +curl http://0.0.0.0:4000/v1/audio/speech \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "tts", + "input": "The quick brown fox jumped over the lazy dog.", + "voice": "alloy" + }' \ + --output speech.mp3 +``` + +With custom parameters: + +```bash +curl http://0.0.0.0:4000/v1/audio/speech \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "tts", + "input": "Custom parameters example.", + "voice": "nova", + "speed": 1.5, + "response_format": "mp3", + "extra_body": { + "vol": 1.2, + "pitch": 1, + "sample_rate": 32000 + } + }' \ + --output custom_speech.mp3 +``` + +## Voice Mappings + +LiteLLM maps OpenAI-compatible voice names to MiniMax voice IDs: + +| OpenAI Voice | MiniMax Voice ID | Description | +|--------------|------------------|-------------| +| alloy | male-qn-qingse | Male voice | +| echo | male-qn-jingying | Male voice | +| fable | female-shaonv | Female voice | +| onyx | male-qn-badao | Male voice | +| nova | female-yujie | Female voice | +| shimmer | female-tianmei | Female voice | + +You can also use any MiniMax-native voice ID directly by passing it as the `voice` parameter. + + +### Streaming (WebSocket) + +:::note +The current implementation uses MiniMax's HTTP endpoint. For WebSocket streaming support, please refer to MiniMax's official documentation at [https://platform.minimax.io/docs](https://platform.minimax.io/docs). +::: + +## Error Handling + +```python +from litellm import speech +import litellm + +try: + response = speech( + model="minimax/speech-2.6-hd", + voice="alloy", + input="Test input", + ) + response.stream_to_file("output.mp3") +except litellm.exceptions.BadRequestError as e: + print(f"Bad request: {e}") +except litellm.exceptions.AuthenticationError as e: + print(f"Authentication failed: {e}") +except Exception as e: + print(f"Error: {e}") +``` + +### Extra Body Parameters + +Pass these via `extra_body`: + +| Parameter | Type | Description | Default | +|-----------|------|-------------|---------| +| vol | float | Volume (0.1 to 10) | 1.0 | +| pitch | int | Pitch adjustment (-12 to 12) | 0 | +| sample_rate | int | Sample rate: 16000, 24000, 32000 | 32000 | +| bitrate | int | Bitrate for MP3: 64000, 128000, 192000, 256000 | 128000 | +| channel | int | Audio channels: 1 (mono) or 2 (stereo) | 1 | +| output_format | string | Output format: "hex" or "url" (url returns a URL valid for 24 hours) | hex | diff --git a/docs/my-website/docs/providers/nano-gpt.md b/docs/my-website/docs/providers/nano-gpt.md new file mode 100644 index 00000000000..4e46c032c75 --- /dev/null +++ b/docs/my-website/docs/providers/nano-gpt.md @@ -0,0 +1,170 @@ +# NanoGPT + +## Overview + +| Property | Details | +|-------|-------| +| Description | NanoGPT is a pay-per-prompt and subscription based AI service providing instant access to over 200+ powerful AI models with no subscriptions or registration required. | +| Provider Route on LiteLLM | `nano-gpt/` | +| Link to Provider Doc | [NanoGPT Website ↗](https://nano-gpt.com) | +| Base URL | `https://nano-gpt.com/api/v1` | +| Supported Operations | [`/chat/completions`](#sample-usage), [`/completions`](#text-completion), [`/embeddings`](#embeddings) | + +
+ +## What is NanoGPT? + +NanoGPT is a flexible AI API service that offers: +- **Pay-Per-Prompt Pricing**: No subscriptions, pay only for what you use +- **200+ AI Models**: Access to text, image, and video generation models +- **No Registration Required**: Get started instantly +- **OpenAI-Compatible API**: Easy integration with existing code +- **Streaming Support**: Real-time response streaming +- **Tool Calling**: Support for function calling + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key +``` + +Get your NanoGPT API key from [nano-gpt.com](https://nano-gpt.com). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="NanoGPT Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# NanoGPT call +response = completion( + model="nano-gpt/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="NanoGPT Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# NanoGPT call with streaming +response = completion( + model="nano-gpt/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +### Tool Calling + +```python showLineNumbers title="NanoGPT Tool Calling" +import os +import litellm + +os.environ["NANOGPT_API_KEY"] = "" + +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"} + } + } + } + } +] + +response = litellm.completion( + model="nano-gpt/model-name", + messages=[{"role": "user", "content": "What's the weather in Paris?"}], + tools=tools +) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export NANOGPT_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: nano-gpt-model + litellm_params: + model: nano-gpt/model-name # Replace with actual model name + api_key: os.environ/NANOGPT_API_KEY +``` + +## Supported OpenAI Parameters + +NanoGPT supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID from 200+ available models | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `n` | integer | Optional. Number of completions to generate | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | +| `response_format` | object | Optional. Response format specification | +| `user` | string | Optional. User identifier | + +## Model Categories + +NanoGPT provides access to multiple model categories: +- **Text Generation**: 200+ LLMs for chat, completion, and analysis +- **Image Generation**: AI models for creating images +- **Video Generation**: AI models for video creation +- **Embedding Models**: Text embedding models for vector search + +## Pricing Model + +NanoGPT offers a flexible pricing structure: +- **Pay-Per-Prompt**: No subscription required +- **No Registration**: Get started immediately +- **Transparent Pricing**: Pay only for what you use + +## API Documentation + +For detailed API documentation, visit [docs.nano-gpt.com](https://docs.nano-gpt.com). + +## Additional Resources + +- [NanoGPT Website](https://nano-gpt.com) +- [NanoGPT API Documentation](https://nano-gpt.com/api) +- [NanoGPT Model List](https://docs.nano-gpt.com/api-reference/endpoint/models) diff --git a/docs/my-website/docs/providers/openai.md b/docs/my-website/docs/providers/openai.md index 509a106d8a4..80645a51ac5 100644 --- a/docs/my-website/docs/providers/openai.md +++ b/docs/my-website/docs/providers/openai.md @@ -495,7 +495,7 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \ |-------|----------------------|------------------| | `gpt-5.1` | `none` | `none`, `low`, `medium`, `high` | | `gpt-5` | `medium` | `minimal`, `low`, `medium`, `high` | -| `gpt-5-mini` | `medium` | `none`, `minimal`, `low`, `medium`, `high` | +| `gpt-5-mini` | `medium` | `minimal`, `low`, `medium`, `high` | | `gpt-5-nano` | `none` | `none`, `low`, `medium`, `high` | | `gpt-5-codex` | `adaptive` | `low`, `medium`, `high` (no `minimal`) | | `gpt-5.1-codex` | `adaptive` | `low`, `medium`, `high` (no `minimal`) | diff --git a/docs/my-website/docs/providers/poe.md b/docs/my-website/docs/providers/poe.md new file mode 100644 index 00000000000..ba4089ae6a4 --- /dev/null +++ b/docs/my-website/docs/providers/poe.md @@ -0,0 +1,139 @@ +# Poe + +## Overview + +| Property | Details | +|-------|-------| +| Description | Poe is Quora's AI platform that provides access to more than 100 models across text, image, video, and voice modalities through a developer-friendly API. | +| Provider Route on LiteLLM | `poe/` | +| Link to Provider Doc | [Poe Website ↗](https://poe.com) | +| Base URL | `https://api.poe.com/v1` | +| Supported Operations | [`/chat/completions`](#sample-usage) | + +
+ +## What is Poe? + +Poe is Quora's comprehensive AI platform that offers: +- **100+ Models**: Access to a wide variety of AI models +- **Multiple Modalities**: Text, image, video, and voice AI +- **Popular Models**: Including OpenAI's GPT series and Anthropic's Claude +- **Developer API**: Easy integration for applications +- **Extensive Reach**: Benefits from Quora's 400M monthly unique visitors + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["POE_API_KEY"] = "" # your Poe API key +``` + +Get your Poe API key from the [Poe platform](https://poe.com). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="Poe Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["POE_API_KEY"] = "" # your Poe API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# Poe call +response = completion( + model="poe/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="Poe Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["POE_API_KEY"] = "" # your Poe API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# Poe call with streaming +response = completion( + model="poe/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export POE_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: poe-model + litellm_params: + model: poe/model-name # Replace with actual model name + api_key: os.environ/POE_API_KEY +``` + +## Supported OpenAI Parameters + +Poe supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID from 100+ available models | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | +| `response_format` | object | Optional. Response format specification | +| `user` | string | Optional. User identifier | + +## Available Model Categories + +Poe provides access to models across multiple providers: +- **OpenAI Models**: Including GPT-4, GPT-4 Turbo, GPT-3.5 Turbo +- **Anthropic Models**: Including Claude 3 Opus, Sonnet, Haiku +- **Other Popular Models**: Various provider models available +- **Multi-Modal**: Text, image, video, and voice models + +## Platform Benefits + +Using Poe through LiteLLM offers several advantages: +- **Unified Access**: Single API for many different models +- **Quora Integration**: Access to large user base and content ecosystem +- **Content Sharing**: Capabilities to share model outputs with followers +- **Content Distribution**: Best AI content distributed to all users +- **Model Discovery**: Efficient way to explore new AI models + +## Developer Resources + +Poe is actively building developer features and welcomes early access requests for API integration. + +## Additional Resources + +- [Poe Website](https://poe.com) +- [Poe AI Quora Space](https://poeai.quora.com) +- [Quora Blog Post about Poe](https://quorablog.quora.com/Poe) diff --git a/docs/my-website/docs/providers/synthetic.md b/docs/my-website/docs/providers/synthetic.md new file mode 100644 index 00000000000..b3ba3d0a9e7 --- /dev/null +++ b/docs/my-website/docs/providers/synthetic.md @@ -0,0 +1,119 @@ +# Synthetic + +## Overview + +| Property | Details | +|-------|-------| +| Description | Synthetic runs open-source AI models in secure datacenters within the US and EU, with a focus on privacy. They never train on your data and auto-delete API data within 14 days. | +| Provider Route on LiteLLM | `synthetic/` | +| Link to Provider Doc | [Synthetic Website ↗](https://synthetic.new) | +| Base URL | `https://api.synthetic.new/openai/v1` | +| Supported Operations | [`/chat/completions`](#sample-usage) | + +
+ +## What is Synthetic? + +Synthetic is a privacy-focused AI platform that provides access to open-source LLMs with the following guarantees: +- **Privacy-First**: Data never used for training +- **Secure Hosting**: Models run in secure datacenters in US and EU +- **Auto-Deletion**: API data automatically deleted within 14 days +- **Open Source**: Runs open-source AI models + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key +``` + +Get your Synthetic API key from [synthetic.new](https://synthetic.new). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="Synthetic Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# Synthetic call +response = completion( + model="synthetic/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="Synthetic Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# Synthetic call with streaming +response = completion( + model="synthetic/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export SYNTHETIC_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: synthetic-model + litellm_params: + model: synthetic/model-name # Replace with actual model name + api_key: os.environ/SYNTHETIC_API_KEY +``` + +## Supported OpenAI Parameters + +Synthetic supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | + +## Privacy & Security + +Synthetic provides enterprise-grade privacy protections: +- Data auto-deleted within 14 days +- No data used for model training +- Secure hosting in US and EU datacenters +- Compliance-friendly architecture + +## Additional Resources + +- [Synthetic Website](https://synthetic.new) diff --git a/docs/my-website/docs/providers/zai.md b/docs/my-website/docs/providers/zai.md index 5055d0c1cdd..937ccd67680 100644 --- a/docs/my-website/docs/providers/zai.md +++ b/docs/my-website/docs/providers/zai.md @@ -19,7 +19,7 @@ import os os.environ['ZAI_API_KEY'] = "" response = completion( - model="zai/glm-4.6", + model="zai/glm-4.7", messages=[ {"role": "user", "content": "hello from litellm"} ], @@ -34,7 +34,7 @@ import os os.environ['ZAI_API_KEY'] = "" response = completion( - model="zai/glm-4.6", + model="zai/glm-4.7", messages=[ {"role": "user", "content": "hello from litellm"} ], @@ -51,7 +51,8 @@ We support ALL Z.AI GLM models, just set `zai/` as a prefix when sending complet | Model Name | Function Call | Notes | |------------|---------------|-------| -| glm-4.6 | `completion(model="zai/glm-4.6", messages)` | Latest flagship model, 200K context | +| glm-4.7 | `completion(model="zai/glm-4.7", messages)` | **Latest flagship**, 200K context, **Reasoning** | +| glm-4.6 | `completion(model="zai/glm-4.6", messages)` | 200K context | | glm-4.5 | `completion(model="zai/glm-4.5", messages)` | 128K context | | glm-4.5v | `completion(model="zai/glm-4.5v", messages)` | Vision model | | glm-4.5-x | `completion(model="zai/glm-4.5-x", messages)` | Premium tier | @@ -62,16 +63,17 @@ We support ALL Z.AI GLM models, just set `zai/` as a prefix when sending complet ## Model Pricing -| Model | Input ($/1M tokens) | Output ($/1M tokens) | Context Window | -|-------|---------------------|----------------------|----------------| -| glm-4.6 | $0.60 | $2.20 | 200K | -| glm-4.5 | $0.60 | $2.20 | 128K | -| glm-4.5v | $0.60 | $1.80 | 128K | -| glm-4.5-x | $2.20 | $8.90 | 128K | -| glm-4.5-air | $0.20 | $1.10 | 128K | -| glm-4.5-airx | $1.10 | $4.50 | 128K | -| glm-4-32b-0414-128k | $0.10 | $0.10 | 128K | -| glm-4.5-flash | **FREE** | **FREE** | 128K | +| Model | Input ($/1M tokens) | Output ($/1M tokens) | Cached Input ($/1M tokens) | Context Window | +|-------|---------------------|----------------------|---------------------------|----------------| +| glm-4.7 | $0.60 | $2.20 | $0.11 | 200K | +| glm-4.6 | $0.60 | $2.20 | - | 200K | +| glm-4.5 | $0.60 | $2.20 | - | 128K | +| glm-4.5v | $0.60 | $1.80 | - | 128K | +| glm-4.5-x | $2.20 | $8.90 | - | 128K | +| glm-4.5-air | $0.20 | $1.10 | - | 128K | +| glm-4.5-airx | $1.10 | $4.50 | - | 128K | +| glm-4-32b-0414-128k | $0.10 | $0.10 | - | 128K | +| glm-4.5-flash | **FREE** | **FREE** | - | 128K | ## Using with LiteLLM Proxy @@ -84,7 +86,7 @@ import os os.environ['ZAI_API_KEY'] = "" response = completion( - model="zai/glm-4.6", + model="zai/glm-4.7", messages=[{"role": "user", "content": "Hello, how are you?"}], ) @@ -98,9 +100,9 @@ print(response.choices[0].message.content) ```yaml model_list: - - model_name: glm-4.6 + - model_name: glm-4.7 litellm_params: - model: zai/glm-4.6 + model: zai/glm-4.7 api_key: os.environ/ZAI_API_KEY - model_name: glm-4.5-flash # Free tier litellm_params: @@ -121,7 +123,7 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer sk-1234' \ -d '{ - "model": "glm-4.6", + "model": "glm-4.7", "messages": [ { "role": "user", diff --git a/docs/my-website/docs/proxy/caching.md b/docs/my-website/docs/proxy/caching.md index 6da977c8b05..87e6a6fdb6e 100644 --- a/docs/my-website/docs/proxy/caching.md +++ b/docs/my-website/docs/proxy/caching.md @@ -1,28 +1,29 @@ -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; +import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Caching +# Caching -:::note +:::note For OpenAI/Anthropic Prompt Caching, go [here](../completion/prompt_caching.md) ::: -Cache LLM Responses. LiteLLM's caching system stores and reuses LLM responses to save costs and reduce latency. When you make the same request twice, the cached response is returned instead of calling the LLM API again. - - +Cache LLM Responses. LiteLLM's caching system stores and reuses LLM responses to save costs and +reduce latency. When you make the same request twice, the cached response is returned instead of +calling the LLM API again. ### Supported Caches - In Memory Cache - Disk Cache -- Redis Cache +- Redis Cache - Qdrant Semantic Cache - Redis Semantic Cache -- s3 Bucket Cache +- S3 Bucket Cache +- GCS Bucket Cache ## Quick Start +
Loading...
;
- }
-
- if (!accessToken || !token || !userRole || !userID) {
- return Loading...
;
- }
- let all_models_on_proxy: any[] = [];
- let all_providers: string[] = [];
-
- // loop through model data and edit each row
- for (let i = 0; i < modelData.data.length; i++) {
- let curr_model = modelData.data[i];
- let litellm_model_name = curr_model?.litellm_params?.model;
- let custom_llm_provider = curr_model?.litellm_params?.custom_llm_provider;
- let model_info = curr_model?.model_info;
-
- let defaultProvider = "openai";
- let provider = "";
- let input_cost = "Undefined";
- let output_cost = "Undefined";
- let max_tokens = "Undefined";
- let max_input_tokens = "Undefined";
- let cleanedLitellmParams = {};
-
- const getProviderFromModel = (model: string) => {
- /**
- * Use model map
- * - check if model in model map
- * - return it's litellm_provider, if so
- */
- if (modelMap !== null && modelMap !== undefined) {
- if (typeof modelMap == "object" && model in modelMap) {
- return modelMap[model]["litellm_provider"];
- }
- }
- return "openai";
- };
-
- // Check if litellm_model_name is null or undefined
- if (litellm_model_name) {
- // Split litellm_model_name based on "/"
- let splitModel = litellm_model_name.split("/");
-
- // Get the first element in the split
- let firstElement = splitModel[0];
-
- // If there is only one element, default provider to openai
- provider = custom_llm_provider;
- if (!provider) {
- provider = splitModel.length === 1 ? getProviderFromModel(litellm_model_name) : firstElement;
- }
- } else {
- // litellm_model_name is null or undefined, default provider to openai
- provider = "-";
- }
-
- if (model_info) {
- input_cost = model_info?.input_cost_per_token;
- output_cost = model_info?.output_cost_per_token;
- max_tokens = model_info?.max_tokens;
- max_input_tokens = model_info?.max_input_tokens;
- }
-
- if (curr_model?.litellm_params) {
- cleanedLitellmParams = Object.fromEntries(
- Object.entries(curr_model?.litellm_params).filter(([key]) => key !== "model" && key !== "api_base"),
- );
- }
-
- modelData.data[i].provider = provider;
- modelData.data[i].input_cost = input_cost;
- modelData.data[i].output_cost = output_cost;
- modelData.data[i].litellm_model_name = litellm_model_name;
- all_providers.push(provider);
-
- // Convert Cost in terms of Cost per 1M tokens
- if (modelData.data[i].input_cost) {
- modelData.data[i].input_cost = (Number(modelData.data[i].input_cost) * 1000000).toFixed(2);
- }
-
- if (modelData.data[i].output_cost) {
- modelData.data[i].output_cost = (Number(modelData.data[i].output_cost) * 1000000).toFixed(2);
- }
-
- modelData.data[i].max_tokens = max_tokens;
- modelData.data[i].max_input_tokens = max_input_tokens;
- modelData.data[i].api_base = curr_model?.litellm_params?.api_base;
- modelData.data[i].cleanedLitellmParams = cleanedLitellmParams;
-
- all_models_on_proxy.push(curr_model.model_name);
- }
- // when users click request access show pop up to allow them to request access
+ const isLoading = isLoadingModels || isLoadingModelCostMap || isLoadingCredentials || isLoadingUISettings;
if (userRole && userRole == "Admin Viewer") {
const { Title, Paragraph } = Typography;
@@ -499,62 +224,20 @@ const ModelsAndEndpointsView: React.FC
- {date &&
- );
- };
-
- const handleOk = () => {
- addModelForm
- .validateFields()
- .then((values: any) => {
- handleAddModelSubmit(values, accessToken, addModelForm, handleRefreshClick);
- })
- .catch((error: any) => {
- const errorMessages =
- error.errorFields
- ?.map((field: any) => {
- return `${field.name.join(".")}: ${field.errors.join(", ")}`;
- })
- .join(" | ") || "Unknown validation error";
- NotificationsManager.fromBackend(`Please fill in the following required fields: ${errorMessages}`);
- });
};
Object.keys(Providers).find((key) => (Providers as { [index: string]: any })[key] === selectedProvider);
@@ -568,7 +251,7 @@ const ModelsAndEndpointsView: React.FCDate: {date}
} - {sortedPayload.map((category: any, idx: number) => { - const roundedValue = parseFloat(category.value.toFixed(5)); - const displayValue = roundedValue === 0 && category.value > 0 ? "<0.00001" : roundedValue.toFixed(5); - return ( -
-
- );
- })}
-
-
-
- {category.dataKey}
-{displayValue}
-
+
+ {selectedModelId && !isLoading ? (
+
+
+
+
+ Request Provider
+
+
+ Missing a provider?
++ The LiteLLM engineering team is constantly adding support for new LLM models, providers, endpoints. If + you don't see the one you need, let us know and we'll prioritize it. +
+
-
- {filteredData.length > 0
- ? `Showing ${pagination.pageIndex * pagination.pageSize + 1} - ${Math.min(
- (pagination.pageIndex + 1) * pagination.pageSize,
- filteredData.length,
- )} of ${filteredData.length} results`
- : "Showing 0 results"}
-
+ {isLoading ? (
+
@@ -366,13 +393,14 @@ const AllModelsTab = ({
getDisplayModelName,
() => {},
() => {},
- setEditModel,
expandedRows,
setExpandedRows,
)}
- data={paginatedData}
+ data={filteredData}
isLoading={false}
- table={tableRef}
+ pagination={pagination}
+ onPaginationChange={setPagination}
+ enablePagination={true}
/>
diff --git a/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/PriceDataManagementTab.tsx b/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/PriceDataManagementTab.tsx
index 4076c19c665..d44d19879d5 100644
--- a/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/PriceDataManagementTab.tsx
+++ b/ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/PriceDataManagementTab.tsx
@@ -1,15 +1,12 @@
import { TabPanel, Text, Title } from "@tremor/react";
import PriceDataReload from "@/components/price_data_reload";
-import { modelCostMap } from "@/components/networking";
import React from "react";
import useAuthorized from "@/app/(dashboard)/hooks/useAuthorized";
+import { useModelCostMap } from "../../hooks/models/useModelCostMap";
-interface PriceDataManagementPanelProps {
- setModelMap: (data: any) => void;
-}
-
-const PriceDataManagementTab = ({ setModelMap }: PriceDataManagementPanelProps) => {
+const PriceDataManagementTab = () => {
const { accessToken } = useAuthorized();
+ const { refetch: refetchModelCostMap } = useModelCostMap();
return (
+
+ {isLoading ? (
+
- )}
+ )}
+
+
+
+
+
+ The Admin UI has been disabled by the administrator. To re-enable it, please update the following
+ environment variable:
+
+
+
+
+ }
+ type="warning"
+ showIcon
+ />
+
+
+
+ );
+ }
+
return (
+ 🚅 LiteLLM
+
+
+ DISABLE_ADMIN_UI=False
+ Public Model Hub
,
}));
@@ -51,7 +51,12 @@ describe("ModelHubTable", () => {
const getUiConfigMock = vi.mocked(networking.getUiConfig);
const modelHubPublicModelsCallMock = vi.mocked(networking.modelHubPublicModelsCall);
- getUiConfigMock.mockResolvedValue({ server_root_path: "/", proxy_base_url: "http://localhost:4000" });
+ getUiConfigMock.mockResolvedValue({
+ server_root_path: "/",
+ proxy_base_url: "http://localhost:4000",
+ auto_redirect_to_sso: false,
+ admin_ui_disabled: false,
+ });
modelHubPublicModelsCallMock.mockResolvedValue([]);
render(
-
- ),
- width: 500,
- okText: "Close",
- maskClosable: true,
- keyboard: true,
- });
return true;
} catch (error) {
@@ -319,29 +293,41 @@ const UsefulLinksManagement: React.FC- Your useful links have been saved and are now visible on the public model hub. -
-
-
- View your updated model hub:
- - Open Public Model Hub → - -
+ Public Model Hub
+
- )}
+ ) : (
+
+
+
+
+ )}
+
diff --git a/ui/litellm-dashboard/src/components/AIHub/forms/MakeAgentPublicForm.test.tsx b/ui/litellm-dashboard/src/components/AIHub/forms/MakeAgentPublicForm.test.tsx
new file mode 100644
index 00000000000..67c6d7d6cc9
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/AIHub/forms/MakeAgentPublicForm.test.tsx
@@ -0,0 +1,505 @@
+import { render, screen, fireEvent, act, waitFor } from "@testing-library/react";
+import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
+import MakeAgentPublicForm from "./MakeAgentPublicForm";
+import { AgentHubData } from "@/components/AIHub/AgentHubTableColumns";
+
+// Mock the networking function
+vi.mock("../../networking", () => ({
+ makeAgentsPublicCall: vi.fn(),
+}));
+
+// Import the mocked function
+import { makeAgentsPublicCall } from "../../networking";
+const mockMakeAgentsPublicCall = vi.mocked(makeAgentsPublicCall);
+
+// Mock antd components
+vi.mock("antd", () => ({
+ Modal: ({ open, title, children, onCancel, footer }: any) =>
+ open ? (
+ ();
+ return {
+ ...actual,
+ Text: ({ children, className }: any) => {children},
+ Title: ({ children }: any) => = ({
+ marginConfig,
+ selectedProvider,
+ marginType,
+ percentageValue,
+ fixedAmountValue,
+ onProviderChange,
+ onMarginTypeChange,
+ onPercentageChange,
+ onFixedAmountChange,
+ onAddProvider,
+}) => {
+ return (
+ = ({
return (
- Configure cost discounts for different LLM providers. Changes are saved automatically.
+ Configure cost discounts and margins for different LLM providers. Changes are saved automatically.
-
+
+ ) : null,
+ Form: Object.assign(({ children, form }: any) => , {
+ useForm: () => [
+ {
+ resetFields: vi.fn(),
+ validateFields: vi.fn(),
+ getFieldsValue: vi.fn(),
+ setFieldsValue: vi.fn(),
+ },
+ vi.fn(),
+ ],
+ Item: ({ children }: any) => {title}
+ {children}
+ {footer}
+ {children}
,
+ }),
+ Steps: Object.assign(
+ ({ children, current, className }: any) => (
+
+ {children}
+
+ ),
+ {
+ Step: ({ title }: any) => {title}
,
+ },
+ ),
+ Button: ({ children, onClick, disabled, loading, ...props }: any) => (
+
+ ),
+ Checkbox: ({ checked, indeterminate, onChange, children, disabled }: any) => (
+
+ ),
+}));
+
+// Mock @tremor/react components
+vi.mock("@tremor/react", () => ({
+ Text: ({ children, className }: any) => {children},
+ Title: ({ children }: any) => {children}
, + Badge: ({ children, color, size }: any) => ( + + {children} + + ), +})); + +describe("MakeAgentPublicForm", () => { + const mockProps = { + visible: true, + onClose: vi.fn(), + accessToken: "test-token", + agentHubData: [ + { + agent_id: "agent-1", + name: "Test Agent 1", + description: "Description 1", + version: "1.0", + is_public: false, + skills: [ + { id: "skill-1", name: "Skill 1", description: "Skill desc" }, + { id: "skill-2", name: "Skill 2", description: "Skill desc" }, + ], + protocolVersion: "1.0", + }, + { + agent_id: "agent-2", + name: "Test Agent 2", + description: "Description 2", + version: "2.0", + is_public: true, + skills: [], + protocolVersion: "1.0", + }, + ] as AgentHubData[], + onSuccess: vi.fn(), + }; + + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(() => { + vi.resetAllMocks(); + }); + + it("should render the component", () => { + render(
+
+ ) : null,
+ Form: Object.assign(({ children, form }: any) => , {
+ useForm: () => [
+ {
+ resetFields: vi.fn(),
+ validateFields: vi.fn(),
+ getFieldsValue: vi.fn(),
+ setFieldsValue: vi.fn(),
+ },
+ vi.fn(),
+ ],
+ Item: ({ children }: any) => {title}
+ {children}
+ {footer}
+ {children}
,
+ }),
+ Steps: Object.assign(
+ ({ children, current, className }: any) => (
+
+ {children}
+
+ ),
+ {
+ Step: ({ title }: any) => {title}
,
+ },
+ ),
+ Button: ({ children, onClick, disabled, loading, ...props }: any) => (
+
+ ),
+ Checkbox: ({ checked, indeterminate, onChange, children, disabled }: any) => (
+
+ ),
+}));
+
+// Additional @tremor/react mocks (Button is already mocked globally)
+vi.mock("@tremor/react", async (importOriginal) => {
+ const actual = await importOriginal{children}
, + Badge: ({ children, color, size }: any) => ( + + {children} + + ), + }; +}); + +describe("MakeMCPPublicForm", () => { + const mockProps = { + visible: true, + onClose: vi.fn(), + accessToken: "test-token", + mcpHubData: [ + { + server_id: "server-1", + server_name: "Test Server 1", + description: "Description 1", + url: "http://example.com/server1", + transport: "http", + status: "active", + mcp_info: { is_public: false }, + allowed_tools: ["tool-1", "tool-2"], + auth_type: "bearer", + credentials: {}, + created_at: "2024-01-01T00:00:00Z", + created_by: "user1", + updated_at: "2024-01-01T00:00:00Z", + updated_by: "user1", + teams: [], + mcp_access_groups: [], + extra_headers: [], + static_headers: {}, + args: [], + env: {}, + }, + { + server_id: "server-2", + server_name: "Test Server 2", + description: "Description 2", + url: "http://example.com/server2", + transport: "websocket", + status: "inactive", + mcp_info: { is_public: true }, + allowed_tools: [], + auth_type: "none", + credentials: {}, + created_at: "2024-01-01T00:00:00Z", + created_by: "user2", + updated_at: "2024-01-01T00:00:00Z", + updated_by: "user2", + teams: [], + mcp_access_groups: [], + extra_headers: [], + static_headers: {}, + args: [], + env: {}, + }, + ] as MCPServerData[], + onSuccess: vi.fn(), + }; + + beforeEach(() => { + vi.clearAllMocks(); + }); + + afterEach(() => { + vi.resetAllMocks(); + }); + + it("should render the component", () => { + render(
+
+ ) : null,
+ Form: Object.assign(({ children, form }: any) => , {
+ useForm: () => [
+ {
+ resetFields: vi.fn(),
+ validateFields: vi.fn(),
+ getFieldsValue: vi.fn(),
+ setFieldsValue: vi.fn(),
+ },
+ vi.fn(),
+ ],
+ Item: ({ children }: any) => {title}
+ {children}
+ {footer}
+ {children}
,
+ }),
+ Steps: Object.assign(
+ ({ children, current, className }: any) => (
+
+ {children}
+
+ ),
+ {
+ Step: ({ title }: any) => {title}
,
+ },
+ ),
+ Button: ({ children, onClick, disabled, loading, ...props }: any) => (
+
+ ),
+ Checkbox: ({ checked, indeterminate, onChange, children, disabled }: any) => (
+
+ ),
+}));
+
+// Mock @tremor/react components
+vi.mock("@tremor/react", () => ({
+ Text: ({ children, className }: any) => {children},
+ Title: ({ children }: any) => {children}
, + Badge: ({ children, color, size }: any) => ( + + {children} + + ), +})); + +// Mock ModelFilters component +vi.mock("../../model_filters", () => ({ + default: ({ onFilteredDataChange, modelHubData }: any) => ( +
+
+
+ ),
+}));
+
+// Mock NotificationsManager
+vi.mock("../../molecules/notifications_manager", () => ({
+ default: {
+ fromBackend: vi.fn(),
+ success: vi.fn(),
+ },
+}));
+
+describe("MakeModelPublicForm", () => {
+ const mockProps = {
+ visible: true,
+ onClose: vi.fn(),
+ accessToken: "test-token",
+ modelHubData: [
+ {
+ model_group: "gpt-4",
+ providers: ["openai"],
+ max_input_tokens: 8192,
+ max_output_tokens: 4096,
+ input_cost_per_token: 0.03,
+ output_cost_per_token: 0.06,
+ mode: "chat",
+ tpm: 10000,
+ rpm: 200,
+ supports_parallel_function_calling: true,
+ supports_vision: false,
+ supports_function_calling: true,
+ supported_openai_params: ["temperature", "max_tokens"],
+ is_public_model_group: false,
+ },
+ {
+ model_group: "gpt-3.5-turbo",
+ providers: ["openai"],
+ max_input_tokens: 4096,
+ max_output_tokens: 2048,
+ input_cost_per_token: 0.0015,
+ output_cost_per_token: 0.002,
+ mode: "chat",
+ tpm: 60000,
+ rpm: 3500,
+ supports_parallel_function_calling: false,
+ supports_vision: false,
+ supports_function_calling: true,
+ supported_openai_params: ["temperature", "max_tokens"],
+ is_public_model_group: true,
+ },
+ ] as ModelGroupInfo[],
+ onSuccess: vi.fn(),
+ };
+
+ beforeEach(() => {
+ vi.clearAllMocks();
+ });
+
+ afterEach(() => {
+ vi.resetAllMocks();
+ });
+
+ it("should render the component", () => {
+ render(
+
+ Provider
+
+
+
+ }
+ rules={[{ required: true, message: "Please select a provider" }]}
+ >
+
+ String(option?.label ?? "").toLowerCase().includes(input.toLowerCase())
+ }
+ >
+
+
+ {Object.entries(Providers).map(([providerEnum, providerDisplayName]) => {
+ const providerValue = provider_map[providerEnum as keyof typeof provider_map];
+ // Only show providers that don't already have a margin configured
+ if (providerValue && marginConfig[providerValue]) {
+ return null;
+ }
+ return (
+
+
+ );
+ })}
+
+
+
+
+ Margin Type
+
+
+
+ }
+ rules={[{ required: true, message: "Please select a margin type" }]}
+ >
+ onMarginTypeChange(e.target.value)}
+ className="w-full"
+ >
+ Percentage-based
+ Fixed Amount
+
+
+
+ {marginType === "percentage" && (
+
+ Margin Percentage
+
+
+
+ }
+ rules={[
+ { required: true, message: "Please enter a margin percentage" },
+ {
+ validator: (_, value) => {
+ if (!value) {
+ return Promise.reject(new Error("Please enter a margin percentage"));
+ }
+ const numValue = parseFloat(value);
+ if (isNaN(numValue) || numValue < 0 || numValue > 1000) {
+ return Promise.reject(new Error("Percentage must be between 0 and 1000"));
+ }
+ return Promise.resolve();
+ },
+ },
+ ]}
+ >
+
+ )}
+
+ {marginType === "fixed" && (
+
+ Fixed Margin Amount
+
+
+
+ }
+ rules={[
+ { required: true, message: "Please enter a fixed amount" },
+ {
+ validator: (_, value) => {
+ if (!value) {
+ return Promise.reject(new Error("Please enter a fixed amount"));
+ }
+ const numValue = parseFloat(value);
+ if (isNaN(numValue) || numValue < 0) {
+ return Promise.reject(new Error("Fixed amount must be non-negative"));
+ }
+ return Promise.resolve();
+ },
+ },
+ ]}
+ >
+
+ )}
+
+
+ );
+};
+
+export default AddMarginForm;
+
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
index 2d530be71eb..7f79a6848bb 100644
--- a/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
@@ -2,7 +2,6 @@ import React from "react";
import { TextInput, Button } from "@tremor/react";
import { Select as AntdSelect, Form, Tooltip } from "antd";
import { InfoCircleOutlined } from "@ant-design/icons";
-import Image from "next/image";
import { Providers, provider_map, providerLogoMap } from "../provider_info_helpers";
import { DiscountConfig } from "./types";
import { handleImageError } from "./provider_display_helpers";
@@ -58,11 +57,9 @@ const AddProviderForm: React.FC
+ Global (All Providers)
+
+
+ 
handleImageError(e, providerDisplayName)}
+ />
+ {providerDisplayName}
+
+
+
+
+ $
+
+
+
+
+
- handleImageError(e, providerDisplayName)}
/>
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
index c356982f189..3b9ea30e128 100644
--- a/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
@@ -1,16 +1,18 @@
-import React, { useState, useEffect, useCallback } from "react";
-import { Title, Text, Button, TabGroup, TabList, Tab, TabPanels, TabPanel } from "@tremor/react";
+import React, { useState, useEffect } from "react";
+import { Title, Text, Button, Accordion, AccordionHeader, AccordionBody, TabGroup, TabList, Tab, TabPanels, TabPanel } from "@tremor/react";
import { Modal, Form } from "antd";
-import { getProxyBaseUrl } from "@/components/networking";
-import NotificationsManager from "../molecules/notifications_manager";
-import { Providers } from "../provider_info_helpers";
-import { CostTrackingSettingsProps, DiscountConfig } from "./types";
-import { getProviderBackendValue } from "./provider_display_helpers";
+import { CostTrackingSettingsProps } from "./types";
import ProviderDiscountTable from "./provider_discount_table";
import AddProviderForm from "./add_provider_form";
+import ProviderMarginTable from "./provider_margin_table";
+import AddMarginForm from "./add_margin_form";
+import PricingCalculator from "./pricing_calculator/index";
import { ExclamationCircleOutlined } from "@ant-design/icons";
import { DocsMenu } from "../HelpLink";
import HowItWorks from "./how_it_works";
+import { useDiscountConfig } from "./use_discount_config";
+import { useMarginConfig } from "./use_margin_config";
+import { fetchAvailableModels, ModelGroup } from "../playground/llm_calls/fetch_models";
const DOCS_LINKS = [
{ label: "Custom pricing for models", href: "https://docs.litellm.ai/docs/proxy/custom_pricing" },
@@ -22,118 +24,65 @@ const CostTrackingSettings: React.FC = ({
userRole,
accessToken
}) => {
- const [discountConfig, setDiscountConfig] = useState({});
const [selectedProvider, setSelectedProvider] = useState(undefined);
const [newDiscount, setNewDiscount] = useState("");
const [isFetching, setIsFetching] = useState(true);
const [isModalVisible, setIsModalVisible] = useState(false);
+ const [isMarginModalVisible, setIsMarginModalVisible] = useState(false);
+ const [selectedMarginProvider, setSelectedMarginProvider] = useState(undefined);
+ const [marginType, setMarginType] = useState<"percentage" | "fixed">("percentage");
+ const [percentageValue, setPercentageValue] = useState("");
+ const [fixedAmountValue, setFixedAmountValue] = useState("");
+ const [models, setModels] = useState([]);
const [form] = Form.useForm();
+ const [marginForm] = Form.useForm();
const [modal, contextHolder] = Modal.useModal();
+
+ const isProxyAdmin = userRole === "proxy_admin" || userRole === "Admin";
- const fetchDiscountConfig = useCallback(async () => {
- setIsFetching(true);
- try {
- const proxyBaseUrl = getProxyBaseUrl();
- const url = proxyBaseUrl
- ? `${proxyBaseUrl}/config/cost_discount_config`
- : "/config/cost_discount_config";
-
- const response = await fetch(url, {
- method: "GET",
- headers: {
- Authorization: `Bearer ${accessToken}`,
- "Content-Type": "application/json",
- },
- });
+ // Use custom hooks for discount and margin config
+ const {
+ discountConfig,
+ fetchDiscountConfig,
+ handleAddProvider: addProvider,
+ handleRemoveProvider: removeProvider,
+ handleDiscountChange,
+ } = useDiscountConfig({ accessToken });
- if (response.ok) {
- const data = await response.json();
- setDiscountConfig(data.values || {});
- } else {
- console.error("Failed to fetch discount config");
- }
- } catch (error) {
- console.error("Error fetching discount config:", error);
- NotificationsManager.fromBackend("Failed to fetch discount configuration");
- } finally {
- setIsFetching(false);
- }
- }, [accessToken]);
+ const {
+ marginConfig,
+ fetchMarginConfig,
+ handleAddMargin: addMargin,
+ handleRemoveMargin: removeMargin,
+ handleMarginChange,
+ } = useMarginConfig({ accessToken });
useEffect(() => {
if (accessToken) {
- fetchDiscountConfig();
- }
- }, [accessToken, fetchDiscountConfig]);
-
- const saveDiscountConfig = async (config: DiscountConfig) => {
- try {
- const proxyBaseUrl = getProxyBaseUrl();
- const url = proxyBaseUrl
- ? `${proxyBaseUrl}/config/cost_discount_config`
- : "/config/cost_discount_config";
-
- const response = await fetch(url, {
- method: "PATCH",
- headers: {
- Authorization: `Bearer ${accessToken}`,
- "Content-Type": "application/json",
- },
- body: JSON.stringify(config),
+ Promise.all([fetchDiscountConfig(), fetchMarginConfig()]).finally(() => {
+ setIsFetching(false);
});
-
- if (response.ok) {
- NotificationsManager.success("Discount configuration updated successfully");
- await fetchDiscountConfig();
- } else {
- const errorData = await response.json();
- const errorMessage = errorData.detail?.error || errorData.detail || "Failed to update settings";
- NotificationsManager.fromBackend(errorMessage);
- }
- } catch (error) {
- console.error("Error updating discount config:", error);
- NotificationsManager.fromBackend("Failed to update discount configuration");
+
+ // Fetch models for pricing calculator (available to all roles)
+ const loadModels = async () => {
+ try {
+ const modelGroups = await fetchAvailableModels(accessToken);
+ setModels(modelGroups.map((m: ModelGroup) => m.model_group));
+ } catch (error) {
+ console.error("Error fetching models:", error);
+ }
+ };
+ loadModels();
}
- };
+ }, [accessToken, fetchDiscountConfig, fetchMarginConfig]);
const handleAddProvider = async () => {
- if (!selectedProvider || !newDiscount) {
- NotificationsManager.fromBackend("Please select a provider and enter discount percentage");
- return;
+ const success = await addProvider(selectedProvider, newDiscount);
+ if (success) {
+ setSelectedProvider(undefined);
+ setNewDiscount("");
+ setIsModalVisible(false);
}
-
- const percentageValue = parseFloat(newDiscount);
- if (isNaN(percentageValue) || percentageValue < 0 || percentageValue > 100) {
- NotificationsManager.fromBackend("Discount must be between 0% and 100%");
- return;
- }
-
- const providerValue = getProviderBackendValue(selectedProvider);
-
- if (!providerValue) {
- NotificationsManager.fromBackend("Invalid provider selected");
- return;
- }
-
- if (discountConfig[providerValue]) {
- NotificationsManager.fromBackend(
- `Discount for ${Providers[selectedProvider as keyof typeof Providers]} already exists. Edit it in the table above.`
- );
- return;
- }
-
- // Convert percentage to decimal for storage
- const discountValue = percentageValue / 100;
- const updatedConfig = {
- ...discountConfig,
- [providerValue]: discountValue,
- };
-
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
- setSelectedProvider(undefined);
- setNewDiscount("");
- setIsModalVisible(false);
};
const handleModalCancel = () => {
@@ -143,7 +92,7 @@ const CostTrackingSettings: React.FC = ({
setNewDiscount("");
};
- const handleFormSubmit = (values: any) => {
+ const handleFormSubmit = () => {
handleAddProvider();
};
@@ -155,27 +104,47 @@ const CostTrackingSettings: React.FC = ({
okText: 'Remove',
okType: 'danger',
cancelText: 'Cancel',
- onOk: async () => {
- const updatedConfig = { ...discountConfig };
- delete updatedConfig[provider];
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
- },
+ onOk: () => removeProvider(provider),
});
};
- const handleDiscountChange = async (provider: string, value: string) => {
- const discountValue = parseFloat(value);
- if (!isNaN(discountValue) && discountValue >= 0 && discountValue <= 1) {
- const updatedConfig = {
- ...discountConfig,
- [provider]: discountValue,
- };
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
+ const handleAddMargin = async () => {
+ const success = await addMargin({
+ selectedProvider: selectedMarginProvider,
+ marginType,
+ percentageValue,
+ fixedAmountValue,
+ });
+ if (success) {
+ setSelectedMarginProvider(undefined);
+ setPercentageValue("");
+ setFixedAmountValue("");
+ setMarginType("percentage");
+ setIsMarginModalVisible(false);
}
};
+ const handleMarginModalCancel = () => {
+ setIsMarginModalVisible(false);
+ marginForm.resetFields();
+ setSelectedMarginProvider(undefined);
+ setPercentageValue("");
+ setFixedAmountValue("");
+ setMarginType("percentage");
+ };
+
+ const handleRemoveMargin = async (provider: string, providerDisplayName: string) => {
+ modal.confirm({
+ title: 'Remove Provider Margin',
+ icon: = ({
-
-
- Provider Discounts
- Test It
-
-
-
- {isFetching ? (
-
-
-
+
+ }
+ open={isMarginModalVisible}
+ width={1000}
+ onCancel={handleMarginModalCancel}
+ footer={null}
+ className="top-8"
+ styles={{
+ body: { padding: "24px" },
+ header: { padding: "24px 24px 0 24px", border: "none" },
+ }}
+ >
+
- Loading configuration...
-
- ) : Object.keys(discountConfig).length > 0 ? (
-
-
- ) : (
-
-
-
- No provider discounts configured
-
-
- Click "Add Provider Discount" to get started
-
-
- )}
-
- = ({
+
+
+ {/* Accordion 1: Provider Discounts - Only for proxy admins */}
+ {isProxyAdmin && (
+
+
+
+
+
+
+ Discounts
+ Test It
+
+
+
+
+
+
+
+
+
+
+ )}
+
+ {/* Accordion 2: Fee/Price Margin - Only for proxy admins */}
+ {isProxyAdmin && (
+
+
+
+
+
+
+ )}
+
+ {/* Accordion 3: Pricing Calculator - Available to all roles */}
+
+
+
+
+
+
+ Provider Discounts
+
+ Apply percentage-based discounts to reduce costs for specific providers
+
-
-
-
+
+
+
+
+
+ {isFetching ? (
+
+ Loading configuration...
+
+ ) : Object.keys(discountConfig).length > 0 ? (
+
+
+
+ No provider discounts configured
+
+
+ Click "Add Provider Discount" to get started
+
+
+ )}
+
+
+
+ Fee/Price Margin
+
+ Add fees or margins to LLM costs for internal billing and cost recovery
+
+
+
+
+
+
+
+ {isFetching ? (
+
+ Loading configuration...
+
+ ) : Object.keys(marginConfig).length > 0 ? (
+
+
+
+ No provider margins configured
+
+
+ Click "Add Provider Margin" to get started
+
+
+ )}
+
+ Pricing Calculator
+
+ Estimate LLM costs based on expected token usage and request volume
+
+
+
+
+ Add Provider Margin
+
+
+ Select a provider (or "Global" for all providers) and configure the margin. You can use percentage-based or fixed amount.
+
+
+
+
);
};
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/index.ts b/ui/litellm-dashboard/src/components/CostTrackingSettings/index.ts
index 11adc414664..feba943154b 100644
--- a/ui/litellm-dashboard/src/components/CostTrackingSettings/index.ts
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/index.ts
@@ -1,8 +1,12 @@
export { default as CostTrackingSettings } from "./cost_tracking_settings";
export { default as ProviderDiscountTable } from "./provider_discount_table";
export { default as AddProviderForm } from "./add_provider_form";
+export { default as ProviderMarginTable } from "./provider_margin_table";
+export { default as AddMarginForm } from "./add_margin_form";
export { default as HowItWorks } from "./how_it_works";
-export type { CostTrackingSettingsProps, DiscountConfig, CostDiscountResponse } from "./types";
+export type { CostTrackingSettingsProps, DiscountConfig, CostDiscountResponse, MarginConfig, CostMarginResponse } from "./types";
export type { ProviderDisplayInfo } from "./provider_display_helpers";
export * from "./provider_display_helpers";
+export { useDiscountConfig } from "./use_discount_config";
+export { useMarginConfig } from "./use_margin_config";
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/cost_results.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/cost_results.tsx
new file mode 100644
index 00000000000..03d5ca0e518
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/cost_results.tsx
@@ -0,0 +1,203 @@
+import React from "react";
+import { Text } from "@tremor/react";
+import { Card, Statistic, Row, Col, Divider, Spin } from "antd";
+import { DollarOutlined, LoadingOutlined } from "@ant-design/icons";
+import { CostEstimateResponse } from "../types";
+import { formatNumberWithCommas } from "@/utils/dataUtils";
+import ExportDropdown from "./export_dropdown";
+
+interface CostResultsProps {
+ result: CostEstimateResponse | null;
+ loading: boolean;
+}
+
+const formatCost = (value: number | null | undefined): string => {
+ if (value === null || value === undefined) return "-";
+ if (value === 0) return "$0";
+ if (value < 0.0001) return `$${value.toExponential(2)}`;
+ if (value < 1) return `$${value.toFixed(4)}`;
+ return `$${formatNumberWithCommas(value, 2, true)}`;
+};
+
+const formatRequests = (value: number | null | undefined): string => {
+ if (value === null || value === undefined) return "-";
+ return formatNumberWithCommas(value, 0, true);
+};
+
+const CostResults: React.FC
+
+ Select a model to see cost estimates
+
+
+ );
+ }
+
+ if (loading && !result) {
+ return (
+
+ Calculating costs...
+
+ );
+ }
+
+ if (!result) return null;
+
+ return (
+
+
+
+ 0 ? "#faad14" : undefined,
+ }}
+ />
+
+
+
+
+ {result.daily_cost !== null && (
+
+
+ 0 ? "#faad14" : undefined,
+ }}
+ />
+
+
+
+ )}
+
+ {result.monthly_cost !== null && (
+
+
+ 0 ? "#faad14" : undefined,
+ }}
+ />
+
+
+
+ )}
+
+ {(result.input_cost_per_token || result.output_cost_per_token) && (
+
+ );
+};
+
+export default CostResults;
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_dropdown.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_dropdown.tsx
new file mode 100644
index 00000000000..e8a681021d6
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_dropdown.tsx
@@ -0,0 +1,71 @@
+import React, { useState, useRef, useEffect } from "react";
+import { Button } from "@tremor/react";
+import { DownloadOutlined, FilePdfOutlined, FileExcelOutlined } from "@ant-design/icons";
+import { CostEstimateResponse } from "../types";
+import { exportToPDF, exportToCSV } from "./export_utils";
+
+interface ExportDropdownProps {
+ result: CostEstimateResponse;
+}
+
+const ExportDropdown: React.FC
+
+
+
+ Cost Estimate
+
+ Model: {result.model} {result.provider && `(${result.provider})`}
+
+
+
+ {loading &&
+
+ Token Pricing:
+ {result.input_cost_per_token && (
+ Input: ${formatNumberWithCommas(result.input_cost_per_token * 1_000_000, 2)}/1M tokens
+ )}
+ {result.input_cost_per_token && result.output_cost_per_token && " | "}
+ {result.output_cost_per_token && (
+ Output: ${formatNumberWithCommas(result.output_cost_per_token * 1_000_000, 2)}/1M tokens
+ )}
+
+ )}
+
+
+
+ {isOpen && (
+
+ );
+};
+
+export default ExportDropdown;
+
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_utils.ts b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_utils.ts
new file mode 100644
index 00000000000..a205efa3bf8
--- /dev/null
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/pricing_calculator/export_utils.ts
@@ -0,0 +1,276 @@
+import { CostEstimateResponse } from "../types";
+import { formatNumberWithCommas } from "@/utils/dataUtils";
+
+const formatCostForExport = (value: number | null | undefined): string => {
+ if (value === null || value === undefined) return "-";
+ if (value === 0) return "$0.00";
+ if (value < 0.01) return `$${value.toFixed(6)}`;
+ if (value < 1) return `$${value.toFixed(4)}`;
+ return `$${formatNumberWithCommas(value, 2)}`;
+};
+
+const formatRequestsForExport = (value: number | null | undefined): string => {
+ if (value === null || value === undefined) return "-";
+ return formatNumberWithCommas(value, 0);
+};
+
+export const exportToPDF = (result: CostEstimateResponse): void => {
+ const printWindow = window.open("", "_blank");
+ if (!printWindow) {
+ alert("Please allow popups to export PDF");
+ return;
+ }
+
+ const html = `
+
+
+
+
+
+
+
+ )}
+ 🚅 LiteLLM Cost Estimate Report
+ + + +Per-Request Cost Breakdown
+| Cost Type | +Amount | +
|---|---|
| Input Cost | +${formatCostForExport(result.input_cost_per_request)} | +
| Output Cost | +${formatCostForExport(result.output_cost_per_request)} | +
| Margin/Fee | +${formatCostForExport(result.margin_cost_per_request)} | +
| Total per Request | +${formatCostForExport(result.cost_per_request)} | +
Daily Cost Estimate (${formatRequestsForExport(result.num_requests_per_day)} requests/day)
+| Cost Type | +Amount | +
|---|---|
| Input Cost | +${formatCostForExport(result.daily_input_cost)} | +
| Output Cost | +${formatCostForExport(result.daily_output_cost)} | +
| Margin/Fee | +${formatCostForExport(result.daily_margin_cost)} | +
| Total Daily | +${formatCostForExport(result.daily_cost)} | +
Monthly Cost Estimate (${formatRequestsForExport(result.num_requests_per_month)} requests/month)
+| Cost Type | +Amount | +
|---|---|
| Input Cost | +${formatCostForExport(result.monthly_input_cost)} | +
| Output Cost | +${formatCostForExport(result.monthly_output_cost)} | +
| Margin/Fee | +${formatCostForExport(result.monthly_margin_cost)} | +
| Total Monthly | +${formatCostForExport(result.monthly_cost)} | +
Token Pricing
+| Token Type | +Price per 1M Tokens | +
|---|---|
| Input Tokens | +$${(result.input_cost_per_token * 1000000).toFixed(2)} | +
| Output Tokens | +$${(result.output_cost_per_token * 1000000).toFixed(2)} | +