+ +## What is NanoGPT? + +NanoGPT is a flexible AI API service that offers: +- **Pay-Per-Prompt Pricing**: No subscriptions, pay only for what you use +- **200+ AI Models**: Access to text, image, and video generation models +- **No Registration Required**: Get started instantly +- **OpenAI-Compatible API**: Easy integration with existing code +- **Streaming Support**: Real-time response streaming +- **Tool Calling**: Support for function calling + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key +``` + +Get your NanoGPT API key from [nano-gpt.com](https://nano-gpt.com). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="NanoGPT Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# NanoGPT call +response = completion( + model="nano-gpt/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="NanoGPT Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["NANOGPT_API_KEY"] = "" # your NanoGPT API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# NanoGPT call with streaming +response = completion( + model="nano-gpt/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +### Tool Calling + +```python showLineNumbers title="NanoGPT Tool Calling" +import os +import litellm + +os.environ["NANOGPT_API_KEY"] = "" + +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"} + } + } + } + } +] + +response = litellm.completion( + model="nano-gpt/model-name", + messages=[{"role": "user", "content": "What's the weather in Paris?"}], + tools=tools +) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export NANOGPT_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: nano-gpt-model + litellm_params: + model: nano-gpt/model-name # Replace with actual model name + api_key: os.environ/NANOGPT_API_KEY +``` + +## Supported OpenAI Parameters + +NanoGPT supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID from 200+ available models | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `n` | integer | Optional. Number of completions to generate | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | +| `response_format` | object | Optional. Response format specification | +| `user` | string | Optional. User identifier | + +## Model Categories + +NanoGPT provides access to multiple model categories: +- **Text Generation**: 200+ LLMs for chat, completion, and analysis +- **Image Generation**: AI models for creating images +- **Video Generation**: AI models for video creation +- **Embedding Models**: Text embedding models for vector search + +## Pricing Model + +NanoGPT offers a flexible pricing structure: +- **Pay-Per-Prompt**: No subscription required +- **No Registration**: Get started immediately +- **Transparent Pricing**: Pay only for what you use + +## API Documentation + +For detailed API documentation, visit [docs.nano-gpt.com](https://docs.nano-gpt.com). + +## Additional Resources + +- [NanoGPT Website](https://nano-gpt.com) +- [NanoGPT API Documentation](https://nano-gpt.com/api) +- [NanoGPT Model List](https://docs.nano-gpt.com/api-reference/endpoint/models) diff --git a/docs/my-website/docs/providers/openai.md b/docs/my-website/docs/providers/openai.md index 509a106d8a4..80645a51ac5 100644 --- a/docs/my-website/docs/providers/openai.md +++ b/docs/my-website/docs/providers/openai.md @@ -495,7 +495,7 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \ |-------|----------------------|------------------| | `gpt-5.1` | `none` | `none`, `low`, `medium`, `high` | | `gpt-5` | `medium` | `minimal`, `low`, `medium`, `high` | -| `gpt-5-mini` | `medium` | `none`, `minimal`, `low`, `medium`, `high` | +| `gpt-5-mini` | `medium` | `minimal`, `low`, `medium`, `high` | | `gpt-5-nano` | `none` | `none`, `low`, `medium`, `high` | | `gpt-5-codex` | `adaptive` | `low`, `medium`, `high` (no `minimal`) | | `gpt-5.1-codex` | `adaptive` | `low`, `medium`, `high` (no `minimal`) | diff --git a/docs/my-website/docs/providers/poe.md b/docs/my-website/docs/providers/poe.md new file mode 100644 index 00000000000..ba4089ae6a4 --- /dev/null +++ b/docs/my-website/docs/providers/poe.md @@ -0,0 +1,139 @@ +# Poe + +## Overview + +| Property | Details | +|-------|-------| +| Description | Poe is Quora's AI platform that provides access to more than 100 models across text, image, video, and voice modalities through a developer-friendly API. | +| Provider Route on LiteLLM | `poe/` | +| Link to Provider Doc | [Poe Website ↗](https://poe.com) | +| Base URL | `https://api.poe.com/v1` | +| Supported Operations | [`/chat/completions`](#sample-usage) | + +
+ +## What is Poe? + +Poe is Quora's comprehensive AI platform that offers: +- **100+ Models**: Access to a wide variety of AI models +- **Multiple Modalities**: Text, image, video, and voice AI +- **Popular Models**: Including OpenAI's GPT series and Anthropic's Claude +- **Developer API**: Easy integration for applications +- **Extensive Reach**: Benefits from Quora's 400M monthly unique visitors + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["POE_API_KEY"] = "" # your Poe API key +``` + +Get your Poe API key from the [Poe platform](https://poe.com). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="Poe Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["POE_API_KEY"] = "" # your Poe API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# Poe call +response = completion( + model="poe/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="Poe Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["POE_API_KEY"] = "" # your Poe API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# Poe call with streaming +response = completion( + model="poe/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export POE_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: poe-model + litellm_params: + model: poe/model-name # Replace with actual model name + api_key: os.environ/POE_API_KEY +``` + +## Supported OpenAI Parameters + +Poe supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID from 100+ available models | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | +| `tools` | array | Optional. List of available tools/functions | +| `tool_choice` | string/object | Optional. Control tool/function calling | +| `response_format` | object | Optional. Response format specification | +| `user` | string | Optional. User identifier | + +## Available Model Categories + +Poe provides access to models across multiple providers: +- **OpenAI Models**: Including GPT-4, GPT-4 Turbo, GPT-3.5 Turbo +- **Anthropic Models**: Including Claude 3 Opus, Sonnet, Haiku +- **Other Popular Models**: Various provider models available +- **Multi-Modal**: Text, image, video, and voice models + +## Platform Benefits + +Using Poe through LiteLLM offers several advantages: +- **Unified Access**: Single API for many different models +- **Quora Integration**: Access to large user base and content ecosystem +- **Content Sharing**: Capabilities to share model outputs with followers +- **Content Distribution**: Best AI content distributed to all users +- **Model Discovery**: Efficient way to explore new AI models + +## Developer Resources + +Poe is actively building developer features and welcomes early access requests for API integration. + +## Additional Resources + +- [Poe Website](https://poe.com) +- [Poe AI Quora Space](https://poeai.quora.com) +- [Quora Blog Post about Poe](https://quorablog.quora.com/Poe) diff --git a/docs/my-website/docs/providers/synthetic.md b/docs/my-website/docs/providers/synthetic.md new file mode 100644 index 00000000000..b3ba3d0a9e7 --- /dev/null +++ b/docs/my-website/docs/providers/synthetic.md @@ -0,0 +1,119 @@ +# Synthetic + +## Overview + +| Property | Details | +|-------|-------| +| Description | Synthetic runs open-source AI models in secure datacenters within the US and EU, with a focus on privacy. They never train on your data and auto-delete API data within 14 days. | +| Provider Route on LiteLLM | `synthetic/` | +| Link to Provider Doc | [Synthetic Website ↗](https://synthetic.new) | +| Base URL | `https://api.synthetic.new/openai/v1` | +| Supported Operations | [`/chat/completions`](#sample-usage) | + +
+ +## What is Synthetic? + +Synthetic is a privacy-focused AI platform that provides access to open-source LLMs with the following guarantees: +- **Privacy-First**: Data never used for training +- **Secure Hosting**: Models run in secure datacenters in US and EU +- **Auto-Deletion**: API data automatically deleted within 14 days +- **Open Source**: Runs open-source AI models + +## Required Variables + +```python showLineNumbers title="Environment Variables" +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key +``` + +Get your Synthetic API key from [synthetic.new](https://synthetic.new). + +## Usage - LiteLLM Python SDK + +### Non-streaming + +```python showLineNumbers title="Synthetic Non-streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key + +messages = [{"content": "What is the capital of France?", "role": "user"}] + +# Synthetic call +response = completion( + model="synthetic/model-name", # Replace with actual model name + messages=messages +) + +print(response) +``` + +### Streaming + +```python showLineNumbers title="Synthetic Streaming Completion" +import os +import litellm +from litellm import completion + +os.environ["SYNTHETIC_API_KEY"] = "" # your Synthetic API key + +messages = [{"content": "Write a short poem about AI", "role": "user"}] + +# Synthetic call with streaming +response = completion( + model="synthetic/model-name", # Replace with actual model name + messages=messages, + stream=True +) + +for chunk in response: + print(chunk) +``` + +## Usage - LiteLLM Proxy Server + +### 1. Save key in your environment + +```bash +export SYNTHETIC_API_KEY="" +``` + +### 2. Start the proxy + +```yaml +model_list: + - model_name: synthetic-model + litellm_params: + model: synthetic/model-name # Replace with actual model name + api_key: os.environ/SYNTHETIC_API_KEY +``` + +## Supported OpenAI Parameters + +Synthetic supports all standard OpenAI-compatible parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `messages` | array | **Required**. Array of message objects with 'role' and 'content' | +| `model` | string | **Required**. Model ID | +| `stream` | boolean | Optional. Enable streaming responses | +| `temperature` | float | Optional. Sampling temperature | +| `top_p` | float | Optional. Nucleus sampling parameter | +| `max_tokens` | integer | Optional. Maximum tokens to generate | +| `frequency_penalty` | float | Optional. Penalize frequent tokens | +| `presence_penalty` | float | Optional. Penalize tokens based on presence | +| `stop` | string/array | Optional. Stop sequences | + +## Privacy & Security + +Synthetic provides enterprise-grade privacy protections: +- Data auto-deleted within 14 days +- No data used for model training +- Secure hosting in US and EU datacenters +- Compliance-friendly architecture + +## Additional Resources + +- [Synthetic Website](https://synthetic.new) diff --git a/docs/my-website/docs/proxy/alerting.md b/docs/my-website/docs/proxy/alerting.md index 4cbcd0cffce..38d6d47be44 100644 --- a/docs/my-website/docs/proxy/alerting.md +++ b/docs/my-website/docs/proxy/alerting.md @@ -215,16 +215,16 @@ general_settings: alerting: ["slack"] alerting_threshold: 0.0001 # (Seconds) set an artificially low threshold for testing alerting alert_to_webhook_url: { - "llm_exceptions": "https://hooks.slack.com/services/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", - "llm_too_slow": "https://hooks.slack.com/services/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", - "llm_requests_hanging": "https://hooks.slack.com/services/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", - "budget_alerts": "https://hooks.slack.com/services/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", - "db_exceptions": "https://hooks.slack.com/services/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", - "daily_reports": "https://hooks.slack.com/services/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", - "spend_reports": "https://hooks.slack.com/services/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", - "cooldown_deployment": "https://hooks.slack.com/services/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", - "new_model_added": "https://hooks.slack.com/services/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", - "outage_alerts": "https://hooks.slack.com/services/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", + "llm_exceptions": "example-slack-webhook-url", + "llm_too_slow": "example-slack-webhook-url", + "llm_requests_hanging": "example-slack-webhook-url", + "budget_alerts": "example-slack-webhook-url", + "db_exceptions": "example-slack-webhook-url", + "daily_reports": "example-slack-webhook-url", + "spend_reports": "example-slack-webhook-url", + "cooldown_deployment": "example-slack-webhook-url", + "new_model_added": "example-slack-webhook-url", + "outage_alerts": "example-slack-webhook-url", } litellm_settings: @@ -399,7 +399,7 @@ curl -X GET --location 'http://0.0.0.0:4000/health/services?service=webhook' \ { "spend": 1, # the spend for the 'event_group' "max_budget": 0, # the 'max_budget' set for the 'event_group' - "token": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b", + "token": "example-api-key-123", "user_id": "default_user_id", "team_id": null, "user_email": null, diff --git a/docs/my-website/docs/proxy/call_hooks.md b/docs/my-website/docs/proxy/call_hooks.md index fa420009cf1..fe865f67e09 100644 --- a/docs/my-website/docs/proxy/call_hooks.md +++ b/docs/my-website/docs/proxy/call_hooks.md @@ -17,6 +17,7 @@ import Image from '@theme/IdealImage'; | `async_pre_call_hook` | Modify incoming request before it's sent to model | Before the LLM API call is made | | `async_moderation_hook` | Run checks on input in parallel to LLM API call | In parallel with the LLM API call | | `async_post_call_success_hook` | Modify outgoing response (non-streaming) | After successful LLM API call, for non-streaming responses | +| `async_post_call_failure_hook` | Transform error responses sent to clients | After failed LLM API call | | `async_post_call_streaming_hook` | Modify outgoing response (streaming) | After successful LLM API call, for streaming responses | See a complete example with our [parallel request rate limiter](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/hooks/parallel_request_limiter.py) @@ -60,7 +61,21 @@ class MyCustomHandler(CustomLogger): # https://docs.litellm.ai/docs/observabilit original_exception: Exception, user_api_key_dict: UserAPIKeyAuth, traceback_str: Optional[str] = None, - ): + ) -> Optional[HTTPException]: + """ + Transform error responses sent to clients. + + Return an HTTPException to replace the original error with a user-friendly message. + Return None to use the original exception. + + Example: + if isinstance(original_exception, litellm.ContextWindowExceededError): + return HTTPException( + status_code=400, + detail="Your prompt is too long. Please reduce the length and try again." + ) + return None # Use original exception + """ pass async def async_post_call_success_hook( @@ -339,3 +354,38 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \ "usage": {} } ``` + +## Advanced - Transform Error Responses + +Transform technical API errors into user-friendly messages using `async_post_call_failure_hook`. Return an `HTTPException` to replace the original error, or `None` to use the original exception. + +```python +from litellm.integrations.custom_logger import CustomLogger +from fastapi import HTTPException +from typing import Optional +import litellm + +class MyErrorTransformer(CustomLogger): + async def async_post_call_failure_hook( + self, + request_data: dict, + original_exception: Exception, + user_api_key_dict: UserAPIKeyAuth, + traceback_str: Optional[str] = None, + ) -> Optional[HTTPException]: + if isinstance(original_exception, litellm.ContextWindowExceededError): + return HTTPException( + status_code=400, + detail="Your prompt is too long. Please reduce the length and try again." + ) + if isinstance(original_exception, litellm.RateLimitError): + return HTTPException( + status_code=429, + detail="Rate limit exceeded. Please try again in a moment." + ) + return None # Use original exception + +proxy_handler_instance = MyErrorTransformer() +``` + +**Result:** Clients receive `"Your prompt is too long..."` instead of `"ContextWindowExceededError: Prompt exceeds context window"`. diff --git a/docs/my-website/docs/proxy/configs.md b/docs/my-website/docs/proxy/configs.md index ba4ca190aa9..bc2f6a13362 100644 --- a/docs/my-website/docs/proxy/configs.md +++ b/docs/my-website/docs/proxy/configs.md @@ -116,7 +116,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \ "role": "user", "content": "what llm are you" } - ], + ] } ' ``` diff --git a/docs/my-website/docs/proxy/cost_tracking.md b/docs/my-website/docs/proxy/cost_tracking.md index 019cd62c620..26a4920c093 100644 --- a/docs/my-website/docs/proxy/cost_tracking.md +++ b/docs/my-website/docs/proxy/cost_tracking.md @@ -722,7 +722,7 @@ curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end ```shell [ { - "api_key": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b", + "api_key": "example-api-key-123", "total_cost": 0.3201286305151999, "total_input_tokens": 36.0, "total_output_tokens": 1593.0, @@ -766,7 +766,7 @@ curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end ```shell [ { - "api_key": "88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b", + "api_key": "example-api-key-123", "total_cost": 0.00013132, "total_input_tokens": 105.0, "total_output_tokens": 872.0, @@ -1151,7 +1151,7 @@ curl -X GET "http://0.0.0.0:4000/spend/logs?request_id=
+
+
+
+
+ The Admin UI has been disabled by the administrator. To re-enable it, please update the following
+ environment variable:
+
+
+
+
+ }
+ type="warning"
+ showIcon
+ />
+
+
+
+ );
+ }
+
return (
+ 🚅 LiteLLM
+
+
+ DISABLE_ADMIN_UI=False
+
+
+ Provider
+
+
+
+ }
+ rules={[{ required: true, message: "Please select a provider" }]}
+ >
+
+ String(option?.label ?? "").toLowerCase().includes(input.toLowerCase())
+ }
+ >
+
+
+ {Object.entries(Providers).map(([providerEnum, providerDisplayName]) => {
+ const providerValue = provider_map[providerEnum as keyof typeof provider_map];
+ // Only show providers that don't already have a margin configured
+ if (providerValue && marginConfig[providerValue]) {
+ return null;
+ }
+ return (
+
+
+ );
+ })}
+
+
+
+
+ Margin Type
+
+
+
+ }
+ rules={[{ required: true, message: "Please select a margin type" }]}
+ >
+ onMarginTypeChange(e.target.value)}
+ className="w-full"
+ >
+ Percentage-based
+ Fixed Amount
+
+
+
+ {marginType === "percentage" && (
+
+ Margin Percentage
+
+
+
+ }
+ rules={[
+ { required: true, message: "Please enter a margin percentage" },
+ {
+ validator: (_, value) => {
+ if (!value) {
+ return Promise.reject(new Error("Please enter a margin percentage"));
+ }
+ const numValue = parseFloat(value);
+ if (isNaN(numValue) || numValue < 0 || numValue > 1000) {
+ return Promise.reject(new Error("Percentage must be between 0 and 1000"));
+ }
+ return Promise.resolve();
+ },
+ },
+ ]}
+ >
+
+ )}
+
+ {marginType === "fixed" && (
+
+ Fixed Margin Amount
+
+
+
+ }
+ rules={[
+ { required: true, message: "Please enter a fixed amount" },
+ {
+ validator: (_, value) => {
+ if (!value) {
+ return Promise.reject(new Error("Please enter a fixed amount"));
+ }
+ const numValue = parseFloat(value);
+ if (isNaN(numValue) || numValue < 0) {
+ return Promise.reject(new Error("Fixed amount must be non-negative"));
+ }
+ return Promise.resolve();
+ },
+ },
+ ]}
+ >
+
+ )}
+
+
+ );
+};
+
+export default AddMarginForm;
+
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
index 2d530be71eb..7f79a6848bb 100644
--- a/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/add_provider_form.tsx
@@ -2,7 +2,6 @@ import React from "react";
import { TextInput, Button } from "@tremor/react";
import { Select as AntdSelect, Form, Tooltip } from "antd";
import { InfoCircleOutlined } from "@ant-design/icons";
-import Image from "next/image";
import { Providers, provider_map, providerLogoMap } from "../provider_info_helpers";
import { DiscountConfig } from "./types";
import { handleImageError } from "./provider_display_helpers";
@@ -58,11 +57,9 @@ const AddProviderForm: React.FC
+ Global (All Providers)
+
+
+ 
handleImageError(e, providerDisplayName)}
+ />
+ {providerDisplayName}
+
+
+
+
+ $
+
+
+
+
+
- handleImageError(e, providerDisplayName)}
/>
diff --git a/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx b/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
index c356982f189..32ffd55efa0 100644
--- a/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
+++ b/ui/litellm-dashboard/src/components/CostTrackingSettings/cost_tracking_settings.tsx
@@ -1,16 +1,16 @@
-import React, { useState, useEffect, useCallback } from "react";
-import { Title, Text, Button, TabGroup, TabList, Tab, TabPanels, TabPanel } from "@tremor/react";
+import React, { useState, useEffect } from "react";
+import { Title, Text, Button, Accordion, AccordionHeader, AccordionBody, TabGroup, TabList, Tab, TabPanels, TabPanel } from "@tremor/react";
import { Modal, Form } from "antd";
-import { getProxyBaseUrl } from "@/components/networking";
-import NotificationsManager from "../molecules/notifications_manager";
-import { Providers } from "../provider_info_helpers";
-import { CostTrackingSettingsProps, DiscountConfig } from "./types";
-import { getProviderBackendValue } from "./provider_display_helpers";
+import { CostTrackingSettingsProps } from "./types";
import ProviderDiscountTable from "./provider_discount_table";
import AddProviderForm from "./add_provider_form";
+import ProviderMarginTable from "./provider_margin_table";
+import AddMarginForm from "./add_margin_form";
import { ExclamationCircleOutlined } from "@ant-design/icons";
import { DocsMenu } from "../HelpLink";
import HowItWorks from "./how_it_works";
+import { useDiscountConfig } from "./use_discount_config";
+import { useMarginConfig } from "./use_margin_config";
const DOCS_LINKS = [
{ label: "Custom pricing for models", href: "https://docs.litellm.ai/docs/proxy/custom_pricing" },
@@ -22,118 +22,51 @@ const CostTrackingSettings: React.FC = ({
userRole,
accessToken
}) => {
- const [discountConfig, setDiscountConfig] = useState({});
const [selectedProvider, setSelectedProvider] = useState(undefined);
const [newDiscount, setNewDiscount] = useState("");
const [isFetching, setIsFetching] = useState(true);
const [isModalVisible, setIsModalVisible] = useState(false);
+ const [isMarginModalVisible, setIsMarginModalVisible] = useState(false);
+ const [selectedMarginProvider, setSelectedMarginProvider] = useState(undefined);
+ const [marginType, setMarginType] = useState<"percentage" | "fixed">("percentage");
+ const [percentageValue, setPercentageValue] = useState("");
+ const [fixedAmountValue, setFixedAmountValue] = useState("");
const [form] = Form.useForm();
+ const [marginForm] = Form.useForm();
const [modal, contextHolder] = Modal.useModal();
- const fetchDiscountConfig = useCallback(async () => {
- setIsFetching(true);
- try {
- const proxyBaseUrl = getProxyBaseUrl();
- const url = proxyBaseUrl
- ? `${proxyBaseUrl}/config/cost_discount_config`
- : "/config/cost_discount_config";
-
- const response = await fetch(url, {
- method: "GET",
- headers: {
- Authorization: `Bearer ${accessToken}`,
- "Content-Type": "application/json",
- },
- });
+ // Use custom hooks for discount and margin config
+ const {
+ discountConfig,
+ fetchDiscountConfig,
+ handleAddProvider: addProvider,
+ handleRemoveProvider: removeProvider,
+ handleDiscountChange,
+ } = useDiscountConfig({ accessToken });
- if (response.ok) {
- const data = await response.json();
- setDiscountConfig(data.values || {});
- } else {
- console.error("Failed to fetch discount config");
- }
- } catch (error) {
- console.error("Error fetching discount config:", error);
- NotificationsManager.fromBackend("Failed to fetch discount configuration");
- } finally {
- setIsFetching(false);
- }
- }, [accessToken]);
+ const {
+ marginConfig,
+ fetchMarginConfig,
+ handleAddMargin: addMargin,
+ handleRemoveMargin: removeMargin,
+ handleMarginChange,
+ } = useMarginConfig({ accessToken });
useEffect(() => {
if (accessToken) {
- fetchDiscountConfig();
- }
- }, [accessToken, fetchDiscountConfig]);
-
- const saveDiscountConfig = async (config: DiscountConfig) => {
- try {
- const proxyBaseUrl = getProxyBaseUrl();
- const url = proxyBaseUrl
- ? `${proxyBaseUrl}/config/cost_discount_config`
- : "/config/cost_discount_config";
-
- const response = await fetch(url, {
- method: "PATCH",
- headers: {
- Authorization: `Bearer ${accessToken}`,
- "Content-Type": "application/json",
- },
- body: JSON.stringify(config),
+ Promise.all([fetchDiscountConfig(), fetchMarginConfig()]).finally(() => {
+ setIsFetching(false);
});
-
- if (response.ok) {
- NotificationsManager.success("Discount configuration updated successfully");
- await fetchDiscountConfig();
- } else {
- const errorData = await response.json();
- const errorMessage = errorData.detail?.error || errorData.detail || "Failed to update settings";
- NotificationsManager.fromBackend(errorMessage);
- }
- } catch (error) {
- console.error("Error updating discount config:", error);
- NotificationsManager.fromBackend("Failed to update discount configuration");
}
- };
+ }, [accessToken, fetchDiscountConfig, fetchMarginConfig]);
const handleAddProvider = async () => {
- if (!selectedProvider || !newDiscount) {
- NotificationsManager.fromBackend("Please select a provider and enter discount percentage");
- return;
+ const success = await addProvider(selectedProvider, newDiscount);
+ if (success) {
+ setSelectedProvider(undefined);
+ setNewDiscount("");
+ setIsModalVisible(false);
}
-
- const percentageValue = parseFloat(newDiscount);
- if (isNaN(percentageValue) || percentageValue < 0 || percentageValue > 100) {
- NotificationsManager.fromBackend("Discount must be between 0% and 100%");
- return;
- }
-
- const providerValue = getProviderBackendValue(selectedProvider);
-
- if (!providerValue) {
- NotificationsManager.fromBackend("Invalid provider selected");
- return;
- }
-
- if (discountConfig[providerValue]) {
- NotificationsManager.fromBackend(
- `Discount for ${Providers[selectedProvider as keyof typeof Providers]} already exists. Edit it in the table above.`
- );
- return;
- }
-
- // Convert percentage to decimal for storage
- const discountValue = percentageValue / 100;
- const updatedConfig = {
- ...discountConfig,
- [providerValue]: discountValue,
- };
-
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
- setSelectedProvider(undefined);
- setNewDiscount("");
- setIsModalVisible(false);
};
const handleModalCancel = () => {
@@ -143,7 +76,7 @@ const CostTrackingSettings: React.FC = ({
setNewDiscount("");
};
- const handleFormSubmit = (values: any) => {
+ const handleFormSubmit = () => {
handleAddProvider();
};
@@ -155,27 +88,47 @@ const CostTrackingSettings: React.FC = ({
okText: 'Remove',
okType: 'danger',
cancelText: 'Cancel',
- onOk: async () => {
- const updatedConfig = { ...discountConfig };
- delete updatedConfig[provider];
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
- },
+ onOk: () => removeProvider(provider),
});
};
- const handleDiscountChange = async (provider: string, value: string) => {
- const discountValue = parseFloat(value);
- if (!isNaN(discountValue) && discountValue >= 0 && discountValue <= 1) {
- const updatedConfig = {
- ...discountConfig,
- [provider]: discountValue,
- };
- setDiscountConfig(updatedConfig);
- await saveDiscountConfig(updatedConfig);
+ const handleAddMargin = async () => {
+ const success = await addMargin({
+ selectedProvider: selectedMarginProvider,
+ marginType,
+ percentageValue,
+ fixedAmountValue,
+ });
+ if (success) {
+ setSelectedMarginProvider(undefined);
+ setPercentageValue("");
+ setFixedAmountValue("");
+ setMarginType("percentage");
+ setIsMarginModalVisible(false);
}
};
+ const handleMarginModalCancel = () => {
+ setIsMarginModalVisible(false);
+ marginForm.resetFields();
+ setSelectedMarginProvider(undefined);
+ setPercentageValue("");
+ setFixedAmountValue("");
+ setMarginType("percentage");
+ };
+
+ const handleRemoveMargin = async (provider: string, providerDisplayName: string) => {
+ modal.confirm({
+ title: 'Remove Provider Margin',
+ icon: = ({
-
-
- Provider Discounts
- Test It
-
-
-
+ {/* Main Content Card with Accordions */}
+ = ({
+
+
+ {/* Accordion 1: Provider Discounts */}
+
+
+
+
+
+
+ Discounts
+ Test It
+
+
+
+
+
+
+
+
+
+
+
+ {/* Accordion 2: Fee/Price Margin */}
+
+
+
+
+
+
+ Provider Discounts
+
+ Apply percentage-based discounts to reduce costs for specific providers
+
+
+
+
+
+
+
+ {isFetching ? (
+
+ Loading configuration...
+
+ ) : Object.keys(discountConfig).length > 0 ? (
+
+
+
+ No provider discounts configured
+
+
+ Click "Add Provider Discount" to get started
+
+
+ )}
+
+
+
+ Fee/Price Margin
+
+ Add fees or margins to LLM costs for internal billing and cost recovery
+
+
+
+
Loading configuration...
- ) : Object.keys(discountConfig).length > 0 ? (
-
- No provider discounts configured
+ No provider margins configured
- Click "Add Provider Discount" to get started
+ Click "Add Provider Margin" to get started
)}
-
-
-
-
-
+
+
+
+
{isFetching ? (
-
+ ) : Object.keys(marginConfig).length > 0 ? (
+
-
- Add Provider Margin
+ + } + open={isMarginModalVisible} + width={1000} + onCancel={handleMarginModalCancel} + footer={null} + className="top-8" + styles={{ + body: { padding: "24px" }, + header: { padding: "24px 24px 0 24px", border: "none" }, + }} + > +
+
+ Select a provider (or "Global" for all providers) and configure the margin. You can use percentage-based or fixed amount.
+
+
+
+
+ Global (All Providers)
+
+ );
+ }
+ const { displayName, logo } = getProviderDisplayInfo(row.provider);
+ return (
+
+ {logo && (
+ 
handleImageError(e, displayName)}
+ />
+ )}
+ {displayName}
+
+ );
+ },
+ },
+ {
+ header: "Margin",
+ cell: (row) => (
+
+ {editingProvider === row.provider ? (
+ <>
+ handleSaveEdit(row.provider)}
+ className="cursor-pointer text-green-600 hover:text-green-700"
+ />
+ {formatMargin(row.margin)}
+ handleStartEdit(row.provider, row.margin)}
+ className="cursor-pointer text-blue-600 hover:text-blue-700"
+ />
+
+ )}
+
+ ),
+ width: "350px",
+ },
+ {
+ header: "Actions",
+ cell: (row) => {
+ const displayName = row.provider === "global" ? "Global" : getProviderDisplayInfo(row.provider).displayName;
+ return (
+
+
+
-
diff --git a/ui/litellm-dashboard/src/components/UsagePage/components/UsagePageView.tsx b/ui/litellm-dashboard/src/components/UsagePage/components/UsagePageView.tsx
index 9766983c36a..920955138b9 100644
--- a/ui/litellm-dashboard/src/components/UsagePage/components/UsagePageView.tsx
+++ b/ui/litellm-dashboard/src/components/UsagePage/components/UsagePageView.tsx
@@ -80,8 +80,8 @@ const UsagePage: React.FC
-
- );
-};
-
/**
- * AllKeysTable – a new table for keys that mimics the table styling used in view_logs.
+ * VirtualKeysTable – a new table for keys that mimics the table styling used in view_logs.
* The team selector and filtering have been removed so that all keys are shown.
*/
-export function AllKeysTable({
- keys,
- setKeys,
- isLoading = false,
- pagination,
- onPageChange,
- pageSize = 50,
- teams,
- selectedTeam,
- setSelectedTeam,
- selectedKeyAlias,
- setSelectedKeyAlias,
- accessToken,
- userID,
- userRole,
- organizations,
- setCurrentOrg,
- refresh,
- onSortChange,
- currentSort,
- premiumUser,
- setAccessToken,
-}: AllKeysTableProps) {
- const [selectedKeyId, setSelectedKeyId] = useState
- Where Team is
-
-
-