diff --git a/docs/my-website/docs/proxy/cost_tracking.md b/docs/my-website/docs/proxy/cost_tracking.md index 7405fd12373..de1a63a4c32 100644 --- a/docs/my-website/docs/proxy/cost_tracking.md +++ b/docs/my-website/docs/proxy/cost_tracking.md @@ -1,22 +1,155 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; +import Image from '@theme/IdealImage'; # 💸 Spend Tracking Track spend for keys, users, and teams across 100+ LLMs. -## Getting Spend Reports - To Charge Other Teams, API Keys +### How to Track Spend with LiteLLM + +**Step 1** + +👉 [Setup LiteLLM with a Database](https://docs.litellm.ai/docs/proxy/deploy) + + +**Step2** Send `/chat/completions` request + + + + + + +```python +import openai +client = openai.OpenAI( + api_key="sk-1234", + base_url="http://0.0.0.0:4000" +) + +response = client.chat.completions.create( + model="llama3", + messages = [ + { + "role": "user", + "content": "this is a test request, write a short poem" + } + ], + user="palantir", + extra_body={ + "metadata": { + "tags": ["jobID:214590dsff09fds", "taskName:run_page_classification"] + } + } +) + +print(response) +``` + + + + +Pass `metadata` as part of the request body + +```shell +curl --location 'http://0.0.0.0:4000/chat/completions' \ + --header 'Content-Type: application/json' \ + --header 'Authorization: Bearer sk-1234' \ + --data '{ + "model": "llama3", + "messages": [ + { + "role": "user", + "content": "what llm are you" + } + ], + "user": "palantir", + "metadata": { + "tags": ["jobID:214590dsff09fds", "taskName:run_page_classification"] + } +}' +``` + + + +```python +from langchain.chat_models import ChatOpenAI +from langchain.prompts.chat import ( + ChatPromptTemplate, + HumanMessagePromptTemplate, + SystemMessagePromptTemplate, +) +from langchain.schema import HumanMessage, SystemMessage +import os + +os.environ["OPENAI_API_KEY"] = "sk-1234" + +chat = ChatOpenAI( + openai_api_base="http://0.0.0.0:4000", + model = "llama3", + user="palantir", + extra_body={ + "metadata": { + "tags": ["jobID:214590dsff09fds", "taskName:run_page_classification"] + } + } +) + +messages = [ + SystemMessage( + content="You are a helpful assistant that im using to make a test request to." + ), + HumanMessage( + content="test from litellm. tell me why it's amazing in 1 sentence" + ), +] +response = chat(messages) + +print(response) +``` + + + + +**Step3 - Verify Spend Tracked** +That's IT. Now Verify your spend was tracked + +The following spend gets tracked in Table `LiteLLM_SpendLogs` + +```json +{ + "api_key": "fe6b0cab4ff5a5a8df823196cc8a450*****", # Hash of API Key used + "user": "default_user", # Internal User (LiteLLM_UserTable) that owns `api_key=sk-1234`. + "team_id": "e8d1460f-846c-45d7-9b43-55f3cc52ac32", # Team (LiteLLM_TeamTable) that owns `api_key=sk-1234` + "request_tags": ["jobID:214590dsff09fds", "taskName:run_page_classification"],# Tags sent in request + "end_user": "palantir", # Customer - the `user` sent in the request + "model_group": "llama3", # "model" passed to LiteLLM + "api_base": "https://api.groq.com/openai/v1/", # "api_base" of model used by LiteLLM + "spend": 0.000002, # Spend in $ + "total_tokens": 100, + "completion_tokens": 80, + "prompt_tokens": 20, + +} +``` + +Navigate to the Usage Tab on the LiteLLM UI (found on https://your-proxy-endpoint/ui) and verify you see spend tracked under `Usage` + + + +## API Endpoints to get Spend +#### Getting Spend Reports - To Charge Other Teams, API Keys Use the `/global/spend/report` endpoint to get daily spend per team, with a breakdown of spend per API Key, Model -### Example Request +##### Example Request ```shell curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end_date=2024-06-30' \ -H 'Authorization: Bearer sk-1234' ``` -### Example Response +##### Example Response @@ -125,7 +258,7 @@ Output from script -## Allowing Non-Proxy Admins to access `/spend` endpoints +#### Allowing Non-Proxy Admins to access `/spend` endpoints Use this when you want non-proxy admins to access `/spend` endpoints @@ -135,7 +268,7 @@ Schedule a [meeting with us to get your Enterprise License](https://calendly.com ::: -### Create Key +##### Create Key Create Key with with `permissions={"get_spend_routes": true}` ```shell curl --location 'http://0.0.0.0:4000/key/generate' \ @@ -146,7 +279,7 @@ curl --location 'http://0.0.0.0:4000/key/generate' \ }' ``` -### Use generated key on `/spend` endpoints +##### Use generated key on `/spend` endpoints Access spend Routes with newly generate keys ```shell @@ -156,14 +289,14 @@ curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end -## Reset Team, API Key Spend - MASTER KEY ONLY +#### Reset Team, API Key Spend - MASTER KEY ONLY Use `/global/spend/reset` if you want to: - Reset the Spend for all API Keys, Teams. The `spend` for ALL Teams and Keys in `LiteLLM_TeamTable` and `LiteLLM_VerificationToken` will be set to `spend=0` - LiteLLM will maintain all the logs in `LiteLLMSpendLogs` for Auditing Purposes -### Request +##### Request Only the `LITELLM_MASTER_KEY` you set can access this route ```shell curl -X POST \ @@ -172,7 +305,7 @@ curl -X POST \ -H 'Content-Type: application/json' ``` -### Expected Responses +##### Expected Responses ```shell {"message":"Spend for all API Keys and Teams reset successfully","status":"success"} @@ -181,11 +314,11 @@ curl -X POST \ -## Spend Tracking for Azure +## Spend Tracking for Azure OpenAI Models Set base model for cost tracking azure image-gen call -### Image Generation +#### Image Generation ```yaml model_list: @@ -200,7 +333,7 @@ model_list: mode: image_generation ``` -### Chat Completions / Embeddings +#### Chat Completions / Embeddings **Problem**: Azure returns `gpt-4` in the response when `azure/gpt-4-1106-preview` is used. This leads to inaccurate cost tracking @@ -219,4 +352,8 @@ model_list: api_version: "2023-07-01-preview" model_info: base_model: azure/gpt-4-1106-preview -``` \ No newline at end of file +``` + +## Custom Input/Output Pricing + +👉 Head to [Custom Input/Output Pricing](https://docs.litellm.ai/docs/proxy/custom_pricing) to setup custom pricing or your models \ No newline at end of file diff --git a/docs/my-website/img/admin_ui_spend.png b/docs/my-website/img/admin_ui_spend.png new file mode 100644 index 00000000000..6a7196f8328 Binary files /dev/null and b/docs/my-website/img/admin_ui_spend.png differ