mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
Merge pull request #3991 from BerriAI/litellm_spend_tracking_docs
[Doc] - Spend tracking with litellm
This commit is contained in:
Ishaan Jaff
GitHub
commit
03a98b2d4d
2 changed files with 150 additions and 13 deletions
|
|
@ -1,22 +1,155 @@
|
|||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import Image from '@theme/IdealImage';
|
||||
|
||||
# 💸 Spend Tracking
|
||||
|
||||
Track spend for keys, users, and teams across 100+ LLMs.
|
||||
|
||||
## Getting Spend Reports - To Charge Other Teams, API Keys
|
||||
### How to Track Spend with LiteLLM
|
||||
|
||||
**Step 1**
|
||||
|
||||
👉 [Setup LiteLLM with a Database](https://docs.litellm.ai/docs/proxy/deploy)
|
||||
|
||||
|
||||
**Step2** Send `/chat/completions` request
|
||||
|
||||
<Tabs>
|
||||
|
||||
|
||||
<TabItem value="openai" label="OpenAI Python v1.0.0+">
|
||||
|
||||
```python
|
||||
import openai
|
||||
client = openai.OpenAI(
|
||||
api_key="sk-1234",
|
||||
base_url="http://0.0.0.0:4000"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="llama3",
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "this is a test request, write a short poem"
|
||||
}
|
||||
],
|
||||
user="palantir",
|
||||
extra_body={
|
||||
"metadata": {
|
||||
"tags": ["jobID:214590dsff09fds", "taskName:run_page_classification"]
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
print(response)
|
||||
```
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="Curl" label="Curl Request">
|
||||
|
||||
Pass `metadata` as part of the request body
|
||||
|
||||
```shell
|
||||
curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header 'Authorization: Bearer sk-1234' \
|
||||
--data '{
|
||||
"model": "llama3",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "what llm are you"
|
||||
}
|
||||
],
|
||||
"user": "palantir",
|
||||
"metadata": {
|
||||
"tags": ["jobID:214590dsff09fds", "taskName:run_page_classification"]
|
||||
}
|
||||
}'
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="langchain" label="Langchain">
|
||||
|
||||
```python
|
||||
from langchain.chat_models import ChatOpenAI
|
||||
from langchain.prompts.chat import (
|
||||
ChatPromptTemplate,
|
||||
HumanMessagePromptTemplate,
|
||||
SystemMessagePromptTemplate,
|
||||
)
|
||||
from langchain.schema import HumanMessage, SystemMessage
|
||||
import os
|
||||
|
||||
os.environ["OPENAI_API_KEY"] = "sk-1234"
|
||||
|
||||
chat = ChatOpenAI(
|
||||
openai_api_base="http://0.0.0.0:4000",
|
||||
model = "llama3",
|
||||
user="palantir",
|
||||
extra_body={
|
||||
"metadata": {
|
||||
"tags": ["jobID:214590dsff09fds", "taskName:run_page_classification"]
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
messages = [
|
||||
SystemMessage(
|
||||
content="You are a helpful assistant that im using to make a test request to."
|
||||
),
|
||||
HumanMessage(
|
||||
content="test from litellm. tell me why it's amazing in 1 sentence"
|
||||
),
|
||||
]
|
||||
response = chat(messages)
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
**Step3 - Verify Spend Tracked**
|
||||
That's IT. Now Verify your spend was tracked
|
||||
|
||||
The following spend gets tracked in Table `LiteLLM_SpendLogs`
|
||||
|
||||
```json
|
||||
{
|
||||
"api_key": "fe6b0cab4ff5a5a8df823196cc8a450*****", # Hash of API Key used
|
||||
"user": "default_user", # Internal User (LiteLLM_UserTable) that owns `api_key=sk-1234`.
|
||||
"team_id": "e8d1460f-846c-45d7-9b43-55f3cc52ac32", # Team (LiteLLM_TeamTable) that owns `api_key=sk-1234`
|
||||
"request_tags": ["jobID:214590dsff09fds", "taskName:run_page_classification"],# Tags sent in request
|
||||
"end_user": "palantir", # Customer - the `user` sent in the request
|
||||
"model_group": "llama3", # "model" passed to LiteLLM
|
||||
"api_base": "https://api.groq.com/openai/v1/", # "api_base" of model used by LiteLLM
|
||||
"spend": 0.000002, # Spend in $
|
||||
"total_tokens": 100,
|
||||
"completion_tokens": 80,
|
||||
"prompt_tokens": 20,
|
||||
|
||||
}
|
||||
```
|
||||
|
||||
Navigate to the Usage Tab on the LiteLLM UI (found on https://your-proxy-endpoint/ui) and verify you see spend tracked under `Usage`
|
||||
|
||||
<Image img={require('../../img/admin_ui_spend.png')} />
|
||||
|
||||
## API Endpoints to get Spend
|
||||
#### Getting Spend Reports - To Charge Other Teams, API Keys
|
||||
|
||||
Use the `/global/spend/report` endpoint to get daily spend per team, with a breakdown of spend per API Key, Model
|
||||
|
||||
### Example Request
|
||||
##### Example Request
|
||||
|
||||
```shell
|
||||
curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end_date=2024-06-30' \
|
||||
-H 'Authorization: Bearer sk-1234'
|
||||
```
|
||||
|
||||
### Example Response
|
||||
##### Example Response
|
||||
<Tabs>
|
||||
|
||||
<TabItem value="response" label="Expected Response">
|
||||
|
|
@ -125,7 +258,7 @@ Output from script
|
|||
|
||||
</Tabs>
|
||||
|
||||
## Allowing Non-Proxy Admins to access `/spend` endpoints
|
||||
#### Allowing Non-Proxy Admins to access `/spend` endpoints
|
||||
|
||||
Use this when you want non-proxy admins to access `/spend` endpoints
|
||||
|
||||
|
|
@ -135,7 +268,7 @@ Schedule a [meeting with us to get your Enterprise License](https://calendly.com
|
|||
|
||||
:::
|
||||
|
||||
### Create Key
|
||||
##### Create Key
|
||||
Create Key with with `permissions={"get_spend_routes": true}`
|
||||
```shell
|
||||
curl --location 'http://0.0.0.0:4000/key/generate' \
|
||||
|
|
@ -146,7 +279,7 @@ curl --location 'http://0.0.0.0:4000/key/generate' \
|
|||
}'
|
||||
```
|
||||
|
||||
### Use generated key on `/spend` endpoints
|
||||
##### Use generated key on `/spend` endpoints
|
||||
|
||||
Access spend Routes with newly generate keys
|
||||
```shell
|
||||
|
|
@ -156,14 +289,14 @@ curl -X GET 'http://localhost:4000/global/spend/report?start_date=2024-04-01&end
|
|||
|
||||
|
||||
|
||||
## Reset Team, API Key Spend - MASTER KEY ONLY
|
||||
#### Reset Team, API Key Spend - MASTER KEY ONLY
|
||||
|
||||
Use `/global/spend/reset` if you want to:
|
||||
- Reset the Spend for all API Keys, Teams. The `spend` for ALL Teams and Keys in `LiteLLM_TeamTable` and `LiteLLM_VerificationToken` will be set to `spend=0`
|
||||
|
||||
- LiteLLM will maintain all the logs in `LiteLLMSpendLogs` for Auditing Purposes
|
||||
|
||||
### Request
|
||||
##### Request
|
||||
Only the `LITELLM_MASTER_KEY` you set can access this route
|
||||
```shell
|
||||
curl -X POST \
|
||||
|
|
@ -172,7 +305,7 @@ curl -X POST \
|
|||
-H 'Content-Type: application/json'
|
||||
```
|
||||
|
||||
### Expected Responses
|
||||
##### Expected Responses
|
||||
|
||||
```shell
|
||||
{"message":"Spend for all API Keys and Teams reset successfully","status":"success"}
|
||||
|
|
@ -181,11 +314,11 @@ curl -X POST \
|
|||
|
||||
|
||||
|
||||
## Spend Tracking for Azure
|
||||
## Spend Tracking for Azure OpenAI Models
|
||||
|
||||
Set base model for cost tracking azure image-gen call
|
||||
|
||||
### Image Generation
|
||||
#### Image Generation
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
|
|
@ -200,7 +333,7 @@ model_list:
|
|||
mode: image_generation
|
||||
```
|
||||
|
||||
### Chat Completions / Embeddings
|
||||
#### Chat Completions / Embeddings
|
||||
|
||||
**Problem**: Azure returns `gpt-4` in the response when `azure/gpt-4-1106-preview` is used. This leads to inaccurate cost tracking
|
||||
|
||||
|
|
@ -219,4 +352,8 @@ model_list:
|
|||
api_version: "2023-07-01-preview"
|
||||
model_info:
|
||||
base_model: azure/gpt-4-1106-preview
|
||||
```
|
||||
```
|
||||
|
||||
## Custom Input/Output Pricing
|
||||
|
||||
👉 Head to [Custom Input/Output Pricing](https://docs.litellm.ai/docs/proxy/custom_pricing) to setup custom pricing or your models
|
||||
BIN
docs/my-website/img/admin_ui_spend.png
Normal file
BIN
docs/my-website/img/admin_ui_spend.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 176 KiB |
Loading…
Add table
Reference in a new issue