scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-08 22:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 78

Add/remove blank lines to make MD linter happy

Browse source 
This removed a lot of yellow squigglies in my VS Code.
This commit is contained in:
Marc Abramowitz 2024-07-10 17:20:23 -07:00
parent 9dadbe52f7
commit 45a9920080
1 changed files with 52 additions and 32 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

84
docs/my-website/docs/proxy/logging.md
View file

@ -15,6 +15,7 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
## Logging Proxy Input/Output - Langfuse
We will use the `--config` to set `litellm.success_callback = ["langfuse"]` this will log all successfull LLM calls to langfuse. Make sure to set `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` in your environment
**Step 1** Install langfuse
@ -24,6 +25,7 @@ pip install langfuse>=2.0.0
```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -34,6 +36,7 @@ litellm_settings:
```
**Step 3**: Set required env variables for logging to langfuse
```shell
export LANGFUSE_PUBLIC_KEY="pk_kk"
export LANGFUSE_SECRET_KEY="sk_ss"
@ -44,11 +47,13 @@ export LANGFUSE_HOST="https://xxx.langfuse.com"
**Step 4**: Start the proxy, make a test request
Start proxy
```shell
litellm --config config.yaml --debug
```
Test Request
```
litellm --test
```
@ -59,7 +64,6 @@ Expected output on Langfuse
### Logging Metadata to Langfuse
<Tabs>
<TabItem value="Curl" label="Curl Request">
@ -85,6 +89,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}
}'
```
</TabItem>
<TabItem value="openai" label="OpenAI v1.0.0+">
@ -118,6 +123,7 @@ response = client.chat.completions.create(
print(response)
```
</TabItem>
<TabItem value="langchain" label="Langchain">
@ -160,7 +166,6 @@ print(response)
</TabItem>
</Tabs>
### Team based Logging to Langfuse
**Example:**
@ -249,6 +254,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}
}'
```
</TabItem>
<TabItem value="openai" label="OpenAI v1.0.0+">
@ -279,6 +285,7 @@ response = client.chat.completions.create(
print(response)
```
</TabItem>
<TabItem value="langchain" label="Langchain">
@ -324,7 +331,6 @@ You will see `raw_request` in your Langfuse Metadata. This is the RAW CURL comma
<Image img={require('../../img/debug_langfuse.png')} />
## Logging Proxy Input/Output in OpenTelemetry format
:::info
@ -340,10 +346,8 @@ OTEL_SERVICE_NAME=<your-service-name>` # default="litellm"
<Tabs>
<TabItem value="Console Exporter" label="Log to console">
**Step 1:** Set callbacks and env vars
Add the following to your env
@ -359,7 +363,6 @@ litellm_settings:
callbacks: ["otel"]
```
**Step 2**: Start the proxy, make a test request
Start proxy
@ -419,7 +422,6 @@ This is the Span from OTEL Logging
</TabItem>
<TabItem value="Honeycomb" label="Log to Honeycomb">
#### Quick Start - Log to Honeycomb
@ -441,7 +443,6 @@ litellm_settings:
callbacks: ["otel"]
```
**Step 2**: Start the proxy, make a test request
Start proxy
@ -466,10 +467,8 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}'
```
</TabItem>
<TabItem value="otel-col" label="Log to OTEL HTTP Collector">
#### Quick Start - Log to OTEL Collector
@ -491,7 +490,6 @@ litellm_settings:
callbacks: ["otel"]
```
**Step 2**: Start the proxy, make a test request
Start proxy
@ -518,7 +516,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
</TabItem>
<TabItem value="otel-col-grpc" label="Log to OTEL GRPC Collector">
#### Quick Start - Log to OTEL GRPC Collector
@ -540,7 +537,6 @@ litellm_settings:
callbacks: ["otel"]
```
**Step 2**: Start the proxy, make a test request
Start proxy
@ -565,7 +561,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
}'
```
</TabItem>
<TabItem value="traceloop" label="Log to Traceloop Cloud">
@ -588,7 +583,6 @@ environment_variables:
TRACELOOP_API_KEY: "XXXXX"
```
**Step 3**: Start the proxy, make a test request
Start proxy
@ -624,11 +618,15 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
❓ Use this when you want to **pass information about the incoming request in a distributed tracing system**
✅ Key change: Pass the **`traceparent` header** in your requests. [Read more about traceparent headers here](https://uptrace.dev/opentelemetry/opentelemetry-traceparent.html#what-is-traceparent-header)
```curl
traceparent: 00-80e1afed08e019fc1110464cfa66635c-7a085853722dc6d2-01
```
Example Usage
1. Make Request to LiteLLM Proxy with `traceparent` header
```python
import openai
import uuid
@ -652,7 +650,6 @@ response = client.chat.completions.create(
)
print(response)
```
```shell
@ -666,12 +663,12 @@ Search for Trace=`80e1afed08e019fc1110464cfa66635c` on your OTEL Collector
<Image img={require('../../img/otel_parent.png')} />
## Custom Callback Class [Async]
Use this when you want to run custom callbacks in `python`
#### Step 1 - Create your custom `litellm` callback class
We use `litellm.integrations.custom_logger` for this, **more details about litellm custom callbacks [here](https://docs.litellm.ai/docs/observability/custom_callback)**
Define your custom callback class in a python file.
@ -774,16 +771,17 @@ proxy_handler_instance = MyCustomHandler()
```
#### Step 2 - Pass your custom callback class in `config.yaml`
We pass the custom callback class defined in **Step1** to the config.yaml.
Set `callbacks` to `python_filename.logger_instance_name`
In the config below, we pass
- python_filename: `custom_callbacks.py`
- logger_instance_name: `proxy_handler_instance`. This is defined in Step 1
`callbacks: custom_callbacks.proxy_handler_instance`
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -796,6 +794,7 @@ litellm_settings:
```
#### Step 3 - Start proxy + test request
```shell
litellm --config proxy_config.yaml
```
@ -817,6 +816,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
```
#### Resulting Log on Proxy
```shell
On Success
Model: gpt-3.5-turbo,
@ -869,7 +869,6 @@ class MyCustomHandler(CustomLogger):
"max_tokens": 10
}
}
```
#### Logging `model_info` set in config.yaml
@ -887,11 +886,13 @@ class MyCustomHandler(CustomLogger):
```
**Expected Output**
```json
{'mode': 'embedding', 'input_cost_per_token': 0.002}
```
### Logging responses from proxy
Both `/chat/completions` and `/embeddings` responses are available as `response_obj`
**Note: for `/chat/completions`, both `stream=True` and `non stream` responses are available as `response_obj`**
@ -905,6 +906,7 @@ class MyCustomHandler(CustomLogger):
```
**Expected Output /chat/completion [for both `stream` and `non-stream` responses]**
```json
ModelResponse(
id='chatcmpl-8Tfu8GoMElwOZuj2JlHBhNHG01PPo',
@ -931,6 +933,7 @@ ModelResponse(
```
**Expected Output /embeddings**
```json
{
'model': 'ada',
@ -950,7 +953,6 @@ ModelResponse(
}
```
## Custom Callback APIs [Async]
:::info
@ -960,10 +962,12 @@ This is an Enterprise only feature [Get Started with Enterprise here](https://gi
:::
Use this if you:
- Want to use custom callbacks written in a non Python programming language
- Want your callbacks to run on a different microservice
#### Step 1. Create your generic logging API endpoint
Set up a generic API endpoint that can receive data in JSON format. The data will be included within a "data" field.
Your server should support the following Request format:
@ -1026,11 +1030,8 @@ async def log_event(request: Request):
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="127.0.0.1", port=4000)
```
#### Step 2. Set your `GENERIC_LOGGER_ENDPOINT` to the endpoint + route we should send callback logs to
```shell
@ -1040,6 +1041,7 @@ os.environ["GENERIC_LOGGER_ENDPOINT"] = "http://localhost:4000/log-event"
#### Step 3. Create a `config.yaml` file and set `litellm_settings`: `success_callback` = ["generic"]
Example litellm proxy config.yaml
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -1051,8 +1053,8 @@ litellm_settings:
Start the LiteLLM Proxy and make a test request to verify the logs reached your callback API
## Logging LLM IO to Galileo
[BETA]
Log LLM I/O on [www.rungalileo.io](https://www.rungalileo.io/)
@ -1075,6 +1077,7 @@ export GALILEO_PASSWORD=""
### Quick Start
1. Add to Config.yaml
```yaml
model_list:
- litellm_params:
@ -1110,7 +1113,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
'
```
🎉 That's it - Expect to see your Logs on your Galileo Dashboard
## Logging Proxy Cost + Usage - OpenMeter
@ -1128,6 +1130,7 @@ export OPENMETER_API_KEY=""
### Quick Start
1. Add to Config.yaml
```yaml
model_list:
- litellm_params:
@ -1163,13 +1166,14 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
'
```
<Image img={require('../../img/openmeter_img_2.png')} />
## Logging Proxy Input/Output - DataDog
We will use the `--config` to set `litellm.success_callback = ["datadog"]` this will log all successfull LLM calls to DataDog
**Step 1**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -1189,6 +1193,7 @@ DD_SITE="us5.datadoghq.com" # your datadog base url
**Step 3**: Start the proxy, make a test request
Start proxy
```shell
litellm --config config.yaml --debug
```
@ -1216,10 +1221,10 @@ Expected output on Datadog
<Image img={require('../../img/dd_small1.png')} />
## Logging Proxy Input/Output - s3 Buckets
We will use the `--config` to set
- `litellm.success_callback = ["s3"]`
This will log all successfull LLM calls to s3 Bucket
@ -1233,6 +1238,7 @@ AWS_REGION_NAME = ""
```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -1252,11 +1258,13 @@ litellm_settings:
**Step 3**: Start the proxy, make a test request
Start proxy
```shell
litellm --config config.yaml --debug
```
Test Request
```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
@ -1276,6 +1284,7 @@ Your logs should be available on the specified s3 Bucket
## Logging Proxy Input/Output - DynamoDB
We will use the `--config` to set
- `litellm.success_callback = ["dynamodb"]`
- `litellm.dynamodb_table_name = "your-table-name"`
@ -1290,6 +1299,7 @@ AWS_REGION_NAME = ""
```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -1303,11 +1313,13 @@ litellm_settings:
**Step 3**: Start the proxy, make a test request
Start proxy
```shell
litellm --config config.yaml --debug
```
Test Request
```shell
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
@ -1395,19 +1407,18 @@ Your logs should be available on DynamoDB
}
```
## Logging Proxy Input/Output - Sentry
If api calls fail (llm/database) you can log those to Sentry:
**Step 1** Install Sentry
```shell
pip install --upgrade sentry-sdk
```
**Step 2**: Save your Sentry_DSN and add `litellm_settings`: `failure_callback`
```shell
export SENTRY_DSN="your-sentry-dsn"
```
@ -1427,11 +1438,13 @@ general_settings:
**Step 3**: Start the proxy, make a test request
Start proxy
```shell
litellm --config config.yaml --debug
```
Test Request
```
litellm --test
```
@ -1449,6 +1462,7 @@ ATHINA_API_KEY = "your-athina-api-key"
```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -1461,11 +1475,13 @@ litellm_settings:
**Step 3**: Start the proxy, make a test request
Start proxy
```shell
litellm --config config.yaml --debug
```
Test Request
```
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
@ -1497,6 +1513,7 @@ AZURE_CONTENT_SAFETY_KEY = "<your-azure-content-safety-key>"
```
**Step 2**: Create a `config.yaml` file and set `litellm_settings`: `success_callback`
```yaml
model_list:
- model_name: gpt-3.5-turbo
@ -1512,11 +1529,13 @@ litellm_settings:
**Step 3**: Start the proxy, make a test request
Start proxy
```shell
litellm --config config.yaml --debug
```
Test Request
```
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
@ -1532,7 +1551,8 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
```
An HTTP 400 error will be returned if the content is detected with a value greater than the threshold set in the `config.yaml`.
The details of the response will describe :
The details of the response will describe:
- The `source` : input text or llm generated text
- The `category` : the category of the content that triggered the moderation
- The `severity` : the severity from 0 to 10