mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
(docs) proxy
This commit is contained in:
parent
dac22df8fe
commit
2587749323
6 changed files with 10 additions and 3 deletions
|
|
@ -1,4 +1,6 @@
|
|||
# Caching
|
||||
Cache LLM Responses
|
||||
|
||||
Caching can be enabled by adding the `cache` key in the `config.yaml`
|
||||
#### Step 1: Add `cache` to the config.yaml
|
||||
```yaml
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# CLI Arguments
|
||||
Cli arguments, --host, --port, --num_workers
|
||||
|
||||
#### --host
|
||||
- **Default:** `'0.0.0.0'`
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
# Config.yaml
|
||||
The Config allows you to set the following params
|
||||
# Proxy Config.yaml
|
||||
Set model list, `api_base`, `api_key`, `temperature` & proxy server settings (`maaster-key`)
|
||||
|
||||
| Param Name | Description |
|
||||
|----------------------|---------------------------------------------------------------|
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
|
||||
# Load Balancing - Multiple Instances of 1 model
|
||||
Use this config to load balance between multiple instances of the same model. The proxy will handle routing requests (using LiteLLM's Router). **Set `rpm` in the config if you want maximize throughput**
|
||||
Load balance multiple instances of the same model
|
||||
|
||||
The proxy will handle routing requests (using LiteLLM's Router). **Set `rpm` in the config if you want maximize throughput**
|
||||
|
||||
#### Example config
|
||||
requests with `model=gpt-3.5-turbo` will be routed across multiple instances of `azure/gpt-3.5-turbo`
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# Logging - OpenTelemetry, Langfuse, ElasticSearch
|
||||
Log Proxy Input, Output, Exceptions to Langfuse, OpenTelemetry
|
||||
## Logging Proxy Input/Output - OpenTelemetry
|
||||
|
||||
### Step 1 Start OpenTelemetry Collecter Docker Container
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
|
||||
# Cost Tracking & Virtual Keys
|
||||
Track Spend and create virtual keys for the proxy
|
||||
|
||||
Grant other's temporary access to your proxy, with keys that expire after a set duration.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue