(docs) proxy

This commit is contained in:
ishaan-jaff 2023-12-02 14:24:38 -08:00
parent dac22df8fe
commit 2587749323
6 changed files with 10 additions and 3 deletions

View file

@ -1,4 +1,6 @@
# Caching
Cache LLM Responses
Caching can be enabled by adding the `cache` key in the `config.yaml`
#### Step 1: Add `cache` to the config.yaml
```yaml

View file

@ -1,4 +1,5 @@
# CLI Arguments
Cli arguments, --host, --port, --num_workers
#### --host
- **Default:** `'0.0.0.0'`

View file

@ -1,5 +1,5 @@
# Config.yaml
The Config allows you to set the following params
# Proxy Config.yaml
Set model list, `api_base`, `api_key`, `temperature` & proxy server settings (`maaster-key`)
| Param Name | Description |
|----------------------|---------------------------------------------------------------|

View file

@ -1,6 +1,8 @@
# Load Balancing - Multiple Instances of 1 model
Use this config to load balance between multiple instances of the same model. The proxy will handle routing requests (using LiteLLM's Router). **Set `rpm` in the config if you want maximize throughput**
Load balance multiple instances of the same model
The proxy will handle routing requests (using LiteLLM's Router). **Set `rpm` in the config if you want maximize throughput**
#### Example config
requests with `model=gpt-3.5-turbo` will be routed across multiple instances of `azure/gpt-3.5-turbo`

View file

@ -1,4 +1,5 @@
# Logging - OpenTelemetry, Langfuse, ElasticSearch
Log Proxy Input, Output, Exceptions to Langfuse, OpenTelemetry
## Logging Proxy Input/Output - OpenTelemetry
### Step 1 Start OpenTelemetry Collecter Docker Container

View file

@ -1,5 +1,6 @@
# Cost Tracking & Virtual Keys
Track Spend and create virtual keys for the proxy
Grant other's temporary access to your proxy, with keys that expire after a set duration.