mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-09 03:18:44 +00:00
docs(prod.md): update prod docs with batch writing info
This commit is contained in:
parent
c9fecc11ed
commit
cdb940d504
1 changed files with 13 additions and 0 deletions
|
|
@ -40,6 +40,19 @@ Use this Docker `CMD`. This will start the proxy with 1 Uvicorn Async Worker
|
|||
CMD ["--port", "4000", "--config", "./proxy_server_config.yaml"]
|
||||
```
|
||||
|
||||
## 2. Batch write spend updates every 60s
|
||||
|
||||
The default proxy batch write is 10s. This is to make it easy to see spend when debugging locally.
|
||||
|
||||
In production, we recommend using a longer interval period of 60s. This reduces the number of connections used to make DB writes.
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
master_key: sk-1234
|
||||
proxy_batch_write_at: 5 # 👈 Frequency of batch writing logs to server (in seconds)
|
||||
```
|
||||
|
||||
|
||||
## 3. Move spend logs to separate server
|
||||
|
||||
Writing each spend log to the db can slow down your proxy. In testing we saw a 70% improvement in median response time, by moving writing spend logs to a separate server.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue