scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-06 08:16:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 78

fix: update worker recommendation (#15702)

Browse source
This commit is contained in:
Alexsander Hamir 2025-10-18 16:24:32 -07:00 • committed by GitHub
parent 3fc49a029f
commit 441aed2c87
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 6 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
docs/my-website/docs/proxy/prod.md
View file

@ -62,20 +62,20 @@ These specifications provide:
- Adequate memory for request processing and caching
## 3. On Kubernetes - Use 1 Uvicorn worker [Suggested CMD]
## 3. On Kubernetes — Match Uvicorn Workers to CPU Count [Suggested CMD]
Use this Docker `CMD`. This will start the proxy with 1 Uvicorn Async Worker
Use this Docker `CMD`. It automatically matches Uvicorn workers to the pod’s CPU count, ensuring each worker uses one core efficiently for better throughput and stable latency.
(Ensure that you're not setting `run_gunicorn` or `num_workers` in the CMD).
```shell
CMD ["--port", "4000", "--config", "./proxy_server_config.yaml"]
CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "$(nproc)"]
```
> Optional: If you observe gradual memory growth under sustained load, consider recycling workers after a fixed number of requests to mitigate leaks. Set this via CLI or environment variable:
> **Optional:** If you observe gradual memory growth under sustained load, consider recycling workers after a fixed number of requests to mitigate leaks.
> You can configure this either via CLI or environment variable:
```shell
# CLI
CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--max_requests_before_restart", "10000"]
CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "$(nproc)", "--max_requests_before_restart", "10000"]
# or ENV (for deployment manifests / containers)
export MAX_REQUESTS_BEFORE_RESTART=10000