mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-07 02:59:05 +00:00
b2aff8be0f
Every pod and uvicorn worker schedules its own CheckBatchCost poller against the shared managed-object table, so two of them can select the same completed batch in one polling window and both write an aretrieve_batch spend log for it, counting that batch's cost twice. Claim the row with a compare-and-swap on batch_processed, and skip the batch when another pod already holds it. The claim sits immediately before the spend log is written rather than before the results fetch, because batch_processed is also what blocks deletion of the files the fetch reads and what keeps an unbilled row selectable by later poll cycles, so claiming up front would strand the spend of any worker that died mid-fetch. A failed spend log write hands the row back. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|---|---|---|
| .. | ||
| auth | ||
| common_utils | ||
| hooks | ||
| management_endpoints | ||
| ui_crud_endpoints | ||
| vector_stores | ||
| __init__.py | ||
| audit_logging_endpoints.py | ||
| enterprise_routes.py | ||
| proxy_server.py | ||
| readme.md | ||
| utils.py | ||
LiteLLM Proxy Enterprise Features - Readme
Overview
This directory contains enterprise features used on the LiteLLM proxy.
Format
Create a file for every group of endpoints (e.g. key_management_endpoints.py, user_management_endpoints.py, etc.)
If there is a broader semantic group of endpoints, create a folder for that group (e.g. management_endpoints, auth_endpoints, etc.)