mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
4178a857ae
The Prisma query engine is a separate Rust process whose resident memory is a high-water mark: it grows with the payload of the largest single statement it executes and glibc never returns that memory to the OS, so a pod's memory floor ratchets up to its worst-ever write and stays there for the life of the worker. Memory-based autoscaling then reads a number that reflects the largest write the pod has ever done rather than what it is doing now. The spend-log flush handed Prisma a fixed 1000 rows per create_many. With store_prompts_in_spend_logs enabled a single row carries the full prompt and response, so one statement can be tens of megabytes and permanently costs hundreds of megabytes of RSS. Row counts cannot express that budget: the same 1000 rows range from well under a megabyte to tens of megabytes. Split each flush into statements bounded by encoded payload size (SPEND_LOG_WRITE_BATCH_MAX_BYTES, default 2MB) on top of the existing 1000-row cap. What is measured is the encoded statement, so the budget counts what actually goes on the wire: the JSON escaping of quotes and newlines, multibyte characters at their encoded width, the field names and separators a 25-column row carries, and the brackets and row separators the rows carry as one collection. Deployments that do not store prompts keep one statement per 1000 rows and are unaffected; prompt-carrying flushes get several small statements instead of one huge one. A row larger than the budget is still written on its own rather than dropped, and a row the serializer refuses counts as zero rather than raising out of the flush and dropping every row queued behind it. Splitting a flush must not multiply what a poison-row flood costs, so the poison-isolation allowance is threaded through every statement of a 1000-row group instead of being handed out fresh per statement. That is only safe because the allowance now counts failed inserts rather than every insert: the one insert a statement needs when nothing is poisoned is not charged, so a healthy flush never runs the allowance down however many statements it splits into, and a statement reached after the allowance is spent is still attempted so clean rows behind a flood still persist. Failed inserts for a group are bounded by the allowance plus one baseline insert per statement, which restores the constant-per-group ceiling the single-statement path had. Resolves LIT-4765 |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| _harness_smoke_test.py | ||
| conftest.py | ||
| test_cache_user_row.py | ||
| test_config_param_cache.py | ||
| test_password_helpers.py | ||
| test_prisma_client_engine_watcher.py | ||
| test_prisma_client_get_data.py | ||
| test_prisma_client_health.py | ||
| test_prisma_client_lifecycle.py | ||
| test_prisma_client_reconnect.py | ||
| test_prisma_client_writes.py | ||
| test_proxy_update_spend.py | ||
| test_send_email.py | ||
| test_spend_functions.py | ||