mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-05 08:07:05 +00:00
4178a857ae
The Prisma query engine is a separate Rust process whose resident memory is a high-water mark: it grows with the payload of the largest single statement it executes and glibc never returns that memory to the OS, so a pod's memory floor ratchets up to its worst-ever write and stays there for the life of the worker. Memory-based autoscaling then reads a number that reflects the largest write the pod has ever done rather than what it is doing now. The spend-log flush handed Prisma a fixed 1000 rows per create_many. With store_prompts_in_spend_logs enabled a single row carries the full prompt and response, so one statement can be tens of megabytes and permanently costs hundreds of megabytes of RSS. Row counts cannot express that budget: the same 1000 rows range from well under a megabyte to tens of megabytes. Split each flush into statements bounded by encoded payload size (SPEND_LOG_WRITE_BATCH_MAX_BYTES, default 2MB) on top of the existing 1000-row cap. What is measured is the encoded statement, so the budget counts what actually goes on the wire: the JSON escaping of quotes and newlines, multibyte characters at their encoded width, the field names and separators a 25-column row carries, and the brackets and row separators the rows carry as one collection. Deployments that do not store prompts keep one statement per 1000 rows and are unaffected; prompt-carrying flushes get several small statements instead of one huge one. A row larger than the budget is still written on its own rather than dropped, and a row the serializer refuses counts as zero rather than raising out of the flush and dropping every row queued behind it. Splitting a flush must not multiply what a poison-row flood costs, so the poison-isolation allowance is threaded through every statement of a 1000-row group instead of being handed out fresh per statement. That is only safe because the allowance now counts failed inserts rather than every insert: the one insert a statement needs when nothing is poisoned is not charged, so a healthy flush never runs the allowance down however many statements it splits into, and a statement reached after the allowance is spent is still attempted so clean rows behind a flood still persist. Failed inserts for a group are bounded by the allowance plus one baseline insert per statement, which restores the constant-per-group ceiling the single-statement path had. Resolves LIT-4765 |
||
|---|---|---|
| .. | ||
| helpers | ||
| prisma_and_spend | ||
| proxy_logging | ||
| __init__.py | ||