mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-20 00:11:50 +00:00
40b8300ac2
The Prisma query engine is a separate process whose resident memory grows with what it is asked to hold and glibc never returns it, so a pod's memory floor ratchets up to its worst statement and stays there for the life of the worker. #34956 bounded a spend-log flush by payload bytes, which caps that floor when prompts are stored and does nothing when they are not: rows carrying only attribution metadata run about 1.2 KB, so a 1000-row statement is roughly 1.2 MB, the 2 MB byte budget never binds, and every statement stays at 1000 rows forever. The engine charges per row as well as per byte. Measured on a container running the same engine build (5.4.2) against real Postgres, with rows shaped like a store_prompts_in_spend_logs=false deployment, writing the same 200,000 rows: rows/statement engine RSS still resident after the flush 1000 179 MB 500 91 MB 250 41 MB 100 19 MB None of those statements came near the byte budget, so the whole difference is row count. The floor is a plateau rather than a leak: 1,000,000 rows written at 1000 per statement settles around 229 MB and stops climbing. Adds SPEND_LOG_WRITE_BATCH_MAX_ROWS, default 100, applied alongside the existing byte budget so whichever binds first splits the statement. Both are needed, since bytes are what track a prompt-carrying row and rows are what track the engine's per-row bookkeeping. One consequence worth naming: a flush now issues more statements, and a statement that fails under a poison flood costs one insert before any isolation runs, so the irreducible floor rises by the statement count. The isolation budget still caps the amplification on top of that, and the tests assert the bound derived from the configured row cap rather than a constant. |
||
|---|---|---|
| .. | ||
| db_transaction_queue | ||
| mcp_server | ||
| conftest.py | ||
| test_autorouter_session_rollup.py | ||
| test_check_migration.py | ||
| test_create_views.py | ||
| test_daily_spend_bulk_upsert.py | ||
| test_db_spend_update_writer.py | ||
| test_db_url_settings.py | ||
| test_exception_handler.py | ||
| test_exception_handler_reconnect_retry.py | ||
| test_gateway_request_tracking.py | ||
| test_prisma_client.py | ||
| test_prisma_planned_engine_restart.py | ||
| test_prisma_self_heal.py | ||
| test_proxy_worker_heartbeat.py | ||
| test_query_engine_reaper.py | ||
| test_rds_iam_token_expiry.py | ||
| test_replica_identity.py | ||
| test_routing_prisma_wrapper.py | ||
| test_spend_log_batching.py | ||
| test_spend_log_tool_index.py | ||
| test_token_auth.py | ||
| test_tool_registry_writer.py | ||