scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102
litellm/tests/test_litellm/proxy
History
Yassin Kortam 40b8300ac2
fix(spend): bound each spend-log write statement by row count as well as bytes (#37758) 
The Prisma query engine is a separate process whose resident memory grows with
what it is asked to hold and glibc never returns it, so a pod's memory floor
ratchets up to its worst statement and stays there for the life of the worker.
#34956 bounded a spend-log flush by payload bytes, which caps that floor when
prompts are stored and does nothing when they are not: rows carrying only
attribution metadata run about 1.2 KB, so a 1000-row statement is roughly
1.2 MB, the 2 MB byte budget never binds, and every statement stays at 1000
rows forever.

The engine charges per row as well as per byte. Measured on a container running
the same engine build (5.4.2) against real Postgres, with rows shaped like a
store_prompts_in_spend_logs=false deployment, writing the same 200,000 rows:

  rows/statement   engine RSS still resident after the flush
  1000             179 MB
  500               91 MB
  250               41 MB
  100               19 MB

None of those statements came near the byte budget, so the whole difference is
row count. The floor is a plateau rather than a leak: 1,000,000 rows written at
1000 per statement settles around 229 MB and stops climbing.

Adds SPEND_LOG_WRITE_BATCH_MAX_ROWS, default 100, applied alongside the
existing byte budget so whichever binds first splits the statement. Both are
needed, since bytes are what track a prompt-carrying row and rows are what
track the engine's per-row bookkeeping.

One consequence worth naming: a flush now issues more statements, and a
statement that fails under a poison flood costs one insert before any
isolation runs, so the irreducible floor rises by the statement count. The
isolation budget still caps the amplification on top of that, and the tests
assert the bound derived from the configured row cap rather than a constant.
2026-08-21 09:49:51 -07:00
..
_experimental/mcp_server test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
a2a fix(a2a): reject malformed protocolVersion suffixes while keeping semver prereleases 2026-07-21 13:24:58 -07:00
agent_endpoints fix(a2a): return SSE (text/event-stream) for message/stream instead of NDJSON (#35037) 2026-08-20 16:19:58 -07:00
analytics_endpoints feat(ui): split failed requests into their own series on the cache dashboard (#34862) 2026-07-29 09:48:17 -07:00
anthropic_endpoints fix(proxy): estimate failed-request input tokens on /v1/messages and count system prompts 2026-08-18 14:21:47 -07:00
auth test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
batches_endpoints test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
client test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
common_utils test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
config_resolvers feat(proxy): add SAML 2.0 SSO for the admin UI (#31429) 2026-07-24 12:51:28 -07:00
credential_endpoints fix(proxy): return the real status code when a credential update is rejected (#36166) 2026-08-07 08:02:52 +00:00
db fix(spend): bound each spend-log write statement by row count as well as bytes (#37758) 2026-08-21 09:49:51 -07:00
discovery_endpoints test: remove tests that never execute 2026-08-12 10:45:38 -07:00
enterprise_billing feat(proxy): push-based OTLP billable-request metering for enterprise deployments (#31592) 2026-07-15 12:12:52 -07:00
experimental/mcp_server
fine_tuning_endpoints fix(proxy): authenticate managed ids before routing 2026-08-08 02:42:34 -07:00
google_endpoints
guardrails test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
health_endpoints feat(proxy): auto-suppress the no-Redis banner for confirmed single-worker deployments 2026-08-14 17:11:05 -07:00
hooks test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
image_endpoints fix(proxy)!: share one destination check between body and path-supplied model 2026-08-05 15:17:43 -07:00
logging_endpoints
management_endpoints fix(proxy): make per-model budgets track spend, enforce, and report the same counter (#37736) 2026-08-21 09:47:52 -07:00
management_helpers test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
memory fix(proxy): give proxy_admin_viewer read parity with proxy_admin (#35851) 2026-08-05 18:33:55 +00:00
middleware feat(proxy): add Amazon Comprehend Medical passthrough provider 2026-08-17 15:44:06 -07:00
ocr_endpoints fix(ocr): validate body req_format in the proxy endpoint and run its tests in CI 2026-08-17 18:29:35 +00:00
openai_files_endpoint test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
pass_through_endpoints fix(proxy): make per-model budgets track spend, enforce, and report the same counter (#37736) 2026-08-21 09:47:52 -07:00
policy_engine test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
prompts fix(proxy): give proxy_admin_viewer read parity with proxy_admin (#35851) 2026-08-05 18:33:55 +00:00
proxy_server test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
public_endpoints feat(cognition): give Cognition its own provider identity 2026-08-20 17:16:53 -07:00
rag_endpoints fix(rag): track LLM completion usage and spend for /v1/rag/query (#32438) 2026-07-17 17:45:27 +00:00
realtime_endpoints fix(realtime): bound Vertex credential resolution and make realtime failures loud 2026-08-20 02:02:00 -07:00
response_api_endpoints fix(responses_api): map bridged chat usage on guardrail-blocked replies 2026-08-14 17:04:27 -07:00
shutdown
spend_tracking Merge pull request #30736 from nitishagar/litellm_fix_raw_key_log_persistence 2026-08-21 00:03:22 -07:00
test_configs
types_utils fix(proxy): resolve router_settings.plugins dotted paths and load plugins from installed packages (#33644) 2026-07-17 11:23:18 -07:00
ui_crud_endpoints feat(ui): let admins supply a dark-mode variant of their custom logo (#37662) 2026-08-20 12:48:55 -07:00
utils fix(spend): bound each spend-log write statement by row count as well as bytes (#37758) 2026-08-21 09:49:51 -07:00
vector_store_endpoints test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
vector_store_files_endpoints fix(proxy): authenticate managed ids before routing 2026-08-08 02:42:34 -07:00
video_endpoints
__init__.py
conftest.py test(proxy): keep a leaked llm_router out of the next test in the worker 2026-08-20 23:19:17 -07:00
test_aiohttp_cleanup_closed.py
test_aiohttp_session_recovery.py
test_api_key_masking_in_errors.py
test_audio_speech_prometheus_hooks.py
test_batch_expiry.py
test_batch_metadata_none_fix.py
test_batch_retrieve_bedrock.py
test_batch_x_litellm_model_encoding.py test(proxy): give the x-litellm-model fallback test deterministic openai env creds 2026-08-17 12:42:57 -07:00
test_blocked_response_usage.py fix(responses_api): map bridged chat usage on guardrail-blocked replies 2026-08-14 17:04:27 -07:00
test_budget_reservation.py test: enforce PT012 so a pytest.raises block cannot hide dead assertions (#37748) 2026-08-20 19:36:26 -07:00
test_caching_routes.py
test_chat_completion_metadata.py
test_claude_code_marketplace.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_common_request_processing.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_component_allowlists.py feat(proxy): authenticate to Azure Postgres with Microsoft Entra ID tokens 2026-08-20 11:50:16 -07:00
test_conftest.py test(proxy): stop monkeypatch.undo re-planting fixture-mocked prisma_client 2026-08-13 20:11:06 -07:00
test_cors_config.py
test_custom_proxy.py
test_dynamic_mcp_route.py fix(proxy): serve aggregate MCP endpoint on bare /mcp instead of 307-redirecting (#34845) 2026-08-14 17:04:32 -07:00
test_empty_model_list.py
test_enforce_user_param.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_fallback_management_endpoints.py
test_fastapi_offline_routes.py
test_filter_models_by_team_access_group.py
test_health_check_functions.py
test_health_check_max_tokens.py
test_init_litellm_callbacks.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_langfuse_passthrough_security.py
test_lazy_openapi_snapshot.py feat(proxy): add GET /v1/indexes to list vector store indexes (#36289) 2026-08-10 15:09:59 -07:00
test_litellm_pre_call_utils.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_max_budget_env_var.py
test_mcp_asgi_response.py
test_model_based_routing_files_batches.py
test_model_deprecations_endpoint.py refactor(proxy): tighten model deprecation typing and cover the endpoint 2026-08-10 22:58:35 +00:00
test_model_dump_with_preserved_fields.py
test_model_id_header_propagation.py feat(proxy): add x-litellm-model-name response header with deployment model string (#33698) 2026-07-17 20:29:42 -07:00
test_model_info_default_limits.py
test_model_level_guardrails.py fix(proxy): apply key/team router_settings.model_group_alias (#35486) 2026-08-03 22:09:47 +00:00
test_model_list_healthy_only.py
test_modify_response_streaming_passthrough.py
test_openai_ws_passthrough_routes.py fix(proxy): close websocket cleanly when OpenAI credentials are missing 2026-08-16 14:40:37 -07:00
test_openapi_schema_validation.py
test_plugin_routes.py
test_pricing_field_strip.py fix(guardrails): bill all chunks on mid-chunking block, strip client guardrail cost metadata, add cost map schema keys 2026-08-18 14:52:23 -07:00
test_prisma_engine_watchdog.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_prisma_migration.py fix(proxy): fail the standalone prisma migration entrypoint on migration errors (#37692) 2026-08-20 16:20:27 -07:00
test_prometheus_cleanup.py
test_provider_url_destination_guard.py fix(proxy)!: share one destination check between body and path-supplied model 2026-08-05 15:17:43 -07:00
test_proxy_cli.py fix(db): apply the configured connection params to the read replica URL (#37691) 2026-08-20 17:23:06 -07:00
test_proxy_logging_hook_detection.py test: enforce PT012 so a pytest.raises block cannot hide dead assertions (#37748) 2026-08-20 19:36:26 -07:00
test_proxy_server.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_proxy_types.py feat(proxy): add project-level ITPM and OTPM quotas 2026-08-05 12:53:22 +05:30
test_proxy_utils.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_pyroscope.py
test_read_model_list.py
test_redis_auth_cache_flag.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_response_model_sanitization.py
test_route_a2a_models.py fix(proxy): single-row read-through resyncs and reload-race hardening 2026-08-18 21:02:12 -07:00
test_route_llm_request.py test: enforce PT012 so a pytest.raises block cannot hide dead assertions (#37748) 2026-08-20 19:36:26 -07:00
test_sensitive_route_auth.py
test_shared_health_check.py
test_spend_log_cleanup.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_swagger_chat_completions.py
test_team_member_update.py
test_team_org_move.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_tools_allowlist_enforcement.py
test_update_llm_router_resilience.py feat(complexity_router): custom classifier plugins via classifier_type 'custom' (#37249) 2026-08-18 14:09:19 -07:00