scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-15 23:31:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 165
litellm/tests/test_litellm/proxy
History
Tin Chi Lo d0c24f2a29 feat(spend): fold auto-router benchmarks into a per-session rollup 
The benchmarks dashboard answered every question by scanning LiteLLM_SpendLogs at
read time: four aggregate queries per auto-router, two of them window functions
over the response JSONB, re-deriving on every page load which model the previous
turn used, how long a tier had been idle, and how big the prefix was last time.
Those are sequential facts and the request that produces them already knows all
of them, so they are now computed once, when the turn happens.

A new LiteLLM_AutoRouterSession row per (session, auto-router) carries both the
counters and the state that classifies the next turn. fold_turn is pure, so every
rate and dollar formula is unit-testable without a database, and the in-memory
queue plus background flusher follow AdaptiveRouterUpdateQueue: atomic increment
upserts, so two pods writing one session compose instead of overwriting. A pod
that has never seen a session loads its row once and classifies from memory
after, which is what keeps a session correct across a restart or a pod move.

The counters are declared once, on TurnDelta. COUNTER_FIELDS derives from that
declaration and the merge, the flush payload and the read query all build off it,
so a metric added there reaches the database and the dashboard without a second
edit. A test asserts the read query aggregates every declared counter; it caught
two that were being written on every request and read by nothing.

The read path is a single aggregate over pre-folded rows covering every
auto-router at once, and touches no per-request table at all. Rollup rows expire
on the existing spend-log retention cutoff, keyed on last activity so a live
conversation is not pruned out from under itself.

Two behaviour fixes came with the move. The turn buckets are now exhaustive: a
session's opening turn used to land in the headline turn count and in none of the
three buckets, so the bucket totals silently disagreed with the headline. And a
turn with no ephemeral cache-creation evidence now reads as the five minute tier
rather than the one hour tier, which had been the default purely because zero is
not less than zero.

Savings come from compute_savings_spend, the same primitive the usage tab uses,
so the two surfaces cannot report different numbers for the same traffic. The
baseline recorded on each row is the one that priced its turns, so the tab names
what the numbers were computed against rather than whatever the config says by
the time someone opens it.
2026-08-03 21:30:40 -07:00
..
_experimental/mcp_server fix(ui): land general login on the keys dashboard, send MCP consent to /ui/connect 2026-08-01 16:09:30 -07:00
a2a fix(a2a): reject malformed protocolVersion suffixes while keeping semver prereleases 2026-07-21 13:24:58 -07:00
agent_endpoints fix(a2a): keep config-defined agents registered and accept the documented agents: key (#35163) 2026-08-01 12:39:13 -07:00
analytics_endpoints feat(ui): split failed requests into their own series on the cache dashboard (#34862) 2026-07-29 09:48:17 -07:00
anthropic_endpoints feat: litellm oss staging (#31935) 2026-07-03 09:27:31 +05:30
auth fix(proxy): backfill null user_email on existing users during JWT auth (#34588) 2026-08-03 12:55:10 -07:00
batches_endpoints fix(proxy): resolve named credentials on provider-only batch and files calls 2026-07-28 18:57:15 -07:00
client feat(cli): read base_url from persistent config file (#35015) 2026-07-29 18:25:05 -07:00
common_utils fix(proxy): publish router_settings changes so peer pods apply them on resync 2026-07-31 22:52:00 -07:00
config_resolvers feat(proxy): add SAML 2.0 SSO for the admin UI (#31429) 2026-07-24 12:51:28 -07:00
db feat(spend): add net auto-router savings to the cost-optimization dashboard (#35521) 2026-08-04 03:10:37 +00:00
discovery_endpoints chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
enterprise_billing feat(proxy): push-based OTLP billable-request metering for enterprise deployments (#31592) 2026-07-15 12:12:52 -07:00
experimental/mcp_server
google_endpoints chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
guardrails fix(guardrails): run bedrock guardrail on MCP tool calls in during_mcp_call mode (#35149) 2026-07-31 11:33:14 -07:00
health_endpoints fix(proxy): authorize /health/test_connection against loaded deployment's team_id (VERIA-441) (#31767) 2026-07-01 17:05:49 -07:00
hooks Merge pull request #35422 from BerriAI/litellm_fix_tpm_only_dynamic_rate_limit 2026-08-01 13:02:11 -07:00
image_endpoints fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) 2026-06-09 22:10:23 +02:00
logging_endpoints feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through callbacks (#31134) 2026-06-24 15:25:10 -07:00
management_endpoints feat(spend): add net auto-router savings to the cost-optimization dashboard (#35521) 2026-08-04 03:10:37 +00:00
management_helpers feat(team): custom metadata validation hook for team create and update (#33353) 2026-08-03 18:37:45 -07:00
memory Litellm memory improvements v2 (#26541) 2026-04-25 19:03:43 -07:00
middleware feat(proxy): push-based OTLP billable-request metering for enterprise deployments (#31592) 2026-07-15 12:12:52 -07:00
openai_files_endpoint fix(proxy): honor key-level model allowlist in provider-only credential resolution 2026-07-28 21:11:50 -07:00
pass_through_endpoints test: patch unified guardrail mapping global instead of loader to fix order-dependent flake 2026-07-30 20:55:32 -07:00
policy_engine fix(policy_engine): warn that the config-defined policy reactivates when all DB versions are deleted 2026-07-30 19:48:35 -07:00
prompts style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
proxy_server fix(proxy): apply key/team router_settings.model_group_alias (#35486) 2026-08-03 22:09:47 +00:00
public_endpoints fix(agents): allow optional securityScheme fields so /public/agent_hub does not 500 (#33897) 2026-07-20 18:13:09 -07:00
rag_endpoints fix(rag): track LLM completion usage and spend for /v1/rag/query (#32438) 2026-07-17 17:45:27 +00:00
realtime_endpoints chore: litellm oss 170626 (#30637) 2026-06-17 21:11:12 -07:00
response_api_endpoints Litellm websocket improvements (#29563) 2026-06-03 11:48:35 -07:00
shutdown feat(proxy): native /health/drain preStop hook for graceful shutdown (#29439) 2026-06-02 16:30:44 -07:00
spend_tracking feat(spend): fold auto-router benchmarks into a per-session rollup 2026-08-03 21:30:40 -07:00
test_configs
types_utils fix(proxy): resolve router_settings.plugins dotted paths and load plugins from installed packages (#33644) 2026-07-17 11:23:18 -07:00
ui_crud_endpoints feat(teams): apply default organization to new teams from default team settings (#35540) 2026-08-03 12:57:12 -07:00
utils fix(spend): bound each spend-log write statement by payload bytes (#34956) 2026-08-01 13:53:36 -07:00
vector_store_endpoints test(vector_store): update credential resolution assertion for team_id kwarg 2026-07-18 18:41:12 +00:00
video_endpoints test(videos): add 1:1 test file scaffold for videos component paths (#30631) 2026-06-29 09:12:51 +05:30
__init__.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
conftest.py fix(proxy): avoid DB outage during planned RDS IAM rotation (#34749) 2026-07-28 13:27:50 -07:00
test_aiohttp_cleanup_closed.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_aiohttp_session_recovery.py fix: use AsyncMock for concurrent test consistency 2026-03-18 00:54:23 +00:00
test_api_key_masking_in_errors.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_audio_speech_prometheus_hooks.py fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) 2026-06-09 22:10:23 +02:00
test_batch_expiry.py fix(proxy): strip LiteLLM policy tracking from OpenAI batch metadata (#28425) 2026-05-26 11:35:42 -07:00
test_batch_metadata_none_fix.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_batch_retrieve_bedrock.py fix(proxy/batches): forward model to retrieve_batch for bedrock 2026-04-29 22:48:03 +02:00
test_batch_x_litellm_model_encoding.py test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) 2026-06-16 10:20:59 -07:00
test_blocked_response_usage.py feat: litellm oss staging (#31935) 2026-07-03 09:27:31 +05:30
test_budget_reservation.py revert(proxy)!: stop enforcing user budget on team keys (#35271) 2026-07-30 20:19:49 +00:00
test_caching_routes.py Litellm oss staging 040626 (#29671) 2026-06-04 11:07:20 -07:00
test_chat_completion_metadata.py
test_common_request_processing.py fix(proxy): apply key/team router_settings.model_group_alias (#35486) 2026-08-03 22:09:47 +00:00
test_component_allowlists.py fix(gateway): keep the Prometheus /metrics Mount in the gateway route trim (#32317) 2026-07-07 18:36:38 +03:00
test_cors_config.py refactor: extract _get_cors_config() for testability, fix no-op CORS tests 2026-04-11 22:24:04 +05:30
test_custom_proxy.py
test_dynamic_mcp_route.py Litellm oss staging 040626 (#29671) 2026-06-04 11:07:20 -07:00
test_empty_model_list.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_enforce_user_param.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_fallback_management_endpoints.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_fastapi_offline_routes.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_filter_models_by_team_access_group.py fix: resolve access group names in _filter_models_by_team_id (#25224) 2026-05-01 11:55:36 +05:30
test_health_check_functions.py feat(proxy): skip disable_background_health_check models on GET /health when flag set (#27716) 2026-05-13 09:49:05 -07:00
test_health_check_max_tokens.py fix: skip health check for semantic auto_router deployments (#31668) 2026-06-29 19:47:20 -07:00
test_langfuse_passthrough_security.py chore(callbacks): guard dynamic integration hosts 2026-04-30 14:27:19 -07:00
test_lazy_openapi_snapshot.py [Fix] Tests: Move Misplaced Import in Lazy OpenAPI Snapshot Test 2026-05-01 14:24:53 -07:00
test_litellm_pre_call_utils.py fix(proxy): redact credential headers from request logging copies (#35678) 2026-08-04 00:07:05 +00:00
test_max_budget_env_var.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_mcp_asgi_response.py fix(mcp): surface upstream 401 for token-forwarding MCP servers (#27847) 2026-05-13 12:03:36 -07:00
test_model_based_routing_files_batches.py test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) 2026-06-16 10:20:59 -07:00
test_model_dump_with_preserved_fields.py
test_model_id_header_propagation.py feat(proxy): add x-litellm-model-name response header with deployment model string (#33698) 2026-07-17 20:29:42 -07:00
test_model_info_default_limits.py fix(proxy): align /v1/model/info with router deployments (#30025) 2026-06-10 19:38:21 -07:00
test_model_level_guardrails.py fix(proxy): apply key/team router_settings.model_group_alias (#35486) 2026-08-03 22:09:47 +00:00
test_model_list_healthy_only.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_modify_response_streaming_passthrough.py fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough (#30800) 2026-06-18 23:29:08 -07:00
test_openapi_schema_validation.py fix(proxy): preserve HTTP operations when injecting WebSocket stubs into OpenAPI schema 2026-05-06 00:28:42 +02:00
test_plugin_routes.py feat: litellm plugin architecture v2 (#30688) 2026-06-20 20:37:22 -07:00
test_pricing_field_strip.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_prometheus_cleanup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_provider_url_destination_guard.py chore(proxy): clean up request parameter validation and provider destination handling (#34189) 2026-07-22 00:57:58 +00:00
test_proxy_cli.py feat(proxy): bound DB statement and lock time via general_settings (#35496) 2026-08-01 13:54:17 -07:00
test_proxy_logging_hook_detection.py fix(proxy): recognize inherited apply_guardrail overrides and keep masking guardrails on their own stream hook 2026-07-30 15:33:49 -07:00
test_proxy_server.py Merge pull request #35423 from BerriAI/litellm_/mock-testing-feature-flag-6e30ad 2026-07-31 20:03:45 -07:00
test_proxy_types.py fix(proxy): allow /key/update to identify the key by key_alias (#34851) 2026-07-29 09:48:08 -07:00
test_proxy_utils.py feat(guardrails): scan and mask MCP tool results via post_mcp_call (#35155) 2026-07-30 14:10:26 -07:00
test_pyroscope.py Implement normalize_nonempty_secret_str function to trim whitespace from secrets and treat empty values as unset. Update proxy_server to use this function for Grafana credentials. Enhance tests to validate the new normalization behavior. 2026-05-04 18:17:31 +00:00
test_read_model_list.py feat: add minimal rust router + axum ai-gateway calling router.realtime (2/2) (#31135) 2026-06-23 19:16:34 -07:00
test_redis_auth_cache_flag.py fix(proxy): share CLI SSO login sessions across workers without enable_redis_auth_cache (#33261) 2026-07-22 10:29:34 -07:00
test_response_model_sanitization.py perf: eliminate per-request callback scanning on proxy hot path (#27858) 2026-05-14 09:28:31 -07:00
test_route_a2a_models.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_route_llm_request.py fix(ci): let the E2E proxy accept the mock testing params its suite sends 2026-08-01 14:57:42 -07:00
test_sensitive_route_auth.py chore(proxy): guard sensitive public endpoints 2026-04-30 11:52:47 -07:00
test_shared_health_check.py feat(proxy): skip disable_background_health_check models on GET /health when flag set (#27716) 2026-05-13 09:49:05 -07:00
test_spend_log_cleanup.py fix(proxy): roll up tool spend daily instead of scanning SpendLogs 2026-07-25 21:52:58 -07:00
test_swagger_chat_completions.py fix(proxy): avoid misleading multi-method operation ids 2026-04-30 20:44:14 -07:00
test_team_member_update.py feat(ui): add budget duration to edit team member form (#29717) 2026-06-06 17:24:55 -07:00
test_team_org_move.py fix(team_endpoints): auto-add SSO team members to org on move (proxy admin only) (#26377) 2026-04-24 08:36:25 -07:00
test_tools_allowlist_enforcement.py fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI (#32258) 2026-07-06 17:34:27 -07:00
test_update_llm_router_resilience.py fix(proxy): stop model writes 500ing on another pod's delete (#35400) 2026-07-31 18:10:48 -07:00