scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-06 02:48:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103
litellm/tests/test_litellm/proxy/management_endpoints
History
tin-berri 4e48d74455
feat(shadow_eval): measure both arms' cost so a job reports what the router would have saved (#38631) 
The attempt row now prices the real arm (the payload's response_cost plus its own
routing classifier when it routed) beside the shadow arm (completion plus the
classifier cost the routing decision writes back), and flags turns litellm's
response cache served. A per-leg funnel table counts the eligible requests that
produced no row (lost the sampling dice, unjudgeable shape, concurrency shed),
so results can weigh judged rows against the traffic they stand for. Job results
gain per-slice and overall arm spends plus the coverage counts, the budget gates
charge the shadow arm's classifier spend against max_budget, and the dashboard
shows the measured cost comparison beside the win rate

Resolves LIT-6358

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 15:13:19 -07:00
..
management_v1 feat(proxy): add paginated GET /public/v1/model_hub (#38636) 2026-08-28 10:02:59 -07:00
policy_endpoints style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
scim fix(scim): apply default_team_params (incl. models) to SCIM-created teams (#38433) 2026-08-26 20:49:21 -07:00
search_endpoints fix(proxy): sync search tools into the router on management writes 2026-08-26 11:46:54 -07:00
usage_endpoints test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_access_group_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_access_group_management.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_activity_tenant_scoping.py fix(proxy): deny agent access when key and team grants resolve to nothing (#36221) 2026-08-07 20:44:11 +00:00
test_auto_router_endpoints.py feat(shadow_eval): measure both arms' cost so a job reports what the router would have saved (#38631) 2026-08-28 15:13:19 -07:00
test_budget_endpoints.py test(budget): annotate the new locals with Final 2026-08-26 17:55:33 -07:00
test_cache_settings_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_callback_management_endpoints.py feat(newrelic): per-team New Relic trace routing via team callbacks (#37603) 2026-08-22 19:13:48 -07:00
test_common_daily_activity.py feat(spend): report prompt caching savings as total and gateway-attributed (#38134) 2026-08-28 00:19:06 -07:00
test_common_utils.py test: merge three stranded twins into the files that shadow them (#37600) 2026-08-20 14:25:23 -07:00
test_compliance_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_config_override_endpoints.py fix(audit): label vault POST as updated when DB row exists 2026-05-01 02:44:47 +00:00
test_coordination_redis_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_cost_estimate_endpoint.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_cost_tracking_settings.py test(cost-estimate): keep the new tests inside the test-quality budgets 2026-08-26 00:32:25 -07:00
test_credential_migration.py feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration (#31215) 2026-06-29 20:14:22 +02:00
test_customer_budget.py feat(proxy): type Customer Management response_model for OpenAPI coverage (#31043) 2026-06-30 09:58:01 -07:00
test_customer_endpoints.py fix(proxy): registry caches stop per-request tag and end-user Postgres reads in auth (#36801) 2026-08-17 18:52:13 +00:00
test_delete_callbacks_endpoint.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_delete_verification_tokens_failed.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_encryption_endpoints.py feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration (#31215) 2026-06-29 20:14:22 +02:00
test_entraid_app_roles.py fix(ui_sso): resolve highest privilege Entra app role, not first in claim (#36728) 2026-08-27 10:26:45 -07:00
test_gateway_request_endpoints.py feat(sgr): make the gateway middleware the source of truth for successful requests (#35717) 2026-08-05 12:40:47 -07:00
test_internal_user_endpoints.py refactor(repositories): type prisma table access with one generic protocol 2026-08-25 12:14:17 +00:00
test_key_management_endpoints.py fix(proxy): reset a key's budget-window counters on spend reset (#38686) 2026-08-28 15:13:03 -07:00
test_mcp_management_endpoints.py feat(mcp): let a resolved OAuth token target a custom upstream header (#38456) 2026-08-27 14:32:01 -07:00
test_model_management_endpoints.py feat(auto_router): write and preview the classifier prompt an edited tier set sends (#38605) 2026-08-28 12:36:52 -07:00
test_org_admin_team_access.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_organization_endpoints.py refactor(repositories): type prisma table access with one generic protocol 2026-08-25 12:14:17 +00:00
test_policy_endpoints.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_project_org_authz.py fix(tests): use canonical litellm_enterprise import path (#27699) 2026-05-12 12:32:57 -07:00
test_ptu_model_settings.py test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
test_router_settings_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_saml_sso.py feat(proxy): add SAML 2.0 SSO for the admin UI (#31429) 2026-07-24 12:51:28 -07:00
test_tag_management_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_team_callback_endpoints.py refactor(repositories): type prisma table access with one generic protocol 2026-08-25 12:14:17 +00:00
test_team_default_params.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_team_endpoints.py test(team): fake the budget table instead of patching new_budget and update_budget 2026-08-26 14:00:45 -07:00
test_team_model_alias_merge.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_tool_management_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_ui_sso.py fix(test): keep the mutmut sentinel out of the cleared environment 2026-08-26 08:19:56 -07:00
test_workflow_management_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00