Ishaan Jaffer
017c6a96b9
fix(proxy/utils): fire-and-forget eval spend-log patch to remove 500ms sleep from response path
2026-04-22 20:45:33 -07:00
Ishaan Jaffer
4fa4cd8a08
fix(evals-ui): align criterion name+weight inputs with flex-end; add /evals route
2026-04-22 20:34:09 -07:00
Ishaan Jaffer
35a5b4dd80
fix(proxy/utils): extract spend-log patch to module-level helper with retry
2026-04-22 19:55:46 -07:00
Ishaan Jaffer
a9f699c683
fix(eval_engine): use llm_router.acompletion for retry, not raw litellm
2026-04-22 19:55:46 -07:00
Ishaan Jaffer
7bae837c17
fix(eval_registry): pass criteria list directly to Prisma Json field, not json.dumps
2026-04-22 19:55:46 -07:00
Ishaan Jaffer
69c9c46fac
fix(eval_utils): harden verdict parsing, fix passed field, fix sync wrapper
2026-04-22 19:55:46 -07:00
Ishaan Jaffer
87b718650a
wire EvalViewer into log detail drawer below guardrail section
2026-04-22 17:14:52 -07:00
Ishaan Jaffer
7971582d51
add EvalViewer component for structured eval display in log detail drawer
2026-04-22 17:14:49 -07:00
Ishaan Jaffer
8d5f43e806
fetch available models in EvalsCatalog on mount so judge model dropdown is populated
...
Without this, navigating directly to Evals without visiting Models first
leaves availableModels empty and the judge model select unusable.
2026-04-22 17:11:18 -07:00
Ishaan Jaffer
3263bb61d9
fix evals API URL prefix: /v1/evals/config -> /litellm_evals/config
2026-04-22 17:11:14 -07:00
Ishaan Jaffer
0a4a453a50
wire page == "evals" routing in page.tsx to EvalsCatalog component
2026-04-22 17:11:11 -07:00
Ishaan Jaffer
cf23b2f99a
update tsconfig jsx to react-jsx for Next.js 16 compatibility
2026-04-22 17:11:08 -07:00
Ishaan Jaffer
3890cb898d
add eval CRUD and agent attachment functions to networking.ts
2026-04-22 16:51:17 -07:00
Ishaan Jaffer
ae144728d8
add Evals [Beta] tab to agent detail page
2026-04-22 16:51:17 -07:00
Ishaan Jaffer
897f65cd4f
add Evals [Beta] nav item after Guardrails in sidebar
2026-04-22 16:51:17 -07:00
Ishaan Jaffer
69c1eab962
add Evals [Beta] catalog page with create/delete UI
2026-04-22 16:51:17 -07:00
Ishaan Jaffer
e84b2bc07d
add eval_information to spend log metadata
2026-04-22 16:51:12 -07:00
Ishaan Jaffer
dfe3414b54
add StandardLoggingEvalInformation to SpendLogsMetadata
2026-04-22 16:51:12 -07:00
Ishaan Jaffer
ab07c03556
wire eval gate into async post_call_success_hook
2026-04-22 16:51:12 -07:00
Ishaan Jaffer
1a138c5da8
register litellm_evals_router in proxy_server
2026-04-22 16:51:12 -07:00
Ishaan Jaffer
da527262d1
add eval_engine: run evals for agent with retry-on-fail loop
2026-04-22 16:51:07 -07:00
Ishaan Jaffer
4943f7de16
add eval_endpoints: CRUD + attach/detach API at /litellm_evals
2026-04-22 16:51:07 -07:00
Ishaan Jaffer
a6bcf2e916
add eval_registry: DB ops for evals CRUD and agent attachment
2026-04-22 16:51:07 -07:00
Ishaan Jaffer
ef284173eb
add litellm/proxy/eval_management package init
2026-04-22 16:51:07 -07:00
Ishaan Jaffer
83aa8b9c84
add migration for LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable
2026-04-22 16:51:01 -07:00
Ishaan Jaffer
f233774225
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to proxy-extras schema.prisma
2026-04-22 16:51:01 -07:00
Ishaan Jaffer
3b5077cbea
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to proxy schema.prisma
2026-04-22 16:51:01 -07:00
Ishaan Jaffer
67d2c8b7f4
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to root schema.prisma
2026-04-22 16:51:01 -07:00
Ishaan Jaffer
e62817a48e
export aevaluate and evaluate from litellm.main
2026-04-22 16:50:56 -07:00
Ishaan Jaffer
9a199f4c8b
add litellm.aevaluate / evaluate SDK primitive for LLM-judge evals
2026-04-22 16:50:56 -07:00
Ishaan Jaffer
9cf52d8006
add EvalFailedError exception with status_code 422
2026-04-22 16:50:49 -07:00
Ishaan Jaffer
d33e2206ad
add EvalCriterion, EvalResult, EvalVerdict, StandardLoggingEvalInformation types
2026-04-22 16:50:46 -07:00
yuneng-jiang
26fcbc93e5
Merge pull request #26044 from BerriAI/litellm_internal_staging
...
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
CodSpeed Benchmarks / benchmarks (push) Has been cancelled
Helm unit test / unit-test (push) Has been cancelled
Read Version from pyproject.toml / read-version (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
[Infra] Promote staging to main
2026-04-18 19:33:24 -07:00
ishaan-berri
2f22a1293e
bump litellm-proxy-extras to 0.4.67 ( #26043 )
...
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
Unit Tests: Security / security (push) Waiting to run
* bump litellm-proxy-extras version to 0.4.67
* bump litellm-proxy-extras pin to 0.4.67 in litellm pyproject
* regenerate uv.lock for litellm-proxy-extras 0.4.67
* bump litellm-enterprise version to 0.1.38
* bump litellm-enterprise pin to 0.1.38 in litellm pyproject
* regenerate uv.lock for litellm-enterprise 0.1.38
2026-04-18 19:03:56 -07:00
yuneng-jiang
9e77e25107
Merge pull request #26038 from BerriAI/yj_bump_apr18
...
bump: version 1.83.9 → 1.83.10
2026-04-18 18:54:11 -07:00
Yuneng Jiang
49ba6b8160
add uv lock
2026-04-18 18:43:09 -07:00
yuneng-jiang
e16bd158c3
Merge pull request #26033 from BerriAI/yj_ui_build_apr18
...
[Infra] Build UI
2026-04-18 18:37:01 -07:00
Yuneng Jiang
4d63a1367e
bump: version 1.83.9 → 1.83.10
2026-04-18 18:31:24 -07:00
Yuneng Jiang
0278d73cd9
Merge remote-tracking branch 'origin/litellm_internal_staging' into yj_ui_build_apr18
2026-04-18 16:48:19 -07:00
Yuneng Jiang
aab3ef8988
chore: update Next.js build artifacts (2026-04-18 23:46 UTC, node v22.16.0)
2026-04-18 16:46:25 -07:00
Yuneng Jiang
ba24e4a1b3
remove next env
2026-04-18 16:45:32 -07:00
ryan-crabbe-berri
67bf18dfe0
Merge pull request #25989 from BerriAI/litellm_feat-multi_threshold_budget_alerts
...
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
Unit Tests: Security / security (push) Waiting to run
feat: configurable multi-threshold budget alerts for virtual keys
2026-04-18 16:21:29 -07:00
Ryan Crabbe
c8b7c1bafa
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_feat-multi_threshold_budget_alerts
2026-04-18 15:26:02 -07:00
Ryan Crabbe
eb6fd98611
Merge remote-tracking branch 'origin/main' into litellm_feat-multi_threshold_budget_alerts
2026-04-18 14:54:26 -07:00
ishaan-berri
ecff06df65
Merge pull request #26032 from BerriAI/litellm_mcp_pkce_fix_v2
...
fix(mcp): restore PKCE-triggering 401 when no stored per-user token exists
2026-04-18 14:52:31 -07:00
Yuneng Jiang
de790fd273
chore: update Next.js build artifacts (2026-04-18 21:49 UTC, node v22.16.0)
2026-04-18 14:49:27 -07:00
shin-berri
85b1b93661
Merge pull request #26022 from BerriAI/litellm_/reverent-kirch-7cf0a6
...
[Infra] Bump proxy dependencies and raise minimum Python to 3.10
2026-04-18 14:44:24 -07:00
yuneng-jiang
e69051916e
Merge pull request #25983 from BerriAI/litellm_yj_apr17
...
[Infra] Merge dev branch
2026-04-18 14:43:04 -07:00
yuneng-jiang
63313bcd77
Merge pull request #25994 from BerriAI/litellm_project_rate_limiting
...
fix: enforce project-level model-specific rate limits in parallel_req…
2026-04-18 14:31:08 -07:00
Ryan Crabbe
0a4b02fe76
fix: tighten recipient_emails guard to reject empty list
...
send_max_budget_alert_email previously guarded with `is not None`, which
accepts `[]` and then crashes on `recipient_emails[0]` inside
_get_email_params. The current caller (_handle_multi_threshold_max_budget_alert)
already filters empty lists upstream, but the public method signature makes
no such guarantee — a future caller passing [] would hit IndexError.
Switch to truthiness so both None and [] fall through to the single-recipient
path.
2026-04-18 14:07:18 -07:00
