Commit graph

37514 commits

Author SHA1 Message Date
Ishaan Jaffer
3263bb61d9
fix evals API URL prefix: /v1/evals/config -> /litellm_evals/config 2026-04-22 17:11:14 -07:00
Ishaan Jaffer
0a4a453a50
wire page == "evals" routing in page.tsx to EvalsCatalog component 2026-04-22 17:11:11 -07:00
Ishaan Jaffer
cf23b2f99a
update tsconfig jsx to react-jsx for Next.js 16 compatibility 2026-04-22 17:11:08 -07:00
Ishaan Jaffer
3890cb898d
add eval CRUD and agent attachment functions to networking.ts 2026-04-22 16:51:17 -07:00
Ishaan Jaffer
ae144728d8
add Evals [Beta] tab to agent detail page 2026-04-22 16:51:17 -07:00
Ishaan Jaffer
897f65cd4f
add Evals [Beta] nav item after Guardrails in sidebar 2026-04-22 16:51:17 -07:00
Ishaan Jaffer
69c1eab962
add Evals [Beta] catalog page with create/delete UI 2026-04-22 16:51:17 -07:00
Ishaan Jaffer
e84b2bc07d
add eval_information to spend log metadata 2026-04-22 16:51:12 -07:00
Ishaan Jaffer
dfe3414b54
add StandardLoggingEvalInformation to SpendLogsMetadata 2026-04-22 16:51:12 -07:00
Ishaan Jaffer
ab07c03556
wire eval gate into async post_call_success_hook 2026-04-22 16:51:12 -07:00
Ishaan Jaffer
1a138c5da8
register litellm_evals_router in proxy_server 2026-04-22 16:51:12 -07:00
Ishaan Jaffer
da527262d1
add eval_engine: run evals for agent with retry-on-fail loop 2026-04-22 16:51:07 -07:00
Ishaan Jaffer
4943f7de16
add eval_endpoints: CRUD + attach/detach API at /litellm_evals 2026-04-22 16:51:07 -07:00
Ishaan Jaffer
a6bcf2e916
add eval_registry: DB ops for evals CRUD and agent attachment 2026-04-22 16:51:07 -07:00
Ishaan Jaffer
ef284173eb
add litellm/proxy/eval_management package init 2026-04-22 16:51:07 -07:00
Ishaan Jaffer
83aa8b9c84
add migration for LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable 2026-04-22 16:51:01 -07:00
Ishaan Jaffer
f233774225
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to proxy-extras schema.prisma 2026-04-22 16:51:01 -07:00
Ishaan Jaffer
3b5077cbea
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to proxy schema.prisma 2026-04-22 16:51:01 -07:00
Ishaan Jaffer
67d2c8b7f4
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to root schema.prisma 2026-04-22 16:51:01 -07:00
Ishaan Jaffer
e62817a48e
export aevaluate and evaluate from litellm.main 2026-04-22 16:50:56 -07:00
Ishaan Jaffer
9a199f4c8b
add litellm.aevaluate / evaluate SDK primitive for LLM-judge evals 2026-04-22 16:50:56 -07:00
Ishaan Jaffer
9cf52d8006
add EvalFailedError exception with status_code 422 2026-04-22 16:50:49 -07:00
Ishaan Jaffer
d33e2206ad
add EvalCriterion, EvalResult, EvalVerdict, StandardLoggingEvalInformation types 2026-04-22 16:50:46 -07:00
yuneng-jiang
26fcbc93e5
Merge pull request #26044 from BerriAI/litellm_internal_staging
Some checks failed
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
CodSpeed Benchmarks / benchmarks (push) Has been cancelled
Helm unit test / unit-test (push) Has been cancelled
Read Version from pyproject.toml / read-version (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
[Infra] Promote staging to main
2026-04-18 19:33:24 -07:00
ishaan-berri
2f22a1293e
bump litellm-proxy-extras to 0.4.67 (#26043)
Some checks are pending
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
Unit Tests: Security / security (push) Waiting to run
* bump litellm-proxy-extras version to 0.4.67

* bump litellm-proxy-extras pin to 0.4.67 in litellm pyproject

* regenerate uv.lock for litellm-proxy-extras 0.4.67

* bump litellm-enterprise version to 0.1.38

* bump litellm-enterprise pin to 0.1.38 in litellm pyproject

* regenerate uv.lock for litellm-enterprise 0.1.38
2026-04-18 19:03:56 -07:00
yuneng-jiang
9e77e25107
Merge pull request #26038 from BerriAI/yj_bump_apr18
bump: version 1.83.9 → 1.83.10
2026-04-18 18:54:11 -07:00
Yuneng Jiang
49ba6b8160
add uv lock 2026-04-18 18:43:09 -07:00
yuneng-jiang
e16bd158c3
Merge pull request #26033 from BerriAI/yj_ui_build_apr18
[Infra] Build UI
2026-04-18 18:37:01 -07:00
Yuneng Jiang
4d63a1367e
bump: version 1.83.9 → 1.83.10 2026-04-18 18:31:24 -07:00
Yuneng Jiang
0278d73cd9
Merge remote-tracking branch 'origin/litellm_internal_staging' into yj_ui_build_apr18 2026-04-18 16:48:19 -07:00
Yuneng Jiang
aab3ef8988
chore: update Next.js build artifacts (2026-04-18 23:46 UTC, node v22.16.0) 2026-04-18 16:46:25 -07:00
Yuneng Jiang
ba24e4a1b3
remove next env 2026-04-18 16:45:32 -07:00
ryan-crabbe-berri
67bf18dfe0
Merge pull request #25989 from BerriAI/litellm_feat-multi_threshold_budget_alerts
Some checks are pending
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
Unit Tests: Security / security (push) Waiting to run
feat: configurable multi-threshold budget alerts for virtual keys
2026-04-18 16:21:29 -07:00
Ryan Crabbe
c8b7c1bafa
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_feat-multi_threshold_budget_alerts 2026-04-18 15:26:02 -07:00
Ryan Crabbe
eb6fd98611
Merge remote-tracking branch 'origin/main' into litellm_feat-multi_threshold_budget_alerts 2026-04-18 14:54:26 -07:00
ishaan-berri
ecff06df65
Merge pull request #26032 from BerriAI/litellm_mcp_pkce_fix_v2
fix(mcp): restore PKCE-triggering 401 when no stored per-user token exists
2026-04-18 14:52:31 -07:00
Yuneng Jiang
de790fd273
chore: update Next.js build artifacts (2026-04-18 21:49 UTC, node v22.16.0) 2026-04-18 14:49:27 -07:00
shin-berri
85b1b93661
Merge pull request #26022 from BerriAI/litellm_/reverent-kirch-7cf0a6
[Infra] Bump proxy dependencies and raise minimum Python to 3.10
2026-04-18 14:44:24 -07:00
yuneng-jiang
e69051916e
Merge pull request #25983 from BerriAI/litellm_yj_apr17
[Infra] Merge dev branch
2026-04-18 14:43:04 -07:00
yuneng-jiang
63313bcd77
Merge pull request #25994 from BerriAI/litellm_project_rate_limiting
fix: enforce project-level model-specific rate limits in parallel_req…
2026-04-18 14:31:08 -07:00
Ryan Crabbe
0a4b02fe76
fix: tighten recipient_emails guard to reject empty list
send_max_budget_alert_email previously guarded with `is not None`, which
accepts `[]` and then crashes on `recipient_emails[0]` inside
_get_email_params. The current caller (_handle_multi_threshold_max_budget_alert)
already filters empty lists upstream, but the public method signature makes
no such guarantee — a future caller passing [] would hit IndexError.

Switch to truthiness so both None and [] fall through to the single-recipient
path.
2026-04-18 14:07:18 -07:00
Ishaan Jaffer
b7813aad41
fix(mcp): restore PKCE-triggering 401 when no stored per-user token exists
Per-user OAuth MCP requests now only skip pre-emptive 401 when a stored token is available, preserving token-reuse behavior while restoring fast PKCE kickoff for first-time or missing-token users.
2026-04-18 14:04:22 -07:00
Ryan Crabbe
029d9bcfc7
refactor: inline _parse_email_list in auth_checks to drop enterprise dep
The lazy import from litellm_enterprise inside _normalize_alert_emails
coupled the core proxy auth path to an optional package. Core should not
depend on enterprise, even lazily — it hides the dependency from static
analysis and inverts the intended layering.

Duplicate the 7-line parser locally. It's pure and unlikely to drift; the
enterprise copy stays where it is for its own callers.
2026-04-18 13:51:00 -07:00
Ryan Crabbe
8b86a3b041
fix: normalize alert-email configs at merge boundary
_merge_budget_alert_email_configs previously called list() directly on each
threshold's value, which raised TypeError on null YAML values and silently
split bare strings into single characters. Both are reachable from user-
supplied global config and per-key metadata, so the crash could fire on
every authenticated request once the metadata was in place.

Route both inputs through a _normalize_alert_emails helper that delegates
to the existing _parse_email_list parser (lazy-imported, matching the
enterprise import pattern used elsewhere in proxy/). The merge body keeps
its tight Dict[str, List[str]] contract.
2026-04-18 13:48:48 -07:00
Ryan Crabbe
09a524a351
fix: narrow CallInfo.max_budget_alert_emails to Dict[str, List[str]]
The Union[str, List[str]] value type was speculative — _merge_budget_alert_email_configs
always returns List[str] values, and no caller produces bare strings. Narrowing to
match the runtime guarantee resolves a mypy invariance error at auth_checks.py:3021
without adding casts or Mapping covariance.
2026-04-18 13:39:33 -07:00
shivam
dbf4f9637f
Merge remote-tracking branch 'upstream/litellm_internal_staging' into litellm_project_rate_limiting 2026-04-18 13:26:24 -07:00
Yuneng Jiang
f483f1e800
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_yj_apr17 2026-04-18 13:19:16 -07:00
Yuneng Jiang
ec590f938b
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/reverent-kirch-7cf0a6 2026-04-18 13:18:45 -07:00
yuneng-jiang
643e941b64
Merge pull request #26026 from BerriAI/litellm_fixPrometheusHelpersPackageCollision
[Fix] Resolve prometheus_helpers file/package shadow breaking /global/spend/logs
2026-04-18 13:18:04 -07:00
Yuneng Jiang
cfdf893226
[Fix] Merge prometheus_helpers.py into prometheus_helpers/__init__.py to resolve file/package collision
A previous refactor added `litellm/integrations/prometheus_helpers.py` as a
sibling to the existing `litellm/integrations/prometheus_helpers/` directory
(which contains `prometheus_api.py` and has no `__init__.py`). The file
shadowed the namespace-package directory, so any deferred
`from litellm.integrations.prometheus_helpers.prometheus_api import ...`
raised `ModuleNotFoundError: 'litellm.integrations.prometheus_helpers' is
not a package` at request time.

Two runtime call sites hit that path:
- /global/spend/logs (spend_management_endpoints.py) returned plain-text 500
  "Internal Server Error" for every call, breaking the Admin UI Usage tab
  and programmatic consumers.
- SlackAlerting.send_fallback_stats_from_prometheus silently failed inside
  its own try/except.

Fix: move prometheus_helpers.py content into prometheus_helpers/__init__.py
and delete the stray .py. The directory becomes a regular package, so both
the package-root import (from ...prometheus_helpers import X) and the
submodule import (from ...prometheus_helpers.prometheus_api import X)
resolve correctly. No call sites change.
2026-04-18 13:04:32 -07:00