Commit graph

37509 commits

Author SHA1 Message Date
Ishaan Jaffer
897f65cd4f
add Evals [Beta] nav item after Guardrails in sidebar 2026-04-22 16:51:17 -07:00
Ishaan Jaffer
69c1eab962
add Evals [Beta] catalog page with create/delete UI 2026-04-22 16:51:17 -07:00
Ishaan Jaffer
e84b2bc07d
add eval_information to spend log metadata 2026-04-22 16:51:12 -07:00
Ishaan Jaffer
dfe3414b54
add StandardLoggingEvalInformation to SpendLogsMetadata 2026-04-22 16:51:12 -07:00
Ishaan Jaffer
ab07c03556
wire eval gate into async post_call_success_hook 2026-04-22 16:51:12 -07:00
Ishaan Jaffer
1a138c5da8
register litellm_evals_router in proxy_server 2026-04-22 16:51:12 -07:00
Ishaan Jaffer
da527262d1
add eval_engine: run evals for agent with retry-on-fail loop 2026-04-22 16:51:07 -07:00
Ishaan Jaffer
4943f7de16
add eval_endpoints: CRUD + attach/detach API at /litellm_evals 2026-04-22 16:51:07 -07:00
Ishaan Jaffer
a6bcf2e916
add eval_registry: DB ops for evals CRUD and agent attachment 2026-04-22 16:51:07 -07:00
Ishaan Jaffer
ef284173eb
add litellm/proxy/eval_management package init 2026-04-22 16:51:07 -07:00
Ishaan Jaffer
83aa8b9c84
add migration for LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable 2026-04-22 16:51:01 -07:00
Ishaan Jaffer
f233774225
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to proxy-extras schema.prisma 2026-04-22 16:51:01 -07:00
Ishaan Jaffer
3b5077cbea
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to proxy schema.prisma 2026-04-22 16:51:01 -07:00
Ishaan Jaffer
67d2c8b7f4
add LiteLLM_EvalsTable and LiteLLM_AgentEvalsTable to root schema.prisma 2026-04-22 16:51:01 -07:00
Ishaan Jaffer
e62817a48e
export aevaluate and evaluate from litellm.main 2026-04-22 16:50:56 -07:00
Ishaan Jaffer
9a199f4c8b
add litellm.aevaluate / evaluate SDK primitive for LLM-judge evals 2026-04-22 16:50:56 -07:00
Ishaan Jaffer
9cf52d8006
add EvalFailedError exception with status_code 422 2026-04-22 16:50:49 -07:00
Ishaan Jaffer
d33e2206ad
add EvalCriterion, EvalResult, EvalVerdict, StandardLoggingEvalInformation types 2026-04-22 16:50:46 -07:00
yuneng-jiang
26fcbc93e5
Merge pull request #26044 from BerriAI/litellm_internal_staging
Some checks failed
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
CodSpeed Benchmarks / benchmarks (push) Has been cancelled
Helm unit test / unit-test (push) Has been cancelled
Read Version from pyproject.toml / read-version (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
[Infra] Promote staging to main
2026-04-18 19:33:24 -07:00
ishaan-berri
2f22a1293e
bump litellm-proxy-extras to 0.4.67 (#26043)
Some checks are pending
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
Unit Tests: Security / security (push) Waiting to run
* bump litellm-proxy-extras version to 0.4.67

* bump litellm-proxy-extras pin to 0.4.67 in litellm pyproject

* regenerate uv.lock for litellm-proxy-extras 0.4.67

* bump litellm-enterprise version to 0.1.38

* bump litellm-enterprise pin to 0.1.38 in litellm pyproject

* regenerate uv.lock for litellm-enterprise 0.1.38
2026-04-18 19:03:56 -07:00
yuneng-jiang
9e77e25107
Merge pull request #26038 from BerriAI/yj_bump_apr18
bump: version 1.83.9 → 1.83.10
2026-04-18 18:54:11 -07:00
Yuneng Jiang
49ba6b8160
add uv lock 2026-04-18 18:43:09 -07:00
yuneng-jiang
e16bd158c3
Merge pull request #26033 from BerriAI/yj_ui_build_apr18
[Infra] Build UI
2026-04-18 18:37:01 -07:00
Yuneng Jiang
4d63a1367e
bump: version 1.83.9 → 1.83.10 2026-04-18 18:31:24 -07:00
Yuneng Jiang
0278d73cd9
Merge remote-tracking branch 'origin/litellm_internal_staging' into yj_ui_build_apr18 2026-04-18 16:48:19 -07:00
Yuneng Jiang
aab3ef8988
chore: update Next.js build artifacts (2026-04-18 23:46 UTC, node v22.16.0) 2026-04-18 16:46:25 -07:00
Yuneng Jiang
ba24e4a1b3
remove next env 2026-04-18 16:45:32 -07:00
ryan-crabbe-berri
67bf18dfe0
Merge pull request #25989 from BerriAI/litellm_feat-multi_threshold_budget_alerts
Some checks are pending
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
Unit Tests: Security / security (push) Waiting to run
feat: configurable multi-threshold budget alerts for virtual keys
2026-04-18 16:21:29 -07:00
Ryan Crabbe
c8b7c1bafa
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_feat-multi_threshold_budget_alerts 2026-04-18 15:26:02 -07:00
Ryan Crabbe
eb6fd98611
Merge remote-tracking branch 'origin/main' into litellm_feat-multi_threshold_budget_alerts 2026-04-18 14:54:26 -07:00
ishaan-berri
ecff06df65
Merge pull request #26032 from BerriAI/litellm_mcp_pkce_fix_v2
fix(mcp): restore PKCE-triggering 401 when no stored per-user token exists
2026-04-18 14:52:31 -07:00
Yuneng Jiang
de790fd273
chore: update Next.js build artifacts (2026-04-18 21:49 UTC, node v22.16.0) 2026-04-18 14:49:27 -07:00
shin-berri
85b1b93661
Merge pull request #26022 from BerriAI/litellm_/reverent-kirch-7cf0a6
[Infra] Bump proxy dependencies and raise minimum Python to 3.10
2026-04-18 14:44:24 -07:00
yuneng-jiang
e69051916e
Merge pull request #25983 from BerriAI/litellm_yj_apr17
[Infra] Merge dev branch
2026-04-18 14:43:04 -07:00
yuneng-jiang
63313bcd77
Merge pull request #25994 from BerriAI/litellm_project_rate_limiting
fix: enforce project-level model-specific rate limits in parallel_req…
2026-04-18 14:31:08 -07:00
Ryan Crabbe
0a4b02fe76
fix: tighten recipient_emails guard to reject empty list
send_max_budget_alert_email previously guarded with `is not None`, which
accepts `[]` and then crashes on `recipient_emails[0]` inside
_get_email_params. The current caller (_handle_multi_threshold_max_budget_alert)
already filters empty lists upstream, but the public method signature makes
no such guarantee — a future caller passing [] would hit IndexError.

Switch to truthiness so both None and [] fall through to the single-recipient
path.
2026-04-18 14:07:18 -07:00
Ishaan Jaffer
b7813aad41
fix(mcp): restore PKCE-triggering 401 when no stored per-user token exists
Per-user OAuth MCP requests now only skip pre-emptive 401 when a stored token is available, preserving token-reuse behavior while restoring fast PKCE kickoff for first-time or missing-token users.
2026-04-18 14:04:22 -07:00
Ryan Crabbe
029d9bcfc7
refactor: inline _parse_email_list in auth_checks to drop enterprise dep
The lazy import from litellm_enterprise inside _normalize_alert_emails
coupled the core proxy auth path to an optional package. Core should not
depend on enterprise, even lazily — it hides the dependency from static
analysis and inverts the intended layering.

Duplicate the 7-line parser locally. It's pure and unlikely to drift; the
enterprise copy stays where it is for its own callers.
2026-04-18 13:51:00 -07:00
Ryan Crabbe
8b86a3b041
fix: normalize alert-email configs at merge boundary
_merge_budget_alert_email_configs previously called list() directly on each
threshold's value, which raised TypeError on null YAML values and silently
split bare strings into single characters. Both are reachable from user-
supplied global config and per-key metadata, so the crash could fire on
every authenticated request once the metadata was in place.

Route both inputs through a _normalize_alert_emails helper that delegates
to the existing _parse_email_list parser (lazy-imported, matching the
enterprise import pattern used elsewhere in proxy/). The merge body keeps
its tight Dict[str, List[str]] contract.
2026-04-18 13:48:48 -07:00
Ryan Crabbe
09a524a351
fix: narrow CallInfo.max_budget_alert_emails to Dict[str, List[str]]
The Union[str, List[str]] value type was speculative — _merge_budget_alert_email_configs
always returns List[str] values, and no caller produces bare strings. Narrowing to
match the runtime guarantee resolves a mypy invariance error at auth_checks.py:3021
without adding casts or Mapping covariance.
2026-04-18 13:39:33 -07:00
shivam
dbf4f9637f
Merge remote-tracking branch 'upstream/litellm_internal_staging' into litellm_project_rate_limiting 2026-04-18 13:26:24 -07:00
Yuneng Jiang
f483f1e800
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_yj_apr17 2026-04-18 13:19:16 -07:00
Yuneng Jiang
ec590f938b
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/reverent-kirch-7cf0a6 2026-04-18 13:18:45 -07:00
yuneng-jiang
643e941b64
Merge pull request #26026 from BerriAI/litellm_fixPrometheusHelpersPackageCollision
[Fix] Resolve prometheus_helpers file/package shadow breaking /global/spend/logs
2026-04-18 13:18:04 -07:00
Yuneng Jiang
cfdf893226
[Fix] Merge prometheus_helpers.py into prometheus_helpers/__init__.py to resolve file/package collision
A previous refactor added `litellm/integrations/prometheus_helpers.py` as a
sibling to the existing `litellm/integrations/prometheus_helpers/` directory
(which contains `prometheus_api.py` and has no `__init__.py`). The file
shadowed the namespace-package directory, so any deferred
`from litellm.integrations.prometheus_helpers.prometheus_api import ...`
raised `ModuleNotFoundError: 'litellm.integrations.prometheus_helpers' is
not a package` at request time.

Two runtime call sites hit that path:
- /global/spend/logs (spend_management_endpoints.py) returned plain-text 500
  "Internal Server Error" for every call, breaking the Admin UI Usage tab
  and programmatic consumers.
- SlackAlerting.send_fallback_stats_from_prometheus silently failed inside
  its own try/except.

Fix: move prometheus_helpers.py content into prometheus_helpers/__init__.py
and delete the stray .py. The directory becomes a regular package, so both
the package-root import (from ...prometheus_helpers import X) and the
submodule import (from ...prometheus_helpers.prometheus_api import X)
resolve correctly. No call sites change.
2026-04-18 13:04:32 -07:00
yuneng-jiang
8e00f61026
Merge pull request #26023 from BerriAI/litellm_/eloquent-feistel-b346ec
[Fix] UI - Keys: strip empty premium fields from key update payload
2026-04-18 13:01:38 -07:00
Yuneng Jiang
f24c8dbf79
chore: bump CircleCI conda envs from python 3.9 to 3.10
Six CI jobs create a miniconda env with python=3.9 before installing
the project; these jobs now fail resolution because the project
requires-python is >=3.10. Bump the conda env python to 3.10 to match
the new floor.
2026-04-18 13:00:03 -07:00
Yuneng Jiang
9bdb3b1772
chore: lower python floor from 3.11 to 3.10
All three dependency bumps in this PR resolve on Python 3.10, so there
is no need to jump the floor all the way to 3.11. Also restore the
py3.10-specific lunary==1.4.36 pin that was collapsed when the floor
was temporarily at 3.11.
2026-04-18 12:50:04 -07:00
Yuneng Jiang
d1e665742b
chore: drop stale python_version markers after floor raise
Now that requires-python starts at 3.11, the "python_version >= '3.9'"
and ">= '3.10'" markers are unconditionally true, and the "< '3.10'"
entries for psycopg, Pillow, pyarrow, langchain, lunary, and pylint can
never resolve. Drop the dead markers and remove the unreachable pins so
the dependency list reflects what actually gets installed.
2026-04-18 12:31:53 -07:00
Yuneng Jiang
cae8b74b0b
Fall back to top-level keyData when resolving previous premium value
Premium fields like policies are echoed at the top level of the
/key/update response, not necessarily mirrored into metadata. Read
metadata first then fall back to the top-level property so an
intentional clear is preserved in either shape.
2026-04-18 12:23:58 -07:00