mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
a50a391066
Backend: - ShadowEvalLogger: background task on async_log_success_event, deterministic sampling, blind pairwise judge, verdict persistence + job counter updates - Endpoints: POST /auto_router/shadow_eval/start (cost estimate), GET /job_id (per-tier results), GET (list), POST /job_id/stop - Schema + migrations: LiteLLM_ShadowEvalJob, LiteLLM_ShadowEvalVerdict tables - Types: StartShadowEvalRequest, GetShadowEvalJobResponse, ShadowEvalResult, ShadowEvalTierResult - Proxy wiring: logger registered in cost_tracking() UI: - useShadowEval.ts: hooks for start/stop mutations + job query with live polling - ShadowEvalSection.tsx: consent-gate form, job status badge, per-tier results table - Integrated into AutoRouterBenchmarksTab alongside existing benchmarks - OpenAPI schema regenerated to include shadow_eval endpoints Tests: unit tests for logger sampling/unmasking/verdict-parsing Linting: all files python3.11 syntax-check + tsx lint ready This ships the full pre-adoption evaluation flow: 1. User starts job: specifies key, router, sampling %, gets upfront cost estimate 2. Sampled requests duplicated through router, judged blind, verdicts persisted 3. Dashboard shows per-tier win rates, cost tracking, live progress 4. Ready for Tyler/Tinder/Access Group pre-launch sign-off Co-Authored-By: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||