mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-17 09:34:48 +00:00
* feat(shadow-eval): let reverse jobs skip samples the baseline group served A reverse shadow eval judges the router's response against baseline_model. When the router itself sent the request to the baseline's model group, the comparison only produces ties and still pays for a shadow and a judge call. Reverse jobs can now opt in with skip_same_model. A sampled request whose served group (router alias resolved, fallback hops respected) is the baseline's own group is counted in a new same_model funnel stage instead of being judged. It makes no provider calls, writes no attempt row, and leaves max_turns and the eval budget untouched. When the served group is unknown the sample is judged as before. The job detail reports same_model_skipped_count even when nothing has been judged yet. The flag defaults to false and is rejected on forward jobs, so existing jobs and requests behave exactly as before. Adds two columns through an additive, schema-only migration. * test(shadow-eval): drive same-model skip tests off explicit completion signals Judged cases now wait on the attempt write the injected prisma reports, then await the pipeline task that made it. Skip cases drain only the tasks the hook created, through a task factory scoped to the hook call, before asserting that no provider call, attempt row, or spend was recorded. This replaces a whole-loop task diff and the assertion on the number of tasks. Adds a skip case with tier parameter overrides, which the policy still treats as the same group, and a case showing a lost sampling roll counts as not_sampled before same_model. Drops the class docstring claim that a same group comparison only produces ties. * style(tests): remove extra shadow evaluation test spacing * fix(ui): send skip_same_model false from the shadow eval start form The generated request type now requires skip_same_model, which broke the dashboard typecheck and builds. The form sends false, the API default, so jobs started from the UI behave exactly as before. * fix(migrations): bound shadow evaluation schema lock waits --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| db_transaction_queue | ||
| mcp_server | ||
| __init__.py | ||
| conftest.py | ||
| fake_prisma_engine.py | ||
| test_autorouter_session_rollup.py | ||
| test_budget_window_spend_writer.py | ||
| test_check_migration.py | ||
| test_connection_budget.py | ||
| test_create_views.py | ||
| test_daily_spend_bulk_upsert.py | ||
| test_db_lookup_gate.py | ||
| test_db_span.py | ||
| test_db_spend_update_writer.py | ||
| test_db_url_settings.py | ||
| test_exception_handler.py | ||
| test_exception_handler_reconnect_retry.py | ||
| test_gateway_request_tracking.py | ||
| test_health_check_latest.py | ||
| test_log_db_metrics.py | ||
| test_log_db_metrics_service_spans.py | ||
| test_master_key_migration.py | ||
| test_model_access_group_spend.py | ||
| test_model_insights_tasks.py | ||
| test_model_usage_rollup.py | ||
| test_native_database.py | ||
| test_object_permission_repository.py | ||
| test_pgbouncer.py | ||
| test_prisma_client.py | ||
| test_prisma_engine.py | ||
| test_prisma_planned_engine_restart.py | ||
| test_prisma_query_span.py | ||
| test_prisma_self_heal.py | ||
| test_proxy_worker_heartbeat.py | ||
| test_query_engine_reaper.py | ||
| test_rds_iam_token_expiry.py | ||
| test_replica_identity.py | ||
| test_request_error_tracking.py | ||
| test_routing_prisma_wrapper.py | ||
| test_scheduled_job_runs.py | ||
| test_shadow_eval_funnel.py | ||
| test_spend_counter_reseed.py | ||
| test_spend_log_batching.py | ||
| test_spend_log_tool_index.py | ||
| test_token_auth.py | ||
| test_tool_registry_writer.py | ||
| test_tool_table_bulk_upsert.py | ||
| test_update_daily_tag_spend.py | ||