litellm/tests/unit/proxy/db
joshua-berri 9e81aa3f40
feat(shadow-eval): skip redundant reverse evaluations on opt-in (#52192)
* feat(shadow-eval): let reverse jobs skip samples the baseline group served

A reverse shadow eval judges the router's response against baseline_model.
When the router itself sent the request to the baseline's model group, the
comparison only produces ties and still pays for a shadow and a judge call.

Reverse jobs can now opt in with skip_same_model. A sampled request whose
served group (router alias resolved, fallback hops respected) is the
baseline's own group is counted in a new same_model funnel stage instead of
being judged. It makes no provider calls, writes no attempt row, and leaves
max_turns and the eval budget untouched. When the served group is unknown
the sample is judged as before. The job detail reports
same_model_skipped_count even when nothing has been judged yet.

The flag defaults to false and is rejected on forward jobs, so existing
jobs and requests behave exactly as before. Adds two columns through an
additive, schema-only migration.

* test(shadow-eval): drive same-model skip tests off explicit completion signals

Judged cases now wait on the attempt write the injected prisma reports,
then await the pipeline task that made it. Skip cases drain only the tasks
the hook created, through a task factory scoped to the hook call, before
asserting that no provider call, attempt row, or spend was recorded. This
replaces a whole-loop task diff and the assertion on the number of tasks.

Adds a skip case with tier parameter overrides, which the policy still
treats as the same group, and a case showing a lost sampling roll counts as
not_sampled before same_model. Drops the class docstring claim that a same
group comparison only produces ties.

* style(tests): remove extra shadow evaluation test spacing

* fix(ui): send skip_same_model false from the shadow eval start form

The generated request type now requires skip_same_model, which broke the
dashboard typecheck and builds. The form sends false, the API default, so
jobs started from the UI behave exactly as before.

* fix(migrations): bound shadow evaluation schema lock waits

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:38:00 -08:00
..
db_transaction_queue test: remove dead skip-marked tests and rewrite the Redis buffer drain test (#52063) 2026-11-16 10:50:52 -08:00
mcp_server refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) 2026-10-27 17:47:50 +00:00
__init__.py ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903) 2026-09-24 22:59:11 +00:00
conftest.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
fake_prisma_engine.py fix(proxy): cancel the Admin UI's raw spend-log reads in Postgres after 30s (#49965) 2026-11-03 16:07:00 -08:00
test_autorouter_session_rollup.py fix(proxy): retain unwritten rollups after database disconnects (#51706) 2026-11-13 13:36:12 -08:00
test_budget_window_spend_writer.py feat(proxy): support budget_limits windows on internal users (#51763) 2026-11-14 07:44:35 -05:00
test_check_migration.py feat(proxy)!: add suppress_startup_output to turn off startup prints (#47708) 2026-10-29 03:47:04 -07:00
test_connection_budget.py feat(db): add the SQLx pool foundation with the writer pool (#49798) 2026-11-14 15:28:10 -08:00
test_create_views.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
test_daily_spend_bulk_upsert.py feat(proxy): enforce customer org, team and member budgets and limits on the request path (#51489) 2026-11-12 20:43:44 -08:00
test_db_lookup_gate.py test: make three flaky unit tests deterministic (#50114) 2026-11-04 14:38:44 -08:00
test_db_span.py fix(otel): name postgres service spans by operation and table (#44240) 2026-10-03 09:20:28 -07:00
test_db_spend_update_writer.py fix(proxy): alert on customer org, customer team, and customer member budgets (#51579) 2026-11-13 09:16:33 -08:00
test_db_url_settings.py feat(db): add an optional native reader pool with explicit routing (#49804) 2026-11-14 15:28:10 -08:00
test_exception_handler.py fix(proxy): retain unwritten rollups after database disconnects (#51706) 2026-11-13 13:36:12 -08:00
test_exception_handler_reconnect_retry.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
test_gateway_request_tracking.py feat(proxy): track failed requests by HTTP status and caller (#45244) 2026-10-09 14:27:56 -07:00
test_health_check_latest.py fix(proxy): show the newest health check per model group on the model hub (#47552) 2026-10-20 16:42:53 -07:00
test_log_db_metrics.py fix(proxy): keep the database error when the log_db_metrics failure hook raises (#44383) 2026-10-03 16:38:15 +00:00
test_log_db_metrics_service_spans.py test: deflake proxy metrics and shutdown drain (#51078) 2026-11-10 21:05:41 -08:00
test_master_key_migration.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_model_access_group_spend.py feat(proxy): enforce customer org, team and member budgets and limits on the request path (#51489) 2026-11-12 20:43:44 -08:00
test_model_insights_tasks.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
test_model_usage_rollup.py feat(alerts): detect daily changes in recorded usage (#50666) 2026-11-11 18:23:02 -08:00
test_native_database.py feat(db): classify errors and recycle poisoned pools like Prisma (#49801) 2026-11-14 15:28:11 -08:00
test_object_permission_repository.py refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
test_pgbouncer.py test(unit): keep unix socket paths in unit tests under the macOS AF_UNIX limit (#45040) 2026-10-28 20:23:22 -07:00
test_prisma_client.py feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations (#48961) 2026-10-29 09:39:26 -07:00
test_prisma_engine.py fix(proxy): let the OS assign Prisma engine ports (#50850) 2026-11-10 07:28:55 +00:00
test_prisma_planned_engine_restart.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
test_prisma_query_span.py chore: move scripts and runtime sources under litellm (#51006) 2026-11-12 04:19:57 +00:00
test_prisma_self_heal.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
test_proxy_worker_heartbeat.py fix(otel): name postgres service spans by operation and table (#44240) 2026-10-03 09:20:28 -07:00
test_query_engine_reaper.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
test_rds_iam_token_expiry.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
test_replica_identity.py test: migrate proxy security, schema migration and MCP legacy tests (#45937) 2026-10-11 16:17:24 -07:00
test_request_error_tracking.py feat(proxy): track failed requests by HTTP status and caller (#45244) 2026-10-09 14:27:56 -07:00
test_routing_prisma_wrapper.py fix(proxy): let the OS assign Prisma engine ports (#50850) 2026-11-10 07:28:55 +00:00
test_scheduled_job_runs.py feat(proxy): run spend log cleanup from a DB-backed scheduled job table (#50131) 2026-11-11 18:22:02 -08:00
test_shadow_eval_funnel.py feat(shadow-eval): skip redundant reverse evaluations on opt-in (#52192) 2026-11-16 21:38:00 -08:00
test_spend_counter_reseed.py feat(proxy): support budget_limits windows on internal users (#51763) 2026-11-14 07:44:35 -05:00
test_spend_log_batching.py test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy (#44012) 2026-10-01 18:14:24 +00:00
test_spend_log_tool_index.py fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385) 2026-11-06 03:54:34 -08:00
test_token_auth.py fix(proxy): sign RDS IAM tokens for the database's own region (#44782) 2026-10-09 15:41:07 -07:00
test_tool_registry_writer.py fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows (#50132) 2026-11-04 17:39:30 -08:00
test_tool_table_bulk_upsert.py fix(proxy): buffer tool discoveries in Redis and write LiteLLM_ToolTable in one statement per drain, on its own lock (#48089) 2026-10-24 12:34:13 -07:00
test_update_daily_tag_spend.py refactor(proxy): expose public names for private proxy helpers (#45170) 2026-10-07 18:59:27 -07:00