scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-07 02:59:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103
litellm/litellm-proxy-extras
History
Exact
Abhimanyu Kapur 1bba6094c5 feat(adaptive_router): add efficiency posterior and latency/throughput reward 
Ports the efficiency-reward piece of the Elastic Unified Router (EURo) design
into the adaptive router's bandit, giving it a second Beta posterior keyed on
latency and completion-token throughput, independent of quality signals.

## What changed

- BanditCell now tracks (alpha_eff, beta_eff) alongside (alpha, beta), both fed
  independently. Efficiency defaults to Beta(1, 1) prior (uninformative).
- AdaptiveRouter.record_efficiency() updates the efficiency posterior from
  observed latency and token count; independent of record_turn()'s quality signals.
- Efficiency reward = gamma * exp(-latency/target) + (1-gamma) * min(tokens/sec/max, 1),
  clamped to [0, 1]. On failure (non-2xx), reward = 0.
- Selection now blends three terms: quality_weight * TS(quality) + efficiency_weight
  * TS(efficiency) + cost_weight * cost. Efficiency weight defaults to 0.0 (opt-in).
- DB: schema.prisma + migration add alpha_eff/beta_eff columns to
  LiteLLM_AdaptiveRouterState; read/write plumbing in update_queue and adaptive_router.
- Snapshot endpoint reports efficiency_mean alongside quality_mean.

## Tests

New tests for apply_efficiency_delta, efficiency_mean property, and flush behavior
with efficiency deltas. All existing bandit and router tests pass.

## Design decisions baked in (as per PHASE1_PLAN.md)

1. Efficiency weight defaults to 0.0 — no silent behavior change for existing
   deployments. Opt-in only.
2. target_latency is static (1.0s default, overridable in config). Auto-calibration
   deferred to Phase 2.
3. Failure handling reuses existing response_status detection — one source of truth.
4. Schema extended in place (not a sibling table) — (router, request_type, model)
   is already the right key for both quality and efficiency.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-08 14:09:19 -07:00
..
dist adding build 2026-03-21 22:55:04 -07:00
litellm_proxy_extras feat(adaptive_router): add efficiency posterior and latency/throughput reward 2026-08-08 14:09:19 -07:00
tests [Fix] v2 resolver: swallow non-connection DB errors; wrap resolve failures 2026-04-21 15:53:07 -07:00
build_and_publish.md build: migrate packaging, CI, and Docker from Poetry to uv (#25007) 2026-04-09 11:46:23 -07:00
LICENSE install prisma migration files - connects litellm proxy to litellm's prisma migration files (#9637) 2025-03-29 15:27:09 -07:00
migration_runbook.md [Infra] Add freshness and destructive guards to migration workflow 2026-04-21 12:00:23 -07:00
pyproject.toml bump: litellm-enterprise 0.1.53 -> 0.1.54, litellm-proxy-extras 0.4.83 -> 0.4.84 2026-08-06 17:01:15 -07:00
README.md build: migrate packaging, CI, and Docker from Poetry to uv (#25007) 2026-04-09 11:46:23 -07:00

README.md

Additional files for the proxy. Reduces the size of the main litellm package.

Currently, only stores the migration.sql files for litellm-proxy.

To install, run:

uv add litellm-proxy-extras

OR

uv tool install 'litellm[proxy]' # installs litellm-proxy-extras and other proxy dependencies

To use the migrations, run:

litellm --use_prisma_migrate