litellm/litellm
ishaan-berri 118ce3cc91
feat: add model leaderboard page (#43649)
* feat(proxy): add model leaderboard analytics

* feat: add model insights task and range constants

* feat: record task type from task tags in model usage rollup

* feat: serve 365 days of model insights by UTC date

* test: cover task tag resolution in model usage rollup

* test: update model insights range limit test to 365 days

* chore: regenerate dashboard api types for model insights

* feat: add model insights aggregation helpers

* test: cover model insights aggregation helpers

* feat: redesign model leaderboard with stacked bars, treemap and ranking

* test: update model leaderboard view test

* feat: mark model leaderboard as beta in sidebar

* chore: sync schema.prisma copies from root

* fix: only treat task: prefixed tags as model insight tasks

* feat: add metric type for model insights ranking

* fix: rank model insights by selected metric and scope detail queries to ranked deployments

* test: plain tags are not model insight tasks

* test: cover metric ranking, deployment scoping and rollup round trip

* fix: build model insights weeks and halves from the requested date range

* test: cover empty weeks and range-based change comparison

* fix: refetch by metric, show load errors and ignore stale responses

* test: cover metric refetch and error state

* feat: define model insight tasks in a JSON file

* feat: return task labels and categories from model insights

* feat: load model insight tasks from JSON

* refactor: validate rollup task tags against the JSON task list

* feat: serve the task list with model insights

* refactor: drop hardcoded task list from constants

* build: ship model insight tasks JSON in the wheel

* test: cover model insight task JSON

* refactor: take task labels and categories from the API

* test: pass task info to task tile builder

* refactor: color treemap by API-provided category

* test: include tasks in model leaderboard fixture

* fix: make daily model usage migration idempotent

* feat: bound the model insights task query size

* fix: compute task breakdown independent of the chart metric

* test: task breakdown is stable across chart metrics

* chore: regenerate lazy openapi snapshot for model insights

* chore: regenerate dashboard api types for model insights

* fix: keep previous ranking dimmed while a new metric loads

* test: cover stale metric state in model leaderboard

* refactor: drop task row cap constant

* fix: return the full task breakdown instead of a truncated one

* test: task query is not truncated

* feat: add task summary types for model insights

* feat: summarise tasks server-side on a separate model insights endpoint

* test: cover the model insights tasks endpoint

* chore: regenerate lazy openapi snapshot for model insights tasks

* chore: regenerate dashboard api types for model insights tasks

* refactor: drop client-side task aggregation

* test: remove client-side task aggregation tests

* feat: load task breakdown separately from the chart metric

* test: task breakdown is not refetched on chart metric change

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-09-28 19:40:04 -07:00
..
_v2 feat(cache): select Rust caching through explicit cache objects (#43601) 2026-09-29 00:01:44 +00:00
a2a_protocol Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5 2026-09-21 12:57:38 -07:00
anthropic_interface fix(proxy): send a real error event when a /v1/messages stream fails (#41826) 2026-09-25 15:36:35 -07:00
assistants refactor(types): replace Any with proven types in 5 files (#43304) 2026-09-27 01:28:02 -07:00
batch_completion
batches refactor: daily fresh tech debt cleanup, rolling PR (2026-09-25) (#43151) 2026-09-26 01:32:55 -07:00
caching feat(cache): select Rust caching through explicit cache objects (#43601) 2026-09-29 00:01:44 +00:00
chat_completions refactor(rust): remove delivery routing abstraction (#43514) 2026-09-28 03:31:22 +00:00
completion_extras feat(lint): add LIT013 flagging *-ok suppressions that suppress nothing and remove the 240 stale ones (#42793) 2026-09-23 17:50:09 -07:00
compression merge: main into litellm_headroom_protect_cached_prefix 2026-09-15 04:12:28 +00:00
containers test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
embeddings feat(embeddings): add native dispatch foundation (#42799) 2026-09-23 21:11:09 +00:00
endpoints/speech/speech_to_completion_bridge
evals
experimental_mcp_client feat(mcp): configure protocol versions and capability discovery (#43169) 2026-09-25 13:13:52 -07:00
files feat(xai): add native xAI batches and files support (#42812) 2026-09-25 15:35:20 -07:00
fine_tuning
google_genai fix(google_genai): drop non-object tool parameters instead of forwarding them 2026-09-19 19:19:33 -07:00
images fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
integrations feat(mcp): scan and pin upstream tool descriptions (#43283) 2026-09-28 18:38:49 -07:00
interactions feat(lint): add LIT013 flagging *-ok suppressions that suppress nothing and remove the 240 stale ones (#42793) 2026-09-23 17:50:09 -07:00
litellm_core_utils refactor(types): replace Any with proven types in 8 files (#43551) 2026-09-28 04:26:41 -07:00
llms feat(fireworks_ai): route and list the auto, auto-instant and firerouter routers (#43641) 2026-09-28 19:08:16 -07:00
messages refactor(rust): remove delivery routing abstraction (#43514) 2026-09-28 03:31:22 +00:00
models feat(mcp): scan and pin upstream tool descriptions (#43283) 2026-09-28 18:38:49 -07:00
ocr refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
passthrough fix(responses): fall back on pre-output stream drops, fail truncated streams, honor request_timeout (#43133) 2026-09-25 14:32:41 -07:00
proxy feat: add model leaderboard page (#43649) 2026-09-28 19:40:04 -07:00
proxy_auth
rag fix(types): read upstream headers through a typed helper 2026-09-21 13:16:58 -07:00
realtime_api refactor(types): replace Any with proven types in 13 files (#42937) 2026-09-24 09:02:02 -07:00
repositories feat: add model leaderboard page (#43649) 2026-09-28 19:40:04 -07:00
rerank_api feat(lint): add LIT013 flagging *-ok suppressions that suppress nothing and remove the 240 stale ones (#42793) 2026-09-23 17:50:09 -07:00
responses refactor(types): replace Any with proven types in 8 files (#43551) 2026-09-28 04:26:41 -07:00
router_strategy feat(router): opt in to prompt-cache cost routing (#43232) 2026-09-26 18:13:13 -07:00
router_utils fix(router): match provider-prefixed fallback keys for bare model groups served by wildcard deployments (#43062) 2026-09-24 18:08:56 -07:00
rust_bridge feat(cache): select Rust caching through explicit cache objects (#43601) 2026-09-29 00:01:44 +00:00
sandbox
search
secret_managers fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
skills
types feat: add model leaderboard page (#43649) 2026-09-28 19:40:04 -07:00
vector_store_files
vector_stores fix(vector_stores): keep config-defined vector stores listed and read-only (#42574) 2026-09-23 04:02:16 +00:00
videos
__init__.py fix(proxy): log key owner identity on expired key auth failures (#43105) 2026-09-28 08:18:48 -07:00
_internal_context.py fix(otel): detach post-response service spans by request phase, name redis spans by operation (#43237) 2026-09-26 10:10:12 -07:00
_lazy_imports.py feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
_lazy_imports_registry.py refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
_logging.py feat(lint): add LIT013 flagging *-ok suppressions that suppress nothing and remove the 240 stale ones (#42793) 2026-09-23 17:50:09 -07:00
_redis.py fix(redis): authenticate sync clusters with IAM credential providers (#40204) 2026-09-23 17:23:55 -05:00
_redis_credential_provider.py
_service_logger.py fix(otel): detach post-response service spans by request phase, name redis spans by operation (#43237) 2026-09-26 10:10:12 -07:00
_uuid.py
_version.py
anthropic_beta_headers_config.json fix(anthropic): forward the per-turn-control beta to Azure AI Foundry (#43415) 2026-09-27 05:02:35 +00:00
anthropic_beta_headers_manager.py fix: drop a blank anthropic-beta header before it reaches the provider 2026-09-19 20:13:59 -07:00
blog_posts.json
budget_manager.py
constants.py feat: add model leaderboard page (#43649) 2026-09-28 19:40:04 -07:00
cost.json
cost_calculator.py fix(cost): apply a deployment's pricing override to realtime sessions (#43114) 2026-09-24 21:03:59 -07:00
exceptions.py fix(spend): return 400 from /spend/calculate for a model with no pricing row (#42497) 2026-09-22 15:44:00 -07:00
main.py fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
model_prices_and_context_window_backup.json feat(fireworks_ai): route and list the auto, auto-instant and firerouter routers (#43641) 2026-09-28 19:08:16 -07:00
policy_templates_backup.json
provider_endpoints_support_backup.json feat(providers): add Prism provider (internal copy of #40914) (#41961) 2026-09-28 16:07:09 -07:00
py.typed
router.py refactor(types): replace Any with proven types in 8 files (#43551) 2026-09-28 04:26:41 -07:00
scheduler.py
setup_wizard.py fix(model_prices): correct Claude Sonnet 5.5 capabilities and provider keys (#43587) 2026-09-28 19:12:31 +00:00
timeout.py
utils.py refactor(types): replace Any with proven types in 8 files (#43551) 2026-09-28 04:26:41 -07:00