* feat(ui): show models under each tier in routing benchmark chart
- Add TierTurnsChart: donut chart showing turns per complexity tier with
tier-assigned models listed below each tier name in the legend
- Only complexity routers show models; quality routers show tier name + %
(quality tiers don't pin specific models)
- Change 'Estimated spend at highest-cost model' wording to 'highest-tier'
to clarify it's the most capable tier's estimated cost, not just the
single-highest model
Closes LIT-5302
* fix(ui): use categorical colors for tier donut, trim redundant turn count
- Tier donut chart now uses a dedicated categorical palette instead of
SEQUENTIAL_COLOR_RAMP, which is a blue monochrome gradient meant for
magnitude series, not distinct categories.
- Space out the tier legend rows (gap-3 -> gap-6) for readability.
- Drop the turn count from "avg saved per session" since Routing by
tier already shows the total turns.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(ui): drop prohibited explanatory comments in TierTurnsChart test
Per repo convention against source comments; the test name and
assertions already communicate the scenario. Addresses Greptile review.
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove dead modulo from color index in TierTurnsChart
The colors array is built with length equal to slices.length, so idx % colors.length
is always a no-op in the render loop. Simplify to idx for clarity.
* fix(ui): wrap CostOptimizationView tests in QueryClientProvider
The tests render CostOptimizationView which uses useCan() → useIsOrgAdmin() →
useOrganizations() and useDailyActivityRange(), both of which call React Query's
useQuery(). Without QueryClientProvider wrapping the render, React Query throws
'No QueryClient set' error.
Also mock the required networking calls (organizationListCall, userDailyActivityCall)
to prevent spurious network errors in test runs.
All 7 tests now pass (CostOptimizationView + CostOptimizationView.activity).
* style(ui): format test files and extract object literal to fix linting
- Format CostOptimizationView.test.tsx with prettier
- Extract getToolSpend mock response to named variable to satisfy eslint
- Pass frontend-lint checks
* fix(ui): hoist mockToolSpendResponse into vi.hoisted to fix test initialization
Extracting the response object to a named variable violated hoisting rules:
vi.mock() factories are evaluated at hoisting time before regular const
declarations. Move mockToolSpendResponse into vi.hoisted() block.
---------
Co-authored-by: Claude <noreply@anthropic.com>
reportAny 16720 -> 15482 and reportExplicitAny 5689 -> 5316 with real types only: no casts, no ignores, no new Any. Whole-tree basedpyright drops 2173 diagnostics with zero per-rule or per-file regressions. Budgets ratcheted: basedpyright -2173, ruff-strict -188, type-discipline -55
Record supports_tool_search on the Bedrock Claude entries in both cost
map files and have _supports_tool_search_on_bedrock read it first via
the provider-resolved capability lookup, keeping the name patterns as a
fallback for ARNs and ids the map cannot resolve. Threads the flag
through ModelInfoBase and drops a dated remark from the pattern list
- formatInstallCommand now produces /plugin install {name}@litellm instead of /plugin marketplace add {source}
- extraKnownMarketplaces snippet fixed: source must be a nested object not a flat string; the flat string caused Claude Code to reject the settings file
- marketplace key renamed from my-org to litellm to match the name the proxy returns in marketplace.json
- setup tab now shows /plugin marketplace add command as primary option with settings.json as secondary
- usage tab now shows a hint to run /plugin marketplace update litellm when a plugin is not found
The /model/new endpoint returns a 400 validation error (type: validation_error)
when 'rpm and tpm must be set to a positive value when enforce_rpm_tpm_on_model_add
is enabled in general_settings' but the frontend's titleFor() keyword matcher
mistitled it as 'Rate Limit Exceeded' because the message contains 'rpm'/'tpm'
substrings, which matched the generic rate-limit keyword check before the more
specific validation check could catch it.
Add "'enforce_rpm_tpm_on_model_add' is enabled" to VALIDATION_MATCH so this
message is classified as a Validation Error, matching the actual HTTP 400
validation_error the backend already returns. A narrow match on the setting
name (rather than the generic "must be set when") avoids overriding the
status-based classification of unrelated 401s, e.g. the PKCE
'GENERIC_CLIENT_ID must be set when PKCE is enabled' error.
Add general_settings toggle 'enforce_rpm_tpm_on_model_add' (default false).
When true, /model/new rejects a model whose rpm or tpm is missing or not a
positive value, so the Admin UI Add Model form surfaces a 400 validation
error instead of silently storing an unbounded model (or one with a
zero/negative limit that would exclude it from routing).
Fixes#36487
video_edit, video_extension, and video_remix called request.body() after
user_api_key_auth had already parsed multipart/form bodies via
_read_request_body(), causing RuntimeError Stream consumed and 500s for
OpenAI SDK clients.
Use _read_request_body consistently and normalize bare-string or JSON-string
video references from form fields into video_id.
Claude Code drives Opus 4.7 with thinking {"type": "adaptive"} plus
output_config {"effort": "max"}. The anthropic-to-openai adapter
forwarded thinking verbatim for Claude models but dropped output_config,
and Bedrock Converse streams zero reasoningContent blocks for adaptive
thinking without an effort tier. Forward the effort subset of
output_config for Bedrock targets, accept it in the converse supported
params, and map it with the model's effort ceiling applied. Re-enable
the skipped e2e compat cell that catches this
Bedrock InvokeModel rejects tool_search_tool_* tool types unless the
request body carries the tool-search-tool-2025-10-19 beta. The model
allowlist gating that beta omitted Haiku 4.5 (and Opus 4.7, supported
since launch per live verification), so every tool-search request on
those models got a Bedrock 400. Add both to the allowlist and re-enable
the e2e compat cell that caught it.
* fix: net prompt-caching savings against the cache-write premium
Prompt-caching savings priced only the cache-read discount and ignored what
the provider charges to create the cache entry. Anthropic bills cache writes
at 1.25x the input rate, so a request that writes a large cache and reads
little from it is a net loss that the dashboard reported as a gain -- or, on
a pure cold write, as a flat zero.
The counterfactual the number answers is "what would this have cost with
caching off", where every token is billed at the input rate. Since
prompt_tokens partitions disjointly into text + reads + writes, that gives
savings = reads * (input - read_rate) - writes * (write_rate - input)
The write term is the premium over the input rate, not the full write cost:
the tokens would have been paid for at the input rate anyway, so only the
markup is attributable to caching.
The premium stays signed rather than clamped. Three models in the pricing map
price writes below input, and clamping would silently drop that saving.
A model with no cache_creation_input_token_cost falls open to the input cost,
yielding a zero premium -- this is why the change is a no-op for the implicit
caching providers (OpenAI, Gemini), which publish no write price, and bites
exactly on Anthropic and Bedrock.
Verified live through the proxy on a mock Anthropic rig across four cases
(cold pure-write, warm pure-read, write-heavy, read-heavy). Reported total
matched the derived net to the cent, including the negatives; the read-only
case is unchanged.
Pre-existing rows are not backfilled, so a range spanning the deploy mixes
gross and net.
* fix: read a zero cache-write price as unpublished, not free
deepseek-chat carries a literal 0.0 cache_creation_input_token_cost. The
fall-open only caught None, so the zero was taken at face value and the
premium became 0 - input_cost -- reporting a fabricated saving of
writes * input_cost on traffic that cached nothing.
No provider gives cache writes away, so a falsy price means the same thing
an absent one does.
* test: pin that the read leg keeps a literal zero price
The two zero prices mean opposite things and the asymmetry was unpinned.
A free cache write is unpublished pricing; a free cache read is real, and
15 models charge for input while serving reads for nothing. Copying the
write leg's falsy fall-open onto the read leg would zero out their savings.
* refactor: resolve caching rates through the established pricing helpers
Addresses Greptile's P1 and P2, and replaces hand-rolled pricing lookup with
the patterns this file and the cost calculator already own:
- Deployment pricing first: rates now resolve through _effective_model_info
(Router.get_deployment_model_info), the same helper the autorouter driver
uses, falling back to _model_info public rates. A deployment with negotiated
cache rates previously priced at the public map -- a 3x error on the repro.
- Individual prices read via _get_cost_per_unit, the cost calculator's
accessor, which also coerces string prices from config.yaml and resolves
service-tier suffixes; the previous raw .get() handled neither.
- Pricing tests no longer monkeypatch litellm.get_model_info; each case now
pins a real pricing-map entry with a fixture-drift assertion, and the
deployment-rate case follows the existing Router-fixture test pattern.
Behaviour on public rates is unchanged: 101 tests pass, including the exact
same live-verified formula.
* fix(cost-optimization): computeCacheLeakage divides net savings by all cached tokens, not reads alone
prompt_caching_savings_spend is net of the cache-write premium since PR #36452.
computeCacheLeakage was still dividing by cache_read_tokens alone, which:
1. Overstates the per-token rate on traffic that writes and reads cache equally:
a 1:1 read:write key shows rate = 0.002, not 0.001, if net savings is /bin/zsh.002
2. Flips the sign on write-heavy traffic: when writes cost more than reads save
(common on Anthropic and Bedrock), the aggregate net can go negative, but
dividing by reads alone would show a positive 'potential savings' for keys
that don't cache yet — recommending they start caching when it's currently
losing money overall
Fix: divide realizedCachingSavings by (cacheReadTokens + cacheCreationTokens),
matching the semantic that a key starting to cache pays those write premiums too.
When the rate is non-positive, price nothing (potentialSavings stays null, renders
as '—'), reusing the existing no-data fallback path. The card can't meaningfully
estimate savings from a losing rate.
Rename discountPerToken → netSavingsPerCachedToken to surface the semantics and
prevent this drift in future.
Update Usage tab and Cache Leakage card tooltips to describe net-of-premium cost.
Add tests for 1:1 read:write traffic and write-heavy negative-net traffic.
A deployment that carries one provider's credentials while resolving to
another is silently broken: litellm ignores the credentials and sends the
request to the resolved provider. The common shape is a Bedrock model
group where one entry lost its route prefix, so `model: claude-sonnet-5`
with aws_region_name set resolves to the first-party Anthropic API and
returns "x-api-key header is required". Because the router load balances
across the group, only the fraction of requests routed to that entry
fails, which reads as an intermittent provider outage rather than a
config error, and nothing at startup says otherwise.
Warn at deployment registration when provider-scoped credential params
(aws_*, vertex_*) sit on a model that resolves elsewhere, naming the
params, the resolved provider, and the likely missing prefix. Warn only:
an operator may be overriding a route deliberately, so this must not
block startup. Deployments litellm cannot classify are left alone.
Resolves LIT-5391
