* feat(ui): show models under each tier in routing benchmark chart
- Add TierTurnsChart: donut chart showing turns per complexity tier with
tier-assigned models listed below each tier name in the legend
- Only complexity routers show models; quality routers show tier name + %
(quality tiers don't pin specific models)
- Change 'Estimated spend at highest-cost model' wording to 'highest-tier'
to clarify it's the most capable tier's estimated cost, not just the
single-highest model
Closes LIT-5302
* fix(ui): use categorical colors for tier donut, trim redundant turn count
- Tier donut chart now uses a dedicated categorical palette instead of
SEQUENTIAL_COLOR_RAMP, which is a blue monochrome gradient meant for
magnitude series, not distinct categories.
- Space out the tier legend rows (gap-3 -> gap-6) for readability.
- Drop the turn count from "avg saved per session" since Routing by
tier already shows the total turns.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(ui): drop prohibited explanatory comments in TierTurnsChart test
Per repo convention against source comments; the test name and
assertions already communicate the scenario. Addresses Greptile review.
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove dead modulo from color index in TierTurnsChart
The colors array is built with length equal to slices.length, so idx % colors.length
is always a no-op in the render loop. Simplify to idx for clarity.
* fix(ui): wrap CostOptimizationView tests in QueryClientProvider
The tests render CostOptimizationView which uses useCan() → useIsOrgAdmin() →
useOrganizations() and useDailyActivityRange(), both of which call React Query's
useQuery(). Without QueryClientProvider wrapping the render, React Query throws
'No QueryClient set' error.
Also mock the required networking calls (organizationListCall, userDailyActivityCall)
to prevent spurious network errors in test runs.
All 7 tests now pass (CostOptimizationView + CostOptimizationView.activity).
* style(ui): format test files and extract object literal to fix linting
- Format CostOptimizationView.test.tsx with prettier
- Extract getToolSpend mock response to named variable to satisfy eslint
- Pass frontend-lint checks
* fix(ui): hoist mockToolSpendResponse into vi.hoisted to fix test initialization
Extracting the response object to a named variable violated hoisting rules:
vi.mock() factories are evaluated at hoisting time before regular const
declarations. Move mockToolSpendResponse into vi.hoisted() block.
---------
Co-authored-by: Claude <noreply@anthropic.com>
reportAny 16720 -> 15482 and reportExplicitAny 5689 -> 5316 with real types only: no casts, no ignores, no new Any. Whole-tree basedpyright drops 2173 diagnostics with zero per-rule or per-file regressions. Budgets ratcheted: basedpyright -2173, ruff-strict -188, type-discipline -55
Record supports_tool_search on the Bedrock Claude entries in both cost
map files and have _supports_tool_search_on_bedrock read it first via
the provider-resolved capability lookup, keeping the name patterns as a
fallback for ARNs and ids the map cannot resolve. Threads the flag
through ModelInfoBase and drops a dated remark from the pattern list
Claude Code drives Opus 4.7 with thinking {"type": "adaptive"} plus
output_config {"effort": "max"}. The anthropic-to-openai adapter
forwarded thinking verbatim for Claude models but dropped output_config,
and Bedrock Converse streams zero reasoningContent blocks for adaptive
thinking without an effort tier. Forward the effort subset of
output_config for Bedrock targets, accept it in the converse supported
params, and map it with the model's effort ceiling applied. Re-enable
the skipped e2e compat cell that catches this
Bedrock InvokeModel rejects tool_search_tool_* tool types unless the
request body carries the tool-search-tool-2025-10-19 beta. The model
allowlist gating that beta omitted Haiku 4.5 (and Opus 4.7, supported
since launch per live verification), so every tool-search request on
those models got a Bedrock 400. Add both to the allowlist and re-enable
the e2e compat cell that caught it.
* fix: net prompt-caching savings against the cache-write premium
Prompt-caching savings priced only the cache-read discount and ignored what
the provider charges to create the cache entry. Anthropic bills cache writes
at 1.25x the input rate, so a request that writes a large cache and reads
little from it is a net loss that the dashboard reported as a gain -- or, on
a pure cold write, as a flat zero.
The counterfactual the number answers is "what would this have cost with
caching off", where every token is billed at the input rate. Since
prompt_tokens partitions disjointly into text + reads + writes, that gives
savings = reads * (input - read_rate) - writes * (write_rate - input)
The write term is the premium over the input rate, not the full write cost:
the tokens would have been paid for at the input rate anyway, so only the
markup is attributable to caching.
The premium stays signed rather than clamped. Three models in the pricing map
price writes below input, and clamping would silently drop that saving.
A model with no cache_creation_input_token_cost falls open to the input cost,
yielding a zero premium -- this is why the change is a no-op for the implicit
caching providers (OpenAI, Gemini), which publish no write price, and bites
exactly on Anthropic and Bedrock.
Verified live through the proxy on a mock Anthropic rig across four cases
(cold pure-write, warm pure-read, write-heavy, read-heavy). Reported total
matched the derived net to the cent, including the negatives; the read-only
case is unchanged.
Pre-existing rows are not backfilled, so a range spanning the deploy mixes
gross and net.
* fix: read a zero cache-write price as unpublished, not free
deepseek-chat carries a literal 0.0 cache_creation_input_token_cost. The
fall-open only caught None, so the zero was taken at face value and the
premium became 0 - input_cost -- reporting a fabricated saving of
writes * input_cost on traffic that cached nothing.
No provider gives cache writes away, so a falsy price means the same thing
an absent one does.
* test: pin that the read leg keeps a literal zero price
The two zero prices mean opposite things and the asymmetry was unpinned.
A free cache write is unpublished pricing; a free cache read is real, and
15 models charge for input while serving reads for nothing. Copying the
write leg's falsy fall-open onto the read leg would zero out their savings.
* refactor: resolve caching rates through the established pricing helpers
Addresses Greptile's P1 and P2, and replaces hand-rolled pricing lookup with
the patterns this file and the cost calculator already own:
- Deployment pricing first: rates now resolve through _effective_model_info
(Router.get_deployment_model_info), the same helper the autorouter driver
uses, falling back to _model_info public rates. A deployment with negotiated
cache rates previously priced at the public map -- a 3x error on the repro.
- Individual prices read via _get_cost_per_unit, the cost calculator's
accessor, which also coerces string prices from config.yaml and resolves
service-tier suffixes; the previous raw .get() handled neither.
- Pricing tests no longer monkeypatch litellm.get_model_info; each case now
pins a real pricing-map entry with a fixture-drift assertion, and the
deployment-rate case follows the existing Router-fixture test pattern.
Behaviour on public rates is unchanged: 101 tests pass, including the exact
same live-verified formula.
* fix(cost-optimization): computeCacheLeakage divides net savings by all cached tokens, not reads alone
prompt_caching_savings_spend is net of the cache-write premium since PR #36452.
computeCacheLeakage was still dividing by cache_read_tokens alone, which:
1. Overstates the per-token rate on traffic that writes and reads cache equally:
a 1:1 read:write key shows rate = 0.002, not 0.001, if net savings is /bin/zsh.002
2. Flips the sign on write-heavy traffic: when writes cost more than reads save
(common on Anthropic and Bedrock), the aggregate net can go negative, but
dividing by reads alone would show a positive 'potential savings' for keys
that don't cache yet — recommending they start caching when it's currently
losing money overall
Fix: divide realizedCachingSavings by (cacheReadTokens + cacheCreationTokens),
matching the semantic that a key starting to cache pays those write premiums too.
When the rate is non-positive, price nothing (potentialSavings stays null, renders
as '—'), reusing the existing no-data fallback path. The card can't meaningfully
estimate savings from a losing rate.
Rename discountPerToken → netSavingsPerCachedToken to surface the semantics and
prevent this drift in future.
Update Usage tab and Cache Leakage card tooltips to describe net-of-premium cost.
Add tests for 1:1 read:write traffic and write-heavy negative-net traffic.
A deployment that carries one provider's credentials while resolving to
another is silently broken: litellm ignores the credentials and sends the
request to the resolved provider. The common shape is a Bedrock model
group where one entry lost its route prefix, so `model: claude-sonnet-5`
with aws_region_name set resolves to the first-party Anthropic API and
returns "x-api-key header is required". Because the router load balances
across the group, only the fraction of requests routed to that entry
fails, which reads as an intermittent provider outage rather than a
config error, and nothing at startup says otherwise.
Warn at deployment registration when provider-scoped credential params
(aws_*, vertex_*) sit on a model that resolves elsewhere, naming the
params, the resolved provider, and the likely missing prefix. Warn only:
an operator may be overriding a route deliberately, so this must not
block startup. Deployments litellm cannot classify are left alone.
Resolves LIT-5391
The existing web_search cells drive Claude Code's client-side WebSearch
tool, which the CLI executes itself and feeds back as a tool_result. The
CLI never emits a web_search_20250305 definition, so those cells stayed
green while the Anthropic-managed server tool 400'd on Bedrock.
Add a cell that posts the server tool to a Bedrock deployment over
/v1/messages and asserts a web_search_tool_result block comes back, and
reword the compat row so it no longer reads as coverage of the server
tool. Model the server tool as a composed base shared with tool_search.
Resolves LIT-5391
The web-search interception hooks stamp
_websearch_interception_emit_native_blocks and
_websearch_interception_converted_stream onto kwargs to carry state across
the agentic loop, but neither was registered in all_litellm_params. The
param builder sweeps anything it does not recognize into the outbound
request, so a provider that validates its body rejects the whole call:
Bedrock Converse answers "_websearch_interception_emit_native_blocks:
Extra inputs are not permitted" with a 400, which breaks every request
interception touches on that route.
Register both alongside their code-interpreter counterparts, which were
already listed for exactly this reason.
Resolves LIT-5391
