* test(ui): characterize default team settings
* refactor(ui): migrate teams settings to shadcn
* chore(ui): prune teams lint suppression
* test(ui): preserve teams settings contracts
* fix(ui): align spend and budget columns
* fix(ui): preserve sub-threshold money formatting
Co-Authored-By: Codex
* fix(ui): use two-decimal summary amounts
Co-Authored-By: Codex
* test(ui): tolerate organization lookup in access checks
Scope denied-role assertions to the protected page endpoints so the
organization membership lookup does not make the tests fail.
Generated with AI
Co-Authored-By: Claude Code
Co-Authored-By: Codex
APScheduler anchors an interval job at now + interval, so every scheduled
background job registered in one proxy startup shares a single firing instant
for the life of the process, and every replica a rollout brought up together
shares that instant too. Each tick the spend flushes, budget reset sweep,
config-in-DB reload, credential reload and cost pollers all hit Postgres at the
same moment, on every pod, competing with request-path auth and budget queries.
Shift each eligible job by a deterministic offset derived from
sha256(job_id, identity), where identity covers the pod and the worker process.
The offset lives in the trigger rather than in a one-off next_run_time, because
a cron trigger recomputes each fire from the wall clock and would otherwise snap
straight back onto the shared instant. An interval job is never offset by more
than one of its own periods.
Only schedules LiteLLM chose are shifted: interval jobs always, cron jobs only
when the id is one of the product's own defaults, so an operator-supplied
crontab keeps the instant it asks for. general_settings.scheduled_job_stagger
turns it off, widens the window, replaces the identity, or pins a job. The
applied offsets are logged once at startup and each fire logs its scheduled
instant against its actual start.
Resolves LIT-5433
255d65192e added useCan to UsageTab, whose useIsOrgAdmin leg calls
useOrganizations (react-query), so every UsageTab test died with 'No
QueryClient set'. Stub the org-admin leg; role gating still flows through
the real hasCapability with the varied userRole.
* feat(proxy): per-key prompt caching auto-injection via enable_prompt_caching
Adds a key-level enable_prompt_caching toggle that auto-injects Anthropic
cache_control breakpoints on requests made with that key, without requiring
the gateway-wide enable_anthropic_prompt_caching flag. The flag lives in key
metadata, is stamped onto the request root by add_key_level_controls, rides
kwargs into both the /chat/completions seeding path and the native
/v1/messages path, and reuses every existing gate (anthropic/bedrock only,
supports_prompt_caching, client markers win). Client-supplied body values are
stripped as an untrusted root control field. Includes the Admin UI switch on
key create and key edit plus a read-only settings row, and dedupes the key
edit view's drifted initial-values objects.
* fix(proxy): drop section comment and suppress LIT011 on key-level prompt caching stamp
* feat(router): add required-AND (&) tag prefix and allow_fail_open flag
Tag routing supported inclusion-OR and independent "!" negation, but had no way
to express a hard "must match all of these" constraint per request, and no way
for a model group to opt into degrading gracefully instead of raising when a
constraint eliminates every deployment.
Adds a "&tag" prefix for required-AND inclusion, composing with existing plain
(OR) and "!" (negate) tags: negation still applies first, then required tags
narrow the survivors, then plain tags apply today's OR/AND preference logic
unchanged. Adds model_info.allow_fail_open (default false) so a chain can opt
into falling back to the default-tagged pool instead of raising
no_deployments_with_tag_routing when "!" or "&" empties the candidate set;
existing chains without the flag keep today's fail-closed behavior exactly.
* fix(router): gate mixed negation on allow_fail_open and stop diluting required-only requests
Two gaps in the initial required-AND/allow_fail_open change: a "!" exclusion
combined with a plain positive tag that emptied the candidate set raised
unconditionally, bypassing allow_fail_open entirely, since the fail-open check
only looked at required-AND exhaustion. And a request using only "&" tags
could get narrowed down to just the deployment matching an incidental
tag_regex/User-Agent preference, silently dropping other deployments that
satisfied the required tags but had no tag_regex at all.
Fixes both: the fail-open check now fires whenever either "!" or "&" leaves
the candidate set empty, not just "&". And regex/header preference no longer
counts as a positive filter when a required-AND ask is present, so a
required-only request returns every deployment satisfying the required tags
regardless of regex/header matching.
Also regenerates ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
model_info.allow_fail_open field, and removes source comments explaining
the router logic per repository convention.
* fix(router): let allow_fail_open cover a non-empty !/& survivor set that fails the plain-tag preference
The unconditional raise inside the has_positive_filter loop was the one
remaining path a chain could hit despite setting allow_fail_open: when "!"
or "&" leaves a non-empty candidate set but none of the survivors match the
request's plain preference tag or carry "default", the request still raised
instead of degrading. Routes that raise through the same allow_fail_open
check used everywhere else, so it now falls back to the default-tagged pool
for opted-in chains and keeps raising unconditionally for everyone else.
This also let the now-redundant pre-loop empty-candidates shortcut be
removed, since the loop reaches the same outcome on its own.
* fix(router): deny allow_fail_open when an unrecognized required tag is masking a satisfiable answer
A caller could add a single "&" tag no deployment in the group has ever
carried to force an empty required-AND set on demand. On a chain with
allow_fail_open, that emptied set fell back to the default-tagged pool
unconditionally, discarding every other constraint merged into the same
request, including ones inherited from key/team policy, even when the rest
of those constraints were still individually satisfiable.
Before falling back, drop any required tag not carried by any deployment in
the group and recompute: if a specific, non-empty answer exists using only
the recognized tags, the unrecognized tag was the actual cause of the
exhaustion, and fail-open must not paper over it. If every required tag is
already recognized, or none are, there's nothing hidden behind an invented
tag, and fail-open proceeds exactly as before; this keeps a single opted-in
deployment's legitimate catch-all behavior working when a caller's tag
simply doesn't exist anywhere in that group.
Ratchets ANN401 and LIT001 budgets down to reflect fixes already earned in
this branch.
* test(router): cover required-AND, allow_fail_open, and unknown-tag denial across fallback chains and model groups
Extends coverage beyond single-hop scenarios: & exhausting a primary group
falls through to a fallback group exactly like ! already does; !, &, and
allow_fail_open composed together across three chained model groups each
raise or fall back independently per-hop; and the unknown-tag denial from
the previous commit is evaluated fresh per hop rather than leaking state
across groups in a fallback chain.
* feat(router): add model_info.enable_tag_filtering per-model-group override
enable_tag_filtering was router-wide only: an operator turning it on for one
model group that needs tag-driven routing exposed every other model group on
the same proxy to the same tag evaluation, even ones that never use tags.
Adds model_info.enable_tag_filtering, checked against any deployment sharing
a model_name, so a chain can flip the router-wide default in either
direction for itself alone: opt a specific group into filtering while the
rest of the proxy stays off, or opt a group out (e.g. an incident-response
catch-all) while the rest of the proxy enforces it.
Precedence, low to high: router-wide default, then the chain override if
set, then the existing request-level escalation (from key/team settings),
which still only ever turns filtering on, never off, over whatever the
router and chain already decided.
Also regenerates ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
field.
* fix(router): gate plain-tag exhaustion on allow_fail_open when the tag is known to the group
A model group where every deployment is tagged "default" (a legitimate
cross-cutting safety-net pattern) never has an empty default_deployments
list, so the existing exhaustion check (len(new)==0 and len(default)==0)
never fired for a plain positive tag that matched nothing among the
currently healthy candidates. The request silently fell through to whatever
"default"-tagged deployment happened to survive, even when allow_fail_open
was never set and the caller's intent (e.g. quality:high) was never honored.
Adds a check for whether the requested tag is part of the group's real
vocabulary at all: if some deployment configured under this model_name
(regardless of current health) genuinely carries the tag, and nothing
healthy currently matches it, the request now raises by default or falls
back per allow_fail_open, through the same _resolve_or_fail_open gate every
other exhaustion path already uses. A tag that's foreign to the group
entirely (e.g. one meant for an unrelated mechanism sharing the same
request-tags list) keeps falling back to the default pool unconditionally,
unchanged, since there's nothing this group's own routing intent could be
violating.
* fix(router): preserve inherited tag constraints when allow_fail_open discards a caller-caused exhaustion
Adds metadata.caller_tags in litellm_pre_call_utils.py, populated only from
what the request itself supplied (header, body tags, body metadata.tags),
never from key/team metadata merged into the same metadata.tags list.
get_deployments_for_tag now uses it to compute a trusted-only pool before
falling open: a required/excluded tag attributable to the caller can be
discarded on fail-open, one inherited from key/team policy cannot. If the
trusted-only pool is itself empty, allow_fail_open raises instead of
silently routing around an unsatisfiable inherited constraint. When
caller_tags carries no information at all (direct SDK Router usage,
bypassing the proxy layer), behavior is unchanged: unconditional fall-open
to the default pool, exactly as before this fix.
* feat(router): add opt-in tag_routing_prefix for collision-proof tag disambiguation
router_settings.tag_routing_prefix lets a caller explicitly mark which
x-litellm-tags/metadata.tags values are routing directives, exempting
them from the known-tag-vocabulary heuristic used to guard fail-open
against caller-invented "&"/"!" tags. Unprefixed tags keep going
through today's existing handling unchanged (hybrid, no migration
required); default "" is a full no-op.
Fixes a bug caught during live-proxy verification: the prefix-stripped
"confirmed" set kept the "&"/"!" marker character, so it never matched
required_set/excluded_set (which _split_tags always strips bare) -- the
entire trusted-required/excluded-tag mechanism silently no-opped for
its primary use case. Adds regression tests for the bare-value mismatch
and updates existing _chain_allows_fail_open/_tag_known_to_group/
_caller_constraint_sets call sites for the new routing_confirmed/
routing_prefix parameters.
* fix(router): resolve model_info.enable_tag_filtering override from the full model group, not just healthy deployments
Cooldown filtering runs before get_deployments_for_tag, so
_chain_tag_filtering_override only saw the survivors of that filter.
A model group whose only enable_tag_filtering-carrying deployment goes
into cooldown lost the override entirely, silently falling back to
the router-wide default and letting any !/&/tag constraint on that
chain be bypassed by driving the one overriding deployment into
cooldown. Resolve the override from every deployment configured for
the model instead, mirroring _tag_known_to_group's existing pattern.
Verified live: with a bad-key deployment carrying the override forced
into real cooldown via allowed_fails=1, an explicit "!provider:openai"
ban on the remaining deployment reproducibly returned 200 via OpenAI
before this fix and 401 (tag filtering still enforced) after it.
* fix(router): avoid Final-reassignment lint error and a MagicMock router fixture gap from tag_routing_prefix
_chain_tag_filtering_override's try/except reassigned a Final-annotated
name across branches, which basedpyright flags as illegal; extracted
the lookup-with-fallback into its own helper so the binding is assigned
once. Also sets tag_routing_prefix on the bare MagicMock router used by
test_router_tag_regex_routing.py's fixture, which otherwise returns an
auto-generated MagicMock (truthy, non-string) for the new attribute and
crashes _strip_routing_prefix's removeprefix() call.
* fix(router): key inherited-tag protection off provenance, not value subtraction
allow_fail_open's trusted-only pool computed "not caller-attributable"
as required_set - caller_required_set. A caller who resubmits the
exact value of an inherited "&"/"!" tag (e.g. an inherited "®ion:eu"
alongside a caller-supplied "®ion:eu" plus a conflicting
"!region:eu") collapses both origins to the same set value, so the
subtraction zeroes out the inherited requirement's protection too,
letting fail-open route outside a key/team-enforced constraint.
Adds metadata.inherited_tags in litellm_pre_call_utils.py: a snapshot
of "tags" taken after key/team/project policy is merged in but before
this request's own caller-supplied tags are merged on top. A required
or excluded tag is now protected from fail-open discard if it has ANY
inherited backing (set intersection with inherited_tags), regardless
of whether the caller also happens to submit the identical value --
this is what set membership alone could never tell apart under the
old subtraction-based approach. caller_tags is kept (documented as the
complementary record) but no longer consulted for this decision.
Verified live: a virtual key with metadata.tags=["®ion:eu"] hit
with header x-litellm-tags: ®ion:eu,!region:eu (the exact
value-collision attack) reproducibly routed to the OpenAI/us
deployment before this fix and stayed on the Anthropic/eu deployment
after it.
* fix(lint): re-ratchet budget ceilings after rebasing onto litellm_internal_staging
Regenerated ruff-strict-budget.json and type-discipline-budget.json
via make lint-ruff-budget-update / lint-type-discipline-budget-update
against the post-rebase merge-base.
* fix(proxy): compute inherited_tags from key/team/project sources directly, not a tags-list snapshot
apply_client_tag_policy_pre_auth (run from user_api_key_auth, for
_tag_max_budget_check) merges the caller's x-litellm-tags header into
the same metadata.tags list before add_litellm_data_to_request ever
runs. The previous inherited_tags snapshot ("whatever's in tags before
this function's own caller-tag merge") therefore misattributed that
caller-controlled value as policy-backed whenever a request arrived
with the header set -- Greptile flagged this as a P1 security finding.
inherited_tags is now built directly from key_metadata/team_metadata/
project_metadata's own "tags" fields, independent of the shared,
pipeline-position-dependent "tags" list's mutation history. Verified
with a direct reproduction mirroring the real pipeline (calling
apply_client_tag_policy_pre_auth on the same data dict before
add_litellm_data_to_request, as user_api_key_auth actually does): the
caller's header tag no longer appears in inherited_tags. Added a
regression test exercising that same call order; confirmed it fails
against the pre-fix snapshot approach and passes against this fix.
* fix(router): make tag_routing_prefix configurable through update_settings/get_settings and UpdateRouterConfig
router_settings.tag_routing_prefix was only ever applied via the
Router() constructor. Router.update_settings's _allowed_settings
(used directly by proxy_server.py's _add_router_settings_from_db_config
for the DB-backed router_settings path) and get_settings's
vars_to_include both omitted it, so an operator relying on that path
had the value silently ignored -- flagged by veria-ai. Also adds it to
UpdateRouterConfig (the pydantic schema behind POST /config/update),
the same bug shape LIT-3152 previously fixed for retry_policy: a field
missing from that schema gets silently dropped by
model_dump(exclude_none=True) before update_settings is ever called.
* chore(ui): regenerate schema.d.ts for UpdateRouterConfig.tag_routing_prefix
Adding tag_routing_prefix to UpdateRouterConfig changed the proxy's
OpenAPI spec; regenerate the dashboard's generated API types to match.
* fix(lint): re-ratchet budget ceilings after rebasing onto litellm_internal_staging
Regenerated ruff-strict-budget.json and type-discipline-budget.json
against the post-rebase merge-base. LIT002/LIT011 ceilings reflect
this branch's true current counts (confirmed unchanged across the
rebase by diffing against the pre-rebase commit); the base's own
counts moved independently.
* fix(lint): replace mutable-collection fallbacks with immutable ones in inherited_tags computation
key_metadata/team_metadata/project_metadata's "tags" fallbacks used
`or {}` / `or []` literals, each a LIT002 mutable-collection-construction
violation that pushed the branch 4 over its ratchet ceiling relative to
a moved base. Swapped to MappingProxyType({}) / () to match the
immutable idiom the rest of tag_based_routing.py already uses; no
behavior change, since both are falsy and only ever read via .get()/
unpacking. Tightens type-discipline-budget.json's LIT002 ceiling back
down to match, fully closing that gap (LIT011 keeps a genuine 1-count
gap from pre-existing, untouched lines in this file, non-gating).
* fix(lint): suppress LIT011 on the two new data[...] mutation sites
Both new lines follow this file's established data[...] mutation
idiom for add_litellm_data_to_request, matching the existing
suppression already on the inherited_tags line.
* test(router): lock in fallback + tag-filtering interaction
Cover the router-level fallbacks mechanism composing with tag-based
routing: a plain negation exhausting a group correctly advances to
the fallback group, the same exclusion tag exhausting every hop
correctly raises, and allow_fail_open resolving locally must not
spuriously trigger an unrelated external fallback.
* chore: retrigger CI now that litellm-docs#814 is merged
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(proxy): add config_updated_at audit timestamp for virtual keys
updated_at carries Prisma's @updatedAt, so every batched spend flush
rewrites it and it cannot distinguish config changes from usage. Add an
additive config_updated_at column stamped only by key management writes
(update, bulk update, regenerate, block, unblock) via a shared helper,
expose it on key responses, and switch the key page's Last Updated to it
with a created_at fallback.
* test(proxy): assert config_updated_at survives key archival
* refactor(proxy): rename config_updated_at to settings_updated_at
* feat(ui): show models under each tier in routing benchmark chart
- Add TierTurnsChart: donut chart showing turns per complexity tier with
tier-assigned models listed below each tier name in the legend
- Only complexity routers show models; quality routers show tier name + %
(quality tiers don't pin specific models)
- Change 'Estimated spend at highest-cost model' wording to 'highest-tier'
to clarify it's the most capable tier's estimated cost, not just the
single-highest model
Closes LIT-5302
* fix(ui): use categorical colors for tier donut, trim redundant turn count
- Tier donut chart now uses a dedicated categorical palette instead of
SEQUENTIAL_COLOR_RAMP, which is a blue monochrome gradient meant for
magnitude series, not distinct categories.
- Space out the tier legend rows (gap-3 -> gap-6) for readability.
- Drop the turn count from "avg saved per session" since Routing by
tier already shows the total turns.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(ui): drop prohibited explanatory comments in TierTurnsChart test
Per repo convention against source comments; the test name and
assertions already communicate the scenario. Addresses Greptile review.
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove dead modulo from color index in TierTurnsChart
The colors array is built with length equal to slices.length, so idx % colors.length
is always a no-op in the render loop. Simplify to idx for clarity.
* fix(ui): wrap CostOptimizationView tests in QueryClientProvider
The tests render CostOptimizationView which uses useCan() → useIsOrgAdmin() →
useOrganizations() and useDailyActivityRange(), both of which call React Query's
useQuery(). Without QueryClientProvider wrapping the render, React Query throws
'No QueryClient set' error.
Also mock the required networking calls (organizationListCall, userDailyActivityCall)
to prevent spurious network errors in test runs.
All 7 tests now pass (CostOptimizationView + CostOptimizationView.activity).
* style(ui): format test files and extract object literal to fix linting
- Format CostOptimizationView.test.tsx with prettier
- Extract getToolSpend mock response to named variable to satisfy eslint
- Pass frontend-lint checks
* fix(ui): hoist mockToolSpendResponse into vi.hoisted to fix test initialization
Extracting the response object to a named variable violated hoisting rules:
vi.mock() factories are evaluated at hoisting time before regular const
declarations. Move mockToolSpendResponse into vi.hoisted() block.
---------
Co-authored-by: Claude <noreply@anthropic.com>
- formatInstallCommand now produces /plugin install {name}@litellm instead of /plugin marketplace add {source}
- extraKnownMarketplaces snippet fixed: source must be a nested object not a flat string; the flat string caused Claude Code to reject the settings file
- marketplace key renamed from my-org to litellm to match the name the proxy returns in marketplace.json
- setup tab now shows /plugin marketplace add command as primary option with settings.json as secondary
- usage tab now shows a hint to run /plugin marketplace update litellm when a plugin is not found
* fix: net prompt-caching savings against the cache-write premium
Prompt-caching savings priced only the cache-read discount and ignored what
the provider charges to create the cache entry. Anthropic bills cache writes
at 1.25x the input rate, so a request that writes a large cache and reads
little from it is a net loss that the dashboard reported as a gain -- or, on
a pure cold write, as a flat zero.
The counterfactual the number answers is "what would this have cost with
caching off", where every token is billed at the input rate. Since
prompt_tokens partitions disjointly into text + reads + writes, that gives
savings = reads * (input - read_rate) - writes * (write_rate - input)
The write term is the premium over the input rate, not the full write cost:
the tokens would have been paid for at the input rate anyway, so only the
markup is attributable to caching.
The premium stays signed rather than clamped. Three models in the pricing map
price writes below input, and clamping would silently drop that saving.
A model with no cache_creation_input_token_cost falls open to the input cost,
yielding a zero premium -- this is why the change is a no-op for the implicit
caching providers (OpenAI, Gemini), which publish no write price, and bites
exactly on Anthropic and Bedrock.
Verified live through the proxy on a mock Anthropic rig across four cases
(cold pure-write, warm pure-read, write-heavy, read-heavy). Reported total
matched the derived net to the cent, including the negatives; the read-only
case is unchanged.
Pre-existing rows are not backfilled, so a range spanning the deploy mixes
gross and net.
* fix: read a zero cache-write price as unpublished, not free
deepseek-chat carries a literal 0.0 cache_creation_input_token_cost. The
fall-open only caught None, so the zero was taken at face value and the
premium became 0 - input_cost -- reporting a fabricated saving of
writes * input_cost on traffic that cached nothing.
No provider gives cache writes away, so a falsy price means the same thing
an absent one does.
* test: pin that the read leg keeps a literal zero price
The two zero prices mean opposite things and the asymmetry was unpinned.
A free cache write is unpublished pricing; a free cache read is real, and
15 models charge for input while serving reads for nothing. Copying the
write leg's falsy fall-open onto the read leg would zero out their savings.
* refactor: resolve caching rates through the established pricing helpers
Addresses Greptile's P1 and P2, and replaces hand-rolled pricing lookup with
the patterns this file and the cost calculator already own:
- Deployment pricing first: rates now resolve through _effective_model_info
(Router.get_deployment_model_info), the same helper the autorouter driver
uses, falling back to _model_info public rates. A deployment with negotiated
cache rates previously priced at the public map -- a 3x error on the repro.
- Individual prices read via _get_cost_per_unit, the cost calculator's
accessor, which also coerces string prices from config.yaml and resolves
service-tier suffixes; the previous raw .get() handled neither.
- Pricing tests no longer monkeypatch litellm.get_model_info; each case now
pins a real pricing-map entry with a fixture-drift assertion, and the
deployment-rate case follows the existing Router-fixture test pattern.
Behaviour on public rates is unchanged: 101 tests pass, including the exact
same live-verified formula.
* fix(cost-optimization): computeCacheLeakage divides net savings by all cached tokens, not reads alone
prompt_caching_savings_spend is net of the cache-write premium since PR #36452.
computeCacheLeakage was still dividing by cache_read_tokens alone, which:
1. Overstates the per-token rate on traffic that writes and reads cache equally:
a 1:1 read:write key shows rate = 0.002, not 0.001, if net savings is /bin/zsh.002
2. Flips the sign on write-heavy traffic: when writes cost more than reads save
(common on Anthropic and Bedrock), the aggregate net can go negative, but
dividing by reads alone would show a positive 'potential savings' for keys
that don't cache yet — recommending they start caching when it's currently
losing money overall
Fix: divide realizedCachingSavings by (cacheReadTokens + cacheCreationTokens),
matching the semantic that a key starting to cache pays those write premiums too.
When the rate is non-positive, price nothing (potentialSavings stays null, renders
as '—'), reusing the existing no-data fallback path. The card can't meaningfully
estimate savings from a losing rate.
Rename discountPerToken → netSavingsPerCachedToken to surface the semantics and
prevent this drift in future.
Update Usage tab and Cache Leakage card tooltips to describe net-of-premium cost.
Add tests for 1:1 read:write traffic and write-heavy negative-net traffic.
* feat(proxy): add per-deployment keepalive_seconds SSE heartbeat for long-running streams
Adds _iter_with_keepalive, _keepalive_from_deployment_config, and
_resolve_keepalive_seconds helpers to proxy_server.py. When enabled
(keepalive_seconds > 0 in request body or deployment litellm_params),
async_data_generator emits ': ping\n\n' SSE comment frames every N
seconds during idle upstream intervals, preventing load-balancer
idle-timeout drops on long chain-of-thought reasoning streams.
The hot path (keepalive_seconds absent or 0) is a plain async-for with
no per-chunk Task wrapping — zero overhead. Includes 8 new unit tests
covering sentinel emission, hot-path pass-through, early-close cleanup,
priority resolution, deployment-config lookup, and end-to-end heartbeat
emission through async_data_generator.
Registers keepalive_seconds in all_litellm_params (types/utils.py) so
the parameter is not stripped from request bodies. Adds the field to
LiteLLMParamsTypedDict and GenericLiteLLMParams (types/router.py) so
deployment YAML config is parsed and validated.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): narrow BaseException to CancelledError to fix BLE001 strict lint gate
* fix: use explicit None check instead of truthiness in keepalive_seconds extraction
`float(raw or 0)` would treat any falsy value (including the integer 0)
as absent and substitute 0.0 before float() saw it. Replace with
`float(raw) if raw is not None else 0.0` so a caller-supplied zero is
correctly passed through to the `value <= 0` guard that disables
keepalive, rather than being silently overwritten.
* fix(proxy): don't guess a deployment's keepalive_seconds when model_id is missing
When a streaming response lacks _hidden_params.model_id, the fallback that
looks up keepalive_seconds by model_name previously returned the first
configured deployment's value, which could apply the wrong interval (or
override an explicit disable) when multiple deployments share the same
model_name with different keepalive_seconds settings. Only resolve the
fallback when every deployment agrees; otherwise leave it unset.
* fix(proxy): also treat an unset keepalive_seconds as disagreement in the fallback
The model_name fallback for keepalive_seconds only compared configured
values, filtering out deployments that leave the field unset entirely.
That meant a deployment with no keepalive_seconds configured could still
inherit a sibling deployment's interval when model_id is unavailable.
Compare the raw per-deployment value (including None for unset) so an
unconfigured deployment never silently adopts another's heartbeat.
* fix(proxy): deployment-level keepalive_seconds: 0 is a hard disable clients can't override
Previously an authenticated client's request-level keepalive_seconds always
took precedence over the deployment default, including when a deployment
operator explicitly set keepalive_seconds: 0 to disable heartbeats. That let
any client re-enable heartbeats for a deployment the operator opted out of,
using them to keep an idle-looking stream alive past a load balancer's idle
timeout and hold a parallel-request slot open longer than intended.
Treat an explicit deployment-level 0 as authoritative: resolve the
deployment's configured value first, and short-circuit to disabled before
ever looking at the request body if the deployment hard-disabled it.
* fix(proxy): a stale (unresolvable) model_id must not fall through to model_name guessing
A populated _hidden_params.model_id names the specific deployment that
served a stream. If that ID no longer resolves (e.g. a deployment removed
by a config reload mid-stream), the resolver was falling through to the
model_name-based fallback, letting a currently-live sibling deployment's
keepalive_seconds silently apply to a stream it never served. Return None
once a populated model_id fails to resolve, rather than degrading to a
guess.
* fix(proxy): keepalive_seconds is operator-only by default; require deployment opt-in for client override
A security review flagged that a client's request-level keepalive_seconds
could unilaterally enable heartbeats for any deployment, even one that
never configured keepalive_seconds at all, letting an authenticated client
defeat load-balancer idle timeouts and hold a parallel-request slot open
for longer than the deployment operator ever intended, with no way for
the operator to prevent it short of explicitly setting keepalive_seconds: 0.
Add allow_client_keepalive_override (default False) to LiteLLMParamsTypedDict
and GenericLiteLLMParams. _resolve_keepalive_seconds now ignores the request
body's keepalive_seconds entirely unless the resolved deployment explicitly
grants override permission; only the deployment's own configured value (or
disabled, if unset) applies otherwise. An explicit deployment-level 0 still
takes priority over everything, including a grant of override permission.
* fix(proxy): register allow_client_keepalive_override in all_litellm_params
Caught during live proxy verification against the real Anthropic API:
allow_client_keepalive_override was added to LiteLLMParamsTypedDict and
GenericLiteLLMParams but never registered in all_litellm_params, so it
leaked straight through into the provider request body as an unrecognized
field. Anthropic rejected every call on a deployment that had this field
configured with a 400 ("Extra inputs are not permitted"), regardless of
its value. Register it alongside keepalive_seconds so it's stripped
before reaching the provider, matching what keepalive_seconds already
does.
* feat(proxy): support keepalive_seconds via x-litellm-keepalive-seconds header
Some clients (e.g. the Vercel AI SDK) can set custom headers more easily
than extra JSON body fields. Add x-litellm-keepalive-seconds, following
the existing x-litellm-timeout/x-litellm-stream-timeout/x-litellm-num-retries
convention in LiteLLMProxyRequestSetup: the header merges into the same
data["keepalive_seconds"] field the request body already populates, so it
goes through the exact same _resolve_keepalive_seconds precedence and the
allow_client_keepalive_override gate -- a header can't enable heartbeats
for a deployment that hasn't opted in any more than the body field can.
Verified live against the real Anthropic API: the header produces real
heartbeats on an opt-in deployment (88 pings over a genuine long-reasoning
stall) and is silently ignored on a deployment without override permission
(0 pings), matching the existing body-field behavior exactly.
* chore: rebase onto litellm_internal_staging, drop unrelated credential_migration.py reformat, fix budget-ratchet drift
Rebased onto the current litellm_internal_staging (merge-base was 5 days
stale). Dropped the now-redundant schema.d.ts-only regen commit entirely
(the new base's own schema.d.ts already supersedes it) and regenerated
schema.d.ts fresh against the new base.
Reverted litellm/proxy/management_endpoints/credential_migration.py to
exactly match litellm_internal_staging: it was a pure reformat with no
semantic change, unrelated to this PR, flagged by review as unnecessary
noise in an encryption-migration file.
Fixed two lint-budget-ratchet failures caused by the base's ceilings
tightening since this branch last synced (other merged work lowered
ANN401/LIT001 budgets; this code was previously under budget and didn't
change):
- _iter_with_keepalive's aiter param: Any -> AsyncIterator[Any], a real
narrowing (it's always the result of .__aiter__()).
- _keepalive_from_deployment_config/_resolve_keepalive_seconds's
request_data param: dict[str, Any] -> Mapping[str, Any], matching the
existing read-only-dict convention already used elsewhere in this file
(_apply_ssrf_general_settings, _build_redis_usage_cache, etc.) for
params that are only ever read, never mutated.
- response/raw params: dropped the explicit `Any` annotation to match
async_data_generator's own (deliberately unannotated) `response` param,
its actual caller.
- litellm_pre_call_utils.py's new headers param: dict -> Mapping[str, str],
same read-only-dict rationale.
* fix(proxy): freeze the transient collections in the keepalive helpers
_iter_with_keepalive and _keepalive_from_deployment_config built a set
literal for asyncio.wait, a set comprehension for the per-deployment
config-agreement check, and two dict-literal fallbacks, all flagged by
the LIT002 mutable-collection-construction gate. Switched to a tuple
for asyncio.wait, a frozenset-wrapped generator plus next(iter(...))
for the config check, and a shared MappingProxyType({}) empty mapping
for the fallbacks.
* fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback
Greptile P1: when a streaming request falls back from model group A to
group B and the response's _hidden_params carries no model_id,
_keepalive_from_deployment_config fell straight through to guessing
via request_data["model"], which still names the pre-fallback group A
since the fallback handler mutates its own local **kwargs copy, not
this dict. request_data[metadata|litellm_metadata]["model_info"]["id"],
by contrast, is mutated on this same dict by
Router._update_kwargs_with_deployment on every attempt including
fallbacks (the same source ProxyLogging._build_litellm_call_info uses
for logging), so check it before falling through to the model-name
guess.
Added two regression tests that fail on the prior code (assert
get_model_list is never called once metadata.model_info.id resolves)
and pass with the fix.
* Revert "fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback"
This reverts commit d779067864.
* fix(proxy): satisfy the new LIT010/ANN001 gates in the keepalive helpers
litellm_internal_staging picked up a LIT010 (every local/module variable
must be declared Final unless it's genuinely rebound) and tightened
ANN001 (missing parameter annotations) since this branch last synced.
Annotated every single-assignment local and module constant with
Final, suppressed pending's loop-carried reassignment with
# rebind-ok, and typed the previously-bare response/raw parameters as
object with isinstance narrowing at their use sites instead of cast
(LIT006 discourages cast; validate into a concrete type instead).
Also swapped the hand-rolled getattr(response, "_hidden_params", None)
+ isinstance(hidden, dict) check for the existing
get_hidden_params_dict() helper already used for this exact purpose
elsewhere in this file and in common_request_processing.py.
* fix(proxy): re-resolve keepalive_seconds per chunk to track mid-stream fallback
Greptile P1: the router can perform a mid-stream fallback to a
different deployment partway through a stream (MidStreamFallbackError
in router.py), and Router._apply_fallback_hidden_params_to_item merges
the fallback deployment's hidden params onto every subsequent chunk.
But _resolve_keepalive_seconds was only ever called once, before
iteration started, against the pre-fallback response wrapper, so a
stream that fell back to a deployment with a different (or disabled)
keepalive policy kept using the original deployment's interval for the
rest of the stream.
_iter_with_keepalive now takes a resolve_keepalive_seconds(item)
callback and re-resolves after every real chunk using that chunk's own
_hidden_params (which do carry the fallback deployment's identity),
rather than trusting the value picked before iteration began. Updated
the three existing timing tests to inject a constant-returning
resolver, since they pin the sentinel/cancellation mechanics rather
than re-resolution, and added two regression tests (interval lowered
and raised mid-stream) that fail against the prior static-resolve
signature and pass with the fix.
* fix(proxy): keep re-resolving keepalive even when a stream starts disabled
Greptile P1: a stream that starts on a deployment with keepalive off
(or unset) skipped _iter_with_keepalive entirely at the call site, so
a mid-stream fallback to a deployment that enables it never got a
chance to activate heartbeats for the rest of that stream, risking the
exact load-balancer idle-timeout this feature exists to prevent.
_iter_with_keepalive now has an internal fast path for
keepalive_seconds <= 0 that still re-resolves after every chunk (no
asyncio.create_task/wait overhead while inactive, same cost as a bare
async for), so activation from a disabled start works the same way
deactivation and interval changes already do. The caller now only
skips wrapping entirely when there's no router to ever fall back
through in the first place (llm_router is None), rather than whenever
the first chunk's deployment happens to start with keepalive off.
Added a regression test that starts keepalive_seconds=0, has the
resolver enable a short interval on a later chunk, and asserts
sentinels appear afterward; it fails against the prior
call-site-gated code and passes with the fix.
* perf(proxy): memoize keepalive resolution per chunk's model_id
_resolve_keepalive_seconds ran a full llm_router.get_deployment() Pydantic
rebuild after every streamed chunk, even when keepalive was unconfigured
anywhere in the deployment list, since async_data_generator wraps every
stream once a router exists. Caching the result by model_id keeps mid-stream
fallback re-resolution correct while paying the router lookup once per
deployment instead of once per token.
* fix(proxy): expire cached keepalive resolution after a bounded TTL
veria-ai flagged that caching by model_id alone lets an already-in-flight
stream keep evading a live config reload (deployment removed, keepalive
disabled, or client override revoked) for the rest of the stream. Expiring
the memo after _KEEPALIVE_CACHE_TTL_SECONDS bounds that window instead of
freezing the resolved value for the stream's full lifetime, while still
avoiding a full deployment rebuild on every chunk in the steady state.
Also fixes add_litellm_data_for_backend_llm_call's now-required request_data
kwarg in the header-merge test, picked up by rebasing onto
litellm_internal_staging.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The prompt editor is a base-ui Dialog at z-index 50. The create form houses it in
the same base-ui Dialog, so it stacks on top, but the edit form was an antd Modal
whose portal computes to z-index 1000, so the editor opened underneath it and was
neither readable nor clickable.
Move the edit form onto the Dialog the create form already uses, which puts the
whole nesting chain in one overlay layer. A dialog opened from inside another
dialog now reads as a drill-down rather than a stack: base-ui stamps
data-nested-dialog-open on the parent while a child is open, so the parent steps
aside instead of showing its own edges around a differently sized child.
Hiding the tab behind all_admin_roles took it away from org admins, who are
entitled to it: /v2/team/list?status=deleted returns 200 for them, scoped to
their own organizations. An org admin is an organization membership rather
than a global role, so their session carries user_role "internal_user" and no
role-based gate can ever see them.
Lift the membership lookup the left nav already did into a shared
useIsOrgAdmin hook, and let a capability opt into allowing org admins.
viewDeletedTeams is the only one that opts in; the backend still refuses org
admins on /v1/tool/list, /policies/list, /prompts/list and /audit, so those
gates stay as they are. The hook also accepts a session role of org_admin, in
case a deployment maps one through SSO.
Drops Any-typed router plumbing, immutable bucketing, generated dashboard API types, and adds endpoint plus resolution-fallback tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
knip derives its entry points from vitest's `test.include`, which does not
cover `test.typecheck.include`, so the new `*.test-d.tsx` file read as an
unused file and failed the lint job. Declare the glob as an entry point.
Also drops the doc comment on `DataTableResolvedProps`; the rationale for
the resolved/public split belongs in the commit that introduced it.
The removed comments restated the test names and the assertions directly
below them. The reasoning they carried is already recorded in the commit
that introduced the fix and in the pull request body.
Workflow Runs, Memory and Guardrails Monitor were visible to every role
while their page-load routes are proxy-admin-only, so a non-admin got a
page shell and a 401. Cost Optimization was half-broken the same way: its
Overall charts run on /user/daily/activity, which every role may call, but
tool spend, prompt caching, prompt compression and auto-router benchmarks
are all proxy-admin-only.
Add viewWorkflowRuns, viewMemory, viewGuardrailUsage and
viewProxyWideCostData, each gating the nav entry, the page and the request
together. The first three hide their page, including the direct-URL path,
since nothing on them works for a non-admin. Cost Optimization keeps its
page and drops only the parts a non-admin cannot read.
Gating both Agentic children left roles with no visible child rendering the
parent as a leaf link to ?page=agentic, which is not a route, so a parent
whose children are all filtered out is now dropped.
Role lists follow what the proxy actually grants: proxy_admin and
proxy_admin_viewer are served, and org admins are not, because
_user_is_org_admin needs an organization_id that a page-load GET never
carries.
* feat(ui): show vector store indexes on the Vector Stores page
Adds a proxy-admin-only Indexes tab listing rows from GET /v1/indexes:
index name, backing vector store, provider index, creator, and created
date. The tab is hidden for non proxy-admin roles to match the
endpoint's gate, and data loads lazily on first visit.
* feat(ui): link index rows to their vector store and creator
Vector Store cells open the store's info view when the name resolves to
a registered store, and Created By cells deep link to the users page via
a new userDetailHref, with the users page reading the user query param
through nuqs so the link is shareable.
* feat(ui): link docs and note supported providers on Indexes tab
* fix(ui): show not-found state instead of infinite loading for missing vector store
DataTable accepted any mix of its 40-odd props and rejected the incoherent
combinations at runtime, from a validator that threw during the first render.
A caller only found out it had wired server sorting without a `sorting` prop
when the page blew up in front of them.
Split the public prop type into mode-keyed unions instead, so the compiler
rejects those combinations at the call site. `validateDataTableConfig` and
`DataTableConfigError` go away; the component body reads an unchanged flat
`DataTableResolvedProps`, which every union member is assignable to, so there
is no narrowing inside it.
All 44 existing call sites typecheck against the new union unchanged, which
`next build` covers. That build only typechecks the app module graph, so the
prop type itself needed a gate of its own: `npm run test:types` runs vitest's
typecheck mode over `*.test-d.tsx`, and the unit workflow now runs it. The
four guards deleted from `DataTable.test.tsx` come back there as compile-time
assertions, and loosening the union back to the flat shape fails all five.
The Old Usage nav entry carried no role restriction, so every role saw it
and the page immediately fired eight /global/spend/* requests that the
proxy withholds from non-admins, producing a wall of 401s.
Gate the nav entry, the page, and both of its mount effects behind a
single viewGlobalSpend capability scoped to proxy_admin and
proxy_admin_viewer, matching what the backend actually serves.
Also drop the session JWT that adminspendByProvider put in the
/global/spend/provider query string; the handler never read it.
The Virtual Keys table and the Logs page team filter both asked for every
team on the proxy, which /v2/team/list and /team/list reject with a 401 for
any role below proxy admin or org admin. Both endpoints answer the same
request with the caller's own teams when it carries a user_id, so send one.
Only the two unscoped call sites change. The remaining callers either
already role-branch or render on surfaces gated to roles the endpoints
answer broadly, and scoping those would shrink the list they see: a proxy
admin scoped to their own id gets nothing back, and an org admin scoped on
/team/list loses the org teams they administer but do not belong to.
The shared helper reads the display-form session role rather than
all_admin_roles, which mixes display labels with raw role names and so does
not match the "Org Admin" value the dashboard actually holds.
Merging staging's flat-cost summary work with the capability gating pushed
EntityUsage.tsx to 815 counted lines, over the 800-line eslint cap. Move the
four pure top-N/rollup helpers to entityUsageAggregations.ts and pass their
inputs explicitly.
TopKeyView and TopModelView were mocked to render static text, so nothing
asserted which breakdown fed which table. The mocks now surface their rows and
a new case pins each table to its own data source.
