Gemini image usage carries prompt_tokens and friends as extra fields on
ResponseAPIUsage, which collided with the bridge's explicit kwargs and
raised TypeError. Exclude keys the bridge already sets explicitly.
Drop the transform overrides that swapped response.usage to the chat
shape, which broke the /v1/responses client contract. Provider extras
like server_side_tool_usage_details already survive validation via
ResponseAPIUsage extra fields, so the shared usage bridge now carries
them onto the bridged chat Usage generically. The web_search_call
output gate also reads dict output items, since items that fail SDK
validation stay plain dicts, and the chat path gains billing tests.
* fix(deps): ship boto3 with the base SDK so bedrock works out of the box
* keep boto3 listed in the proxy extra as well
* scrub ambient AWS env vars in the base SDK bedrock smoke check
* fix(alerting): dedupe scheduled Slack spend reports across pods
Every pod ran its own weekly/monthly spend report jobs, prometheus
fallback stats cron, and daily report loop, so deployments with
multiple replicas or uvicorn workers received one copy per pod.
Gate each scheduled send behind the shared PodLockManager redis lock.
The lock is never released: its TTL (the full reporting window for the
weekly interval job, whose per-pod anchors drift by boot time and
jitter) doubles as a sent-this-window marker. acquire_lock returning
None (no redis wired) proceeds, preserving single-pod behavior.
Also generalize the pod lock could-not-acquire log line, which claimed
to be about spend tracking for every consumer.
Fixes#14809
* fix(alerting): harden spend report locks after adversarial review
Weekly lock TTL gets an hour haircut: with ttl equal to the interval,
the winner re-fires just before its own key expires, reacquires without
a TTL refresh, and the key then lapses in time for a trailing pod to
re-send. Job/lock ids move to litellm/constants.py per convention, and
spend_report_frequency now rejects non-positive day counts, which
previously coerced to an every-second schedule and would now compute a
negative lock TTL that silently never sends.
Adds the missing test coverage the review flagged: startup_event's
pod_lock_manager wiring (identity-asserted), the prometheus closure's
positive path, and the ungated immediate prometheus send pinned to
exactly one await.
* test(alerting): consolidate spend_report_frequency validator coverage
Drops a duplicate non-positive-days test and parametrizes the survivor
over the suffix half of the validator too
* fix(alerting): route the startup prometheus fallback send through the pod lock
Greptile caught that the boot-time send still ran once per pod when
PROMETHEUS_URL is set, the same duplication class this PR removes
* fix(alerting): make report lock acquisition non-reentrant
Greptile caught that a pod booting within an hour of the fallback stats
cron sent twice: the startup send takes the lock, then the cron fire
hits acquire_lock's reacquire branch, which returns True for the
holder. Window-marker gates now pass allow_reentrant=False so a live
lock blocks everyone including its holder; leader-election consumers
keep the reentrant default
* test(proxy): give spec'd ProxyLogging mocks a db_spend_update_writer
_initialize_slack_alerting_jobs now reads it for the pod lock manager,
and spec=ProxyLogging blocks instance-only attributes
255d65192e added useCan to UsageTab, whose useIsOrgAdmin leg calls
useOrganizations (react-query), so every UsageTab test died with 'No
QueryClient set'. Stub the org-admin leg; role gating still flows through
the real hasCapability with the varied userRole.
* feat(proxy): per-key prompt caching auto-injection via enable_prompt_caching
Adds a key-level enable_prompt_caching toggle that auto-injects Anthropic
cache_control breakpoints on requests made with that key, without requiring
the gateway-wide enable_anthropic_prompt_caching flag. The flag lives in key
metadata, is stamped onto the request root by add_key_level_controls, rides
kwargs into both the /chat/completions seeding path and the native
/v1/messages path, and reuses every existing gate (anthropic/bedrock only,
supports_prompt_caching, client markers win). Client-supplied body values are
stripped as an untrusted root control field. Includes the Admin UI switch on
key create and key edit plus a read-only settings row, and dedupes the key
edit view's drifted initial-values objects.
* fix(proxy): drop section comment and suppress LIT011 on key-level prompt caching stamp
* feat(router): add required-AND (&) tag prefix and allow_fail_open flag
Tag routing supported inclusion-OR and independent "!" negation, but had no way
to express a hard "must match all of these" constraint per request, and no way
for a model group to opt into degrading gracefully instead of raising when a
constraint eliminates every deployment.
Adds a "&tag" prefix for required-AND inclusion, composing with existing plain
(OR) and "!" (negate) tags: negation still applies first, then required tags
narrow the survivors, then plain tags apply today's OR/AND preference logic
unchanged. Adds model_info.allow_fail_open (default false) so a chain can opt
into falling back to the default-tagged pool instead of raising
no_deployments_with_tag_routing when "!" or "&" empties the candidate set;
existing chains without the flag keep today's fail-closed behavior exactly.
* fix(router): gate mixed negation on allow_fail_open and stop diluting required-only requests
Two gaps in the initial required-AND/allow_fail_open change: a "!" exclusion
combined with a plain positive tag that emptied the candidate set raised
unconditionally, bypassing allow_fail_open entirely, since the fail-open check
only looked at required-AND exhaustion. And a request using only "&" tags
could get narrowed down to just the deployment matching an incidental
tag_regex/User-Agent preference, silently dropping other deployments that
satisfied the required tags but had no tag_regex at all.
Fixes both: the fail-open check now fires whenever either "!" or "&" leaves
the candidate set empty, not just "&". And regex/header preference no longer
counts as a positive filter when a required-AND ask is present, so a
required-only request returns every deployment satisfying the required tags
regardless of regex/header matching.
Also regenerates ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
model_info.allow_fail_open field, and removes source comments explaining
the router logic per repository convention.
* fix(router): let allow_fail_open cover a non-empty !/& survivor set that fails the plain-tag preference
The unconditional raise inside the has_positive_filter loop was the one
remaining path a chain could hit despite setting allow_fail_open: when "!"
or "&" leaves a non-empty candidate set but none of the survivors match the
request's plain preference tag or carry "default", the request still raised
instead of degrading. Routes that raise through the same allow_fail_open
check used everywhere else, so it now falls back to the default-tagged pool
for opted-in chains and keeps raising unconditionally for everyone else.
This also let the now-redundant pre-loop empty-candidates shortcut be
removed, since the loop reaches the same outcome on its own.
* fix(router): deny allow_fail_open when an unrecognized required tag is masking a satisfiable answer
A caller could add a single "&" tag no deployment in the group has ever
carried to force an empty required-AND set on demand. On a chain with
allow_fail_open, that emptied set fell back to the default-tagged pool
unconditionally, discarding every other constraint merged into the same
request, including ones inherited from key/team policy, even when the rest
of those constraints were still individually satisfiable.
Before falling back, drop any required tag not carried by any deployment in
the group and recompute: if a specific, non-empty answer exists using only
the recognized tags, the unrecognized tag was the actual cause of the
exhaustion, and fail-open must not paper over it. If every required tag is
already recognized, or none are, there's nothing hidden behind an invented
tag, and fail-open proceeds exactly as before; this keeps a single opted-in
deployment's legitimate catch-all behavior working when a caller's tag
simply doesn't exist anywhere in that group.
Ratchets ANN401 and LIT001 budgets down to reflect fixes already earned in
this branch.
* test(router): cover required-AND, allow_fail_open, and unknown-tag denial across fallback chains and model groups
Extends coverage beyond single-hop scenarios: & exhausting a primary group
falls through to a fallback group exactly like ! already does; !, &, and
allow_fail_open composed together across three chained model groups each
raise or fall back independently per-hop; and the unknown-tag denial from
the previous commit is evaluated fresh per hop rather than leaking state
across groups in a fallback chain.
* feat(router): add model_info.enable_tag_filtering per-model-group override
enable_tag_filtering was router-wide only: an operator turning it on for one
model group that needs tag-driven routing exposed every other model group on
the same proxy to the same tag evaluation, even ones that never use tags.
Adds model_info.enable_tag_filtering, checked against any deployment sharing
a model_name, so a chain can flip the router-wide default in either
direction for itself alone: opt a specific group into filtering while the
rest of the proxy stays off, or opt a group out (e.g. an incident-response
catch-all) while the rest of the proxy enforces it.
Precedence, low to high: router-wide default, then the chain override if
set, then the existing request-level escalation (from key/team settings),
which still only ever turns filtering on, never off, over whatever the
router and chain already decided.
Also regenerates ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
field.
* fix(router): gate plain-tag exhaustion on allow_fail_open when the tag is known to the group
A model group where every deployment is tagged "default" (a legitimate
cross-cutting safety-net pattern) never has an empty default_deployments
list, so the existing exhaustion check (len(new)==0 and len(default)==0)
never fired for a plain positive tag that matched nothing among the
currently healthy candidates. The request silently fell through to whatever
"default"-tagged deployment happened to survive, even when allow_fail_open
was never set and the caller's intent (e.g. quality:high) was never honored.
Adds a check for whether the requested tag is part of the group's real
vocabulary at all: if some deployment configured under this model_name
(regardless of current health) genuinely carries the tag, and nothing
healthy currently matches it, the request now raises by default or falls
back per allow_fail_open, through the same _resolve_or_fail_open gate every
other exhaustion path already uses. A tag that's foreign to the group
entirely (e.g. one meant for an unrelated mechanism sharing the same
request-tags list) keeps falling back to the default pool unconditionally,
unchanged, since there's nothing this group's own routing intent could be
violating.
* fix(router): preserve inherited tag constraints when allow_fail_open discards a caller-caused exhaustion
Adds metadata.caller_tags in litellm_pre_call_utils.py, populated only from
what the request itself supplied (header, body tags, body metadata.tags),
never from key/team metadata merged into the same metadata.tags list.
get_deployments_for_tag now uses it to compute a trusted-only pool before
falling open: a required/excluded tag attributable to the caller can be
discarded on fail-open, one inherited from key/team policy cannot. If the
trusted-only pool is itself empty, allow_fail_open raises instead of
silently routing around an unsatisfiable inherited constraint. When
caller_tags carries no information at all (direct SDK Router usage,
bypassing the proxy layer), behavior is unchanged: unconditional fall-open
to the default pool, exactly as before this fix.
* feat(router): add opt-in tag_routing_prefix for collision-proof tag disambiguation
router_settings.tag_routing_prefix lets a caller explicitly mark which
x-litellm-tags/metadata.tags values are routing directives, exempting
them from the known-tag-vocabulary heuristic used to guard fail-open
against caller-invented "&"/"!" tags. Unprefixed tags keep going
through today's existing handling unchanged (hybrid, no migration
required); default "" is a full no-op.
Fixes a bug caught during live-proxy verification: the prefix-stripped
"confirmed" set kept the "&"/"!" marker character, so it never matched
required_set/excluded_set (which _split_tags always strips bare) -- the
entire trusted-required/excluded-tag mechanism silently no-opped for
its primary use case. Adds regression tests for the bare-value mismatch
and updates existing _chain_allows_fail_open/_tag_known_to_group/
_caller_constraint_sets call sites for the new routing_confirmed/
routing_prefix parameters.
* fix(router): resolve model_info.enable_tag_filtering override from the full model group, not just healthy deployments
Cooldown filtering runs before get_deployments_for_tag, so
_chain_tag_filtering_override only saw the survivors of that filter.
A model group whose only enable_tag_filtering-carrying deployment goes
into cooldown lost the override entirely, silently falling back to
the router-wide default and letting any !/&/tag constraint on that
chain be bypassed by driving the one overriding deployment into
cooldown. Resolve the override from every deployment configured for
the model instead, mirroring _tag_known_to_group's existing pattern.
Verified live: with a bad-key deployment carrying the override forced
into real cooldown via allowed_fails=1, an explicit "!provider:openai"
ban on the remaining deployment reproducibly returned 200 via OpenAI
before this fix and 401 (tag filtering still enforced) after it.
* fix(router): avoid Final-reassignment lint error and a MagicMock router fixture gap from tag_routing_prefix
_chain_tag_filtering_override's try/except reassigned a Final-annotated
name across branches, which basedpyright flags as illegal; extracted
the lookup-with-fallback into its own helper so the binding is assigned
once. Also sets tag_routing_prefix on the bare MagicMock router used by
test_router_tag_regex_routing.py's fixture, which otherwise returns an
auto-generated MagicMock (truthy, non-string) for the new attribute and
crashes _strip_routing_prefix's removeprefix() call.
* fix(router): key inherited-tag protection off provenance, not value subtraction
allow_fail_open's trusted-only pool computed "not caller-attributable"
as required_set - caller_required_set. A caller who resubmits the
exact value of an inherited "&"/"!" tag (e.g. an inherited "®ion:eu"
alongside a caller-supplied "®ion:eu" plus a conflicting
"!region:eu") collapses both origins to the same set value, so the
subtraction zeroes out the inherited requirement's protection too,
letting fail-open route outside a key/team-enforced constraint.
Adds metadata.inherited_tags in litellm_pre_call_utils.py: a snapshot
of "tags" taken after key/team/project policy is merged in but before
this request's own caller-supplied tags are merged on top. A required
or excluded tag is now protected from fail-open discard if it has ANY
inherited backing (set intersection with inherited_tags), regardless
of whether the caller also happens to submit the identical value --
this is what set membership alone could never tell apart under the
old subtraction-based approach. caller_tags is kept (documented as the
complementary record) but no longer consulted for this decision.
Verified live: a virtual key with metadata.tags=["®ion:eu"] hit
with header x-litellm-tags: ®ion:eu,!region:eu (the exact
value-collision attack) reproducibly routed to the OpenAI/us
deployment before this fix and stayed on the Anthropic/eu deployment
after it.
* fix(lint): re-ratchet budget ceilings after rebasing onto litellm_internal_staging
Regenerated ruff-strict-budget.json and type-discipline-budget.json
via make lint-ruff-budget-update / lint-type-discipline-budget-update
against the post-rebase merge-base.
* fix(proxy): compute inherited_tags from key/team/project sources directly, not a tags-list snapshot
apply_client_tag_policy_pre_auth (run from user_api_key_auth, for
_tag_max_budget_check) merges the caller's x-litellm-tags header into
the same metadata.tags list before add_litellm_data_to_request ever
runs. The previous inherited_tags snapshot ("whatever's in tags before
this function's own caller-tag merge") therefore misattributed that
caller-controlled value as policy-backed whenever a request arrived
with the header set -- Greptile flagged this as a P1 security finding.
inherited_tags is now built directly from key_metadata/team_metadata/
project_metadata's own "tags" fields, independent of the shared,
pipeline-position-dependent "tags" list's mutation history. Verified
with a direct reproduction mirroring the real pipeline (calling
apply_client_tag_policy_pre_auth on the same data dict before
add_litellm_data_to_request, as user_api_key_auth actually does): the
caller's header tag no longer appears in inherited_tags. Added a
regression test exercising that same call order; confirmed it fails
against the pre-fix snapshot approach and passes against this fix.
* fix(router): make tag_routing_prefix configurable through update_settings/get_settings and UpdateRouterConfig
router_settings.tag_routing_prefix was only ever applied via the
Router() constructor. Router.update_settings's _allowed_settings
(used directly by proxy_server.py's _add_router_settings_from_db_config
for the DB-backed router_settings path) and get_settings's
vars_to_include both omitted it, so an operator relying on that path
had the value silently ignored -- flagged by veria-ai. Also adds it to
UpdateRouterConfig (the pydantic schema behind POST /config/update),
the same bug shape LIT-3152 previously fixed for retry_policy: a field
missing from that schema gets silently dropped by
model_dump(exclude_none=True) before update_settings is ever called.
* chore(ui): regenerate schema.d.ts for UpdateRouterConfig.tag_routing_prefix
Adding tag_routing_prefix to UpdateRouterConfig changed the proxy's
OpenAPI spec; regenerate the dashboard's generated API types to match.
* fix(lint): re-ratchet budget ceilings after rebasing onto litellm_internal_staging
Regenerated ruff-strict-budget.json and type-discipline-budget.json
against the post-rebase merge-base. LIT002/LIT011 ceilings reflect
this branch's true current counts (confirmed unchanged across the
rebase by diffing against the pre-rebase commit); the base's own
counts moved independently.
* fix(lint): replace mutable-collection fallbacks with immutable ones in inherited_tags computation
key_metadata/team_metadata/project_metadata's "tags" fallbacks used
`or {}` / `or []` literals, each a LIT002 mutable-collection-construction
violation that pushed the branch 4 over its ratchet ceiling relative to
a moved base. Swapped to MappingProxyType({}) / () to match the
immutable idiom the rest of tag_based_routing.py already uses; no
behavior change, since both are falsy and only ever read via .get()/
unpacking. Tightens type-discipline-budget.json's LIT002 ceiling back
down to match, fully closing that gap (LIT011 keeps a genuine 1-count
gap from pre-existing, untouched lines in this file, non-gating).
* fix(lint): suppress LIT011 on the two new data[...] mutation sites
Both new lines follow this file's established data[...] mutation
idiom for add_litellm_data_to_request, matching the existing
suppression already on the inherited_tags line.
* test(router): lock in fallback + tag-filtering interaction
Cover the router-level fallbacks mechanism composing with tag-based
routing: a plain negation exhausting a group correctly advances to
the fallback group, the same exclusion tag exhausting every hop
correctly raises, and allow_fail_open resolving locally must not
spuriously trigger an unrelated external fallback.
* chore: retrigger CI now that litellm-docs#814 is merged
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(proxy): add config_updated_at audit timestamp for virtual keys
updated_at carries Prisma's @updatedAt, so every batched spend flush
rewrites it and it cannot distinguish config changes from usage. Add an
additive config_updated_at column stamped only by key management writes
(update, bulk update, regenerate, block, unblock) via a shared helper,
expose it on key responses, and switch the key page's Last Updated to it
with a created_at fallback.
* test(proxy): assert config_updated_at survives key archival
* refactor(proxy): rename config_updated_at to settings_updated_at
* feat(ui): show models under each tier in routing benchmark chart
- Add TierTurnsChart: donut chart showing turns per complexity tier with
tier-assigned models listed below each tier name in the legend
- Only complexity routers show models; quality routers show tier name + %
(quality tiers don't pin specific models)
- Change 'Estimated spend at highest-cost model' wording to 'highest-tier'
to clarify it's the most capable tier's estimated cost, not just the
single-highest model
Closes LIT-5302
* fix(ui): use categorical colors for tier donut, trim redundant turn count
- Tier donut chart now uses a dedicated categorical palette instead of
SEQUENTIAL_COLOR_RAMP, which is a blue monochrome gradient meant for
magnitude series, not distinct categories.
- Space out the tier legend rows (gap-3 -> gap-6) for readability.
- Drop the turn count from "avg saved per session" since Routing by
tier already shows the total turns.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(ui): drop prohibited explanatory comments in TierTurnsChart test
Per repo convention against source comments; the test name and
assertions already communicate the scenario. Addresses Greptile review.
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove dead modulo from color index in TierTurnsChart
The colors array is built with length equal to slices.length, so idx % colors.length
is always a no-op in the render loop. Simplify to idx for clarity.
* fix(ui): wrap CostOptimizationView tests in QueryClientProvider
The tests render CostOptimizationView which uses useCan() → useIsOrgAdmin() →
useOrganizations() and useDailyActivityRange(), both of which call React Query's
useQuery(). Without QueryClientProvider wrapping the render, React Query throws
'No QueryClient set' error.
Also mock the required networking calls (organizationListCall, userDailyActivityCall)
to prevent spurious network errors in test runs.
All 7 tests now pass (CostOptimizationView + CostOptimizationView.activity).
* style(ui): format test files and extract object literal to fix linting
- Format CostOptimizationView.test.tsx with prettier
- Extract getToolSpend mock response to named variable to satisfy eslint
- Pass frontend-lint checks
* fix(ui): hoist mockToolSpendResponse into vi.hoisted to fix test initialization
Extracting the response object to a named variable violated hoisting rules:
vi.mock() factories are evaluated at hoisting time before regular const
declarations. Move mockToolSpendResponse into vi.hoisted() block.
---------
Co-authored-by: Claude <noreply@anthropic.com>
reportAny 16720 -> 15482 and reportExplicitAny 5689 -> 5316 with real types only: no casts, no ignores, no new Any. Whole-tree basedpyright drops 2173 diagnostics with zero per-rule or per-file regressions. Budgets ratcheted: basedpyright -2173, ruff-strict -188, type-discipline -55
Record supports_tool_search on the Bedrock Claude entries in both cost
map files and have _supports_tool_search_on_bedrock read it first via
the provider-resolved capability lookup, keeping the name patterns as a
fallback for ARNs and ids the map cannot resolve. Threads the flag
through ModelInfoBase and drops a dated remark from the pattern list
