Reverts #32005. Team-scoped keys are governed by the team and team-member
budgets only; the key owner personal max_budget no longer applies to them,
restoring the hierarchy that existed before that PR.
The skip_user_budget_on_team_key opt-out existed solely to turn the new
behavior back off, so it is removed along with the behavior: the
ConfigGeneralSettings field, the /config/list allowed_args entry that
surfaced it as an Admin UI toggle, and the argument threaded through
reserve_budget_for_request and _get_budget_counters.
Regression tests cover both enforcement points in the restored direction:
test_common_checks_personal_user_budget_skipped_for_team_key for the
read-time check and test_should_not_reserve_user_budget_counter_for_team_key
for the optimistic reservation path.
(cherry picked from commit 6f1625d23b)
The "Savings over time" chart plotted a single floating dot for short
ranges: the daily rollup keys spend by YYYY-MM-DD, so a one-day range is
one point by construction. Rather than stand up an hourly SpendLogs data
source, read that same daily rollup and make the cumulative line legible.
- Cumulative | Per day toggle. Cumulative accumulates within the range;
Per day shows the raw stacked bars.
- Cumulative prepends a synthetic $0 point at the range start
(withStartAnchor) so the line rises from zero to the running total
instead of floating. An empty series is left untouched so the chart's
own "No data" state shows.
- Order the daily series oldest-first (the rollup arrives newest-first)
so the axis reads left to right and the total accumulates forward.
- Header legend, dots on small series, and a "No data" guard on BarChart.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 1fa40bd168)
Swap the antd Collapse "How savings are calculated" panel for click-triggered
shadcn Popovers on each SummaryCard, so the explanation sits next to the
metric it describes instead of in one combined block.
(cherry picked from commit 3c287576b2)
The cost-optimization page was backported onto this line with its leftnav
entry but without its pageDescriptions entry, so page_utils.test.ts failed
two assertions: the page fell back to the "No description available"
placeholder, and it showed up in missingDescriptions.
Restores the entry verbatim from litellm_internal_staging, which makes
page_metadata.ts byte-identical to staging.
Dropping className="mt-3" from <Tabs> in #34885 shortened the line
enough that prettier wants the props back on one line; the PR merged
with frontend-lint red, so the pick inherits the violation.
* Fix cache leakage card layout to keep date picker on right and prevent content overlap
Removes flex-wrap and mt-3 to ensure date picker stays pinned to the right side of the card header regardless of zoom level, preventing it from covering card content below
* Remove overflow-hidden from Card to allow dropdowns and overlays to display fully
Fixes date picker dropdown being clipped when opened in cards like the Cache Leakage Card. By removing overflow-hidden from the Card container, popovers, dropdowns, and other overflow content can now display properly without being clipped by the card boundaries.
* Make cache leakage card descriptions consistent with line clamping
Adds line-clamp-2 to ensure both 'by model' and 'by virtual key' cards maintain consistent height. Removes conditional anthropic-specific text that caused height variations between dimensions.
(cherry picked from commit d91fd084f7)
* feat(spend): track prompt compression saved tokens in daily spend aggregates (#33810)
* feat(spend): track prompt compression saved tokens in daily spend aggregates
Native compression interception now records tokens_before/after/saved into the
request litellm_metadata so savings land in the SpendLog metadata JSON under a
typed compression_savings key. A single normalizer
(extract_compression_saved_tokens) sums that key with Headroom guardrail
tokens_saved; the two writers are disjoint and run at different stages, so
summing never double-counts. The spend-log redactor now preserves purely
numeric compression stats inside guardrail_response so Headroom savings
survive the store_prompts_in_spend_logs=false default. compression_saved_tokens
is threaded through BaseDailySpendTransaction, queue aggregation, the daily
upsert blocks, a new BigInt column on all six daily spend tables, and the
daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups)
* fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader
* feat(spend): aggregate compression and prompt caching dollar savings in daily rollups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): update daily spend aggregation fixtures for savings columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add Cost Optimization dashboard page
New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows.
Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 3f3295b33f)
* Merge pull request #33733 from BerriAI/litellm_lit_4162_bedrock_batch_tags
feat(bedrock): forward bedrock_tags to CreateModelInvocationJob for batch jobs
(cherry picked from commit 3819ee5dc0)
* bump: version 0.4.79 → 0.4.79.post1
---------
Co-authored-by: tin-berri <tin@berri.ai>
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* feat(chat-ui): add personal Logs view scoped to the current user
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chat-ui): show request payload from proxy_server_request in logs detail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chat-ui): address logs panel review feedback (stable detail key, error state)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): add ID-JAG egress auth as a v2 outbound-credentials arm
Adds the oauth2_id_jag MCP egress auth mode (draft-ietf-oauth-identity-assertion-authz-grant,
shipped by Okta as "AI agent token exchange") as a first-class arm of the v2
outbound_credentials resolver rather than a standalone v1 handler.
ID-JAG is a two-leg flow: an RFC 8693 token exchange swaps the caller's id_token for an
ID-JAG assertion at the IdP org authorization server, then an RFC 7523 jwt-bearer grant
presents that assertion to the MCP's resource authorization server for the access token
used to call the upstream. The gateway authenticates to both endpoints with a private-key
JWT client_assertion, falling back to client_secret when no key is configured.
The mode is modeled as IdJagConfig in the AuthConfig discriminated union, with client auth
as a ClientAuth tagged union (private_key_jwt or client_secret) so required fields are
enforced at construction and illegal states are unrepresentable. A new token_endpoint
collaborator performs the authenticated OAuth token-endpoint call and caches the result
with per-key single-flight; the resolver's _id_jag arm runs the two legs and returns an
httpx.Auth or a typed CredError. A missing caller identity token fails closed
(precondition_required), so an ID-JAG server never falls back to a static credential. The
v1->v2 adapter maps oauth2_id_jag servers onto IdJagConfig and the existing live v2 path
resolves them, so no standalone handler, has_id_jag_config flag, or resolve_mcp_auth
precedence branch is needed.
The ID-JAG client_private_key is encrypted at rest alongside client_secret.
* fix(mcp): sort token_endpoint imports to satisfy the I001 budget gate
* fix(mcp): give token_endpoint pyright suppressions reasons for the LIT004 budget
The freshly-merged base ratcheted the LIT004 ceiling down, so the six
unexplained pyright suppressions in token_endpoint.py went over budget.
Annotate each with why the boundary is untyped (litellm http handler and
InMemoryCache are untyped; response.json() is validated by
_TokenEndpointResponse in fetch) so the gate counts them as explained.
* fix(mcp): enforce ID-JAG exchange over caller auth overrides and redact token endpoint from client errors
For oauth2_id_jag servers the v2 resolver mints the upstream assertion from the caller's identity token; a caller-supplied x-mcp-auth / x-mcp-<alias>-authorization override or a conflicting injected Authorization must not disable that exchange and forward an arbitrary bearer, so IdJagConfig now joins authorization_code and token_exchange as a resolver-owned mode that keeps the v2 spec and ignores the override.
The token endpoint error branches previously returned the configured endpoint URL in the client-visible 503 detail. The endpoint now stays in server-side logs and clients get a generic token-exchange failure.
* fix(mcp): bind the ID-JAG token cache to the exchange config and map token endpoint network errors to typed CredErrors
* fix(mcp): fail closed when an oauth2_id_jag server is half-configured instead of deferring to v1 static credentials
* fix(mcp): evict the cached ID-JAG bearer on an upstream 401 so the retry re-exchanges
* fix(mcp): map an unsignable client assertion to a typed misconfigured error instead of an unhandled 500
* fix(mcp): redact credential fields from the server-registry debug dump
* refactor(ui): consolidate Add/Edit credential modals into one CredentialModal
AddCredentialModal and EditCredentialModal were ~90% identical: the same
provider select, ProviderSpecificFields, and submit/filter logic, differing
only in title, button text, edit-mode prefill, and the disabled credential
name. Replace both with a single CredentialModal driven by a mode: 'add' |
'edit' prop, and point the two call sites in credentials.tsx at it.
Removes ~120 lines of duplication and drops the no-explicit-any and
no-restricted-imports baselines. The two per-file tests merge into one
CredentialModal.test.tsx covering both modes (add: editable empty name;
edit: prefilled, disabled name; provider fields render).
* refactor(ui): derive credential name disabled state from mode, not data
The disabled flag on the credential name field was tied to whether
existingCredential?.credential_name is truthy, an artifact of the old
EditCredentialModal. Drive it from the isEdit flag like the rest of the
component so mode='add' with a stray existingCredential can't disable the
field and mode='edit' with an empty name can't leave it editable. Behavior
is unchanged for real call sites; adds a regression test for the edit-with-
empty-name case.
* refactor(ui): prefill credential form declaratively instead of via useEffect
The edit-mode form was seeded with an imperative form.setFieldsValue inside
a useEffect that also set React state (setSelectedProvider), an antd anti-
pattern carried over from the old EditCredentialModal. Both call sites mount
the modal fresh with existingCredential already present (conditional && plus
destroyOnHidden), so there is no 'prop arrives after mount' case to handle.
Replace it with antd's declarative initialValues on the Form and a lazy
useState initializer for the provider. Removes the effect, its
react-hooks/set-state-in-effect suppression and exhaustive-deps warning, and
one any cast; behavior is unchanged (edit now shows the real provider on
first paint instead of flashing the default). Existing tests cover prefill
and the disabled name field.
* feat: add Straiker guardrail integration
Implements LLM security guardrails via Straiker with prompt and response inspection, multi-mode execution (pre_call, post_call), and configurable blocking or redaction of flagged content across providers, streaming, images, and tool calls.
* fix(guardrails): harden straiker source attribution and error-path consistency
Use the operator-configured source for Straiker application attribution instead of a caller-supplied agent_id metadata value, so a caller cannot spoof which application a detection is attributed to. Make _fail reuse _block so a post_call error raises ModifyResponseException like a deliberate post_call block rather than GuardrailRaisedException, and type the blocking helper as NoReturn so the type checker enforces that execution never falls through the BLOCKED branch. Serialize the webhook payload once and send it as raw content to avoid re-serializing on the size check and on every retry.
* fix(guardrails): read straiker config and metadata from all supported shapes
Handle a dict optional_params in _get_config_value so nested guardrail
settings loaded from YAML or the DB (timeout, unreachable_fallback, and
the rest) are applied instead of silently falling back to defaults;
previously only attribute-style access was supported. Build the webhook
metadata bag from the merged metadata so client tags stored under
litellm_metadata on routes like /v1/messages reach Straiker the same way
identity and application fields already do, and widen the internal-key
skip prefix to user_api so proxy-injected budget values are not
forwarded.
* fix(guardrails): fail safe on straiker interventions without redactions
Block instead of passing content through when Straiker returns
GUARDRAIL_INTERVENED without replacement texts, so a positive
intervention verdict can never silently forward the original flagged
content. Fix the streamed-request detection to read the request body
from proxy_server_request.body, where the proxy stores it, instead of a
top-level body key that is never populated; the previous fallback was
dead, so a streamed response whose stream flag was not lifted to the top
level would have been redacted rather than blocked while buffering
replayed the original chunks.
* revert(guardrails): restore straiker caller agent_id application attribution
Restore the original behavior where a request-scoped agent_id in metadata
sets the Straiker application source, falling back to the configured
source. This is the integration's intended per-application attribution;
litellm already resolves a key-owned agent_id ahead of any caller-supplied
value, so a configured key cannot be spoofed.
* revert(guardrails): restore straiker webhook metadata scoping
Restore the original behavior where the Straiker webhook metadata bag is
built from request-scoped metadata only. Forwarding litellm_metadata was
a scope change to what the integration sends to Straiker; keep the
author's intended scoping.
* fix(guardrails): keep proxy key material out of straiker webhook metadata
Widen the internal-key skip prefix from user_api_key_ to user_api so the
proxy-injected user_api_key hash and user_api_end_user_max_budget are not
copied into the Straiker webhook metadata bag. The narrower prefix missed
the bare user_api_key name, leaking the hashed key to the vendor. Keeps
the request-scoped metadata source unchanged.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
Editing an existing LLM credential and changing only the api_base also
overwrote the stored api_key with its masked display value (e.g. sk****IA).
The edit form pre-fills fields from the credential the backend returns, whose
secrets come back masked, and the update handler sent every field straight
back; the endpoint then encrypted and stored the asterisks over the real key.
Run credential_values through stripMaskedSecrets before the PATCH so masked
placeholders are never sent, mirroring the guard the model edit form already
uses. The isMaskedSecret / stripMaskedSecrets helpers move out of
model_info_view into a shared utils module so both call sites share one
implementation.
Add a Playwright e2e that seeds a credential, edits only the api base in the
LLM Credentials tab, and asserts the outgoing PATCH no longer carries the
masked api_key while the new base persists.
The tag delete action moved into a Base UI dropdown menu when the tags
table was migrated onto the shared DataTable. That menu is modal by
default and holds a pointer-events lock on the page while it opens and
closes, which left the hand-rolled inline confirmation modal unclickable,
so deleting a tag stopped working
Replace the inline modal with the shared DeleteResourceModal, which
renders through an antd Modal portal that manages its own pointer-events
and z-index, matching every other table's delete flow. Add a deleting
loading state so the confirm button reflects progress and cannot be
double-clicked
Cover the wiring with a regression test that drives the delete flow
through the shared modal and asserts tagDeleteCall runs with the tag name
The toggle and ttl descriptions were a wall of text, with a panel intro that
mostly repeated the toggle description. Drop the intro and cut both descriptions
to one or two lines, keeping a one-clause note that the cache is shared across
callers on the same upstream credentials.
Rather than mixing the flag and its ttl into the generic General settings table
(which also surfaced the confusing Not Set / In Config / In DB provenance badges),
give prompt caching a dedicated tab with a purpose-built toggle and ttl dropdown.
Each registry field gains an optional tab, surfaced as ConfigList.field_tab, so
the General tab renders the ungrouped fields and the caching fields render on
their own tab. The update, persist and reset endpoints are unchanged.
The value cell was a ternary chain over field_type; adding Select made it a fourth
level and tripped no-nested-ternary. Early returns read better than a deeper chain
and let the suppression baseline ratchet down.
Register enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl on the
General Settings table so caching can be turned on without hand-writing config.
The registry could not express either field: validation was hardcoded to a float in
(0, 1], reset set every field to None (not a bool for a boolean flag), and the listing
reported any non-None value as 'In Config', which a False default would always trip.
Validation now dispatches on the declared type and reset restores each field's own
default. ConfigList carries field_options so the table can render a Select for enums
instead of no editor at all.
* feat(complexity-router): user-triggered escalation keywords
Add an escalation_keywords config option to the complexity router so a user
can force a bump to the next-higher complexity tier by including a phrase in
their message (a stronger model, but not one they get to choose). Defaults to
['LITELLM ESCALATE'] when unset, case-sensitive so it only fires on the
deliberate shouted form; admins can override the list or set [] to disable.
Escalation applies across every routing path: heuristic/LLM classification,
literal and semantic keyword_tier_rules overrides, adaptive routing, and
session affinity (where it bumps relative to the pinned model and persists the
higher tier for the rest of the session). Capped at the highest configured
tier and skips unconfigured intermediate tiers.
Expose it in the Auto-Router v2 UI as an Escalation Keywords field wired into
the complexity_router_config payload.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity-router): validate escalation keywords and pin at tier ceiling
Strip blank/whitespace escalation keywords so an empty phrase can't match every message and escalate all traffic. Keep the exact pinned model when a session escalates at the highest configured tier instead of randomly hopping to a peer in a multi-model pool.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): migrate AI Hub, public hub, and MCP Toolsets tables onto shared DataTable
* test(ui): stub skillHubPublicCall in the public model hub networking mock
The policy attachment form fetched /team/list with the caller's own
user_id, which the backend treats as a membership filter even for proxy
admins. Admins only saw teams they were personally a member of, and the
scope validation added in #32131 then rejected every other valid team
alias as nonexistent. Drop the user_id filter; the policies page is
admin-only and /team/list without user_id returns all teams for admin
roles.
Fixes LIT-4199
Migrate Projects, Project Keys, Vector-store Documents, Logging
Callbacks, and Pass-through Endpoints onto the shared DataTable and
cell library, each split into a thin container plus a columns file.
Row actions move into the unified overflow menu, empty states get the
rich icon+title+body treatment, and parents that never tracked loading
gain an initial-load-only isLoading flag so tables render skeletons
instead of flashing the empty state.
Project Keys switches from a hand-rolled antd Pagination header to
server pagination through the shared footer, and its section is
rewired into ProjectDetailsPage, which had regressed to a hardcoded
"No keys to display" empty card while the section sat orphaned.
pass_through_settings.tsx moves to PascalCase under
components/PassThroughSettings/ and drops its dead modelData prop and
tremor imports; its stale eslint bulk-suppressions are pruned.
Team/key router_settings.enable_tag_filtering was stored and echoed by
/team/info but never applied at request time: the per-request override
whitelist in route_llm_request.py dropped it, tag filtering only read the
router-level flag, and UpdateRouterConfig silently discarded the field on
/key/generate and /config/update. Requests from teams with the toggle on
were load balanced across all deployments instead of tag-matched ones.
- add enable_tag_filtering to the router_settings_override whitelist and
strip any client-supplied copy from the request body first, so only the
key/team value reaches the router
- run tag filtering when the request carries enable_tag_filtering=True; a
request-level False cannot disable a router-level True, so per-request
settings can only scope down, never escape the global policy
- add the field to UpdateRouterConfig so key and config update paths stop
dropping it, and to all_litellm_params so it never leaks into provider
request bodies
- allow it through Router.update_settings/get_settings so the global UI
toggle persists across DB config reloads
Resolves LIT-4390
Anthropic only caches a prompt when the request carries explicit cache_control
breakpoints, unlike OpenAI where prompt caching is automatic and needs no
configuration. Today litellm can inject those breakpoints server-side, but only
when an admin hand-writes cache_control_injection_points into a model's
litellm_params (or router_settings.default_litellm_params). Clients such as
Claude Code and Claude Desktop never set cache_control themselves, and the
admin recipe is easy to miss, so Anthropic traffic through the proxy silently
pays full price on every repeated prefix.
This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When
it is on and the request has no injection points configured and no
client-supplied cache_control, litellm synthesizes a default pair of breakpoints
(the system prompt and the trailing turn) so the stable prefix is cached while
the breakpoint advances with the conversation. It is wired into both surfaces:
/chat/completions seeds the points before the existing prompt-management gate, and
/v1/messages resolves them in maybe_inject_cache_control, so the existing
AnthropicCacheControlHook applies them unchanged and keeps its four-block cap and
its refusal to overwrite client breakpoints.
The default is off, so no existing deployment changes behavior. Injection is
gated to providers that actually consume cache_control markers (anthropic and
bedrock) and to models the cost map flags as supporting prompt caching; note that
supports_prompt_caching alone is not a sufficient gate, since OpenAI, Azure and
Gemini models report it as well but never take cache_control markers. The default
ttl is Anthropic's 5 minute ephemeral cache, with an optional
anthropic_prompt_caching_ttl of "5m" or "1h"; ttl is also added to
ChatCompletionCachedContent, which the bedrock and anthropic transforms already
read at runtime but the type never declared
Resolves LIT-4478
* fix(ui): navigate to /ui/login/ with trailing slash via hard navigation
Logged-out redirects targeted /ui/login without the trailing slash, so
Starlette's StaticFiles(html=True) mount answered with a 307 whose
absolute Location is built from the scheme the container sees. Behind a
TLS-terminating reverse proxy uvicorn does not trust X-Forwarded-Proto
by default, so the redirect downgraded https to http and stranded users
on an unreachable URL (#33454). The auth guard also used the Next client
router for this navigation, which first requests an RSC payload that the
static export cannot serve, producing 404s before falling back to a full
page load.
Centralize the login URL in getLoginUrl(), which always emits the
trailing slash so no server redirect fires, and use
window.location.replace for the login redirects so no RSC fetch is
attempted.
* test(ui): expect trailing slash in expired-token login redirect
* fix(ui): stop sending the complexity-router pseudo-model to /health/test_connection
Test Connection on a saved auto-router model sent the raw
"auto_router/complexity_router" model string to the generic
health-check endpoint, which always failed with "Unmapped LLM
provider" since it's a routing-strategy config, not a real
completion endpoint.
Reuse the per-tier connection test already built for the Add Auto
Router wizard: for complexity-router models, test each configured
tier's underlying model group instead of the router pseudo-model.
Semantic-type auto routers (auto_router_config) have no equivalent
tier-based test yet, so the button is hidden for them instead of
guaranteed to fail.
* fix(ui): address review feedback on auto-router test connection fix
Type the complexity-router config parsing instead of using `any`, use
NotificationsManager.warning instead of fromBackend for the
client-generated "no tiers configured" message, remove comments added
in the previous commit, and also test the deployment's configured
complexity_router_default_model as a fallback target when it isn't
already covered by a configured tier (matches the fallback Router
itself uses for unconfigured tiers).
