* feat(spend): track prompt compression saved tokens in daily spend aggregates (#33810)
* feat(spend): track prompt compression saved tokens in daily spend aggregates
Native compression interception now records tokens_before/after/saved into the
request litellm_metadata so savings land in the SpendLog metadata JSON under a
typed compression_savings key. A single normalizer
(extract_compression_saved_tokens) sums that key with Headroom guardrail
tokens_saved; the two writers are disjoint and run at different stages, so
summing never double-counts. The spend-log redactor now preserves purely
numeric compression stats inside guardrail_response so Headroom savings
survive the store_prompts_in_spend_logs=false default. compression_saved_tokens
is threaded through BaseDailySpendTransaction, queue aggregation, the daily
upsert blocks, a new BigInt column on all six daily spend tables, and the
daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups)
* fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader
* feat(spend): aggregate compression and prompt caching dollar savings in daily rollups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): update daily spend aggregation fixtures for savings columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add Cost Optimization dashboard page
New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows.
Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 3f3295b33f)
* Merge pull request #33733 from BerriAI/litellm_lit_4162_bedrock_batch_tags
feat(bedrock): forward bedrock_tags to CreateModelInvocationJob for batch jobs
(cherry picked from commit 3819ee5dc0)
* bump: version 0.4.79 → 0.4.79.post1
---------
Co-authored-by: tin-berri <tin@berri.ai>
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* feat(chat-ui): add personal Logs view scoped to the current user
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chat-ui): show request payload from proxy_server_request in logs detail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chat-ui): address logs panel review feedback (stable detail key, error state)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): add ID-JAG egress auth as a v2 outbound-credentials arm
Adds the oauth2_id_jag MCP egress auth mode (draft-ietf-oauth-identity-assertion-authz-grant,
shipped by Okta as "AI agent token exchange") as a first-class arm of the v2
outbound_credentials resolver rather than a standalone v1 handler.
ID-JAG is a two-leg flow: an RFC 8693 token exchange swaps the caller's id_token for an
ID-JAG assertion at the IdP org authorization server, then an RFC 7523 jwt-bearer grant
presents that assertion to the MCP's resource authorization server for the access token
used to call the upstream. The gateway authenticates to both endpoints with a private-key
JWT client_assertion, falling back to client_secret when no key is configured.
The mode is modeled as IdJagConfig in the AuthConfig discriminated union, with client auth
as a ClientAuth tagged union (private_key_jwt or client_secret) so required fields are
enforced at construction and illegal states are unrepresentable. A new token_endpoint
collaborator performs the authenticated OAuth token-endpoint call and caches the result
with per-key single-flight; the resolver's _id_jag arm runs the two legs and returns an
httpx.Auth or a typed CredError. A missing caller identity token fails closed
(precondition_required), so an ID-JAG server never falls back to a static credential. The
v1->v2 adapter maps oauth2_id_jag servers onto IdJagConfig and the existing live v2 path
resolves them, so no standalone handler, has_id_jag_config flag, or resolve_mcp_auth
precedence branch is needed.
The ID-JAG client_private_key is encrypted at rest alongside client_secret.
* fix(mcp): sort token_endpoint imports to satisfy the I001 budget gate
* fix(mcp): give token_endpoint pyright suppressions reasons for the LIT004 budget
The freshly-merged base ratcheted the LIT004 ceiling down, so the six
unexplained pyright suppressions in token_endpoint.py went over budget.
Annotate each with why the boundary is untyped (litellm http handler and
InMemoryCache are untyped; response.json() is validated by
_TokenEndpointResponse in fetch) so the gate counts them as explained.
* fix(mcp): enforce ID-JAG exchange over caller auth overrides and redact token endpoint from client errors
For oauth2_id_jag servers the v2 resolver mints the upstream assertion from the caller's identity token; a caller-supplied x-mcp-auth / x-mcp-<alias>-authorization override or a conflicting injected Authorization must not disable that exchange and forward an arbitrary bearer, so IdJagConfig now joins authorization_code and token_exchange as a resolver-owned mode that keeps the v2 spec and ignores the override.
The token endpoint error branches previously returned the configured endpoint URL in the client-visible 503 detail. The endpoint now stays in server-side logs and clients get a generic token-exchange failure.
* fix(mcp): bind the ID-JAG token cache to the exchange config and map token endpoint network errors to typed CredErrors
* fix(mcp): fail closed when an oauth2_id_jag server is half-configured instead of deferring to v1 static credentials
* fix(mcp): evict the cached ID-JAG bearer on an upstream 401 so the retry re-exchanges
* fix(mcp): map an unsignable client assertion to a typed misconfigured error instead of an unhandled 500
* fix(mcp): redact credential fields from the server-registry debug dump
* refactor(ui): consolidate Add/Edit credential modals into one CredentialModal
AddCredentialModal and EditCredentialModal were ~90% identical: the same
provider select, ProviderSpecificFields, and submit/filter logic, differing
only in title, button text, edit-mode prefill, and the disabled credential
name. Replace both with a single CredentialModal driven by a mode: 'add' |
'edit' prop, and point the two call sites in credentials.tsx at it.
Removes ~120 lines of duplication and drops the no-explicit-any and
no-restricted-imports baselines. The two per-file tests merge into one
CredentialModal.test.tsx covering both modes (add: editable empty name;
edit: prefilled, disabled name; provider fields render).
* refactor(ui): derive credential name disabled state from mode, not data
The disabled flag on the credential name field was tied to whether
existingCredential?.credential_name is truthy, an artifact of the old
EditCredentialModal. Drive it from the isEdit flag like the rest of the
component so mode='add' with a stray existingCredential can't disable the
field and mode='edit' with an empty name can't leave it editable. Behavior
is unchanged for real call sites; adds a regression test for the edit-with-
empty-name case.
* refactor(ui): prefill credential form declaratively instead of via useEffect
The edit-mode form was seeded with an imperative form.setFieldsValue inside
a useEffect that also set React state (setSelectedProvider), an antd anti-
pattern carried over from the old EditCredentialModal. Both call sites mount
the modal fresh with existingCredential already present (conditional && plus
destroyOnHidden), so there is no 'prop arrives after mount' case to handle.
Replace it with antd's declarative initialValues on the Form and a lazy
useState initializer for the provider. Removes the effect, its
react-hooks/set-state-in-effect suppression and exhaustive-deps warning, and
one any cast; behavior is unchanged (edit now shows the real provider on
first paint instead of flashing the default). Existing tests cover prefill
and the disabled name field.
* feat: add Straiker guardrail integration
Implements LLM security guardrails via Straiker with prompt and response inspection, multi-mode execution (pre_call, post_call), and configurable blocking or redaction of flagged content across providers, streaming, images, and tool calls.
* fix(guardrails): harden straiker source attribution and error-path consistency
Use the operator-configured source for Straiker application attribution instead of a caller-supplied agent_id metadata value, so a caller cannot spoof which application a detection is attributed to. Make _fail reuse _block so a post_call error raises ModifyResponseException like a deliberate post_call block rather than GuardrailRaisedException, and type the blocking helper as NoReturn so the type checker enforces that execution never falls through the BLOCKED branch. Serialize the webhook payload once and send it as raw content to avoid re-serializing on the size check and on every retry.
* fix(guardrails): read straiker config and metadata from all supported shapes
Handle a dict optional_params in _get_config_value so nested guardrail
settings loaded from YAML or the DB (timeout, unreachable_fallback, and
the rest) are applied instead of silently falling back to defaults;
previously only attribute-style access was supported. Build the webhook
metadata bag from the merged metadata so client tags stored under
litellm_metadata on routes like /v1/messages reach Straiker the same way
identity and application fields already do, and widen the internal-key
skip prefix to user_api so proxy-injected budget values are not
forwarded.
* fix(guardrails): fail safe on straiker interventions without redactions
Block instead of passing content through when Straiker returns
GUARDRAIL_INTERVENED without replacement texts, so a positive
intervention verdict can never silently forward the original flagged
content. Fix the streamed-request detection to read the request body
from proxy_server_request.body, where the proxy stores it, instead of a
top-level body key that is never populated; the previous fallback was
dead, so a streamed response whose stream flag was not lifted to the top
level would have been redacted rather than blocked while buffering
replayed the original chunks.
* revert(guardrails): restore straiker caller agent_id application attribution
Restore the original behavior where a request-scoped agent_id in metadata
sets the Straiker application source, falling back to the configured
source. This is the integration's intended per-application attribution;
litellm already resolves a key-owned agent_id ahead of any caller-supplied
value, so a configured key cannot be spoofed.
* revert(guardrails): restore straiker webhook metadata scoping
Restore the original behavior where the Straiker webhook metadata bag is
built from request-scoped metadata only. Forwarding litellm_metadata was
a scope change to what the integration sends to Straiker; keep the
author's intended scoping.
* fix(guardrails): keep proxy key material out of straiker webhook metadata
Widen the internal-key skip prefix from user_api_key_ to user_api so the
proxy-injected user_api_key hash and user_api_end_user_max_budget are not
copied into the Straiker webhook metadata bag. The narrower prefix missed
the bare user_api_key name, leaking the hashed key to the vendor. Keeps
the request-scoped metadata source unchanged.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
Editing an existing LLM credential and changing only the api_base also
overwrote the stored api_key with its masked display value (e.g. sk****IA).
The edit form pre-fills fields from the credential the backend returns, whose
secrets come back masked, and the update handler sent every field straight
back; the endpoint then encrypted and stored the asterisks over the real key.
Run credential_values through stripMaskedSecrets before the PATCH so masked
placeholders are never sent, mirroring the guard the model edit form already
uses. The isMaskedSecret / stripMaskedSecrets helpers move out of
model_info_view into a shared utils module so both call sites share one
implementation.
Add a Playwright e2e that seeds a credential, edits only the api base in the
LLM Credentials tab, and asserts the outgoing PATCH no longer carries the
masked api_key while the new base persists.
The tag delete action moved into a Base UI dropdown menu when the tags
table was migrated onto the shared DataTable. That menu is modal by
default and holds a pointer-events lock on the page while it opens and
closes, which left the hand-rolled inline confirmation modal unclickable,
so deleting a tag stopped working
Replace the inline modal with the shared DeleteResourceModal, which
renders through an antd Modal portal that manages its own pointer-events
and z-index, matching every other table's delete flow. Add a deleting
loading state so the confirm button reflects progress and cannot be
double-clicked
Cover the wiring with a regression test that drives the delete flow
through the shared modal and asserts tagDeleteCall runs with the tag name
The toggle and ttl descriptions were a wall of text, with a panel intro that
mostly repeated the toggle description. Drop the intro and cut both descriptions
to one or two lines, keeping a one-clause note that the cache is shared across
callers on the same upstream credentials.
Rather than mixing the flag and its ttl into the generic General settings table
(which also surfaced the confusing Not Set / In Config / In DB provenance badges),
give prompt caching a dedicated tab with a purpose-built toggle and ttl dropdown.
Each registry field gains an optional tab, surfaced as ConfigList.field_tab, so
the General tab renders the ungrouped fields and the caching fields render on
their own tab. The update, persist and reset endpoints are unchanged.
The value cell was a ternary chain over field_type; adding Select made it a fourth
level and tripped no-nested-ternary. Early returns read better than a deeper chain
and let the suppression baseline ratchet down.
Register enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl on the
General Settings table so caching can be turned on without hand-writing config.
The registry could not express either field: validation was hardcoded to a float in
(0, 1], reset set every field to None (not a bool for a boolean flag), and the listing
reported any non-None value as 'In Config', which a False default would always trip.
Validation now dispatches on the declared type and reset restores each field's own
default. ConfigList carries field_options so the table can render a Select for enums
instead of no editor at all.
* feat(complexity-router): user-triggered escalation keywords
Add an escalation_keywords config option to the complexity router so a user
can force a bump to the next-higher complexity tier by including a phrase in
their message (a stronger model, but not one they get to choose). Defaults to
['LITELLM ESCALATE'] when unset, case-sensitive so it only fires on the
deliberate shouted form; admins can override the list or set [] to disable.
Escalation applies across every routing path: heuristic/LLM classification,
literal and semantic keyword_tier_rules overrides, adaptive routing, and
session affinity (where it bumps relative to the pinned model and persists the
higher tier for the rest of the session). Capped at the highest configured
tier and skips unconfigured intermediate tiers.
Expose it in the Auto-Router v2 UI as an Escalation Keywords field wired into
the complexity_router_config payload.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity-router): validate escalation keywords and pin at tier ceiling
Strip blank/whitespace escalation keywords so an empty phrase can't match every message and escalate all traffic. Keep the exact pinned model when a session escalates at the highest configured tier instead of randomly hopping to a peer in a multi-model pool.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): migrate AI Hub, public hub, and MCP Toolsets tables onto shared DataTable
* test(ui): stub skillHubPublicCall in the public model hub networking mock
The policy attachment form fetched /team/list with the caller's own
user_id, which the backend treats as a membership filter even for proxy
admins. Admins only saw teams they were personally a member of, and the
scope validation added in #32131 then rejected every other valid team
alias as nonexistent. Drop the user_id filter; the policies page is
admin-only and /team/list without user_id returns all teams for admin
roles.
Fixes LIT-4199
Migrate Projects, Project Keys, Vector-store Documents, Logging
Callbacks, and Pass-through Endpoints onto the shared DataTable and
cell library, each split into a thin container plus a columns file.
Row actions move into the unified overflow menu, empty states get the
rich icon+title+body treatment, and parents that never tracked loading
gain an initial-load-only isLoading flag so tables render skeletons
instead of flashing the empty state.
Project Keys switches from a hand-rolled antd Pagination header to
server pagination through the shared footer, and its section is
rewired into ProjectDetailsPage, which had regressed to a hardcoded
"No keys to display" empty card while the section sat orphaned.
pass_through_settings.tsx moves to PascalCase under
components/PassThroughSettings/ and drops its dead modelData prop and
tremor imports; its stale eslint bulk-suppressions are pruned.
Team/key router_settings.enable_tag_filtering was stored and echoed by
/team/info but never applied at request time: the per-request override
whitelist in route_llm_request.py dropped it, tag filtering only read the
router-level flag, and UpdateRouterConfig silently discarded the field on
/key/generate and /config/update. Requests from teams with the toggle on
were load balanced across all deployments instead of tag-matched ones.
- add enable_tag_filtering to the router_settings_override whitelist and
strip any client-supplied copy from the request body first, so only the
key/team value reaches the router
- run tag filtering when the request carries enable_tag_filtering=True; a
request-level False cannot disable a router-level True, so per-request
settings can only scope down, never escape the global policy
- add the field to UpdateRouterConfig so key and config update paths stop
dropping it, and to all_litellm_params so it never leaks into provider
request bodies
- allow it through Router.update_settings/get_settings so the global UI
toggle persists across DB config reloads
Resolves LIT-4390
Anthropic only caches a prompt when the request carries explicit cache_control
breakpoints, unlike OpenAI where prompt caching is automatic and needs no
configuration. Today litellm can inject those breakpoints server-side, but only
when an admin hand-writes cache_control_injection_points into a model's
litellm_params (or router_settings.default_litellm_params). Clients such as
Claude Code and Claude Desktop never set cache_control themselves, and the
admin recipe is easy to miss, so Anthropic traffic through the proxy silently
pays full price on every repeated prefix.
This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When
it is on and the request has no injection points configured and no
client-supplied cache_control, litellm synthesizes a default pair of breakpoints
(the system prompt and the trailing turn) so the stable prefix is cached while
the breakpoint advances with the conversation. It is wired into both surfaces:
/chat/completions seeds the points before the existing prompt-management gate, and
/v1/messages resolves them in maybe_inject_cache_control, so the existing
AnthropicCacheControlHook applies them unchanged and keeps its four-block cap and
its refusal to overwrite client breakpoints.
The default is off, so no existing deployment changes behavior. Injection is
gated to providers that actually consume cache_control markers (anthropic and
bedrock) and to models the cost map flags as supporting prompt caching; note that
supports_prompt_caching alone is not a sufficient gate, since OpenAI, Azure and
Gemini models report it as well but never take cache_control markers. The default
ttl is Anthropic's 5 minute ephemeral cache, with an optional
anthropic_prompt_caching_ttl of "5m" or "1h"; ttl is also added to
ChatCompletionCachedContent, which the bedrock and anthropic transforms already
read at runtime but the type never declared
Resolves LIT-4478
* fix(ui): navigate to /ui/login/ with trailing slash via hard navigation
Logged-out redirects targeted /ui/login without the trailing slash, so
Starlette's StaticFiles(html=True) mount answered with a 307 whose
absolute Location is built from the scheme the container sees. Behind a
TLS-terminating reverse proxy uvicorn does not trust X-Forwarded-Proto
by default, so the redirect downgraded https to http and stranded users
on an unreachable URL (#33454). The auth guard also used the Next client
router for this navigation, which first requests an RSC payload that the
static export cannot serve, producing 404s before falling back to a full
page load.
Centralize the login URL in getLoginUrl(), which always emits the
trailing slash so no server redirect fires, and use
window.location.replace for the login redirects so no RSC fetch is
attempted.
* test(ui): expect trailing slash in expired-token login redirect
* fix(ui): stop sending the complexity-router pseudo-model to /health/test_connection
Test Connection on a saved auto-router model sent the raw
"auto_router/complexity_router" model string to the generic
health-check endpoint, which always failed with "Unmapped LLM
provider" since it's a routing-strategy config, not a real
completion endpoint.
Reuse the per-tier connection test already built for the Add Auto
Router wizard: for complexity-router models, test each configured
tier's underlying model group instead of the router pseudo-model.
Semantic-type auto routers (auto_router_config) have no equivalent
tier-based test yet, so the button is hidden for them instead of
guaranteed to fail.
* fix(ui): address review feedback on auto-router test connection fix
Type the complexity-router config parsing instead of using `any`, use
NotificationsManager.warning instead of fromBackend for the
client-generated "no tiers configured" message, remove comments added
in the previous commit, and also test the deployment's configured
complexity_router_default_model as a fallback target when it isn't
already covered by a configured tier (matches the fallback Router
itself uses for unconfigured tiers).
Discover the issuer from the upstream and persist it trust-on-first-use (fill-empty-only,
frozen thereafter), so admins do not have to type it; an admin-configured issuer always wins
and is never overwritten. Re-pointing the server url now clears the discovered issuer and
endpoints (matching the existing auth_type-change clearing) so a new upstream re-discovers
instead of anchoring on the previous upstream's issuer. Adds the issuer to the per-user OAuth
token identity so re-pointing it purges stale tokens. Surfaces the issuer as an optional,
auto-discovered, overridable field in the create and edit MCP server forms.
C901 gate shows +1 vs staging; that is inherited from the #33317 stack base (delta 0 against
* fix(ui): remove New badge from Projects sidebar item
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add reusable BetaBadge and use it for Projects sidebar item
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): rename disableShowNewBadge flag to disableShowBadges and make BetaBadge respect it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): use blue for BetaBadge to match existing New badge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(ui): keep disableShowNewBadge localStorage key to preserve existing user opt-outs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: remove non-functional md artifacts from PR
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Clients calling the standalone apply_guardrail endpoint had no way to pass
per-request configuration to custom guardrail implementations. This adds an
optional metadata field to ApplyGuardrailRequest and forwards it to
CustomGuardrail.apply_guardrail via request_data, only when the client sends
it. The messages guard is aligned to the same is-not-None semantics so an
explicitly-sent empty list is forwarded instead of silently dropped.
The Admin UI's Guardrail Test Playground gains an optional Metadata JSON
input (validated client-side) wired through applyGuardrail in networking.tsx,
so parameterized guardrails can be exercised from the dashboard.
Tests cover metadata alone, metadata with messages, explicit empty values,
the omitted-field passthrough, and the UI panel's parse/error behavior
Co-authored-by: Yassin Kortam <yassin@berri.ai>
CHAT_ROUTES was built once at module load via migratedHref, capturing an
empty server root path before the UI-config bootstrap sets it. Under
SERVER_ROOT_PATH every chat route came out unprefixed, so router.push
hard-navigated to a 404; during a send that page unload also aborted the
streaming request, so the first message of a new conversation never
rendered and only a second send appeared to work. Compute the routes at
render time via getChatRoutes(), update the send URL with a shallow
history.pushState instead of a router navigation, and source the active
conversation id from the hook's local state so it propagates without a
router round-trip
* feat(proxy): push-based OTLP billable-request metering for enterprise deployments
Adds opt-in, license-gated metering that counts 2xx HTTP requests to LLM
inference, MCP, and A2A endpoints and exports them over mutual TLS to a global
OpenTelemetry Collector for request-based billing.
A pure ASGI middleware (BillableRequestMetricsMiddleware) classifies each
request by route and records one count per 2xx response via an injected
recorder. The recorder (BillingMetricsRecorder) owns a dedicated OTEL meter
provider and an OTLP/gRPC exporter authenticated with client certificates, kept
isolated from the global meter provider so a customer's own OTEL metrics are
untouched. The recorder is built only when a valid LITELLM_LICENSE is present
and the cert material is configured; otherwise the middleware is a transparent
pass-through.
Deployment identity rides on the mTLS client certificate rather than the
payload, so the secret license key is never sent as an attribute or header; only
the license org id travels as a resource attribute for cross-checking.
Resolves LIT-4089
* fix(proxy): align billable-request metering with the global collector
- switch the exporter to OTLP/HTTP with a TLS client certificate. The
collector front end terminates mutual TLS and validates the client cert
against our CA; server verification uses the system trust store, so the
CA env var is now an optional override for private collectors
- resolve the metrics recorder on the first request via a factory instead
of at import time, so deployments that provide the license and cert env
vars through the YAML config's environment_variables export correctly
- close the metering bypass: classify /images/edits, /images/variations,
/v1/messages, /v1/videos, video remix, /v1/ocr and Gemini generateContent
as billable, and gate LLM routes to POST so GET reads (list videos, fetch
a response) do not bill. Verified live: the collector count matches the
UI usage page successful_requests exactly, with failures excluded on both
sides
* fix(proxy): wrap enterprise billing import in try-except per code-quality gate
The check_unsafe_enterprise_import gate requires every import from an
enterprise-pathed module to be guarded. Annotate the factory with the
middleware's BillingRecorder protocol so no enterprise type import is
needed at type-check time
* chore: satisfy strict lint gates in billing modules
- builtin generics per UP006 (dict/tuple instead of typing.Dict/Tuple)
- noqa the deliberate blind catch that keeps metering from breaking startup
- sort proxy_server import blocks split by the guarded enterprise import
* fix(proxy): bill provider passthrough, search, and rag routes
Route-inventory audit against LiteLLMRoutes.llm_api_routes found more
SpendLogs-producing surfaces the classifier missed: provider passthrough
(/bedrock, /vertex-ai, /cohere and the rest of mapped_pass_through_routes),
/v1/search and vector-store search, and the rag ingest/query routes. All are
counted by the dashboard usage page, so missing them undercounts billing.
The passthrough prefix list is read from LiteLLMRoutes so new providers are
picked up without touching this module. /langfuse is excluded: it forwards
observability traffic and writes no SpendLogs row. Known limitation recorded
in the PR: /v1/realtime is a websocket flow the HTTP middleware does not see
* fix(proxy): bill MCP and A2A requests by protocol transport routes only
The billable-request classifier matched the whole /v1/mcp prefix, so
management and discovery reads such as GET /v1/mcp/tools and GET
/v1/mcp/server counted as billable MCP requests, while real MCP tool
calls on the /{server}/mcp and /toolset/{name}/mcp aliases were missed
because their route handlers rewrite the ASGI scope only after this
middleware has already classified the original path. Classify MCP by the
concrete transport surface (the /mcp streamable-HTTP and SSE sub-app plus
the single-segment server and toolset aliases) and exclude the /v1/mcp
management API. Apply the same shape to A2A, which had the identical
issue: only the /message/send invoke route bills, not /v1/a2a/discover or
the .well-known agent-card reads.
* fix(proxy): harden billable-request classification and recorder lifecycle
Exact-match Anthropic /v1/messages so OpenAI Assistants thread-message
routes no longer bill, add Google Interactions create routes, guard
recorder.record() so a broken exporter can never fail a served request,
lock lazy recorder resolution against concurrent first requests, and
disable metering on empty-string env config instead of accepting a
blank endpoint
* chore(ui): regenerate eslint metrics after staging merge
* docs(proxy): state the lower-bound billing contract in middleware comments
* fix(proxy): bill mcp-rest tool calls and bare a2a agent invokes
POST /mcp-rest/tools/call executes a tool and fires the same MCP spend
logging as the /mcp transport, and POST /a2a/{agent_id} is the JSON-RPC
invoke route whose method (message/send or message/stream) travels in
the body; both returned 2xx without being recorded
* fix(proxy): flush billable-request counts on proxy shutdown
PeriodicExportingMetricReader buffers up to one export interval of
counts; without a final flush every restart silently dropped them. The
factory registers the recorder it builds and proxy_shutdown_event pops
and flushes it, bounded by a 5s timeout so a dead collector cannot
stall shutdown
* fix(proxy): stop billing bare a2a task RPCs and close the shutdown race
POST /a2a/{agent_id} multiplexes JSON-RPC methods off the request body. Only
message/send and message/stream write a SpendLogs row; tasks/get, tasks/cancel
and the pushNotificationConfig RPCs are forwarded upstream and write none.
Classifying the bare path as billable counted those task RPCs and pushed the
metric above the dashboard's successful-request count. Since a path-only
classifier cannot read the body, the bare route no longer bills; the explicit
/message/send routes still do. Missing a bare-path invoke undercounts, which is
the only direction this metric is allowed to drift. The /mcp transport keeps
billing every method because its list path logs a SpendLogs row too.
The billing middleware also sat outside InFlightRequestsMiddleware, and it
records after the inner app returns. A request could therefore be counted as
drained while its record() had not yet run, letting proxy_shutdown_event flush
and stop the exporter underneath it. Registering it before the in-flight
tracker nests it inside, so wait_for_drain covers the record
* test(proxy): stub the OTLP exporter in the recorder-build test
test_premium_with_full_config_builds_recorder built a real MeterProvider, so
the shutdown flush resolved collector.example and opened a TLS connection from
a unit test. The exporter is now stubbed, and a getaddrinfo spy asserts nothing
resolves the collector host so the stub cannot be quietly dropped later
* fix(helm): truncate the helm.sh/chart label to 63 bytes
Kubernetes caps a label value at 63 bytes and .Chart.Version is unbounded. CI
publishes branch builds as 0.0.0-branch-<branch>-<sha>, so helm.sh/chart
rendered as a 64 byte value and the API server rejected every labeled resource
with "must be no more than 63 bytes", including the migrations Job. The
litellm-helm chart already guards this through a litellm.chart helper; this
adds the same helper here.
Swept the rest of the chart for label and name values built from unbounded
input. .Chart.Version appeared only in this label. The remaining candidates all
derive from .Release.Name, which helm itself caps at 53 characters, so they
cannot overflow; three of them are selector labels feeding immutable Deployment
matchLabels, where adding trunc would risk churn for no gain. They are left
alone deliberately.
Verified with a new helm-unittest suite, tests/chart_label_tests.yaml, which
overrides chart.version per test:
helm unittest -f 'tests/*.yaml' helm/litellm # 13 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
The truncation cases fail against the previous helper. Reproduced the original
overflow by rendering with the real branch version and measuring the label:
helm template rel helm/litellm -f helm/litellm/tests/values/required.yaml \
| grep helm.sh/chart # 64 bytes before, 63 after
* feat(proxy): accept inline PEM for the billing-metrics mTLS credentials
LITELLM_BILLING_METRICS_CLIENT_CERT, _CLIENT_KEY and _CA_CERT took a filesystem
path. ECS injects Secrets Manager values as environment content and cannot mount
them as files, so a licensed deployment there could not turn metering on.
Each variable now takes either a path or the PEM itself. Inline PEM, detected by
the "-----BEGIN" prefix, is written once when the recorder is built into a 0700
temp dir as a 0600 file, and the config points at that path. The OTLP exporter
still only ever sees paths. A write failure disables metering through the
existing failure-as-None path rather than raising, and path-valued variables are
passed through untouched, so nothing changes for deployments that mount files.
The mixed case works too: mount the CA, inject the client credentials
* feat(helm): add first-class billingMetrics values to the componentized chart
Turning enterprise billable-request metering on meant hand-rolling the env vars
and the cert volume through gateway.extraEnv and gateway.volumes. This adds a
top-level billingMetrics block, off by default, consumed only by the gateway
since that is the component serving billable traffic.
When enabled it renders LITELLM_BILLING_METRICS_ENDPOINT plus the two cert paths
and mounts secretName read-only at /etc/litellm/billing-mtls. caSecretName is
optional and only needed for private collectors whose server certificate is not
on the public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and
adds the CA env var. exportIntervalMs is passed through only when set.
Enabling without secretName or with an empty endpoint fails the render with a
named message rather than producing a gateway that silently never exports.
The generic gateway.volumes, gateway.volumeMounts and gateway.extraEnv paths are
untouched and still compose with this, so existing overlays keep working.
The chart has no values.schema.json and no README, so there is nothing further to
update. Verified with a new helm-unittest suite:
helm unittest -f 'tests/*.yaml' helm/litellm # 23 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
* feat(terraform): billing-metrics variables for the aws and gcp templates
* feat(helm): add billingMetrics values to the classic chart
The componentized chart just gained a first-class billingMetrics block; this
mirrors it in litellm-helm so enabling enterprise billable-request metering no
longer means hand-rolling the env vars and the cert volume through envVars and
volumes.
When enabled the proxy Deployment renders LITELLM_BILLING_METRICS_ENDPOINT plus
the two cert paths, and mounts secretName read-only at /etc/litellm/billing-mtls.
secretName defaults to litellm-billing-metrics-mtls, the conventional name, so
enabling the block is enough once that Secret exists. caSecretName is optional
and only needed for private collectors whose server certificate is not on the
public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and adds the
CA env var. exportIntervalMs is passed through only when set.
The env entries render after envVars and extraEnvVars, so a user-supplied
LITELLM_BILLING_METRICS_ENDPOINT cannot silently redirect the export under
Kubernetes last-wins duplicate-env semantics; this is the same ordering the
migrations Job relies on for DISABLE_SCHEMA_UPDATE.
Enabling with an emptied secretName or endpoint fails the render with a named
message rather than producing a proxy that silently never exports.
The generic volumes, volumeMounts, envVars and extraEnvVars paths are untouched
and still compose with this, so existing overlays keep working. The chart has no
values.schema.json; README parameters and a setup section are updated.
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 68 passed (54 + 14 new)
helm lint helm/litellm-helm # 0 failed
* test(helm): pin that the migrations job never mounts the billing cert
The componentized chart's suite asserts the backend Deployment stays clear of the
billing wiring, since only the gateway serves billable traffic. The classic chart
has no backend, but it does have a second pod: the migrations Job, which renders
its own env from envVars and extraEnvVars. Nothing today wires the billing
include into it, and nothing stopped a future edit from doing so.
Asserts absence of the env, and that the Job grows no volumes or volumeMounts at
all. Both are notExists rather than notContains because the Job renders neither
key by default, so a notContains would fail on an unknown path instead of
checking the absence it looks like it is checking.
* fix(helm): meter the backend too, it serves the MCP transport
Scoping billingMetrics to the gateway was wrong. Applying each component's own
route allowlist to the proxy app shows the split is 75 billable routes on the
gateway and one on the backend: /{mcp_server_name}/mcp, the named-server MCP
transport, which writes a SpendLogs row on success. Metering only the gateway
would have silently dropped every MCP transport call from the counter, an
undercount proportional to a customer's MCP traffic.
The backend deployment now renders the same env and mounts the same read-only
cert secret. The migrations job still gets neither; it runs prisma and serves no
traffic, and a test pins that.
helm unittest -f 'tests/*.yaml' helm/litellm # 25 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
This also aligns the chart with the terraform templates, which inject the
credentials into both components.
* fix(proxy): never log billing credential values when they fail to resolve
Accepting inline PEM turned the cert env vars into secret-bearing values, but
the disable warning still echoed them. A value that is neither a readable path
nor `-----BEGIN`-prefixed PEM, for example a key with a preamble or a malformed
secret, fell through to the path branch and was written to the proxy logs
verbatim, exposing the client certificate or private key to anyone who can read
them.
The warning now names the offending environment variables and tells the operator
what a valid value looks like, without ever printing one
* Revert "fix(helm): truncate the helm.sh/chart label to 63 bytes"
This reverts commit 4f7f706a63.
Version hygiene belongs to the pipeline that mints chart versions, not to the
chart. The build workflow now caps the version slug so litellm-<version> fits
the 63 byte label budget, which removes the overflow at the source rather than
silently truncating a value operators use to identify the build.
Drops the litellm.chart helper, restores the direct helm.sh/chart printf, and
removes tests/chart_label_tests.yaml. Both chart suites stay green:
helm unittest -f 'tests/*.yaml' helm/litellm # 20 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
* feat(helm): default billingMetrics.secretName to the conventional name
The componentized chart required an explicit secretName while the classic chart
defaults to litellm-billing-metrics-mtls. Both now default to it, so the common
path is to create that Secret with tls.crt and tls.key and set enabled: true.
The required() guard stays, and with a default it now only fires when someone
explicitly blanks the override, which the tests pin from both sides
* feat(proxy): log once when billing metrics are actually enabled
build_billing_metrics_recorder returned None silently when the deployment was
not licensed, while every other disable path logged a warning. An operator
reading logs could not tell "metering active" from "metering off because this
component never saw the license", and a component can carry the cert mount and
the billing env and still meter nothing. That is the undercount direction the
metric is not allowed to drift in.
A successful build now emits one info line naming the collector endpoint and the
export interval; neither the certificate contents nor the license appear. The
unlicensed path logs at debug rather than warning, because unlicensed is the
common case and a warning there would be noise on every OSS proxy
* fix(terraform): fail the plan on a partial billing-metrics config
Each PEM secret is created only when its own variable is non-empty, so setting
billing_metrics_endpoint with a certificate but no key applied cleanly and left
the proxy logging "missing config" and never exporting. Silent non-export is the
undercount direction this metric must not drift in, and every other surface
fails fast on a half-configured metering block.
Both templates now carry a lifecycle precondition requiring the client
certificate and its key together whenever the endpoint is set. It lives on the
gateway task definition (aws) and the gateway Cloud Run service (gcp) rather
than on the secret resources, because those are themselves count-gated on the
PEM being present and would never evaluate in the failing case. Cross-variable
`validation` blocks would need terraform 1.9; versions.tf pins >= 1.6, and
preconditions work there.
ca_cert_pem stays optional, so an empty value still falls back to the system
trust store.
endpoint cert key result
"" any any metering off, no secrets created
set set set metering on
set missing either plan fails
Verified each row with `terraform console` against the condition, and reran
`terraform fmt -check` and `terraform validate` in both directories
* docs(terraform): record why the billing guard sits on the gateway resource
The precondition cannot live on the cert secret, which is count-gated on
the cert itself and so has zero instances in exactly the case the guard
must catch. That makes the guard's correctness depend on this resource
staying unconditional, which nothing else records and no test enforces
* fix(terraform): guard the backend against a partial billing config too
The precondition only sat on the gateway, but the backend receives the billing
endpoint as well, because it serves the named-server MCP transport and meters
it. A targeted apply of just the backend task or service would therefore skip
the guard entirely and provision a component holding a billing endpoint with no
credentials to use it, which is the silent never-export failure the guard exists
to prevent.
Both templates now carry the same precondition on the backend resource. The
condition and truth table are unchanged; ca_cert_pem stays optional.
terraform fmt -check and terraform validate clean in both directories
* docs(team): document mcp_rpm_limit in update_team docstring
* chore(ui): regenerate schema.d.ts for update_team docstring change
token_storage_ttl_seconds previously won outright over the token's expires_in, so a TTL longer than the token's lifetime kept the Redis fast path serving an expired bearer until eviction, while the stored refresh_token sat unused because refresh only runs on the DB read-through
The configured TTL is now capped at expires_in minus the expiry buffer. Shorter TTLs and servers without the field behave exactly as before, and the TTL still applies verbatim when the upstream reports no expires_in. The dashboard tooltips on the create and edit forms are updated to describe the capped behavior
Rewrites the three Batch A tables from hand-rolled TanStack + tremor renderers
into thin DataTable consumers with a separate ColumnDef module each, matching
the Guardrails and Tags migrations. Row actions move into a per-row overflow
menu (edit/copy/delete for vector stores; copy for everyone plus admin-gated
delete for prompts and skills). The vector stores parent gains an isLoading
flag resolved on every exit path so the table shows the shared skeleton
instead of flashing the empty state. Table files are renamed to PascalCase
and stale eslint bulk-suppressions for the rewritten files are pruned.
