Commit graph

41724 commits

Author SHA1 Message Date
Tin Chi Lo
fa444265e0 feat(cache_warming): warm only the models a session has been served on
Some checks failed
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
Warming resolved its target set from configuration, so every active session was
replayed against a representative of every tier on every interval whether or not it
had ever been routed there. Most sessions never leave their starting tier, so that
spent N replays per interval to keep caches warm that nobody would read, and for a
pooled tier it warmed the wrong member entirely.

The set is now per session and comes from what the session actually did. Capture
records each served model into a per-session Redis set inside the same atomic script
that writes the record, sharing the session's hash tag, so the touched set cannot
disagree with the record it belongs to and expires with it. The refresher replays
exactly that set.

This is the intended shape of the feature: keep a session's own caches alive so
returning to a tier it has already used is a read, rather than pre-warming tiers on
speculation. The first switch to a new tier is a normal cache write, and every visit
after it is warm. warm_models changes meaning accordingly, from a pre-warm list to
an allowlist that narrows what a session may be warmed on, and resolve_warm_models
now returns every model across the tier pools since it bounds eligibility rather
than naming the targets.

Tests that expected a replay on a tier the seeded session had never visited now
declare a session that has been to both, which is the case the feature serves.
2026-07-30 11:03:48 -07:00
Tin Chi Lo
ed39e60a07 fix(cache_warming): warm the model the session was actually served
A tier may be a pool the router picks from at random, but the warm set resolved to
one representative per tier, so for a pooled tier the member holding the session's
cache was usually not warmed at all. That inverted the feature: the session's own
cache expired at the provider TTL and its next turn on the same tier paid a full
cache write, while warming spent on a pool member the session had never touched and
might never be routed to.

The warm set is now per session, leading with the record's served_model and then the
tier representatives, so a session's own cache is always refreshed. Eligibility is
still resolved once per tick over the union, so no extra model-list lookups happen
per session.

Single-model tiers, which every existing test and the live proof used, are
unaffected: the served model is the tier representative there, which is why the gap
did not surface earlier.
2026-07-30 10:32:39 -07:00
Tin Chi Lo
bb0d5684f7 test(cache_warming): fold the SCIM gate into the ceiling matrix
The standalone predicate test asserted four return values of
user_is_scim_deactivated and never exercised warming, which is coverage of the
helper rather than of the feature; the helper is already pinned by the auth and MCP
paths that own its behavior. The ceiling matrix is the right home, since it is the
test that answers whether a refusal actually stops a replay, so SCIM deactivation
joins the blocked key, expired key, blocked team, denied model and budget arms and
asserts the session is not warmed. Removing the gate fails that arm.
2026-07-30 02:15:28 -07:00
Tin Chi Lo
4bef86c1e1 fix(cache_warming): let the concurrency bound also bound decompressed payloads
Every session in a tick is started at once, so anything a session materializes
before acquiring its replay slot scales with max_sessions rather than with the
concurrency setting. The payload was inflated above the semaphore and held for the
whole replay, and capture admits payloads up to eight times the compressed cap, so
one tick could hold a decompressed payload per active session instead of per replay
in flight.

Decompression moves inside the slot, which is now held across the session's whole
due set. The replay ceiling is unchanged, since a session's models are replayed in
sequence inside its slot, and the CPU burst of inflating is now serialized to the
same bound rather than running for every session at once.

The bound had no test at all; the rig tracked peak in-flight replays and nothing
asserted on it. One test now pins both halves through a real tick: replays in
flight and payloads inflated must both respect max_concurrent_replays. It fails on
the previous ordering with the payload count at the session count, not the bound.
2026-07-30 01:20:08 -07:00
Tin Chi Lo
ca956b5477 fix(cache_warming): read the caller's redaction opt-out from its owner
should_redact_message_logging takes a model_call_details dict whose required shape
is implicit: the header and global forms are read off litellm_params, but the
per-request form is read from standard_callback_dynamic_params at the top level.
Capture passed only litellm_params, so a caller setting turn_off_message_logging
in the request body had its opt-out silently dropped and its prompts retained in
Redis anyway; the two forms that happened to be tested, headers and the global
setting, both worked, which is why the gap survived. The gate's own docstring
claimed the per-request form was honored.

The value is owned by the logging object, which initializes it from the request in
its constructor, so it is read off the object rather than re-derived, and the key
is spelled exactly as the request path spells it when it builds the same dict for
the same predicate. Where no logging object rides the request (SDK-direct use)
the behavior is unchanged, and those callers consent through cache_warming.enabled
itself.

The matrix test already named a per-request case but exercised the header leg, so
it now drives all three body spellings (root, metadata, litellm_metadata) through
a real Logging object built the way function_setup builds it, plus a
no-opt-out control proving the same path still captures. The three body cases fail
without this change and the control does not.
2026-07-30 00:25:51 -07:00
Tin Chi Lo
bd8511dd51 refactor: extract user_is_scim_deactivated as one shared predicate
The SCIM-deactivation check was spelled out inline at five call sites (the standard auth
builder, both MCP admission arms, and both bridge-refresh revalidation paths). Each one
re-derived the same three conditions, and cache warming, which resolves a user through
get_user_object exactly like those five do, shipped without the check at all; a sixth
caller getting it wrong is what a copy-pasted predicate guarantees eventually.

The predicate now lives beside get_user_object in auth_checks.py, since the resolver is
what hands back a live-looking row for a deactivated user and every caller owes the check
afterwards. It returns a bool rather than raising, so each caller keeps its own reaction:
the standard builder raises, the MCP arms return 401, the bridge returns a status, and the
warming refresher raises a ProxyException before it can spend against a deactivated
owner's key. Only an explicit scim_active of False deactivates, so a missing user or
absent metadata still fails open exactly as before at every site.
2026-07-29 23:59:00 -07:00
tin
16bf7efd41 test(complexity_router): derive the warm-pick record key from the shared caller scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-30 05:38:11 +00:00
Tin Chi Lo
a4db51b1a1 fix(complexity_router): scope keyless callers by their stamped tenancy, not one shared bucket
caller_scope fell back to the literal "unscoped" whenever user_api_key_hash was
absent, which is the normal shape for JWT and other keyless proxy principals.
Distinct tenants that reused a session_id therefore collided on one Redis record,
so the last writer's payload and attribution won and later replays could spend
under the wrong team or user.

Scoping now derives from the strongest identity the request actually carries, in
one shared owner (core_helpers.get_caller_scope) that capture, the affinity-aware
pick and the session-affinity pin all call. This mirrors how the v3 rate limiter
builds an api_key descriptor only when api_key is present and separate descriptors
per user, team and organization (parallel_request_limiter_v3.py:1988), and how
DeploymentAffinityCheck declines to scope rather than sharing a bucket.
"unscoped" now means only what it says: no proxy identity at all, which is direct
SDK use where there is one tenant by construction.
2026-07-29 22:27:50 -07:00
Tin Chi Lo
343201e4b9 fix(complexity_router): never fall back a warming replay to an unvalidated group
Eligibility, every-member cacheability, deployment affinity and pricing are
properties of the target model group. When a replay failed and the key had not
explicitly disabled fallbacks, the router could send it to a fallback group
carrying none of that validation, spending the customer's money without warming
the tier the session will actually switch to.

Fallbacks are now disabled at the dispatch site rather than in the request body,
so no key-level control or pre-call mutation can re-enable them. router.py:6157
raises before fallbacks, context_window_fallbacks and content_policy_fallbacks
are consulted, so the single flag covers all three kinds; a failed replay simply
retries on the next tick, which was already the documented intent.
2026-07-29 21:16:52 -07:00
tin
22635302bd fix(complexity_router): stamp a cache-warming replay warm only when it lands
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-30 04:10:06 +00:00
Tin Chi Lo
00e1fe8822 feat(complexity_router): multi-model prompt-cache warming for the auto-router
Provider prompt caches are per-model, so every mid-session tier switch the
complexity auto-router makes lands on a cold cache and pays the full cache write
again. Opt-in cache_warming captures each session's latest payload at the routing
decision and a leader-elected background refresher replays it with max_tokens=1
against every cacheable tier model just under the provider cache TTL, so the
switch is a pure cache read.

A replay is a request, so it is admitted through the request path's own entry
points rather than beside them. For each replay the refresher assembles a request
body, reserves budget through the same wrapper auth calls right after
common_checks, stamps identity with the proxy's own stamper, applies every
key-level, team-level and project-level control, applies the key and team scoped
dynamic logging settings, runs ProxyLogging.pre_call_hook, applies the
fully-blocked-model check, and hands the dict that hook returns to
Router.acompletion or Router.aanthropic_messages. post_call_failure_hook runs on
every rejection and every dispatch failure, so the parallel request slot, the
reserved TPM tokens and the budget reservation all come back. Warming therefore
inherits both halves of every contract it touches (the limiter's descriptors
across every scope with its own configured window, its RPM and max-parallel
check, its upfront reservation and the stash its success callback reconciles
from; the key, team, user, end-user, organization and tag budget counters; every
configured guardrail and pipeline, including the ones defined on the deployment)
instead of reimplementing them. That deletes nine functions and the admission
block they served.

Warming writes no spend logs of its own, so the replay rows are the only record
of warming cost that will exist. They carry the customer's own tags and
spend_logs_metadata with the litellm_cache_warming tag alongside rather than
instead, and they fan out to the key and team scoped loggers, so warming is
included in per-tag chargeback and filterable out of it. Two Request-free blocks
of add_litellm_data_to_request are extracted verbatim as
LiteLLMProxyRequestSetup.add_key_team_project_metadata and
apply_dynamic_logging_settings so both callers share them; the move is
statement-for-statement identical, with no behavior change on the request path.
Ordering there is load-bearing: add_key_level_controls resets data["cache"] and
refills it from key metadata, so it runs after the body is built and a key's own
cache controls override warming's response-cache bypass exactly as they override
a caller's.

Blocked and expired keys are still checked locally because common_checks
dereferences the FastAPI Request; extracting a Request-free core so its other
gates bind on a replay too is a follow-up. Sessions on a key that declares
max_iterations are skipped entirely, because that limiter counts every request on
a session_id and cannot be consulted without incrementing it.

Metadata precedence (litellm_metadata before metadata, stringified) had three
implementations; core_helpers.iter_request_metadata_dicts and
get_request_metadata_field are now the single owner and DeploymentAffinityCheck
deletes its four private copies to delegate to them.

Resolves LIT-4865
2026-07-29 20:59:17 -07:00
Mateo Wang
47f1fb394e
Merge pull request #35172 from BerriAI/litellm_vertex_cache_skip_tool_final
fix(vertex_ai): skip context caching when the cached block ends on a model turn
2026-07-29 20:24:42 -07:00
Mateo Wang
072a6eef50
Merge pull request #32601 from BerriAI/litellm_fix_managed_file_id_idempotent_regression
test(managed-files): lock in store_unified_file_id idempotency on batch retrieve
2026-07-29 19:56:05 -07:00
mateo-berri
819dc7812a fix(vertex_ai): evaluate cached-block terminal turn after system extraction 2026-07-29 19:47:26 -07:00
tin-berri
04177e7419
Merge pull request #35164 from BerriAI/litellm_classifier_reqlog
fix(complexity_router): capture the classifier request body in spend logs
2026-07-29 19:30:39 -07:00
mateo-berri
a5c79e7708 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_managed_file_id_idempotent_regression 2026-07-29 19:10:57 -07:00
Mateo Wang
56d51bc32e
build(makefile): give local basedpyright runs the node heap CI uses (#35173) 2026-07-30 02:00:31 +00:00
Mateo Wang
9f9d72b50f
Merge pull request #34603 from ljogeiger/litellm_vertex_function_call_id
fix(vertex_ai): forward function_call id on Vertex Gemini 3+ tool turns
2026-07-29 18:51:54 -07:00
mateo-berri
04d702c46a test(managed-files): call store_unified_file_id twice and assert upsert payloads 2026-07-29 18:51:25 -07:00
Tin Chi Lo
074eda5222 fix(complexity_router): use Mapping instead of dict in turn_off_message_logging helper parameter
Accept read-only Mapping[str, Any] instead of mutable dict[str, Any] in
_effective_turn_off_message_logging's parameter to satisfy LIT001
(mutable collections in type annotations). Convert to dict for the
function that expects Dict.
2026-07-29 18:45:51 -07:00
Mateo Wang
52fc276f05
Merge pull request #32587 from BerriAI/litellm_fix_batch_model_access_hash_32580
fix(auth): resolve managed batch/file deployment model_id to model name for team access checks
2026-07-29 18:43:39 -07:00
mateo-berri
dbc0d23c1e fix(vertex_ai): skip context caching when the cached block ends on a model turn 2026-07-29 18:38:29 -07:00
Tin Chi Lo
3d5b8e5960 fix(complexity_router): propagate turn_off_message_logging to internal sub-calls
The classifier and semantic-embedding sub-calls now capture proxy_server_request,
but neither forwarded the caller's turn_off_message_logging opt-out. A caller who
disabled message logging still had their prompt stored in the clear in these
internal sub-calls' spend-log rows, since should_redact_message_logging reads the
flag per-call and this internal call never inherited it.
2026-07-29 18:28:48 -07:00
tin
78207064d8 fix(complexity_router): log the classifier request on chat completions too
The classifier read its metadata only from litellm_metadata, which the proxy
populates just for LITELLM_METADATA_ROUTES (/v1/messages, /v1/responses, ...);
/v1/chat/completions puts it under metadata, so the classifier call arrived
unattributed and _should_track_cost_callback dropped it, leaving no spend-log
row at all for the captured request body to show up in.

Also log response_format in the wire shape litellm actually sends
(type_to_response_format_param) instead of the bare pydantic JSON schema

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-29 18:28:48 -07:00
Tin Chi Lo
2a84c39762 fix(complexity_router): capture the classifier request body in spend logs 2026-07-29 18:28:48 -07:00
ryan-crabbe-berri
581f5c319e
feat(cli): read base_url from persistent config file (#35015)
* feat(cli): read base_url from persistent config file

Adds a lite config command group (set/get/unset) backed by
~/.litellm/config.json so users no longer need to export
LITELLM_PROXY_URL in every shell session. Resolution order is
--base-url flag, then LITELLM_PROXY_URL, then the config file,
then the localhost default. A config-file base_url counts as an
explicit server choice for lite auth print-token, matching the
env var semantics it replaces.

* fix(cli): harden config persistence after review feedback

Rejects base_url values containing a query string or fragment,
including bare trailing ? or # which parse as empty but still
corrupt every joined request URL. Writes config.json and token.json
atomically through a shared write_private_json helper (0600 at
creation, fsync, os.replace) so an interrupted save can no longer
truncate the file or leave it world-readable. Warns on stderr when
an existing config file is invalid instead of silently ignoring it,
including invalid UTF-8. Resolves the eager --version flag through
the same env, config file, default chain as every other command,
and reads the config file once per invocation so base_url and
base_url_explicit always come from the same snapshot.

* fix(cli): resolve --version after option parsing

The eager --version callback ran before --base-url and --api-key were
parsed, so it could not see an explicitly named server. Combined with
the env fallback added for config-file support, that sent the resolved
API key to whichever server the config file pointed at even when the
user named a different one on the command line. Making the flag a
normal option and handling it in the group callback gives the version
request the same flag, env, config, default precedence as every other
command, and lets the stored-token lookup stay origin-checked.
2026-07-29 18:25:05 -07:00
Mateo Wang
6fe1e73699
Merge pull request #34847 from BerriAI/litellm_fix_vertex_batch_read_per_model_bucket
fix(vertex_ai): honor per-model gcs_bucket_name on managed-file read path
2026-07-29 18:10:47 -07:00
tin-berri
732364e260
Merge pull request #34990 from BerriAI/litellm_fix_mcp_oauth_discovered_issuer_anchoring
fix(mcp): never write discovery results to the row, heal already-stamped rows, and retry failed discovery with backoff
2026-07-29 18:07:44 -07:00
Tin Chi Lo
7041f5768f fix(mcp): never write discovery results to the row, heal rows a release already stamped, and retry failed discovery with backoff
An interactive oauth2 MCP server created with explicit endpoint URLs and no issuer served
400 "authorization url is not configured" from /authorize about a minute after creation,
with the admin's endpoints intact in the row the whole time (#34985). Discovery wrote its
trust-on-first-use issuer into the same column an admin writes, so the next registry build
read the gateway's own output back as an admin pin, anchored the server to RFC 8414
section 3.3, and discarded the stored endpoint columns; one transient metadata fetch
failure then had nothing to serve, and the reload fast path pinned the broken entry until
an unrelated config write

The core of the fix is a deletion. The gateway no longer writes discovery results anywhere:
the OAuth columns and credentials.scopes carry admin intent alone, and everything discovery
learns lives on the in-memory registry entry, as the existing carry-forward already
assumes. With no gateway write there is no value whose provenance a later build can
misread, so the accidental anchoring cannot be expressed

Deleting the write cannot fix a row a released version already stamped, which still reads
as pinned, so a one-time startup heal clears those stamps. The signal is necessarily a
heuristic: updated_by records only the most recent writer and no audit trail says which
field it touched. A row is therefore healed only on the full signature of the defect, which
is discovery as the last writer plus an issuer plus at least one configured endpoint column
that anchoring is actively discarding; rows with an issuer but no configured endpoints are
left alone, since for them both paths resolve from the same upstream document. Every heal
logs the cleared value so an admin who pinned deliberately can re-pin, and the heal records
its own actor, which makes it idempotent

The reload fast path exempts servers missing an endpoint their flow needs, so failed
discovery retries on the normal reload cadence rather than waiting for a config write. Flow
requirements are read through effective_oauth2_flow, the column-first shape-fallback judge
every flow decision uses, so a legacy null-flow M2M row is classified exactly as the
request path classifies it instead of re-discovering forever; a dcr_bridge server with no
configured client needs its registration endpoint for the relay arm, and an entra_obo
server needs a scope, both of which discovery can supply. Retries back off per server,
doubling from one reload cadence to a fifteen-minute cap, so a permanently unresolvable
server cannot re-run the RFC 9728 to 8414 chain and re-log its warning every cycle forever

Deployments with store_model_in_db unset or false loaded MCP servers exactly once at
startup, leaving that retry with no driver, so they now refresh the registry on the same
reload interval. That job deliberately calls a reload-only entry point rather than the
startup composite, keeping the one-time oauth2_flow backfill and issuer heal out of a
recurring path

Losing the persisted trust-on-first-use issuer also means the issuer column no longer
changes underneath the OAuth token identity, so user tokens are purged only when an admin
actually edits the server

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-29 17:51:22 -07:00
Yassin Kortam
ae74b06ae3
fix(tests): assert Content variants are identified by type, not by the discriminator keyword (#35161)
`test_content_schema_uses_discriminator` fetches Google's live Interactions
OpenAPI document and required an OpenAPI `discriminator` on the `Content` union.
Google has since dropped that keyword and now pins `type` with a `const` on each
variant instead, so the assertion fails on the current spec and the `misc` shard
is red on every open PR against staging

The information the transformation actually needs did not change: a content part
is still routed by reading its `type`, and each variant still declares exactly
one distinct value for it. So the test now asserts that property directly, and
accepts either spelling, a `discriminator` on the union or a `const` (or
single-value `enum`) on each member

It stays a real check rather than a weakened one. Against the live spec it fails
if TextContent loses its type property, if `text` is renamed, if two variants
claim the same type value, if `Content` stops being a union of named variants,
or if a discriminator appears on some property other than `type`
2026-07-29 17:49:30 -07:00
ryan-crabbe-berri
0a6b372126
feat(ui): link organization teams to their team detail pages (#35120)
Some checks failed
CodSpeed Benchmarks / benchmarks (push) Waiting to run
UI Unit Tests / ui-unit-tests (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
* feat(ui): link organization teams to their team detail pages

On the organization info page the teams shown for an org were plain
badges, so walking to a team meant copying its id and finding it by
hand on the teams page

Team badges now link to /teams?team=<team_id>, which opens that team's
detail page directly since #35112. Adds a shared BadgeLink (a badge
rendered as a real anchor with modifier-aware client-side navigation,
so cmd-click opens a new tab) and a teamDetailHref builder for reuse by
future entity links

* fix(ui): format BadgeLink, split its modifier-click chain, and size it up

prettier wanted the Badge props wrapped, and local/no-long-condition-chain
flagged the four-way modifier-click guard; the guard is now two named
conditions. Linked badges also render slightly larger (text-sm, roomier
padding) than plain badges so clickable entries stand out

* feat(ui): size org model badges to match the linked team badges

BadgeLink's href is now optional; without one it renders the same
enlarged plain badge (no pointer, no hover), so the org page's model
badges share the component and the size while staying non-clickable
2026-07-29 17:20:59 -07:00
devin-ai-integration[bot]
551e5d097c
feat(dashscope): add qwen3.7-plus and qwen3.7-max to the model cost map (#35123)
* feat(dashscope): add qwen3.7-plus and qwen3.7-max to the model cost map

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: limit backup cost map diff to the new dashscope entries

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost_calculator): adjust tier-only alias assertion for mapped qwen3.7-plus

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(dashscope): drop redundant cost map pinning tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost_calculator): point tier-only alias check at an unmapped model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-29 14:22:22 -07:00
mubashir1osmani
5182dfa66b
test(e2e): remove the Presidio guardrail suite (#35129)
Drops tests/e2e/guardrails/test_presidio_guardrail_e2e.py and the
PresidioParamsBody it was the only caller of.

Both cases were red on most stage runs between 07-25 and 07-29: pre_call
failed 6 of 11 runs, post_call 6 of 11, with post_call reporting the raw
address reaching the caller while apply_to_output was set.

The cause was propagation, not masking. GuardrailsClient.register() posts
/guardrails and returns immediately with no readiness wait, unlike
ProxyClient._await_model_servable or GuardrailsClient._await_team, and the
data plane only picks a new guardrail up on its next periodic DB sync. Calls
issued before that sync pass the raw value through. #34833 has since made
both cases poll to the deadline, and on the current build each masks on the
first attempt, so the suite is expected to be green now; it is being removed
because it spends real provider money on every retry and because a pod
replaced mid-poll still reproduces the old failure.

The three guardrail.presidio.* rows stay in coverage_registry/guardrail.yaml
and go uncovered on purpose, so Presidio reads as a tier-P0 gap in Grafana
rather than dropping out of the denominator.
2026-07-29 14:20:39 -07:00
Yassin Kortam
440b1bcf65
fix(otel): make OTLP export work against Grafana Cloud (#35060)
Three defects kept LiteLLM's OTel metrics from reaching an OTLP backend.

OTEL_EXPORTER_OTLP_HEADERS is W3C Baggage encoded per the OTLP spec, so its
values are percent-encoded. litellm split the string on "," and "=" and passed
the raw value straight to the exporter, so a vendor that documents
"Authorization=Basic%20<token>" got a literal "%20" on the wire and the backend
rejected the credential. Grafana Cloud documents exactly that shape, which made
its OTLP gateway unreachable. Header parsing now delegates to the OTel SDK's own
W3C Baggage parser in liberal mode, so percent-encoded values decode and values
that were never encoded keep working. It moves from model/utils.py to
plumbing/providers.py because model/ is deliberately free of opentelemetry
imports; providers.parse_headers was already the entry point every caller used.

The OTLP metric exporters then overrode histogram temporality to delta.
Prometheus and Mimir, which back Grafana Cloud's OTLP gateway, reject delta
histograms outright: the gateway answers 400 "invalid temporality and type
combination" and drops the entire batch, so every GenAI metric was silently lost
while traces kept flowing. Backends that prefer delta still accept cumulative, so
the SDK default is the compatible choice in both directions, and the enterprise
billing exporter already relies on it.

Three GenAI instruments also carried names no convention or backend defines, so
nothing downstream could chart them. Time to first token and time per output
token take their semconv names, gen_ai.server.time_to_first_token and
gen_ai.server.time_per_output_token; the gen_ai.client.response.* spellings
litellm used are not conventions at all. Cost has no semconv instrument, so it
takes gen_ai.usage.cost, the name backends already query for spend. All three are
listed verbatim in Grafana Cloud's AI Observability integration reference, so its
prebuilt panels find them. Both engines now read the names from the shared Metric
constants rather than repeating string literals, so v1 and v2 cannot drift.

The renames are breaking for anyone charting the former names; the docs and the
release changelog carry the migration note.
2026-07-29 13:43:33 -07:00
devin-ai-integration[bot]
ea783cc35c
refactor(rust): make litellm-core the callable messages() SDK; drop the ai-gateway handler (#35044)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-07-29 13:41:31 -07:00
ryan-crabbe-berri
bf5334bc59
fix(router): drop duplicate Mapping import that fails ruff F811 (#35122)
router.py imports Mapping from collections.abc and again from typing,
which ruff flags as a redefinition and fails the lint CI job on every
open PR. Keep the collections.abc import
2026-07-29 12:32:55 -07:00
Mateo Wang
8b03315ac6
Merge pull request #35028 from BerriAI/litellm_batch_provider_credentials
fix(proxy): resolve named credentials on provider-only batch and files calls
2026-07-29 11:59:09 -07:00
ryan-crabbe-berri
ba7d8ae17f
feat(ui): deep link organization detail page via ?org= query param (#35117)
* feat(ui): deep link organization detail page via ?org= query param

The organizations page kept the selected organization in React state, so
an org detail page had no URL: it could not be shared, bookmarked, or
opened from another page, and the browser back button dropped you out of
the page instead of closing the detail view

Adds useOrgDetailRouting reading ?org= (same pattern as the api-keys,
models, logs, and teams deep links) and derives the open organization in
OrganizationsPanel from the URL

* fix(ui): reset org edit mode on plain row selection and type test mocks

Greptile P1: with the selected org now URL-derived, browser Back leaves
the detail view without running onClose, so a stale editOrg=true made
the next plain row click open on the Settings tab. Reset the flag on
row selection, matching the teams page

Greptile P2: type the panel test's captured table and detail-view props
from the real components instead of any
2026-07-29 11:56:37 -07:00
ryan-crabbe-berri
bb769702b1
feat(ui): deep link team detail page via ?team= query param (#35112)
The teams page kept the selected team in React state, so a team detail
page had no URL: it could not be shared, bookmarked, or opened from
another page, and the browser back button dropped you out of the page
instead of closing the detail view

Adds useTeamDetailRouting reading ?team= (same pattern as the api-keys,
models, and logs deep links) and derives the open team in Teams.tsx from
the URL. TeamInfo now also derives team-admin rights from the fetched
team data, so team admins arriving via a deep link are not stuck with a
read-only view
2026-07-29 11:35:06 -07:00
Mateo Wang
c56e657097
Merge pull request #34266 from BerriAI/litellm_team-model-allowlist-stale-qqg50q
fix(proxy): stop serving stale team model allowlist after /team/update
2026-07-29 11:00:38 -07:00
yuneng-jiang
74244ddd45
Merge pull request #35041 from BerriAI/litellm_/ui-perf-regression-d7888c
fix(ui): point the navbar and sidebar logos at the dashboard home route
2026-07-29 10:59:53 -07:00
Mateo Wang
2348ccc977
Merge pull request #35107 from BerriAI/litellm_usage_public_model_names
fix(ui): show public model names in usage breakdowns
2026-07-29 10:42:13 -07:00
ryan-crabbe-berri
fdea50daa2
feat(ui): shareable log links via log_id query param on the logs page (#34879)
* feat(ui): shareable log links via log_id query param on the logs page

Clicking a log row now writes ?log_id=<request_id> to the URL, closing the
drawer removes it, and loading the logs page with ?log_id= opens the drawer
for that log. When the log is not in the loaded page, it is fetched by
request_id (the backend already drops the date window for id lookups), so
links keep working for logs of any age. Drawer open state derives from the
URL, mirroring the models page ?model= pattern.

* fix(ui): close the log drawer on browser back after opening via session id

Session opens now write ?session_id= to the URL instead of holding local
state, so back removes both params and the drawer closes (Greptile P1).
Session views become shareable links as a side effect. In-drawer log
switching now replaces the history entry instead of pushing, so back
always closes the drawer in one step rather than replaying every viewed
log.

* fix(proxy): scope /spend/logs/session/ui to the requesting user's visible logs

Non-admin callers now only receive session rows they could already see on
/spend/logs/ui: their own logs plus logs of teams where they hold the
spend-logs permission. Previously any authenticated user could read any
session's log metadata by id, which shareable ?session_id= links made
trivial to trigger. Admin views are unchanged. Also, clicking a log row
now clears a lingering ?session_id= from the URL so the drawer shows the
clicked log instead of a stale session (Greptile P1).
2026-07-29 17:01:09 +00:00
mateo-berri
d17387e2e1 fix(proxy): fall back on empty-string model_group in aggregated usage SQL 2026-07-29 09:59:34 -07:00
ryan-crabbe-berri
9b7a6b9b90
feat(ui): split failed requests into their own series on the cache dashboard (#34862)
* feat(ui): chart failed requests as their own series on the cache dashboard

Spend logs for failed requests are stored with an empty call_type, so the
Cache Hits vs API Requests chart lumped them into an Unknown bar that read
as normal LLM API traffic. The activity query now also returns a per-group
failed_rows count (status = 'failure') and the dashboard charts it as a
third stacked series, so failures are visibly separate from successful
requests and cache hits. The chart data transform moves into a pure
summarizeCacheActivity helper with unit tests; header stats keep their
existing semantics (cache hit ratio still counts failures in the
denominator).

* refactor(ui): move cache dashboard aggregation server-side with a typed response

The /global/activity/cache_hits endpoint previously returned raw per
(key, call_type, model) spend-log aggregates typed as LiteLLM_SpendLogs
(wrong), and the dashboard reduced them in the browser: grouping by
call_type, relabeling empty call_type as Unknown, and computing the stat
card totals. All of that now happens server-side. The SQL groups per
call_type and splits cache hits vs successful vs failed requests, a new
cache_activity module validates rows into Pydantic models and computes
totals plus the key-alias/model filter options, and the endpoint declares
a real response_model so schema.d.ts types it correctly. The dashboard
consumes it through a typed $api react-query hook (filters ride the
query key and are applied in SQL instead of the browser), the hand-rolled
summarizeCacheActivity transform and the adminGlobalCacheActivity fetch
helper are deleted, and the refresh button now actually refetches.

The endpoint is UI-internal (hidden from the public swagger), so the
response reshape is not a public API break.
2026-07-29 09:48:17 -07:00
ryan-crabbe-berri
40878a1ed5
fix(proxy): allow /key/update to identify the key by key_alias (#34851)
* fix(proxy): allow /key/update to identify the key by key_alias

* fix(ui): drop machine-dependent union-order churn from generated schema.d.ts
2026-07-29 09:48:08 -07:00
ryan-crabbe-berri
fe1670fc06
fix(ui): size object permissions card grid by container width (#35019)
The card variant used viewport breakpoints (md:grid-cols-2 lg:grid-cols-3)
but every card usage sits in a one-third-width grid cell, so on desktop the
narrow card still rendered three internal columns of roughly 100px each and
the text spilled out of its boxes. Switch to Tailwind container queries so
the internal column count follows the card's own width
2026-07-29 09:47:54 -07:00
mateo-berri
802ed1c74f fix(ui): show public model names in usage breakdowns 2026-07-29 09:46:49 -07:00
Mateo Wang
e299d2b970
Merge pull request #35074 from BerriAI/litellm_daily_any_cleanup_07_29_2026
chore(typing): clear basedpyright Any errors in proxy management endpoints
2026-07-29 09:42:19 -07:00
ryan-crabbe-berri
25ebe5600a
fix(scim): stop provisioning nested group ids as internal users (#34997)
* fix(scim): stop provisioning nested group ids as internal users

POST/PUT/PATCH /scim/v2/Groups treated every member.value as a user id, so
with the default scim_upsert_user=true an unknown id was auto-created as an
internal user. Entra sends nested groups as members carrying "type": "Group",
which meant every nested group produced a phantom internal user whose id and
email were the group GUID, and those users counted toward licensed seats.

Group members are now classified before they are used: members typed "Group"
are skipped without a database hit, an id that names an existing team is
skipped too (Okta sends untyped ids through filtered paths, so the type alone
is not enough), and only ids that resolve to a user, or that resolve to
nothing at all, keep today's behavior. The user lookup runs before the team
lookup so a user whose id collides with a team id keeps syncing.

The type was previously dropped at parse time on POST/PUT because SCIMMember
had no such field, and on PATCH because the raw member dicts were reduced to
bare ids; both paths now share one resolver and one parser that preserves it.

Member removals no longer upsert: a remove of an id we do not know is an
idempotent no-op rather than a reason to create a user and immediately drop
it, and strict mode (scim_upsert_user=false) no longer rejects it. Removal of
an id that is on the roster but has no user row still cleans up membership.

Responses now state members are of type "User" instead of emitting a null,
and the advertised Group schema documents the members.type sub-attribute.

* fix(scim): harden group member classification after adversarial review

Removals now bypass classification and drop exactly the ids they name,
restoring cleanup of roster entries the old bug left behind. The
team-id fallback only applies to untyped members, so an explicit User
type always provisions even when the id collides with a team. Member
types are normalized before matching; a type other than User or Group
only skips when the id is not an existing user. Non-string type values
are tolerated as absent on every verb instead of failing validation.
Admitted member ids are deduped order-preserving, which also closes a
pre-existing duplicate-row hazard on group creation.

* fix(scim): only treat scim-managed teams as nested groups

A PR reviewer flagged that an untyped SCIM member whose id collides
with an admin-created team was silently skipped, suppressing that
user's provisioning. SCIM group writes (POST, PUT, and every PATCH)
now stamp the team with scim_managed metadata, and the typeless
team-id skip only applies to teams carrying that marker or the
scim_data blob older PUTs already wrote. Admin-created teams stay
unmarked, so a colliding untyped member provisions the user in
permissive mode and returns the standard unknown-user 400 in strict
mode. Teams SCIM touched before this change adopt the marker on their
next group write.
2026-07-29 09:38:17 -07:00