Commit graph

47270 commits

Author SHA1 Message Date
yuneng-jiang
912a3f88f8
Merge pull request #40346 from BerriAI/litellm_backport_40335_rc_1_101_0
Some checks are pending
LiteLLM Rust / rustfmt, clippy, test (push) Waiting to run
LiteLLM Rust / release wheel (push) Waiting to run
fix(otel): backport auth spans and callback merge to rc/1.101.0 (#40335)
2026-09-08 18:31:32 -07:00
Yuneng Jiang
bcddffcbdc
fix(otel): backport auth spans and callback merge to rc/1.101.0
Cherry-pick PR #40335, restoring the Datadog auth span and last-wins callback credential merging.

(cherry picked from commit 43a1b2992a)
2026-09-08 18:28:11 -07:00
yuneng-jiang
f0b6d66b84
Merge pull request #40325 from BerriAI/litellm_backport_37667_rc_1_101_0
feat(team): backport team admin callbacks to rc/1.101.0 (#37667)
2026-09-08 16:42:17 -07:00
yuneng-jiang
f41fc719c1
Merge pull request #40324 from BerriAI/litellm_backport_40312_rc_1_101_0
chore(ui): backport dashboard dependencies to rc/1.101.0 (#40312)
2026-09-08 16:42:09 -07:00
Yuneng Jiang
b6e73cccdb
feat(team): backport team admin callbacks to rc/1.101.0
Backport #37667 without conflict resolution or implementation changes

(cherry picked from commit 90731576e3)
2026-09-08 16:35:28 -07:00
Yuneng Jiang
d8b9fcf399
chore(ui): backport dashboard dependencies to rc/1.101.0
Backport #40312 without conflict resolution or implementation changes

(cherry picked from commit 8280d7ca9d, mainline parent 1)
2026-09-08 16:35:28 -07:00
yuneng-jiang
cbf0c76869
Merge pull request #40321 from BerriAI/litellm_backport_39654_rc_1_101_0
feat(otel): backport tenant trace destinations to rc/1.101.0 (#39654)
2026-09-08 16:34:13 -07:00
Yuneng Jiang
09976043ac
feat(otel): backport tenant trace destinations to rc/1.101.0
Backport #39654 without conflict resolution or implementation changes

(cherry picked from commit 3165951e6d)
2026-09-08 16:29:16 -07:00
yuneng-jiang
744156d297
Merge pull request #40316 from BerriAI/litellm_backport_40203_rc_1_101_0
feat: backport MongoDB sidecar to rc/1.101.0
2026-09-08 15:56:32 -07:00
yuneng-jiang
935bb5260f
feat: backport MongoDB sidecar to rc/1.101.0
(cherry picked from commit b8d573c5f9)
2026-09-08 15:52:58 -07:00
ryan-crabbe-berri
eeb7732fc1
Merge pull request #39994 from BerriAI/litellm_hotfix_mongodb_extra_docker_images
Some checks failed
Unit Tests / caching-local (push) Has been cancelled
Unit Tests / core-utils (push) Has been cancelled
Unit Tests / enterprise-routing (push) Has been cancelled
Unit Tests / integrations (push) Has been cancelled
Unit Tests / All Other Providers (push) Has been cancelled
Unit Tests / Vertex AI (push) Has been cancelled
Unit Tests / misc (push) Has been cancelled
Unit Tests / proxy-auth (push) Has been cancelled
Unit Tests / proxy-infra (push) Has been cancelled
Unit Tests / proxy-server (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Unit Tests: Documentation Validation / documentation (push) Has been cancelled
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
Unit Tests / enterprise-package (push) Has been cancelled
Unit Tests / proxy-endpoints (push) Has been cancelled
Unit Tests / proxy-extras (push) Has been cancelled
Unit Tests / responses-caching-types (push) Has been cancelled
Unit Tests: Proxy DB Operations / auth-checks (push) Has been cancelled
Unit Tests: Proxy DB Operations / budgets (push) Has been cancelled
Unit Tests: Proxy DB Operations / custom-logging (push) Has been cancelled
Unit Tests: Proxy DB Operations / db-and-spend (push) Has been cancelled
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Has been cancelled
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Has been cancelled
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Has been cancelled
Unit Tests: Proxy DB Operations / key-generation (push) Has been cancelled
Unit Tests: Proxy DB Operations / logging-misc (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-runtime (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-server-core (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-utils (push) Has been cancelled
fix(docker): ship pymongo in the proxy images for the MongoDB vector store
2026-09-05 18:12:56 -07:00
Yuneng Jiang
16fd14f537
fix(docker): ship pymongo in the proxy images for the MongoDB vector store
The MongoDB Atlas vector store provider imports pymongo lazily from the
opt-in `mongodb` extra, but none of the shipped images installed that
extra. Any image-based deployment that configured a MongoDB vector store
failed at search time with "requires the 'pymongo' package", which the
user cannot fix without extending the image

Adds `--extra mongodb` to every uv sync in the root Dockerfile,
Dockerfile.database, Dockerfile.non_root, and the gateway component
image. The backend component does not serve /vector_stores so it is left
as is. The extra resolves from the existing uv.lock to pymongo 4.17.0
plus dnspython 2.8.0, no lock change needed
2026-09-05 18:03:24 -07:00
yuneng-jiang
e56b26c255
Merge pull request #39849 from BerriAI/litellm_internal_staging
chore(ci): promote internal staging to main
2026-09-05 16:41:13 -07:00
yuneng-jiang
bde58f9b41
Merge pull request #39959 from BerriAI/litellm_/release-ui-build-e256a3
chore: rebuild Admin UI bundle for the next release
2026-09-05 16:40:13 -07:00
Yuneng Jiang
0e118342ac
chore: update Next.js build artifacts (2026-09-05 23:25 UTC, node v24.19.0) 2026-09-05 16:25:57 -07:00
yuneng-jiang
b1e2f5bc0b
Merge pull request #39969 from BerriAI/revert-39121-litellm_lazy_sdk_import
revert: perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS (#39121)
2026-09-05 16:22:55 -07:00
devin-ai-integration[bot]
9a5564f00e
fix(ui): read Usage Total Requests tile from gateway request counts (#39963) 2026-09-05 16:20:56 -07:00
Yuneng Jiang
eee162b317
fix(lazy_imports): type import_map as Mapping to stay under the LIT001 budget
The revert restored the dict annotation that #39121 had loosened to Mapping,
and the LIT001 ceiling has been ratcheted down since, so the gate rejected the
one reintroduced hit. Annotation only, no behavior change.
2026-09-05 16:12:45 -07:00
ryan-crabbe-berri
587311d0e4
Merge pull request #39968 from BerriAI/litellm_remove_dead_model_hub_public_dialog
fix(ui): remove unreachable AI Hub dialog that put the session key in a URL
2026-09-05 16:10:15 -07:00
yuneng-jiang
45cf1a7ef1
Revert "perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS (…"
This reverts commit c091dd4608.
2026-09-05 16:07:09 -07:00
ryan-crabbe-berri
79a5614c18 fix(ui): remove unreachable AI Hub dialog that put the session key in a URL
The Public Model Hub dialog in ModelHubTable was never opened (its open setter had no callers), but its See Page button navigated to /model_hub_table?key=<session key>. Delete the dialog, its state, the handler and the unused router import so the path cannot be revived.
2026-09-05 16:04:50 -07:00
yuneng-jiang
1b25132863
Merge pull request #39953 from BerriAI/litellm_/litellm-e2e-flaky-test-2159ae
test(e2e): judge /v1/messages streaming on the clock, not on the provider's delta count
2026-09-05 16:04:45 -07:00
ryan-crabbe-berri
a0c9095cfc
Merge pull request #39960 from BerriAI/litellm_cache_hit_guardrail_spend
fix(proxy): keep guardrail cost in spend on cache hits
2026-09-05 15:53:34 -07:00
ryan-crabbe-berri
63156a7bd6 test(proxy): explain the proxy_server patches in the cache-hit regression test
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 15:44:37 -07:00
ryan-crabbe-berri
acddd21860 fix(proxy): keep guardrail cost in spend on cache hits
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 15:32:02 -07:00
Mateo Wang
bf51dea36b
Merge pull request #39862 from BerriAI/litellm_lit_6992_cohere_parse
feat(ocr): add Cohere Parse support for cohere and azure_ai
2026-09-05 15:16:16 -07:00
yuneng-jiang
6a4fb2bbe8
Merge pull request #39938 from BerriAI/litellm_e2e_vertex_cache_first_call
test(e2e): prove Vertex context caching on the first cold call and on the spend row
2026-09-05 15:10:15 -07:00
Yuneng Jiang
cd976624d1
test(e2e): drop the explanatory sentence from the StreamingResponse docstring 2026-09-05 14:53:12 -07:00
Yuneng Jiang
b55a4317a6
test(e2e): annotate new stream-timing locals as Final and trim the docstrings 2026-09-05 14:49:56 -07:00
Yassin Kortam
0cb759772c
fix(ui): show indirectly granted and name-keyed MCP servers in the tool matrix (#35154)
* fix(ui): show indirectly granted and name-keyed MCP servers in the tool matrix

The MCP tool permission editor was fed the direct server list only, so a server a
principal reaches through an access group or a toolset never appeared in the matrix.
That single blind spot produced two opposite bugs depending on how a save handler
filtered mcp_tool_permissions: filtering by the selected servers deletes an indirect
server's allowlist, and because a missing entry means "no restriction from this
level", the principal silently gains every tool on it; not filtering leaves a stale
entry that keeps a removed access group's server reachable, since a server named
under mcp_tool_permissions is entitled on purpose.

The editor now resolves the selected access groups and toolsets to their servers and
renders them alongside the direct ones, badged with where the grant comes from, so an
admin can see and clear an inherited server's tools like any other. Resolution reuses
the data the selector already loads: access groups resolve from each server's
mcp_access_groups, toolsets from the toolset's own tool list. When that data cannot be
loaded the editor says so instead of rendering an empty list, because an absent
inherited server reads as "there are none". Servers named only by an
mcp_tool_permissions key are listed too, which is what makes a leftover entry
visible; the opt-out sentinel still renders nothing, since it short-circuits the
backend resolver to zero servers.

Opening the editor no longer applies the delete-blocked-by-default allowlist to an
inherited server. Writing an entry for one would narrow a grant the admin never
touched just by opening the form; direct servers keep that default.

Both components also matched on server_id alone, while the backend accepts a server
id, name or alias interchangeably. A grant or allowlist written by API or config with
a name rendered as a selected server with no tools under it, which reads as "this
server has no tools". Matching now covers all three identifiers, and an edit writes
back to the key the entry already uses rather than forking a second id-keyed entry.

The same mismatch could also put one server under several keys at once, its id and
its name for instance. The backend unions every key's list, so reading one key
understated what was in force and writing one key left the others granting. The
resolver now reports, per server, the key an edit keeps, the equivalent keys it
supersedes, and the union those keys allow; the card renders the union and every
write goes through one function that writes the kept key and drops the superseded
ones. A key that also names a DIFFERENT server, which happens when two servers share
a name, is never dropped, because dropping it would strip the neighbouring server's
restriction; the card names such a key and says its tools stay allowed until the
servers no longer share the name, so an admin is told rather than left to infer it
from an edit that bounces back.

A third divergence from the backend sat in the same matching. The backend resolves an
identifier with exact-id precedence: a string that is a registry server id names that
server and stops, and only a string that is no server's id falls back to name and alias,
which can name several. Matching all three fields at once meant a server merely named
after another server's id joined the matrix as if it had been selected, and because it
landed there as a directly selected server it also received the delete-blocked default
write on open. Since an mcp_tool_permissions key is itself a grant source, saving then
handed out a server nobody granted, with no admin gesture involved. Identifier
resolution now mirrors the backend's precedence, and a key is read as this server's only
when it resolves back to it, so an entry that belongs to the id's owner is neither read
into this server's allowlist nor overwritten by an edit made against it.

A toolset grant was also invisible to the tool matrix. The backend unions a toolset's
tools with whatever mcp_tool_permissions allows, so a toolset-only grant restricts the
server to that toolset's tools; the editor read the map alone, found no entry and
rendered every tool on the server as allowed. Deselecting one from that state wrote all
the others as a permission entry, and the union turned a revocation into a grant of
every tool the toolset never included. The resolved entry now carries the toolset's
tools, so the matrix opens on what is actually in force, the delete-blocked default is
withheld from a server a toolset restricts, and a write keeps out the tools only the
toolset accounts for so a grant that ends with the toolset does not become a standing
one. Those tools cannot be revoked from this screen at all, since the backend unions
them in; they render allowed and locked and the card says which of them a toolset holds
open and where to go to revoke them.

That guard originally covered only the keys an edit supersedes, on the assumption that
the key it keeps names one server. It does not when a shared key is a server's only
entry: it then becomes the key an edit writes, and writing it moves the other server's
allowlist too, which is the widening the guard exists to prevent. The key an edit writes
is now the first one naming this server and no other, falling back to the server's own
id, so a shared key is never written through and an edit against one card cannot reach
the server behind the other. Both cards say the shared key holds tools open, since
neither can revoke them.

No owner's save handler changes here. With the full effective set now available to
the editor, the key and team handlers can filter against it instead of guessing,
which makes the internal-user surface's unfiltered save redundant

Resolves LIT-4963
Resolves LIT-4958

* chore: drop tsbuildinfo churn from merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): satisfy dashboard lint budgets

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep MCP tool allowlists for indirect grants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep standing MCP grants on team save

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(ui): format TeamInfo and hoist inline object args

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep MCP tool allowlists for team servers granted indirectly (#35153)

* fix(ui): filter team MCP tool allowlists against the effective server set

Saving a team filtered mcp_tool_permissions down to the directly selected servers. A server reached
through an access group or a toolset is never in that list, so any save dropped its entry, including
a save that only changed the team alias. Because the resolver unions tool-permission keys into the
entitled server set and treats a missing entry as "no restriction from this level", the team kept
the server and lost the tool allowlist on it

Filtering on the direct list alone cannot get this right in either direction. Keeping every entry a
level did not directly select leaves a removed access group's server reachable through its own stale
entry, which breaks revocation. Dropping on deselection alone widens a server that an access group
still supplies

The save handler now resolves the effective server set with resolveEffectiveMcpServers and keeps an
entry only when something other than the entry itself still grants that server: a direct selection, a
selected access group, or a selected toolset. Unified access group ids are added when that selection
is untouched, since the loaded server list is then still accurate

When the server or toolset list cannot be resolved, every entry is kept and the admin is told the
allowlists were saved unchanged. Pruning on incomplete knowledge is the direction that silently
widens, so it only happens when the editor can show the server became unreachable. A failed lookup
and a changed access group selection are separate cases in a tagged union, so the notice names what
actually happened instead of describing the intentional one as a failure, and both hooks gate the
filter symmetrically so a save fired before toolsets settle cannot resolve against an empty toolset
list

Resolves LIT-4961

* fix(ui): resolve team MCP grants from access group metadata and refuse unsafe saves

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): resolve team access group grants from team info when the access group list is role-gated

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): match every selected access group by id instead of by count

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): reload team access group grants at save time

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep frontend lint budget within limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): cover a standing allowlist no group grant covers at load or save

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(ui): keep MCP grant inputs in named variables for the lint budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): guard MCP default write on toolset load, keep create toolsets, fix flat view

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 14:49:30 -07:00
Yuneng Jiang
d56affa814
test(e2e): judge /v1/messages streaming on the clock, not on the provider's delta count
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.

The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.

Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
2026-09-05 14:44:21 -07:00
ryan-crabbe-berri
1745d74293
Merge pull request #39853 from BerriAI/litellm_guardrail_usage_cost_ui
feat(ui): show guardrail usage units and cost on the Guardrails Monitor
2026-09-05 14:41:11 -07:00
ryan-crabbe-berri
b99d8ac38e refactor(ui): keep guardrail usage code under the inline-object-arg lint budget
The staging merge pushed local/no-large-inline-object-arg to 567 against a 554 ceiling, and 15 of those hits came from this branch. useGuardrailsUsageDetail now takes the guardrail id positionally with the date window as its second argument, the usageUnits tests build CounterMath rows through a positional helper, and the overview fixture spreads a base row inside the array instead of calling a factory

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 14:27:08 -07:00
yuneng-jiang
9222a4de2d
Merge pull request #39946 from BerriAI/litellm_e2e_messages_stream_delta_count
test(e2e): stream a longer /v1/messages reply so the delta-count pin has margin
2026-09-05 14:26:55 -07:00
devin-ai-integration[bot]
a46a076b2a
fix(proxy): reject ambiguous name or alias keys in mcp_tool_permissions on write (#39947)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 14:22:26 -07:00
Yassin Kortam
9832d6e4a6
fix(mcp): scan and mask MCP tool call arguments in unified guardrails (#35142)
* fix(mcp): scan and mask MCP tool call arguments in unified guardrails

A guardrail configured with mode pre_mcp_call was handed only a synthetic
tool definition (name plus an empty parameters schema), so it never saw the
argument values it was configured to inspect, and any rewrite it returned was
discarded. Detection could not fire and masking could not take effect, while
the applied-guardrails metadata still reported the guardrail as having run.

Pass every string leaf of the tool call arguments as texts, and fold the
guardrail's rewritten leaves back into modified_arguments, which is the channel
the MCP call path reads to decide what to send upstream. The leaf walk reuses
the json_string_leaves / with_json_string_leaves helpers the tool result path
already uses, so both directions share one bounded traversal.

Two guardrails running concurrently under run_in_parallel scan the same payload
snapshot, so each returns a full replacement derived from the original leaf.
Rewrites of the same leaf to different values are rejected rather than silently
losing one redaction; a leaf that already holds this guardrail's own replacement
is convergent and still masks, which is what the bundled content filter does
when it rewrites the arguments itself as well as through texts.

* fix(mcp): annotate guardrail argument rewrites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): isolate MCP guardrail callback state

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: ratchet LIT010 budget after merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): remove duplicate Bedrock hook parameter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): fail closed when guardrail rewrites cannot be mapped to MCP arguments

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): patch the guardrail translation mappings cache where staging now keeps it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 20:51:25 +00:00
devin-ai-integration[bot]
80839bb33c
feat(proxy): serve Prometheus /metrics from a separate process via --prometheus_metrics_port (#39889)
* feat(proxy): serve Prometheus /metrics from a separate process via --prometheus_metrics_port

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): ruff format prometheus_metrics_server

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): fail fast when the separate metrics server cannot start and force the multiproc dir whenever it is enabled

- wait for the child's /health before starting uvicorn; raise a ClickException if it exits first (port in use)
- create PROMETHEUS_MULTIPROC_DIR whenever --prometheus_metrics_port is set, so DB-configured prometheus callbacks work
- honour lowercase prometheus_multiproc_dir; validate the port before spawning
- cover main() entry point, readiness, bind failure and wildcard-host probing in tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): pin metrics-server readiness to the child pid so another service on the port cannot pass the health check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): probe metrics-server readiness through the shared HTTPHandler instead of bare httpx.get

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): serve only /metrics on the prometheus metrics port

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): validate metrics server CLI args with pydantic instead of typing.cast

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): satisfy metrics server lint gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 13:26:09 -07:00
Yuneng Jiang
d6bc8fe289
test(e2e): ask the streamed /v1/messages pin for a reply long enough to span several deltas
Anthropic now returns the 64-token 'count to 20' reply in one to three content_block_delta events, measured directly against api.anthropic.com and through proxies at 7672399 and 49a1145 alike, so the incrementality assertion (at least two deltas) failed in litellm-e2e builds 125, 130 and the 278 rerun with no proxy change behind it. A 'count to 100' reply at max_tokens 400 arrived in five to fifty deltas across every measured run
2026-09-05 13:23:38 -07:00
ryan-crabbe-berri
0be8bb98b0 Merge branch 'litellm_internal_staging' into litellm_guardrail_usage_cost_ui 2026-09-05 13:09:45 -07:00
devin-ai-integration[bot]
5df0e12e0f
feat(guardrails): add non-blocking flag() verdict to custom code guardrails (#39728)
Custom code guardrails could only allow(), block(reason) or modify(). This adds flag(reason, metadata={}) which lets the request or response through unchanged and records a guardrail_flagged entry carrying the guardrail name, configured mode, evaluated input_type (request or response), reason and structured metadata. The new status is threaded through the request-level guardrail_status aggregation, the Guardrails Monitor rollup (flagged_count), Request Logs (action=flagged, most severe phase wins when a guardrail runs pre and post call) and the Request Logs detail view in the dashboard, which now renders FLAGGED with warning styling instead of falling into FAILED.

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 13:08:03 -07:00
Yuneng Jiang
b56e4f80a4
test(e2e): make each cold cache call a single-assignment helper so its result stays Final 2026-09-05 13:05:03 -07:00
Yuneng Jiang
1c0172b477
style(e2e): annotate the cold-call locals as Final 2026-09-05 13:04:33 -07:00
cursor[bot]
8544faec91
fix(ci): grant pull_requests write for release wheel reporter (#39922)
* fix(ci): grant pull_requests write for release wheel reporter

The reporter posts a PR comment via github.rest.issues.createComment.
GitHub requires both issues=write and pull_requests=write to comment on
a PR issue, as returned in x-accepted-github-permissions. The workflow
had pull-requests: read, so the POST failed with 403 'Resource not
accessible by integration'. Bumping to pull-requests: write fixes the
create path; the read-only pulls.get call still works. Same-repo scope
is preserved by the existing head_repository.full_name check.

Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>

* fix(ci): scope release wheel reporter permissions to pull requests

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
Co-authored-by: Yujong Lee <yujong@berri.ai>
2026-09-05 13:04:31 -07:00
yucheng-berri
948e5755eb
test(e2e): cover presidio post_call, tool_permission, and weave logging cells (#39279)
* test(e2e): cover presidio post_call, tool_permission, and weave logging cells

Five registry cells in Logging & Guardrails had no covering test. Each one now
has a live scenario read back from the real destination:

- guardrail.presidio.post_call.masks: an output-scoped Presidio guardrail
  anonymizes the PII the model repeats back. The prompt also asks for the
  address's local part, which Presidio does not mask, so one response proves the
  model saw the raw address (no pre-call masking) while the address itself comes
  back as <EMAIL_ADDRESS>
- guardrail.tool_permission.pre_call.blocks / .allows: an allow-list of one tool.
  A request declaring an unlisted tool is rejected 400 naming it; a request
  declaring the permitted tool is served and carries
  x-litellm-applied-guardrails, so the allow half cannot pass by the guardrail
  never running
- logging.niche_integrations.success.logs_spend / .failure.logs_spend: a
  key-scoped weave_otel callback delivers to the real Weave project, read back
  through Weave's query API. Success asserts exactly one call whose
  llm.response.cost equals the x-litellm-response-cost header; failure asserts
  one ERROR-status call naming the provider exception and carrying no cost

Logging & Guardrails coverage goes 24/59 to 29/59. No registry rows are added.

* test(e2e): make the tool-permission allow case deterministic and scope the Weave read-back

Review follow-ups on the coverage PR.

- the allow scenario forced the outcome to depend on whether the model felt like
  calling an optional tool, and checked for the tool name as a substring of the
  whole body, which a prose mention would satisfy. It now sends
  tool_choice="required" and asserts the parsed response carries exactly one tool
  call, for the permitted tool
- the Weave read-back queried the newest 200 calls of a shared project and
  filtered client-side, so busy traffic could push the target out of the window
  and read as a delivery failure. The query now scopes server-side to the
  litellm_request op and to calls started after the request, and pages through
  the window with offset
- the reader builds its results as tuples instead of accumulating into lists

Also unblocks the lint gate: `basedpyright tests/e2e` runs only on PRs that touch
tests/e2e, and it has been failing on staging for three FakeItem arguments in
test_junit_properties.py. The stand-in now goes through one typed adapter that
says why, so the gate is green without touching junit_properties.py itself.

* test(e2e): scope the presidio post_call guardrail to email and phone

Running the suite three times in a row caught a real flake: Presidio's broader
recognizers sometimes claim the email's local part as an NRP entity, so the
answer came back as `<NRP>\n<EMAIL_ADDRESS>\n<PHONE_NUMBER>` and the assertion
that the raw local part survives failed. That token is what tells output masking
apart from input masking, so it has to survive.

The post_call guardrail now registers pii_entities_config for EMAIL_ADDRESS and
PHONE_NUMBER only, which is also the narrower thing the scenario means. Verified
against the exact marker that failed, plus two others.

* test(e2e): mark weave logging cells stage red

* test(e2e): use per-test stage red skips for the weave logging cells
2026-09-05 13:03:28 -07:00
Yuneng Jiang
b98f8ee2c5
test(e2e): retry a fresh prefix when Vertex rejects the cache create on its minimum-token check 2026-09-05 13:00:24 -07:00
devin-ai-integration[bot]
c091dd4608
perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS (#39121)
* perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: resolve litellm.proxy submodules lazily so litellm.proxy._types stays importable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: correct SlackAlerting lazy mapping and keep eager encoding path importable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: register module-valued public names as module aliases instead of symbol imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: justify module-alias cache write with rebind-ok

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 19:58:35 +00:00
ryan-crabbe-berri
29b93b57aa feat(ui): show the guardrail cost math in a popover table
The "How is this calculated?" hover was a plain-text tooltip. It is now a
popover (opens on hover or click) with a title, the formula, a table of
one row per counter or guardrail (units, × price, = cost, with unpriced
units called out under the row) and a total row, so the math reads as a
worked sum instead of a sentence.

Refs LIT-5652
2026-09-05 12:58:18 -07:00
Yuneng Jiang
def734923f
test(e2e): prove Vertex context caching on the first cold call and on the spend row 2026-09-05 12:55:30 -07:00
Yassin Kortam
17e13126cc
feat(mcp): warn when an oauth2_id_jag server outruns the SSO provider's assertion capture (#35394)
* feat(mcp): warn when an oauth2_id_jag server outruns the SSO provider's assertion capture

Only the generic OIDC login path captures the IdP id_token that an oauth2_id_jag MCP
server spends as its RFC 8693 subject token. Under Google, Microsoft, SAML or no SSO at
all, registration succeeds and then every ID-JAG credential resolution fails for every
user, with nothing in the logs, the config or the API response to say why.

Report the mismatch from the two places it is knowable: when an oauth2_id_jag server is
created or updated through the management endpoint, and at SSO callback time when a login
hands the arm nothing while such a server is registered. Provider selection mirrors the
callback's precedence, so a generic client id sitting behind GOOGLE_CLIENT_ID does not
clear the warning.

* test(sso): update merged CLI diagnostic patch target

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(mcp): warn about the ID-JAG capture gap for config-declared servers and on the SSO debug page (#39350)

* feat(sso): surface the ID-JAG capture gap on the SSO debug page

/sso/debug/callback is where an operator lands when they are already trying to work out
why ID-JAG is failing, so the reason belongs on it. The annotation appears only when the
active SSO provider captures no identity assertion AND an oauth2_id_jag server is
registered for that gap to break; a deployment without both renders the page it rendered
before, byte for byte. Only the provider name and the remedy are rendered, never a
configured value, and an unreachable MCP table costs the page its annotation rather than
the page itself.

The payload carries the one mutable-ok in this work. Conditionally including a member of a
JSON document has to construct a mapping, and the rejected alternatives are recorded on the
helper so the next reader does not rediscover them.

Held out of the diagnosability PR deliberately: that PR is already reviewed and green, and
this surface ships with the remaining config-load warning as one follow-up.

* feat(mcp): warn at config load when an oauth2_id_jag server outruns the SSO provider's assertion capture

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(sso): trim comments on the ID-JAG debug page diagnostic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): clean up merged imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): satisfy type discipline for diagnostic payload

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(sso): keep the optional ID-JAG payload member on one line for ruff format

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): use Python 3.10-compatible assert_never

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): keep the ID-JAG capture-gap diagnostic out of the unauthenticated debug page

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(sso): inject the retention check and log via caplog so the ID-JAG tests pass the test-quality gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(sso): keep the debug-page outage test on the capture-gap path

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): annotate the retention check type alias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 12:43:09 -07:00
Yassin Kortam
110f654f34
feat(mcp): renew the stored SSO identity assertion behind ID-JAG (#35401)
* feat(mcp): renew the stored SSO identity assertion behind ID-JAG

The oauth2_id_jag arm asserts the id_token captured at the user's last
interactive SSO login, and nothing ever renewed it, so an agent holding a
brokered LiteLLM key could act for that user only until that token's exp.
The assertion already carried the IdP refresh token beside it; this
redeems it.

RefreshingSSOAssertionStore wraps the database reader and satisfies the
same protocol, so the egress arm is unchanged. Renewal is lazy and
single-flighted per user through the same RefreshCoordinator the
authorization_code arm uses, since an IdP that rotates refresh tokens
treats two concurrent redemptions as replay. A refusal leaves the expired
assertion in place so the reader still challenges the user; an
unreachable IdP surfaces as a store outage instead.

* fix(mcp): let a cross-replica loser settle the SSO assertion renewal itself

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): satisfy type discipline lint budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ci): rerun checks after docs main added the missing router setting row

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): answer a cross-replica loser retryable instead of re-electing it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): bypass stale assertion cache during renewal

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: ratchet type-discipline budget after merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 12:43:02 -07:00