Commit graph

47292 commits

Author SHA1 Message Date
Yucheng He
e6c5594b8a feat(otel v2): let a tenant destination export alongside the operator's own
Override stays the default: a key or team destination replaces the operator's
exporter for that backend. Operators running one org-wide backend across every
team set litellm_settings.otel_tenant_destination_mode to additive, and the
same trace lands in both places. A team that names the operator's own project
is still written once, since the fan-out skips a destination the operator's
exporter is already sending that span to.
2026-09-05 17:29:08 -07:00
Yucheng He
3ba9133fe8 fix(otel v2): guard the fan-out's closed state with the lock that gates it
An Event read on its own leaves room for shutdown to run in the gap. A cache
miss then inserted a live exporter into a map that had been cleared, and a
shed processor landed behind sentinels every drain worker had exited on.
The drain pool takes its queue by injection so both interleavings are
reachable from a test without patching.
2026-09-05 17:29:08 -07:00
Yucheng He
c5003c88cd fix(otel v2): retire the drain workers with the fan-out that started them
A proxy that rebuilds its telemetry builds another fan-out, so workers that
outlive the one that started them are two more threads per reload. Shutdown
now retires them once everything queued is closed, and a processor shed
afterwards is closed inline rather than queued to nobody.
2026-09-05 17:29:08 -07:00
Yucheng He
5e98a5d5fc fix(otel v2): close no destination processor under a span still in flight
The fan-out now refuses new work once shutdown starts and waits out the
spans already being forwarded, so teardown neither drops a trace mid-forward
nor hands the next caller an exporter nothing will ever close. The wait is
bounded so a dead collector cannot hold the proxy open.
2026-09-05 17:29:08 -07:00
Yucheng He
3480ec9bef fix(otel v2): give the fan-out its own drain pool instead of a lazy singleton
functools.lru_cache does not hold a lock across the call it caches, so
concurrent first evictions each finish building a queue and start its
workers, and every queue but the winner is abandoned with two daemon
threads blocked on it forever.
2026-09-05 17:29:08 -07:00
Yucheng He
1338433b22 fix(otel v2): drain shed destination processors on daemon workers
A ThreadPoolExecutor joins its workers at interpreter exit, so one unreachable
tenant collector would hold the whole proxy open for its export timeout on the
way down. Two long-lived daemon workers off a queue keep the thread count
bounded without blocking shutdown.
2026-09-05 17:29:08 -07:00
Yucheng He
f1cab32449 fix(otel v2): deliver tenant destinations from the published global provider
Scoping the fan-out by callback name in the previous commit left every backend
that is not the canonical logger with a one-span trace: only the published
global provider sees the FastAPI server span, the auth span and the post-call
database spans, so an arize-only proxy handed a team's Langfuse just the model
call. Attach the fan-out once, to that provider, and let it forward every
destination.

An overridden backend now skips per-request tracer routing outright rather than
only clearing its credential headers, since a key or team otel_service_name was
still enough to detach the model call onto a second provider. The destination
carries that service name as a resource attribute instead.

Shed processors drain on a two-thread pool rather than a thread each, so a
tenant cycling its destination config cannot spawn threads as fast as it sends
requests.
2026-09-05 17:29:08 -07:00
Yucheng He
fb3b32d22d fix(otel v2): scope the fan-out to its own backend and close shed processors off the export path
Three problems in the fan-out, two of them in the eviction added last round:

- Every v2 logger carries its own provider and emits its own copy of a gen-AI
  span, so a proxy running two of them handed the tenant the same model call
  twice. A provider now forwards only destinations for the backend it speaks
  for; the tenant's own backend always has a logger, since naming it in the key
  or team config is what builds one. Reproduced live against a self-hosted
  Langfuse on an arize-only proxy and on the bare `otel` callback.
- Eviction could close a processor another thread was still exporting through,
  which drops that span. Exports are now counted, and a retired processor is
  closed only once its count reaches zero.
- That close ran inside `on_end`, where `shutdown` flushes over the network, so
  one unreachable tenant collector stalled every other tenant's spans. It now
  runs on a short-lived thread, which also retires the retiree cap: a retiree
  drains as soon as its export finishes.
2026-09-05 17:29:08 -07:00
Yucheng He
e9df4458c7 fix(otel v2): keep the destination merge immutable
The per-backend var merge seeded a plain dict and the gated exporter list a plain
list, both of which the LIT budget counts. Wrap the merge in MappingProxyType and
hand the exporters back as a tuple.
2026-09-05 17:29:08 -07:00
Yucheng He
1779fcf4a7 fix(otel v2): read the tenant's stored callback config the way the sibling parser does
Three divergences between the destination resolver and
`convert_key_logging_metadata_to_callback`, which read the same stored config:

- A key whose callbacks are disabled stores an empty list, and `or` treated that
  as "the key configured nothing", so the request inherited the team's
  destination. The sibling parser treats an empty list as configured.
- Two entries naming one backend now merge their `callback_vars` last-wins,
  matching the sibling, instead of the resolver taking the first entry and the
  per-request tracer routing taking the last.
- `credential_gated_exporters` dropped any exporter whose kind had no transport,
  which also dropped an `in_memory` exporter the operator asked for. The
  placeholder is the spec with every field still at its default, so that is what
  the predicate now says.

Arize's `allow_missing_credentials` branch was unreachable: `get_arize_config`
resolves every credential with `os.environ.get` and always supplies an endpoint,
so it never raises. Dropped it and corrected the protocol docstring.
2026-09-05 17:29:08 -07:00
Yucheng He
f5fb73f716 refactor(otel v2): reuse the proxy's own destination allowlist for tenant hosts
A tenant-supplied Langfuse host is the same threat as a URL-valued `model`, so
it now goes through `is_url_destination_allowed_by_host` against
`provider_url_destination_allowed_hosts` instead of a second, DNS-based check
of its own. The DNS lookup would have blocked the asyncio auth path on a
hostname the caller picked, and its cached verdicts could blackhole a real host
after one resolver blip.

Evicting a destination processor now retires it to drain rather than shutting it
down, since `on_end` hands a processor back and exports outside the lock. The
retirees are capped so they cannot accumulate a thread each.

`credential_gated_exporters` tells the synthesized stdout placeholder from a
real exporter by transport rather than by the literal kind `console`, so an
unrecognized kind is not mistaken for a configured collector, and an exporter
the operator did configure survives. That also stops a weave test's env writes
from making this look like a real OTLP exporter later in the same CI worker.
2026-09-05 17:29:08 -07:00
Yucheng He
1f6b80e659 fix(otel v2): validate tenant destinations and match each backend's own endpoint
Review round on the tenant destination routing.

- A key/team Langfuse host is user-supplied input, so it goes through the
  proxy's SSRF guard. A private address is refused, the operator keeps the
  trace, and the warning names user_url_allowed_hosts. The operator's own
  LANGFUSE_HOST is not checked.
- Arize and Weave destinations now resolve their endpoint and transport
  through the backend's own config, so an ARIZE_HTTP_ENDPOINT collector and
  a self-hosted WANDB_HOST are honoured instead of the cloud default.
- A half-configured backend no longer resolves: several dynamic header
  builders gate each credential separately, so an api key with no space id
  produced a non-empty but unusable header set that suppressed the
  operator's exporter.
- A callback_type of "failure" no longer takes over the trace. The
  destination is resolved during auth, before the outcome is known.
- The fan-out cache evicts without shutting the processor down, matching
  ArizePhoenixLogger: a concurrent on_end may still hold it.
- The stdout placeholder is identified by what it does rather than by
  equality with an import-time default, so an operator's OTEL_EXPORTER_OTLP_*
  collector survives the credential-less path.
2026-09-05 17:29:08 -07:00
Yucheng He
bcba86e426 feat(otel v2): send a key's or team's whole trace to its own destination
A key or team that configures its own Langfuse, Arize, Weave or New Relic
credentials used to get a single detached span in its account while the rest of
the request trace stayed on the operator's backend, so neither side held a
complete trace. Resolve the destination during auth, forward every span of the
request to it, and hold the same request back from the operator's exporter for
that backend, so the tenant gets the tree the operator would have seen and the
operator gets nothing for that request.

Also let a credential-mandatory preset build without the operator's own env
credentials. Without that, a proxy whose teams each bring their own account fell
back to the legacy integration and never ran a line of the v2 path.
2026-09-05 17:29:07 -07:00
ryan-crabbe-berri
a9f8a8d794
Merge pull request #39978 from BerriAI/litellm_remove_migrated_pages_shim
refactor(ui): route the sidebar by pathname and shrink the ?page= shim to a redirect table
2026-09-05 17:17:34 -07:00
yucheng-berri
6e05ac5d97
feat(guardrails): add inspect_embeddings toggle for AIM and Cato (#39918)
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks

`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:

- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
  list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
  redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.

Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.

* feat(guardrails): add inspect_embeddings toggle for AIM and Cato

* fix(guardrails): redact batched embedding input on anonymize

A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.

* fix(guardrails): reject partial embedding redactions

* fix(guardrails): avoid unnecessary batch type check

* style(tests): drop trailing blank line in cato guardrail tests

* fix(guardrails): reject malformed batch redactions

* fix(guardrails): reject malformed batch redactions

* fix(guardrails): reject aim redactions with no text content

The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.

* fix(guardrails): validate all aim redaction paths

Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.

* fix(guardrails): preserve aim output redaction alignment

AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.

* test(guardrails): cover aim output anonymize alignment and malformed redactions

---------

Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
2026-09-05 17:15:46 -07:00
yucheng-berri
d515a285b1
fix(azure_sentinel): split batches under the 1MB ingestion cap (#39880)
* fix(azure_sentinel): split batches under the 1MB ingestion cap and keep undelivered records queued

Azure Monitor rejects any Logs Ingestion body over 1MB with a 413. The Sentinel logger
posted the whole queue as one body and cleared it in a finally block, so an oversize
batch, a transient 5xx, or a failed token call dropped every queued record, and records
logged while a send was in flight were cleared with it. Both the standard and the audit
queue share the sender.

Move Datadog's proactive size split and 413 halving into a shared helper,
litellm/integrations/batch_utils.send_batch_with_413_split, and route Sentinel through it
with a 1MB size check. A lone record that still 413s is dropped, everything a transient
failure leaves undelivered goes back to the front of its queue, and the retry queue is
capped at max_queue_size so an unreachable workspace cannot grow memory without bound

* fix(azure_sentinel): retry undelivered records on the flush timer only

Requeued records made every later event cross the batch_size threshold, so a
down ingestion endpoint got one full-queue resend per request. Threshold sends
now go through flush_queue, so they take the flush lock instead of racing the
timer, and they stand down while records are awaiting retry.

A record that cannot be serialized raised out of the size probe and killed the
periodic flush task. The probe now runs inside the failure handling, so the
batch is split and only the record that cannot be serialized is dropped.

* fix(azure_sentinel): decide threshold sends under the flush lock

Concurrent callbacks all read logs_awaiting_retry before the first send
finished, so each one resent the whole queue once that send failed. The
flag and the batch_size threshold are now rechecked while holding the
flush lock, and each queue sends only itself instead of going through
flush_queue, which was retrying the other queue too.

* test(azure_sentinel): cover successful threshold waiters

* fix(azure_sentinel): preserve cancelled batches for retry

* fix(azure_sentinel): requeue only the undelivered part of a cancelled split

A batch over the ingestion cap goes out in pieces, so a cancellation partway
through requeued pieces the destination had already accepted and sent them a
second time on the next flush

The split helper now raises a cancellation carrying the records it never
delivered, and Azure Sentinel requeues those instead of the whole batch

* fix(azure_sentinel): drop batches a permanent rejection will never accept

A non-413 4xx from the ingestion endpoint or from the OAuth token call means the request
will fail the same way on every retry, so requeueing it held the batch, and every record
logged behind it, until the queue cap dropped them. Retryable statuses (5xx, 408, 429)
still keep the whole batch, and a shared classifier gives Datadog the same rule

The serialization probe now catches any exception, not just TypeError and ValueError,
because safe_dumps hands pydantic models to model_dump and can raise anything. It also
splits on record count, so a recovery flush sends batch_size records per request instead
of serializing the whole requeued queue to measure it

Both integrations re-raise a cancelled send as exactly asyncio.CancelledError. Python
3.12's asyncio.wait_for only translates the exact class into TimeoutError, so the
BatchSendCancelled subclass escaped the logging worker as an unhandled error

The awaiting-retry flag now follows the queue that survived the max_queue_size trim, so
a deployment with the cap at zero is not left waiting for a timer flush with nothing
queued to retry

* chore(logging): document mutable queue ownership

Annotate the queue detach and requeue constructions required by the logger's appendable queue contract so the type-discipline budget stays clean

* fix(datadog): preserve non-413 retry behavior

Keep Datadog's existing contract of requeuing every non-413 HTTP failure while Azure Sentinel applies its permanent-client-error policy through the shared splitter

* fix(batch_utils): requeue by default and let Sentinel opt into dropping

The shared splitter's default non-success handler is now requeue_after_http_error, the behavior Datadog had before the extraction, so a caller that omits the argument keeps its records. Azure Sentinel passes undelivered_after_http_error explicitly to drop permanent 4xx rejections

Also drops an explicit return None the strict ruff gate flags in the test helper
2026-09-05 17:15:36 -07:00
Mateo Wang
56a61cf016
Merge pull request #39764 from BerriAI/litellm_govcloud_profiles_lit6421
feat(pricing): add GovCloud pricing for every live but unpriced Bedrock model
2026-09-05 17:15:22 -07:00
ryan-crabbe-berri
e1fb8affe3
Merge pull request #36841 from BerriAI/litellm_lite_pi
feat(cli): add lite pi to run the pi coding agent through the proxy
2026-09-05 17:14:11 -07:00
ryan-crabbe-berri
0c45e28dbd test(ui): query sidebar links by role so the testing-library budgets stay under their ceilings 2026-09-05 17:07:27 -07:00
Mateo Wang
0aa346cba5
Merge pull request #39972 from BerriAI/litellm_lit_7027_emulated_file_search_scope
fix(file_search): scope emulated file_search to the request's vector stores
2026-09-05 17:07:04 -07:00
ryan-crabbe-berri
2dfa14648a refactor(ui): drop comments that restate the redirect table and home route 2026-09-05 16:50:20 -07:00
Mateo Wang
02cbff4918
Merge pull request #39964 from BerriAI/litellm_lit_7050_redact_failure_traceback
fix(proxy): redact provider keys from pass-through failure tracebacks
2026-09-05 16:48:04 -07:00
mateo-berri
8bf03c10fd fix(file_search): escape the dropped vector_store_id in the warning
Format the model-picked id with %r so control characters in it cannot
break the log line. The regression test for the unlisted id keeps to
generic scoping wording
2026-09-05 16:44:54 -07:00
ryan-crabbe-berri
1258d84221 refactor(ui): route the sidebar by pathname and shrink the ?page= shim to a redirect table
The sidebar and header were still keyed on legacy ?page= ids and mapped
back and forth through MIGRATED_PAGES, legacyPageHref and
legacyKeyForPathname. Leaves are now plain Next links to their path
route, the active item and breadcrumb come from usePathname, and the
setPage/defaultSelectedKey prop chain is gone.

The id-to-route table moves next to the dashboard root page as its only
consumer. That redirect now forwards the remaining query params instead
of dropping them, so deep links such as the proxy's MCP env-var setup
link (?page=mcp-servers&fill_env_vars=) no longer rely on the target page
reading the pre-redirect URL during its first render. The proxy builds
that link as /ui/mcp-servers?fill_env_vars= directly, and the Playground
warnings link to the real routes instead of relative ?page= URLs.

migratedHref is renamed uiHref, the /ui base-path helper it always was.
2026-09-05 16:44:11 -07:00
yuneng-jiang
bde58f9b41
Merge pull request #39959 from BerriAI/litellm_/release-ui-build-e256a3
chore: rebuild Admin UI bundle for the next release
2026-09-05 16:40:13 -07:00
mateo-berri
55a6132b31 fix(file_search): log when a model-picked vector_store_id is dropped
Emulated file_search now warns when the model returns a vector_store_id that is
not one of the request's stores, naming the dropped id and the stores that were
searched instead. H16 asserts the warning is emitted exactly once.
2026-09-05 16:31:24 -07:00
Yuneng Jiang
0e118342ac
chore: update Next.js build artifacts (2026-09-05 23:25 UTC, node v24.19.0) 2026-09-05 16:25:57 -07:00
yuneng-jiang
b1e2f5bc0b
Merge pull request #39969 from BerriAI/revert-39121-litellm_lazy_sdk_import
revert: perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS (#39121)
2026-09-05 16:22:55 -07:00
devin-ai-integration[bot]
9a5564f00e
fix(ui): read Usage Total Requests tile from gateway request counts (#39963) 2026-09-05 16:20:56 -07:00
mateo-berri
27c55a21d9 test(proxy): type the traceback-recording hook to match CustomLogger
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
2026-09-05 16:19:29 -07:00
Yuneng Jiang
eee162b317
fix(lazy_imports): type import_map as Mapping to stay under the LIT001 budget
The revert restored the dict annotation that #39121 had loosened to Mapping,
and the LIT001 ceiling has been ratcheted down since, so the gate rejected the
one reintroduced hit. Annotation only, no behavior change.
2026-09-05 16:12:45 -07:00
mateo-berri
942e6cb3cd fix(file_search): scope emulated file_search to the request's vector stores
The emulated file_search handler searched whatever vector_store_id the
model returned, so a model steered to an id outside the request's
file_search tool reached a store the per-key vector store permission
check never saw. An id outside the request's stores now falls back to
those stores; an id that is one of them still narrows the search to it.
2026-09-05 16:12:43 -07:00
ryan-crabbe-berri
587311d0e4
Merge pull request #39968 from BerriAI/litellm_remove_dead_model_hub_public_dialog
fix(ui): remove unreachable AI Hub dialog that put the session key in a URL
2026-09-05 16:10:15 -07:00
yuneng-jiang
45cf1a7ef1
Revert "perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS (…"
This reverts commit c091dd4608.
2026-09-05 16:07:09 -07:00
ryan-crabbe-berri
79a5614c18 fix(ui): remove unreachable AI Hub dialog that put the session key in a URL
The Public Model Hub dialog in ModelHubTable was never opened (its open setter had no callers), but its See Page button navigated to /model_hub_table?key=<session key>. Delete the dialog, its state, the handler and the unused router import so the path cannot be revived.
2026-09-05 16:04:50 -07:00
yuneng-jiang
1b25132863
Merge pull request #39953 from BerriAI/litellm_/litellm-e2e-flaky-test-2159ae
test(e2e): judge /v1/messages streaming on the clock, not on the provider's delta count
2026-09-05 16:04:45 -07:00
mateo-berri
4ec5a6761c fix(proxy): redact provider keys from pass-through failure tracebacks
A failed pass-through call logged the httpx traceback, whose message
quotes the upstream URL with the provider API key in its query string,
into the spend log's error information and into every failure callback.
The error information built for logging now redacts its traceback and
error message, and the traceback is redacted once before the failure
callbacks receive it.
2026-09-05 15:58:33 -07:00
ryan-crabbe-berri
a0c9095cfc
Merge pull request #39960 from BerriAI/litellm_cache_hit_guardrail_spend
fix(proxy): keep guardrail cost in spend on cache hits
2026-09-05 15:53:34 -07:00
ryan-crabbe-berri
63156a7bd6 test(proxy): explain the proxy_server patches in the cache-hit regression test
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 15:44:37 -07:00
ryan-crabbe-berri
acddd21860 fix(proxy): keep guardrail cost in spend on cache hits
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 15:32:02 -07:00
Mateo Wang
bf51dea36b
Merge pull request #39862 from BerriAI/litellm_lit_6992_cohere_parse
feat(ocr): add Cohere Parse support for cohere and azure_ai
2026-09-05 15:16:16 -07:00
yuneng-jiang
6a4fb2bbe8
Merge pull request #39938 from BerriAI/litellm_e2e_vertex_cache_first_call
test(e2e): prove Vertex context caching on the first cold call and on the spend row
2026-09-05 15:10:15 -07:00
Yuneng Jiang
cd976624d1
test(e2e): drop the explanatory sentence from the StreamingResponse docstring 2026-09-05 14:53:12 -07:00
Yuneng Jiang
b55a4317a6
test(e2e): annotate new stream-timing locals as Final and trim the docstrings 2026-09-05 14:49:56 -07:00
Yassin Kortam
0cb759772c
fix(ui): show indirectly granted and name-keyed MCP servers in the tool matrix (#35154)
* fix(ui): show indirectly granted and name-keyed MCP servers in the tool matrix

The MCP tool permission editor was fed the direct server list only, so a server a
principal reaches through an access group or a toolset never appeared in the matrix.
That single blind spot produced two opposite bugs depending on how a save handler
filtered mcp_tool_permissions: filtering by the selected servers deletes an indirect
server's allowlist, and because a missing entry means "no restriction from this
level", the principal silently gains every tool on it; not filtering leaves a stale
entry that keeps a removed access group's server reachable, since a server named
under mcp_tool_permissions is entitled on purpose.

The editor now resolves the selected access groups and toolsets to their servers and
renders them alongside the direct ones, badged with where the grant comes from, so an
admin can see and clear an inherited server's tools like any other. Resolution reuses
the data the selector already loads: access groups resolve from each server's
mcp_access_groups, toolsets from the toolset's own tool list. When that data cannot be
loaded the editor says so instead of rendering an empty list, because an absent
inherited server reads as "there are none". Servers named only by an
mcp_tool_permissions key are listed too, which is what makes a leftover entry
visible; the opt-out sentinel still renders nothing, since it short-circuits the
backend resolver to zero servers.

Opening the editor no longer applies the delete-blocked-by-default allowlist to an
inherited server. Writing an entry for one would narrow a grant the admin never
touched just by opening the form; direct servers keep that default.

Both components also matched on server_id alone, while the backend accepts a server
id, name or alias interchangeably. A grant or allowlist written by API or config with
a name rendered as a selected server with no tools under it, which reads as "this
server has no tools". Matching now covers all three identifiers, and an edit writes
back to the key the entry already uses rather than forking a second id-keyed entry.

The same mismatch could also put one server under several keys at once, its id and
its name for instance. The backend unions every key's list, so reading one key
understated what was in force and writing one key left the others granting. The
resolver now reports, per server, the key an edit keeps, the equivalent keys it
supersedes, and the union those keys allow; the card renders the union and every
write goes through one function that writes the kept key and drops the superseded
ones. A key that also names a DIFFERENT server, which happens when two servers share
a name, is never dropped, because dropping it would strip the neighbouring server's
restriction; the card names such a key and says its tools stay allowed until the
servers no longer share the name, so an admin is told rather than left to infer it
from an edit that bounces back.

A third divergence from the backend sat in the same matching. The backend resolves an
identifier with exact-id precedence: a string that is a registry server id names that
server and stops, and only a string that is no server's id falls back to name and alias,
which can name several. Matching all three fields at once meant a server merely named
after another server's id joined the matrix as if it had been selected, and because it
landed there as a directly selected server it also received the delete-blocked default
write on open. Since an mcp_tool_permissions key is itself a grant source, saving then
handed out a server nobody granted, with no admin gesture involved. Identifier
resolution now mirrors the backend's precedence, and a key is read as this server's only
when it resolves back to it, so an entry that belongs to the id's owner is neither read
into this server's allowlist nor overwritten by an edit made against it.

A toolset grant was also invisible to the tool matrix. The backend unions a toolset's
tools with whatever mcp_tool_permissions allows, so a toolset-only grant restricts the
server to that toolset's tools; the editor read the map alone, found no entry and
rendered every tool on the server as allowed. Deselecting one from that state wrote all
the others as a permission entry, and the union turned a revocation into a grant of
every tool the toolset never included. The resolved entry now carries the toolset's
tools, so the matrix opens on what is actually in force, the delete-blocked default is
withheld from a server a toolset restricts, and a write keeps out the tools only the
toolset accounts for so a grant that ends with the toolset does not become a standing
one. Those tools cannot be revoked from this screen at all, since the backend unions
them in; they render allowed and locked and the card says which of them a toolset holds
open and where to go to revoke them.

That guard originally covered only the keys an edit supersedes, on the assumption that
the key it keeps names one server. It does not when a shared key is a server's only
entry: it then becomes the key an edit writes, and writing it moves the other server's
allowlist too, which is the widening the guard exists to prevent. The key an edit writes
is now the first one naming this server and no other, falling back to the server's own
id, so a shared key is never written through and an edit against one card cannot reach
the server behind the other. Both cards say the shared key holds tools open, since
neither can revoke them.

No owner's save handler changes here. With the full effective set now available to
the editor, the key and team handlers can filter against it instead of guessing,
which makes the internal-user surface's unfiltered save redundant

Resolves LIT-4963
Resolves LIT-4958

* chore: drop tsbuildinfo churn from merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): satisfy dashboard lint budgets

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep MCP tool allowlists for indirect grants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep standing MCP grants on team save

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(ui): format TeamInfo and hoist inline object args

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep MCP tool allowlists for team servers granted indirectly (#35153)

* fix(ui): filter team MCP tool allowlists against the effective server set

Saving a team filtered mcp_tool_permissions down to the directly selected servers. A server reached
through an access group or a toolset is never in that list, so any save dropped its entry, including
a save that only changed the team alias. Because the resolver unions tool-permission keys into the
entitled server set and treats a missing entry as "no restriction from this level", the team kept
the server and lost the tool allowlist on it

Filtering on the direct list alone cannot get this right in either direction. Keeping every entry a
level did not directly select leaves a removed access group's server reachable through its own stale
entry, which breaks revocation. Dropping on deselection alone widens a server that an access group
still supplies

The save handler now resolves the effective server set with resolveEffectiveMcpServers and keeps an
entry only when something other than the entry itself still grants that server: a direct selection, a
selected access group, or a selected toolset. Unified access group ids are added when that selection
is untouched, since the loaded server list is then still accurate

When the server or toolset list cannot be resolved, every entry is kept and the admin is told the
allowlists were saved unchanged. Pruning on incomplete knowledge is the direction that silently
widens, so it only happens when the editor can show the server became unreachable. A failed lookup
and a changed access group selection are separate cases in a tagged union, so the notice names what
actually happened instead of describing the intentional one as a failure, and both hooks gate the
filter symmetrically so a save fired before toolsets settle cannot resolve against an empty toolset
list

Resolves LIT-4961

* fix(ui): resolve team MCP grants from access group metadata and refuse unsafe saves

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): resolve team access group grants from team info when the access group list is role-gated

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): match every selected access group by id instead of by count

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): reload team access group grants at save time

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep frontend lint budget within limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): cover a standing allowlist no group grant covers at load or save

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(ui): keep MCP grant inputs in named variables for the lint budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): guard MCP default write on toolset load, keep create toolsets, fix flat view

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 14:49:30 -07:00
Yuneng Jiang
d56affa814
test(e2e): judge /v1/messages streaming on the clock, not on the provider's delta count
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.

The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.

Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
2026-09-05 14:44:21 -07:00
ryan-crabbe-berri
1745d74293
Merge pull request #39853 from BerriAI/litellm_guardrail_usage_cost_ui
feat(ui): show guardrail usage units and cost on the Guardrails Monitor
2026-09-05 14:41:11 -07:00
ryan-crabbe-berri
b99d8ac38e refactor(ui): keep guardrail usage code under the inline-object-arg lint budget
The staging merge pushed local/no-large-inline-object-arg to 567 against a 554 ceiling, and 15 of those hits came from this branch. useGuardrailsUsageDetail now takes the guardrail id positionally with the date window as its second argument, the usageUnits tests build CounterMath rows through a positional helper, and the overview fixture spreads a base row inside the array instead of calling a factory

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 14:27:08 -07:00
yuneng-jiang
9222a4de2d
Merge pull request #39946 from BerriAI/litellm_e2e_messages_stream_delta_count
test(e2e): stream a longer /v1/messages reply so the delta-count pin has margin
2026-09-05 14:26:55 -07:00
devin-ai-integration[bot]
a46a076b2a
fix(proxy): reject ambiguous name or alias keys in mcp_tool_permissions on write (#39947)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 14:22:26 -07:00