The End User filter listed every row of LiteLLM_EndUserTable, which is both
unscoped and the wrong source. Team admins and internal users can open the
Logs page, and their log view is already restricted to their own requests
plus the teams they administer, but the filter dropdown offered them every
end user on the proxy.
Team attribution only exists on spend logs, so /customer/aliases now reads
LiteLLM_SpendLogs and applies the same scoping /spend/logs/ui does: a proxy
admin sees the whole window, everyone else sees ("user" = caller OR team_id
IN permitted_teams), reusing _get_permitted_team_ids_for_spend_logs so the
two paths cannot drift. A caller with neither matches FALSE rather than
falling through to unscoped, and a failed team lookup degrades to
own-rows-only.
Querying spend logs safely is the other half. start_date/end_date are now
required, so the query always has the indexed startTime bound, and the
inner scan is capped at MAX_SPENDLOG_ROWS_TO_SCAN_FOR_FILTERS rows ordered
by startTime DESC. DISTINCT therefore runs over a bounded row set instead
of the whole table the way /global/all_end_users does.
Also adds /customer/aliases to spend_tracking_routes. Without it RouteChecks
rejects INTERNAL_USER and INTERNAL_USER_VIEW_ONLY before the handler runs,
which would have made the scoping above dead code; a test pins the route to
the same access tier as /spend/logs/ui.
The dropdown now shows the end users present in the window the table is
showing, so the filter list matches what it filters. formatLogsWindow is
shared with the logs query so the two windows cannot diverge.
The guardrail-information writer picked its metadata bucket with a hand-rolled
precedence that preferred a caller-supplied `metadata` field, while every reader
resolves the bucket through `get_metadata_variable_name_from_kwargs`, which
prefers `litellm_metadata`. The two rules agree only when the caller sends no
`metadata` of its own. Routes in `LITELLM_METADATA_ROUTES` seed `litellm_metadata`,
so on /v1/messages and /v1/responses a caller that sends `metadata` sent the entry
to a dict nothing reads; the spend log then reported `guardrail_status: not_run`
with no `guardrail_information` even though the guardrail ran and the
`x-litellm-applied-guardrails` header was present.
Give the resolver one owner. `get_or_create_metadata_bucket` moves from the proxy
layer into core_helpers next to the resolver it calls, so `litellm/integrations`
can reach it without a proxy dependency, and the byte-identical duplicate of
`get_metadata_variable_name_from_kwargs` in callback_utils is deleted. The writer
now shares that owner with `add_guardrail_to_applied_guardrails_header`, so the
response header and the spend log can no longer disagree.
Two readers had to move with it or the fix would be a no-op on the affected
routes. `_sync_guardrail_info_to_logging_obj`, which bridges request_data into the
spend-log payload for passthrough routes, picked the first truthy bucket, so a
non-empty caller `metadata` short-circuited it. The otel failure-path span reader
`_emit_guardrail_spans_from_request_data` read a hard-coded `metadata` key, which
also dropped the span whenever the entry lived in `litellm_metadata`.
Model Armor already resolved the bucket for its file-scan results but wrote its
text-scan and post-call results, and read them back in `_process_response`,
through a hard-coded `metadata` key; on a seeded route that split the record so a
file scan's evidence never reached the logger. All four Model Armor sites now use
the shared resolver. The unified guardrail hook seeds `litellm_metadata` on every
route, so the OpenAI moderation entry lands there too; spend-log output is
unchanged because `merge_litellm_metadata` reads both buckets.
Enterprise MCP users mint virtual keys against tool access groups rather than
explicit server ids. Nothing exercised that end to end.
Registers the upstream MCP server tagged with a server-side access group
(mcp_access_groups), then mints one key granted that group and one granted a
different group. Asserts the granted key sees the tagged server's tools on
tools/list and the other key sees none, so access-group scoping can't leak the
server across the boundary.
Adds mcp_access_groups support to the e2e MCP client (server registration, key
generation, ObjectPermission) and the registry cell
mcp.list_tools.api_key.access_group_scoped.
* feat(ui): migrate the Create Organization form to shadcn and react-hook-form
* fix(ui): guard double submit, test escape-close reset, drop dead organizationCreateCall
* fix(ui): block org create dialog dismissal while a create is in flight
* fix(ui): render budget duration labels instead of raw values in the shadcn Select
TableHeaderSortDropdown had no importers left; the shared DataTable's
DataTableSortHeader covers the same ascending/descending/reset menu on Base UI.
knip did not flag it because its own test file counted as a usage.
* fix(ui): bind key duration input to one Form.Item so pre-filled expiry submits
The Create and Edit key forms kept the displayed expiry in a Tremor TextInput's
local state while the value actually submitted lived in a separate hidden antd
Form.Item. After the first create, form.resetFields() cleared the hidden field
but not the local state, so a second create showed a stale "1d" that was never
sent unless the user deleted and retyped it
Wrap the visible input in a real Form.Item (name="duration") inside
KeyLifecycleSettings and drop both hidden mirror fields plus the local
durationValue state, so what is displayed is always what is submitted. The
Regenerate key flow already used this pattern
* test(ui): restore custom rotation interval coverage in real-form harness
The KeyLifecycleSettings test rewrite dropped the custom interval branch:
selecting Custom interval, typing a value, propagation to the parent, and
hiding the input when switching back to a predefined interval. Cover it in
the real antd Form harness, asserting the parent-held rotationInterval state
instead of a mocked callback
Rebuilds the Router Settings > Routing Groups table on the shared DataTable
and cell library, the last antd entity grid in the dashboard.
The table splits into a thin RoutingGroupsTable container plus
RoutingGroupsTableColumns, with the usage snippets moving to their own
RoutingGroupUsagePanel on ui/tabs and the shared CodeBlock instead of antd
Tabs and Paragraph copyable. Models render through the shared ModelsCell so
long lists collapse behind "+N more" rather than wrapping the row, and the
two inline icon buttons become a single overflow menu with Edit and Delete.
antd gave the snippet panel its own chevron column; under the shared pattern
a row has two click targets, the name cell and the overflow menu, so clicking
the group name now opens the panel. Column set, order, actions, and the
backend row order are otherwise unchanged.
Tremor's TabPanels hands each child an index via React.Children.map, while
the selected index comes from HeadlessUI counting only real Tab elements.
An empty fragment, false, or null still consumes a panel index but
contributes no tab, so the team-only Agent Activity conditional made the
two lists drift for every non-team entity type: Key Activity resolved to
the empty slot and rendered nothing at all, and Endpoint Activity rendered
the key metrics
Drive both lists from a single tab array so adding or removing a
conditional tab touches one place and the indices cannot diverge
The dashboard declared the server-owned object_permission shape by hand
in five places, each with a different subset of fields and none matching
the OpenAPI schema. That is what hid LIT-4766: KeyResponse.object_permission
never declared mcp_toolsets, so a form that wrote the field without reading
it compiled cleanly and silently wiped the grant
Replace four of those copies with one alias over the generated
LiteLLM_ObjectPermissionTable. The agent shape stays separate because the
agent endpoint really does return a narrower type, so it points at its own
generated AgentObjectPermission
The http_handler pair only received its full mutation verdict after the
first removal batch landed; these five tests pass unchanged when every
function they execute is mutated and the owning file killed none of
their scored mutants. The ssl tests excluded from mutation scoring are
untouched.
Opening Logs > Filters fetched the entire customer table through
/customer/list, which is an unbounded find_many that eagerly loads the
budget and object-permission relations for every row. On a proxy with
61k customers that is a 20 MB, 7.6 s response; the dropdown then built an
option per row and rendered all of them, since the combobox does not
virtualize. The result was a multi-second freeze every time the drawer
opened.
Adds GET /customer/aliases, a projection of user_id alone with page/size/
search, mirroring /key/aliases. The End User field now uses
PaginatedSearchSelect behind an infinite query, the same shape the Key
Alias and Model filters already use, so it fetches 50 rows at a time and
pushes the typed query to the server.
The response reports has_more instead of a total count. A total needs
COUNT(*) over the whole match set on every keystroke, which is the cost
this endpoint exists to avoid; ordering by the user_id primary key and
fetching one row past the page lets Postgres stop early and still tells
the client whether to request more.
LIKE metacharacters in the search term are escaped, because end-user ids
routinely contain underscores and an unescaped one silently widens the
match.
Drops the now-unused accessToken prop threaded from RequestLogsPanel
through RequestLogsTable into the filters.
Under otel_v2 a single MCP tool call surfaced in APM as two disconnected
traces joined only by a span link: the HTTP transport transaction
POST /{mcp_server_name}/mcp and the tools/call span carrying
error.type=MCPToolResultError. resolve_mcp_span_context parented the MCP
span to the W3C trace context the client propagates in params._meta
(SEP-414) and recorded the transport as a link, so with no traceparent
propagated (the common case today, including MCP Inspector) the span
started its own root trace.
Nest the MCP span under the transport span when nothing is propagated, so
the call stays in one trace; the propagated-context path is unchanged and
still parents to the remote context and links the transport per the OTel
GenAI MCP semconv.
The transport has to be resolved per message rather than read from the
request-root ContextVar. A stateful streamable-HTTP session runs every
message on the single task the session's initialize POST spawned, so that
ContextVar is frozen at initialize inside the handler: live capture on
staging showed the tools/call span linking the initialize POST rather than
the POST that carried it, and nesting on that anchor would hang every tool
call of a session off the first request's already-ended span. The gateway
now resolves the current request's span on the ASGI task and carries it to
the handler on the authenticated-user object, the same way per-request auth
already crosses that boundary.
* fix: handle explicit outputInfo: null in Vertex AI batch response
Vertex AI can return HTTP 200 for a create_batch/get_batch call with an
explicit "outputInfo": null body (the output directory is assigned
asynchronously and may not be populated yet at response time).
_get_output_file_id_from_vertex_ai_batch_response did:
response.get("outputInfo", OutputInfo()).get("gcsOutputDirectory", "")
dict.get(key, default) only substitutes default when the key is absent,
not when it is present but explicitly None, so this crashed with:
AttributeError: 'NoneType' object has no attribute 'get'
surfaced to callers as an opaque openai.InternalServerError 500 from
litellm.create_batch()/retrieve_batch() for any Vertex AI batch job,
regardless of whether the job ultimately succeeds.
Fixed by guarding with `response.get("outputInfo") or OutputInfo()`,
matching the existing null-safe pattern already used by the sibling
_get_input_file_id_from_vertex_ai_batch_response for inputConfig. The
existing outputConfig fallback branch (a few lines below) already
handles this case correctly once it's reachable - it just never was.
Added 2 regression tests covering outputInfo: null with and without an
outputConfig fallback available.
* test: drop explanatory comment from regression test
---------
Co-authored-by: htourinho-clgx <htourinho@cotality.com>
The gateway acts as an MCP client toward upstream MCP servers, and the MCP
authorization spec requires an MCP client to send the RFC 8707 resource
parameter on both the authorization request and every token request. The
gateway sent it on none of its upstream OAuth legs, so an authorization server
that requires resource indicators rejected the exchange with invalid_target
with no way to configure around it.
Authorization servers disagree irreconcilably and nothing advertises which
camp they are in, so this is a per-server opt-in rather than a default: most
providers ignore the parameter, some hard-reject it and carry audience in
scopes instead, and strict or MCP-native ones refuse to mint a correctly
scoped token without it. The new upstream_resource setting is unset by
default, which keeps today's requests byte-identical.
Both outbound OAuth stacks resolve the value from the server exactly once and
carry it structurally rather than attaching it per call site. In v1 every
plain-OAuth2 token leg builds its body through one helper that resolves the
resource in the same call as the mandatory client authentication; in v2 the
adapter, the single place an MCPServer becomes an outbound config, resolves it
onto the client_credentials config that the HTTP/SSE M2M path uses, and it
joins the config's mint identity so retargeting a live server refreshes the
token rather than serving the previous audience's. A leg cannot authenticate
without also naming the resource its sibling legs named, which is what an
attach-per-call-site approach kept getting wrong.
The setting is non-secret admin config sharing a blob with real secrets, and
the backend classifies which key is which rather than nulling the blob
wholesale or gating on its truthiness: redaction returns admin config to an
admin, session inheritance ignores it when deciding whether a real credential
was supplied and carries it onto the derived server, and the edit form renders
the same shared OAuth component as create so the field exists on both, an
emptied field submitting an explicit null that the credential merge drops.
Router.acompletion() takes messages positionally, so splatting a body that omits it raised a TypeError that the generic handler mapped to a 500. Validate the required body param at the routing boundary and raise the existing 400 contract instead.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails
Pre-call guardrails run sequentially because each may mutate the request
payload and later guardrails depend on earlier mutations. Deployments with
several slow block-only pre_call guardrails (external moderation, Bedrock,
LLM-judge) therefore pay the sum of their latencies. during_call guardrails
run concurrently but alongside the LLM call, so a violating payload has
already been sent, which is unacceptable when the request must never reach
the model.
This adds a per-guardrail run_in_parallel flag (default off). Guardrails that
opt in are pulled out of the sequential loop and run concurrently via
asyncio.gather after every sequential (payload-mutating) guardrail has run, so
they observe the mutated payload and still form a hard barrier before the LLM
call; the first to raise blocks the request. Their returned data is discarded
since they are declared block-only.
The flag is wired from LitellmParams onto the guardrail instance at the same
generic choke point in initialize_guardrail that already sets
skip_system_message_in_guardrail, so no per-provider initializer needs to
change.
* feat(guardrails): extend run_in_parallel opt-in to post_call guardrails
post_call_success_hook ran guardrails sequentially for the same reason
pre_call did: response-modifying guardrails thread the response forward. But
block-only output scanners (which read the response and reject on violation
without changing it) serialize for no benefit and add latency.
This reuses the existing run_in_parallel flag for the post_call hook. Opted-in
post_call guardrails are pulled out of the sequential loop and run concurrently
via asyncio.gather after the sequential (response-modifying) guardrails and
before the non-guardrail CustomLogger callbacks, so they inspect the final
response and still block it from reaching the client if any raises. Their
returned response is discarded since they are block-only.
The apply_guardrail path sets data["guardrail_to_apply"] immediately before
awaiting, and unified_guardrail pops it before its first suspension point, so
concurrent guardrails never race on that key under asyncio's cooperative
scheduling.
* fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes
Addresses review feedback on the run_in_parallel opt-in.
asyncio.gather propagated the first exception without cancelling or awaiting
the siblings, so a block at t=0 left the other guardrails running as
unobserved background tasks (wasted external calls plus event-loop warnings),
and a fast SensitiveDataRouteException/ModifyResponseException could return a
reroute or passthrough before a slower block finished, letting crafted input
bypass the block. Both the pre_call and post_call parallel batches now gather
with return_exceptions=True so every guardrail runs to completion, then raise
any blocking exception ahead of a flow-changing one.
The registry choke point wrote bool(None)==False onto every instance when the
config omitted run_in_parallel, silently disabling a constructor-set default;
it now only writes when the config provides an explicit value.
* fix(guardrails): record lifecycle logs for every concurrently-run guardrail
The log_guardrail_information decorator skipped its auto-record when it saw
that the count of standard_logging_guardrail_information entries in the shared
request_data had grown during the wrapped call, taking that as proof the
wrapped function had recorded its own richer entry. That heuristic breaks the
moment guardrails run concurrently (parallel pre_call/post_call, during_call):
a sibling guardrail's append inflates the shared count, so a guardrail that did
not self-record wrongly concludes it already did and drops its own entry. The
result is that enabling run_in_parallel silently loses per-guardrail lifecycle
logs, so the Admin UI Request Lifecycle timeline and downstream loggers
(Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent
guardrails.
Replace the shared-count heuristic with a ContextVar flag set when a guardrail
records its own entry. asyncio copies the context into each gathered task, so
the flag is isolated per concurrent guardrail while still catching the
self-record-then-skip-auto-record case within a single invocation.
* test(guardrails): declare run_in_parallel on post_call guardrail mocks
The post_call partition reads run_in_parallel on every CustomGuardrail
callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is
set in __init__, not on the class) so the attribute access raised, and even
a class-level default would return a truthy child mock that wrongly routes
the double into the parallel batch. Declare the flag False on the shared
mock factories so these pre-existing hook tests exercise the sequential
path they assert on.
* fix(guardrails): harden run_in_parallel reads and address review feedback
Read run_in_parallel via getattr(..., False) in the pre_call and post_call
partitions so a third-party CustomGuardrail subclass that overrides __init__
without chaining super().__init__() no longer raises AttributeError on a path
that previously worked. Drop the redundant in-function GuardrailEventHooks
import in _run_parallel_post_call_guardrails (already imported module-level).
Remove the flaky wall-clock upper-bound assertions from the two concurrency
tests; the all-start-before-any-end overlap assertion is the timing-independent
signal that actually proves concurrency.
litellm already supports Google, Microsoft and generic OIDC SSO through
fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an
unused enum value. This adds real SAML 2.0 single sign-on for the admin UI.
A new SAMLAuthHandler validates signed assertions with the OneLogin
python3-saml toolkit and maps them onto a CustomOpenID, then reuses the
shared post-login path every other provider goes through, so provisioning,
role/team mapping and the UI session JWT are unchanged. Both SP-initiated
and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are
bound to the browser that started them via an HttpOnly state cookie plus a
cached AuthnRequest id, and the ACS rejects any response whose InResponseTo
doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound
so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected
by a consumed-assertion guard whose lifetime tracks each assertion's
NotOnOrAfter, and both the replay guard and the login-state binding go
through the proxy's shared in-memory + Redis cache for multi-instance
deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the
free-SSO-user Enterprise gate after the assertion is validated, so an
unvalidated POST can no longer drive the billable-user count query.
SAML is configurable from the admin UI SSO settings (IdP metadata URL or
inline XML, SP entity ID, and an allow-unsolicited toggle), which persists
the SAML_* environment variables the handler reads, exactly like the Google,
Microsoft and generic OIDC providers.
python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels
bundle the native libraries so no system packages are required, and the
import is guarded so the proxy still starts without the package with the
SAML routes returning a clear 501.
Resolves LIT-4016
The gateway image installed --extra proxy/proxy-runtime/extra_proxy/semantic-router
but not bedrock-realtime, so aws-sdk-bedrock-runtime was absent. Bedrock Nova Sonic
speech-to-speech uses InvokeModelWithBidirectionalStream (which boto3 cannot do) via
that package, so realtime requests failed at startup with 'Missing
aws_sdk_bedrock_runtime'. Add the extra to both uv sync stages; it is already in
uv.lock so --frozen resolves, and the package/marker (python>=3.12) matches the
python3.13 image.
The accumulated-JSON fallback ran json.loads over the whole buffer after every fragment and, on failure, kept the buffer without resetting it. A buffer that ever held more than one concatenated JSON value could never parse (json raises on trailing data), so it returned None on every subsequent chunk while growing without bound - an unrecoverable per-request CPU spin. Parse one value at a time from the front with raw_decode and keep the remainder, draining trailing values on later calls and at end of stream.
The async streaming error paths fired the sync failure_handler in a thread
and the async_failure_handler via create_task at the same time, so both
mutated the shared logging object concurrently and could crash pydantic-core.
Route failure logging through a single guarded dispatch_failure_handlers, so
the sync handler only runs after the async one completes.
* fix(guardrails/model_armor): handle None metadata in post_call _process_response
On batch routes data["metadata"] is normalized to None (present key, None
value), so request_data.get("metadata", {}) returned None and _process_response
raised 'NoneType' object has no attribute 'get', 500ing every /v1/batches create
with a post_call Model Armor guardrail (regression from v1.93.0 activating the
post_call hook). Coalesce a falsy metadata to {}
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Clean up test case documentation
Remove regression comment from test_process_response_with_none_metadata_does_not_crash.
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The a2a completion-bridge tests registered the agent with only
custom_llm_provider and model, so the bridge's litellm.acompletion had no
api_key and relied on the gateway resolving ANTHROPIC_API_KEY from its ambient
env. When that env var is absent, POST /a2a/{id} returns 500 with
"Missing Anthropic API Key" and every message/send test (completion bridge,
pinned v0.3/v1.0 message shapes, semver serves) fails while the
register/discovery/rejection tests still pass.
Give A2ABridgeParams an optional api_key and register the bridge agent with
api_key="os.environ/ANTHROPIC_API_KEY", matching how the rest of the suite
wires anthropic-backed models (e.g. the ratelimit redis tests). The agent now
carries its provider key explicitly instead of depending on ambient gateway env.
Registers claude-opus-5 across the cost maps and provider lists so the model
prices, reports its real 1M/128K limits, and advertises its capabilities instead
of falling through the generalization patterns at zero cost.
Adds the first-party entry plus the Bedrock (base, global, us, eu, au, jp),
Vertex AI, and Azure AI variants. Pricing matches Opus 4.8 at $5/$25 per MTok
with the usual 1.1x regional premium on the cross-region inference profiles, and
fast mode is priced at 2x through provider_specific_entry on the first-party
entry only.
Two fields deliberately differ from Opus 4.8: prompt_cache_min_tokens drops to
512, and bedrock_output_config_effort_ceiling is omitted because Bedrock accepts
output_config.effort="max" for Opus 5.
25 test functions across three files pass unchanged when every function
they execute is mutated; the owning file killed zero of their scored
mutants. Four zero-kill tests tied to the fix in #31288 are kept for
rewrite instead of removal.
The suggested CIDR chip was a click-only span both before and after the shadcn
migration, so keyboard users could not reach or activate it. Render it as a
Button, which brings focus and Enter/Space activation with it, and cover the
keyboard path with a test that fails against the old span.
