Realtime websocket sessions emit events outside the OpenAIRealtimeEvents
union (e.g. rate_limits.updated, response.function_call_arguments.delta,
surfaced when logged_real_time_event_types="*"). Building
LiteLLMRealtimeStreamLoggingObject revalidated every stored event against
the 16-member union, producing thousands of ValidationErrors per session
(12,670 for a ~281-event session). That synchronous work blocked the
asyncio event loop, degrading realtime time-to-first-audio and dial latency
and tripping readiness probes, and the raised error discarded the session
usage so no cost was tracked.
Type results as SkipValidation[OpenAIRealtimeStreamList] and serialize the
events verbatim, so already-formed event dicts are not revalidated. The
flood drops from 12,670 errors to 0 and the combined usage survives to the
cost calculator.
Resolves LIT-3919
Resolves LIT-3920
POST /team/new with any budget_limits returned 500 because
jsonify_team_object serialized members_with_roles but left budget_limits
as a raw Python list, which Prisma's Json column rejects. /team/update
and /key/generate worked only because each json.dumps the windows itself.
Serialize budget_limits in the shared helper, guarded by isinstance(list)
so the pre-serialized /team/update path is unaffected.
Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by #30160, #30787 and #30788.
- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
async). Large agentic tool-use / thinking streams can make assembly re-raise
as APIError from inside the except-StopIteration handler, where the sibling
except does not catch it, so it escaped __next__/__anext__ and dropped the
request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
stream_chunk_builder returns None or raises, rebuild usage from the
message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
(it is read from there, not from kwargs), matching the gemini/cohere/openai
handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
operators don't have to reverse-map spend back to a key
* docs: add MCP server change guidelines
* Add outbound_credentials directory and update AGENTS.md
Updated AGENTS.md to include new outbound_credentials directory and its files, along with additional comments on existing files.
---------
Co-authored-by: tin-berri <tin@berri.ai>
The login flow stores a post-login return URL in the litellm_return_url
cookie (5 minute TTL). globalSetup snapshots cookies into the per-role
storageState that every spec reuses, so when the snapshot races ahead
of the app consuming that cookie, each test inheriting it gets
redirected to the stale URL (/ui/?login=success) mid-assertion the
first time it mounts a page. That one rogue navigation is behind the
recurring e2e failures whose call logs all show "navigated to
/ui/?login=success" while waiting for an element; which specs die
varies run to run with snapshot timing. Clear the cookie right before
saving the snapshot so no test starts with a pending redirect.
The request logs table column displayed "Key Name" while its accessor
(metadata.user_api_key_alias) and the corresponding filter both use the
"Key Alias" label; this aligns the column header with that naming.
* feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools
GET /v1/mcp/tools returns the MCP tools available to the calling key, the same
data already exposed through /mcp/tools/list and /mcp-rest/tools/list, both of
which are in llm_api_routes. The /v1/mcp/tools path was in no route group, so
virtual keys created from the UI (which default to allowed_routes=["llm_api_routes"])
got a 403 listing tools one way but not the other.
Add it to mcp_inference_routes. Unlike /v1/mcp/server, this path has no
management write counterpart, so it does not need the method-aware carve-out
used for server discovery.
* test(proxy): parametrize MCP inference route check over the full endpoint set
The Add Model provider dropdown is driven by provider_create_fields.json
(served at /public/providers/fields), and Bedrock Mantle had no entry, so it
could not be selected even though the backend provider, its models, and the UI
enum/logo mappings already existed.
Add a bedrock_mantle entry exposing the credential fields the provider actually
honors: an optional bearer api_key for BYOK, the AWS SigV4 chain, a region, and
an api_base override. Selecting it now populates the bedrock_mantle models from
the cost map via the existing getProviderModels filter.
Also resolve the provider logo when getProviderLogoAndName is given the enum key
(e.g. BedrockMantle) rather than the slug, which the dropdown passes; previously
only slugs that lowercase-matched their key (like bedrock) resolved a logo.
* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel
A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.
* refactor(ui): centralize no-mcp-servers sentinel in a shared constant
The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.
* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes
Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.
AWS auth parameters in the Bedrock and SageMaker path could be expanded against
the process environment when credentials were built. Config-sourced references
are already expanded at load time, so restrict expansion to that path: a
reference still present at request time is treated as caller-supplied input and
is left as-is, and the web-identity helper rejects environment-variable
references before resolving the token.
Also rework the ambient AWS_* fallback as a single pass that pairs each value
with its own env-var name, fixing a latent index misalignment that left
AWS_EXTERNAL_ID unresolved.
Adds regression tests covering the resolution behavior.
GET /model/info returned an empty list for a team key whose team only has team-scoped BYOK deployments, even though /v1/models and a master key both returned them. _get_caller_byok_team_scope resolved the caller's allowed teams only from user_api_key_dict.user_id and the bound user's team memberships. A team or service key has user_id=None, so the helper returned an empty set and _byok_row_outside_caller_teams then dropped every team BYOK row
This includes the key's own team_id in the allowed-team set across every non-admin branch, since a team key is authoritatively scoped to its team regardless of whether a bound user is resolvable or a formal member of that team
Completes the work in #30025, which aligned /v1/model/info with router deployments but missed the team-key case in the scope helper it introduced
* fix: prevent key-level metadata.tags from leaking into Bedrock passthrough body
* test: cover bedrock key-tag litellm_metadata pre-seed in common_checks
Add a regression test asserting key-level tags on a bedrock passthrough
request land in litellm_metadata and never leak into the provider-facing
metadata field, which closes the codecov/patch gap on the auth_checks
pre-seed line. Also drop the now-stale comment that hardcoded
metadata["headers"]; the headers are written under whichever metadata
field _get_metadata_variable_name selects.
* refactor(auth): pre-seed litellm_metadata from LITELLM_METADATA_ROUTES
The auth-time pre-seed in common_checks hardcoded "bedrock", so any other
route later added to LITELLM_METADATA_ROUTES would reintroduce GH#30629 (key
tags leaking into the provider-facing metadata field) without a matching
update here. Key off the shared constant instead, and extend the regression
test to cover a non-bedrock metadata route so the route-agnostic behavior is
locked in.
* fix(auth): pre-seed litellm_metadata before header-tag merge
apply_client_tag_policy_pre_auth runs in user_api_key_auth.py before
common_checks, so it resolved get_metadata_variable_name_from_kwargs to
'metadata' (litellm_metadata was not yet present). common_checks then
pre-seeded litellm_metadata on LITELLM_METADATA_ROUTES, after which
apply_key_tags_pre_auth and _tag_max_budget_check both targeted
litellm_metadata, leaving header tags stranded in metadata and invisible
to per-tag budget enforcement on Bedrock and other matching routes.
Extract the pre-seed into LiteLLMProxyRequestSetup.pre_seed_litellm_metadata_for_route
and invoke it before apply_client_tag_policy_pre_auth so all tag merges
and the budget-check read agree on the same metadata key.
* test(auth): guard early litellm_metadata pre-seed wiring for header tags
Bugbot's autofix added a pre-seed of litellm_metadata in
_run_centralized_common_checks before apply_client_tag_policy_pre_auth, so
x-litellm-tags header tags land in litellm_metadata and stay visible to
_tag_max_budget_check on LITELLM_METADATA_ROUTES. Its test replayed that call
order in the test body, so removing the production call site still passed.
Add a wiring-level regression that drives the real _run_centralized_common_checks
and asserts header tags land in litellm_metadata (not metadata) for bedrock and
/v1/messages. Dropping the pre-seed call site now fails the test.
---------
Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The Usage dashboard "Spend Per User" chart rendered raw UUIDs (and
default_user_id) instead of emails. /user/daily/activity passed
entity_metadata_field=None, so every user entity in the breakdown
carried empty metadata; the chart could only fall back to the user_id.
The frontend resolved labels from a separately paginated user list, so
any spender not on a loaded page showed as a UUID.
Resolve the email/alias for the user_ids actually on the page (mirroring
how api key metadata is already resolved) and attach it to the entity
metadata, so the chart labels each spender with their email and falls
back to the UUID only when no email is on file. get_daily_activity gains
an optional resolve_entity_metadata hook so the user endpoint can do this
page-scoped lookup without loading the whole user table.
Resolves LIT-3889
Map the SCIM enterprise extension block
(urn:ietf:params:scim:schemas:extension:enterprise:2.0:User) onto SCIMUser
so create and PUT persist employeeNumber, costCenter, organization,
division, department, and manager into LiteLLM_UserTable.metadata under
scim_enterprise, and round-trip them back out on read. This lets financial
reporting group spend by fields like cost center and department.
The enterprise block holds directory-only HR attributes, so it is kept out
of the generic user management responses (/user/info, /v2/user/info, and
/user/list), which non-proxy-admin callers such as team and org admins can
use to read other users. The data still lands in metadata for reporting and
still round-trips through the SCIM read endpoints, which build their response
from the user row directly.
Resolves LIT-3617
