Add a RESTful PATCH /team/{team_id} that partially updates a team using RFC 7386 JSON Merge Patch. team_id comes from the path, and metadata is merged with the team's stored metadata instead of being replaced wholesale the way POST /team/update does: an omitted key is preserved, key: null deletes it, and any other value overwrites, recursing into nested objects. Every other field behaves the same as POST /team/update
The handler delegates to the existing update path, so authorization, budget checks, system-managed-key stripping, metadata encryption, cache refresh, and audit logging are shared rather than reimplemented. POST /team/update is untouched, so the change is purely additive
A trailing slash on --base-url (or LITELLM_PROXY_URL) produced
double-slash URLs like https://host//sso/cli/start, which 404s. Normalize
once in the CLI's top-level group callback so every subcommand benefits.
The /guardrails/usage/{overview,detail,logs} endpoints resolved guardrails only
from the litellm_guardrailstable Prisma table, so guardrails defined in
config.yaml (which live only in IN_MEMORY_GUARDRAIL_HANDLER) were invisible:
detail 404'd, overview omitted them or rendered them as Custom/Guardrail
orphans, and logs missed their logical-name alias.
Add config-owned accessors (list_config_guardrails, get_config_guardrail_by_id)
to the in-memory handler and use them in the usage endpoints, mirroring the
union/fallback already used by list_guardrails_v2 and get_guardrail_info. Also
preserve guardrail_info when storing a config guardrail (type/description were
dropped at initialize time) and read the Prisma-row / dict / LitellmParams
shapes uniformly.
Resolves LIT-2529
* fix(proxy): match list/dict guardrail_mode in compliance mode checks
* test(compliance): cover ComplianceChecker guardrail_mode shapes (str/list/dict/None)
* fix(proxy): trust only Mode.default in compliance mode matching (ignore tag overrides)
* fix(proxy): match dict guardrail_mode only when every branch runs in mode (no false-compliant)
* fix(proxy): treat multi-mode guardrail_mode as unresolved (no false-compliant)
The list branch previously counted a guardrail configured with mode:
[pre_call, post_call] under every listed mode. But when the writer cannot
infer the concrete hook that fired (apply_guardrail invocations), the raw
list is logged, and an image-only request that only reaches the post-call
path still records both modes. That let a pre_call compliance check pass on
a request that only ran post_call.
Match the tightened dict semantics: a list now counts for mode only when
every listed mode equals mode. Same trade-off (under-report instead of
false-COMPLIANT). Speculative set support is dropped (spend logs are
JSON-serialized, sets do not cross the wire).
Tests updated to reflect the tightened list semantics, deduplicated (single
TestModeMatching class), and shortened. The invariant is now expressed as
a computed check: True implies every branch runs in the matched mode.
---------
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
gemini/gemini-3.1-flash-image, vertex_ai/gemini-3-pro-image, and
vertex_ai/gemini-3.1-flash-image existed in the root pricing JSON but not in
litellm/model_prices_and_context_window_backup.json, leaving deployments with
LITELLM_LOCAL_MODEL_COST_MAP=True unprotected. Copies the root entries into
the backup verbatim and extends the regression test to cover all ten gemini
image models, asserting each exists in the local cost map so a missing backup
entry fails the test instead of passing vacuously
* fix(proxy): build redis usage cache from REDIS_* env when cache backend is not Redis
Selecting a semantic (or any non-Redis-KV) response cache left
redis_usage_cache unset, silently downgrading cross-pod rate limits,
parallel-request limits, spend coordination, and the pod lock manager
to per-pod in-memory state. Fall back to a standalone RedisCache built
from REDIS_* environment variables, mirroring the existing
use_redis_transaction_buffer escape hatch, which now shares the same
helper.
Resolves LIT-3861
* feat(proxy): configure the coordination redis independently of the response cache
Adds general_settings.coordination_redis, an explicit block for the Redis
the proxy uses for cross-pod rate limits, parallel-request limits, spend
tracking, the pod lock manager, and shared health checks. Resolution order
is the explicit block, then a plain-Redis response-cache backend, then the
REDIS_* environment. Cluster and sentinel targets are supported, and a
cluster target now builds a RedisClusterCache so cluster-aware consumers
take the cluster path.
Admins can configure it from the Caching page of the dashboard via
/coordination_redis/settings, which reports which source is in effect,
redacts credentials on read, and offers a connection test. Settings saved
there are read back at startup so they take effect on restart.
Also fixes redis client construction so an explicitly configured host
outranks REDIS_URL in the environment. Previously the url branch stripped
the caller's host and port, so an explicit block, or a connection test
typed into the dashboard, silently targeted whatever REDIS_URL named
* fix(ui): move coordination_redis_settings into renamed _components directory
---------
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
* feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls
The GenAI semantic conventions record failures of a GenAI client operation as
a log-based event named gen_ai.client.operation.exception, carrying the
exception.type / exception.message / exception.stacktrace trio at severity
WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM
call produced only the deprecated error.* span attributes, a generic exception
span event without a stacktrace, and the stacktrace under the vendor key
litellm.provider.error.stack_trace.
Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring
the metrics plumbing) and record the event behind the enable_events flag, which
until now was defined but consumed nowhere. An operator-configured LoggerProvider
global is reused so the events ride their existing logs pipeline; an explicit
NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all.
The existing span-side error surface (error.type, error.message, the exception
span event, and the litellm.provider.error.* detail keys) is untouched for
backwards compatibility.
* fix(otel): always ride the semconv-required exception pair on the GenAI event
Filtering the event attributes on truthiness conflated "absent" with "empty",
so an empty exception.type or exception.message would have been dropped, leaving
an event with neither semconv-required field. Build the attributes so the pair is
unconditional and only the recommended stacktrace is omitted when the payload
carries none.
* docs(otel): document the events plumbing module in the package README
* test(otel): cover the log exporter selection and logs endpoint normalization
The new logs plumbing had no coverage for exporter-kind selection, the
console fallback for an unrecognized kind, the /v1/logs signal-path rewriting
that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch
processor split.
Selecting a semantic (or any non-Redis-KV) response cache left
redis_usage_cache unset, silently downgrading cross-pod rate limits,
parallel-request limits, spend coordination, and the pod lock manager
to per-pod in-memory state. Fall back to a standalone RedisCache built
from REDIS_* environment variables, mirroring the existing
use_redis_transaction_buffer escape hatch, which now shares the same
helper.
Resolves LIT-3861
vertex_ai/gemini-2.5-flash-image, vertex_ai/gemini-3-pro-image-preview,
vertex_ai/gemini-3.1-flash-image-preview, gemini/gemini-3-pro-image-preview,
and gemini/gemini-3.1-flash-image-preview were missing supports_reasoning
entries; _supports_factory then fell through to the vertex_ai provider-level
config which returns true, causing requests with reasoning_effort to be sent
to an API that rejects them.
Hoisting every role system entry into the top-level system field mutates
the cache prefix whenever a client such as Claude Code appends a new
mid-conversation system message, invalidating the prompt cache for the
entire message history on Bedrock Invoke. Bedrock only rejects a system
entry at messages.0, so hoist just the leading run and forward the rest
in place
A correctly signed JWT whose user_id or server_id claim was an empty string
passed claims validation but raised ValidationError from the EnvelopeIdentity
constructor inside open_envelope, breaking its never-raises guarantee. The
claims model now mirrors the identity's min_length constraints, so any claim
set that validates also constructs, and the empty-identity case maps to
MalformedPayload like every other bad claim shape.
Pure, unwired module: mints and opens the single client-held bearer that
carries both a litellm identity and the encrypted upstream OAuth grant with
zero server-side storage. HS256 JWT signing (same approach as the BYOK
session bearer) plus the existing encrypt_value/decrypt_value symmetric
helpers, with all key material and the clock injected as parameters. Opening
returns typed frozen error values (not_an_envelope, bad_signature, expired,
malformed_payload, decrypt_failed); minting rejects envelopes over
MAX_ENVELOPE_BYTES with a typed error instead of truncating. Error values
and reprs never carry token material.
For MCP servers with auth_type=oauth2 + delegate_auth_to_upstream=true, a
client-supplied upstream token that the upstream rejects was masked: the
upstream 401 raised during tools/list is absorbed by the list handler, so on a
single-server route a rejected token became HTTP 200 with an empty tool list.
Clients showed "0 tools" instead of re-authenticating, and monitoring never saw
an unauthorized signal.
Extend the connect-time preflight _check_passthrough_upstream_auth to probe
delegate-auth servers with the caller's bare Authorization bearer, reusing the
existing _probe_upstream_auth and the RFC 6750 challenge builder, so a rejected
token fails the connect with 401 + WWW-Authenticate error="invalid_token" and a
compliant client re-runs the upstream OAuth flow.
The bare Authorization header is a valid upstream token only when admission took
the delegate bypass, so the delegate target is resolved through
get_mcp_server_by_name (the same resolver admission uses) rather than the wider
allowed-server prefix/access-group matching. A name that reaches a delegate
server only via server_id or an access group is admitted as a real LiteLLM key,
so probing it would leak that key upstream; requiring the admission-resolver
match closes that gap. The probe is gated to single-server routes (matching the
OBO preflight), keyed to the caller's authorized set by server_id, and the
challenge echoes the requested name so aliased routes get the same
resource_metadata URL as the tokenless preemptive challenge. Tokenless requests
keep flowing to the preemptive discovery challenge unchanged.
Resolves LIT-4194
* fix(spend-logs): honor store_prompts_in_spend_logs for guardrail_information (LIT-4314)
_get_spend_logs_metadata passed guardrail_information entries through
verbatim, so guardrail hooks that echo the LLM request into
guardrail_response leaked the raw prompt into LiteLLM_SpendLogs.metadata
regardless of store_prompts_in_spend_logs. This mirrored the pre-existing
gap for the other prompt-carrying fields (vector_store_request_metadata,
error_information, etc.), which already sanitize via
_should_store_prompts_and_responses_in_spend_logs.
Add _sanitize_guardrail_information_for_spend_logs alongside the other
per-field sanitizers and wire it into _get_spend_logs_metadata. When the
flag is False the sanitizer replaces guardrail_request and
guardrail_response with REDACTED_BY_LITELM_STRING while preserving every
other typed field on the entry (name, provider, mode, status, timings,
action, violation_categories, risk_score, masked_entity_count, ...) so
guardrail dashboards keep working. When the flag is True (or the field
is None) the entries pass through unchanged.
Widen StandardLoggingGuardrailInformation.guardrail_request from
Optional[dict] to Optional[Union[dict, str]] so the redacted sentinel
satisfies the TypedDict without needing a cast; guardrail_response
already accepted str.
Regression tests cover the three cases (flag=False redacts,
flag=True passes through, None passes through) plus an end-to-end
get_logging_payload path that fails if the wire-in at line 139 is
reverted.
* chore(spend-logs): review nits (one-shot dict build, scrub identifier in tests)
- _redact_prompt_fields_in_guardrail_entry now returns the redacted
dict in one expression instead of seed-then-mutate (TYPE-3)
- swap the illustrative guardrail_name in the new test fixtures for
a generic 'demo-echo-guard' identifier
* chore(spend-logs): only redact guardrail prompt fields when caller supplied them
Greptile P2: the sanitizer was unconditionally writing REDACTED_BY_LITELM
into both guardrail_request and guardrail_response on the copy, so entries
that never carried one of those fields (e.g. a guardrail that only emits
a guardrail_response) came out with a phantom guardrail_request key added.
Guard both assignments with an in-check so the output shape is stable.
Add a mutation-checked regression test that fails if either guard is
removed.
* fix(spend-logs): also redact match_details and classification in guardrail_information
The initial LIT-4314 fix redacted guardrail_request and guardrail_response,
but two other typed fields on StandardLoggingGuardrailInformation also
carry raw prompt content when a first-party guardrail populates them:
- litellm_content_filter/content_filter.py:1676 sets classification =
dict(CompetitorIntentDetection), whose evidence[*].match is a substring
taken directly from the user's normalized prompt (see
litellm_content_filter/competitor_intent/base.py:184-194).
- block_code_execution/block_code_execution.py:571 sets match_details =
guardrail_response = [dict(d) for d in detections], where detections
carry the fenced-code-block content extracted from the user's message.
Reproduced live against localhost:4000 with store_prompts_in_spend_logs
false and a custom guardrail passing tracing_detail with both fields:
before this commit the raw prompt shows up in metadata.guardrail_information[0]
under match_details and classification; after, both are the sentinel.
Widen the two TypedDict fields to Optional[Union[..., str]] so the
sentinel string satisfies the schema without a cast, and consolidate
the redaction set into a tuple so future prompt-carrying additions are
one-line changes.
* fix(spend-logs): normalize non-list guardrail_information shapes in sanitizer
xecguard's logging hook (xecguard.py:246) assigns a bare dict to
standard_logging_object['guardrail_information'] instead of a list,
violating the typed contract Optional[List[StandardLoggingGuardrailInformation]].
Without defensive normalization, _sanitize_guardrail_information_for_spend_logs
iterates the dict's string keys and _redact_prompt_fields_in_guardrail_entry
raises TypeError on {**'guardrail_name'}, which get_logging_payload's
downstream update_database catches with a broad except and silently drops
the entire spend-log write for that request.
Normalize a bare-dict input to a single-item list at the sanitizer's
entry point, and skip any non-dict entries defensively (matching OTEL's
existing isinstance filter at opentelemetry.py:1751-1753 for the same
field). Downstream readers already model this defensively; make the
spend-log write path match.
The root cause is xecguard's writer, not the sanitizer. That is being
tracked as a separate ticket; this PR keeps xecguard-enabled deploys
from silently losing spend logs when store_prompts_in_spend_logs=false.
* fix(types): declare guardrail Union members str-first to avoid poisoning typing cache
CPython's typing module caches Union[...] order-insensitively (first-
construction wins), and litellm/types/utils.py has no 'from __future__
import annotations', so its unions are constructed eagerly at import
time -- before any proxy model. Declaring guardrail_request,
classification, and match_details with dict-first ordering seeds the
typing cache with a dict-first tuple, and later proxy models that
declare custom_llm_provider / model_aliases / vertex_credentials as
Optional[Union[str, dict]] pick up the same dict-first object.
Downstream, Pydantic's get_args() then reports anyOf in dict-first
order, FastAPI emits the OpenAPI accordingly, and 'npm run gen:api'
produces a schema.d.ts diff on unrelated fields, tripping the schema-
sync CI check.
Behaviorally identical in Python and at the wire; the flip only reorders
the union members so the first construction matches how the codebase
had always declared these unions, and 'npm run gen:api' now produces a
zero diff against the committed schema.d.ts.
The multi-server list path already relays an upstream 401 from a client-forwarded
server (true_passthrough / oauth_delegate) as an MCPUpstreamAuthError so the caller
re-runs its own upstream OAuth. The single-server REST call path did not: an upstream
401 was masked as a graceful isError result, so an MCP client holding an expired
upstream token never learned it had to re-authenticate
Relay the upstream 401 on the call path too. For these modes the manager calls the
client with raise_on_error=True, extracts the WWW-Authenticate through the existing
upstream-auth exception walk, and raises MCPUpstreamAuthError; the REST endpoint turns
it into a real 401 + WWW-Authenticate. Only 401 is treated as a re-auth signal (a 403 is
a genuine authorization failure that re-auth will not fix, so it stays a masked isError
with a visible warning), matching the list path and MCPUpstreamAuthError's contract. The
legacy oauth2 + delegate_auth_to_upstream mode is deliberately left off the call-path
relay since it is being removed
To keep this expected caller-must-reauth signal from tripping error-rate alerts, the
client layer logs at debug when the caller opted into raise_on_error and therefore owns
the exception (both call_tool/list_tools and the run_with_session helper they share, so an
expected re-auth emits no warning per call either), the manager's non-auth branch logs the
exception type only (never str(e), which for an httpx error embeds the upstream URL a
credential can hide in), and the streamable and REST handlers log the relayed 401 at info
rather than as an error with a traceback
Tests cover the manager raising on a client-forwarded 401 while keeping a 403/503 as a
masked isError, the client-layer debug-vs-error logging split, the streamable handler's
informational isError, and the REST endpoint relaying both the direct and virtual
mcp_tool_call branches as a real 401 + WWW-Authenticate; each was mutation-checked to fail
when the corresponding behavior is broken
Keep the full provider exception in server-side logs only; the client
receives a fixed actionable message. Also follow implicit exception
context when detecting context window overflows and pin the detection
variants plus the redaction in tests
Resolves LIT-4284
When the embedding model exceeded its context window, the MCP semantic
tool filter silently passed all tools through and reported N->N success
in the filter header; when the overflow happened while embedding tool
descriptions at router build time, the hook was never registered at all
and filtering was silently disabled
Semantic filtering now fails closed on context window overflows: the
request is rejected with HTTP 400 and a message that names the embedding
model and advises switching to one with a larger context window or
disabling the filter. Build time overflows are recorded on the filter so
the hook still registers and blocks MCP tool requests with the same
actionable error while leaving native-only requests untouched. The
dashboard test panel renders the backend message in an error banner
instead of a success state. OpenAI's embedding overflow message
(maximum input length is N tokens) now maps to ContextWindowExceededError
Team-level callback_vars (e.g. langsmith_api_key) get spread into
data["metadata"] as four aliases (user_api_key_metadata,
user_api_key_team_metadata, user_api_key_auth_metadata,
user_api_key_auth). When a guardrail hook echoes that metadata into
its guardrail_response, the plaintext credential landed five times
inside LiteLLM_SpendLogs.metadata.standard_logging_guardrail_information[i].guardrail_response
and every downstream sink that reads it (OTel via emit_guardrail_span,
Langfuse, custom loggers).
Add a purpose-built payload walker (mask_credentials_in_payload) that
only masks strings under sensitive-named keys and preserves every
other value (None, ints, floats, bools, tuples, typed objects) verbatim.
The walker reuses SensitiveDataMasker.is_sensitive_key so the pattern
list stays in one place, and unwraps Pydantic models via model_dump()
so nested UserAPIKeyAuth values reached by the walk get scanned as
plain dicts (they are JSON-serialized downstream anyway).
Apply the walker at add_standard_logging_guardrail_information_to_request_data
after the existing secret_fields pop and match/regex redaction, so
every downstream sink sees masked values from a single seam.
The per-row delete_many loop becomes a single delete filtered to the enumerated OAuth users'
(user_id IN, server_id) pairs; same rows deleted, same BYOK-sparing precision, same count-mismatch
detection, one round-trip instead of N
LiteLLM_MCPUserCredentials stores BYOK API keys in the same column as per-user
OAuth tokens, so the purge on a mint-relevant config change now deletes only
rows whose payload decodes as an OAuth2 credential, each by its
(user_id, server_id) pair, instead of every row for the server. An api_key
server whose url changes purges nothing. delete_mcp_server now also
invalidates each enumerated user's cached token so a re-created server reusing
the id cannot serve tokens minted for the deleted one, and both cache drops
are best-effort
The snapshot read only feeds the stale-token purge decision; leaving it unguarded meant a failed
read would 500 an edit whose update would have succeeded, and it broke
test_edit_mcp_server_redacts_credentials, whose mocked prisma is not awaitable on the un-patched
get_mcp_server path. A failure now logs and skips the purge, consistent with the purge half already
being best-effort. Adds the first endpoint-level coverage of the edit purge wiring: purge on a
mint-relevant change, no purge when the identity is unchanged, and edit success with purge skipped
when the snapshot read raises
