* Add Rubrik as officially-supported guardrail plugin
Adds tool blocking and batch logging integration with an external Rubrik
webhook service. The plugin validates LLM tool calls against a policy
service (fail-open on errors) and batch-logs all requests/responses.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Update Rubrik docs: config.yaml as primary, env vars as fallback
Restructures the Quick Start to present config.yaml as the recommended
approach with tabbed UI, and environment variables as an alternative
fallback.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add Rubrik env vars to config_settings reference
Fixes documentation validation by adding RUBRIK_API_KEY,
RUBRIK_BATCH_SIZE, RUBRIK_SAMPLING_RATE, and RUBRIK_WEBHOOK_URL
to the environment settings reference table.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add fallback message when blocking service returns empty explanation
Prevents whitespace-only violation message when the tool blocking
service blocks tools but returns an empty content field.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When litellm migrated from Poetry to uv (PR #24905, v1.83.1), the core
dependency specifications in pyproject.toml changed from Poetry bare-version
strings (e.g. openai = "2.30.0") to PEP 621 exact pins (openai==2.24.0).
Poetry bare-version strings are actually caret ranges (^X.Y.Z == >=X.Y.Z,<X+1),
but PEP 621 == is exact. This means every downstream package that installs
litellm as a library dependency is now forced to downgrade aiohttp, pydantic,
openai, click, and 8 other common packages to exact old versions.
Fix: restore range specifiers for the 12 core runtime dependencies. The
optional extras (proxy, proxy-runtime, etc.) are consumed primarily by
Docker images where exact pins are appropriate and are left unchanged.
The uv.lock file continues to provide exact reproducibility for Docker
builds and CI.
Fixes: #26154
- preserve existing shared backend `mode` when router deployment registration
reuses a provider/model key already in `litellm.model_cost` (prevents alias
with `mode: chat` from downgrading shared `chatgpt/gpt-5.4` from `responses`
to `chat` and triggering 403s on /v1/chat/completions)
- teach the ChatGPT Responses parser to recover `response.output_item.done`
entries when `response.completed.output` is empty
- add defensive /responses -> /chat/completions bridge fallback that
reconstructs output items from raw SSE when `raw_response.output` is empty
- regression coverage for shared alias routing, empty completed.output
parsing, and SSE bridge recovery
Closes#25403
Co-authored-by: afoninsky <andrey.afoninsky@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: reuse cached credentials in VertexAIPartnerModels instead of creating new VertexLLM per request
VertexAIPartnerModels.completion() was creating a throwaway VertexLLM()
instance on every call to get an access token, bypassing the credential
cache inherited from VertexBase. This caused a fresh token fetch for
every single request, adding significant latency overhead.
Fix: call super().__init__() to initialize VertexBase's credential cache,
and use self._ensure_access_token() instead of a new VertexLLM instance.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: apply same credential caching fix to VertexAIGemmaModels and VertexAIModelGardenModels
Same bug as VertexAIPartnerModels: both classes had `pass` in __init__
instead of `super().__init__()`, and created throwaway VertexLLM()
instances per request instead of using self._ensure_access_token().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(vertex_ai): single-flight credential refresh to prevent thundering herd
When GCP credentials expire under high concurrency, all requests
simultaneously call credentials.refresh() via asyncify, saturating the
40-thread anyio pool and blocking the proxy for 20+ seconds.
This adds:
- Per-credential asyncio.Lock in get_access_token_async for single-flight
refresh (1 coroutine refreshes, others wait on the lock)
- Background refresh when token_state is STALE (usable but near expiry),
returning the current token immediately with zero added latency
- threading.Lock on the sync get_access_token path
- Uses google-auth's TokenState enum (FRESH/STALE/INVALID) instead of
reimplementing expiry logic
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address PR review comments
- Use asyncio.create_task() instead of deprecated get_event_loop().create_task()
- Track in-flight background refresh tasks to prevent duplicate refreshes
when multiple STALE-path callers pass through the lock before the first
background task completes
- Add token validation in the STALE branch (consistent with FRESH/INVALID)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: lazy-import TokenState to avoid breaking when google-auth is not installed
Also extract helper methods to bring get_access_token_async under the
PLR0915 statement limit (50).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: apply Black formatting to test file and update uv.lock
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove user-provided project_id from log messages (CodeQL log injection)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: avoid leaking token value in error message, log type instead
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: restore uv.lock to match litellm_oss_branch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove project_id from remaining log message (CodeQL log injection)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove remaining project_id from log and error messages
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace hardcoded _is_claude_4_6_model() string matching with
supports_output_config flag in model_prices_and_context_window.json,
accessed via _supports_factory(). This follows the project's established
pattern for model capability checks (per AGENTS.md rule #8).
Bedrock Invoke now conditionally preserves output_config for models
that declare supports_output_config=true (currently Claude 4.6 models),
while stripping it for older models to avoid request rejection.
Ref: https://github.com/BerriAI/litellm/issues/22797
* fix(redis): cache GCP IAM token to prevent async event loop blocking
## Problem
GCPIAMCredentialProvider.get_credentials() calls _generate_gcp_iam_access_token
on every Redis connection establishment. This function performs synchronous HTTP
and gRPC calls (google-auth + google-cloud-iam) which block Python's asyncio
event loop while running.
Under concurrent load (e.g. connection pool warm-up, parallel health checks),
multiple connections are established simultaneously, each triggering an
independent blocking IAM token refresh. These refreshes serialise behind each
other inside the single-threaded event loop, causing individual Redis spans to
take 20-25 seconds instead of milliseconds.
Observed in production via Datadog APM: a single INCRBYFLOAT Redis span took
25.6 seconds (90% of a 28.4s trace), with GCP metadata + GenerateAccessToken
gRPC calls visible inside the span. This cascaded into aiohttp SocketTimeoutError
on upstream LLM API calls — not because the upstream was slow, but because the
event loop was frozen and the 30-second sock_read timer fired on a connection
that was never given CPU time.
## Fix
Add a module-level token cache (dict keyed by service account, value is
(token, expiry_monotonic)). _get_cached_gcp_iam_token() returns the cached
token on cache hit (no I/O), and refreshes only when expired using
double-checked locking so only one thread performs the network round-trip.
GCP IAM tokens are valid for 1 hour; the cache TTL is set to 55 minutes
(_GCP_IAM_TOKEN_TTL_SECONDS = 3300) to refresh safely before expiry.
The cache is shared across all GCPIAMCredentialProvider instances for the same
service account, so N concurrent Redis connections on the same pod share a
single token and avoid N concurrent blocking refreshes.
get_credentials_async() already used asyncio.to_thread (non-blocking), and is
updated to call _get_cached_gcp_iam_token so it also benefits from caching.
## Tests
- Updated existing test that expected a fresh token on every call to reflect
the new caching behaviour.
- Added tests for: cache hit (no redundant I/O), cache expiry and refresh,
and cache sharing across multiple provider instances.
- Added autouse fixture to clear the module-level cache between tests.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(redis): remove unused Optional import from _redis_credential_provider.py
* refactor(redis): improve documentation for GCPIAMCredentialProvider class
Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication.
* refactor(redis): improve documentation for GCPIAMCredentialProvider class
Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication.
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(memory): jsonify metadata before Prisma writes on /v1/memory
The POST/PUT memory endpoints handed bare dicts (and bare `None`) to
prisma-client-python for the `Json?` `metadata` column, which the client
rejects with `MissingRequiredValueError` / `DataError: metadata should
be of any of the following types: NullableJsonNullValueInput, Json`.
Both the create and upsert paths now route writes through the existing
`jsonify_object` helper used elsewhere in the proxy for `Json?` columns
(e.g. `LiteLLM_VerificationToken.budget_limits`), and omit metadata
when None so the column defaults to SQL NULL via the schema.
Explicit `metadata: null` on PUT is now a no-op for the column to match
how the rest of the proxy handles nullable JSON fields (no
`JsonNull`/`DbNull` sentinel exists in prisma-client-python — see
RobertCraigie/prisma-client-py#714). A payload with only `metadata: null`
returns 400 instead of a misleading 200.
Made-with: Cursor
* fix(memory): JSON-encode non-dict metadata before Prisma writes
`jsonify_object` only stringifies dict values, so list-shaped metadata
still hit Prisma as raw Python objects and triggered the same
DataError this PR is meant to fix. `metadata` is typed `Optional[Any]`
so list payloads are valid input. Replace `jsonify_object` with a
local `_serialize_metadata_for_prisma` helper that always `json.dumps`
non-string values, applied at all three write sites
(POST create, PUT update, PUT-create). Adds regression tests for
list metadata on each path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): always json.dumps metadata, not just non-strings
The str-passthrough in `_serialize_metadata_for_prisma` left plain
Python strings (e.g. `metadata: "hello"`) unencoded — Postgres `jsonb`
rejects bare-word strings as invalid JSON, reproducing the same
DataError this PR is meant to fix. Always `json.dumps` regardless of
input type so all `Optional[Any]` shapes (dict, list, scalar, str)
become valid JSON. Adds a regression test for plain-string metadata.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): encode explicit metadata:null as JSON null to clear field
prisma-client-python has no JsonNull/DbNull sentinel for writing a
true SQL NULL on `Json?` columns (RobertCraigie/prisma-client-py#714),
so an earlier iteration of this PR treated `PUT {"metadata": null}`
as a no-op. That doesn't match the natural caller expectation that
explicit-null clears the field.
Encode it as the JSON literal `null` instead — stored as Postgres
`jsonb 'null'`, which prisma deserializes back to Python `None` on
read. Subsequent reads return `metadata: null`, so the field is
effectively cleared from the caller's perspective. Strict SQL NULL
remains unreachable via the typed client and would require raw SQL.
Also clean up stale `jsonify_object` references in test mock comments
(replaced by `_serialize_metadata_for_prisma`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(memory ui): use shared DeleteResourceModal for memory deletion
Swap the imperative `Modal.confirm` in MemoryView for the shared
`DeleteResourceModal`, so memory deletion matches the rest of the
dashboard: type-to-confirm guard on the key, in-flight loading state
on the OK button, cancel disabled while the request is pending, and
the modal stays open on error so the user can retry.
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): jsonify metadata before Prisma writes on /v1/memory
The POST/PUT memory endpoints handed bare dicts (and bare `None`) to
prisma-client-python for the `Json?` `metadata` column, which the client
rejects with `MissingRequiredValueError` / `DataError: metadata should
be of any of the following types: NullableJsonNullValueInput, Json`.
Both the create and upsert paths now route writes through the existing
`jsonify_object` helper used elsewhere in the proxy for `Json?` columns
(e.g. `LiteLLM_VerificationToken.budget_limits`), and omit metadata
when None so the column defaults to SQL NULL via the schema.
Explicit `metadata: null` on PUT is now a no-op for the column to match
how the rest of the proxy handles nullable JSON fields (no
`JsonNull`/`DbNull` sentinel exists in prisma-client-python — see
RobertCraigie/prisma-client-py#714). A payload with only `metadata: null`
returns 400 instead of a misleading 200.
Made-with: Cursor
* fix(memory): JSON-encode non-dict metadata before Prisma writes
`jsonify_object` only stringifies dict values, so list-shaped metadata
still hit Prisma as raw Python objects and triggered the same
DataError this PR is meant to fix. `metadata` is typed `Optional[Any]`
so list payloads are valid input. Replace `jsonify_object` with a
local `_serialize_metadata_for_prisma` helper that always `json.dumps`
non-string values, applied at all three write sites
(POST create, PUT update, PUT-create). Adds regression tests for
list metadata on each path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): always json.dumps metadata, not just non-strings
The str-passthrough in `_serialize_metadata_for_prisma` left plain
Python strings (e.g. `metadata: "hello"`) unencoded — Postgres `jsonb`
rejects bare-word strings as invalid JSON, reproducing the same
DataError this PR is meant to fix. Always `json.dumps` regardless of
input type so all `Optional[Any]` shapes (dict, list, scalar, str)
become valid JSON. Adds a regression test for plain-string metadata.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code review noted the previous test reimplemented the proxy's
try/except/finally around post_call_success_hook, so it would not catch a
regression that re-introduced the duplicate-log bug in the production code
path. Extract the gating logic into
`ProxyBaseLLMRequestProcessing._flush_deferred_async_logging` so tests
exercise the production helper directly.
The proxy finally block becomes a single call to the helper. Tests now
invoke the helper itself and additionally assert (via inspect.getsource)
that base_process_llm_request continues to delegate to the helper rather
than inlining the gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a non-streaming request had any post-call guardrail registered, the
proxy deferred the async-success logging closure until after
post_call_success_hook ran. The finally block fired that closure even when
the hook raised — the propagating HTTPException then routed through
post_call_failure_hook → _handle_logging_proxy_only_error, which writes its
own failure spend log via async_failure_handler. The result was two spend
log rows per blocked request: one Success exposing the blocked LLM response
and one Failure. Reproduces with both pre and post bedrock guardrails
configured for a team when the post-call OUTPUT scan blocks the response.
Gate the deferred closure on _exception_raised so the failure path remains
the single source of truth for blocked requests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(guardrails): apply team-level guardrails alongside global policy guardrails
Two bugs prevented team-direct guardrails from being automatically applied
when using a team-scoped API key:
1. Auth caching: `valid_token.team_metadata` was never refreshed from the
freshly-fetched team object at the "Check 6" step in
`_user_api_key_auth_builder`. Guardrails added to a team after the key
was first cached were therefore invisible to `move_guardrails_to_metadata`.
Fix: propagate `_team_obj.metadata` → `valid_token.team_metadata` after
every "Check 6" team fetch (user_api_key_auth.py).
2. Guardrail execution: `get_guardrail_from_metadata` checked
`data["litellm_metadata"]` before `data["metadata"]`. When a request
carried a non-empty `litellm_metadata` without a "guardrails" key, the
merged guardrail list written to `data["metadata"]` by
`move_guardrails_to_metadata` was shadowed and the guardrail received an
empty requested-guardrails list (custom_guardrail.py).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix merge conflict
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors the existing api_base handling in _load_credentials_from_config():
when a vector_store_config references litellm_credential_name and the stored
credential does not itself define aws_sts_endpoint or aws_web_identity_token,
any caller-supplied value on those keys is removed. Keeps endpoint and
identity fields aligned with the credential definition.
* fix(bedrock guardrail): dedupe post-call log when only post_call is configured
When a Bedrock guardrail runs with only post_call configured, the post-call
trace section showed the same guardrail twice (one entry per parallel
INPUT/OUTPUT API call). Add skip_logging param to make_bedrock_api_request
and pass it for the INPUT scan so the OUTPUT scan stands as the single
canonical post_call log entry. INPUT exceptions still propagate, so the
input-side blocking behavior is preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(bedrock guardrail): post_call only scans OUTPUT, not INPUT
post_call is the response-validation hook by definition — input scanning
belongs to pre_call / during_call. The previous code ran an extra INPUT
scan in post_call when no pre/during hook was configured, which produced
a duplicate "post-call" entry in the trace and was semantically wrong
for a "post-call" event.
Drops the should_validate_input branch and parallel asyncio.gather in both
async_post_call_success_hook and async_post_call_streaming_iterator_hook
in favor of a single OUTPUT scan. Reverts the now-unneeded skip_logging
parameter on make_bedrock_api_request introduced in the previous commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fall back to email match when looking up the caller in
members_with_roles — email-onboarded members may have user_id=None on the
stored entry, which caused a false 404 for valid members. (P1)
- Replace 3 raw Prisma queries with get_team_object / get_team_membership /
get_user_object so the endpoint reuses the cache + retry layer the rest of
the proxy uses. (P2)
- Allow internal_user role to reach /team/{team_id}/members/me by adding the
route to LiteLLMRoutes.self_managed_routes (the handler already enforces
member-of-team access).
- Return null from the UI fetch on 404 instead of throwing, so a proxy admin
who isn't a team member sees the existing empty state rather than an error
string in the always-visible tab. (P2)
- Move the fetch out of networking.tsx into a colocated React Query hook
(useMyTeamMember) next to MyUserTab; TeamInfo now passes only teamId.
- Tooltip + empty-state copy on Model Scope: drop "(all team models)"
parenthetical and the redundant tooltip line.
- Tests: build real LiteLLM_TeamMembership / LiteLLM_BudgetTableFull
fixtures (with created_at) so the Pydantic Union resolves to the Full
variant; add an assertion that budget_reset_at survives end-to-end; add a
test for the email-only member match path.
* feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants)
Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet
shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page
is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the
established precedent for azure/gpt-5.4* (which were in the cost map
before the Azure rollout) so cost tracking and capability flags work
the moment customers deploy.
Schema follows the existing azure/gpt-5.4* shape:
- Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat,
$60/$360 pro per 1M, with priority tier 2x base
- Azure variants drop the flex/batches keys (Azure has no flex tier)
but keep priority pricing, matching gpt-5.4* precedent
- mode=chat for the thinking model, mode=responses for pro
reasoning_effort capability flags mirror the OpenAI variants exactly
since Azure proxies the same API contract: minimal rejection on both
chat and pro, low/none rejection on pro. Once #26456 (which sets
supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*)
lands, OpenAI and Azure flag profiles align.
Tests pin entry presence + pricing for all four Azure variants and
verify the live-API-derived reasoning_effort flags.
* test: register supports_low_reasoning_effort in cost-map JSON schema
azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch
carry supports_low_reasoning_effort=false. The strict
'additionalProperties: false' schema in
test_aaamodel_prices_and_context_window_json_is_valid rejected the new
key. Register it alongside the other supports_*_reasoning_effort
entries.
Note: the runtime side of this flag (code that reads it) lands in
#26456. Until that PR merges the flag is inert for both Azure and
OpenAI pro entries, but having the schema accept it lets cost-map
tests pass on either merge order.
Adds a new "My User" tab on the team detail page (between Overview and
Virtual Keys) so non-admin team members can see their own spend, budget,
budget reset date, rate limits, model scope, and team role.
Backend
- New `GET /team/{team_id}/members/me` endpoint that resolves the caller
from the API key and returns only their own LiteLLM_TeamMembership row
plus minimal team context (alias, role, email). Returns 404 if the
caller is not a member of the team. Avoids exposing other members'
data, which would happen if we filtered `/team/info` client-side.
- New `TeamMemberInfoResponse` Pydantic model.
Frontend
- New `MyUserTab` component (antd) — read-only summary cards.
- New `teamMemberMeCall` helper in networking.tsx.
- Tab is visible to all team members (including non-admins).
When litellm_credential_name resolves to no values (name not registered
in litellm.credential_list), bail out before touching the resolved
config. Prevents stripping a caller-supplied api_base in the no-match
case.
Mirrors the existing vertex_credentials handling for the newer
vertex_ai_credentials field, and extends the dynamic masker's sensitive
pattern set to recognize the plural form so other plural-named credential
fields are also covered.
When a vector_store config references a stored credential by name,
overwrite caller-supplied values on the same keys with the credential
values, and drop api_base from the resolved config when the credential
does not define one. Keeps the api_key / api_base pair consistent with
the credential definition.
Bring `/server/oauth/{server_id}/authorize`, `/token`, and `/register`
in line with `fetch_mcp_server`: the helper that resolves the server now
also applies the per-caller access policy. Admin-view callers are
unrestricted; non-admins must have the server in their allowed-servers
set; servers resolved from the admin-only `/server/oauth/session`
temporary cache reject non-admins.
dimensions=1 was silently dropped before #24415 wired the param through
to Vertex. Now that it's forwarded, Vertex rejects it (valid values:
128, 256, 512, 1408). Switch the test to a valid dimension.
