MCP servers created through the UI are persisted to the database independent of
store_model_in_db, but the in-memory registry that GET /v1/mcp/server reads was
hydrated from the database only through add_deployment, which runs solely when
store_model_in_db is True. On a DB-backed single-instance proxy with
store_model_in_db unset the registry started empty after a restart, so the MCP
Servers page showed nothing until an add or edit triggered a reload.
Hydrate the registry from the database on startup regardless of store_model_in_db
via a new ProxyConfig.init_mcp_servers_from_db, honoring supported_db_objects.
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
Include top-level scalar fields from standard_logging_metadata in the
combined metadata dict used by custom_prometheus_metadata_labels. Previously
only nested sub-dicts (requester_metadata, user_api_key_auth_metadata,
spend_logs_metadata) were spread into combined_metadata, so fields like
user_api_key_project_alias were inaccessible and always resolved to None.
Co-authored-by: unknown <>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): audit remaining system-wide settings updates
Extends the audit logging framework introduced in the parent PR to the
rest of the LiteLLM_Config writers and the two adjacent settings tables:
/config/update (general, environment_variables, litellm_settings,
router_settings sections), /config/field/update, /config/field/delete,
/config/callback/delete, /update/default_team_settings,
/update/mcp_semantic_filter_settings, /add/allowed_ip,
/delete/allowed_ip, /update/sso_settings, /update/ui_theme_settings,
/update/ui_settings.
Each writer records the actor, action, the affected config section, and
a redacted before/after snapshot. SSO and UI settings rows use their own
table_name (LiteLLM_SSOConfig, LiteLLM_UISettings). The /config/callback
and /update/sso_settings audits fire BEFORE the proxy reload and the env
cleanup step respectively, so a failure in either leaves the audit row
intact.
The audit-actor parameter on _update_litellm_setting is now required
rather than optional; the chokepoint covers default_team and
mcp_semantic_filter for free, and a future caller that forgets the
actor fails loudly instead of silently skipping the audit. The two
direct-calling tests pass a dummy actor.
The environment_variables section redacts every value rather than
relying on key-name matching, because it carries credentials under
non-secret-looking uppercase keys (e.g. DATABASE_URL).
* fix(proxy): capture redacted SSO before-snapshot in audit log
Greptile review of #31754 flagged update_sso_settings as the one endpoint
where before_value is permanently None, so the LiteLLM_SSOConfig audit
trail has no pre-change state. An auditor reviewing a secret-rotation
event could see what the SSO settings were changed to but not what they
were before.
Read the existing SSO row before the upsert, decrypt it via
proxy_config._decrypt_db_variables, and pass it as before_value.
create_config_audit_log's secret-name redaction then masks the
*_client_secret fields, so neither the old nor the new plaintext secret
lands in the audit row.
Add a regression test asserting the before-snapshot reflects the
pre-change values for non-secret fields (google_client_id) and is
redacted for secret fields (google_client_secret). Mutation-checked
against reverting to before_value=None.
The pre-existing SSO tests now also mock litellm_ssoconfig.find_unique
since the endpoint reads it; the read returns None for tests that do not
care about the before-state.
* fix: remove committed zero init migration
* refactor(proxy): audit config writes via asyncio.create_task everywhere
PR A's chokepoint audit call was refactored from a blocking await to
asyncio.create_task so that a post-save audit-log failure could not
surface as a 500 to the caller. The 12 other audit call sites added in
this PR were still using await, reintroducing the exact 500-after-commit
exposure at every sibling endpoint. Wrap them all in asyncio.create_task
to match the model_management_endpoints / key_management_endpoints /
hooks / config_override_endpoints / team_callback_endpoints /
cache_settings_endpoints house pattern, so the codebase tells one story.
The two direct-invocation tests (test_update_config_general_settings and
test_delete_config_general_settings, which call the handler in-process
rather than via TestClient) yield with `await asyncio.sleep(0)` after the
handler returns so the scheduled audit task runs before the assertion.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: add cache control injection support for v1/messages endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: normalize string content to list for Anthropic-native cache_control injection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: simplify cache control injection, fix system=[] bug, fix handler system type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: extract cache control logic into static helper on AnthropicCacheControlHook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails/headroom): add CCR (compress-cache-retrieve) support via agentic loop
When Headroom's /v1/compress returns messages containing hash markers
(hash=[a-f0-9]{24}), inject a headroom_retrieve tool into the request.
When the LLM calls that tool, intercept via async_should_run_agentic_loop
and async_build_agentic_loop_plan, call GET /v1/retrieve/{hash} on the
Headroom sidecar, and replay the LLM with the original content as a tool
result -- all transparent to the caller.
* style: run ruff format on headroom guardrail and tests
* fix(guardrails/headroom): detect headroom_retrieve calls in both OpenAI and Anthropic response formats
* test(guardrails/headroom): add test for Anthropic content block format detection in CCR loop
* ci: trigger CI checks
* fix(guardrails/headroom): replace List/Dict with list/dict to fix UP006 ruff violations
* fix(guardrails/headroom): replace except Exception with except ValueError to fix BLE001
* fix(guardrails/headroom): add Responses API output format detection for CCR tool calls
* refactor(guardrails/headroom): extract format-specific helpers to fix C901 complexity
* fix(guardrails/headroom): scope CCR retrieval to hashes produced by current request
Previously any LLM-supplied hash in a headroom_retrieve tool call was
forwarded to the Headroom retrieve API, letting a crafted tool call
fetch arbitrary cached content. Validate the hash against the set
produced by compressing the current request's messages before calling
retrieve.
* fix(guardrails/headroom): track issued hashes server-side, fix Responses API replay shape
Hash validation now also checks an in-memory cache of hashes actually
returned by /v1/compress, not just whether the hash text appears
somewhere in the request's messages. The message-text check alone is
forgeable: an attacker can plant a hash-shaped string in their own
prompt and have it treated as valid.
Responses API follow-up now emits function_call/function_call_output
items keyed by call_id instead of chat-style assistant/tool messages,
since the Responses API does not accept the latter as input. Also
fixes call_id/id field priority when extracting tool calls from
Responses API output, since call_id (not id) is what must match
between the function_call and its output.
* fix(guardrails/headroom): drop redundant quoted type annotations
UP037 flags quotes on annotations that are already lazily evaluated
via `from __future__ import annotations`.
* test(guardrails/headroom): add missing pytest.mark.asyncio decorators
Functional under asyncio_mode=auto, but every other async test in the
file has the decorator for consistency.
* fix(guardrails/headroom): scope CCR hashes per call_id, fix Anthropic replay shape
Two real gaps found in review:
1. The instance-wide issued-hash cache combined with a message-text
check did not actually scope retrieval to the request that produced
the hash. A hash issued for request A stays in the shared cache
until TTL expiry, and the message-text check is satisfied by any
request whose own messages happen to echo that hash string. Request
B could plant A's hash in its own prompt and retrieve A's content.
Fixed by keying the issued-hash cache by litellm_call_id, matching
the pattern already used in compression_interception: a hash is
only honored when it was issued under the exact call_id resolving
for the current request.
2. The Anthropic Messages replay path fell through to the chat-style
assistant/tool-message builder, which Anthropic does not accept.
Anthropic requires the tool_use block echoed in an assistant message
paired with a tool_result block in a user message, keyed by
tool_use_id. Added a dedicated branch for this shape.
* docs: note proactive API-fragmentation helper convention
Add a bullet to the coding-conventions list: look for or add a shared
helper when logic branches on API surface (chat completions vs
Anthropic Messages vs Responses API), instead of duplicating
format-detection per module.
* fix(guardrails/headroom): fix Anthropic tool-shape detection, extract shared cross-API tool util
Live e2e testing against the real Anthropic API surfaced two bugs the
mocked unit tests couldn't catch because they used MagicMock responses
instead of realistic response shapes:
1. has_headroom_retrieve_tool only recognized OpenAI-shaped function
tools. By the time an Anthropic Messages response reaches the
agentic-loop gate, the tool this guardrail injected has already been
transformed into Anthropic's native shape (type: "custom", top-level
"name"), so the gate never fired for real Anthropic requests.
2. AnthropicMessagesResponse is a TypedDict, so real responses are
plain dicts at runtime, not objects with attribute access. The
extractors and format detectors used bare getattr(), which silently
returns nothing for dict responses instead of reading the actual
key.
Extracted the cross-API-surface tool-call extraction and tool-presence
check into litellm/litellm_core_utils/prompt_templates/factory.py
(get_tool_calls_from_response, has_tool_with_name) so this format
fragmentation is handled in one place instead of being duplicated
per-guardrail, and reused the existing repair-aware
parse_tool_call_arguments from common_utils instead of a naive
json.loads. headroom.py now delegates to these shared helpers.
Confirmed live against the real Anthropic API: the retrieve loop now
fires and successfully retrieves the correct hash's content through
the full compress -> tool-call -> retrieve -> replay round-trip.
* fix(guardrails/headroom): fix ruff-strict UP006/I001 budget violations
Use lowercase list/dict generics in the new factory.py tool-call
helpers instead of typing.List/Dict, drop the now-unused Tuple import
in headroom.py, and reorder the new factory import ahead of the
llms.custom_httpx import to satisfy import sorting.
* fix(guardrails/headroom): match Anthropic tools without a type field
Anthropic's documented client tool format is just name + input_schema;
type: "custom" is only one possible value, not a requirement. Match
any non-OpenAI-shaped tool on its top-level name instead of requiring
type == "custom".
* feat(proxy): support object_permission in default_key_generate_params
default_key_generate_params filled in a fixed whitelist of scalar fields
plus a full-replace for models/metadata, but never touched object_permission,
so admins had no way to set a default (e.g. mcp_tool_search_enabled,
vector_stores) applied to every new key. Merge object_permission field-by-field
instead of replacing it wholesale, so a caller-supplied field (e.g. mcp_servers)
is preserved alongside defaulted fields the caller left unset.
* ci: retrigger proxy_pass_through_endpoint_tests (suspected flake, unrelated to this PR's diff)
* fix(proxy): apply default object_permission after team-scope validation
Injecting the default before validate_key_vector_stores_against_team /
validate_key_search_tools_against_team ran meant a default containing a
team-scoped field (e.g. vector_stores) looked like a caller-requested
permission, turning ordinary non-admin personal key creation into a 403.
Merge the default into data_json after those checks instead, and guard
against a non-dict default value.
* feat(sandbox): reuse e2b container across requests when metadata.session_id is set
When a client passes `metadata.session_id` in a /chat/completions request
alongside a code_interpreter tool, the proxy now routes all requests sharing
that session_id to the same sandbox container. State (variables, imports,
installed packages) persists across requests within the session.
Without a session_id the existing ephemeral behavior is unchanged: one
container per agentic loop, deleted immediately after.
The sandbox key is derived from session_id rather than a per-request UUID.
The cleanup and post-loop hooks skip deletion for session-scoped containers.
TTL-based pruning (15 min idle) still applies and refreshes on every use,
so an active session never expires mid-use. The session_id-scoped key is
registered in all_litellm_params and the proxy strip-list so it never
leaks to the upstream LLM provider.
* fix(sandbox): scope session sandbox key to API key identity; add per-identity LRU cap
Two security issues addressed:
1. Cross-user sandbox isolation: the session_id supplied by the client is now
combined with the server-minted user_api_key_hash to form the cache key
(format: "{hash}:{session_id}" when authenticated, bare session_id for
non-proxy use). Two tenants sharing the same session_id no longer share a
sandbox.
2. Bounded session allocation: each API key identity is capped at
_SESSION_SCOPED_PER_IDENTITY_CAP (10) live session-scoped containers. When
a new session is opened beyond the cap, the least-recently-used entry for
that identity is evicted and its sandbox deleted, preventing unbounded
accumulation via rotating session IDs.
The container cache tuple gains a fourth element (identity: str | None) so
eviction can filter by identity without parsing key formats. Tests added for
both properties.
* fix(websearch): wire chat completion agentic loop to correct hooks
maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller.
Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path.
Regression test added.
* fix(websearch): strip tool_choice from follow-up request
When the original request forces tool_choice to litellm_web_search,
the follow-up request after search execution inherited that tool_choice,
causing the model to call the search tool again instead of synthesizing
an answer from the results.
* fix(websearch): inject api_key into agentic hook kwargs for anthropic messages
Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's
synthesis call after executing Exa/Perplexity searches) were missing api_key
because the named api_key param in async_anthropic_messages_handler was never
merged into the kwargs dict forwarded downstream. Result: every /v1/messages
websearch follow-up failed with "x-api-key header is required" and the caller
received the raw tool_use response instead of the synthesized answer.
* ci: trigger CI run
* fix(websearch): support unified agentic hooks alongside chat-completion-specific hooks
CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with
_agentic_loop_api_surface to handle both surfaces from one hook. The chat
completion loop must also check _gate_overridden so callbacks using the
unified hook pattern still fire for chat completions.
* fix(websearch): strip tool_choice from legacy chat completion follow-up call
The _execute_chat_completion_agentic_loop path merged original optional_params
(which includes forced tool_choice) into follow-up params without explicit
removal. _build_chat_completion_request_patch already excluded tool_choice from
its optional_params output, but dict.update() with a missing key leaves the
original value intact. Explicit pop after the merge removes it.
* fix(websearch): always strip tool_choice from plan-path follow-up params
The tool_choice removal was gated on patch.tools is not None. WebSearch sets
tools via patch.optional_params not patch.tools, so the gate was False and
forced tool_choice from the original request survived into the synthesis call.
Move the pop outside the patch.tools branch so it applies unconditionally.
* feat(proxy): audit default user settings updates
Adds audit logging for the customer-impacting path: PATCH
/update/internal_user_settings, which is what the admin dashboard hits
when an admin changes Default User Settings and which today leaves no
record of who changed what.
Introduces the small framework that future system-wide settings audits
will share: a CONFIG_TABLE_NAME enum value, a create_config_audit_log
helper that reuses the existing create_object_audit_log path (so the
enterprise gate and store_audit_logs flag still apply), and a
_dump_redacted_config helper that strips secret leaves before the row is
written using the same matcher /config/field/info applies for non-admins.
The helper handles environment_variables as a special case where every
value is redacted, since that section carries credentials under
non-secret-looking uppercase keys (e.g. DATABASE_URL).
Only update_internal_user_settings is wired up in this change. Coverage
for the other LiteLLM_Config writers (/config/update sections,
/config/field/update, /config/field/delete, /config/callback/delete,
default_team_settings, mcp_semantic_filter, allowed_ip, sso_settings,
ui_theme, ui_settings) is intentionally a follow-up so each can be
verified live against the credential-bearing fields it actually carries.
The audit-actor parameter on _update_litellm_setting is optional today so
non-audited callers keep working unchanged; the follow-up will make it
required once every caller is wired up.
* fix(proxy): make audit-log call non-blocking and serializer defensive
Greptile review of #31753 surfaced three robustness issues with the
audit-log call path. The settings change always commits; these fixes
prevent post-commit audit failures from surfacing as 500 responses.
Switch the audit-log call in _update_litellm_setting from a blocking
await to asyncio.create_task, matching the create_object_audit_log
pattern every other call site uses (model_management_endpoints etc.).
A transient prisma blip or a JSON serialization error in the audit row
no longer turns a successful save_config into a 500 the caller sees.
Add default=str to both json.dumps calls in _dump_redacted_config so a
YAML-loaded value with a non-JSON-native leaf (datetime, custom object)
serializes cleanly. The sibling audit-log serializers in
team_endpoints.py already pass default=str for the same reason.
Tighten the redact_all_values branch to redact wholesale for non-dict
inputs rather than silently falling through to the key-name matcher;
defensive against a future change that stores a section as a list or
scalar.
Each fix has a regression test mutation-checked against reverting the
fix.
* refactor(proxy): drop unreachable non-dict redact_all_values branch
The defensive non-dict fallback in _dump_redacted_config emitted
json.dumps("REDACTED") which, if ever hit, would crash LiteLLM_AuditLogs
construction (mask_api_keys validator calls json.loads on the already-
parsed bare string). Reachability is zero: redact_all_values is True
only for param_name=="environment_variables", which is always a dict.
Delete the dead branch and its test rather than ship provably-wrong
defensive code with a test that green-lights it.
* feat(ui): add provider auth editing to the model edit view
Provider API keys / auth could previously only be changed by hand-editing
the raw litellm_params JSON, so there was no first-class way to rotate a
model's key. Adds an Authentication section that renders the correct
provider-specific fields (reusing ProviderSpecificFields) keyed off the
model's custom_llm_provider; fields are blank ("leave blank to keep
current") so untouched secrets are preserved and only entered values are
PATCHed and encrypted at rest.
Resolves LIT-3169
* refactor(ui): simplify model auth editing; fix stale credential branch
Drop the onFieldsResolved/authFieldKeys round trip: the parent now resolves
provider auth field keys itself via the new useProviderAuthFieldKeys hook
(same metadata ProviderSpecificFields renders), removing the report-up effect
and its stable-reference footgun. ProviderSpecificFields keeps only
excludeKeys (real need: suppress duplicate visible inputs).
Fix the stale Authentication branch: derive it from the live
litellm_credential_name form value (Form.useWatch) instead of the server
snapshot, so clearing/adding a credential mid-edit shows the right UI. Also
skip inline auth updates entirely when a named credential is selected, so we
never submit a credential name and raw inline auth together.
* fix(ui): don't leak freshly-entered model auth secrets to display/console
The auth values a user types are still sent in the PATCH request, but:
- strip them from the locally-stored litellm_params after save so the
read-only LiteLLM Params JSON doesn't render the plaintext key
- remove the debug console.log in modelPatchUpdateCall that dumped the
full update payload (incl. api_key / vertex_credentials) to the browser
console on every model update
Backend stores these encrypted and returns them masked on refetch.
* fix(ui): don't require blank auth fields in model edit context
Auth fields render blank ('leave blank to keep'), but required metadata
(e.g. OpenAI api_key) added a required validation rule that blocked
onFinish entirely — making it impossible to save any unrelated edit
without re-entering the secret. Add a disableRequired prop to
ProviderSpecificFields and set it in the model edit Authentication section.
* fix(ui): rotate model credentials in a dedicated modal so a normal save can't overwrite secrets
The model edit form seeded the read-only LiteLLM Params textarea with the whole
litellm_params blob and re-sent all of it on every save. Because /model/info
redacts secrets by masking them ("azur****BBCC") rather than removing them, any
save re-encrypted the asterisk mask over the real value and silently destroyed
credentials such as azure_ad_token, aws_session_token, watsonx token/zen_api_key
and the OCI key fields. api_key, client_secret, vertex_credentials and the AWS
access/secret keys were safe only because the backend strips those entirely
Credential rotation now lives in a dedicated UpdateModelCredentialsModal that
PATCHes only the fields the user types, decoupled from the params blob; the
backend already merges partial litellm_params, so the rest of the deployment is
left untouched. The general edit form drops masked values from both the textarea
seed and the outbound payload, so a normal save can never carry a redacted secret
Also removes the now-unused inline auth section and its excludeKeys and
useProviderAuthFieldKeys plumbing, strips secret-leaking console.logs from the
provider upload handler and the model-update response, and fixes a
react-hooks/use-memo error that was failing the frontend-lint CI job
* chore(ui): ratchet no-explicit-any lint metric to 2013
Removing the credential-echoing console.log (and its info: any param) from the
provider upload handler dropped the tracked count by one; update the committed
baseline so the Check lint budgets CI step is not stale
* refactor(ui): scope the model credential modal to api-key rotation only
Narrows UpdateModelCredentialsModal to a single API Key field. On submit it
PATCHes only { api_key }, so the backend merge leaves every other deployment
param untouched; a model authed via azure_ad_token, AWS keys, or a Vertex JSON
won't have anything to rotate here yet, which is the intended scope for now.
Drops the multi-field provider rendering this added earlier, which also removes
the now-unused disableRequired prop from ProviderSpecificFields and reverts that
shared component to its prior shape. The "Update API Key" trigger button is now
an antd Button rather than a TremorButton, so the feature introduces no tremor.
* refactor(ui): convert the model detail toolbar buttons from tremor to antd
Switches Test Connection, Re-use Credentials and Delete Model to antd Button so
the toolbar matches the Update API Key button and no longer mixes libraries;
Delete Model uses antd's danger styling instead of hand-rolled red classes
* style(ui): make the api-key modal submit button primary and drop the Need Help link
Two NVIDIA-Riva-specific fields consumed by the audio-transcription
handler via the provider's `optional_params` passthrough were not
covered by the proxy's existing banned-request-body list or the
admin-config clearing list applied on `api_base` BYOK override:
* `nvcf_function_id`
* `use_ssl`
Add both to `_BANNED_REQUEST_BODY_PARAMS` in
`litellm/proxy/auth/auth_utils.py` and to the kwargs-only list in
`_admin_config_fields_to_clear_on_base_override()` in
`litellm/router_utils/clientside_credential_handler.py`, next to the
analogous provider-specific entries already there (`aws_bedrock_*`,
OCI provider fields, etc.). Same admin opt-ins as every other entry
on those lists (`general_settings.allow_client_side_credentials`
proxy-wide, or `configurable_clientside_auth_params` per deployment).
Regression tests in `tests/test_litellm/proxy/auth/test_auth_utils.py`
cover root-level rejection, the historical `api_key` bypass, both
admin opt-in paths (proxy-wide and per-deployment), nested-container
smuggling via the existing recursive walk, and clearing on
`api_base` override. Mutation check verified.
Resolves VERIA-493
* test(pass-through): de-flake vertex spend-log assertion by re-billing
The vertex pass-through spend-log test asserted that a single billed
generateContent call moved the global spend aggregate within a fixed
wait. CI failures show the call returning a valid response with real
usage, yet spend never increasing over a 240s poll.
Pass-through spend logging is best-effort: the success handler is
enqueued on a background worker that can drop or time out an individual
event under load and never retries it, so one billed call occasionally
never reaches LiteLLM_SpendLogs. Waiting longer cannot recover a dropped
event; only re-issuing the call can.
Re-bill the call up to a few times and require at least one to be
tracked, mirroring the sibling jest test that already retries. The test
still fails hard if cost tracking is actually broken, since then every
call records nothing. Also sum spend across all returned days instead of
matching the runner's local 'today', removing a separate UTC-rollover
flake.
* test(pass-through): route vertex spend-log test through proxy via direct HTTP
The vertexai SDK, configured with location="global" and an http api_endpoint
override, intermittently sends generateContent to the public Vertex endpoint
instead of the proxy. Proxy logs from a failing run show all 46 of the test's
own spend-log polls reaching the proxy while zero generateContent calls did, so
LiteLLM never saw the billed call and no spend was ever recorded; re-billing
through the SDK could not help because every retry bypassed the proxy too.
Issue the pass-through request directly over HTTP so it always hits the proxy,
minting a Google token from the same service-account credentials, then assert
that the specific call's own spend log lands with spend > 0, a gemini model, and
custom_llm_provider vertex_ai. A small best-effort retry covers the rare case
where the background logging worker drops a single event; failing every attempt
still fails hard so the test keeps its teeth if cost tracking breaks.
* test(pass-through): reuse LITE_LLM_ENDPOINT and drop needless async in get_tracked_spend
* ci(codspeed): pin benchmark runner to ubuntu-24.04
ubuntu-latest resolves to different runner images between the BASE
(main/staging) and HEAD (PR) runs, so CodSpeed reports 'Different
runtime environments detected' and emits false-positive regressions
(e.g. a -25.2% swing on test_completion_multi_turn in #31684, an MCP
auth fix with no LLM code changes). Pinning the runner to a fixed
image keeps BASE and HEAD on the same hardware so 1 ms swings on a
~3 ms benchmark stop blocking unrelated PRs.
Fixes#31738
* ci(codspeed): stop running benchmarks on litellm_internal_staging
The CodSpeed check flip-flops on internal staging and on PRs targeting
it (e.g. "+11.75% improvement" on one run, "-25.36% regression" on the
next) because the comparison flags "different runtime environments" and
the benchmarks are only 3-4 ms, so sub-millisecond runner noise swings
the result by 25-30%. Pinning the runner to ubuntu-24.04 in this PR
helps the head side, but the internal_staging base is still recorded on
the old unpinned runner, so comparisons keep flapping until the pin
merges and the base is re-baselined.
Until that settles, the red X's on internal staging make the OSS
project look unhealthy and confuse contributors, so drop the
litellm_internal_staging push and pull_request triggers and keep
CodSpeed running on main only.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(token_counter): count legacy function_call.arguments (VERIA-492)
token_counter handled the modern assistant tool_calls field but had no
branch for the legacy OpenAI function_call payload. The value is a dict,
so it skipped every special-cased branch in _count_messages and fell
through to the unsupported-key continue, letting arbitrary text in
function_call.arguments slip past the count.
Resolves VERIA-492
* refactor(token_counter): raise on unexpected key in _count_function_call_tokens
Address Greptile P2: the helper's fallback branch previously applied
function_call logic to any key that wasn't tool_calls. Make the contract
explicit so a future caller can't silently miscount.
The Presidio streaming post-call hooks (_stream_apply_output_masking for
apply_to_output and _stream_pii_unmasking for output_parse_pii) collected every
upstream chunk, reassembled the full completion with stream_chunk_builder at
end-of-stream, ran Presidio over it, then emitted one reconstructed SSE chunk.
Time-to-first-token collapsed to the total generation time and token-by-token
streaming was lost whenever Presidio output handling was enabled. With the
default presidio_filter_scope both, an apply_to_output masking instance is always
created, so even the unmask configuration buffered the stream.
Both paths now transform and forward chunks as they arrive. The unmask path
replaces placeholder tokens per chunk, holding back only the trailing run that
could still grow into a token so a placeholder split across SSE chunks
(<PER + SON_1>) is still rewritten atomically. The mask path emits a prefix only
when masking it in isolation matches the corresponding prefix of masking the
whole buffer, with a lookahead margin still buffered past the cut, so an entity
straddling the cut is detected and held until complete; past
_PRESIDIO_STREAM_MAX_BUFFER the run is bounded without splitting an entity.
Tool-call and legacy function-call argument fragments are accumulated per choice
and transformed once the choice closes, content is buffered independently per
choice index for correct n>1 streaming, raw Anthropic SSE bytes and /v1/responses
events pass through with any held content flushed first so events never reorder,
and a masking error redacts only the affected chunk (fail closed, keeping
finish_reason) while the stream continues.
Resolves LIT-3222
RealTimeStreaming.log_messages dispatched the success handler with a bare
asyncio.create_task, bypassing GLOBAL_LOGGING_WORKER (which gives a per-coroutine
timeout and a concurrency cap). On a long-lived realtime websocket a slow logging
callback left one suspended task per logged turn, each pinning that turn's
assembled response, accumulating without bound (~12-15k in-flight under load in a
repro) until OOM. Route realtime success logging through the bounded worker so
in-flight logging is capped and a hung callback is cancelled at the worker
timeout.
The chat and responses streaming success-logging paths are intentionally left
unchanged: their success callbacks must complete within the call's event-loop run
(the non-streaming path pairs the worker with a synchronous callback; the
streaming path has no such companion), so deferring them through the worker would
drop logs for one-shot SDK calls and breaks test_async_custom_handler_stream.
Bounding those paths needs a load-shedding approach and is left to a follow-up.
The e2e runs against the real proxy, so a contract drift already fails the test at
runtime; tsc only checks the spec against schema.d.ts, a generated snapshot, so a
backend change with a stale snapshot would pass tsc while the live test still
catches it. The dedicated tsconfig + script + CI step were circular ceremony for
that. Keep the zero-runtime-cost type-only import, which still catches mistakes in
the editor, and make its comment honest about what enforces the contract.
When redis_startup_nodes is set the async cluster client was built with no health check and no TCP keepalive, so a connection silently dropped by a cluster restart (e.g. ElastiCache Serverless maintenance) stayed in the pool and got reused while dead; the first command after the restart stalled in re-initialization until the LoggingWorker timeout cancelled it, surfacing as CancelledError then TimeoutError on the spend-counter path
Build the async cluster client with a 25s health_check_interval and socket_keepalive so an idle connection is PING-validated and reconnected before reuse, and expose both through the cluster kwarg allow-list so an explicit value from config still wins
Resolves LIT-4083
update_spend_logs flushes the queue with a single create_many per batch, so one
row carrying bytes Postgres refuses (a residual NUL byte is the canonical case)
fails the entire insert and drops every good spend log alongside it. PR #29515
strips NUL bytes from the JSON columns, but the scalar string columns (end_user,
model, session_id, ...) still flow through unsanitized, so a poisoned row can
still reach the write and take a batch of up to 1000 good rows down with it.
On a genuine data-layer rejection the batch is now bisected so the good rows
still persist and only the offending row is dropped and logged with its
request_id. The classification lives in PrismaDBExceptionHandler.is_prisma_data_error
(matched by exact type so systemic subclasses like a missing table are not
mistaken for a single poison row), which keeps prisma an in-function import and
litellm.proxy.utils importable without the proxy extra. Transport failures,
including the "can't reach database server" outage that prisma mislabels as a
DataError, are re-raised unchanged so the existing connection-retry path still
runs and a transient outage never turns into silent per-row data loss.
The bisection carries a per-batch isolation budget so an authenticated caller
flooding poisoned rows cannot amplify one failed bulk insert into ~2N failed
inserts and N log lines; once the budget is spent the still-failing remainder
is dropped wholesale under a single log line.
Resolves LIT-4103
The Vertex AI and Azure AI Sonnet 5 entries carried supports_output_config:
true, which the gen-5 siblings (vertex_ai/claude-opus-4-8, azure_ai/claude-fable-5,
etc.) do not. The flag only feeds AnthropicConfig._model_supports_effort_param,
which already returns true for these entries via supports_xhigh/max_reasoning_effort,
so output_config.effort still forwards on both routes. Removing it is behavior
neutral and matches the existing per-platform convention for gen-5 Claude.
* feat(messages): passthrough /v1/messages to native endpoints via supported_endpoints
The unified /v1/messages proxy endpoint always translated inbound Anthropic
requests down to /v1/chat/completions (or the Responses API for openai) when the
deployment's provider lacked a native Anthropic-messages config, dropping
Anthropic-only features like cache_control and thinking. Some customers run
OpenAI-compatible servers (self-hosted vLLM, DeepSeek's Anthropic endpoint, etc.)
that also natively expose /v1/messages and want the raw Anthropic payload
forwarded untranslated, while keeping provider openai so /v1/chat/completions to
the same deployment stays native.
Opt in per deployment via model_info.supported_endpoints containing
/v1/messages. When present, the gate routes to a generic, provider-agnostic
OpenAILikeAnthropicMessagesConfig that POSTs the Anthropic payload to
{api_base}/v1/messages with Bearer auth, instead of translating. Default
behavior is unchanged. Generalizes and supersedes the hosted_vllm-only,
env-var-toggled PR #28745.
* fix(messages): preserve standard-cased caller headers in native passthrough
The OpenAI-like Anthropic passthrough config only checked for lowercase header
names before injecting Bearer auth, anthropic-version, and content-type
defaults. A caller sending standard-cased Authorization, Anthropic-Version, or
Content-Type was treated as missing those headers, so LiteLLM added duplicate
lowercase variants and overwrote the caller's credential/version at the HTTP
layer. Header presence is now checked case-insensitively and the merge no longer
mutates the caller dict.
Also moves the feature docs out of the main repo (docs live in litellm-docs).
* fix(openai_like/messages): delegate to parent transform and inject anthropic-beta headers
The passthrough config bypassed the parent transform and skipped header beta injection. Both gaps cause native /v1/messages features (context management, advisor tool, fast mode, structured outputs, reasoning_effort, advisor stripping) to silently degrade on opted-in deployments. Reuse the parent's pipeline and call _update_headers_with_anthropic_beta after merging defaults
* fix: normalize anthropic-beta header key case before beta injection
* style: collapse anthropic-beta header normalization to single line
ruff format --check requires the comprehension on one line (it fits within
the 120 char limit); fixes the lint job failure on the bugbot autofix commit
* fix(messages): forward anthropic-beta to native passthrough upstream
The shared anthropic_messages HTTP handler ran update_headers_with_filtered_beta
with the deployment's custom_llm_provider after validate. For the native
/v1/messages passthrough that provider is openai, which has no beta-header
mapping, so every anthropic-beta value (caller-supplied or feature-derived for
speed/context_management/etc.) was stripped to empty before the upstream
request, breaking beta passthrough to the Anthropic-compatible endpoint.
Beta filtering only makes sense on cross-provider translation paths where the
upstream cannot understand Anthropic betas. Gate it on a new
should_filter_anthropic_beta_headers() that defaults to True (bedrock, vertex_ai,
native anthropic unchanged) and is overridden to False by
OpenAILikeAnthropicMessagesConfig, whose upstream is a native Anthropic endpoint,
so betas pass through verbatim.
* chore: remove accidentally committed local QA logs and config
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Address an adversarial review of the Loadbalancing e2e:
- The "typed against the backend schema" claim was hollow: nothing type-checked
e2e_tests (the root tsconfig excludes it and no CI step runs tsc), so a
contract drift would compile and run unchanged. Add e2e_tests/tsconfig.json, a
typecheck:e2e script, and a CircleCI step so the schema typing actually gates.
- The two describe blocks both mutate the proxy's shared router_settings, and the
Loadbalancing save echoes the whole settings object, so they could clobber each
other under local fullyParallel. Run the file serially.
- patchRouterSettings swallowed a failed seed, which surfaced later as a
misleading UI timeout. Assert the write succeeded, and rely on the server-side
merge instead of echoing the whole settings object back (drops a cast and a GET).
- Empty routing_groups already reproduces the bug, so drop the non-empty seed and
its model coupling.
* fix(mcp): stop one unauthenticated server from emptying the aggregate tools/list
On the aggregate MCP route (/mcp), the gateway fans out to every server the caller can access and
flattens their tools. _fetch_and_filter_server_tools re-raises MCPUpstreamAuthError unconditionally
(added with the OAuth passthrough feature in #28356) so it surfaces a 401 on single-server routes,
but on the aggregate route that exception propagates through the asyncio.gather fan-out and the
outer handler turns it into an empty list. The result: a single delegate/passthrough OAuth server
the user has not authenticated (e.g. a delegate-auth server) zeroes the tools of every other server,
including the ones that resolve fine, so the client connects and sees no tools.
Surface the upstream auth error only when a single server was explicitly targeted (so that route
still drives the upstream OAuth flow); across the aggregate, absorb it to [] for that one server so
the rest still list their tools. This restores the graceful per-server degradation that predated
#28356.
Adds regression tests: the aggregate keeps a healthy server's tools when a sibling raises
MCPUpstreamAuthError, and a single-server listing still surfaces it.
* fix(mcp): decide aggregate vs single-server listing by route scope, not server count
Addresses review: keying the surface-vs-absorb decision off the server count (len(allowed_mcp_servers),
and even len(mcp_servers)) misclassifies an aggregate /mcp request from a key that can access exactly
one server as a targeted single-server listing, so that one server's MCPUpstreamAuthError re-raises and
empties the aggregate again for one-server permission sets.
Use the path-derived single-server scope instead: _mcp_gateway_server_name, set by
_gateway_initialize_instructions_request_scope only when the request path names exactly one upstream
server (/<server>/mcp) and never from client headers, is None on the aggregate route (/mcp) regardless
of how many servers the key can access. Single-server routes still surface the upstream-auth challenge;
the aggregate absorbs it per server.
Adds a regression test that an aggregate request with a single accessible server still absorbs, plus
renames the single-server test to drive the route scope explicitly. The new test fails on the
count-based logic.
* fixing aggregation error
* style(mcp): collapse single-line debug log to satisfy ruff format
Drives the real save flow against a live proxy: seeds a present routing_groups
array (the LIT-4057 trigger) via the typed /config/update contract, changes
num_retries on the Loadbalancing tab, and asserts the POST returns 200 instead
of 422, the success toast appears, and the value still shows after a reload (the
ticket's "refresh shows old values" symptom). The round-trip is typed against the
OpenAPI-generated backend schema (ConfigYAML write, RouterSettingsResponse read)
through a type-only import, so a backend contract drift fails the type check.
Register claude-sonnet-5 across the Anthropic, Bedrock (base + global/us/eu/au/jp
cross-region inference profiles), Vertex AI, and Azure AI cost-map entries in both
the root and bundled-backup model maps, plus BEDROCK_CONVERSE_MODELS and the
setup-wizard provider list.
Sonnet 5 ships with the gen-5 adaptive-thinking profile (adaptive thinking always
on, no extended thinking, effort defaults to high), so the entries mirror the
Fable 5 / Opus 4.8 sampling-param and prefill restrictions rather than the older
Sonnet 4.6 behavior: supports_sampling_params and supports_assistant_prefill are
false while supports_adaptive_thinking, supports_xhigh_reasoning_effort, and
supports_max_reasoning_effort are true. Pricing follows standard Sonnet rates
($3 / $15 per MTok) with the 10% regional premium on the us/eu/au/jp profiles.
Add a reasoning-effort grid entry for the Anthropic direct route and a regression
test pinning pricing, capabilities, regional premiums, backup parity, and bare-name
provider resolution.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Address Greptile P2: the routing_groups test read setCallbacksCall.mock.calls[0][1]
immediately after the now-async save handler, so any latency in the mock would throw
an opaque TypeError instead of a clean assertion failure. Assert through
toHaveBeenCalledWith inside waitFor with expect.not.objectContaining, dropping the
index access and the cast. Also drop the ticket id from the test names.
The Loadbalancing tab rendered routing_groups as a generic text input and
sent its array value back as the JSON string "[]", which fails Pydantic
list validation on POST /config/update and returns 422. routing_groups has
its own dedicated Routing Groups tab, so this tab must neither render nor
write it; exclude it the same way retry_policy and model_group_retry_policy
are excluded for the Model Retry Settings tab.
The save was also fire-and-forget: setCallbacksCall was not awaited, so the
rejected promise escaped the try/catch and the success toast fired
unconditionally, showing success even when the backend rejected the change.
Await the call, gate the success toast on resolution, and surface the error.
Adds `!` prefix negation to tag-based routing so callers can exclude deployments by exact tag value without enumerating every allowed alternative. `!provider:anthropic` removes all deployments tagged exactly `provider:anthropic` before routing, and positive and negation tags compose. Matching is exact literal membership (frozenset intersection), so there is no regex or ReDoS surface for client-supplied tags. Ban-only requests that carry only negation tags stay within the default pool, mirroring untagged-request semantics so callers can't use negation to escape it. Fallback chains keep working because get_deployments_for_tag runs on each routing hop
Copy of #31680; implementation credit to @deepanshululla
Co-authored-by: deepanshululla <15312873+deepanshululla@users.noreply.github.com>
* feat(guardrails): expose streaming knobs on generic_guardrail_api
Wire streaming_end_of_stream_only and streaming_sampling_rate through
optional params, initialize_guardrail, and get_config_model so the
generic guardrail API participates in UnifiedLLMGuardrails streaming
checks with configurable cadence and end-of-stream-only mode.
* fix(guardrails): use builtin type[] in get_config_model return
Avoids a new UP006 violation that tripped the ruff strict-rule budget
gate on the PR lint job.
* fix(guardrails): default optional streaming knobs to None
Non-None Pydantic defaults on GenericGuardrailAPIOptionalParams made
_get_config_value treat unset nested fields as explicit values, which
shadowed top-level litellm_params streaming flags whenever any other
optional_params key was present. Real defaults stay in the constructor.
* fix(guardrails): address review nits on generic_guardrail_api streaming
Validate streaming_sampling_rate >= 1 in the constructor and Pydantic
optional_params (ge=1), and add /v1/responses streaming coverage through
the unified post-call hook so Responses API usage is exercised alongside
chat completions.
* fix(guardrails): read nested streaming config from dict optional_params
Guardrail API/UI delivers optional_params as a plain dict, so getattr was
silently ignoring streaming_sampling_rate and streaming_end_of_stream_only.
Handle both dict and model shapes in _get_config_value with regression tests.
* fix(guardrails): clear ruff findings in generic_guardrail_api tests/types
* style(guardrails): ruff format generic_guardrail_api modules
---------
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
* fix(ui): allow any git host on the skills add form (LIT-4053)
The skills add form only accepted GitHub URLs: its URL parser bailed on
any host that did not start with github.com, so GitLab, Bitbucket, and
self-hosted repos (and any repo subfolder on them) were rejected before a
request was ever sent. The backend already accepts arbitrary git hosts
via its url and git-subdir sources, with no host allowlist, so this was a
client-side restriction only.
Generalize the parser into an exported, host-agnostic parseSkillSource:
GitHub URLs keep their github / git-subdir shorthand, every other host is
treated as a raw repo url, and an optional Subfolder path field turns any
repo into a git-subdir source (url + path). When a pasted GitHub
tree/blob URL already encodes a subfolder, the field is cleared and
disabled so a contradictory source can never be submitted.
The parser is hardened to match the backend contract: query strings and
fragments are stripped, the host match is case-insensitive and drops a
leading www., the extracted and field-entered subfolder paths are both
validated against the same regex the server uses, a real file-extension
allowlist (not "any dot") decides whether a trailing blob segment is a
file, a branch-only tree URL falls back to the repo, non-GitHub URLs
require at least an org/repo, and the suggested skill name is kebab-cased
so it satisfies the name field's own rule.
The git-subdir source is now handled in the display helpers
(getSourceDisplayText, getSourceLink, formatInstallCommand), which
previously showed it as "Unknown source" with no link. The submit path
is fully typed (RegisterPluginRequest plus an AddPluginFormValues
interface), removing the two prior any usages; as a result an
author with an email but no name is dropped rather than sent, since the
backend requires the author name.
No backend changes. Tests cover the full host/subfolder matrix at the
parser level plus form-submit assertions on the exact source payload.
* refactor(ui): sync skill register types to the generated OpenAPI schema, surface backend errors
Replace the hand-maintained, already-drifted API types for the skills add
flow with the generated ones from schema.d.ts: PluginAuthor now aliases
components["schemas"]["PluginAuthor"], the registration payload is a new
SkillRegisterRequest (the generated RegisterPluginRequest envelope with
source narrowed to our PluginSource union, since the backend types source
as a loose string map, and version kept optional since the backend
defaults it), and the dead, mismatched RegisterPluginResponse is deleted.
registerClaudeCodePlugin's inline payload type (which was missing the
git-subdir path field entirely) is replaced with SkillRegisterRequest, so
the networking layer and the form can no longer drift from the backend.
Error handling: the add-skill form swallowed the real failure and always
showed "Failed to register skill". registerClaudeCodePlugin already
derives the backend message and throws it, so the form now surfaces it
("Failed to register skill: <reason>"), and the networking helper falls
back to the raw body / status when the error response is not JSON instead
of throwing a JSON parse error. A regression test asserts the backend
message reaches the user.
* fix(ui): reject credentialed git URLs on the skills form
A repo URL with embedded user-info (user:token@host) passed the raw-host
parser and was stored verbatim as the skill source, which is served on
the unauthenticated /public/skill_hub and marketplace.json feeds, leaking
the credentials. Reject any host segment containing '@'.
* fix(ui): validate skill repo URLs through one WHATWG URL gate
Replace the ad-hoc string parsing (stripScheme / splitHost / manual
scheme, @, ?# checks) with a single parseRepoUrl gate built on the URL
parser, so every malformed/unsafe class is handled in one place and the
URL stored on the public skill feeds is always canonical. It enforces
https (rejecting http/ssh/git/file/javascript/data and protocol-relative
//host), rejects embedded credentials (user:token@host, including
userinfo-confusion like github.com@evil.com), rejects IP-literal hosts
(loopback/private/metadata and obfuscated/IPv6 forms), and rebuilds the
stored url from origin+pathname so query strings, fragments, and trailing
slashes can never be published. The GitHub org/repo shorthand is now
charset-validated like the other paths, so junk can't reach the stored
repo. Closes both Veria findings (credentialed and http sources) plus the
adversarial-review follow-ups, with regression tests for each class.
Guard the per-request CPU cost of the chat completion, MCP tool and A2A
message transforms against regressions on every commit. All benchmarks are
pure in-process work with no network I/O so they stay deterministic under
CodSpeed's simulation mode, and they import under the base dependency set the
benchmark job installs.
Inference covers the full SDK overhead via mock_response (simple, multi-turn,
tools, streaming) plus convert_to_model_response_object as a deterministic
anchor. MCP covers the client-side tool translation and the proxy server-side
tool-name prefix round-trip. A2A covers the client request/response transforms
and the proxy server-ingress message conversion.
Adds the mcp and a2a-sdk packages to the benchmark run since those transform
modules need them, and broadens the workflow triggers to litellm_internal_staging
so the internal branch flow is benchmarked too.
* feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2
Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method
The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path
Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool
* fix(otel): anchor MCP spans to params._meta trace context, not the transport span
MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles
Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports
This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug
* fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing
The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context
extracted the client's W3C Baggage from params._meta into the span's parent context.
The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span,
and the list-tools/tool-call mappers don't set those identity keys, so nothing
overwrites them. A malicious MCP client could send
params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=...
and have those identity attributes attributed to its spans.
Extract trace context only (traceparent/tracestate) in the propagator, and stop
collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the
client's trace context, the actual goal, needs only trace context; remote baggage had
no legitimate consumer here. Regression tests at both layers assert a spoofed
params._meta.baggage never lands as a span identity attribute.
* style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers
The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and
UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead
of `Dict`, and narrow the optional-import guards to `except ImportError` (the only
failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of
a blind `except Exception`.
* fix(otel): stamp authenticated identity baggage onto MCP spans
Parenting MCP spans to the client's params._meta trace context over an empty
Context() meant the tool-call and tools/list spans carried no team/key/metadata
identity at all, so they couldn't be attributed or filtered by team in a traces
backend. The LLM-call span already re-seeds identity from the parsed, authenticated
StandardLoggingPayload rather than trusting ambient/remote context; extract that into
a shared _seed_identity_baggage helper and run both MCP emitters through it.
Identity comes only from the authenticated payload, never the client carrier, so this
keeps the earlier spoofing fix intact while restoring attribution. Regression tests
assert the authenticated team lands on both MCP spans and that a spoofed
params._meta.baggage value can't override it.
* refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY
The proxy auth path calls phase_span() and seed_request_identity() in
litellm/integrations/otel/runtime.py on every request, each doing a
try/except lazy import of litellm.integrations.otel.logger. When the
OpenTelemetry SDK is not installed (the default), that import raises, and
CPython never caches a failed import, so every request re-scanned sys.path
and contended on the import lock. At 750 concurrent users this cost about
12% throughput versus v1.85.0.
Resolve the hooks once and cache the outcome, absence included, with
functools.cache, so the import is attempted a single time instead of per
request. Throughput returns to the v1.85.0 baseline.
* feat(proxy): type Customer Management response_model for OpenAPI coverage
Add response_model to the five remaining untyped /customer operations
(block, unblock, new, update, delete) so the generated OpenAPI schema
documents a concrete response body. new/update reuse the canonical
LiteLLM_EndUserTable (matching info/list); block, unblock, and delete
get small dedicated models in
litellm/types/proxy/management_endpoints/customer_endpoints.py.
Together with the already-typed info/list/daily-activity routes this
brings the Customer Management group to full response_model coverage.
Regression tests assert each public /customer/* route declares the
expected response_model and that /customer/new surfaces a typed schema
in app.openapi(), so dropping a response_model fails CI.
* fix(proxy): keep budget_id in typed customer responses
Address review feedback on the Customer Management response_model typing.
Greptile flagged that response_model=LiteLLM_EndUserTable on /customer/new
and /customer/update silently drops fields the raw Prisma model_dump()
echoed. Checking the schema, budget_id is the only such scalar column that
was missing from the Pydantic model (created_at/updated_at/tpm_limit do not
exist on litellm_endusertable), so add budget_id to LiteLLM_EndUserTable.
This restores budget_id on new/update and also fixes the pre-existing gap
where /customer/info and /customer/list (already typed) dropped it, which
the UI Customer type expects. A regression test pins budget_id surviving the
response_model filter on /customer/update.
Also document UnblockUsersResponse.blocked_users via a Field description: it
holds the users that remain blocked after the call. The key name predates
this PR and is kept to avoid a backwards-incompatible rename on a beta route.
* fix(proxy): keep nested budget fields in customer responses
response_model=LiteLLM_EndUserTable nests the budget as the narrow write
allowlist LiteLLM_BudgetTable, which silently drops the server-managed
fields the customer endpoints used to return (budget_reset_at, created_at).
Introduce CustomerResponse, a thin response model that nests
LiteLLM_BudgetTableFull (the repo's budget response model), and apply it on
/customer/new, /customer/update, /customer/info and /customer/list. list
also builds CustomerResponse so its budget isn't narrowed at construction
time. created_by/updated_at/updated_by remain omitted, matching how budgets
are returned elsewhere.
The shared LiteLLM_EndUserTable is left untouched: it's constructed in many
places that pass narrow budget instances, and pydantic v2 won't coerce a
budget instance into a wider nested model. Typing only at the response
boundary (where the handler hands FastAPI a dict) sidesteps that. A
regression test pins budget_reset_at + created_at through the filter and
asserts the internal audit fields stay out.
* test(proxy): add golden-master characterization tests for customer responses
Lock the exact JSON body each customer-object endpoint (info/list/new/update)
and delete return today, so the upcoming type-safety refactor of the handlers
is only allowed to land if it reproduces these byte for byte. Pins null-field
inclusion, the nested budget shape (server fields kept, audit fields dropped),
and object_permission reverse-relation stripping. Green against current code.
* refactor(proxy): make the customer response flow type-safe
Replace the untyped dict + bolt-on response_model pattern on the customer
object endpoints with explicit typed construction. A single mapper,
_to_customer_response, validates a DB row into CustomerResponse at one
Any -> typed seam; new/update/info/list now return it (or a list of it) and
carry real -> CustomerResponse / -> List[CustomerResponse] return
annotations, and delete returns DeleteCustomersResponse. basedpyright now
verifies the handlers' return shapes instead of a runtime filter doing it
silently.
This also deletes the four copy-pasted object_permission reverse-relation
cleanup loops: pydantic's extra=ignore drops those undeclared fields during
validation, so the loops were dead code (proven by the golden-master tests,
which stay byte-for-byte green). basedpyright errors on the file drop from
140 to 116, all from removed dict plumbing.
CustomerResponse stays a thin subclass of LiteLLM_EndUserTable so it inherits
the existing validators/config unchanged (behavior preservation); only the
nested budget type is widened.
* refactor(proxy): annotate customer response mapper param as BaseModel
Address review nit: the mapper's untyped `record` added an ANN001 violation.
The incoming rows are pydantic v2 models, so type the param as BaseModel
rather than object (object has no model_dump, which would just move the
problem to basedpyright). This clears the ANN001 and also drops three
basedpyright unknown-type violations the untyped param was adding.
* style(test): ruff format customer endpoint tests
* test(proxy): give customer budget test update mocks a valid model_dump
The type-safe response refactor validates the update result via
_to_customer_response (CustomerResponse.model_validate(record.model_dump())).
These budget tests mocked the end-user update to return a bare MagicMock,
so model_dump() yielded a MagicMock that fails validation. Give each update
mock a minimal valid dict; the tests assert on the prisma calls, not the body.
* chore(ui): regenerate API types from proxy OpenAPI spec
* fix(ui): make generated API types stable across Python versions
Python 3.13 strips a docstring's common leading indentation at compile
time while 3.12 keeps it, so app.openapi() emits differently-indented
description strings depending on the interpreter. The dashboard type
generator ran locally on 3.13 and in CI on 3.12, so schema.d.ts drifted
and the "Verify schema.d.ts matches the proxy OpenAPI spec" check failed
Normalize every description through inspect.cleandoc in the spec dump so
the output is identical regardless of interpreter, then regenerate
