The test_sso_key_generate_shows_deprecation_banner test gets 403 in CI
because GOOGLE_CLIENT_ID/MICROSOFT_CLIENT_ID env vars are set, triggering
the premium_user check. Clear these env vars in the test.
Use monkeypatch to directly replace module attributes instead of
unittest.mock.patch context managers. This is more robust in CI
environments with parallel test workers where patch contexts may
not properly intercept module-level function calls.
* Pyroscope: require PYROSCOPE_APP_NAME and PYROSCOPE_SERVER_ADDRESS, add UTF-8 locale hint
- No defaults for PYROSCOPE_APP_NAME or PYROSCOPE_SERVER_ADDRESS; fail at startup if unset when Pyroscope is enabled
- Set LANG/LC_ALL to C.UTF-8 when unset to reduce malformed_profile (invalid UTF-8) rejections
- Startup message suggests PYTHONUTF8=1 if server rejects profiles
- Simplify LITELLM_ENABLE_PYROSCOPE in config_settings; document Pyroscope env vars as required with no default
- Add pyroscope_profiling to sidebar (Alerting & Monitoring)
- pyproject.toml: pyroscope-io as required dep on non-Windows (marker), in proxy extra
* proxy: add PYROSCOPE_SAMPLE_RATE env, use verbose logging, fix int type
- Add optional PYROSCOPE_SAMPLE_RATE env (integer, no default)
- Pass sample_rate to pyroscope.configure() as int for pyroscope-io
- Replace print with verbose_proxy_logger (info/warning)
- Document PYROSCOPE_SAMPLE_RATE in config_settings.md
* Address Greptile PR feedback: Pyroscope optional, docs, tests, docstring
- pyproject.toml: mark pyroscope-io as optional=true (proxy extra only)
- Add docs/my-website/docs/proxy/pyroscope_profiling.md (fix broken sidebar link)
- Add tests/test_litellm/proxy/test_pyroscope.py for _init_pyroscope()
- proxy_server: fix _init_pyroscope docstring (required server/app name, sample rate as int)
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: populate identity fields in proxy admin JWT early-return path
When is_proxy_admin is True, the UserAPIKeyAuth early-return now includes
user_id, team_id, team_alias, team_metadata, org_id, and end_user_id
resolved from the JWT. Previously only user_role and parent_otel_span
were set, causing blank Team Name and Internal User in Request Logs UI.
* test: add unit tests for proxy admin JWT identity fields
* fix: add custom_body parameter to endpoint_func in create_pass_through_route
The bedrock_proxy_route calls `endpoint_func(custom_body=data)` to
pass a pre-parsed, SigV4-signed request body. However, the
`endpoint_func` closure created by `create_pass_through_route` does
not accept a `custom_body` keyword argument, causing:
TypeError: endpoint_func() got an unexpected keyword argument 'custom_body'
Add `custom_body: Optional[dict] = None` to both `endpoint_func`
definitions (adapter-based and URL-based). In the URL-based path,
when `custom_body` is provided by the caller, use it instead of
re-parsing the body from the raw request.
Fixes#16999
* Add tests for custom_body handling in create_pass_through_route
Address reviewer feedback on PR #20849:
- Document why the adapter-based endpoint_func accepts custom_body
for signature compatibility but does not forward it (the underlying
chat_completion_pass_through_endpoint does not support it).
- Add test_create_pass_through_route_custom_body_url_target: verifies
that when a caller (e.g. bedrock_proxy_route) supplies custom_body,
it takes precedence over the body parsed from the raw request.
- Add test_create_pass_through_route_no_custom_body_falls_back:
verifies that the default path (no custom_body) correctly uses the
request-parsed body, preserving existing behavior.
Both tests are fully mocked following the project's CONTRIBUTING.md
guidelines and the patterns established in the existing test file.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: themavik <themavik@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: add reference to example_openai_endpoint repo for self-hosting fake OpenAI proxy (#21006)
- Updated benchmarks.md with a section on setting up fake OpenAI endpoints
- Updated load_test.md to mention the self-hosted option
- Updated load_test_advanced.md with a tip box about the example repo
Reference: https://github.com/BerriAI/example_openai_endpoint
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* MCP fixes
* fix(oldteams.tsx): show policies when creating
* fix(proxy/_types.py): ensure mcp rest endpoints can be called by virtual key
ensures UI works with virtual key testing mcp endpoints
* refactor: migrate get object permissions table logic to happen in user api key auth - allows functions to trust user api key object they receive has what they need
* fix(rest_endpoints.py): filter for allowed tools based on what key has access to
* fix(mcp_server_manager.py): ensure only allowed MCP's are returned to the user, via rest endpoints
* Guardrails - add toxic/abusive content filter guardrails
* fix(streaming): preserve usage data from post-finish_reason chunks in OpenAI-compatible streaming
Fixes#16112
OpenRouter and other OpenAI-compatible providers send a usage chunk after
the finish_reason='stop' chunk when stream_options.include_usage is True.
The OpenAIChatCompletionStreamingHandler.chunk_parser() was not passing
the usage field to ModelResponseStream, causing real token counts from the
provider to be lost and falling back to inaccurate estimates.
* fix: resolve merge conflict in test file
- Fix typo in test method name (extra space)
- Move test_prompt_cache_key_in_optional_params to its own class
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(vertex_ai): forward extra_body to completion transformation handler
The responses() function accepted extra_body as a named parameter but
did not pass it to response_api_handler when responses_api_provider_config
was None (completion transformation path), silently dropping it.
Also adds deep-merge support for extra_body in Vertex AI Gemini
transformation, so dict values like generationConfig are merged rather
than replaced.
* refactor(vertex_ai): extract _merge_extra_body to fix PLR0915 lint
Move the extra_body merge loop into a helper function to keep
_transform_request_body under the 50-statement limit.
* fix: allow Management keys to access user/daily/activity and team/daily/activity
* feat(vertex): surface trafficType via generic provider_specific_fields in Responses API
Extract Vertex AI's trafficType from usageMetadata in both streaming and
non-streaming paths, storing it in _hidden_params["provider_specific_fields"].
The Responses API transformation layer generically passes any
_hidden_params["provider_specific_fields"] dict to the ResponsesAPIResponse,
avoiding provider-specific logic in the bridge.
Also fix stream_chunk_builder to propagate _hidden_params from the last
streaming chunk to the rebuilt ModelResponse, ensuring provider metadata
survives the chunk→response rebuild.
---------
Co-authored-by: naaa760 <neh6a683@gmail.com>
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
OAuth tokens (sk-ant-oat*) require Authorization: Bearer header per
Anthropic's OAuth specification, but were being sent via x-api-key
which Anthropic rejects with 'invalid x-api-key'.
- optionally_handle_anthropic_oauth: detect OAuth tokens in api_key
param (standard chat flow), not just Authorization header
- get_anthropic_headers: use Authorization: Bearer + required OAuth
headers for OAuth tokens, x-api-key for regular API keys
- Passthrough messages: skip x-api-key when Authorization is set
- Add oauth-2025-04-20 to beta headers whitelist config
* feat: add curated MCP server registry for discovery UI
Curated list of 31 well-known MCP servers with names, icons,
categories, transport config, and registry URLs. Includes HTTP
endpoints for GitHub, Atlassian, Sentry, Snowflake, and Cloudflare.
* feat: add GET /v1/mcp/discover endpoint for MCP discovery
Admin-only endpoint that serves the curated MCP registry with
optional query and category filters. Used by the UI discovery modal.
* feat: add DiscoverableMCPServer types for MCP discovery
* feat: add fetchDiscoverableMCPServers network function
* feat: add MCP discovery modal component
Compact list-row layout with category filters, search, and
grouped server list. Follows dev-tool aesthetic.
* feat: wire MCP discovery modal into server management page
Add MCP Server button now opens discovery modal. Card click
pre-fills the create form. Custom Server opens blank form.
* feat: add prefill from discovery and back-to-registry link
Create form accepts prefillData from discovery selection and
shows a Browse MCP Registry link to return to discovery modal.
* test: add unit tests for MCP discovery endpoint and registry
Tests for registry JSON structure validation and endpoint
query/category filtering logic. 15 tests total.
* fix: sync registry with official MCP API and fix stdio prefill
- Updated transport types and URLs from registry.modelcontextprotocol.io API
- GitHub: streamable-http at api.githubcopilot.com/mcp/
- GitLab: streamable-http at gitlab.com/api/v4/mcp (remote only)
- Atlassian: SSE at mcp.atlassian.com/v1/sse (remote only)
- Linear: SSE at mcp.linear.app/sse (remote only)
- Notion: SSE at mcp.notion.com/sse (remote only)
- Stripe: streamable-http at mcp.stripe.com (remote only)
- Exa: streamable-http at mcp.exa.ai/mcp (remote only)
- Cloudflare: SSE at bindings.mcp.cloudflare.com/sse (remote only)
- Sentry: stdio via @sentry/mcp-server (npm, correct package)
- Snowflake: stdio via snowflake-labs-mcp (pypi/uvx, not npm)
- Brave Search: stdio via @brave/brave-search-mcp-server (correct package)
- Fixed stdio prefill to generate stdio_config JSON instead of separate fields
- Discovery modal matches create modal width and header style
- Back arrow positioned on left of create modal header
* Update ui/litellm-dashboard/src/components/mcp_tools/mcp_discovery.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/management_endpoints/mcp_management_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: address Greptile review feedback
- Move `import json` and `import os` to module top level
- Move mcp_registry.json into litellm/proxy/ for pip distribution
- Fix `Text` component: destructure from antd Typography instead of deprecated Tremor
- Update test fixture path to match new registry location
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: inject NPM_CONFIG_CACHE into STDIO MCP subprocess env for Docker
npm/npx needs a writable cache directory. In containers the default
(~/.npm) may not exist or be read-only, causing STDIO MCP servers
launched via npx to fail with ENOENT. Inject NPM_CONFIG_CACHE=/tmp/.npm_mcp_cache
into the subprocess env when not already set.
* test: add unit test for NPM_CONFIG_CACHE injection in STDIO MCP
Verifies that NPM_CONFIG_CACHE is auto-injected when not set, and
preserved when explicitly provided. Also moves the import to module
level per code style rules.
* Update litellm/constants.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Fixes AT&T customer issue where API keys are returned in plain text
in error responses.
Changes:
1. user_api_key_auth.py: Mask the API key in the AssertionError when a
key doesn't start with 'sk-' (e.g. key with leading space). Shows
first 4 + last 4 chars with **** in between instead of the full key.
2. key_management_endpoints.py: Same masking for the key format
validation error when creating keys with invalid prefix.
3. presidio.py: Sanitize exceptions from Presidio analyze/anonymize
calls to prevent leaking original request text (which may contain
API keys) in error responses. Error messages now show only the
exception type, not the full payload.
* Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support (#20619)
* fix: fix styling
* fix(custom_code_guardrail.py): add http support for custom code guardrails
allows users to call external guardrails on litellm with minimal code changes (no custom handlers)
Test guardrail integrations more easily
* feat(a2a/): add guardrails for agent interactions
allows the same guardrails for llm's to be applied to agents as well
* fix(a2a/): support passing guardrails to a2a from the UI
* style(code-editor): allow editing custom code guardrails on ui + add examples of pre/post calls for custom code guardrails
* feat(mcp/): support custom code guardrails for mcp calls
allows custom code guardrails to work on mcp input
* feat(chatui.tsx): support guardrails on mcp tool calls on playground
* fix(mypy): resolve missing return statements and type casting issues (#20618)
* fix(mypy): resolve missing return statements and type casting issues
* fix(pangea): use elif to prevent UnboundLocalError and handle None messages
Address Greptile review feedback:
- Make branches mutually exclusive using elif to prevent input_messages from being overwritten
- Handle case where data.get('messages') returns None to avoid passing invalid payload to Pangea API
---------
Co-authored-by: Shin <shin@openclaw.ai>
* [Feat] MCP Gateway - Allow setting MCP Servers as Private/Public available on Internet (#20607)
* update MCPAuthenticatedUser
* add available_on_public_internet for MCPs
* update claude.md
* init IPAddressUtils
* init available_on_public_internet
* add on REST endpoints
* filter with IP
* TestIsInternalIp
* _extract_mcp_headers_from_request
* init get_mcp_client_ip
* _get_general_settings
* allowed_server_ids
* address PR comments
* get_mcp_server_by_name fix
* fix server
* fix review comments
* get_public_mcp_servers
* address _get_allowed_mcp_servers
* fixing user_id
* [Feat] IP-Based Access Control for MCP Servers (#20620)
* update MCPAuthenticatedUser
* add available_on_public_internet for MCPs
* update claude.md
* init IPAddressUtils
* init available_on_public_internet
* add on REST endpoints
* filter with IP
* TestIsInternalIp
* _extract_mcp_headers_from_request
* init get_mcp_client_ip
* _get_general_settings
* allowed_server_ids
* address PR comments
* get_mcp_server_by_name fix
* fix server
* fix review comments
* get_public_mcp_servers
* address _get_allowed_mcp_servers
* test fix
* fix linting
* inint ui types
* add ui for managing MCP private/public
* add ui
* fixes
* add to schema
* add types
* fix endpoint
* add endpoint
* update manager
* test mcp
* dont use external party for ip address
* Add OpenAI/Azure release test suite with HTTP client lifecycle regression detection (#20622)
* docs (#20626)
* docs
* fix(mypy): resolve type checking errors in 5 files (#20627)
- a2a_protocol/exception_mapping_utils.py: Fix type ignore comment for None assignment
- caching/redis_cache.py: Add type ignore for async ping return type
- caching/redis_cluster_cache.py: Add type ignore for async ping return type
- llms/deprecated_providers/palm.py: Add type ignore for palm.generate_text
- proxy/auth/handle_jwt.py: Add type ignore for jwt.decode options argument
All changes add appropriate type: ignore comments to handle library typing inconsistencies.
* fix(test): update deprecated gemini embedding model (#20621)
Replace text-embedding-004 with gemini-embedding-001.
The old model was deprecated and returns 404:
'models/text-embedding-004 is not found for API version v1beta'
Co-authored-by: Shin <shin@openclaw.ai>
* ui new buil
* fix(http_handler): bypass cache when shared_session is provided for aiohttp tracing
When users pass a shared_session with trace_configs to acompletion(),
the get_async_httpx_client() function was ignoring it and returning
a cached client without the user's tracing configuration.
This fix bypasses the cache when shared_session is provided, ensuring
the user's ClientSession (with its trace_configs, connector settings, etc.)
is actually used for the request.
Fixes#20174
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Shin <shin@openclaw.ai>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com>
Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com>
* fix: empty guardrails/policies arrays should not trigger enterprise license check (#20304)
The UI sends empty arrays for enterprise-only fields (guardrails, policies,
logging) even when the user has not configured these features. The backend
`is not None` check treated `[]` as a truthy intent to use the feature,
falsely requiring an enterprise license for basic team operations.
Backend: Add `and updated_kv[field] != [] and updated_kv[field] != {}`
guards in `_update_metadata_fields` so empty collections are skipped.
UI: Conditionally omit guardrails, logging, and policies from the
payload when empty instead of defaulting to `[]`.
Fixes#20304
* fix: allow clearing fields with empty collections while skipping enterprise check
Address PR review feedback:
1. Move the empty-collection guard into _update_metadata_field (singular)
so that empty lists/dicts skip only the premium license check but still
get written into metadata. This lets users intentionally clear a
previously-set field (e.g. guardrails: []) without being blocked, while
the UI's default empty arrays still don't trigger a false enterprise
error.
2. Remove sys.path hack from test file; use standard imports that work
with pytest discovery.
3. Add tests verifying that empty collections are moved into metadata
(field clearing works) even though they bypass the premium check.
Fixes#20304
* fix(proxy): add regression tests for #20441 - ensure <script> tags in LLM messages are not blocked
The 403 Forbidden error when sending messages containing `<script>` is caused
by external WAF/reverse proxy infrastructure (confirmed by the standard nginx
HTML 403 response format), not by LiteLLM's own content filtering. However,
these regression tests ensure that:
1. The content filter guardrail's built-in patterns do not match HTML tags
2. Messages containing <script> and other HTML tags pass through the content
filter unchanged when no explicit HTML-blocking rules are configured
3. The HTTP request body parser correctly handles JSON payloads containing
HTML content without modification
These tests guard against accidentally introducing HTML/XSS filtering that
would break legitimate LLM API usage (e.g., discussing HTML/JavaScript code).
Closes#20441
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When an OpenAI-compatible upstream provider emits minimal streaming event
payloads that omit required fields (e.g. created_at, output, output_index,
content_index), Pydantic raises a ValidationError crashing the SSE stream
and returning HTTP 500.
Fall back to model_construct() on ValidationError, consistent with the
existing pattern in transform_response_api_response for non-streaming.
Fixes https://github.com/BerriAI/litellm/issues/20570
Signed-off-by: Varun Chawla <varun_6april@hotmail.com>
HuggingFace Text Embeddings Inference (TEI) returns embeddings as raw
arrays [[0.1, 0.2, ...]] instead of wrapped format {"embedding": [...]}.
This change handles both formats:
- Raw array: [[...]] (TEI, some HF models)
- Wrapped: {"embedding": [[...]]} (standard HF format)
Fixes SagemakerError: "HF response missing 'embedding' field" when using
TEI containers on SageMaker.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix(oldteams.tsx): show policies when creating
* fix(proxy/_types.py): ensure mcp rest endpoints can be called by virtual key
ensures UI works with virtual key testing mcp endpoints
* refactor: migrate get object permissions table logic to happen in user api key auth - allows functions to trust user api key object they receive has what they need
* fix(rest_endpoints.py): filter for allowed tools based on what key has access to
* fix(mcp_server_manager.py): ensure only allowed MCP's are returned to the user, via rest endpoints
