`_check_byok_credential` previously returned silently when `prisma_client`
was None, bypassing BYOK ownership validation during database-outage
windows. Any proxy-authenticated user could invoke BYOK-protected MCP
tools without a stored credential during the outage window.
Now raises HTTP 503 with a structured error so the flow fails closed.
Regression test asserts 503 is raised when `prisma_client` is None.
Reported by @brodmart in GHSA-6762-2m23-5mxp.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds three new UISettings flags gated by check_org_admin_feature_access:
- disable_key_generate_for_org_admin
- disable_team_create_for_org_admin
- disable_model_add_for_org_admin
When enabled by a proxy admin, users with the ORG_ADMIN role receive a 403
on /key/generate, /team/new, and /model/new respectively. All other roles
(proxy admin, internal user, team) are unaffected and continue through
their existing auth checks. Flags are persisted to litellm_uisettings and
synced into general_settings via _RUNTIME_GENERAL_SETTINGS_FLAGS so the
enforcement helper can read them at request time.
Scope the header-driven team fetch to LLM API routes so admin
management routes keep the pre-existing bypass behavior (no
phantom teams, no 404s on mgmt calls). Team context is threaded
onto UserAPIKeyAuth so spend logs, rate limits, and team_models
attribution are correctly applied when admins act on behalf of
a team via x-litellm-team-id.
* fix(proxy): honor object_permission for managed vector store access
* perf(proxy): preload team object_permission on UserAPIKeyAuth
Populate team_object_permission during virtual-key and JWT auth when the
team is loaded, so can_user_access_vector_store uses it in memory first
and only falls back to get_object_permission by id when missing.
Made-with: Cursor
Three concerns raised by bot reviewers, all addressed:
1. CodeQL cyclic-import warning
``experimental_pass_through/transformation.py`` imported from the
parent ``..transformation`` module, which CodeQL flagged as a
potential cycle. Extracted the helper into a new leaf module
``vertex_ai_partner_models/anthropic/output_params_utils.py`` that
has no heavy imports of its own. Both transformation files now
import from it cleanly. Renamed the helper from the underscore-
prefixed ``_sanitize_vertex_anthropic_output_params`` to the
public ``sanitize_vertex_anthropic_output_params`` since it is now
shared across modules.
2. Greptile P2: redundant ``None`` guard on ``extra_kwargs``
``handler.py`` had two ``extra_kwargs = extra_kwargs if ... else {}``
coercions; the second was a no-op because line 220 already
coerced. Removed the second one and added a NOTE comment so future
readers understand ``extra_kwargs`` is guaranteed non-None at the
point of use.
3. Greptile P2: misleading "already translated" docstring
The docstring claimed the translator above mapped
``output_config.format`` to ``response_format``, but Greptile
correctly traced the code and found that only the legacy top-level
``output_format`` was being translated — ``output_config.format``
was being silently dropped on the adapter path. Two-part fix:
a. Code: extended ``_translate_output_format_to_openai`` to accept
both shapes (top-level ``output_format`` AND
``output_config.format`` sub-key). Top-level still takes
precedence when both are supplied. This means callers using the
newer Anthropic Structured Outputs API now have their schema
properly forwarded to non-Anthropic backends as
``response_format``.
b. Tests: rewrote the misleading docstring to describe what
actually happens, plus added two new tests:
* ``test_output_format_top_level_still_translates`` —
regression guard for the legacy path
* ``test_output_format_takes_precedence_over_output_config_format``
— documents the precedence rule explicitly
Tests: 28/28 pass (was 26/26 before; +2 for the new translation
behavior + precedence). All run in ~0.5s, no real network calls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves the silent strip of Anthropic Structured Outputs across the
Vertex AI Claude transformation paths and the Anthropic-adapter
re-merge. Consolidates and supersedes four stalled community PRs
addressing overlapping aspects of the same root bug:
- #23475 (Vertex AI Claude blanket-strip removal)
- #23396 (Vertex AI Claude conditional passthrough)
- #23706 (Anthropic adapter exclude output_config from non-Anthropic
backends)
- #22727 (Anthropic adapter strip output_config for non-Anthropic
backends)
Closes / addresses: #23380 (Vertex AI Claude output_config drop),
related: #26423, #25079, #24549, #25971, #25957, #26163, #24856.
What was broken
---------------
* Vertex AI Claude paths called ``data.pop("output_config")`` and
``data.pop("output_format")`` unconditionally even when Vertex
accepted those fields. Callers asking for Structured Outputs got a
200 with prose and never knew the schema constraints had been
silently dropped (often masked for months by permissive fallback
parsers).
* The ``/v1/messages`` -> ``/chat/completions`` adapter
(``LiteLLMMessagesToCompletionTransformationHandler``) re-merged the
raw Anthropic-shaped ``output_config`` into ``completion_kwargs``
AFTER the translator already mapped its meaningful parts to
``response_format`` / ``reasoning_effort``. Non-Anthropic backends
(Azure OpenAI, Fireworks, Bedrock Nova, etc.) then 400'd with
"Extra inputs are not permitted".
Approach
--------
Vertex AI Claude (chat-completion + experimental_pass_through paths):
Replace the unconditional pop with a sanitizer
``_sanitize_vertex_anthropic_output_params`` that strips only the
Vertex-unsupported keys (today: ``effort``) from ``output_config``
while forwarding ``format`` and the legacy top-level
``output_format``. Defensive: non-dict ``output_config`` values are
dropped to avoid sending malformed payloads downstream.
Greptile P1 from PR #23396 addressed: when ``output_config`` carries
both ``format`` and ``effort``, the prior conditional pass-through
forwarded ``effort`` and reproduced the 400. The new helper filters
per-key.
Anthropic ``/v1/messages`` adapter:
Add ``output_config`` to a named module-level constant
``ANTHROPIC_ONLY_REQUEST_KEYS`` and wire it into ``excluded_keys`` so
the post-translation re-merge skips re-adding the raw key. This
fixes the 400 on non-Anthropic backends and avoids the conflicting
duplicate (``response_format`` + raw ``output_config``) on
Anthropic-family backends.
Greptile P2 from PR #23706 addressed: the constant gives reviewers
one grep target instead of an inline literal that silently grows.
Greptile P2 from PR #22727 addressed: ``extra_kwargs or {}`` is
replaced with explicit ``is None`` checks so empty-dict callers no
longer skip the fallback path.
Tests
-----
* tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/
test_vertex_ai_partner_models_anthropic_transformation.py:
- 5 new/updated cases plus a direct unit test for
``_sanitize_vertex_anthropic_output_params``.
- Updated ``test_vertex_ai_claude_sonnet_4_5_structured_output_fix``
so its mock-injected ``output_format`` is asserted to FLOW THROUGH
(the original test asserted the now-buggy strip behavior).
* tests/test_litellm/llms/anthropic/experimental_pass_through/
adapters/test_handler_output_config_passthrough.py (new):
- Constant export sanity, output_config strip with ``effort`` only,
output_config strip with ``format`` only, regression guard that
unrelated extras still flow, explicit-empty-dict path, and the
``extra_kwargs=None`` no-crash path.
Test-quality fixes incorporated from Greptile review on the
superseded PRs:
* No ``inspect.getsource`` source-text assertions (PR #24114 / #23475).
* ``sys.path`` insertion is anchored to ``__file__`` (PR #23706).
* Assertion messages are positional, not tuple (PR #24114-class bug).
* No ``or {}`` masking explicit empty dicts in helper signatures
(PR #22727).
Verified locally: 26/26 pass with this commit. The new tests
fail (or fail to import) on ``main`` without it.
Out of scope
------------
* The ``max_tokens`` capping logic from PR #22727 — independent
concern, deserves its own PR with a focused test plan.
* Architectural rework of the ``excluded_keys`` mechanism (Greptile
P2 on PR #23706 noted point-fix growth). The named constant gives
maintainers a clear place to extend; a registry-based approach
would be a follow-up.
Co-Authored-By: netbrah <netbrah>
Co-Authored-By: s-zx <s-zx>
Co-Authored-By: invoicepulse <invoicepulse>
Co-Authored-By: cfdude <cfdude>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(team_endpoints): auto-add SSO team members to org for proxy admins
* test: proxy_admin vs team_admin security boundary for team→org move
* screenshots: before/after for team-org SSO fix
* fix(team_endpoints): restore staging security features dropped in SSO commit
Co-Authored-By: Ishaan Jaff <ishaan@berri.ai>
* style: black formatting for team_endpoints
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Has been cancelled
Use decoded managed container model_id to resolve deployment credentials for container file calls and add regressions to verify provider/model metadata decoding and api_base selection.
Made-with: Cursor
- _get_masked_values now recurses into nested dict values and covers
additional field name patterns (credentials, password, passwd)
- _row_to_submission_item applies masking before returning litellm_params
- list_guardrails_v2 filters DB and in-memory guardrails to the caller's
team memberships for non-admin users; admins still see all guardrails
- approve_guardrail_submission propagates team_id into the in-memory
guardrail dict so ownership is preserved after approval
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
MCP server CRUD endpoints (/v1/mcp/server*) were bundled with MCP
tool-call / passthrough endpoints under llm_api_routes, so setting
DISABLE_LLM_API_ENDPOINTS=true on admin-only nodes also blocked the
Admin UI from listing, adding, or attaching MCP servers.
Separate mcp_inference_routes (data-plane, gated by
DISABLE_LLM_API_ENDPOINTS) from mcp_management_routes (control-plane,
gated by DISABLE_ADMIN_ENDPOINTS). Keep mcp_routes as a union for
backward compat with allowed_routes=["mcp_routes"] virtual key configs.
Upgrade is_management_route to pattern-aware matching so
/v1/mcp/server/{path:path} resolves for concrete IDs.
Temporary MCP OAuth sessions were kept in process-local memory, so on
multi-instance/LB proxy deployments a session created on instance A could
not be found when the follow-up /server/oauth/{server_id}/... request
landed on instance B.
Persist temporary session records to Redis (encrypted with the existing
proxy encryption helpers) as a best-effort L2 cache alongside the current
in-memory L1. Convert get_cached_temporary_mcp_server to async and await
it from the authorize/token/register OAuth endpoints.
Made-with: Cursor
Vertex multi-region endpoints (e.g. us, eu) use the rep host pattern, not
{geo}-aiplatform.googleapis.com. Regional IDs still contain a hyphen.
common_utils.get_vertex_base_url centralizes the rule for SDK/API URL building.
Proxy pass-through duplicates the same branching in a local get_vertex_base_url
(with trailing slashes) to avoid importing from common_utils there; live
WebSocket passthrough uses the same multi-region host logic for wss://.
Tests cover us/eu for the common_utils helper.
Made-with: Cursor
Sibling tests were mutating litellm.proxy.proxy_server.master_key and
prisma_client with raw setattr. Values leaked across tests in the same
xdist worker, flipping the auth short-circuit in user_api_key_auth and
causing unrelated tests (e.g. test_ui_view_session_spend_logs_pagination)
to return 401 instead of 200.
Replace raw setattr with monkeypatch in the two offending files and add
an autouse conftest fixture that snapshots/restores the known-leaky
module globals for every proxy test.
PR feedback (greptile P1 / veria high): with the previous change, a team
storing mcp_tool_permissions={"my-alias": ["read_file"]} would pass the
server-access check (because the alias expanded to a concrete id in the
allowed-servers list) but the per-server tool lookup still did
dict.get(server_id) against the raw name-keyed dict — missing, returning
None, which callers treat as "no restrictions" → all tools allowed instead
of only the declared ones.
Add MCPServerManager.expand_tool_permissions() that rewrites the dict so
every key is a concrete server_id where possible (tool lists from keys
pointing at the same server are unioned). Unresolved keys pass through
unchanged so stale id-keyed restrictions still apply when the same string
is used for lookup. Wire the helper into the four dict-lookup sites:
get_allowed_tools_for_server (key + team paths), the agent tool lookup,
and the rest_endpoints.py tool filter.
Also switch expand_permission_list to pass through unresolved entries
(rather than dropping them) so existing test fixtures that use bare string
placeholders continue to work. The downstream access check denies unknown
entries when compared to the concrete request server_id, so security
posture is unchanged.
Sanitize the debug log to use %r formatting so an admin-controlled
identifier with newlines can't forge log entries (CodeQL log-injection
warning).
Two fixes to proxy-db CI:
1. test_realtime_webrtc_endpoints.py's `proxy_app` fixture mutated the
module-global `proxy_server.master_key` without restoring it, leaking
state into any test that shared the same xdist worker. Under
--dist=loadscope with 2 workers (GHA proxy-endpoints), this caused the
google_endpoints tests to fail with "No api key passed in." because
user_api_key_auth saw a set master_key and a missing API key on the
test request. The fixture now saves and restores the original value.
2. Address the Greptile note that the semantic shard design has no
catch-all, so a new test file added to tests/proxy_unit_tests/ without
a matrix entry would silently skip CI. Adds an assert-shard-coverage
job that enumerates test_*.py files and fails the workflow if any are
not referenced by a matrix entry, with a clear message telling the
author which semantic shard to place it in. All proxy-db shards now
depend on this guard.
The mocked async_increment_cache_pipeline is invoked from Router's
deployment_callback_on_success, registered as an async success callback.
Those callbacks are enqueued to GLOBAL_LOGGING_WORKER and run on a
background task, so the mock may not have been called yet when the test
asserts on it. Flush the worker before asserting.
Two independent deflakes:
1. test_ui_view_spend_logs_unauthorized (unit) was returning 400 instead
of 401/403 when earlier tests in the file left proxy-auth globals
(prisma_client, master_key, user_custom_auth, general_settings,
user_api_key_cache) in a state that let invalid tokens pass auth and
fall through to the endpoint's own start_date/end_date validation.
Add an autouse fixture that pins those globals to their import-time
defaults for every test in the file. Harden the assertion to include
response body so future flakes are diagnosable.
2. test_basic_spend_accuracy (CI job proxy_spend_accuracy_tests) depends
on the Redis transaction buffer flushing spend to Postgres. The buffer
uses a single global pod-lock key (cronjob_lock:db_spend_update_job)
and a single global buffer list key. Pointing the proxy at the shared
remote Redis means concurrent CI pipelines contend for the same lock
and can drain each other's buffer into the wrong database. Add a
start_redis reusable command that boots a per-job redis:7-alpine
container (digest-pinned), and switch proxy_spend_accuracy_tests to
REDIS_HOST=host.docker.internal:6379 so lock and buffer state are
isolated per CI run.
- Add gpt-5.5 to GPT5_MODELS parametrized list so both OpenAIGPT5Config
and AzureOpenAIGPT5Config routing tests cover the new model.
- Add test_generic_cost_per_token_gpt55 verifying the new entry's
cost-map values ($5/$0.50/$30 per 1M) and that generic_cost_per_token
returns the expected prompt/completion costs.
* feat: add gpt-5.5 to model cost map
Add gpt-5.5 entry with pricing from OpenAI flagship page:
input $5/1M, cached input $0.50/1M, output $30/1M, 272K context.
* test: add gpt-5.5 coverage for model cost map and gpt-5 routing
- Add gpt-5.5 to GPT5_MODELS parametrized list so both OpenAIGPT5Config
and AzureOpenAIGPT5Config routing tests cover the new model.
- Add test_generic_cost_per_token_gpt55 verifying the new entry's
cost-map values ($5/$0.50/$30 per 1M) and that generic_cost_per_token
returns the expected prompt/completion costs.
