From a79e0902467bfdfa4103cf5e5e7ce4ca2e331173 Mon Sep 17 00:00:00 2001 From: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Date: Tue, 22 Sep 2026 16:23:37 -0700 Subject: [PATCH] docs: plan built-in management MCP server --- mcp_management_devin_review.md | 77 +++++++++ mcp_management_implementation_plan.md | 221 ++++++++++++++++++++++++++ 2 files changed, 298 insertions(+) create mode 100644 mcp_management_devin_review.md create mode 100644 mcp_management_implementation_plan.md diff --git a/mcp_management_devin_review.md b/mcp_management_devin_review.md new file mode 100644 index 00000000000..8fc4082f8ab --- /dev/null +++ b/mcp_management_devin_review.md @@ -0,0 +1,77 @@ +# Devin independent review transcript + +Session: https://app.devin.ai/sessions/c94fda7266fe4e3bb40e3ac3adf08dea + +This is the original review, including findings challenged in subsequent exchanges. The implementation plan is the authoritative revised proposal + +Corrections applied after the initial review: the literal `/litellm-management/mcp` does match `/{mcp_server_name}/mcp`; response serialization requires separate validation; raw keys can appear in supplied arguments and echoed results; stateless JSON needs no per-session semaphore; the admin role is rechecked per tool; management MCP routes must form a new permission group outside the legacy `mcp_routes` union + +Part 1 of 3: review text, findings F1 to F5 (baseline 7177d3b, worktree /home/ubuntu/repos/litellm-base, planning only) + +Verdict. The product scope (dedicated disabled-by-default endpoint, ten explicit tools, admin-only writes, zero upstream MCP servers) is sound and is the smallest useful delivery. The execution architecture is not: an in-process ASGI client that re-enters the full FastAPI app from inside an MCP request is contradicted by the middleware stack at this baseline (admission semaphore, in-flight and billing counters, size limit, root-path rewrite) and by the way user_api_key_auth reads the body. Replace decision 3 with direct dispatch that resolves the two FastAPI dependencies itself (user_api_key_auth against a synthesized per-tool request, then the endpoint coroutine with typed kwargs). That keeps every REST authorization, hook, cache and audit behavior because those live in user_api_key_auth and management_endpoint_wrapper, not in the router. Priorities: P0 blocks the design, P1 must be resolved before implementation, P2 should be resolved, P3 advisory + +F1 (P0) In-process ASGI re-entry deadlocks on the admission semaphore and double counts every worker-level gauge. Confirmed and broader than admission control. AdmissionControlMiddleware.__call__ (litellm/proxy/middleware/admission_control_middleware.py:147-192) acquires a per-worker asyncio.Semaphore for every non-exempt HTTP request and holds it in try/finally around await self.app(...), so the slot is held for the entire outer MCP POST including the SSE stream the SDK keeps open while the tool runs. A nested httpx.ASGITransport request is a second HTTP scope and needs a second slot. With max_in_flight_requests_per_worker: 1 the inner call waits admission_queue_timeout_seconds (default 1.0) then gets a synthetic 503 overloaded_error for a request that never ran; with N slots, N concurrent outer tool calls saturate the worker and every inner call 503s. The exempt set is a hard-coded frozenset (_EXEMPT_PATHS, lines 14-25), so the only keep-the-middleware escape is a scope marker or path exemption, which is the trusted-internal bypass the plan says it does not want. The same double entry applies to InFlightRequestsMiddleware (in_flight_requests_middleware.py:31-42, drives /health/backlog and /health/drain, so one tool call counts as two in-flight requests and delays drain), BillableRequestMetricsMiddleware (billable_request_metrics_middleware.py:246-273, path-classified so /key/generate may be counted twice), RequestSizeLimitMiddleware, SecurityHeadersMiddleware, PrometheusAuthMiddleware, CORS and the root-path middleware (proxy_server.py:2357-2400 and the add_middleware calls after it). Starlette builds this stack once at first request, so there is no supported way to hand a middleware-free inner app to ASGITransport short of reaching into app.router (undocumented seam) or a second FastAPI app sharing routers, which the plan rejects. On get_async_asgi_client (litellm/llms/custom_httpx/asgi_handler.py:19-42): agreed it is the only ASGI client to use if one survives; context-local app, root_path and client, pooled transport, raise_app_exceptions False, follow_redirects False, timeout None so the caller owns the deadline. Its one caller, litellm/proxy/native_compaction.py:59-64, targets scope["app"], so native compaction has the same exposure today with max_in_flight 1; worth its own ticket and proof the problem is real. It does not change the recommendation: it removes the new-infra objection but leaves recursion, gauge double counting and F2 untouched, so it is only the right tool for a future tool that truly needs an HTTP round trip (streaming or passthrough), which none of the ten MVP tools do. Recommendation: drop decision 3 as written. If a feasibility gate stays, its first test is max_in_flight_requests_per_worker: 1 plus one create_virtual_key call, which fails at this baseline by construction + +F2 (P0) The REST auth dependency reads the request body, so any adapter must synthesize a real per-tool request, not reuse the outer MCP scope. user_api_key_auth calls _read_request_body_deferring_parse_failure (litellm/proxy/auth/user_api_key_auth.py:3302-3307), and route checks consume request_data (route_checks.py:322-327, /user/update field allowlisting at 872-883) and request.method (route_checks.py:894-899). Existing MCP admission works around this by overriding .body() to b"{}" on a request built from the MCP scope (auth/user_api_key_auth_mcp.py, process_mcp_request), fine for admission because /mcp policy is body-independent, not fine for per-tool REST re-authorization: /key/generate checks inspect team_id and friends from the body and viewer policy branches on the verb. The adapter must build a starlette Request whose scope carries the catalog's method, path, client and headers and whose body is the validated argument model as JSON. Passing the outer MCP Request (POST /litellm-management/mcp) into user_api_key_auth would authorize every tool as a POST to the MCP route and silently defeat "REST remains authoritative". Recommendation: make this an explicit requirement in decision 3 and require a test asserting user_api_key_auth sees method GET and route /key/list for list_virtual_keys + +F3 (P1) Admin-viewer support collides with route policy twice; you have resolved this as deferred, recorded here for the plan text. _check_proxy_admin_viewer_access (route_checks.py:835-930) first rejects anything is_llm_api_route (line 863), which is true for mcp_inference_routes (line 407, _types.py:533-549), so an inference classification denies viewers at admission regardless of tool. Under a control-plane classification the verb branch applies: safe methods allow, unsafe deny unless /user/update, /user/password/change or the legacy allow sets (894-930); Streamable HTTP initialize, tools/list and tools/call are POST, so viewers are denied again. Only after admission does per-tool REST authorization already do the right thing: GET /key/list and /key/info pass the safe branch, POST /key/generate|update|delete hit _ADMIN_VIEWER_BLOCKED_WRITE_ROUTES (908-916), and the access-group handlers enforce _require_admin_view versus _require_proxy_admin themselves (access_group_endpoints.py:157-175). mcp_management_routes (_types.py:552-556) is gated by DISABLE_ADMIN_ENDPOINTS and folded into management_routes (line 734); mcp_inference_routes is gated by DISABLE_LLM_API_ENDPOINTS and folded into llm_api_routes (line 622). The new endpoint must be in the management half. Viewer support would need exactly one change, a POST allowance for PROXY_ADMIN_VIEW_ONLY on the literal route and its trailing-slash form, with per-tool enforcement REST-derived, plus regression tests (viewer initialize and tools/list succeed, viewer create_virtual_key gets a tool error carrying the REST 403, viewer POST to /mcp still denied). With deferral, admission rejects initialize for any role other than PROXY_ADMIN, and the plan must stop stating "existing admin-view credentials for read tools" as fact + +F4 (P1) SDK session managers are single-run and the existing ones are lazily initialized inside the request handler. StreamableHTTPSessionManager.run() sets _has_started and raises RuntimeError on a second entry (mcp 2.2.0, mcp/server/streamable_http_manager.py:131-154; handle_request raises if the task group is missing, line 185). The existing server creates two managers at import time and enters them either from its own sub-app lifespan (_experimental/mcp_server/server.py:779-786) or lazily on first request via initialize_session_managers (line 724), because the main app lifespan is proxy_startup_event (proxy_server.py:1682) and the sub-app lifespan never runs through _stream_mcp_asgi_response. A management server must not reuse these managers (bound to the aggregate Server and its handlers) nor share _stateful_session_owners, _mcp_active_toolset_id, _mcp_proxy_mode or the AuthContextMiddleware context vars, otherwise a management session id becomes a valid aggregate session id and vice versa. Recommendation: the management module owns its own MCPServer (2.2.0 high-level API, mcp/server/mcpserver/server.py, exposes session_manager and streamable_http_app()) or a low-level Server plus one StreamableHTTPSessionManager, entered once from proxy_startup_event when the flag is on and exited on shutdown; do not copy the lazy-init pattern. Your resolution of stateless plus json_response True matches my recommendation: every request re-authenticates, the ten tools carry no session state, stateless removes the reused-session authority class of bugs and the per-owner cap and fingerprint logic you would otherwise duplicate, and JSON responses keep the outer request short + +F5 (P1) Route naming and split placement. /litellm-management/mcp has two segments and cannot match @app.api_route("/{mcp_server_name}/mcp") or the toolset route, so registration order is not load bearing; the internal mount in server.py is on the sub-app and only reached through _stream_mcp_asgi_response, so no public collision. What does matter: _resolve_mcp_csv_tokens will treat litellm-management as a server alias if an operator registers an MCP server with that name and the aggregate route would then serve /litellm-management/mcp when the flag is off, so reserve the alias at mcp_server_manager registration, not only at management startup. Split: backend/routes/allowlist.py keeps /mcp exact, /mcp mount, /mcp/ prefix and /{mcp_server_name}/, none covering /litellm-management/; add the exact path and trailing-slash form to BACKEND_EXACT_PATHS and confirm _is_backend_route in backend/main.py keeps them after lifespan filtering. Add nothing to gateway/routes/allowlist.py. Route policy: add the literal to mcp_management_routes or a new management_mcp_routes folded into management_routes (_types.py:734) so DISABLE_ADMIN_ENDPOINTS and allowed_routes=["management_routes"] behave, and test is_llm_api_route("/litellm-management/mcp") is false. Trailing slash: the aggregate route documents why a 307 breaks clients behind TLS terminators (proxy_server.py:19598-19601); register both spellings with include_in_schema=False rather than relying on redirect_slashes + +--- + +Part 2 of 3: findings F6 to F10, recommended architecture, amendments + +F6 (P1) Admission for the management endpoint must be a separate minimal auth path, not MCPRequestHandler.process_mcp_request. That function returns upstream auth headers, OAuth headers, requested server lists and raw headers, and has anonymous admission branches for passthrough and DCR plus /.well-known/ public routing. None of that is wanted, and inheriting it means a future DCR anonymity change silently changes management admission. Management admission should be: extract Authorization or x-litellm-api-key, build a Request from the scope with path set to the management route and empty JSON body, call user_api_key_auth, require user_role == PROXY_ADMIN, reject everything else with 401/403 at the ASGI boundary before the SDK sees the request. Reuse reject_disallowed_mcp_client and the client-IP helpers only if they are role-agnostic. Name process_mcp_request in the plan as the thing not to call + +F7 (P2) Recursive admission limits. With re-entry removed the surface is the MCP request itself (bounded by admission control), one user_api_key_auth at admission, one per tool call, one endpoint coroutine per tool call; no fan-out. Two bounds to state: cap tools/call concurrency per connection with a small semaphore in the management server (the SDK dispatches concurrent calls on one stream, so one agent could pin a worker's DB pool), and cap list_virtual_keys.size at the REST maximum (le=100, key_management_endpoints.py:6378) by reusing the same Query constraints in the argument model. list_access_groups has no server-side pagination (access_group_endpoints.py:580-592); the oversized-response rule should name a byte threshold and be a tool error, not a transport error + +F8 (P2) Secret handling. info_key_fn and list_keys return hashed tokens; the only raw key in scope is GenerateKeyResponse.key. New leak paths from MCP: CallToolResult.structuredContent and content[].text (intended, once), the SDK OpenTelemetry hooks (mcp/server/_otel.py) if mcp tracing is enabled, and LiteLLM's own MCP call logging path in operations.py, which must not be reused for management tools. info_key_fn accepts a plaintext key as its query parameter (docstring at key_management_endpoints.py:4257-4275), so the get_virtual_key description must tell agents to pass the hash and the dispatcher must never log argument payloads for that tool. Redaction is by field name on argument and result models, not regex. Inner hooks inside management_endpoint_wrapper (management_helpers/utils.py:755-830) run before the result reaches the dispatcher and are reached unchanged by direct dispatch. _emit_management_endpoint_otel_span (671-752) runs only when user_api_key_dict.parent_otel_span is set, reads the body from kwargs["http_request"] (698, so empty unless the dispatcher passes the synthetic request under that name, which REST does not either) and strips key, token, api_key and friends from the result via _CREDENTIAL_FIELDS (711-730), so the generated key does not reach OTEL today. send_management_endpoint_alert (560-580) forwards request_kwargs to send_virtual_key_event_slack, which prints every kwarg except user_api_key_dict with repr (integrations/SlackAlerting/slack_alerting.py:2067-2071); that includes data, so a caller-supplied GenerateKeyRequest.key and KeyRequest.keys reach Slack exactly as over REST, and a Request kwarg would be printed as its repr. Rule for the plan: the dispatcher passes the same kwargs REST would (data, user_api_key_dict, litellm_changed_by, and request only where the signature has it), never adds MCP correlation into kwargs, and carries correlation on user_api_key_dict.request_route or a context var. Test: capture the Slack alert payload and the OTEL logging payload for create_virtual_key and assert the generated key is absent from both + +F9 (P2) create_virtual_key is not proxy-admin-only at REST; /key/generate is reachable by internal users and team admins through route_checks and the ownership checks inside generate_key_fn. With direct dispatch the REST layer will not enforce the MVP narrowing, so the admission role gate in F6 is the only place it happens. Write that down so a later "add team admins" change knows the single line to relax + +F10 (P3) File layout. server.py, catalog.py, dispatcher.py under _experimental/mcp_server/management/ is reasonable. The catalog is a tuple of frozen dataclasses (tool_name, method, path, arg_model, response_type, endpoint coroutine, kwargs mapping, read_only, destructive, redact_fields) per the repo immutability rules, and the dispatcher is constructed with user_api_key_auth and the endpoint coroutines injected so tests substitute them without monkeypatching. Query-parameter tools need small typed argument models because the REST functions take individual Query params; no **kwargs from a dict + +Recommended architecture, direct dispatch with resolved dependencies. Every tool call: (1) validate MCP arguments into the catalog's Pydantic model (existing GenerateKeyRequest, UpdateKeyRequest, KeyRequest, AccessGroupCreateRequest, AccessGroupUpdateRequest, or a new typed query model); unknown-field policy is whatever the model declares. (2) Build a synthetic starlette Request whose scope copies client, scheme, server, root_path and the credential headers from the admitted MCP scope, sets method and path from the catalog entry, and whose body is the model serialized with exclude_unset=True so omitted versus null versus empty array survives (update_key_fn documents merge-patch semantics on exactly this, key_management_endpoints.py:3280-3283). (3) Call user_api_key_auth(request=..., api_key=...) with that request; this runs route checks, allowed_routes, custom auth, IP restrictions, disabled-route gating, expiry and revocation against the real REST route and verb. (4) Call the endpoint coroutine with typed kwargs: data, user_api_key_dict, litellm_changed_by, and request=synthetic for update_key_fn and list_keys; management_endpoint_wrapper operates on kwargs (utils.py:764-807) so audit, alerts and the key/user/team cache deletions run unchanged; access-group path parameters are plain positional args. (5) Convert the return to structuredContent plus text; convert exceptions to tool errors with the same public helpers the app handlers use (detail in part 3). Why this beats the ASGI client: no middleware touched, no marker or exemption, cannot deadlock, no double-counted gauges, and its two dependencies are exactly the two things FastAPI would have injected. The trusted-internal-admission variant (mark inner scopes so AdmissionControlMiddleware skips them) still leaves the other three middlewares double counting and F2 untouched. A service-layer adapter calling generate_key_helper_fn or the repositories loses management_endpoint_wrapper, the access-group handler role checks and the cache patching, so it is worse than calling the endpoint coroutine + +Amendments, in plan order. Scope: replace "existing admin-view credentials for read tools" with "deferred; admission accepts PROXY_ADMIN only". Decision 1: reserve the litellm-management alias at MCP server registration; classify under management_routes, never mcp_inference_routes; add the two backend exact paths; drop the literal-before-dynamic ordering claim. Decision 3: replace the ASGI client with direct dispatch, the synthetic-request requirement, the exclude_unset rule, process_mcp_request named as not reused, and the statement that proxy-admin narrowing is enforced at admission only. Decision 4: stateless True, json_response True, drop the session-reuse acceptance tests stateless makes moot, keep "two concurrent callers get independent results". Decision 5: add the inner-hook kwargs rule, per-connection tool-call cap, get_virtual_key hash guidance, field-name redaction on argument and result models, exclusion from aggregate MCP call logging and SDK OTel hooks. Sequence step 1: replace "in-process request/auth feasibility tests" with "direct-dispatch parity tests" including the max_in_flight 1 regression documenting why ASGI was rejected. Validation: is_llm_api_route false, DISABLE_ADMIN_ENDPOINTS hides the route, DISABLE_LLM_API_ENDPOINTS does not, viewer POST to /mcp still denied, alias reservation. On key_aliases: KeyRequest already carries both keys and key_aliases, so keeping both with the model-derived schema is parity, not new scope; agreed + +--- + +Part 3 of 3: direct-dispatch parity, answering your challenge point by point, with the remaining conditions + +Nothing in the design uses private FastAPI DI. It rests on one fact I verified: calling user_api_key_auth as a plain coroutine with a synthetic Request is already an established public seam in this codebase, not something new. Callers at this baseline: langfuse_endpoints.py:176 and llm_passthrough_endpoints.py:326 (user_api_key_auth(request=request, api_key=f"Bearer {k}")), prometheus_auth_middleware.py:65, the MCP admission code itself at user_api_key_auth_mcp.py:482, 539, 552 and 831, and user_api_key_auth_websocket_for_model (user_api_key_auth.py:641-660), which builds a synthetic http scope from a websocket exactly the way the dispatcher would. The fastapi.Security(...) header parameters are ordinary keyword defaults; you pass api_key="Bearer " and leave the rest. The endpoint coroutines are likewise plain async functions whose Depends and Header defaults are only meaningful to the router; passing every parameter explicitly is a normal call + +Dependencies beyond user_api_key_auth. There are none on the ten routes. key_management_endpoints.py:841 is APIRouter() with no dependencies; access_group_endpoints.py:42-44 is APIRouter(tags=[...]) only; the app at proxy_server.py:1674-1685 declares no dependencies=. Route-level: every one of the ten has exactly [Depends(user_api_key_auth)] plus the same object as a parameter. One thing to know: access_group_endpoints is a lazy feature (litellm/proxy/_lazy_features.py:297-299, included on first hit of its prefixes), so the dispatcher imports the module directly and does not depend on the router having been included + +DISABLE_ADMIN_ENDPOINTS. It is not a router concern; it runs inside user_api_key_auth: _authorize_authenticated_request calls RouteChecks.should_call_route (user_api_key_auth.py:3172), which in the enterprise build is EnterpriseRouteChecks.should_call_route (enterprise/litellm_enterprise/proxy/auth/route_checks.py:49-71) and raises 403 when is_management_route(route) and the flag is set. So each synthetic per-tool request (/key/generate etc.) gets the check for free, and the management route itself gets it at admission once it sits in management_routes. Custom auth is the same story: user_custom_auth is consumed inside _user_api_key_auth_builder (user_api_key_auth.py:1487), so it runs, with the condition that it now sees the synthetic request rather than the outer MCP one; that is the correct input, and the plan should say so + +Request and header injection. The only Header on these routes is litellm_changed_by on generate, update and delete keys. Decision to record: forward the outer MCP request's litellm-changed-by header value if present, else None, since REST callers can already set it and it only feeds the audit trail. The Request parameter appears on update_key_fn and list_keys; pass the synthetic request. Path params for access groups are positional. Query params for list_keys and info_key_fn come from the typed argument model, field by field + +response_model serialization. FastAPI validates the return value against response_model and encodes it. Direct dispatch replaces that with the catalog's declared response type: TypeAdapter(response_type).dump_python(result, mode="json") for the routes that declare one (GenerateKeyResponse at key_management_endpoints.py:1930; AccessGroupResponse and list[AccessGroupResponse] at access_group_endpoints.py:510, 582, 597, 618, 821-840), and fastapi.encoders.jsonable_encoder (public API, what FastAPI uses when no response_model is set) for update, delete, info and list keys. Both are public APIs, and the catalog entry pins the type explicitly so it is audited rather than read off the route. The 201 on create_access_group is a transport detail with no MCP equivalent and is dropped + +Validation and error shape. REST argument errors go through the RequestValidationError handler (proxy_server.py:1965-1977): a problem-details body via request_validation_problem for paths under MANAGEMENT_V1_PREFIX, else {"detail": jsonable_encoder(exc.errors())} with 422. The dispatcher catches pydantic ValidationError from step 1 and calls those same two public helpers depending on the catalog path, so the tool error body equals the REST body. Exception wrapping: three app handlers matter, ProxyException (1835-1850, body {"error": exc.to_dict()} plus call id, status exc.code), ManagementProblem (1897-1899, problem_response(exc.problem)), and the Exception fallthrough (1979-1995, which turns a DB-unavailable chain into a 503 ProxyException via PrismaDBExceptionHandler and otherwise logs a bug report and returns a generic 500). Key endpoints already normalize most failures with handle_exception_on_proxy (utils.py:8091-8116) into ProxyException; HTTPException falls to Starlette's default {"detail": ...}. The dispatcher maps in that order using the same functions: ProxyException -> its to_dict and code; ManagementProblem -> problem_response body and status; HTTPException -> status_code and detail; anything else -> is_database_service_unavailable_error_in_chain then service_unavailable_proxy_exception, else generic 500 with no traceback. No reimplementation, one audited match statement + +Remaining conditions, each a line in the revised plan or a test. One, the catalog is a reviewed ten-entry table (method, path, arg model, response type, coroutine, kwargs mapping) and the drift guard is behavioral: for every tool a parity test issues the same operation over REST and over MCP against the same test DB and asserts equal status class, equal body shape and equal persisted state, including the error cases above. Two, the synthetic scope must carry type http, method, path, root_path, query_string for GET tools, headers (authorization, litellm-changed-by), client copied from the admitted scope so IP allowlists match, a fresh state dict, and a receive callable that yields the JSON body once; _read_request_body caches on the request, so one Request per tool call. Three, OTEL: user_api_key_auth calls _ensure_parent_otel_span_on_request_state on the synthetic request (3305), so each tool call opens its own server span; decide whether to parent it under the outer MCP span or accept sibling spans, and note that the app exception handlers also close spans via _close_dangling_otel_server_span, which the dispatcher must do itself on the failure path. Four, budget reservation and rate limiting inside common_checks behave as for the same REST route because it is the same function; a test with a key that has rpm_limit set on management routes should confirm the counter moves once per tool call, not twice. Five, the admission-time role gate (PROXY_ADMIN only) is the only place the MVP narrowing lives (F9). Six, per-tool user_api_key_auth means one extra key lookup per tool call versus REST's one; it hits the same cache and is the price of parity, worth stating. Seven, the two hooks in F8 receive kwargs identical to REST, no MCP fields added. Eight, cancellation: if the outer MCP request is cancelled mid-dispatch the endpoint coroutine is cancelled with it, which for writes means the unknown-outcome message the plan already specifies; the dispatcher should shield nothing and retry nothing + +That closes my review pending your revised draft; I will re-review the revision against the same baseline when you send it + +--- + +## Final cross-check, 2026-09-22 23:21:51 UTC + +Final cross-check of the revised draft against baseline 7177d3b. I agree with the scoped design and the acceptance tests as written, subject to one correction that I consider a blocker and two wording fixes. No runtime proof is claimed + +Blocker, route-list placement. The draft says "classify as a management route governed by DISABLE_ADMIN_ENDPOINTS" and the acceptance test says "generic MCP inference permission does not grant administration". Those two conflict if the literal goes into mcp_management_routes as my F5 suggested: _types.py:560 defines mcp_routes = mcp_inference_routes + mcp_management_routes for backwards compatibility, keys can carry allowed_routes=["mcp_routes"], and the JWT default team_allowed_routes (_types.py:5203-5209) includes "mcp_routes". Any route in mcp_management_routes is therefore route-allowed to every mcp_routes caller, and the admission role gate would be the only thing standing between an inference-scoped key and the management endpoint. Fix: define a new management_mcp_routes tuple containing the two spellings, fold it into management_routes (the sum at _types.py:689-734) and nothing else, never into mcp_routes or mcp_management_routes, and turn the acceptance line into an explicit test: a key with allowed_routes=["mcp_routes"] and PROXY_ADMIN role is denied at admission, while allowed_routes=["management_routes"] is allowed. Please also amend my F5 wording in the review record accordingly + +Wording fix, quota. "Must not double-reserve quota for one management action" implies quota is reserved on these routes today. Model budget reservation and the parallel-request limiter apply only where is_llm_api_route is true (auth_checks.py:700-732, 873), and the rpm pre-call hooks run from LLM endpoints, not management endpoints, so nothing is reserved on either the admission route or the per-tool route. Rephrase to the invariant you own: two cached auth lookups per tool call, no budget reservation and no parallel-request counter movement for either request, asserted by a test rather than assumed + +Wording fix, viewer rationale. "MCP uses POST for reads too, so viewer support needs a dedicated transport-policy design rather than an HTTP-verb exception applied broadly" is right; add that the per-tool REST verb check already does the right thing for viewers on the synthetic requests, so the future design is admission-only + +Confirmations from this pass: delete_access_group is declared status 204 and returns None (access_group_endpoints.py:714-720), so the "successful empty result" rule is correct; the literal-before-dynamic ordering, disabled 404 responders, alias reservation and BACKEND_EXACT_PATHS entries match the code; the inner-hook owners listed (management_helpers/utils.py, slack_alerting.py) are the right two; the stateless bounds (one tool per request, 30 s deadline, 1 MiB result, existing worker admission, no per-session semaphore) are consistent with the transport; and "reused session identifiers cannot import authority" remains a valid test in stateless mode because the SDK ignores Mcp-Session-Id there and you want that asserted + +With the route-list correction applied, you can record my agreement in the review record + +Resolution: all three final amendments were applied to the implementation plan. Agreement is on the plan and required tests, not implementation approval or runtime proof diff --git a/mcp_management_implementation_plan.md b/mcp_management_implementation_plan.md new file mode 100644 index 00000000000..ca2702ebe07 --- /dev/null +++ b/mcp_management_implementation_plan.md @@ -0,0 +1,221 @@ +# Built-in LiteLLM management MCP: implementation plan + +Status: independently reviewed and agreed with Devin on 2026-09-22; ready for user review, not implementation approval or runtime verification + +Code baseline: `origin/main` at `7177d3b6d11dc208e2531f05df4df7a1f33b228a`, inspected on 2026-09-22 + +## Objective + +Let an authenticated agent manage LiteLLM through a built-in MCP endpoint, beginning with virtual keys and access groups. The MCP interface must preserve the existing management API's authorization, validation, persistence, cache invalidation, and audit behavior + +The long-term goal is broad management API coverage. The first release is a deliberately bounded foundation, not an automatic export of every HTTP route + +## Verified code context + +All paths below refer to the baseline commit, rather than the unrelated working branch in the local checkout + +| Area | Existing code and implication | +| --- | --- | +| SDK and images | `pyproject.toml` requires `mcp>=2.2.0,<3`; `uv.lock` resolves 2.2.0; `Dockerfile` and `gateway/Dockerfile` install frozen dependencies with the proxy extra | +| Current MCP routes | `litellm/proxy/proxy_server.py`: `aggregate_mcp_route`, `proxy_mcp_route`, toolset routes, and dynamic `/{mcp_server_name}/mcp` routes already exist | +| MCP transport | `litellm/proxy/_experimental/mcp_server/server.py` owns SDK transport/session wiring and delegates tool operations to `operations.py` | +| Existing policy | `operations.py`, `contracts.py`, and `auth/user_api_key_auth_mcp.py` operate on upstream MCP server grants. `_raise_if_initialize_grants_no_mcp_servers` can deny callers without upstream grants. Management access must not depend on granting an unrelated upstream MCP server | +| OpenAPI conversion | `openapi_to_mcp_generator.py::build_input_schema` copies body properties but does not fully resolve body schema references or preserve all nested constraints. `_register_openapi_tools` in `mcp_server_manager.py` uses this converter | +| Local tool registry | `tool_registry.py` supports configured handlers. It is not a ready-made management authorization or audit boundary | +| Worker admission | `middleware/admission_control_middleware.py::AdmissionControlMiddleware` acquires a worker permit for each non-exempt HTTP request. Re-entering the full application can self-starve and repeat middleware accounting | +| Existing internal client | `litellm/llms/custom_httpx/asgi_handler.py::get_async_asgi_client` already pools ASGI clients with context-local client/root-path state. It does not solve full-application re-entry and is not needed by the selected design | +| Inner management hooks | `management_helpers/utils.py::management_endpoint_wrapper` executes alerts, OTEL, and cache updates before returning. `integrations/SlackAlerting/slack_alerting.py::send_virtual_key_event_slack` renders kwargs, so outer MCP redaction alone cannot prevent secret exposure | +| Virtual keys | `management_endpoints/key_management_endpoints.py` has authenticated generate/list/info/update/delete routes. `generate_key_fn` uses `GenerateKeyRequest`, `management_endpoint_wrapper`, hooks, and caller permission checks | +| Access groups | `management_endpoints/access_group_endpoints.py` exposes `/v1/access_group` CRUD. `_require_proxy_admin` controls writes; `_require_admin_view` controls reads. Team/key cache synchronization is already implemented there | +| Access-group models | `litellm/types/access_group.py` defines create/update/response models, including assigned keys, teams, models, agents, and MCP servers | +| Split deployment | `gateway/main.py` and `backend/main.py` filter the shared application routes through their component allowlists. `backend/routes/allowlist.py` retains `/mcp`; `gateway/routes/allowlist.py` does not retain the aggregate `/mcp` route or mount | + +Correction to the initial feasibility summary: having MCP SDK dependencies in the standalone gateway image does not itself make the aggregate MCP endpoint reachable there. The full proxy and backend are the appropriate starting hosts + +## Scope and product contract + +### First release + +- A dedicated built-in endpoint, proposed as `/litellm-management/mcp`, enabled explicitly by deployment configuration +- Stateless Streamable HTTP with JSON responses, using the official MCP SDK already installed in the repository; no long-lived SSE stream for these finite operations +- Full-proxy and backend-component support; operators can route the endpoint to the backend behind the same public hostname as the gateway +- Virtual-key and access-group tools listed below +- Initial callers: existing proxy-admin credentials only +- Explicitly narrower access than REST is acceptable for the first release. Admin-viewer, team-admin, organization-admin, internal-user, and broader delegated OAuth support are subsequent work. MCP uses POST for reads too, so viewer support needs a narrow admission-policy design. Per-tool REST verb checks already distinguish viewer reads and writes on the synthetic requests +- Works with no external MCP servers configured or granted +- Disabled means the endpoint is unavailable and no management tools appear in existing catalogs + +### Outside the first release + +- Exporting every OpenAPI operation or changing the generic OpenAPI converter +- Automatically adding administration tools to the existing aggregate `/mcp` catalog +- Gateway-to-backend credential delegation, token exchange, or a new authorization server +- Dashboard configuration UI, new database tables, or new migrations +- Inference streaming, uploads, realtime sessions, provider credentials, SSO settings, arbitrary HTTP execution, or arbitrary SQL +- New durable approval, idempotency, or job systems + +### Initial tools + +Names are proposed stable public contracts. Descriptions must identify effects and required privileges + +| Tool | Existing API | Release behavior | +| --- | --- | --- | +| `list_virtual_keys` | `GET /key/list` | Reuse existing pagination and filters; redact secrets | +| `get_virtual_key` | `GET /key/info` | Prefer the key hash, as REST recommends; accept existing identifier semantics; remove echoed raw keys from the safe metadata result | +| `create_virtual_key` | `POST /key/generate` | Proxy-admin only; return the newly generated key once in the authorized tool response | +| `update_virtual_key` | `POST /key/update` | Proxy-admin only; preserve omitted versus explicitly supplied fields | +| `delete_virtual_keys` | `POST /key/delete` | Proxy-admin only; preserve existing batch and `key_aliases` semantics; destructive annotation; redact echoed raw keys | +| `list_access_groups` | `GET /v1/access_group` | Preserve current response semantics; do not invent server-side pagination | +| `get_access_group` | `GET /v1/access_group/{access_group_id}` | Proxy-admin for this release; retain endpoint checks | +| `create_access_group` | `POST /v1/access_group` | Proxy-admin only; existing relation validation and cache updates | +| `update_access_group` | `PUT /v1/access_group/{access_group_id}` | Proxy-admin only; preserve null, empty-array, and omitted-field semantics | +| `delete_access_group` | `DELETE /v1/access_group/{access_group_id}` | Proxy-admin only; preserve cleanup semantics; destructive annotation | + +Every tool requires proxy-admin admission in this release. Any later viewer support must check each actual REST contract and keep direct calls to hidden write tools denied + +## Architecture decisions + +### 1. A dedicated management surface + +Use a small management MCP application under `litellm/proxy/_experimental/mcp_server/management/`, with explicit catalog and execution owners. Reuse the official SDK and appropriate existing HTTP/auth helpers, while keeping management policy distinct from upstream MCP server entitlements + +Register literal handlers for `/litellm-management/mcp` and `/litellm-management/mcp/` before the dynamic MCP alias route, with OpenAPI inclusion disabled. The former does match `/{mcp_server_name}/mcp`, so order matters. Keep the literal handlers as 404 responders when disabled to prevent alias fallthrough, reserve `litellm-management` at config/database MCP registration, and report existing conflicts explicitly. Add both exact paths to `BACKEND_EXACT_PATHS`; make no gateway allowlist changes + +Own a separate SDK server and stateless session manager, configured for JSON responses. Create and enter the manager once per application lifespan in `proxy_startup_event`, close it on shutdown, and create a fresh manager for a later lifespan. Do not reuse the aggregate server's manager or auth/session ContextVars, and do not copy its lazy startup pattern + +Do not change the existing aggregate server's no-upstream-grants check to accommodate management tools. That would broaden unrelated access paths + +### 2. Explicit tools with schemas derived from existing models + +Use an explicit immutable allowlisted mapping of tool name, HTTP method, fixed path, request model, query/path model, response projection, and read/write metadata. Inject auth and handler dependencies for tests. Use typed per-tool call adapters rather than generic untyped kwargs or callable reflection + +Derive JSON Schema from the existing Pydantic request types, with typed wrappers for query/path parameters. Preserve references through self-contained definitions, required fields, enums, nested lists, unions, defaults, and nullable values. Validate arguments using the same model contract before dispatch + +Keep tool schemas deliberate and agent-readable. Do not bind arbitrary functions or blindly register every OpenAPI operation. New tools require a reviewed mapping and behavioral tests + +### 3. Explicit dispatch preserving REST policy and handler behavior + +The reviewed direction replaces full-app ASGI re-entry with an explicit ten-tool dispatcher. For each catalog entry, construct a fresh per-tool Request with the mapped REST method, route, validated arguments, verified client context, and caller credential. Authenticate and authorize that request through the existing `user_api_key_auth`, require the returned caller to remain `PROXY_ADMIN`, then call the existing decorated endpoint coroutine with every required argument explicitly supplied + +Build the synthetic request from trusted `client`, `scheme`, `server`, and `root_path` values with fresh request state. Replace stale outer route/endpoint/path parameters with the correct REST values; construct GET query strings from the query model and mutation bodies from the request model using `exclude_unset=True`. Preserve null versus missing fields. Supply a receive function that yields the body once, and retain that Request for auth and any handler that accepts it + +Pass `request` to `update_key_fn` and `list_keys`; pass every query parameter explicitly to `list_keys` and `info_key_fn`. Preserve the existing `litellm-changed-by` header where the key handlers accept it, but record verified caller identity separately from this caller-declared attribution. Pass no additional MCP kwargs to decorated functions. Import lazy access-group handlers explicitly rather than assuming their router has already loaded + +The adapter must preserve the caller credential for the same LiteLLM service, request correlation, and the verified client context required by IP restrictions. It must not accept arbitrary forwarded identity headers, tool-supplied credentials, target URLs, or elevated server credentials + +Check authentication and the admin role restriction at MCP admission, then re-run existing endpoint authorization on every tool execution. Classify the new endpoint explicitly as a management route, governed by `DISABLE_ADMIN_ENDPOINTS`, not as MCP inference. It must not inherit generic MCP admission fallbacks or become public through deployment overrides. The existing REST policy remains authoritative for object ownership, route permissions, and custom authorization. Never call a route function with unresolved FastAPI `Depends`, `Header`, or `Query` defaults + +Define a new `management_mcp_routes` group containing only the two endpoint spellings and include it in `management_routes`. Do not include it in `mcp_inference_routes`, `mcp_management_routes`, or their backwards-compatible `mcp_routes` union. The latter also grants existing MCP server CRUD and appears in JWT defaults, so reusing it would broaden an existing key's route permission. Require both the admin role and appropriate management route permission; per-tool authorization additionally checks the concrete REST route + +Full-app ASGI re-entry is excluded: `AdmissionControlMiddleware` holds the outer request's permit while an inner REST request acquires another, causing self-starvation at one permit or saturation. It also repeats request accounting and middleware. The existing `get_async_asgi_client` helper is useful elsewhere but does not remove these effects + +Direct dispatch does not automatically provide FastAPI request parsing, dependency injection, or response-model filtering. Before adding each tool, audit its application/router/route dependencies, request/header/query arguments, decorators, response type, and exceptions. At this baseline, the selected routes use `user_api_key_auth` with no additional application/router dependencies. Implement explicit typed argument adapters and safe response projections with existing Pydantic models; do not use private FastAPI dependency-solving APIs or a generic reflection dispatcher. Add REST-versus-MCP behavioral comparison tests for these contracts + +For declared response types, validate with the relevant model or `TypeAdapter.validate_python` before JSON serialization. Serialization alone is not validation. For endpoints without a response model, use explicit typed safe-result projections and the public JSON encoder. Do not expose arbitrary metadata or raw handler dictionaries blindly. Treat access-group deletion's 204/None as a successful empty result, not invalid JSON + +Normalize `ValidationError` into the corresponding REST validation structure with body/query/path locations, removing credential-bearing input details. Preserve useful status classes for `ProxyException`, `ManagementProblem`, and `HTTPException` through existing public helpers. Recognize database-unavailable chains using the existing classifier; otherwise return a generic safe server error. Promise semantic parity and deliberate secret redaction, not byte-identical output where the MCP transport or projections differ + +The outer HTTP request still traverses normal middleware once. Preserve client IP/root path/correlation from trusted request context, and re-check route policy against the actual per-tool operation. Do not add a user-controllable middleware bypass marker or an admission exemption for management traffic + +### 4. Credential and session boundaries + +Start with header-configured clients using credentials already accepted by LiteLLM's management APIs. Build dedicated management admission around the existing credential parser and `user_api_key_auth`, using the literal management path and an isolated empty body for transport admission before the SDK consumes the original request. Preserve the configured custom-auth behavior and test it at both admission and per-tool authorization. Do not call `MCPRequestHandler.process_mcp_request`, which includes upstream passthrough/DCR flows irrelevant to administration + +An upstream MCP OAuth credential, MCP session bearer, browser cookie, or external server grant is not automatically a management credential. Fail closed if management authentication is unavailable, even in a development deployment with no master key + +Do not forward a token issued for the MCP resource to a different backend audience. Split deployments initially route the public management MCP URL directly to the backend; any later internal forwarding needs an explicit audience/delegation design + +Authenticate each request and check the role again after per-tool auth. Stateless mode stores no caller/session authority, and arbitrary incoming session IDs must not import aggregate-server authority. Revocation behavior must be no weaker than the current REST auth cache behavior + +### 5. Results, failures, and operational behavior + +- Return typed structured results where supported and a compatible textual representation; preserve useful backend errors without echoing credentials or tracebacks +- Map tool execution failures to MCP tool error results; keep transport/authentication failures at their appropriate protocol boundary +- Never retry mutations automatically. A timeout or cancellation after dispatch may have an unknown outcome; tell the caller to inspect state before repeating a create operation +- Mark read operations read-only and deletions destructive. Client confirmation is advisory and not a substitute for authorization +- Newly created virtual keys are intentionally sensitive tool output. Exclude raw arguments/results containing secrets from MCP traces, callbacks, analytics, audit payloads, and error messages. Metadata-only reads must not expose usable keys. Cover caller-supplied keys and get/delete echoes as well as newly generated keys +- Reuse and extend field-aware redaction at the inner alert/OTEL owners before they stringify inputs. A focused alert-redaction change is in scope because preserving the existing hook unchanged would expose raw key inputs through this feature. Keep audit events and safe metadata; do not disable auditing. Capture hook payloads in tests without actually sending messages +- Do not reuse aggregate MCP tool I/O logging for management tools. Ensure SDK telemetry excludes arguments/results. Correlate the synthetic request span with the outer request and explicitly close spans on success, failure, and cancellation using existing tracing helpers +- Keep the existing management audit events and attribute them to the verified caller. Add only missing management-MCP correlation metadata; do not assume every endpoint already emits identical audit events +- Proposed initial bounds: one tool execution per stateless request with no batch fan-out, a 30-second execution deadline, and a 1 MiB encoded MCP result limit. Keep existing worker admission as the concurrency control; do not add a per-session semaphore to a stateless protocol. Preserve REST `list_virtual_keys` page size of 1 to 100. Oversized responses are explicit tool errors without silent truncation; if a mutation already completed, report that the result could not be returned and warn against blind retry +- Attach the SDK server's lifecycle to the existing application lifespan; do not start an application lifespan per tool call or duplicate database startup + +## Implementation sequence and code ownership + +1. **Prove the boundary**: audit the ten handlers' full REST contracts and current auth routing; write dispatcher-versus-REST behavioral tests, including one available admission permit; verify official SDK lifecycle against the locked version +2. **Add endpoint and typed config**: implement the disabled-by-default flag, dedicated management transport, exact backend routing, collision detection, and shutdown cleanup. Extend `proxy_server.py` only for wiring, not management business logic +3. **Add typed catalog and dispatcher**: explicit ten-tool mappings, model-derived schemas, credential-safe REST execution, deterministic role-filtered discovery, and safe result/error conversion +4. **Validate behavior and document operation**: full-proxy/backend tests, real client flows, configuration and ingress examples, limitations, and rollback instructions +5. **Expand separately after acceptance**: teams/users/organizations, budgets/models, spend/analytics, then remaining management families. Review each family's permission and secret contracts before adding it. Aggregate catalog integration and delegated identities are separate follow-ups + +Expected new capability owners: `management/server.py`, `management/catalog.py`, and `management/dispatcher.py`, adjusted only if repository conventions favor fewer files. Existing transport/auth helpers should be factored only where this feature requires it + +Existing owners potentially touched: `proxy_server.py`, `_types.py::ConfigGeneralSettings` and route classifications, `backend/routes/allowlist.py`, `auth/user_api_key_auth.py`, `auth/route_checks.py`, MCP alias registration/config owners, `management_helpers/utils.py`, `integrations/SlackAlerting/slack_alerting.py`, and existing tracing/redaction helpers. No endpoint business-logic rewrite is planned + +## Validation and acceptance criteria + +### Automated behavior + +- Disabled endpoint, enabled endpoint, bare/trailing-slash paths, alias collision, and backend/full-proxy route availability +- Management route classification: `DISABLE_ADMIN_ENDPOINTS` blocks it, `DISABLE_LLM_API_ENDPOINTS` does not, and generic MCP inference permission does not grant administration +- A proxy-admin key restricted to `allowed_routes=["mcp_routes"]` is denied at management MCP admission; the equivalent key with `allowed_routes=["management_routes"]` is admitted. Explicit MCP transport permission never bypasses concrete per-tool REST route restrictions +- Startup/shutdown/repeated test lifespans own fresh SDK managers; cancellation closes tracing and releases worker admission capacity +- Initialize/list/call using real SDK types with zero configured upstream servers +- Admin read/write success; denied admin-viewer and other non-admin callers; denied anonymous/expired/revoked credentials; direct calls to hidden tools remain denied +- REST authorization parity for custom auth, route restrictions, verified client IP, and object restrictions; no service-account substitution +- One available worker admission permit must support a successful call; check saturated workers and request accounting. Admission plus per-tool auth may perform two cached lookups. At this baseline these management requests must cause no LLM budget reservation or LLM parallel-request counter movement; assert that invariant in tests +- Two concurrent callers and reused session identifiers cannot leak credentials, tool visibility, results, or authority +- Schema fidelity for references, nested access-group assignments, enums, null, empty lists, false, zero, omitted fields, and unknown fields under the existing model policy +- Create/update/delete effects match direct REST calls, including team/key cache invalidation and existing hooks +- Error propagation for invalid arguments, missing resources, conflicts, database failures, oversized results, timeout, cancellation, and unknown mutation outcome +- No raw generated or supplied key in captured logs, alerts, traces, callbacks, validation errors, or get/delete metadata output; key creation returns its secret only in the authorized result +- Existing aggregate MCP discovery/calls, OpenAPI-backed tools, and upstream authentication remain unchanged + +New tests mirror the new capability under `tests/test_litellm/proxy/_experimental/mcp_server/management/`. Extend existing mapped tests when changing `test_operations.py`, `test_dynamic_mcp_route.py`, auth behavior, key endpoints, or access-group endpoints. Follow `tests/e2e/AGENTS.md` for real end-to-end coverage + +Place management MCP end-to-end scenarios under `tests/e2e/management/`, exercising the actual LiteLLM server and database. The existing `tests/e2e/mcp/` suite specifically requires real external MCP servers and is not the appropriate home for this built-in management surface + +### Real before/after evidence + +After implementation is authorized, present local setup steps and obtain setup/service approval before installation or startup. Use source builds tied to the merge base and implementation tip + +With identical client steps, show the absent management feature at the base and successful initialize/list/create/read/update/delete flows at the tip. Cover keys and access groups, non-admin denial, full proxy, and split ingress-to-backend routing. Record commit hashes, commands, sanitized results, and cleanup. No paid LLM calls are needed for protocol/API verification + +### Submission requirements + +Before a future implementation PR, re-read repository guidelines, the PR template, and current CI. Use the verified default branch and a `litellm_` branch name. Run applicable full local CI lint/type/budget gates against the merge base and affected backend tests; do not increase budgets or add blanket suppressions + +Measure changed executable-line coverage separately from touched-function branch coverage. Aim for at least 95% changed-line coverage for this feature and cover all meaningful authorization and secret-handling branches. Require completed current-tip Codecov evidence, required CI, contributor agreement, and current-tip Veria, Greptile confidence at least 4/5, and Bugbot reviews with zero unresolved actionable findings before declaring the implementation ready + +## Rollout and rollback + +Enable first in a test deployment, then an administrator-only pilot. Configuration removal disables the endpoint without schema rollback. Keep the management endpoint separate from the existing aggregate catalog during the pilot + +Success means an agent can complete the ten supported operations with the same observable authorization and state changes as REST, and operators can identify the caller without recording secrets + +## Independent review record + +Devin session: [Independent plan cross-check](https://app.devin.ai/sessions/c94fda7266fe4e3bb40e3ac3adf08dea) + +The user explicitly requested a fresh session. The complete draft was shared with Devin on 2026-09-22 for independent code inspection and iterative review, with no implementation authorized + +Agreement status: agreed on the scoped design and acceptance criteria after applying Devin's final route-permission correction. No unresolved design blockers remain from this review; implementation tests and runtime proof remain future work + +| Review exchange | Resolution | +| --- | --- | +| Initial independent review and local admission finding | Removed recursive ASGI dispatch. Explicit per-tool auth and decorated-handler adapters preserve the relevant REST behavior without consuming worker capacity twice | +| Viewer and transport scope | Admin-only first release; stateless JSON over Streamable HTTP; keep existing key-alias deletion semantics | +| Challenge to direct-handler parity | Added explicit query/header/body adaptation, response validation before serialization, safe result projections, public error normalization, and parity tests | +| Corrections to review details | The dynamic alias pattern does match the proposed literal, so literal ordering and disabled 404 handling remain required. Serialization alone does not validate. Raw keys occur in supplied arguments and echoed results too. Recheck the admin role after per-tool auth | +| Stateless resource bounds | Existing worker admission, one execution per request, 30-second deadline, 1 MiB result limit; no new per-session semaphore | +| Final independent cross-check | Created a separate `management_mcp_routes` permission group rather than joining the legacy `mcp_routes` union. Added explicit restricted-admin-key tests and corrected the quota/viewer wording | + +Devin's final response at 2026-09-22 23:21:51 UTC, event `event-01a0cb6cc3ab7640bf1508df6681edff`, explicitly authorizes recording agreement after that route-list correction. All three final amendments are applied above + +The original findings and subsequent correction record are retained in `mcp_management_devin_review.md`. That transcript contains superseded review assertions; this plan is the authoritative proposal + +## References + +- [Baseline repository](https://github.com/BerriAI/litellm/tree/7177d3b6d11dc208e2531f05df4df7a1f33b228a) +- [MCP tools specification](https://modelcontextprotocol.io/specification/2026-07-28/server/tools) +- [MCP security guidance](https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices)