litellm/litellm-rust/crates/gateway-mcp/AGENTS.md

MCP gateway

Native MCP routing using the official Rust SDK, pinned to rmcp 3.4.1. The crate exposes a mountable Axum router and an SDK ServerHandler

The HTTP router serves /mcp, /{server}/mcp, /mcp/{server}, /mcp/{server}/mcp, legacy /mcp/sse and /mcp/sse/messages, /mcp/enabled, and /mcp-rest/tools/list and /mcp-rest/tools/call. Streamable HTTP supports stateless requests and legacy initialize/session flows. The caller supplies the shutdown token, allowed hosts, allowed origins, and server identity through HttpConfig

NativeGateway aggregates upstream tools, prompts, resources, and resource templates. Tool and prompt calls resolve registered server prefixes, including prefixes containing hyphens. Resource names receive prefixes while their URIs remain unchanged, and reads require exactly one selected server. Tool allowlists apply to discovery and execution. Upstream pagination is drained with cycle detection. Aggregate tool listings retain healthy servers and expose sanitized outcomes under _meta["litellm.ai/server_outcomes"], while scoped REST listings preserve upstream failures

Every HTTP failure, including extractor rejections and the legacy SSE and session transport checks, is an Error with its status from status() and the {"detail"} envelope. The detail is the litellm_error::Report of the error, except for upstream and authorization failures, whose causes can carry upstream URLs and backend detail. Startup failures are ConnectError, whose Debug redacts the SDK's initialization cause for the same reason

Integration

Construct Server entries from connected SDK peers and pass a Registry to NativeGateway::new. Keep their RunningService handles alive in the host and cancel them during shutdown. SDK peers can use stdio, Streamable HTTP, or another SDK transport. Connection establishment, outbound credentials, HTTP client pools, and reconnection remain owned by the host

Mount router(operations, config) behind the gateway's authentication middleware before merging it with other endpoint routers. Registry is an authorized static catalog, not an authenticator. Production hosts must authenticate all requests, including initialize, SSE GET, session POST, and DELETE. Authentication middleware may insert SessionOwner into request extensions to bind sessions to a stable principal across credential rotation. Otherwise sessions bind to a digest of the presented credentials and requested server scope

Implement ServerResolver to select authorized peers and tool permissions for each request. It receives HTTP headers and trusted extensions through Context.parts, plus the downstream MCP request context when available. Path and x-mcp-servers scopes can narrow that selection, never broaden it. Outbound headers are not copied automatically

Wrap Operations to integrate guardrails, spend logging, virtual tools, and other policy. Both MCP and REST execution use this interface. Database management, OAuth/BYOK services, toolsets, OpenAPI conversion, and semantic search are integration responsibilities in this stage, rather than new implementations in this crate

RelayClient::for_request forwards sampling, elicitation, roots, and progress to the originating downstream client. Create it for that request's upstream connection and retain the connection until execution completes. Do not share this relay across callers. Sampling can instead be handled by an injected SDK ClientHandler backed by inference

Local interoperability check

From the repository root, this loopback-only example runs the existing Python MCP fixture through the SDK's stdio transport. It has no admission authentication and is intended for local testing

cargo run --manifest-path litellm-rust/Cargo.toml -p litellm-gateway-mcp --example stdio_proxy -- \
  .venv/bin/python -c 'import runpy; runpy.run_path("tests/integration/_support/mcp_e2e_upstream_server.py")["mcp"].run(transport="stdio")'

curl -sS http://127.0.0.1:4000/mcp-rest/tools/call \
  -H 'Content-Type: application/json' \
  -d '{"server_id":"local","name":"add","arguments":{"a":2,"b":3}}'

{"content":[{"type":"text","text":"5"}],"structuredContent":{"result":5},"isError":false}

curl -sS http://127.0.0.1:4000/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -H 'MCP-Protocol-Version: 2025-11-25' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"local-multiply","arguments":{"a":6,"b":7}}}'

data: {"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"42"}],"structuredContent":{"result":42},"isError":false}}