3b9a59348f
* fix(vertex_ai): drop the caller's anthropic-version header on Claude /v1/messages Vertex reads the API version from the body's anthropic_version and answers 400 when a forwarded anthropic-version header names a different one, which anthropic SDK 2.x always does. Strip the header on the Python and LITELLM_RUST Vertex messages routes. * fix(vertex_ai): carry the caller's anthropic-version into the Claude /v1/messages body Vertex reads the API version from the body alone, so the caller's anthropic-version header now becomes the body's anthropic_version instead of a pinned vertex-2023-10-16. The 2023-01-01 and 2023-06-01 first-party versions keep vertex-2023-10-16, and the header itself is still never sent. Same change on the Python proxy and the Rust gateway. * fix(vertex_ai): follow drop_params for an anthropic-version Vertex does not accept Only versions Vertex accepts in the body (vertex-2023-10-16, 2026-10-21) are carried into anthropic_version. 2023-06-01, 2023-01-01, a blank header, or no header keep vertex-2023-10-16. Any other value drops to vertex-2023-10-16 under drop_params and answers a 400 naming drop_params without it, on the Python proxy and the Rust gateway. * refactor(vertex_ai): return the unsupported anthropic-version error as a value and keep Rust helpers and tests in place * fix(rust gateway): resolve os.environ drop_params and accept the same flag words as Python The gateway read drop_params: os.environ/NAME and words like yes as off, so the drop_params opt-out from the new anthropic-version 400 did not work there * docs(rust router): name Router::from_config in the crate README --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| AGENTS.md | ||
| Cargo.toml | ||
| Dockerfile | ||
| Dockerfile.dockerignore | ||
| mcp_config.yaml | ||
| README.md | ||
Rust gateway
The binary reads YAML from LITELLM_CONFIG and mounts configured MCP servers alongside inference routes. MCP startup requires a resolvable general_settings.master_key. Every MCP and REST request uses the gateway's shared authentication middleware
Run MCP locally
From the repository root, start the Python fixture in one terminal:
MCP_HOST=127.0.0.1 MCP_PORT=8090 .venv/bin/python tests/integration/_support/mcp_e2e_upstream_server.py
Start the gateway in another terminal with the example configuration:
LITELLM_MASTER_KEY=local-mcp-key \
LITELLM_CONFIG=litellm-rust/crates/gateway/mcp_config.yaml \
HOST=127.0.0.1 PORT=4000 \
cargo run --manifest-path litellm-rust/Cargo.toml -p litellm-gateway
Call a tool through REST:
curl -sS http://127.0.0.1:4000/mcp-rest/tools/call \
-H 'Authorization: Bearer local-mcp-key' \
-H 'Content-Type: application/json' \
-d '{"server_id":"math","name":"add","arguments":{"a":2,"b":3}}'
MCP clients can connect to /mcp or the scoped /math/mcp endpoint with the same admission credential. Tool names on the MCP endpoint carry the configured alias or server-name prefix
Configuration behavior
Config::load resolves nested include files once in breadth-first order. Included lists append and other top-level values replace earlier values. Paths resolve relative to the declaring file, with the root config directory as a fallback
environment_variables provides scalar overrides to the injected secret source and HTTP settings without changing process globals. Credentials, static headers, URLs, and stdio environment values can reference os.environ/NAME. Model providers and MCP share the host's HTTP settings, including TLS certificates and environment proxies. Explicit environment HTTP settings take precedence over litellm_settings
Each MCP entry supports HTTP or stdio transport, an optional pinned server_id, alias, tool allowlist, timeout in seconds, and positive max_concurrent_requests. Omitted or empty tool allowlists allow all upstream tools. HTTP entries accept static_headers and static API-key, bearer, basic, authorization, or token credentials. Stdio entries accept command, args, and env. Use explicit secret references in env to pass configured credentials to a child process
The gateway initializes upstreams before opening its listener. Startup fails when credentials cannot be resolved or an upstream cannot initialize. Shutdown cancels upstream connections and terminates owned stdio children
The config schema preserves Python sections, but parsing a section does not implement its service. MCP OAuth, database-backed server management, guardrails, spend accounting, local Python tools, and legacy SSE upstream connections remain unimplemented. Unsupported MCP settings and configured guardrail policies fail at startup. Incoming legacy SSE clients remain supported