574fcea691
* fix(redis): share one native circuit breaker between RedisCache and the Rust stores RedisCache and the native Redis client each kept their own breaker, so once the cache learned the coordination Redis was wedged, a spend-counter write through the native store still paid a full socket timeout. Every Redis target now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected into both the RedisCache guard and the native client. The Python FSM is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(rust): cargo fmt the shared breaker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(redis): close the breaker when the recovery probe gets a Redis reply error A command error such as WRONGTYPE means Redis answered, but the probe was left unsettled and its drop reopened the breaker for another recovery timeout. Both the RedisCache guard and the native client now settle such a probe as a success. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw. * fix(redis): key the shared breaker by server and close the three audit gaps The breaker registry now keys on the Redis server (host and port, unix socket, cluster node set or sentinel service) instead of the full connection target, so caches that differ only in db, credentials or url spelling share one breaker. A READONLY reply from a replica mid-failover no longer counts as Redis answering, so it can't reset a run of failures. A new RedisCache refreshes the breaker state gauge, which the deleted Python breaker did at construction. A shared native client is only reused by a store that passes the same breaker. * refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations lib.rs now only declares modules and re-exports, with resolution, target, tls, auth and endpoint each in their own file. Derived FromPyObject already reads a struct field as the attribute of the same name, so #[pyo3(attribute)] on the breaker and store settings did nothing. * refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot Derived FromPyObject already reads each field as the attribute of the same name. The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in the same order. * refactor(redis): return a HashMap from the three memory-tier snapshots The snapshots read cache-memory's HashMap, so there is no order to keep. PyO3 builds the same dict from a std HashMap, which replaces the hand-written PyDict loops in the spend counter and router usage stores and the IndexMap in the cooldown store. The rate-limit slot registry keeps IndexMap because it hands back a Python dict in the order Python wrote it. * refactor(redis): share Redis breakers and clients through one RedisHosts per host redis-connect's RedisHosts now owns the per-server breaker and the shared clients that python-bridge kept in two process-wide statics. A client takes breaker settings and admits through its server's breaker, so no caller can pair a client with a breaker of its own, and clients are keyed by Resolution alone. This drops RedisClient::guarded_by and Breaker::same. python-bridge keeps one RedisHosts per process and calls it with the GIL released. The gateway builds one per connect, so its coordination and response cache on the same server now share a breaker. RedisStoreSettings.breaker is back to breaker settings, as on main. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| AGENTS.md | ||
| Cargo.toml | ||
| Dockerfile | ||
| Dockerfile.dockerignore | ||
| mcp_config.yaml | ||
| README.md | ||
Rust gateway
The binary reads YAML from LITELLM_CONFIG and mounts configured MCP servers alongside inference routes. MCP startup requires a resolvable general_settings.master_key. Every MCP and REST request uses the gateway's shared authentication middleware
Run MCP locally
From the repository root, start the Python fixture in one terminal:
MCP_HOST=127.0.0.1 MCP_PORT=8090 .venv/bin/python tests/integration/_support/mcp_e2e_upstream_server.py
Start the gateway in another terminal with the example configuration:
LITELLM_MASTER_KEY=local-mcp-key \
LITELLM_CONFIG=litellm-rust/crates/gateway/mcp_config.yaml \
HOST=127.0.0.1 PORT=4000 \
cargo run --manifest-path litellm-rust/Cargo.toml -p litellm-gateway
Call a tool through REST:
curl -sS http://127.0.0.1:4000/mcp-rest/tools/call \
-H 'Authorization: Bearer local-mcp-key' \
-H 'Content-Type: application/json' \
-d '{"server_id":"math","name":"add","arguments":{"a":2,"b":3}}'
MCP clients can connect to /mcp or the scoped /math/mcp endpoint with the same admission credential. Tool names on the MCP endpoint carry the configured alias or server-name prefix
Configuration behavior
Config::load resolves nested include files once in breadth-first order. Included lists append and other top-level values replace earlier values. Paths resolve relative to the declaring file, with the root config directory as a fallback
environment_variables provides scalar overrides to the injected secret source and HTTP settings without changing process globals. Credentials, static headers, URLs, and stdio environment values can reference os.environ/NAME. Model providers and MCP share the host's HTTP settings, including TLS certificates and environment proxies. Explicit environment HTTP settings take precedence over litellm_settings
Each MCP entry supports HTTP or stdio transport, an optional pinned server_id, alias, tool allowlist, timeout in seconds, and positive max_concurrent_requests. Omitted or empty tool allowlists allow all upstream tools. HTTP entries accept static_headers and static API-key, bearer, basic, authorization, or token credentials. Stdio entries accept command, args, and env. Use explicit secret references in env to pass configured credentials to a child process
The gateway initializes upstreams before opening its listener. Startup fails when credentials cannot be resolved or an upstream cannot initialize. Shutdown cancels upstream connections and terminates owned stdio children
The config schema preserves Python sections, but parsing a section does not implement its service. MCP OAuth, database-backed server management, guardrails, spend accounting, local Python tools, and legacy SSE upstream connections remain unimplemented. Unsupported MCP settings and configured guardrail policies fail at startup. Incoming legacy SSE clients remain supported