* feat(proxy): supervise the opt-in Rust forwarding server
With LITELLM_RUST on, `litellm` and `python -m gateway.launch` start a
supervisor that runs the Rust gateway on the public port and Python on a
private loopback listener. Python still serves every route and sees each
request as if the client had connected to it directly.
Rust sends the client's TCP peer in `x-litellm-rust-peer`. The supervised
server turns server-level proxy headers off, and a middleware restores
the peer and then applies the operator's own trust, so client IPs,
trusted proxies and every IP-based setting behave the same with Rust on
or off.
The supervisor is transparent to Python. Signals pass through, Python's
exit status is the result, and both children shut down when the
supervisor's end of their stdin closes, so nothing outlives it. Rust
starts only once Python listens. When forwarding cannot start (no
gateway binary, TLS, a Unix socket, Hypercorn or Granian), the entry
point warns and serves from Python alone.
Every forwarding decision lives in litellm/proxy/rust_forward, paired
with the gateway-forward crate. OS differences sit behind a Platform
protocol, with Windows supported for local CLI runs. AGENTS.md files in
the touched folders state the boundary. The `--rust_gateway` flag is
gone in favor of the environment variable.
* refactor(proxy): switch Rust forwarding on with LITELLM_RUST_FIRST
Forwarding gets its own switch, independent of LITELLM_RUST, which keeps
selecting only the Rust paths inside the SDK. Neither turns the other on.
---------
Co-authored-by: narmstrong-berri <narmstrong@berri.ai>
* fix(proxy): default the uvicorn worker healthcheck timeout to 60 s so multi-worker boots survive a slow import
* fix(proxy): warn on uvicorn below 0.37 when multi-worker boots cannot get the longer healthcheck
uvicorn 0.33 to 0.36 has no timeout_worker_healthcheck, so the 60 s default never reaches it and a slow worker start still crash-loops with no hint. Print an upgrade warning when the proxy runs more than one uvicorn worker on such a version, keep granian out of the uvicorn-only path, and type the new tests.
* fix(proxy): name --run_granian in the healthcheck help text and add per-worker grace guidance to the helm chart
* fix(proxy): keep the litellm command light so spawned workers answer uvicorn's first health ping
Each worker uvicorn spawns re-runs the parent's main script before it starts
answering health pings. The litellm console script imported the whole SDK at
the top, so on a busy or CPU-limited host a new worker missed uvicorn's 5 s
ping and was killed in a loop. The script now runs litellm_cli.main:proxy_server,
which imports the proxy only when called, and gateway.launch imports LiteLLM
inside main the same way.
This replaces the earlier 60 s healthcheck default, so uvicorn's 5 s
healthcheck and SIGTERM timing stay as they are.
* fix(proxy): load the worker app before its log config and without GC pauses
* fix(gateway): ship litellm_cli in the gateway image
gateway.launch imports without_gc_pauses from litellm_cli, which the
gateway image never copied, so python -m gateway.launch failed at import.
Also build the gateway image when the launcher or litellm_cli changes.
* fix(proxy): apply the worker log config before the proxy app import, as earlier releases did
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): keep the spend sidecar socket path under the macOS AF_UNIX limit
* test(unit): share the short socket fixtures across unit tests
Move socket_dir and socket_path to tests/unit/conftest.py so the datadog
redis test's absent socket fails for the right reason on a Mac, rename
the pgbouncer helper's directory argument, and record the rule in
tests/unit/AGENTS.md
* test(unit): fall back to /tmp when TMPDIR leaves no room for a socket path
socket_dir now uses /tmp when TMPDIR plus the longest socket suffix
would pass macOS's sun_path limit, and its teardown ignores errors so a
failed test reports only its own failure
* test(unit): keep socket fixtures inside tmp_path behind a short link
socket_dir now links a short temp directory to tmp_path, so artifacts keep
pytest's retention and a killed run leaks only an empty link. The fallback
to /tmp counts the temp directory in bytes, and socket_tempdir points
tempfile at it for the pooler tests whose socket directory production picks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(ci): fix four CircleCI regressions on main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ci): stop reloading auth_checks in unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(constants): cover CLI JWT expiry env parsing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(gateway): restore proxy lifespan after importing gateway.main in launch tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
