mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
95ecfbbac6
* feat(proxy): supervise the opt-in Rust forwarding server With LITELLM_RUST on, `litellm` and `python -m gateway.launch` start a supervisor that runs the Rust gateway on the public port and Python on a private loopback listener. Python still serves every route and sees each request as if the client had connected to it directly. Rust sends the client's TCP peer in `x-litellm-rust-peer`. The supervised server turns server-level proxy headers off, and a middleware restores the peer and then applies the operator's own trust, so client IPs, trusted proxies and every IP-based setting behave the same with Rust on or off. The supervisor is transparent to Python. Signals pass through, Python's exit status is the result, and both children shut down when the supervisor's end of their stdin closes, so nothing outlives it. Rust starts only once Python listens. When forwarding cannot start (no gateway binary, TLS, a Unix socket, Hypercorn or Granian), the entry point warns and serves from Python alone. Every forwarding decision lives in litellm/proxy/rust_forward, paired with the gateway-forward crate. OS differences sit behind a Platform protocol, with Windows supported for local CLI runs. AGENTS.md files in the touched folders state the boundary. The `--rust_gateway` flag is gone in favor of the environment variable. * refactor(proxy): switch Rust forwarding on with LITELLM_RUST_FIRST Forwarding gets its own switch, independent of LITELLM_RUST, which keeps selecting only the Rust paths inside the SDK. Neither turns the other on. --------- Co-authored-by: narmstrong-berri <narmstrong@berri.ai> |
||
|---|---|---|
| .. | ||
| test_audio_transcription_model_resolution.py | ||
| test_basic_proxy_startup.py | ||
| test_callback_delete.py | ||
| test_callback_leak_contracts.py | ||
| test_callback_settings_boot.py | ||
| test_codex_catalog_boot.py | ||
| test_config_declared_model_behaviours.py | ||
| test_effective_settings.py | ||
| test_fips_mode_boot.py | ||
| test_json_logs_startup.py | ||
| test_lazy_routes_flag.py | ||
| test_multi_worker_reload.py | ||
| test_multi_worker_slow_import_boot.py | ||
| test_openapi_docs_gate.py | ||
| test_openapi_docs_route.py | ||
| test_reload_worker_pool.py | ||
| test_rust_forward.py | ||
| test_rust_forward_lifecycle.py | ||
| test_rust_forward_traffic.py | ||
| test_search_embedding_model_settings.py | ||
| test_startup_output.py | ||
| test_unit_conftest_isolation.py | ||