scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-14 09:03:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 214
litellm/litellm-rust/crates
History
Exact
yujonglee 95ecfbbac6
feat(proxy): supervise the opt-in Rust forwarding server (#49606) 
* feat(proxy): supervise the opt-in Rust forwarding server

With LITELLM_RUST on, `litellm` and `python -m gateway.launch` start a
supervisor that runs the Rust gateway on the public port and Python on a
private loopback listener. Python still serves every route and sees each
request as if the client had connected to it directly.

Rust sends the client's TCP peer in `x-litellm-rust-peer`. The supervised
server turns server-level proxy headers off, and a middleware restores
the peer and then applies the operator's own trust, so client IPs,
trusted proxies and every IP-based setting behave the same with Rust on
or off.

The supervisor is transparent to Python. Signals pass through, Python's
exit status is the result, and both children shut down when the
supervisor's end of their stdin closes, so nothing outlives it. Rust
starts only once Python listens. When forwarding cannot start (no
gateway binary, TLS, a Unix socket, Hypercorn or Granian), the entry
point warns and serves from Python alone.

Every forwarding decision lives in litellm/proxy/rust_forward, paired
with the gateway-forward crate. OS differences sit behind a Platform
protocol, with Windows supported for local CLI runs. AGENTS.md files in
the touched folders state the boundary. The `--rust_gateway` flag is
gone in favor of the environment variable.

* refactor(proxy): switch Rust forwarding on with LITELLM_RUST_FIRST

Forwarding gets its own switch, independent of LITELLM_RUST, which keeps
selecting only the Rust paths inside the SDK. Neither turns the other on.

---------

Co-authored-by: narmstrong-berri <narmstrong@berri.ai>
2026-11-11 19:40:00 +00:00
..
auth refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
auth-aws feat(rust): add litellm-s3 crate, AwsCredentialsProvider and cost map fetch (#50547) 2026-11-10 02:37:35 +00:00
auth-azure fix(auth): align cloud credential loading and expiry with the Python SDKs (#49589) 2026-11-05 07:18:19 +09:00
auth-gcp fix(auth): align cloud credential loading and expiry with the Python SDKs (#49589) 2026-11-05 07:18:19 +09:00
auth-types fix(auth): align cloud credential loading and expiry with the Python SDKs (#49589) 2026-11-05 07:18:19 +09:00
cache refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
cache-azure-blob refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
cache-disk refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
cache-gcs fix(auth): align cloud credential loading and expiry with the Python SDKs (#49589) 2026-11-05 07:18:19 +09:00
cache-memory feat(redis): extend the native Redis foundation for the store families (#50169) 2026-11-06 12:30:07 +09:00
cache-qdrant-semantic refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
cache-response refactor(cache): remove experimental Redis and v2 caches (#50005) 2026-11-04 04:54:08 +00:00
cache-s3 feat(rust): add litellm-s3 crate, AwsCredentialsProvider and cost map fetch (#50547) 2026-11-10 02:37:35 +00:00
cache-testing refactor(rust): align the cache crates with Python and wire every native backend (#42530) 2026-09-22 13:13:02 -07:00
callbacks-legacy-python fix(messages): bill a native stream the client abandoned like python (#50425) 2026-11-10 00:24:21 +00:00
circuit-breaker fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473) 2026-11-07 17:50:06 +09:00
config feat(redis): extend the native Redis foundation for the store families (#50169) 2026-11-06 12:30:07 +09:00
core-utils fix(anthropic): map pause_turn and return refusal explanations on chat and responses (#49192) 2026-10-31 02:53:59 -07:00
coroutine refactor(rust-bridge): return an unstarted Execution from native entrypoints and drive it in Python (#46299) 2026-10-14 17:50:44 -07:00
cost feat(model-catalog): add fedramp_uplift_multiplier (#49950) 2026-11-03 14:55:51 -08:00
db feat(db): add typed daily end-user spend upsert (#48237) 2026-10-26 17:05:08 -07:00
db-testing feat(rust): add redis-client crate for standalone Redis (#48269) 2026-10-26 18:41:46 -07:00
error refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
framer refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
gateway feat(proxy): supervise the opt-in Rust forwarding server (#49606) 2026-11-11 19:40:00 +00:00
gateway-auth refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
gateway-forward feat(proxy): supervise the opt-in Rust forwarding server (#49606) 2026-11-11 19:40:00 +00:00
gateway-inference refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
gateway-management refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
gateway-mcp refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
gateway-ui refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
host fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
host-http fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
host-native fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
host-python refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
http refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
inference fix(messages): raise native messages failures as python's errors (#50424) 2026-11-10 00:24:20 +00:00
inference-chat fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
inference-messages fix(anthropic-messages): apply additional_drop_params before the unsupported param checks (#50753) 2026-11-10 01:35:35 +00:00
inference-ocr fix(auth): align cloud credential loading and expiry with the Python SDKs (#49589) 2026-11-05 07:18:19 +09:00
inference-responses fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
inference-testing refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
inference-transcription feat(rust): one typed decline from core up to each host (#47772) 2026-10-28 09:22:11 -07:00
litellm refactor(rust): expand logging and test coverage across gateway and Anthropic messages (#43295) 2026-09-26 08:04:19 +00:00
llms fix(messages): raise native messages failures as python's errors (#50424) 2026-11-10 00:24:20 +00:00
llms-types fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
model-catalog feat(cost): bill batch cached image input tokens for OpenAI image models (#50997) 2026-11-10 19:49:15 +00:00
model-catalog-refresh refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
python-bridge fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order (#50971) 2026-11-10 18:35:19 -08:00
python-compat feat(redis): extend the native Redis foundation for the store families (#50169) 2026-11-06 12:30:07 +09:00
redis-client fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473) 2026-11-07 17:50:06 +09:00
redis-connect fix(rust): install the rustls crypto provider at the Python host boundary (#50502) 2026-11-10 07:31:36 +09:00
redis-resolve feat(redis): extend the native Redis foundation for the store families (#50169) 2026-11-06 12:30:07 +09:00
redis-store fix(proxy): show committed spend in the remaining budget gauges (#49199) 2026-11-09 20:05:37 +00:00
redis-testing feat(redis): extend the native Redis foundation for the store families (#50169) 2026-11-06 12:30:07 +09:00
redis-types fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473) 2026-11-07 17:50:06 +09:00
router fix(vertex_ai)!: send the caller's anthropic-version in the Vertex Claude /v1/messages body, not the header (#48814) 2026-10-28 18:06:58 -07:00
router-core fix(management): keep classifier deployments serving during reloads (#51111) 2026-11-11 18:48:06 +00:00
router-types refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
s3 feat(rust): add litellm-s3 crate, AwsCredentialsProvider and cost map fetch (#50547) 2026-11-10 02:37:35 +00:00
secret-value fix(auth): align cloud credential loading and expiry with the Python SDKs (#49589) 2026-11-05 07:18:19 +09:00
secrets refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
secrets-aws fix(auth): align cloud credential loading and expiry with the Python SDKs (#49589) 2026-11-05 07:18:19 +09:00
secrets-azure refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
secrets-cyberark refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
secrets-google refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
secrets-hashicorp refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
secrets-types refactor(rust): extract SecretValue and use it for Redis credentials (#49186) 2026-10-30 11:31:04 -07:00
spend-clickhouse refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
storage-clickhouse refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
testkit refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
token-counter refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
token-counter-fast refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
token-counter-huggingface refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
token-counter-tiktoken refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) 2026-10-28 09:22:11 -07:00
tracing feat(rust): add structured route lifecycle tracing (#43466) 2026-09-27 19:18:58 -07:00