refactor(rust): replace ai gateway with inference crate

.github/workflows/test-rust.yml

@@ -75,7 +75,7 @@ jobs:
         run: cargo clippy -p litellm-core --all-targets --features bedrock-auth --locked -- -D warnings
 
       - name: Run Clippy with all gateway runtime features
-        run: cargo clippy -p litellm-ai-gateway --all-targets --all-features --locked -- -D warnings
+        run: cargo clippy -p litellm-gateway-inference --all-targets --all-features --locked -- -D warnings
 
       - name: Run Clippy for the gateway server
         run: cargo clippy -p litellm-gateway-server --all-targets --locked -- -D warnings

litellm-rust/ADDING_A_PROVIDER.md

@@ -26,4 +26,4 @@ variants of it. The test for a good abstraction is that adding the next provider
 is a few declarative lines, not a new file of duplicated flow. Only diverge from
 the base when behavior is genuinely different, and say so explicitly in the PR.
 
-**Calling:** hosts invoke the core entrypoint. The Python bridge and the reusable `ai-gateway` runtime both call `litellm_core::messages::messages`, while `gateway-server` adapts HTTP requests to that runtime. Never add a provider handler to either host layer. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md).
+**Calling:** hosts invoke the core entrypoint. The Python bridge and the reusable `gateway-inference` runtime both call `litellm_core::messages::messages`, while `gateway-server` adapts HTTP requests to that runtime. Never add a provider handler to either host layer. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md).

litellm-rust/AGENTS.md

@@ -8,12 +8,12 @@ litellm-rust has six crates. A crate is a layer, shared foundation, or separatel
 |-------|------|
 | litellm-core | The LiteLLM SDK in Rust. One public entrypoint per top-level call (`messages::messages()`), owning types, transforms, provider resolution, auth, and the provider HTTP call. Call it, get a typed response. |
 | litellm-config | Config-loading boundary. Returns resolved core deployment data and optionally delegates loading to Python. |
-| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by the server and Python bridge. Owns transport-neutral orchestration and legacy call modules, but no Axum or Tower dependencies. |
-| litellm-gateway-server | The Axum binary and HTTP/WebSocket host. Owns routes, auth extractors, application state, startup config, and HTTP-only Tower dependencies. |
+| litellm-gateway-inference | Framework-independent gateway runtime and integrations shared by the server and Python bridge. Owns transport-neutral orchestration and legacy call modules, but no Axum or Tower dependencies. |
+| litellm-gateway-server | The root Axum binary and composition crate. Owns route composition, auth extractors, application state, startup config, and HTTP-only Tower dependencies, then delegates domain behavior to extracted gateway crates. |
 | litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
 | litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
 
-Dependency direction is acyclic: `litellm-config` depends on `litellm-core`; `litellm-ai-gateway` depends on core; `litellm-gateway-server` depends on gateway, core, and config; and `litellm-python-bridge` depends on the reusable domain layers and `litellm-python-interop`. Reusable crates must not depend on `litellm-gateway-server`. The interop foundation depends on no LiteLLM domain crate.
+Dependency direction is acyclic: `litellm-config` depends on `litellm-core`; `litellm-gateway-inference` depends on core; `litellm-gateway-server` depends on inference, core, and config; and `litellm-python-bridge` depends on the reusable domain layers and `litellm-python-interop`. Reusable crates must not depend on `litellm-gateway-server`. The interop foundation depends on no LiteLLM domain crate.
 
 ## Where a route lives
 
@@ -29,7 +29,7 @@ core/src/messages/
   client.rs          # the shared reqwest client
 ```
 
-Provider handlers never live in `gateway-server`. `ocr`, `audio_transcription`, and realtime provider I/O are still hosted in `ai-gateway` from before this rule; they move to `core` as they are touched.
+Provider handlers never live in `gateway-server`. `ocr`, `audio_transcription`, and realtime provider I/O are still hosted in `gateway-inference` from before this rule; they move to `core` as they are touched.
 
 Adding a crate: default to a module. A new crate requires a real trigger: separate artifact (binary/cdylib), proc-macro, shared foundation, or publishable standalone. A new provider or route is none of these.
 

litellm-rust/CLAUDE.md

@@ -25,8 +25,8 @@ the base when behavior is genuinely different, and say so explicitly in the PR.
 
 `litellm-core` **is** the LiteLLM SDK in Rust: it makes the LLM call.
 `litellm-config` is the config-loading boundary and returns resolved core types.
-`litellm-ai-gateway` is the framework-independent gateway runtime and integration
-layer. `litellm-gateway-server` is the HTTP/WebSocket server in front of it, and
+`litellm-gateway-inference` is the framework-independent inference domain.
+`litellm-gateway-server` is the root HTTP/WebSocket composition crate, and
 `litellm-python-bridge` exposes reusable Rust APIs to the Python SDK.
 `litellm-python-interop` holds domain-neutral PyO3 primitives shared by
 Python-facing Rust code. A crate is a layer, shared foundation, or separate host,
@@ -83,7 +83,7 @@ Env reads in `core` are limited to credential fallback inside a route's
 no key is passed. Everything else config-shaped is resolved by the host and
 passed in.
 
-Legacy call runtimes still hosted in `ai-gateway` (`ocr`,
+Legacy call runtimes still hosted in `gateway-inference` (`ocr`,
 `audio_transcription`, realtime provider I/O) predate this rule and are being
 moved into `core` route modules; do not add new ones there, and prefer moving
 one when you touch it. Axum routes, auth extractors, application state, and
@@ -181,7 +181,7 @@ cd litellm-rust
 cargo fmt --check
 cargo clippy --workspace --all-targets -- -D warnings
 cargo clippy -p litellm-core --all-targets --features bedrock-auth -- -D warnings
-cargo clippy -p litellm-ai-gateway --all-targets --all-features -- -D warnings
+cargo clippy -p litellm-gateway-inference --all-targets --all-features -- -D warnings
 cargo clippy -p litellm-gateway-server --all-targets -- -D warnings
 cargo test --workspace
 cargo test -p litellm-core --features bedrock-auth

litellm-rust/Cargo.lock

@@ -1404,22 +1404,6 @@ version = "0.2.186"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
 
-[[package]]
-name = "litellm-ai-gateway"
-version = "0.1.0"
-dependencies = [
- "base64",
- "futures-channel",
- "futures-util",
- "litellm-core",
- "reqwest",
- "serde",
- "serde_json",
- "tokio",
- "tokio-tungstenite",
- "tracing",
-]
-
 [[package]]
 name = "litellm-config"
 version = "0.1.0"
@@ -1453,15 +1437,31 @@ dependencies = [
  "tracing-subscriber",
 ]
 
+[[package]]
+name = "litellm-gateway-inference"
+version = "0.1.0"
+dependencies = [
+ "base64",
+ "futures-channel",
+ "futures-util",
+ "litellm-core",
+ "reqwest",
+ "serde",
+ "serde_json",
+ "tokio",
+ "tokio-tungstenite",
+ "tracing",
+]
+
 [[package]]
 name = "litellm-gateway-server"
 version = "0.1.0"
 dependencies = [
  "axum",
  "futures-util",
- "litellm-ai-gateway",
  "litellm-config",
  "litellm-core",
+ "litellm-gateway-inference",
  "reqwest",
  "serde",
  "serde_json",
@@ -1478,8 +1478,8 @@ version = "0.1.0"
 dependencies = [
  "criterion",
  "futures-util",
- "litellm-ai-gateway",
  "litellm-core",
+ "litellm-gateway-inference",
  "litellm-python-interop",
  "pyo3",
  "pyo3-async-runtimes",

litellm-rust/Cargo.toml

@@ -2,7 +2,7 @@
 members = [
     "crates/core",
     "crates/config",
-    "crates/ai-gateway",
+    "crates/gateway-inference",
     "crates/gateway-server",
     "crates/python-interop",
     "crates/python-bridge",
@@ -20,7 +20,7 @@ tracing = "0.1"
 tracing-subscriber = { version = "0.3", default-features = false, features = ["registry", "std"] }
 litellm-core = { path = "crates/core" }
 litellm-config = { path = "crates/config" }
-litellm-ai-gateway = { path = "crates/ai-gateway", default-features = false }
+litellm-gateway-inference = { path = "crates/gateway-inference", default-features = false }
 litellm-gateway-server = { path = "crates/gateway-server" }
 litellm-python-interop = { path = "crates/python-interop" }
 axum = "0.7"

litellm-rust/README.md

@@ -26,8 +26,8 @@ coverage and production evidence.
 |-------|------|
 | litellm-core | The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router. |
 | litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. |
-| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by server and Python hosts. |
-| litellm-gateway-server | Axum binary, HTTP/WebSocket routes, auth extractors, application state, and HTTP-only dependencies. |
+| litellm-gateway-inference | Framework-independent inference services and integrations shared by server and Python hosts. |
+| litellm-gateway-server | Root Axum binary and composition crate. Owns routes, auth, state, startup, and HTTP-only dependencies. |
 | litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
 | litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
 
@@ -41,8 +41,8 @@ crates/
     src/messages/   mod.rs (entrypoint), types, transformation, prepare, handler, client
     src/providers/anthropic/messages/transformation.rs
   config/         Config loading and resolved deployments.
-  ai-gateway/     Framework-independent gateway runtime and integrations.
-  gateway-server/ Axum binary and HTTP/WebSocket host.
+  gateway-inference/ Framework-independent inference runtime and integrations.
+  gateway-server/    Root Axum binary and route composition host.
   python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
   python-bridge/  PyO3 API adapter for Python LiteLLM.
 ```

litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md

@@ -16,7 +16,7 @@ Rules for adding or changing an LLM provider/route in `litellm-rust`. `messages`
 
 ## Boundaries
 
-7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `ai-gateway` = framework-independent gateway runtime and integrations; `gateway-server` = HTTP/WS routing, extractors, auth of *our* callers, application state, and streaming to the client; `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint or reusable gateway runtime; they never build a provider request.
+7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `gateway-inference` = framework-independent gateway runtime and integrations; `gateway-server` = HTTP/WS routing, extractors, auth of *our* callers, application state, and streaming to the client; `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint or reusable gateway runtime; they never build a provider request.
 8. Generic/route files contain zero provider-specific branches. A provider is one module under `core/src/providers/<provider>/<route>/`; a route is a module, never a new crate.
 9. Route entry point stays thin: `core::<route>::<route>()` -> `prepare_*` -> handler (or `CallLifecycle::run_request`, which owns the pre_call -> during_call -> provider call -> success/failure order and phase timing). Axum handlers validate and delegate to a service that calls the entrypoint; no business logic in them.
 10. Constants (URLs, env-var names, API versions, error messages) live in a crate `constants.rs`, never inline. Config-shaped env reads happen at the host/config layer with the `DEFAULT_*` fallback defined in `constants.rs`; the only env read in `core` is the credential fallback in a route's `prepare.rs`.

litellm-rust/crates/ai-gateway/ARCHITECTURE.md

@@ -1,17 +0,0 @@
-# ai-gateway architecture
-
-`litellm-ai-gateway` is between hosts and `litellm-core`. It exposes
-framework-independent runtime services and callback integrations without
-depending on an HTTP framework
-
-```mermaid
-flowchart LR
-  S[litellm-gateway-server] --> G[litellm-ai-gateway runtime]
-  B[litellm-python-bridge] --> G
-  G --> C[litellm-core]
-  S --> C
-  S --> F[litellm-config]
-```
-
-The server may depend on the runtime, core, and config crates. Reusable crates
-must not depend on the server

litellm-rust/crates/core/src/call_lifecycle/README.md

@@ -51,7 +51,7 @@ of editing OCR, chat, messages, responses, completions, or provider modules.
 Each migrated call type should use this folder shape:
 
 ```text
-litellm-rust/crates/ai-gateway/src/<call_type>/
+litellm-rust/crates/gateway-inference/src/<call_type>/
   mod.rs          # thin public entrypoint
   types.rs        # public request, prepared request, provider request, response types
   prepare.rs      # model/provider/callback/guardrail setup

litellm-rust/crates/core/tests/workspace_crate_allowlist.rs

@@ -1,7 +1,7 @@
 //! Enforcement: the litellm-rust workspace has exactly six crates.
 //!
 //! `core` (the Rust SDK), `config` (the config-loading boundary),
-//! `ai-gateway` (framework-independent gateway runtime),
+//! `gateway-inference` (framework-independent gateway runtime),
 //! `gateway-server` (the HTTP/WebSocket host),
 //! `python-interop` (domain-neutral PyO3 primitives), and `python-bridge` (the
 //! PyO3 cdylib). Adding or removing a crate must be a
@@ -22,7 +22,7 @@ use std::path::{Path, PathBuf};
 const EXPECTED_MEMBERS: &[&str] = &[
     "crates/core",
     "crates/config",
-    "crates/ai-gateway",
+    "crates/gateway-inference",
     "crates/gateway-server",
     "crates/python-interop",
     "crates/python-bridge",
@@ -32,7 +32,7 @@ const EXPECTED_MEMBERS: &[&str] = &[
 const EXPECTED_CRATE_DIRS: &[&str] = &[
     "core",
     "config",
-    "ai-gateway",
+    "gateway-inference",
     "gateway-server",
     "python-interop",
     "python-bridge",

litellm-rust/crates/gateway-inference/AGENTS.md

@@ -1,11 +1,11 @@
-# ai-gateway folder architecture
+# gateway-inference folder architecture
 
-This crate is a reusable, framework-independent gateway runtime and integration
-library. It is used by `litellm-gateway-server` and `litellm-python-bridge`
+This crate is the reusable, framework-independent inference domain. It is used
+by `litellm-gateway-server` and `litellm-python-bridge`
 
 Axum routes, auth extractors, application state, startup, and Tower dependencies
 belong in `litellm-gateway-server`. This crate must not depend on the server
 
-Transport-neutral route orchestration lives under `src/runtime/`. Existing OCR,
+Transport-neutral inference orchestration lives under `src/runtime/`. Existing OCR,
 audio transcription, realtime I/O, and callback integrations remain here as
 migration seams until their provider call paths move into `litellm-core`

litellm-rust/crates/gateway-inference/ARCHITECTURE.md

@@ -0,0 +1,17 @@
+# gateway-inference architecture
+
+`litellm-gateway-inference` is the first extracted gateway domain. It sits
+between hosts and `litellm-core`, exposing framework-independent inference
+services and callback integrations without depending on an HTTP framework
+
+```mermaid
+flowchart LR
+  S[litellm-gateway-server] --> G[litellm-gateway-inference runtime]
+  B[litellm-python-bridge] --> G
+  G --> C[litellm-core]
+  S --> C
+  S --> F[litellm-config]
+```
+
+The server may depend on the runtime, core, and config crates. Reusable crates
+must not depend on the server

litellm-rust/crates/gateway-inference/Cargo.toml

@@ -1,12 +1,12 @@
 [package]
-name = "litellm-ai-gateway"
+name = "litellm-gateway-inference"
 version = "0.1.0"
 edition.workspace = true
 license.workspace = true
 repository.workspace = true
 
 [lib]
-name = "litellm_ai_gateway"
+name = "litellm_gateway_inference"
 
 [dependencies]
 tracing.workspace = true

litellm-rust/crates/gateway-inference/README.md

@@ -1,9 +1,9 @@
-# LiteLLM AI Gateway Runtime
+# LiteLLM Gateway Inference
 
-`litellm-ai-gateway` is the reusable, framework-independent runtime used by the
-Rust gateway server and Python bridge
+`litellm-gateway-inference` is the reusable, framework-independent inference
+domain used by the Rust gateway server and Python bridge
 
-It owns gateway integrations, transport-neutral route orchestration, and legacy
+It owns inference integrations, transport-neutral orchestration, and legacy
 OCR, audio transcription, and realtime I/O that have not yet moved into
 `litellm-core`. It has no Axum or Tower dependency
 

litellm-rust/crates/gateway-inference/src/constants.rs

@@ -1,4 +1,4 @@
-//! Crate-level constants for the ai-gateway.
+//! Crate-level constants for the gateway-inference.
 //!
 //! Per `litellm-rust/CLAUDE.md`, magic numbers and fixed strings live here
 //! (the Rust mirror of Python's `litellm/constants.py`), not inline in feature

litellm-rust/crates/gateway-inference/src/integrations/README.md

@@ -26,7 +26,7 @@ Implement `CustomLogger` when Rust code needs to observe terminal success or
 failure events. Method names intentionally match Python `CustomLogger` names.
 
 ```rust
-use litellm_ai_gateway::integrations::custom_logger::{
+use litellm_gateway_inference::integrations::custom_logger::{
     CallbackTiming, CallbackValue, CustomLogger, LogFuture, ModelCallDetails,
 };
 
@@ -80,7 +80,7 @@ during-call checks. Method names intentionally match Python `CustomGuardrail`
 entrypoints inherited from Python `CustomLogger`.
 
 ```rust
-use litellm_ai_gateway::integrations::custom_guardrail::{
+use litellm_gateway_inference::integrations::custom_guardrail::{
     CustomGuardrail, GuardrailContext, GuardrailDecision, GuardrailEventHook,
     GuardrailFuture, GuardrailRequest,
 };
@@ -104,7 +104,7 @@ impl CustomGuardrail for BlocklistedPromptGuardrail {
         Box::pin(async move {
             if request.data.to_string().contains("blocked phrase") {
                 return Ok(GuardrailDecision::Block(
-                    litellm_ai_gateway::integrations::custom_guardrail::GuardrailError::blocked(
+                    litellm_gateway_inference::integrations::custom_guardrail::GuardrailError::blocked(
                         "blocked phrase detected",
                     ),
                 ));

litellm-rust/crates/gateway-inference/src/integrations/custom_logger/mod.rs

@@ -65,7 +65,7 @@ impl CustomLoggerRunner {
                 .await
             {
                 report.dropped += 1;
-                eprintln!("litellm-ai-gateway: async_log_success_event dropped: {err}");
+                eprintln!("litellm-gateway-inference: async_log_success_event dropped: {err}");
             }
         }
         report
@@ -89,7 +89,7 @@ impl CustomLoggerRunner {
                 .await
             {
                 report.dropped += 1;
-                eprintln!("litellm-ai-gateway: async_log_failure_event dropped: {err}");
+                eprintln!("litellm-gateway-inference: async_log_failure_event dropped: {err}");
             }
         }
         report

litellm-rust/crates/gateway-inference/src/integrations/litellm_python_proxy_api/mod.rs

@@ -186,12 +186,12 @@ async fn flush(client: &Client, url: &str, master_key: &str, batch: &mut Vec<Log
         Ok(resp) if resp.status().is_success() => {}
         Ok(resp) => {
             eprintln!(
-                "litellm-ai-gateway: callback logs POST returned {} to {url}",
+                "litellm-gateway-inference: callback logs POST returned {} to {url}",
                 resp.status()
             );
         }
         Err(err) => {
-            eprintln!("litellm-ai-gateway: callback logs POST failed to {url}: {err}");
+            eprintln!("litellm-gateway-inference: callback logs POST failed to {url}: {err}");
         }
     }
 }

litellm-rust/crates/gateway-inference/src/io/realtime.rs

@@ -295,7 +295,7 @@ mod tests {
 
     /// Live end-to-end check against OpenAI. Ignored by default (CI never runs
     /// it); run explicitly with `OPENAI_API_KEY` set:
-    ///   `cargo test -p litellm-ai-gateway realtime_invokes_openai -- --ignored --nocapture`
+    ///   `cargo test -p litellm-gateway-inference realtime_invokes_openai -- --ignored --nocapture`
     #[tokio::test]
     #[ignore = "hits the live OpenAI realtime API; needs OPENAI_API_KEY"]
     async fn realtime_invokes_openai_and_responds() {

litellm-rust/crates/gateway-inference/tests/ocr_lifecycle.rs

@@ -1,16 +1,16 @@
 use std::sync::{Arc, Mutex};
 use std::time::Duration;
 
-use litellm_ai_gateway::integrations::custom_guardrail::{
+use litellm_core::error::Error;
+use litellm_gateway_inference::integrations::custom_guardrail::{
     CustomGuardrail, GuardrailContext, GuardrailDecision, GuardrailError, GuardrailEventHook,
     GuardrailFuture, GuardrailRequest,
 };
-use litellm_ai_gateway::integrations::custom_logger::{
+use litellm_gateway_inference::integrations::custom_logger::{
     CallbackTiming, CallbackValue, CustomLogger, LogFuture, ModelCallDetails,
 };
-use litellm_ai_gateway::integrations::types::RequestMetadata;
-use litellm_ai_gateway::ocr::{OcrRequest, ocr};
-use litellm_core::error::Error;
+use litellm_gateway_inference::integrations::types::RequestMetadata;
+use litellm_gateway_inference::ocr::{OcrRequest, ocr};
 use serde_json::{Map, Value, json};
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::net::{TcpListener, TcpStream};

litellm-rust/crates/gateway-server/AGENTS.md

@@ -3,7 +3,7 @@
 The Axum server that fronts the reusable Rust gateway runtime. It owns HTTP/WS
 transport, startup config, auth extractors, and application state only.
 Deployment selection and transport-neutral orchestration live in
-`litellm-ai-gateway` or `core::router`, and provider calls live behind core route
+`litellm-gateway-inference` or `core::router`, and provider calls live behind core route
 entrypoints such as `litellm_core::messages::messages`. No provider handler
 lives here.
 
@@ -28,7 +28,7 @@ src/
   `transport`). See `routes/AGENTS.md`.
 - **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's
   args; it runs during extraction. Never re-implement the check per route.
-- **Handlers are thin.** A handler validates and delegates to `litellm-ai-gateway::runtime`. No
+- **Handlers are thin.** A handler validates and delegates to `litellm-gateway-inference::runtime`. No
   business logic, no provider calls, no transforms in handlers.
 - **Runtime services call `core`, they don't reimplement it.** The reusable
   service picks the deployment and calls the `core` route entrypoint. Provider

litellm-rust/crates/gateway-server/ARCHITECTURE.md

@@ -1,13 +1,14 @@
 # gateway-server architecture
 
-The Rust gateway server owns the Axum binary, HTTP/WebSocket routes, auth
-extractors, application state, and startup config. It delegates transport-neutral
-orchestration and callback integrations to `litellm-ai-gateway`
+The Rust gateway server is the root composition crate. It owns the Axum binary,
+HTTP/WebSocket routes, auth extractors, application state, and startup config.
+It delegates inference behavior to `litellm-gateway-inference`; future domain
+crates plug into this composition root the same way
 
 ```mermaid
 flowchart LR
   C[client] <--> S[litellm-gateway-server<br/>Axum host]
-  S --> G[litellm-ai-gateway<br/>runtime and integrations]
+  S --> G[litellm-gateway-inference<br/>runtime and integrations]
   G --> K[litellm-core]
   G <--> O[OpenAI realtime]
   G -. spend tracking callback .-> P[litellm proxy]

litellm-rust/crates/gateway-server/Cargo.toml

@@ -9,13 +9,13 @@ repository.workspace = true
 name = "litellm_gateway_server"
 
 [[bin]]
-name = "litellm-ai-gateway"
+name = "litellm-gateway-server"
 path = "src/main.rs"
 
 [dependencies]
 axum = { workspace = true, features = ["ws"] }
 futures-util.workspace = true
-litellm-ai-gateway.workspace = true
+litellm-gateway-inference.workspace = true
 litellm-config.workspace = true
 litellm-core = { workspace = true, features = ["bedrock-auth"] }
 reqwest.workspace = true

litellm-rust/crates/gateway-server/Dockerfile

@@ -5,7 +5,7 @@
 # which is not in any PyPI release yet) AND build the rust workspace under
 # litellm-rust/.
 #
-#   docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-ai-gateway .
+#   docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-gateway-server .
 #
 # No secrets live in this file. Runtime config (LITELLM_MASTER_KEY,
 # OPENAI_API_KEY referenced by config.yaml, etc.) is injected as environment
@@ -41,7 +41,7 @@ RUN cargo chef cook --locked --release \
 # Now copy the real sources and build the gateway binary. Deps are already cooked
 # above, so this step only recompiles the gateway crate.
 COPY litellm-rust/ .
-RUN cargo build --locked --release -p litellm-gateway-server --bin litellm-ai-gateway --features python-config
+RUN cargo build --locked --release -p litellm-gateway-server --bin litellm-gateway-server --features python-config
 
 # ---- Runtime ----------------------------------------------------------------
 # python:3.11-slim-bookworm ships libpython3.11, matching the builder's PyO3
@@ -64,7 +64,7 @@ RUN pip install --no-cache-dir ".[proxy]"
 
 # The compiled gateway binary (pure-Rust realtime hot path; Python is load-time
 # only).
-COPY --from=builder /build/litellm-rust/target/release/litellm-ai-gateway /usr/local/bin/litellm-ai-gateway
+COPY --from=builder /build/litellm-rust/target/release/litellm-gateway-server /usr/local/bin/litellm-gateway-server
 
 # Default config.yaml. A real deploy can override this (e.g. mount a Render
 # secret file at the same path) — never bake secrets into the image.
@@ -83,4 +83,4 @@ RUN useradd --system --no-create-home --uid 10001 appuser \
     && chown -R appuser:appuser /app
 USER appuser
 
-ENTRYPOINT ["/usr/local/bin/litellm-ai-gateway"]
+ENTRYPOINT ["/usr/local/bin/litellm-gateway-server"]

litellm-rust/crates/gateway-server/README.md

@@ -12,8 +12,8 @@ dials OpenAI upstream, and splices the two sockets frame-by-frame.
 |-------|------|
 | litellm-core | The LiteLLM SDK in Rust — per-route entrypoints (`messages::messages()`) that resolve the provider, transform, and make the call; plus types, provider transforms, and the router. |
 | litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. |
-| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by the server and Python bridge. |
-| litellm-gateway-server | Axum binary, HTTP/WebSocket routes, auth extractors, application state, startup config, and HTTP-only dependencies. |
+| litellm-gateway-inference | Framework-independent gateway runtime and integrations shared by the server and Python bridge. |
+| litellm-gateway-server | Root Axum binary and composition crate. Owns routes, auth, state, startup config, and HTTP-only dependencies. |
 | litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
 | litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. |
 
@@ -46,7 +46,7 @@ model_list:
 ```
 
 ```bash
-LITELLM_CONFIG_PATH=./config.yaml ./litellm-ai-gateway
+LITELLM_CONFIG_PATH=./config.yaml ./litellm-gateway-server
 ```
 
 At boot `litellm-config` calls into `litellm.proxy.read_model_list` and returns
@@ -113,13 +113,13 @@ repo's source** (the config reader is newer than any PyPI release), so the build
 
 ```bash
 # from the repo root
-docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-ai-gateway .
+docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-gateway-server .
 
 docker run --rm -p 4001:4001 \
   -e HOST=0.0.0.0 -e PORT=4001 \
   -e LITELLM_MASTER_KEY=sk-local \
   -e OPENAI_API_KEY=$OPENAI_API_KEY \
-  litellm-ai-gateway          # LITELLM_CONFIG_PATH defaults to /app/config.yaml
+  litellm-gateway-server          # LITELLM_CONFIG_PATH defaults to /app/config.yaml
 
 # smoke test
 curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/health/readiness   # -> 200
@@ -134,7 +134,7 @@ To use your own config, mount it over the default:
 docker run --rm -p 4001:4001 \
   -e HOST=0.0.0.0 -e LITELLM_MASTER_KEY=sk-local -e OPENAI_API_KEY=$OPENAI_API_KEY \
   -v $(pwd)/my-config.yaml:/app/config.yaml:ro \
-  litellm-ai-gateway
+  litellm-gateway-server
 ```
 
 ### Cargo-only (no Docker)
@@ -170,7 +170,7 @@ deploy. To use a non-default model_list, mount a **Render Secret File** at
 curl -X POST https://api.render.com/v1/services \
   -H "Authorization: Bearer $RENDER_API_KEY" -H "Content-Type: application/json" \
   -d '{
-    "type": "web_service", "name": "litellm-rust-ai-gateway",
+    "type": "web_service", "name": "litellm-rust-gateway-server",
     "ownerId": "<owner-id>", "repo": "https://github.com/BerriAI/litellm",
     "branch": "<branch-with-this-dockerfile>",
     "serviceDetails": {

litellm-rust/crates/gateway-server/render.yaml

@@ -11,7 +11,7 @@
 # them in the Render dashboard or via the API, never inline here.
 services:
   - type: web
-    name: litellm-rust-ai-gateway
+    name: litellm-rust-gateway-server
     runtime: docker
     plan: standard
     dockerfilePath: ./litellm-rust/crates/gateway-server/Dockerfile

litellm-rust/crates/gateway-server/src/main.rs

@@ -2,16 +2,16 @@
 //!
 //! The binary owns startup and config, then mounts the Axum routes from
 //! `litellm_gateway_server`. Transport-neutral runtime and integrations come
-//! from `litellm_ai_gateway`.
+//! from `litellm_gateway_inference`.
 
 use std::sync::Arc;
 
-use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
-use litellm_ai_gateway::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger;
-use litellm_ai_gateway::io::realtime_pool::{PoolConfig, RealtimePool, upstream_key};
 #[cfg(feature = "python-config")]
 use litellm_config::load_model_list;
 use litellm_core::router::{Deployment, LiteLLMParams, Router};
+use litellm_gateway_inference::integrations::custom_logger::CustomLogger;
+use litellm_gateway_inference::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger;
+use litellm_gateway_inference::io::realtime_pool::{PoolConfig, RealtimePool, upstream_key};
 use litellm_gateway_server::routes;
 use litellm_gateway_server::state::AppState;
 
@@ -72,7 +72,7 @@ async fn main() {
     let listener = tokio::net::TcpListener::bind((host.as_str(), port))
         .await
         .expect("failed to bind listener");
-    eprintln!("litellm-ai-gateway listening on {host}:{port}");
+    eprintln!("litellm-gateway-server listening on {host}:{port}");
     axum::serve(listener, routes::app(state))
         .await
         .expect("server error");

litellm-rust/crates/gateway-server/src/routes/AGENTS.md

@@ -17,7 +17,7 @@ async fn handle(...) -> impl IntoResponse { ... }
 
 ## Runtime boundary
 When a route has transport-neutral orchestration, put it under
-`litellm-ai-gateway::runtime` and test it there. The route file stays the Axum
+`litellm-gateway-inference::runtime` and test it there. The route file stays the Axum
 surface: router, handler, and socket or SSE adapter. Never build a provider
 request, resolve a provider key, or perform the provider call in this crate.
 
@@ -25,7 +25,7 @@ request, resolve a provider key, or perform the provider call in this crate.
 - **Auth is an extractor, not a manual call.** A handler requires auth by adding
   `crate::auth::RequireMasterKey` to its arguments; it runs during extraction.
   Never re-implement the check per route.
-- **Handlers contain no business logic; `litellm-ai-gateway` contains no Axum types.**
+- **Handlers contain no business logic; `litellm-gateway-inference` contains no Axum types.**
 - **No provider handlers in this crate.** Transforms, auth headers, and the
   provider HTTP call live in `core/src/<route>/`.
 - A route owns its paths in its own `router()`; `mod.rs` only merges.

litellm-rust/crates/gateway-server/src/routes/messages/mod.rs

@@ -7,8 +7,8 @@ use axum::http::StatusCode;
 use axum::http::header::{CACHE_CONTROL, CONTENT_TYPE, HeaderMap, HeaderValue};
 use axum::response::{IntoResponse, Response};
 use axum::routing::post;
-use litellm_ai_gateway::runtime::messages::{MessagesResponse, run};
 use litellm_core::Error;
+use litellm_gateway_inference::runtime::messages::{MessagesResponse, run};
 use serde_json::{Map, Value};
 
 use crate::auth::RequireMasterKey;
@@ -149,7 +149,7 @@ mod tests {
 
     use super::super::app;
     use crate::state::AppState;
-    use litellm_ai_gateway::io::realtime_pool::RealtimePool;
+    use litellm_gateway_inference::io::realtime_pool::RealtimePool;
 
     fn state(model: &str, api_base: String, master_key: Option<&str>) -> AppState {
         state_with_provider(model, model, api_base, master_key)

litellm-rust/crates/gateway-server/src/routes/realtime/README.md

@@ -2,8 +2,8 @@
 
 Proxies OpenAI's realtime WebSocket. `mod.rs` is the Axum surface (handler and
 socket-to-events adapter); the pure logic lives in
-`litellm-ai-gateway::runtime::realtime`. The pool lives in
-`litellm-ai-gateway::io::realtime_pool`.
+`litellm-gateway-inference::runtime::realtime`. The pool lives in
+`litellm-gateway-inference::io::realtime_pool`.
 
 ## Connection pooling
 
@@ -84,4 +84,4 @@ upstream sockets, which is why warm sockets are short-lived
   attempts against a broken key so it can't exhaust upstream rate limits and degrade
   valid cold-path traffic; the backoff resets the moment a dial succeeds.
 
-Benchmarks and repro: `../../../../ai-gateway/benchmarks/realtime/README.md`.
+Benchmarks and repro: `../../../../gateway-inference/benchmarks/realtime/README.md`.

litellm-rust/crates/gateway-server/src/routes/realtime/mod.rs

@@ -2,7 +2,7 @@
 //!
 //! This file is the **axum surface**: `router()`, the handler, and the small
 //! socket↔events adapter. The pure logic lives in
-//! [`litellm_ai_gateway::runtime::realtime`]. Auth is the `RequireMasterKey`
+//! [`litellm_gateway_inference::runtime::realtime`]. Auth is the `RequireMasterKey`
 //! extractor, so the handler stays thin.
 
 use std::sync::Arc;
@@ -16,13 +16,13 @@ use axum::http::StatusCode;
 use axum::response::Response;
 use axum::routing::get;
 use futures_util::{SinkExt, StreamExt};
-use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
-use litellm_ai_gateway::integrations::types::RequestMetadata;
-use litellm_ai_gateway::io::realtime_pool::RealtimePool;
-use litellm_ai_gateway::realtime::streaming::{RealTimeStreaming, SessionStatus};
-use litellm_ai_gateway::runtime::realtime;
 use litellm_core::realtime::types::RealtimeEvent;
 use litellm_core::router::Router as ModelRouter;
+use litellm_gateway_inference::integrations::custom_logger::CustomLogger;
+use litellm_gateway_inference::integrations::types::RequestMetadata;
+use litellm_gateway_inference::io::realtime_pool::RealtimePool;
+use litellm_gateway_inference::realtime::streaming::{RealTimeStreaming, SessionStatus};
+use litellm_gateway_inference::runtime::realtime;
 use serde::Deserialize;
 
 use crate::auth::RequireMasterKey;
@@ -84,7 +84,7 @@ async fn handle(
 }
 
 /// Adapt the axum socket (text frames) to the typed-event `Stream`/`Sink` the
-/// runtime wants, keeping axum types out of `litellm-ai-gateway`.
+/// runtime wants, keeping axum types out of `litellm-gateway-inference`.
 ///
 /// This is also the realtime-logging seam: every upstream→client event (the
 /// direction carrying `session.created` and `response.done` with usage) is fed

litellm-rust/crates/gateway-server/src/routes/responses/mod.rs

@@ -9,11 +9,11 @@ use axum::http::StatusCode;
 use axum::response::Response;
 use axum::routing::get;
 use futures_util::{Sink, SinkExt, StreamExt};
-use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
-use litellm_ai_gateway::integrations::types::RequestMetadata;
-use litellm_ai_gateway::runtime::responses;
 use litellm_core::responses::types::{ResponsesErrorFrame, ResponsesWsEvent, ResponsesWsEventType};
 use litellm_core::router::Router as ModelRouter;
+use litellm_gateway_inference::integrations::custom_logger::CustomLogger;
+use litellm_gateway_inference::integrations::types::RequestMetadata;
+use litellm_gateway_inference::runtime::responses;
 use serde::Deserialize;
 
 use crate::auth::RequireMasterKey;
@@ -241,8 +241,8 @@ mod tests {
     use crate::state::AppState;
     use axum::body::Body;
     use axum::http::Request;
-    use litellm_ai_gateway::io::realtime_pool::RealtimePool;
     use litellm_core::router::Router as ModelRouter;
+    use litellm_gateway_inference::io::realtime_pool::RealtimePool;
     use serde_json::json;
     use std::pin::Pin;
     use std::sync::Arc;

litellm-rust/crates/gateway-server/src/state.rs

@@ -1,8 +1,8 @@
 use std::sync::Arc;
 
-use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
-use litellm_ai_gateway::io::realtime_pool::RealtimePool;
 use litellm_core::router::Router;
+use litellm_gateway_inference::integrations::custom_logger::CustomLogger;
+use litellm_gateway_inference::io::realtime_pool::RealtimePool;
 
 /// Shared application state handed to every route handler.
 #[derive(Clone)]

litellm-rust/crates/python-bridge/Cargo.toml

@@ -17,14 +17,14 @@ panic-test = []
 trace-parity = [
     "dep:tracing",
     "litellm-core/observability",
-    "litellm-ai-gateway/trace-parity",
+    "litellm-gateway-inference/trace-parity",
 ]
 
 [dependencies]
 futures-util.workspace = true
 tracing = { workspace = true, optional = true }
 litellm-core = { workspace = true, features = ["bedrock-auth"] }
-litellm-ai-gateway = { workspace = true, default-features = false }
+litellm-gateway-inference = { workspace = true, default-features = false }
 litellm-python-interop.workspace = true
 pyo3.workspace = true
 pyo3-async-runtimes.workspace = true

litellm-rust/crates/python-bridge/src/lib.rs

@@ -6,7 +6,7 @@ mod function_trace;
 mod marshal;
 mod routes;
 
-use litellm_ai_gateway::io::responses_ws::ResponsesWebSocketConnection as RustResponsesWebSocketConnection;
+use litellm_gateway_inference::io::responses_ws::ResponsesWebSocketConnection as RustResponsesWebSocketConnection;
 use pyo3::prelude::*;
 use pyo3::types::PyAny;
 use serde_json::Value;

litellm-rust/crates/python-bridge/src/routes/gateway_messages.rs

@@ -11,7 +11,7 @@ fn gateway_messages<'py>(
     api_base: String,
     #[pyo3(from_py_with = litellm_python_interop::from_py)] body: Value,
 ) -> PyResult<Bound<'py, PyAny>> {
-    let future = litellm_ai_gateway::trace_parity::messages_request(
+    let future = litellm_gateway_inference::trace_parity::messages_request(
         model_alias,
         provider_model,
         api_base,

litellm-rust/crates/python-bridge/src/routes/ocr.rs

@@ -1,7 +1,7 @@
 use litellm_core::Error;
 use std::future::Future;
 
-use litellm_ai_gateway::io::ocr::{OcrRequest, ocr as run_ocr};
+use litellm_gateway_inference::io::ocr::{OcrRequest, ocr as run_ocr};
 use pyo3::prelude::*;
 use serde_json::Value;