diff --git a/.github/workflows/test-rust.yml b/.github/workflows/test-rust.yml index beeb9c46151..4fcc89953d3 100644 --- a/.github/workflows/test-rust.yml +++ b/.github/workflows/test-rust.yml @@ -75,7 +75,7 @@ jobs: run: cargo clippy -p litellm-core --all-targets --features bedrock-auth --locked -- -D warnings - name: Run Clippy with all gateway runtime features - run: cargo clippy -p litellm-ai-gateway --all-targets --all-features --locked -- -D warnings + run: cargo clippy -p litellm-gateway-inference --all-targets --all-features --locked -- -D warnings - name: Run Clippy for the gateway server run: cargo clippy -p litellm-gateway-server --all-targets --locked -- -D warnings diff --git a/litellm-rust/ADDING_A_PROVIDER.md b/litellm-rust/ADDING_A_PROVIDER.md index d7198cfcecf..7db399a6b60 100644 --- a/litellm-rust/ADDING_A_PROVIDER.md +++ b/litellm-rust/ADDING_A_PROVIDER.md @@ -26,4 +26,4 @@ variants of it. The test for a good abstraction is that adding the next provider is a few declarative lines, not a new file of duplicated flow. Only diverge from the base when behavior is genuinely different, and say so explicitly in the PR. -**Calling:** hosts invoke the core entrypoint. The Python bridge and the reusable `ai-gateway` runtime both call `litellm_core::messages::messages`, while `gateway-server` adapts HTTP requests to that runtime. Never add a provider handler to either host layer. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md). +**Calling:** hosts invoke the core entrypoint. The Python bridge and the reusable `gateway-inference` runtime both call `litellm_core::messages::messages`, while `gateway-server` adapts HTTP requests to that runtime. Never add a provider handler to either host layer. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md). diff --git a/litellm-rust/AGENTS.md b/litellm-rust/AGENTS.md index 066239be46c..54f6c00327c 100644 --- a/litellm-rust/AGENTS.md +++ b/litellm-rust/AGENTS.md @@ -8,12 +8,12 @@ litellm-rust has six crates. A crate is a layer, shared foundation, or separatel |-------|------| | litellm-core | The LiteLLM SDK in Rust. One public entrypoint per top-level call (`messages::messages()`), owning types, transforms, provider resolution, auth, and the provider HTTP call. Call it, get a typed response. | | litellm-config | Config-loading boundary. Returns resolved core deployment data and optionally delegates loading to Python. | -| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by the server and Python bridge. Owns transport-neutral orchestration and legacy call modules, but no Axum or Tower dependencies. | -| litellm-gateway-server | The Axum binary and HTTP/WebSocket host. Owns routes, auth extractors, application state, startup config, and HTTP-only Tower dependencies. | +| litellm-gateway-inference | Framework-independent gateway runtime and integrations shared by the server and Python bridge. Owns transport-neutral orchestration and legacy call modules, but no Axum or Tower dependencies. | +| litellm-gateway-server | The root Axum binary and composition crate. Owns route composition, auth extractors, application state, startup config, and HTTP-only Tower dependencies, then delegates domain behavior to extracted gateway crates. | | litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | | litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. | -Dependency direction is acyclic: `litellm-config` depends on `litellm-core`; `litellm-ai-gateway` depends on core; `litellm-gateway-server` depends on gateway, core, and config; and `litellm-python-bridge` depends on the reusable domain layers and `litellm-python-interop`. Reusable crates must not depend on `litellm-gateway-server`. The interop foundation depends on no LiteLLM domain crate. +Dependency direction is acyclic: `litellm-config` depends on `litellm-core`; `litellm-gateway-inference` depends on core; `litellm-gateway-server` depends on inference, core, and config; and `litellm-python-bridge` depends on the reusable domain layers and `litellm-python-interop`. Reusable crates must not depend on `litellm-gateway-server`. The interop foundation depends on no LiteLLM domain crate. ## Where a route lives @@ -29,7 +29,7 @@ core/src/messages/ client.rs # the shared reqwest client ``` -Provider handlers never live in `gateway-server`. `ocr`, `audio_transcription`, and realtime provider I/O are still hosted in `ai-gateway` from before this rule; they move to `core` as they are touched. +Provider handlers never live in `gateway-server`. `ocr`, `audio_transcription`, and realtime provider I/O are still hosted in `gateway-inference` from before this rule; they move to `core` as they are touched. Adding a crate: default to a module. A new crate requires a real trigger: separate artifact (binary/cdylib), proc-macro, shared foundation, or publishable standalone. A new provider or route is none of these. diff --git a/litellm-rust/CLAUDE.md b/litellm-rust/CLAUDE.md index e594818a810..7f394da2860 100644 --- a/litellm-rust/CLAUDE.md +++ b/litellm-rust/CLAUDE.md @@ -25,8 +25,8 @@ the base when behavior is genuinely different, and say so explicitly in the PR. `litellm-core` **is** the LiteLLM SDK in Rust: it makes the LLM call. `litellm-config` is the config-loading boundary and returns resolved core types. -`litellm-ai-gateway` is the framework-independent gateway runtime and integration -layer. `litellm-gateway-server` is the HTTP/WebSocket server in front of it, and +`litellm-gateway-inference` is the framework-independent inference domain. +`litellm-gateway-server` is the root HTTP/WebSocket composition crate, and `litellm-python-bridge` exposes reusable Rust APIs to the Python SDK. `litellm-python-interop` holds domain-neutral PyO3 primitives shared by Python-facing Rust code. A crate is a layer, shared foundation, or separate host, @@ -83,7 +83,7 @@ Env reads in `core` are limited to credential fallback inside a route's no key is passed. Everything else config-shaped is resolved by the host and passed in. -Legacy call runtimes still hosted in `ai-gateway` (`ocr`, +Legacy call runtimes still hosted in `gateway-inference` (`ocr`, `audio_transcription`, realtime provider I/O) predate this rule and are being moved into `core` route modules; do not add new ones there, and prefer moving one when you touch it. Axum routes, auth extractors, application state, and @@ -181,7 +181,7 @@ cd litellm-rust cargo fmt --check cargo clippy --workspace --all-targets -- -D warnings cargo clippy -p litellm-core --all-targets --features bedrock-auth -- -D warnings -cargo clippy -p litellm-ai-gateway --all-targets --all-features -- -D warnings +cargo clippy -p litellm-gateway-inference --all-targets --all-features -- -D warnings cargo clippy -p litellm-gateway-server --all-targets -- -D warnings cargo test --workspace cargo test -p litellm-core --features bedrock-auth diff --git a/litellm-rust/Cargo.lock b/litellm-rust/Cargo.lock index 6ead3ea2ce1..a58664b0052 100644 --- a/litellm-rust/Cargo.lock +++ b/litellm-rust/Cargo.lock @@ -1404,22 +1404,6 @@ version = "0.2.186" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" -[[package]] -name = "litellm-ai-gateway" -version = "0.1.0" -dependencies = [ - "base64", - "futures-channel", - "futures-util", - "litellm-core", - "reqwest", - "serde", - "serde_json", - "tokio", - "tokio-tungstenite", - "tracing", -] - [[package]] name = "litellm-config" version = "0.1.0" @@ -1453,15 +1437,31 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "litellm-gateway-inference" +version = "0.1.0" +dependencies = [ + "base64", + "futures-channel", + "futures-util", + "litellm-core", + "reqwest", + "serde", + "serde_json", + "tokio", + "tokio-tungstenite", + "tracing", +] + [[package]] name = "litellm-gateway-server" version = "0.1.0" dependencies = [ "axum", "futures-util", - "litellm-ai-gateway", "litellm-config", "litellm-core", + "litellm-gateway-inference", "reqwest", "serde", "serde_json", @@ -1478,8 +1478,8 @@ version = "0.1.0" dependencies = [ "criterion", "futures-util", - "litellm-ai-gateway", "litellm-core", + "litellm-gateway-inference", "litellm-python-interop", "pyo3", "pyo3-async-runtimes", diff --git a/litellm-rust/Cargo.toml b/litellm-rust/Cargo.toml index 5cf3a5178b4..840c33aa81a 100644 --- a/litellm-rust/Cargo.toml +++ b/litellm-rust/Cargo.toml @@ -2,7 +2,7 @@ members = [ "crates/core", "crates/config", - "crates/ai-gateway", + "crates/gateway-inference", "crates/gateway-server", "crates/python-interop", "crates/python-bridge", @@ -20,7 +20,7 @@ tracing = "0.1" tracing-subscriber = { version = "0.3", default-features = false, features = ["registry", "std"] } litellm-core = { path = "crates/core" } litellm-config = { path = "crates/config" } -litellm-ai-gateway = { path = "crates/ai-gateway", default-features = false } +litellm-gateway-inference = { path = "crates/gateway-inference", default-features = false } litellm-gateway-server = { path = "crates/gateway-server" } litellm-python-interop = { path = "crates/python-interop" } axum = "0.7" diff --git a/litellm-rust/README.md b/litellm-rust/README.md index 7a756903c30..e5f634f75fe 100644 --- a/litellm-rust/README.md +++ b/litellm-rust/README.md @@ -26,8 +26,8 @@ coverage and production evidence. |-------|------| | litellm-core | The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router. | | litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. | -| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by server and Python hosts. | -| litellm-gateway-server | Axum binary, HTTP/WebSocket routes, auth extractors, application state, and HTTP-only dependencies. | +| litellm-gateway-inference | Framework-independent inference services and integrations shared by server and Python hosts. | +| litellm-gateway-server | Root Axum binary and composition crate. Owns routes, auth, state, startup, and HTTP-only dependencies. | | litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | | litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. | @@ -41,8 +41,8 @@ crates/ src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client src/providers/anthropic/messages/transformation.rs config/ Config loading and resolved deployments. - ai-gateway/ Framework-independent gateway runtime and integrations. - gateway-server/ Axum binary and HTTP/WebSocket host. + gateway-inference/ Framework-independent inference runtime and integrations. + gateway-server/ Root Axum binary and route composition host. python-interop/ Domain-neutral PyO3 conversion and GIL primitives. python-bridge/ PyO3 API adapter for Python LiteLLM. ``` diff --git a/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md b/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md index a529a71f803..2b9f61cff02 100644 --- a/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md +++ b/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md @@ -16,7 +16,7 @@ Rules for adding or changing an LLM provider/route in `litellm-rust`. `messages` ## Boundaries -7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `ai-gateway` = framework-independent gateway runtime and integrations; `gateway-server` = HTTP/WS routing, extractors, auth of *our* callers, application state, and streaming to the client; `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint or reusable gateway runtime; they never build a provider request. +7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `gateway-inference` = framework-independent gateway runtime and integrations; `gateway-server` = HTTP/WS routing, extractors, auth of *our* callers, application state, and streaming to the client; `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint or reusable gateway runtime; they never build a provider request. 8. Generic/route files contain zero provider-specific branches. A provider is one module under `core/src/providers///`; a route is a module, never a new crate. 9. Route entry point stays thin: `core::::()` -> `prepare_*` -> handler (or `CallLifecycle::run_request`, which owns the pre_call -> during_call -> provider call -> success/failure order and phase timing). Axum handlers validate and delegate to a service that calls the entrypoint; no business logic in them. 10. Constants (URLs, env-var names, API versions, error messages) live in a crate `constants.rs`, never inline. Config-shaped env reads happen at the host/config layer with the `DEFAULT_*` fallback defined in `constants.rs`; the only env read in `core` is the credential fallback in a route's `prepare.rs`. diff --git a/litellm-rust/crates/ai-gateway/ARCHITECTURE.md b/litellm-rust/crates/ai-gateway/ARCHITECTURE.md deleted file mode 100644 index 796a3b6506f..00000000000 --- a/litellm-rust/crates/ai-gateway/ARCHITECTURE.md +++ /dev/null @@ -1,17 +0,0 @@ -# ai-gateway architecture - -`litellm-ai-gateway` is between hosts and `litellm-core`. It exposes -framework-independent runtime services and callback integrations without -depending on an HTTP framework - -```mermaid -flowchart LR - S[litellm-gateway-server] --> G[litellm-ai-gateway runtime] - B[litellm-python-bridge] --> G - G --> C[litellm-core] - S --> C - S --> F[litellm-config] -``` - -The server may depend on the runtime, core, and config crates. Reusable crates -must not depend on the server diff --git a/litellm-rust/crates/core/src/call_lifecycle/README.md b/litellm-rust/crates/core/src/call_lifecycle/README.md index 692e249ef27..c11584ef61a 100644 --- a/litellm-rust/crates/core/src/call_lifecycle/README.md +++ b/litellm-rust/crates/core/src/call_lifecycle/README.md @@ -51,7 +51,7 @@ of editing OCR, chat, messages, responses, completions, or provider modules. Each migrated call type should use this folder shape: ```text -litellm-rust/crates/ai-gateway/src// +litellm-rust/crates/gateway-inference/src// mod.rs # thin public entrypoint types.rs # public request, prepared request, provider request, response types prepare.rs # model/provider/callback/guardrail setup diff --git a/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs b/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs index 2d82905882d..7c2cec91988 100644 --- a/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs +++ b/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs @@ -1,7 +1,7 @@ //! Enforcement: the litellm-rust workspace has exactly six crates. //! //! `core` (the Rust SDK), `config` (the config-loading boundary), -//! `ai-gateway` (framework-independent gateway runtime), +//! `gateway-inference` (framework-independent gateway runtime), //! `gateway-server` (the HTTP/WebSocket host), //! `python-interop` (domain-neutral PyO3 primitives), and `python-bridge` (the //! PyO3 cdylib). Adding or removing a crate must be a @@ -22,7 +22,7 @@ use std::path::{Path, PathBuf}; const EXPECTED_MEMBERS: &[&str] = &[ "crates/core", "crates/config", - "crates/ai-gateway", + "crates/gateway-inference", "crates/gateway-server", "crates/python-interop", "crates/python-bridge", @@ -32,7 +32,7 @@ const EXPECTED_MEMBERS: &[&str] = &[ const EXPECTED_CRATE_DIRS: &[&str] = &[ "core", "config", - "ai-gateway", + "gateway-inference", "gateway-server", "python-interop", "python-bridge", diff --git a/litellm-rust/crates/ai-gateway/AGENTS.md b/litellm-rust/crates/gateway-inference/AGENTS.md similarity index 53% rename from litellm-rust/crates/ai-gateway/AGENTS.md rename to litellm-rust/crates/gateway-inference/AGENTS.md index 91812cc3385..822dec7b70e 100644 --- a/litellm-rust/crates/ai-gateway/AGENTS.md +++ b/litellm-rust/crates/gateway-inference/AGENTS.md @@ -1,11 +1,11 @@ -# ai-gateway folder architecture +# gateway-inference folder architecture -This crate is a reusable, framework-independent gateway runtime and integration -library. It is used by `litellm-gateway-server` and `litellm-python-bridge` +This crate is the reusable, framework-independent inference domain. It is used +by `litellm-gateway-server` and `litellm-python-bridge` Axum routes, auth extractors, application state, startup, and Tower dependencies belong in `litellm-gateway-server`. This crate must not depend on the server -Transport-neutral route orchestration lives under `src/runtime/`. Existing OCR, +Transport-neutral inference orchestration lives under `src/runtime/`. Existing OCR, audio transcription, realtime I/O, and callback integrations remain here as migration seams until their provider call paths move into `litellm-core` diff --git a/litellm-rust/crates/gateway-inference/ARCHITECTURE.md b/litellm-rust/crates/gateway-inference/ARCHITECTURE.md new file mode 100644 index 00000000000..cb406405ddd --- /dev/null +++ b/litellm-rust/crates/gateway-inference/ARCHITECTURE.md @@ -0,0 +1,17 @@ +# gateway-inference architecture + +`litellm-gateway-inference` is the first extracted gateway domain. It sits +between hosts and `litellm-core`, exposing framework-independent inference +services and callback integrations without depending on an HTTP framework + +```mermaid +flowchart LR + S[litellm-gateway-server] --> G[litellm-gateway-inference runtime] + B[litellm-python-bridge] --> G + G --> C[litellm-core] + S --> C + S --> F[litellm-config] +``` + +The server may depend on the runtime, core, and config crates. Reusable crates +must not depend on the server diff --git a/litellm-rust/crates/ai-gateway/Cargo.toml b/litellm-rust/crates/gateway-inference/Cargo.toml similarity index 91% rename from litellm-rust/crates/ai-gateway/Cargo.toml rename to litellm-rust/crates/gateway-inference/Cargo.toml index 6c838110a20..cacec934933 100644 --- a/litellm-rust/crates/ai-gateway/Cargo.toml +++ b/litellm-rust/crates/gateway-inference/Cargo.toml @@ -1,12 +1,12 @@ [package] -name = "litellm-ai-gateway" +name = "litellm-gateway-inference" version = "0.1.0" edition.workspace = true license.workspace = true repository.workspace = true [lib] -name = "litellm_ai_gateway" +name = "litellm_gateway_inference" [dependencies] tracing.workspace = true diff --git a/litellm-rust/crates/ai-gateway/README.md b/litellm-rust/crates/gateway-inference/README.md similarity index 56% rename from litellm-rust/crates/ai-gateway/README.md rename to litellm-rust/crates/gateway-inference/README.md index dfd3af80164..f8ca138bf4e 100644 --- a/litellm-rust/crates/ai-gateway/README.md +++ b/litellm-rust/crates/gateway-inference/README.md @@ -1,9 +1,9 @@ -# LiteLLM AI Gateway Runtime +# LiteLLM Gateway Inference -`litellm-ai-gateway` is the reusable, framework-independent runtime used by the -Rust gateway server and Python bridge +`litellm-gateway-inference` is the reusable, framework-independent inference +domain used by the Rust gateway server and Python bridge -It owns gateway integrations, transport-neutral route orchestration, and legacy +It owns inference integrations, transport-neutral orchestration, and legacy OCR, audio transcription, and realtime I/O that have not yet moved into `litellm-core`. It has no Axum or Tower dependency diff --git a/litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md b/litellm-rust/crates/gateway-inference/benchmarks/realtime/README.md similarity index 100% rename from litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md rename to litellm-rust/crates/gateway-inference/benchmarks/realtime/README.md diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/hooks.rs b/litellm-rust/crates/gateway-inference/src/audio_transcription/hooks.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/audio_transcription/hooks.rs rename to litellm-rust/crates/gateway-inference/src/audio_transcription/hooks.rs diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/mod.rs b/litellm-rust/crates/gateway-inference/src/audio_transcription/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/audio_transcription/mod.rs rename to litellm-rust/crates/gateway-inference/src/audio_transcription/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/prepare.rs b/litellm-rust/crates/gateway-inference/src/audio_transcription/prepare.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/audio_transcription/prepare.rs rename to litellm-rust/crates/gateway-inference/src/audio_transcription/prepare.rs diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/tests.rs b/litellm-rust/crates/gateway-inference/src/audio_transcription/tests.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/audio_transcription/tests.rs rename to litellm-rust/crates/gateway-inference/src/audio_transcription/tests.rs diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/types.rs b/litellm-rust/crates/gateway-inference/src/audio_transcription/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/audio_transcription/types.rs rename to litellm-rust/crates/gateway-inference/src/audio_transcription/types.rs diff --git a/litellm-rust/crates/ai-gateway/src/client.rs b/litellm-rust/crates/gateway-inference/src/client.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/client.rs rename to litellm-rust/crates/gateway-inference/src/client.rs diff --git a/litellm-rust/crates/ai-gateway/src/constants.rs b/litellm-rust/crates/gateway-inference/src/constants.rs similarity index 96% rename from litellm-rust/crates/ai-gateway/src/constants.rs rename to litellm-rust/crates/gateway-inference/src/constants.rs index 1258cb4eb3b..c83215c8199 100644 --- a/litellm-rust/crates/ai-gateway/src/constants.rs +++ b/litellm-rust/crates/gateway-inference/src/constants.rs @@ -1,4 +1,4 @@ -//! Crate-level constants for the ai-gateway. +//! Crate-level constants for the gateway-inference. //! //! Per `litellm-rust/CLAUDE.md`, magic numbers and fixed strings live here //! (the Rust mirror of Python's `litellm/constants.py`), not inline in feature diff --git a/litellm-rust/crates/ai-gateway/src/integrations/README.md b/litellm-rust/crates/gateway-inference/src/integrations/README.md similarity index 94% rename from litellm-rust/crates/ai-gateway/src/integrations/README.md rename to litellm-rust/crates/gateway-inference/src/integrations/README.md index 16a162dac57..11065811dcc 100644 --- a/litellm-rust/crates/ai-gateway/src/integrations/README.md +++ b/litellm-rust/crates/gateway-inference/src/integrations/README.md @@ -26,7 +26,7 @@ Implement `CustomLogger` when Rust code needs to observe terminal success or failure events. Method names intentionally match Python `CustomLogger` names. ```rust -use litellm_ai_gateway::integrations::custom_logger::{ +use litellm_gateway_inference::integrations::custom_logger::{ CallbackTiming, CallbackValue, CustomLogger, LogFuture, ModelCallDetails, }; @@ -80,7 +80,7 @@ during-call checks. Method names intentionally match Python `CustomGuardrail` entrypoints inherited from Python `CustomLogger`. ```rust -use litellm_ai_gateway::integrations::custom_guardrail::{ +use litellm_gateway_inference::integrations::custom_guardrail::{ CustomGuardrail, GuardrailContext, GuardrailDecision, GuardrailEventHook, GuardrailFuture, GuardrailRequest, }; @@ -104,7 +104,7 @@ impl CustomGuardrail for BlocklistedPromptGuardrail { Box::pin(async move { if request.data.to_string().contains("blocked phrase") { return Ok(GuardrailDecision::Block( - litellm_ai_gateway::integrations::custom_guardrail::GuardrailError::blocked( + litellm_gateway_inference::integrations::custom_guardrail::GuardrailError::blocked( "blocked phrase detected", ), )); diff --git a/litellm-rust/crates/ai-gateway/src/integrations/custom_guardrail/mod.rs b/litellm-rust/crates/gateway-inference/src/integrations/custom_guardrail/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/custom_guardrail/mod.rs rename to litellm-rust/crates/gateway-inference/src/integrations/custom_guardrail/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/custom_guardrail/types.rs b/litellm-rust/crates/gateway-inference/src/integrations/custom_guardrail/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/custom_guardrail/types.rs rename to litellm-rust/crates/gateway-inference/src/integrations/custom_guardrail/types.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/custom_logger/mod.rs b/litellm-rust/crates/gateway-inference/src/integrations/custom_logger/mod.rs similarity index 98% rename from litellm-rust/crates/ai-gateway/src/integrations/custom_logger/mod.rs rename to litellm-rust/crates/gateway-inference/src/integrations/custom_logger/mod.rs index 792717dacfc..ce139b84ce9 100644 --- a/litellm-rust/crates/ai-gateway/src/integrations/custom_logger/mod.rs +++ b/litellm-rust/crates/gateway-inference/src/integrations/custom_logger/mod.rs @@ -65,7 +65,7 @@ impl CustomLoggerRunner { .await { report.dropped += 1; - eprintln!("litellm-ai-gateway: async_log_success_event dropped: {err}"); + eprintln!("litellm-gateway-inference: async_log_success_event dropped: {err}"); } } report @@ -89,7 +89,7 @@ impl CustomLoggerRunner { .await { report.dropped += 1; - eprintln!("litellm-ai-gateway: async_log_failure_event dropped: {err}"); + eprintln!("litellm-gateway-inference: async_log_failure_event dropped: {err}"); } } report diff --git a/litellm-rust/crates/ai-gateway/src/integrations/custom_logger/types.rs b/litellm-rust/crates/gateway-inference/src/integrations/custom_logger/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/custom_logger/types.rs rename to litellm-rust/crates/gateway-inference/src/integrations/custom_logger/types.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/mod.rs b/litellm-rust/crates/gateway-inference/src/integrations/litellm_python_proxy_api/mod.rs similarity index 97% rename from litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/mod.rs rename to litellm-rust/crates/gateway-inference/src/integrations/litellm_python_proxy_api/mod.rs index 3dad18cb7a3..782f6e4457b 100644 --- a/litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/mod.rs +++ b/litellm-rust/crates/gateway-inference/src/integrations/litellm_python_proxy_api/mod.rs @@ -186,12 +186,12 @@ async fn flush(client: &Client, url: &str, master_key: &str, batch: &mut Vec {} Ok(resp) => { eprintln!( - "litellm-ai-gateway: callback logs POST returned {} to {url}", + "litellm-gateway-inference: callback logs POST returned {} to {url}", resp.status() ); } Err(err) => { - eprintln!("litellm-ai-gateway: callback logs POST failed to {url}: {err}"); + eprintln!("litellm-gateway-inference: callback logs POST failed to {url}: {err}"); } } } diff --git a/litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/types.rs b/litellm-rust/crates/gateway-inference/src/integrations/litellm_python_proxy_api/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/types.rs rename to litellm-rust/crates/gateway-inference/src/integrations/litellm_python_proxy_api/types.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/mod.rs b/litellm-rust/crates/gateway-inference/src/integrations/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/mod.rs rename to litellm-rust/crates/gateway-inference/src/integrations/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/types.rs b/litellm-rust/crates/gateway-inference/src/integrations/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/types.rs rename to litellm-rust/crates/gateway-inference/src/integrations/types.rs diff --git a/litellm-rust/crates/ai-gateway/src/io/audio_transcription.rs b/litellm-rust/crates/gateway-inference/src/io/audio_transcription.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/io/audio_transcription.rs rename to litellm-rust/crates/gateway-inference/src/io/audio_transcription.rs diff --git a/litellm-rust/crates/ai-gateway/src/io/mod.rs b/litellm-rust/crates/gateway-inference/src/io/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/io/mod.rs rename to litellm-rust/crates/gateway-inference/src/io/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/io/ocr.rs b/litellm-rust/crates/gateway-inference/src/io/ocr.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/io/ocr.rs rename to litellm-rust/crates/gateway-inference/src/io/ocr.rs diff --git a/litellm-rust/crates/ai-gateway/src/io/realtime.rs b/litellm-rust/crates/gateway-inference/src/io/realtime.rs similarity index 99% rename from litellm-rust/crates/ai-gateway/src/io/realtime.rs rename to litellm-rust/crates/gateway-inference/src/io/realtime.rs index b2e4635fbe7..bee8b77a4d1 100644 --- a/litellm-rust/crates/ai-gateway/src/io/realtime.rs +++ b/litellm-rust/crates/gateway-inference/src/io/realtime.rs @@ -295,7 +295,7 @@ mod tests { /// Live end-to-end check against OpenAI. Ignored by default (CI never runs /// it); run explicitly with `OPENAI_API_KEY` set: - /// `cargo test -p litellm-ai-gateway realtime_invokes_openai -- --ignored --nocapture` + /// `cargo test -p litellm-gateway-inference realtime_invokes_openai -- --ignored --nocapture` #[tokio::test] #[ignore = "hits the live OpenAI realtime API; needs OPENAI_API_KEY"] async fn realtime_invokes_openai_and_responds() { diff --git a/litellm-rust/crates/ai-gateway/src/io/realtime_pool.rs b/litellm-rust/crates/gateway-inference/src/io/realtime_pool.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/io/realtime_pool.rs rename to litellm-rust/crates/gateway-inference/src/io/realtime_pool.rs diff --git a/litellm-rust/crates/ai-gateway/src/io/responses_ws.rs b/litellm-rust/crates/gateway-inference/src/io/responses_ws.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/io/responses_ws.rs rename to litellm-rust/crates/gateway-inference/src/io/responses_ws.rs diff --git a/litellm-rust/crates/ai-gateway/src/lib.rs b/litellm-rust/crates/gateway-inference/src/lib.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/lib.rs rename to litellm-rust/crates/gateway-inference/src/lib.rs diff --git a/litellm-rust/crates/ai-gateway/src/ocr/common_utils.rs b/litellm-rust/crates/gateway-inference/src/ocr/common_utils.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/ocr/common_utils.rs rename to litellm-rust/crates/gateway-inference/src/ocr/common_utils.rs diff --git a/litellm-rust/crates/ai-gateway/src/ocr/handler.rs b/litellm-rust/crates/gateway-inference/src/ocr/handler.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/ocr/handler.rs rename to litellm-rust/crates/gateway-inference/src/ocr/handler.rs diff --git a/litellm-rust/crates/ai-gateway/src/ocr/hooks.rs b/litellm-rust/crates/gateway-inference/src/ocr/hooks.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/ocr/hooks.rs rename to litellm-rust/crates/gateway-inference/src/ocr/hooks.rs diff --git a/litellm-rust/crates/ai-gateway/src/ocr/mod.rs b/litellm-rust/crates/gateway-inference/src/ocr/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/ocr/mod.rs rename to litellm-rust/crates/gateway-inference/src/ocr/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/ocr/prepare.rs b/litellm-rust/crates/gateway-inference/src/ocr/prepare.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/ocr/prepare.rs rename to litellm-rust/crates/gateway-inference/src/ocr/prepare.rs diff --git a/litellm-rust/crates/ai-gateway/src/ocr/types.rs b/litellm-rust/crates/gateway-inference/src/ocr/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/ocr/types.rs rename to litellm-rust/crates/gateway-inference/src/ocr/types.rs diff --git a/litellm-rust/crates/ai-gateway/src/realtime/mod.rs b/litellm-rust/crates/gateway-inference/src/realtime/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/realtime/mod.rs rename to litellm-rust/crates/gateway-inference/src/realtime/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/realtime/streaming.rs b/litellm-rust/crates/gateway-inference/src/realtime/streaming.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/realtime/streaming.rs rename to litellm-rust/crates/gateway-inference/src/realtime/streaming.rs diff --git a/litellm-rust/crates/ai-gateway/src/runtime/messages.rs b/litellm-rust/crates/gateway-inference/src/runtime/messages.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/runtime/messages.rs rename to litellm-rust/crates/gateway-inference/src/runtime/messages.rs diff --git a/litellm-rust/crates/ai-gateway/src/runtime/mod.rs b/litellm-rust/crates/gateway-inference/src/runtime/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/runtime/mod.rs rename to litellm-rust/crates/gateway-inference/src/runtime/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/runtime/realtime.rs b/litellm-rust/crates/gateway-inference/src/runtime/realtime.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/runtime/realtime.rs rename to litellm-rust/crates/gateway-inference/src/runtime/realtime.rs diff --git a/litellm-rust/crates/ai-gateway/src/runtime/responses.rs b/litellm-rust/crates/gateway-inference/src/runtime/responses.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/runtime/responses.rs rename to litellm-rust/crates/gateway-inference/src/runtime/responses.rs diff --git a/litellm-rust/crates/ai-gateway/src/trace_parity.rs b/litellm-rust/crates/gateway-inference/src/trace_parity.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/trace_parity.rs rename to litellm-rust/crates/gateway-inference/src/trace_parity.rs diff --git a/litellm-rust/crates/ai-gateway/tests/ocr_lifecycle.rs b/litellm-rust/crates/gateway-inference/tests/ocr_lifecycle.rs similarity index 98% rename from litellm-rust/crates/ai-gateway/tests/ocr_lifecycle.rs rename to litellm-rust/crates/gateway-inference/tests/ocr_lifecycle.rs index c3a89f4394d..176ff1be6f3 100644 --- a/litellm-rust/crates/ai-gateway/tests/ocr_lifecycle.rs +++ b/litellm-rust/crates/gateway-inference/tests/ocr_lifecycle.rs @@ -1,16 +1,16 @@ use std::sync::{Arc, Mutex}; use std::time::Duration; -use litellm_ai_gateway::integrations::custom_guardrail::{ +use litellm_core::error::Error; +use litellm_gateway_inference::integrations::custom_guardrail::{ CustomGuardrail, GuardrailContext, GuardrailDecision, GuardrailError, GuardrailEventHook, GuardrailFuture, GuardrailRequest, }; -use litellm_ai_gateway::integrations::custom_logger::{ +use litellm_gateway_inference::integrations::custom_logger::{ CallbackTiming, CallbackValue, CustomLogger, LogFuture, ModelCallDetails, }; -use litellm_ai_gateway::integrations::types::RequestMetadata; -use litellm_ai_gateway::ocr::{OcrRequest, ocr}; -use litellm_core::error::Error; +use litellm_gateway_inference::integrations::types::RequestMetadata; +use litellm_gateway_inference::ocr::{OcrRequest, ocr}; use serde_json::{Map, Value, json}; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; diff --git a/litellm-rust/crates/gateway-server/AGENTS.md b/litellm-rust/crates/gateway-server/AGENTS.md index 8eec0bf9ac6..1cf61e94e91 100644 --- a/litellm-rust/crates/gateway-server/AGENTS.md +++ b/litellm-rust/crates/gateway-server/AGENTS.md @@ -3,7 +3,7 @@ The Axum server that fronts the reusable Rust gateway runtime. It owns HTTP/WS transport, startup config, auth extractors, and application state only. Deployment selection and transport-neutral orchestration live in -`litellm-ai-gateway` or `core::router`, and provider calls live behind core route +`litellm-gateway-inference` or `core::router`, and provider calls live behind core route entrypoints such as `litellm_core::messages::messages`. No provider handler lives here. @@ -28,7 +28,7 @@ src/ `transport`). See `routes/AGENTS.md`. - **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's args; it runs during extraction. Never re-implement the check per route. -- **Handlers are thin.** A handler validates and delegates to `litellm-ai-gateway::runtime`. No +- **Handlers are thin.** A handler validates and delegates to `litellm-gateway-inference::runtime`. No business logic, no provider calls, no transforms in handlers. - **Runtime services call `core`, they don't reimplement it.** The reusable service picks the deployment and calls the `core` route entrypoint. Provider diff --git a/litellm-rust/crates/gateway-server/ARCHITECTURE.md b/litellm-rust/crates/gateway-server/ARCHITECTURE.md index 9902368bb2d..ce7fce5dec5 100644 --- a/litellm-rust/crates/gateway-server/ARCHITECTURE.md +++ b/litellm-rust/crates/gateway-server/ARCHITECTURE.md @@ -1,13 +1,14 @@ # gateway-server architecture -The Rust gateway server owns the Axum binary, HTTP/WebSocket routes, auth -extractors, application state, and startup config. It delegates transport-neutral -orchestration and callback integrations to `litellm-ai-gateway` +The Rust gateway server is the root composition crate. It owns the Axum binary, +HTTP/WebSocket routes, auth extractors, application state, and startup config. +It delegates inference behavior to `litellm-gateway-inference`; future domain +crates plug into this composition root the same way ```mermaid flowchart LR C[client] <--> S[litellm-gateway-server
Axum host] - S --> G[litellm-ai-gateway
runtime and integrations] + S --> G[litellm-gateway-inference
runtime and integrations] G --> K[litellm-core] G <--> O[OpenAI realtime] G -. spend tracking callback .-> P[litellm proxy] diff --git a/litellm-rust/crates/gateway-server/Cargo.toml b/litellm-rust/crates/gateway-server/Cargo.toml index 0ed6b747971..5fd51b2c0ab 100644 --- a/litellm-rust/crates/gateway-server/Cargo.toml +++ b/litellm-rust/crates/gateway-server/Cargo.toml @@ -9,13 +9,13 @@ repository.workspace = true name = "litellm_gateway_server" [[bin]] -name = "litellm-ai-gateway" +name = "litellm-gateway-server" path = "src/main.rs" [dependencies] axum = { workspace = true, features = ["ws"] } futures-util.workspace = true -litellm-ai-gateway.workspace = true +litellm-gateway-inference.workspace = true litellm-config.workspace = true litellm-core = { workspace = true, features = ["bedrock-auth"] } reqwest.workspace = true diff --git a/litellm-rust/crates/gateway-server/Dockerfile b/litellm-rust/crates/gateway-server/Dockerfile index 3cbaeae5875..9e7ce071e3a 100644 --- a/litellm-rust/crates/gateway-server/Dockerfile +++ b/litellm-rust/crates/gateway-server/Dockerfile @@ -5,7 +5,7 @@ # which is not in any PyPI release yet) AND build the rust workspace under # litellm-rust/. # -# docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-ai-gateway . +# docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-gateway-server . # # No secrets live in this file. Runtime config (LITELLM_MASTER_KEY, # OPENAI_API_KEY referenced by config.yaml, etc.) is injected as environment @@ -41,7 +41,7 @@ RUN cargo chef cook --locked --release \ # Now copy the real sources and build the gateway binary. Deps are already cooked # above, so this step only recompiles the gateway crate. COPY litellm-rust/ . -RUN cargo build --locked --release -p litellm-gateway-server --bin litellm-ai-gateway --features python-config +RUN cargo build --locked --release -p litellm-gateway-server --bin litellm-gateway-server --features python-config # ---- Runtime ---------------------------------------------------------------- # python:3.11-slim-bookworm ships libpython3.11, matching the builder's PyO3 @@ -64,7 +64,7 @@ RUN pip install --no-cache-dir ".[proxy]" # The compiled gateway binary (pure-Rust realtime hot path; Python is load-time # only). -COPY --from=builder /build/litellm-rust/target/release/litellm-ai-gateway /usr/local/bin/litellm-ai-gateway +COPY --from=builder /build/litellm-rust/target/release/litellm-gateway-server /usr/local/bin/litellm-gateway-server # Default config.yaml. A real deploy can override this (e.g. mount a Render # secret file at the same path) — never bake secrets into the image. @@ -83,4 +83,4 @@ RUN useradd --system --no-create-home --uid 10001 appuser \ && chown -R appuser:appuser /app USER appuser -ENTRYPOINT ["/usr/local/bin/litellm-ai-gateway"] +ENTRYPOINT ["/usr/local/bin/litellm-gateway-server"] diff --git a/litellm-rust/crates/gateway-server/README.md b/litellm-rust/crates/gateway-server/README.md index b6ede1ae47a..536a4d52e55 100644 --- a/litellm-rust/crates/gateway-server/README.md +++ b/litellm-rust/crates/gateway-server/README.md @@ -12,8 +12,8 @@ dials OpenAI upstream, and splices the two sockets frame-by-frame. |-------|------| | litellm-core | The LiteLLM SDK in Rust — per-route entrypoints (`messages::messages()`) that resolve the provider, transform, and make the call; plus types, provider transforms, and the router. | | litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. | -| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by the server and Python bridge. | -| litellm-gateway-server | Axum binary, HTTP/WebSocket routes, auth extractors, application state, startup config, and HTTP-only dependencies. | +| litellm-gateway-inference | Framework-independent gateway runtime and integrations shared by the server and Python bridge. | +| litellm-gateway-server | Root Axum binary and composition crate. Owns routes, auth, state, startup config, and HTTP-only dependencies. | | litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | | litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. | @@ -46,7 +46,7 @@ model_list: ``` ```bash -LITELLM_CONFIG_PATH=./config.yaml ./litellm-ai-gateway +LITELLM_CONFIG_PATH=./config.yaml ./litellm-gateway-server ``` At boot `litellm-config` calls into `litellm.proxy.read_model_list` and returns @@ -113,13 +113,13 @@ repo's source** (the config reader is newer than any PyPI release), so the build ```bash # from the repo root -docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-ai-gateway . +docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-gateway-server . docker run --rm -p 4001:4001 \ -e HOST=0.0.0.0 -e PORT=4001 \ -e LITELLM_MASTER_KEY=sk-local \ -e OPENAI_API_KEY=$OPENAI_API_KEY \ - litellm-ai-gateway # LITELLM_CONFIG_PATH defaults to /app/config.yaml + litellm-gateway-server # LITELLM_CONFIG_PATH defaults to /app/config.yaml # smoke test curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/health/readiness # -> 200 @@ -134,7 +134,7 @@ To use your own config, mount it over the default: docker run --rm -p 4001:4001 \ -e HOST=0.0.0.0 -e LITELLM_MASTER_KEY=sk-local -e OPENAI_API_KEY=$OPENAI_API_KEY \ -v $(pwd)/my-config.yaml:/app/config.yaml:ro \ - litellm-ai-gateway + litellm-gateway-server ``` ### Cargo-only (no Docker) @@ -170,7 +170,7 @@ deploy. To use a non-default model_list, mount a **Render Secret File** at curl -X POST https://api.render.com/v1/services \ -H "Authorization: Bearer $RENDER_API_KEY" -H "Content-Type: application/json" \ -d '{ - "type": "web_service", "name": "litellm-rust-ai-gateway", + "type": "web_service", "name": "litellm-rust-gateway-server", "ownerId": "", "repo": "https://github.com/BerriAI/litellm", "branch": "", "serviceDetails": { diff --git a/litellm-rust/crates/gateway-server/render.yaml b/litellm-rust/crates/gateway-server/render.yaml index 295049fbcde..21791f2a1b7 100644 --- a/litellm-rust/crates/gateway-server/render.yaml +++ b/litellm-rust/crates/gateway-server/render.yaml @@ -11,7 +11,7 @@ # them in the Render dashboard or via the API, never inline here. services: - type: web - name: litellm-rust-ai-gateway + name: litellm-rust-gateway-server runtime: docker plan: standard dockerfilePath: ./litellm-rust/crates/gateway-server/Dockerfile diff --git a/litellm-rust/crates/gateway-server/src/main.rs b/litellm-rust/crates/gateway-server/src/main.rs index aac266be0d7..a64d0721f15 100644 --- a/litellm-rust/crates/gateway-server/src/main.rs +++ b/litellm-rust/crates/gateway-server/src/main.rs @@ -2,16 +2,16 @@ //! //! The binary owns startup and config, then mounts the Axum routes from //! `litellm_gateway_server`. Transport-neutral runtime and integrations come -//! from `litellm_ai_gateway`. +//! from `litellm_gateway_inference`. use std::sync::Arc; -use litellm_ai_gateway::integrations::custom_logger::CustomLogger; -use litellm_ai_gateway::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger; -use litellm_ai_gateway::io::realtime_pool::{PoolConfig, RealtimePool, upstream_key}; #[cfg(feature = "python-config")] use litellm_config::load_model_list; use litellm_core::router::{Deployment, LiteLLMParams, Router}; +use litellm_gateway_inference::integrations::custom_logger::CustomLogger; +use litellm_gateway_inference::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger; +use litellm_gateway_inference::io::realtime_pool::{PoolConfig, RealtimePool, upstream_key}; use litellm_gateway_server::routes; use litellm_gateway_server::state::AppState; @@ -72,7 +72,7 @@ async fn main() { let listener = tokio::net::TcpListener::bind((host.as_str(), port)) .await .expect("failed to bind listener"); - eprintln!("litellm-ai-gateway listening on {host}:{port}"); + eprintln!("litellm-gateway-server listening on {host}:{port}"); axum::serve(listener, routes::app(state)) .await .expect("server error"); diff --git a/litellm-rust/crates/gateway-server/src/routes/AGENTS.md b/litellm-rust/crates/gateway-server/src/routes/AGENTS.md index bdb74929049..3cdcae77ccc 100644 --- a/litellm-rust/crates/gateway-server/src/routes/AGENTS.md +++ b/litellm-rust/crates/gateway-server/src/routes/AGENTS.md @@ -17,7 +17,7 @@ async fn handle(...) -> impl IntoResponse { ... } ## Runtime boundary When a route has transport-neutral orchestration, put it under -`litellm-ai-gateway::runtime` and test it there. The route file stays the Axum +`litellm-gateway-inference::runtime` and test it there. The route file stays the Axum surface: router, handler, and socket or SSE adapter. Never build a provider request, resolve a provider key, or perform the provider call in this crate. @@ -25,7 +25,7 @@ request, resolve a provider key, or perform the provider call in this crate. - **Auth is an extractor, not a manual call.** A handler requires auth by adding `crate::auth::RequireMasterKey` to its arguments; it runs during extraction. Never re-implement the check per route. -- **Handlers contain no business logic; `litellm-ai-gateway` contains no Axum types.** +- **Handlers contain no business logic; `litellm-gateway-inference` contains no Axum types.** - **No provider handlers in this crate.** Transforms, auth headers, and the provider HTTP call live in `core/src//`. - A route owns its paths in its own `router()`; `mod.rs` only merges. diff --git a/litellm-rust/crates/gateway-server/src/routes/messages/mod.rs b/litellm-rust/crates/gateway-server/src/routes/messages/mod.rs index f057e3a9411..379a6c8c27f 100644 --- a/litellm-rust/crates/gateway-server/src/routes/messages/mod.rs +++ b/litellm-rust/crates/gateway-server/src/routes/messages/mod.rs @@ -7,8 +7,8 @@ use axum::http::StatusCode; use axum::http::header::{CACHE_CONTROL, CONTENT_TYPE, HeaderMap, HeaderValue}; use axum::response::{IntoResponse, Response}; use axum::routing::post; -use litellm_ai_gateway::runtime::messages::{MessagesResponse, run}; use litellm_core::Error; +use litellm_gateway_inference::runtime::messages::{MessagesResponse, run}; use serde_json::{Map, Value}; use crate::auth::RequireMasterKey; @@ -149,7 +149,7 @@ mod tests { use super::super::app; use crate::state::AppState; - use litellm_ai_gateway::io::realtime_pool::RealtimePool; + use litellm_gateway_inference::io::realtime_pool::RealtimePool; fn state(model: &str, api_base: String, master_key: Option<&str>) -> AppState { state_with_provider(model, model, api_base, master_key) diff --git a/litellm-rust/crates/gateway-server/src/routes/realtime/README.md b/litellm-rust/crates/gateway-server/src/routes/realtime/README.md index 358490b2939..5e7f12411d0 100644 --- a/litellm-rust/crates/gateway-server/src/routes/realtime/README.md +++ b/litellm-rust/crates/gateway-server/src/routes/realtime/README.md @@ -2,8 +2,8 @@ Proxies OpenAI's realtime WebSocket. `mod.rs` is the Axum surface (handler and socket-to-events adapter); the pure logic lives in -`litellm-ai-gateway::runtime::realtime`. The pool lives in -`litellm-ai-gateway::io::realtime_pool`. +`litellm-gateway-inference::runtime::realtime`. The pool lives in +`litellm-gateway-inference::io::realtime_pool`. ## Connection pooling @@ -84,4 +84,4 @@ upstream sockets, which is why warm sockets are short-lived attempts against a broken key so it can't exhaust upstream rate limits and degrade valid cold-path traffic; the backoff resets the moment a dial succeeds. -Benchmarks and repro: `../../../../ai-gateway/benchmarks/realtime/README.md`. +Benchmarks and repro: `../../../../gateway-inference/benchmarks/realtime/README.md`. diff --git a/litellm-rust/crates/gateway-server/src/routes/realtime/mod.rs b/litellm-rust/crates/gateway-server/src/routes/realtime/mod.rs index e8f8df4913a..d123324e8a3 100644 --- a/litellm-rust/crates/gateway-server/src/routes/realtime/mod.rs +++ b/litellm-rust/crates/gateway-server/src/routes/realtime/mod.rs @@ -2,7 +2,7 @@ //! //! This file is the **axum surface**: `router()`, the handler, and the small //! socket↔events adapter. The pure logic lives in -//! [`litellm_ai_gateway::runtime::realtime`]. Auth is the `RequireMasterKey` +//! [`litellm_gateway_inference::runtime::realtime`]. Auth is the `RequireMasterKey` //! extractor, so the handler stays thin. use std::sync::Arc; @@ -16,13 +16,13 @@ use axum::http::StatusCode; use axum::response::Response; use axum::routing::get; use futures_util::{SinkExt, StreamExt}; -use litellm_ai_gateway::integrations::custom_logger::CustomLogger; -use litellm_ai_gateway::integrations::types::RequestMetadata; -use litellm_ai_gateway::io::realtime_pool::RealtimePool; -use litellm_ai_gateway::realtime::streaming::{RealTimeStreaming, SessionStatus}; -use litellm_ai_gateway::runtime::realtime; use litellm_core::realtime::types::RealtimeEvent; use litellm_core::router::Router as ModelRouter; +use litellm_gateway_inference::integrations::custom_logger::CustomLogger; +use litellm_gateway_inference::integrations::types::RequestMetadata; +use litellm_gateway_inference::io::realtime_pool::RealtimePool; +use litellm_gateway_inference::realtime::streaming::{RealTimeStreaming, SessionStatus}; +use litellm_gateway_inference::runtime::realtime; use serde::Deserialize; use crate::auth::RequireMasterKey; @@ -84,7 +84,7 @@ async fn handle( } /// Adapt the axum socket (text frames) to the typed-event `Stream`/`Sink` the -/// runtime wants, keeping axum types out of `litellm-ai-gateway`. +/// runtime wants, keeping axum types out of `litellm-gateway-inference`. /// /// This is also the realtime-logging seam: every upstream→client event (the /// direction carrying `session.created` and `response.done` with usage) is fed diff --git a/litellm-rust/crates/gateway-server/src/routes/responses/mod.rs b/litellm-rust/crates/gateway-server/src/routes/responses/mod.rs index 0f708d0897d..1bf600766be 100644 --- a/litellm-rust/crates/gateway-server/src/routes/responses/mod.rs +++ b/litellm-rust/crates/gateway-server/src/routes/responses/mod.rs @@ -9,11 +9,11 @@ use axum::http::StatusCode; use axum::response::Response; use axum::routing::get; use futures_util::{Sink, SinkExt, StreamExt}; -use litellm_ai_gateway::integrations::custom_logger::CustomLogger; -use litellm_ai_gateway::integrations::types::RequestMetadata; -use litellm_ai_gateway::runtime::responses; use litellm_core::responses::types::{ResponsesErrorFrame, ResponsesWsEvent, ResponsesWsEventType}; use litellm_core::router::Router as ModelRouter; +use litellm_gateway_inference::integrations::custom_logger::CustomLogger; +use litellm_gateway_inference::integrations::types::RequestMetadata; +use litellm_gateway_inference::runtime::responses; use serde::Deserialize; use crate::auth::RequireMasterKey; @@ -241,8 +241,8 @@ mod tests { use crate::state::AppState; use axum::body::Body; use axum::http::Request; - use litellm_ai_gateway::io::realtime_pool::RealtimePool; use litellm_core::router::Router as ModelRouter; + use litellm_gateway_inference::io::realtime_pool::RealtimePool; use serde_json::json; use std::pin::Pin; use std::sync::Arc; diff --git a/litellm-rust/crates/gateway-server/src/state.rs b/litellm-rust/crates/gateway-server/src/state.rs index cd4a5e5deed..29648e6a7b5 100644 --- a/litellm-rust/crates/gateway-server/src/state.rs +++ b/litellm-rust/crates/gateway-server/src/state.rs @@ -1,8 +1,8 @@ use std::sync::Arc; -use litellm_ai_gateway::integrations::custom_logger::CustomLogger; -use litellm_ai_gateway::io::realtime_pool::RealtimePool; use litellm_core::router::Router; +use litellm_gateway_inference::integrations::custom_logger::CustomLogger; +use litellm_gateway_inference::io::realtime_pool::RealtimePool; /// Shared application state handed to every route handler. #[derive(Clone)] diff --git a/litellm-rust/crates/python-bridge/Cargo.toml b/litellm-rust/crates/python-bridge/Cargo.toml index bda09a7d840..5e2ff9e5f7b 100644 --- a/litellm-rust/crates/python-bridge/Cargo.toml +++ b/litellm-rust/crates/python-bridge/Cargo.toml @@ -17,14 +17,14 @@ panic-test = [] trace-parity = [ "dep:tracing", "litellm-core/observability", - "litellm-ai-gateway/trace-parity", + "litellm-gateway-inference/trace-parity", ] [dependencies] futures-util.workspace = true tracing = { workspace = true, optional = true } litellm-core = { workspace = true, features = ["bedrock-auth"] } -litellm-ai-gateway = { workspace = true, default-features = false } +litellm-gateway-inference = { workspace = true, default-features = false } litellm-python-interop.workspace = true pyo3.workspace = true pyo3-async-runtimes.workspace = true diff --git a/litellm-rust/crates/python-bridge/src/lib.rs b/litellm-rust/crates/python-bridge/src/lib.rs index 384f0be5a1b..c5f9092684a 100644 --- a/litellm-rust/crates/python-bridge/src/lib.rs +++ b/litellm-rust/crates/python-bridge/src/lib.rs @@ -6,7 +6,7 @@ mod function_trace; mod marshal; mod routes; -use litellm_ai_gateway::io::responses_ws::ResponsesWebSocketConnection as RustResponsesWebSocketConnection; +use litellm_gateway_inference::io::responses_ws::ResponsesWebSocketConnection as RustResponsesWebSocketConnection; use pyo3::prelude::*; use pyo3::types::PyAny; use serde_json::Value; diff --git a/litellm-rust/crates/python-bridge/src/routes/gateway_messages.rs b/litellm-rust/crates/python-bridge/src/routes/gateway_messages.rs index 97ff93f299a..2616c64ebe7 100644 --- a/litellm-rust/crates/python-bridge/src/routes/gateway_messages.rs +++ b/litellm-rust/crates/python-bridge/src/routes/gateway_messages.rs @@ -11,7 +11,7 @@ fn gateway_messages<'py>( api_base: String, #[pyo3(from_py_with = litellm_python_interop::from_py)] body: Value, ) -> PyResult> { - let future = litellm_ai_gateway::trace_parity::messages_request( + let future = litellm_gateway_inference::trace_parity::messages_request( model_alias, provider_model, api_base, diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr.rs b/litellm-rust/crates/python-bridge/src/routes/ocr.rs index cc2f8e43cea..d59f6666ebb 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr.rs @@ -1,7 +1,7 @@ use litellm_core::Error; use std::future::Future; -use litellm_ai_gateway::io::ocr::{OcrRequest, ocr as run_ocr}; +use litellm_gateway_inference::io::ocr::{OcrRequest, ocr as run_ocr}; use pyo3::prelude::*; use serde_json::Value;