This commit is contained in:
yujonglee 2026-11-03 07:20:25 +00:00 • committed by GitHub
commit c6bb5f4483
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
27 changed files with 2469 additions and 6 deletions

View file

@ -7,7 +7,7 @@ metadata:
# Rust gateway
`crates/gateway` mounts endpoint routers and serves the listener. `crates/gateway-inference`, `gateway-mcp` and `gateway-ui` own their endpoints, `gateway-auth` is the inbound authentication middleware and `gateway-management` holds management logic that has no router yet. A crate's `AGENTS.md`, where it has one, says what belongs in it. Read the reference for the part you are touching
`crates/gateway` mounts endpoint routers and serves the listener. `crates/gateway-inference`, `gateway-mcp` and `gateway-ui` own their endpoints, `gateway-auth` is the inbound authentication middleware and `gateway-management` holds management logic that has no router yet. `gateway-forward` is the reverse proxy that relays every request to the Python proxy and has no routes of its own. A crate's `AGENTS.md`, where it has one, says what belongs in it. Read the reference for the part you are touching
- [surfaces](references/surfaces.md)
- Each API surface mirrors the envelope the Python proxy returns there, so the shapes differ on purpose

View file

@ -3812,6 +3812,7 @@ dependencies = [
"litellm-config",
"litellm-error",
"litellm-gateway-auth",
"litellm-gateway-forward",
"litellm-gateway-inference",
"litellm-gateway-mcp",
"litellm-gateway-ui",
@ -3858,6 +3859,28 @@ dependencies = [
"veil",
]
[[package]]
name = "litellm-gateway-forward"
version = "0.1.0"
dependencies = [
"axum",
"bytes",
"envy",
"futures-util",
"http-body-util",
"httparse",
"hyper 1.10.1",
"hyper-util",
"proptest",
"rstest",
"serde",
"thiserror 2.0.19",
"tokio",
"tokio-tungstenite",
"tokio-util",
"tracing",
]
[[package]]
name = "litellm-gateway-inference"
version = "0.1.0"
@ -7570,6 +7593,7 @@ dependencies = [
"bytes",
"futures-core",
"futures-sink",
"futures-util",
"pin-project-lite",
"tokio",
]

View file

@ -25,6 +25,7 @@ litellm-inference-ocr = { path = "crates/inference-ocr" }
litellm-inference-testing = { path = "crates/inference-testing" }
litellm-gateway-mcp = { path = "crates/gateway-mcp" }
litellm-gateway = { path = "crates/gateway" }
litellm-gateway-forward = { path = "crates/gateway-forward" }
litellm-gateway-inference = { path = "crates/gateway-inference" }
litellm-gateway-auth = { path = "crates/gateway-auth" }
litellm-gateway-management = { path = "crates/gateway-management" }

View file

@ -0,0 +1,33 @@
# Forwarding gateway
`Forward` is a reverse proxy in front of the Python proxy on loopback. Python must see each request as if the client had connected to it directly, because every Python route (inference, MCP, pass-through, UI, WebSocket) is served behind it without Rust knowing which one it is
- Never branch on a route or path. A behavior one endpoint needs is a protocol behavior every endpoint gets
- Forward the method, request target, headers, and body byte for byte. Only drop hop-by-hop headers (RFC 9110 section 7.6.1) and set `x-litellm-rust-peer`, which replaces any client copy
- Stream both bodies. Never buffer, cap, or time out a body, since uploads and provider streams have no upper bound
- A client disconnect closes the upstream connection at every stage, because that is how Python learns to stop a provider stream
- An upstream that dies mid-response fails the client body. Never end it cleanly, or SSE clients treat a truncated stream as complete
- Pooled upstream connections are retired well before Python's keep-alive timeout, so a request never lands on a connection Python is closing
- WebSocket is the one protocol switch Python serves. Upgrades are relayed as raw bytes after the 101 and drained on shutdown
## Layout
Every forwarding decision has one home. `rewrite.rs` owns what Python and the client receive, as pure functions over requests and responses. `Rejection` in `error.rs` is the only place that picks a status for a request Rust answers itself. `settings.rs` parses the environment into typed values, so an `Upstream` is a loopback origin by construction. `forward.rs` and `websocket.rs` only move requests, bodies and upgraded connections and never touch a header themselves
## Environment
`Settings` reads only these variables, and only when `LITELLM_FORWARD_UPSTREAM` is set. The Python supervisor in `litellm/proxy/rust_forward/` sets every one of them from the values Python actually runs with, so all of them are required and a missing or invalid one fails startup naming it. The crate never reads Python's own settings (`HOST`, `PORT`, `KEEPALIVE_TIMEOUT`, `GRACEFUL_SHUTDOWN_TIMEOUT`) and keeps no copy of Python's defaults, since Python can get those values from CLI flags the environment never shows
| Variable | Meaning |
|---|---|
| `LITELLM_FORWARD_UPSTREAM` | Python's loopback origin, e.g. `http://127.0.0.1:53211` |
| `LITELLM_FORWARD_HOST` | public bind host |
| `LITELLM_FORWARD_PORT` | public bind port |
| `LITELLM_FORWARD_UPSTREAM_KEEPALIVE` | seconds uvicorn keeps an idle connection open. Pooled connections are retired at half of it |
| `LITELLM_FORWARD_SHUTDOWN_DRAIN` | seconds to drain on shutdown, already including Python's graceful timeout and teardown headroom |
## Tests
Test through real sockets with `tests/support`, never `oneshot`. `Gateway` runs the forwarder on a listener like production, `recording` is a well-behaved upstream that captures what Python would receive, and `raw_upstream` and `Raw` drive bytes by hand for framing, timing and disconnects a real server would not produce. Waits measured in minutes use paused tokio time instead of real sleeps
The forwarder sees HTTP/1.1 messages, not routes, so the tests are split into one file per traffic shape uvicorn can emit: `exchange` (plain request and response), `slow_head` (minutes before response headers), `stream` (incremental response bodies), `idle_stream` (long-lived streams that go quiet), `upload`, `download`, `websocket`, `client_identity`, `lifecycle` (disconnects, pooling, shutdown) and `roundtrip` (the forwarding invariant for generated messages). A new route never gets its own test. Add its behavior to the shape it already has, and when it needs a wire behavior none of the files covers, add a new shape file for it

View file

@ -0,0 +1,29 @@
[package]
name = "litellm-gateway-forward"
version = "0.1.0"
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
[dependencies]
axum.workspace = true
envy = "0.4.2"
hyper = { version = "1", features = ["http1"] }
hyper-util = { workspace = true, features = ["client-legacy", "http1", "http2", "tokio"] }
serde.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["io-util"] }
tokio-util = { version = "0.7", features = ["rt"] }
tracing.workspace = true
[dev-dependencies]
bytes.workspace = true
futures-util.workspace = true
http-body-util = "0.1"
httparse = "1"
hyper = { version = "1", features = ["client", "http2"] }
proptest.workspace = true
rstest.workspace = true
tokio = { workspace = true, features = ["io-util", "sync", "test-util"] }
tokio-tungstenite.workspace = true

View file

@ -0,0 +1,60 @@
use axum::{
http::StatusCode,
response::{IntoResponse, Response},
};
#[derive(Debug, thiserror::Error)]
pub enum Error {
#[error("invalid forwarding upstream URI")]
UpstreamUri(#[source] axum::http::uri::InvalidUri),
#[error(
"LITELLM_FORWARD_UPSTREAM must be a loopback IPv4 http origin without a path, query or credentials"
)]
Upstream,
#[error("{0} must be set in forward mode")]
MissingSetting(String),
#[error("invalid forwarding setting: {0}")]
InvalidSetting(String),
#[error("forwarding gateway I/O failed: {0}")]
Io(#[from] std::io::Error),
}
/// Why a request got an answer from Rust instead of from Python.
#[derive(Debug, thiserror::Error)]
pub(crate) enum Rejection {
#[error("request authority is not a valid Host header")]
InvalidAuthority,
#[error("python proxy unreachable: {0}")]
Unreachable(#[source] hyper_util::client::legacy::Error),
#[error("python proxy switched protocols without being asked to")]
UnrequestedUpgrade,
#[error("python proxy websocket upgrade failed: {0}")]
UpgradeFailed(#[source] hyper::Error),
#[error("forwarding gateway is shutting down")]
ShuttingDown,
#[error("could not build the request for the python proxy")]
Internal,
}
impl Rejection {
fn status(&self) -> StatusCode {
match self {
Self::InvalidAuthority => StatusCode::BAD_REQUEST,
Self::Unreachable(_) | Self::UnrequestedUpgrade | Self::UpgradeFailed(_) => {
StatusCode::BAD_GATEWAY
}
Self::ShuttingDown => StatusCode::SERVICE_UNAVAILABLE,
Self::Internal => StatusCode::INTERNAL_SERVER_ERROR,
}
}
}
impl IntoResponse for Rejection {
fn into_response(self) -> Response {
let status = self.status();
if status.is_server_error() {
tracing::warn!(error = %self, "request not forwarded");
}
status.into_response()
}
}

View file

@ -0,0 +1,128 @@
use std::{net::SocketAddr, time::Duration};
use axum::{
Router,
body::Body,
extract::{ConnectInfo, DefaultBodyLimit, Request, State},
http::{StatusCode, Uri},
response::Response,
serve::ListenerExt,
};
use hyper_util::{
client::legacy::{Client, connect::HttpConnector},
rt::{TokioExecutor, TokioTimer},
};
use tokio::net::TcpListener;
use tokio_util::sync::CancellationToken;
use crate::{Rejection, Upstream, rewrite, websocket::Upgrades};
// uvicorn sets no response header limit and Python copies every provider header into its
// response, so hyper's default of 100 would turn ordinary responses into 502s.
const PYTHON_RESPONSE_HEADERS: usize = 1000;
#[derive(Clone)]
pub struct Forward {
upstream: Uri,
client: Client<HttpConnector, Body>,
upgrades: Upgrades,
}
impl Forward {
/// `upstream_keepalive` is how long the upstream keeps an idle connection open. Pooled
/// connections are dropped well before that, so a request never lands on one it is closing.
pub fn new(upstream: &Upstream, upstream_keepalive: Duration) -> Self {
let mut connector = HttpConnector::new();
connector.set_nodelay(true);
let client = Client::builder(TokioExecutor::new())
.http1_max_headers(PYTHON_RESPONSE_HEADERS)
.pool_timer(TokioTimer::new())
.pool_idle_timeout(upstream_keepalive / 2)
.build(connector);
Self {
upstream: upstream.uri().clone(),
client,
upgrades: Upgrades::default(),
}
}
pub fn router(self) -> Router {
Router::new()
.fallback(forward)
.layer(DefaultBodyLimit::disable())
.with_state(self)
}
/// Serves until `shutdown` resolves, then lets open requests and WebSockets finish for up to
/// `drain` before closing whatever is left. Python shuts down alongside and ends long streams
/// itself, so `drain` only backstops a Python that never does.
pub async fn serve(
self,
listener: TcpListener,
shutdown: impl Future<Output = ()> + Send + 'static,
drain: Duration,
) -> std::io::Result<()> {
let upgrades = self.upgrades.clone();
let _close_on_drop = upgrades.close_on_drop();
let stopping = CancellationToken::new();
let stop = stopping.clone();
let draining = upgrades.clone();
let listener = listener.tap_io(|stream| {
if let Err(error) = stream.set_nodelay(true) {
tracing::warn!(%error, "could not disable Nagle on a client connection");
}
});
let server = axum::serve(
listener,
self.router()
.into_make_service_with_connect_info::<SocketAddr>(),
)
.with_graceful_shutdown(async move {
shutdown.await;
draining.begin_shutdown();
stop.cancel();
});
let drained = async {
let result = server.await;
if result.is_err() {
upgrades.close();
}
upgrades.wait().await;
result
};
let deadline = async {
stopping.cancelled().await;
tokio::time::sleep(drain).await;
tracing::warn!(
?drain,
"closing connections still open after the shutdown drain"
);
};
let result = tokio::select! {
result = drained => result,
() = deadline => Ok(()),
};
upgrades.close();
upgrades.wait().await;
result
}
}
async fn forward(
State(state): State<Forward>,
ConnectInfo(peer): ConnectInfo<SocketAddr>,
mut request: Request,
) -> Result<Response, Rejection> {
let downstream = state.upgrades.take_handshake(&mut request)?;
let request = rewrite::upstream_request(request, &state.upstream, peer, downstream.is_some())?;
let response = state.client.request(request);
let response = match downstream {
Some(_) => state.upgrades.until_closed(response).await?,
None => response.await,
}
.map_err(Rejection::Unreachable)?;
if response.status() == StatusCode::SWITCHING_PROTOCOLS {
return state.upgrades.switch(downstream, response).await;
}
Ok(rewrite::downstream_response(response).map(Body::new))
}

View file

@ -0,0 +1,26 @@
mod error;
mod forward;
mod rewrite;
mod settings;
mod websocket;
pub use error::Error;
pub(crate) use error::Rejection;
pub use forward::Forward;
pub use rewrite::PEER_HEADER;
pub use settings::{Settings, Upstream};
/// Binds the public listener and forwards to Python until `shutdown` resolves, then drains for
/// `settings.shutdown_drain`.
pub async fn run(
settings: Settings,
shutdown: impl Future<Output = ()> + Send + 'static,
) -> Result<(), Error> {
let forward = Forward::new(&settings.upstream, settings.upstream_keepalive);
let listener = tokio::net::TcpListener::bind((settings.host.as_str(), settings.port)).await?;
tracing::info!(address = %listener.local_addr()?, "forwarding gateway listening");
forward
.serve(listener, shutdown, settings.shutdown_drain)
.await?;
Ok(())
}

View file

@ -0,0 +1,111 @@
use std::net::SocketAddr;
use axum::{
body::HttpBody,
extract::Request,
http::{HeaderMap, HeaderValue, Response, Uri, Version, header},
};
use crate::Rejection;
pub const PEER_HEADER: &str = "x-litellm-rust-peer";
/// The request Python receives: the client's method, target, headers and body unchanged, sent to
/// `upstream` over HTTP/1.1 without the client connection's own headers, plus the client's TCP
/// peer in `PEER_HEADER`, which replaces any copy the client sent.
pub(crate) fn upstream_request(
mut request: Request,
upstream: &Uri,
peer: SocketAddr,
websocket: bool,
) -> Result<Request, Rejection> {
// HTTP/2 carries the public host in :authority, which retargeting at the upstream replaces.
if let Some(authority) = request.uri().authority()
&& !request.headers().contains_key(header::HOST)
{
let host =
HeaderValue::from_str(authority.as_str()).map_err(|_| Rejection::InvalidAuthority)?;
request.headers_mut().insert(header::HOST, host);
}
let mut uri = upstream.clone().into_parts();
uri.path_and_query = request.uri().path_and_query().cloned();
*request.uri_mut() = Uri::from_parts(uri).map_err(|_| Rejection::Internal)?;
*request.version_mut() = Version::HTTP_11;
let chunked = request.body().size_hint().exact().is_none();
let headers = request.headers_mut();
strip_hop_by_hop(headers);
if chunked {
// hyper sends no body for a GET with an unknown length unless told it is chunked.
headers.insert(
header::TRANSFER_ENCODING,
HeaderValue::from_static("chunked"),
);
}
if websocket {
restore_websocket_handshake(headers);
}
headers.insert(
PEER_HEADER,
HeaderValue::from_str(&peer.to_string()).map_err(|_| Rejection::Internal)?,
);
Ok(request)
}
/// The response the client receives: Python's, without the upstream connection's own headers.
pub(crate) fn downstream_response<B>(mut response: Response<B>) -> Response<B> {
strip_hop_by_hop(response.headers_mut());
response
}
/// Python's 101 as the client receives it, still naming the protocol both sides switch to.
pub(crate) fn switched_response<B>(mut response: Response<B>) -> Response<B> {
strip_hop_by_hop(response.headers_mut());
restore_websocket_handshake(response.headers_mut());
response
}
pub(crate) fn is_websocket_handshake(headers: &HeaderMap) -> bool {
headers
.get(header::UPGRADE)
.is_some_and(|value| value.as_bytes().eq_ignore_ascii_case(b"websocket"))
&& headers
.get_all(header::CONNECTION)
.iter()
.filter_map(|value| value.to_str().ok())
.flat_map(|value| value.split(','))
.any(|token| token.trim().eq_ignore_ascii_case("upgrade"))
}
// Per-connection headers (RFC 9110 section 7.6.1) describe the client's connection to Rust, not
// Rust's connection to Python, so a proxy must drop them instead of forwarding them.
fn strip_hop_by_hop(headers: &mut HeaderMap) {
let connection_headers: Vec<_> = headers
.get_all(header::CONNECTION)
.iter()
.filter_map(|value| value.to_str().ok())
.flat_map(|value| value.split(','))
.filter_map(|name| header::HeaderName::from_bytes(name.trim().as_bytes()).ok())
.collect();
for name in connection_headers {
headers.remove(name);
}
for name in [
"connection",
"keep-alive",
"proxy-authenticate",
"proxy-authorization",
"te",
"trailer",
"transfer-encoding",
"upgrade",
"proxy-connection",
] {
headers.remove(name);
}
}
// Stripping drops the handshake's Connection and Upgrade, so each hop gets them back.
fn restore_websocket_handshake(headers: &mut HeaderMap) {
headers.insert(header::CONNECTION, HeaderValue::from_static("upgrade"));
headers.insert(header::UPGRADE, HeaderValue::from_static("websocket"));
}

View file

@ -0,0 +1,101 @@
use std::{ffi::OsString, net::SocketAddr, str::FromStr, time::Duration};
use axum::http::Uri;
use serde::Deserialize;
use crate::Error;
const UPSTREAM: &str = "LITELLM_FORWARD_UPSTREAM";
/// The Python proxy's private origin. Only a loopback IPv4 address parses, so nothing can point
/// the forwarder at another host.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Upstream(Uri);
impl Upstream {
pub(crate) fn uri(&self) -> &Uri {
&self.0
}
}
impl FromStr for Upstream {
type Err = Error;
fn from_str(value: &str) -> Result<Self, Error> {
let uri: Uri = value.parse().map_err(Error::UpstreamUri)?;
let address = uri
.authority()
.filter(|_| uri.scheme_str() == Some("http"))
.filter(|authority| !authority.as_str().contains('@'))
.filter(|_| uri.path() == "/" && uri.query().is_none())
.and_then(|authority| authority.as_str().parse::<SocketAddr>().ok());
match address {
Some(SocketAddr::V4(address)) if address.ip().is_loopback() => Ok(Self(uri)),
_ => Err(Error::Upstream),
}
}
}
#[derive(Deserialize)]
struct Environment {
litellm_forward_upstream: String,
litellm_forward_host: String,
litellm_forward_port: u16,
litellm_forward_upstream_keepalive: f64,
litellm_forward_shutdown_drain: f64,
}
/// Everything forward mode runs with. The Python supervisor resolves each value from the settings
/// Python actually runs with, so none of them is guessed or defaulted here.
#[derive(Debug, Clone, PartialEq)]
pub struct Settings {
pub upstream: Upstream,
pub host: String,
pub port: u16,
pub upstream_keepalive: Duration,
pub shutdown_drain: Duration,
}
impl Settings {
/// `None` when `LITELLM_FORWARD_UPSTREAM` is unset, which means the gateway is not forwarding.
pub fn from_env() -> Result<Option<Self>, Error> {
Self::from_vars(std::env::vars_os())
}
pub fn from_vars(
variables: impl IntoIterator<Item = (OsString, OsString)>,
) -> Result<Option<Self>, Error> {
let variables: Vec<(String, String)> = variables
.into_iter()
.filter_map(|(key, value)| Some((key.into_string().ok()?, value.into_string().ok()?)))
.collect();
if !variables.iter().any(|(key, _)| key == UPSTREAM) {
return Ok(None);
}
let environment: Environment = envy::from_iter(variables).map_err(|error| match error {
envy::Error::MissingValue(field) => Error::MissingSetting(field.to_uppercase()),
envy::Error::Custom(message) => Error::InvalidSetting(message),
})?;
Ok(Some(Self {
upstream: environment.litellm_forward_upstream.parse()?,
host: environment.litellm_forward_host,
port: environment.litellm_forward_port,
upstream_keepalive: seconds(
"LITELLM_FORWARD_UPSTREAM_KEEPALIVE",
environment.litellm_forward_upstream_keepalive,
)?,
shutdown_drain: seconds(
"LITELLM_FORWARD_SHUTDOWN_DRAIN",
environment.litellm_forward_shutdown_drain,
)?,
}))
}
}
fn seconds(variable: &str, value: f64) -> Result<Duration, Error> {
Duration::try_from_secs_f64(value).map_err(|_| {
Error::InvalidSetting(format!(
"{variable} must be a non-negative number of seconds, got {value}"
))
})
}

View file

@ -0,0 +1,102 @@
use axum::{
body::Body,
extract::Request,
http::{Method, Version},
response::Response,
};
use hyper::{
body::Incoming,
upgrade::{OnUpgrade, Upgraded},
};
use hyper_util::rt::TokioIo;
use tokio_util::{
sync::{CancellationToken, DropGuard},
task::TaskTracker,
};
use crate::{Rejection, rewrite};
#[derive(Clone, Default)]
pub(crate) struct Upgrades {
tasks: TaskTracker,
draining: CancellationToken,
closed: CancellationToken,
}
impl Upgrades {
pub fn begin_shutdown(&self) {
self.draining.cancel();
}
pub fn take_handshake(&self, request: &mut Request) -> Result<Option<OnUpgrade>, Rejection> {
if request.method() != Method::GET
|| request.version() != Version::HTTP_11
|| !rewrite::is_websocket_handshake(request.headers())
{
return Ok(None);
}
if self.draining.is_cancelled() {
return Err(Rejection::ShuttingDown);
}
Ok(Some(hyper::upgrade::on(request)))
}
pub async fn until_closed<T>(&self, future: impl Future<Output = T>) -> Result<T, Rejection> {
tokio::select! {
_ = self.closed.cancelled() => Err(Rejection::ShuttingDown),
output = future => Ok(output),
}
}
pub async fn switch(
&self,
downstream: Option<OnUpgrade>,
mut response: Response<Incoming>,
) -> Result<Response, Rejection> {
let Some(downstream) = downstream else {
return Err(Rejection::UnrequestedUpgrade);
};
if !rewrite::is_websocket_handshake(response.headers()) {
return Err(Rejection::UnrequestedUpgrade);
}
let upstream = self
.until_closed(hyper::upgrade::on(&mut response))
.await?
.map_err(Rejection::UpgradeFailed)?;
self.relay(downstream, upstream);
Ok(rewrite::switched_response(response).map(|_| Body::empty()))
}
pub fn close(&self) {
self.closed.cancel();
}
pub fn close_on_drop(&self) -> DropGuard {
self.closed.clone().drop_guard()
}
pub async fn wait(&self) {
self.tasks.close();
self.tasks.wait().await;
}
fn relay(&self, downstream: OnUpgrade, upstream: Upgraded) {
let closed = self.closed.clone();
self.tasks.spawn(async move {
tokio::select! {
_ = closed.cancelled() => {}
_ = async {
let Ok(downstream) = downstream.await else {
tracing::debug!("websocket upgrade did not complete");
return;
};
let mut downstream = TokioIo::new(downstream);
let mut upstream = TokioIo::new(upstream);
if let Err(error) = tokio::io::copy_bidirectional(&mut downstream, &mut upstream).await {
tracing::debug!(%error, "websocket relay disconnected");
}
} => {}
}
});
}
}

View file

@ -0,0 +1,68 @@
mod support;
use axum::{body::Body, http::StatusCode, response::Response};
use litellm_gateway_forward::PEER_HEADER;
use rstest::rstest;
use support::{Gateway, recording, within};
fn ok() -> Response {
Response::new(Body::from("ok"))
}
#[rstest]
#[tokio::test]
async fn python_sees_forwarding_headers_untouched_and_the_tcp_peer_rust_saw() {
let (upstream, mut received) = recording(ok).await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send(
"GET /sso/callback HTTP/1.1\r\n\
host: public.example\r\n\
x-forwarded-for: 192.0.2.1\r\n\
x-forwarded-for: 192.0.2.2\r\n\
x-forwarded-proto: https\r\n\
forwarded: for=192.0.2.1;proto=https\r\n\
x-litellm-rust-peer: 10.1.2.3:1\r\n\r\n",
)
.await;
assert_eq!(
within(raw.response_head()).await.unwrap().status,
StatusCode::OK
);
let request = received.recv().await.unwrap();
let forwarded_for: Vec<_> = request.headers.get_all("x-forwarded-for").iter().collect();
assert_eq!(forwarded_for, ["192.0.2.1", "192.0.2.2"]);
assert_eq!(request.headers["x-forwarded-proto"], "https");
assert_eq!(request.headers["forwarded"], "for=192.0.2.1;proto=https");
let peers: Vec<_> = request.headers.get_all(PEER_HEADER).iter().collect();
assert_eq!(peers, [raw.local_address().to_string().as_str()]);
}
#[rstest]
#[tokio::test]
async fn http2_clients_reach_python_with_their_public_host() {
let (upstream, mut received) = recording(ok).await;
let gateway = Gateway::start(upstream).await;
let stream = tokio::net::TcpStream::connect(gateway.address)
.await
.unwrap();
let (mut sender, connection) = hyper::client::conn::http2::handshake(
hyper_util::rt::TokioExecutor::new(),
hyper_util::rt::TokioIo::new(stream),
)
.await
.unwrap();
tokio::spawn(connection);
let request = axum::http::Request::get("http://public.example/v1/models")
.body(Body::empty())
.unwrap();
let response = within(sender.send_request(request)).await.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
received.recv().await.unwrap().headers["host"],
"public.example"
);
}

View file

@ -0,0 +1,80 @@
mod support;
use std::sync::Arc;
use axum::{body::Body, extract::Request, http::StatusCode, response::Response};
use http_body_util::BodyExt;
use rstest::rstest;
use support::{Gateway, client, raw_upstream, recording, within};
use tokio::sync::Notify;
#[rstest]
#[tokio::test]
async fn streams_a_body_without_a_content_length_as_python_produces_it() {
let first: &[u8] = &[0xff, 0xfb, 0x90, 0x64, 0x00, 0x0d, 0x0a];
let rest = vec![0xa5; 64 * 1024];
let finish = Arc::new(Notify::new());
let finished = finish.clone();
let tail = rest.clone();
let upstream = raw_upstream(move |mut python| {
let finished = finished.clone();
let tail = tail.clone();
async move {
python.request_head().await.unwrap();
python
.send([&b"HTTP/1.1 200 OK\r\ncontent-type: audio/mpeg\r\ntransfer-encoding: chunked\r\n\r\n7\r\n"[..], first, b"\r\n"].concat())
.await;
finished.notified().await;
python
.send([format!("{:x}\r\n", tail.len()).as_bytes(), &tail, b"\r\n0\r\n\r\n"].concat())
.await;
}
})
.await;
let gateway = Gateway::start(upstream).await;
let mut response = within(
client().request(
Request::post(gateway.url("/v1/audio/speech"))
.body(Body::empty())
.unwrap(),
),
)
.await
.unwrap();
assert!(!response.headers().contains_key("content-length"));
let frame = within(response.body_mut().frame()).await.unwrap().unwrap();
assert_eq!(frame.into_data().unwrap(), first);
finish.notify_one();
let body = within(response.into_body().collect())
.await
.unwrap()
.to_bytes();
assert_eq!(body, rest);
}
#[rstest]
#[tokio::test]
async fn range_requests_and_partial_content_pass_unchanged() {
let (upstream, mut received) = recording(|| -> Response {
Response::builder()
.status(StatusCode::PARTIAL_CONTENT)
.header("content-range", "bytes 2-5/10")
.header("accept-ranges", "bytes")
.body(Body::from("2345"))
.unwrap()
})
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("GET /v1/videos/video_1/content HTTP/1.1\r\nhost: x\r\nrange: bytes=2-5\r\nif-range: \"v1\"\r\n\r\n")
.await;
let head = within(raw.response_head()).await.unwrap();
assert_eq!(head.status, StatusCode::PARTIAL_CONTENT);
assert_eq!(head.header("content-range"), Some(&b"bytes 2-5/10"[..]));
assert_eq!(raw.body(head.content_length().unwrap()).await, b"2345");
let request = received.recv().await.unwrap();
assert_eq!(request.headers["range"], "bytes=2-5");
assert_eq!(request.headers["if-range"], "\"v1\"");
}

View file

@ -0,0 +1,320 @@
mod support;
use axum::{
Router,
body::Body,
extract::Request,
http::{HeaderValue, StatusCode},
response::Response,
};
use rstest::rstest;
use support::{Gateway, client, raw_upstream, recording, serve, within};
fn ok() -> Response {
Response::new(Body::from("ok"))
}
#[rstest]
#[tokio::test]
async fn forwards_the_request_as_the_client_sent_it_minus_hop_headers() {
let (upstream, mut received) = recording(|| {
Response::builder()
.status(StatusCode::UNAUTHORIZED)
.header("connection", "x-upstream-hop")
.header("x-upstream-hop", "remove")
.header("x-response", "keep")
.body(Body::from("python error"))
.unwrap()
})
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send(
"PATCH /v1/messages?x=a%2Fb&x=c HTTP/1.1\r\n\
host: public.example\r\n\
authorization: Bearer test\r\n\
connection: x-remove, keep-alive\r\n\
x-remove: remove\r\n\
proxy-authorization: remove\r\n\
te: trailers\r\n\
content-length: 7\r\n\r\npayload",
)
.await;
let head = within(raw.response_head()).await.unwrap();
assert_eq!(head.status, StatusCode::UNAUTHORIZED);
assert_eq!(head.header("x-response"), Some(&b"keep"[..]));
assert_eq!(head.header("x-upstream-hop"), None);
let body = raw.body(head.content_length().unwrap()).await;
assert_eq!(body, b"python error");
let request = received.recv().await.unwrap();
assert_eq!(request.method, "PATCH");
assert_eq!(request.uri, "/v1/messages?x=a%2Fb&x=c");
assert_eq!(request.headers["host"], "public.example");
assert_eq!(request.headers["authorization"], "Bearer test");
assert!(!request.headers.contains_key("x-remove"));
assert!(!request.headers.contains_key("proxy-authorization"));
assert!(!request.headers.contains_key("te"));
assert!(!request.headers.contains_key("connection"));
assert_eq!(request.body, "payload");
}
#[rstest]
#[case::get("GET")]
#[case::post("POST")]
#[case::put("PUT")]
#[case::delete("DELETE")]
#[case::options("OPTIONS")]
#[case::extension("PURGE")]
#[tokio::test]
async fn forwards_every_method(#[case] method: &str) {
let (upstream, mut received) = recording(ok).await;
let gateway = Gateway::start(upstream).await;
let response = client()
.request(
Request::builder()
.method(method)
.uri(gateway.url("/mcp"))
.body(Body::from("body"))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let request = received.recv().await.unwrap();
assert_eq!(request.method, method);
assert_eq!(request.body, "body");
}
#[rstest]
#[case::double_slash("//v1//models")]
#[case::encoded_slash("/v1/files/a%2Fb/content")]
#[case::encoded_query("/v1/models?q=%E2%9C%93&empty=&flag")]
#[case::utf8_bytes_escaped("/mcp/%F0%9F%A6%80")]
#[tokio::test]
async fn forwards_the_request_target_byte_for_byte(#[case] target: &str) {
let (upstream, mut received) = recording(ok).await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send(format!("GET {target} HTTP/1.1\r\nhost: x\r\n\r\n"))
.await;
assert_eq!(
within(raw.response_head()).await.unwrap().status,
StatusCode::OK
);
assert_eq!(received.recv().await.unwrap().uri, target);
}
#[rstest]
#[tokio::test]
async fn absolute_form_targets_reach_python_as_origin_form() {
let (upstream, mut received) = recording(ok).await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send(
"GET http://elsewhere.example/v1/models?x=1 HTTP/1.1\r\nhost: elsewhere.example\r\n\r\n",
)
.await;
assert_eq!(
within(raw.response_head()).await.unwrap().status,
StatusCode::OK
);
let request = received.recv().await.unwrap();
assert_eq!(request.uri, "/v1/models?x=1");
}
#[rstest]
#[tokio::test]
async fn relays_a_response_with_more_headers_than_hyper_allows_by_default() {
let upstream = serve(axum::Router::new().fallback(|| async {
let headers = (0..150).map(|index| (format!("llm_provider-x-{index}"), "v"));
let mut response = ok();
for (name, value) in headers {
response.headers_mut().insert(
axum::http::HeaderName::from_bytes(name.as_bytes()).unwrap(),
HeaderValue::from_static(value),
);
}
response
}))
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("GET /v1/chat/completions HTTP/1.1\r\nhost: public.example\r\n\r\n")
.await;
let head = within(raw.response_head()).await.unwrap();
assert_eq!(head.status, StatusCode::OK);
assert_eq!(head.header("llm_provider-x-149"), Some(&b"v"[..]));
}
#[rstest]
#[tokio::test]
async fn keeps_repeated_response_headers_and_opaque_header_bytes() {
let (upstream, mut received) = recording(|| {
Response::builder()
.header("set-cookie", "a=1; Path=/")
.header("set-cookie", "b=2; HttpOnly")
.header("x-opaque", HeaderValue::from_bytes(b"caf\xe9").unwrap())
.body(Body::empty())
.unwrap()
})
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send(b"GET / HTTP/1.1\r\nhost: x\r\nx-opaque: caf\xe9\r\n\r\n")
.await;
let head = within(raw.response_head()).await.unwrap();
let cookies: Vec<_> = head
.headers
.iter()
.filter(|(name, _)| name == "set-cookie")
.map(|(_, value)| value.as_slice())
.collect();
assert_eq!(cookies, [&b"a=1; Path=/"[..], b"b=2; HttpOnly"]);
assert_eq!(head.header("x-opaque"), Some(&b"caf\xe9"[..]));
assert_eq!(
received.recv().await.unwrap().headers["x-opaque"].as_bytes(),
b"caf\xe9"
);
}
#[rstest]
#[tokio::test]
async fn answers_pipelined_requests_in_order() {
let upstream = serve(
Router::new().fallback(|request: Request| async move { request.uri().path().to_owned() }),
)
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("GET /first HTTP/1.1\r\nhost: x\r\n\r\nHEAD /second HTTP/1.1\r\nhost: x\r\n\r\nGET /third HTTP/1.1\r\nhost: x\r\n\r\n")
.await;
let first = within(raw.response_head()).await.unwrap();
assert_eq!(raw.body(first.content_length().unwrap()).await, b"/first");
let second = within(raw.response_head()).await.unwrap();
assert_eq!(second.content_length(), Some(7));
let third = within(raw.response_head()).await.unwrap();
assert_eq!(raw.body(third.content_length().unwrap()).await, b"/third");
}
#[rstest]
#[case::no_content(StatusCode::NO_CONTENT)]
#[case::not_modified(StatusCode::NOT_MODIFIED)]
#[tokio::test]
async fn bodiless_statuses_keep_the_connection_reusable(#[case] status: StatusCode) {
let upstream = serve(Router::new().fallback(move |request: Request| async move {
match request.uri().path() {
"/next" => (StatusCode::OK, "next"),
_ => (status, ""),
}
}))
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("GET /empty HTTP/1.1\r\nhost: x\r\n\r\nGET /next HTTP/1.1\r\nhost: x\r\n\r\n")
.await;
assert_eq!(within(raw.response_head()).await.unwrap().status, status);
let next = within(raw.response_head()).await.unwrap();
assert_eq!(raw.body(next.content_length().unwrap()).await, b"next");
}
#[rstest]
#[tokio::test]
async fn informational_responses_keep_the_connection_reusable() {
let upstream = raw_upstream(|mut python| async move {
while let Some(head) = python.request_head().await {
match head.path.as_str() {
"/hints" => {
python
.send("HTTP/1.1 103 Early Hints\r\nlink: </ui/app.css>; rel=preload\r\n\r\nHTTP/1.1 200 OK\r\ncontent-length: 5\r\n\r\nfinal")
.await
}
_ => {
python
.send("HTTP/1.1 200 OK\r\ncontent-length: 4\r\n\r\nnext")
.await
}
}
}
})
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("GET /hints HTTP/1.1\r\nhost: x\r\n\r\nGET /next HTTP/1.1\r\nhost: x\r\n\r\n")
.await;
let mut hinted = within(raw.response_head()).await.unwrap();
while hinted.status.is_informational() {
hinted = within(raw.response_head()).await.unwrap();
}
assert_eq!(hinted.status, StatusCode::OK);
assert_eq!(raw.body(hinted.content_length().unwrap()).await, b"final");
let next = within(raw.response_head()).await.unwrap();
assert_eq!(raw.body(next.content_length().unwrap()).await, b"next");
}
#[rstest]
#[case::found(
StatusCode::FOUND,
"https://idp.example/authorize?state=a%2Fb&redirect_uri=https%3A%2F%2Fpublic.example%2Fsso%2Fcallback"
)]
#[case::see_other(StatusCode::SEE_OTHER, "/ui/?login=success")]
#[case::temporary(StatusCode::TEMPORARY_REDIRECT, "./mcp/oauth/authorize?client_id=x")]
#[tokio::test]
async fn redirects_reach_the_client_with_location_and_cookies_unchanged(
#[case] status: StatusCode,
#[case] location: &'static str,
) {
let (upstream, _received) = recording(move || {
Response::builder()
.status(status)
.header("location", location)
.header(
"set-cookie",
"token=a%2Fb; Path=/; HttpOnly; Secure; SameSite=Lax",
)
.header("set-cookie", "state=; Max-Age=0")
.body(Body::empty())
.unwrap()
})
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("GET /sso/key/generate HTTP/1.1\r\nhost: public.example\r\n\r\n")
.await;
let head = within(raw.response_head()).await.unwrap();
assert_eq!(head.status, status);
assert_eq!(head.header("location"), Some(location.as_bytes()));
let cookies: Vec<_> = head
.headers
.iter()
.filter(|(name, _)| name == "set-cookie")
.map(|(_, value)| value.as_slice())
.collect();
assert_eq!(
cookies,
[
&b"token=a%2Fb; Path=/; HttpOnly; Secure; SameSite=Lax"[..],
b"state=; Max-Age=0"
]
);
}
#[rstest]
#[tokio::test]
async fn non_websocket_upgrades_reach_python_as_plain_requests() {
let (upstream, mut received) = recording(ok).await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("GET / HTTP/1.1\r\nhost: x\r\nconnection: upgrade, http2-settings\r\nupgrade: h2c\r\nhttp2-settings: AAMAAABkAAQAoAAAAAIAAAAA\r\n\r\n")
.await;
assert_eq!(
within(raw.response_head()).await.unwrap().status,
StatusCode::OK
);
let request = received.recv().await.unwrap();
assert!(!request.headers.contains_key("upgrade"));
assert!(!request.headers.contains_key("connection"));
assert!(!request.headers.contains_key("http2-settings"));
}

View file

@ -0,0 +1,82 @@
mod support;
use std::time::Duration;
use axum::{
body::Body,
extract::Request,
http::StatusCode,
response::{IntoResponse, Response},
};
use http_body_util::BodyExt;
use rstest::rstest;
use support::{Gateway, client, raw_upstream, recording, within};
use tokio::{sync::mpsc, time::Instant};
/// An MCP client holds a GET stream open for server-initiated messages, which may not come for
/// many minutes, so neither hop may close a stream for being idle.
#[rstest]
#[tokio::test(start_paused = true)]
async fn an_idle_mcp_stream_stays_open_in_both_directions() {
let idle = Duration::from_secs(20 * 60);
let (session, mut sessions) = mpsc::unbounded_channel();
let upstream = raw_upstream(move |mut python| {
let session = session.clone();
async move {
let head = python.request_head().await.unwrap();
session
.send(head.header("mcp-session-id").map(<[u8]>::to_vec))
.unwrap();
python
.send("HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\nmcp-session-id: session-1\r\ntransfer-encoding: chunked\r\n\r\n")
.await;
tokio::time::sleep(idle).await;
python.send("c\r\ndata: ping\n\n\r\n").await;
}
})
.await;
let gateway = Gateway::start(upstream).await;
let started = Instant::now();
let mut response = client()
.request(
Request::get(gateway.url("/github/mcp"))
.header("accept", "text/event-stream")
.header("mcp-session-id", "session-1")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.headers()["mcp-session-id"], "session-1");
assert_eq!(
sessions.recv().await.unwrap().as_deref(),
Some(&b"session-1"[..])
);
let frame = response.body_mut().frame().await.unwrap().unwrap();
assert_eq!(frame.into_data().unwrap(), "data: ping\n\n");
assert!(started.elapsed() >= idle);
}
#[rstest]
#[tokio::test]
async fn ends_a_session_with_a_bodiless_delete() {
let (upstream, mut received) = recording(|| -> Response {
([("mcp-session-id", "session-1")], StatusCode::NO_CONTENT).into_response()
})
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("DELETE /mcp HTTP/1.1\r\nhost: x\r\nmcp-session-id: session-1\r\n\r\n")
.await;
let head = within(raw.response_head()).await.unwrap();
assert_eq!(head.status, StatusCode::NO_CONTENT);
assert_eq!(head.header("mcp-session-id"), Some(&b"session-1"[..]));
let request = received.recv().await.unwrap();
assert_eq!(request.method, "DELETE");
assert_eq!(request.headers["mcp-session-id"], "session-1");
assert!(!request.headers.contains_key("transfer-encoding"));
assert!(request.body.is_empty());
}

View file

@ -0,0 +1,169 @@
mod support;
use std::{
sync::Arc,
time::{Duration, Instant},
};
use axum::{body::Body, extract::Request, http::StatusCode};
use http_body_util::BodyExt;
use rstest::rstest;
use support::{DRAIN, Gateway, Raw, client, raw_upstream, unreachable_upstream, within};
use tokio::sync::{Notify, mpsc};
/// Python stops a provider stream when it sees the client go away, so a client disconnect must
/// close Rust's connection to Python at every stage of the exchange.
#[rstest]
#[case::waiting_for_headers(
"POST /v1/chat/completions HTTP/1.1\r\nhost: x\r\ncontent-length: 0\r\n\r\n",
false
)]
#[case::mid_stream(
"POST /v1/chat/completions HTTP/1.1\r\nhost: x\r\ncontent-length: 0\r\n\r\n",
true
)]
#[case::mid_upload(
"POST /v1/files HTTP/1.1\r\nhost: x\r\ncontent-length: 1000000\r\n\r\npartial",
false
)]
#[tokio::test]
async fn client_disconnect_closes_the_upstream_connection(
#[case] request: &'static str,
#[case] headers_sent: bool,
) {
let (event, mut events) = mpsc::unbounded_channel();
let upstream = raw_upstream(move |mut python: Raw| {
let event = event.clone();
async move {
python.request_head().await.unwrap();
if headers_sent {
python
.send("HTTP/1.1 200 OK\r\ntransfer-encoding: chunked\r\n\r\n6\r\ndata: \r\n")
.await;
}
event.send("received").unwrap();
python.closed().await;
event.send("closed").unwrap();
}
})
.await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send(request).await;
assert_eq!(within(events.recv()).await, Some("received"));
if headers_sent {
assert_eq!(
within(raw.response_head()).await.unwrap().status,
StatusCode::OK
);
}
drop(raw);
assert_eq!(within(events.recv()).await, Some("closed"));
}
/// uvicorn closes a connection that sat idle past its keep-alive timeout. A request Rust sends on
/// that connection as uvicorn closes it is lost, so Rust must retire idle connections first.
#[rstest]
#[case::reused_while_fresh(Duration::from_millis(0))]
#[case::past_the_pool_timeout(Duration::from_millis(250))]
#[case::past_python_keepalive(Duration::from_millis(450))]
#[tokio::test]
async fn never_reuses_a_connection_python_is_closing_for_idleness(#[case] idle: Duration) {
let keepalive = Duration::from_millis(400);
let upstream = raw_upstream(move |mut python: Raw| async move {
let mut idle_since = None::<Instant>;
while python.request_head().await.is_some() {
if idle_since.is_some_and(|since| since.elapsed() >= keepalive) {
return;
}
python
.send("HTTP/1.1 200 OK\r\ncontent-length: 2\r\n\r\nok")
.await;
idle_since = Some(Instant::now());
}
})
.await;
let gateway = Gateway::with_keepalive(upstream, keepalive).await;
let client = client();
let first = within(client.request(Request::get(gateway.url("/")).body(Body::empty()).unwrap()))
.await
.unwrap();
first.into_body().collect().await.unwrap();
tokio::time::sleep(idle).await;
let second =
within(client.request(Request::get(gateway.url("/")).body(Body::empty()).unwrap()))
.await
.unwrap();
assert_eq!(second.status(), StatusCode::OK);
}
#[rstest]
#[case::health("/health/liveliness")]
#[case::messages("/v1/messages")]
#[tokio::test]
async fn unreachable_python_is_a_502(#[case] path: &str) {
let gateway = Gateway::start(unreachable_upstream()).await;
let response =
within(client().request(Request::get(gateway.url(path)).body(Body::empty()).unwrap()))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_GATEWAY);
}
#[rstest]
#[tokio::test]
async fn shutdown_lets_an_in_flight_stream_finish() {
let release = Arc::new(Notify::new());
let released = release.clone();
let upstream = raw_upstream(move |mut python: Raw| {
let released = released.clone();
async move {
python.request_head().await.unwrap();
python
.send("HTTP/1.1 200 OK\r\ntransfer-encoding: chunked\r\n\r\n5\r\nfirst\r\n")
.await;
released.notified().await;
python.send("4\r\nlast\r\n0\r\n\r\n").await;
}
})
.await;
let gateway = Gateway::start(upstream).await;
let response =
within(client().request(Request::get(gateway.url("/")).body(Body::empty()).unwrap()))
.await
.unwrap();
let server = gateway.shutdown();
tokio::time::sleep(Duration::from_millis(100)).await;
assert!(!server.is_finished());
release.notify_one();
let body = within(response.into_body().collect())
.await
.unwrap()
.to_bytes();
assert_eq!(body, "firstlast");
within(server).await.unwrap().unwrap();
}
#[rstest]
#[tokio::test]
async fn shutdown_stops_waiting_for_a_stream_still_open_after_the_drain() {
let upstream = raw_upstream(|mut python: Raw| async move {
python.request_head().await.unwrap();
python
.send("HTTP/1.1 200 OK\r\ntransfer-encoding: chunked\r\n\r\n5\r\nfirst\r\n")
.await;
std::future::pending::<()>().await;
})
.await;
let gateway = Gateway::start(upstream).await;
let response =
within(client().request(Request::get(gateway.url("/")).body(Body::empty()).unwrap()))
.await
.unwrap();
let started = std::time::Instant::now();
within(gateway.shutdown()).await.unwrap().unwrap();
assert!(started.elapsed() >= DRAIN);
drop(response);
}

View file

@ -0,0 +1,203 @@
mod support;
use std::{
convert::Infallible,
sync::{Arc, Mutex},
};
use axum::{
body::{Body, Bytes},
http::{HeaderMap, HeaderName, HeaderValue, Method, Request, StatusCode},
response::Response,
};
use http_body_util::BodyExt;
use litellm_gateway_forward::PEER_HEADER;
use proptest::{prelude::*, test_runner::TestRunner};
use support::{Gateway, client, recording};
#[derive(Clone, Debug)]
struct Message {
headers: Vec<(String, Vec<Vec<u8>>)>,
body: Vec<u8>,
chunked: bool,
}
#[derive(Clone, Debug)]
struct Exchange {
method: Method,
target: String,
request: Message,
status: StatusCode,
response: Message,
}
fn header_value() -> impl Strategy<Value = Vec<u8>> {
prop::collection::vec(prop_oneof![0x21u8..=0x7e, 0x80u8..=0xff], 1..24)
}
fn message() -> impl Strategy<Value = Message> {
(
prop::collection::btree_map(
"x-[a-z0-9-]{1,12}",
prop::collection::vec(header_value(), 1..4),
0..12,
),
prop::collection::vec(any::<u8>(), 0..32 * 1024),
any::<bool>(),
)
.prop_map(|(headers, body, chunked)| Message {
headers: headers.into_iter().collect(),
body,
chunked,
})
}
fn target() -> impl Strategy<Value = String> {
let segment = prop_oneof!["[A-Za-z0-9._~!$&'()*+,;=:@-]{0,8}", "%[0-9A-F]{2}"];
(
prop::collection::vec(segment, 0..5),
prop::option::of("[A-Za-z0-9=&._~-]{0,20}"),
)
.prop_map(|(segments, query)| {
let path = format!("/{}", segments.join("/"));
match query {
Some(query) => format!("{path}?{query}"),
None => path,
}
})
}
fn exchange() -> impl Strategy<Value = Exchange> {
let method = prop_oneof![
Just(Method::GET),
Just(Method::POST),
Just(Method::PUT),
Just(Method::PATCH),
Just(Method::DELETE),
Just(Method::OPTIONS),
Just(Method::from_bytes(b"PURGE").unwrap()),
];
let status = (200u16..600)
.prop_filter("bodiless status", |code| ![204, 205, 304].contains(code))
.prop_map(|code| StatusCode::from_u16(code).unwrap());
(method, target(), message(), status, message()).prop_map(
|(method, target, request, status, response)| Exchange {
method,
target,
request,
status,
response,
},
)
}
fn body(message: &Message) -> Body {
if !message.chunked {
return Body::from(message.body.clone());
}
let chunks: Vec<Result<Bytes, Infallible>> = message
.body
.chunks(1000)
.map(|chunk| Ok(Bytes::copy_from_slice(chunk)))
.collect();
Body::from_stream(futures_util::stream::iter(chunks))
}
fn apply(headers: &mut HeaderMap, message: &Message) {
if message.chunked {
headers.insert("transfer-encoding", HeaderValue::from_static("chunked"));
}
for (name, values) in &message.headers {
for value in values {
headers.append(
HeaderName::from_bytes(name.as_bytes()).unwrap(),
HeaderValue::from_bytes(value).unwrap(),
);
}
}
}
fn assert_carried(headers: &HeaderMap, message: &Message, framing_and_hop: &[&str]) {
for (name, values) in &message.headers {
let carried: Vec<&[u8]> = headers
.get_all(name)
.iter()
.map(HeaderValue::as_bytes)
.collect();
let sent: Vec<&[u8]> = values.iter().map(Vec::as_slice).collect();
assert_eq!(carried, sent, "header {name}");
}
for name in headers.keys() {
assert!(
framing_and_hop.contains(&name.as_str())
|| message
.headers
.iter()
.any(|(sent, _)| sent == name.as_str()),
"unexpected header {name}"
);
}
}
/// Any request a client can send and any response Python can give cross the forwarder unchanged,
/// apart from the framing and peer headers the forwarder owns.
#[test]
fn every_exchange_crosses_the_forwarder_unchanged() {
let runtime = tokio::runtime::Runtime::new().unwrap();
let current = Arc::new(Mutex::new(None::<Exchange>));
let responding = current.clone();
let (gateway, received) = runtime.block_on(async {
let (upstream, received) = recording(move || {
let exchange = responding.lock().unwrap().clone().unwrap();
let mut response = Response::new(body(&exchange.response));
*response.status_mut() = exchange.status;
apply(response.headers_mut(), &exchange.response);
response
})
.await;
(
Gateway::start(upstream).await,
tokio::sync::Mutex::new(received),
)
});
let client = client();
TestRunner::new(ProptestConfig::with_cases(256))
.run(&exchange(), |exchange| {
*current.lock().unwrap() = Some(exchange.clone());
runtime.block_on(async {
while received.lock().await.try_recv().is_ok() {}
let mut request = Request::builder()
.method(exchange.method.clone())
.uri(gateway.url(&exchange.target))
.body(body(&exchange.request))
.unwrap();
apply(request.headers_mut(), &exchange.request);
let response = support::within(client.request(request)).await.unwrap();
let python = received.lock().await.recv().await.unwrap();
assert_eq!(python.method, exchange.method);
assert_eq!(python.uri, exchange.target.as_str());
assert_eq!(python.body, exchange.request.body);
assert_carried(
&python.headers,
&exchange.request,
&["host", "content-length", "transfer-encoding", PEER_HEADER],
);
assert_eq!(python.headers.get_all(PEER_HEADER).iter().count(), 1);
assert_eq!(response.status(), exchange.status);
let headers = response.headers().clone();
assert_carried(
&headers,
&exchange.response,
&["date", "content-length", "transfer-encoding"],
);
let body = support::within(response.into_body().collect())
.await
.unwrap();
assert_eq!(body.to_bytes(), exchange.response.body);
});
Ok(())
})
.unwrap();
}

View file

@ -0,0 +1,109 @@
use std::{ffi::OsString, time::Duration};
use litellm_gateway_forward::{Error, Settings};
use rstest::rstest;
const COMPLETE: [(&str, &str); 5] = [
("LITELLM_FORWARD_UPSTREAM", "http://127.0.0.1:53211"),
("LITELLM_FORWARD_HOST", "0.0.0.0"),
("LITELLM_FORWARD_PORT", "4000"),
("LITELLM_FORWARD_UPSTREAM_KEEPALIVE", "2"),
("LITELLM_FORWARD_SHUTDOWN_DRAIN", "40"),
];
fn parse(variables: &[(&str, &str)]) -> Result<Option<Settings>, Error> {
Settings::from_vars(
variables
.iter()
.map(|(key, value)| (OsString::from(key), OsString::from(value))),
)
}
fn without(variable: &str) -> Vec<(&'static str, &'static str)> {
COMPLETE
.into_iter()
.filter(|(key, _)| *key != variable)
.collect()
}
fn with<'a>(variable: &'a str, value: &'a str) -> Vec<(&'a str, &'a str)> {
without(variable)
.into_iter()
.chain([(variable, value)])
.collect()
}
#[rstest]
fn reads_every_variable() {
assert_eq!(
parse(&COMPLETE).unwrap(),
Some(Settings {
upstream: "http://127.0.0.1:53211".parse().unwrap(),
host: "0.0.0.0".into(),
port: 4000,
upstream_keepalive: Duration::from_secs(2),
shutdown_drain: Duration::from_secs(40),
})
);
}
#[rstest]
fn without_an_upstream_the_gateway_is_not_forwarding() {
assert_eq!(parse(&without("LITELLM_FORWARD_UPSTREAM")).unwrap(), None);
}
#[rstest]
#[case::host("LITELLM_FORWARD_HOST")]
#[case::port("LITELLM_FORWARD_PORT")]
#[case::upstream_keepalive("LITELLM_FORWARD_UPSTREAM_KEEPALIVE")]
#[case::shutdown_drain("LITELLM_FORWARD_SHUTDOWN_DRAIN")]
fn a_missing_variable_fails_naming_it(#[case] variable: &str) {
let error = parse(&without(variable)).unwrap_err();
assert!(error.to_string().contains(variable), "{error}");
}
#[rstest]
#[case::port_overflow("LITELLM_FORWARD_PORT", "65536")]
#[case::port_text("LITELLM_FORWARD_PORT", "http")]
#[case::keepalive_text("LITELLM_FORWARD_UPSTREAM_KEEPALIVE", "soon")]
#[case::keepalive_negative("LITELLM_FORWARD_UPSTREAM_KEEPALIVE", "-1")]
#[case::drain_text("LITELLM_FORWARD_SHUTDOWN_DRAIN", "soon")]
#[case::drain_negative("LITELLM_FORWARD_SHUTDOWN_DRAIN", "-1")]
#[case::drain_infinite("LITELLM_FORWARD_SHUTDOWN_DRAIN", "inf")]
fn an_invalid_variable_fails_naming_it(#[case] variable: &str, #[case] value: &str) {
let error = parse(&with(variable, value)).unwrap_err();
assert!(error.to_string().contains(variable), "{error}");
}
#[rstest]
fn fractional_drain_seconds_are_used_as_given() {
let settings = parse(&with("LITELLM_FORWARD_SHUTDOWN_DRAIN", "12.5"))
.unwrap()
.unwrap();
assert_eq!(settings.shutdown_drain, Duration::from_millis(12_500));
}
#[rstest]
#[case::keepalive("KEEPALIVE_TIMEOUT")]
#[case::graceful_shutdown("GRACEFUL_SHUTDOWN_TIMEOUT")]
#[case::host("HOST")]
#[case::port("PORT")]
fn python_settings_are_never_read(#[case] variable: &str) {
let mut variables = COMPLETE.to_vec();
variables.push((variable, "9"));
assert_eq!(parse(&variables).unwrap(), parse(&COMPLETE).unwrap());
}
#[rstest]
fn a_non_loopback_upstream_fails_startup() {
let error = parse(&with("LITELLM_FORWARD_UPSTREAM", "http://192.0.2.1:4000")).unwrap_err();
assert!(
error.to_string().contains("LITELLM_FORWARD_UPSTREAM"),
"{error}"
);
}

View file

@ -0,0 +1,41 @@
mod support;
use std::time::Duration;
use axum::{body::Body, extract::Request, http::StatusCode};
use http_body_util::BodyExt;
use rstest::rstest;
use support::{Gateway, client, raw_upstream};
use tokio::time::Instant;
/// Reasoning models with `stream=false`, image and video generation, and model health checks send
/// no response headers until the provider finishes, which can take many minutes.
#[rstest]
#[tokio::test(start_paused = true)]
async fn waits_for_response_headers_as_long_as_python_takes() {
let thinking = Duration::from_secs(30 * 60);
let upstream = raw_upstream(move |mut python| async move {
python.request_head().await.unwrap();
tokio::time::sleep(thinking).await;
python
.send("HTTP/1.1 200 OK\r\ncontent-length: 5\r\n\r\nimage")
.await;
})
.await;
let gateway = Gateway::start(upstream).await;
let started = Instant::now();
let response = client()
.request(
Request::post(gateway.url("/v1/images/generations"))
.body(Body::from("{}"))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert!(started.elapsed() >= thinking);
let body = response.into_body().collect().await.unwrap().to_bytes();
assert_eq!(body, "image");
}

View file

@ -0,0 +1,136 @@
mod support;
use std::sync::Arc;
use axum::{body::Body, extract::Request, http::StatusCode};
use http_body_util::BodyExt;
use rstest::rstest;
use support::{Gateway, client, raw_upstream, within};
use tokio::sync::Notify;
fn chunk(bytes: &[u8]) -> Vec<u8> {
[format!("{:x}\r\n", bytes.len()).as_bytes(), bytes, b"\r\n"].concat()
}
#[rstest]
#[tokio::test]
async fn each_event_reaches_the_client_before_python_produces_the_next() {
let produce = Arc::new(Notify::new());
let next = produce.clone();
let upstream = raw_upstream(move |mut python| {
let next = next.clone();
async move {
python.request_head().await.unwrap();
python
.send("HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\ntransfer-encoding: chunked\r\n\r\n")
.await;
for event in ["data: 1\n\n", "data: 2\n\n", "data: [DONE]\n\n"] {
python.send(chunk(event.as_bytes())).await;
next.notified().await;
}
python.send("0\r\n\r\n").await;
}
})
.await;
let gateway = Gateway::start(upstream).await;
let mut response = within(
client().request(
Request::post(gateway.url("/v1/chat/completions"))
.body(Body::from(r#"{"stream":true}"#))
.unwrap(),
),
)
.await
.unwrap();
for event in ["data: 1\n\n", "data: 2\n\n", "data: [DONE]\n\n"] {
let frame = within(response.body_mut().frame()).await.unwrap().unwrap();
assert_eq!(frame.into_data().unwrap(), event);
produce.notify_one();
}
assert!(within(response.body_mut().frame()).await.is_none());
}
#[rstest]
#[tokio::test]
async fn relays_a_binary_framed_stream_unchanged() {
let message: Vec<u8> = [
&[
0x00, 0x00, 0x00, 0x3d, 0x00, 0x00, 0x00, 0x20, 0x8a, 0x0d, 0x0a, 0xff,
][..],
b"\x0b:event-type\x07\x00\x05chunk\x0d:content-type\x07\x00\x10application/json",
b"0\r\n\r\n{\"bytes\":\"eyJ0eXBlIjoiY29udGVudCJ9\"}",
&[0xc3, 0x28, 0x00, 0x0d],
]
.concat();
let sent = message.clone();
let upstream = raw_upstream(move |mut python| {
let sent = sent.clone();
async move {
python.request_head().await.unwrap();
python
.send("HTTP/1.1 200 OK\r\ncontent-type: application/vnd.amazon.eventstream\r\ntransfer-encoding: chunked\r\n\r\n")
.await;
for piece in sent.chunks(7) {
python.send(chunk(piece)).await;
}
python.send("0\r\n\r\n").await;
}
})
.await;
let gateway = Gateway::start(upstream).await;
let response = within(
client().request(
Request::post(gateway.url("/bedrock/model/m/converse-stream"))
.body(Body::empty())
.unwrap(),
),
)
.await
.unwrap();
assert_eq!(
response.headers()["content-type"],
"application/vnd.amazon.eventstream"
);
let body = within(response.into_body().collect())
.await
.unwrap()
.to_bytes();
assert_eq!(body, message);
}
#[rstest]
#[tokio::test]
async fn relays_a_body_delimited_by_the_upstream_closing() {
let upstream = raw_upstream(|mut python| async move {
python.request_head().await.unwrap();
python
.send("HTTP/1.1 200 OK\r\nconnection: close\r\n\r\nuntil eof")
.await;
})
.await;
let gateway = Gateway::start(upstream).await;
let response = client().request(Request::get(gateway.url("/")).body(Body::empty()).unwrap());
let body = within(response).await.unwrap().into_body().collect();
assert_eq!(within(body).await.unwrap().to_bytes(), "until eof");
}
#[rstest]
#[tokio::test]
async fn an_upstream_dying_mid_stream_fails_the_client_body_instead_of_ending_it() {
let upstream = raw_upstream(|mut python| async move {
python.request_head().await.unwrap();
python
.send("HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\ntransfer-encoding: chunked\r\n\r\n6\r\ndata: \r\n")
.await;
})
.await;
let gateway = Gateway::start(upstream).await;
let response =
within(client().request(Request::get(gateway.url("/")).body(Body::empty()).unwrap()))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert!(within(response.into_body().collect()).await.is_err());
}

View file

@ -0,0 +1,305 @@
#![allow(
dead_code,
reason = "each test binary uses a different subset of the harness"
)]
use std::{future::Future, net::SocketAddr, time::Duration};
use axum::{
Router,
body::{Body, Bytes},
extract::Request,
http::{HeaderMap, Method, StatusCode, Uri},
};
use http_body_util::BodyExt;
use hyper_util::{
client::legacy::{Client, connect::HttpConnector},
rt::TokioExecutor,
};
use litellm_gateway_forward::Forward;
use tokio::{
io::{AsyncReadExt, AsyncWriteExt},
net::{TcpListener, TcpStream},
sync::{mpsc, oneshot},
task::JoinHandle,
};
pub const WAIT: Duration = Duration::from_secs(5);
pub const UVICORN_KEEPALIVE: Duration = Duration::from_secs(5);
pub const DRAIN: Duration = Duration::from_secs(1);
pub async fn within<T>(future: impl Future<Output = T>) -> T {
tokio::time::timeout(WAIT, future).await.expect("timed out")
}
/// A running forwarder in front of `upstream`, listening on a real socket like production.
pub struct Gateway {
pub address: SocketAddr,
stop: oneshot::Sender<()>,
server: JoinHandle<std::io::Result<()>>,
}
impl Gateway {
pub async fn start(upstream: SocketAddr) -> Self {
Self::with_keepalive(upstream, UVICORN_KEEPALIVE).await
}
pub async fn with_keepalive(upstream: SocketAddr, keepalive: Duration) -> Self {
let forward = Forward::new(&format!("http://{upstream}").parse().unwrap(), keepalive);
let public = TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = public.local_addr().unwrap();
let (stop, stopped) = oneshot::channel::<()>();
let server = tokio::spawn(forward.serve(
public,
async {
stopped.await.ok();
},
DRAIN,
));
Self {
address,
stop,
server,
}
}
pub fn url(&self, path_and_query: &str) -> Uri {
format!("http://{}{path_and_query}", self.address)
.parse()
.unwrap()
}
pub async fn connect(&self) -> Raw {
Raw::new(TcpStream::connect(self.address).await.unwrap())
}
/// Starts graceful shutdown and returns the server task, which ends once in-flight work drains
/// or `DRAIN` passes.
pub fn shutdown(self) -> JoinHandle<std::io::Result<()>> {
self.stop.send(()).unwrap();
self.server
}
}
pub fn client() -> Client<HttpConnector, Body> {
Client::builder(TokioExecutor::new()).build_http()
}
/// What the upstream received, after the forwarder rewrote it.
pub struct Received {
pub method: Method,
pub uri: Uri,
pub headers: HeaderMap,
pub body: Bytes,
}
/// An axum upstream that records every request it fully reads and answers with `respond`.
pub async fn recording(
respond: impl Fn() -> axum::response::Response + Clone + Send + Sync + 'static,
) -> (SocketAddr, mpsc::UnboundedReceiver<Received>) {
let (sender, receiver) = mpsc::unbounded_channel();
let address = serve(Router::new().fallback(move |request: Request| {
let sender = sender.clone();
let respond = respond.clone();
async move {
let (parts, body) = request.into_parts();
let body = body.collect().await.unwrap().to_bytes();
sender
.send(Received {
method: parts.method,
uri: parts.uri,
headers: parts.headers,
body,
})
.unwrap();
respond()
}
}))
.await;
(address, receiver)
}
pub async fn serve(router: Router) -> SocketAddr {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
tokio::spawn(async move { axum::serve(listener, router).await.unwrap() });
address
}
/// An upstream that hands each accepted connection to `handle` as raw bytes, for behavior a
/// well-behaved HTTP server would never produce.
pub async fn raw_upstream<F, Fut>(handle: F) -> SocketAddr
where
F: Fn(Raw) -> Fut + Send + Sync + 'static,
Fut: Future<Output = ()> + Send + 'static,
{
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
let (stream, _) = listener.accept().await.unwrap();
tokio::spawn(handle(Raw::new(stream)));
}
});
address
}
pub fn unreachable_upstream() -> SocketAddr {
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
listener.local_addr().unwrap()
}
pub struct Head {
pub status: StatusCode,
pub headers: Vec<(String, Vec<u8>)>,
pub path: String,
}
impl Head {
pub fn header(&self, name: &str) -> Option<&[u8]> {
self.headers
.iter()
.find(|(key, _)| key.eq_ignore_ascii_case(name))
.map(|(_, value)| value.as_slice())
}
pub fn content_length(&self) -> Option<usize> {
self.header("content-length")
.map(|value| std::str::from_utf8(value).unwrap().parse().unwrap())
}
}
/// One TCP connection driven byte by byte, so tests control framing, timing and disconnects.
/// Bytes read past a message head stay buffered for the next read.
pub struct Raw {
stream: TcpStream,
buffer: Vec<u8>,
}
impl Raw {
pub fn new(stream: TcpStream) -> Self {
Self {
stream,
buffer: Vec::new(),
}
}
pub fn into_stream(self) -> TcpStream {
assert!(self.buffer.is_empty(), "buffered bytes would be lost");
self.stream
}
pub fn local_address(&self) -> SocketAddr {
self.stream.local_addr().unwrap()
}
pub async fn send(&mut self, bytes: impl AsRef<[u8]>) {
self.stream.write_all(bytes.as_ref()).await.unwrap();
}
/// Reads one request head, or `None` when the peer closed the connection first.
pub async fn request_head(&mut self) -> Option<Head> {
self.head(|bytes| {
let mut headers = [httparse::EMPTY_HEADER; 256];
let mut request = httparse::Request::new(&mut headers);
let httparse::Status::Complete(length) = request.parse(bytes).unwrap() else {
return None;
};
Some((
length,
Head {
status: StatusCode::OK,
headers: owned(request.headers),
path: request.path.unwrap().to_owned(),
},
))
})
.await
}
pub async fn response_head(&mut self) -> Option<Head> {
self.head(|bytes| {
let mut headers = [httparse::EMPTY_HEADER; 256];
let mut response = httparse::Response::new(&mut headers);
let httparse::Status::Complete(length) = response.parse(bytes).unwrap() else {
return None;
};
Some((
length,
Head {
status: StatusCode::from_u16(response.code.unwrap()).unwrap(),
headers: owned(response.headers),
path: String::new(),
},
))
})
.await
}
async fn head(&mut self, parse: impl Fn(&[u8]) -> Option<(usize, Head)>) -> Option<Head> {
loop {
if let Some((length, head)) = parse(&self.buffer) {
self.buffer.drain(..length);
return Some(head);
}
if self.fill().await == 0 {
return None;
}
}
}
/// Reads exactly `length` body bytes, using what is already buffered first.
pub async fn body(&mut self, length: usize) -> Vec<u8> {
while self.buffer.len() < length {
assert_ne!(self.fill().await, 0, "closed mid-body");
}
self.buffer.drain(..length).collect()
}
/// Reads a whole chunked body and returns its decoded bytes.
pub async fn chunked_body(&mut self) -> Vec<u8> {
let mut decoded = Vec::new();
loop {
let size_line = self.line().await;
let size =
usize::from_str_radix(size_line.split(';').next().unwrap().trim(), 16).unwrap();
if size == 0 {
while !self.line().await.is_empty() {}
return decoded;
}
decoded.extend(self.body(size).await);
assert_eq!(self.body(2).await, b"\r\n");
}
}
async fn line(&mut self) -> String {
loop {
if let Some(end) = self.buffer.windows(2).position(|window| window == b"\r\n") {
let line = String::from_utf8(self.buffer[..end].to_vec()).unwrap();
self.buffer.drain(..end + 2);
return line;
}
assert_ne!(self.fill().await, 0, "closed mid-line");
}
}
async fn fill(&mut self) -> usize {
let mut chunk = [0; 8192];
let read = self.stream.read(&mut chunk).await.unwrap_or(0);
self.buffer.extend_from_slice(&chunk[..read]);
read
}
/// Resolves once the peer closes the connection, discarding anything it still sends.
pub async fn closed(&mut self) {
let mut chunk = [0; 8192];
while self.stream.read(&mut chunk).await.unwrap_or(0) != 0 {}
}
}
fn owned(headers: &[httparse::Header<'_>]) -> Vec<(String, Vec<u8>)> {
headers
.iter()
.map(|header| (header.name.to_owned(), header.value.to_vec()))
.collect()
}

View file

@ -0,0 +1,92 @@
mod support;
use std::convert::Infallible;
use axum::{
Router,
body::{Body, Bytes},
extract::Request,
http::StatusCode,
response::Response,
};
use http_body_util::BodyExt;
use rstest::rstest;
use support::{Gateway, client, recording, serve, within};
use tokio::sync::mpsc;
fn ok() -> Response {
Response::new(Body::from("ok"))
}
#[rstest]
#[tokio::test]
async fn streams_upload_and_response_incrementally_without_a_body_limit() {
let upstream = serve(
Router::new().fallback(|request: Request| async { Response::new(request.into_body()) }),
)
.await;
let gateway = Gateway::start(upstream).await;
let (sender, receiver) = mpsc::channel::<Result<Bytes, Infallible>>(1);
let stream = futures_util::stream::unfold(receiver, |mut receiver| async {
receiver.recv().await.map(|chunk| (chunk, receiver))
});
sender.send(Ok(Bytes::from_static(b"first"))).await.unwrap();
let mut response = within(
client().request(
Request::post(gateway.url("/v1/files"))
.body(Body::from_stream(stream))
.unwrap(),
),
)
.await
.unwrap();
let frame = within(response.body_mut().frame()).await.unwrap().unwrap();
assert_eq!(frame.into_data().unwrap(), "first");
let large = Bytes::from(vec![b'x'; 8 * 1024 * 1024]);
sender.send(Ok(large.clone())).await.unwrap();
drop(sender);
let rest = within(response.into_body().collect())
.await
.unwrap()
.to_bytes();
assert_eq!(rest, large);
}
#[rstest]
#[tokio::test]
async fn answers_expect_continue_so_large_uploads_start() {
let (upstream, mut received) = recording(ok).await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send("POST /v1/audio/transcriptions HTTP/1.1\r\nhost: x\r\nexpect: 100-continue\r\ncontent-length: 4\r\n\r\n")
.await;
let interim = within(raw.response_head()).await.unwrap();
assert_eq!(interim.status, StatusCode::CONTINUE);
raw.send("data").await;
assert_eq!(
within(raw.response_head()).await.unwrap().status,
StatusCode::OK
);
assert_eq!(received.recv().await.unwrap().body, "data");
}
#[rstest]
#[case::get("GET")]
#[case::delete("DELETE")]
#[case::options("OPTIONS")]
#[case::post("POST")]
#[tokio::test]
async fn forwards_a_chunked_body_on_any_method(#[case] method: &str) {
let (upstream, mut received) = recording(ok).await;
let gateway = Gateway::start(upstream).await;
let mut raw = gateway.connect().await;
raw.send(format!(
"{method} /v1/search HTTP/1.1\r\nhost: x\r\ntransfer-encoding: chunked\r\n\r\n4\r\nbody\r\n0\r\n\r\n"
))
.await;
assert_eq!(
within(raw.response_head()).await.unwrap().status,
StatusCode::OK
);
assert_eq!(received.recv().await.unwrap().body, "body");
}

View file

@ -0,0 +1,21 @@
use litellm_gateway_forward::Upstream;
use rstest::rstest;
#[rstest]
#[case::https("https://127.0.0.1:4000")]
#[case::public("http://192.0.2.1:4000")]
#[case::hostname("http://localhost:4000")]
#[case::ipv6("http://[::1]:4000")]
#[case::path("http://127.0.0.1:4000/api")]
#[case::query("http://127.0.0.1:4000/?x=y")]
#[case::credentials("http://user@127.0.0.1:4000")]
fn rejects_anything_but_a_loopback_http_origin(#[case] upstream: &str) {
assert!(upstream.parse::<Upstream>().is_err());
}
#[rstest]
#[case::standard("http://127.0.0.1:4000")]
#[case::loopback_range("http://127.0.0.2:4000/")]
fn accepts_a_loopback_http_origin(#[case] upstream: &str) {
assert!(upstream.parse::<Upstream>().is_ok());
}

View file

@ -0,0 +1,198 @@
mod support;
use std::{
net::SocketAddr,
time::{Duration, Instant},
};
use axum::{
Router,
http::{HeaderValue, StatusCode},
response::IntoResponse,
};
use futures_util::{SinkExt, StreamExt};
use litellm_gateway_forward::PEER_HEADER;
use rstest::rstest;
use support::{DRAIN, Gateway, raw_upstream, serve, within};
use tokio::net::TcpStream;
use tokio_tungstenite::{
WebSocketStream, accept_hdr_async, client_async,
tungstenite::{
Error, Message,
client::IntoClientRequest,
handshake::server::{ErrorResponse, Request as Handshake, Response as Accepted},
protocol::{CloseFrame, frame::coding::CloseCode},
},
};
async fn connect(
address: SocketAddr,
connection: &'static str,
) -> Result<WebSocketStream<TcpStream>, Box<Error>> {
let mut request = format!("ws://{address}/v1/realtime?model=m")
.into_client_request()
.unwrap();
let headers = request.headers_mut();
headers.insert("authorization", HeaderValue::from_static("Bearer test"));
headers.insert(
"sec-websocket-protocol",
HeaderValue::from_static("realtime"),
);
headers.insert("connection", HeaderValue::from_static(connection));
let stream = TcpStream::connect(address).await.unwrap();
let (socket, response) = client_async(request, stream).await.map_err(Box::new)?;
assert_eq!(response.headers()["sec-websocket-protocol"], "realtime");
Ok(socket)
}
/// Greets, echoes every message, and closes with 4000 when told to "bye".
#[expect(
clippy::result_large_err,
reason = "tungstenite's handshake callback returns ErrorResponse"
)]
async fn echo() -> SocketAddr {
raw_upstream(|python| async move {
let callback = |request: &Handshake, mut response: Accepted| {
assert_eq!(request.uri(), "/v1/realtime?model=m");
assert_eq!(request.headers()["authorization"], "Bearer test");
assert!(request.headers().contains_key(PEER_HEADER));
response.headers_mut().insert(
"sec-websocket-protocol",
HeaderValue::from_static("realtime"),
);
Ok::<_, ErrorResponse>(response)
};
let mut socket = accept_hdr_async(python.into_stream(), callback)
.await
.unwrap();
socket.send(Message::text("greeting")).await.unwrap();
while let Some(Ok(message)) = socket.next().await {
if message == Message::text("bye") {
socket
.close(Some(CloseFrame {
code: CloseCode::Library(4000),
reason: "done".into(),
}))
.await
.unwrap();
continue;
}
if message.is_close() {
break;
}
socket.send(message).await.unwrap();
}
})
.await
}
#[rstest]
#[tokio::test]
async fn relays_messages_both_ways(
#[values("Upgrade", "keep-alive, Upgrade", "upgrade")] connection: &'static str,
#[values(
Message::text("hello"),
Message::binary(vec![0, 1, 2]),
Message::binary(vec![7; 4 * 1024 * 1024])
)]
message: Message,
) {
let gateway = Gateway::start(echo().await).await;
let mut socket = within(connect(gateway.address, connection)).await.unwrap();
assert_eq!(
socket.next().await.unwrap().unwrap(),
Message::text("greeting")
);
socket.send(message.clone()).await.unwrap();
assert_eq!(within(socket.next()).await.unwrap().unwrap(), message);
socket.close(None).await.unwrap();
while socket.next().await.is_some() {}
drop(socket);
within(gateway.shutdown()).await.unwrap().unwrap();
}
#[rstest]
#[tokio::test]
async fn relays_a_close_python_initiates() {
let gateway = Gateway::start(echo().await).await;
let mut socket = within(connect(gateway.address, "Upgrade")).await.unwrap();
socket.next().await.unwrap().unwrap();
socket.send(Message::text("bye")).await.unwrap();
let Message::Close(Some(frame)) = within(socket.next()).await.unwrap().unwrap() else {
panic!("expected python's close frame");
};
assert_eq!(frame.code, CloseCode::Library(4000));
assert_eq!(frame.reason, "done");
}
#[rstest]
#[tokio::test]
async fn passes_an_upstream_rejection_through() {
let upstream = serve(
Router::new()
.fallback(|| async { (StatusCode::FORBIDDEN, "origin not allowed").into_response() }),
)
.await;
let gateway = Gateway::start(upstream).await;
let error = within(connect(gateway.address, "Upgrade"))
.await
.unwrap_err();
let Error::Http(response) = *error else {
panic!("expected the upstream rejection, got {error}");
};
assert_eq!(response.status(), StatusCode::FORBIDDEN);
assert_eq!(response.body().as_deref(), Some(&b"origin not allowed"[..]));
}
#[rstest]
#[tokio::test]
async fn shutdown_closes_open_sockets_after_drain() {
let gateway = Gateway::start(echo().await).await;
let mut socket = within(connect(gateway.address, "Upgrade")).await.unwrap();
socket.next().await.unwrap().unwrap();
let started = Instant::now();
within(gateway.shutdown()).await.unwrap().unwrap();
assert!(started.elapsed() >= DRAIN);
assert!(!matches!(socket.next().await, Some(Ok(Message::Text(_)))));
}
#[rstest]
#[tokio::test]
async fn shutdown_relays_the_close_python_sends_while_draining() {
let gateway = Gateway::start(echo().await).await;
let mut socket = within(connect(gateway.address, "Upgrade")).await.unwrap();
socket.next().await.unwrap().unwrap();
let server = gateway.shutdown();
tokio::time::sleep(Duration::from_millis(100)).await;
socket.send(Message::text("bye")).await.unwrap();
let Message::Close(Some(frame)) = within(socket.next()).await.unwrap().unwrap() else {
panic!("expected python's close frame");
};
assert_eq!(frame.code, CloseCode::Library(4000));
while socket.next().await.is_some() {}
drop(socket);
let started = Instant::now();
within(server).await.unwrap().unwrap();
assert!(started.elapsed() < DRAIN / 2);
}
#[rstest]
#[tokio::test]
async fn refuses_new_sockets_while_draining() {
let gateway = Gateway::start(echo().await).await;
let mut open = within(connect(gateway.address, "Upgrade")).await.unwrap();
open.next().await.unwrap().unwrap();
let address = gateway.address;
let server = gateway.shutdown();
tokio::time::sleep(Duration::from_millis(100)).await;
assert!(
TcpStream::connect(address).await.is_err() || connect(address, "Upgrade").await.is_err()
);
open.send(Message::text("still open")).await.unwrap();
assert_eq!(
within(open.next()).await.unwrap().unwrap(),
Message::text("still open")
);
drop(open);
within(server).await.unwrap().unwrap();
}

View file

@ -12,6 +12,7 @@ envy = "0.4.2"
http-body-util = "0.1"
litellm-inference.workspace = true
litellm-gateway-inference.workspace = true
litellm-gateway-forward.workspace = true
litellm-gateway-mcp.workspace = true
tokio-util = "0.7"
thiserror.workspace = true

View file

@ -10,7 +10,7 @@ use serde_json::json;
mod settings;
use settings::{Settings, UiSettings};
use settings::{LogSettings, Settings, UiSettings};
struct StderrSink {
level: Level,
@ -41,6 +41,15 @@ impl Sink for StderrSink {
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
if let Some(forward) = litellm_gateway_forward::Settings::from_env()? {
let log: LogSettings = settings::from_iter(std::env::vars_os())?;
Logger::new(StderrSink {
level: log.log_level(),
})
.install_global()?;
litellm_gateway_forward::run(forward, shutdown_signal()).await?;
return Ok(());
}
let settings: Settings = settings::from_iter(std::env::vars_os())?;
let level = settings.log_level();
Logger::new(StderrSink { level }).install_global()?;

View file

@ -4,6 +4,23 @@ use litellm_secret_value::SecretValue;
use litellm_tracing::Level;
use serde::{Deserialize, de::DeserializeOwned};
#[derive(Deserialize)]
pub(super) struct LogSettings {
rust_log: Option<String>,
}
impl LogSettings {
pub fn log_level(&self) -> Level {
log_level(self.rust_log.as_deref())
}
}
fn log_level(rust_log: Option<&str>) -> Level {
rust_log
.and_then(|value| value.parse().ok())
.unwrap_or(Level::INFO)
}
#[derive(Deserialize)]
#[serde(default)]
pub(super) struct Settings {
@ -26,10 +43,7 @@ impl Default for Settings {
impl Settings {
pub fn log_level(&self) -> Level {
self.rust_log
.as_deref()
.and_then(|value| value.parse().ok())
.unwrap_or(Level::INFO)
log_level(self.rust_log.as_deref())
}
}