scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-29 06:33:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 201

docs(rust): focus the rust-error gateway page on the error flow and add a rust-gateway skill

Browse source 
The former axum page toured gateway-inference: its status match, the envelope
table, an extractor, the unsupported route and the Sse closure. Those details
belong to the gateway crates and go stale with them. The page is now gateway.md
and follows an error from the provider failure through each From into
IntoResponse, where it is classified once and rendered by surface and framing.

The gateway-structure material moves to a new rust-gateway skill: envelopes per
API surface, extractors and path registration, SSE relay, and the router test
harness.
This commit is contained in:
Yujong Lee 2026-10-23 11:22:40 -07:00
parent 57b392c624
commit ca8c6c698f
8 changed files with 137 additions and 77 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
litellm-rust/.agents/skills/rust-error/SKILL.md
View file

@ -22,8 +22,8 @@ description: Define, wrap, convert, classify, report or test Rust error types in
- Classification states facts. Retry and fallback policy lives with the caller that knows idempotency
- [boundaries](references/boundaries.md)
- The gateway and the Python bridge are the only places an error becomes text for someone else, and both ask the same predicates in the same order
- [axum](references/axum.md)
- One `Error` with `IntoResponse` per gateway crate. `status()` is pure and separate from the envelope, extractors reject with the same type
- [gateway](references/gateway.md)
- One `Error` with `IntoResponse` per gateway crate. Every failure flows through the same `From` chain, is classified once in `status()` and rendered by surface and framing
- [pyo3](references/pyo3.md)
- Two `create_exception!` classes with the payload in `args`, `ValueError` for the caller's fault and `RuntimeError` otherwise, decoded once in Python
- [testing](references/testing.md)

74
litellm-rust/.agents/skills/rust-error/references/axum.md
View file

@ -1,74 +0,0 @@
# Gateway HTTP
Each gateway crate has one `Error` with `IntoResponse`, so a handler returns `Result<_, Error>` and every `?` ends in the right status and envelope. `crates/gateway-inference/src/error.rs` is the reference. Axum's own conventions are in its [error handling docs](https://docs.rs/axum/latest/axum/error_handling/index.html)
## Status and body are separate
`status()` picks the code and is pure. `body(request_id)` builds the Anthropic envelope, `openai_response()` the OpenAI one, and `sse_frame()` reuses `body()` for a stream that fails after its headers went out. Keeping them apart is what lets a streaming failure and a JSON failure of the same error agree on everything but framing
```rust
pub fn status(&self) -> StatusCode {
match self {
Self::Auth(error) => error.status(),
Self::Unsupported(_) => StatusCode::NOT_IMPLEMENTED,
Self::BodyTooLarge => StatusCode::PAYLOAD_TOO_LARGE,
Self::InvalidBody(_) | Self::UnknownModel(_) => StatusCode::BAD_REQUEST,
Self::Internal(_) => StatusCode::INTERNAL_SERVER_ERROR,
Self::Ocr(error) => ocr_status(error),
Self::Route(error) => route_status(error),
}
}
```
Variants the gateway declares itself are matched. A nested crate's error goes to one function that asks its predicates in order: unsupported, then the provider's own status, then a missing credential, then the caller's fault, else 500. Do not reach into the nested enum's variants from the gateway, add a predicate on the nested error instead, see [classification](classification.md)
## Envelopes follow the API surface
Each surface mirrors what the Python proxy returns there, so the shapes differ on purpose
| Surface | Envelope |
|---|---|
| `/v1/messages` | Anthropic, `{"type": "error", "error": {"type", "message"}, "request_id"}` from `body()` |
| OpenAI-shaped routes | `{"error": {"message", "type", "param", "code"}}` from `openai_response()` |
| MCP | `{"detail": message}` in `crates/gateway-mcp` |
| UI | `{"error": {"message"}}` in `crates/gateway-ui` |
| auth rejections | plain text in `crates/gateway-auth` |
The error `type` string is derived from the status, so a new failure never names its own type. A provider body already in the surface's shape passes through untouched, anything else has its message extracted and re-enveloped. The body was bounded by `truncate_error_body` when the error was built, see [boundaries](boundaries.md)
## Extractors reject with the same Error
A custom extractor sets `type Rejection = Error`, so a bad body gets the surface's envelope rather than axum's plain-text rejection
```rust
impl<S: Send + Sync> FromRequest<S> for JsonObject {
type Rejection = Error;
async fn from_request(request: Request, state: &S) -> Result<Self, Self::Rejection> {
let body = Bytes::from_request(request, state).await.map_err(|error| {
if error.status() == StatusCode::PAYLOAD_TOO_LARGE {
Error::BodyTooLarge
} else {
Error::InvalidBody(error.to_string())
}
})?;
object(&body).map(Self)
}
}
```
When the handler needs the request id in the error body, take the extractor as `Result<JsonObject, Error>` so the handler maps the rejection itself, as `messages::create` does. `Infallible` is the right rejection for an extractor that reads an optional header
A path the gateway knows but does not serve, such as `/v1/completions`, is routed to `request::unsupported`, which returns `Error::Unsupported(path)` so the client gets a 501 in the surface's envelope rather than axum's bare 404
## Streaming
Once the response headers are sent the status cannot change. `litellm_host_http::Sse::new(Json, |error| Bytes::from(Error::from(error).sse_frame()))` takes the closure that renders a mid-stream failure as an `event: error` frame. `litellm_host_http::Error<E>` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, which the `From` maps to `Internal`
## Known gaps
`InvalidBody(String)` and `Internal(String)` flatten their cause into a string, against [messages and causes](messages-and-causes.md). A new variant carries the typed cause
## Testing
`status()` and `body()` are pure, so unit tests call them on errors built the way production builds them, one `#[case]` per fault, as `status_follows_who_is_at_fault` and `body_follows_the_anthropic_exception_mapping` do. For the full path, `tests/support/mod.rs` builds a `Router` and `post()` drives it with `tower::ServiceExt::oneshot`. Assert the status and the fields the surface promises, not the whole body

2
litellm-rust/.agents/skills/rust-error/references/boundaries.md
View file

@ -4,7 +4,7 @@ An error becomes text for someone else in exactly two places, and both are expli
| Boundary | File | Reference |
|---|---|---|
| Gateway HTTP response | `crates/gateway-inference/src/error.rs` | [axum](axum.md) |
| Gateway HTTP response | `crates/gateway-inference/src/error.rs` | [gateway](gateway.md) |
| Python bridge | `crates/python-bridge/src/errors.rs` and `crates/python-bridge/src/routes/ocr/errors.rs` | [pyo3](pyo3.md) |
Both ask a nested error the same questions in the same order

54
litellm-rust/.agents/skills/rust-error/references/gateway.md Normal file
View file

@ -0,0 +1,54 @@
# Gateway responses
The gateway is where a Rust error becomes an HTTP response. This page is about the path an error takes to get there and the one place it is classified. The gateway crates' own conventions, such as which envelope each API surface uses, which extractors exist and how the router is composed, live in the [`rust-gateway`](../../rust-gateway/SKILL.md) skill
## The flow
```text
reqwest / provider failure
-> litellm_http::Error From<reqwest::Error>, strips the URL, classifies once
-> litellm_inference::Error::Http(_) From, one transparent variant, predicates delegate
-> gateway_inference::Error::Route(_) From, one variant per nested crate
-> handler returns Result<_, Error> every `?` ends here
-> IntoResponse status() picks the code, a renderer builds the body
```
Each hop is one `From` that nests the lower error in one variant, see [conversions](conversions.md). No hop re-derives what the lower crate already decided, and nothing between the provider and `IntoResponse` formats an error for the client. That is what makes "one `Error` with `IntoResponse` per gateway crate" work: a handler can `?` any failure from any depth and the response comes out right
## Classify once
`status()` is pure and runs once per error. Variants the gateway declares itself, such as a bad body, an unknown model or an oversized body, are matched at the top of its own enum. A nested crate's error goes to one function that asks the predicates in the order [boundaries](boundaries.md) gives, which here means
| Answer | Status |
|---|---|
| `declined()` | 501, the gateway matched and parsed the request but core cannot serve it, nothing reached the provider |
| `upstream_response()` | the provider's own status, 502 when it is not a valid code |
| `is_missing_credential()` | 401 |
| `is_request()` | 400 |
| otherwise | 500 |
Do not reach into a nested enum's variants from the gateway. When a new nested failure needs a different status, add a predicate on the nested error, see [classification](classification.md)
## Render by surface and framing
Rendering is separate from classification. The surface picks the envelope, which is why `gateway_inference::Error` has both `body(request_id)` for the Anthropic shape and `openai_response()` for the OpenAI shape, and the moment picks the framing: a JSON body before the headers went out, an `event: error` SSE frame from `sse_frame()` after. Both framings render the same `body()`, so a streaming failure and a JSON failure of the same error agree on everything but framing
Two rules hold for every surface. The error `type` string is derived from the status, so a new failure never names its own type. A provider body already in the surface's shape passes through untouched and anything else has its message extracted, with the body bounded by `truncate_error_body` when the error was built, see [boundaries](boundaries.md)
## Every way in uses the same Error
A request can fail before the handler runs, in the handler, or after the headers went out. All three end in the same `Error`, so they get the same classification and envelope
- An extractor sets `type Rejection = Error`, so a bad body is the surface's envelope rather than axum's plain text
- A path the gateway knows but does not serve returns `Error::Unsupported(path)` from one handler, so the client gets a 501 in the surface's envelope rather than a bare 404
- A stream that fails after its headers went out hands `Error::from(error).sse_frame()` to `litellm_host_http::Sse::new`. `litellm_host_http::Error<E>` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, and the `From` maps the second to `Internal`
The axum hooks behind each line are in [`rust-gateway`](../../rust-gateway/SKILL.md)
## Known gaps
`InvalidBody(String)` and `Internal(String)` flatten their cause into a string, against [messages and causes](messages-and-causes.md). A new variant carries the typed cause
## Testing
`status()` and `body()` are pure, so unit tests call them on errors built the way production builds them, one `#[case]` per fault, as `status_follows_who_is_at_fault` and `body_follows_the_anthropic_exception_mapping` in `crates/gateway-inference/src/error.rs` do. Assert the status and the fields the surface promises, not the whole body. Driving a whole router belongs to the gateway crate's own tests, see [`rust-gateway`](../../rust-gateway/references/testing.md)

20
litellm-rust/.agents/skills/rust-gateway/SKILL.md Normal file
View file

@ -0,0 +1,20 @@
---
name: rust-gateway
description: Add or change an HTTP route, extractor, response envelope, stream or router mount in the litellm-rust gateway crates (crates/gateway and crates/gateway-*). Use when adding an endpoint, parsing a request body, choosing an error envelope for an API surface, relaying SSE, or writing a router-level test
---
# Rust gateway
`crates/gateway` mounts endpoint routers and serves the listener. `crates/gateway-inference` owns the inference endpoints, `gateway-mcp`, `gateway-ui`, `gateway-management` and `gateway-auth` own theirs. Each crate's `AGENTS.md` says what belongs in it. Read the reference for the part you are touching
- [surfaces](references/surfaces.md)
- Each API surface mirrors the envelope the Python proxy returns there, so the shapes differ on purpose
- The error type string is derived from the status and provider bodies pass through bounded
- [requests](references/requests.md)
- Extractors reject with the crate's `Error`, handlers take the body as a `Result` when the error needs the request id
- Every alias of a path is registered together, a known but unserved path goes to one handler
- Streams relay through `litellm_host_http::Sse`, with the error frame rendered by the crate's `Error`
- [testing](references/testing.md)
- Build the router through `tests/support` and drive it with `oneshot`, assert the fields the surface promises
How an error is classified before it reaches a response is the [`rust-error`](../rust-error/SKILL.md) skill, in particular its [gateway](../rust-error/references/gateway.md) page. Axum's own conventions are in its [docs](https://docs.rs/axum/latest/axum/)

34
litellm-rust/.agents/skills/rust-gateway/references/requests.md Normal file
View file

@ -0,0 +1,34 @@
# Requests
## Extractors reject with the crate's Error
A custom extractor sets `type Rejection = Error`, so a bad body gets the surface's envelope rather than axum's plain-text rejection. `JsonObject` and `InferenceBody` in `crates/gateway-inference/src/request.rs` are the pattern
```rust
impl<S: Send + Sync> FromRequest<S> for JsonObject {
type Rejection = Error;
async fn from_request(request: Request, state: &S) -> Result<Self, Self::Rejection> {
let body = Bytes::from_request(request, state).await.map_err(|error| {
if error.status() == StatusCode::PAYLOAD_TOO_LARGE {
Error::BodyTooLarge
} else {
Error::InvalidBody(error.to_string())
}
})?;
object(&body).map(Self)
}
}
```
When the handler needs something from the request in the error body, such as the request id on `/v1/messages`, take the extractor as `Result<JsonObject, Error>` so the handler maps the rejection itself, as `messages::create` does. `Infallible` is the right rejection for an extractor that only reads an optional header, as `RequestId` does
## Paths
Every alias of a route is registered together in `crates/gateway-inference/src/lib.rs`, the bare path, its `/v1` form and the deployment-prefixed form, otherwise one alias silently behaves differently from the others. A path the gateway knows but does not serve, such as `/v1/completions`, is routed to `request::unsupported`, which returns `Error::Unsupported(path)` so the client gets a 501 in the surface's envelope rather than a bare 404
`crates/gateway` merges the mounted routers and adds the shared inbound middleware as `route_layer`s, so authentication runs only for matched routes. Endpoint paths and parsing stay in the mounted crate, see the two crates' `AGENTS.md`
## Streaming
Once the response headers are sent the status cannot change. A streaming handler builds `litellm_host_http::Sse::new(Json, |error| Bytes::from(Error::from(error).sse_frame()))`, where the closure renders a mid-stream failure as an `event: error` frame through the same `body()` a JSON failure would use. `litellm_host_http::Error<E>` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, and the crate's `From` maps the second to `Internal`

17
litellm-rust/.agents/skills/rust-gateway/references/surfaces.md Normal file
View file

@ -0,0 +1,17 @@
# Envelopes follow the API surface
Each surface mirrors what the Python proxy returns there, so the shapes differ on purpose
| Surface | Envelope | Where |
|---|---|---|
| `/v1/messages` | Anthropic, `{"type": "error", "error": {"type", "message"}, "request_id"}` | `Error::body()` in `crates/gateway-inference/src/error.rs` |
| OpenAI-shaped routes | `{"error": {"message", "type", "param", "code"}}` | `Error::openai_response()` in the same file |
| MCP | `{"detail": message}` | `crates/gateway-mcp/src/error.rs` |
| UI | status plus a message | `crates/gateway-ui/src/error.rs` |
| auth rejections | plain text | `crates/gateway-auth/src/error.rs` |
Two rules hold on every surface. The error `type` string is derived from the status by `error_type()`, so a new failure never names its own type. A provider body already in the surface's shape passes through untouched, anything else has its message extracted and re-enveloped, and the body was bounded by `truncate_error_body` when the error was built
`/v1/messages` adds the request id to the envelope when the provider did not set one, which is why its handler takes the body as `Result<JsonObject, Error>` and renders the rejection itself, see [requests](requests.md)
A new surface gets its own renderer next to the others, named after the shape it produces, and its status still comes from `Error::status()`. Picking the status is the [`rust-error`](../../rust-error/references/gateway.md) skill's concern, not the surface's

9
litellm-rust/.agents/skills/rust-gateway/references/testing.md Normal file
View file

@ -0,0 +1,9 @@
# Testing
Test HTTP contracts in the crate that owns the endpoint: status codes, forwarded headers, error envelopes and streaming behavior. Inference and provider behavior is tested in the `inference-<fmt>` and `llms` crates, and mounting and server lifecycle in `crates/gateway`
`crates/gateway-inference/tests/support/mod.rs` builds a `Router` with injected deployments, pools and settings through `app()` and its variants, and `post()` drives it with `tower::ServiceExt::oneshot`. `json()` reads the response body back. Use these instead of building a router or a client by hand in a test
Assert the status and the fields the surface promises, not the whole body. A test that pins a whole envelope breaks on every unrelated field, one that checks `status`, `error.type` and the presence of `request_id` catches the regressions that matter
`status()` and `body()` of the error type are pure, so a classification case belongs in the error module's unit tests, see the [`rust-error`](../../rust-error/references/gateway.md) skill. Router tests cover what only the full path shows, such as an extractor rejection arriving in the right envelope or an SSE error frame after the headers went out