scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-29 06:33:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 200

refactor(rust-error): map bridge and gateway-inference errors once and render them with Report

Browse source 
RustUpstreamError has one constructor that always sets headers, argument
checks raise typed errors mapped once, OCR asks upstream_response(), the
Responses stream error event comes from the error type, gateway-inference
loses its string variants, and auth faults classify through one
litellm_auth predicate on every route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Yujong Lee 2026-10-23 14:05:20 -07:00
parent b5b542a68a
commit b99aa6b687
57 changed files with 1158 additions and 649 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
litellm-rust/.agents/skills/rust-error/references/boundaries.md
View file

@ -23,7 +23,7 @@ else { 500 at the gateway, RuntimeError at the bridge }
So a new failure in a nested crate only needs the right predicate answers to land correctly on both. If you find yourself adding an arm for your variant at a boundary, check whether a predicate is missing first
The two boundaries are not identical. The gateway has variants of its own for body parsing, an unknown model and an unsupported path, and the OCR bridge has Python parity arms such as `FileNotFoundError` for a missing file. Those arms match the boundary's own variants or the top level of a public enum, never a nested error's variants. A missing API key is 401 at the gateway, `RuntimeError` at the inference bridge and `ValueError` at the OCR bridge, each pinned by a test to what Python's own path does
The two boundaries are not identical. The gateway has variants of its own for body parsing, an unknown model and an unsupported path, and the OCR bridge has Python parity arms such as `FileNotFoundError` for a missing file. Those arms match the boundary's own variants or the top level of a public enum, never a nested error's variants. A missing API key is 401 at the gateway and `RuntimeError` at both bridges, pinned by tests to what Python's own path does
## What reaches the client

4
litellm-rust/.agents/skills/rust-error/references/gateway.md
View file

@ -31,12 +31,12 @@ Do not reach into a nested enum's variants from the gateway. When a new nested f
## Render by surface and framing
Rendering is separate from classification. The surface picks the envelope and the moment picks the framing, a JSON body before the headers went out and an `event: error` SSE frame after. Both framings of one surface render the same envelope, so a streaming failure and a JSON failure of the same error agree on everything but framing
Rendering is separate from classification. The surface picks the envelope and the moment picks the framing, a JSON body before the headers went out and an `event: error` SSE frame after. Both framings of one surface render the same status, type and `Report` message, so a streaming failure and a JSON failure of the same error agree on everything but framing
| Surface | JSON | SSE |
|---|---|---|
| Anthropic, `/v1/messages` | `body(request_id)` | `sse_frame()` |
| OpenAI, chat, responses and the rest | `openai_response()` | `openai_sse_frame()` |
| OpenAI, chat, responses and the rest | `openai_response()` | `responses_sse_frame()`, the Responses API `error` event |
Two rules hold for both. The error `type` string is derived from the status by `error_type()`, so a new failure never names its own type, and the message of anything that is not a provider answer is the `litellm_error::Report` of the error. The Anthropic envelope passes a provider body already in its shape through untouched and extracts the message from anything else. The OpenAI envelope always wraps, because that is what the Python proxy returns there

4
litellm-rust/.agents/skills/rust-error/references/pyo3.md
View file

@ -22,7 +22,7 @@ The two native classes come from `create_exception!` in `crates/python-bridge/sr
Python decodes a `RustUpstreamError` in `litellm/rust_bridge/failures.py`, which validates `args` and `headers` with a `TypeAdapter` and rebuilds the provider's `httpx.Response`. `litellm/rust_bridge/runtime.py` catches `RustBridgeDeclined` and reads its reason. `_raise_upstream` in `runtime.py` still reads `args` a second time by hand for the `attempt` path, which is the one decode to fold into `failures.py`. A new fact goes into `args` or `headers`, and the decoder learns to read it in the same change
Two routes also flag a request fault with a boolean attribute, read by their Python route module: `ocr_request_format_error` in `host/routes/ocr.py` and `messages_request_error` in `host/routes/messages.py`. OCR additionally sets `status_code` and `message` for Python parity. Do not add another attribute, nothing on the Python side types them
Two routes also flag a request fault with a boolean attribute, read by their Python route module: `ocr_request_format_error` in `host/routes/ocr.py` and `messages_request_error` in `host/routes/messages.py`. OCR additionally sets `status_code` and `message` for Python parity, and `native_settings_error`, which only the OCR host reads back to tell a settings failure from the caller's exception. Do not add another attribute, nothing on the Python side types them
## Why args, not a typed subclass
@ -41,7 +41,7 @@ OCR puts Python parity arms in front of that order, each matching a top-level va
| `FileRead` whose cause is `NotFound` | `FileNotFoundError` naming the path |
| other `FileRead` | `OSError` with the io message |
OCR also treats every auth failure as the caller's, so a missing `litellm_auth` key is a `ValueError` there, while the inference routes raise `RuntimeError`
Both routes ask `litellm_auth::Error::is_request()` through their own `is_request()`, so a missing key or API base is a `RuntimeError` on both, and a missing setting the caller can pass, such as a region, is a `ValueError`
Use the standard class whose Python meaning matches. `PyFileNotFoundError` for a missing file is right, a `RuntimeError` whose text says the file is missing is not. `import_exception!` exists for an exception defined in Python, none is used today because `failures.py` maps to LiteLLM's public exceptions through `litellm.exception_type`

2
litellm-rust/.agents/skills/rust-gateway/references/requests.md
View file

@ -31,4 +31,4 @@ Every alias the Python proxy registers for a route is registered together in `cr
## Streaming
Once the response headers are sent the status cannot change. A streaming handler builds `litellm_host_http::Sse::new(Json, |error| Bytes::from(Error::from(error).sse_frame()))`, where the closure renders a mid-stream failure as an `event: error` frame through the same envelope a JSON failure on that surface uses, `sse_frame()` for Anthropic and `openai_sse_frame()` for OpenAI. `litellm_host_http::Error<E>` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, and the crate's `From` maps the second to `Internal`
Once the response headers are sent the status cannot change. A streaming handler builds `litellm_host_http::Sse::new(Json, |error| Bytes::from(Error::from(error).sse_frame()))`, where the closure renders a mid-stream failure as an `event: error` frame with the same status and message a JSON failure on that surface carries, `sse_frame()` for Anthropic and `responses_sse_frame()` for the Responses API's `error` event. `litellm_host_http::Error<E>` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, and the crate's `From` maps the second to `Internal`

3
litellm-rust/.agents/skills/rust-gateway/references/surfaces.md
View file

@ -5,7 +5,8 @@ Each surface mirrors what the Python proxy returns there, so the shapes differ o
| Surface | Envelope | Where |
|---|---|---|
| `/v1/messages` | Anthropic, `{"type": "error", "error": {"type", "message"}, "request_id"}` | `Error::body()` and `sse_frame()` in `crates/gateway-inference/src/error.rs` |
| OpenAI-shaped routes | `{"error": {"message", "type", "param", "code"}}` | `Error::openai_response()` and `openai_sse_frame()` in the same file |
| OpenAI-shaped routes | `{"error": {"message", "type", "param", "code"}}` | `Error::openai_response()` in the same file |
| `/v1/responses` stream | the Responses `error` event, `{"type": "error", "code", "message", "param"}` | `Error::responses_sse_frame()` in the same file |
| MCP | `{"detail": message}` | `crates/gateway-mcp/src/error.rs` |
| UI | `{"error": {"message"}}`, plus `Retry-After` on 429 | `crates/gateway-ui/src/error.rs` |
| auth rejections | plain text, the top-level message only | `crates/gateway-auth/src/error.rs` |

28
litellm-rust/crates/auth-types/src/error.rs
View file

@ -48,8 +48,6 @@ impl Error {
Self::Provider(Source::new(cause))
}
/// The environment lacks the credential the provider needs, as opposed to the caller
/// having sent a bad one.
pub fn is_missing_credential(&self) -> bool {
match self {
Self::MissingApiKey { .. } => true,
@ -67,22 +65,19 @@ impl Error {
}
}
/// A connection setting the caller can pass on the call, such as a region, a project or an
/// API base, is unset everywhere it is read from.
pub fn is_missing_setting(&self) -> bool {
pub fn is_request(&self) -> bool {
match self {
Self::MissingApiBase { .. } => true,
Self::MissingParam { spec, .. } => match spec.kind {
ParamKind::Setting => true,
ParamKind::Credential => false,
},
Self::InvalidConfiguration(_)
| Self::CredentialAcquisition(_)
| Self::EmptyCallerCredential { .. }
| Self::Provider(_)
| Self::CredentialChain(_)
| Self::MissingApiKey { .. }
| Self::InvalidHeader => false,
| Self::InvalidHeader => true,
Self::MissingParam { spec, .. } => match spec.kind {
ParamKind::Setting => true,
ParamKind::Credential => false,
},
Self::MissingApiKey { .. } | Self::MissingApiBase { .. } => false,
}
}
}
@ -273,14 +268,15 @@ mod tests {
#[rstest]
#[case::vertex_project(VertexParams::PROJECT.missing("Vertex AI"), true)]
#[case::aws_region(AwsParams::REGION.missing("AWS"), true)]
#[case::api_base(Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" }, true)]
#[case::api_base(Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" }, false)]
#[case::vertex_credentials(VertexParams::CREDENTIALS.missing("Vertex AI"), false)]
#[case::api_key(Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" }, false)]
#[case::invalid_configuration(Error::InvalidConfiguration(ErrorDetail::PlanNotAllowed), false)]
fn an_absent_connection_setting_is_a_missing_setting(
#[case::invalid_configuration(Error::InvalidConfiguration(ErrorDetail::PlanNotAllowed), true)]
#[case::invalid_header(Error::InvalidHeader, true)]
fn a_setting_the_caller_can_pass_is_theirs_to_fix_but_credentials_and_api_bases_are_not(
#[case] error: Error,
#[case] expected: bool,
) {
assert_eq!(error.is_missing_setting(), expected);
assert_eq!(error.is_request(), expected);
}
}

2
litellm-rust/crates/gateway-inference/src/audio_transcription.rs
View file

@ -27,7 +27,7 @@ async fn handle(
upload,
}: InferenceBody,
) -> Result<Value, Error> {
let deployment = request::resolve_deployment(gateway, &body)?;
let deployment = request::resolve_deployment(gateway, "/audio/transcriptions", &body)?;
request::authorize_model(identity, deployment, &body).await?;
let audio = match upload {
Some(upload) => {

20
litellm-rust/crates/gateway-inference/src/caching.rs
View file

@ -5,7 +5,7 @@ use litellm_gateway_auth::AuthenticatedRequest;
use serde::Deserialize;
use serde_json::{Map, Value};
use crate::Error;
use crate::BodyError;
#[derive(Default, Deserialize)]
#[serde(default, deny_unknown_fields)]
@ -24,10 +24,9 @@ type Prepared = (Map<String, Value>, CacheOptions);
pub(crate) fn prepare(
identity: &AuthenticatedRequest,
body: Map<String, Value>,
) -> Result<Prepared, Error> {
) -> Result<Prepared, BodyError> {
let controls: Controls = match body.get("cache").filter(|value| !value.is_null()) {
Some(value) => serde_json::from_value(value.clone())
.map_err(|error| Error::InvalidBody(error.to_string()))?,
Some(value) => serde_json::from_value(value.clone()).map_err(BodyError::CacheControls)?,
None => Controls::default(),
};
let caching: Option<bool> = body
@ -35,7 +34,7 @@ pub(crate) fn prepare(
.filter(|value| !value.is_null())
.map(|value| serde_json::from_value(value.clone()))
.transpose()
.map_err(|error| Error::InvalidBody(error.to_string()))?;
.map_err(BodyError::Caching)?;
let caller = identity.caller();
let options = CacheOptions {
policy: CachePolicy {
@ -62,11 +61,12 @@ pub(crate) fn prepare(
))
}
fn duration(seconds: f64) -> Result<Duration, Error> {
Duration::try_from_secs_f64(seconds)
.ok()
.filter(|duration| !duration.is_zero())
.ok_or_else(|| Error::InvalidBody("cache durations must be finite and positive".into()))
fn duration(seconds: f64) -> Result<Duration, BodyError> {
match Duration::try_from_secs_f64(seconds) {
Ok(duration) if !duration.is_zero() => Ok(duration),
Ok(_) => Err(BodyError::CacheDuration(None)),
Err(error) => Err(BodyError::CacheDuration(Some(error))),
}
}
#[derive(Clone, Default)]

2
litellm-rust/crates/gateway-inference/src/chat_completions.rs
View file

@ -40,7 +40,7 @@ async fn handle(
identity: &AuthenticatedRequest,
body: Map<String, Value>,
) -> Result<Response, Error> {
let deployment = request::resolve_deployment(gateway, &body)?;
let deployment = request::resolve_deployment(gateway, "/chat/completions", &body)?;
request::authorize_model(identity, deployment, &body).await?;
let (body, cache_options) = crate::caching::prepare(identity, body)?;
let route = gateway.chat_completions.clone();

202
litellm-rust/crates/gateway-inference/src/error.rs
View file

@ -11,12 +11,12 @@ use serde_json::{Map, Value, json};
pub enum Error {
#[error(transparent)]
Auth(#[from] litellm_gateway_auth::Error),
#[error("invalid request body: {0}")]
InvalidBody(String),
#[error("invalid request body")]
InvalidBody(#[from] BodyError),
#[error(
"/v1/messages: Invalid model name passed in model={0}. Call `/v1/models` to view available models for your key."
"{route}: Invalid model name passed in model={model}. Call `/v1/models` to view available models for your key."
)]
UnknownModel(String),
UnknownModel { route: &'static str, model: String },
#[error(transparent)]
Route(#[from] InferenceError),
#[error(transparent)]
@ -25,8 +25,30 @@ pub enum Error {
Unsupported(String),
#[error("request body exceeds the size limit")]
BodyTooLarge,
#[error("{0}")]
Internal(String),
#[error("unexpected HTTP host operation")]
Internal,
}
#[derive(Debug, thiserror::Error)]
pub enum BodyError {
#[error("could not read the body")]
Read(#[source] axum::extract::rejection::BytesRejection),
#[error("invalid JSON")]
Json(#[source] serde_json::Error),
#[error("expected a JSON object")]
NotAnObject,
#[error("invalid multipart body")]
Multipart(#[source] axum::extract::multipart::MultipartRejection),
#[error("invalid multipart field")]
MultipartField(#[source] axum::extract::multipart::MultipartError),
#[error("model is required")]
MissingModel,
#[error("invalid cache controls")]
CacheControls(#[source] serde_json::Error),
#[error("invalid caching flag")]
Caching(#[source] serde_json::Error),
#[error("cache durations must be finite and positive")]
CacheDuration(#[source] Option<std::time::TryFromFloatSecsError>),
}
impl IntoResponse for Error {
@ -39,7 +61,7 @@ impl From<litellm_host_http::Error<InferenceError>> for Error {
fn from(error: litellm_host_http::Error<InferenceError>) -> Self {
match error {
litellm_host_http::Error::Call(error) => Self::Route(error),
litellm_host_http::Error::Protocol => Self::Internal(error.to_string()),
litellm_host_http::Error::Protocol => Self::Internal,
}
}
}
@ -50,29 +72,37 @@ impl Error {
Self::Auth(error) => error.status(),
Self::Unsupported(_) => StatusCode::NOT_IMPLEMENTED,
Self::BodyTooLarge => StatusCode::PAYLOAD_TOO_LARGE,
Self::InvalidBody(_) | Self::UnknownModel(_) => StatusCode::BAD_REQUEST,
Self::Internal(_) => StatusCode::INTERNAL_SERVER_ERROR,
Self::InvalidBody(_) | Self::UnknownModel { .. } => StatusCode::BAD_REQUEST,
Self::Internal => StatusCode::INTERNAL_SERVER_ERROR,
Self::Ocr(error) => ocr_status(error),
Self::Route(error) => route_status(error),
}
}
pub fn openai_response(self) -> Response {
(self.status(), Json(self.openai_body())).into_response()
}
/// The OpenAI envelope as an `event: error` frame, for a stream that fails after its
/// headers went out.
pub fn responses_sse_frame(&self) -> String {
let event = json!({
"type": "error",
"code": self.status().as_u16().to_string(),
"message": litellm_error::Report::new(self).to_string(),
"param": null,
});
format!("event: error\ndata: {event}\n\n")
}
fn openai_body(&self) -> Value {
let status = self.status();
let message = match &self {
Self::UnknownModel(model) => format!("Invalid model name passed in model={model}"),
_ => litellm_error::Report::new(&self).to_string(),
};
(
status,
Json(json!({"error": {
"message": message,
"type": error_type(status),
"param": null,
"code": status.as_u16(),
}})),
)
.into_response()
json!({"error": {
"message": litellm_error::Report::new(self).to_string(),
"type": error_type(status),
"param": null,
"code": status.as_u16(),
}})
}
/// The Anthropic error envelope Python's `AnthropicExceptionMapping` builds: an upstream
@ -121,6 +151,9 @@ fn ocr_status(error: &OcrError) -> StatusCode {
if error.is_unsupported() {
return StatusCode::NOT_IMPLEMENTED;
}
if let Some((status, ..)) = error.upstream_response() {
return StatusCode::from_u16(status).unwrap_or(StatusCode::BAD_GATEWAY);
}
if error.is_missing_credential() {
return StatusCode::UNAUTHORIZED;
}
@ -191,8 +224,19 @@ mod tests {
Error::Route(litellm_http::Error::upstream(status, body.into()).into())
}
fn invalid_json() -> Error {
BodyError::Json(serde_json::from_str::<Value>("{").unwrap_err()).into()
}
fn unknown_model() -> Error {
Error::UnknownModel {
route: "/v1/messages",
model: "nope".into(),
}
}
#[rstest]
#[case::anthropic_body_passes_through(
#[case::anthropic_body(
upstream(529, r#"{"type":"error","error":{"type":"overloaded_error","message":"busy","extra":1}}"#),
Some("req_1"),
json!({"type": "error", "error": {"type": "overloaded_error", "message": "busy", "extra": 1}, "request_id": "req_1"}),
@ -202,30 +246,7 @@ mod tests {
Some("caller"),
json!({"type": "error", "error": {"type": "x", "message": "m"}, "request_id": "upstream"}),
)]
#[case::bedrock_detail(
upstream(403, r#"{"detail":{"message":"denied"}}"#),
None,
json!({"type": "error", "error": {"type": "permission_error", "message": "denied"}}),
)]
#[case::aws_message(
upstream(429, r#"{"Message":"slow down"}"#),
None,
json!({"type": "error", "error": {"type": "rate_limit_error", "message": "slow down"}}),
)]
#[case::plain_text_with_unmapped_status(
upstream(502, "bad gateway"),
None,
json!({"type": "error", "error": {"type": "api_error", "message": "bad gateway"}}),
)]
#[case::unknown_model(
Error::UnknownModel("nope".into()),
None,
json!({"type": "error", "error": {
"type": "invalid_request_error",
"message": "/v1/messages: Invalid model name passed in model=nope. Call `/v1/models` to view available models for your key.",
}}),
)]
fn body_follows_the_anthropic_exception_mapping(
fn an_anthropic_provider_body_passes_through(
#[case] error: Error,
#[case] request_id: Option<&str>,
#[case] expected: Value,
@ -233,6 +254,39 @@ mod tests {
assert_eq!(error.body(request_id), expected);
}
#[rstest]
#[case::bedrock_detail(
upstream(403, r#"{"detail":{"message":"denied"}}"#),
"permission_error",
"denied"
)]
#[case::aws_message(
upstream(429, r#"{"Message":"slow down"}"#),
"rate_limit_error",
"slow down"
)]
#[case::plain_text_with_unmapped_status(
upstream(502, "bad gateway"),
"api_error",
"bad gateway"
)]
#[case::unknown_model(
unknown_model(),
"invalid_request_error",
"/v1/messages: Invalid model name passed in model=nope. Call `/v1/models` to view available models for your key."
)]
fn other_failures_get_the_anthropic_envelope_with_their_message(
#[case] error: Error,
#[case] error_type: &str,
#[case] message: &str,
) {
let body = error.body(Some("req_1"));
assert_eq!(body["type"], "error");
assert_eq!(body["error"]["type"], error_type);
assert_eq!(body["error"]["message"], message);
assert_eq!(body["request_id"], "req_1");
}
#[rstest]
#[case::upstream_status(upstream(429, ""), StatusCode::TOO_MANY_REQUESTS)]
#[case::rejected_request(
@ -256,7 +310,59 @@ mod tests {
),
StatusCode::INTERNAL_SERVER_ERROR,
)]
#[case::invalid_body(invalid_json(), StatusCode::BAD_REQUEST)]
#[case::missing_model(BodyError::MissingModel.into(), StatusCode::BAD_REQUEST)]
#[case::unknown_model(unknown_model(), StatusCode::BAD_REQUEST)]
#[case::oversized_body(Error::BodyTooLarge, StatusCode::PAYLOAD_TOO_LARGE)]
#[case::unserved_path(Error::Unsupported("/v1/completions".into()), StatusCode::NOT_IMPLEMENTED)]
#[case::host_protocol(
litellm_host_http::Error::<InferenceError>::Protocol.into(),
StatusCode::INTERNAL_SERVER_ERROR
)]
#[case::ocr_provider_answer(
Error::Ocr(OcrError::Provider { status: 429, body: "slow".into(), headers: vec![] }),
StatusCode::TOO_MANY_REQUESTS
)]
#[case::ocr_missing_credential(
Error::Ocr(OcrError::MissingCredential { credential: "Azure AI credentials", guidance: "set a key" }),
StatusCode::UNAUTHORIZED
)]
#[case::ocr_request_fault(Error::Ocr(OcrError::MissingDocumentUrl), StatusCode::BAD_REQUEST)]
#[case::ocr_response_fault(
Error::Ocr(OcrError::EmptyContent),
StatusCode::INTERNAL_SERVER_ERROR
)]
fn status_follows_who_is_at_fault(#[case] error: Error, #[case] status: StatusCode) {
assert_eq!(error.status(), status);
}
#[test]
fn a_body_fault_names_the_body_and_keeps_its_cause_once() {
let cause = serde_json::from_str::<Value>("{").unwrap_err().to_string();
let body = invalid_json().openai_body();
let message = body["error"]["message"].as_str().unwrap();
assert!(
message.starts_with("invalid request body: invalid JSON: "),
"{message}"
);
assert_eq!(message.matches(&cause).count(), 1, "{message}");
assert_eq!(body["error"]["type"], "invalid_request_error");
assert_eq!(body["error"]["code"], 400);
}
#[rstest]
#[case::provider_answer(upstream(429, "slow down"))]
#[case::caller_fault(invalid_json())]
#[case::host_bug(litellm_host_http::Error::<InferenceError>::Protocol.into())]
fn the_responses_sse_frame_is_the_responses_error_event(#[case] error: Error) {
let frame = error.responses_sse_frame();
let data = frame
.strip_prefix("event: error\ndata: ")
.and_then(|data| data.strip_suffix("\n\n"))
.unwrap_or_else(|| panic!("an event: error frame, got {frame:?}"));
let event = serde_json::from_str::<Value>(data).unwrap();
assert_eq!(event["type"], "error");
assert_eq!(event["code"], error.status().as_u16().to_string());
assert_eq!(event["message"], error.openai_body()["error"]["message"]);
}
}

2
litellm-rust/crates/gateway-inference/src/lib.rs
View file

@ -25,7 +25,7 @@ use litellm_inference_transcription::AudioTranscriptionRoute;
use litellm_llms::base_llm::ocr::{handler::OcrClient, settings::OcrSettings};
use litellm_secrets::source::SecretSource;
pub use error::Error;
pub use error::{BodyError, Error};
pub use litellm_router::{Deployment, Router as ModelRouter};
pub use request::{JsonObject, RequestId};

2
litellm-rust/crates/gateway-inference/src/messages.rs
View file

@ -46,7 +46,7 @@ async fn handle(
headers: &HeaderMap,
body: Map<String, Value>,
) -> Result<Response, Error> {
let deployment = request::resolve_deployment(gateway, &body)?;
let deployment = request::resolve_deployment(gateway, "/v1/messages", &body)?;
request::authorize_model(identity, deployment, &body).await?;
let (body, cache_options) = crate::caching::prepare(identity, body)?;
let route = gateway.messages.clone();

2
litellm-rust/crates/gateway-inference/src/ocr.rs
View file

@ -35,7 +35,7 @@ async fn handle(
.get("x-req-format")
.and_then(|value| value.to_str().ok())
.map(str::to_owned);
let deployment = request::resolve_deployment(gateway, &body)?;
let deployment = request::resolve_deployment(gateway, "/ocr", &body)?;
request::authorize_model(identity, deployment, &body).await?;
let document = match upload {
Some(upload) => OcrDocumentInput::Bytes {

26
litellm-rust/crates/gateway-inference/src/request.rs
View file

@ -6,7 +6,7 @@ use axum::{
};
use serde_json::{Map, Value};
use crate::{Deployment, Error, Gateway};
use crate::{BodyError, Deployment, Error, Gateway};
pub(crate) const MAX_FILE_BYTES: usize = 50 * 1024 * 1024;
pub(crate) const MAX_BODY_BYTES: usize = MAX_FILE_BYTES + 1024 * 1024;
@ -43,10 +43,10 @@ impl<S: Send + Sync> FromRequest<S> for JsonObject {
if error.status() == axum::http::StatusCode::PAYLOAD_TOO_LARGE {
Error::BodyTooLarge
} else {
Error::InvalidBody(error.to_string())
BodyError::Read(error).into()
}
})?;
object(&body).map(Self)
Ok(Self(object(&body)?))
}
}
@ -77,16 +77,16 @@ impl<S: Send + Sync> FromRequest<S> for InferenceBody {
}
let multipart = Multipart::from_request(request, state)
.await
.map_err(|error| Error::InvalidBody(error.to_string()))?;
.map_err(BodyError::Multipart)?;
parse_multipart(multipart).await
}
}
fn object(body: &[u8]) -> Result<Map<String, Value>, Error> {
fn object(body: &[u8]) -> Result<Map<String, Value>, BodyError> {
match serde_json::from_slice(body) {
Ok(Value::Object(body)) => Ok(body),
Ok(_) => Err(Error::InvalidBody("expected a JSON object".into())),
Err(error) => Err(Error::InvalidBody(error.to_string())),
Ok(_) => Err(BodyError::NotAnObject),
Err(error) => Err(BodyError::Json(error)),
}
}
@ -98,7 +98,7 @@ pub(crate) async fn authorize_model(
let name = body
.get("model")
.and_then(Value::as_str)
.ok_or_else(|| Error::InvalidBody("model is required".into()))?;
.ok_or(BodyError::MissingModel)?;
let access = litellm_gateway_auth::AccessRequest::Model {
name: name.into(),
deployment: deployment.model.clone(),
@ -112,16 +112,20 @@ pub(crate) async fn authorize_model(
pub(crate) fn resolve_deployment<'a>(
gateway: &'a Gateway,
route: &'static str,
body: &Map<String, Value>,
) -> Result<&'a Deployment, Error> {
let model = body
.get("model")
.and_then(Value::as_str)
.ok_or_else(|| Error::InvalidBody("model is required".into()))?;
.ok_or(BodyError::MissingModel)?;
gateway
.models
.get(model)
.ok_or_else(|| Error::UnknownModel(model.to_owned()))
.ok_or_else(|| Error::UnknownModel {
route,
model: model.to_owned(),
})
}
async fn parse_multipart(mut multipart: Multipart) -> Result<InferenceBody, Error> {
@ -159,7 +163,7 @@ fn multipart_error(error: axum::extract::multipart::MultipartError) -> Error {
if error.status() == axum::http::StatusCode::PAYLOAD_TOO_LARGE {
return Error::BodyTooLarge;
}
Error::InvalidBody(error.to_string())
BodyError::MultipartField(error).into()
}
pub(crate) async fn unsupported(OriginalUri(uri): OriginalUri) -> impl IntoResponse {

9
litellm-rust/crates/gateway-inference/src/responses.rs
View file

@ -4,7 +4,6 @@ use axum::{Json, body::Bytes, extract::State, response::Response};
use litellm_gateway_auth::AuthenticatedRequest;
use litellm_host_http::Sse;
use litellm_inference_responses::{route::Responses, types::ResponsesCall};
use serde_json::json;
use crate::{Error, Gateway, JsonObject, request};
@ -13,7 +12,7 @@ pub(crate) async fn create(
identity: AuthenticatedRequest,
JsonObject(body): JsonObject,
) -> Result<Response, Error> {
let deployment = request::resolve_deployment(&gateway, &body)?;
let deployment = request::resolve_deployment(&gateway, "/responses", &body)?;
request::authorize_model(&identity, deployment, &body).await?;
let (body, cache_options) = crate::caching::prepare(&identity, body)?;
let route = gateway.responses.clone();
@ -40,11 +39,7 @@ pub(crate) async fn create(
};
let machine = route.machine(call, cache_options.policy);
let stream = Sse::<Responses, _, _>::new(Json, |error| {
let error = Error::from(error);
Bytes::from(format!(
"event: error\ndata: {}\n\n",
json!({"type": "error", "code": error.status().as_u16().to_string(), "message": error.to_string(), "param": null})
))
Bytes::from(Error::from(error).responses_sse_frame())
});
let headers = crate::caching::CacheHeaders::default();
let response = litellm_host_http::serve(machine, (), headers.clone(), stream, None).await?;

26
litellm-rust/crates/gateway-inference/tests/messages.rs
View file

@ -6,7 +6,6 @@ use axum::{
};
use rstest::rstest;
use serde_json::json;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tower::ServiceExt;
use wiremock::{
Mock, MockServer, ResponseTemplate,
@ -90,34 +89,11 @@ async fn invalid_messages_stays_an_anthropic_error() {
assert_eq!(body["error"]["type"], "invalid_request_error");
}
/// Answers with the SSE head and one event, then drops the connection short of the
/// announced body length.
async fn truncating_upstream() -> String {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let base = format!("http://{}", listener.local_addr().unwrap());
tokio::spawn(async move {
let (mut socket, _) = listener.accept().await.unwrap();
let mut request = vec![0; 4096];
let _ = socket.read(&mut request).await;
socket
.write_all(
format!(
"HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\ncontent-length: {}\r\n\r\n{FIRST_EVENT}",
FIRST_EVENT.len() * 2
)
.as_bytes(),
)
.await
.unwrap();
});
base
}
const FIRST_EVENT: &str = "event: message_start\ndata: {}\n\n";
#[tokio::test]
async fn a_stream_that_fails_after_opening_ends_with_an_sse_error_frame() {
let base = truncating_upstream().await;
let base = support::truncating_upstream(FIRST_EVENT).await;
let request = Request::post("/v1/messages")
.header("content-type", "application/json")
.body(Body::from(

29
litellm-rust/crates/gateway-inference/tests/responses.rs
View file

@ -116,3 +116,32 @@ async fn responses_authorize_the_public_model_before_provider_execution() {
assert_eq!(response.status(), 403);
assert!(upstream.received_requests().await.unwrap().is_empty());
}
const RESPONSE_CREATED: &str = "event: response.created\ndata: {\"type\":\"response.created\",\"response\":{\"id\":\"response-1\",\"model\":\"test-model\",\"output\":[]}}\n\n";
#[rstest]
#[tokio::test]
async fn a_stream_that_fails_after_opening_ends_with_the_responses_error_event() {
let base = support::truncating_upstream(RESPONSE_CREATED).await;
let response = support::post(
support::app("openai/test-model", &base),
"/v1/responses",
json!({"model": "public/model", "input": "hello", "stream": true}),
)
.await;
assert_eq!(response.status(), 200);
let body = to_bytes(response.into_body(), 4096).await.unwrap();
let text = std::str::from_utf8(&body).unwrap();
let frame = text
.strip_prefix(RESPONSE_CREATED)
.and_then(|rest| rest.strip_prefix("event: error\ndata: "))
.unwrap_or_else(|| panic!("the delivered event then one error frame, got {text:?}"));
let error: serde_json::Value = serde_json::from_str(frame.trim_end()).unwrap();
assert_eq!(error["type"], "error");
assert_eq!(error["code"], "500");
assert!(
error["message"]
.as_str()
.is_some_and(|message| !message.is_empty())
);
}

24
litellm-rust/crates/gateway-inference/tests/support/mod.rs
View file

@ -15,6 +15,7 @@ use litellm_http::{HttpClientPool, HttpSettings, Resolution, media::PublicDnsRes
use litellm_inference::resources::CoreResources;
use litellm_secrets::{SecretValue, source::SecretSource};
use serde_json::Value;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tower::ServiceExt;
pub struct NoSecrets;
@ -166,3 +167,26 @@ impl litellm_gateway_auth::IdentityResolver for TestPermissions {
})
}
}
/// Answers with the SSE head and one event, then drops the connection short of the
/// announced body length.
pub async fn truncating_upstream(first_event: &'static str) -> String {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let base = format!("http://{}", listener.local_addr().unwrap());
tokio::spawn(async move {
let (mut socket, _) = listener.accept().await.unwrap();
let mut request = vec![0; 4096];
let _ = socket.read(&mut request).await;
socket
.write_all(
format!(
"HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\ncontent-length: {}\r\n\r\n{first_event}",
first_event.len() * 2
)
.as_bytes(),
)
.await
.unwrap();
});
base
}

3
litellm-rust/crates/gateway-mcp/tests/gateway/native.rs
View file

@ -152,8 +152,7 @@ async fn routes_prompts_and_resources_without_rewriting_uris(#[future] harness:
.gateway
.execute(read.clone(), context(None))
.await
.err()
.expect("read without a server scope");
.expect_err("read without a server scope");
assert_eq!(ambiguous.status(), axum::http::StatusCode::BAD_REQUEST);
let result = harness
.gateway

5
litellm-rust/crates/gateway/tests/server.rs
View file

@ -94,7 +94,10 @@ async fn authenticates_before_serving_mounted_inference_routes(
let response = request.send().await.unwrap();
assert_eq!(response.status().as_u16(), status);
if status == 400 {
let expected = Error::UnknownModel("unconfigured-model".into());
let expected = Error::UnknownModel {
route: "/v1/messages",
model: "unconfigured-model".into(),
};
assert_eq!(
response.json::<Value>().await.unwrap(),
expected.body(Some("gateway-test"))

5
litellm-rust/crates/inference-chat/src/common_utils.rs
View file

@ -57,9 +57,8 @@ pub(super) fn resolve_request(
})?;
if messages.is_empty() {
return Err(Error::invalid_request(
litellm_llms::ErrorDetail::Expected {
subject: "chat completions messages",
expected: "non-empty",
litellm_llms::ErrorDetail::NoMessages {
route: "chat completions",
},
));
}

2
litellm-rust/crates/inference-chat/tests/request.rs
View file

@ -250,7 +250,7 @@ async fn rejects_a_model_with_no_resolvable_provider() {
}
#[rstest]
#[case::empty(json!([]), "chat completions messages must be non-empty")]
#[case::empty(json!([]), "chat completions requires at least one message")]
#[case::malformed(json!("not a list"), "invalid chat completions messages")]
#[tokio::test]
async fn rejects_an_empty_or_malformed_message_list(#[case] messages: Value, #[case] reason: &str) {

48
litellm-rust/crates/inference/src/error.rs
View file

@ -55,7 +55,7 @@ impl Error {
| Self::Unsupported(_)
| Self::Headers(_) => true,
Self::Llm(error) => error.is_request(),
Self::Auth(error) => !error.is_missing_credential(),
Self::Auth(error) => error.is_request(),
Self::Http(error) => error.is_request(),
Self::Secret(_) | Self::Host(_) | Self::PostCallHook(_) => false,
}
@ -120,6 +120,21 @@ impl Error {
}
}
pub fn rejected_request(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
Self::Llm(error) => error.rejected_request(),
Self::InvalidProvider(_)
| Self::UnresolvedProvider { .. }
| Self::Unsupported(_)
| Self::Auth(_)
| Self::Headers(_)
| Self::Http(_)
| Self::Secret(_)
| Self::Host(_)
| Self::PostCallHook(_) => None,
}
}
pub fn upstream_response(&self) -> Option<(u16, &str)> {
match self {
Self::Llm(error) => error.upstream_response(),
@ -277,6 +292,37 @@ mod tests {
assert!(error.dispatched());
}
#[rstest]
#[case::invalid_request(
Error::invalid_request(ErrorDetail::Expected {
subject: "messages",
expected: "non-empty",
}),
Some("messages must be non-empty")
)]
#[case::missing_field(
LlmError::MissingField("max_tokens").into(),
Some("missing required field: max_tokens")
)]
#[case::invalid_response(
Error::invalid_response(ErrorDetail::MissingField {
subject: "response",
field: "usage",
}),
None
)]
#[case::provider_rejection(litellm_http::Error::upstream(400, "bad".into()).into(), None)]
#[case::unresolved_provider(Error::InvalidProvider("unknown".into()), None)]
fn a_rejected_request_names_only_what_the_caller_got_wrong(
#[case] error: Error,
#[case] expected: Option<&str>,
) {
let rejection = error
.rejected_request()
.map(|rejection| litellm_error::Report::new(rejection).to_string());
assert_eq!(rejection.as_deref(), expected);
}
#[test]
fn a_post_call_hook_failure_is_not_the_providers_answer() {
let error =

3
litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs
View file

@ -503,8 +503,7 @@ async fn poll_operation(
let response =
tokio::time::timeout_at(deadline, litellm_http::request::http_request(builder))
.await
.map_err(|_| Error::PollTimeout)?
.map_err(litellm_http::Error::from)?;
.map_err(|_| Error::PollTimeout)??;
let retry = response
.headers()
.get(reqwest::header::RETRY_AFTER)

6
litellm-rust/crates/llms/src/base_llm/ocr/error.rs
View file

@ -3,6 +3,8 @@ use std::{io, path::PathBuf, sync::Arc};
use litellm_core_utils::call_arguments::ArgumentError;
use litellm_error::Source;
pub type Headers = [(String, String)];
#[derive(Clone, Debug, thiserror::Error)]
pub enum Error {
#[error("upstream OCR error ({status}): {body}")]
@ -144,7 +146,7 @@ impl Error {
}
}
pub fn upstream_response(&self) -> Option<(u16, &str, &[(String, String)])> {
pub fn upstream_response(&self) -> Option<(u16, &str, &Headers)> {
match self {
Self::Provider {
status,
@ -301,7 +303,7 @@ impl Error {
| Self::InvalidModel { .. }
| Self::Params(_)
| Self::Headers(_) => true,
Self::Auth(error) => !error.is_missing_credential(),
Self::Auth(error) => error.is_request(),
Self::Http(error) => error.is_request(),
Self::Provider { .. }
| Self::FileRead { .. }

14
litellm-rust/crates/llms/src/error.rs
View file

@ -28,7 +28,7 @@ impl Error {
| Self::MissingField(_)
| Self::InvalidRequest(_)
| Self::Unsupported(_) => true,
Self::Auth(error) => !error.is_missing_credential(),
Self::Auth(error) => error.is_request(),
Self::Http(error) => error.is_request(),
Self::InvalidResponse(_) => false,
}
@ -72,6 +72,16 @@ impl Error {
}
}
/// Why the transformation rejected the caller's request, rendered as Python's own
/// validation renders it, without this layer's "invalid request" prefix.
pub fn rejected_request(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
Self::InvalidRequest(detail) => Some(detail),
Self::InvalidType { .. } | Self::MissingField(_) => Some(self),
Self::InvalidResponse(_) | Self::Unsupported(_) | Self::Auth(_) | Self::Http(_) => None,
}
}
pub fn upstream_response(&self) -> Option<(u16, &str)> {
match self {
Self::Http(error) => error.upstream_response(),
@ -99,6 +109,8 @@ pub enum ErrorDetail {
#[source]
cause: Source,
},
#[error("{route} requires at least one message")]
NoMessages { route: &'static str },
#[error("{subject} must be {expected}")]
Expected {
subject: &'static str,

4
litellm-rust/crates/llms/src/reducto/ocr/transformation.rs
View file

@ -539,9 +539,7 @@ async fn upload_bytes_async(
headers,
litellm_http::request::HeaderPolicy::Except(&["content-type", "content-length"]),
);
let response = litellm_http::request::http_request(builder)
.await
.map_err(litellm_http::Error::from)?;
let response = litellm_http::request::http_request(builder).await?;
let uploaded = crate::base_llm::ocr::handler::read_json_response::<ReductoUploadResponse>(
response,
false,

40
litellm-rust/crates/python-bridge/src/cache/mod.rs vendored
View file

@ -12,6 +12,46 @@ use pyo3::{
prelude::*,
};
/// A cache call the bridge rejects itself, before or after the cache ran.
#[derive(Debug, thiserror::Error)]
pub(crate) enum HandleError {
#[error("recreate the v2 cache after fork")]
Forked,
#[error("cache limits must be positive")]
Limits,
#[error("namespace and a positive cache limit are required")]
RedisLimits,
#[error("cache durations must be finite and positive")]
Duration(#[source] Option<std::time::TryFromFloatSecsError>),
#[error("unsupported v2 cache control: {0}")]
Control(String),
#[error("native inference needs a v2 cache, the catalog keeps Python caches on Python")]
NotNative,
#[error("cache operation without configured cache")]
Unconfigured,
#[error("cache arguments unavailable")]
MissingArguments,
#[error("cache reply without pending operation")]
NothingPending,
}
impl From<HandleError> for PyErr {
fn from(error: HandleError) -> Self {
let message = litellm_error::Report::new(&error).to_string();
match error {
HandleError::Limits
| HandleError::RedisLimits
| HandleError::Duration(_)
| HandleError::Control(_) => PyValueError::new_err(message),
HandleError::Forked
| HandleError::NotNative
| HandleError::Unconfigured
| HandleError::MissingArguments
| HandleError::NothingPending => PyRuntimeError::new_err(message),
}
}
}
fn cache_error(error: Error) -> PyErr {
let message = litellm_error::Report::new(&error).to_string();
match error {

37
litellm-rust/crates/python-bridge/src/cache/native/v2.rs vendored
View file

@ -1,4 +1,4 @@
use crate::cache::cache_error;
use crate::cache::{HandleError, cache_error};
use std::{
sync::Arc,
time::{Duration, SystemTime, UNIX_EPOCH},
@ -14,7 +14,7 @@ use litellm_cache_response::{
CacheEntry, CacheKeyInput, ExactResponseCache, ResponseCache, ResponseCacheCodec,
ResponseCacheConfig, ResponseCacheRequest, ResponseCacheService,
};
use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict};
use pyo3::{prelude::*, types::PyDict};
#[pyclass(
frozen,
@ -37,9 +37,7 @@ enum Storage {
impl NativeCacheHandle {
fn check_process(&self) -> PyResult<()> {
if self.pid != std::process::id() {
return Err(pyo3::exceptions::PyRuntimeError::new_err(
"recreate the v2 cache after fork",
));
return Err(HandleError::Forked.into());
}
Ok(())
}
@ -61,7 +59,7 @@ impl NativeCacheHandle {
fn memory(ttl: f64, capacity: usize, max_entry_bytes: usize) -> PyResult<Self> {
let ttl = duration(ttl)?;
if capacity == 0 || max_entry_bytes == 0 {
return Err(PyValueError::new_err("cache limits must be positive"));
return Err(HandleError::Limits.into());
}
let storage = Arc::new(InMemoryCache::new(Some(capacity), Some(ttl)));
let backend = Arc::new(ResponseCache::new(storage.clone()).with_config(
@ -89,9 +87,7 @@ impl NativeCacheHandle {
) -> PyResult<Self> {
let ttl = duration(ttl)?;
if namespace.is_empty() || max_entry_bytes == 0 {
return Err(PyValueError::new_err(
"namespace and a positive cache limit are required",
));
return Err(HandleError::RedisLimits.into());
}
let storage = Arc::new(
release_gil(py, || {
@ -252,11 +248,12 @@ impl NativeCacheHandle {
}
}
fn duration(seconds: f64) -> PyResult<Duration> {
Duration::try_from_secs_f64(seconds)
.ok()
.filter(|value| !value.is_zero())
.ok_or_else(|| PyValueError::new_err("cache durations must be finite and positive"))
fn duration(seconds: f64) -> Result<Duration, HandleError> {
match Duration::try_from_secs_f64(seconds) {
Ok(duration) if !duration.is_zero() => Ok(duration),
Ok(_) => Err(HandleError::Duration(None)),
Err(error) => Err(HandleError::Duration(Some(error))),
}
}
pub(in crate::cache) fn native_handle<'py>(
@ -277,11 +274,7 @@ pub(in crate::cache) fn configured(
Option<Arc<dyn ResponseCacheService>>,
litellm_cache_response::CacheOptions,
)> {
let handle = native_handle(configured)?.ok_or_else(|| {
pyo3::exceptions::PyRuntimeError::new_err(
"native inference needs a v2 cache, the catalog keeps Python caches on Python",
)
})?;
let handle = native_handle(configured)?.ok_or(HandleError::NotNative)?;
let cache = handle.extract::<PyRef<'_, NativeCacheHandle>>()?;
cache.check_process()?;
let controls = kwargs.get_item("cache")?.filter(|value| !value.is_none());
@ -296,9 +289,7 @@ pub(in crate::cache) fn configured(
name.as_str(),
"no-cache" | "no-store" | "ttl" | "s-maxage" | "s-max-age" | "use-cache"
) {
return Err(PyValueError::new_err(format!(
"unsupported v2 cache control: {name}"
)));
return Err(HandleError::Control(name).into());
}
}
}
@ -316,7 +307,7 @@ pub(in crate::cache) fn configured(
.map(|values| values.get_item(name))
.transpose()?
.flatten()
.map(|value| duration(value.extract()?))
.map(|value| Ok(duration(value.extract()?)?))
.transpose()
};
Ok((

28
litellm-rust/crates/python-bridge/src/cache/python/host.rs vendored
View file

@ -9,6 +9,7 @@ use pyo3::{
use serde_json::Value;
use super::service::CacheCall;
use crate::cache::HandleError;
enum Pending {
Lookup(Reply<Result<Option<Value>, Error>>),
@ -43,16 +44,12 @@ impl PythonCache {
pub fn begin(&mut self, py: Python<'_>, call: CacheCall) -> PyResult<Option<Py<PyAny>>> {
let Some(cache) = self.cache.as_ref() else {
return Err(pyo3::exceptions::PyRuntimeError::new_err(
"cache operation without configured cache",
));
return Err(HandleError::Unconfigured.into());
};
let arguments = self
.arguments
.as_ref()
.ok_or_else(|| {
pyo3::exceptions::PyRuntimeError::new_err("cache arguments unavailable")
})?
.ok_or(HandleError::MissingArguments)?
.bind(py)
.copy()?;
let (method, result) = match call {
@ -97,12 +94,13 @@ impl PythonCache {
py: Python<'_>,
result: PyResult<Py<PyAny>>,
) -> PyResult<Option<Py<PyAny>>> {
if let Err(error) = &result
&& !error.is_instance_of::<pyo3::exceptions::PyException>(py)
{
self.pending = None;
return Err(result.err().unwrap());
}
let result = match result {
Err(error) if !error.is_instance_of::<pyo3::exceptions::PyException>(py) => {
self.pending = None;
return Err(error);
}
result => result,
};
match self.pending.take() {
Some(Pending::Lookup(reply)) => {
let value = result.map_err(Error::unavailable).and_then(|value| {
@ -119,11 +117,7 @@ impl PythonCache {
Some(Pending::Store(reply)) => {
reply.send(result.map(|_| ()).map_err(Error::unavailable));
}
None => {
return Err(pyo3::exceptions::PyRuntimeError::new_err(
"cache reply without pending operation",
));
}
None => return Err(HandleError::NothingPending.into()),
}
Ok(None)
}

63
litellm-rust/crates/python-bridge/src/coercion.rs
View file

@ -1,4 +1,5 @@
use litellm_core_utils::serde_compat::parse_str_bool;
use litellm_error::Report;
use pyo3::{
exceptions::{PyAttributeError, PyRuntimeError, PyValueError},
prelude::*,
@ -9,15 +10,20 @@ use pyo3::{
pub(crate) enum ProjectionError {
Python(PyErr),
InvalidConfiguration(ProjectionDetail),
UnsupportedLiveObject(String),
UnsupportedLiveObject(ProjectionDetail),
InternalSchemaFailure(ProjectionDetail),
}
#[derive(Debug, thiserror::Error)]
pub(crate) enum ProjectionDetail {
#[error("{0}")]
Message(String),
#[error("{field}: {source}")]
#[error("{group}.{name}: expected {expected}, got {actual}")]
Expected {
group: &'static str,
name: &'static str,
expected: &'static str,
actual: String,
},
#[error("{field}")]
InvalidField {
field: &'static str,
#[source]
@ -30,12 +36,6 @@ pub(crate) enum ProjectionDetail {
},
}
impl From<String> for ProjectionDetail {
fn from(message: String) -> Self {
Self::Message(message)
}
}
impl From<PyErr> for ProjectionError {
fn from(error: PyErr) -> Self {
Self::Python(error)
@ -46,12 +46,12 @@ impl From<ProjectionError> for PyErr {
fn from(error: ProjectionError) -> Self {
match error {
ProjectionError::Python(error) => error,
ProjectionError::InvalidConfiguration(detail) => {
PyValueError::new_err(detail.to_string())
ProjectionError::InvalidConfiguration(detail)
| ProjectionError::UnsupportedLiveObject(detail) => {
PyValueError::new_err(Report::new(&detail).to_string())
}
ProjectionError::UnsupportedLiveObject(message) => PyValueError::new_err(message),
ProjectionError::InternalSchemaFailure(detail) => {
PyRuntimeError::new_err(detail.to_string())
PyRuntimeError::new_err(Report::new(&detail).to_string())
}
}
}
@ -124,26 +124,26 @@ impl<'py> Field<'py> {
&& getter.is(object.getattr("__getattribute__")?))
}
pub(crate) fn path(&self) -> String {
format!("{}.{}", self.group, self.name)
}
/// A member of this field's collection, reported under the same path.
pub(crate) fn member(&self, value: Bound<'py, PyAny>) -> Self {
Self::new(self.group, self.name, value)
}
pub(crate) fn expected(&self, expected: &str) -> Result<String, ProjectionError> {
Ok(format!(
"{}: expected {expected}, got {}",
self.path(),
self.value.get_type().name()?
))
pub(crate) fn expected(
&self,
expected: &'static str,
) -> Result<ProjectionDetail, ProjectionError> {
Ok(ProjectionDetail::Expected {
group: self.group,
name: self.name,
expected,
actual: self.value.get_type().name()?.to_string(),
})
}
pub(crate) fn invalid(&self, expected: &str) -> ProjectionError {
pub(crate) fn invalid(&self, expected: &'static str) -> ProjectionError {
match self.expected(expected) {
Ok(message) => ProjectionError::InvalidConfiguration(message.into()),
Ok(detail) => ProjectionError::InvalidConfiguration(detail),
Err(error) => error,
}
}
@ -163,24 +163,23 @@ impl<'py> Field<'py> {
pub(crate) fn schema_bool(&self) -> Result<bool, ProjectionError> {
if !self.value.is_instance_of::<PyBool>() {
return Err(ProjectionError::InternalSchemaFailure(
self.expected("a Boolean")?.into(),
self.expected("a Boolean")?,
));
}
Ok(self.exact_true())
}
pub(crate) fn strict_string(&self) -> Result<String, ProjectionError> {
let value = self
.value
.cast::<PyString>()
.map_err(|_| self.invalid("a string"))?;
let Ok(value) = self.value.cast::<PyString>() else {
return Err(self.invalid("a string"));
};
Ok(value.to_str()?.to_owned())
}
pub(crate) fn schema_string(&self) -> Result<String, ProjectionError> {
if !self.value.is_instance_of::<PyString>() {
return Err(ProjectionError::InternalSchemaFailure(
self.expected("a string")?.into(),
self.expected("a string")?,
));
}
self.strict_string()

7
litellm-rust/crates/python-bridge/src/diagnostics.rs
View file

@ -18,8 +18,11 @@ pub(crate) fn process_state_started() -> bool {
/// so the runtime can never start. Raises if it already has. Forked workers are unaffected.
#[pyfunction]
pub(crate) fn reserve_process_for_forking() -> PyResult<()> {
litellm_host_python::reserve_process_for_forking()
.map_err(|_| PyRuntimeError::new_err("the native runtime already started in this process"))
litellm_host_python::reserve_process_for_forking().map_err(
|litellm_host_python::RuntimeAlreadyStarted| {
PyRuntimeError::new_err("the native runtime already started in this process")
},
)
}
#[cfg(feature = "panic-test")]

186
litellm-rust/crates/python-bridge/src/errors.rs
View file

@ -1,3 +1,6 @@
use std::error::Error as StdError;
use litellm_error::Report;
use litellm_inference::Error;
use pyo3::{
exceptions::{PyRuntimeError, PyValueError},
@ -15,21 +18,25 @@ pyo3::create_exception!(
_native,
RustUpstreamError,
pyo3::exceptions::PyException,
"The provider call was already issued and failed. Args are (status, message); status is 0 when there was no HTTP response."
"The provider call was already issued and the provider answered with a failure. Args are (status, body); the headers attribute holds the provider's (name, value) response headers."
);
pub(crate) fn route_error_to_pyerr(error: Error) -> PyErr {
match error.upstream_response() {
Some((status, body)) => RustUpstreamError::new_err((status, body.to_owned())),
None => by_fault(
error.is_request(),
litellm_error::Report::new(&error).to_string(),
),
}
const MESSAGES_REQUEST_ERROR: &str = "messages_request_error";
pub(crate) fn upstream_error(
py: Python<'_>,
status: u16,
body: &str,
headers: &[(String, String)],
) -> PyResult<PyErr> {
let error = RustUpstreamError::new_err((status, body.to_owned()));
error.value(py).setattr("headers", headers.to_vec())?;
Ok(error)
}
/// A request the caller got wrong is a `ValueError`; anything else is a `RuntimeError`.
pub(crate) fn by_fault(is_request: bool, message: String) -> PyErr {
pub(crate) fn by_fault(is_request: bool, error: &(dyn StdError + 'static)) -> PyErr {
let message = Report::new(error).to_string();
if is_request {
PyValueError::new_err(message)
} else {
@ -37,10 +44,59 @@ pub(crate) fn by_fault(is_request: bool, message: String) -> PyErr {
}
}
/// How a route flags a rejected request for its Python route module, which raises the public
/// exception for it instead of mapping the native one.
#[derive(Clone, Copy)]
pub(crate) enum RequestMarker {
Unmarked,
Messages,
}
/// Maps a route failure once. A Python exception raised inside the call is returned as it is,
/// everything else is mapped and handed to `public`, the route's public exception mapping.
pub(crate) fn route_failure(
py: Python<'_>,
error: Error,
marker: RequestMarker,
public: impl FnOnce(PyErr) -> PyResult<PyErr>,
) -> PyResult<PyErr> {
if let Some(raised) = crate::secrets::python_error(py, &error) {
return Ok(raised);
}
if let Some((status, body)) = error.upstream_response() {
return public(upstream_error(py, status, body, &[])?);
}
let native = match (marker, error.rejected_request()) {
(RequestMarker::Messages, Some(rejection)) => {
let native = PyValueError::new_err(Report::new(rejection).to_string());
native.value(py).setattr(MESSAGES_REQUEST_ERROR, true)?;
native
}
(RequestMarker::Messages, None) | (RequestMarker::Unmarked, _) => {
by_fault(error.is_request(), &error)
}
};
public(native)
}
pub(crate) fn route_error_to_pyerr(error: Error) -> PyErr {
Python::attach(|py| {
route_failure(py, error, RequestMarker::Unmarked, Ok).unwrap_or_else(std::convert::identity)
})
}
#[cfg(test)]
mod tests {
use super::*;
fn upstream_payload(py: Python<'_>, failure: &PyErr) -> ((u16, String), Vec<(String, String)>) {
let value = failure.value(py);
(
value.getattr("args").unwrap().extract().unwrap(),
value.getattr("headers").unwrap().extract().unwrap(),
)
}
#[rstest::rstest]
#[case::unsupported(Error::Unsupported("test capability"), true)]
#[case::invalid_provider(Error::InvalidProvider("unknown".into()), true)]
@ -73,20 +129,37 @@ mod tests {
}
#[rstest::rstest]
fn transport_status_survives_python_mapping() {
#[case::unmarked(RequestMarker::Unmarked)]
#[case::messages(RequestMarker::Messages)]
fn the_provider_answer_passes_through_with_a_headers_attribute(#[case] marker: RequestMarker) {
Python::initialize();
Python::attach(|py| {
let upstream =
route_error_to_pyerr(litellm_http::Error::upstream(429, "slow down".into()).into());
let error = litellm_http::Error::upstream(429, "slow down".into()).into();
let upstream = route_failure(py, error, marker, Ok).unwrap();
assert!(upstream.is_instance_of::<RustUpstreamError>(py));
assert_eq!(
upstream_payload(py, &upstream),
((429, "slow down".into()), Vec::new())
);
assert!(
upstream
.value(py)
.getattr("args")
.getattr_opt(MESSAGES_REQUEST_ERROR)
.unwrap()
.extract::<(u16, String)>()
.unwrap(),
(429, "slow down".into())
.is_none()
);
});
}
#[test]
fn upstream_error_keeps_the_provider_headers() {
Python::initialize();
Python::attach(|py| {
let headers = [("Retry-After".to_string(), "17".to_string())];
let upstream = upstream_error(py, 503, "busy", &headers).unwrap();
assert_eq!(
upstream_payload(py, &upstream),
((503, "busy".into()), headers.to_vec())
);
});
}
@ -104,4 +177,83 @@ mod tests {
assert!(invalid.is_instance_of::<PyValueError>(py));
});
}
#[test]
fn a_wrapped_failure_renders_every_layer() {
Python::initialize();
Python::attach(|py| {
let error = Error::post_call(
litellm_http::Error::upstream(500, "hook backend down".into()).into(),
);
let report = Report::new(&error).to_string();
let failure = route_error_to_pyerr(error);
assert_eq!(failure.value(py).to_string(), report);
assert!(report.contains("hook backend down"), "{report}");
});
}
#[rstest::rstest]
#[case::rejected_request(
Error::invalid_request(litellm_llms::ErrorDetail::UnsupportedParameter {
model: "claude".into(),
param: "top_k".into(),
value: "5".into(),
hint: String::new(),
}),
Some("claude does not support top_k=5. To drop unsupported params, set `litellm.drop_params = True`.")
)]
#[case::missing_field(
litellm_llms::Error::MissingField("max_tokens").into(),
Some("missing required field: max_tokens")
)]
#[case::unresolvable_provider(Error::InvalidProvider("openai".into()), None)]
#[case::invalid_response(
Error::invalid_response(litellm_llms::ErrorDetail::MissingField {
subject: "response",
field: "usage",
}),
None
)]
fn only_messages_request_rejections_carry_the_marker_and_python_text(
#[case] error: Error,
#[case] rejection: Option<&str>,
) {
Python::initialize();
Python::attach(|py| {
let unmarked = route_failure(py, error.clone(), RequestMarker::Unmarked, Ok).unwrap();
assert!(
unmarked
.value(py)
.getattr_opt(MESSAGES_REQUEST_ERROR)
.unwrap()
.is_none()
);
let native = route_failure(py, error, RequestMarker::Messages, Ok).unwrap();
let marked = native
.value(py)
.getattr_opt(MESSAGES_REQUEST_ERROR)
.unwrap()
.map(|value| value.extract::<bool>().unwrap());
assert_eq!(marked.unwrap_or(false), rejection.is_some());
if let Some(rejection) = rejection {
assert!(native.is_instance_of::<PyValueError>(py));
assert_eq!(native.value(py).to_string(), rejection);
}
});
}
#[test]
fn a_python_secret_manager_exception_skips_the_public_mapping() {
Python::initialize();
Python::attach(|py| {
let raised = pyo3::exceptions::PyKeyError::new_err("vault down");
let identity = raised.value(py).clone().unbind();
let error = Error::from(crate::secrets::read_error(py, raised));
let surfaced = route_failure(py, error, RequestMarker::Unmarked, |_| {
panic!("the raised exception must not be remapped")
})
.unwrap();
assert!(surfaced.value(py).is(identity.bind(py)));
});
}
}

8
litellm-rust/crates/python-bridge/src/logger/mod.rs
View file

@ -81,8 +81,12 @@ impl NativeDiagnosticProcessor {
}
}
fn processing_error(_: fancy_regex::Error) -> PyErr {
PyRuntimeError::new_err("diagnostic processing failed")
#[derive(Debug, thiserror::Error)]
#[error("diagnostic processing failed")]
struct ProcessingError(#[source] fancy_regex::Error);
fn processing_error(error: fancy_regex::Error) -> PyErr {
PyRuntimeError::new_err(litellm_error::Report::new(&ProcessingError(error)).to_string())
}
struct PythonSink {

50
litellm-rust/crates/python-bridge/src/marshal.rs
View file

@ -21,10 +21,27 @@ pub(crate) struct RouteOptions {
pub(crate) timeout: Option<Duration>,
}
fn required_object(name: &'static str, value: Value) -> PyResult<Map<String, Value>> {
/// A caller argument the bridge cannot use, raised as a `ValueError`.
#[derive(Debug, thiserror::Error)]
pub(crate) enum ArgumentError {
#[error("{0} is required")]
Required(&'static str),
#[error("{0} must be a dict")]
NotADict(&'static str),
#[error("header values must be strings")]
HeaderValue,
}
impl From<ArgumentError> for PyErr {
fn from(error: ArgumentError) -> Self {
PyValueError::new_err(litellm_error::Report::new(&error).to_string())
}
}
fn required_object(name: &'static str, value: Value) -> Result<Map<String, Value>, ArgumentError> {
match value {
Value::Object(values) => Ok(values),
_ => Err(PyValueError::new_err(format!("{name} must be a dict"))),
_ => Err(ArgumentError::NotADict(name)),
}
}
@ -35,7 +52,7 @@ fn optional_object(
if value.is_none() {
return Ok(None);
}
required_object(name, from_py_argument(value)?).map(Some)
Ok(Some(required_object(name, from_py_argument(value)?)?))
}
pub(crate) fn optional_timeout(timeout_seconds: Option<f64>) -> Option<Duration> {
@ -57,11 +74,11 @@ pub(crate) fn python_timeout_seconds(py: Python<'_>, timeout: Py<PyAny>) -> PyRe
pub(crate) fn required_field<'py>(
fields: &Bound<'py, PyDict>,
name: &str,
name: &'static str,
) -> PyResult<Bound<'py, PyAny>> {
fields
Ok(fields
.get_item(name)?
.ok_or_else(|| PyValueError::new_err(format!("{name} is required")))
.ok_or(ArgumentError::Required(name))?)
}
pub(crate) fn optional_field<T: DeserializeOwned>(
@ -171,21 +188,18 @@ pub(crate) fn request_input_sources<'a>(
.collect())
}
pub(crate) fn marshal_headers(headers: Option<Value>) -> PyResult<HashMap<String, String>> {
let value = match headers {
Some(headers) => headers,
None => Value::Object(Map::new()),
};
let Value::Object(headers) = value else {
return Err(PyValueError::new_err("headers must be a dict"));
pub(crate) fn marshal_headers(
headers: Option<Value>,
) -> Result<HashMap<String, String>, ArgumentError> {
let headers = match headers {
Some(headers) => required_object("headers", headers)?,
None => Map::new(),
};
headers
.into_iter()
.map(|(name, value)| {
value
.as_str()
.map(|value| (name, value.to_string()))
.ok_or_else(|| PyValueError::new_err("header values must be strings"))
.map(|(name, value)| match value {
Value::String(value) => Ok((name, value)),
_ => Err(ArgumentError::HeaderValue),
})
.collect()
}

10
litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs
View file

@ -13,7 +13,7 @@ use crate::{
};
async fn execute(
http: Result<litellm_http::Client, litellm_http::Error>,
http: litellm_http::Client,
secrets: std::sync::Arc<dyn litellm_secrets::source::SecretSource>,
audio: Value,
optional_params: Map<String, Value>,
@ -27,7 +27,7 @@ async fn execute(
extra_headers,
timeout,
} = options;
AudioTranscriptionRoute::new(http?, crate::http::resources().auth.clone(), secrets)
AudioTranscriptionRoute::new(http, crate::http::resources().auth.clone(), secrets)
.execute(AudioTranscriptionRequest {
model: &model,
audio,
@ -48,7 +48,8 @@ pub(crate) fn transcription(py: Python<'_>, call: NativeCall<'_>) -> PyResult<Py
litellm_host_python::from_py_argument(&required_field(&arguments, "audio")?)?;
let options = value_route_options(&arguments)?;
let optional_params = optional_object_field(&arguments, "optional_params")?.unwrap_or_default();
let http = crate::http::provider_client(py, &call.kwargs, false)?;
let http = crate::http::provider_client(py, &call.kwargs, false)?
.map_err(crate::http::client_error)?;
let secrets = crate::secrets::source(py)?;
run_sync(
py,
@ -67,7 +68,8 @@ pub(crate) fn atranscription<'py>(
litellm_host_python::from_py_argument(&required_field(&arguments, "audio")?)?;
let options = value_route_options(&arguments)?;
let optional_params = optional_object_field(&arguments, "optional_params")?.unwrap_or_default();
let http = crate::http::provider_client(py, &call.kwargs, true)?;
let http =
crate::http::provider_client(py, &call.kwargs, true)?.map_err(crate::http::client_error)?;
let secrets = crate::secrets::source(py)?;
run_async(
py,

4
litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs
View file

@ -78,8 +78,8 @@ impl PythonBinding for ChatCompletionsPythonHost {
fn map_error(&self, py: Python<'_>, error: Error) -> PyResult<PyErr> {
self.0.error(py, error)
}
fn host_error(_: &PyErr) -> Error {
Error::invalid_request(litellm_llms::ErrorDetail::HostCall)
fn host_error(error: &PyErr) -> Error {
crate::routes::inference::host_call_error(error)
}
}

47
litellm-rust/crates/python-bridge/src/routes/inference.rs
View file

@ -9,7 +9,6 @@ use litellm_host_python::{PythonBinding, PythonHostCalls, from_py};
use litellm_inference::Error;
use litellm_secrets::source::SecretSource;
use pyo3::{
exceptions::PyValueError,
gc::{PyTraverseError, PyVisit},
prelude::*,
types::PyDict,
@ -19,8 +18,10 @@ use serde_json::{Map, Value};
use super::{NativeCall, RequestView};
use crate::{
errors::{RustUpstreamError, route_error_to_pyerr},
marshal::{RouteOptions, optional_timeout, public_response, python_timeout_seconds},
errors::{RequestMarker, route_failure},
marshal::{
ArgumentError, RouteOptions, optional_timeout, public_response, python_timeout_seconds,
},
};
pub(super) struct InferenceHost {
@ -43,7 +44,7 @@ impl InferenceHost {
&mut self,
py: Python<'_>,
arguments: &Bound<'_, PyDict>,
input: &str,
input: &'static str,
) -> PyResult<ProjectedCall> {
self.view.resolve(py, arguments)?;
let argument = |name: &str| self.argument(py, arguments, name);
@ -56,7 +57,7 @@ impl InferenceHost {
.map(|value| python_timeout_seconds(py, value.unbind()))
.transpose()?
.flatten();
let model = string("model")?.ok_or_else(|| PyValueError::new_err("model is required"))?;
let model = string("model")?.ok_or(ArgumentError::Required("model"))?;
let custom_llm_provider = string("custom_llm_provider")?;
let provider = get_custom_llm_provider(&model, custom_llm_provider.as_deref())
.map_or("", |resolved| resolved.custom_llm_provider);
@ -81,10 +82,7 @@ impl InferenceHost {
.transpose()?,
timeout: optional_timeout(timeout),
},
input: from_py(
&argument(input)?
.ok_or_else(|| PyValueError::new_err(format!("{input} is required")))?,
)?,
input: from_py(&argument(input)?.ok_or(ArgumentError::Required(input))?)?,
params,
})
}
@ -119,29 +117,20 @@ impl InferenceHost {
}
pub fn error(&self, py: Python<'_>, error: Error) -> PyResult<PyErr> {
if let Error::Secret(source) = &error
&& let Some(original) = crate::secrets::python_error(py, source.source_error())
{
return Ok(original);
}
let native = match error.upstream_response() {
Some((status, body)) => {
let error = RustUpstreamError::new_err((status, body.to_owned()));
error
.value(py)
.setattr("headers", Vec::<(String, String)>::new())?;
error
}
None => route_error_to_pyerr(error),
};
let mapped = py
.import(self.module)?
.getattr("map_failure")?
.call1((native.value(py), self.view.request(py)?))?;
Ok(PyErr::from_value(mapped))
route_failure(py, error, RequestMarker::Unmarked, |native| {
let mapped = py
.import(self.module)?
.getattr("map_failure")?
.call1((native.value(py), self.view.request(py)?))?;
Ok(PyErr::from_value(mapped))
})
}
}
pub(super) fn host_call_error(_: &PyErr) -> Error {
Error::invalid_request(litellm_llms::ErrorDetail::HostCall)
}
pub(super) trait InferenceRoute: Sized + 'static {
type Protocol: Protocol<Error = Error>;
const OPERATION: LoggingOperation;

83
litellm-rust/crates/python-bridge/src/routes/messages/host.rs
View file

@ -11,7 +11,7 @@ use litellm_llms::base_llm::messages::context::MessagesModelCapabilities;
use litellm_llms_types::headers::ProviderSpecificHeaders;
use litellm_router_types::LitellmParams;
use pyo3::{
exceptions::{PyException, PyValueError},
exceptions::PyException,
gc::{PyTraverseError, PyVisit},
prelude::*,
types::{PyBytes, PyDict},
@ -19,14 +19,16 @@ use pyo3::{
use serde_json::{Map, Value};
use crate::{
errors::{RustUpstreamError, route_error_to_pyerr},
marshal::{optional_timeout, project_optional_fields, public_response, python_timeout_seconds},
errors::{RequestMarker, route_failure},
marshal::{
ArgumentError, optional_timeout, project_optional_fields, public_response,
python_timeout_seconds,
},
python_settings::missing_module,
routes::RequestView,
};
const ROUTE_HOST_MODULE: &str = "litellm.rust_bridge.host.routes.messages";
const REQUEST_ERROR_MARKER: &str = "messages_request_error";
fn merge_headers(
forwarded: Option<Map<String, Value>>,
@ -67,29 +69,6 @@ fn project_litellm_params<'py>(
Ok(litellm_params(fields.into_iter().chain(folded).collect()))
}
fn native_error(py: Python<'_>, error: Error) -> PyResult<PyErr> {
if let Some((status, body)) = error.upstream_response() {
let error = RustUpstreamError::new_err((status, body.to_owned()));
error
.value(py)
.setattr("headers", Vec::<(String, String)>::new())?;
return Ok(error);
}
match error {
Error::Llm(litellm_llms::Error::InvalidRequest(message)) => {
let error = PyValueError::new_err(litellm_error::Report::new(&message).to_string());
error.value(py).setattr(REQUEST_ERROR_MARKER, true)?;
Ok(error)
}
Error::Llm(litellm_llms::Error::MissingField(field)) => {
let error = PyValueError::new_err(format!("missing required field: {field}"));
error.value(py).setattr(REQUEST_ERROR_MARKER, true)?;
Ok(error)
}
other => Ok(route_error_to_pyerr(other)),
}
}
/// The Python side of the Messages route: projects the prepared arguments and builds the
/// public response, chunks and exceptions.
pub(super) struct MessagesPythonHost {
@ -114,9 +93,8 @@ impl MessagesPythonHost {
let string = |name: &str| -> PyResult<Option<String>> {
argument(name)?.map(|value| value.extract()).transpose()
};
let model = string("model")?.ok_or_else(|| PyValueError::new_err("model is required"))?;
let messages =
argument("messages")?.ok_or_else(|| PyValueError::new_err("messages is required"))?;
let model = string("model")?.ok_or(ArgumentError::Required("model"))?;
let messages = argument("messages")?.ok_or(ArgumentError::Required("messages"))?;
let fields = self.view.body_params(py, arguments, ROUTE_HOST_MODULE)?;
let body = [
("model".to_string(), Value::String(model.clone())),
@ -275,16 +253,13 @@ impl PythonBinding for MessagesPythonHost {
}
fn map_error(&self, py: Python<'_>, error: Error) -> PyResult<PyErr> {
if let Error::Secret(source) = &error
&& let Some(original) = crate::secrets::python_error(py, source.source_error())
{
return Ok(original);
}
Ok(self.map_failure(py, native_error(py, error)?))
route_failure(py, error, RequestMarker::Messages, |native| {
Ok(self.map_failure(py, native))
})
}
fn host_error(_: &PyErr) -> Error {
Error::invalid_request(litellm_llms::ErrorDetail::HostCall)
fn host_error(error: &PyErr) -> Error {
crate::routes::inference::host_call_error(error)
}
}
@ -463,36 +438,4 @@ mod tests {
}
});
}
#[rstest]
#[case::rejected_request(
Error::invalid_request(litellm_llms::ErrorDetail::UnsupportedParameter {
model: "claude".into(),
param: "top_k".into(),
value: "5".into(),
hint: String::new(),
}),
true
)]
#[case::missing_field(litellm_llms::Error::MissingField("max_tokens").into(), true)]
#[case::unresolvable_provider(Error::InvalidProvider("openai".into()), false)]
#[case::upstream_failure(
litellm_http::Error::upstream(400, "bad".into()).into(),
false,
)]
fn only_request_rejections_carry_the_request_error_marker(
#[case] error: Error,
#[case] marked: bool,
) {
Python::initialize();
Python::attach(|py| {
let native = native_error(py, error).unwrap();
let marker = native
.value(py)
.getattr_opt(REQUEST_ERROR_MARKER)
.unwrap()
.map(|value| value.extract::<bool>().unwrap());
assert_eq!(marker.unwrap_or(false), marked);
});
}
}

51
litellm-rust/crates/python-bridge/src/routes/ocr/document.rs
View file

@ -9,6 +9,29 @@ use pyo3::{
types::{PyBytes, PyString, PyType},
};
/// A `type='file'` document the bridge cannot read, with the text Python's OCR path raises.
#[derive(Debug, thiserror::Error)]
enum DocumentError {
#[error(
"document with type='file' must include a 'file' field containing a pathlib.Path, file-like object, or bytes"
)]
MissingFile,
#[error(
"OCR file input does not accept bare str values. Pass bytes, a pathlib.Path, or a file-like object."
)]
BareString,
#[error(
"Unsupported file input type: {0}. Expected pathlib.Path, bytes, or a file-like object."
)]
UnsupportedType(String),
}
impl From<DocumentError> for PyErr {
fn from(error: DocumentError) -> Self {
PyValueError::new_err(litellm_error::Report::new(&error).to_string())
}
}
/// A `type='file'` document as projected: paths and bytes are typed inputs already; a
/// file-like object is a reader the projection consumes once every other field is read.
pub(super) enum FileDocumentInput {
@ -45,25 +68,18 @@ impl FromPyObject<'_, '_> for FileDocumentInput {
Err(error) if error.is_instance_of::<pyo3::exceptions::PyKeyError>(py) => None,
Err(error) => return Err(error),
};
let missing = || {
PyValueError::new_err(
"document with type='file' must include a 'file' field containing a pathlib.Path, file-like object, or bytes",
)
};
let file = document.get_item("file").map_err(|error| {
if error.is_instance_of::<pyo3::exceptions::PyKeyError>(py) {
missing()
} else {
error
let file = match document.get_item("file") {
Ok(file) => file,
Err(error) if error.is_instance_of::<pyo3::exceptions::PyKeyError>(py) => {
return Err(DocumentError::MissingFile.into());
}
})?;
Err(error) => return Err(error),
};
if file.is_none() {
return Err(missing());
return Err(DocumentError::MissingFile.into());
}
if file.is_instance_of::<PyString>() {
return Err(PyValueError::new_err(
"OCR file input does not accept bare str values. Pass bytes, a pathlib.Path, or a file-like object.",
));
return Err(DocumentError::BareString.into());
}
static PATH_LIKE: PyOnceLock<Py<PyType>> = PyOnceLock::new();
if file.is_instance(PATH_LIKE.import(py, "os", "PathLike")?)? {
@ -81,10 +97,7 @@ impl FromPyObject<'_, '_> for FileDocumentInput {
}
match PythonFileReader::from_file_like(&file)? {
Some(reader) => Ok(Self::Deferred { reader, mime_type }),
None => Err(PyValueError::new_err(format!(
"Unsupported file input type: {}. Expected pathlib.Path, bytes, or a file-like object.",
file.get_type(),
))),
None => Err(DocumentError::UnsupportedType(file.get_type().to_string()).into()),
}
}
}

258
litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs
View file

@ -1,136 +1,138 @@
use std::io::ErrorKind;
use litellm_error::Report;
use litellm_llms::base_llm::ocr::error::Error;
use pyo3::{
exceptions::{PyFileNotFoundError, PyOSError},
prelude::*,
};
use crate::errors::{RustUpstreamError, by_fault};
use crate::errors::{by_fault, upstream_error};
const REQUEST_FORMAT_ERROR: &str = "ocr_request_format_error";
pub(super) fn to_pyerr(error: Error) -> PyErr {
let status = error.http_status_code();
let mapped = Python::attach(|py| -> PyResult<PyErr> {
Ok(match error {
Error::Provider {
status,
body,
headers,
} => upstream_error(py, status, body, headers)?,
Error::Http(http) if let Some(status) = http.status() => {
let body = http
.upstream_response()
.map_or_else(|| http.to_string(), |(_, body)| body.to_owned());
upstream_error(py, status, body, Vec::new())?
}
Error::RequestFormat => {
let error = by_fault(true, Error::RequestFormat.to_string());
error
.value(py)
.setattr("ocr_request_format_error", true)
.ok();
error
}
Error::FileRead { path, cause } if cause.kind() == std::io::ErrorKind::NotFound => {
PyFileNotFoundError::new_err(format!("File not found: {}", path.display()))
}
Error::FileRead { cause, .. } => PyOSError::new_err(cause.to_string()),
other => by_fault(
is_request(&other),
litellm_error::Report::new(&other).to_string(),
),
})
})
.unwrap_or_else(|error| error);
attach_status(mapped, status)
Python::attach(|py| native(py, error).unwrap_or_else(std::convert::identity))
}
fn is_request(error: &Error) -> bool {
error.is_request()
|| matches!(
error,
Error::Auth(_)
| Error::InvalidProvider(_)
| Error::MissingField(_)
| Error::MissingDocumentUrl
)
}
fn upstream_error(
/// Maps an OCR route failure once. A Python exception raised inside the call is returned as it
/// is, everything else is mapped and handed to `public`, the route's public exception mapping.
pub(super) fn route_failure(
py: Python<'_>,
status: u16,
body: String,
headers: Vec<(String, String)>,
error: Error,
public: impl FnOnce(PyErr) -> PyResult<PyErr>,
) -> PyResult<PyErr> {
let error = RustUpstreamError::new_err((status, body));
error.value(py).setattr("headers", headers)?;
Ok(error)
match crate::secrets::python_error(py, &error) {
Some(raised) => Ok(raised),
None => public(native(py, error)?),
}
}
fn attach_status(error: PyErr, status: Option<u16>) -> PyErr {
if let Some(status) = status {
Python::attach(|py| {
let value = error.value(py);
value.setattr("status_code", status).ok();
value.setattr("message", value.to_string()).ok();
});
fn native(py: Python<'_>, error: Error) -> PyResult<PyErr> {
let mapped = match (&error, error.upstream_response()) {
(_, Some((status, body, headers))) => upstream_error(py, status, body, headers)?,
(Error::RequestFormat, None) => {
let mapped = by_fault(true, &error);
mapped.value(py).setattr(REQUEST_FORMAT_ERROR, true)?;
mapped
}
(Error::FileRead { path, cause }, None) if cause.kind() == ErrorKind::NotFound => {
PyFileNotFoundError::new_err(format!("File not found: {}", path.display()))
}
(Error::FileRead { cause, .. }, None) => {
PyOSError::new_err(Report::new(cause.as_ref()).to_string())
}
(_, None) => by_fault(error.is_request(), &error),
};
if let Some(status) = error.http_status_code() {
let value = mapped.value(py);
value.setattr("status_code", status)?;
value.setattr("message", value.str()?)?;
}
error
Ok(mapped)
}
#[cfg(test)]
mod tests {
use pyo3::exceptions::PyValueError;
use pyo3::exceptions::{PyRuntimeError, PyValueError};
use rstest::rstest;
use super::*;
use crate::errors::RustUpstreamError;
#[test]
fn preserves_python_validation_and_provider_details() {
fn status_code(py: Python<'_>, mapped: &PyErr) -> Option<u16> {
mapped
.value(py)
.getattr_opt("status_code")
.unwrap()
.map(|status| status.extract().unwrap())
}
fn file_read(kind: ErrorKind) -> Error {
Error::FileRead {
path: "/missing/scan.pdf".into(),
cause: std::sync::Arc::new(std::io::Error::new(kind, "disk said no")),
}
}
#[rstest]
#[case::provider(
Error::Provider {
status: 429,
body: r#"{"message":"rate limited"}"#.into(),
headers: vec![("Retry-After".into(), "17".into())],
},
vec![("Retry-After".into(), "17".into())]
)]
#[case::transport(litellm_http::Error::upstream(429, r#"{"message":"rate limited"}"#.into()).into(), vec![])]
fn a_provider_answer_keeps_its_status_body_and_headers(
#[case] error: Error,
#[case] headers: Vec<(String, String)>,
) {
Python::initialize();
Python::attach(|py| {
let mapped = to_pyerr(Error::MissingDocumentUrl);
assert!(mapped.is_instance_of::<pyo3::exceptions::PyValueError>(py));
assert_eq!(
mapped.value(py).to_string(),
Error::MissingDocumentUrl.to_string()
);
assert_eq!(
mapped
.value(py)
.getattr("status_code")
.unwrap()
.extract::<u16>()
.unwrap(),
400
);
let mapped = to_pyerr(Error::Provider {
status: 429,
body: r#"{"message":"rate limited"}"#.to_string(),
headers: vec![("Retry-After".to_string(), "17".to_string())],
});
let mapped = to_pyerr(error);
assert!(mapped.is_instance_of::<RustUpstreamError>(py));
let headers: Vec<(String, String)> = mapped
.value(py)
.getattr("headers")
.and_then(|headers| headers.extract())
.expect("OCR failures retain provider headers");
assert_eq!(headers, vec![("Retry-After".to_string(), "17".to_string())]);
let args: (u16, String) = mapped
.value(py)
.getattr("args")
.and_then(|args| args.extract())
.expect("OCR failures retain status and unprefixed provider message");
assert_eq!(args, (429, r#"{"message":"rate limited"}"#.to_string()));
let value = mapped.value(py);
let args: (u16, String) = value.getattr("args").unwrap().extract().unwrap();
assert_eq!(args, (429, r#"{"message":"rate limited"}"#.into()));
let kept: Vec<(String, String)> = value.getattr("headers").unwrap().extract().unwrap();
assert_eq!(kept, headers);
assert_eq!(status_code(py, &mapped), Some(429));
});
}
let mapped = to_pyerr(Error::Features);
#[rstest]
#[case::missing_document_url(Error::MissingDocumentUrl)]
#[case::invalid_features(Error::Features)]
#[case::invalid_provider(Error::InvalidProvider("unknown".into()))]
fn request_faults_are_value_errors_with_status_400(#[case] error: Error) {
Python::initialize();
Python::attach(|py| {
let report = Report::new(&error).to_string();
let mapped = to_pyerr(error);
assert!(mapped.is_instance_of::<PyValueError>(py));
assert_eq!(
mapped
.value(py)
.getattr("status_code")
.unwrap()
.extract::<u16>()
.unwrap(),
400
);
assert_eq!(mapped.value(py).to_string(), report);
assert_eq!(status_code(py, &mapped), Some(400));
});
}
#[rstest]
#[case::missing_api_key(litellm_auth::Error::MissingApiKey {
provider: "Mistral",
environment_variable: "MISTRAL_API_KEY",
}
.into())]
#[case::missing_credential(Error::MissingCredential {
credential: "Azure AI credentials",
guidance: "set AZURE_AI_API_KEY",
})]
fn a_missing_credential_is_a_statusless_runtime_error(#[case] error: Error) {
Python::initialize();
Python::attach(|py| {
let mapped = to_pyerr(error);
assert!(mapped.is_instance_of::<PyRuntimeError>(py));
assert_eq!(status_code(py, &mapped), None);
});
}
@ -143,19 +145,12 @@ mod tests {
assert!(mapped.is_instance_of::<PyValueError>(py));
assert!(
value
.getattr("ocr_request_format_error")
.getattr(REQUEST_FORMAT_ERROR)
.unwrap()
.extract::<bool>()
.unwrap()
);
assert_eq!(
value
.getattr("status_code")
.unwrap()
.extract::<u16>()
.unwrap(),
400
);
assert_eq!(status_code(py, &mapped), Some(400));
assert_eq!(
value
.getattr("message")
@ -167,18 +162,11 @@ mod tests {
});
}
fn file_read(kind: std::io::ErrorKind) -> Error {
Error::FileRead {
path: "/missing/scan.pdf".into(),
cause: std::sync::Arc::new(std::io::Error::new(kind, "disk said no")),
}
}
#[test]
fn missing_files_map_to_file_not_found_naming_the_path() {
Python::initialize();
Python::attach(|py| {
let mapped = to_pyerr(file_read(std::io::ErrorKind::NotFound));
let mapped = to_pyerr(file_read(ErrorKind::NotFound));
assert!(mapped.is_instance_of::<PyFileNotFoundError>(py));
assert_eq!(
mapped.value(py).to_string(),
@ -191,28 +179,32 @@ mod tests {
fn other_file_read_failures_map_to_os_error_with_the_io_message() {
Python::initialize();
Python::attach(|py| {
let mapped = to_pyerr(file_read(std::io::ErrorKind::PermissionDenied));
let mapped = to_pyerr(file_read(ErrorKind::PermissionDenied));
assert!(mapped.is_instance_of::<PyOSError>(py));
assert!(!mapped.is_instance_of::<PyFileNotFoundError>(py));
assert_eq!(mapped.value(py).to_string(), "disk said no");
});
}
#[rstest::rstest]
#[rstest]
#[case::oversized(Error::TooLarge { limit: 7 })]
#[case::malformed_field(Error::ResponseField { path: "pages[0].index".into(), cause: None })]
fn response_failures_are_statusless_runtime_errors(#[case] error: Error) {
#[case::malformed_field(Error::response_field(
"pages[0].index",
"-1".parse::<u32>().unwrap_err()
))]
fn response_failures_are_statusless_runtime_errors(
#[case] error: Error,
#[values("status_code", "ocr_request_format_error", "headers")] attribute: &str,
) {
Python::initialize();
Python::attach(|py| {
let message = error.to_string();
let report = Report::new(&error).to_string();
let mapped = to_pyerr(error);
let value = mapped.value(py);
assert!(mapped.is_instance_of::<pyo3::exceptions::PyRuntimeError>(py));
assert!(mapped.is_instance_of::<PyRuntimeError>(py));
assert!(!mapped.is_instance_of::<RustUpstreamError>(py));
assert_eq!(value.to_string(), message);
for attribute in ["status_code", "ocr_request_format_error", "headers"] {
assert!(!value.hasattr(attribute).unwrap(), "{attribute}");
}
assert_eq!(value.to_string(), report);
assert!(!value.hasattr(attribute).unwrap());
});
}
}

9
litellm-rust/crates/python-bridge/src/routes/ocr/host.rs
View file

@ -12,7 +12,7 @@ use pyo3::{
};
use super::{
errors::to_pyerr as ocr_error_to_pyerr,
errors::route_failure,
project::{OcrHostHandles, SETTINGS_ERROR_MARKER, project_request},
};
use crate::{marshal::public_response, routes::RequestView};
@ -135,12 +135,7 @@ impl PythonBinding for OcrPythonHost {
}
fn map_error(&self, py: Python<'_>, error: Error) -> PyResult<PyErr> {
if let Error::Secret(source) = &error
&& let Some(original) = crate::secrets::python_error(py, source)
{
return Ok(original);
}
Ok(self.map_failure(py, ocr_error_to_pyerr(error)))
route_failure(py, error, |native| Ok(self.map_failure(py, native)))
}
fn host_error(_: &PyErr) -> Error {

18
litellm-rust/crates/python-bridge/src/routes/ocr/project.rs
View file

@ -6,14 +6,16 @@ use litellm_inference_ocr::{
wire::{OcrWireRequest, consumed_optional_params, decode_document, decode_request_input},
};
use litellm_llms::base_llm::ocr::error::Error;
use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict};
use pyo3::{prelude::*, types::PyDict};
use serde_json::{Map, Value};
use super::{document::FileDocumentInput, errors::to_pyerr as ocr_error_to_pyerr};
use crate::{
coercion::FieldSpec,
credentials::{self, CallerTokenProvider},
marshal::{project_optional_fields, python_timeout_seconds, request_input_sources},
marshal::{
ArgumentError, project_optional_fields, python_timeout_seconds, request_input_sources,
},
python_settings::{PythonSettings, Snapshot},
};
@ -30,9 +32,9 @@ struct OcrArguments<'a, 'py> {
}
impl<'py> OcrArguments<'_, 'py> {
fn lookup(&self, name: &str) -> PyResult<Bound<'py, PyAny>> {
litellm_host_python::lookup(self.kwargs, self.bound, name)?
.ok_or_else(|| PyValueError::new_err(format!("missing argument: {name}")))
fn lookup(&self, name: &'static str) -> PyResult<Bound<'py, PyAny>> {
Ok(litellm_host_python::lookup(self.kwargs, self.bound, name)?
.ok_or(ArgumentError::Required(name))?)
}
fn model(&self) -> PyResult<String> {
@ -219,8 +221,10 @@ fn fold_module_globals(
pub(super) const SETTINGS_ERROR_MARKER: &str = "native_settings_error";
fn settings_error(py: Python<'_>, error: PyErr) -> PyErr {
error.value(py).setattr(SETTINGS_ERROR_MARKER, true).ok();
error
match error.value(py).setattr(SETTINGS_ERROR_MARKER, true) {
Ok(()) => error,
Err(failure) => failure,
}
}
#[cfg(test)]

16
litellm-rust/crates/python-bridge/src/routes/responses/host.rs
View file

@ -141,9 +141,9 @@ impl PythonBinding for ResponsesPythonHost {
head: <Responses as litellm_host::protocol::Protocol>::StreamHead,
) -> PyResult<Py<PyAny>> {
let _ = head;
Err(pyo3::exceptions::PyRuntimeError::new_err(
"native Python Responses streaming is not supported",
))
Err(crate::errors::route_error_to_pyerr(Error::Unsupported(
"native Python responses streaming",
)))
}
fn encode_chunk(
@ -152,16 +152,16 @@ impl PythonBinding for ResponsesPythonHost {
chunk: <Responses as litellm_host::protocol::Protocol>::Chunk,
) -> PyResult<Py<PyAny>> {
let _ = chunk;
Err(pyo3::exceptions::PyRuntimeError::new_err(
"native Python Responses streaming is not supported",
))
Err(crate::errors::route_error_to_pyerr(Error::Unsupported(
"native Python responses streaming",
)))
}
fn map_error(&self, py: Python<'_>, error: Error) -> PyResult<PyErr> {
self.host.error(py, error)
}
fn host_error(_: &PyErr) -> Error {
Error::invalid_request(litellm_llms::ErrorDetail::HostCall)
fn host_error(error: &PyErr) -> Error {
crate::routes::inference::host_call_error(error)
}
}

4
litellm-rust/crates/python-bridge/src/secrets/config.rs
View file

@ -128,9 +128,7 @@ pub(crate) fn read(py: Python<'_>) -> PyResult<SecretManagerSnapshot> {
.system
.is_some_and(|system| system != backend.system())
{
return Err(pyo3::exceptions::PyValueError::new_err(
"native secret manager system does not match configuration",
));
return Err(super::ManagerError::SystemMismatch.into());
}
Ok(SecretManagerSnapshot {
client: SecretManagerClient::Native(Box::new(backend)),

76
litellm-rust/crates/python-bridge/src/secrets/error.rs
View file

@ -1,7 +1,66 @@
use std::fmt;
use litellm_error::Report;
use litellm_secrets::Error;
use pyo3::{exceptions::PyBaseException, prelude::*};
use pyo3::{
exceptions::{PyBaseException, PyRuntimeError, PyValueError},
prelude::*,
};
/// A secret manager call the bridge rejects, with the text Python's own managers raise.
#[derive(Debug, thiserror::Error)]
pub(crate) enum ManagerError {
#[error("native secret manager must be recreated after fork")]
Forked,
#[error("unknown secret manager system")]
UnknownSystem(#[source] serde_json::Error),
#[error("invalid secret manager settings")]
Settings(#[source] litellm_error::Source),
#[error("native secret manager system does not match configuration")]
SystemMismatch,
#[error("secret manager does not support provider reads")]
ProviderReads,
#[error("invalid timeout")]
Timeout(#[source] std::time::TryFromFloatSecsError),
#[error("No secret found in CyberArk Secret Manager for {0}")]
CyberarkSecretMissing(String),
#[error("Current secret {0} not found")]
CurrentMissing(String),
#[error("Failed to verify new secret {0}")]
ReplacementMissing(String),
#[error("New secret value mismatch")]
ReplacementMismatch,
#[error("native secret manager mutation is unavailable")]
MutationUnsupported,
#[error(transparent)]
Secret(Error),
}
impl ManagerError {
pub(crate) fn settings(error: impl std::error::Error + Send + Sync + 'static) -> Self {
Self::Settings(litellm_error::Source::new(error))
}
}
impl From<ManagerError> for PyErr {
fn from(error: ManagerError) -> Self {
let message = Report::new(&error).to_string();
match error {
ManagerError::Forked => PyRuntimeError::new_err(message),
ManagerError::UnknownSystem(_)
| ManagerError::Settings(_)
| ManagerError::SystemMismatch
| ManagerError::ProviderReads
| ManagerError::Timeout(_)
| ManagerError::CyberarkSecretMissing(_)
| ManagerError::CurrentMissing(_)
| ManagerError::ReplacementMissing(_)
| ManagerError::ReplacementMismatch
| ManagerError::MutationUnsupported
| ManagerError::Secret(_) => PyValueError::new_err(message),
}
}
}
#[derive(thiserror::Error)]
#[error("Python secret manager failed")]
@ -19,17 +78,18 @@ pub(super) fn external_error(py: Python<'_>, error: PyErr) -> Error {
)))
}
pub(super) fn read_error(py: Python<'_>, error: PyErr) -> Error {
pub(crate) fn read_error(py: Python<'_>, error: PyErr) -> Error {
Error::ExternalRead(litellm_error::Source::new(PythonSecretError(
error.into_value(py),
)))
}
pub(crate) fn python_error(py: Python<'_>, error: &Error) -> Option<PyErr> {
let (Error::ExternalManager(source) | Error::ExternalRead(source)) = error else {
return None;
};
source
.downcast_ref::<PythonSecretError>()
/// The exception a Python secret manager raised anywhere in the chain, so a route surfaces it
/// unchanged instead of wrapping it.
pub(crate) fn python_error(
py: Python<'_>,
error: &(dyn std::error::Error + 'static),
) -> Option<PyErr> {
litellm_error::find::<PythonSecretError>(error)
.map(|error| PyErr::from_value(error.0.clone_ref(py).into_bound(py).into_any()))
}

4
litellm-rust/crates/python-bridge/src/secrets/mod.rs
View file

@ -11,7 +11,9 @@ mod vault;
use std::sync::Arc;
pub(crate) use error::python_error;
#[cfg(test)]
pub(crate) use error::read_error;
pub(crate) use error::{ManagerError, python_error};
use litellm_secrets::source::{EnvironmentSecrets, SecretSource};
use pyo3::prelude::*;
use python::PythonSecrets;

69
litellm-rust/crates/python-bridge/src/secrets/mutation.rs
View file

@ -1,7 +1,10 @@
use super::operations::{PythonMutationError, PythonMutationResponse};
use litellm_error::Report;
use litellm_host_python::{json_loads, to_py};
use litellm_secrets::cyberark;
use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict};
use pyo3::{prelude::*, types::PyDict};
use super::ManagerError;
pub(super) fn mutation_value(
result: Result<PythonMutationResponse, PythonMutationError>,
@ -23,18 +26,16 @@ pub(super) fn mutation_value(
&serde_json::json!({"status": "error", "message": message}),
)
}
Err(PythonMutationError::CurrentMissing(name)) => Err(PyValueError::new_err(format!(
"Current secret {name} not found"
))),
Err(PythonMutationError::ReplacementMissing(name)) => Err(PyValueError::new_err(format!(
"Failed to verify new secret {name}"
))),
Err(PythonMutationError::ReplacementMismatch) => {
Err(PyValueError::new_err("New secret value mismatch"))
Err(PythonMutationError::CurrentMissing(name)) => {
Err(ManagerError::CurrentMissing(name).into())
}
Err(PythonMutationError::Unsupported) => Err(PyValueError::new_err(
"native secret manager mutation is unavailable",
)),
Err(PythonMutationError::ReplacementMissing(name)) => {
Err(ManagerError::ReplacementMissing(name).into())
}
Err(PythonMutationError::ReplacementMismatch) => {
Err(ManagerError::ReplacementMismatch.into())
}
Err(PythonMutationError::Unsupported) => Err(ManagerError::MutationUnsupported.into()),
})
}
@ -43,18 +44,35 @@ fn cyberark_failure(
name: &str,
failure: cyberark::WriteFailure,
) -> PyResult<String> {
let message = match failure.source {
let message = if failure.source.is_unsafe_secret_name() {
format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?)
} else {
cyberark_message(
py,
failure.source,
failure.authentication,
failure.request_url.as_ref(),
)?
};
Ok(if failure.authentication {
format!("Could not authenticate to CyberArk Conjur: {message}")
} else {
message
})
}
fn cyberark_message(
py: Python<'_>,
source: cyberark::Error,
authentication: bool,
request_url: Option<&reqwest::Url>,
) -> PyResult<String> {
Ok(match source {
cyberark::Error::Status(status) | cyberark::Error::AuthStatus(status) => {
let url = failure
.request_url
.as_ref()
.map_or("", reqwest::Url::as_str);
let url = request_url.map_or("", reqwest::Url::as_str);
http_message(py, "POST", url, status)?
}
cyberark::Error::Operation(litellm_secrets_types::Error::UnsafeSecretName) => {
format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?)
}
cyberark::Error::Http(source) if failure.authentication => match os_error_code(&source) {
cyberark::Error::Http(source) if authentication => match os_error_code(&source) {
Some(code) => {
let reason = py.import("os")?.getattr("strerror")?.call1((code,))?;
py.import("builtins")?
@ -63,17 +81,12 @@ fn cyberark_failure(
.str()?
.extract()?
}
None => cyberark::Error::Http(source).to_string(),
None => Report::new(&cyberark::Error::Http(source)).to_string(),
},
cyberark::Error::Http(source) if source.is_connect() => {
"All connection attempts failed".to_owned()
}
source => source.to_string(),
};
Ok(if failure.authentication {
format!("Could not authenticate to CyberArk Conjur: {message}")
} else {
message
source => Report::new(&source).to_string(),
})
}

12
litellm-rust/crates/python-bridge/src/secrets/provider.rs
View file

@ -6,7 +6,9 @@ use litellm_secrets_types::{
AwsOperationContext, CyberarkOperationContext, GoogleOperationContext,
HashicorpOperationContext, SecretOperationContext,
};
use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict};
use pyo3::{prelude::*, types::PyDict};
use super::ManagerError;
pub(super) fn read_request(
system: KeyManagementSystem,
@ -39,9 +41,7 @@ pub(super) fn read_request(
SecretOperationContext::Google(GoogleOperationContext::default())
}
_ => {
return Err(PyValueError::new_err(
"secret manager does not support provider reads",
));
return Err(ManagerError::ProviderReads.into());
}
};
Ok(PythonReadRequest {
@ -96,9 +96,7 @@ fn read_timeout(value: Option<&Bound<'_, PyAny>>) -> PyResult<Option<Duration>>
Err(_) => value.getattr("read")?.extract::<Option<f64>>()?,
};
seconds
.map(|value| {
Duration::try_from_secs_f64(value).map_err(|_| PyValueError::new_err("invalid timeout"))
})
.map(|value| Ok(Duration::try_from_secs_f64(value).map_err(ManagerError::Timeout)?))
.transpose()
}

46
litellm-rust/crates/python-bridge/src/secrets/runtime.rs
View file

@ -7,12 +7,9 @@ use litellm_secrets::{
read_secret_from_python_manager,
};
use litellm_secrets_types::PythonSecretRead;
use pyo3::{
exceptions::{PyAttributeError, PyRuntimeError, PyValueError},
prelude::*,
types::PyDict,
};
use pyo3::{exceptions::PyAttributeError, prelude::*, types::PyDict};
use super::ManagerError;
use crate::execution::{run_async_value, run_sync_value};
const SECRET_MANAGER_MODULE: &str = "litellm.rust_bridge.host.secret_manager";
@ -35,9 +32,7 @@ pub(crate) struct NativeSecretManager {
impl NativeSecretManager {
pub(super) fn backend(&self) -> PyResult<SecretManager> {
if self.pid != std::process::id() {
return Err(PyRuntimeError::new_err(
"native secret manager must be recreated after fork",
));
return Err(ManagerError::Forked.into());
}
Ok(self.backend.clone())
}
@ -60,7 +55,7 @@ impl NativeSecretManager {
enterprise_enabled,
)
.await
.map_err(|error| PyValueError::new_err(error.to_string()))
.map_err(|error| ManagerError::Secret(error).into())
})?;
Ok(Self {
backend,
@ -82,7 +77,7 @@ impl NativeSecretManager {
enterprise_enabled: bool,
) -> PyResult<Self> {
let system = serde_json::from_value(serde_json::Value::String(system.to_owned()))
.map_err(|_| PyValueError::new_err("unknown secret manager system"))?;
.map_err(ManagerError::UnknownSystem)?;
let settings = parse_settings(settings)?;
Self::build(
py,
@ -146,7 +141,7 @@ impl NativeSecretManager {
.collect::<PyResult<Vec<_>>>()?;
let settings =
from_py::<serde_json::Map<String, serde_json::Value>>(&config.getattr("settings")?)
.map_err(|_| PyValueError::new_err("invalid secret manager settings"))?;
.map_err(ManagerError::settings)?;
let attributes = config
.getattr("settings_attributes")?
.extract::<Vec<String>>()?;
@ -161,11 +156,11 @@ impl NativeSecretManager {
system: serde_json::from_value(serde_json::Value::String(
config.getattr("system")?.extract()?,
))
.map_err(|_| PyValueError::new_err("unknown secret manager system"))?,
.map_err(ManagerError::UnknownSystem)?,
settings: serde_json::from_value(serde_json::Value::Object(
settings.into_iter().chain(setting_overrides).collect(),
))
.map_err(|_| PyValueError::new_err("invalid secret manager settings"))?,
.map_err(ManagerError::settings)?,
environment: captured
.into_iter()
.filter(|(key, _)| !environment_attributes.contains_key(key))
@ -190,12 +185,8 @@ impl NativeSecretManager {
}
#[getter]
fn system(&self) -> String {
serde_json::to_value(self.configuration.system)
.expect("serializable system")
.as_str()
.expect("string system")
.to_owned()
fn system(&self) -> &'static str {
self.configuration.system.into()
}
#[pyo3(signature = (name, settings=None))]
@ -238,7 +229,7 @@ impl NativeSecretManager {
run_sync_value(py, async move {
super::operations::read_python_provider(&backend, &request, &ProcessEnvironment)
.await
.map_err(|error| PyValueError::new_err(error.to_string()))
.map_err(|error| ManagerError::Secret(error).into())
.and_then(|value| python_secret_value(value, &request.secret_name))
})
}
@ -264,7 +255,7 @@ impl NativeSecretManager {
run_async_value(py, async move {
super::operations::read_python_provider(&backend, &request, &ProcessEnvironment)
.await
.map_err(|error| PyValueError::new_err(error.to_string()))
.map_err(|error| ManagerError::Secret(error).into())
.and_then(|value| python_secret_value(value, &request.secret_name))
})
}
@ -438,8 +429,8 @@ fn cached(
fn parse_settings(value: Option<&Bound<'_, PyAny>>) -> PyResult<KeyManagementSettings> {
value
.map(|value| {
serde_json::from_value(from_py::<serde_json::Value>(value)?)
.map_err(|_| PyValueError::new_err("invalid secret manager settings"))
Ok(serde_json::from_value(from_py::<serde_json::Value>(value)?)
.map_err(ManagerError::settings)?)
})
.transpose()
.map(Option::unwrap_or_default)
@ -466,11 +457,10 @@ fn python_read_error(
name: &str,
error: litellm_secrets::Error,
) -> PyErr {
let message = match (system, error) {
match (system, error) {
(KeyManagementSystem::Cyberark, litellm_secrets::Error::ManagedSecretMissing) => {
format!("No secret found in CyberArk Secret Manager for {name}")
ManagerError::CyberarkSecretMissing(name.to_owned()).into()
}
(_, error) => error.to_string(),
};
PyValueError::new_err(message)
(_, error) => ManagerError::Secret(error).into(),
}
}

5
litellm-rust/crates/python-bridge/src/secrets/vault.rs
View file

@ -2,6 +2,7 @@ mod operation;
pub(super) use operation::{Failure, FailureKind, FailureStage, delete, rotate, write};
use litellm_error::Report;
use litellm_secrets::hashicorp::{Error, RawOperationError};
use pyo3::prelude::*;
@ -63,7 +64,7 @@ pub(super) fn failure_value(
fn cause_message(py: Python<'_>, kind: FailureKind, context: &ErrorContext) -> PyResult<String> {
Ok(match kind {
FailureKind::Native(RawOperationError::Local(error)) => error.to_string(),
FailureKind::Native(RawOperationError::Local(error)) => Report::new(&error).to_string(),
FailureKind::UnsafeName(name) => {
format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?)
}
@ -126,7 +127,7 @@ fn cause_message(py: Python<'_>, kind: FailureKind, context: &ErrorContext) -> P
}) => {
let message = match source {
Error::LoginStatus { status } => http_message(py, "POST", &url, status)?,
error => error.to_string(),
error => Report::new(&error).to_string(),
};
let mechanism = if certificate { "TLS cert" } else { "AppRole" };
format!("Could not authenticate to Vault via {mechanism}: {message}")

10
litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs
View file

@ -2,7 +2,7 @@ use std::collections::HashMap;
use litellm_secrets::{
SecretValue,
hashicorp::{Error, HashicorpVault, RawOperationError},
hashicorp::{HashicorpVault, RawOperationError},
};
use litellm_secrets_types::{HashicorpOperationContext, SecretWriteContext};
use serde_json::value::RawValue;
@ -53,12 +53,10 @@ impl Failure {
}
fn native_failure(name: &str, error: RawOperationError) -> Failure {
match error {
RawOperationError::Local(Error::InvalidSecretName(_)) => {
FailureKind::UnsafeName(name.to_owned()).into()
}
error => FailureKind::Native(error).into(),
if error.is_unsafe_secret_name() {
return FailureKind::UnsafeName(name.to_owned()).into();
}
FailureKind::Native(error).into()
}
pub(crate) async fn write(

121
litellm-rust/crates/python-bridge/src/tokenizer.rs
View file

@ -8,7 +8,6 @@ use std::sync::Arc;
#[cfg(feature = "fast")]
use std::sync::OnceLock;
#[cfg(any(feature = "tiktoken", feature = "huggingface"))]
use litellm_error::Report;
use litellm_host_python::{enter_native, release_gil};
#[cfg(feature = "fast")]
@ -16,10 +15,11 @@ use litellm_token_counter::fast::{FastCounter, FastTokenizer};
use litellm_token_counter::{Error, TextCodec};
use pyo3::{exceptions::PyUnicodeEncodeError, prelude::*, types::PyString};
#[cfg(any(feature = "tiktoken", feature = "huggingface"))]
use pyo3::exceptions::PyValueError;
#[cfg(feature = "huggingface")]
use pyo3::{exceptions::PyIOError, types::PyDict};
use pyo3::{
exceptions::{PyIOError, PyTypeError, PyValueError},
types::PyDict,
};
#[cfg(feature = "tiktoken")]
use pyo3::{
exceptions::{PyKeyError, PyRuntimeError},
@ -53,6 +53,70 @@ pub(crate) fn load_tiktoken(py: Python<'_>, encoding: &str) -> PyResult<Tiktoken
.map_err(|error| token_count_error_to_pyerr(error.into()))
}
/// A tokenizer call the bridge rejects itself, mapped once onto the class each tokenizer
/// library raises for it.
#[derive(Debug, thiserror::Error)]
enum TokenizerError {
#[cfg(not(all(feature = "tiktoken", feature = "huggingface")))]
#[error("tokenizer backend requires the {0} feature")]
MissingFeature(&'static str),
#[cfg(all(feature = "tiktoken", feature = "huggingface"))]
#[error("requires a tiktoken encoding")]
NotTiktoken,
#[cfg(all(feature = "tiktoken", feature = "huggingface"))]
#[error("requires a Hugging Face tokenizer")]
NotHuggingFace,
#[cfg(feature = "tiktoken")]
#[error("this encoding was built without its vocabulary")]
NoVocabulary,
#[cfg(feature = "tiktoken")]
#[error(transparent)]
EncodeSpecial(Error),
#[cfg(feature = "tiktoken")]
#[error(transparent)]
DecodeBytes(Error),
#[cfg(feature = "huggingface")]
#[error("input must match is_pretokenized")]
PretokenizedMismatch,
#[cfg(feature = "huggingface")]
#[error("invalid {what} direction {value:?}: expected 'left' or 'right'")]
Direction { what: &'static str, value: String },
#[cfg(feature = "huggingface")]
#[error("could not read the downloaded tokenizer")]
Download(#[source] std::io::Error),
#[cfg(feature = "huggingface")]
#[error(transparent)]
EncodingJson(Error),
}
impl From<TokenizerError> for PyErr {
fn from(error: TokenizerError) -> Self {
let message = Report::new(&error).to_string();
match error {
#[cfg(not(all(feature = "tiktoken", feature = "huggingface")))]
TokenizerError::MissingFeature(_) => RustBridgeDeclined::new_err(message),
#[cfg(all(feature = "tiktoken", feature = "huggingface"))]
TokenizerError::NotTiktoken | TokenizerError::NotHuggingFace => {
PyValueError::new_err(message)
}
#[cfg(feature = "tiktoken")]
TokenizerError::NoVocabulary | TokenizerError::EncodeSpecial(_) => {
PyRuntimeError::new_err(message)
}
#[cfg(feature = "tiktoken")]
TokenizerError::DecodeBytes(_) => PyKeyError::new_err(message),
#[cfg(feature = "huggingface")]
TokenizerError::Direction { .. } | TokenizerError::EncodingJson(_) => {
PyValueError::new_err(message)
}
#[cfg(feature = "huggingface")]
TokenizerError::PretokenizedMismatch => PyTypeError::new_err(message),
#[cfg(feature = "huggingface")]
TokenizerError::Download(_) => PyIOError::new_err(message),
}
}
}
pub(crate) enum Codec {
#[cfg(feature = "tiktoken")]
Tiktoken(TiktokenTokenizer),
@ -102,9 +166,7 @@ impl Tokenizer {
#[cfg(not(feature = "tiktoken"))]
{
let _ = (py, encoding);
Err(RustBridgeDeclined::new_err(
"tokenizer backend requires the tiktoken feature",
))
Err(TokenizerError::MissingFeature("tiktoken").into())
}
}
@ -120,9 +182,7 @@ impl Tokenizer {
#[cfg(not(feature = "huggingface"))]
{
let _ = (py, tokenizer_json);
Err(RustBridgeDeclined::new_err(
"tokenizer backend requires the huggingface feature",
))
Err(TokenizerError::MissingFeature("huggingface").into())
}
}
@ -146,16 +206,14 @@ impl Tokenizer {
.getattr("hf_hub_download")?
.call((), Some(&kwargs))?
.extract()?;
let json =
release_gil(py, || std::fs::read_to_string(path)).map_err(PyIOError::new_err)?;
let json = release_gil(py, || std::fs::read_to_string(path))
.map_err(TokenizerError::Download)?;
Self::from_json(py, &json)
}
#[cfg(not(feature = "huggingface"))]
{
let _ = (py, identifier, revision, token);
Err(RustBridgeDeclined::new_err(
"tokenizer backend requires the huggingface feature",
))
Err(TokenizerError::MissingFeature("huggingface").into())
}
}
@ -201,7 +259,7 @@ impl Tokenizer {
let tokenizer = self.tiktoken()?;
let text = self.text(text)?;
release_gil(py, || tokenizer.encode_special(&text, &allowed))
.map_err(|error| PyRuntimeError::new_err(Report::new(&error).to_string()))
.map_err(|error| TokenizerError::EncodeSpecial(error.into()).into())
}
/// tiktoken's `encode_with_unstable`: `(stable_tokens, completions)`.
@ -261,7 +319,7 @@ impl Tokenizer {
enter_native()?;
let tokenizer = self.tiktoken()?;
let bytes = release_gil(py, || tokenizer.decode_bytes(&ids))
.map_err(|error| PyKeyError::new_err(Report::new(&error).to_string()))?;
.map_err(|error| TokenizerError::DecodeBytes(error.into()))?;
Ok(PyBytes::new(py, &bytes))
}
@ -482,15 +540,16 @@ impl Tokenizer {
match *self.inner {
Codec::Tiktoken(ref tokenizer) => Ok(tokenizer),
#[cfg(feature = "huggingface")]
Codec::HuggingFace(_) => Err(PyValueError::new_err("requires a tiktoken encoding")),
Codec::HuggingFace(_) => Err(TokenizerError::NotTiktoken.into()),
}
}
#[cfg(feature = "tiktoken")]
fn vocabulary(&self) -> PyResult<&Vocabulary> {
self.tiktoken()?.vocabulary().ok_or_else(|| {
PyRuntimeError::new_err("this encoding was built without its vocabulary")
})
Ok(self
.tiktoken()?
.vocabulary()
.ok_or(TokenizerError::NoVocabulary)?)
}
#[cfg(feature = "huggingface")]
@ -498,7 +557,7 @@ impl Tokenizer {
match *self.inner {
Codec::HuggingFace(ref tokenizer) => Ok(tokenizer),
#[cfg(feature = "tiktoken")]
Codec::Tiktoken(_) => Err(PyValueError::new_err("requires a Hugging Face tokenizer")),
Codec::Tiktoken(_) => Err(TokenizerError::NotHuggingFace.into()),
}
}
}
@ -532,21 +591,21 @@ impl Sequence {
match (self, is_pretokenized) {
(Self::Text(text), false) => Ok(text.into()),
(Self::Words(words), true) => Ok(words.into()),
_ => Err(pyo3::exceptions::PyTypeError::new_err(
"input must match is_pretokenized",
)),
_ => Err(TokenizerError::PretokenizedMismatch.into()),
}
}
}
#[cfg(feature = "huggingface")]
fn direction<T>(value: &str, left: T, right: T, what: &str) -> PyResult<T> {
fn direction<T>(value: &str, left: T, right: T, what: &'static str) -> PyResult<T> {
match value {
"left" => Ok(left),
"right" => Ok(right),
other => Err(PyValueError::new_err(format!(
"invalid {what} direction {other:?}: expected 'left' or 'right'"
))),
other => Err(TokenizerError::Direction {
what,
value: other.to_owned(),
}
.into()),
}
}
@ -566,7 +625,7 @@ impl HuggingFaceEncoding {
fn new(json: Option<&str>) -> PyResult<Self> {
let inner = match json {
Some(json) => encoding_from_json(json)
.map_err(|error| PyValueError::new_err(Report::new(&error).to_string()))?,
.map_err(|error| TokenizerError::EncodingJson(error.into()))?,
None => Encoding::default(),
};
Ok(Self { inner })
@ -588,7 +647,7 @@ impl HuggingFaceEncoding {
py: Python<'py>,
) -> PyResult<(Bound<'py, pyo3::types::PyType>, (String,))> {
let json = encoding_to_json(&self.inner)
.map_err(|error| PyValueError::new_err(Report::new(&error).to_string()))?;
.map_err(|error| TokenizerError::EncodingJson(error.into()))?;
Ok((py.get_type::<Self>(), (json,)))
}

18
litellm-rust/crates/secrets-cyberark/src/error.rs
View file

@ -37,3 +37,21 @@ pub enum Error {
#[error(transparent)]
Operation(#[from] litellm_secrets_types::Error),
}
impl Error {
pub fn is_unsafe_secret_name(&self) -> bool {
match self {
Self::Operation(error) => error.is_unsafe_secret_name(),
Self::Timeout
| Self::Http(_)
| Self::AuthStatus(_)
| Self::Status(_)
| Self::MissingCredentials
| Self::ClientCertificate(_)
| Self::Client(_)
| Self::RefreshInterval(_)
| Self::Endpoint(_)
| Self::EnterpriseRequired => false,
}
}
}

24
litellm-rust/crates/secrets-hashicorp/src/error.rs
View file

@ -48,3 +48,27 @@ pub enum Error {
#[error("invalid HashiCorp Vault refresh interval")]
RefreshInterval(#[source] Option<std::num::ParseIntError>),
}
impl Error {
pub fn is_unsafe_secret_name(&self) -> bool {
match self {
Self::InvalidSecretName(error) | Self::Operation(error) => {
error.is_unsafe_secret_name()
}
Self::EnterpriseRequired
| Self::Client(_)
| Self::ClientSettings(_)
| Self::TlsIdentity { .. }
| Self::LoginStatus { .. }
| Self::MalformedLogin(_)
| Self::NoAuthConfigured
| Self::Status { .. }
| Self::MalformedPayload(_)
| Self::NonStringValue
| Self::DataKeyConflictsWithDescription
| Self::CasVersionOverflow(_)
| Self::Timeout
| Self::RefreshInterval(_) => false,
}
}
}

12
litellm-rust/crates/secrets-hashicorp/src/secret_manager/raw.rs
View file

@ -32,6 +32,18 @@ pub enum RawOperationError {
Timeout { method: String, elapsed: Duration },
}
impl RawOperationError {
pub fn is_unsafe_secret_name(&self) -> bool {
match self {
Self::Local(error) => error.is_unsafe_secret_name(),
Self::Authentication { .. }
| Self::Http { .. }
| Self::Transport(_)
| Self::Timeout { .. } => false,
}
}
}
impl HashicorpVault {
pub async fn write_raw(
&self,

12
litellm-rust/crates/secrets-types/src/error.rs
View file

@ -11,3 +11,15 @@ pub enum Error {
#[error("secret manager received an incompatible operation context")]
InvalidOperationContext,
}
impl Error {
pub fn is_unsafe_secret_name(&self) -> bool {
match self {
Self::UnsafeSecretName => true,
Self::CurrentSecretMissing
| Self::NewSecretMissing
| Self::NewSecretMismatch
| Self::InvalidOperationContext => false,
}
}
}