mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
feat(rust): one typed decline from core up to each host (#47772)
* feat(rust): one typed decline from core up to each host Rust core says it cannot serve a request with one Decline that travels unchanged through llms and inference up to the host. The Python bridge raises RustBridgeDeclined for it in route_failure before the public exception mapping, so dispatch falls back to legacy Python instead of failing the call, and the gateway answers 501. Every route collapses errors after the send with Error::after_send, which keeps a provider answer and a response fault as they are and nests anything else as the source of an invalid response, so a billed call can never read as a decline. Response-side cases that used to decline are now ErrorDetail::Unhandled response faults. The rust-error skill documents declined() as the first question both boundaries ask, and the Python suites pin that a native decline reaches the dispatcher before any provider IO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(rust): keep post-provider hook failures terminal --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
parent
7b60fd061a
commit
323d3956d6
43 changed files with 558 additions and 345 deletions
|
|
@ -23,6 +23,7 @@ description: Define, wrap, convert, classify, report or test Rust error types in
|
|||
- Classification states facts. Retry and fallback policy lives with the caller that knows idempotency
|
||||
- [boundaries](references/boundaries.md)
|
||||
- An error becomes text for someone else only in one mapping function per error type, in the gateway crates, the Python bridge and host-python. The inference gateway and the bridge decide by the same predicates
|
||||
- A failure before the provider was called is a decline. Both ask `declined()` first and hand it back to the host unchanged
|
||||
- [gateway](references/gateway.md)
|
||||
- One `Error` with `IntoResponse` per gateway crate that serves a router. Every failure flows through the same `From` chain, is classified once in `status()` and rendered by surface and framing
|
||||
- [pyo3](references/pyo3.md)
|
||||
|
|
|
|||
|
|
@ -12,11 +12,11 @@ A boundary is where an error becomes something another program reads: an HTTP re
|
|||
|
||||
## Asking the nested error
|
||||
|
||||
The inference gateway and the Python bridge's route mapping both receive a nested route error and decide by the same predicates, so a new failure lands correctly on both once its predicates answer right
|
||||
The inference gateway and the Python bridge's route mapping both receive a nested route error and decide by the same predicates in the same order, so a new failure lands correctly on both once its predicates answer right
|
||||
|
||||
| Predicate | Gateway | Bridge |
|
||||
|---|---|---|
|
||||
| `is_unsupported()` | 501 | `ValueError`, since `is_request()` also answers true |
|
||||
| `declined()` | 501 | `RustBridgeDeclined`, without the route's public mapping |
|
||||
| `upstream_response()` | the provider's status and body | `RustUpstreamError` |
|
||||
| `is_missing_credential()` | 401 | no class of its own, a `RuntimeError` because `is_request()` answers false |
|
||||
| `is_request()` | 400 | `ValueError` |
|
||||
|
|
@ -26,6 +26,12 @@ If you find yourself adding an arm for your variant at a boundary, check whether
|
|||
|
||||
The two boundaries are not identical. The gateway has variants of its own for body parsing, an unknown model and an unsupported path, and the OCR bridge has Python parity arms such as `FileNotFoundError` for a missing file. Those arms match the boundary's own variants or the top level of a public enum, never a nested error's variants. A missing API key is 401 at the gateway and `RuntimeError` at both bridges, pinned by tests to what Python's own path does
|
||||
|
||||
## Declines
|
||||
|
||||
A failure before anything reached the provider is a decline, and `declined()` on the route error names its reason. It means another host may serve the same request as it was received, so each boundary hands it to the host unchanged: the gateway answers 501 and the Python bridge raises `RustBridgeDeclined`, never a public exception or a `ValueError`. See [gateway](gateway.md) and [pyo3](pyo3.md) for each host's side
|
||||
|
||||
Anything after the send is terminal, never a decline. `declined()` answers `None` for a response fault and for a post-call hook failure even when the wrapped error was a decline, and every route that transforms the provider's answer collapses its errors with `Error::after_send`, which keeps a provider answer and a response fault as they are and nests anything else as the source of an invalid response, so a decline can never describe a call that was billed. Ask `declined()`, do not match the variant behind it, because that after-send cut lives in the predicate
|
||||
|
||||
## What reaches the client
|
||||
|
||||
A boundary renders anything that is not a provider answer with `litellm_error::Report`, so the reader sees every layer of the chain once. That is why messages are written for a reader, each layer leaves its cause to `source()`, and reqwest URLs are stripped on entry. A boundary that must not show the chain, such as the bridge's TLS settings error that names the setting but not the file, renders fixed text of its own and says so in its test
|
||||
|
|
|
|||
|
|
@ -13,9 +13,9 @@ Downstream crates ask a lower error questions instead of matching its nested var
|
|||
| `status()` | an HTTP status to report |
|
||||
| `is_connect()` | the connection was never made, for a host's own connect-error mapping |
|
||||
| `tls_source()` | which TLS setting to blame in a config error |
|
||||
| `is_computed_header()` | the caller forwarded a header the signer owns, which `litellm_inference::Error` turns into `Unsupported` |
|
||||
| `is_computed_header()` | the caller forwarded a header the signer owns, which `litellm_inference::Error` turns into a decline |
|
||||
|
||||
Upper types add the questions their callers ask. `litellm_inference::Error` and the OCR error answer `is_missing_credential()` and `is_unsupported()`, which the gateway asks instead of reaching into `Auth` or `Unsupported`
|
||||
Upper types add the questions their callers ask. `litellm_inference::Error` and the OCR error answer `is_missing_credential()`, and `litellm_llms::Error` and `litellm_inference::Error` answer `declined()`, which the boundaries ask first instead of reaching into `Auth` or `Declined`, see [boundaries](boundaries.md)
|
||||
|
||||
```rust
|
||||
// reaches through two crates' variants and breaks when either changes
|
||||
|
|
@ -39,7 +39,7 @@ pub fn is_request(&self) -> bool {
|
|||
|
||||
Add a predicate when a real caller needs a new distinction, and name it after the fact the caller needs, in the caller's words, not after the variant's spelling. An upper type's predicates call through to the nested error, as in `Self::Http(error) => error.is_request()` and `Self::Auth(error) => error.is_missing_credential()`
|
||||
|
||||
A public enum may be matched one level down by the type that owns the variant, such as the gateway matching its own `Unsupported` or the OCR bridge matching `FileRead`. Going a second level down, into a variant of a variant, is what to avoid, the gateway used to do that for `MissingApiKey` and `Unsupported`
|
||||
A public enum may be matched one level down by the type that owns the variant, such as the gateway matching its own `Unsupported` or the OCR bridge matching `FileRead`. Going a second level down, into a variant of a variant, is what to avoid, the gateway used to do that for `MissingApiKey` and the old `Unsupported` route variant
|
||||
|
||||
## Facts, not policy
|
||||
|
||||
|
|
|
|||
|
|
@ -56,7 +56,7 @@ When a lower error needs a different meaning for every caller, do it in the uppe
|
|||
impl From<litellm_http::Error> for Error {
|
||||
fn from(error: litellm_http::Error) -> Self {
|
||||
if error.is_computed_header() {
|
||||
return Self::Unsupported("request forwards a header AWS SigV4 computes");
|
||||
return Self::Declined(Decline::new("request forwards a header AWS SigV4 computes"));
|
||||
}
|
||||
Self::Http(error)
|
||||
}
|
||||
|
|
@ -70,7 +70,7 @@ All four inference routes classify a header the signer computes the same way, be
|
|||
| The upper layer | Do | Example |
|
||||
|---|---|---|
|
||||
| has nothing to add | nest in one transparent variant and delegate predicates | `Self::Http(error) => error.is_request()` |
|
||||
| changes what the failure means | translate to an upper variant, keep the cause attached when it helps | computed header -> `Unsupported` |
|
||||
| changes what the failure means | translate to an upper variant, keep the cause attached when it helps | computed header -> `Declined` |
|
||||
| knows something the lower one does not | a variant with the extra fields plus the cause as its source | `Read { path, tls_source, source }` |
|
||||
|
||||
Never re-declare a lower crate's variants in the upper type with an upper-layer name and a 1:1 mapping function. OCR once did that for every media download failure, `MediaError::BlockedUrl => Error::BlockedDocumentUrl` and so on, which doubled every change and let the two lists drift.
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ Each hop is one `From` that nests the lower error in one variant, see [conversio
|
|||
|
||||
| Answer | Status |
|
||||
|---|---|
|
||||
| `is_unsupported()` | 501, the gateway matched and parsed the request but core cannot serve it, nothing reached the provider |
|
||||
| `declined()` | 501, for a route error, the gateway matched and parsed the request but core cannot serve it, nothing reached the provider |
|
||||
| `upstream_response()` | the provider's own status, 502 when it is not a valid code |
|
||||
| `is_missing_credential()` | 401 |
|
||||
| `is_request()` | 400, for a route error |
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ The bridge builds with pyo3's `abi3-py310` feature. Under abi3, `#[pyclass(exten
|
|||
|
||||
## Mapping order
|
||||
|
||||
A Python exception raised inside the call, such as a secret manager callback's, is returned first as it is. Then the provider's answer passes through as `RustUpstreamError`. On `/v1/messages` a request the route rejected becomes a `ValueError` flagged with `messages_request_error`. Everything else is split by `is_request()` into `ValueError` or `RuntimeError`, with the message rendered by `litellm_error::Report`. A missing API key is a `RuntimeError` because `is_request()` answers false for it, and `missing_api_key_stays_a_runtime_error_while_other_auth_failures_are_value_errors` pins that
|
||||
A core decline is returned first, as `RustBridgeDeclined` with the decline's reason as its only arg, without calling the route's public mapping. Then a Python exception raised inside the call, such as a secret manager callback's, is returned as it is. Then the provider's answer passes through as `RustUpstreamError`. On `/v1/messages` a request the route rejected becomes a `ValueError` flagged with `messages_request_error`. Everything else is split by `is_request()` into `ValueError` or `RuntimeError`, with the message rendered by `litellm_error::Report`. A missing API key is a `RuntimeError` because `is_request()` answers false for it, and `missing_api_key_stays_a_runtime_error_while_other_auth_failures_are_value_errors` pins that
|
||||
|
||||
OCR puts Python parity arms in front of that order, each matching a top-level variant of its own public enum
|
||||
|
||||
|
|
@ -47,7 +47,9 @@ Use the standard class whose Python meaning matches. `PyFileNotFoundError` for a
|
|||
|
||||
## Declining
|
||||
|
||||
`RustBridgeDeclined` means no provider call was made and the host may retry on its own path. It is raised before dispatch, by the tokenizer, embeddings and token counter routes. Never raise it for a failure after the request went out, the host would repeat a billed call. `route_failures_are_terminal` in `errors.rs` pins that for the inference routes
|
||||
`RustBridgeDeclined` means no provider call was made and the host may run its own path for the same call. The tokenizer, embeddings and token counter routes raise it from their own pre-checks, and `route_failure` raises it for a route error whose `declined()` answers, so a decline from deep inside a transformation reaches Python the same way. Never raise it for a failure after the request went out, the host would repeat a billed call. `a_core_decline_hands_the_call_back_to_python_before_the_public_mapping` and `route_failures_are_terminal` in `errors.rs` pin both sides
|
||||
|
||||
Route hosts pass a terminal failure through the Python route module's `map_failure`, which runs `litellm.exception_type` and returns the public exception a Python caller expects. A decline must skip that step. Once wrapped it looks like a provider failure to the dispatcher and the host no longer falls back, so `route_failure` returns it before it calls `public`
|
||||
|
||||
## Rules
|
||||
|
||||
|
|
|
|||
16
litellm-rust/crates/core-utils/src/decline.rs
Normal file
16
litellm-rust/crates/core-utils/src/decline.rs
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
/// Rust core cannot serve this request, and nothing was sent to the provider yet.
|
||||
///
|
||||
/// Every error on the call path carries it unchanged, so the host can hand the same request
|
||||
/// to another implementation: the SDK runs legacy Python, the gateway forwards to the proxy.
|
||||
/// Raise it only before the provider call. Anything after the send is terminal.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)]
|
||||
#[error("declined by the rust path: {reason}")]
|
||||
pub struct Decline {
|
||||
pub reason: &'static str,
|
||||
}
|
||||
|
||||
impl Decline {
|
||||
pub const fn new(reason: &'static str) -> Self {
|
||||
Self { reason }
|
||||
}
|
||||
}
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
pub mod call_arguments;
|
||||
pub mod core_helpers;
|
||||
pub mod decline;
|
||||
pub mod dot_notation_indexing;
|
||||
pub mod exception_mapping_utils;
|
||||
pub mod get_llm_provider_logic;
|
||||
|
|
|
|||
|
|
@ -149,7 +149,7 @@ impl Error {
|
|||
}
|
||||
|
||||
fn route_status(error: &InferenceError) -> StatusCode {
|
||||
if error.is_unsupported() {
|
||||
if error.declined().is_some() {
|
||||
return StatusCode::NOT_IMPLEMENTED;
|
||||
}
|
||||
if let Some((status, _)) = error.upstream_response() {
|
||||
|
|
@ -165,9 +165,6 @@ fn route_status(error: &InferenceError) -> StatusCode {
|
|||
}
|
||||
|
||||
fn ocr_status(error: &OcrError) -> StatusCode {
|
||||
if error.is_unsupported() {
|
||||
return StatusCode::NOT_IMPLEMENTED;
|
||||
}
|
||||
if let Some((status, ..)) = error.upstream_response() {
|
||||
return StatusCode::from_u16(status).unwrap_or(StatusCode::BAD_GATEWAY);
|
||||
}
|
||||
|
|
@ -326,6 +323,12 @@ mod tests {
|
|||
|
||||
#[rstest]
|
||||
#[case::upstream_status(upstream(429, ""), StatusCode::TOO_MANY_REQUESTS)]
|
||||
#[case::declined(
|
||||
Error::Route(InferenceError::from(litellm_http::Error::computed_header(
|
||||
"authorization"
|
||||
))),
|
||||
StatusCode::NOT_IMPLEMENTED
|
||||
)]
|
||||
#[case::rejected_request(
|
||||
Error::Route(InferenceError::invalid_request(litellm_llms::ErrorDetail::Expected {
|
||||
subject: "top_k",
|
||||
|
|
|
|||
|
|
@ -66,11 +66,10 @@ where
|
|||
MachineStep::Suspended(request) => request,
|
||||
};
|
||||
let answered = match request {
|
||||
HostRequest::Intercept(InterceptRequest::ResultReady { facts, reply }) => self
|
||||
.interceptors
|
||||
.result_ready(facts)
|
||||
.await
|
||||
.map(|()| reply.send(())),
|
||||
HostRequest::Intercept(InterceptRequest::ResultReady { facts, reply }) => {
|
||||
reply.send(self.interceptors.result_ready(facts).await);
|
||||
Ok(())
|
||||
}
|
||||
HostRequest::HostCall(call) => self.services.handle_host_call(call).await,
|
||||
HostRequest::Intercept(InterceptRequest::BeforeProviderRequest {
|
||||
wire,
|
||||
|
|
@ -82,10 +81,8 @@ where
|
|||
.await
|
||||
.map(|wire| reply.send(wire)),
|
||||
HostRequest::Intercept(InterceptRequest::AfterProviderResponse { raw, reply }) => {
|
||||
self.interceptors
|
||||
.after_provider_response(raw)
|
||||
.await
|
||||
.map(|()| reply.send(()))
|
||||
reply.send(self.interceptors.after_provider_response(raw).await);
|
||||
Ok(())
|
||||
}
|
||||
HostRequest::Stream(StreamDelivery::Open(head, reply)) => {
|
||||
self.demand = Some(reply);
|
||||
|
|
|
|||
|
|
@ -299,7 +299,7 @@ where
|
|||
facts,
|
||||
}))
|
||||
.await?;
|
||||
reply.send(());
|
||||
reply.send(Ok(()));
|
||||
Ok(())
|
||||
}
|
||||
HostRequest::Intercept(InterceptRequest::AfterProviderResponse { raw, reply }) => {
|
||||
|
|
@ -307,7 +307,7 @@ where
|
|||
ExecutionEvent::ProviderResponseReceived { raw: &raw },
|
||||
))
|
||||
.await?;
|
||||
reply.send(());
|
||||
reply.send(Ok(()));
|
||||
Ok(())
|
||||
}
|
||||
HostRequest::Stream(StreamDelivery::Open(head, reply)) => {
|
||||
|
|
|
|||
|
|
@ -212,7 +212,7 @@ mod tests {
|
|||
panic!("on_event yields Emit");
|
||||
};
|
||||
assert_eq!(raw.body, "raw");
|
||||
reply.send(());
|
||||
reply.send(Ok(()));
|
||||
|
||||
let Ok(MachineStep::Complete((sent, ()))) = machine.resume().await else {
|
||||
panic!("the call completes with the answers");
|
||||
|
|
|
|||
|
|
@ -109,7 +109,7 @@ where
|
|||
.request_reply(|reply| {
|
||||
HostRequest::Intercept(InterceptRequest::ResultReady { facts, reply })
|
||||
})
|
||||
.await
|
||||
.await?
|
||||
}
|
||||
|
||||
async fn after_provider_response(&self, raw: RawResponse) -> Result<(), P::Error> {
|
||||
|
|
@ -117,7 +117,7 @@ where
|
|||
.request_reply(|reply| {
|
||||
HostRequest::Intercept(InterceptRequest::AfterProviderResponse { raw, reply })
|
||||
})
|
||||
.await
|
||||
.await?
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -15,14 +15,14 @@ pub trait Protocol: Send + Sync + 'static {
|
|||
|
||||
pub enum HostRequest<P: Protocol> {
|
||||
HostCall(P::HostCall),
|
||||
Intercept(InterceptRequest),
|
||||
Intercept(InterceptRequest<P::Error>),
|
||||
Stream(StreamDelivery<P>),
|
||||
}
|
||||
|
||||
pub enum InterceptRequest {
|
||||
pub enum InterceptRequest<E> {
|
||||
ResultReady {
|
||||
facts: crate::interceptors::ExecutionFacts,
|
||||
reply: Reply<()>,
|
||||
reply: Reply<Result<(), E>>,
|
||||
},
|
||||
BeforeProviderRequest {
|
||||
wire: Box<WireRequest>,
|
||||
|
|
@ -31,7 +31,7 @@ pub enum InterceptRequest {
|
|||
},
|
||||
AfterProviderResponse {
|
||||
raw: RawResponse,
|
||||
reply: Reply<()>,
|
||||
reply: Reply<Result<(), E>>,
|
||||
},
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -63,7 +63,7 @@ pub(super) fn resolve_request(
|
|||
));
|
||||
}
|
||||
if let Some(reason) = config.unsupported_reason(&messages, &call.optional_params) {
|
||||
return Err(Error::Unsupported(reason.0));
|
||||
return Err(Error::Declined(reason));
|
||||
}
|
||||
Ok(ResolvedChatCompletionsRequest {
|
||||
messages,
|
||||
|
|
|
|||
|
|
@ -128,31 +128,12 @@ pub(super) async fn execute(
|
|||
config
|
||||
.transform_response(provider.model, ProviderChatResponseData { body })
|
||||
.map_err(Error::from)
|
||||
.map_err(as_response_error)
|
||||
.map_err(Error::after_send)
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Re-tag an error raised while normalizing a response the provider already
|
||||
/// returned.
|
||||
///
|
||||
/// A config reports the same variants on either side of the call: a missing
|
||||
/// field or an unsupported block can mean "this request cannot be translated"
|
||||
/// before the request is sent and "this response cannot be normalized" here. Only the
|
||||
/// second kind has already been billed, and a host that keeps a reference
|
||||
/// implementation must not retry those, so collapse them to one variant that
|
||||
/// can only mean the provider was already called.
|
||||
pub(super) fn as_response_error(err: Error) -> Error {
|
||||
if err.is_response() || err.upstream_response().is_some() {
|
||||
return err;
|
||||
}
|
||||
Error::invalid_response(litellm_llms::ErrorDetail::invalid(
|
||||
"chat completions response",
|
||||
err,
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::sync::Mutex;
|
||||
|
|
@ -313,40 +294,4 @@ mod tests {
|
|||
);
|
||||
assert!(interceptors.raw.into_inner().unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
#[case::missing_field(litellm_llms::Error::MissingField("usage").into())]
|
||||
#[case::unsupported(Error::Unsupported("non-text response content block"))]
|
||||
#[case::invalid_request(Error::invalid_request(litellm_llms::ErrorDetail::Expected {
|
||||
subject: "whatever",
|
||||
expected: "anything else",
|
||||
}))]
|
||||
#[case::auth(Error::Auth(litellm_auth::Error::InvalidHeader))]
|
||||
fn response_errors_collapse_to_a_fault_that_can_only_mean_already_sent(
|
||||
#[case] original: Error,
|
||||
) {
|
||||
let collapsed = as_response_error(original);
|
||||
assert!(collapsed.is_response(), "{collapsed:?}");
|
||||
assert!(!collapsed.is_request(), "{collapsed:?}");
|
||||
assert!(!collapsed.is_unsupported(), "{collapsed:?}");
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
fn a_response_fault_is_not_wrapped_twice() {
|
||||
let original = Error::invalid_response(litellm_llms::ErrorDetail::MissingField {
|
||||
subject: "response",
|
||||
field: "usage",
|
||||
});
|
||||
let report = litellm_error::Report::new(&as_response_error(original)).to_string();
|
||||
assert!(!report.contains("chat completions response"), "{report}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_provider_answer_keeps_its_status_and_body() {
|
||||
let upstream = Error::from(litellm_http::Error::upstream(429, "slow down".into()));
|
||||
assert_eq!(
|
||||
as_response_error(upstream).upstream_response(),
|
||||
Some((429, "slow down"))
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -354,7 +354,7 @@ async fn rejects_a_forwarded_header_the_signer_computes(#[case] forwarded: &str)
|
|||
let error = complete(call)
|
||||
.await
|
||||
.expect_err("conflicting signing headers fail");
|
||||
assert!(matches!(error, Error::Unsupported(_)), "{error:?}");
|
||||
assert!(error.declined().is_some(), "{error:?}");
|
||||
assert!(received(&upstream).await.is_empty());
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -316,6 +316,7 @@ fn decode_response(
|
|||
config
|
||||
.transform_anthropic_messages_response(model, response)
|
||||
.map_err(Error::from)
|
||||
.map_err(Error::after_send)
|
||||
}
|
||||
|
||||
fn streaming_response(
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_host::lifecycle::ExecutionEvent;
|
||||
use std::sync::Mutex;
|
||||
|
||||
|
|
@ -23,6 +24,7 @@ struct RecordingHost {
|
|||
optional_params: Mutex<Vec<Value>>,
|
||||
facts: Mutex<Vec<ExecutionFacts>>,
|
||||
reject_result: bool,
|
||||
reject_response: bool,
|
||||
}
|
||||
|
||||
impl RecordingHost {
|
||||
|
|
@ -34,6 +36,7 @@ impl RecordingHost {
|
|||
optional_params: Mutex::new(Vec::new()),
|
||||
facts: Mutex::new(Vec::new()),
|
||||
reject_result: false,
|
||||
reject_response: false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -81,7 +84,7 @@ impl litellm_host::interceptors::Interceptors<<Messages as litellm_host::protoco
|
|||
async fn result_ready(&self, facts: ExecutionFacts) -> Result<(), Error> {
|
||||
self.facts.lock().unwrap().push(facts);
|
||||
if self.reject_result {
|
||||
return Err(Error::Unsupported("result rejected"));
|
||||
return Err(Error::Declined(Decline::new("result rejected")));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -101,6 +104,9 @@ impl litellm_host::interceptors::Interceptors<<Messages as litellm_host::protoco
|
|||
&self,
|
||||
raw: litellm_host::interceptors::RawResponse,
|
||||
) -> Result<(), <Messages as litellm_host::protocol::Protocol>::Error> {
|
||||
if self.reject_response {
|
||||
return Err(Error::Declined(Decline::new("response rejected")));
|
||||
}
|
||||
litellm_host::lifecycle::CallObserver::observe(
|
||||
self,
|
||||
litellm_host::lifecycle::CallEvent::Execution(
|
||||
|
|
@ -121,6 +127,7 @@ async fn rejected_results_are_not_delivered_or_cached(
|
|||
call: MessagesCall,
|
||||
#[case] streaming: bool,
|
||||
#[case] hosted: bool,
|
||||
#[values(false, true)] cached: bool,
|
||||
) {
|
||||
use futures_util::TryStreamExt;
|
||||
use litellm_cache_memory::InMemoryCache;
|
||||
|
|
@ -142,13 +149,8 @@ async fn rejected_results_are_not_delivered_or_cached(
|
|||
)))),
|
||||
CacheScope::Shared,
|
||||
));
|
||||
for (reject, expected_requests, cached) in [
|
||||
(true, 1, false),
|
||||
(false, 2, false),
|
||||
(true, 2, true),
|
||||
(false, 2, true),
|
||||
] {
|
||||
let request = authenticated(
|
||||
let request = || {
|
||||
authenticated(
|
||||
with_fields(
|
||||
MessagesCall {
|
||||
body: call.body.clone(),
|
||||
|
|
@ -157,36 +159,41 @@ async fn rejected_results_are_not_delivered_or_cached(
|
|||
json!({"stream": streaming}),
|
||||
),
|
||||
upstream.uri(),
|
||||
);
|
||||
let host = RecordingHost {
|
||||
reject_result: reject,
|
||||
..RecordingHost::passthrough(request)
|
||||
};
|
||||
let result = if hosted {
|
||||
litellm_host_native::in_process::run_hosted(
|
||||
route.clone().machine(host.request().unwrap(), None),
|
||||
host.runtime(),
|
||||
)
|
||||
)
|
||||
};
|
||||
let host = RecordingHost {
|
||||
reject_result: true,
|
||||
..RecordingHost::passthrough(request())
|
||||
};
|
||||
if cached {
|
||||
let output = route.execute(request(), &(), None).await.unwrap();
|
||||
if let MessagesCallResponse::Stream { chunks, .. } = output {
|
||||
chunks.try_collect::<Vec<_>>().await.unwrap();
|
||||
}
|
||||
}
|
||||
let result = if hosted {
|
||||
litellm_host_native::in_process::run_hosted(
|
||||
route.clone().machine(host.request().unwrap(), None),
|
||||
host.runtime(),
|
||||
)
|
||||
.await
|
||||
.map(|_| ())
|
||||
} else {
|
||||
route
|
||||
.execute(host.request().unwrap(), &host, None)
|
||||
.await
|
||||
.map(|_| ())
|
||||
} else {
|
||||
match route.execute(host.request().unwrap(), &host, None).await {
|
||||
Ok(MessagesCallResponse::Complete(_)) => Ok(()),
|
||||
Ok(MessagesCallResponse::Stream { chunks, .. }) => {
|
||||
chunks.try_collect::<Vec<_>>().await.map(|_| ())
|
||||
}
|
||||
Err(error) => Err(error),
|
||||
}
|
||||
};
|
||||
assert_eq!(
|
||||
result,
|
||||
if reject {
|
||||
Err(Error::Unsupported("result rejected"))
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
);
|
||||
assert_eq!(received(&upstream).await.len(), expected_requests);
|
||||
};
|
||||
let error = result.unwrap_err();
|
||||
assert_eq!(error.declined().is_some(), cached);
|
||||
assert_eq!(error.dispatched(), !cached);
|
||||
assert!(
|
||||
litellm_error::Report::new(&error)
|
||||
.to_string()
|
||||
.contains("result rejected")
|
||||
);
|
||||
assert_eq!(received(&upstream).await.len(), 1);
|
||||
{
|
||||
let facts = host.facts.lock().unwrap();
|
||||
assert_eq!(facts.len(), 1);
|
||||
assert_eq!(
|
||||
|
|
@ -194,6 +201,46 @@ async fn rejected_results_are_not_delivered_or_cached(
|
|||
cached
|
||||
);
|
||||
}
|
||||
let output = route.execute(request(), &(), None).await.unwrap();
|
||||
if let MessagesCallResponse::Stream { chunks, .. } = output {
|
||||
chunks.try_collect::<Vec<_>>().await.unwrap();
|
||||
}
|
||||
assert_eq!(received(&upstream).await.len(), if cached { 1 } else { 2 });
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::native(false)]
|
||||
#[case::hosted(true)]
|
||||
#[tokio::test]
|
||||
async fn response_hook_declines_are_terminal(call: MessagesCall, #[case] hosted: bool) {
|
||||
let upstream = upstream([message_response()]).await;
|
||||
let host = RecordingHost {
|
||||
reject_response: true,
|
||||
..RecordingHost::passthrough(authenticated(call, upstream.uri()))
|
||||
};
|
||||
let route = messages_route(no_secrets());
|
||||
let result = if hosted {
|
||||
litellm_host_native::in_process::run_hosted(
|
||||
route.machine(host.request().unwrap(), None),
|
||||
host.runtime(),
|
||||
)
|
||||
.await
|
||||
.map(|_| ())
|
||||
} else {
|
||||
route
|
||||
.execute(host.request().unwrap(), &host, None)
|
||||
.await
|
||||
.map(|_| ())
|
||||
};
|
||||
let error = result.unwrap_err();
|
||||
assert!(error.declined().is_none());
|
||||
assert!(error.dispatched());
|
||||
assert!(
|
||||
litellm_error::Report::new(&error)
|
||||
.to_string()
|
||||
.contains("response rejected")
|
||||
);
|
||||
assert_eq!(received(&upstream).await.len(), 1);
|
||||
}
|
||||
|
||||
async fn run_through(host: &RecordingHost) -> Result<MessagesOutput, Error> {
|
||||
|
|
|
|||
|
|
@ -41,11 +41,10 @@ async fn drive_until(
|
|||
Err(error) => break Err(error),
|
||||
};
|
||||
let answer = match op {
|
||||
HostRequest::Intercept(InterceptRequest::ResultReady { facts, reply }) => host
|
||||
.result_ready(facts)
|
||||
.await
|
||||
.map(|()| reply.send(()))
|
||||
.map_err(HostFailure::Error),
|
||||
HostRequest::Intercept(InterceptRequest::ResultReady { facts, reply }) => {
|
||||
reply.send(host.result_ready(facts).await);
|
||||
Ok(())
|
||||
}
|
||||
HostRequest::Stream(stream) => match stream {
|
||||
litellm_host::protocol::StreamDelivery::Open(head, _) => match head {},
|
||||
litellm_host::protocol::StreamDelivery::Chunk(chunk, _) => match chunk {},
|
||||
|
|
@ -64,10 +63,8 @@ async fn drive_until(
|
|||
}
|
||||
HostRequest::Intercept(InterceptRequest::AfterProviderResponse { raw, reply }) => {
|
||||
ops.push("response");
|
||||
host.after_provider_response(raw)
|
||||
.await
|
||||
.map(|()| reply.send(()))
|
||||
.map_err(HostFailure::Error)
|
||||
reply.send(host.after_provider_response(raw).await);
|
||||
Ok(())
|
||||
}
|
||||
};
|
||||
if let Err(failure) = answer {
|
||||
|
|
@ -85,10 +82,10 @@ async fn drive_until_notified(machine: &mut OcrMachine, host: &LocalOcrHost, sto
|
|||
_ = stop.notified() => break,
|
||||
step = machine.resume() => {
|
||||
match step.unwrap() {
|
||||
MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::ResultReady { reply, .. })) => reply.send(()),
|
||||
MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::ResultReady { reply, .. })) => reply.send(Ok(())),
|
||||
MachineStep::Suspended(HostRequest::HostCall(op)) => host.handle_host_call(op).await.unwrap(),
|
||||
MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::BeforeProviderRequest { wire, reply, .. })) => reply.send(*wire),
|
||||
MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::AfterProviderResponse { reply, .. })) => reply.send(()),
|
||||
MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::AfterProviderResponse { reply, .. })) => reply.send(Ok(())),
|
||||
MachineStep::Suspended(HostRequest::Stream(stream)) => match stream {
|
||||
litellm_host::protocol::StreamDelivery::Open(head, _) => match head {},
|
||||
litellm_host::protocol::StreamDelivery::Chunk(chunk, _) => match chunk {},
|
||||
|
|
|
|||
|
|
@ -134,6 +134,7 @@ pub(super) async fn execute(
|
|||
.transform_response_api_response(value)
|
||||
.map(ResponsesOutput::Complete)
|
||||
.map_err(Error::from)
|
||||
.map_err(Error::after_send)
|
||||
},
|
||||
)
|
||||
.await
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ mod handler;
|
|||
use std::sync::Arc;
|
||||
|
||||
use litellm_auth::AuthServices;
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_core_utils::get_llm_provider_logic::{
|
||||
CustomLlmProvider, LlmProviders, get_custom_llm_provider,
|
||||
};
|
||||
|
|
@ -24,6 +25,8 @@ use litellm_llms::{
|
|||
use litellm_secrets::source::SecretSource;
|
||||
use types::{ResponsesCall, ResponsesOutput, ResponsesProvider};
|
||||
|
||||
const NOT_NATIVE: Decline = Decline::new("native HTTP responses provider");
|
||||
|
||||
fn responses_provider(provider: LlmProviders) -> Option<&'static dyn BaseResponsesApiConfig> {
|
||||
match provider {
|
||||
LlmProviders::Openai => Some(&OpenAiResponsesApiConfig),
|
||||
|
|
@ -111,9 +114,8 @@ impl ResponsesRoute {
|
|||
}
|
||||
let provider = provider_name
|
||||
.parse::<LlmProviders>()
|
||||
.map_err(|_| Error::Unsupported("native HTTP responses provider"))?;
|
||||
let config = responses_provider(provider)
|
||||
.ok_or(Error::Unsupported("native HTTP responses provider"))?;
|
||||
.map_err(|_| Error::Declined(NOT_NATIVE))?;
|
||||
let config = responses_provider(provider).ok_or(Error::Declined(NOT_NATIVE))?;
|
||||
let wire_model = match provider {
|
||||
LlmProviders::Openai if resolved_model.contains('/') => {
|
||||
return Err(Error::InvalidProvider(call.model));
|
||||
|
|
@ -123,7 +125,7 @@ impl ResponsesRoute {
|
|||
format!("openrouter/{resolved_model}")
|
||||
}
|
||||
LlmProviders::Openrouter => resolved_model.into(),
|
||||
_ => return Err(Error::Unsupported("native HTTP responses provider")),
|
||||
_ => return Err(Error::Declined(NOT_NATIVE)),
|
||||
};
|
||||
let provider = ResponsesProvider {
|
||||
identity: ProviderIdentity {
|
||||
|
|
|
|||
|
|
@ -103,10 +103,7 @@ async fn unsupported_providers_fail_before_transport(
|
|||
)
|
||||
.await;
|
||||
|
||||
assert!(matches!(
|
||||
result,
|
||||
Err(Error::Unsupported("native HTTP responses provider"))
|
||||
));
|
||||
assert!(matches!(result, Err(ref error) if error.declined().is_some()));
|
||||
assert!(received(&upstream).await.is_empty());
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -67,7 +67,8 @@ pub(super) async fn execute(
|
|||
error,
|
||||
))
|
||||
})?;
|
||||
Ok(config
|
||||
.transform_audio_transcription_response(provider.model, response_json)?
|
||||
.into_json())
|
||||
config
|
||||
.transform_audio_transcription_response(provider.model, response_json)
|
||||
.map(|response| response.into_json())
|
||||
.map_err(|error| Error::from(error).after_send())
|
||||
}
|
||||
|
|
|
|||
|
|
@ -56,7 +56,7 @@
|
|||
- The [rust-error skill](../../.agents/skills/rust-error/SKILL.md) shapes each crate's error. This section decides which crate and module a failure belongs to
|
||||
- A failure is declared once, by the lowest crate that raises it
|
||||
- every crate above nests it in one variant and delegates its predicates, as `Error::Llm`, `Error::Auth` and `Error::Http` do, or translates it once in the shared `From` when its meaning changes, as `From<litellm_llms::Error>` does for `Unsupported` and `From<litellm_http::Error>` does for a header the signer computes
|
||||
- `Error` collects route failures for every format crate and never re-declares a variant a lower crate raises. Routes build transformation failures with `Error::invalid_request` and `Error::invalid_response`, and callers ask `is_unsupported()`, `is_request()`, `is_response()`, `is_missing_credential()`, `dispatched()`, `upstream_response()` and `rejected_request()` instead of matching nested variants
|
||||
- `Error` collects route failures for every format crate and never re-declares a variant a lower crate raises. Routes build transformation failures with `Error::invalid_request` and `Error::invalid_response` and collapse anything raised after the send with `Error::after_send`, and callers ask `declined()`, `is_request()`, `is_response()`, `is_missing_credential()`, `dispatched()`, `upstream_response()` and `rejected_request()` instead of matching nested variants
|
||||
- Scope follows the concept, not the first caller
|
||||
- an error type under `litellm-llms`'s `<provider>/` is private to that provider: no other provider and nothing in `base_llm` may import it
|
||||
- a failure two providers or two routes can hit (wire framing, stream event decoding, a malformed provider response) belongs to the crate that owns the concept: `litellm-framer` for framing, `litellm_llms::Error` for the transformation layer
|
||||
|
|
|
|||
|
|
@ -377,5 +377,66 @@ async fn publish(
|
|||
facts: facts.clone(),
|
||||
}));
|
||||
}
|
||||
interceptors.result_ready(facts).await
|
||||
let from_provider = facts.source == ResultSource::Provider;
|
||||
interceptors.result_ready(facts).await.map_err(|error| {
|
||||
if from_provider {
|
||||
Error::post_call(error)
|
||||
} else {
|
||||
error
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_host::interceptors::{RawResponse, RequestContext, WireRequest};
|
||||
|
||||
use super::*;
|
||||
|
||||
struct RejectResult;
|
||||
|
||||
impl Interceptors<Error> for RejectResult {
|
||||
async fn before_provider_request(
|
||||
&self,
|
||||
wire: WireRequest,
|
||||
_: RequestContext,
|
||||
) -> Result<WireRequest, Error> {
|
||||
Ok(wire)
|
||||
}
|
||||
|
||||
async fn after_provider_response(&self, _: RawResponse) -> Result<(), Error> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn result_ready(&self, _: ExecutionFacts) -> Result<(), Error> {
|
||||
Err(Error::Declined(Decline::new("result rejected")))
|
||||
}
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
#[case::provider(ResultSource::Provider, false)]
|
||||
#[case::cache(ResultSource::Cache { key: "cached".into() }, true)]
|
||||
#[tokio::test]
|
||||
async fn result_rejection_only_declines_without_a_provider_call(
|
||||
#[case] source: ResultSource,
|
||||
#[case] may_decline: bool,
|
||||
) {
|
||||
let facts = ExecutionFacts {
|
||||
provider: ProviderIdentity {
|
||||
model: "model".into(),
|
||||
provider: "provider".into(),
|
||||
},
|
||||
cost: Cost::settle(&source, None),
|
||||
source,
|
||||
};
|
||||
let error = publish(facts, &RejectResult, None).await.unwrap_err();
|
||||
assert_eq!(error.declined().is_some(), may_decline);
|
||||
assert_eq!(error.dispatched(), !may_decline);
|
||||
assert!(
|
||||
litellm_error::Report::new(&error)
|
||||
.to_string()
|
||||
.contains("result rejected")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
use litellm_cache_response::CachePolicy;
|
||||
use litellm_host::{
|
||||
interceptors::{ExecutionFacts, Interceptors, RawResponse},
|
||||
interceptors::{ExecutionFacts, Interceptors, RawResponse, ResultSource},
|
||||
lifecycle::{CallEvent, ExecutionEvent},
|
||||
observation::ObservationSender,
|
||||
};
|
||||
|
|
@ -28,7 +28,17 @@ impl<'a, I: Interceptors<Error>> CallContext<'a, I> {
|
|||
facts: facts.clone(),
|
||||
}));
|
||||
}
|
||||
self.interceptors.result_ready(facts).await
|
||||
let from_provider = facts.source == ResultSource::Provider;
|
||||
self.interceptors
|
||||
.result_ready(facts)
|
||||
.await
|
||||
.map_err(|error| {
|
||||
if from_provider {
|
||||
Error::post_call(error)
|
||||
} else {
|
||||
error
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn response_received(&self, body: &str) -> Result<(), Error> {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
use std::sync::Arc;
|
||||
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_host::machine::MachineFault;
|
||||
use litellm_llms::{Error as LlmError, ErrorDetail};
|
||||
|
||||
|
|
@ -9,8 +10,8 @@ pub enum Error {
|
|||
InvalidProvider(String),
|
||||
#[error("invalid provider: unable to resolve custom_llm_provider for {route} request")]
|
||||
UnresolvedProvider { route: &'static str },
|
||||
#[error("unsupported by the rust path: {0}")]
|
||||
Unsupported(&'static str),
|
||||
#[error(transparent)]
|
||||
Declined(Decline),
|
||||
#[error(transparent)]
|
||||
Llm(LlmError),
|
||||
#[error(transparent)]
|
||||
|
|
@ -50,14 +51,11 @@ impl Error {
|
|||
/// the provider's answer.
|
||||
pub fn is_request(&self) -> bool {
|
||||
match self {
|
||||
Self::InvalidProvider(_)
|
||||
| Self::UnresolvedProvider { .. }
|
||||
| Self::Unsupported(_)
|
||||
| Self::Headers(_) => true,
|
||||
Self::InvalidProvider(_) | Self::UnresolvedProvider { .. } | Self::Headers(_) => true,
|
||||
Self::Llm(error) => error.is_request(),
|
||||
Self::Auth(error) => error.is_request(),
|
||||
Self::Http(error) => error.is_request(),
|
||||
Self::Secret(_) | Self::Host(_) | Self::PostCallHook(_) => false,
|
||||
Self::Declined(_) | Self::Secret(_) | Self::Host(_) | Self::PostCallHook(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -67,7 +65,7 @@ impl Error {
|
|||
Self::Http(error) => error.is_response(),
|
||||
Self::InvalidProvider(_)
|
||||
| Self::UnresolvedProvider { .. }
|
||||
| Self::Unsupported(_)
|
||||
| Self::Declined(_)
|
||||
| Self::Auth(_)
|
||||
| Self::Headers(_)
|
||||
| Self::Secret(_)
|
||||
|
|
@ -82,7 +80,7 @@ impl Error {
|
|||
Self::Auth(error) => error.is_missing_credential(),
|
||||
Self::InvalidProvider(_)
|
||||
| Self::UnresolvedProvider { .. }
|
||||
| Self::Unsupported(_)
|
||||
| Self::Declined(_)
|
||||
| Self::Headers(_)
|
||||
| Self::Http(_)
|
||||
| Self::Secret(_)
|
||||
|
|
@ -91,21 +89,34 @@ impl Error {
|
|||
}
|
||||
}
|
||||
|
||||
pub fn is_unsupported(&self) -> bool {
|
||||
pub fn declined(&self) -> Option<&Decline> {
|
||||
match self {
|
||||
Self::Unsupported(_) => true,
|
||||
Self::Declined(decline) => Some(decline),
|
||||
Self::Llm(error) => error.declined(),
|
||||
Self::InvalidProvider(_)
|
||||
| Self::UnresolvedProvider { .. }
|
||||
| Self::Llm(_)
|
||||
| Self::Auth(_)
|
||||
| Self::Headers(_)
|
||||
| Self::Http(_)
|
||||
| Self::Secret(_)
|
||||
| Self::Host(_)
|
||||
| Self::PostCallHook(_) => false,
|
||||
| Self::PostCallHook(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Re-tag an error raised while handling an answer the provider already returned.
|
||||
///
|
||||
/// A config reports the same variants on either side of the call, so a missing field or
|
||||
/// a decline can mean "this request cannot be translated" before the send and "this
|
||||
/// response cannot be normalized" after it. Only the second kind was billed, and a host
|
||||
/// that keeps a reference implementation must not retry it.
|
||||
pub fn after_send(self) -> Self {
|
||||
if self.is_response() || self.upstream_response().is_some() {
|
||||
return self;
|
||||
}
|
||||
Self::invalid_response(ErrorDetail::invalid("provider response", self))
|
||||
}
|
||||
|
||||
pub fn dispatched(&self) -> bool {
|
||||
match self {
|
||||
Self::Llm(error) => error.dispatched(),
|
||||
|
|
@ -113,7 +124,7 @@ impl Error {
|
|||
Self::Host(_) | Self::PostCallHook(_) => true,
|
||||
Self::InvalidProvider(_)
|
||||
| Self::UnresolvedProvider { .. }
|
||||
| Self::Unsupported(_)
|
||||
| Self::Declined(_)
|
||||
| Self::Auth(_)
|
||||
| Self::Headers(_)
|
||||
| Self::Secret(_) => false,
|
||||
|
|
@ -125,7 +136,7 @@ impl Error {
|
|||
Self::Llm(error) => error.rejected_request(),
|
||||
Self::InvalidProvider(_)
|
||||
| Self::UnresolvedProvider { .. }
|
||||
| Self::Unsupported(_)
|
||||
| Self::Declined(_)
|
||||
| Self::Auth(_)
|
||||
| Self::Headers(_)
|
||||
| Self::Http(_)
|
||||
|
|
@ -141,7 +152,7 @@ impl Error {
|
|||
Self::Http(error) => error.upstream_response(),
|
||||
Self::InvalidProvider(_)
|
||||
| Self::UnresolvedProvider { .. }
|
||||
| Self::Unsupported(_)
|
||||
| Self::Declined(_)
|
||||
| Self::Auth(_)
|
||||
| Self::Headers(_)
|
||||
| Self::Secret(_)
|
||||
|
|
@ -156,7 +167,7 @@ impl From<litellm_http::Error> for Error {
|
|||
// Python drops the caller's copy and prefers a forwarded Authorization
|
||||
// over the signature, so leave the request to it.
|
||||
if error.is_computed_header() {
|
||||
return Self::Unsupported("request forwards a header AWS SigV4 computes");
|
||||
return Self::Declined(Decline::new("request forwards a header AWS SigV4 computes"));
|
||||
}
|
||||
Self::Http(error)
|
||||
}
|
||||
|
|
@ -165,7 +176,7 @@ impl From<litellm_http::Error> for Error {
|
|||
impl From<LlmError> for Error {
|
||||
fn from(error: LlmError) -> Self {
|
||||
match error {
|
||||
LlmError::Unsupported(reason) => Self::Unsupported(reason),
|
||||
LlmError::Declined(decline) => Self::Declined(decline),
|
||||
LlmError::Http(error) => Self::from(error),
|
||||
LlmError::Auth(error) => Self::Auth(error),
|
||||
error @ (LlmError::InvalidType { .. }
|
||||
|
|
@ -203,6 +214,7 @@ impl Eq for SecretError {}
|
|||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::Error;
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_host::machine::MachineFault;
|
||||
use litellm_llms::{Error as LlmError, ErrorDetail};
|
||||
use rstest::rstest;
|
||||
|
|
@ -273,13 +285,66 @@ mod tests {
|
|||
LlmError::Http(litellm_http::Error::computed_header("authorization")).into()
|
||||
)]
|
||||
fn a_header_the_signer_computes_is_left_to_the_host_on_every_route(#[case] error: Error) {
|
||||
assert!(error.is_unsupported(), "{error:?}");
|
||||
assert!(error.is_request());
|
||||
assert!(error.declined().is_some(), "{error:?}");
|
||||
assert!(!error.dispatched());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_transformation_unsupported_stays_a_decline() {
|
||||
assert!(Error::from(LlmError::Unsupported("test capability")).is_unsupported());
|
||||
fn a_transformation_decline_survives_into_the_route_error() {
|
||||
let decline = Decline::new("unrecognized request parameter");
|
||||
let error = Error::from(LlmError::Declined(decline));
|
||||
assert_eq!(error.declined(), Some(&decline));
|
||||
assert!(!error.is_request());
|
||||
assert!(!error.dispatched());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_post_call_hook_failure_is_never_a_decline() {
|
||||
let hook = Error::post_call(Error::Declined(Decline::new("hook")));
|
||||
assert_eq!(hook.declined(), None);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::declined(Error::Declined(Decline::new("non-text response content block")))]
|
||||
#[case::transformation_decline(LlmError::Declined(Decline::new("streaming")).into())]
|
||||
#[case::missing_field(LlmError::MissingField("usage").into())]
|
||||
#[case::invalid_request(Error::invalid_request(ErrorDetail::Expected {
|
||||
subject: "whatever",
|
||||
expected: "anything else",
|
||||
}))]
|
||||
#[case::auth(Error::Auth(litellm_auth::Error::InvalidHeader))]
|
||||
#[case::host_driver(MachineFault::Abandoned.into())]
|
||||
fn errors_after_the_send_are_response_faults_and_never_a_decline(#[case] error: Error) {
|
||||
let report = litellm_error::Report::new(&error).to_string();
|
||||
let collapsed = error.after_send();
|
||||
assert!(collapsed.is_response(), "{collapsed:?}");
|
||||
assert!(!collapsed.is_request(), "{collapsed:?}");
|
||||
assert!(collapsed.dispatched(), "{collapsed:?}");
|
||||
assert_eq!(collapsed.declined(), None);
|
||||
let collapsed_report = litellm_error::Report::new(&collapsed).to_string();
|
||||
assert!(collapsed_report.ends_with(&report), "{collapsed_report}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_response_fault_is_not_wrapped_twice_after_the_send() {
|
||||
let original = Error::invalid_response(ErrorDetail::MissingField {
|
||||
subject: "response",
|
||||
field: "usage",
|
||||
});
|
||||
let report = litellm_error::Report::new(&original).to_string();
|
||||
assert_eq!(
|
||||
litellm_error::Report::new(&original.after_send()).to_string(),
|
||||
report
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_provider_answer_stays_as_is_after_the_send() {
|
||||
let upstream = Error::from(litellm_http::Error::upstream(429, "slow down".into()));
|
||||
assert_eq!(
|
||||
upstream.after_send().upstream_response(),
|
||||
Some((429, "slow down"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ use litellm_cache_memory::InMemoryCache;
|
|||
use litellm_cache_response::{
|
||||
CacheOptions, CachePolicy, CacheScope, ResponseCache, ResponseCacheConfig, ResponseCacheService,
|
||||
};
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_host::{
|
||||
call::{CallOutput, OutputOf},
|
||||
interceptors::{
|
||||
|
|
@ -204,7 +205,8 @@ async fn streamed(
|
|||
.chunks(3)
|
||||
.map(|bytes| Ok(Bytes::copy_from_slice(bytes)))
|
||||
.collect::<Vec<_>>();
|
||||
let ending = fail.then_some(Err(Error::Unsupported("test transport failure")));
|
||||
let ending =
|
||||
fail.then_some(Err(Error::Declined(Decline::new("test transport failure"))));
|
||||
Ok(CallOutput::Stream {
|
||||
head: (),
|
||||
chunks: stream::iter(chunks.into_iter().chain(ending)).boxed(),
|
||||
|
|
@ -315,7 +317,7 @@ async fn a_provider_failure_never_populates_the_cache(cache: Arc<dyn ResponseCac
|
|||
None,
|
||||
|| async {
|
||||
calls.fetch_add(1, Ordering::SeqCst);
|
||||
Err(Error::Unsupported("test provider failure"))
|
||||
Err(Error::Declined(Decline::new("test provider failure")))
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
|
@ -482,7 +484,7 @@ impl Interceptors<Error> for CacheHitAccounting {
|
|||
);
|
||||
self.key.set(key).unwrap();
|
||||
if self.reject {
|
||||
return Err(Error::Unsupported("cache accounting rejected"));
|
||||
return Err(Error::Declined(Decline::new("cache accounting rejected")));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -569,10 +571,10 @@ async fn cache_hits_notify_accounting_once_and_propagate_its_failure(
|
|||
.await
|
||||
};
|
||||
if reject {
|
||||
assert!(matches!(
|
||||
result,
|
||||
Err(Error::Unsupported("cache accounting rejected"))
|
||||
));
|
||||
assert_eq!(
|
||||
result.unwrap_err().declined(),
|
||||
Some(&Decline::new("cache accounting rejected"))
|
||||
);
|
||||
} else {
|
||||
assert_eq!(result.unwrap(), expected);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_llms_types::formats::batches::{BatchRequestCounts, BatchResponse, BatchStatus};
|
||||
use litellm_llms_types::formats::messages::MessagesResponse;
|
||||
use litellm_llms_types::providers::anthropic::{BATCHES_PATH, MESSAGES_PATH};
|
||||
|
|
@ -127,7 +128,9 @@ impl AnthropicBatchesConfig for AnthropicBatchesTransformation {
|
|||
}
|
||||
|
||||
fn transform_create_batch_request(&self) -> Result<Value, Error> {
|
||||
Err(Error::Unsupported("Anthropic message batch creation"))
|
||||
Err(Error::Declined(Decline {
|
||||
reason: "Anthropic message batch creation",
|
||||
}))
|
||||
}
|
||||
|
||||
fn transform_create_batch_response(
|
||||
|
|
@ -135,7 +138,9 @@ impl AnthropicBatchesConfig for AnthropicBatchesTransformation {
|
|||
_response: AnthropicMessageBatch,
|
||||
_now: i64,
|
||||
) -> Result<BatchResponse, Error> {
|
||||
Err(Error::Unsupported("Anthropic message batch creation"))
|
||||
Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"Anthropic message batch creation",
|
||||
)))
|
||||
}
|
||||
|
||||
fn retrieve_batch_url(
|
||||
|
|
@ -337,16 +342,22 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn preserves_python_placeholder_for_batch_creation() {
|
||||
assert!(matches!(
|
||||
ANTHROPIC_BATCHES_TRANSFORMATION.transform_create_batch_request(),
|
||||
Err(Error::Unsupported("Anthropic message batch creation"))
|
||||
));
|
||||
#[rstest::rstest]
|
||||
fn batch_creation_only_declines_before_a_provider_response() {
|
||||
let error = ANTHROPIC_BATCHES_TRANSFORMATION
|
||||
.transform_create_batch_request()
|
||||
.unwrap_err();
|
||||
assert!(!error.dispatched());
|
||||
assert_eq!(
|
||||
error.declined().map(|decline| decline.reason),
|
||||
Some("Anthropic message batch creation")
|
||||
);
|
||||
let response: AnthropicMessageBatch = serde_json::from_value(json!({})).unwrap();
|
||||
assert!(matches!(
|
||||
ANTHROPIC_BATCHES_TRANSFORMATION.transform_create_batch_response(response, 0),
|
||||
Err(Error::Unsupported("Anthropic message batch creation"))
|
||||
));
|
||||
let error = ANTHROPIC_BATCHES_TRANSFORMATION
|
||||
.transform_create_batch_response(response, 0)
|
||||
.unwrap_err();
|
||||
assert!(error.dispatched());
|
||||
assert!(error.is_response());
|
||||
assert!(error.declined().is_none());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
use litellm_auth::SecretValue;
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_core_utils::{
|
||||
core_helpers::{finish_reason_for, unix_now, usage_from_parts},
|
||||
prompt_templates::factory::{Conversation, build_conversation},
|
||||
|
|
@ -24,7 +25,7 @@ use crate::{
|
|||
streaming::StreamShape,
|
||||
transformation::{
|
||||
BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData,
|
||||
Unsupported, ValidatedEnvironment, unsupported_message, unsupported_param,
|
||||
ValidatedEnvironment, unsupported_message, unsupported_param,
|
||||
},
|
||||
},
|
||||
},
|
||||
|
|
@ -126,15 +127,14 @@ impl BaseConfig for AnthropicConfig {
|
|||
serde_json::from_value(response.body).map_err(|error| {
|
||||
Error::InvalidResponse(crate::ErrorDetail::invalid("messages response", error))
|
||||
})?;
|
||||
// The route declines tool and thinking requests, so a non-text block
|
||||
// means the response carries something this path never asked for.
|
||||
// Decline rather than silently dropping it; the host falls back.
|
||||
if body
|
||||
.content
|
||||
.iter()
|
||||
.any(|block| matches!(block, TextResponseBlock::Other))
|
||||
{
|
||||
return Err(Error::Unsupported("non-text response content block"));
|
||||
return Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"non-text response content block",
|
||||
)));
|
||||
}
|
||||
let text: String = body
|
||||
.content
|
||||
|
|
@ -203,7 +203,7 @@ impl BaseConfig for AnthropicConfig {
|
|||
&self,
|
||||
messages: &[ChatMessage],
|
||||
optional_params: &Map<String, Value>,
|
||||
) -> Option<Unsupported> {
|
||||
) -> Option<Decline> {
|
||||
unsupported_param(self.supported_openai_param_mappings(), &[], optional_params)
|
||||
.or_else(|| messages.iter().find_map(unsupported_message))
|
||||
// Anthropic rejects a request whose first turn is not a user turn.
|
||||
|
|
@ -211,7 +211,7 @@ impl BaseConfig for AnthropicConfig {
|
|||
// core cannot observe, so decline instead of guessing.
|
||||
.or_else(|| {
|
||||
(!build_conversation(messages).opens_on_user_turn())
|
||||
.then_some(Unsupported("conversation does not open on a user turn"))
|
||||
.then_some(Decline::new("conversation does not open on a user turn"))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -260,9 +260,10 @@ mod tests {
|
|||
anthropic::chat::transformation::ANTHROPIC_CHAT_COMPLETIONS_CONFIG,
|
||||
base_llm::{
|
||||
auth::AuthScheme,
|
||||
chat::transformation::{BaseConfig, ProviderChatResponseData, Unsupported},
|
||||
chat::transformation::{BaseConfig, ProviderChatResponseData},
|
||||
},
|
||||
};
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_llms_types::formats::chat_completions::{ChatCompletionsResponse, ChatMessage};
|
||||
use rstest::rstest;
|
||||
use serde_json::{Map, Value, json};
|
||||
|
|
@ -290,7 +291,7 @@ mod tests {
|
|||
.transform_response("claude-sonnet-4-5", ProviderChatResponseData { body })
|
||||
}
|
||||
|
||||
fn reason(msgs: Value, opts: Value) -> Option<Unsupported> {
|
||||
fn reason(msgs: Value, opts: Value) -> Option<Decline> {
|
||||
ANTHROPIC_CHAT_COMPLETIONS_CONFIG.unsupported_reason(&messages(msgs), ¶ms(opts))
|
||||
}
|
||||
|
||||
|
|
@ -398,7 +399,7 @@ mod tests {
|
|||
json!([{"role": "user", "content": "hi"}]),
|
||||
json!({"top_k": 40})
|
||||
),
|
||||
Some(Unsupported("unrecognized request parameter"))
|
||||
Some(Decline::new("unrecognized request parameter"))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -409,7 +410,7 @@ mod tests {
|
|||
json!([{"role": "user", "content": "hi"}]),
|
||||
json!({"stream": true, "max_tokens": 16})
|
||||
),
|
||||
Some(Unsupported("streaming"))
|
||||
Some(Decline::new("streaming"))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -434,7 +435,7 @@ mod tests {
|
|||
fn declines_any_param_outside_the_allowlist(#[case] param: Value) {
|
||||
assert_eq!(
|
||||
reason(json!([{"role": "user", "content": "hi"}]), param.clone()),
|
||||
Some(Unsupported("unrecognized request parameter")),
|
||||
Some(Decline::new("unrecognized request parameter")),
|
||||
"expected {param} to decline"
|
||||
);
|
||||
}
|
||||
|
|
@ -452,7 +453,7 @@ mod tests {
|
|||
]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported("unrecognized message field"))
|
||||
Some(Decline::new("unrecognized message field"))
|
||||
);
|
||||
assert_eq!(
|
||||
reason(
|
||||
|
|
@ -462,7 +463,7 @@ mod tests {
|
|||
]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported("unrecognized message field"))
|
||||
Some(Decline::new("unrecognized message field"))
|
||||
);
|
||||
assert_eq!(
|
||||
reason(
|
||||
|
|
@ -472,7 +473,7 @@ mod tests {
|
|||
]}]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported("non-text message content"))
|
||||
Some(Decline::new("non-text message content"))
|
||||
);
|
||||
assert_eq!(
|
||||
reason(
|
||||
|
|
@ -482,7 +483,7 @@ mod tests {
|
|||
]}]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported("non-text message content"))
|
||||
Some(Decline::new("non-text message content"))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -493,7 +494,7 @@ mod tests {
|
|||
// than declining to Python before it.
|
||||
assert_eq!(
|
||||
reason(json!([{"role": "user", "content": []}]), json!({})),
|
||||
Some(Unsupported("message without content"))
|
||||
Some(Decline::new("message without content"))
|
||||
);
|
||||
assert_eq!(
|
||||
reason(
|
||||
|
|
@ -514,7 +515,7 @@ mod tests {
|
|||
]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported("conversation does not open on a user turn"))
|
||||
Some(Decline::new("conversation does not open on a user turn"))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -647,7 +648,7 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn declines_a_response_carrying_a_non_text_block() {
|
||||
fn a_response_carrying_a_non_text_block_is_invalid() {
|
||||
let err = transform_response(json!({
|
||||
"model": "claude-sonnet-4-5",
|
||||
"content": [{"type": "tool_use", "id": "t1", "name": "f", "input": {}}],
|
||||
|
|
@ -655,7 +656,12 @@ mod tests {
|
|||
"usage": {"input_tokens": 1, "output_tokens": 1}
|
||||
}))
|
||||
.expect_err("non-text block");
|
||||
assert_eq!(err, Error::Unsupported("non-text response content block"));
|
||||
assert_eq!(
|
||||
err,
|
||||
Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"non-text response content block"
|
||||
))
|
||||
);
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_llms_types::formats::chat_completions::{
|
||||
ChatCompletionsResponse, ChatMessage, ChatMessageContent,
|
||||
};
|
||||
|
|
@ -29,16 +30,6 @@ const IGNORABLE_MESSAGE_FIELDS: &[&str] = &["name"];
|
|||
|
||||
pub use crate::base_llm::auth::{Headers, ValidatedEnvironment};
|
||||
|
||||
/// Why a request cannot be served by the Rust path.
|
||||
///
|
||||
/// The core declines rather than guessing: the host turns this into a
|
||||
/// transparent fallback to the Python implementation, which covers the full
|
||||
/// surface. Acceptance is an allowlist, so a parameter or message shape the
|
||||
/// core has never seen declines by construction instead of being translated
|
||||
/// wrong.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct Unsupported(pub &'static str);
|
||||
|
||||
pub trait BaseConfig: Sync {
|
||||
fn secret_names(&self) -> Vec<&'static str>;
|
||||
|
||||
|
|
@ -97,7 +88,7 @@ pub trait BaseConfig: Sync {
|
|||
&self,
|
||||
messages: &[ChatMessage],
|
||||
optional_params: &Map<String, Value>,
|
||||
) -> Option<Unsupported> {
|
||||
) -> Option<Decline> {
|
||||
unsupported_param(
|
||||
self.supported_openai_param_mappings(),
|
||||
self.config_params(),
|
||||
|
|
@ -111,13 +102,13 @@ pub fn unsupported_param(
|
|||
supported: &'static [(&'static str, &'static str)],
|
||||
config: &'static [&'static str],
|
||||
optional_params: &Map<String, Value>,
|
||||
) -> Option<Unsupported> {
|
||||
) -> Option<Decline> {
|
||||
if optional_params
|
||||
.get(STREAM_PARAM)
|
||||
.and_then(Value::as_bool)
|
||||
.unwrap_or(false)
|
||||
{
|
||||
return Some(Unsupported("streaming"));
|
||||
return Some(Decline::new("streaming"));
|
||||
}
|
||||
optional_params
|
||||
.keys()
|
||||
|
|
@ -128,29 +119,29 @@ pub fn unsupported_param(
|
|||
.any(|(_, provider_name)| *provider_name == key)
|
||||
&& !config.contains(&key.as_str())
|
||||
})
|
||||
.then_some(Unsupported("unrecognized request parameter"))
|
||||
.then_some(Decline::new("unrecognized request parameter"))
|
||||
}
|
||||
|
||||
/// Message shapes the core can translate faithfully: text content, either a
|
||||
/// plain string or a non-empty list of parts that are all
|
||||
/// `{"type": "text", "text": ...}`. Tool calls, tool results, and multimodal
|
||||
/// parts decline so Python's fuller translation handles them.
|
||||
pub fn unsupported_message(message: &ChatMessage) -> Option<Unsupported> {
|
||||
pub fn unsupported_message(message: &ChatMessage) -> Option<Decline> {
|
||||
if message
|
||||
.extra
|
||||
.keys()
|
||||
.any(|key| !IGNORABLE_MESSAGE_FIELDS.contains(&key.as_str()))
|
||||
{
|
||||
return Some(Unsupported("unrecognized message field"));
|
||||
return Some(Decline::new("unrecognized message field"));
|
||||
}
|
||||
if !matches!(message.role.as_str(), "system" | "user" | "assistant") {
|
||||
return Some(Unsupported("unrecognized message role"));
|
||||
return Some(Decline::new("unrecognized message role"));
|
||||
}
|
||||
match &message.content {
|
||||
None => Some(Unsupported("message without content")),
|
||||
None => Some(Decline::new("message without content")),
|
||||
Some(ChatMessageContent::Text(_)) => None,
|
||||
Some(ChatMessageContent::Parts(parts)) if parts.is_empty() => {
|
||||
Some(Unsupported("message without content"))
|
||||
Some(Decline::new("message without content"))
|
||||
}
|
||||
Some(ChatMessageContent::Parts(parts)) => parts
|
||||
.iter()
|
||||
|
|
@ -159,6 +150,6 @@ pub fn unsupported_message(message: &ChatMessage) -> Option<Unsupported> {
|
|||
|| part.get("text").and_then(Value::as_str).is_none()
|
||||
|| part.as_object().is_some_and(|object| object.len() != 2)
|
||||
})
|
||||
.then_some(Unsupported("non-text message content")),
|
||||
.then_some(Decline::new("non-text message content")),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -87,8 +87,6 @@ pub enum Error {
|
|||
PollOrigin,
|
||||
#[error("OCR polling timed out")]
|
||||
PollTimeout,
|
||||
#[error("unsupported by the rust path: {0}")]
|
||||
Unsupported(&'static str),
|
||||
#[error("invalid provider: {0}")]
|
||||
InvalidProvider(String),
|
||||
#[error("invalid model: {provider} has no model {model:?} - use one of: {}", supported.join(", "))]
|
||||
|
|
@ -212,7 +210,6 @@ impl Error {
|
|||
| Self::InvalidPollUrl { .. }
|
||||
| Self::PollOrigin
|
||||
| Self::PollTimeout
|
||||
| Self::Unsupported(_)
|
||||
| Self::InvalidProvider(_)
|
||||
| Self::InvalidModel { .. }
|
||||
| Self::MissingCredential { .. }
|
||||
|
|
@ -255,7 +252,6 @@ impl Error {
|
|||
| Self::InvalidPollUrl { .. }
|
||||
| Self::PollOrigin
|
||||
| Self::PollTimeout
|
||||
| Self::Unsupported(_)
|
||||
| Self::InvalidProvider(_)
|
||||
| Self::InvalidModel { .. }
|
||||
| Self::Secret(_)
|
||||
|
|
@ -268,49 +264,6 @@ impl Error {
|
|||
}
|
||||
}
|
||||
|
||||
pub fn is_unsupported(&self) -> bool {
|
||||
match self {
|
||||
Self::Unsupported(_) => true,
|
||||
Self::Provider { .. }
|
||||
| Self::EmptyFile
|
||||
| Self::FileRead { .. }
|
||||
| Self::DocumentTask(_)
|
||||
| Self::InvalidMimeType(_)
|
||||
| Self::UnsupportedDocument { .. }
|
||||
| Self::RejectedDocument { .. }
|
||||
| Self::RequestFormat
|
||||
| Self::RequestField { .. }
|
||||
| Self::Arguments(_)
|
||||
| Self::MissingField(_)
|
||||
| Self::MissingDocumentUrl
|
||||
| Self::InvalidDataUri
|
||||
| Self::InlineDocumentTooLarge
|
||||
| Self::Pages(_)
|
||||
| Self::Features
|
||||
| Self::DotModel
|
||||
| Self::TooLarge { .. }
|
||||
| Self::ResponseField { .. }
|
||||
| Self::EmptyContent
|
||||
| Self::OperationStatus(_)
|
||||
| Self::NumericRange(_)
|
||||
| Self::PollLocation
|
||||
| Self::InvalidPollUrl { .. }
|
||||
| Self::PollOrigin
|
||||
| Self::PollTimeout
|
||||
| Self::InvalidProvider(_)
|
||||
| Self::InvalidModel { .. }
|
||||
| Self::MissingCredential { .. }
|
||||
| Self::Secret(_)
|
||||
| Self::Host(_)
|
||||
| Self::HostCall
|
||||
| Self::Auth(_)
|
||||
| Self::Params(_)
|
||||
| Self::Headers(_)
|
||||
| Self::Http(_)
|
||||
| Self::DocumentDownload(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn http_status_code(&self) -> Option<u16> {
|
||||
if let Some((status, ..)) = self.upstream_response() {
|
||||
return Some(status);
|
||||
|
|
@ -357,7 +310,6 @@ impl Error {
|
|||
| Self::InvalidPollUrl { .. }
|
||||
| Self::PollOrigin
|
||||
| Self::PollTimeout
|
||||
| Self::Unsupported(_)
|
||||
| Self::MissingCredential { .. }
|
||||
| Self::Secret(_)
|
||||
| Self::Host(_)
|
||||
|
|
@ -394,7 +346,6 @@ impl Error {
|
|||
| Self::Pages(_)
|
||||
| Self::Features
|
||||
| Self::DotModel
|
||||
| Self::Unsupported(_)
|
||||
| Self::InvalidProvider(_)
|
||||
| Self::InvalidModel { .. }
|
||||
| Self::MissingCredential { .. }
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ use litellm_auth_aws::{
|
|||
constants::{AWS_BEARER_TOKEN_BEDROCK, BEDROCK_RUNTIME_ENDPOINT_TEMPLATE, BEDROCK_SERVICE},
|
||||
resolve_bedrock_region,
|
||||
};
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_core_utils::{
|
||||
core_helpers::{finish_reason_for, unix_now, usage_from_parts},
|
||||
prompt_templates::factory::{Conversation, TurnRole, build_conversation},
|
||||
|
|
@ -26,7 +27,7 @@ use crate::{
|
|||
streaming::StreamShape,
|
||||
transformation::{
|
||||
BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData,
|
||||
Unsupported, ValidatedEnvironment, unsupported_message, unsupported_param,
|
||||
ValidatedEnvironment, unsupported_message, unsupported_param,
|
||||
},
|
||||
},
|
||||
},
|
||||
|
|
@ -164,10 +165,10 @@ impl BaseConfig for AmazonConverseConfig {
|
|||
field: "usage",
|
||||
}));
|
||||
}
|
||||
// The route declines tool requests, so anything other than a text block
|
||||
// is something this path never asked for. Decline; the host falls back.
|
||||
if response.message_content_is_non_text() {
|
||||
return Err(Error::Unsupported("non-text response content block"));
|
||||
return Err(Error::InvalidResponse(ErrorDetail::Unhandled(
|
||||
"non-text response content block",
|
||||
)));
|
||||
}
|
||||
let text = response.content_text();
|
||||
let computed = usage_from_parts(
|
||||
|
|
@ -264,7 +265,7 @@ impl BaseConfig for AmazonConverseConfig {
|
|||
&self,
|
||||
messages: &[ChatMessage],
|
||||
optional_params: &Map<String, Value>,
|
||||
) -> Option<Unsupported> {
|
||||
) -> Option<Decline> {
|
||||
unsupported_param(
|
||||
self.supported_openai_param_mappings(),
|
||||
CONFIG_PARAMS,
|
||||
|
|
@ -278,7 +279,7 @@ impl BaseConfig for AmazonConverseConfig {
|
|||
messages
|
||||
.iter()
|
||||
.any(has_blank_text)
|
||||
.then_some(Unsupported("blank message text"))
|
||||
.then_some(Decline::new("blank message text"))
|
||||
})
|
||||
// Converse has no assistant prefill: Python inserts a continue turn
|
||||
// when a conversation opens or closes on an assistant message, and
|
||||
|
|
@ -291,7 +292,7 @@ impl BaseConfig for AmazonConverseConfig {
|
|||
.turns
|
||||
.last()
|
||||
.is_some_and(|turn| turn.role == TurnRole::Assistant);
|
||||
(!conversation.opens_on_user_turn() || ends_on_assistant).then_some(Unsupported(
|
||||
(!conversation.opens_on_user_turn() || ends_on_assistant).then_some(Decline::new(
|
||||
"conversation does not run user turn to user turn",
|
||||
))
|
||||
})
|
||||
|
|
@ -353,11 +354,12 @@ mod tests {
|
|||
Error,
|
||||
base_llm::{
|
||||
auth::AuthScheme,
|
||||
chat::transformation::{BaseConfig, ProviderChatResponseData, Unsupported},
|
||||
chat::transformation::{BaseConfig, ProviderChatResponseData},
|
||||
},
|
||||
bedrock::chat::converse_transformation::BEDROCK_CHAT_COMPLETIONS_CONFIG,
|
||||
};
|
||||
use litellm_auth::CredentialPlacement;
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_llms_types::formats::chat_completions::{ChatCompletionsResponse, ChatMessage};
|
||||
use rstest::rstest;
|
||||
use serde_json::{Map, Value, json};
|
||||
|
|
@ -391,7 +393,7 @@ mod tests {
|
|||
)
|
||||
}
|
||||
|
||||
fn reason(msgs: Value, opts: Value) -> Option<Unsupported> {
|
||||
fn reason(msgs: Value, opts: Value) -> Option<Decline> {
|
||||
BEDROCK_CHAT_COMPLETIONS_CONFIG.unsupported_reason(&messages(msgs), ¶ms(opts))
|
||||
}
|
||||
|
||||
|
|
@ -467,7 +469,7 @@ mod tests {
|
|||
json!([{"role": "user", "content": "hi"}]),
|
||||
json!({"stream": true})
|
||||
),
|
||||
Some(Unsupported("streaming"))
|
||||
Some(Decline::new("streaming"))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -478,7 +480,7 @@ mod tests {
|
|||
json!([{"role": "user", "content": "hi"}]),
|
||||
json!({"topK": 40})
|
||||
),
|
||||
Some(Unsupported("unrecognized request parameter"))
|
||||
Some(Decline::new("unrecognized request parameter"))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -492,7 +494,7 @@ mod tests {
|
|||
fn declines_tools_and_other_params_outside_the_allowlist(#[case] param: Value) {
|
||||
assert_eq!(
|
||||
reason(json!([{"role": "user", "content": "hi"}]), param.clone()),
|
||||
Some(Unsupported("unrecognized request parameter")),
|
||||
Some(Decline::new("unrecognized request parameter")),
|
||||
"expected {param} to decline"
|
||||
);
|
||||
}
|
||||
|
|
@ -509,7 +511,7 @@ mod tests {
|
|||
json!([{"role": "user", "content": content}, {"role": "user", "content": "hi"}]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported("blank message text")),
|
||||
Some(Decline::new("blank message text")),
|
||||
"expected blank content {content} to decline"
|
||||
);
|
||||
}
|
||||
|
|
@ -521,7 +523,7 @@ mod tests {
|
|||
// host before the call rather than an error after it.
|
||||
assert_eq!(
|
||||
reason(json!([{"role": "user", "content": []}]), json!({})),
|
||||
Some(Unsupported("message without content"))
|
||||
Some(Decline::new("message without content"))
|
||||
);
|
||||
assert_eq!(
|
||||
reason(
|
||||
|
|
@ -542,7 +544,7 @@ mod tests {
|
|||
]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported(
|
||||
Some(Decline::new(
|
||||
"conversation does not run user turn to user turn"
|
||||
))
|
||||
);
|
||||
|
|
@ -554,7 +556,7 @@ mod tests {
|
|||
]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported(
|
||||
Some(Decline::new(
|
||||
"conversation does not run user turn to user turn"
|
||||
))
|
||||
);
|
||||
|
|
@ -814,7 +816,7 @@ mod tests {
|
|||
]}]),
|
||||
json!({})
|
||||
),
|
||||
Some(Unsupported("non-text message content"))
|
||||
Some(Decline::new("non-text message content"))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -841,7 +843,7 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn declines_a_response_carrying_a_tool_use_block() {
|
||||
fn a_response_carrying_a_tool_use_block_is_invalid() {
|
||||
let err = transform_response(json!({
|
||||
"output": {"message": {"content": [
|
||||
{"toolUse": {"toolUseId": "t1", "name": "f", "input": {}}}
|
||||
|
|
@ -850,7 +852,12 @@ mod tests {
|
|||
"usage": {"inputTokens": 1, "outputTokens": 1}
|
||||
}))
|
||||
.expect_err("tool use block");
|
||||
assert_eq!(err, Error::Unsupported("non-text response content block"));
|
||||
assert_eq!(
|
||||
err,
|
||||
Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"non-text response content block"
|
||||
))
|
||||
);
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
use litellm_core_utils::decline::Decline;
|
||||
use std::convert::Infallible;
|
||||
|
||||
use crate::{
|
||||
|
|
@ -130,9 +131,9 @@ impl BaseMessagesConfig for AmazonAnthropicClaudeMessagesConfig {
|
|||
_request: MessagesRequest,
|
||||
_context: &MessagesTransformContext,
|
||||
) -> Result<MessagesRequest, Error> {
|
||||
Err(Error::Unsupported(
|
||||
Err(Error::Declined(Decline::new(
|
||||
"Bedrock invoke messages request shaping",
|
||||
))
|
||||
)))
|
||||
}
|
||||
|
||||
fn secret_names(&self) -> &'static [&'static str] {
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_error::Source;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
|
||||
|
|
@ -13,8 +14,8 @@ pub enum Error {
|
|||
InvalidRequest(#[source] ErrorDetail),
|
||||
#[error("invalid response")]
|
||||
InvalidResponse(#[source] ErrorDetail),
|
||||
#[error("unsupported: {0}")]
|
||||
Unsupported(&'static str),
|
||||
#[error(transparent)]
|
||||
Declined(Decline),
|
||||
#[error(transparent)]
|
||||
Auth(#[from] litellm_auth::Error),
|
||||
#[error(transparent)]
|
||||
|
|
@ -24,13 +25,10 @@ pub enum Error {
|
|||
impl Error {
|
||||
pub fn is_request(&self) -> bool {
|
||||
match self {
|
||||
Self::InvalidType { .. }
|
||||
| Self::MissingField(_)
|
||||
| Self::InvalidRequest(_)
|
||||
| Self::Unsupported(_) => true,
|
||||
Self::InvalidType { .. } | Self::MissingField(_) | Self::InvalidRequest(_) => true,
|
||||
Self::Auth(error) => error.is_request(),
|
||||
Self::Http(error) => error.is_request(),
|
||||
Self::InvalidResponse(_) => false,
|
||||
Self::InvalidResponse(_) | Self::Declined(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -41,7 +39,7 @@ impl Error {
|
|||
Self::InvalidType { .. }
|
||||
| Self::MissingField(_)
|
||||
| Self::InvalidRequest(_)
|
||||
| Self::Unsupported(_)
|
||||
| Self::Declined(_)
|
||||
| Self::Auth(_) => false,
|
||||
}
|
||||
}
|
||||
|
|
@ -53,7 +51,7 @@ impl Error {
|
|||
| Self::MissingField(_)
|
||||
| Self::InvalidRequest(_)
|
||||
| Self::InvalidResponse(_)
|
||||
| Self::Unsupported(_)
|
||||
| Self::Declined(_)
|
||||
| Self::Http(_) => false,
|
||||
}
|
||||
}
|
||||
|
|
@ -67,8 +65,8 @@ impl Error {
|
|||
Self::InvalidType { .. }
|
||||
| Self::MissingField(_)
|
||||
| Self::InvalidRequest(_)
|
||||
| Self::InvalidResponse(_)
|
||||
| Self::Unsupported(_) => true,
|
||||
| Self::InvalidResponse(_) => true,
|
||||
Self::Declined(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -78,7 +76,19 @@ impl Error {
|
|||
match self {
|
||||
Self::InvalidRequest(detail) => Some(detail),
|
||||
Self::InvalidType { .. } | Self::MissingField(_) => Some(self),
|
||||
Self::InvalidResponse(_) | Self::Unsupported(_) | Self::Auth(_) | Self::Http(_) => None,
|
||||
Self::InvalidResponse(_) | Self::Declined(_) | Self::Auth(_) | Self::Http(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn declined(&self) -> Option<&Decline> {
|
||||
match self {
|
||||
Self::Declined(decline) => Some(decline),
|
||||
Self::InvalidType { .. }
|
||||
| Self::MissingField(_)
|
||||
| Self::InvalidRequest(_)
|
||||
| Self::InvalidResponse(_)
|
||||
| Self::Auth(_)
|
||||
| Self::Http(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -89,7 +99,7 @@ impl Error {
|
|||
| Self::MissingField(_)
|
||||
| Self::InvalidRequest(_)
|
||||
| Self::InvalidResponse(_)
|
||||
| Self::Unsupported(_)
|
||||
| Self::Declined(_)
|
||||
| Self::Auth(_) => None,
|
||||
}
|
||||
}
|
||||
|
|
@ -154,6 +164,8 @@ pub enum ErrorDetail {
|
|||
MissingApiBase,
|
||||
#[error("the host call raised an exception")]
|
||||
HostCall,
|
||||
#[error("{0} is not handled by the rust path")]
|
||||
Unhandled(&'static str),
|
||||
#[error("invalid {subject} on line {line}")]
|
||||
InvalidLine {
|
||||
subject: &'static str,
|
||||
|
|
|
|||
|
|
@ -246,14 +246,20 @@ fn normalize_choice(position: usize, choice: &Value) -> Result<ChatCompletionsCh
|
|||
// normalized type cannot carry tool calls at all. Declining is
|
||||
// terminal at this point, but passing back an empty assistant turn
|
||||
// would fabricate the reply.
|
||||
return Err(Error::Unsupported("tool call response"));
|
||||
return Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"tool call response",
|
||||
)));
|
||||
}
|
||||
if message.get("refusal").is_some_and(|value| !value.is_null()) {
|
||||
return Err(Error::Unsupported("refusal response"));
|
||||
return Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"refusal response",
|
||||
)));
|
||||
}
|
||||
let content = message.get("content");
|
||||
if content.is_some_and(|value| !value.is_null() && !value.is_string()) {
|
||||
return Err(Error::Unsupported("non-text response content"));
|
||||
return Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"non-text response content",
|
||||
)));
|
||||
}
|
||||
Ok(ChatCompletionsChoice {
|
||||
index: choice
|
||||
|
|
@ -282,10 +288,11 @@ mod tests {
|
|||
Error,
|
||||
base_llm::{
|
||||
auth::AuthScheme,
|
||||
chat::transformation::{BaseConfig, ProviderChatResponseData, Unsupported},
|
||||
chat::transformation::{BaseConfig, ProviderChatResponseData},
|
||||
},
|
||||
openai_like::chat::transformation::OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG,
|
||||
};
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use litellm_llms_types::formats::chat_completions::{ChatCompletionsResponse, ChatMessage};
|
||||
use rstest::rstest;
|
||||
use serde_json::{Map, Value, json};
|
||||
|
|
@ -321,7 +328,7 @@ mod tests {
|
|||
.transform_response("some-model", ProviderChatResponseData { body })
|
||||
}
|
||||
|
||||
fn reason(msgs: Value, opts: Value) -> Option<Unsupported> {
|
||||
fn reason(msgs: Value, opts: Value) -> Option<Decline> {
|
||||
OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG.unsupported_reason(&messages(msgs), ¶ms(opts))
|
||||
}
|
||||
|
||||
|
|
@ -535,9 +542,7 @@ mod tests {
|
|||
}
|
||||
|
||||
#[rstest]
|
||||
fn a_tool_call_response_declines_instead_of_dropping_the_calls() {
|
||||
// The `json_mode` rewrite needs a request flag the route does not carry, so
|
||||
// a tool-call answer falls back to Python rather than losing the calls.
|
||||
fn a_tool_call_response_fails_instead_of_dropping_the_calls() {
|
||||
assert_eq!(
|
||||
transform_response(json!({
|
||||
"model": "m",
|
||||
|
|
@ -554,12 +559,14 @@ mod tests {
|
|||
"finish_reason": "tool_calls",
|
||||
}],
|
||||
})),
|
||||
Err(Error::Unsupported("tool call response"))
|
||||
Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"tool call response"
|
||||
)))
|
||||
);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
fn a_refusal_declines_instead_of_returning_an_empty_reply() {
|
||||
fn a_refusal_fails_instead_of_returning_an_empty_reply() {
|
||||
assert_eq!(
|
||||
transform_response(json!({
|
||||
"model": "m",
|
||||
|
|
@ -568,12 +575,14 @@ mod tests {
|
|||
"finish_reason": "stop",
|
||||
}],
|
||||
})),
|
||||
Err(Error::Unsupported("refusal response"))
|
||||
Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"refusal response"
|
||||
)))
|
||||
);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
fn a_non_text_response_content_declines() {
|
||||
fn a_non_text_response_content_is_invalid() {
|
||||
assert_eq!(
|
||||
transform_response(json!({
|
||||
"model": "m",
|
||||
|
|
@ -582,7 +591,9 @@ mod tests {
|
|||
"finish_reason": "stop",
|
||||
}],
|
||||
})),
|
||||
Err(Error::Unsupported("non-text response content"))
|
||||
Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled(
|
||||
"non-text response content"
|
||||
)))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -592,7 +603,7 @@ mod tests {
|
|||
fn declines(#[case] opts: Value, #[case] expected: &'static str) {
|
||||
assert_eq!(
|
||||
reason(json!([{"role": "user", "content": "hi"}]), opts),
|
||||
Some(Unsupported(expected))
|
||||
Some(Decline::new(expected))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -622,7 +633,7 @@ mod tests {
|
|||
json!([{"role": "user", "content": "hi"}]),
|
||||
json!({"tools": [{"type": "function", "function": {"name": "f"}}]}),
|
||||
),
|
||||
Some(Unsupported("unrecognized request parameter"))
|
||||
Some(Decline::new("unrecognized request parameter"))
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -62,14 +62,18 @@ pub(crate) enum RequestMarker {
|
|||
Messages,
|
||||
}
|
||||
|
||||
/// Maps a route failure once. A Python exception raised inside the call is returned as it is,
|
||||
/// everything else is mapped and handed to `public`, the route's public exception mapping.
|
||||
/// Maps a route failure once. A core decline and a Python exception raised inside the call are
|
||||
/// returned as they are, everything else is mapped and handed to `public`, the route's public
|
||||
/// exception mapping.
|
||||
pub(crate) fn route_failure(
|
||||
py: Python<'_>,
|
||||
error: Error,
|
||||
marker: RequestMarker,
|
||||
public: impl FnOnce(PyErr) -> PyResult<PyErr>,
|
||||
) -> PyResult<PyErr> {
|
||||
if let Some(decline) = error.declined() {
|
||||
return Ok(RustBridgeDeclined::new_err(decline.reason));
|
||||
}
|
||||
if let Some(raised) = crate::secrets::python_error(py, &error) {
|
||||
return Ok(raised);
|
||||
}
|
||||
|
|
@ -97,6 +101,7 @@ pub(crate) fn route_error_to_pyerr(error: Error) -> PyErr {
|
|||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use litellm_core_utils::decline::Decline;
|
||||
use reqwest::header::{HeaderValue, RETRY_AFTER};
|
||||
|
||||
use super::*;
|
||||
|
|
@ -110,7 +115,6 @@ mod tests {
|
|||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
#[case::unsupported(Error::Unsupported("test capability"), true)]
|
||||
#[case::invalid_provider(Error::InvalidProvider("unknown".into()), true)]
|
||||
#[case::invalid_request(
|
||||
Error::invalid_request(litellm_llms::ErrorDetail::Expected {
|
||||
|
|
@ -130,6 +134,10 @@ mod tests {
|
|||
}),
|
||||
false
|
||||
)]
|
||||
#[case::post_call_decline(
|
||||
Error::post_call(Error::Declined(Decline::new("result rejected"))),
|
||||
false
|
||||
)]
|
||||
fn route_failures_are_terminal(#[case] error: Error, #[case] is_request: bool) {
|
||||
Python::initialize();
|
||||
Python::attach(|py| {
|
||||
|
|
@ -140,6 +148,34 @@ mod tests {
|
|||
});
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
#[case::unmarked(RequestMarker::Unmarked)]
|
||||
#[case::messages(RequestMarker::Messages)]
|
||||
fn a_core_decline_hands_the_call_back_to_python_before_the_public_mapping(
|
||||
#[case] marker: RequestMarker,
|
||||
) {
|
||||
Python::initialize();
|
||||
Python::attach(|py| {
|
||||
let error = Error::from(litellm_llms::Error::Declined(Decline::new(
|
||||
"unrecognized request parameter",
|
||||
)));
|
||||
let declined = route_failure(py, error, marker, |_| {
|
||||
panic!("a decline must not reach the public mapping")
|
||||
})
|
||||
.unwrap();
|
||||
assert!(declined.is_instance_of::<RustBridgeDeclined>(py));
|
||||
assert_eq!(
|
||||
declined
|
||||
.value(py)
|
||||
.getattr("args")
|
||||
.unwrap()
|
||||
.extract::<(String,)>()
|
||||
.unwrap(),
|
||||
("unrecognized request parameter".into(),)
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
#[case::unmarked(RequestMarker::Unmarked)]
|
||||
#[case::messages(RequestMarker::Messages)]
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
use litellm_core_utils::decline::Decline;
|
||||
use std::convert::Infallible;
|
||||
|
||||
use crate::routes::inference::InferenceHost;
|
||||
|
|
@ -44,9 +45,9 @@ impl PythonBinding for ChatCompletionsPythonHost {
|
|||
.get("stream")
|
||||
.is_some_and(|value| value == &serde_json::Value::Bool(true))
|
||||
{
|
||||
return Err(InvokeError::Native(Error::Unsupported(
|
||||
return Err(InvokeError::Native(Error::Declined(Decline::new(
|
||||
"native Python chat_completions streaming",
|
||||
)));
|
||||
))));
|
||||
}
|
||||
Ok(call)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
use litellm_core_utils::decline::Decline;
|
||||
use std::convert::Infallible;
|
||||
|
||||
use crate::routes::inference::InferenceHost;
|
||||
|
|
@ -118,9 +119,9 @@ impl PythonBinding for ResponsesPythonHost {
|
|||
.get("stream")
|
||||
.is_some_and(|value| value == &serde_json::Value::Bool(true))
|
||||
{
|
||||
return Err(InvokeError::Native(Error::Unsupported(
|
||||
return Err(InvokeError::Native(Error::Declined(Decline::new(
|
||||
"native Python responses streaming",
|
||||
)));
|
||||
))));
|
||||
}
|
||||
Ok(call)
|
||||
}
|
||||
|
|
@ -141,9 +142,11 @@ impl PythonBinding for ResponsesPythonHost {
|
|||
head: <Responses as litellm_host::protocol::Protocol>::StreamHead,
|
||||
) -> PyResult<Py<PyAny>> {
|
||||
let _ = head;
|
||||
Err(crate::errors::route_error_to_pyerr(Error::Unsupported(
|
||||
"native Python responses streaming",
|
||||
)))
|
||||
Err(crate::errors::route_error_to_pyerr(
|
||||
Error::invalid_response(litellm_llms::ErrorDetail::Unhandled(
|
||||
"native Python responses streaming",
|
||||
)),
|
||||
))
|
||||
}
|
||||
|
||||
fn encode_chunk(
|
||||
|
|
@ -152,9 +155,11 @@ impl PythonBinding for ResponsesPythonHost {
|
|||
chunk: <Responses as litellm_host::protocol::Protocol>::Chunk,
|
||||
) -> PyResult<Py<PyAny>> {
|
||||
let _ = chunk;
|
||||
Err(crate::errors::route_error_to_pyerr(Error::Unsupported(
|
||||
"native Python responses streaming",
|
||||
)))
|
||||
Err(crate::errors::route_error_to_pyerr(
|
||||
Error::invalid_response(litellm_llms::ErrorDetail::Unhandled(
|
||||
"native Python responses streaming",
|
||||
)),
|
||||
))
|
||||
}
|
||||
|
||||
fn map_error(&self, py: Python<'_>, error: Error) -> PyResult<PyErr> {
|
||||
|
|
|
|||
|
|
@ -67,3 +67,16 @@ async def test_empty_messages_fail_before_provider_io(server: RecordingServer, a
|
|||
await call(server, asynchronous=asynchronous, messages=[])
|
||||
|
||||
assert not server.requests
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("asynchronous", (False, True))
|
||||
async def test_a_native_decline_reaches_the_dispatcher_before_provider_io(
|
||||
server: RecordingServer, asynchronous: bool
|
||||
) -> None:
|
||||
server.expected_requests = 0
|
||||
|
||||
with pytest.raises(RuntimeError, match="declined the request: native Python chat_completions streaming"):
|
||||
await call(server, asynchronous=asynchronous, stream=True)
|
||||
|
||||
assert not server.requests
|
||||
|
|
|
|||
|
|
@ -70,6 +70,6 @@ async def call(server: RecordingServer, *, asynchronous: bool = True, **kwargs:
|
|||
|
||||
SYNC_DEPLOYMENT_HOOK: Final = gap("native sync calls skip the async pre-call deployment hook")
|
||||
STREAMING: Final = gap(
|
||||
"native Responses streaming is unsupported, the catalog keeps it on Python", raises=litellm.APIConnectionError
|
||||
"native Responses streaming is unsupported, the catalog keeps it on Python", raises=RuntimeError
|
||||
)
|
||||
BODY_HOOKS: Final = gap("native Responses does not run the handler body hooks")
|
||||
|
|
|
|||
|
|
@ -56,3 +56,16 @@ async def test_provider_failure_is_terminal_and_shared_with_callbacks(
|
|||
assert len(failure) == 1
|
||||
assert _OBJECT.validate_python(failure[0].kwargs)["exception"] is caught.value
|
||||
assert not any("success" in name for name in recorder.names)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("asynchronous", (False, True))
|
||||
async def test_a_native_decline_reaches_the_dispatcher_before_provider_io(
|
||||
server: RecordingServer, asynchronous: bool
|
||||
) -> None:
|
||||
server.expected_requests = 0
|
||||
|
||||
with pytest.raises(RuntimeError, match="declined the request: native Python responses streaming"):
|
||||
await call(server, asynchronous=asynchronous, stream=True)
|
||||
|
||||
assert not server.requests
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue