mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042)
* build(rust): upgrade reqwest to 0.13 and move MCP out of litellm-http The workspace moves to reqwest 0.13, so litellm-http no longer carries a second reqwest just for rmcp. The mcp feature, its pool and its proxy conversion go away, and the gateway hands rmcp a pooled NoRedirect client that already carries the TLS, proxy and timeout settings The Client test helpers collapse into for_test, a no-redirect client without TLS or proxy settings, and wrap_for_test for the few tests that need a hand-built reqwest client Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(http): honor transport settings for Rust WebSockets The Responses WebSocket connected through tokio-tungstenite with its own native-roots TLS, so it ignored the configured CA bundle, verify flag, client certificate, proxies and timeouts. The handshake now goes through a pooled reqwest client (ClientVariant::WebSocket, HTTP/1.1, no redirects) and upgrades the response, so it shares the settings every other outbound request uses The upgrade response is checked for the Sec-WebSocket-Accept key, the Upgrade and Connection headers and a subprotocol that was offered Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(http): build outbound requests as reqwest::Request OutboundRequest was a hand-rolled copy of reqwest::Request. Routes now build a reqwest::Request directly and RequestSigner reads the body bytes back out of it, so the signer still covers exactly what is sent. SigV4 signing moves onto HeaderMap and takes the method, which drops the BTreeMap plumbing and the header casing table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(http): make litellm_http::Error the one opaque error transport::Error and media::Error fold into litellm_http::Error, an Arc<Repr> newtype whose private variants keep the typed reqwest, io, rustls and tungstenite causes and own every message. Callers can no longer build an error with a message of their own. They ask through dispatched, is_request, status, upstream_response and tls_source One From<reqwest::Error> now classifies connect, network and timeout and strips the URL. The WebSocket handshake reports each failed check as its own variant and owns its connect timeout. OCR drops the media variants it mirrored, and inference::Error maps a header the signer computes to Unsupported for every route, not only chat The error rules move out of AGENTS.md into the rust-error skill, which also records the decisions behind this design Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(rust): split rust-error skill into thin SKILL.md with references Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rust-error): render causes once and ask predicates at the boundaries The four WebSocket handshake variants collapse into Handshake(HandshakeFault), and media download failures no longer answer dispatched() since they happen before the provider call ErrorDetail in llms and auth-types, the inference error and the OCR error now store their cause in a field named cause with no #[source], so to_string() carries it exactly once and source() is None. Tests that walked the chain now destructure the variant and downcast the cause The gateway asks is_unsupported() and is_missing_credential() on the inference and OCR errors instead of matching through nested enums, and litellm_auth::Error answers is_missing_credential() for both The rust-error skill gains axum and pyo3 references, boundaries.md describes the real predicate order and the known divergences, the testing example asserts dispatched() correctly, and shape, classification, conversions, messages-and-causes and recipes agree with the code again * docs(rust): focus the rust-error gateway page on the error flow and add a rust-gateway skill The former axum page toured gateway-inference: its status match, the envelope table, an extractor, the unsupported route and the Sse closure. Those details belong to the gateway crates and go stale with them. The page is now gateway.md and follows an error from the provider failure through each From into IntoResponse, where it is classified once and rendered by surface and framing. The gateway-structure material moves to a new rust-gateway skill: envelopes per API surface, extractors and path registration, SSE relay, and the router test harness. * test(inference): pin provider URL redaction on every route's transport error Chat, Messages, Responses and transcription each get a route-level test that fails if a reqwest error reaches the caller with its URL. The Messages stream decoder also takes body errors through litellm_http::Error like every other reqwest error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(rust-error): adopt std source chains and add litellm-error Report Messages describe their own layer and the cause is the source, boundaries render the chain once with litellm_error::Report. Corrects skill claims the audit found stale against the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rust-error): keep causes as sources across http, auth, secrets, cache and utility crates Messages describe their own layer and every cause is a typed source, predicates match exhaustively, and boundaries render with litellm_error::Report. Media downloads no longer count as dispatched, Error::upstream bounds the provider body itself, and a forwarded header SigV4 cannot sign is the caller's fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rust-error): keep causes as sources in llms and the gateway crates ErrorDetail loses its string variant, OCR provider variants become data, host faults stop counting as the caller's, a tunnelled transport error classifies the same on every stream path, and each gateway crate has one Error with one status() and extractors that reject in its envelope. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rust-error): nest llms errors in inference and ask predicates in route tests litellm_inference::Error nests litellm_llms::Error instead of mirroring its variants, host driver faults get their own variant, every predicate matches exhaustively, and route tests ask predicates instead of matching transport variants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rust-error): map bridge and gateway-inference errors once and render them with Report RustUpstreamError has one constructor that always sets headers, argument checks raise typed errors mapped once, OCR asks upstream_response(), the Responses stream error event comes from the error type, gateway-inference loses its string variants, and auth faults classify through one litellm_auth predicate on every route. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(rust-error): carry OCR provider headers as a HeaderMap and keep outages off the caller The OCR provider error and get_error_class take reqwest's HeaderMap, the bridge turns it into Python's (name, value) pairs once. Credential acquisition and provider SDK failures are no longer the caller's fault, the UI hides auth and backend causes, and chat keeps a provider answer as is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(rust-error): correct skill claims against the aligned code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rust-error): fix the remaining sweep findings in gateway, llms, auth and cache crates Auth rejections hide their backend cause on every inference renderer, path parameters reject in the surface envelope, Textract's unsupported document and Azure DI's unparseable poll URL get typed variants, the credential chain exposes its last failure as the source, gateway-mcp asks predicates over rmcp errors, and the S3 and Azure blob transports classify through litellm_http::Error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rust-error): fix the remaining sweep findings on the Python side host-python keeps a native error's chain and returns blocking-pool cancellation instead of panicking, Vault transport failures convert through litellm_http::Error so no URL reaches Python, the bridge asks predicates instead of matching three levels into Vault errors, and the last string-built PyErrs become typed errors mapped once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(deps): refresh compatible dependencies flagged by OSV * refactor(rust): remove redundant client construction comment * test(rust): cover configured MCP proxy and CA bundle * fix(http): preserve native WebSocket trust and optional protocols * fix(http): allow plain WebSockets without native roots * fix(http): normalize provider URLs before building outbound requests * fix(rust): preserve OCR download status and reduce error size * fix(rust): preserve upstream bodies and bound diagnostics only * test(rust): preserve bridge serialization boundary in body regression --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
yujonglee
GitHub
parent
424e5c069c
commit
7b60fd061a
406 changed files with 9261 additions and 4747 deletions
35
litellm-rust/.agents/skills/rust-error/SKILL.md
Normal file
35
litellm-rust/.agents/skills/rust-error/SKILL.md
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
---
|
||||
name: rust-error
|
||||
description: Define, wrap, convert, classify, report or test Rust error types in litellm-rust with thiserror. Use when adding an error type or variant, mapping one crate's error into another, turning an error into a gateway response or Python exception, deciding retry or fault from an error, or replacing call sites that build errors from strings
|
||||
---
|
||||
|
||||
# Rust errors
|
||||
|
||||
`crates/http/src/error.rs` is the reference implementation. Read the reference for the part you are touching before editing, each one carries the reasoning and worked examples behind its rules
|
||||
|
||||
- [shape](references/shape.md)
|
||||
- One top-level `Error` per crate in `src/error.rs`. A public enum by default, an opaque struct over a private `Repr` enum when the variants must stay private. No `#[non_exhaustive]`
|
||||
- A variant is a failure mode a caller handles differently. Provider names and guidance are data, never variants
|
||||
- [messages and causes](references/messages-and-causes.md)
|
||||
- Every message is written once, in `#[error(...)]`, and describes its own layer only. Call sites pass typed facts, never a string
|
||||
- The cause is the `source()`, never part of the message. Keep it typed, never stringify or drop it
|
||||
- `reqwest::Error` enters our types only through `litellm_http::Error`, which strips its URL
|
||||
- Anything shown to someone else is rendered with `litellm_error::Report`, never `to_string()`
|
||||
- [conversions](references/conversions.md)
|
||||
- One `From` per source and target pair, only when the conversion needs no context. It classifies and sanitizes once
|
||||
- Nest a lower crate's error in one variant. Translate when the meaning changes, never mirror its variants
|
||||
- [classification](references/classification.md)
|
||||
- Callers ask predicates named after their decision, they do not match nested variants. Predicates match exhaustively
|
||||
- Classification states facts. Retry and fallback policy lives with the caller that knows idempotency
|
||||
- [boundaries](references/boundaries.md)
|
||||
- An error becomes text for someone else only in one mapping function per error type, in the gateway crates, the Python bridge and host-python. The inference gateway and the bridge decide by the same predicates
|
||||
- [gateway](references/gateway.md)
|
||||
- One `Error` with `IntoResponse` per gateway crate that serves a router. Every failure flows through the same `From` chain, is classified once in `status()` and rendered by surface and framing
|
||||
- [pyo3](references/pyo3.md)
|
||||
- Two `create_exception!` classes with the payload in `args`, `ValueError` for the caller's fault and `RuntimeError` otherwise, decoded in `failures.py`
|
||||
- [testing](references/testing.md)
|
||||
- Test predicates, chain shape and redaction on errors built the way production builds them
|
||||
- [recipes](references/recipes.md)
|
||||
- Common tasks walked through end to end
|
||||
|
||||
The workspace uses thiserror 2 for every error type, and no library crate uses `anyhow` or `Box<dyn Error>` in a public signature. The [derive docs](https://docs.rs/thiserror/latest/thiserror/) are the authority on macro behavior, and the [`std::error::Error` docs](https://doc.rust-lang.org/std/error/trait.Error.html) on what `Display` and `source()` mean
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
# Boundaries
|
||||
|
||||
A boundary is where an error becomes something another program reads: an HTTP response, a Python exception, a stored or logged message. Each boundary has one mapping function per error type it receives, and nothing outside those functions formats an error for someone else
|
||||
|
||||
| Boundary | Mapping | Reference |
|
||||
|---|---|---|
|
||||
| Inference HTTP response | `Error::status()` and its renderers in `crates/gateway-inference/src/error.rs` | [gateway](gateway.md) |
|
||||
| Other gateway surfaces | the `IntoResponse` of each `crates/gateway-*` crate's `Error` | [gateway](gateway.md) |
|
||||
| Python bridge, inference and OCR routes | `route_failure` in `crates/python-bridge/src/errors.rs` and in `crates/python-bridge/src/routes/ocr/errors.rs` | [pyo3](pyo3.md) |
|
||||
| Python bridge, everything else | one `From<Error> for PyErr` or mapping function per crate error, all in `crates/python-bridge` | [pyo3](pyo3.md) |
|
||||
| Python host runtime | `From<Error> for PyErr` in `crates/host-python/src/error.rs` | [pyo3](pyo3.md) |
|
||||
|
||||
## Asking the nested error
|
||||
|
||||
The inference gateway and the Python bridge's route mapping both receive a nested route error and decide by the same predicates, so a new failure lands correctly on both once its predicates answer right
|
||||
|
||||
| Predicate | Gateway | Bridge |
|
||||
|---|---|---|
|
||||
| `is_unsupported()` | 501 | `ValueError`, since `is_request()` also answers true |
|
||||
| `upstream_response()` | the provider's status and body | `RustUpstreamError` |
|
||||
| `is_missing_credential()` | 401 | no class of its own, a `RuntimeError` because `is_request()` answers false |
|
||||
| `is_request()` | 400 | `ValueError` |
|
||||
| none of these | 500 | `RuntimeError` |
|
||||
|
||||
If you find yourself adding an arm for your variant at a boundary, check whether a predicate is missing first
|
||||
|
||||
The two boundaries are not identical. The gateway has variants of its own for body parsing, an unknown model and an unsupported path, and the OCR bridge has Python parity arms such as `FileNotFoundError` for a missing file. Those arms match the boundary's own variants or the top level of a public enum, never a nested error's variants. A missing API key is 401 at the gateway and `RuntimeError` at both bridges, pinned by tests to what Python's own path does
|
||||
|
||||
## What reaches the client
|
||||
|
||||
A boundary renders anything that is not a provider answer with `litellm_error::Report`, so the reader sees every layer of the chain once. That is why messages are written for a reader, each layer leaves its cause to `source()`, and reqwest URLs are stripped on entry. A boundary that must not show the chain, such as the bridge's TLS settings error that names the setting but not the file, renders fixed text of its own and says so in its test
|
||||
|
||||
Provider bodies are passed through intact for gateway and Python error mapping. `litellm_http::Error::upstream(status, body)` retains the complete body, but its `Display`, `Debug` and `Report` diagnostics show only the first 256 characters and mark the cut. Do not log the raw body or `RustUpstreamError` args. OCR's `Provider` error also keeps the whole body for Python's `get_error_class`, bounded by the connection's response size limit
|
||||
|
||||
Never format `Debug` of an error into a response or a Python exception. A type whose sources may hold secrets derives `veil::Redact`, so its `Debug` is safe to log
|
||||
|
||||
## Logging
|
||||
|
||||
Propagate with context, and let the terminal handler own the one log or trace event for a failed call. Do not log at every `?`, and do not log an error you also return. Diagnostic tracing excludes raw error strings and bodies, see [`rust-tracing`](../../rust-tracing/SKILL.md)
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
# Classification
|
||||
|
||||
## Ask, don't match
|
||||
|
||||
Downstream crates ask a lower error questions instead of matching its nested variants. `litellm_http::Error` answers these
|
||||
|
||||
| Predicate | Caller decides |
|
||||
|---|---|
|
||||
| `dispatched()` | whether retrying elsewhere could repeat a billed call |
|
||||
| `is_request()` | the caller's input is at fault, so 400 and no reroute |
|
||||
| `is_response()` | the provider answered with something we cannot use |
|
||||
| `upstream_response()` | the provider's own status and body to pass through |
|
||||
| `status()` | an HTTP status to report |
|
||||
| `is_connect()` | the connection was never made, for a host's own connect-error mapping |
|
||||
| `tls_source()` | which TLS setting to blame in a config error |
|
||||
| `is_computed_header()` | the caller forwarded a header the signer owns, which `litellm_inference::Error` turns into `Unsupported` |
|
||||
|
||||
Upper types add the questions their callers ask. `litellm_inference::Error` and the OCR error answer `is_missing_credential()` and `is_unsupported()`, which the gateway asks instead of reaching into `Auth` or `Unsupported`
|
||||
|
||||
```rust
|
||||
// reaches through two crates' variants and breaks when either changes
|
||||
Self::Route(InferenceError::Auth(litellm_auth::Error::MissingApiKey { .. })) => StatusCode::UNAUTHORIZED,
|
||||
|
||||
// asks the question it cares about
|
||||
Self::Route(error) if error.is_missing_credential() => StatusCode::UNAUTHORIZED,
|
||||
```
|
||||
|
||||
A predicate is an exhaustive `match` with no `_` arm, never `matches!`, so a new variant does not compile until it has answered every question on purpose
|
||||
|
||||
```rust
|
||||
pub fn is_request(&self) -> bool {
|
||||
match &*self.0 {
|
||||
Repr::Request(_) | Repr::BlockedUrl | Repr::DownloadTooLarge => true,
|
||||
Repr::Connect(_) | Repr::Network(_) | Repr::Timeout(_) | Repr::Status { .. } => false,
|
||||
// ... every other variant, listed by name
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Add a predicate when a real caller needs a new distinction, and name it after the fact the caller needs, in the caller's words, not after the variant's spelling. An upper type's predicates call through to the nested error, as in `Self::Http(error) => error.is_request()` and `Self::Auth(error) => error.is_missing_credential()`
|
||||
|
||||
A public enum may be matched one level down by the type that owns the variant, such as the gateway matching its own `Unsupported` or the OCR bridge matching `FileRead`. Going a second level down, into a variant of a variant, is what to avoid, the gateway used to do that for `MissingApiKey` and `Unsupported`
|
||||
|
||||
## Facts, not policy
|
||||
|
||||
A predicate reports what happened. The decision to retry, reroute or fall back depends on things the error cannot know, such as whether the request is idempotent, whether its body can be replayed, the deadline and the attempt budget. That decision lives with the caller. `dispatched()` exists for the router's reroute decision, so every transport failure answers it even before that caller is wired
|
||||
|
||||
A timeout is the main trap. A request that timed out may already have reached the provider and been billed, so `Timeout` answers `dispatched() == true` and is not safe to retry as if it were a connect failure. Only a failure that provably happened before any byte reached the provider answers `false`: connecting, building or signing the request, and media downloads, which fetch the caller's document before the provider call is made
|
||||
|
||||
```rust
|
||||
pub fn dispatched(&self) -> bool {
|
||||
match &*self.0 {
|
||||
Repr::Network(_) | Repr::Timeout(_) | Repr::Status { .. } => true,
|
||||
Repr::Connect(_) | Repr::RequestBody(_) | Repr::Download(_) | Repr::BlockedUrl => false,
|
||||
// ... every other variant, listed by name
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When the transport cannot tell whether a request reached the provider, it must answer `true`. Wrongly answering `false` risks a duplicate billed call, wrongly answering `true` only costs a reroute
|
||||
|
||||
Keep one predicate per question. `status()` currently mixes the provider's status, the media download's status and a synthesized 408 for timeouts, so a caller that wants "did the provider answer" must use `upstream_response()` instead. Do not add more meanings to an existing predicate, add a new one
|
||||
|
||||
## Equality
|
||||
|
||||
`Arc<Repr>` errors compare with `Arc::ptr_eq`, so a clone equals its original and two separately built errors never compare equal. That exists only so a wrapping enum such as `litellm_inference::Error` can keep its `PartialEq` derive. Do not give a new type pointer equality for any other reason, and do not rely on it in tests. Assert predicates instead, see [testing](testing.md)
|
||||
|
|
@ -0,0 +1,92 @@
|
|||
# Conversions
|
||||
|
||||
## One From per source and target pair
|
||||
|
||||
A `From<Foreign> for Ours` exists when the conversion is obvious, infallible and needs no context from the call site. It does all classification and sanitizing for that pair, so every call site is `?` or `.map_err(Ours::from)`
|
||||
|
||||
```rust
|
||||
impl From<reqwest::Error> for Error {
|
||||
fn from(error: reqwest::Error) -> Self {
|
||||
let error = error.without_url();
|
||||
if error.is_timeout() {
|
||||
Repr::Timeout(error).into()
|
||||
} else if error.is_connect() || error.is_builder() {
|
||||
Repr::Connect(error).into()
|
||||
} else {
|
||||
Repr::Network(error).into()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A call site never re-derives that classification
|
||||
|
||||
```rust
|
||||
// every route used to repeat this, and each drifted a little
|
||||
.map_err(|err| {
|
||||
if err.is_connect() || err.is_builder() {
|
||||
Error::Transport(Connect(err.to_string()))
|
||||
} else {
|
||||
Error::Transport(Network(err.to_string()))
|
||||
}
|
||||
})?
|
||||
|
||||
// now
|
||||
.map_err(litellm_http::Error::from)?
|
||||
```
|
||||
|
||||
The same foreign type may convert into several of our types, one `From` each. What is not allowed is two places converting the same pair differently
|
||||
|
||||
## When the call site knows something the From cannot
|
||||
|
||||
One `io::Error` can mean "reading the CA bundle failed" or "resolving the media host failed". A global `From<io::Error>` cannot know which, so there is none. Name the operation at the call site instead
|
||||
|
||||
```rust
|
||||
.map_err(Repr::Resolve)?
|
||||
.map_err(|source| Repr::Read { path, tls_source, source })?
|
||||
```
|
||||
|
||||
Do not drop meaningful context just to use `?`
|
||||
|
||||
## Reclassify once, in the shared From
|
||||
|
||||
When a lower error needs a different meaning for every caller, do it in the upper type's `From`, not per route
|
||||
|
||||
```rust
|
||||
impl From<litellm_http::Error> for Error {
|
||||
fn from(error: litellm_http::Error) -> Self {
|
||||
if error.is_computed_header() {
|
||||
return Self::Unsupported("request forwards a header AWS SigV4 computes");
|
||||
}
|
||||
Self::Http(error)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
All four inference routes classify a header the signer computes the same way, because none of them does it themselves
|
||||
|
||||
## Nest, translate or add context, never mirror
|
||||
|
||||
| The upper layer | Do | Example |
|
||||
|---|---|---|
|
||||
| has nothing to add | nest in one transparent variant and delegate predicates | `Self::Http(error) => error.is_request()` |
|
||||
| changes what the failure means | translate to an upper variant, keep the cause attached when it helps | computed header -> `Unsupported` |
|
||||
| knows something the lower one does not | a variant with the extra fields plus the cause as its source | `Read { path, tls_source, source }` |
|
||||
|
||||
Never re-declare a lower crate's variants in the upper type with an upper-layer name and a 1:1 mapping function. OCR once did that for every media download failure, `MediaError::BlockedUrl => Error::BlockedDocumentUrl` and so on, which doubled every change and let the two lists drift.
|
||||
|
||||
## Third-party and host faults
|
||||
|
||||
Converting a fault from a crate we do not own into a string variant is still flattening. Prefer a variant that carries the fault, and when an existing `ErrorDetail` constructor fits, use it
|
||||
|
||||
```rust
|
||||
// flattens
|
||||
.map_err(|error| Error::InvalidRequest(error.to_string().into()))
|
||||
|
||||
// keeps the cause typed
|
||||
.map_err(|error| Error::InvalidRequest(ErrorDetail::invalid("header", error)))
|
||||
```
|
||||
|
||||
## Through a foreign error type
|
||||
|
||||
A trait we implement for another crate sometimes forces its error type on us, such as `io::Error` from a body stream or `sqlx::Error` from a driver. Wrap ours as that type's source (`io::Error::other(error)`, `azure_core::Error::new(kind, error)`) and recover it with `downcast_ref` on the other side, never `with_message(error.to_string())` or `from_msg`. The error must come back out as the same value it went in as, so one failure classifies the same way on every path
|
||||
56
litellm-rust/.agents/skills/rust-error/references/gateway.md
Normal file
56
litellm-rust/.agents/skills/rust-error/references/gateway.md
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
# Gateway responses
|
||||
|
||||
The gateway is where a Rust error becomes an HTTP response. This page is about the path an error takes to get there and the one place it is classified. The gateway crates' own conventions, such as which envelope each API surface uses, which extractors exist and how the router is composed, live in the [`rust-gateway`](../../rust-gateway/SKILL.md) skill
|
||||
|
||||
## The flow
|
||||
|
||||
```text
|
||||
reqwest / provider failure
|
||||
-> litellm_http::Error From<reqwest::Error>, strips the URL, classifies once
|
||||
-> litellm_inference::Error::Http(_) From, one transparent variant, predicates delegate
|
||||
-> gateway_inference::Error::Route(_) From, one variant per nested crate
|
||||
-> handler returns Result<_, Error> every `?` ends here
|
||||
-> IntoResponse status() picks the code, a renderer builds the body
|
||||
```
|
||||
|
||||
Each hop is one `From` that nests the lower error in one variant, see [conversions](conversions.md). No hop re-derives what the lower crate already decided, and nothing between the provider and `IntoResponse` formats an error for the client. That is what makes "one `Error` with `IntoResponse` per gateway crate that serves a router" work: a handler can `?` any failure from any depth and the response comes out right
|
||||
|
||||
## Classify once
|
||||
|
||||
`status()` is pure and runs once per error. Variants the gateway declares itself, such as a bad body, an unknown model or an oversized body, are matched at the top of its own enum. A nested route error goes to `route_status` and an OCR error to `ocr_status`, which ask the predicates [boundaries](boundaries.md) lists
|
||||
|
||||
| Answer | Status |
|
||||
|---|---|
|
||||
| `is_unsupported()` | 501, the gateway matched and parsed the request but core cannot serve it, nothing reached the provider |
|
||||
| `upstream_response()` | the provider's own status, 502 when it is not a valid code |
|
||||
| `is_missing_credential()` | 401 |
|
||||
| `is_request()` | 400, for a route error |
|
||||
| `http_status_code()` | the transport's status, for an OCR error, such as 408 for a timeout |
|
||||
| otherwise | 500 |
|
||||
|
||||
Do not reach into a nested enum's variants from the gateway. When a new nested failure needs a different status, add a predicate on the nested error, see [classification](classification.md)
|
||||
|
||||
## Render by surface and framing
|
||||
|
||||
Rendering is separate from classification. The surface picks the envelope and the moment picks the framing, a JSON body before the headers went out and an `event: error` SSE frame after. Both framings of one surface render the same status and `Report` message, so a streaming failure and a JSON failure of the same error agree on everything but framing
|
||||
|
||||
| Surface | JSON | SSE |
|
||||
|---|---|---|
|
||||
| Anthropic, `/v1/messages` | `body(request_id)` | `sse_frame()` |
|
||||
| OpenAI, chat, responses and the rest | `openai_response()` | `responses_sse_frame()`, the Responses API `error` event |
|
||||
|
||||
The message of anything that is not a provider answer is the `litellm_error::Report` of the error, and in the OpenAI and Anthropic JSON envelopes the error `type` string is derived from the status by `error_type()`, so a new failure never names its own type. The Responses `error` event has the fixed type `error` and the status as its `code`. The Anthropic envelope passes a provider body already in its shape through untouched and extracts the message from anything else. The OpenAI envelope always wraps, because that is what the Python proxy returns there
|
||||
|
||||
## Every way in uses the same Error
|
||||
|
||||
A request can fail before the handler runs, in the handler, or after the headers went out. All three end in the same `Error`, so they get the same classification and envelope
|
||||
|
||||
- An extractor sets `type Rejection = Error`, so a bad body is the surface's envelope rather than axum's plain text
|
||||
- A path the gateway knows but does not serve returns `Error::Unsupported(path)` from one handler, so the client gets a 501 in the surface's envelope rather than a bare 404
|
||||
- A stream that fails after its headers went out hands the surface's SSE frame of `Error::from(error)` to `litellm_host_http::Sse::new`. `litellm_host_http::Error<E>` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, and the `From` maps the second to `Internal`
|
||||
|
||||
The axum hooks behind each line are in [`rust-gateway`](../../rust-gateway/SKILL.md)
|
||||
|
||||
## Testing
|
||||
|
||||
`status()` and `body()` are pure, so unit tests call them on errors built the way production builds them, one `#[case]` per fault, as `status_follows_who_is_at_fault`, `an_anthropic_provider_body_passes_through` and `other_failures_get_the_anthropic_envelope_with_their_message` in `crates/gateway-inference/src/error.rs` do. Assert the status and the fields the surface promises, not the whole body, except where passing the body through untouched is the contract. Driving a whole router belongs to the gateway crate's own tests, see [`rust-gateway`](../../rust-gateway/references/testing.md)
|
||||
|
|
@ -0,0 +1,112 @@
|
|||
# Messages and causes
|
||||
|
||||
## Messages belong to the variant
|
||||
|
||||
Every message template is written once, in the variant's `#[error(...)]`. A caller passes the typed facts the template needs and nothing else
|
||||
|
||||
```rust
|
||||
// none of these may appear at a call site
|
||||
Error::Network("WebSocket handshake returned a wrong Sec-WebSocket-Accept".into())
|
||||
Error::Request(format!("bad header {name}"))
|
||||
Error::Signature(error.to_string())
|
||||
|
||||
// the call site names what happened, the variant owns the words
|
||||
Repr::Handshake(HandshakeFault::Accept).into()
|
||||
Error::signature(error)
|
||||
Error::NonAsciiHeader { name }
|
||||
```
|
||||
|
||||
When a check fails for a distinct reason, that reason is a variant or a field, never a string inside a generic variant. A `Message(String)` variant, or a `From<String>` / `From<&str>` into an error, is the hole every stringly call site goes through, so neither exists
|
||||
|
||||
Format arguments can call functions and reach into fields, so a hand-written `Display` is rarely needed
|
||||
|
||||
```rust
|
||||
#[error("could not read {}", path.display())]
|
||||
#[error("invalid model: {provider} has no model {model:?} - use one of: {}", supported.join(", "))]
|
||||
```
|
||||
|
||||
Keep messages short and lowercase, with no trailing period, unless Python parity pins the exact text. A reporter appends `: {cause}` after them. Never format `Debug` (`{:?}`) of an error, or of a type with no `Display`, into a message
|
||||
|
||||
## The cause is the source, never the message
|
||||
|
||||
This is the rule from the [`std::error::Error` docs](https://doc.rust-lang.org/std/error/trait.Error.html): an error's `Display` describes that layer only, and the error underneath is returned by `source()`. thiserror wires `source()` from a field marked `#[source]`, a `#[from]` field, or a field named `source`
|
||||
|
||||
```rust
|
||||
// prints the cause twice under any reporter
|
||||
#[error("could not read {}: {source}", path.display())]
|
||||
Read { path: PathBuf, source: io::Error },
|
||||
#[error("invalid JSON: {0}")]
|
||||
Json(#[from] serde_json::Error),
|
||||
|
||||
// this layer's words, the cause is the source
|
||||
#[error("could not read {}", path.display())]
|
||||
Read { path: PathBuf, source: io::Error },
|
||||
#[error("invalid JSON")]
|
||||
Json(#[from] serde_json::Error),
|
||||
```
|
||||
|
||||
| Field | thiserror does | Use for |
|
||||
|---|---|---|
|
||||
| `source: E` or `#[source] cause: E` | exposes it as `source()` | a cause with context fields next to it |
|
||||
| `#[from] E` | implies `#[source]` and adds `From` | context-free wrapping, see [conversions](conversions.md) |
|
||||
| `#[error(transparent)]` | forwards both `Display` and `source()` to the inner value | nesting another of our errors that adds no text |
|
||||
| no attribute, any other name | nothing, the field is plain data | facts the message renders, such as a path or a status |
|
||||
|
||||
`#[error(transparent)]` is an invisible layer, not an extra chain node, so a transparent wrapper's `source()` is the inner value's `source()`
|
||||
|
||||
A field that holds an error but is neither rendered nor a source hides that error from every reader. Every error-typed field is a source. A fieldless reason enum, such as `HandshakeFault` or `PagesFault`, is data and not a cause, so the variant renders it in its message
|
||||
|
||||
`source()` is one chain, so an error with several causes, such as `CredentialChain` trying each credential in turn, makes the most relevant one its source and renders the others in its message, each with `Report`
|
||||
|
||||
## Keep causes typed
|
||||
|
||||
Store the lower-level error itself, never its string, and never drop it with `map_err(|_| ...)`
|
||||
|
||||
| Flattening or dropping loses | Because |
|
||||
|---|---|
|
||||
| the chain | a `String` has no `source()` |
|
||||
| downcasting | the type is gone |
|
||||
| redaction | `reqwest::Error`'s `Display` includes the request URL, which can carry credentials in the path or query |
|
||||
|
||||
If the cause is not `Clone` and the error must be, hold it as `#[source] litellm_error::Source`, a cloneable `Arc<dyn Error + Send + Sync>` that is transparent to `Display` and `source()`, or make the crate's error opaque. `litellm_error::find::<T>(&error)` walks the chain and looks through `Source`, so it is how anyone reaches a typed cause. A third-party `Box<dyn Error + Send + Sync>` converts with `Source::from`, and an `anyhow::Error` a dependency returns goes through `Box` first, `Source::from(Box::<dyn Error + Send + Sync>::from(error))`, and a dependency's own boxed alias such as `tokenizers::Error` is held as it is. A source boxed to keep the variant small is a `Box<E>` node of the chain, so it is reached as `find::<Box<E>>`. Never put `#[error(transparent)]` on a `Source`, the erased error would then be no node of the chain and `find` could not reach it. Do not stringify or drop a cause to get `Clone` or `PartialEq`
|
||||
|
||||
```rust
|
||||
// loses the type and leaks the URL
|
||||
.map_err(|error| Error::Network(error.to_string()))
|
||||
// loses everything
|
||||
.map_err(|_| Error::Transport)
|
||||
|
||||
// keeps the cause, the From impl strips the URL
|
||||
.map_err(litellm_http::Error::from)
|
||||
```
|
||||
|
||||
A cause that must not be shown at all, such as a secret manager's response, is still a source. The type derives [`veil::Redact`](https://docs.rs/veil) with `#[redact]` on the field, so `Debug` hides it, and the boundary decides how much of the chain to show, see [boundaries](boundaries.md)
|
||||
|
||||
## Strip reqwest URLs on entry
|
||||
|
||||
Every `reqwest::Error` enters our types through `From<reqwest::Error> for litellm_http::Error`, which calls `without_url()`. A crate that talks HTTP holds `litellm_http::Error`, never `reqwest::Error`. When a dependency's own error wraps a `reqwest::Error`, as vaultrs and rustify do, one `From` takes the reqwest error out and converts it, as `secrets-hashicorp` does. That way so every later `Display`, `Debug` or chain walk is safe from that URL
|
||||
|
||||
`without_url()` only removes the URL reqwest attached. It does not scrub headers, response bodies, provider messages, file paths or anything a variant stores next to it. Those need their own policy, see [boundaries](boundaries.md)
|
||||
|
||||
## Rendering
|
||||
|
||||
Anything that shows an error to someone else renders the whole chain with `litellm_error::Report`, which writes each layer once joined by `: `, the single-line form of the unstable [`std::error::Report`](https://doc.rust-lang.org/std/error/struct.Report.html). `to_string()` of an error is only its top layer and is never what a client, a Python exception or a log line shows
|
||||
|
||||
```rust
|
||||
let message = litellm_error::Report::new(&error).to_string();
|
||||
// "could not reach the provider: error sending request: client error (Connect): tcp connect error: Connection refused"
|
||||
```
|
||||
|
||||
## #[from] limits
|
||||
|
||||
`#[from]` only works on a variant whose only fields are the source and an optional backtrace. A variant that needs context such as a path, a phase or a `TlsSource` takes its cause through an explicit `map_err` or a named constructor
|
||||
|
||||
```rust
|
||||
std::fs::read(&path).map_err(|source| Repr::Read {
|
||||
path: path.clone(),
|
||||
tls_source: TlsSource::CaBundle,
|
||||
source,
|
||||
})?;
|
||||
```
|
||||
|
||||
`#[backtrace]` and automatic `provide()` need a nightly compiler and are not used here
|
||||
74
litellm-rust/.agents/skills/rust-error/references/pyo3.md
Normal file
74
litellm-rust/.agents/skills/rust-error/references/pyo3.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# Python bridge
|
||||
|
||||
`crates/python-bridge` turns a Rust error into a Python exception through one mapping function per error type. A `#[pyfunction]` returns `Err(PyErr)` and pyo3 raises it, so those functions are the whole boundary
|
||||
|
||||
| Error | Mapping |
|
||||
|---|---|
|
||||
| `litellm_inference::Error` from a route | `errors::route_failure`, with `route_error_to_pyerr` as its shorthand for routes that set no marker |
|
||||
| the OCR error | `route_failure` and `to_pyerr` in `routes/ocr/errors.rs` |
|
||||
| token counter, cache, spend, secrets, argument checks | one `From<Error> for PyErr` or mapping function next to the code that registers the functions |
|
||||
|
||||
A route host never builds a `PyErr` from a route error itself. It calls the mapping, and the mapping is the only place that knows the exception classes and their attributes
|
||||
|
||||
## Exception classes
|
||||
|
||||
| Class | Meaning | Payload |
|
||||
|---|---|---|
|
||||
| `RustUpstreamError` | the provider call was issued and the provider answered with a failure | `args == (status, body)` and a `headers` attribute of `(name, value)` pairs, always set |
|
||||
| `RustBridgeDeclined` | the route stopped before calling the provider, so the host may run its own path | `args == (message,)` |
|
||||
| a standard class from `pyo3::exceptions` | whichever Python class means the same thing, such as `ValueError` or `TypeError` for the caller's input, `FileNotFoundError` or `OSError` for a file, `KeyError` or `OverflowError` where Python raises them, and `RuntimeError` for anything else | the rendered chain, or a parity text where Python pins one |
|
||||
|
||||
The two native classes come from `create_exception!` in `crates/python-bridge/src/errors.rs` and are exported with `#[pymodule_export]`, which is how the [pyo3 guide](https://github.com/PyO3/pyo3/blob/main/guide/src/exception.md) makes an exception importable. Their docstrings are the contract Python reads. `errors::upstream_error(py, status, body, headers)` is the one constructor of `RustUpstreamError`, so the payload cannot differ between routes
|
||||
|
||||
Python decodes a `RustUpstreamError` in `litellm/rust_bridge/failures.py`, which validates `args` and `headers` with a `TypeAdapter` and rebuilds the provider's `httpx.Response`. `litellm/rust_bridge/runtime.py` catches `RustBridgeDeclined` and reads its reason. `_raise_upstream` in `runtime.py` still reads `args` a second time by hand for the `attempt` path, which is the one decode to fold into `failures.py`. A new fact goes into `args` or `headers`, and the decoder learns to read it in the same change
|
||||
|
||||
Two routes also flag a request fault with a boolean attribute, read by their Python route module: `ocr_request_format_error` in `host/routes/ocr.py` and `messages_request_error` in `host/routes/messages.py`. OCR additionally sets `status_code` and `message` for Python parity, and `native_settings_error`, which only the OCR host reads back to tell a settings failure from the caller's exception. Do not add another attribute, nothing on the Python side types them
|
||||
|
||||
## Why args, not a typed subclass
|
||||
|
||||
The bridge builds with pyo3's `abi3-py310` feature. Under abi3, `#[pyclass(extends = PyException)]` is only allowed on Python 3.12 and newer, so a class with typed fields is not available. Even where it is allowed, pyo3 passes an empty tuple to the native base's `tp_new`, so `.args` and `str()` would need manual overrides. The payload therefore travels in `args`, which a `create_exception!` class fills from `new_err`
|
||||
|
||||
## Mapping order
|
||||
|
||||
A Python exception raised inside the call, such as a secret manager callback's, is returned first as it is. Then the provider's answer passes through as `RustUpstreamError`. On `/v1/messages` a request the route rejected becomes a `ValueError` flagged with `messages_request_error`. Everything else is split by `is_request()` into `ValueError` or `RuntimeError`, with the message rendered by `litellm_error::Report`. A missing API key is a `RuntimeError` because `is_request()` answers false for it, and `missing_api_key_stays_a_runtime_error_while_other_auth_failures_are_value_errors` pins that
|
||||
|
||||
OCR puts Python parity arms in front of that order, each matching a top-level variant of its own public enum
|
||||
|
||||
| Arm | Exception |
|
||||
|---|---|
|
||||
| the provider answered | `RustUpstreamError` with the provider's headers |
|
||||
| `RequestFormat` | `ValueError` flagged with `ocr_request_format_error` |
|
||||
| `FileRead` whose cause is `NotFound` | `FileNotFoundError` naming the path |
|
||||
| other `FileRead` | `OSError` with the io message |
|
||||
|
||||
Both routes ask `litellm_auth::Error::is_request()` through their own `is_request()`, so a missing key or API base is a `RuntimeError` on both, and a missing setting the caller can pass, such as a region, is a `ValueError`
|
||||
|
||||
Use the standard class whose Python meaning matches. `PyFileNotFoundError` for a missing file is right, a `RuntimeError` whose text says the file is missing is not. `import_exception!` exists for an exception defined in Python, none is used today because `failures.py` maps to LiteLLM's public exceptions through `litellm.exception_type`
|
||||
|
||||
## Declining
|
||||
|
||||
`RustBridgeDeclined` means no provider call was made and the host may retry on its own path. It is raised before dispatch, by the tokenizer, embeddings and token counter routes. Never raise it for a failure after the request went out, the host would repeat a billed call. `route_failures_are_terminal` in `errors.rs` pins that for the inference routes
|
||||
|
||||
## Rules
|
||||
|
||||
- Never format `Debug` of an error into a `PyErr`
|
||||
- Never build a `PyErr` from a string at a call site. A failed argument check is a variant of a typed error, as `ProjectionError` in `coercion.rs` is, mapped once
|
||||
- A Python exception raised inside the call stays that exception. Return it as is, never wrap it in a Rust message or stringify it into a new `ValueError`. The one policy exception is a caller-supplied callable, whose failure `callable::wrap_failure` reports as a `RuntimeError` with the original as `__cause__`
|
||||
- A `panic!`, `unwrap` or `expect` inside a `#[pyfunction]` or `#[getter]` reaches Python as `pyo3_runtime.PanicException`, which is not an `Exception` subclass. Model the failure as a value instead
|
||||
- When the mapping itself calls Python, as `setattr` does, propagate its failure with `?` rather than discarding it with `.ok()`
|
||||
|
||||
## Testing
|
||||
|
||||
Build the Rust error the way production builds it, map it, and assert the contract Python sees. `Python::initialize()` once, then `Python::attach`
|
||||
|
||||
```rust
|
||||
Python::initialize();
|
||||
Python::attach(|py| {
|
||||
let failure = route_error_to_pyerr(litellm_http::Error::upstream(429, "slow down".into()).into());
|
||||
assert!(failure.is_instance_of::<RustUpstreamError>(py));
|
||||
let args: (u16, String) = failure.value(py).getattr("args").unwrap().extract().unwrap();
|
||||
assert_eq!(args, (429, "slow down".into()));
|
||||
});
|
||||
```
|
||||
|
||||
One `#[case]` per fault class, asserting `is_instance_of` for the class and `extract` for the payload. The Python side of the contract is tested in `tests/unit/rust_bridge`, through `failures.py`
|
||||
64
litellm-rust/.agents/skills/rust-error/references/recipes.md
Normal file
64
litellm-rust/.agents/skills/rust-error/references/recipes.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
# Recipes
|
||||
|
||||
## A new failure in litellm_http
|
||||
|
||||
Say media downloads must reject a `Content-Type` outside an allow-list
|
||||
|
||||
1. Add a `Repr` variant with typed facts and the message, `#[error("media type {0:?} is not allowed")] DisallowedMediaType(String)`
|
||||
2. Return it where the check fails, `return Err(Repr::DisallowedMediaType(content_type.to_owned()).into())`
|
||||
3. Add an arm to every predicate's exhaustive `match`, the compiler lists them, the way its neighbor `DownloadTooLarge` answers. A media download happens before the provider call, so it is not in `dispatched()`. The caller chose the URL, so it is in `is_request()`
|
||||
4. Leave both boundaries alone. With those answers the gateway returns 400 and Python raises `ValueError`
|
||||
5. Test it with a mock server answering `Content-Type: application/x-msdownload` and assert `error.is_request()`
|
||||
|
||||
## A provider-specific failure
|
||||
|
||||
A second OCR provider needs a missing-key error whose text names its environment variable and how to set it
|
||||
|
||||
1. Look for an existing provider-neutral variant whose template fits. `litellm_auth::Error::MissingApiKey { provider, environment_variable }` renders `Missing {provider} API Key - Set ...`
|
||||
2. If none fits, find what varies between providers with the same failure, here the environment variable and the guidance after the dash, and add one neutral variant with those as fields, `#[error("Missing {environment_variable} - {guidance}")] MissingCredential { environment_variable: &'static str, guidance: &'static str }`
|
||||
3. Pass the provider's values from its transformation. The next provider passes its own and adds no variant, and `is_missing_credential()` lists `MissingCredential { .. }` once
|
||||
|
||||
## Wrapping a new lower crate in litellm_inference
|
||||
|
||||
Say routes start calling `litellm_cache` and its errors must surface
|
||||
|
||||
1. Add one variant, `#[error(transparent)] Cache(#[from] litellm_cache::Error)`
|
||||
2. Extend every predicate's `match` with an arm that calls through to the cache error's own predicates, adding a predicate there if it has none
|
||||
3. Do not copy the cache error's variants into `litellm_inference::Error`
|
||||
|
||||
## A foreign error that needs context
|
||||
|
||||
Reading a file fails with `io::Error`, and the message must name the file
|
||||
|
||||
1. A variant with the context and the cause, `Read { path: PathBuf, source: io::Error }` with `#[error("could not read {}", path.display())]`
|
||||
2. Build it with `map_err(|source| Repr::Read { path: path.clone(), source })`, not a `From<io::Error>`
|
||||
3. If the error must be `Clone`, the crate's error is opaque or the field is `litellm_error::Source`
|
||||
|
||||
## Migrating a stringly error
|
||||
|
||||
Call sites look like `Error::Network(error.to_string())`, `Error::Request(format!(...))` or `.map_err(|_| Error::Unavailable)`
|
||||
|
||||
1. List every call site of the string variant and group them by what actually failed
|
||||
2. Each group becomes a variant holding the typed cause or facts. Groups no caller tells apart share one
|
||||
3. Where a foreign error was stringified with classification logic around it, move that logic into the `From` and turn the call sites into `?`
|
||||
4. Delete the string variant. If it cannot be deleted yet, nothing new may construct it
|
||||
5. Re-run the boundary tests, since messages a client sees may change
|
||||
|
||||
## Moving a crate to source chains
|
||||
|
||||
A variant reads `#[error("invalid JSON: {0}")] Json(#[from] serde_json::Error)` or stores `cause` and renders `{cause}`
|
||||
|
||||
1. Drop the cause from the template, `#[error("invalid JSON")]`
|
||||
2. Make the cause the source, `#[from]`, `#[source]` or a field named `source`
|
||||
3. Find what renders this error for someone else and make it use `litellm_error::Report`
|
||||
4. Update tests that asserted the cause in `to_string()` to assert it in the report
|
||||
|
||||
## Making a public enum opaque
|
||||
|
||||
Downstream crates match nested variants, or a variant needs a non-`Clone` cause
|
||||
|
||||
1. Rename the enum to `pub(crate) enum Repr` and add `pub struct Error(Arc<Repr>)` with `#[error(transparent)]` and `From<Repr>`
|
||||
2. For each downstream `match`, find the question it asks and add a predicate. Replace the `match` with the predicate
|
||||
3. For each downstream construction, add a constructor taking typed data, or move the failure into this crate
|
||||
4. Swap test assertions to predicates and add the `#[cfg(test)] repr()` accessor for in-crate tests
|
||||
5. Implement `PartialEq` with `Arc::ptr_eq` only if a wrapping enum's derive needs it
|
||||
105
litellm-rust/.agents/skills/rust-error/references/shape.md
Normal file
105
litellm-rust/.agents/skills/rust-error/references/shape.md
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
# Error shape
|
||||
|
||||
## One top-level Error per crate
|
||||
|
||||
A crate's errors live in `src/error.rs` and are re-exported from `lib.rs` as `Error`. A caller of any public function in the crate should only ever see `Result<_, crate::Error>`, so one `?` and one set of predicates cover the whole crate
|
||||
|
||||
This deliberately departs from the advice to put an error type next to each unit of fallibility ([Sabrina Jewson, "Modular Errors"](https://sabrinajewson.org/blog/errors)). Our callers do not branch on which function failed, they ask the predicates in [classification](classification.md), and the boundaries map one type per crate. Splitting by function would multiply those predicates and mappings without giving any caller a decision it lacks. Layering, the other half of that advice, is kept: each crate nests the one below it, see [conversions](conversions.md)
|
||||
|
||||
That one type takes one of two forms
|
||||
|
||||
| Form | Use when | Example |
|
||||
|---|---|---|
|
||||
| Public enum | Callers outside the crate match variants, and every variant holds our own data or another crate's `Error` that is already `Clone` and comparable | `litellm_inference::Error`, `litellm_auth::Error` |
|
||||
| Opaque struct over a private `Repr` | The variants hold third-party causes, or the variants must be free to change without breaking callers | `litellm_http::Error` |
|
||||
|
||||
Both forms are still one top-level `Error`. The opaque form just moves the enum behind `Repr`, where it is `pub(crate)`
|
||||
|
||||
```rust
|
||||
// public enum
|
||||
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
|
||||
pub enum Error {
|
||||
#[error("missing required field: {0}")]
|
||||
MissingField(&'static str),
|
||||
#[error(transparent)]
|
||||
Http(litellm_http::Error),
|
||||
}
|
||||
|
||||
// opaque struct
|
||||
#[derive(Clone, Debug, thiserror::Error)]
|
||||
#[error(transparent)]
|
||||
pub struct Error(Arc<Repr>);
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub(crate) enum Repr {
|
||||
#[error("could not reach the provider")]
|
||||
Connect(#[source] reqwest::Error),
|
||||
#[error("upstream request failed with status {status}: {body}")]
|
||||
Status { status: u16, body: String },
|
||||
}
|
||||
```
|
||||
|
||||
Start with the public enum. Move to opaque when a variant needs a non-`Clone` third-party cause, or when downstream crates keep matching variants in ways that duplicate classification. `reqwest::Error`, `std::io::Error` and `serde_json::Error` are built the opaque way
|
||||
|
||||
In the opaque form other crates cannot name a variant, so they cannot build an error with their own message. Public constructors exist only for errors another crate genuinely produces, take typed data and never a message, as in `Error::upstream(status, body)`, `Error::computed_header(name)`, `Error::unsignable_body()` and `Error::signature(cause)`
|
||||
|
||||
`Arc` makes cloning a refcount bump, so `Repr` never has to be `Clone`. `Box<Repr>` is enough when the goal is only a small `Result`. `Arc` is for when callers clone
|
||||
|
||||
Every crate here is internal to the workspace, so do not mark errors or variants `#[non_exhaustive]`. It only forces a wildcard arm on the other workspace crates and turns off the exhaustiveness check that makes a new variant visit every `match`. When the variants must be free to change, use the opaque form
|
||||
|
||||
## What else may sit next to Error
|
||||
|
||||
These are not a second top-level error and need no folding in
|
||||
|
||||
- Leaf data types whose fields are typed and whose message is in `#[error(...)]`, such as `litellm_http::request::HeaderError`. A wrapping crate nests it as one variant
|
||||
- Plain data used by predicates, such as `litellm_http::TlsSource`
|
||||
- A private fault enum one variant carries, such as `PemFault` inside `Repr::InvalidPem`
|
||||
|
||||
## Variants are failure modes
|
||||
|
||||
A public variant is something a caller handles differently: a phase, a status code, retry versus no retry, fault on the request versus the environment, or a message Python parity pins exactly. Failures no caller tells apart share one variant and differ in their data
|
||||
|
||||
```rust
|
||||
// one variant per provider, so every new provider adds a variant and a match arm
|
||||
#[error("Missing Azure API Base - set AZURE_API_BASE")]
|
||||
MissingAzureApiBase,
|
||||
#[error("Missing Vertex API Base - set VERTEXAI_API_BASE")]
|
||||
MissingVertexApiBase,
|
||||
|
||||
// one failure mode, the provider and its guidance are data
|
||||
#[error("Missing {provider} API Base - {guidance}")]
|
||||
MissingApiBase { provider: &'static str, guidance: &'static str },
|
||||
```
|
||||
|
||||
An exact parity message does not justify a provider-specific variant when caller-supplied data can produce the same text
|
||||
|
||||
Inside a private `Repr` a variant costs no API, so one per distinct reason is fine when each one carries a different message. When several only differ in a word and share every predicate answer, prefer one variant with a fieldless reason enum, so predicates list one arm instead of four, as `Repr::Handshake(HandshakeFault)` does
|
||||
|
||||
```rust
|
||||
#[error("WebSocket handshake returned {0}")]
|
||||
Handshake(HandshakeFault),
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub(crate) enum HandshakeFault {
|
||||
#[error("a wrong Sec-WebSocket-Accept")]
|
||||
Accept,
|
||||
#[error("an invalid Upgrade header")]
|
||||
Upgrade,
|
||||
#[error("an invalid Connection header")]
|
||||
Connection,
|
||||
#[error("an invalid subprotocol")]
|
||||
Subprotocol,
|
||||
}
|
||||
```
|
||||
|
||||
## When to split
|
||||
|
||||
Split into a separate type only when different functions fail in disjoint ways and different callers see them, so no caller matches variants its function can never return. A type per function overdoes it
|
||||
|
||||
Name a split type after what went wrong, not after the function that returns it. `ConnectError` reads better than `OpenSessionError`, and a unit struct is fine for a single failure mode
|
||||
|
||||
A split type still converts into the crate's `Error` when a caller further up needs one `Result` type
|
||||
|
||||
`litellm-llms` has one recorded exception, the OCR error in `src/base_llm/ocr/error.rs`, kept apart until OCR folds into `litellm_llms::Error`, see `crates/inference-ocr/AGENTS.md`. Do not add another
|
||||
|
||||
A utility crate whose modules are unrelated, such as `core-utils`, has no crate-wide failure, so each module that can fail owns one error next to its code instead of a `src/error.rs`. The rest of this skill applies to each of them
|
||||
37
litellm-rust/.agents/skills/rust-error/references/testing.md
Normal file
37
litellm-rust/.agents/skills/rust-error/references/testing.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# Testing errors
|
||||
|
||||
Assert behavior through predicates and accessors, on errors produced the way production produces them
|
||||
|
||||
```rust
|
||||
// guesses the message and the variant, and cannot be written once the type is opaque
|
||||
assert_eq!(error, Error::Transport(TransportError::Http { status: 429, body: "slow down".into() }));
|
||||
|
||||
// asserts what callers rely on
|
||||
assert_eq!(error.upstream_response(), Some((429, "slow down")));
|
||||
assert!(error.dispatched());
|
||||
assert!(!error.is_request());
|
||||
```
|
||||
|
||||
## Producing the error
|
||||
|
||||
In order of preference
|
||||
|
||||
1. A real failing call. Bind a `TcpListener`, drop it, then connect to its address for a connect failure. Accept and never answer for a timeout
|
||||
2. A real cause, such as `tungstenite::Error::ConnectionClosed` or a `serde_json::from_str("{")` error, passed through our `From`
|
||||
3. A public constructor, such as `Error::upstream(500, "boom".into())`
|
||||
|
||||
Inside the owning crate a test may match the private representation through the `#[cfg(test)] fn repr(&self) -> &Repr` accessor in `crates/http/src/error.rs`, as the `media.rs` and `tls.rs` tests do. Anywhere else, reach the cause the test expects with `litellm_error::find::<T>(&error)`. A transparent layer is no node of the chain, so `find` reaches foreign causes underneath it but never the nested error of ours itself. Ask that error's predicates through the outer type instead, which is what callers do too
|
||||
|
||||
## What to cover
|
||||
|
||||
| Property | Test shape |
|
||||
|---|---|
|
||||
| classification | one `#[case]` per input class, asserting the predicate |
|
||||
| redaction | build from a URL holding `api_key=secret`, assert the message lacks `secret` and the path |
|
||||
| cause kept | `litellm_error::find::<T>(&error)` returns the expected cause |
|
||||
| rendered once | the cause's text appears exactly once in `litellm_error::Report::new(&error).to_string()` and not in `to_string()` |
|
||||
| boundary mapping | the gateway status and body, or the Python exception type and its `status` attribute |
|
||||
|
||||
Assert exact message text only where it is a contract, such as a message Python parity pins. Everywhere else assert that the message contains the facts it must carry. Matching a fieldless variant with `matches!` is fine, full `assert_eq!` on an error that holds a cause is not, since causes compare by pointer
|
||||
|
||||
A test that returns `Result` reports only `Debug` of the error on failure. Prefer `expect` with a short reason, or `unwrap_err` followed by assertions on the predicates
|
||||
20
litellm-rust/.agents/skills/rust-gateway/SKILL.md
Normal file
20
litellm-rust/.agents/skills/rust-gateway/SKILL.md
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
---
|
||||
name: rust-gateway
|
||||
description: Add or change an HTTP route, extractor, response envelope, stream or router mount in the litellm-rust gateway crates (crates/gateway and crates/gateway-*). Use when adding an endpoint, parsing a request body, choosing an error envelope for an API surface, relaying SSE, or writing a router-level test
|
||||
---
|
||||
|
||||
# Rust gateway
|
||||
|
||||
`crates/gateway` mounts endpoint routers and serves the listener. `crates/gateway-inference`, `gateway-mcp` and `gateway-ui` own their endpoints, `gateway-auth` is the inbound authentication middleware and `gateway-management` holds management logic that has no router yet. A crate's `AGENTS.md`, where it has one, says what belongs in it. Read the reference for the part you are touching
|
||||
|
||||
- [surfaces](references/surfaces.md)
|
||||
- Each API surface mirrors the envelope the Python proxy returns there, so the shapes differ on purpose
|
||||
- The error type string is derived from the status and provider bodies pass through bounded
|
||||
- [requests](references/requests.md)
|
||||
- Extractors reject with the crate's `Error`, handlers take the body as a `Result` when the error needs the request id
|
||||
- Every alias of a path is registered together, a known but unserved path goes to one handler
|
||||
- Streams relay through `litellm_host_http::Sse`, with the error frame rendered by the crate's `Error`
|
||||
- [testing](references/testing.md)
|
||||
- Build the router through `tests/support` and drive it with `oneshot`, assert the fields the surface promises
|
||||
|
||||
How an error is classified before it reaches a response is the [`rust-error`](../rust-error/SKILL.md) skill, in particular its [gateway](../rust-error/references/gateway.md) page. Axum's own conventions are in its [docs](https://docs.rs/axum/latest/axum/)
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
# Requests
|
||||
|
||||
## Extractors reject with the crate's Error
|
||||
|
||||
A custom extractor sets `type Rejection = Error`, so a bad body gets the surface's envelope rather than axum's plain-text rejection. Never take axum's own `Json`, `Query` or `Path` in a handler, their rejection is plain text. The rejection is the variant's source, never its string. `JsonObject` and `InferenceBody` in `crates/gateway-inference/src/request.rs` are the pattern
|
||||
|
||||
```rust
|
||||
impl<S: Send + Sync> FromRequest<S> for JsonObject {
|
||||
type Rejection = Error;
|
||||
|
||||
async fn from_request(request: Request, state: &S) -> Result<Self, Self::Rejection> {
|
||||
let body = Bytes::from_request(request, state).await.map_err(|error| {
|
||||
if error.status() == StatusCode::PAYLOAD_TOO_LARGE {
|
||||
Error::BodyTooLarge
|
||||
} else {
|
||||
BodyError::Read(error).into()
|
||||
}
|
||||
})?;
|
||||
Ok(Self(object(&body)?))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When the handler needs something from the request in the error body, such as the request id on `/v1/messages`, take the extractor as `Result<JsonObject, Error>` so the handler maps the rejection itself, as `messages::create` does. `Infallible` is the right rejection for an extractor that only reads an optional header, as `RequestId` does
|
||||
|
||||
## Paths
|
||||
|
||||
Every alias the Python proxy registers for a route is registered together in `crates/gateway-inference/src/lib.rs`, such as the bare path, its `/v1` form and the `/engines/{model}` and `/openai/deployments/{model}` forms, otherwise one alias silently behaves differently from the others. A path the gateway knows but does not serve, such as `/v1/completions`, is routed to `request::unsupported`, which returns `Error::Unsupported(path)` so the client gets a 501 in the surface's envelope rather than a bare 404
|
||||
|
||||
`crates/gateway` merges the mounted routers and adds the shared inbound middleware as `route_layer`s, so authentication runs only for matched routes. Endpoint paths and parsing stay in the mounted crate, see the two crates' `AGENTS.md`
|
||||
|
||||
## Streaming
|
||||
|
||||
Once the response headers are sent the status cannot change. A streaming handler builds `litellm_host_http::Sse::new(Json, |error| Bytes::from(Error::from(error).sse_frame()))`, where the closure renders a mid-stream failure as an `event: error` frame with the same status and message a JSON failure on that surface carries, `sse_frame()` for Anthropic and `responses_sse_frame()` for the Responses API's `error` event. `litellm_host_http::Error<E>` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, and the crate's `From` maps the second to `Internal`
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
# Envelopes follow the API surface
|
||||
|
||||
Each surface mirrors what the Python proxy returns there, so the shapes differ on purpose
|
||||
|
||||
| Surface | Envelope | Where |
|
||||
|---|---|---|
|
||||
| `/v1/messages` | Anthropic, `{"type": "error", "error": {"type", "message"}, "request_id"}` | `Error::body()` and `sse_frame()` in `crates/gateway-inference/src/error.rs` |
|
||||
| OpenAI-shaped routes | `{"error": {"message", "type", "param", "code"}}` | `Error::openai_response()` in the same file |
|
||||
| `/v1/responses` stream | the Responses `error` event, `{"type": "error", "code", "message", "param"}` | `Error::responses_sse_frame()` in the same file |
|
||||
| MCP | `{"detail": message}` | `crates/gateway-mcp/src/error.rs` |
|
||||
| UI | `{"error": {"message"}}`, plus `Retry-After` on 429 | `crates/gateway-ui/src/error.rs` |
|
||||
| auth rejections | plain text, the top-level message only | `crates/gateway-auth/src/error.rs` |
|
||||
|
||||
Every surface takes its status from one `status()` on its crate's `Error` and renders the message of anything that is not a provider answer with `litellm_error::Report`. In `gateway-inference` the error `type` string is derived from the status by `error_type()`, so a new failure never names its own type. The Anthropic envelope passes a provider body already in its shape through untouched and extracts the message from anything else, and the OpenAI envelope always wraps, as the Python proxy does. A provider body is bounded by `litellm_http::Error::upstream` when the error is built
|
||||
|
||||
Two surfaces show less than the chain on purpose, and a test in each pins it. An auth rejection goes to a caller who has not authenticated, so it never shows what a secret manager or key store said. An MCP upstream or authorization failure shows only its own layer, because rmcp renders the transport error, URL included, into its own `Display`, and an authorizer's cause is the same backend detail auth rejections hide. The inference surfaces show only the top layer of an auth rejection for the same reason
|
||||
|
||||
`/v1/messages` adds the request id to the envelope when the provider did not set one, which is why its handler takes the body as `Result<JsonObject, Error>` and renders the rejection itself, see [requests](requests.md)
|
||||
|
||||
A new surface gets its own renderer next to the others, named after the shape it produces, and its status still comes from `Error::status()`. Picking the status is the [`rust-error`](../../rust-error/references/gateway.md) skill's concern, not the surface's
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# Testing
|
||||
|
||||
Test HTTP contracts in the crate that owns the endpoint: status codes, forwarded headers, error envelopes and streaming behavior. Inference and provider behavior is tested in the `inference-<fmt>` and `llms` crates, and mounting and server lifecycle in `crates/gateway`
|
||||
|
||||
`crates/gateway-inference/tests/support/mod.rs` builds a `Router` with injected deployments, pools and settings through `app()` and its variants, and `post()` drives it with `tower::ServiceExt::oneshot`. `json()` reads the response body back. Use these instead of building a router or a client by hand in a test
|
||||
|
||||
Assert the status and the fields the surface promises, not the whole body. A test that pins a whole envelope breaks on every unrelated field, one that checks `status`, `error.type` and the presence of `request_id` catches the regressions that matter
|
||||
|
||||
`status()` and `body()` of the error type are pure, so a classification case belongs in the error module's unit tests, see the [`rust-error`](../../rust-error/references/gateway.md) skill. Router tests cover what only the full path shows, such as an extractor rejection arriving in the right envelope or an SSE error frame after the headers went out
|
||||
|
|
@ -4,6 +4,8 @@ For diagnostic tracing changes, follow [.agents/skills/rust-tracing/SKILL.md](.a
|
|||
|
||||
For string-valued enums and their Serde conversions, follow [.agents/skills/rust-string-enums/SKILL.md](.agents/skills/rust-string-enums/SKILL.md)
|
||||
|
||||
For error types, error variants and conversions between crate errors, follow [.agents/skills/rust-error/SKILL.md](.agents/skills/rust-error/SKILL.md)
|
||||
|
||||
For fieldless enums, derive `strum::VariantArray` and use `VARIANTS` instead of a hand-listed `ALL` array. Derive Strum string conversions instead of hand-written variant-to-string matches
|
||||
|
||||
Use the derived conversions directly (`<&'static str>::from(x)` / `.into()`, `str::parse`) with no `as_str`/`parse` wrapper that only delegates, and spell each variant with explicit `#[strum(serialize = "...")]` instead of `serialize_all`
|
||||
|
|
@ -24,13 +26,3 @@ Use [`#[rstest]`](https://docs.rs/rstest/latest/rstest/attr.rstest.html) for new
|
|||
|
||||
- Never loop over inputs (`for`, `.iter().for_each`, `.all`) inside a test body. Give each input its own `#[case::name(...)]`, or use `#[values(...)]` for a cross product
|
||||
- Exception: a test pinning a Rust table against a repo-owned data file (for example `include_str!` of a JSON config) may iterate that file's entries
|
||||
|
||||
## Error definitions
|
||||
|
||||
- A crate's errors live in `src/error.rs`, defined with `thiserror`, and re-exported from `lib.rs`
|
||||
- Put message templates in the variant's `#[error(...)]` declaration. Callers pass only the small typed arguments needed to fill them, never `Error::Variant(format!(...))` or a preformatted message. Keep the smallest set of neutral variants that callers need to distinguish. Different wording or providers do not justify new variants
|
||||
- Default to one top-level `Error` enum per crate, with one variant per failure mode and a `#[error(...)]` message on each. A failure mode is something a caller handles differently (phase, status code, retry, a message Python parity pins exactly). Failures no caller tells apart share one variant and differ only in its message
|
||||
- Keep shared error enums minimal and provider-neutral. Provider names, credential types, configuration fields, and setup guidance belong in caller-supplied data, not dedicated variants or hardcoded shared messages. Reuse a variant for the same failure mode across providers, such as `MissingApiBase { provider: "Azure", guidance: "..." }`. An exact parity message does not justify a provider-specific variant when caller-supplied context can preserve it
|
||||
- Wrap a lower-level error as a variant with `#[from]` or `#[source]` instead of flattening it to a string
|
||||
- Exception: split into separate types when different functions fail in disjoint ways, especially when different callers see them. A shared enum would force every caller to match variants its function can never return
|
||||
- Name a split type after what went wrong (a unit struct is fine for a single failure mode), not after the function that returns it
|
||||
|
|
|
|||
66
litellm-rust/Cargo.lock
generated
66
litellm-rust/Cargo.lock
generated
|
|
@ -3388,6 +3388,7 @@ dependencies = [
|
|||
"bytes",
|
||||
"futures-util",
|
||||
"litellm-auth-types",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"moka",
|
||||
"reqwest",
|
||||
|
|
@ -3407,6 +3408,7 @@ dependencies = [
|
|||
"azure_core",
|
||||
"azure_identity",
|
||||
"litellm-auth-types",
|
||||
"litellm-error",
|
||||
"moka",
|
||||
"rstest",
|
||||
"serde_json",
|
||||
|
|
@ -3425,6 +3427,7 @@ dependencies = [
|
|||
"google-cloud-auth",
|
||||
"http 1.4.2",
|
||||
"litellm-auth-types",
|
||||
"litellm-error",
|
||||
"moka",
|
||||
"rstest",
|
||||
"serde",
|
||||
|
|
@ -3439,6 +3442,7 @@ dependencies = [
|
|||
name = "litellm-auth-types"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"litellm-error",
|
||||
"rstest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
|
|
@ -3452,6 +3456,7 @@ dependencies = [
|
|||
name = "litellm-cache"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"litellm-error",
|
||||
"rstest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
|
|
@ -3477,6 +3482,7 @@ dependencies = [
|
|||
"reqwest",
|
||||
"rstest",
|
||||
"serde_json",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"url",
|
||||
"wiremock",
|
||||
|
|
@ -3495,6 +3501,7 @@ dependencies = [
|
|||
"serde-pickle",
|
||||
"serde_json",
|
||||
"tempfile",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
|
|
@ -3507,11 +3514,13 @@ dependencies = [
|
|||
"litellm-auth-types",
|
||||
"litellm-cache",
|
||||
"litellm-cache-testing",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"percent-encoding",
|
||||
"reqwest",
|
||||
"rstest",
|
||||
"serde_json",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"wiremock",
|
||||
]
|
||||
|
|
@ -3534,6 +3543,7 @@ dependencies = [
|
|||
"futures-util",
|
||||
"litellm-cache",
|
||||
"litellm-cache-testing",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"qdrant-client",
|
||||
"reqwest",
|
||||
|
|
@ -3554,11 +3564,13 @@ version = "0.1.0"
|
|||
dependencies = [
|
||||
"litellm-cache",
|
||||
"litellm-cache-testing",
|
||||
"litellm-error",
|
||||
"r2d2",
|
||||
"redis",
|
||||
"redis-test",
|
||||
"rstest",
|
||||
"serde_json",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
|
|
@ -3611,6 +3623,7 @@ dependencies = [
|
|||
"litellm-auth-aws",
|
||||
"litellm-cache",
|
||||
"litellm-cache-testing",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"reqwest",
|
||||
"rstest",
|
||||
|
|
@ -3633,11 +3646,13 @@ dependencies = [
|
|||
"litellm-cache",
|
||||
"litellm-cache-redis",
|
||||
"litellm-cache-testing",
|
||||
"litellm-error",
|
||||
"redis",
|
||||
"redis-test",
|
||||
"rstest",
|
||||
"serde_json",
|
||||
"sha2 0.10.9",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"uuid",
|
||||
]
|
||||
|
|
@ -3647,6 +3662,7 @@ name = "litellm-callbacks-legacy-python"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"litellm-auth",
|
||||
"litellm-error",
|
||||
"litellm-host",
|
||||
"litellm-host-python",
|
||||
"proptest",
|
||||
|
|
@ -3654,6 +3670,7 @@ dependencies = [
|
|||
"rstest",
|
||||
"serde_json",
|
||||
"strum",
|
||||
"thiserror 2.0.19",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -3669,6 +3686,7 @@ name = "litellm-config"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"litellm-auth-types",
|
||||
"litellm-error",
|
||||
"litellm-router-types",
|
||||
"rstest",
|
||||
"serde",
|
||||
|
|
@ -3686,6 +3704,7 @@ name = "litellm-core-utils"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"fancy-regex 0.19.2",
|
||||
"litellm-error",
|
||||
"litellm-llms-types",
|
||||
"litellm-tracing",
|
||||
"rstest",
|
||||
|
|
@ -3713,6 +3732,7 @@ dependencies = [
|
|||
"criterion",
|
||||
"proptest",
|
||||
"rstest",
|
||||
"thiserror 2.0.19",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -3742,6 +3762,14 @@ dependencies = [
|
|||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "litellm-error"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"rstest",
|
||||
"thiserror 2.0.19",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "litellm-framer"
|
||||
version = "0.1.0"
|
||||
|
|
@ -3769,6 +3797,7 @@ dependencies = [
|
|||
"litellm-auth-types",
|
||||
"litellm-circuit-breaker",
|
||||
"litellm-config",
|
||||
"litellm-error",
|
||||
"litellm-gateway-auth",
|
||||
"litellm-gateway-inference",
|
||||
"litellm-gateway-mcp",
|
||||
|
|
@ -3802,6 +3831,7 @@ dependencies = [
|
|||
"futures-util",
|
||||
"litellm-auth-types",
|
||||
"litellm-config",
|
||||
"litellm-error",
|
||||
"litellm-secrets",
|
||||
"rstest",
|
||||
"serde",
|
||||
|
|
@ -3825,6 +3855,7 @@ dependencies = [
|
|||
"litellm-auth",
|
||||
"litellm-cache-memory",
|
||||
"litellm-cache-response",
|
||||
"litellm-error",
|
||||
"litellm-gateway-auth",
|
||||
"litellm-host",
|
||||
"litellm-host-http",
|
||||
|
|
@ -3844,6 +3875,7 @@ dependencies = [
|
|||
"serde_json",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"tokio-tungstenite",
|
||||
"tower",
|
||||
"wiremock",
|
||||
]
|
||||
|
|
@ -3854,6 +3886,7 @@ version = "0.1.0"
|
|||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"litellm-auth-types",
|
||||
"litellm-error",
|
||||
"litellm-gateway-auth",
|
||||
"rand 0.8.7",
|
||||
"rstest",
|
||||
|
|
@ -3871,6 +3904,7 @@ dependencies = [
|
|||
"http 1.4.2",
|
||||
"litellm-auth-types",
|
||||
"litellm-config",
|
||||
"litellm-error",
|
||||
"litellm-secrets",
|
||||
"moka",
|
||||
"rmcp",
|
||||
|
|
@ -3885,6 +3919,7 @@ dependencies = [
|
|||
"tower",
|
||||
"url",
|
||||
"uuid",
|
||||
"veil",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -3894,9 +3929,11 @@ dependencies = [
|
|||
"axum",
|
||||
"axum-login",
|
||||
"base64 0.22.1",
|
||||
"futures-util",
|
||||
"governor",
|
||||
"jsonwebtoken",
|
||||
"litellm-auth-types",
|
||||
"litellm-error",
|
||||
"litellm-gateway-auth",
|
||||
"rand 0.8.7",
|
||||
"rstest",
|
||||
|
|
@ -3921,6 +3958,7 @@ dependencies = [
|
|||
"litellm-coroutine",
|
||||
"rstest",
|
||||
"serde_json",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
|
|
@ -3958,6 +3996,7 @@ dependencies = [
|
|||
"bytes",
|
||||
"futures-util",
|
||||
"litellm-coroutine",
|
||||
"litellm-error",
|
||||
"litellm-host",
|
||||
"parking_lot",
|
||||
"pyo3",
|
||||
|
|
@ -3978,6 +4017,7 @@ dependencies = [
|
|||
"http 1.4.2",
|
||||
"hyper-util",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"rcgen",
|
||||
"reqwest",
|
||||
"rstest",
|
||||
|
|
@ -3992,6 +4032,7 @@ dependencies = [
|
|||
"tokio-rustls 0.26.4",
|
||||
"tokio-tungstenite",
|
||||
"tracing",
|
||||
"url",
|
||||
"veil",
|
||||
"webpki-roots",
|
||||
]
|
||||
|
|
@ -4007,6 +4048,7 @@ dependencies = [
|
|||
"litellm-cache-memory",
|
||||
"litellm-cache-response",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-framer",
|
||||
"litellm-host",
|
||||
"litellm-http",
|
||||
|
|
@ -4034,6 +4076,7 @@ dependencies = [
|
|||
"litellm-cache-memory",
|
||||
"litellm-cache-response",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-host",
|
||||
"litellm-host-native",
|
||||
"litellm-http",
|
||||
|
|
@ -4043,7 +4086,6 @@ dependencies = [
|
|||
"litellm-llms-types",
|
||||
"litellm-secrets",
|
||||
"litellm-tracing",
|
||||
"reqwest",
|
||||
"rstest",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
|
|
@ -4062,6 +4104,7 @@ dependencies = [
|
|||
"litellm-cache-memory",
|
||||
"litellm-cache-response",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-framer",
|
||||
"litellm-host",
|
||||
"litellm-host-native",
|
||||
|
|
@ -4093,6 +4136,7 @@ dependencies = [
|
|||
"litellm-auth-aws",
|
||||
"litellm-auth-gcp",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-host",
|
||||
"litellm-host-native",
|
||||
"litellm-http",
|
||||
|
|
@ -4110,6 +4154,7 @@ dependencies = [
|
|||
"time",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"url",
|
||||
"wiremock",
|
||||
]
|
||||
|
||||
|
|
@ -4125,6 +4170,7 @@ dependencies = [
|
|||
"litellm-cache-memory",
|
||||
"litellm-cache-response",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-host",
|
||||
"litellm-host-native",
|
||||
"litellm-http",
|
||||
|
|
@ -4162,6 +4208,7 @@ dependencies = [
|
|||
"futures-util",
|
||||
"litellm-auth",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"litellm-inference",
|
||||
"litellm-inference-testing",
|
||||
|
|
@ -4190,6 +4237,7 @@ dependencies = [
|
|||
"litellm-auth-azure",
|
||||
"litellm-auth-gcp",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-framer",
|
||||
"litellm-host",
|
||||
"litellm-http",
|
||||
|
|
@ -4230,6 +4278,7 @@ version = "0.1.0"
|
|||
dependencies = [
|
||||
"indexmap 2.14.0",
|
||||
"jsonschema",
|
||||
"litellm-error",
|
||||
"litellm-llms-types",
|
||||
"rstest",
|
||||
"schemars 1.2.2",
|
||||
|
|
@ -4256,6 +4305,7 @@ dependencies = [
|
|||
"litellm-cache-response",
|
||||
"litellm-callbacks-legacy-python",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-host",
|
||||
"litellm-host-python",
|
||||
"litellm-http",
|
||||
|
|
@ -4286,6 +4336,7 @@ dependencies = [
|
|||
"serde_with",
|
||||
"sqlx",
|
||||
"strum",
|
||||
"tempfile",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"tokio-tungstenite",
|
||||
|
|
@ -4363,6 +4414,7 @@ dependencies = [
|
|||
"google-cloud-auth",
|
||||
"google-cloud-kms-v1",
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"litellm-python-compat",
|
||||
"litellm-secrets-aws",
|
||||
|
|
@ -4421,6 +4473,7 @@ dependencies = [
|
|||
"sha2 0.10.9",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"url",
|
||||
"veil",
|
||||
"wiremock",
|
||||
]
|
||||
|
|
@ -4444,6 +4497,7 @@ dependencies = [
|
|||
"tempfile",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"url",
|
||||
"veil",
|
||||
"wiremock",
|
||||
]
|
||||
|
|
@ -4470,6 +4524,7 @@ dependencies = [
|
|||
"serde_json",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"url",
|
||||
"veil",
|
||||
"wiremock",
|
||||
]
|
||||
|
|
@ -4479,7 +4534,10 @@ name = "litellm-secrets-hashicorp"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"litellm-core-utils",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"litellm-secrets-types",
|
||||
"reqwest",
|
||||
"rstest",
|
||||
"rustify",
|
||||
"rustify_derive",
|
||||
|
|
@ -4513,6 +4571,7 @@ name = "litellm-spend-clickhouse"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"flate2",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"litellm-storage-clickhouse",
|
||||
"rstest",
|
||||
|
|
@ -4533,6 +4592,7 @@ name = "litellm-storage-clickhouse"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"flate2",
|
||||
"litellm-error",
|
||||
"litellm-http",
|
||||
"rstest",
|
||||
"serde",
|
||||
|
|
@ -4552,6 +4612,7 @@ version = "0.1.0"
|
|||
dependencies = [
|
||||
"flate2",
|
||||
"futures-util",
|
||||
"litellm-http",
|
||||
"reqwest",
|
||||
"rstest",
|
||||
"semver",
|
||||
|
|
@ -4575,6 +4636,7 @@ dependencies = [
|
|||
"criterion",
|
||||
"indexmap 2.14.0",
|
||||
"itoa",
|
||||
"litellm-error",
|
||||
"litellm-token-counter-fast",
|
||||
"litellm-token-counter-huggingface",
|
||||
"litellm-token-counter-tiktoken",
|
||||
|
|
@ -4591,6 +4653,7 @@ name = "litellm-token-counter-fast"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"litellm-error",
|
||||
"rand 0.8.7",
|
||||
"rstest",
|
||||
"rustc-hash",
|
||||
|
|
@ -4616,6 +4679,7 @@ name = "litellm-token-counter-tiktoken"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"litellm-error",
|
||||
"once_cell",
|
||||
"rstest",
|
||||
"rustc-hash",
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ repository = "https://github.com/BerriAI/litellm"
|
|||
|
||||
[workspace.dependencies]
|
||||
litellm-config = { path = "crates/config" }
|
||||
litellm-error = { path = "crates/error" }
|
||||
litellm-router = { path = "crates/router" }
|
||||
litellm-router-types = { path = "crates/router-types" }
|
||||
litellm-tracing = { path = "crates/tracing" }
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ repository.workspace = true
|
|||
|
||||
[dependencies]
|
||||
litellm-auth-types.workspace = true
|
||||
litellm-error.workspace = true
|
||||
litellm-http.workspace = true
|
||||
reqwest.workspace = true
|
||||
|
||||
|
|
|
|||
|
|
@ -15,13 +15,13 @@ use reqwest::header::{HeaderMap, HeaderName, HeaderValue};
|
|||
|
||||
use litellm_auth_types::AwsParams;
|
||||
|
||||
use super::Error;
|
||||
use super::constants::{
|
||||
AWS_ACCESS_KEY_ID, AWS_EXTERNAL_ID, AWS_PROFILE_NAME, AWS_REGION_NAME, AWS_ROLE_ARN,
|
||||
AWS_ROLE_NAME, AWS_SECRET_ACCESS_KEY, AWS_SESSION_NAME, AWS_SESSION_TOKEN,
|
||||
AWS_SIGNED_HEADER_NAMES, AWS_STS_ENDPOINT, AWS_WEB_IDENTITY_TOKEN, AWS_WEB_IDENTITY_TOKEN_FILE,
|
||||
DEFAULT_BEDROCK_REGION, DEFAULT_SESSION_NAME_PREFIX, SIGV4_COMPUTED_HEADER_NAMES,
|
||||
};
|
||||
use crate::error::{CredentialFlow, Error, SigningStage};
|
||||
|
||||
const STATIC_CREDENTIALS_TTL: Duration = Duration::from_secs(3600 - 60);
|
||||
const AMBIENT_CREDENTIALS_TTL: Duration = Duration::from_secs(600);
|
||||
|
|
@ -271,7 +271,7 @@ impl AwsAuthService {
|
|||
provider
|
||||
.provide_credentials()
|
||||
.await
|
||||
.map_err(|error| Error::AwsProfile(error.to_string()))
|
||||
.map_err(Error::credentials(CredentialFlow::Profile))
|
||||
}
|
||||
AwsAuthFlow::AssumeRole { role, session_name } => {
|
||||
if is_already_running_as_role(&role, &resolved).await? {
|
||||
|
|
@ -287,7 +287,7 @@ impl AwsAuthService {
|
|||
let credentials = provider
|
||||
.provide_credentials()
|
||||
.await
|
||||
.map_err(|error| Error::AwsDefaultChain(error.to_string()))?;
|
||||
.map_err(Error::credentials(CredentialFlow::DefaultChain))?;
|
||||
self.set_cached_credentials(
|
||||
key,
|
||||
credentials.clone(),
|
||||
|
|
@ -327,7 +327,7 @@ impl AwsAuthService {
|
|||
provider
|
||||
.provide_credentials()
|
||||
.await
|
||||
.map_err(|error| Error::AwsAssumeRole(error.to_string()))
|
||||
.map_err(Error::credentials(CredentialFlow::AssumeRole))
|
||||
}
|
||||
AwsAuthFlow::WebIdentity {
|
||||
token,
|
||||
|
|
@ -350,12 +350,14 @@ impl AwsAuthService {
|
|||
.web_identity_token(token)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|error| Error::AwsWebIdentity(error.to_string()))?;
|
||||
.map_err(Error::credentials(CredentialFlow::WebIdentity))?;
|
||||
let credentials = response
|
||||
.credentials()
|
||||
.ok_or(Error::AwsMissingWebIdentityCredentials)?;
|
||||
let expiration = SystemTime::try_from(*credentials.expiration())
|
||||
.map_err(|error| Error::AwsWebIdentityExpiration(error.to_string()))?;
|
||||
.ok_or(Error::MissingWebIdentityCredentials)?;
|
||||
let expiration =
|
||||
SystemTime::try_from(*credentials.expiration()).map_err(|source| {
|
||||
Error::WebIdentityExpiration(litellm_error::Source::new(source))
|
||||
})?;
|
||||
Ok(Credentials::new(
|
||||
credentials.access_key_id(),
|
||||
credentials.secret_access_key(),
|
||||
|
|
@ -376,7 +378,7 @@ impl AwsAuthService {
|
|||
let credentials = provider
|
||||
.provide_credentials()
|
||||
.await
|
||||
.map_err(|error| Error::AwsDefaultChain(error.to_string()))?;
|
||||
.map_err(Error::credentials(CredentialFlow::DefaultChain))?;
|
||||
self.set_cached_credentials(
|
||||
key,
|
||||
credentials.clone(),
|
||||
|
|
@ -480,14 +482,16 @@ pub fn sign_headers(
|
|||
.settings(SigningSettings::default())
|
||||
.build()
|
||||
.map(SigningParams::from)
|
||||
.map_err(|error| Error::AwsSigningParameters(error.to_string()))?;
|
||||
.map_err(Error::signing(SigningStage::Parameters))?;
|
||||
let header_refs = headers
|
||||
.iter()
|
||||
.map(|(name, value)| {
|
||||
value
|
||||
.to_str()
|
||||
.map(|value| (name.as_str(), value))
|
||||
.map_err(|_| Error::AwsSignableRequest(format!("{name} is not visible ASCII")))
|
||||
.map_err(|_| Error::NonAsciiHeader {
|
||||
name: name.to_string(),
|
||||
})
|
||||
})
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
let request = SignableRequest::new(
|
||||
|
|
@ -496,23 +500,20 @@ pub fn sign_headers(
|
|||
header_refs.into_iter(),
|
||||
SignableBody::Bytes(body),
|
||||
)
|
||||
.map_err(|error| Error::AwsSignableRequest(error.to_string()))?;
|
||||
.map_err(Error::signing(SigningStage::Request))?;
|
||||
let (instructions, _) = sign(request, ¶ms)
|
||||
.map_err(|error| Error::AwsSigning(error.to_string()))?
|
||||
.map_err(Error::signing(SigningStage::Signature))?
|
||||
.into_parts();
|
||||
instructions
|
||||
.headers()
|
||||
.map(|(name, value)| {
|
||||
HeaderName::from_bytes(name.as_bytes())
|
||||
.map_err(|error| error.to_string())
|
||||
.and_then(|name| {
|
||||
HeaderValue::from_str(value)
|
||||
.map(|value| (name, value))
|
||||
.map_err(|error| error.to_string())
|
||||
})
|
||||
let name = HeaderName::from_bytes(name.as_bytes())
|
||||
.map_err(Error::signing(SigningStage::Output))?;
|
||||
let value =
|
||||
HeaderValue::from_str(value).map_err(Error::signing(SigningStage::Output))?;
|
||||
Ok((name, value))
|
||||
})
|
||||
.collect::<Result<HeaderMap, _>>()
|
||||
.map_err(Error::AwsSigning)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Model-id and region parsing shared by every Bedrock route.
|
||||
|
|
@ -1081,6 +1082,55 @@ mod tests {
|
|||
assert!(!signed.contains_key("X-Amz-Security-Token"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_header_value_outside_visible_ascii_is_named_instead_of_signed() {
|
||||
let (url, body, mut headers) = parity_inputs();
|
||||
headers.insert("x-amz-meta-note", HeaderValue::from_str("café").unwrap());
|
||||
let credentials = Credentials::new("AKIDEXAMPLE", "secret", None, None, "test");
|
||||
|
||||
let error = sign_headers(
|
||||
"POST",
|
||||
&url,
|
||||
&body,
|
||||
&headers,
|
||||
SigningScope {
|
||||
region: "us-east-1",
|
||||
service: BEDROCK_SERVICE,
|
||||
credentials: &credentials,
|
||||
time: UNIX_EPOCH,
|
||||
},
|
||||
)
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(
|
||||
error,
|
||||
Error::NonAsciiHeader {
|
||||
name: "x-amz-meta-note".into()
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_shared_auth_error_nests_the_aws_failure_and_its_cause() {
|
||||
let cause = std::io::Error::new(std::io::ErrorKind::NotFound, "profile not found");
|
||||
let error =
|
||||
litellm_auth_types::Error::from(Error::credentials(CredentialFlow::Profile)(cause));
|
||||
|
||||
assert!(matches!(
|
||||
litellm_error::find::<Error>(&error),
|
||||
Some(Error::Credentials {
|
||||
flow: CredentialFlow::Profile,
|
||||
..
|
||||
})
|
||||
));
|
||||
let report = litellm_error::Report::new(&error).to_string();
|
||||
assert!(
|
||||
report.contains("could not load AWS profile credentials"),
|
||||
"{report}"
|
||||
);
|
||||
assert_eq!(report.matches("profile not found").count(), 1, "{report}");
|
||||
}
|
||||
|
||||
#[ignore]
|
||||
#[tokio::test]
|
||||
async fn live_bedrock_invoke_model_returns_200() -> Result<(), Box<dyn std::error::Error>> {
|
||||
|
|
|
|||
|
|
@ -1,46 +1,73 @@
|
|||
use thiserror::Error as ThisError;
|
||||
use litellm_error::Source;
|
||||
|
||||
#[derive(Clone, Debug, ThisError, PartialEq, Eq)]
|
||||
#[derive(Clone, Debug, thiserror::Error, PartialEq, Eq)]
|
||||
pub enum Error {
|
||||
#[error("AWS profile credentials failed: {0}")]
|
||||
AwsProfile(String),
|
||||
#[error("AWS default credentials failed: {0}")]
|
||||
AwsDefaultChain(String),
|
||||
#[error("AWS role credentials failed: {0}")]
|
||||
AwsAssumeRole(String),
|
||||
#[error("AWS web identity credentials failed: {0}")]
|
||||
AwsWebIdentity(String),
|
||||
#[error("AWS web identity expiration was invalid: {0}")]
|
||||
AwsWebIdentityExpiration(String),
|
||||
#[error("AWS signing parameters failed: {0}")]
|
||||
AwsSigningParameters(String),
|
||||
#[error("AWS signable request failed: {0}")]
|
||||
AwsSignableRequest(String),
|
||||
#[error("AWS request signing failed: {0}")]
|
||||
AwsSigning(String),
|
||||
#[error("could not load AWS {flow} credentials")]
|
||||
Credentials {
|
||||
flow: CredentialFlow,
|
||||
#[source]
|
||||
source: Source,
|
||||
},
|
||||
#[error("AWS web identity response had no credentials")]
|
||||
AwsMissingWebIdentityCredentials,
|
||||
MissingWebIdentityCredentials,
|
||||
#[error("AWS web identity response had an invalid expiration")]
|
||||
WebIdentityExpiration(#[source] Source),
|
||||
#[error("header {name} has a value that is not visible ASCII")]
|
||||
NonAsciiHeader { name: String },
|
||||
#[error("{stage}")]
|
||||
Signing {
|
||||
stage: SigningStage,
|
||||
#[source]
|
||||
source: Source,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)]
|
||||
pub enum CredentialFlow {
|
||||
#[error("profile")]
|
||||
Profile,
|
||||
#[error("default chain")]
|
||||
DefaultChain,
|
||||
#[error("assumed role")]
|
||||
AssumeRole,
|
||||
#[error("web identity")]
|
||||
WebIdentity,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)]
|
||||
pub enum SigningStage {
|
||||
#[error("SigV4 signing parameters are invalid")]
|
||||
Parameters,
|
||||
#[error("SigV4 cannot sign this request")]
|
||||
Request,
|
||||
#[error("SigV4 could not compute the signature")]
|
||||
Signature,
|
||||
#[error("SigV4 produced an invalid header")]
|
||||
Output,
|
||||
}
|
||||
|
||||
impl Error {
|
||||
pub(crate) fn credentials<E: std::error::Error + Send + Sync + 'static>(
|
||||
flow: CredentialFlow,
|
||||
) -> impl FnOnce(E) -> Self {
|
||||
move |source| Self::Credentials {
|
||||
flow,
|
||||
source: Source::new(source),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn signing<E: std::error::Error + Send + Sync + 'static>(
|
||||
stage: SigningStage,
|
||||
) -> impl FnOnce(E) -> Self {
|
||||
move |source| Self::Signing {
|
||||
stage,
|
||||
source: Source::new(source),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<Error> for litellm_auth_types::Error {
|
||||
fn from(error: Error) -> Self {
|
||||
Self::ProviderAuthentication(error.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::Error;
|
||||
|
||||
#[test]
|
||||
fn converts_to_shared_auth_error_without_losing_context() {
|
||||
let error = litellm_auth_types::Error::from(Error::AwsProfile("profile not found".into()));
|
||||
|
||||
assert_eq!(
|
||||
error,
|
||||
litellm_auth_types::Error::ProviderAuthentication(
|
||||
"AWS profile credentials failed: profile not found".into()
|
||||
)
|
||||
);
|
||||
Self::provider(error)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,5 +5,5 @@ mod signer;
|
|||
|
||||
pub use aws::*;
|
||||
pub use aws_credential_types::Credentials;
|
||||
pub use error::Error;
|
||||
pub use error::{CredentialFlow, Error, SigningStage};
|
||||
pub use signer::SigV4Signer;
|
||||
|
|
|
|||
|
|
@ -53,13 +53,13 @@ impl RequestSigner for SigV4Signer {
|
|||
.keys()
|
||||
.find(|name| is_sigv4_computed_header(name.as_str()))
|
||||
{
|
||||
return Err(litellm_http::Error::ComputedHeader(name.to_string()));
|
||||
return Err(litellm_http::Error::computed_header(name.as_str()));
|
||||
}
|
||||
let body = match request.body() {
|
||||
None => &[],
|
||||
Some(body) => body.as_bytes().ok_or_else(|| {
|
||||
litellm_http::Error::Signature("a streaming body cannot be signed".into())
|
||||
})?,
|
||||
Some(body) => body
|
||||
.as_bytes()
|
||||
.ok_or_else(litellm_http::Error::unsignable_body)?,
|
||||
};
|
||||
sign_headers(
|
||||
request.method().as_str(),
|
||||
|
|
@ -73,7 +73,13 @@ impl RequestSigner for SigV4Signer {
|
|||
time: (self.clock)(),
|
||||
},
|
||||
)
|
||||
.map_err(|error| litellm_http::Error::Signature(error.to_string()))
|
||||
.map_err(|error| match error {
|
||||
Error::NonAsciiHeader { .. } => litellm_http::Error::unsignable_header(error),
|
||||
Error::Credentials { .. }
|
||||
| Error::MissingWebIdentityCredentials
|
||||
| Error::WebIdentityExpiration(_)
|
||||
| Error::Signing { .. } => litellm_http::Error::signature(error),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -183,10 +189,28 @@ mod tests {
|
|||
)
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(
|
||||
error,
|
||||
litellm_http::Error::ComputedHeader("authorization".into())
|
||||
);
|
||||
assert!(error.is_computed_header());
|
||||
assert!(error.to_string().contains("authorization"), "{error}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_forwarded_header_that_cannot_be_signed_is_the_callers_fault() {
|
||||
let error = signed(
|
||||
json(
|
||||
URL,
|
||||
&[("x-amz-meta-note".into(), "café".into())],
|
||||
&json_value!({}),
|
||||
None,
|
||||
)
|
||||
.unwrap(),
|
||||
&signer("textract"),
|
||||
)
|
||||
.unwrap_err();
|
||||
|
||||
let report = litellm_error::Report::new(&error).to_string();
|
||||
assert!(report.contains("x-amz-meta-note"), "{report}");
|
||||
assert!(error.is_request());
|
||||
assert!(!error.dispatched());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -198,7 +222,7 @@ mod tests {
|
|||
|
||||
assert!(matches!(
|
||||
signed(request, &signer("textract")),
|
||||
Err(litellm_http::Error::Signature(_))
|
||||
Err(error) if !error.dispatched() && !error.is_request()
|
||||
));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -18,5 +18,6 @@ azure_core = "1.0.0"
|
|||
azure_identity = { version = "1.0.0", features = ["tokio"] }
|
||||
|
||||
[dev-dependencies]
|
||||
litellm-error.workspace = true
|
||||
rstest.workspace = true
|
||||
tokio.workspace = true
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ use azure_identity::{
|
|||
};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use litellm_auth_types::Error;
|
||||
use litellm_auth_types::{Error, ErrorDetail};
|
||||
use litellm_auth_types::{InputSource, ResolvedCredential, SecretValue, Sourced};
|
||||
|
||||
use super::credential_provider_cache::{
|
||||
|
|
@ -133,7 +133,9 @@ impl NativeAzureTokenAcquirer {
|
|||
let token = credential
|
||||
.get_token(&[scope.as_str()], None)
|
||||
.await
|
||||
.map_err(|error| Error::CredentialAcquisition(error.to_string().into()))?;
|
||||
.map_err(|error| {
|
||||
Error::CredentialAcquisition(ErrorDetail::failed("Azure token request", error))
|
||||
})?;
|
||||
let expires_on = u64::try_from(token.expires_on.unix_timestamp())
|
||||
.ok()
|
||||
.map(|seconds| UNIX_EPOCH + Duration::from_secs(seconds));
|
||||
|
|
@ -250,11 +252,8 @@ fn validate_authority(request: &NativeAzureRequest) -> Result<(), Error> {
|
|||
let Some(authority) = authority else {
|
||||
return Ok(());
|
||||
};
|
||||
let url = url::Url::parse(authority.value()).map_err(|_| {
|
||||
Error::InvalidConfiguration(
|
||||
"Azure authority must be an HTTPS origin without credentials, query, or fragment"
|
||||
.into(),
|
||||
)
|
||||
let url = url::Url::parse(authority.value()).map_err(|error| {
|
||||
Error::InvalidConfiguration(ErrorDetail::failed("Azure authority parsing", error))
|
||||
})?;
|
||||
if url.scheme() != "https"
|
||||
|| url.host_str().is_none()
|
||||
|
|
@ -264,10 +263,10 @@ fn validate_authority(request: &NativeAzureRequest) -> Result<(), Error> {
|
|||
|| url.fragment().is_some()
|
||||
|| !matches!(url.path(), "" | "/")
|
||||
{
|
||||
return Err(Error::InvalidConfiguration(
|
||||
"Azure authority must be an HTTPS origin without credentials, query, or fragment"
|
||||
.into(),
|
||||
));
|
||||
return Err(Error::InvalidConfiguration(ErrorDetail::InvalidType {
|
||||
field: "Azure authority".into(),
|
||||
expected: "an HTTPS origin without credentials, query, or fragment",
|
||||
}));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -377,7 +376,7 @@ fn trusted_source(sources: &[InputSource]) -> InputSource {
|
|||
|
||||
fn mixed_sources<T>() -> Result<T, Error> {
|
||||
Err(Error::InvalidConfiguration(
|
||||
"request-controlled Azure auth inputs cannot be combined with host credentials".into(),
|
||||
ErrorDetail::MixedInputSources { provider: "Azure" },
|
||||
))
|
||||
}
|
||||
|
||||
|
|
@ -444,7 +443,7 @@ fn build_credential(
|
|||
.map(|credential| credential as Arc<dyn TokenCredential>),
|
||||
}
|
||||
.map_err(|error| {
|
||||
Error::InvalidConfiguration(litellm_auth_types::ErrorDetail::failed(
|
||||
Error::InvalidConfiguration(ErrorDetail::failed(
|
||||
"Azure credential initialization",
|
||||
error,
|
||||
))
|
||||
|
|
@ -662,12 +661,11 @@ mod tests {
|
|||
))
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(
|
||||
error,
|
||||
litellm_auth_types::Error::InvalidConfiguration(
|
||||
assert!(
|
||||
litellm_error::Report::new(&error).to_string().contains(
|
||||
"request-controlled Azure auth inputs cannot be combined with host credentials"
|
||||
.into()
|
||||
)
|
||||
),
|
||||
"{error:?}"
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -695,12 +693,11 @@ mod tests {
|
|||
authority,
|
||||
))
|
||||
.unwrap_err();
|
||||
assert_eq!(
|
||||
error,
|
||||
litellm_auth_types::Error::InvalidConfiguration(
|
||||
assert!(
|
||||
litellm_error::Report::new(&error).to_string().contains(
|
||||
"Azure authority must be an HTTPS origin without credentials, query, or fragment"
|
||||
.into()
|
||||
)
|
||||
),
|
||||
"{error:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
use litellm_auth_types::Error;
|
||||
use litellm_auth_types::{
|
||||
CredentialFileRef, CredentialLookup, CredentialRef, InputSource, ResolvedCredential,
|
||||
SecretValue, Sourced, TokenProviderHandle,
|
||||
};
|
||||
use litellm_auth_types::{Error, ErrorDetail};
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
|
|
@ -91,9 +91,9 @@ impl AzureAuthService {
|
|||
AzureCredentialPlan::Caller(caller) => {
|
||||
let credential = caller.acquire().await?;
|
||||
if credential.secret().expose().is_empty() {
|
||||
return Err(Error::EmptyCallerCredential(
|
||||
"Azure AD token provider returned an empty token",
|
||||
));
|
||||
return Err(Error::EmptyCallerCredential {
|
||||
caller: "Azure AD token provider",
|
||||
});
|
||||
}
|
||||
Ok(Some(Sourced::new(credential, InputSource::Deployment)))
|
||||
}
|
||||
|
|
@ -106,11 +106,9 @@ impl AzureAuthService {
|
|||
} => {
|
||||
let assertion = resolve_reference(inputs, env_lookup, reference.value())
|
||||
.await?
|
||||
.ok_or_else(|| {
|
||||
Error::CredentialAcquisition(
|
||||
"Azure OIDC reference did not resolve to a value".into(),
|
||||
)
|
||||
})?;
|
||||
.ok_or(Error::CredentialAcquisition(ErrorDetail::Unresolved {
|
||||
subject: "Azure OIDC reference",
|
||||
}))?;
|
||||
let request = ValidatedAzureRequest::new(NativeAzureRequest::ClientAssertion {
|
||||
tenant_id,
|
||||
client_id,
|
||||
|
|
@ -142,7 +140,13 @@ impl AzureAuthService {
|
|||
Err(error) => failures.push(error),
|
||||
}
|
||||
}
|
||||
Err(Error::CredentialChain(failures))
|
||||
match failures.pop() {
|
||||
Some(last) => Err(Error::CredentialChain {
|
||||
last: Box::new(last),
|
||||
earlier: failures,
|
||||
}),
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
AzureCredentialPlan::Missing => Ok(None),
|
||||
}
|
||||
|
|
@ -173,7 +177,12 @@ pub(crate) fn select_auth_plan(
|
|||
.map(|selector| Sourced::new(selector, value.source()))
|
||||
})
|
||||
.transpose()
|
||||
.map_err(|_| Error::InvalidConfiguration("invalid Azure credential selector".into()))?;
|
||||
.map_err(|error| {
|
||||
Error::InvalidConfiguration(ErrorDetail::failed(
|
||||
"Azure credential selector parsing",
|
||||
error,
|
||||
))
|
||||
})?;
|
||||
let federated_token_file = configured_string(
|
||||
&inputs.federated_token_file,
|
||||
AZURE_FEDERATED_TOKEN_FILE_ENV,
|
||||
|
|
@ -263,9 +272,12 @@ fn select_native_plan(
|
|||
let selection_source = selected.source();
|
||||
|
||||
match selected.into_value() {
|
||||
AzureCredentialType::ClientSecretCredential => Err(Error::InvalidConfiguration(
|
||||
"ClientSecretCredential requires tenant_id, client_id, and client_secret".into(),
|
||||
)),
|
||||
AzureCredentialType::ClientSecretCredential => {
|
||||
Err(Error::InvalidConfiguration(ErrorDetail::MissingSettings {
|
||||
credential: "ClientSecretCredential",
|
||||
settings: "tenant_id, client_id, and client_secret",
|
||||
}))
|
||||
}
|
||||
AzureCredentialType::WorkloadIdentityCredential => {
|
||||
Ok(AzureCredentialPlan::Native(ValidatedAzureRequest::new(
|
||||
workload_request(tenant_id, client_id, federated_token_file, scope, authority)?,
|
||||
|
|
@ -349,22 +361,21 @@ fn workload_request(
|
|||
authority: Option<Sourced<String>>,
|
||||
) -> Result<NativeAzureRequest, Error> {
|
||||
Ok(NativeAzureRequest::WorkloadIdentity {
|
||||
tenant_id: tenant_id.ok_or_else(|| {
|
||||
Error::InvalidConfiguration("WorkloadIdentityCredential requires tenant_id".into())
|
||||
})?,
|
||||
client_id: client_id.ok_or_else(|| {
|
||||
Error::InvalidConfiguration("WorkloadIdentityCredential requires client_id".into())
|
||||
})?,
|
||||
token_file_path: token_file_path.ok_or_else(|| {
|
||||
Error::InvalidConfiguration(
|
||||
"WorkloadIdentityCredential requires azure_federated_token_file".into(),
|
||||
)
|
||||
})?,
|
||||
tenant_id: tenant_id.ok_or(workload_requires("tenant_id"))?,
|
||||
client_id: client_id.ok_or(workload_requires("client_id"))?,
|
||||
token_file_path: token_file_path.ok_or(workload_requires("azure_federated_token_file"))?,
|
||||
scope,
|
||||
authority,
|
||||
})
|
||||
}
|
||||
|
||||
fn workload_requires(settings: &'static str) -> Error {
|
||||
Error::InvalidConfiguration(ErrorDetail::MissingSettings {
|
||||
credential: "WorkloadIdentityCredential",
|
||||
settings,
|
||||
})
|
||||
}
|
||||
|
||||
fn configured_string(
|
||||
configured: &ConfigValue<String>,
|
||||
environment_name: &str,
|
||||
|
|
@ -410,11 +421,10 @@ async fn resolve_reference(
|
|||
.map_or(CredentialLookup::Missing, CredentialLookup::Found),
|
||||
CredentialRef::None => return Ok(None),
|
||||
CredentialRef::File(_) | CredentialRef::Request(_) | CredentialRef::Host(_) => {
|
||||
let resolver = inputs.credential_resolver.as_ref().ok_or_else(|| {
|
||||
Error::InvalidConfiguration(
|
||||
"credential reference requires a host credential resolver".into(),
|
||||
)
|
||||
})?;
|
||||
let resolver = inputs
|
||||
.credential_resolver
|
||||
.as_ref()
|
||||
.ok_or(Error::InvalidConfiguration(ErrorDetail::MissingResolver))?;
|
||||
resolver.resolve(reference).await?
|
||||
}
|
||||
};
|
||||
|
|
@ -433,7 +443,7 @@ fn oidc_reference(
|
|||
let value = token.value().expose();
|
||||
if token.source() == InputSource::Request && value.starts_with("oidc/") {
|
||||
return Err(Error::InvalidConfiguration(
|
||||
"request-controlled Azure credential references are not allowed".into(),
|
||||
ErrorDetail::RequestControlledReference { provider: "Azure" },
|
||||
));
|
||||
}
|
||||
if let Some(name) = value.strip_prefix("oidc/env/") {
|
||||
|
|
@ -457,19 +467,17 @@ fn oidc_reference(
|
|||
}
|
||||
if value.starts_with("oidc/") {
|
||||
return Err(Error::InvalidConfiguration(
|
||||
"unsupported OIDC reference".into(),
|
||||
ErrorDetail::UnsupportedReference,
|
||||
));
|
||||
}
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
fn non_empty_reference(value: &str, kind: &str) -> Result<String, Error> {
|
||||
fn non_empty_reference(value: &str, kind: &'static str) -> Result<String, Error> {
|
||||
if value.is_empty() {
|
||||
return Err(Error::InvalidConfiguration(
|
||||
litellm_auth_types::ErrorDetail::Empty {
|
||||
subject: kind.into(),
|
||||
},
|
||||
));
|
||||
return Err(Error::InvalidConfiguration(ErrorDetail::Empty {
|
||||
subject: kind,
|
||||
}));
|
||||
}
|
||||
Ok(value.to_string())
|
||||
}
|
||||
|
|
@ -518,7 +526,9 @@ mod tests {
|
|||
expires_on: None,
|
||||
})
|
||||
} else {
|
||||
Err(Error::CredentialAcquisition(kind.into()))
|
||||
Err(Error::CredentialAcquisition(
|
||||
litellm_auth_types::ErrorDetail::Unresolved { subject: kind },
|
||||
))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
@ -624,7 +634,11 @@ mod tests {
|
|||
)))
|
||||
.expect_err("unsupported backend must fail validation");
|
||||
|
||||
assert!(error.to_string().contains("unsupported OIDC reference"));
|
||||
assert!(
|
||||
litellm_error::Report::new(&error)
|
||||
.to_string()
|
||||
.contains("unsupported OIDC reference")
|
||||
);
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
|
|
@ -649,11 +663,11 @@ mod tests {
|
|||
})
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(
|
||||
error,
|
||||
Error::InvalidConfiguration(
|
||||
"request-controlled Azure credential references are not allowed".into()
|
||||
)
|
||||
assert!(
|
||||
litellm_error::Report::new(&error)
|
||||
.to_string()
|
||||
.contains("request-controlled Azure credential references are not allowed"),
|
||||
"{error:?}"
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -715,7 +729,16 @@ mod tests {
|
|||
.await
|
||||
.unwrap_err();
|
||||
|
||||
assert!(matches!(error, Error::CredentialChain(errors) if errors.len() == 2));
|
||||
assert!(matches!(&error, Error::CredentialChain { earlier, .. } if earlier.len() == 1));
|
||||
let report = litellm_error::Report::new(&error).to_string();
|
||||
assert!(
|
||||
report.contains("managed-identity did not resolve"),
|
||||
"{report}"
|
||||
);
|
||||
assert!(
|
||||
report.contains("developer-tools did not resolve"),
|
||||
"{report}"
|
||||
);
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
|
|
@ -763,7 +786,9 @@ mod tests {
|
|||
|
||||
assert_eq!(
|
||||
error,
|
||||
Error::EmptyCallerCredential("Azure AD token provider returned an empty token")
|
||||
Error::EmptyCallerCredential {
|
||||
caller: "Azure AD token provider"
|
||||
}
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -24,5 +24,6 @@ google-cloud-auth = { workspace = true, optional = true }
|
|||
http = { workspace = true, optional = true }
|
||||
|
||||
[dev-dependencies]
|
||||
litellm-error.workspace = true
|
||||
rstest.workspace = true
|
||||
tempfile.workspace = true
|
||||
|
|
|
|||
|
|
@ -73,13 +73,17 @@ impl VertexAuth {
|
|||
.or_else(|| non_empty_env(env_lookup, VERTEXAI_API_KEY_ENV));
|
||||
let project_id = get_vertex_ai_project(config, env_lookup);
|
||||
|
||||
if !has_authorization && static_token.is_none() {
|
||||
let access = self.get_access_token(config, env_lookup).await?;
|
||||
return Ok(VertexEnvironment {
|
||||
headers: apply_credential(headers, &access.token, CredentialPlacement::Bearer)?,
|
||||
project_id: project_id.unwrap_or(access.project_id),
|
||||
});
|
||||
}
|
||||
let static_token = match (has_authorization, static_token) {
|
||||
(true, _) => None,
|
||||
(false, Some(token)) => Some(token),
|
||||
(false, None) => {
|
||||
let access = self.get_access_token(config, env_lookup).await?;
|
||||
return Ok(VertexEnvironment {
|
||||
headers: apply_credential(headers, &access.token, CredentialPlacement::Bearer)?,
|
||||
project_id: project_id.unwrap_or(access.project_id),
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
let project_id = match project_id {
|
||||
Some(project_id) => project_id,
|
||||
|
|
@ -90,14 +94,9 @@ impl VertexAuth {
|
|||
.await?
|
||||
}
|
||||
};
|
||||
let headers = if has_authorization {
|
||||
headers
|
||||
} else {
|
||||
apply_credential(
|
||||
headers,
|
||||
static_token.as_deref().expect("static token was checked"),
|
||||
CredentialPlacement::Bearer,
|
||||
)?
|
||||
let headers = match static_token {
|
||||
Some(token) => apply_credential(headers, &token, CredentialPlacement::Bearer)?,
|
||||
None => headers,
|
||||
};
|
||||
Ok(VertexEnvironment {
|
||||
headers,
|
||||
|
|
@ -234,14 +233,13 @@ enum CredentialCacheKey {
|
|||
Adc,
|
||||
}
|
||||
|
||||
const BASE64_CREDENTIALS: &str = "The `vertex_credentials` value is base64-encoded JSON, not the JSON itself. \
|
||||
Set it to a readable file path, or to the credentials JSON itself.";
|
||||
|
||||
fn load_trusted_service_account(configured: &str) -> Result<CustomServiceAccount, Error> {
|
||||
if is_base64_encoded_json(configured) {
|
||||
return Err(Error::InvalidConfiguration(ErrorDetail::Message(
|
||||
BASE64_CREDENTIALS.into(),
|
||||
)));
|
||||
return Err(Error::InvalidConfiguration(
|
||||
ErrorDetail::Base64EncodedJson {
|
||||
field: "vertex_credentials",
|
||||
},
|
||||
));
|
||||
}
|
||||
if names_a_credentials_file(configured) {
|
||||
CustomServiceAccount::from_file(configured)
|
||||
|
|
@ -299,7 +297,8 @@ mod tests {
|
|||
else {
|
||||
panic!("{configured:?} should not load");
|
||||
};
|
||||
assert!(error.to_string().contains(expected), "{error}");
|
||||
let report = litellm_error::Report::new(&error).to_string();
|
||||
assert!(report.contains(expected), "{report}");
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
|
|
@ -321,7 +320,7 @@ mod tests {
|
|||
.await
|
||||
{
|
||||
Ok(_) => panic!("{configured:?} should not load"),
|
||||
Err(error) => error.to_string(),
|
||||
Err(error) => litellm_error::Report::new(&error).to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -55,7 +55,7 @@ impl CredentialsProvider for GoogleCredentials {
|
|||
entity_tag: EntityTag::new(),
|
||||
data,
|
||||
})
|
||||
.map_err(|_| CredentialsError::from_msg(false, "Google authentication failed"))
|
||||
.map_err(|error| CredentialsError::from_source(false, error))
|
||||
}
|
||||
|
||||
async fn universe_domain(&self) -> Option<String> {
|
||||
|
|
@ -102,5 +102,9 @@ mod tests {
|
|||
);
|
||||
let error = credentials.headers(Extensions::new()).await.unwrap_err();
|
||||
assert!(!format!("{error:?}").contains("private"));
|
||||
assert_eq!(
|
||||
std::error::Error::source(&error).and_then(|cause| cause.downcast_ref::<Error>()),
|
||||
Some(&Error::InvalidHeader)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ license.workspace = true
|
|||
repository.workspace = true
|
||||
|
||||
[dependencies]
|
||||
litellm-error.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
subtle.workspace = true
|
||||
|
|
|
|||
|
|
@ -86,9 +86,9 @@ impl CredentialPlan {
|
|||
Self::Caller(caller) => {
|
||||
let credential = caller.acquire().await?;
|
||||
if credential.secret().expose().is_empty() {
|
||||
return Err(Error::EmptyCallerCredential(
|
||||
"credential caller returned an empty credential",
|
||||
));
|
||||
return Err(Error::EmptyCallerCredential {
|
||||
caller: "credential caller",
|
||||
});
|
||||
}
|
||||
Ok(CredentialPlanResolution::Resolved(credential))
|
||||
}
|
||||
|
|
@ -104,8 +104,8 @@ mod tests {
|
|||
CredentialLookup, CredentialLookupFuture, CredentialPlan, CredentialPlanResolution,
|
||||
CredentialRef, CredentialResolver, CredentialResolverHandle,
|
||||
};
|
||||
use crate::Error;
|
||||
use crate::SecretValue;
|
||||
use crate::{Error, ErrorDetail};
|
||||
|
||||
#[derive(Debug)]
|
||||
struct HostResolver;
|
||||
|
|
@ -150,9 +150,10 @@ mod tests {
|
|||
impl CredentialResolver for FailingResolver {
|
||||
fn resolve<'a>(&'a self, _reference: &'a CredentialRef) -> CredentialLookupFuture<'a> {
|
||||
Box::pin(async {
|
||||
Err(Error::CredentialAcquisition(
|
||||
"host credential lookup failed".into(),
|
||||
))
|
||||
Err(Error::CredentialAcquisition(ErrorDetail::failed(
|
||||
"host credential lookup",
|
||||
std::io::Error::from(std::io::ErrorKind::PermissionDenied),
|
||||
)))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -168,9 +169,14 @@ mod tests {
|
|||
.await
|
||||
.expect_err("acquisition errors cannot become fallback");
|
||||
|
||||
assert_eq!(
|
||||
error,
|
||||
Error::CredentialAcquisition("host credential lookup failed".into())
|
||||
assert!(
|
||||
matches!(error, Error::CredentialAcquisition(_)),
|
||||
"{error:?}"
|
||||
);
|
||||
assert!(
|
||||
litellm_error::Report::new(&error)
|
||||
.to_string()
|
||||
.contains("host credential lookup failed")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,17 +1,23 @@
|
|||
use thiserror::Error as ThisError;
|
||||
use litellm_error::{Report, Source};
|
||||
|
||||
#[derive(Clone, Debug, ThisError, PartialEq, Eq)]
|
||||
use crate::{ParamKind, ParamSpec};
|
||||
|
||||
#[derive(Clone, Debug, thiserror::Error, PartialEq, Eq)]
|
||||
pub enum Error {
|
||||
#[error("invalid authentication configuration: {0}")]
|
||||
#[error("invalid authentication configuration")]
|
||||
InvalidConfiguration(#[source] ErrorDetail),
|
||||
#[error("credential acquisition failed: {0}")]
|
||||
#[error("credential acquisition failed")]
|
||||
CredentialAcquisition(#[source] ErrorDetail),
|
||||
#[error("credential caller failed: {0}")]
|
||||
EmptyCallerCredential(&'static str),
|
||||
#[error("{0}")]
|
||||
ProviderAuthentication(String),
|
||||
#[error("credential acquisition failed: {}", .0.iter().map(ToString::to_string).collect::<Vec<_>>().join("; "))]
|
||||
CredentialChain(Vec<Error>),
|
||||
#[error("{caller} returned an empty credential")]
|
||||
EmptyCallerCredential { caller: &'static str },
|
||||
#[error("provider authentication failed")]
|
||||
Provider(#[source] Source),
|
||||
#[error("no credential in the chain succeeded{}", earlier_attempts(earlier))]
|
||||
CredentialChain {
|
||||
#[source]
|
||||
last: Box<Error>,
|
||||
earlier: Vec<Error>,
|
||||
},
|
||||
#[error(
|
||||
"Missing {provider} API Key - Set `api_key` or the {environment_variable} environment variable"
|
||||
)]
|
||||
|
|
@ -22,7 +28,7 @@ pub enum Error {
|
|||
#[error("Missing {provider} {} - {}", .spec.setting, .spec.guidance())]
|
||||
MissingParam {
|
||||
provider: &'static str,
|
||||
spec: &'static crate::ParamSpec,
|
||||
spec: &'static ParamSpec,
|
||||
},
|
||||
#[error("Missing {provider} API Base - {guidance}")]
|
||||
MissingApiBase {
|
||||
|
|
@ -33,78 +39,282 @@ pub enum Error {
|
|||
InvalidHeader,
|
||||
}
|
||||
|
||||
fn earlier_attempts(earlier: &[Error]) -> String {
|
||||
if earlier.is_empty() {
|
||||
return String::new();
|
||||
}
|
||||
let attempts = earlier
|
||||
.iter()
|
||||
.map(|failure| Report::new(failure).to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.join("; ");
|
||||
format!(" (earlier attempts: {attempts})")
|
||||
}
|
||||
|
||||
impl Error {
|
||||
pub fn provider(cause: impl std::error::Error + Send + Sync + 'static) -> Self {
|
||||
Self::Provider(Source::new(cause))
|
||||
}
|
||||
|
||||
pub fn is_missing_credential(&self) -> bool {
|
||||
match self {
|
||||
Self::MissingApiKey { .. } => true,
|
||||
Self::MissingParam { spec, .. } => match spec.kind {
|
||||
ParamKind::Credential => true,
|
||||
ParamKind::Setting => false,
|
||||
},
|
||||
Self::InvalidConfiguration(_)
|
||||
| Self::CredentialAcquisition(_)
|
||||
| Self::EmptyCallerCredential { .. }
|
||||
| Self::Provider(_)
|
||||
| Self::CredentialChain { .. }
|
||||
| Self::MissingApiBase { .. }
|
||||
| Self::InvalidHeader => false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_request(&self) -> bool {
|
||||
match self {
|
||||
Self::InvalidConfiguration(_)
|
||||
| Self::EmptyCallerCredential { .. }
|
||||
| Self::InvalidHeader => true,
|
||||
Self::CredentialAcquisition(_) | Self::Provider(_) | Self::CredentialChain { .. } => {
|
||||
false
|
||||
}
|
||||
Self::MissingParam { spec, .. } => match spec.kind {
|
||||
ParamKind::Setting => true,
|
||||
ParamKind::Credential => false,
|
||||
},
|
||||
Self::MissingApiKey { .. } | Self::MissingApiBase { .. } => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
|
||||
pub enum ErrorDetail {
|
||||
#[error("{0}")]
|
||||
Message(String),
|
||||
#[error("{field} must be {expected}")]
|
||||
InvalidType {
|
||||
field: String,
|
||||
expected: &'static str,
|
||||
},
|
||||
#[error("{subject} cannot be empty")]
|
||||
Empty { subject: String },
|
||||
Empty { subject: &'static str },
|
||||
#[error("credential header {0} already exists")]
|
||||
DuplicateHeader(&'static str),
|
||||
#[error("{operation} failed: {source}")]
|
||||
#[error("credential plan is not allowed by the provider auth policy")]
|
||||
PlanNotAllowed,
|
||||
#[error("{credential} requires {settings}")]
|
||||
MissingSettings {
|
||||
credential: &'static str,
|
||||
settings: &'static str,
|
||||
},
|
||||
#[error("credentials cannot be sent to a request-controlled {provider} endpoint")]
|
||||
RequestControlledEndpoint { provider: &'static str },
|
||||
#[error("request-controlled {provider} credential references are not allowed")]
|
||||
RequestControlledReference { provider: &'static str },
|
||||
#[error("request-controlled {provider} auth inputs cannot be combined with host credentials")]
|
||||
MixedInputSources { provider: &'static str },
|
||||
#[error("unsupported OIDC reference")]
|
||||
UnsupportedReference,
|
||||
#[error("credential reference requires a host credential resolver")]
|
||||
MissingResolver,
|
||||
#[error("{subject} did not resolve to a value")]
|
||||
Unresolved { subject: &'static str },
|
||||
#[error(
|
||||
"The `{field}` value is base64-encoded JSON, not the JSON itself. Set it to a readable file path, or to the credentials JSON itself."
|
||||
)]
|
||||
Base64EncodedJson { field: &'static str },
|
||||
#[error("{operation} failed")]
|
||||
Failed {
|
||||
operation: &'static str,
|
||||
#[source]
|
||||
source: ErrorSource,
|
||||
cause: Source,
|
||||
},
|
||||
}
|
||||
|
||||
impl ErrorDetail {
|
||||
pub fn failed(
|
||||
operation: &'static str,
|
||||
source: impl std::error::Error + Send + Sync + 'static,
|
||||
cause: impl std::error::Error + Send + Sync + 'static,
|
||||
) -> Self {
|
||||
Self::Failed {
|
||||
operation,
|
||||
source: ErrorSource::new(source),
|
||||
cause: Source::new(cause),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<String> for ErrorDetail {
|
||||
fn from(message: String) -> Self {
|
||||
Self::Message(message)
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::io;
|
||||
|
||||
use litellm_error::{Report, Source};
|
||||
use rstest::rstest;
|
||||
|
||||
use super::{Error, ErrorDetail};
|
||||
use crate::{AwsParams, VertexParams};
|
||||
|
||||
#[rstest]
|
||||
#[case::api_key(
|
||||
Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" },
|
||||
"Missing Example API Key - Set `api_key` or the EXAMPLE_API_KEY environment variable"
|
||||
)]
|
||||
#[case::another_api_key(
|
||||
Error::MissingApiKey { provider: "Custom", environment_variable: "CUSTOM_KEY" },
|
||||
"Missing Custom API Key - Set `api_key` or the CUSTOM_KEY environment variable"
|
||||
)]
|
||||
#[case::api_base(
|
||||
Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" },
|
||||
"Missing Example API Base - Pass api_base"
|
||||
)]
|
||||
#[case::another_api_base(
|
||||
Error::MissingApiBase { provider: "Custom", guidance: "Set CUSTOM_ENDPOINT" },
|
||||
"Missing Custom API Base - Set CUSTOM_ENDPOINT"
|
||||
)]
|
||||
fn missing_inputs_keep_the_python_text(#[case] error: Error, #[case] expected: &str) {
|
||||
assert_eq!(error.to_string(), expected);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::configuration(
|
||||
Error::InvalidConfiguration(ErrorDetail::Empty { subject: "credential selector" }),
|
||||
"invalid authentication configuration: credential selector cannot be empty"
|
||||
)]
|
||||
#[case::acquisition(
|
||||
Error::CredentialAcquisition(ErrorDetail::Unresolved { subject: "OIDC reference" }),
|
||||
"credential acquisition failed: OIDC reference did not resolve to a value"
|
||||
)]
|
||||
#[case::empty_caller(
|
||||
Error::EmptyCallerCredential { caller: "token provider" },
|
||||
"token provider returned an empty credential"
|
||||
)]
|
||||
fn the_report_names_every_layer_in_order(#[case] error: Error, #[case] expected: &str) {
|
||||
assert!(
|
||||
Report::new(&error).to_string().contains(expected),
|
||||
"{}",
|
||||
Report::new(&error)
|
||||
);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::configuration(Error::InvalidConfiguration(ErrorDetail::failed(
|
||||
"test credential",
|
||||
denied()
|
||||
)))]
|
||||
#[case::acquisition(Error::CredentialAcquisition(ErrorDetail::failed(
|
||||
"test credential",
|
||||
denied()
|
||||
)))]
|
||||
fn a_failure_keeps_its_cause_as_the_source_and_renders_it_once(#[case] error: Error) {
|
||||
let cause =
|
||||
std::iter::successors(std::error::Error::source(&error), |cause| cause.source())
|
||||
.find_map(|cause| cause.downcast_ref::<Source>()?.downcast_ref::<io::Error>())
|
||||
.expect("the io cause is in the source chain");
|
||||
assert_eq!(cause.kind(), io::ErrorKind::PermissionDenied);
|
||||
let report = Report::new(&error).to_string();
|
||||
assert_eq!(report.matches("access denied").count(), 1, "{report}");
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::configuration(Error::InvalidConfiguration(ErrorDetail::failed(
|
||||
"test credential",
|
||||
denied()
|
||||
)))]
|
||||
#[case::acquisition(Error::CredentialAcquisition(ErrorDetail::failed(
|
||||
"test credential",
|
||||
denied()
|
||||
)))]
|
||||
fn a_contextual_failure_leaves_its_cause_out_of_its_own_message(#[case] error: Error) {
|
||||
assert!(!error.to_string().contains("access denied"), "{error}");
|
||||
assert!(
|
||||
Report::new(&error)
|
||||
.to_string()
|
||||
.contains("test credential failed: access denied")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_provider_failure_keeps_the_provider_error_as_its_source() {
|
||||
let error = Error::provider(denied());
|
||||
assert_eq!(
|
||||
litellm_error::find::<io::Error>(&error).map(io::Error::kind),
|
||||
Some(io::ErrorKind::PermissionDenied)
|
||||
);
|
||||
assert!(!error.to_string().contains("access denied"), "{error}");
|
||||
let report = Report::new(&error).to_string();
|
||||
assert_eq!(report.matches("access denied").count(), 1, "{report}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_chain_keeps_its_last_failure_as_the_source_and_renders_every_attempt_once() {
|
||||
let error = Error::CredentialChain {
|
||||
last: Box::new(Error::provider(denied())),
|
||||
earlier: vec![
|
||||
Error::CredentialAcquisition(ErrorDetail::Unresolved {
|
||||
subject: "OIDC reference",
|
||||
}),
|
||||
Error::EmptyCallerCredential {
|
||||
caller: "token provider",
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
assert_eq!(
|
||||
litellm_error::find::<io::Error>(&error).map(io::Error::kind),
|
||||
Some(io::ErrorKind::PermissionDenied)
|
||||
);
|
||||
let report = Report::new(&error).to_string();
|
||||
assert_eq!(report.matches("access denied").count(), 1, "{report}");
|
||||
assert_eq!(
|
||||
report.matches("OIDC reference did not resolve").count(),
|
||||
1,
|
||||
"{report}"
|
||||
);
|
||||
assert_eq!(
|
||||
report
|
||||
.matches("token provider returned an empty credential")
|
||||
.count(),
|
||||
1,
|
||||
"{report}"
|
||||
);
|
||||
}
|
||||
|
||||
fn denied() -> io::Error {
|
||||
io::Error::new(io::ErrorKind::PermissionDenied, "access denied")
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::api_key(Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" }, true)]
|
||||
#[case::vertex_credentials(VertexParams::CREDENTIALS.missing("Vertex AI"), true)]
|
||||
#[case::aws_secret_access_key(AwsParams::SECRET_ACCESS_KEY.missing("AWS"), true)]
|
||||
#[case::vertex_project(VertexParams::PROJECT.missing("Vertex AI"), false)]
|
||||
#[case::aws_region(AwsParams::REGION.missing("AWS"), false)]
|
||||
#[case::api_base(Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" }, false)]
|
||||
#[case::invalid_configuration(Error::InvalidConfiguration(ErrorDetail::PlanNotAllowed), false)]
|
||||
fn only_an_absent_credential_is_a_missing_credential(
|
||||
#[case] error: Error,
|
||||
#[case] expected: bool,
|
||||
) {
|
||||
assert_eq!(error.is_missing_credential(), expected);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::vertex_project(VertexParams::PROJECT.missing("Vertex AI"), true)]
|
||||
#[case::aws_region(AwsParams::REGION.missing("AWS"), true)]
|
||||
#[case::api_base(Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" }, false)]
|
||||
#[case::vertex_credentials(VertexParams::CREDENTIALS.missing("Vertex AI"), false)]
|
||||
#[case::api_key(Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" }, false)]
|
||||
#[case::invalid_configuration(Error::InvalidConfiguration(ErrorDetail::PlanNotAllowed), true)]
|
||||
#[case::invalid_header(Error::InvalidHeader, true)]
|
||||
#[case::identity_provider_outage(
|
||||
Error::CredentialAcquisition(ErrorDetail::MissingResolver),
|
||||
false
|
||||
)]
|
||||
#[case::provider_sdk_failure(Error::provider(denied()), false)]
|
||||
fn a_setting_the_caller_can_pass_is_theirs_to_fix_but_credentials_and_api_bases_are_not(
|
||||
#[case] error: Error,
|
||||
#[case] expected: bool,
|
||||
) {
|
||||
assert_eq!(error.is_request(), expected);
|
||||
}
|
||||
}
|
||||
|
||||
impl From<&str> for ErrorDetail {
|
||||
fn from(message: &str) -> Self {
|
||||
Self::Message(message.into())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct ErrorSource(std::sync::Arc<dyn std::error::Error + Send + Sync>);
|
||||
|
||||
impl std::ops::Deref for ErrorSource {
|
||||
type Target = dyn std::error::Error + Send + Sync;
|
||||
|
||||
fn deref(&self) -> &Self::Target {
|
||||
self.0.as_ref()
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Display for ErrorSource {
|
||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
std::fmt::Display::fmt(&self.0, formatter)
|
||||
}
|
||||
}
|
||||
|
||||
impl ErrorSource {
|
||||
pub fn new(error: impl std::error::Error + Send + Sync + 'static) -> Self {
|
||||
Self(std::sync::Arc::new(error))
|
||||
}
|
||||
}
|
||||
|
||||
impl PartialEq for ErrorSource {
|
||||
fn eq(&self, other: &Self) -> bool {
|
||||
std::sync::Arc::ptr_eq(&self.0, &other.0)
|
||||
}
|
||||
}
|
||||
|
||||
impl Eq for ErrorSource {}
|
||||
|
|
|
|||
|
|
@ -21,9 +21,9 @@ pub fn apply_credential(
|
|||
placement: CredentialPlacement,
|
||||
) -> Result<Vec<(String, String)>, Error> {
|
||||
if credential.trim().is_empty() {
|
||||
return Err(Error::InvalidConfiguration(
|
||||
"credential cannot be empty".into(),
|
||||
));
|
||||
return Err(Error::InvalidConfiguration(crate::ErrorDetail::Empty {
|
||||
subject: "credential",
|
||||
}));
|
||||
}
|
||||
if headers
|
||||
.iter()
|
||||
|
|
@ -71,6 +71,10 @@ mod tests {
|
|||
)
|
||||
.expect_err("provider policy must handle existing credentials");
|
||||
|
||||
assert!(error.to_string().contains("already exists"));
|
||||
assert!(
|
||||
litellm_error::Report::new(&error)
|
||||
.to_string()
|
||||
.contains("already exists")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -51,9 +51,9 @@ pub use credential::{
|
|||
CredentialFileRef, CredentialLookup, CredentialLookupFuture, CredentialPlan,
|
||||
CredentialPlanResolution, CredentialRef, CredentialResolver, CredentialResolverHandle,
|
||||
};
|
||||
pub use error::{Error, ErrorDetail, ErrorSource};
|
||||
pub use error::{Error, ErrorDetail};
|
||||
pub use http::CredentialPlacement;
|
||||
pub use params::{AwsParams, ParamSpec, VertexParams};
|
||||
pub use params::{AwsParams, ParamKind, ParamSpec, VertexParams};
|
||||
pub use policy::{CredentialPlanKind, CredentialRule, ExistingHeaderBehavior, ProviderAuthPolicy};
|
||||
pub use secret::SecretValue;
|
||||
pub use token::{ResolvedCredential, TokenFuture, TokenProvider, TokenProviderHandle};
|
||||
|
|
|
|||
|
|
@ -4,15 +4,17 @@ use serde::{Deserialize, Serialize};
|
|||
use serde_json::{Map, Value};
|
||||
use veil::Redact;
|
||||
|
||||
use super::ParamSpec;
|
||||
use super::{ParamKind, ParamSpec};
|
||||
|
||||
const fn spec(
|
||||
setting: &'static str,
|
||||
kind: ParamKind,
|
||||
wire: &'static [&'static str],
|
||||
env: &'static [&'static str],
|
||||
) -> ParamSpec {
|
||||
ParamSpec {
|
||||
setting,
|
||||
kind,
|
||||
wire,
|
||||
module_global: None,
|
||||
env,
|
||||
|
|
@ -53,40 +55,67 @@ pub struct AwsParams {
|
|||
impl AwsParams {
|
||||
pub const ACCESS_KEY_ID: ParamSpec = spec(
|
||||
"access key id",
|
||||
ParamKind::Credential,
|
||||
&["aws_access_key_id"],
|
||||
&["AWS_ACCESS_KEY_ID"],
|
||||
);
|
||||
pub const SECRET_ACCESS_KEY: ParamSpec = spec(
|
||||
"secret access key",
|
||||
ParamKind::Credential,
|
||||
&["aws_secret_access_key"],
|
||||
&["AWS_SECRET_ACCESS_KEY"],
|
||||
);
|
||||
pub const SESSION_TOKEN: ParamSpec = spec(
|
||||
"session token",
|
||||
ParamKind::Credential,
|
||||
&["aws_session_token"],
|
||||
&["AWS_SESSION_TOKEN"],
|
||||
);
|
||||
pub const REGION: ParamSpec = spec(
|
||||
"region",
|
||||
ParamKind::Setting,
|
||||
&["aws_region_name"],
|
||||
&["AWS_REGION_NAME", "AWS_REGION"],
|
||||
);
|
||||
pub const SESSION_NAME: ParamSpec =
|
||||
spec("session name", &["aws_session_name"], &["AWS_SESSION_NAME"]);
|
||||
pub const PROFILE_NAME: ParamSpec =
|
||||
spec("profile name", &["aws_profile_name"], &["AWS_PROFILE_NAME"]);
|
||||
pub const ROLE_NAME: ParamSpec = spec("role name", &["aws_role_name"], &["AWS_ROLE_NAME"]);
|
||||
pub const SESSION_NAME: ParamSpec = spec(
|
||||
"session name",
|
||||
ParamKind::Setting,
|
||||
&["aws_session_name"],
|
||||
&["AWS_SESSION_NAME"],
|
||||
);
|
||||
pub const PROFILE_NAME: ParamSpec = spec(
|
||||
"profile name",
|
||||
ParamKind::Setting,
|
||||
&["aws_profile_name"],
|
||||
&["AWS_PROFILE_NAME"],
|
||||
);
|
||||
pub const ROLE_NAME: ParamSpec = spec(
|
||||
"role name",
|
||||
ParamKind::Setting,
|
||||
&["aws_role_name"],
|
||||
&["AWS_ROLE_NAME"],
|
||||
);
|
||||
pub const WEB_IDENTITY_TOKEN: ParamSpec = spec(
|
||||
"web identity token",
|
||||
ParamKind::Credential,
|
||||
&["aws_web_identity_token"],
|
||||
&["AWS_WEB_IDENTITY_TOKEN"],
|
||||
);
|
||||
pub const STS_ENDPOINT: ParamSpec =
|
||||
spec("STS endpoint", &["aws_sts_endpoint"], &["AWS_STS_ENDPOINT"]);
|
||||
pub const EXTERNAL_ID: ParamSpec =
|
||||
spec("external id", &["aws_external_id"], &["AWS_EXTERNAL_ID"]);
|
||||
pub const STS_ENDPOINT: ParamSpec = spec(
|
||||
"STS endpoint",
|
||||
ParamKind::Setting,
|
||||
&["aws_sts_endpoint"],
|
||||
&["AWS_STS_ENDPOINT"],
|
||||
);
|
||||
pub const EXTERNAL_ID: ParamSpec = spec(
|
||||
"external id",
|
||||
ParamKind::Setting,
|
||||
&["aws_external_id"],
|
||||
&["AWS_EXTERNAL_ID"],
|
||||
);
|
||||
pub const BEDROCK_RUNTIME_ENDPOINT: ParamSpec = spec(
|
||||
"Bedrock runtime endpoint",
|
||||
ParamKind::Setting,
|
||||
&["aws_bedrock_runtime_endpoint"],
|
||||
&["AWS_BEDROCK_RUNTIME_ENDPOINT"],
|
||||
);
|
||||
|
|
|
|||
|
|
@ -10,11 +10,18 @@ pub use vertex::VertexParams;
|
|||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct ParamSpec {
|
||||
pub setting: &'static str,
|
||||
pub kind: ParamKind,
|
||||
pub wire: &'static [&'static str],
|
||||
pub module_global: Option<&'static str>,
|
||||
pub env: &'static [&'static str],
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ParamKind {
|
||||
Credential,
|
||||
Setting,
|
||||
}
|
||||
|
||||
impl ParamSpec {
|
||||
/// The first non-blank value: each wire name through `params`, then each environment
|
||||
/// name through `env`.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ use serde::{Deserialize, Deserializer, Serialize, de::Unexpected};
|
|||
use serde_json::Value;
|
||||
use veil::Redact;
|
||||
|
||||
use super::ParamSpec;
|
||||
use super::{ParamKind, ParamSpec};
|
||||
|
||||
/// The Vertex AI fields of Python's `GenericLiteLLMParams`, both the current names and the
|
||||
/// `vertex_ai_*` spellings `VertexBase.safe_get_vertex_ai_*` still read.
|
||||
|
|
@ -27,18 +27,21 @@ pub struct VertexParams {
|
|||
impl VertexParams {
|
||||
pub const CREDENTIALS: ParamSpec = ParamSpec {
|
||||
setting: "credentials",
|
||||
kind: ParamKind::Credential,
|
||||
wire: &["vertex_credentials", "vertex_ai_credentials"],
|
||||
module_global: None,
|
||||
env: &["VERTEXAI_CREDENTIALS"],
|
||||
};
|
||||
pub const PROJECT: ParamSpec = ParamSpec {
|
||||
setting: "project",
|
||||
kind: ParamKind::Setting,
|
||||
wire: &["vertex_project", "vertex_ai_project"],
|
||||
module_global: Some("vertex_project"),
|
||||
env: &["VERTEXAI_PROJECT"],
|
||||
};
|
||||
pub const LOCATION: ParamSpec = ParamSpec {
|
||||
setting: "location",
|
||||
kind: ParamKind::Setting,
|
||||
wire: &["vertex_location", "vertex_ai_location"],
|
||||
module_global: Some("vertex_location"),
|
||||
env: &["VERTEXAI_LOCATION", "VERTEX_LOCATION"],
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
use crate::http::apply_credential;
|
||||
use crate::{CredentialPlacement, Error, ResolvedCredential};
|
||||
use crate::{CredentialPlacement, Error, ErrorDetail, ResolvedCredential};
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum CredentialPlanKind {
|
||||
|
|
@ -31,11 +31,18 @@ pub struct ProviderAuthPolicy {
|
|||
|
||||
impl ProviderAuthPolicy {
|
||||
pub fn has_existing_credential(&self, headers: &[(String, String)]) -> bool {
|
||||
headers.iter().any(|(name, _)| {
|
||||
self.accepted_existing_headers
|
||||
.iter()
|
||||
.any(|accepted| name.eq_ignore_ascii_case(accepted))
|
||||
})
|
||||
self.existing_credential_header(headers).is_some()
|
||||
}
|
||||
|
||||
fn existing_credential_header(&self, headers: &[(String, String)]) -> Option<&'static str> {
|
||||
self.accepted_existing_headers
|
||||
.iter()
|
||||
.copied()
|
||||
.find(|accepted| {
|
||||
headers
|
||||
.iter()
|
||||
.any(|(name, _)| name.eq_ignore_ascii_case(accepted))
|
||||
})
|
||||
}
|
||||
|
||||
pub fn apply(
|
||||
|
|
@ -44,11 +51,11 @@ impl ProviderAuthPolicy {
|
|||
kind: CredentialPlanKind,
|
||||
credential: &ResolvedCredential,
|
||||
) -> Result<Vec<(String, String)>, Error> {
|
||||
if self.has_existing_credential(&headers) {
|
||||
if let Some(existing) = self.existing_credential_header(&headers) {
|
||||
return match self.existing_header_behavior {
|
||||
ExistingHeaderBehavior::Preserve => Ok(headers),
|
||||
ExistingHeaderBehavior::Reject => Err(Error::InvalidConfiguration(
|
||||
"credential header already exists".into(),
|
||||
ErrorDetail::DuplicateHeader(existing),
|
||||
)),
|
||||
};
|
||||
}
|
||||
|
|
@ -56,11 +63,7 @@ impl ProviderAuthPolicy {
|
|||
.rules
|
||||
.iter()
|
||||
.find(|rule| rule.kind == kind)
|
||||
.ok_or_else(|| {
|
||||
Error::InvalidConfiguration(
|
||||
"credential plan is not allowed by the provider auth policy".into(),
|
||||
)
|
||||
})?;
|
||||
.ok_or(Error::InvalidConfiguration(ErrorDetail::PlanNotAllowed))?;
|
||||
apply_credential(headers, credential.secret().expose(), rule.placement)
|
||||
}
|
||||
}
|
||||
|
|
@ -108,6 +111,10 @@ mod tests {
|
|||
)
|
||||
.unwrap_err();
|
||||
|
||||
assert!(error.to_string().contains("not allowed"));
|
||||
assert!(
|
||||
litellm_error::Report::new(&error)
|
||||
.to_string()
|
||||
.contains("not allowed")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,74 +0,0 @@
|
|||
use litellm_auth_types::Error;
|
||||
use rstest::rstest;
|
||||
|
||||
#[rstest]
|
||||
#[case::api_key(
|
||||
Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" },
|
||||
"Missing Example API Key - Set `api_key` or the EXAMPLE_API_KEY environment variable"
|
||||
)]
|
||||
#[case::another_api_key(
|
||||
Error::MissingApiKey { provider: "Custom", environment_variable: "CUSTOM_KEY" },
|
||||
"Missing Custom API Key - Set `api_key` or the CUSTOM_KEY environment variable"
|
||||
)]
|
||||
#[case::api_base(
|
||||
Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" },
|
||||
"Missing Example API Base - Pass api_base"
|
||||
)]
|
||||
#[case::another_api_base(
|
||||
Error::MissingApiBase { provider: "Custom", guidance: "Set CUSTOM_ENDPOINT" },
|
||||
"Missing Custom API Base - Set CUSTOM_ENDPOINT"
|
||||
)]
|
||||
#[case::configuration(
|
||||
Error::InvalidConfiguration("credential selector is invalid".into()),
|
||||
"invalid authentication configuration: credential selector is invalid"
|
||||
)]
|
||||
#[case::acquisition(
|
||||
Error::CredentialAcquisition("token expired".into()),
|
||||
"credential acquisition failed: token expired"
|
||||
)]
|
||||
#[case::caller(
|
||||
Error::EmptyCallerCredential("empty token"),
|
||||
"credential caller failed: empty token"
|
||||
)]
|
||||
#[case::provider(
|
||||
Error::ProviderAuthentication("provider rejected credentials".into()),
|
||||
"provider rejected credentials"
|
||||
)]
|
||||
#[case::chain(
|
||||
Error::CredentialChain(vec![
|
||||
Error::CredentialAcquisition("token expired".into()),
|
||||
Error::EmptyCallerCredential("empty token"),
|
||||
]),
|
||||
"credential acquisition failed: credential acquisition failed: token expired; credential caller failed: empty token"
|
||||
)]
|
||||
fn display_preserves_failure_phase_and_caller_context(
|
||||
#[case] error: Error,
|
||||
#[case] expected: &str,
|
||||
) {
|
||||
assert_eq!(error.to_string(), expected);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::configuration(true)]
|
||||
#[case::acquisition(false)]
|
||||
fn contextual_failures_keep_the_original_source(#[case] configuration: bool) {
|
||||
use litellm_auth_types::ErrorDetail;
|
||||
|
||||
let detail = ErrorDetail::failed(
|
||||
"test credential",
|
||||
std::io::Error::from(std::io::ErrorKind::PermissionDenied),
|
||||
);
|
||||
let error = if configuration {
|
||||
Error::InvalidConfiguration(detail)
|
||||
} else {
|
||||
Error::CredentialAcquisition(detail)
|
||||
};
|
||||
let source = std::iter::successors(Some(&error as &dyn std::error::Error), |error| {
|
||||
error.source()
|
||||
})
|
||||
.find_map(|error| error.downcast_ref::<std::io::Error>())
|
||||
.expect("the original credential error remains available");
|
||||
assert_eq!(source.kind(), std::io::ErrorKind::PermissionDenied);
|
||||
assert!(error.to_string().contains("test credential failed:"));
|
||||
assert!(error.to_string().ends_with(&source.to_string()));
|
||||
}
|
||||
|
|
@ -1,8 +1,9 @@
|
|||
use litellm_auth_types::{AwsParams, ParamSpec};
|
||||
use litellm_auth_types::{AwsParams, ParamKind, ParamSpec};
|
||||
use rstest::rstest;
|
||||
|
||||
const LOCATION: ParamSpec = ParamSpec {
|
||||
setting: "location",
|
||||
kind: ParamKind::Setting,
|
||||
wire: &["vertex_location", "vertex_ai_location"],
|
||||
module_global: Some("vertex_location"),
|
||||
env: &["VERTEXAI_LOCATION", "VERTEX_LOCATION"],
|
||||
|
|
|
|||
|
|
@ -1,5 +1,4 @@
|
|||
use std::{
|
||||
error::Error as StdError,
|
||||
future::{Future, poll_fn},
|
||||
sync::{
|
||||
Arc,
|
||||
|
|
@ -53,7 +52,7 @@ async fn callbacks_acquire_fresh_credentials_on_demand(#[case] access_token: boo
|
|||
|
||||
#[rstest]
|
||||
#[tokio::test]
|
||||
async fn callback_errors_preserve_the_original_source() {
|
||||
async fn callback_errors_keep_their_typed_cause() {
|
||||
let provider = TokenProviderHandle::from_callback(|| async {
|
||||
Err(Error::CredentialAcquisition(ErrorDetail::failed(
|
||||
"caller credential",
|
||||
|
|
@ -62,13 +61,15 @@ async fn callback_errors_preserve_the_original_source() {
|
|||
});
|
||||
|
||||
let error = provider.acquire().await.unwrap_err();
|
||||
assert!(matches!(error, Error::CredentialAcquisition(_)));
|
||||
let source = std::iter::successors(Some(&error as &(dyn StdError + 'static)), |error| {
|
||||
(*error).source()
|
||||
})
|
||||
.find_map(|error| error.downcast_ref::<std::io::Error>())
|
||||
.unwrap();
|
||||
assert_eq!(source.kind(), std::io::ErrorKind::PermissionDenied);
|
||||
let Error::CredentialAcquisition(ErrorDetail::Failed { cause, .. }) = &error else {
|
||||
panic!("the callback's error passes through unchanged: {error:?}");
|
||||
};
|
||||
assert_eq!(
|
||||
cause
|
||||
.downcast_ref::<std::io::Error>()
|
||||
.map(std::io::Error::kind),
|
||||
Some(std::io::ErrorKind::PermissionDenied)
|
||||
);
|
||||
}
|
||||
|
||||
struct Release(Arc<AtomicBool>);
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ azure_core = "1.1.0"
|
|||
azure_storage_blob = "1.1.0"
|
||||
futures-util.workspace = true
|
||||
reqwest.workspace = true
|
||||
thiserror.workspace = true
|
||||
tokio.workspace = true
|
||||
url.workspace = true
|
||||
|
||||
|
|
|
|||
|
|
@ -18,6 +18,10 @@ use url::Url;
|
|||
|
||||
use crate::{credential::AzureBlobCredential, transport::ReqwestTransport};
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("account URL cannot hold a container path")]
|
||||
struct NotABaseUrl;
|
||||
|
||||
pub struct AzureBlobCache<C> {
|
||||
container: BlobContainerClient,
|
||||
codec: C,
|
||||
|
|
@ -57,12 +61,12 @@ impl<C: CacheCodec> AzureBlobCache<C> {
|
|||
codec: C,
|
||||
runtime: Handle,
|
||||
) -> Result<Self, Error> {
|
||||
let parsed = Url::parse(account_url).map_err(|_| Error::Unavailable)?;
|
||||
let parsed = Url::parse(account_url).map_err(Error::unavailable)?;
|
||||
let account_url = parsed.as_str().trim_end_matches('/').to_string();
|
||||
let container_url = {
|
||||
let mut url = parsed;
|
||||
url.path_segments_mut()
|
||||
.map_err(|()| Error::Unavailable)?
|
||||
.map_err(|()| Error::unavailable(NotABaseUrl))?
|
||||
.pop_if_empty()
|
||||
.push(container);
|
||||
url
|
||||
|
|
@ -75,7 +79,7 @@ impl<C: CacheCodec> AzureBlobCache<C> {
|
|||
..BlobContainerClientOptions::default()
|
||||
}),
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let cache = Self {
|
||||
container: client,
|
||||
codec,
|
||||
|
|
@ -101,7 +105,7 @@ impl<C: CacheCodec> AzureBlobCache<C> {
|
|||
Err(error) if is_storage_error(&error, StorageErrorCode::ContainerAlreadyExists) => {
|
||||
Ok(())
|
||||
}
|
||||
Err(_) => Err(Error::Unavailable),
|
||||
Err(error) => Err(Error::unavailable(error)),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -116,7 +120,7 @@ impl<C: CacheCodec> AzureBlobCache<C> {
|
|||
{
|
||||
Ok(_) => Ok(()),
|
||||
Err(error) if !overwrite && is_already_present(&error) => Ok(()),
|
||||
Err(_) => Err(Error::Unavailable),
|
||||
Err(error) => Err(Error::unavailable(error)),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -126,13 +130,9 @@ impl<C: CacheCodec> AzureBlobCache<C> {
|
|||
Err(error) if is_storage_error(&error, StorageErrorCode::BlobNotFound) => {
|
||||
return Ok(None);
|
||||
}
|
||||
Err(_) => return Err(Error::Unavailable),
|
||||
Err(error) => return Err(Error::unavailable(error)),
|
||||
};
|
||||
let bytes = response
|
||||
.body
|
||||
.collect()
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
let bytes = response.body.collect().await.map_err(Error::unavailable)?;
|
||||
self.codec.decode(&bytes).map(Some)
|
||||
}
|
||||
|
||||
|
|
@ -140,16 +140,16 @@ impl<C: CacheCodec> AzureBlobCache<C> {
|
|||
let mut pages = self
|
||||
.container
|
||||
.list_blobs(None)
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.into_pages();
|
||||
while let Some(page) = pages.try_next().await.map_err(|_| Error::Unavailable)? {
|
||||
let page = page.into_model().map_err(|_| Error::Unavailable)?;
|
||||
while let Some(page) = pages.try_next().await.map_err(Error::unavailable)? {
|
||||
let page = page.into_model().map_err(Error::unavailable)?;
|
||||
for name in page.blob_items.into_iter().filter_map(|item| item.name) {
|
||||
self.container
|
||||
.blob_client(&name)
|
||||
.delete(None)
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
|
|
|
|||
|
|
@ -17,6 +17,10 @@ const LLM_TOKEN_ENV: &str = "AZURE_AD_TOKEN";
|
|||
|
||||
type EnvLookup = Arc<dyn Fn(&str) -> Option<String> + Send + Sync>;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("no Azure credential is available for blob storage")]
|
||||
struct NoCredential;
|
||||
|
||||
pub struct AzureBlobCredential {
|
||||
service: AzureAuthService,
|
||||
env_lookup: EnvLookup,
|
||||
|
|
@ -62,15 +66,8 @@ impl TokenCredential for AzureBlobCredential {
|
|||
&lookup,
|
||||
)
|
||||
.await
|
||||
.map_err(|error| {
|
||||
azure_core::Error::with_message(ErrorKind::Credential, error.to_string())
|
||||
})?
|
||||
.ok_or_else(|| {
|
||||
azure_core::Error::with_message(
|
||||
ErrorKind::Credential,
|
||||
"no Azure credential is available for blob storage",
|
||||
)
|
||||
})?;
|
||||
.map_err(|error| azure_core::Error::new(ErrorKind::Credential, error))?
|
||||
.ok_or_else(|| azure_core::Error::new(ErrorKind::Credential, NoCredential))?;
|
||||
let (token, expires_on) = match credential.into_value() {
|
||||
ResolvedCredential::AccessToken { token, expires_on } => (token, expires_on),
|
||||
ResolvedCredential::Static(token) => (token, None),
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ impl HttpClient for ReqwestTransport {
|
|||
Body::SeekableStream(stream) => outgoing.body(reqwest::Body::wrap_stream(stream)),
|
||||
};
|
||||
let response = outgoing.send().await.map_err(|error| {
|
||||
let error = litellm_http::Error::from(error);
|
||||
let kind = if error.is_connect() {
|
||||
ErrorKind::Connection
|
||||
} else {
|
||||
|
|
@ -41,9 +42,9 @@ impl HttpClient for ReqwestTransport {
|
|||
);
|
||||
}
|
||||
}
|
||||
let body = response
|
||||
.bytes_stream()
|
||||
.map_err(|error| azure_core::Error::new(ErrorKind::Io, error));
|
||||
let body = response.bytes_stream().map_err(|error| {
|
||||
azure_core::Error::new(ErrorKind::Io, litellm_http::Error::from(error))
|
||||
});
|
||||
Ok(AsyncRawResponse::new(status, headers, Box::pin(body)))
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -128,7 +128,7 @@ fn connect_surfaces_service_failures() {
|
|||
ResponseCacheCodec,
|
||||
runtime.handle().clone(),
|
||||
));
|
||||
assert!(matches!(result, Err(Error::Unavailable)));
|
||||
assert!(matches!(result, Err(Error::Unavailable(_))));
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
@ -263,7 +263,7 @@ fn malformed_blobs_are_invalid_entries(fixture: Fixture, #[case] key: &str, #[ca
|
|||
fixture.service.seed_blob(key, bytes);
|
||||
assert!(matches!(
|
||||
fixture.cache.get_cache(key, &no_ttl()),
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(_))
|
||||
));
|
||||
}
|
||||
|
||||
|
|
@ -378,15 +378,15 @@ fn service_failures_map_to_unavailable(fixture: Fixture) {
|
|||
fixture.service.set_failing(true);
|
||||
assert!(matches!(
|
||||
fixture.cache.get_cache("key", &no_ttl()),
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(_))
|
||||
));
|
||||
assert!(matches!(
|
||||
fixture.cache.set_cache("key", entry(json!(1)), &no_ttl()),
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(_))
|
||||
));
|
||||
assert!(matches!(
|
||||
fixture.cache.flush_cache(),
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(_))
|
||||
));
|
||||
assert!(matches!(
|
||||
fixture.runtime.block_on(
|
||||
|
|
@ -394,7 +394,7 @@ fn service_failures_map_to_unavailable(fixture: Fixture) {
|
|||
.cache
|
||||
.async_set_cache_pipeline(vec![("k".into(), entry(json!(1)))], no_ttl())
|
||||
),
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(_))
|
||||
));
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ rand.workspace = true
|
|||
rusqlite = { version = "0.39", features = ["bundled"] }
|
||||
serde-pickle = "1.2"
|
||||
serde_json.workspace = true
|
||||
thiserror.workspace = true
|
||||
tokio.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ impl<S: CacheCodec, D: DiskStore, A: ValueAdapter> DiskCache<S, D, A> {
|
|||
{
|
||||
tokio::task::spawn_blocking(move || operation(&store))
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -162,7 +162,7 @@ impl<S: CacheCodec, D: DiskStore, A: ValueAdapter> BatchCache for DiskCache<S, D
|
|||
.map(|key| match self.get_cache(key, context) {
|
||||
Ok(Some(value)) => Ok(BatchEntry::Hit(value)),
|
||||
Ok(None) => Ok(BatchEntry::Miss),
|
||||
Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid),
|
||||
Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid),
|
||||
Err(error) => Err(error),
|
||||
})
|
||||
.collect()
|
||||
|
|
@ -186,7 +186,7 @@ impl<S: CacheCodec, D: DiskStore, A: ValueAdapter> BatchCache for DiskCache<S, D
|
|||
Some(value) => match self.decode_stored(value) {
|
||||
Ok(Some(value)) => Ok(BatchEntry::Hit(value)),
|
||||
Ok(None) => Ok(BatchEntry::Miss),
|
||||
Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid),
|
||||
Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid),
|
||||
Err(error) => Err(error),
|
||||
},
|
||||
})
|
||||
|
|
@ -272,7 +272,7 @@ fn increment<A: ValueAdapter, D: DiskStore>(
|
|||
Ok((stored, ttl.map(|ttl| unix_now() + ttl.as_secs_f64())))
|
||||
};
|
||||
store.update(key, unix_now(), &mut apply)?;
|
||||
result.ok_or(Error::InvalidEntry)
|
||||
result.ok_or(Error::InvalidEntry(None))
|
||||
}
|
||||
|
||||
fn unix_now() -> f64 {
|
||||
|
|
|
|||
|
|
@ -3,8 +3,8 @@ use py_literal::Value;
|
|||
use serde_json::{Map, Number};
|
||||
|
||||
pub(crate) fn from_pickle(bytes: &[u8]) -> Result<Value, Error> {
|
||||
let value = serde_pickle::value_from_slice(bytes, Default::default())
|
||||
.map_err(|_| Error::InvalidEntry)?;
|
||||
let value =
|
||||
serde_pickle::value_from_slice(bytes, Default::default()).map_err(Error::invalid_entry)?;
|
||||
from_pickle_value(value)
|
||||
}
|
||||
|
||||
|
|
@ -70,7 +70,7 @@ fn from_pickle_hashable(value: serde_pickle::HashableValue) -> Result<Value, Err
|
|||
}
|
||||
|
||||
fn integer(value: String) -> Result<Value, Error> {
|
||||
value.parse().map_err(|_| Error::InvalidEntry)
|
||||
value.parse().map_err(Error::invalid_entry)
|
||||
}
|
||||
|
||||
pub(crate) fn from_json(value: serde_json::Value) -> Value {
|
||||
|
|
@ -101,7 +101,7 @@ pub(crate) fn from_json(value: serde_json::Value) -> Value {
|
|||
pub(crate) fn from_json_text(value: &str) -> Result<Value, Error> {
|
||||
serde_json::from_str(value)
|
||||
.map(from_json)
|
||||
.map_err(|_| Error::InvalidEntry)
|
||||
.map_err(Error::invalid_entry)
|
||||
}
|
||||
|
||||
pub(crate) fn is_truthy(value: &Value) -> bool {
|
||||
|
|
@ -131,7 +131,7 @@ pub(crate) fn to_f64(value: &Value) -> Option<f64> {
|
|||
}
|
||||
|
||||
pub(crate) fn to_json(value: &Value) -> Result<Vec<u8>, Error> {
|
||||
serde_json::to_vec(&to_json_value(value)?).map_err(|_| Error::InvalidEntry)
|
||||
serde_json::to_vec(&to_json_value(value)?).map_err(Error::invalid_entry)
|
||||
}
|
||||
|
||||
fn to_json_value(value: &Value) -> Result<serde_json::Value, Error> {
|
||||
|
|
@ -142,12 +142,12 @@ fn to_json_value(value: &Value) -> Result<serde_json::Value, Error> {
|
|||
value
|
||||
.to_string()
|
||||
.parse::<Number>()
|
||||
.map_err(|_| Error::InvalidEntry)?,
|
||||
.map_err(Error::invalid_entry)?,
|
||||
),
|
||||
Value::Float(value) => {
|
||||
serde_json::Value::Number(Number::from_f64(*value).ok_or(Error::InvalidEntry)?)
|
||||
serde_json::Value::Number(Number::from_f64(*value).ok_or(Error::InvalidEntry(None))?)
|
||||
}
|
||||
Value::Complex(_) | Value::Bytes(_) => return Err(Error::InvalidEntry),
|
||||
Value::Complex(_) | Value::Bytes(_) => return Err(Error::InvalidEntry(None)),
|
||||
Value::String(value) => serde_json::Value::String(value.clone()),
|
||||
Value::Tuple(values) | Value::List(values) | Value::Set(values) => {
|
||||
serde_json::Value::Array(
|
||||
|
|
@ -162,7 +162,7 @@ fn to_json_value(value: &Value) -> Result<serde_json::Value, Error> {
|
|||
.iter()
|
||||
.map(|(key, value)| {
|
||||
let Value::String(key) = key else {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
};
|
||||
Ok((key.clone(), to_json_value(value)?))
|
||||
})
|
||||
|
|
|
|||
|
|
@ -48,13 +48,13 @@ struct Row {
|
|||
impl DiskcacheSqliteStore {
|
||||
pub fn open(directory: impl AsRef<Path>) -> Result<Self, Error> {
|
||||
let directory = directory.as_ref().to_path_buf();
|
||||
fs::create_dir_all(&directory).map_err(|_| Error::Unavailable)?;
|
||||
let directory = std::path::absolute(&directory).map_err(|_| Error::Unavailable)?;
|
||||
fs::create_dir_all(&directory).map_err(Error::unavailable)?;
|
||||
let directory = std::path::absolute(&directory).map_err(Error::unavailable)?;
|
||||
let database = directory.join("cache.db");
|
||||
let connection = Connection::open(database).map_err(|_| Error::Unavailable)?;
|
||||
let connection = Connection::open(database).map_err(Error::unavailable)?;
|
||||
connection
|
||||
.busy_timeout(std::time::Duration::from_secs(60))
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
|
||||
let mut settings = read_settings(&connection)?;
|
||||
for (key, value) in default_settings() {
|
||||
|
|
@ -74,7 +74,7 @@ impl DiskcacheSqliteStore {
|
|||
value
|
||||
)",
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
for (key, value) in &settings {
|
||||
if !matches!(key.as_str(), "count" | "size" | "hits" | "misses") {
|
||||
connection
|
||||
|
|
@ -82,7 +82,7 @@ impl DiskcacheSqliteStore {
|
|||
"INSERT OR REPLACE INTO Settings VALUES (?, ?)",
|
||||
params![key, value],
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
}
|
||||
for (key, value) in [
|
||||
|
|
@ -96,7 +96,7 @@ impl DiskcacheSqliteStore {
|
|||
"INSERT OR IGNORE INTO Settings VALUES (?, ?)",
|
||||
params![key, value],
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
connection
|
||||
.execute_batch(
|
||||
|
|
@ -117,7 +117,7 @@ impl DiskcacheSqliteStore {
|
|||
CREATE UNIQUE INDEX IF NOT EXISTS Cache_key_raw ON Cache(key, raw);
|
||||
CREATE INDEX IF NOT EXISTS Cache_expire_time ON Cache(expire_time);",
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
|
||||
let eviction_policy = setting_string(&settings, "eviction_policy")
|
||||
.unwrap_or_else(|| "least-recently-stored".to_string());
|
||||
|
|
@ -128,23 +128,27 @@ impl DiskcacheSqliteStore {
|
|||
.execute_batch(
|
||||
"CREATE INDEX IF NOT EXISTS Cache_store_time ON Cache(store_time)",
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
"least-recently-used" => {
|
||||
connection
|
||||
.execute_batch(
|
||||
"CREATE INDEX IF NOT EXISTS Cache_access_time ON Cache(access_time)",
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
"least-frequently-used" => {
|
||||
connection
|
||||
.execute_batch(
|
||||
"CREATE INDEX IF NOT EXISTS Cache_access_count ON Cache(access_count)",
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
_ => {
|
||||
return Err(Error::unavailable(SettingFault::EvictionPolicy(
|
||||
eviction_policy,
|
||||
)));
|
||||
}
|
||||
_ => return Err(Error::Unavailable),
|
||||
}
|
||||
connection
|
||||
.execute_batch(
|
||||
|
|
@ -170,12 +174,12 @@ impl DiskcacheSqliteStore {
|
|||
UPDATE Settings SET value = value - OLD.size
|
||||
WHERE key = \"size\"; END;",
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
|
||||
let min_file_size = setting_i64(&settings, "disk_min_file_size")
|
||||
.unwrap_or(DEFAULT_DISK_MIN_FILE_SIZE)
|
||||
.try_into()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let size_limit = setting_i64(&settings, "size_limit").unwrap_or(DEFAULT_SIZE_LIMIT);
|
||||
let cull_limit = setting_i64(&settings, "cull_limit").unwrap_or(DEFAULT_CULL_LIMIT);
|
||||
let statistics = setting_i64(&settings, "statistics").unwrap_or_default() != 0;
|
||||
|
|
@ -207,7 +211,7 @@ impl DiskcacheSqliteStore {
|
|||
|row| row.get::<_, Option<String>>(0),
|
||||
)
|
||||
.optional()
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.flatten()
|
||||
{
|
||||
cleanup.push(old_filename);
|
||||
|
|
@ -221,7 +225,7 @@ impl DiskcacheSqliteStore {
|
|||
|row| row.get::<_, i64>(0),
|
||||
)
|
||||
.optional()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
if let Some(rowid) = rowid {
|
||||
connection
|
||||
.execute(
|
||||
|
|
@ -230,7 +234,7 @@ impl DiskcacheSqliteStore {
|
|||
WHERE rowid = ?",
|
||||
params![now, expire_time, now, size, mode, filename, value, rowid],
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
} else {
|
||||
connection
|
||||
.execute(
|
||||
|
|
@ -240,7 +244,7 @@ impl DiskcacheSqliteStore {
|
|||
) VALUES (?, 1, ?, ?, ?, 0, NULL, ?, ?, ?, ?)",
|
||||
params![key, now, expire_time, now, size, mode, filename, value],
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
cleanup.extend(self.cull(connection, now)?);
|
||||
Ok(cleanup)
|
||||
|
|
@ -257,13 +261,13 @@ impl DiskcacheSqliteStore {
|
|||
WHERE expire_time IS NOT NULL AND expire_time < ?
|
||||
ORDER BY expire_time LIMIT ?",
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.query_map(params![now, self.cull_limit], |row| {
|
||||
Ok((row.get::<_, i64>(0)?, row.get::<_, Option<String>>(1)?))
|
||||
})
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.collect::<Result<Vec<_>, _>>()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
for (_, filename) in &expired {
|
||||
if let Some(filename) = filename {
|
||||
cleanup.push(filename.clone());
|
||||
|
|
@ -272,7 +276,7 @@ impl DiskcacheSqliteStore {
|
|||
for (rowid, _) in &expired {
|
||||
connection
|
||||
.execute("DELETE FROM Cache WHERE rowid = ?", params![rowid])
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
let remaining = self.cull_limit - i64::try_from(expired.len()).unwrap_or(self.cull_limit);
|
||||
if remaining <= 0 || self.volume(connection)? < self.size_limit {
|
||||
|
|
@ -283,19 +287,23 @@ impl DiskcacheSqliteStore {
|
|||
"least-recently-stored" => "store_time",
|
||||
"least-recently-used" => "access_time",
|
||||
"least-frequently-used" => "access_count",
|
||||
_ => return Err(Error::Unavailable),
|
||||
policy => {
|
||||
return Err(Error::unavailable(SettingFault::EvictionPolicy(
|
||||
policy.to_owned(),
|
||||
)));
|
||||
}
|
||||
};
|
||||
let rows = connection
|
||||
.prepare(&format!(
|
||||
"SELECT rowid, filename FROM Cache ORDER BY {order} LIMIT ?"
|
||||
))
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.query_map(params![remaining], |row| {
|
||||
Ok((row.get::<_, i64>(0)?, row.get::<_, Option<String>>(1)?))
|
||||
})
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.collect::<Result<Vec<_>, _>>()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
for (_, filename) in &rows {
|
||||
if let Some(filename) = filename {
|
||||
cleanup.push(filename.clone());
|
||||
|
|
@ -304,7 +312,7 @@ impl DiskcacheSqliteStore {
|
|||
for (rowid, _) in rows {
|
||||
connection
|
||||
.execute("DELETE FROM Cache WHERE rowid = ?", params![rowid])
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
Ok(cleanup)
|
||||
}
|
||||
|
|
@ -312,15 +320,15 @@ impl DiskcacheSqliteStore {
|
|||
fn volume(&self, connection: &Connection) -> Result<i64, Error> {
|
||||
let page_count: i64 = connection
|
||||
.query_row("PRAGMA page_count", [], |row| row.get(0))
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let page_size: i64 = connection
|
||||
.query_row("PRAGMA page_size", [], |row| row.get(0))
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let size: i64 = connection
|
||||
.query_row("SELECT value FROM Settings WHERE key = 'size'", [], |row| {
|
||||
row.get(0)
|
||||
})
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
Ok(page_count.saturating_mul(page_size).saturating_add(size))
|
||||
}
|
||||
}
|
||||
|
|
@ -331,13 +339,13 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
}
|
||||
|
||||
fn get(&self, key: &str, now: f64) -> Result<Option<StoredValue>, Error> {
|
||||
let connection = self.connection.lock().map_err(|_| Error::Unavailable)?;
|
||||
let connection = self.connection.lock()?;
|
||||
let select = "SELECT rowid, expire_time, mode, filename, value FROM Cache
|
||||
WHERE key = ? AND raw = 1 AND (expire_time IS NULL OR expire_time > ?)";
|
||||
let row = connection
|
||||
.query_row(select, params![key, now], row_from_query)
|
||||
.optional()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
if !self.statistics && !has_get_update(&self.eviction_policy) {
|
||||
return row
|
||||
.map(|row| fetch_row(&self.directory, row))
|
||||
|
|
@ -348,7 +356,7 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
let row = connection
|
||||
.query_row(select, params![key, now], row_from_query)
|
||||
.optional()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let Some(row) = row else {
|
||||
if self.statistics {
|
||||
connection
|
||||
|
|
@ -356,7 +364,7 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
"UPDATE Settings SET value = value + 1 WHERE key = 'misses'",
|
||||
[],
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
return Ok(None);
|
||||
};
|
||||
|
|
@ -369,14 +377,14 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
"UPDATE Settings SET value = value + 1 WHERE key = 'hits'",
|
||||
[],
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
} else if !hit && self.statistics {
|
||||
connection
|
||||
.execute(
|
||||
"UPDATE Settings SET value = value + 1 WHERE key = 'misses'",
|
||||
[],
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
if has_get_update(&self.eviction_policy) && hit {
|
||||
let update = match self.eviction_policy.as_str() {
|
||||
|
|
@ -384,16 +392,20 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
"least-frequently-used" => {
|
||||
"UPDATE Cache SET access_count = access_count + 1 WHERE rowid = ?"
|
||||
}
|
||||
_ => return Err(Error::Unavailable),
|
||||
policy => {
|
||||
return Err(Error::unavailable(SettingFault::EvictionPolicy(
|
||||
policy.to_owned(),
|
||||
)));
|
||||
}
|
||||
};
|
||||
if self.eviction_policy == "least-recently-used" {
|
||||
connection
|
||||
.execute(update, params![now, rowid])
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
} else {
|
||||
connection
|
||||
.execute(update, params![rowid])
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
}
|
||||
value
|
||||
|
|
@ -409,7 +421,7 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
) -> Result<(), Error> {
|
||||
let columns = store_value(&self.directory, self.min_file_size, value)?;
|
||||
let new_filename = columns.filename.clone();
|
||||
let connection = self.connection.lock().map_err(|_| Error::Unavailable)?;
|
||||
let connection = self.connection.lock()?;
|
||||
let result = transactional(&connection, |connection| {
|
||||
self.set_locked(connection, key, columns, expire_time, now)
|
||||
});
|
||||
|
|
@ -428,7 +440,7 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
}
|
||||
|
||||
fn pop(&self, key: &str, now: f64) -> Result<Option<StoredValue>, Error> {
|
||||
let connection = self.connection.lock().map_err(|_| Error::Unavailable)?;
|
||||
let connection = self.connection.lock()?;
|
||||
let selected = transactional(&connection, |connection| {
|
||||
let row = connection
|
||||
.query_row(
|
||||
|
|
@ -439,13 +451,13 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
row_from_query,
|
||||
)
|
||||
.optional()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let Some(row) = row else {
|
||||
return Ok(None);
|
||||
};
|
||||
connection
|
||||
.execute("DELETE FROM Cache WHERE rowid = ?", params![row.rowid])
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
Ok(Some(row))
|
||||
})?;
|
||||
let Some(row) = selected else {
|
||||
|
|
@ -460,7 +472,7 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
}
|
||||
|
||||
fn clear(&self) -> Result<(), Error> {
|
||||
let connection = self.connection.lock().map_err(|_| Error::Unavailable)?;
|
||||
let connection = self.connection.lock()?;
|
||||
let mut last_rowid = 0_i64;
|
||||
loop {
|
||||
let batch = transactional(&connection, |connection| {
|
||||
|
|
@ -469,13 +481,13 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
"SELECT rowid, filename FROM Cache
|
||||
WHERE rowid > ? ORDER BY rowid LIMIT 100",
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.query_map(params![last_rowid], |row| {
|
||||
Ok((row.get::<_, i64>(0)?, row.get::<_, Option<String>>(1)?))
|
||||
})
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.collect::<Result<Vec<_>, _>>()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
if rows.is_empty() {
|
||||
return Ok(rows);
|
||||
}
|
||||
|
|
@ -486,7 +498,7 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
.join(",");
|
||||
connection
|
||||
.execute(&format!("DELETE FROM Cache WHERE rowid IN ({ids})"), [])
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
Ok(rows)
|
||||
})?;
|
||||
if batch.is_empty() {
|
||||
|
|
@ -509,7 +521,7 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
now: f64,
|
||||
apply: &mut dyn FnMut(Option<StoredValue>) -> Result<(StoredValue, Option<f64>), Error>,
|
||||
) -> Result<(), Error> {
|
||||
let connection = self.connection.lock().map_err(|_| Error::Unavailable)?;
|
||||
let connection = self.connection.lock()?;
|
||||
let mut created_filename = None;
|
||||
let result = transactional(&connection, |connection| {
|
||||
let current = connection
|
||||
|
|
@ -521,7 +533,7 @@ impl DiskStore for DiskcacheSqliteStore {
|
|||
row_from_query,
|
||||
)
|
||||
.optional()
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.map(|row| fetch_row(&self.directory, row))
|
||||
.transpose()?
|
||||
.flatten();
|
||||
|
|
@ -582,21 +594,35 @@ fn read_settings(connection: &Connection) -> Result<HashMap<String, Value>, Erro
|
|||
};
|
||||
statement
|
||||
.query_map([], |row| Ok((row.get(0)?, row.get(1)?)))
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.collect::<Result<HashMap<_, _>, _>>()
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
enum SettingFault {
|
||||
#[error("unknown eviction policy {0:?}")]
|
||||
EvictionPolicy(String),
|
||||
#[error("setting {0:?} is not an SQLite pragma")]
|
||||
NotPragma(String),
|
||||
#[error("setting {0:?} has an unsupported value type")]
|
||||
PragmaValue(String),
|
||||
}
|
||||
|
||||
fn apply_pragma(connection: &Connection, key: &str, value: &Value) -> Result<(), Error> {
|
||||
let pragma = key.strip_prefix("sqlite_").ok_or(Error::Unavailable)?;
|
||||
let pragma = key
|
||||
.strip_prefix("sqlite_")
|
||||
.ok_or_else(|| Error::unavailable(SettingFault::NotPragma(key.to_owned())))?;
|
||||
match value {
|
||||
Value::Integer(value) => connection
|
||||
.pragma_update(None, pragma, value)
|
||||
.map_err(|_| Error::Unavailable),
|
||||
.map_err(Error::unavailable),
|
||||
Value::Text(value) => connection
|
||||
.pragma_update(None, pragma, value)
|
||||
.map_err(|_| Error::Unavailable),
|
||||
_ => Err(Error::Unavailable),
|
||||
.map_err(Error::unavailable),
|
||||
_ => Err(Error::unavailable(SettingFault::PragmaValue(
|
||||
key.to_owned(),
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -634,7 +660,7 @@ fn fetch_row(directory: &Path, row: Row) -> Result<Option<StoredValue>, Error> {
|
|||
Value::Text(value) => Ok(Some(StoredValue::Text(value))),
|
||||
Value::Integer(value) => Ok(Some(StoredValue::Integer(value))),
|
||||
Value::Real(value) => Ok(Some(StoredValue::Float(value))),
|
||||
Value::Null => Err(Error::InvalidEntry),
|
||||
Value::Null => Err(Error::InvalidEntry(None)),
|
||||
},
|
||||
MODE_BINARY | MODE_PICKLE => {
|
||||
let bytes = match row.value {
|
||||
|
|
@ -645,7 +671,7 @@ fn fetch_row(directory: &Path, row: Row) -> Result<Option<StoredValue>, Error> {
|
|||
};
|
||||
value
|
||||
}
|
||||
_ => return Err(Error::InvalidEntry),
|
||||
_ => return Err(Error::InvalidEntry(None)),
|
||||
};
|
||||
Ok(Some(if row.mode == MODE_BINARY {
|
||||
StoredValue::Bytes(bytes)
|
||||
|
|
@ -663,24 +689,24 @@ fn fetch_row(directory: &Path, row: Row) -> Result<Option<StoredValue>, Error> {
|
|||
}
|
||||
Value::Blob(value) => value,
|
||||
Value::Text(value) => value.into_bytes(),
|
||||
_ => return Err(Error::InvalidEntry),
|
||||
_ => return Err(Error::InvalidEntry(None)),
|
||||
};
|
||||
Ok(Some(StoredValue::Text(
|
||||
String::from_utf8(bytes).map_err(|_| Error::InvalidEntry)?,
|
||||
String::from_utf8(bytes).map_err(Error::invalid_entry)?,
|
||||
)))
|
||||
}
|
||||
_ => Err(Error::InvalidEntry),
|
||||
_ => Err(Error::InvalidEntry(None)),
|
||||
}
|
||||
}
|
||||
|
||||
fn read_file(directory: &Path, filename: Option<&str>) -> Result<Option<Vec<u8>>, Error> {
|
||||
let Some(filename) = filename else {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
};
|
||||
match fs::read(directory.join(filename)) {
|
||||
Ok(value) => Ok(Some(value)),
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None),
|
||||
Err(_) => Err(Error::Unavailable),
|
||||
Err(error) => Err(Error::unavailable(error)),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -712,7 +738,7 @@ fn store_value(
|
|||
let bytes = value.into_bytes();
|
||||
let filename = write_file(directory, &bytes)?;
|
||||
Ok(StoredColumns {
|
||||
size: i64::try_from(bytes.len()).map_err(|_| Error::Unavailable)?,
|
||||
size: i64::try_from(bytes.len()).map_err(Error::unavailable)?,
|
||||
mode: MODE_TEXT,
|
||||
filename: Some(filename),
|
||||
value: None,
|
||||
|
|
@ -727,7 +753,7 @@ fn store_value(
|
|||
StoredValue::Bytes(value) => {
|
||||
let filename = write_file(directory, &value)?;
|
||||
Ok(StoredColumns {
|
||||
size: i64::try_from(value.len()).map_err(|_| Error::Unavailable)?,
|
||||
size: i64::try_from(value.len()).map_err(Error::unavailable)?,
|
||||
mode: MODE_BINARY,
|
||||
filename: Some(filename),
|
||||
value: None,
|
||||
|
|
@ -742,7 +768,7 @@ fn store_value(
|
|||
StoredValue::Pickle(value) => {
|
||||
let filename = write_file(directory, &value)?;
|
||||
Ok(StoredColumns {
|
||||
size: i64::try_from(value.len()).map_err(|_| Error::Unavailable)?,
|
||||
size: i64::try_from(value.len()).map_err(Error::unavailable)?,
|
||||
mode: MODE_PICKLE,
|
||||
filename: Some(filename),
|
||||
value: None,
|
||||
|
|
@ -761,14 +787,14 @@ fn write_file(directory: &Path, bytes: &[u8]) -> Result<String, Error> {
|
|||
let filename = format!("{}/{}/{}.val", &hex[..2], &hex[2..4], &hex[4..]);
|
||||
let path = directory.join(&filename);
|
||||
if let Some(parent) = path.parent() {
|
||||
fs::create_dir_all(parent).map_err(|_| Error::Unavailable)?;
|
||||
fs::create_dir_all(parent).map_err(Error::unavailable)?;
|
||||
}
|
||||
let mut file = OpenOptions::new()
|
||||
.write(true)
|
||||
.create_new(true)
|
||||
.open(path)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
file.write_all(bytes).map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
file.write_all(bytes).map_err(Error::unavailable)?;
|
||||
Ok(filename)
|
||||
}
|
||||
|
||||
|
|
@ -789,12 +815,12 @@ fn transactional<T>(
|
|||
) -> Result<T, Error> {
|
||||
connection
|
||||
.execute_batch("BEGIN IMMEDIATE")
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
match operation(connection) {
|
||||
Ok(value) => {
|
||||
connection
|
||||
.execute_batch("COMMIT")
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
Ok(value)
|
||||
}
|
||||
Err(error) => {
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ fn python_read_cases(#[case] row: StoredValue, #[case] expected: ReadExpectation
|
|||
match expected {
|
||||
ReadExpectation::Bytes(expected) => assert_eq!(result.unwrap().unwrap(), expected),
|
||||
ReadExpectation::Miss => assert_eq!(result.unwrap(), None),
|
||||
ReadExpectation::Invalid => assert!(matches!(result, Err(Error::InvalidEntry))),
|
||||
ReadExpectation::Invalid => assert!(matches!(result, Err(Error::InvalidEntry(_)))),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -13,9 +13,11 @@ litellm-auth-types.workspace = true
|
|||
litellm-cache.workspace = true
|
||||
percent-encoding.workspace = true
|
||||
reqwest.workspace = true
|
||||
thiserror.workspace = true
|
||||
tokio.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
litellm-error.workspace = true
|
||||
litellm-http = { workspace = true, features = ["test-support"] }
|
||||
litellm-cache-testing.workspace = true
|
||||
rstest.workspace = true
|
||||
|
|
|
|||
|
|
@ -25,6 +25,14 @@ pub fn key_prefix(gcs_path: Option<&str>) -> String {
|
|||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("GCS answered {0}")]
|
||||
struct UnexpectedStatus(reqwest::StatusCode);
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("GCS cache worker thread panicked")]
|
||||
struct WorkerPanicked;
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct GcsConfig {
|
||||
pub bucket_name: String,
|
||||
|
|
@ -114,9 +122,9 @@ impl<S: CacheCodec> GcsCache<S> {
|
|||
.body(payload)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(|error| Error::unavailable(litellm_http::Error::from(error)))?;
|
||||
if !response.status().is_success() {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(UnexpectedStatus(response.status())));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -135,18 +143,18 @@ impl<S: CacheCodec> GcsCache<S> {
|
|||
.header(reqwest::header::CONTENT_TYPE, "application/json")
|
||||
.send()
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(|error| Error::unavailable(litellm_http::Error::from(error)))?;
|
||||
if response.status() == reqwest::StatusCode::NOT_FOUND {
|
||||
return Ok(None);
|
||||
}
|
||||
if !response.status().is_success() {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(UnexpectedStatus(response.status())));
|
||||
}
|
||||
let body = response.bytes().await.map_err(|_| Error::Unavailable)?;
|
||||
self.codec
|
||||
.decode(&body)
|
||||
.map(Some)
|
||||
.map_err(|_| Error::InvalidEntry)
|
||||
let body = response
|
||||
.bytes()
|
||||
.await
|
||||
.map_err(|error| Error::unavailable(litellm_http::Error::from(error)))?;
|
||||
self.codec.decode(&body).map(Some)
|
||||
}
|
||||
|
||||
fn run_sync<T, F>(future: F) -> Result<T, Error>
|
||||
|
|
@ -158,7 +166,7 @@ impl<S: CacheCodec> GcsCache<S> {
|
|||
tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
.and_then(|runtime| runtime.block_on(future))
|
||||
};
|
||||
match tokio::runtime::Handle::try_current() {
|
||||
|
|
@ -169,7 +177,7 @@ impl<S: CacheCodec> GcsCache<S> {
|
|||
scope
|
||||
.spawn(|| run(future))
|
||||
.join()
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(|_| Error::unavailable(WorkerPanicked))
|
||||
.and_then(|result| result)
|
||||
}),
|
||||
Err(_) => run(future),
|
||||
|
|
@ -242,7 +250,7 @@ impl<S: CacheCodec> BatchCache for GcsCache<S> {
|
|||
match self.async_get_cache(&key, &context).await {
|
||||
Ok(Some(value)) => Ok(BatchEntry::Hit(value)),
|
||||
Ok(None) => Ok(BatchEntry::Miss),
|
||||
Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid),
|
||||
Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid),
|
||||
Err(error) => Err(error),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ impl TokenSource for GcpTokenSource {
|
|||
self.auth
|
||||
.access_token(&self.config, &|name| std::env::var(name).ok())
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -55,19 +55,27 @@ async fn set_writes_encoded_object_and_headers(#[future(awt)] server: MockServer
|
|||
Ok(Some(json!({"value": "entry"})))
|
||||
)]
|
||||
#[case::missing("missing", ResponseTemplate::new(404), Ok(None))]
|
||||
#[case::server_error("server-error", ResponseTemplate::new(500), Err(Error::Unavailable))]
|
||||
#[case::unauthorized("unauthorized", ResponseTemplate::new(401), Err(Error::Unavailable))]
|
||||
#[case::server_error(
|
||||
"server-error",
|
||||
ResponseTemplate::new(500),
|
||||
Err("cache is unavailable: GCS answered 500 Internal Server Error")
|
||||
)]
|
||||
#[case::unauthorized(
|
||||
"unauthorized",
|
||||
ResponseTemplate::new(401),
|
||||
Err("cache is unavailable: GCS answered 401 Unauthorized")
|
||||
)]
|
||||
#[case::invalid(
|
||||
"invalid",
|
||||
ResponseTemplate::new(200).set_body_string("not json"),
|
||||
Err(Error::InvalidEntry)
|
||||
Err("invalid cache entry: ")
|
||||
)]
|
||||
#[tokio::test]
|
||||
async fn get_maps_statuses_and_decode_failures(
|
||||
#[future(awt)] server: MockServer,
|
||||
#[case] key: &str,
|
||||
#[case] response: ResponseTemplate,
|
||||
#[case] expected: Result<Option<Value>, Error>,
|
||||
#[case] expected: Result<Option<Value>, &str>,
|
||||
) {
|
||||
Mock::given(method("GET"))
|
||||
.and(path(format!("/storage/v1/b/bucket/o/{key}")))
|
||||
|
|
@ -76,8 +84,21 @@ async fn get_maps_statuses_and_decode_failures(
|
|||
.mount(&server)
|
||||
.await;
|
||||
let cache = support::cache(&server, None);
|
||||
assert_eq!(cache.get_cache(key, &context()), expected);
|
||||
assert_eq!(cache.async_get_cache(key, &context()).await, expected);
|
||||
let outcome = |result: Result<Option<Value>, Error>| {
|
||||
result.map_err(|error| litellm_error::Report::new(&error).to_string())
|
||||
};
|
||||
let sync = outcome(cache.get_cache(key, &context()));
|
||||
let asynchronous = outcome(cache.async_get_cache(key, &context()).await);
|
||||
match expected {
|
||||
Ok(value) => {
|
||||
assert_eq!(sync, Ok(value.clone()));
|
||||
assert_eq!(asynchronous, Ok(value));
|
||||
}
|
||||
Err(prefix) => {
|
||||
assert!(sync.unwrap_err().starts_with(prefix));
|
||||
assert!(asynchronous.unwrap_err().starts_with(prefix));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
@ -257,7 +278,7 @@ struct FailingTokenSource;
|
|||
|
||||
impl TokenSource for FailingTokenSource {
|
||||
fn bearer_token(&self) -> Pin<Box<dyn Future<Output = Result<String, Error>> + Send + '_>> {
|
||||
Box::pin(async { Err(Error::Unavailable) })
|
||||
Box::pin(async { Err(Error::Unavailable(None)) })
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -267,14 +288,14 @@ async fn token_source_failure_skips_http(#[future(awt)] server: MockServer) {
|
|||
let cache = support::cache_with_token(&server, None, Arc::new(FailingTokenSource));
|
||||
assert_eq!(
|
||||
cache.get_cache("key", &context()).unwrap_err(),
|
||||
Error::Unavailable
|
||||
Error::Unavailable(None)
|
||||
);
|
||||
assert_eq!(
|
||||
cache
|
||||
.async_set_cache("key", json!(1), context())
|
||||
.await
|
||||
.unwrap_err(),
|
||||
Error::Unavailable
|
||||
Error::Unavailable(None)
|
||||
);
|
||||
assert_eq!(server.received_requests().await.unwrap().len(), 0);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ use std::{
|
|||
cmp::Reverse,
|
||||
collections::{BinaryHeap, HashMap, HashSet},
|
||||
hash::Hash,
|
||||
sync::{Arc, Mutex},
|
||||
sync::{Arc, Mutex, MutexGuard},
|
||||
time::{Duration, SystemTime, UNIX_EPOCH},
|
||||
};
|
||||
|
||||
|
|
@ -84,6 +84,10 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
}
|
||||
}
|
||||
|
||||
fn lock(&self) -> Result<MutexGuard<'_, CacheState<V>>, Error> {
|
||||
Ok(self.state.lock()?)
|
||||
}
|
||||
|
||||
pub fn set_cache(
|
||||
&self,
|
||||
key: impl Into<String>,
|
||||
|
|
@ -94,13 +98,13 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
return Ok(CacheWrite::Disabled);
|
||||
}
|
||||
let now = (self.now)();
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
self.store(&mut state, key.into(), value, ttl, now)
|
||||
}
|
||||
|
||||
pub fn get_cache(&self, key: &str) -> Result<Option<V>, Error> {
|
||||
let now = (self.now)();
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
if state
|
||||
.expirations
|
||||
.get(key)
|
||||
|
|
@ -124,7 +128,7 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
/// fits.
|
||||
pub fn evict_cache(&self) -> Result<(), Error> {
|
||||
let now = (self.now)();
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
Self::evict(&mut state, self.max_size_in_memory, now, None);
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -132,7 +136,7 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
/// `evict_element_if_expired`: `true` when `key` had expired and was removed.
|
||||
pub fn evict_element_if_expired(&self, key: &str) -> Result<bool, Error> {
|
||||
let now = (self.now)();
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
let expired = state
|
||||
.expirations
|
||||
.get(key)
|
||||
|
|
@ -153,12 +157,7 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
|
||||
/// The number of stored entries, expired ones included until they are evicted.
|
||||
pub fn len(&self) -> Result<usize, Error> {
|
||||
Ok(self
|
||||
.state
|
||||
.lock()
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.values
|
||||
.len())
|
||||
Ok(self.lock()?.values.len())
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> Result<bool, Error> {
|
||||
|
|
@ -167,12 +166,7 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
|
||||
/// Entries in the expiration heap, stale ones included; bounded by eviction.
|
||||
pub fn expiration_heap_len(&self) -> Result<usize, Error> {
|
||||
Ok(self
|
||||
.state
|
||||
.lock()
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.expiration_heap
|
||||
.len())
|
||||
Ok(self.lock()?.expiration_heap.len())
|
||||
}
|
||||
|
||||
pub fn max_size_in_memory(&self) -> usize {
|
||||
|
|
@ -184,13 +178,7 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
}
|
||||
|
||||
pub fn expires_at(&self, key: &str) -> Result<Option<Duration>, Error> {
|
||||
Ok(self
|
||||
.state
|
||||
.lock()
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.expirations
|
||||
.get(key)
|
||||
.copied())
|
||||
Ok(self.lock()?.expirations.get(key).copied())
|
||||
}
|
||||
|
||||
pub async fn async_get_ttl(&self, key: &str) -> Result<Option<Duration>, Error> {
|
||||
|
|
@ -198,7 +186,7 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
}
|
||||
|
||||
pub async fn async_get_oldest_n_keys(&self, count: usize) -> Result<Vec<String>, Error> {
|
||||
let state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let state = self.lock()?;
|
||||
let mut expirations = state
|
||||
.expirations
|
||||
.iter()
|
||||
|
|
@ -213,13 +201,13 @@ impl<V: Clone> InMemoryCache<V> {
|
|||
}
|
||||
|
||||
pub fn delete_cache(&self, key: &str) -> Result<(), Error> {
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
Self::remove(&mut state, key);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn flush_cache(&self) -> Result<(), Error> {
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
state.values.clear();
|
||||
state.expirations.clear();
|
||||
state.expiration_heap.clear();
|
||||
|
|
@ -316,7 +304,7 @@ where
|
|||
return Ok(candidate);
|
||||
}
|
||||
let now = (self.now)();
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
Self::evict(&mut state, self.max_size_in_memory, now, Some(key));
|
||||
let existing = state
|
||||
.values
|
||||
|
|
@ -351,7 +339,7 @@ impl CounterCache for InMemoryCache<f64> {
|
|||
return Ok(amount);
|
||||
}
|
||||
let now = (self.now)();
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
let value = Self::live(&mut state, key, now).unwrap_or_default() + amount;
|
||||
self.store(&mut state, key.into(), value, self.get_ttl(&context), now)?;
|
||||
Ok(value)
|
||||
|
|
@ -424,7 +412,7 @@ where
|
|||
return Ok(values);
|
||||
}
|
||||
let now = (self.now)();
|
||||
let mut state = self.state.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut state = self.lock()?;
|
||||
let mut stored = Self::live(&mut state, key, now).unwrap_or_default();
|
||||
stored.extend(values.iter().cloned());
|
||||
self.store(&mut state, key.into(), stored, ttl, now)?;
|
||||
|
|
|
|||
|
|
@ -52,7 +52,7 @@ fn measured(capacity: usize) -> InMemoryCache<String> {
|
|||
Some(4),
|
||||
Some(Arc::new(|value: &String| {
|
||||
if value.is_empty() {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
Ok(value.len())
|
||||
})),
|
||||
|
|
@ -275,7 +275,7 @@ fn evict_element_if_expired_reports_removal(clock: Clock) {
|
|||
#[case::fits("ok", Ok(true))]
|
||||
#[case::at_limit("four", Ok(true))]
|
||||
#[case::too_large("oversized", Ok(false))]
|
||||
#[case::measure_error("", Err(Error::InvalidEntry))]
|
||||
#[case::measure_error("", Err(Error::InvalidEntry(None)))]
|
||||
fn check_value_size_applies_the_entry_limit(
|
||||
#[case] value: &str,
|
||||
#[case] expected: Result<bool, Error>,
|
||||
|
|
@ -310,7 +310,7 @@ fn disabled_size_limited_and_validated_writes_are_observable() {
|
|||
assert_eq!(cache.get_cache("small").unwrap(), Some("ok".into()));
|
||||
assert_eq!(
|
||||
cache.set_cache("invalid", String::new(), None),
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
assert_eq!(cache.get_cache("invalid").unwrap(), None);
|
||||
cache.delete_cache("small").unwrap();
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ tokio.workspace = true
|
|||
uuid.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
litellm-error.workspace = true
|
||||
litellm-http = { workspace = true, features = ["test-support"] }
|
||||
futures-executor = "0.3"
|
||||
litellm-cache-testing.workspace = true
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ impl<E: Embedder, C: CacheCodec> QdrantSemanticCache<E, C> {
|
|||
let exists = client
|
||||
.collection_exists(config.collection_name.clone())
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
if !exists {
|
||||
client
|
||||
.create_collection(
|
||||
|
|
@ -48,7 +48,7 @@ impl<E: Embedder, C: CacheCodec> QdrantSemanticCache<E, C> {
|
|||
.quantization_config(quantization(&config.quantization)),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
let _ = client
|
||||
.create_field_index(CreateFieldIndexCollectionBuilder::new(
|
||||
|
|
@ -99,13 +99,13 @@ impl<E: Embedder, C: CacheCodec> QdrantSemanticCache<E, C> {
|
|||
.async_embed(&prompt, context.metadata.as_ref())
|
||||
.await?;
|
||||
let response =
|
||||
String::from_utf8(self.codec.encode(&value)?).map_err(|_| Error::InvalidEntry)?;
|
||||
String::from_utf8(self.codec.encode(&value)?).map_err(Error::invalid_entry)?;
|
||||
let payload = Payload::try_from(json!({
|
||||
"litellm_cache_key": key,
|
||||
"text": prompt,
|
||||
"response": response,
|
||||
}))
|
||||
.map_err(|_| Error::InvalidEntry)?;
|
||||
.map_err(Error::invalid_entry)?;
|
||||
self.client
|
||||
.upsert_points(
|
||||
UpsertPointsBuilder::new(
|
||||
|
|
@ -119,7 +119,7 @@ impl<E: Embedder, C: CacheCodec> QdrantSemanticCache<E, C> {
|
|||
.wait(true),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
|
@ -152,7 +152,7 @@ impl<E: Embedder, C: CacheCodec> QdrantSemanticCache<E, C> {
|
|||
),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let Some(point) = result.result.into_iter().next() else {
|
||||
return Ok(SemanticLookup::miss(Some(0.0)));
|
||||
};
|
||||
|
|
@ -170,7 +170,7 @@ impl<E: Embedder, C: CacheCodec> QdrantSemanticCache<E, C> {
|
|||
let response = payload
|
||||
.get("response")
|
||||
.and_then(Value::as_str)
|
||||
.ok_or(Error::InvalidEntry)?;
|
||||
.ok_or(Error::InvalidEntry(None))?;
|
||||
Ok(SemanticLookup {
|
||||
value: Some(self.codec.decode(response.as_bytes())?),
|
||||
similarity: Some(similarity),
|
||||
|
|
|
|||
|
|
@ -55,10 +55,9 @@ impl Embedder for OpenAiEmbedder {
|
|||
}
|
||||
.send()
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.error_for_status()
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
let body: Value = response.json().await.map_err(|_| Error::Unavailable)?;
|
||||
.and_then(reqwest::Response::error_for_status)
|
||||
.map_err(http_error)?;
|
||||
let body: Value = response.json().await.map_err(http_error)?;
|
||||
body.get("data")
|
||||
.and_then(Value::as_array)
|
||||
.and_then(|data| data.first())
|
||||
|
|
@ -70,6 +69,14 @@ impl Embedder for OpenAiEmbedder {
|
|||
.map(|value| value.as_f64().map(|value| value as f32))
|
||||
.collect::<Option<Vec<_>>>()
|
||||
})
|
||||
.ok_or(Error::Unavailable)
|
||||
.ok_or(Error::unavailable(MissingEmbedding))
|
||||
}
|
||||
}
|
||||
|
||||
fn http_error(error: reqwest::Error) -> Error {
|
||||
Error::unavailable(litellm_http::Error::from(error))
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("embeddings response has no embedding")]
|
||||
struct MissingEmbedding;
|
||||
|
|
|
|||
|
|
@ -134,13 +134,19 @@ async fn posts_embeddings_request_and_parses_vector() {
|
|||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::error_status("500 Internal Server Error", "{}", 0, None, Err(Error::Unavailable))]
|
||||
#[case::error_status(
|
||||
"500 Internal Server Error",
|
||||
"{}",
|
||||
0,
|
||||
None,
|
||||
Err(Error::Unavailable(None))
|
||||
)]
|
||||
#[case::timed_out(
|
||||
"200 OK",
|
||||
r#"{"data":[{"embedding":[0.1,0.2]}]}"#,
|
||||
500,
|
||||
Some(Duration::from_millis(200)),
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(None))
|
||||
)]
|
||||
#[case::within_timeout(
|
||||
"200 OK",
|
||||
|
|
@ -149,7 +155,7 @@ async fn posts_embeddings_request_and_parses_vector() {
|
|||
Some(Duration::from_secs(1)),
|
||||
Ok(vec![0.1, 0.2])
|
||||
)]
|
||||
#[case::missing_embedding("200 OK", r#"{"data":[]}"#, 0, None, Err(Error::Unavailable))]
|
||||
#[case::missing_embedding("200 OK", r#"{"data":[]}"#, 0, None, Err(Error::Unavailable(None)))]
|
||||
#[tokio::test]
|
||||
async fn status_timeout_and_body_errors_are_unavailable(
|
||||
#[case] status: &str,
|
||||
|
|
@ -164,7 +170,14 @@ async fn status_timeout_and_body_errors_are_unavailable(
|
|||
litellm_http::Client::for_test(),
|
||||
config(server.base_url(), timeout),
|
||||
);
|
||||
assert_eq!(embedder.async_embed("hello", None).await, expected);
|
||||
let result = embedder.async_embed("hello", None).await;
|
||||
if let Err(error) = &result {
|
||||
let report = litellm_error::Report::new(error).to_string();
|
||||
assert!(!report.contains(&server.base_url()), "{report}");
|
||||
}
|
||||
let kind =
|
||||
|result: Result<Vec<f32>, Error>| result.map_err(|error| std::mem::discriminant(&error));
|
||||
assert_eq!(kind(result), kind(expected));
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
|
|||
|
|
@ -53,7 +53,10 @@ impl Embedder for FixedEmbedder {
|
|||
.lock()
|
||||
.unwrap()
|
||||
.push((input.to_owned(), metadata.cloned()));
|
||||
self.vectors.get(input).cloned().ok_or(Error::Unavailable)
|
||||
self.vectors
|
||||
.get(input)
|
||||
.cloned()
|
||||
.ok_or(Error::Unavailable(None))
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -396,9 +399,9 @@ async fn lookup_reports_python_semantic_similarity(
|
|||
|
||||
#[rstest]
|
||||
#[case::codec_decodes_the_payload(Some(json!("{\"a\":1}")), Ok(Some(json!({"a": 1}))))]
|
||||
#[case::undecodable_response(Some(json!("not json")), Err(Error::InvalidEntry))]
|
||||
#[case::non_string_response(Some(json!(1)), Err(Error::InvalidEntry))]
|
||||
#[case::missing_response(None, Err(Error::InvalidEntry))]
|
||||
#[case::undecodable_response(Some(json!("not json")), Err(Error::InvalidEntry(None)))]
|
||||
#[case::non_string_response(Some(json!(1)), Err(Error::InvalidEntry(None)))]
|
||||
#[case::missing_response(None, Err(Error::InvalidEntry(None)))]
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn stored_responses_go_through_the_codec(
|
||||
#[case] response: Option<JsonValue>,
|
||||
|
|
@ -419,9 +422,12 @@ async fn stored_responses_go_through_the_codec(
|
|||
.into(),
|
||||
});
|
||||
|
||||
let kind = |result: Result<Option<JsonValue>, Error>| {
|
||||
result.map_err(|error| std::mem::discriminant(&error))
|
||||
};
|
||||
assert_eq!(
|
||||
cache.async_get_cache("key", &context("hello")).await,
|
||||
expected
|
||||
kind(cache.async_get_cache("key", &context("hello")).await),
|
||||
kind(expected)
|
||||
);
|
||||
server.stop();
|
||||
}
|
||||
|
|
@ -434,7 +440,7 @@ async fn embedding_failures_propagate() {
|
|||
|
||||
assert_eq!(
|
||||
cache.async_get_cache("key", &context("unknown")).await,
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(None))
|
||||
);
|
||||
server.stop();
|
||||
}
|
||||
|
|
@ -493,10 +499,10 @@ async fn stopped_qdrant_server_maps_to_unavailable() {
|
|||
let cache = connect(&server, [("hello", vec![1.0, 0.0])]).await;
|
||||
server.stop();
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
assert_eq!(
|
||||
assert!(matches!(
|
||||
cache.async_get_cache("key", &context("hello")).await,
|
||||
Err(Error::Unavailable)
|
||||
);
|
||||
Err(Error::Unavailable(Some(_)))
|
||||
));
|
||||
}
|
||||
|
||||
/// `_payload_matches_cache_key` compares `str(cached_key) == str(key)`, so a point whose stored
|
||||
|
|
|
|||
|
|
@ -63,13 +63,13 @@ impl Inner {
|
|||
.arg(CACHE_KEY_FIELD)
|
||||
.arg(tag)
|
||||
.query::<()>(connection)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
if let Some(ttl) = ttl {
|
||||
redis::cmd("EXPIRE")
|
||||
.arg(&hash_key)
|
||||
.arg(ttl_seconds(ttl))
|
||||
.query::<()>(connection)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -111,7 +111,7 @@ impl Inner {
|
|||
.arg("vector")
|
||||
.arg(vector_buffer(vector))
|
||||
.query::<redis::Value>(connection)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let Some(fields) = first_document(&result) else {
|
||||
return Ok(SemanticLookup::miss(Some(0.0)));
|
||||
};
|
||||
|
|
|
|||
|
|
@ -40,10 +40,10 @@ impl Index {
|
|||
Some(false) => self.isolated(connection, dims)?,
|
||||
None => match create_index(connection, &self.name, dims) {
|
||||
Ok(()) => self.name.clone(),
|
||||
Err(_) => match index_compatible(connection, &self.name, dims)? {
|
||||
Err(error) => match index_compatible(connection, &self.name, dims)? {
|
||||
Some(true) => self.name.clone(),
|
||||
Some(false) => self.isolated(connection, dims)?,
|
||||
None => return Err(Error::Unavailable),
|
||||
None => return Err(error),
|
||||
},
|
||||
},
|
||||
};
|
||||
|
|
@ -59,7 +59,7 @@ impl Index {
|
|||
redis::cmd("FT.DROPINDEX")
|
||||
.arg(&name)
|
||||
.query::<()>(connection)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
create_index(connection, &name, dims)?;
|
||||
Ok(name)
|
||||
}
|
||||
|
|
@ -109,7 +109,7 @@ fn create_index(connection: &mut ConnectionRef<'_>, name: &str, dims: usize) ->
|
|||
.arg("SEPARATOR")
|
||||
.arg(",")
|
||||
.query::<()>(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
|
||||
fn index_compatible(
|
||||
|
|
@ -123,14 +123,18 @@ fn index_compatible(
|
|||
{
|
||||
Ok(info) => info,
|
||||
Err(error) if unknown_index(&error) => return Ok(None),
|
||||
Err(_) => return Err(Error::Unavailable),
|
||||
Err(error) => return Err(Error::unavailable(error)),
|
||||
};
|
||||
Ok(Some(schema_compatible(&info, dims)))
|
||||
}
|
||||
|
||||
fn unknown_index(error: &redis::RedisError) -> bool {
|
||||
let message = error.to_string().to_lowercase();
|
||||
message.contains("unknown") && message.contains("index")
|
||||
error
|
||||
.code()
|
||||
.is_some_and(|code| code.eq_ignore_ascii_case("unknown"))
|
||||
&& error
|
||||
.detail()
|
||||
.is_some_and(|detail| detail.to_ascii_lowercase().starts_with("index"))
|
||||
}
|
||||
|
||||
struct Attribute {
|
||||
|
|
|
|||
|
|
@ -79,7 +79,7 @@ fn s(value: &str) -> redis::Value {
|
|||
}
|
||||
|
||||
fn unknown_index_error() -> redis::RedisError {
|
||||
redis::RedisError::from((redis::ErrorKind::Extension, "Unknown index name"))
|
||||
support::server_error("Unknown index name")
|
||||
}
|
||||
|
||||
fn info_missing(index: &str) -> MockCmd {
|
||||
|
|
@ -320,7 +320,7 @@ fn store_writes_the_redisvl_hash(
|
|||
#[case::below_distance_threshold("key1", "0.05", None, Ok(Some(entry())))]
|
||||
#[case::above_distance_threshold("key1", "0.5", None, Ok(None))]
|
||||
#[case::other_cache_key("other", "0.05", None, Ok(None))]
|
||||
#[case::malformed_response("key1", "0.05", Some(b"not json!".as_slice()), Err(Error::InvalidEntry))]
|
||||
#[case::malformed_response("key1", "0.05", Some(b"not json!".as_slice()), Err(Error::InvalidEntry(None)))]
|
||||
fn lookup_applies_threshold_scope_and_codec(
|
||||
#[case] stored_tag: &str,
|
||||
#[case] distance: &str,
|
||||
|
|
@ -343,7 +343,10 @@ fn lookup_applies_threshold_scope_and_codec(
|
|||
FakeEmbedder::new(&[]),
|
||||
);
|
||||
|
||||
assert_eq!(cache.get_cache("key1", &context), expected);
|
||||
let kind = |result: Result<Option<Value>, Error>| {
|
||||
result.map_err(|error| std::mem::discriminant(&error))
|
||||
};
|
||||
assert_eq!(kind(cache.get_cache("key1", &context)), kind(expected));
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
@ -451,10 +454,7 @@ fn create_index_race_rechecks_schema_and_stores(entry: Value, context: SemanticC
|
|||
info_missing(INDEX),
|
||||
MockCmd::new(
|
||||
create_index_command(INDEX, 3),
|
||||
Err::<&str, _>(redis::RedisError::from((
|
||||
redis::ErrorKind::Extension,
|
||||
"Index already exists",
|
||||
))),
|
||||
Err::<&str, _>(support::server_error("Index already exists")),
|
||||
),
|
||||
info(INDEX, compatible_info(3)),
|
||||
hset(INDEX, PROMPT, "key1", &VECTOR, &entry),
|
||||
|
|
@ -673,10 +673,10 @@ fn shared_base_index_across_dimensions_replaces_the_isolated_index(entry: Value)
|
|||
.set_cache("key1", entry.clone(), &context())
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
worker_b.get_cache("key1", &context()).unwrap_err(),
|
||||
Error::Unavailable
|
||||
);
|
||||
assert!(matches!(
|
||||
worker_b.get_cache("key1", &context()),
|
||||
Err(Error::Unavailable(Some(_)))
|
||||
));
|
||||
}
|
||||
|
||||
#[fixture]
|
||||
|
|
@ -732,10 +732,10 @@ fn live_shared_index_is_replaced_across_dimensions(redis_stack_url: Option<Strin
|
|||
.set_cache("key1", entry.clone(), &context())
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
worker_b.get_cache("key1", &context()).unwrap_err(),
|
||||
Error::Unavailable
|
||||
);
|
||||
assert!(matches!(
|
||||
worker_b.get_cache("key1", &context()),
|
||||
Err(Error::Unavailable(Some(_)))
|
||||
));
|
||||
|
||||
let mut connection = redis::Client::open(url).unwrap().get_connection().unwrap();
|
||||
for index in [&base, &isolated] {
|
||||
|
|
|
|||
|
|
@ -74,7 +74,7 @@ impl FakeSearch {
|
|||
"FT.CREATE" => {
|
||||
let name = text(1);
|
||||
if state.indexes.contains_key(&name) {
|
||||
return Err(error("Index already exists"));
|
||||
return Err(server_error("Index already exists"));
|
||||
}
|
||||
let position = |token: &str| args.iter().position(|arg| arg == token.as_bytes());
|
||||
let prefix = args[position("PREFIX").unwrap() + 2].clone();
|
||||
|
|
@ -94,7 +94,7 @@ impl FakeSearch {
|
|||
let index = state
|
||||
.indexes
|
||||
.get(&text(1))
|
||||
.ok_or_else(|| error("Unknown index name"))?;
|
||||
.ok_or_else(|| server_error("Unknown index name"))?;
|
||||
Ok(index_info(index))
|
||||
}
|
||||
"FT.DROPINDEX" => {
|
||||
|
|
@ -118,7 +118,7 @@ impl FakeSearch {
|
|||
let index = state
|
||||
.indexes
|
||||
.get(&text(1))
|
||||
.ok_or_else(|| error("no such index"))?;
|
||||
.ok_or_else(|| server_error("no such index"))?;
|
||||
let query = text(2);
|
||||
let tag = query_tag(&query);
|
||||
let params = args.iter().position(|arg| arg == b"PARAMS").unwrap();
|
||||
|
|
@ -155,7 +155,7 @@ impl FakeSearch {
|
|||
]))
|
||||
}
|
||||
"PING" => Ok(redis::Value::SimpleString("PONG".into())),
|
||||
_ => Err(error("unsupported command")),
|
||||
_ => Err(server_error("unsupported command")),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -192,8 +192,11 @@ impl redis::ConnectionLike for FakeSearch {
|
|||
}
|
||||
}
|
||||
|
||||
fn error(message: &'static str) -> redis::RedisError {
|
||||
redis::RedisError::from((redis::ErrorKind::Extension, message))
|
||||
pub fn server_error(reply: &str) -> redis::RedisError {
|
||||
redis::parse_redis_value(format!("-{reply}\r\n").as_bytes())
|
||||
.expect("error reply")
|
||||
.extract_error()
|
||||
.expect_err("server error")
|
||||
}
|
||||
|
||||
fn bulk(bytes: &[u8]) -> redis::Value {
|
||||
|
|
|
|||
|
|
@ -7,8 +7,10 @@ repository.workspace = true
|
|||
|
||||
[dependencies]
|
||||
litellm-cache.workspace = true
|
||||
litellm-error.workspace = true
|
||||
redis = { workspace = true, features = ["cluster", "tls-rustls"] }
|
||||
r2d2 = "0.8.10"
|
||||
thiserror.workspace = true
|
||||
tokio.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
|
|
|
|||
|
|
@ -141,7 +141,7 @@ where
|
|||
redis::Value::Nil => Ok(None),
|
||||
redis::Value::BulkString(bytes) => self.codec.decode(&bytes).map(Some),
|
||||
redis::Value::SimpleString(text) => self.codec.decode(text.as_bytes()).map(Some),
|
||||
_ => Err(Error::InvalidEntry),
|
||||
_ => Err(Error::InvalidEntry(None)),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -152,7 +152,7 @@ where
|
|||
match self.decode_response(value) {
|
||||
Ok(Some(value)) => Ok(BatchEntry::Hit(value)),
|
||||
Ok(None) => Ok(BatchEntry::Miss),
|
||||
Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid),
|
||||
Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid),
|
||||
Err(error) => Err(error),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ fn stored_bytes(value: redis::Value) -> Result<Option<Vec<u8>>, Error> {
|
|||
redis::Value::Nil => Ok(None),
|
||||
redis::Value::BulkString(bytes) => Ok(Some(bytes)),
|
||||
redis::Value::SimpleString(text) => Ok(Some(text.into_bytes())),
|
||||
_ => Err(Error::InvalidEntry),
|
||||
_ => Err(Error::InvalidEntry(None)),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -37,13 +37,13 @@ where
|
|||
{
|
||||
let payload = codec.encode(&candidate)?;
|
||||
if payload.is_empty() {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
for _ in 0..CLAIM_ATTEMPTS {
|
||||
let current = stored_bytes(
|
||||
connection
|
||||
.get::<_, redis::Value>(key)
|
||||
.map_err(|_| Error::Unavailable)?,
|
||||
.map_err(Error::unavailable)?,
|
||||
)?
|
||||
.filter(|bytes| !bytes.is_empty());
|
||||
let existing = current
|
||||
|
|
@ -63,14 +63,18 @@ where
|
|||
.arg(write)
|
||||
.arg(u8::from(refresh))
|
||||
.query::<bool>(connection)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
if applied {
|
||||
return Ok(existing.unwrap_or(candidate));
|
||||
}
|
||||
}
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::unavailable(ClaimContended))
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("claim kept losing the race after {CLAIM_ATTEMPTS} attempts")]
|
||||
struct ClaimContended;
|
||||
|
||||
impl<S, C> ClaimCache for RedisCache<S, C>
|
||||
where
|
||||
S: CacheCodec + Clone + 'static,
|
||||
|
|
|
|||
|
|
@ -36,19 +36,19 @@ where
|
|||
) -> Result<T, Error> {
|
||||
match self {
|
||||
Self::Pool(pool) => {
|
||||
let mut pooled = pool.get().map_err(|_| Error::Unavailable)?;
|
||||
let mut pooled = pool.get().map_err(Error::unavailable)?;
|
||||
let result = operation(&mut ConnectionRef::Node(&mut pooled.connection));
|
||||
pooled.failed = matches!(result, Err(Error::Unavailable));
|
||||
pooled.failed = matches!(result, Err(Error::Unavailable(_)));
|
||||
result
|
||||
}
|
||||
Self::Cluster(pool) => {
|
||||
let mut pooled = pool.get().map_err(|_| Error::Unavailable)?;
|
||||
let mut pooled = pool.get().map_err(Error::unavailable)?;
|
||||
let result = operation(&mut ConnectionRef::Cluster(&mut pooled.connection));
|
||||
pooled.failed = matches!(result, Err(Error::Unavailable));
|
||||
pooled.failed = matches!(result, Err(Error::Unavailable(_)));
|
||||
result
|
||||
}
|
||||
Self::Fixed(connection) => {
|
||||
let mut connection = connection.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut connection = connection.lock()?;
|
||||
operation(&mut ConnectionRef::Node(&mut *connection))
|
||||
}
|
||||
}
|
||||
|
|
@ -61,7 +61,7 @@ where
|
|||
{
|
||||
tokio::task::spawn_blocking(move || connections.execute(operation))
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
}
|
||||
|
||||
pub fn fixed(connection: C) -> Self {
|
||||
|
|
@ -95,7 +95,7 @@ fn pool<M: r2d2::ManageConnection>(manager: M) -> Result<r2d2::Pool<M>, Error> {
|
|||
.connection_timeout(REDIS_TIMEOUT)
|
||||
.test_on_check_out(false)
|
||||
.build(manager)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
|
||||
fn close_idle<M, T>(pool: &r2d2::Pool<M>)
|
||||
|
|
@ -123,7 +123,7 @@ impl ConnectionManager {
|
|||
fn open(url: &str) -> Result<Self, Error> {
|
||||
redis::Client::open(url)
|
||||
.map(Self)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -151,14 +151,26 @@ impl r2d2::ManageConnection for ConnectionManager {
|
|||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
enum TopologyFault {
|
||||
#[error("Redis Cluster needs at least one startup node")]
|
||||
NoStartupNodes,
|
||||
#[error("Redis Cluster nodes need a TCP address")]
|
||||
NonTcpAddress,
|
||||
#[error("Redis Cluster answered a fan-out command without a per-node map")]
|
||||
FanOutReply,
|
||||
#[error("scan target does not match the connection topology")]
|
||||
ScanTarget,
|
||||
}
|
||||
|
||||
pub(crate) struct ClusterConnectionManager(ClusterClient);
|
||||
|
||||
impl ClusterConnectionManager {
|
||||
fn open(url: &str, startup_nodes: &[RedisNode]) -> Result<Self, Error> {
|
||||
if startup_nodes.is_empty() {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(TopologyFault::NoStartupNodes));
|
||||
}
|
||||
let info = url.into_connection_info().map_err(|_| Error::Unavailable)?;
|
||||
let info = url.into_connection_info().map_err(Error::unavailable)?;
|
||||
let nodes = startup_nodes
|
||||
.iter()
|
||||
.map(|node| node_info(&info, node))
|
||||
|
|
@ -168,7 +180,7 @@ impl ClusterConnectionManager {
|
|||
.response_timeout(REDIS_TIMEOUT)
|
||||
.build()
|
||||
.map(Self)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -185,7 +197,7 @@ fn node_info(info: &ConnectionInfo, node: &RedisNode) -> Result<ConnectionInfo,
|
|||
insecure: *insecure,
|
||||
tls_params: tls_params.clone(),
|
||||
},
|
||||
_ => return Err(Error::Unavailable),
|
||||
_ => return Err(Error::unavailable(TopologyFault::NonTcpAddress)),
|
||||
};
|
||||
Ok(info.clone().set_addr(addr))
|
||||
}
|
||||
|
|
@ -298,7 +310,7 @@ impl ConnectionRef<'_> {
|
|||
.and_then(|replies| pipeline.query(&mut Replies(Some(replies))))
|
||||
}
|
||||
}
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
|
||||
pub(crate) fn scan(
|
||||
|
|
@ -311,7 +323,7 @@ impl ConnectionRef<'_> {
|
|||
Self::Node(connection) => {
|
||||
let page = scan_command(0, pattern, count)
|
||||
.query::<ScanPage>(*connection)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
vec![(None, page)]
|
||||
}
|
||||
Self::Cluster(connection) => connection
|
||||
|
|
@ -322,7 +334,7 @@ impl ConnectionRef<'_> {
|
|||
Some(ResponsePolicy::Special),
|
||||
)),
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
.and_then(primary_pages)?
|
||||
.into_iter()
|
||||
.map(|(node, page)| (Some(node), page))
|
||||
|
|
@ -362,7 +374,7 @@ impl ConnectionRef<'_> {
|
|||
|
||||
pub(crate) fn node_text(&mut self, command: &redis::Cmd) -> Result<String, Error> {
|
||||
match self {
|
||||
Self::Node(connection) => command.query(*connection).map_err(|_| Error::Unavailable),
|
||||
Self::Node(connection) => command.query(*connection).map_err(Error::unavailable),
|
||||
Self::Cluster(connection) => {
|
||||
let value = connection
|
||||
.route_command(
|
||||
|
|
@ -372,18 +384,16 @@ impl ConnectionRef<'_> {
|
|||
Some(ResponsePolicy::Special),
|
||||
)),
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let redis::Value::Map(entries) = value else {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(TopologyFault::FanOutReply));
|
||||
};
|
||||
let mut replies = entries
|
||||
.into_iter()
|
||||
.map(|(node, reply)| {
|
||||
Ok((
|
||||
redis::from_redis_value::<String>(node)
|
||||
.map_err(|_| Error::Unavailable)?,
|
||||
redis::from_redis_value::<String>(reply)
|
||||
.map_err(|_| Error::Unavailable)?,
|
||||
redis::from_redis_value::<String>(node).map_err(Error::unavailable)?,
|
||||
redis::from_redis_value::<String>(reply).map_err(Error::unavailable)?,
|
||||
))
|
||||
})
|
||||
.collect::<Result<Vec<(String, String)>, Error>>()?;
|
||||
|
|
@ -400,7 +410,7 @@ impl ConnectionRef<'_> {
|
|||
pub(crate) fn flushall(&mut self) -> Result<(), Error> {
|
||||
let command = redis::cmd("FLUSHALL");
|
||||
match self {
|
||||
Self::Node(connection) => command.query(*connection).map_err(|_| Error::Unavailable),
|
||||
Self::Node(connection) => command.query(*connection).map_err(Error::unavailable),
|
||||
Self::Cluster(connection) => connection
|
||||
.route_command(
|
||||
&command,
|
||||
|
|
@ -410,7 +420,7 @@ impl ConnectionRef<'_> {
|
|||
)),
|
||||
)
|
||||
.map(|_| ())
|
||||
.map_err(|_| Error::Unavailable),
|
||||
.map_err(Error::unavailable),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -424,7 +434,7 @@ impl ConnectionRef<'_> {
|
|||
let command = scan_command(cursor, pattern, count);
|
||||
match (self, node) {
|
||||
(Self::Node(connection), None) => {
|
||||
command.query(*connection).map_err(|_| Error::Unavailable)
|
||||
command.query(*connection).map_err(Error::unavailable)
|
||||
}
|
||||
(Self::Cluster(connection), Some(node)) => connection
|
||||
.route_command(
|
||||
|
|
@ -434,9 +444,9 @@ impl ConnectionRef<'_> {
|
|||
port: node.port(),
|
||||
}),
|
||||
)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.and_then(|value| redis::from_redis_value(value).map_err(|_| Error::Unavailable)),
|
||||
_ => Err(Error::Unavailable),
|
||||
.map_err(Error::unavailable)
|
||||
.and_then(|value| redis::from_redis_value(value).map_err(Error::unavailable)),
|
||||
_ => Err(Error::unavailable(TopologyFault::ScanTarget)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -445,14 +455,14 @@ type ScanPage = (u64, Vec<String>);
|
|||
|
||||
fn primary_pages(value: redis::Value) -> Result<Vec<(NodeAddress, ScanPage)>, Error> {
|
||||
let redis::Value::Map(entries) = value else {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(TopologyFault::FanOutReply));
|
||||
};
|
||||
entries
|
||||
.into_iter()
|
||||
.map(|(node, page)| {
|
||||
let node = redis::from_redis_value::<String>(node).map_err(|_| Error::Unavailable)?;
|
||||
let node = NodeAddress::try_from(node.as_str()).map_err(|_| Error::Unavailable)?;
|
||||
let page = redis::from_redis_value::<ScanPage>(page).map_err(|_| Error::Unavailable)?;
|
||||
let node = redis::from_redis_value::<String>(node).map_err(Error::unavailable)?;
|
||||
let node = NodeAddress::try_from(node.as_str()).map_err(Error::unavailable)?;
|
||||
let page = redis::from_redis_value::<ScanPage>(page).map_err(Error::unavailable)?;
|
||||
Ok((node, page))
|
||||
})
|
||||
.collect()
|
||||
|
|
|
|||
|
|
@ -100,7 +100,7 @@ fn increment(
|
|||
.arg(amount)
|
||||
.arg(ttl)
|
||||
.query(connection)
|
||||
.map_err(|_| Error::Unavailable);
|
||||
.map_err(Error::unavailable);
|
||||
}
|
||||
connection
|
||||
.query_pipeline(
|
||||
|
|
@ -140,7 +140,7 @@ where
|
|||
}
|
||||
|
||||
fn count(value: redis::Value) -> Result<Option<i64>, Error> {
|
||||
redis::from_redis_value(value).map_err(|_| Error::InvalidEntry)
|
||||
redis::from_redis_value(value).map_err(Error::invalid_entry)
|
||||
}
|
||||
|
||||
impl<S, C> BoundedCounterCache for RedisCache<S, C>
|
||||
|
|
@ -182,7 +182,7 @@ where
|
|||
.arg(value)
|
||||
.arg(ttl)
|
||||
.query(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
|
@ -201,5 +201,5 @@ fn increment_with_floor(
|
|||
.arg(amount)
|
||||
.arg(ttl)
|
||||
.query(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ where
|
|||
redis::cmd("TTL")
|
||||
.arg(key)
|
||||
.query::<i64>(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
.await?;
|
||||
Ok(u64::try_from(ttl).ok().map(Duration::from_secs))
|
||||
|
|
@ -36,7 +36,7 @@ where
|
|||
.arg(key)
|
||||
.arg(ttl)
|
||||
.query(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,11 +11,11 @@ where
|
|||
C: redis::ConnectionLike + Send + 'static,
|
||||
{
|
||||
fn sync_ping(&self) -> Result<bool, Error> {
|
||||
self.execute(|connection| connection.ping().map_err(|_| Error::Unavailable))
|
||||
self.execute(|connection| connection.ping().map_err(Error::unavailable))
|
||||
}
|
||||
|
||||
async fn ping(&self) -> Result<bool, Error> {
|
||||
self.run(|connection| connection.ping().map_err(|_| Error::Unavailable))
|
||||
self.run(|connection| connection.ping().map_err(Error::unavailable))
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
|
@ -33,9 +33,10 @@ where
|
|||
RedisTopology::Cluster { .. } => "Redis Cluster",
|
||||
};
|
||||
let ping = self
|
||||
.run(|connection| Ok(connection.ping().map_err(|error| error.to_string())))
|
||||
.run(|connection| Ok(connection.ping().map_err(Error::unavailable)))
|
||||
.await
|
||||
.unwrap_or_else(|error| Err(error.to_string()));
|
||||
.and_then(|ping| ping)
|
||||
.map_err(|error| litellm_error::Report::new(&error).to_string());
|
||||
Ok(match ping {
|
||||
Ok(true) => CacheConnectionResult {
|
||||
status: CacheConnectionStatus::Success,
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ where
|
|||
ttl: Option<Duration>,
|
||||
) -> Result<Self::SetResult, Error> {
|
||||
if values.is_empty() {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
let key = self.namespaced_key(key);
|
||||
let ttl = self.ttl_or_default(ttl);
|
||||
|
|
@ -58,7 +58,7 @@ where
|
|||
|
||||
async fn async_rpush(&self, key: &str, values: Vec<Self::QueueValue>) -> Result<usize, Error> {
|
||||
if values.is_empty() {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
let key = self.namespaced_key(key);
|
||||
self.run(move |connection| {
|
||||
|
|
@ -66,7 +66,7 @@ where
|
|||
.arg(key)
|
||||
.arg(values)
|
||||
.query(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
|
@ -78,7 +78,7 @@ where
|
|||
max_len: usize,
|
||||
) -> Result<usize, Error> {
|
||||
if values.is_empty() {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
let key = self.namespaced_key(key);
|
||||
let start = i64::try_from(max_len).map_or(i64::MIN, |max_len| -max_len);
|
||||
|
|
@ -108,7 +108,7 @@ where
|
|||
let mut pipeline = redis::pipe();
|
||||
for operation in operations {
|
||||
if operation.values.is_empty() {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
pipeline
|
||||
.cmd("RPUSH")
|
||||
|
|
@ -130,7 +130,7 @@ where
|
|||
.run(move |connection| {
|
||||
command
|
||||
.query::<redis::Value>(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
.await?;
|
||||
lpop_result(value, count.is_some())
|
||||
|
|
@ -182,7 +182,7 @@ fn redis_bytes(value: redis::Value) -> Result<Vec<u8>, Error> {
|
|||
match value {
|
||||
redis::Value::BulkString(bytes) => Ok(bytes),
|
||||
redis::Value::SimpleString(text) => Ok(text.into_bytes()),
|
||||
_ => Err(Error::InvalidEntry),
|
||||
_ => Err(Error::InvalidEntry(None)),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -221,6 +221,6 @@ fn lpop_result(value: redis::Value, multiple: bool) -> Result<RedisLpopResult, E
|
|||
.collect::<Result<Vec<_>, _>>()
|
||||
.map(RedisLpopResult::Values),
|
||||
value if !multiple => redis_bytes(value).map(RedisLpopResult::Value),
|
||||
_ => Err(Error::InvalidEntry),
|
||||
_ => Err(Error::InvalidEntry(None)),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -116,7 +116,7 @@ fn eval(
|
|||
.arg(keys)
|
||||
.arg(arguments)
|
||||
.query(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
|
||||
impl<S, C> ScriptCache for RedisCache<S, C>
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ where
|
|||
self.execute(|connection| {
|
||||
connection
|
||||
.set_ex::<_, _, ()>(key, payload, ttl)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -41,7 +41,7 @@ where
|
|||
let value = self.execute(|connection| {
|
||||
connection
|
||||
.get::<_, redis::Value>(key)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})?;
|
||||
self.decode_response(value)
|
||||
}
|
||||
|
|
@ -58,7 +58,7 @@ where
|
|||
self.run(move |connection| {
|
||||
connection
|
||||
.set_ex::<_, _, ()>(key, payload, ttl)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
|
@ -73,7 +73,7 @@ where
|
|||
.run(move |connection| {
|
||||
connection
|
||||
.get::<_, redis::Value>(key)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
})
|
||||
.await?;
|
||||
self.decode_response(value)
|
||||
|
|
@ -158,7 +158,7 @@ pub(crate) fn mget(
|
|||
redis::cmd("MGET")
|
||||
.arg(keys)
|
||||
.query(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
.map_err(Error::unavailable)
|
||||
}
|
||||
|
||||
impl<S, C> DeleteCache for RedisCache<S, C>
|
||||
|
|
@ -168,12 +168,12 @@ where
|
|||
{
|
||||
fn delete_cache(&self, key: &str) -> Result<(), Error> {
|
||||
let key = self.namespaced_key(key);
|
||||
self.execute(|connection| connection.del::<_, ()>(key).map_err(|_| Error::Unavailable))
|
||||
self.execute(|connection| connection.del::<_, ()>(key).map_err(Error::unavailable))
|
||||
}
|
||||
|
||||
async fn async_delete_cache(&self, key: &str) -> Result<(), Error> {
|
||||
let key = self.namespaced_key(key);
|
||||
self.run(move |connection| connection.del::<_, ()>(key).map_err(|_| Error::Unavailable))
|
||||
self.run(move |connection| connection.del::<_, ()>(key).map_err(Error::unavailable))
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
|
@ -188,7 +188,7 @@ where
|
|||
return Ok(0);
|
||||
}
|
||||
let keys = self.namespaced_keys(&keys);
|
||||
self.run(move |connection| connection.del(keys).map_err(|_| Error::Unavailable))
|
||||
self.run(move |connection| connection.del(keys).map_err(Error::unavailable))
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
|
@ -225,7 +225,7 @@ fn flush_matching(connection: &mut ConnectionRef<'_>, pattern: &str) -> Result<(
|
|||
if !keys.is_empty() {
|
||||
connection
|
||||
.del::<_, usize>(keys)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
}
|
||||
Ok(true)
|
||||
})
|
||||
|
|
|
|||
|
|
@ -178,11 +178,11 @@ async fn codec_errors_propagate_without_writing_partial_batches(context: ExactCa
|
|||
]);
|
||||
assert_eq!(
|
||||
cache.set_cache("invalid", 255, &context),
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
assert_eq!(
|
||||
cache.async_set_cache("invalid", 255, context.clone()).await,
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
assert_eq!(
|
||||
cache
|
||||
|
|
@ -191,7 +191,7 @@ async fn codec_errors_propagate_without_writing_partial_batches(context: ExactCa
|
|||
context.clone(),
|
||||
)
|
||||
.await,
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
assert_eq!(
|
||||
cache
|
||||
|
|
@ -200,15 +200,15 @@ async fn codec_errors_propagate_without_writing_partial_batches(context: ExactCa
|
|||
("invalid".into(), 255, None),
|
||||
])
|
||||
.await,
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
assert_eq!(
|
||||
cache.get_cache("invalid", &context),
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
assert_eq!(
|
||||
cache.async_get_cache("invalid", &context).await,
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -516,7 +516,7 @@ async fn sets_add_members_and_arm_the_default_ttl() {
|
|||
cache
|
||||
.async_set_cache_sadd("members", Vec::new(), None)
|
||||
.await,
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -547,7 +547,7 @@ async fn queues_push_and_pop_namespaced_lists() {
|
|||
);
|
||||
assert_eq!(
|
||||
cache.async_rpush("queue", Vec::new()).await,
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
assert_eq!(
|
||||
cache.async_lpop("queue", Some(2)).await.unwrap(),
|
||||
|
|
@ -673,15 +673,15 @@ async fn rpush_and_trim_raises_when_a_queued_command_fails() {
|
|||
)])
|
||||
.with_namespace(Some("ns".into()));
|
||||
|
||||
assert_eq!(
|
||||
assert!(matches!(
|
||||
cache
|
||||
.async_rpush_and_trim("buf", vec!["c".into(), "d".into()], 3)
|
||||
.await,
|
||||
Err(Error::Unavailable)
|
||||
);
|
||||
Err(Error::Unavailable(Some(_)))
|
||||
));
|
||||
assert_eq!(
|
||||
cache.async_rpush_and_trim("buf", Vec::new(), 3).await,
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -1122,24 +1122,30 @@ async fn test_connection_reports_success_with_the_python_message() {
|
|||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::unexpected_reply(Ok("NOPE"), "Redis ping returned False", false)]
|
||||
#[case::unexpected_reply(Ok("NOPE"), "Redis ping returned False", None)]
|
||||
#[case::connection_refused(
|
||||
Err(redis::RedisError::from((redis::ErrorKind::Io, "connection refused"))),
|
||||
"Redis connection failed:",
|
||||
true
|
||||
Some("connection refused")
|
||||
)]
|
||||
#[tokio::test]
|
||||
async fn test_connection_failures_use_the_python_result_contract(
|
||||
#[case] reply: redis::RedisResult<&'static str>,
|
||||
#[case] message: &str,
|
||||
#[case] has_error: bool,
|
||||
#[case] cause: Option<&str>,
|
||||
) {
|
||||
let cache = json_cache(vec![MockCmd::new(redis::cmd("PING"), reply)]);
|
||||
|
||||
let result = cache.test_connection().await.unwrap();
|
||||
assert_eq!(result.status, CacheConnectionStatus::Failed);
|
||||
assert!(result.message.starts_with(message), "{}", result.message);
|
||||
assert_eq!(result.error.is_some(), has_error);
|
||||
assert_eq!(result.error.is_some(), cause.is_some());
|
||||
assert!(cause.is_none_or(|cause| {
|
||||
result
|
||||
.error
|
||||
.as_deref()
|
||||
.is_some_and(|error| error.contains(cause))
|
||||
}));
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
|
|||
|
|
@ -66,7 +66,7 @@ fn constructor_rejects_unusable_cluster_configs(
|
|||
JsonCodec::new(),
|
||||
)
|
||||
.err();
|
||||
assert!(matches!(error, Some(Error::Unavailable)));
|
||||
assert!(matches!(error, Some(Error::Unavailable(_))));
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ impl CacheCodec for TaggedByteCodec {
|
|||
|
||||
fn encode(&self, value: &u8) -> Result<Vec<u8>, Error> {
|
||||
if *value > 127 {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
Ok(vec![self.0, *value])
|
||||
}
|
||||
|
|
@ -24,7 +24,7 @@ impl CacheCodec for TaggedByteCodec {
|
|||
fn decode(&self, bytes: &[u8]) -> Result<u8, Error> {
|
||||
match bytes {
|
||||
[tag, value] if *tag == self.0 => Ok(*value),
|
||||
_ => Err(Error::InvalidEntry),
|
||||
_ => Err(Error::InvalidEntry(None)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ impl WriteBuffer {
|
|||
now: Duration,
|
||||
) -> Result<(), Error> {
|
||||
let pending = {
|
||||
let mut entries = self.entries.lock().map_err(|_| Error::Unavailable)?;
|
||||
let mut entries = self.entries.lock()?;
|
||||
entries.push((request.clone(), response, now));
|
||||
(entries.len() >= self.flush_size).then(|| std::mem::take(&mut *entries))
|
||||
};
|
||||
|
|
@ -39,7 +39,7 @@ impl WriteBuffer {
|
|||
}
|
||||
|
||||
pub fn clear(&self) -> Result<(), Error> {
|
||||
self.entries.lock().map_err(|_| Error::Unavailable)?.clear();
|
||||
self.entries.lock()?.clear();
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,24 +14,24 @@ impl CacheCodec for ResponseCacheCodec {
|
|||
.timestamp
|
||||
.is_some_and(|timestamp| !timestamp.is_finite())
|
||||
{
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
// Python reads a `response` that is either a dict or a serialized string, so every
|
||||
// other shape is written serialized. A string on the wire is therefore always a
|
||||
// serialized response, which keeps string-valued responses unambiguous.
|
||||
if value.timestamp.is_none() || value.response.is_object() {
|
||||
return serde_json::to_vec(value).map_err(|_| Error::InvalidEntry);
|
||||
return serde_json::to_vec(value).map_err(Error::invalid_entry);
|
||||
}
|
||||
let response = serde_json::to_string(&value.response).map_err(|_| Error::InvalidEntry)?;
|
||||
let response = serde_json::to_string(&value.response).map_err(Error::invalid_entry)?;
|
||||
serde_json::to_vec(&CacheEntry {
|
||||
timestamp: value.timestamp,
|
||||
response: Value::String(response),
|
||||
})
|
||||
.map_err(|_| Error::InvalidEntry)
|
||||
.map_err(Error::invalid_entry)
|
||||
}
|
||||
|
||||
fn decode(&self, bytes: &[u8]) -> Result<CacheEntry, Error> {
|
||||
let text = std::str::from_utf8(bytes).map_err(|_| Error::InvalidEntry)?;
|
||||
let text = std::str::from_utf8(bytes).map_err(Error::invalid_entry)?;
|
||||
let value = decode_value(text)?;
|
||||
let Some(timestamp) = value.get("timestamp") else {
|
||||
return Ok(CacheEntry {
|
||||
|
|
@ -40,9 +40,9 @@ impl CacheCodec for ResponseCacheCodec {
|
|||
});
|
||||
};
|
||||
let Some(timestamp) = timestamp.as_f64().filter(|timestamp| timestamp.is_finite()) else {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
};
|
||||
let response = match value.get("response").ok_or(Error::InvalidEntry)? {
|
||||
let response = match value.get("response").ok_or(Error::InvalidEntry(None))? {
|
||||
Value::String(text) => decode_value(text)?,
|
||||
response => response.clone(),
|
||||
};
|
||||
|
|
@ -58,25 +58,25 @@ fn decode_value(text: &str) -> Result<Value, Error> {
|
|||
return Ok(value);
|
||||
}
|
||||
check_literal_depth(text)?;
|
||||
let literal: py_literal::Value = text.parse().map_err(|_| Error::InvalidEntry)?;
|
||||
let literal: py_literal::Value = text.parse().map_err(Error::invalid_entry)?;
|
||||
literal_value(literal, 0)
|
||||
}
|
||||
|
||||
fn literal_value(value: py_literal::Value, depth: usize) -> Result<Value, Error> {
|
||||
use py_literal::Value as Literal;
|
||||
if depth > 128 {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
match value {
|
||||
Literal::String(text) => Ok(Value::String(text)),
|
||||
Literal::Boolean(value) => Ok(Value::Bool(value)),
|
||||
Literal::None => Ok(Value::Null),
|
||||
Literal::Integer(value) => {
|
||||
serde_json::from_str(&value.to_string()).map_err(|_| Error::InvalidEntry)
|
||||
serde_json::from_str(&value.to_string()).map_err(Error::invalid_entry)
|
||||
}
|
||||
Literal::Float(value) => serde_json::Number::from_f64(value)
|
||||
.map(Value::Number)
|
||||
.ok_or(Error::InvalidEntry),
|
||||
.ok_or(Error::InvalidEntry(None)),
|
||||
Literal::List(values) | Literal::Tuple(values) => values
|
||||
.into_iter()
|
||||
.map(|value| literal_value(value, depth + 1))
|
||||
|
|
@ -86,13 +86,13 @@ fn literal_value(value: py_literal::Value, depth: usize) -> Result<Value, Error>
|
|||
.into_iter()
|
||||
.map(|(key, value)| {
|
||||
let Literal::String(key) = key else {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
};
|
||||
Ok((key, literal_value(value, depth + 1)?))
|
||||
})
|
||||
.collect::<Result<serde_json::Map<_, _>, _>>()
|
||||
.map(Value::Object),
|
||||
_ => Err(Error::InvalidEntry),
|
||||
_ => Err(Error::InvalidEntry(None)),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -118,7 +118,7 @@ fn check_literal_depth(text: &str) -> Result<(), Error> {
|
|||
'[' | '{' | '(' => {
|
||||
depth += 1;
|
||||
if depth > 128 {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
}
|
||||
']' | '}' | ')' => depth = depth.saturating_sub(1),
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ use std::{sync::Arc, time::Duration};
|
|||
|
||||
use litellm_cache::{
|
||||
BaseCache, BatchCache, BatchEntry, CacheConnectionResult, CacheContext, ConnectionCache, Error,
|
||||
FlushCache,
|
||||
FlushCache, ShortBatch,
|
||||
semantic::{SemanticCache, SemanticLookup},
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
|
@ -114,7 +114,7 @@ where
|
|||
.get_cache(&cache_key(&request.key), &request.context)
|
||||
{
|
||||
Ok(entry) => entry,
|
||||
Err(Error::InvalidEntry) => None,
|
||||
Err(Error::InvalidEntry(_)) => None,
|
||||
Err(error) => return Err(error),
|
||||
};
|
||||
Ok(Self::fresh_or_miss(entry, now, request.max_age))
|
||||
|
|
@ -134,7 +134,7 @@ where
|
|||
.await
|
||||
{
|
||||
Ok(entry) => entry,
|
||||
Err(Error::InvalidEntry) => None,
|
||||
Err(Error::InvalidEntry(_)) => None,
|
||||
Err(error) => return Err(error),
|
||||
};
|
||||
Ok(Self::fresh_or_miss(entry, now, request.max_age))
|
||||
|
|
@ -338,7 +338,10 @@ where
|
|||
now: Duration,
|
||||
) -> Result<PartialHits, Error> {
|
||||
if readable.len() != entries.len() {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(ShortBatch {
|
||||
requested: readable.len(),
|
||||
returned: entries.len(),
|
||||
}));
|
||||
}
|
||||
let mut values = vec![None; requests.len()];
|
||||
for ((index, request), entry) in readable.into_iter().zip(entries) {
|
||||
|
|
@ -361,7 +364,7 @@ where
|
|||
value: Self::fresh_or_miss(lookup.value, now, max_age),
|
||||
similarity: lookup.similarity,
|
||||
}),
|
||||
Err(Error::InvalidEntry) => Ok(SemanticLookup::miss(None)),
|
||||
Err(Error::InvalidEntry(_)) => Ok(SemanticLookup::miss(None)),
|
||||
Err(error) => Err(error),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -44,25 +44,23 @@ fn decode_reads_every_python_envelope(#[case] bytes: &[u8], #[case] response: Va
|
|||
#[case::non_utf8(vec![0xff, 0xfe])]
|
||||
#[case::deep_nesting(format!("{}None{}", "[".repeat(1000), "]".repeat(1000)).into_bytes())]
|
||||
fn decode_rejects_invalid_entries(#[case] bytes: Vec<u8>) {
|
||||
assert_eq!(
|
||||
ResponseCacheCodec.decode(&bytes).unwrap_err(),
|
||||
Error::InvalidEntry
|
||||
);
|
||||
assert!(matches!(
|
||||
ResponseCacheCodec.decode(&bytes),
|
||||
Err(Error::InvalidEntry(_))
|
||||
));
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[case::nan(f64::NAN)]
|
||||
#[case::infinity(f64::INFINITY)]
|
||||
fn encode_rejects_non_finite_timestamps(#[case] timestamp: f64) {
|
||||
assert_eq!(
|
||||
ResponseCacheCodec
|
||||
.encode(&CacheEntry {
|
||||
timestamp: Some(timestamp),
|
||||
response: json!({}),
|
||||
})
|
||||
.unwrap_err(),
|
||||
Error::InvalidEntry
|
||||
);
|
||||
assert!(matches!(
|
||||
ResponseCacheCodec.encode(&CacheEntry {
|
||||
timestamp: Some(timestamp),
|
||||
response: json!({}),
|
||||
}),
|
||||
Err(Error::InvalidEntry(_))
|
||||
));
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
|
|||
|
|
@ -158,8 +158,16 @@ fn scored(timestamp: f64, similarity: f64) -> Result<SemanticLookup<CacheEntry>,
|
|||
)]
|
||||
#[case::no_search(Ok(SemanticLookup::miss(None)), true, Ok(SemanticLookup::miss(None)))]
|
||||
#[case::disabled_reads_skip_the_backend(scored(95.0, 0.95), false, Ok(SemanticLookup::miss(None)))]
|
||||
#[case::invalid_entry_is_a_miss(Err(Error::InvalidEntry), true, Ok(SemanticLookup::miss(None)))]
|
||||
#[case::backend_errors_propagate(Err(Error::Unavailable), true, Err(Error::Unavailable))]
|
||||
#[case::invalid_entry_is_a_miss(
|
||||
Err(Error::InvalidEntry(None)),
|
||||
true,
|
||||
Ok(SemanticLookup::miss(None))
|
||||
)]
|
||||
#[case::backend_errors_propagate(
|
||||
Err(Error::Unavailable(None)),
|
||||
true,
|
||||
Err(Error::Unavailable(None))
|
||||
)]
|
||||
#[tokio::test]
|
||||
async fn semantic_lookup_applies_freshness_to_the_value_only(
|
||||
#[case] backend: Result<SemanticLookup<CacheEntry>, Error>,
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ reqwest.workspace = true
|
|||
tokio.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
litellm-error.workspace = true
|
||||
litellm-http = { workspace = true, features = ["test-support"] }
|
||||
litellm-cache-testing.workspace = true
|
||||
rstest.workspace = true
|
||||
|
|
|
|||
|
|
@ -46,7 +46,7 @@ impl ProvideCredentials for S3Credentials {
|
|||
self.auth
|
||||
.resolve_credentials(self.config.clone(), &self.env)
|
||||
.await
|
||||
.map_err(|_| CredentialsError::provider_error("S3 cache authentication failed"))
|
||||
.map_err(CredentialsError::provider_error)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -125,7 +125,7 @@ impl<C: CacheCodec> S3Cache<C> {
|
|||
}
|
||||
None => request.cache_control("immutable, max-age=31536000, s-maxage=31536000"),
|
||||
};
|
||||
request.send().await.map_err(|_| Error::Unavailable)?;
|
||||
request.send().await.map_err(Error::unavailable)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
|
@ -152,7 +152,7 @@ impl<C: CacheCodec> S3Cache<C> {
|
|||
return Ok(None);
|
||||
}
|
||||
}
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(error));
|
||||
}
|
||||
};
|
||||
if let Some(expires) = output.expires_string()
|
||||
|
|
@ -165,7 +165,7 @@ impl<C: CacheCodec> S3Cache<C> {
|
|||
.body
|
||||
.collect()
|
||||
.await
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.map_err(Error::unavailable)?
|
||||
.into_bytes();
|
||||
self.codec.decode(&bytes).map(Some)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -29,9 +29,10 @@ impl HttpConnector for ReqwestHttpClient {
|
|||
.try_into_http1x()
|
||||
.map_err(|error| ConnectorError::other(error.into(), None))?
|
||||
.map(reqwest::Body::wrap);
|
||||
let request = reqwest::Request::try_from(request)
|
||||
let request = litellm_http::outbound::from_http(request)
|
||||
.map_err(|error| ConnectorError::other(error.into(), None))?;
|
||||
let response = client.execute(request).await.map_err(|error| {
|
||||
let error = litellm_http::Error::from(error);
|
||||
if error.is_timeout() {
|
||||
ConnectorError::timeout(error.into())
|
||||
} else {
|
||||
|
|
|
|||
|
|
@ -147,15 +147,15 @@ async fn async_set_signs_with_the_configured_keys(#[future(awt)] server: MockSer
|
|||
#[case::malformed(
|
||||
"malformed",
|
||||
ResponseTemplate::new(200).set_body_string("not a cache entry"),
|
||||
Err(Error::InvalidEntry)
|
||||
Err("invalid cache entry: ")
|
||||
)]
|
||||
#[case::server_error("broken", ResponseTemplate::new(500), Err(Error::Unavailable))]
|
||||
#[case::server_error("broken", ResponseTemplate::new(500), Err("cache is unavailable: "))]
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn get_maps_s3_responses(
|
||||
#[future(awt)] server: MockServer,
|
||||
#[case] key: &str,
|
||||
#[case] response: ResponseTemplate,
|
||||
#[case] expected: Result<Option<Value>, Error>,
|
||||
#[case] expected: Result<Option<Value>, &str>,
|
||||
) {
|
||||
Mock::given(method("GET"))
|
||||
.and(path(format!("/cache-bucket/team/{key}")))
|
||||
|
|
@ -164,9 +164,21 @@ async fn get_maps_s3_responses(
|
|||
.await;
|
||||
let cache = support::cache(&server.uri());
|
||||
let context = ExactCacheContext::default();
|
||||
|
||||
assert_eq!(cache.get_cache(key, &context), expected);
|
||||
assert_eq!(cache.async_get_cache(key, &context).await, expected);
|
||||
let outcome = |result: Result<Option<Value>, Error>| {
|
||||
result.map_err(|error| litellm_error::Report::new(&error).to_string())
|
||||
};
|
||||
let sync = outcome(cache.get_cache(key, &context));
|
||||
let asynchronous = outcome(cache.async_get_cache(key, &context).await);
|
||||
match expected {
|
||||
Ok(value) => {
|
||||
assert_eq!(sync, Ok(value.clone()));
|
||||
assert_eq!(asynchronous, Ok(value));
|
||||
}
|
||||
Err(prefix) => {
|
||||
assert!(sync.unwrap_err().starts_with(prefix));
|
||||
assert!(asynchronous.unwrap_err().starts_with(prefix));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
|
|||
|
|
@ -10,9 +10,11 @@ litellm-cache.workspace = true
|
|||
litellm-cache-redis.workspace = true
|
||||
redis = { workspace = true, features = ["tls-rustls"] }
|
||||
sha2.workspace = true
|
||||
thiserror.workspace = true
|
||||
uuid = { version = "1", features = ["v4"] }
|
||||
|
||||
[dev-dependencies]
|
||||
litellm-error.workspace = true
|
||||
litellm-cache-testing.workspace = true
|
||||
redis-test.workspace = true
|
||||
rstest.workspace = true
|
||||
|
|
|
|||
|
|
@ -23,8 +23,7 @@ pub(crate) fn ensure_index(
|
|||
) -> Result<(), Error> {
|
||||
if index
|
||||
.dimension
|
||||
.lock()
|
||||
.map_err(|_| Error::Unavailable)?
|
||||
.lock()?
|
||||
.is_some_and(|existing| existing == dimension)
|
||||
{
|
||||
return Ok(());
|
||||
|
|
@ -50,24 +49,42 @@ pub(crate) fn ensure_index(
|
|||
.arg("DISTANCE_METRIC")
|
||||
.arg("COSINE")
|
||||
.query::<String>(connection)
|
||||
.map(|_| ())
|
||||
.map_err(|error| error.to_string());
|
||||
if let Err(message) = create {
|
||||
if !message.to_ascii_lowercase().contains("already exists") {
|
||||
return Err(Error::Unavailable);
|
||||
.map(|_| ());
|
||||
if let Err(error) = create {
|
||||
if !index_exists(&error) {
|
||||
return Err(Error::unavailable(error));
|
||||
}
|
||||
let info = redis::cmd("FT.INFO")
|
||||
.arg(&index.name)
|
||||
.query::<redis::Value>(connection)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
if index_dimension_from_info(&info).is_some_and(|existing| existing != dimension) {
|
||||
return Err(Error::Unavailable);
|
||||
.map_err(Error::unavailable)?;
|
||||
if let Some(existing) =
|
||||
index_dimension_from_info(&info).filter(|existing| *existing != dimension)
|
||||
{
|
||||
return Err(Error::unavailable(DimensionMismatch {
|
||||
existing,
|
||||
dimension,
|
||||
}));
|
||||
}
|
||||
}
|
||||
*index.dimension.lock().map_err(|_| Error::Unavailable)? = Some(dimension);
|
||||
*index.dimension.lock()? = Some(dimension);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn index_exists(error: &redis::RedisError) -> bool {
|
||||
error.code().is_some()
|
||||
&& error
|
||||
.detail()
|
||||
.is_some_and(|detail| detail.to_ascii_lowercase().contains("already exists"))
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("index exists with dimension {existing}, not {dimension}")]
|
||||
struct DimensionMismatch {
|
||||
existing: usize,
|
||||
dimension: usize,
|
||||
}
|
||||
|
||||
/// `_extract_index_dim`: flatten each attribute one level and read the value after
|
||||
/// `dimensions`.
|
||||
fn index_dimension_from_info(value: &redis::Value) -> Option<usize> {
|
||||
|
|
|
|||
|
|
@ -53,9 +53,7 @@ pub(crate) fn write_document(
|
|||
.arg(ttl.as_secs())
|
||||
.ignore();
|
||||
}
|
||||
pipeline
|
||||
.query::<()>(connection)
|
||||
.map_err(|_| Error::Unavailable)
|
||||
pipeline.query::<()>(connection).map_err(Error::unavailable)
|
||||
}
|
||||
|
||||
/// The KNN-1 search within `scope`: the closest document's similarity, and its stored response
|
||||
|
|
@ -83,7 +81,7 @@ pub(crate) fn search_document(
|
|||
.arg("DIALECT")
|
||||
.arg(2)
|
||||
.query::<redis::Value>(connection)
|
||||
.map_err(|_| Error::Unavailable)?;
|
||||
.map_err(Error::unavailable)?;
|
||||
let Some(fields) = search_fields(response)? else {
|
||||
return Ok(SemanticLookup::miss(Some(0.0)));
|
||||
};
|
||||
|
|
@ -91,7 +89,7 @@ pub(crate) fn search_document(
|
|||
fields
|
||||
.iter()
|
||||
.find_map(|(field, value)| (field == name).then(|| value.clone()))
|
||||
.ok_or(Error::InvalidEntry)
|
||||
.ok_or(Error::InvalidEntry(None))
|
||||
};
|
||||
let response = field("response")?;
|
||||
let similarity = 1.0 - parse_f64(&field("vector_distance")?)?;
|
||||
|
|
@ -105,24 +103,24 @@ type SearchFields = Vec<(String, Vec<u8>)>;
|
|||
|
||||
fn search_fields(value: redis::Value) -> Result<Option<SearchFields>, Error> {
|
||||
let redis::Value::Array(values) = value else {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
};
|
||||
let total = parse_i64(values.first().ok_or(Error::InvalidEntry)?)?;
|
||||
let total = parse_i64(values.first().ok_or(Error::InvalidEntry(None))?)?;
|
||||
if total <= 0 || values.len() < 3 {
|
||||
return Ok(None);
|
||||
}
|
||||
let redis::Value::Array(fields) = &values[2] else {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
};
|
||||
let (pairs, remainder) = fields.as_chunks::<2>();
|
||||
if !remainder.is_empty() {
|
||||
return Err(Error::InvalidEntry);
|
||||
return Err(Error::InvalidEntry(None));
|
||||
}
|
||||
pairs
|
||||
.iter()
|
||||
.map(|pair| {
|
||||
Ok((
|
||||
value_text(&pair[0]).ok_or(Error::InvalidEntry)?,
|
||||
value_text(&pair[0]).ok_or(Error::InvalidEntry(None))?,
|
||||
value_bytes(&pair[1])?,
|
||||
))
|
||||
})
|
||||
|
|
@ -132,16 +130,16 @@ fn search_fields(value: redis::Value) -> Result<Option<SearchFields>, Error> {
|
|||
|
||||
fn parse_i64(value: &redis::Value) -> Result<i64, Error> {
|
||||
value_text(value)
|
||||
.ok_or(Error::InvalidEntry)?
|
||||
.ok_or(Error::InvalidEntry(None))?
|
||||
.parse()
|
||||
.map_err(|_| Error::InvalidEntry)
|
||||
.map_err(Error::invalid_entry)
|
||||
}
|
||||
|
||||
fn parse_f64(value: &[u8]) -> Result<f64, Error> {
|
||||
std::str::from_utf8(value)
|
||||
.map_err(|_| Error::InvalidEntry)?
|
||||
.map_err(Error::invalid_entry)?
|
||||
.parse()
|
||||
.map_err(|_| Error::InvalidEntry)
|
||||
.map_err(Error::invalid_entry)
|
||||
}
|
||||
|
||||
pub(crate) fn value_text(value: &redis::Value) -> Option<String> {
|
||||
|
|
@ -158,6 +156,6 @@ fn value_bytes(value: &redis::Value) -> Result<Vec<u8>, Error> {
|
|||
redis::Value::BulkString(bytes) => Ok(bytes.clone()),
|
||||
redis::Value::SimpleString(value) => Ok(value.as_bytes().to_vec()),
|
||||
redis::Value::Int(value) => Ok(value.to_string().into_bytes()),
|
||||
_ => Err(Error::InvalidEntry),
|
||||
_ => Err(Error::InvalidEntry(None)),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -83,10 +83,7 @@ fn ok() -> redis::RedisResult<redis::Value> {
|
|||
}
|
||||
|
||||
fn already_exists() -> redis::RedisResult<redis::Value> {
|
||||
Err(redis::RedisError::from((
|
||||
redis::ErrorKind::Io,
|
||||
"already exists",
|
||||
)))
|
||||
Err(support::server_error("Index already exists"))
|
||||
}
|
||||
|
||||
fn bulk(value: &[u8]) -> redis::Value {
|
||||
|
|
@ -250,19 +247,29 @@ fn second_set_skips_create_after_dimension_is_cached(entry: Value, context: Sema
|
|||
|
||||
#[rstest]
|
||||
#[case::nested_matching(nested_dimension_info(3), Ok(()))]
|
||||
#[case::nested_mismatch(nested_dimension_info(2), Err(Error::Unavailable))]
|
||||
#[case::nested_mismatch(
|
||||
nested_dimension_info(2),
|
||||
Err("cache is unavailable: index exists with dimension 2, not 3")
|
||||
)]
|
||||
#[case::flat_matching(flat_dimension_info("3"), Ok(()))]
|
||||
#[case::flat_mismatch(flat_dimension_info("2"), Err(Error::Unavailable))]
|
||||
#[case::flat_mismatch(
|
||||
flat_dimension_info("2"),
|
||||
Err("cache is unavailable: index exists with dimension 2, not 3")
|
||||
)]
|
||||
#[case::unreported_dimension_is_accepted(redis::Value::Array(vec![]), Ok(()))]
|
||||
fn existing_index_dimension_must_match_embedding(
|
||||
#[case] info: redis::Value,
|
||||
#[case] expected: Result<(), Error>,
|
||||
#[case] expected: Result<(), &str>,
|
||||
entry: Value,
|
||||
context: SemanticCacheContext,
|
||||
) {
|
||||
let recording = recording([already_exists(), Ok(info)]);
|
||||
|
||||
assert_eq!(recording.cache.set_cache("key", entry, &context), expected);
|
||||
let result = recording.cache.set_cache("key", entry, &context);
|
||||
assert_eq!(
|
||||
result.map_err(|error| litellm_error::Report::new(&error).to_string()),
|
||||
expected.map_err(str::to_owned)
|
||||
);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
@ -274,10 +281,10 @@ fn index_creation_failures_are_unavailable(
|
|||
) {
|
||||
let recording = recording([reply]);
|
||||
|
||||
assert_eq!(
|
||||
assert!(matches!(
|
||||
recording.cache.set_cache("key", entry, &context),
|
||||
Err(Error::Unavailable)
|
||||
);
|
||||
Err(Error::Unavailable(Some(_)))
|
||||
));
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
@ -291,10 +298,10 @@ fn index_creation_failures_are_unavailable(
|
|||
bulk(b"document"),
|
||||
redis::Value::Array(vec![bulk(b"vector_distance"), bulk(b"0.1")]),
|
||||
]),
|
||||
Err(Error::InvalidEntry)
|
||||
Err(Error::InvalidEntry(None))
|
||||
)]
|
||||
#[case::unparsable_distance(search_hit(b"not-json", "abc"), Err(Error::InvalidEntry))]
|
||||
#[case::undecodable_response(search_hit(b"not-json", "0.1"), Err(Error::InvalidEntry))]
|
||||
#[case::unparsable_distance(search_hit(b"not-json", "abc"), Err(Error::InvalidEntry(None)))]
|
||||
#[case::undecodable_response(search_hit(b"not-json", "0.1"), Err(Error::InvalidEntry(None)))]
|
||||
fn get_applies_threshold_and_decodes_entry(
|
||||
#[case] reply: redis::Value,
|
||||
#[case] expected: Result<Option<Value>, Error>,
|
||||
|
|
@ -302,7 +309,13 @@ fn get_applies_threshold_and_decodes_entry(
|
|||
) {
|
||||
let recording = recording([ok(), Ok(reply)]);
|
||||
|
||||
assert_eq!(recording.cache.get_cache("key", &context), expected);
|
||||
let kind = |result: Result<Option<Value>, Error>| {
|
||||
result.map_err(|error| std::mem::discriminant(&error))
|
||||
};
|
||||
assert_eq!(
|
||||
kind(recording.cache.get_cache("key", &context)),
|
||||
kind(expected)
|
||||
);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
|
|
|
|||
|
|
@ -74,7 +74,7 @@ impl FakeSearch {
|
|||
"FT.CREATE" => {
|
||||
let name = text(1);
|
||||
if state.indexes.contains_key(&name) {
|
||||
return Err(error("Index already exists"));
|
||||
return Err(server_error("Index already exists"));
|
||||
}
|
||||
let position = |token: &str| args.iter().position(|arg| arg == token.as_bytes());
|
||||
let prefix = args[position("PREFIX").unwrap() + 2].clone();
|
||||
|
|
@ -94,7 +94,7 @@ impl FakeSearch {
|
|||
let index = state
|
||||
.indexes
|
||||
.get(&text(1))
|
||||
.ok_or_else(|| error("Unknown index name"))?;
|
||||
.ok_or_else(|| server_error("Unknown index name"))?;
|
||||
Ok(index_info(index))
|
||||
}
|
||||
"FT.DROPINDEX" => {
|
||||
|
|
@ -118,7 +118,7 @@ impl FakeSearch {
|
|||
let index = state
|
||||
.indexes
|
||||
.get(&text(1))
|
||||
.ok_or_else(|| error("no such index"))?;
|
||||
.ok_or_else(|| server_error("no such index"))?;
|
||||
let query = text(2);
|
||||
let tag = query_tag(&query);
|
||||
let params = args.iter().position(|arg| arg == b"PARAMS").unwrap();
|
||||
|
|
@ -155,7 +155,7 @@ impl FakeSearch {
|
|||
]))
|
||||
}
|
||||
"PING" => Ok(redis::Value::SimpleString("PONG".into())),
|
||||
_ => Err(error("unsupported command")),
|
||||
_ => Err(server_error("unsupported command")),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -192,8 +192,11 @@ impl redis::ConnectionLike for FakeSearch {
|
|||
}
|
||||
}
|
||||
|
||||
fn error(message: &'static str) -> redis::RedisError {
|
||||
redis::RedisError::from((redis::ErrorKind::Extension, message))
|
||||
pub fn server_error(reply: &str) -> redis::RedisError {
|
||||
redis::parse_redis_value(format!("-{reply}\r\n").as_bytes())
|
||||
.expect("error reply")
|
||||
.extract_error()
|
||||
.expect_err("server error")
|
||||
}
|
||||
|
||||
fn bulk(bytes: &[u8]) -> redis::Value {
|
||||
|
|
|
|||
1
litellm-rust/crates/cache/Cargo.toml
vendored
1
litellm-rust/crates/cache/Cargo.toml
vendored
|
|
@ -6,6 +6,7 @@ license.workspace = true
|
|||
repository.workspace = true
|
||||
|
||||
[dependencies]
|
||||
litellm-error.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json = { workspace = true, features = ["preserve_order"] }
|
||||
strum.workspace = true
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ pub trait BatchCache: BaseCache {
|
|||
.map(|key| match self.get_cache(key, context) {
|
||||
Ok(Some(value)) => Ok(BatchEntry::Hit(value)),
|
||||
Ok(None) => Ok(BatchEntry::Miss),
|
||||
Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid),
|
||||
Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid),
|
||||
Err(error) => Err(error),
|
||||
})
|
||||
.collect()
|
||||
|
|
@ -65,7 +65,7 @@ pub trait BatchCache: BaseCache {
|
|||
entries.push(match self.async_get_cache(&key, &context).await {
|
||||
Ok(Some(value)) => BatchEntry::Hit(value),
|
||||
Ok(None) => BatchEntry::Miss,
|
||||
Err(Error::InvalidEntry) => BatchEntry::Invalid,
|
||||
Err(Error::InvalidEntry(_)) => BatchEntry::Invalid,
|
||||
Err(error) => return Err(error),
|
||||
});
|
||||
}
|
||||
|
|
|
|||
4
litellm-rust/crates/cache/src/codec.rs
vendored
4
litellm-rust/crates/cache/src/codec.rs
vendored
|
|
@ -41,10 +41,10 @@ where
|
|||
type Value = V;
|
||||
|
||||
fn encode(&self, value: &Self::Value) -> Result<Vec<u8>, Error> {
|
||||
serde_json::to_vec(value).map_err(|_| Error::InvalidEntry)
|
||||
serde_json::to_vec(value).map_err(Error::invalid_entry)
|
||||
}
|
||||
|
||||
fn decode(&self, bytes: &[u8]) -> Result<Self::Value, Error> {
|
||||
serde_json::from_slice(bytes).map_err(|_| Error::InvalidEntry)
|
||||
serde_json::from_slice(bytes).map_err(Error::invalid_entry)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
14
litellm-rust/crates/cache/src/dual.rs
vendored
14
litellm-rust/crates/cache/src/dual.rs
vendored
|
|
@ -2,7 +2,7 @@ use std::{sync::Arc, time::Duration};
|
|||
|
||||
use crate::{
|
||||
BaseCache, BatchCache, BatchEntry, BulkDeleteCache, CacheContext, ClaimCache, CounterCache,
|
||||
DeleteCache, Error, FlushCache, IncrementOperation, SetCache, TtlCache,
|
||||
DeleteCache, Error, FlushCache, IncrementOperation, SetCache, ShortBatch, TtlCache,
|
||||
};
|
||||
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
|
|
@ -98,7 +98,7 @@ impl<L1, L2> DualCache<L1, L2> {
|
|||
fn remote<T>(&self, result: Result<T, Error>) -> Result<Option<T>, Error> {
|
||||
match result {
|
||||
Ok(value) => Ok(Some(value)),
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(_))
|
||||
if self.remote_failure_policy == RemoteFailurePolicy::UseLocal =>
|
||||
{
|
||||
Ok(None)
|
||||
|
|
@ -136,7 +136,10 @@ where
|
|||
remote: Vec<BatchEntry<V>>,
|
||||
) -> Result<Vec<BatchEntry<V>>, Error> {
|
||||
if missing.len() != remote.len() {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(ShortBatch {
|
||||
requested: missing.len(),
|
||||
returned: remote.len(),
|
||||
}));
|
||||
}
|
||||
for (index, entry) in missing.into_iter().zip(remote) {
|
||||
if let BatchEntry::Hit(value) = &entry {
|
||||
|
|
@ -389,7 +392,10 @@ where
|
|||
}
|
||||
let values = self.l2.async_increment_pipeline(operations.clone()).await?;
|
||||
if values.len() != operations.len() {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::unavailable(ShortBatch {
|
||||
requested: operations.len(),
|
||||
returned: values.len(),
|
||||
}));
|
||||
}
|
||||
for (operation, value) in operations.iter().zip(&values) {
|
||||
self.l1
|
||||
|
|
|
|||
62
litellm-rust/crates/cache/src/error.rs
vendored
62
litellm-rust/crates/cache/src/error.rs
vendored
|
|
@ -1,9 +1,11 @@
|
|||
use litellm_error::Source;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
|
||||
pub enum Error {
|
||||
#[error("cache is unavailable")]
|
||||
Unavailable,
|
||||
Unavailable(#[source] Option<Source>),
|
||||
#[error("invalid cache entry")]
|
||||
InvalidEntry,
|
||||
InvalidEntry(#[source] Option<Source>),
|
||||
#[error("flushing Redis requires an explicit namespace")]
|
||||
UnscopedFlush,
|
||||
#[error("operation is not supported by this cache")]
|
||||
|
|
@ -11,3 +13,59 @@ pub enum Error {
|
|||
#[error("semantic cache requires request messages")]
|
||||
MissingPrompt,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("cache backend returned {returned} results for {requested} keys")]
|
||||
pub struct ShortBatch {
|
||||
pub requested: usize,
|
||||
pub returned: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("cache lock is poisoned")]
|
||||
struct Poisoned;
|
||||
|
||||
impl<T> From<std::sync::PoisonError<T>> for Error {
|
||||
fn from(_: std::sync::PoisonError<T>) -> Self {
|
||||
Self::unavailable(Poisoned)
|
||||
}
|
||||
}
|
||||
|
||||
impl Error {
|
||||
pub fn unavailable(cause: impl std::error::Error + Send + Sync + 'static) -> Self {
|
||||
Self::Unavailable(Some(Source::new(cause)))
|
||||
}
|
||||
|
||||
pub fn invalid_entry(cause: impl std::error::Error + Send + Sync + 'static) -> Self {
|
||||
Self::InvalidEntry(Some(Source::new(cause)))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use rstest::rstest;
|
||||
|
||||
use super::Error;
|
||||
|
||||
#[rstest]
|
||||
#[case::unavailable(
|
||||
Error::unavailable(std::io::Error::other("connection refused")),
|
||||
"cache is unavailable"
|
||||
)]
|
||||
#[case::invalid_entry(
|
||||
Error::invalid_entry(std::io::Error::other("connection refused")),
|
||||
"invalid cache entry"
|
||||
)]
|
||||
fn the_backend_cause_is_the_source_and_rendered_once(
|
||||
#[case] error: Error,
|
||||
#[case] message: &str,
|
||||
) {
|
||||
let report = litellm_error::Report::new(&error).to_string();
|
||||
assert_eq!(report, format!("{message}: connection refused"));
|
||||
assert!(!error.to_string().contains("connection refused"));
|
||||
let cause = std::error::Error::source(&error)
|
||||
.and_then(|cause| cause.downcast_ref::<litellm_error::Source>())
|
||||
.expect("cause");
|
||||
assert!(cause.downcast_ref::<std::io::Error>().is_some());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
2
litellm-rust/crates/cache/src/lib.rs
vendored
2
litellm-rust/crates/cache/src/lib.rs
vendored
|
|
@ -23,4 +23,4 @@ pub use codec::{CacheCodec, JsonCodec};
|
|||
pub use dual::{
|
||||
DEFAULT_DELETE_BATCH_SIZE, DualCache, ReadPolicy, RemoteFailurePolicy, WritePolicy,
|
||||
};
|
||||
pub use error::Error;
|
||||
pub use error::{Error, ShortBatch};
|
||||
|
|
|
|||
10
litellm-rust/crates/cache/tests/caching.rs
vendored
10
litellm-rust/crates/cache/tests/caching.rs
vendored
|
|
@ -67,7 +67,7 @@ impl BaseCache for TestCache {
|
|||
}
|
||||
|
||||
fn set_cache(&self, _: &str, _: Self::Value, _: &ExactCacheContext) -> Result<(), Error> {
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(None))
|
||||
}
|
||||
|
||||
async fn async_set_cache(
|
||||
|
|
@ -77,7 +77,7 @@ impl BaseCache for TestCache {
|
|||
context: ExactCacheContext,
|
||||
) -> Result<(), Error> {
|
||||
if key == "unavailable" {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::Unavailable(None));
|
||||
}
|
||||
self.writes
|
||||
.lock()
|
||||
|
|
@ -118,7 +118,7 @@ impl BaseCache for RecordingCounter {
|
|||
impl CounterCache for RecordingCounter {
|
||||
fn increment_cache(&self, key: &str, amount: f64, _: ExactCacheContext) -> Result<f64, Error> {
|
||||
if key == "unavailable" {
|
||||
return Err(Error::Unavailable);
|
||||
return Err(Error::Unavailable(None));
|
||||
}
|
||||
let mut total = self.total.lock().unwrap();
|
||||
*total += amount;
|
||||
|
|
@ -226,7 +226,7 @@ async fn default_batch_operations_use_async_writes_and_stop_on_failure(cache: Te
|
|||
context.clone(),
|
||||
)
|
||||
.await,
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(None))
|
||||
);
|
||||
assert_eq!(
|
||||
*cache.writes.lock().unwrap(),
|
||||
|
|
@ -323,7 +323,7 @@ async fn default_increment_pipeline_stops_at_the_first_failure() {
|
|||
operation("skipped", 1.0, None),
|
||||
])
|
||||
.await,
|
||||
Err(Error::Unavailable)
|
||||
Err(Error::Unavailable(None))
|
||||
);
|
||||
let keys = cache
|
||||
.increments
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue