From 7b60fd061a5e0500fa075600d3c36fd35be48f43 Mon Sep 17 00:00:00 2001 From: yujonglee Date: Wed, 28 Oct 2026 09:22:11 -0700 Subject: [PATCH] refactor(rust): adopt one error discipline across litellm-rust with source chains and Report (#48042) * build(rust): upgrade reqwest to 0.13 and move MCP out of litellm-http The workspace moves to reqwest 0.13, so litellm-http no longer carries a second reqwest just for rmcp. The mcp feature, its pool and its proxy conversion go away, and the gateway hands rmcp a pooled NoRedirect client that already carries the TLS, proxy and timeout settings The Client test helpers collapse into for_test, a no-redirect client without TLS or proxy settings, and wrap_for_test for the few tests that need a hand-built reqwest client Co-Authored-By: Claude Opus 5.5 * fix(http): honor transport settings for Rust WebSockets The Responses WebSocket connected through tokio-tungstenite with its own native-roots TLS, so it ignored the configured CA bundle, verify flag, client certificate, proxies and timeouts. The handshake now goes through a pooled reqwest client (ClientVariant::WebSocket, HTTP/1.1, no redirects) and upgrades the response, so it shares the settings every other outbound request uses The upgrade response is checked for the Sec-WebSocket-Accept key, the Upgrade and Connection headers and a subprotocol that was offered Co-Authored-By: Claude Opus 5.5 * refactor(http): build outbound requests as reqwest::Request OutboundRequest was a hand-rolled copy of reqwest::Request. Routes now build a reqwest::Request directly and RequestSigner reads the body bytes back out of it, so the signer still covers exactly what is sent. SigV4 signing moves onto HeaderMap and takes the method, which drops the BTreeMap plumbing and the header casing table Co-Authored-By: Claude Opus 5.5 * refactor(http): make litellm_http::Error the one opaque error transport::Error and media::Error fold into litellm_http::Error, an Arc newtype whose private variants keep the typed reqwest, io, rustls and tungstenite causes and own every message. Callers can no longer build an error with a message of their own. They ask through dispatched, is_request, status, upstream_response and tls_source One From now classifies connect, network and timeout and strips the URL. The WebSocket handshake reports each failed check as its own variant and owns its connect timeout. OCR drops the media variants it mirrored, and inference::Error maps a header the signer computes to Unsupported for every route, not only chat The error rules move out of AGENTS.md into the rust-error skill, which also records the decisions behind this design Co-Authored-By: Claude Opus 5.5 * docs(rust): split rust-error skill into thin SKILL.md with references Co-Authored-By: Claude Opus 5.5 * refactor(rust-error): render causes once and ask predicates at the boundaries The four WebSocket handshake variants collapse into Handshake(HandshakeFault), and media download failures no longer answer dispatched() since they happen before the provider call ErrorDetail in llms and auth-types, the inference error and the OCR error now store their cause in a field named cause with no #[source], so to_string() carries it exactly once and source() is None. Tests that walked the chain now destructure the variant and downcast the cause The gateway asks is_unsupported() and is_missing_credential() on the inference and OCR errors instead of matching through nested enums, and litellm_auth::Error answers is_missing_credential() for both The rust-error skill gains axum and pyo3 references, boundaries.md describes the real predicate order and the known divergences, the testing example asserts dispatched() correctly, and shape, classification, conversions, messages-and-causes and recipes agree with the code again * docs(rust): focus the rust-error gateway page on the error flow and add a rust-gateway skill The former axum page toured gateway-inference: its status match, the envelope table, an extractor, the unsupported route and the Sse closure. Those details belong to the gateway crates and go stale with them. The page is now gateway.md and follows an error from the provider failure through each From into IntoResponse, where it is classified once and rendered by surface and framing. The gateway-structure material moves to a new rust-gateway skill: envelopes per API surface, extractors and path registration, SSE relay, and the router test harness. * test(inference): pin provider URL redaction on every route's transport error Chat, Messages, Responses and transcription each get a route-level test that fails if a reqwest error reaches the caller with its URL. The Messages stream decoder also takes body errors through litellm_http::Error like every other reqwest error Co-Authored-By: Claude Opus 5.5 * docs(rust-error): adopt std source chains and add litellm-error Report Messages describe their own layer and the cause is the source, boundaries render the chain once with litellm_error::Report. Corrects skill claims the audit found stale against the code. Co-Authored-By: Claude Opus 5.5 * refactor(rust-error): keep causes as sources across http, auth, secrets, cache and utility crates Messages describe their own layer and every cause is a typed source, predicates match exhaustively, and boundaries render with litellm_error::Report. Media downloads no longer count as dispatched, Error::upstream bounds the provider body itself, and a forwarded header SigV4 cannot sign is the caller's fault. Co-Authored-By: Claude Opus 5.5 * refactor(rust-error): keep causes as sources in llms and the gateway crates ErrorDetail loses its string variant, OCR provider variants become data, host faults stop counting as the caller's, a tunnelled transport error classifies the same on every stream path, and each gateway crate has one Error with one status() and extractors that reject in its envelope. Co-Authored-By: Claude Opus 5.5 * refactor(rust-error): nest llms errors in inference and ask predicates in route tests litellm_inference::Error nests litellm_llms::Error instead of mirroring its variants, host driver faults get their own variant, every predicate matches exhaustively, and route tests ask predicates instead of matching transport variants. Co-Authored-By: Claude Opus 5.5 * refactor(rust-error): map bridge and gateway-inference errors once and render them with Report RustUpstreamError has one constructor that always sets headers, argument checks raise typed errors mapped once, OCR asks upstream_response(), the Responses stream error event comes from the error type, gateway-inference loses its string variants, and auth faults classify through one litellm_auth predicate on every route. Co-Authored-By: Claude Opus 5.5 * fix(rust-error): carry OCR provider headers as a HeaderMap and keep outages off the caller The OCR provider error and get_error_class take reqwest's HeaderMap, the bridge turns it into Python's (name, value) pairs once. Credential acquisition and provider SDK failures are no longer the caller's fault, the UI hides auth and backend causes, and chat keeps a provider answer as is. Co-Authored-By: Claude Opus 5.5 * docs(rust-error): correct skill claims against the aligned code Co-Authored-By: Claude Opus 5.5 * refactor(rust-error): fix the remaining sweep findings in gateway, llms, auth and cache crates Auth rejections hide their backend cause on every inference renderer, path parameters reject in the surface envelope, Textract's unsupported document and Azure DI's unparseable poll URL get typed variants, the credential chain exposes its last failure as the source, gateway-mcp asks predicates over rmcp errors, and the S3 and Azure blob transports classify through litellm_http::Error. Co-Authored-By: Claude Opus 5.5 * refactor(rust-error): fix the remaining sweep findings on the Python side host-python keeps a native error's chain and returns blocking-pool cancellation instead of panicking, Vault transport failures convert through litellm_http::Error so no URL reaches Python, the bridge asks predicates instead of matching three levels into Vault errors, and the last string-built PyErrs become typed errors mapped once. Co-Authored-By: Claude Opus 5.5 * build(deps): refresh compatible dependencies flagged by OSV * refactor(rust): remove redundant client construction comment * test(rust): cover configured MCP proxy and CA bundle * fix(http): preserve native WebSocket trust and optional protocols * fix(http): allow plain WebSockets without native roots * fix(http): normalize provider URLs before building outbound requests * fix(rust): preserve OCR download status and reduce error size * fix(rust): preserve upstream bodies and bound diagnostics only * test(rust): preserve bridge serialization boundary in body regression --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .../.agents/skills/rust-error/SKILL.md | 35 + .../rust-error/references/boundaries.md | 39 + .../rust-error/references/classification.md | 66 ++ .../rust-error/references/conversions.md | 92 +++ .../skills/rust-error/references/gateway.md | 56 ++ .../references/messages-and-causes.md | 112 +++ .../skills/rust-error/references/pyo3.md | 74 ++ .../skills/rust-error/references/recipes.md | 64 ++ .../skills/rust-error/references/shape.md | 105 +++ .../skills/rust-error/references/testing.md | 37 + .../.agents/skills/rust-gateway/SKILL.md | 20 + .../rust-gateway/references/requests.md | 34 + .../rust-gateway/references/surfaces.md | 20 + .../skills/rust-gateway/references/testing.md | 9 + litellm-rust/AGENTS.md | 12 +- litellm-rust/Cargo.lock | 66 +- litellm-rust/Cargo.toml | 1 + litellm-rust/crates/auth-aws/Cargo.toml | 1 + litellm-rust/crates/auth-aws/src/aws.rs | 94 ++- litellm-rust/crates/auth-aws/src/error.rs | 101 ++- litellm-rust/crates/auth-aws/src/lib.rs | 2 +- litellm-rust/crates/auth-aws/src/signer.rs | 44 +- litellm-rust/crates/auth-azure/Cargo.toml | 1 + litellm-rust/crates/auth-azure/src/native.rs | 43 +- litellm-rust/crates/auth-azure/src/resolve.rs | 119 ++-- litellm-rust/crates/auth-gcp/Cargo.toml | 1 + litellm-rust/crates/auth-gcp/src/auth.rs | 45 +- litellm-rust/crates/auth-gcp/src/sdk.rs | 6 +- litellm-rust/crates/auth-types/Cargo.toml | 1 + .../crates/auth-types/src/credential.rs | 26 +- litellm-rust/crates/auth-types/src/error.rs | 326 +++++++-- litellm-rust/crates/auth-types/src/http.rs | 12 +- litellm-rust/crates/auth-types/src/lib.rs | 4 +- .../crates/auth-types/src/params/aws.rs | 49 +- .../crates/auth-types/src/params/mod.rs | 7 + .../crates/auth-types/src/params/vertex.rs | 5 +- litellm-rust/crates/auth-types/src/policy.rs | 35 +- litellm-rust/crates/auth-types/tests/error.rs | 74 -- .../crates/auth-types/tests/params.rs | 3 +- litellm-rust/crates/auth-types/tests/token.rs | 19 +- .../crates/cache-azure-blob/Cargo.toml | 1 + .../crates/cache-azure-blob/src/cache.rs | 30 +- .../crates/cache-azure-blob/src/credential.rs | 15 +- .../crates/cache-azure-blob/src/transport.rs | 7 +- .../crates/cache-azure-blob/tests/cache.rs | 12 +- litellm-rust/crates/cache-disk/Cargo.toml | 1 + litellm-rust/crates/cache-disk/src/cache.rs | 8 +- .../crates/cache-disk/src/python/value.rs | 18 +- litellm-rust/crates/cache-disk/src/sqlite.rs | 168 +++-- .../crates/cache-disk/tests/python_compat.rs | 2 +- litellm-rust/crates/cache-gcs/Cargo.toml | 2 + litellm-rust/crates/cache-gcs/src/cache.rs | 32 +- litellm-rust/crates/cache-gcs/src/token.rs | 2 +- litellm-rust/crates/cache-gcs/tests/cache.rs | 39 +- litellm-rust/crates/cache-memory/src/cache.rs | 48 +- .../crates/cache-memory/tests/cache.rs | 6 +- .../crates/cache-qdrant-semantic/Cargo.toml | 1 + .../crates/cache-qdrant-semantic/src/cache.rs | 14 +- .../cache-qdrant-semantic/src/embedder.rs | 17 +- .../cache-qdrant-semantic/tests/embedder.rs | 21 +- .../cache-qdrant-semantic/tests/qdrant.rs | 26 +- .../crates/cache-redis-semantic/src/cache.rs | 6 +- .../crates/cache-redis-semantic/src/index.rs | 18 +- .../cache-redis-semantic/tests/cache.rs | 30 +- .../cache-redis-semantic/tests/support/mod.rs | 15 +- litellm-rust/crates/cache-redis/Cargo.toml | 2 + litellm-rust/crates/cache-redis/src/cache.rs | 4 +- litellm-rust/crates/cache-redis/src/claim.rs | 14 +- .../crates/cache-redis/src/connection.rs | 74 +- .../crates/cache-redis/src/counter.rs | 8 +- litellm-rust/crates/cache-redis/src/keys.rs | 4 +- .../crates/cache-redis/src/lifecycle.rs | 9 +- litellm-rust/crates/cache-redis/src/queue.rs | 16 +- litellm-rust/crates/cache-redis/src/script.rs | 2 +- litellm-rust/crates/cache-redis/src/store.rs | 18 +- .../crates/cache-redis/tests/cache.rs | 38 +- .../crates/cache-redis/tests/cluster.rs | 2 +- .../crates/cache-redis/tests/support/mod.rs | 4 +- .../crates/cache-response/src/buffer.rs | 4 +- .../crates/cache-response/src/codec.rs | 28 +- .../crates/cache-response/src/response.rs | 13 +- .../crates/cache-response/tests/codec.rs | 24 +- .../crates/cache-response/tests/response.rs | 12 +- litellm-rust/crates/cache-s3/Cargo.toml | 1 + litellm-rust/crates/cache-s3/src/auth.rs | 2 +- litellm-rust/crates/cache-s3/src/cache.rs | 6 +- litellm-rust/crates/cache-s3/src/transport.rs | 3 +- litellm-rust/crates/cache-s3/tests/cache.rs | 24 +- .../crates/cache-valkey-semantic/Cargo.toml | 2 + .../crates/cache-valkey-semantic/src/index.rs | 39 +- .../cache-valkey-semantic/src/search.rs | 28 +- .../cache-valkey-semantic/tests/cache.rs | 43 +- .../tests/support/mod.rs | 15 +- litellm-rust/crates/cache/Cargo.toml | 1 + litellm-rust/crates/cache/src/capabilities.rs | 4 +- litellm-rust/crates/cache/src/codec.rs | 4 +- litellm-rust/crates/cache/src/dual.rs | 14 +- litellm-rust/crates/cache/src/error.rs | 62 +- litellm-rust/crates/cache/src/lib.rs | 2 +- litellm-rust/crates/cache/tests/caching.rs | 10 +- litellm-rust/crates/cache/tests/codec.rs | 10 +- litellm-rust/crates/cache/tests/dual.rs | 35 +- .../crates/callbacks-legacy-python/Cargo.toml | 2 + .../callbacks-legacy-python/src/adapter.rs | 7 +- .../callbacks-legacy-python/src/error.rs | 17 + .../crates/callbacks-legacy-python/src/lib.rs | 1 + litellm-rust/crates/config/Cargo.toml | 1 + litellm-rust/crates/config/src/error.rs | 11 +- litellm-rust/crates/config/src/includes.rs | 35 +- litellm-rust/crates/config/tests/config.rs | 10 +- litellm-rust/crates/core-utils/Cargo.toml | 1 + .../crates/core-utils/src/call_arguments.rs | 19 +- .../src/exception_mapping_utils/mod.rs | 16 +- litellm-rust/crates/core-utils/src/params.rs | 2 + .../crates/core-utils/src/url_utils.rs | 2 +- litellm-rust/crates/cost/Cargo.toml | 3 + litellm-rust/crates/cost/src/error.rs | 27 + litellm-rust/crates/cost/src/lib.rs | 88 +-- litellm-rust/crates/cost/tests/calculation.rs | 20 +- litellm-rust/crates/error/Cargo.toml | 13 + litellm-rust/crates/error/src/lib.rs | 132 ++++ litellm-rust/crates/framer/src/error.rs | 8 +- litellm-rust/crates/framer/src/framer.rs | 13 - litellm-rust/crates/gateway-auth/AGENTS.md | 4 +- litellm-rust/crates/gateway-auth/Cargo.toml | 1 + litellm-rust/crates/gateway-auth/src/error.rs | 108 +-- litellm-rust/crates/gateway-auth/src/keys.rs | 10 +- litellm-rust/crates/gateway-auth/src/lib.rs | 2 +- litellm-rust/crates/gateway-auth/src/ui.rs | 29 +- .../crates/gateway-auth/tests/auth.rs | 12 +- .../crates/gateway-auth/tests/keys.rs | 22 +- .../crates/gateway-inference/Cargo.toml | 2 + .../src/audio_transcription.rs | 2 +- .../crates/gateway-inference/src/caching.rs | 20 +- .../gateway-inference/src/chat_completions.rs | 9 +- .../crates/gateway-inference/src/error.rs | 348 ++++++--- .../crates/gateway-inference/src/lib.rs | 2 +- .../crates/gateway-inference/src/messages.rs | 2 +- .../crates/gateway-inference/src/ocr.rs | 2 +- .../crates/gateway-inference/src/request.rs | 39 +- .../crates/gateway-inference/src/responses.rs | 9 +- .../gateway-inference/tests/messages.rs | 26 +- .../gateway-inference/tests/responses.rs | 29 + .../crates/gateway-inference/tests/routes.rs | 16 +- .../gateway-inference/tests/support/mod.rs | 24 + .../crates/gateway-management/Cargo.toml | 1 + .../crates/gateway-management/src/error.rs | 2 +- .../crates/gateway-management/tests/keys.rs | 10 +- .../gateway-management/tests/keys/support.rs | 15 +- litellm-rust/crates/gateway-mcp/AGENTS.md | 2 + litellm-rust/crates/gateway-mcp/Cargo.toml | 2 + .../crates/gateway-mcp/src/configured.rs | 120 ++-- litellm-rust/crates/gateway-mcp/src/error.rs | 381 +++++++++- litellm-rust/crates/gateway-mcp/src/legacy.rs | 88 +-- litellm-rust/crates/gateway-mcp/src/lib.rs | 3 +- litellm-rust/crates/gateway-mcp/src/native.rs | 23 +- .../crates/gateway-mcp/src/native/catalog.rs | 15 +- .../crates/gateway-mcp/src/native/registry.rs | 18 +- .../crates/gateway-mcp/src/request.rs | 44 ++ litellm-rust/crates/gateway-mcp/src/routes.rs | 14 +- .../crates/gateway-mcp/src/sessions.rs | 12 +- .../crates/gateway-mcp/tests/gateway/http.rs | 115 ++- .../gateway-mcp/tests/gateway/native.rs | 10 +- litellm-rust/crates/gateway-ui/Cargo.toml | 2 + .../crates/gateway-ui/src/dashboard.rs | 16 +- litellm-rust/crates/gateway-ui/src/error.rs | 119 +++- litellm-rust/crates/gateway-ui/src/session.rs | 35 +- .../crates/gateway-ui/tests/assets.rs | 18 + .../crates/gateway-ui/tests/routes.rs | 53 +- litellm-rust/crates/gateway/Cargo.toml | 1 + litellm-rust/crates/gateway/src/auth.rs | 8 +- litellm-rust/crates/gateway/src/error.rs | 4 +- litellm-rust/crates/gateway/src/lib.rs | 2 +- litellm-rust/crates/gateway/tests/mcp.rs | 146 ++-- litellm-rust/crates/gateway/tests/server.rs | 5 +- litellm-rust/crates/host-python/Cargo.toml | 1 + .../crates/host-python/src/binding.rs | 2 +- litellm-rust/crates/host-python/src/driver.rs | 2 +- litellm-rust/crates/host-python/src/error.rs | 16 +- .../crates/host-python/src/fork_gate.rs | 14 +- litellm-rust/crates/host-python/src/gil.rs | 14 +- litellm-rust/crates/host-python/src/handle.rs | 32 +- .../crates/host-python/src/marshal.rs | 3 +- .../crates/host-python/src/runtime.rs | 31 +- .../crates/host-python/tests/support/mod.rs | 2 + litellm-rust/crates/host/Cargo.toml | 1 + .../crates/host/src/machine/coroutine.rs | 6 +- litellm-rust/crates/http/Cargo.toml | 2 + litellm-rust/crates/http/src/config.rs | 20 +- litellm-rust/crates/http/src/error.rs | 672 +++++++++++++++++- litellm-rust/crates/http/src/lib.rs | 1 - litellm-rust/crates/http/src/media.rs | 122 ++-- litellm-rust/crates/http/src/outbound.rs | 40 +- litellm-rust/crates/http/src/pool.rs | 5 +- litellm-rust/crates/http/src/request.rs | 38 +- litellm-rust/crates/http/src/tls.rs | 39 +- litellm-rust/crates/http/src/transport.rs | 109 --- litellm-rust/crates/http/src/websocket.rs | 64 +- litellm-rust/crates/http/tests/websocket.rs | 53 +- litellm-rust/crates/inference-chat/Cargo.toml | 2 +- .../crates/inference-chat/src/common_utils.rs | 8 +- .../crates/inference-chat/src/handler.rs | 127 ++-- .../inference-chat/tests/chat_completions.rs | 50 +- .../crates/inference-chat/tests/request.rs | 43 +- .../crates/inference-messages/Cargo.toml | 1 + .../inference-messages/src/common_utils.rs | 16 +- .../crates/inference-messages/src/handler.rs | 72 +- .../crates/inference-messages/src/types.rs | 4 +- .../tests/bedrock_native/main.rs | 9 +- .../inference-messages/tests/messages/host.rs | 9 +- .../inference-messages/tests/messages/main.rs | 2 +- .../tests/messages/request.rs | 26 +- .../tests/messages/response.rs | 59 +- .../tests/messages/secrets.rs | 2 +- .../tests/messages/stream.rs | 24 +- .../tests/messages/tencent.rs | 10 +- .../tests/openrouter/main.rs | 15 +- litellm-rust/crates/inference-ocr/Cargo.toml | 2 + .../crates/inference-ocr/src/document.rs | 27 +- .../inference-ocr/src/provider_config.rs | 32 +- .../crates/inference-ocr/src/route.rs | 5 +- .../crates/inference-ocr/src/types.rs | 4 +- litellm-rust/crates/inference-ocr/src/wire.rs | 28 +- .../inference-ocr/tests/ocr/aws_textract.rs | 15 +- .../inference-ocr/tests/ocr/azure_ai.rs | 11 +- .../tests/ocr/azure_document_intelligence.rs | 34 +- .../crates/inference-ocr/tests/ocr/cohere.rs | 11 +- .../inference-ocr/tests/ocr/documents.rs | 7 +- .../inference-ocr/tests/ocr/lifecycle.rs | 7 +- .../crates/inference-ocr/tests/ocr/machine.rs | 35 +- .../crates/inference-ocr/tests/ocr/main.rs | 12 +- .../crates/inference-ocr/tests/ocr/mistral.rs | 25 +- .../crates/inference-ocr/tests/ocr/reducto.rs | 8 +- .../inference-ocr/tests/ocr/vertex_ai.rs | 10 +- .../crates/inference-responses/Cargo.toml | 1 + .../crates/inference-responses/src/handler.rs | 37 +- .../inference-responses/src/websocket.rs | 55 +- .../inference-responses/tests/openrouter.rs | 18 +- .../inference-responses/tests/responses.rs | 30 +- .../inference-responses/tests/support/mod.rs | 3 + .../crates/inference-transcription/Cargo.toml | 1 + .../inference-transcription/src/handler.rs | 19 +- .../tests/audio_transcription.rs | 38 +- litellm-rust/crates/inference/AGENTS.md | 6 +- litellm-rust/crates/inference/Cargo.toml | 1 + litellm-rust/crates/inference/src/error.rs | 340 +++++++-- litellm-rust/crates/inference/src/outbound.rs | 8 +- litellm-rust/crates/inference/src/provider.rs | 6 +- .../crates/inference/tests/caching.rs | 4 +- litellm-rust/crates/llms/Cargo.toml | 1 + .../src/anthropic/batches/transformation.rs | 45 +- .../crates/llms/src/anthropic/chat/handler.rs | 163 ----- .../crates/llms/src/anthropic/chat/mod.rs | 1 - .../llms/src/anthropic/chat/transformation.rs | 34 +- .../llms/src/anthropic/messages/handler.rs | 19 +- .../ocr/analyze_transformation.rs | 2 +- .../llms/src/aws_textract/ocr/common_utils.rs | 90 +-- .../src/aws_textract/ocr/transformation.rs | 4 +- .../ocr/cohere_parse_transformation.rs | 6 +- .../llms/src/azure_ai/ocr/common_utils.rs | 4 +- .../document_intelligence/transformation.rs | 47 +- .../llms/src/azure_ai/ocr/transformation.rs | 11 +- .../llms/src/base_llm/base_model_iterator.rs | 9 + .../crates/llms/src/base_llm/ocr/document.rs | 73 +- .../crates/llms/src/base_llm/ocr/error.rs | 440 +++++++++--- .../crates/llms/src/base_llm/ocr/handler.rs | 89 +-- .../llms/src/base_llm/ocr/transformation.rs | 30 +- litellm-rust/crates/llms/src/base_llm/sse.rs | 55 +- .../src/bedrock/audio_transcription/mod.rs | 33 +- .../bedrock/chat/converse_transformation.rs | 69 +- .../llms/src/bedrock/chat/invoke_handler.rs | 66 +- .../messages/transformation.rs | 3 +- .../llms/src/cohere/ocr/transformation.rs | 50 +- litellm-rust/crates/llms/src/error.rs | 177 +++-- litellm-rust/crates/llms/src/lib.rs | 2 +- .../llms/src/mistral/ocr/transformation.rs | 9 +- .../src/openai/responses/transformation.rs | 18 +- .../src/openai_like/chat/transformation.rs | 19 +- .../llms/src/openai_like/common_utils.rs | 9 +- .../llms/src/reducto/ocr/transformation.rs | 47 +- .../llms/src/vertex_ai/ocr/common_utils.rs | 4 +- .../vertex_ai/ocr/deepseek_transformation.rs | 56 +- .../llms/src/vertex_ai/ocr/transformation.rs | 5 +- litellm-rust/crates/model-catalog/Cargo.toml | 1 + .../crates/model-catalog/src/catalog.rs | 6 +- .../crates/model-catalog/src/error.rs | 8 +- .../crates/model-catalog/src/validation.rs | 114 +-- .../tests/registry_validation.rs | 4 +- litellm-rust/crates/python-bridge/AGENTS.md | 7 +- litellm-rust/crates/python-bridge/Cargo.toml | 2 + .../crates/python-bridge/src/cache/mod.rs | 93 ++- .../python-bridge/src/cache/native/v2.rs | 37 +- .../python-bridge/src/cache/python/host.rs | 34 +- .../python-bridge/src/cache/python/service.rs | 19 +- .../python-bridge/src/cache/selection.rs | 2 +- .../crates/python-bridge/src/callable.rs | 44 +- .../crates/python-bridge/src/coercion.rs | 138 ++-- .../crates/python-bridge/src/credentials.rs | 13 +- .../crates/python-bridge/src/diagnostics.rs | 5 +- .../crates/python-bridge/src/errors.rs | 253 ++++++- litellm-rust/crates/python-bridge/src/http.rs | 79 +- .../crates/python-bridge/src/logger/mod.rs | 8 +- .../crates/python-bridge/src/marshal.rs | 130 ++-- .../src/routes/audio_transcription.rs | 10 +- .../src/routes/chat_completions/host.rs | 2 +- .../src/routes/clickhouse_spend.rs | 58 +- .../python-bridge/src/routes/embeddings.rs | 15 +- .../python-bridge/src/routes/inference.rs | 48 +- .../python-bridge/src/routes/messages/host.rs | 76 +- .../python-bridge/src/routes/ocr/document.rs | 113 +-- .../python-bridge/src/routes/ocr/errors.rs | 315 ++++---- .../python-bridge/src/routes/ocr/host.rs | 13 +- .../python-bridge/src/routes/ocr/project.rs | 19 +- .../src/routes/responses/host.rs | 14 +- .../python-bridge/src/routes/token_counter.rs | 26 +- .../python-bridge/src/secrets/config.rs | 4 +- .../crates/python-bridge/src/secrets/error.rs | 84 ++- .../crates/python-bridge/src/secrets/mod.rs | 4 +- .../python-bridge/src/secrets/mutation.rs | 60 +- .../python-bridge/src/secrets/operations.rs | 25 +- .../python-bridge/src/secrets/provider.rs | 12 +- .../python-bridge/src/secrets/runtime.rs | 46 +- .../crates/python-bridge/src/secrets/vault.rs | 189 ++--- .../src/secrets/vault/operation.rs | 10 +- .../crates/python-bridge/src/tokenizer.rs | 123 +++- .../crates/python-compat/src/error.rs | 8 +- .../crates/python-compat/src/literal.rs | 34 +- .../crates/python-compat/src/pickle.rs | 13 +- litellm-rust/crates/secrets-aws/src/auth.rs | 4 +- litellm-rust/crates/secrets-aws/src/error.rs | 4 +- .../secrets-aws/src/secret_manager/read.rs | 17 +- .../secrets-aws/tests/secret_manager/reads.rs | 6 +- litellm-rust/crates/secrets-azure/Cargo.toml | 1 + .../crates/secrets-azure/src/error.rs | 20 +- .../crates/secrets-azure/src/key_vault.rs | 15 +- .../crates/secrets-azure/tests/key_vault.rs | 18 +- .../crates/secrets-cyberark/Cargo.toml | 1 + .../crates/secrets-cyberark/src/error.rs | 63 +- .../crates/secrets-cyberark/src/lib.rs | 4 +- .../secrets-cyberark/src/secret_manager.rs | 30 +- .../src/secret_manager/client.rs | 32 +- .../src/secret_manager/read.rs | 8 +- .../src/secret_manager/write.rs | 8 +- .../tests/secret_manager/configuration.rs | 12 +- litellm-rust/crates/secrets-google/Cargo.toml | 1 + .../crates/secrets-google/src/error.rs | 38 +- .../secrets-google/src/secret_manager.rs | 20 +- .../secrets-google/tests/secret_manager.rs | 6 +- .../crates/secrets-hashicorp/Cargo.toml | 3 + .../crates/secrets-hashicorp/src/config.rs | 6 +- .../crates/secrets-hashicorp/src/error.rs | 122 +++- .../crates/secrets-hashicorp/src/lib.rs | 4 +- .../secrets-hashicorp/src/secret_manager.rs | 30 +- .../src/secret_manager/client.rs | 12 +- .../src/secret_manager/raw.rs | 132 +++- .../src/secret_manager/write.rs | 2 +- .../tests/secret_manager/configuration.rs | 35 +- .../tests/secret_manager/reads.rs | 51 +- .../crates/secrets-types/src/error.rs | 12 + .../crates/secrets-types/tests/cache.rs | 20 +- litellm-rust/crates/secrets/Cargo.toml | 1 + .../crates/secrets/src/compatibility.rs | 6 +- litellm-rust/crates/secrets/src/error.rs | 14 +- litellm-rust/crates/secrets/src/handler.rs | 10 +- litellm-rust/crates/secrets/src/oidc.rs | 20 +- litellm-rust/crates/secrets/tests/aws.rs | 2 +- litellm-rust/crates/secrets/tests/google.rs | 2 +- .../crates/secrets/tests/resolution.rs | 133 ++-- .../crates/spend-clickhouse/Cargo.toml | 1 + .../crates/spend-clickhouse/src/error.rs | 33 +- .../crates/spend-clickhouse/src/insert.rs | 117 ++- .../crates/spend-clickhouse/src/lib.rs | 4 +- .../crates/storage-clickhouse/Cargo.toml | 2 + .../crates/storage-clickhouse/src/error.rs | 70 +- .../crates/storage-clickhouse/src/insert.rs | 13 +- .../crates/storage-clickhouse/src/lib.rs | 8 +- .../crates/storage-clickhouse/src/migrate.rs | 24 +- .../crates/storage-clickhouse/src/read.rs | 26 +- .../storage-clickhouse/tests/migrations.rs | 29 +- .../storage-clickhouse/tests/transport.rs | 105 ++- litellm-rust/crates/testkit/Cargo.toml | 1 + .../crates/testkit/src/agent/configure.rs | 5 +- litellm-rust/crates/testkit/src/error.rs | 14 +- .../crates/testkit/src/install/fetch.rs | 4 +- .../crates/testkit/src/install/mod.rs | 26 +- litellm-rust/crates/testkit/src/session.rs | 12 +- .../crates/token-counter-fast/Cargo.toml | 1 + .../crates/token-counter-fast/src/error.rs | 19 +- .../crates/token-counter-fast/src/tiktoken.rs | 10 +- .../token-counter-huggingface/src/error.rs | 17 +- .../token-counter-huggingface/src/lib.rs | 4 +- .../crates/token-counter-tiktoken/Cargo.toml | 1 + .../token-counter-tiktoken/src/error.rs | 49 +- .../crates/token-counter-tiktoken/src/lib.rs | 29 +- .../token-counter-tiktoken/src/ranks.rs | 242 ++++--- litellm-rust/crates/token-counter/Cargo.toml | 1 + .../crates/token-counter/src/counter.rs | 10 +- .../crates/token-counter/src/error.rs | 91 ++- litellm-rust/crates/token-counter/src/fast.rs | 12 - .../crates/token-counter/src/huggingface.rs | 11 - litellm-rust/crates/token-counter/src/lib.rs | 2 +- .../crates/token-counter/src/python_json.rs | 16 +- .../crates/token-counter/src/tiktoken.rs | 18 +- .../crates/token-counter/src/tools.rs | 6 +- .../crates/token-counter/src/types.rs | 4 +- .../token-counter/tests/token_counter.rs | 20 +- 406 files changed, 9261 insertions(+), 4747 deletions(-) create mode 100644 litellm-rust/.agents/skills/rust-error/SKILL.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/boundaries.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/classification.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/conversions.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/gateway.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/messages-and-causes.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/pyo3.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/recipes.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/shape.md create mode 100644 litellm-rust/.agents/skills/rust-error/references/testing.md create mode 100644 litellm-rust/.agents/skills/rust-gateway/SKILL.md create mode 100644 litellm-rust/.agents/skills/rust-gateway/references/requests.md create mode 100644 litellm-rust/.agents/skills/rust-gateway/references/surfaces.md create mode 100644 litellm-rust/.agents/skills/rust-gateway/references/testing.md delete mode 100644 litellm-rust/crates/auth-types/tests/error.rs create mode 100644 litellm-rust/crates/callbacks-legacy-python/src/error.rs create mode 100644 litellm-rust/crates/cost/src/error.rs create mode 100644 litellm-rust/crates/error/Cargo.toml create mode 100644 litellm-rust/crates/error/src/lib.rs delete mode 100644 litellm-rust/crates/framer/src/framer.rs create mode 100644 litellm-rust/crates/gateway-mcp/src/request.rs delete mode 100644 litellm-rust/crates/http/src/transport.rs delete mode 100644 litellm-rust/crates/llms/src/anthropic/chat/handler.rs diff --git a/litellm-rust/.agents/skills/rust-error/SKILL.md b/litellm-rust/.agents/skills/rust-error/SKILL.md new file mode 100644 index 00000000000..2f02394c494 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/SKILL.md @@ -0,0 +1,35 @@ +--- +name: rust-error +description: Define, wrap, convert, classify, report or test Rust error types in litellm-rust with thiserror. Use when adding an error type or variant, mapping one crate's error into another, turning an error into a gateway response or Python exception, deciding retry or fault from an error, or replacing call sites that build errors from strings +--- + +# Rust errors + +`crates/http/src/error.rs` is the reference implementation. Read the reference for the part you are touching before editing, each one carries the reasoning and worked examples behind its rules + +- [shape](references/shape.md) + - One top-level `Error` per crate in `src/error.rs`. A public enum by default, an opaque struct over a private `Repr` enum when the variants must stay private. No `#[non_exhaustive]` + - A variant is a failure mode a caller handles differently. Provider names and guidance are data, never variants +- [messages and causes](references/messages-and-causes.md) + - Every message is written once, in `#[error(...)]`, and describes its own layer only. Call sites pass typed facts, never a string + - The cause is the `source()`, never part of the message. Keep it typed, never stringify or drop it + - `reqwest::Error` enters our types only through `litellm_http::Error`, which strips its URL + - Anything shown to someone else is rendered with `litellm_error::Report`, never `to_string()` +- [conversions](references/conversions.md) + - One `From` per source and target pair, only when the conversion needs no context. It classifies and sanitizes once + - Nest a lower crate's error in one variant. Translate when the meaning changes, never mirror its variants +- [classification](references/classification.md) + - Callers ask predicates named after their decision, they do not match nested variants. Predicates match exhaustively + - Classification states facts. Retry and fallback policy lives with the caller that knows idempotency +- [boundaries](references/boundaries.md) + - An error becomes text for someone else only in one mapping function per error type, in the gateway crates, the Python bridge and host-python. The inference gateway and the bridge decide by the same predicates +- [gateway](references/gateway.md) + - One `Error` with `IntoResponse` per gateway crate that serves a router. Every failure flows through the same `From` chain, is classified once in `status()` and rendered by surface and framing +- [pyo3](references/pyo3.md) + - Two `create_exception!` classes with the payload in `args`, `ValueError` for the caller's fault and `RuntimeError` otherwise, decoded in `failures.py` +- [testing](references/testing.md) + - Test predicates, chain shape and redaction on errors built the way production builds them +- [recipes](references/recipes.md) + - Common tasks walked through end to end + +The workspace uses thiserror 2 for every error type, and no library crate uses `anyhow` or `Box` in a public signature. The [derive docs](https://docs.rs/thiserror/latest/thiserror/) are the authority on macro behavior, and the [`std::error::Error` docs](https://doc.rust-lang.org/std/error/trait.Error.html) on what `Display` and `source()` mean diff --git a/litellm-rust/.agents/skills/rust-error/references/boundaries.md b/litellm-rust/.agents/skills/rust-error/references/boundaries.md new file mode 100644 index 00000000000..253cf52a7db --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/boundaries.md @@ -0,0 +1,39 @@ +# Boundaries + +A boundary is where an error becomes something another program reads: an HTTP response, a Python exception, a stored or logged message. Each boundary has one mapping function per error type it receives, and nothing outside those functions formats an error for someone else + +| Boundary | Mapping | Reference | +|---|---|---| +| Inference HTTP response | `Error::status()` and its renderers in `crates/gateway-inference/src/error.rs` | [gateway](gateway.md) | +| Other gateway surfaces | the `IntoResponse` of each `crates/gateway-*` crate's `Error` | [gateway](gateway.md) | +| Python bridge, inference and OCR routes | `route_failure` in `crates/python-bridge/src/errors.rs` and in `crates/python-bridge/src/routes/ocr/errors.rs` | [pyo3](pyo3.md) | +| Python bridge, everything else | one `From for PyErr` or mapping function per crate error, all in `crates/python-bridge` | [pyo3](pyo3.md) | +| Python host runtime | `From for PyErr` in `crates/host-python/src/error.rs` | [pyo3](pyo3.md) | + +## Asking the nested error + +The inference gateway and the Python bridge's route mapping both receive a nested route error and decide by the same predicates, so a new failure lands correctly on both once its predicates answer right + +| Predicate | Gateway | Bridge | +|---|---|---| +| `is_unsupported()` | 501 | `ValueError`, since `is_request()` also answers true | +| `upstream_response()` | the provider's status and body | `RustUpstreamError` | +| `is_missing_credential()` | 401 | no class of its own, a `RuntimeError` because `is_request()` answers false | +| `is_request()` | 400 | `ValueError` | +| none of these | 500 | `RuntimeError` | + +If you find yourself adding an arm for your variant at a boundary, check whether a predicate is missing first + +The two boundaries are not identical. The gateway has variants of its own for body parsing, an unknown model and an unsupported path, and the OCR bridge has Python parity arms such as `FileNotFoundError` for a missing file. Those arms match the boundary's own variants or the top level of a public enum, never a nested error's variants. A missing API key is 401 at the gateway and `RuntimeError` at both bridges, pinned by tests to what Python's own path does + +## What reaches the client + +A boundary renders anything that is not a provider answer with `litellm_error::Report`, so the reader sees every layer of the chain once. That is why messages are written for a reader, each layer leaves its cause to `source()`, and reqwest URLs are stripped on entry. A boundary that must not show the chain, such as the bridge's TLS settings error that names the setting but not the file, renders fixed text of its own and says so in its test + +Provider bodies are passed through intact for gateway and Python error mapping. `litellm_http::Error::upstream(status, body)` retains the complete body, but its `Display`, `Debug` and `Report` diagnostics show only the first 256 characters and mark the cut. Do not log the raw body or `RustUpstreamError` args. OCR's `Provider` error also keeps the whole body for Python's `get_error_class`, bounded by the connection's response size limit + +Never format `Debug` of an error into a response or a Python exception. A type whose sources may hold secrets derives `veil::Redact`, so its `Debug` is safe to log + +## Logging + +Propagate with context, and let the terminal handler own the one log or trace event for a failed call. Do not log at every `?`, and do not log an error you also return. Diagnostic tracing excludes raw error strings and bodies, see [`rust-tracing`](../../rust-tracing/SKILL.md) diff --git a/litellm-rust/.agents/skills/rust-error/references/classification.md b/litellm-rust/.agents/skills/rust-error/references/classification.md new file mode 100644 index 00000000000..59c3a1eb3f0 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/classification.md @@ -0,0 +1,66 @@ +# Classification + +## Ask, don't match + +Downstream crates ask a lower error questions instead of matching its nested variants. `litellm_http::Error` answers these + +| Predicate | Caller decides | +|---|---| +| `dispatched()` | whether retrying elsewhere could repeat a billed call | +| `is_request()` | the caller's input is at fault, so 400 and no reroute | +| `is_response()` | the provider answered with something we cannot use | +| `upstream_response()` | the provider's own status and body to pass through | +| `status()` | an HTTP status to report | +| `is_connect()` | the connection was never made, for a host's own connect-error mapping | +| `tls_source()` | which TLS setting to blame in a config error | +| `is_computed_header()` | the caller forwarded a header the signer owns, which `litellm_inference::Error` turns into `Unsupported` | + +Upper types add the questions their callers ask. `litellm_inference::Error` and the OCR error answer `is_missing_credential()` and `is_unsupported()`, which the gateway asks instead of reaching into `Auth` or `Unsupported` + +```rust +// reaches through two crates' variants and breaks when either changes +Self::Route(InferenceError::Auth(litellm_auth::Error::MissingApiKey { .. })) => StatusCode::UNAUTHORIZED, + +// asks the question it cares about +Self::Route(error) if error.is_missing_credential() => StatusCode::UNAUTHORIZED, +``` + +A predicate is an exhaustive `match` with no `_` arm, never `matches!`, so a new variant does not compile until it has answered every question on purpose + +```rust +pub fn is_request(&self) -> bool { + match &*self.0 { + Repr::Request(_) | Repr::BlockedUrl | Repr::DownloadTooLarge => true, + Repr::Connect(_) | Repr::Network(_) | Repr::Timeout(_) | Repr::Status { .. } => false, + // ... every other variant, listed by name + } +} +``` + +Add a predicate when a real caller needs a new distinction, and name it after the fact the caller needs, in the caller's words, not after the variant's spelling. An upper type's predicates call through to the nested error, as in `Self::Http(error) => error.is_request()` and `Self::Auth(error) => error.is_missing_credential()` + +A public enum may be matched one level down by the type that owns the variant, such as the gateway matching its own `Unsupported` or the OCR bridge matching `FileRead`. Going a second level down, into a variant of a variant, is what to avoid, the gateway used to do that for `MissingApiKey` and `Unsupported` + +## Facts, not policy + +A predicate reports what happened. The decision to retry, reroute or fall back depends on things the error cannot know, such as whether the request is idempotent, whether its body can be replayed, the deadline and the attempt budget. That decision lives with the caller. `dispatched()` exists for the router's reroute decision, so every transport failure answers it even before that caller is wired + +A timeout is the main trap. A request that timed out may already have reached the provider and been billed, so `Timeout` answers `dispatched() == true` and is not safe to retry as if it were a connect failure. Only a failure that provably happened before any byte reached the provider answers `false`: connecting, building or signing the request, and media downloads, which fetch the caller's document before the provider call is made + +```rust +pub fn dispatched(&self) -> bool { + match &*self.0 { + Repr::Network(_) | Repr::Timeout(_) | Repr::Status { .. } => true, + Repr::Connect(_) | Repr::RequestBody(_) | Repr::Download(_) | Repr::BlockedUrl => false, + // ... every other variant, listed by name + } +} +``` + +When the transport cannot tell whether a request reached the provider, it must answer `true`. Wrongly answering `false` risks a duplicate billed call, wrongly answering `true` only costs a reroute + +Keep one predicate per question. `status()` currently mixes the provider's status, the media download's status and a synthesized 408 for timeouts, so a caller that wants "did the provider answer" must use `upstream_response()` instead. Do not add more meanings to an existing predicate, add a new one + +## Equality + +`Arc` errors compare with `Arc::ptr_eq`, so a clone equals its original and two separately built errors never compare equal. That exists only so a wrapping enum such as `litellm_inference::Error` can keep its `PartialEq` derive. Do not give a new type pointer equality for any other reason, and do not rely on it in tests. Assert predicates instead, see [testing](testing.md) diff --git a/litellm-rust/.agents/skills/rust-error/references/conversions.md b/litellm-rust/.agents/skills/rust-error/references/conversions.md new file mode 100644 index 00000000000..db46e4e3aa9 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/conversions.md @@ -0,0 +1,92 @@ +# Conversions + +## One From per source and target pair + +A `From for Ours` exists when the conversion is obvious, infallible and needs no context from the call site. It does all classification and sanitizing for that pair, so every call site is `?` or `.map_err(Ours::from)` + +```rust +impl From for Error { + fn from(error: reqwest::Error) -> Self { + let error = error.without_url(); + if error.is_timeout() { + Repr::Timeout(error).into() + } else if error.is_connect() || error.is_builder() { + Repr::Connect(error).into() + } else { + Repr::Network(error).into() + } + } +} +``` + +A call site never re-derives that classification + +```rust +// every route used to repeat this, and each drifted a little +.map_err(|err| { + if err.is_connect() || err.is_builder() { + Error::Transport(Connect(err.to_string())) + } else { + Error::Transport(Network(err.to_string())) + } +})? + +// now +.map_err(litellm_http::Error::from)? +``` + +The same foreign type may convert into several of our types, one `From` each. What is not allowed is two places converting the same pair differently + +## When the call site knows something the From cannot + +One `io::Error` can mean "reading the CA bundle failed" or "resolving the media host failed". A global `From` cannot know which, so there is none. Name the operation at the call site instead + +```rust +.map_err(Repr::Resolve)? +.map_err(|source| Repr::Read { path, tls_source, source })? +``` + +Do not drop meaningful context just to use `?` + +## Reclassify once, in the shared From + +When a lower error needs a different meaning for every caller, do it in the upper type's `From`, not per route + +```rust +impl From for Error { + fn from(error: litellm_http::Error) -> Self { + if error.is_computed_header() { + return Self::Unsupported("request forwards a header AWS SigV4 computes"); + } + Self::Http(error) + } +} +``` + +All four inference routes classify a header the signer computes the same way, because none of them does it themselves + +## Nest, translate or add context, never mirror + +| The upper layer | Do | Example | +|---|---|---| +| has nothing to add | nest in one transparent variant and delegate predicates | `Self::Http(error) => error.is_request()` | +| changes what the failure means | translate to an upper variant, keep the cause attached when it helps | computed header -> `Unsupported` | +| knows something the lower one does not | a variant with the extra fields plus the cause as its source | `Read { path, tls_source, source }` | + +Never re-declare a lower crate's variants in the upper type with an upper-layer name and a 1:1 mapping function. OCR once did that for every media download failure, `MediaError::BlockedUrl => Error::BlockedDocumentUrl` and so on, which doubled every change and let the two lists drift. + +## Third-party and host faults + +Converting a fault from a crate we do not own into a string variant is still flattening. Prefer a variant that carries the fault, and when an existing `ErrorDetail` constructor fits, use it + +```rust +// flattens +.map_err(|error| Error::InvalidRequest(error.to_string().into())) + +// keeps the cause typed +.map_err(|error| Error::InvalidRequest(ErrorDetail::invalid("header", error))) +``` + +## Through a foreign error type + +A trait we implement for another crate sometimes forces its error type on us, such as `io::Error` from a body stream or `sqlx::Error` from a driver. Wrap ours as that type's source (`io::Error::other(error)`, `azure_core::Error::new(kind, error)`) and recover it with `downcast_ref` on the other side, never `with_message(error.to_string())` or `from_msg`. The error must come back out as the same value it went in as, so one failure classifies the same way on every path diff --git a/litellm-rust/.agents/skills/rust-error/references/gateway.md b/litellm-rust/.agents/skills/rust-error/references/gateway.md new file mode 100644 index 00000000000..1ec8ecd0acb --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/gateway.md @@ -0,0 +1,56 @@ +# Gateway responses + +The gateway is where a Rust error becomes an HTTP response. This page is about the path an error takes to get there and the one place it is classified. The gateway crates' own conventions, such as which envelope each API surface uses, which extractors exist and how the router is composed, live in the [`rust-gateway`](../../rust-gateway/SKILL.md) skill + +## The flow + +```text +reqwest / provider failure + -> litellm_http::Error From, strips the URL, classifies once + -> litellm_inference::Error::Http(_) From, one transparent variant, predicates delegate + -> gateway_inference::Error::Route(_) From, one variant per nested crate + -> handler returns Result<_, Error> every `?` ends here + -> IntoResponse status() picks the code, a renderer builds the body +``` + +Each hop is one `From` that nests the lower error in one variant, see [conversions](conversions.md). No hop re-derives what the lower crate already decided, and nothing between the provider and `IntoResponse` formats an error for the client. That is what makes "one `Error` with `IntoResponse` per gateway crate that serves a router" work: a handler can `?` any failure from any depth and the response comes out right + +## Classify once + +`status()` is pure and runs once per error. Variants the gateway declares itself, such as a bad body, an unknown model or an oversized body, are matched at the top of its own enum. A nested route error goes to `route_status` and an OCR error to `ocr_status`, which ask the predicates [boundaries](boundaries.md) lists + +| Answer | Status | +|---|---| +| `is_unsupported()` | 501, the gateway matched and parsed the request but core cannot serve it, nothing reached the provider | +| `upstream_response()` | the provider's own status, 502 when it is not a valid code | +| `is_missing_credential()` | 401 | +| `is_request()` | 400, for a route error | +| `http_status_code()` | the transport's status, for an OCR error, such as 408 for a timeout | +| otherwise | 500 | + +Do not reach into a nested enum's variants from the gateway. When a new nested failure needs a different status, add a predicate on the nested error, see [classification](classification.md) + +## Render by surface and framing + +Rendering is separate from classification. The surface picks the envelope and the moment picks the framing, a JSON body before the headers went out and an `event: error` SSE frame after. Both framings of one surface render the same status and `Report` message, so a streaming failure and a JSON failure of the same error agree on everything but framing + +| Surface | JSON | SSE | +|---|---|---| +| Anthropic, `/v1/messages` | `body(request_id)` | `sse_frame()` | +| OpenAI, chat, responses and the rest | `openai_response()` | `responses_sse_frame()`, the Responses API `error` event | + +The message of anything that is not a provider answer is the `litellm_error::Report` of the error, and in the OpenAI and Anthropic JSON envelopes the error `type` string is derived from the status by `error_type()`, so a new failure never names its own type. The Responses `error` event has the fixed type `error` and the status as its `code`. The Anthropic envelope passes a provider body already in its shape through untouched and extracts the message from anything else. The OpenAI envelope always wraps, because that is what the Python proxy returns there + +## Every way in uses the same Error + +A request can fail before the handler runs, in the handler, or after the headers went out. All three end in the same `Error`, so they get the same classification and envelope + +- An extractor sets `type Rejection = Error`, so a bad body is the surface's envelope rather than axum's plain text +- A path the gateway knows but does not serve returns `Error::Unsupported(path)` from one handler, so the client gets a 501 in the surface's envelope rather than a bare 404 +- A stream that fails after its headers went out hands the surface's SSE frame of `Error::from(error)` to `litellm_host_http::Sse::new`. `litellm_host_http::Error` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, and the `From` maps the second to `Internal` + +The axum hooks behind each line are in [`rust-gateway`](../../rust-gateway/SKILL.md) + +## Testing + +`status()` and `body()` are pure, so unit tests call them on errors built the way production builds them, one `#[case]` per fault, as `status_follows_who_is_at_fault`, `an_anthropic_provider_body_passes_through` and `other_failures_get_the_anthropic_envelope_with_their_message` in `crates/gateway-inference/src/error.rs` do. Assert the status and the fields the surface promises, not the whole body, except where passing the body through untouched is the contract. Driving a whole router belongs to the gateway crate's own tests, see [`rust-gateway`](../../rust-gateway/references/testing.md) diff --git a/litellm-rust/.agents/skills/rust-error/references/messages-and-causes.md b/litellm-rust/.agents/skills/rust-error/references/messages-and-causes.md new file mode 100644 index 00000000000..80017423a42 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/messages-and-causes.md @@ -0,0 +1,112 @@ +# Messages and causes + +## Messages belong to the variant + +Every message template is written once, in the variant's `#[error(...)]`. A caller passes the typed facts the template needs and nothing else + +```rust +// none of these may appear at a call site +Error::Network("WebSocket handshake returned a wrong Sec-WebSocket-Accept".into()) +Error::Request(format!("bad header {name}")) +Error::Signature(error.to_string()) + +// the call site names what happened, the variant owns the words +Repr::Handshake(HandshakeFault::Accept).into() +Error::signature(error) +Error::NonAsciiHeader { name } +``` + +When a check fails for a distinct reason, that reason is a variant or a field, never a string inside a generic variant. A `Message(String)` variant, or a `From` / `From<&str>` into an error, is the hole every stringly call site goes through, so neither exists + +Format arguments can call functions and reach into fields, so a hand-written `Display` is rarely needed + +```rust +#[error("could not read {}", path.display())] +#[error("invalid model: {provider} has no model {model:?} - use one of: {}", supported.join(", "))] +``` + +Keep messages short and lowercase, with no trailing period, unless Python parity pins the exact text. A reporter appends `: {cause}` after them. Never format `Debug` (`{:?}`) of an error, or of a type with no `Display`, into a message + +## The cause is the source, never the message + +This is the rule from the [`std::error::Error` docs](https://doc.rust-lang.org/std/error/trait.Error.html): an error's `Display` describes that layer only, and the error underneath is returned by `source()`. thiserror wires `source()` from a field marked `#[source]`, a `#[from]` field, or a field named `source` + +```rust +// prints the cause twice under any reporter +#[error("could not read {}: {source}", path.display())] +Read { path: PathBuf, source: io::Error }, +#[error("invalid JSON: {0}")] +Json(#[from] serde_json::Error), + +// this layer's words, the cause is the source +#[error("could not read {}", path.display())] +Read { path: PathBuf, source: io::Error }, +#[error("invalid JSON")] +Json(#[from] serde_json::Error), +``` + +| Field | thiserror does | Use for | +|---|---|---| +| `source: E` or `#[source] cause: E` | exposes it as `source()` | a cause with context fields next to it | +| `#[from] E` | implies `#[source]` and adds `From` | context-free wrapping, see [conversions](conversions.md) | +| `#[error(transparent)]` | forwards both `Display` and `source()` to the inner value | nesting another of our errors that adds no text | +| no attribute, any other name | nothing, the field is plain data | facts the message renders, such as a path or a status | + +`#[error(transparent)]` is an invisible layer, not an extra chain node, so a transparent wrapper's `source()` is the inner value's `source()` + +A field that holds an error but is neither rendered nor a source hides that error from every reader. Every error-typed field is a source. A fieldless reason enum, such as `HandshakeFault` or `PagesFault`, is data and not a cause, so the variant renders it in its message + +`source()` is one chain, so an error with several causes, such as `CredentialChain` trying each credential in turn, makes the most relevant one its source and renders the others in its message, each with `Report` + +## Keep causes typed + +Store the lower-level error itself, never its string, and never drop it with `map_err(|_| ...)` + +| Flattening or dropping loses | Because | +|---|---| +| the chain | a `String` has no `source()` | +| downcasting | the type is gone | +| redaction | `reqwest::Error`'s `Display` includes the request URL, which can carry credentials in the path or query | + +If the cause is not `Clone` and the error must be, hold it as `#[source] litellm_error::Source`, a cloneable `Arc` that is transparent to `Display` and `source()`, or make the crate's error opaque. `litellm_error::find::(&error)` walks the chain and looks through `Source`, so it is how anyone reaches a typed cause. A third-party `Box` converts with `Source::from`, and an `anyhow::Error` a dependency returns goes through `Box` first, `Source::from(Box::::from(error))`, and a dependency's own boxed alias such as `tokenizers::Error` is held as it is. A source boxed to keep the variant small is a `Box` node of the chain, so it is reached as `find::>`. Never put `#[error(transparent)]` on a `Source`, the erased error would then be no node of the chain and `find` could not reach it. Do not stringify or drop a cause to get `Clone` or `PartialEq` + +```rust +// loses the type and leaks the URL +.map_err(|error| Error::Network(error.to_string())) +// loses everything +.map_err(|_| Error::Transport) + +// keeps the cause, the From impl strips the URL +.map_err(litellm_http::Error::from) +``` + +A cause that must not be shown at all, such as a secret manager's response, is still a source. The type derives [`veil::Redact`](https://docs.rs/veil) with `#[redact]` on the field, so `Debug` hides it, and the boundary decides how much of the chain to show, see [boundaries](boundaries.md) + +## Strip reqwest URLs on entry + +Every `reqwest::Error` enters our types through `From for litellm_http::Error`, which calls `without_url()`. A crate that talks HTTP holds `litellm_http::Error`, never `reqwest::Error`. When a dependency's own error wraps a `reqwest::Error`, as vaultrs and rustify do, one `From` takes the reqwest error out and converts it, as `secrets-hashicorp` does. That way so every later `Display`, `Debug` or chain walk is safe from that URL + +`without_url()` only removes the URL reqwest attached. It does not scrub headers, response bodies, provider messages, file paths or anything a variant stores next to it. Those need their own policy, see [boundaries](boundaries.md) + +## Rendering + +Anything that shows an error to someone else renders the whole chain with `litellm_error::Report`, which writes each layer once joined by `: `, the single-line form of the unstable [`std::error::Report`](https://doc.rust-lang.org/std/error/struct.Report.html). `to_string()` of an error is only its top layer and is never what a client, a Python exception or a log line shows + +```rust +let message = litellm_error::Report::new(&error).to_string(); +// "could not reach the provider: error sending request: client error (Connect): tcp connect error: Connection refused" +``` + +## #[from] limits + +`#[from]` only works on a variant whose only fields are the source and an optional backtrace. A variant that needs context such as a path, a phase or a `TlsSource` takes its cause through an explicit `map_err` or a named constructor + +```rust +std::fs::read(&path).map_err(|source| Repr::Read { + path: path.clone(), + tls_source: TlsSource::CaBundle, + source, +})?; +``` + +`#[backtrace]` and automatic `provide()` need a nightly compiler and are not used here diff --git a/litellm-rust/.agents/skills/rust-error/references/pyo3.md b/litellm-rust/.agents/skills/rust-error/references/pyo3.md new file mode 100644 index 00000000000..81a7a515726 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/pyo3.md @@ -0,0 +1,74 @@ +# Python bridge + +`crates/python-bridge` turns a Rust error into a Python exception through one mapping function per error type. A `#[pyfunction]` returns `Err(PyErr)` and pyo3 raises it, so those functions are the whole boundary + +| Error | Mapping | +|---|---| +| `litellm_inference::Error` from a route | `errors::route_failure`, with `route_error_to_pyerr` as its shorthand for routes that set no marker | +| the OCR error | `route_failure` and `to_pyerr` in `routes/ocr/errors.rs` | +| token counter, cache, spend, secrets, argument checks | one `From for PyErr` or mapping function next to the code that registers the functions | + +A route host never builds a `PyErr` from a route error itself. It calls the mapping, and the mapping is the only place that knows the exception classes and their attributes + +## Exception classes + +| Class | Meaning | Payload | +|---|---|---| +| `RustUpstreamError` | the provider call was issued and the provider answered with a failure | `args == (status, body)` and a `headers` attribute of `(name, value)` pairs, always set | +| `RustBridgeDeclined` | the route stopped before calling the provider, so the host may run its own path | `args == (message,)` | +| a standard class from `pyo3::exceptions` | whichever Python class means the same thing, such as `ValueError` or `TypeError` for the caller's input, `FileNotFoundError` or `OSError` for a file, `KeyError` or `OverflowError` where Python raises them, and `RuntimeError` for anything else | the rendered chain, or a parity text where Python pins one | + +The two native classes come from `create_exception!` in `crates/python-bridge/src/errors.rs` and are exported with `#[pymodule_export]`, which is how the [pyo3 guide](https://github.com/PyO3/pyo3/blob/main/guide/src/exception.md) makes an exception importable. Their docstrings are the contract Python reads. `errors::upstream_error(py, status, body, headers)` is the one constructor of `RustUpstreamError`, so the payload cannot differ between routes + +Python decodes a `RustUpstreamError` in `litellm/rust_bridge/failures.py`, which validates `args` and `headers` with a `TypeAdapter` and rebuilds the provider's `httpx.Response`. `litellm/rust_bridge/runtime.py` catches `RustBridgeDeclined` and reads its reason. `_raise_upstream` in `runtime.py` still reads `args` a second time by hand for the `attempt` path, which is the one decode to fold into `failures.py`. A new fact goes into `args` or `headers`, and the decoder learns to read it in the same change + +Two routes also flag a request fault with a boolean attribute, read by their Python route module: `ocr_request_format_error` in `host/routes/ocr.py` and `messages_request_error` in `host/routes/messages.py`. OCR additionally sets `status_code` and `message` for Python parity, and `native_settings_error`, which only the OCR host reads back to tell a settings failure from the caller's exception. Do not add another attribute, nothing on the Python side types them + +## Why args, not a typed subclass + +The bridge builds with pyo3's `abi3-py310` feature. Under abi3, `#[pyclass(extends = PyException)]` is only allowed on Python 3.12 and newer, so a class with typed fields is not available. Even where it is allowed, pyo3 passes an empty tuple to the native base's `tp_new`, so `.args` and `str()` would need manual overrides. The payload therefore travels in `args`, which a `create_exception!` class fills from `new_err` + +## Mapping order + +A Python exception raised inside the call, such as a secret manager callback's, is returned first as it is. Then the provider's answer passes through as `RustUpstreamError`. On `/v1/messages` a request the route rejected becomes a `ValueError` flagged with `messages_request_error`. Everything else is split by `is_request()` into `ValueError` or `RuntimeError`, with the message rendered by `litellm_error::Report`. A missing API key is a `RuntimeError` because `is_request()` answers false for it, and `missing_api_key_stays_a_runtime_error_while_other_auth_failures_are_value_errors` pins that + +OCR puts Python parity arms in front of that order, each matching a top-level variant of its own public enum + +| Arm | Exception | +|---|---| +| the provider answered | `RustUpstreamError` with the provider's headers | +| `RequestFormat` | `ValueError` flagged with `ocr_request_format_error` | +| `FileRead` whose cause is `NotFound` | `FileNotFoundError` naming the path | +| other `FileRead` | `OSError` with the io message | + +Both routes ask `litellm_auth::Error::is_request()` through their own `is_request()`, so a missing key or API base is a `RuntimeError` on both, and a missing setting the caller can pass, such as a region, is a `ValueError` + +Use the standard class whose Python meaning matches. `PyFileNotFoundError` for a missing file is right, a `RuntimeError` whose text says the file is missing is not. `import_exception!` exists for an exception defined in Python, none is used today because `failures.py` maps to LiteLLM's public exceptions through `litellm.exception_type` + +## Declining + +`RustBridgeDeclined` means no provider call was made and the host may retry on its own path. It is raised before dispatch, by the tokenizer, embeddings and token counter routes. Never raise it for a failure after the request went out, the host would repeat a billed call. `route_failures_are_terminal` in `errors.rs` pins that for the inference routes + +## Rules + +- Never format `Debug` of an error into a `PyErr` +- Never build a `PyErr` from a string at a call site. A failed argument check is a variant of a typed error, as `ProjectionError` in `coercion.rs` is, mapped once +- A Python exception raised inside the call stays that exception. Return it as is, never wrap it in a Rust message or stringify it into a new `ValueError`. The one policy exception is a caller-supplied callable, whose failure `callable::wrap_failure` reports as a `RuntimeError` with the original as `__cause__` +- A `panic!`, `unwrap` or `expect` inside a `#[pyfunction]` or `#[getter]` reaches Python as `pyo3_runtime.PanicException`, which is not an `Exception` subclass. Model the failure as a value instead +- When the mapping itself calls Python, as `setattr` does, propagate its failure with `?` rather than discarding it with `.ok()` + +## Testing + +Build the Rust error the way production builds it, map it, and assert the contract Python sees. `Python::initialize()` once, then `Python::attach` + +```rust +Python::initialize(); +Python::attach(|py| { + let failure = route_error_to_pyerr(litellm_http::Error::upstream(429, "slow down".into()).into()); + assert!(failure.is_instance_of::(py)); + let args: (u16, String) = failure.value(py).getattr("args").unwrap().extract().unwrap(); + assert_eq!(args, (429, "slow down".into())); +}); +``` + +One `#[case]` per fault class, asserting `is_instance_of` for the class and `extract` for the payload. The Python side of the contract is tested in `tests/unit/rust_bridge`, through `failures.py` diff --git a/litellm-rust/.agents/skills/rust-error/references/recipes.md b/litellm-rust/.agents/skills/rust-error/references/recipes.md new file mode 100644 index 00000000000..80ba1512f77 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/recipes.md @@ -0,0 +1,64 @@ +# Recipes + +## A new failure in litellm_http + +Say media downloads must reject a `Content-Type` outside an allow-list + +1. Add a `Repr` variant with typed facts and the message, `#[error("media type {0:?} is not allowed")] DisallowedMediaType(String)` +2. Return it where the check fails, `return Err(Repr::DisallowedMediaType(content_type.to_owned()).into())` +3. Add an arm to every predicate's exhaustive `match`, the compiler lists them, the way its neighbor `DownloadTooLarge` answers. A media download happens before the provider call, so it is not in `dispatched()`. The caller chose the URL, so it is in `is_request()` +4. Leave both boundaries alone. With those answers the gateway returns 400 and Python raises `ValueError` +5. Test it with a mock server answering `Content-Type: application/x-msdownload` and assert `error.is_request()` + +## A provider-specific failure + +A second OCR provider needs a missing-key error whose text names its environment variable and how to set it + +1. Look for an existing provider-neutral variant whose template fits. `litellm_auth::Error::MissingApiKey { provider, environment_variable }` renders `Missing {provider} API Key - Set ...` +2. If none fits, find what varies between providers with the same failure, here the environment variable and the guidance after the dash, and add one neutral variant with those as fields, `#[error("Missing {environment_variable} - {guidance}")] MissingCredential { environment_variable: &'static str, guidance: &'static str }` +3. Pass the provider's values from its transformation. The next provider passes its own and adds no variant, and `is_missing_credential()` lists `MissingCredential { .. }` once + +## Wrapping a new lower crate in litellm_inference + +Say routes start calling `litellm_cache` and its errors must surface + +1. Add one variant, `#[error(transparent)] Cache(#[from] litellm_cache::Error)` +2. Extend every predicate's `match` with an arm that calls through to the cache error's own predicates, adding a predicate there if it has none +3. Do not copy the cache error's variants into `litellm_inference::Error` + +## A foreign error that needs context + +Reading a file fails with `io::Error`, and the message must name the file + +1. A variant with the context and the cause, `Read { path: PathBuf, source: io::Error }` with `#[error("could not read {}", path.display())]` +2. Build it with `map_err(|source| Repr::Read { path: path.clone(), source })`, not a `From` +3. If the error must be `Clone`, the crate's error is opaque or the field is `litellm_error::Source` + +## Migrating a stringly error + +Call sites look like `Error::Network(error.to_string())`, `Error::Request(format!(...))` or `.map_err(|_| Error::Unavailable)` + +1. List every call site of the string variant and group them by what actually failed +2. Each group becomes a variant holding the typed cause or facts. Groups no caller tells apart share one +3. Where a foreign error was stringified with classification logic around it, move that logic into the `From` and turn the call sites into `?` +4. Delete the string variant. If it cannot be deleted yet, nothing new may construct it +5. Re-run the boundary tests, since messages a client sees may change + +## Moving a crate to source chains + +A variant reads `#[error("invalid JSON: {0}")] Json(#[from] serde_json::Error)` or stores `cause` and renders `{cause}` + +1. Drop the cause from the template, `#[error("invalid JSON")]` +2. Make the cause the source, `#[from]`, `#[source]` or a field named `source` +3. Find what renders this error for someone else and make it use `litellm_error::Report` +4. Update tests that asserted the cause in `to_string()` to assert it in the report + +## Making a public enum opaque + +Downstream crates match nested variants, or a variant needs a non-`Clone` cause + +1. Rename the enum to `pub(crate) enum Repr` and add `pub struct Error(Arc)` with `#[error(transparent)]` and `From` +2. For each downstream `match`, find the question it asks and add a predicate. Replace the `match` with the predicate +3. For each downstream construction, add a constructor taking typed data, or move the failure into this crate +4. Swap test assertions to predicates and add the `#[cfg(test)] repr()` accessor for in-crate tests +5. Implement `PartialEq` with `Arc::ptr_eq` only if a wrapping enum's derive needs it diff --git a/litellm-rust/.agents/skills/rust-error/references/shape.md b/litellm-rust/.agents/skills/rust-error/references/shape.md new file mode 100644 index 00000000000..4d51c2cee6b --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/shape.md @@ -0,0 +1,105 @@ +# Error shape + +## One top-level Error per crate + +A crate's errors live in `src/error.rs` and are re-exported from `lib.rs` as `Error`. A caller of any public function in the crate should only ever see `Result<_, crate::Error>`, so one `?` and one set of predicates cover the whole crate + +This deliberately departs from the advice to put an error type next to each unit of fallibility ([Sabrina Jewson, "Modular Errors"](https://sabrinajewson.org/blog/errors)). Our callers do not branch on which function failed, they ask the predicates in [classification](classification.md), and the boundaries map one type per crate. Splitting by function would multiply those predicates and mappings without giving any caller a decision it lacks. Layering, the other half of that advice, is kept: each crate nests the one below it, see [conversions](conversions.md) + +That one type takes one of two forms + +| Form | Use when | Example | +|---|---|---| +| Public enum | Callers outside the crate match variants, and every variant holds our own data or another crate's `Error` that is already `Clone` and comparable | `litellm_inference::Error`, `litellm_auth::Error` | +| Opaque struct over a private `Repr` | The variants hold third-party causes, or the variants must be free to change without breaking callers | `litellm_http::Error` | + +Both forms are still one top-level `Error`. The opaque form just moves the enum behind `Repr`, where it is `pub(crate)` + +```rust +// public enum +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum Error { + #[error("missing required field: {0}")] + MissingField(&'static str), + #[error(transparent)] + Http(litellm_http::Error), +} + +// opaque struct +#[derive(Clone, Debug, thiserror::Error)] +#[error(transparent)] +pub struct Error(Arc); + +#[derive(Debug, thiserror::Error)] +pub(crate) enum Repr { + #[error("could not reach the provider")] + Connect(#[source] reqwest::Error), + #[error("upstream request failed with status {status}: {body}")] + Status { status: u16, body: String }, +} +``` + +Start with the public enum. Move to opaque when a variant needs a non-`Clone` third-party cause, or when downstream crates keep matching variants in ways that duplicate classification. `reqwest::Error`, `std::io::Error` and `serde_json::Error` are built the opaque way + +In the opaque form other crates cannot name a variant, so they cannot build an error with their own message. Public constructors exist only for errors another crate genuinely produces, take typed data and never a message, as in `Error::upstream(status, body)`, `Error::computed_header(name)`, `Error::unsignable_body()` and `Error::signature(cause)` + +`Arc` makes cloning a refcount bump, so `Repr` never has to be `Clone`. `Box` is enough when the goal is only a small `Result`. `Arc` is for when callers clone + +Every crate here is internal to the workspace, so do not mark errors or variants `#[non_exhaustive]`. It only forces a wildcard arm on the other workspace crates and turns off the exhaustiveness check that makes a new variant visit every `match`. When the variants must be free to change, use the opaque form + +## What else may sit next to Error + +These are not a second top-level error and need no folding in + +- Leaf data types whose fields are typed and whose message is in `#[error(...)]`, such as `litellm_http::request::HeaderError`. A wrapping crate nests it as one variant +- Plain data used by predicates, such as `litellm_http::TlsSource` +- A private fault enum one variant carries, such as `PemFault` inside `Repr::InvalidPem` + +## Variants are failure modes + +A public variant is something a caller handles differently: a phase, a status code, retry versus no retry, fault on the request versus the environment, or a message Python parity pins exactly. Failures no caller tells apart share one variant and differ in their data + +```rust +// one variant per provider, so every new provider adds a variant and a match arm +#[error("Missing Azure API Base - set AZURE_API_BASE")] +MissingAzureApiBase, +#[error("Missing Vertex API Base - set VERTEXAI_API_BASE")] +MissingVertexApiBase, + +// one failure mode, the provider and its guidance are data +#[error("Missing {provider} API Base - {guidance}")] +MissingApiBase { provider: &'static str, guidance: &'static str }, +``` + +An exact parity message does not justify a provider-specific variant when caller-supplied data can produce the same text + +Inside a private `Repr` a variant costs no API, so one per distinct reason is fine when each one carries a different message. When several only differ in a word and share every predicate answer, prefer one variant with a fieldless reason enum, so predicates list one arm instead of four, as `Repr::Handshake(HandshakeFault)` does + +```rust +#[error("WebSocket handshake returned {0}")] +Handshake(HandshakeFault), + +#[derive(Debug, thiserror::Error)] +pub(crate) enum HandshakeFault { + #[error("a wrong Sec-WebSocket-Accept")] + Accept, + #[error("an invalid Upgrade header")] + Upgrade, + #[error("an invalid Connection header")] + Connection, + #[error("an invalid subprotocol")] + Subprotocol, +} +``` + +## When to split + +Split into a separate type only when different functions fail in disjoint ways and different callers see them, so no caller matches variants its function can never return. A type per function overdoes it + +Name a split type after what went wrong, not after the function that returns it. `ConnectError` reads better than `OpenSessionError`, and a unit struct is fine for a single failure mode + +A split type still converts into the crate's `Error` when a caller further up needs one `Result` type + +`litellm-llms` has one recorded exception, the OCR error in `src/base_llm/ocr/error.rs`, kept apart until OCR folds into `litellm_llms::Error`, see `crates/inference-ocr/AGENTS.md`. Do not add another + +A utility crate whose modules are unrelated, such as `core-utils`, has no crate-wide failure, so each module that can fail owns one error next to its code instead of a `src/error.rs`. The rest of this skill applies to each of them diff --git a/litellm-rust/.agents/skills/rust-error/references/testing.md b/litellm-rust/.agents/skills/rust-error/references/testing.md new file mode 100644 index 00000000000..4aa22b4a293 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-error/references/testing.md @@ -0,0 +1,37 @@ +# Testing errors + +Assert behavior through predicates and accessors, on errors produced the way production produces them + +```rust +// guesses the message and the variant, and cannot be written once the type is opaque +assert_eq!(error, Error::Transport(TransportError::Http { status: 429, body: "slow down".into() })); + +// asserts what callers rely on +assert_eq!(error.upstream_response(), Some((429, "slow down"))); +assert!(error.dispatched()); +assert!(!error.is_request()); +``` + +## Producing the error + +In order of preference + +1. A real failing call. Bind a `TcpListener`, drop it, then connect to its address for a connect failure. Accept and never answer for a timeout +2. A real cause, such as `tungstenite::Error::ConnectionClosed` or a `serde_json::from_str("{")` error, passed through our `From` +3. A public constructor, such as `Error::upstream(500, "boom".into())` + +Inside the owning crate a test may match the private representation through the `#[cfg(test)] fn repr(&self) -> &Repr` accessor in `crates/http/src/error.rs`, as the `media.rs` and `tls.rs` tests do. Anywhere else, reach the cause the test expects with `litellm_error::find::(&error)`. A transparent layer is no node of the chain, so `find` reaches foreign causes underneath it but never the nested error of ours itself. Ask that error's predicates through the outer type instead, which is what callers do too + +## What to cover + +| Property | Test shape | +|---|---| +| classification | one `#[case]` per input class, asserting the predicate | +| redaction | build from a URL holding `api_key=secret`, assert the message lacks `secret` and the path | +| cause kept | `litellm_error::find::(&error)` returns the expected cause | +| rendered once | the cause's text appears exactly once in `litellm_error::Report::new(&error).to_string()` and not in `to_string()` | +| boundary mapping | the gateway status and body, or the Python exception type and its `status` attribute | + +Assert exact message text only where it is a contract, such as a message Python parity pins. Everywhere else assert that the message contains the facts it must carry. Matching a fieldless variant with `matches!` is fine, full `assert_eq!` on an error that holds a cause is not, since causes compare by pointer + +A test that returns `Result` reports only `Debug` of the error on failure. Prefer `expect` with a short reason, or `unwrap_err` followed by assertions on the predicates diff --git a/litellm-rust/.agents/skills/rust-gateway/SKILL.md b/litellm-rust/.agents/skills/rust-gateway/SKILL.md new file mode 100644 index 00000000000..66b93ee128d --- /dev/null +++ b/litellm-rust/.agents/skills/rust-gateway/SKILL.md @@ -0,0 +1,20 @@ +--- +name: rust-gateway +description: Add or change an HTTP route, extractor, response envelope, stream or router mount in the litellm-rust gateway crates (crates/gateway and crates/gateway-*). Use when adding an endpoint, parsing a request body, choosing an error envelope for an API surface, relaying SSE, or writing a router-level test +--- + +# Rust gateway + +`crates/gateway` mounts endpoint routers and serves the listener. `crates/gateway-inference`, `gateway-mcp` and `gateway-ui` own their endpoints, `gateway-auth` is the inbound authentication middleware and `gateway-management` holds management logic that has no router yet. A crate's `AGENTS.md`, where it has one, says what belongs in it. Read the reference for the part you are touching + +- [surfaces](references/surfaces.md) + - Each API surface mirrors the envelope the Python proxy returns there, so the shapes differ on purpose + - The error type string is derived from the status and provider bodies pass through bounded +- [requests](references/requests.md) + - Extractors reject with the crate's `Error`, handlers take the body as a `Result` when the error needs the request id + - Every alias of a path is registered together, a known but unserved path goes to one handler + - Streams relay through `litellm_host_http::Sse`, with the error frame rendered by the crate's `Error` +- [testing](references/testing.md) + - Build the router through `tests/support` and drive it with `oneshot`, assert the fields the surface promises + +How an error is classified before it reaches a response is the [`rust-error`](../rust-error/SKILL.md) skill, in particular its [gateway](../rust-error/references/gateway.md) page. Axum's own conventions are in its [docs](https://docs.rs/axum/latest/axum/) diff --git a/litellm-rust/.agents/skills/rust-gateway/references/requests.md b/litellm-rust/.agents/skills/rust-gateway/references/requests.md new file mode 100644 index 00000000000..395f29f5c64 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-gateway/references/requests.md @@ -0,0 +1,34 @@ +# Requests + +## Extractors reject with the crate's Error + +A custom extractor sets `type Rejection = Error`, so a bad body gets the surface's envelope rather than axum's plain-text rejection. Never take axum's own `Json`, `Query` or `Path` in a handler, their rejection is plain text. The rejection is the variant's source, never its string. `JsonObject` and `InferenceBody` in `crates/gateway-inference/src/request.rs` are the pattern + +```rust +impl FromRequest for JsonObject { + type Rejection = Error; + + async fn from_request(request: Request, state: &S) -> Result { + let body = Bytes::from_request(request, state).await.map_err(|error| { + if error.status() == StatusCode::PAYLOAD_TOO_LARGE { + Error::BodyTooLarge + } else { + BodyError::Read(error).into() + } + })?; + Ok(Self(object(&body)?)) + } +} +``` + +When the handler needs something from the request in the error body, such as the request id on `/v1/messages`, take the extractor as `Result` so the handler maps the rejection itself, as `messages::create` does. `Infallible` is the right rejection for an extractor that only reads an optional header, as `RequestId` does + +## Paths + +Every alias the Python proxy registers for a route is registered together in `crates/gateway-inference/src/lib.rs`, such as the bare path, its `/v1` form and the `/engines/{model}` and `/openai/deployments/{model}` forms, otherwise one alias silently behaves differently from the others. A path the gateway knows but does not serve, such as `/v1/completions`, is routed to `request::unsupported`, which returns `Error::Unsupported(path)` so the client gets a 501 in the surface's envelope rather than a bare 404 + +`crates/gateway` merges the mounted routers and adds the shared inbound middleware as `route_layer`s, so authentication runs only for matched routes. Endpoint paths and parsing stay in the mounted crate, see the two crates' `AGENTS.md` + +## Streaming + +Once the response headers are sent the status cannot change. A streaming handler builds `litellm_host_http::Sse::new(Json, |error| Bytes::from(Error::from(error).sse_frame()))`, where the closure renders a mid-stream failure as an `event: error` frame with the same status and message a JSON failure on that surface carries, `sse_frame()` for Anthropic and `responses_sse_frame()` for the Responses API's `error` event. `litellm_host_http::Error` separates `Call(E)`, the route's own error, from `Protocol`, a host bug, and the crate's `From` maps the second to `Internal` diff --git a/litellm-rust/.agents/skills/rust-gateway/references/surfaces.md b/litellm-rust/.agents/skills/rust-gateway/references/surfaces.md new file mode 100644 index 00000000000..08d4bbca4ea --- /dev/null +++ b/litellm-rust/.agents/skills/rust-gateway/references/surfaces.md @@ -0,0 +1,20 @@ +# Envelopes follow the API surface + +Each surface mirrors what the Python proxy returns there, so the shapes differ on purpose + +| Surface | Envelope | Where | +|---|---|---| +| `/v1/messages` | Anthropic, `{"type": "error", "error": {"type", "message"}, "request_id"}` | `Error::body()` and `sse_frame()` in `crates/gateway-inference/src/error.rs` | +| OpenAI-shaped routes | `{"error": {"message", "type", "param", "code"}}` | `Error::openai_response()` in the same file | +| `/v1/responses` stream | the Responses `error` event, `{"type": "error", "code", "message", "param"}` | `Error::responses_sse_frame()` in the same file | +| MCP | `{"detail": message}` | `crates/gateway-mcp/src/error.rs` | +| UI | `{"error": {"message"}}`, plus `Retry-After` on 429 | `crates/gateway-ui/src/error.rs` | +| auth rejections | plain text, the top-level message only | `crates/gateway-auth/src/error.rs` | + +Every surface takes its status from one `status()` on its crate's `Error` and renders the message of anything that is not a provider answer with `litellm_error::Report`. In `gateway-inference` the error `type` string is derived from the status by `error_type()`, so a new failure never names its own type. The Anthropic envelope passes a provider body already in its shape through untouched and extracts the message from anything else, and the OpenAI envelope always wraps, as the Python proxy does. A provider body is bounded by `litellm_http::Error::upstream` when the error is built + +Two surfaces show less than the chain on purpose, and a test in each pins it. An auth rejection goes to a caller who has not authenticated, so it never shows what a secret manager or key store said. An MCP upstream or authorization failure shows only its own layer, because rmcp renders the transport error, URL included, into its own `Display`, and an authorizer's cause is the same backend detail auth rejections hide. The inference surfaces show only the top layer of an auth rejection for the same reason + +`/v1/messages` adds the request id to the envelope when the provider did not set one, which is why its handler takes the body as `Result` and renders the rejection itself, see [requests](requests.md) + +A new surface gets its own renderer next to the others, named after the shape it produces, and its status still comes from `Error::status()`. Picking the status is the [`rust-error`](../../rust-error/references/gateway.md) skill's concern, not the surface's diff --git a/litellm-rust/.agents/skills/rust-gateway/references/testing.md b/litellm-rust/.agents/skills/rust-gateway/references/testing.md new file mode 100644 index 00000000000..10c1f01c472 --- /dev/null +++ b/litellm-rust/.agents/skills/rust-gateway/references/testing.md @@ -0,0 +1,9 @@ +# Testing + +Test HTTP contracts in the crate that owns the endpoint: status codes, forwarded headers, error envelopes and streaming behavior. Inference and provider behavior is tested in the `inference-` and `llms` crates, and mounting and server lifecycle in `crates/gateway` + +`crates/gateway-inference/tests/support/mod.rs` builds a `Router` with injected deployments, pools and settings through `app()` and its variants, and `post()` drives it with `tower::ServiceExt::oneshot`. `json()` reads the response body back. Use these instead of building a router or a client by hand in a test + +Assert the status and the fields the surface promises, not the whole body. A test that pins a whole envelope breaks on every unrelated field, one that checks `status`, `error.type` and the presence of `request_id` catches the regressions that matter + +`status()` and `body()` of the error type are pure, so a classification case belongs in the error module's unit tests, see the [`rust-error`](../../rust-error/references/gateway.md) skill. Router tests cover what only the full path shows, such as an extractor rejection arriving in the right envelope or an SSE error frame after the headers went out diff --git a/litellm-rust/AGENTS.md b/litellm-rust/AGENTS.md index e62a9463cb5..9e7a55eb232 100644 --- a/litellm-rust/AGENTS.md +++ b/litellm-rust/AGENTS.md @@ -4,6 +4,8 @@ For diagnostic tracing changes, follow [.agents/skills/rust-tracing/SKILL.md](.a For string-valued enums and their Serde conversions, follow [.agents/skills/rust-string-enums/SKILL.md](.agents/skills/rust-string-enums/SKILL.md) +For error types, error variants and conversions between crate errors, follow [.agents/skills/rust-error/SKILL.md](.agents/skills/rust-error/SKILL.md) + For fieldless enums, derive `strum::VariantArray` and use `VARIANTS` instead of a hand-listed `ALL` array. Derive Strum string conversions instead of hand-written variant-to-string matches Use the derived conversions directly (`<&'static str>::from(x)` / `.into()`, `str::parse`) with no `as_str`/`parse` wrapper that only delegates, and spell each variant with explicit `#[strum(serialize = "...")]` instead of `serialize_all` @@ -24,13 +26,3 @@ Use [`#[rstest]`](https://docs.rs/rstest/latest/rstest/attr.rstest.html) for new - Never loop over inputs (`for`, `.iter().for_each`, `.all`) inside a test body. Give each input its own `#[case::name(...)]`, or use `#[values(...)]` for a cross product - Exception: a test pinning a Rust table against a repo-owned data file (for example `include_str!` of a JSON config) may iterate that file's entries - -## Error definitions - -- A crate's errors live in `src/error.rs`, defined with `thiserror`, and re-exported from `lib.rs` -- Put message templates in the variant's `#[error(...)]` declaration. Callers pass only the small typed arguments needed to fill them, never `Error::Variant(format!(...))` or a preformatted message. Keep the smallest set of neutral variants that callers need to distinguish. Different wording or providers do not justify new variants -- Default to one top-level `Error` enum per crate, with one variant per failure mode and a `#[error(...)]` message on each. A failure mode is something a caller handles differently (phase, status code, retry, a message Python parity pins exactly). Failures no caller tells apart share one variant and differ only in its message -- Keep shared error enums minimal and provider-neutral. Provider names, credential types, configuration fields, and setup guidance belong in caller-supplied data, not dedicated variants or hardcoded shared messages. Reuse a variant for the same failure mode across providers, such as `MissingApiBase { provider: "Azure", guidance: "..." }`. An exact parity message does not justify a provider-specific variant when caller-supplied context can preserve it -- Wrap a lower-level error as a variant with `#[from]` or `#[source]` instead of flattening it to a string -- Exception: split into separate types when different functions fail in disjoint ways, especially when different callers see them. A shared enum would force every caller to match variants its function can never return -- Name a split type after what went wrong (a unit struct is fine for a single failure mode), not after the function that returns it diff --git a/litellm-rust/Cargo.lock b/litellm-rust/Cargo.lock index e998fe1abbb..9c45c5c4b1c 100644 --- a/litellm-rust/Cargo.lock +++ b/litellm-rust/Cargo.lock @@ -3388,6 +3388,7 @@ dependencies = [ "bytes", "futures-util", "litellm-auth-types", + "litellm-error", "litellm-http", "moka", "reqwest", @@ -3407,6 +3408,7 @@ dependencies = [ "azure_core", "azure_identity", "litellm-auth-types", + "litellm-error", "moka", "rstest", "serde_json", @@ -3425,6 +3427,7 @@ dependencies = [ "google-cloud-auth", "http 1.4.2", "litellm-auth-types", + "litellm-error", "moka", "rstest", "serde", @@ -3439,6 +3442,7 @@ dependencies = [ name = "litellm-auth-types" version = "0.1.0" dependencies = [ + "litellm-error", "rstest", "serde", "serde_json", @@ -3452,6 +3456,7 @@ dependencies = [ name = "litellm-cache" version = "0.1.0" dependencies = [ + "litellm-error", "rstest", "serde", "serde_json", @@ -3477,6 +3482,7 @@ dependencies = [ "reqwest", "rstest", "serde_json", + "thiserror 2.0.19", "tokio", "url", "wiremock", @@ -3495,6 +3501,7 @@ dependencies = [ "serde-pickle", "serde_json", "tempfile", + "thiserror 2.0.19", "tokio", ] @@ -3507,11 +3514,13 @@ dependencies = [ "litellm-auth-types", "litellm-cache", "litellm-cache-testing", + "litellm-error", "litellm-http", "percent-encoding", "reqwest", "rstest", "serde_json", + "thiserror 2.0.19", "tokio", "wiremock", ] @@ -3534,6 +3543,7 @@ dependencies = [ "futures-util", "litellm-cache", "litellm-cache-testing", + "litellm-error", "litellm-http", "qdrant-client", "reqwest", @@ -3554,11 +3564,13 @@ version = "0.1.0" dependencies = [ "litellm-cache", "litellm-cache-testing", + "litellm-error", "r2d2", "redis", "redis-test", "rstest", "serde_json", + "thiserror 2.0.19", "tokio", ] @@ -3611,6 +3623,7 @@ dependencies = [ "litellm-auth-aws", "litellm-cache", "litellm-cache-testing", + "litellm-error", "litellm-http", "reqwest", "rstest", @@ -3633,11 +3646,13 @@ dependencies = [ "litellm-cache", "litellm-cache-redis", "litellm-cache-testing", + "litellm-error", "redis", "redis-test", "rstest", "serde_json", "sha2 0.10.9", + "thiserror 2.0.19", "tokio", "uuid", ] @@ -3647,6 +3662,7 @@ name = "litellm-callbacks-legacy-python" version = "0.1.0" dependencies = [ "litellm-auth", + "litellm-error", "litellm-host", "litellm-host-python", "proptest", @@ -3654,6 +3670,7 @@ dependencies = [ "rstest", "serde_json", "strum", + "thiserror 2.0.19", ] [[package]] @@ -3669,6 +3686,7 @@ name = "litellm-config" version = "0.1.0" dependencies = [ "litellm-auth-types", + "litellm-error", "litellm-router-types", "rstest", "serde", @@ -3686,6 +3704,7 @@ name = "litellm-core-utils" version = "0.1.0" dependencies = [ "fancy-regex 0.19.2", + "litellm-error", "litellm-llms-types", "litellm-tracing", "rstest", @@ -3713,6 +3732,7 @@ dependencies = [ "criterion", "proptest", "rstest", + "thiserror 2.0.19", ] [[package]] @@ -3742,6 +3762,14 @@ dependencies = [ "tokio", ] +[[package]] +name = "litellm-error" +version = "0.1.0" +dependencies = [ + "rstest", + "thiserror 2.0.19", +] + [[package]] name = "litellm-framer" version = "0.1.0" @@ -3769,6 +3797,7 @@ dependencies = [ "litellm-auth-types", "litellm-circuit-breaker", "litellm-config", + "litellm-error", "litellm-gateway-auth", "litellm-gateway-inference", "litellm-gateway-mcp", @@ -3802,6 +3831,7 @@ dependencies = [ "futures-util", "litellm-auth-types", "litellm-config", + "litellm-error", "litellm-secrets", "rstest", "serde", @@ -3825,6 +3855,7 @@ dependencies = [ "litellm-auth", "litellm-cache-memory", "litellm-cache-response", + "litellm-error", "litellm-gateway-auth", "litellm-host", "litellm-host-http", @@ -3844,6 +3875,7 @@ dependencies = [ "serde_json", "thiserror 2.0.19", "tokio", + "tokio-tungstenite", "tower", "wiremock", ] @@ -3854,6 +3886,7 @@ version = "0.1.0" dependencies = [ "base64 0.22.1", "litellm-auth-types", + "litellm-error", "litellm-gateway-auth", "rand 0.8.7", "rstest", @@ -3871,6 +3904,7 @@ dependencies = [ "http 1.4.2", "litellm-auth-types", "litellm-config", + "litellm-error", "litellm-secrets", "moka", "rmcp", @@ -3885,6 +3919,7 @@ dependencies = [ "tower", "url", "uuid", + "veil", ] [[package]] @@ -3894,9 +3929,11 @@ dependencies = [ "axum", "axum-login", "base64 0.22.1", + "futures-util", "governor", "jsonwebtoken", "litellm-auth-types", + "litellm-error", "litellm-gateway-auth", "rand 0.8.7", "rstest", @@ -3921,6 +3958,7 @@ dependencies = [ "litellm-coroutine", "rstest", "serde_json", + "thiserror 2.0.19", "tokio", ] @@ -3958,6 +3996,7 @@ dependencies = [ "bytes", "futures-util", "litellm-coroutine", + "litellm-error", "litellm-host", "parking_lot", "pyo3", @@ -3978,6 +4017,7 @@ dependencies = [ "http 1.4.2", "hyper-util", "litellm-core-utils", + "litellm-error", "rcgen", "reqwest", "rstest", @@ -3992,6 +4032,7 @@ dependencies = [ "tokio-rustls 0.26.4", "tokio-tungstenite", "tracing", + "url", "veil", "webpki-roots", ] @@ -4007,6 +4048,7 @@ dependencies = [ "litellm-cache-memory", "litellm-cache-response", "litellm-core-utils", + "litellm-error", "litellm-framer", "litellm-host", "litellm-http", @@ -4034,6 +4076,7 @@ dependencies = [ "litellm-cache-memory", "litellm-cache-response", "litellm-core-utils", + "litellm-error", "litellm-host", "litellm-host-native", "litellm-http", @@ -4043,7 +4086,6 @@ dependencies = [ "litellm-llms-types", "litellm-secrets", "litellm-tracing", - "reqwest", "rstest", "serde_json", "tokio", @@ -4062,6 +4104,7 @@ dependencies = [ "litellm-cache-memory", "litellm-cache-response", "litellm-core-utils", + "litellm-error", "litellm-framer", "litellm-host", "litellm-host-native", @@ -4093,6 +4136,7 @@ dependencies = [ "litellm-auth-aws", "litellm-auth-gcp", "litellm-core-utils", + "litellm-error", "litellm-host", "litellm-host-native", "litellm-http", @@ -4110,6 +4154,7 @@ dependencies = [ "time", "tokio", "tracing", + "url", "wiremock", ] @@ -4125,6 +4170,7 @@ dependencies = [ "litellm-cache-memory", "litellm-cache-response", "litellm-core-utils", + "litellm-error", "litellm-host", "litellm-host-native", "litellm-http", @@ -4162,6 +4208,7 @@ dependencies = [ "futures-util", "litellm-auth", "litellm-core-utils", + "litellm-error", "litellm-http", "litellm-inference", "litellm-inference-testing", @@ -4190,6 +4237,7 @@ dependencies = [ "litellm-auth-azure", "litellm-auth-gcp", "litellm-core-utils", + "litellm-error", "litellm-framer", "litellm-host", "litellm-http", @@ -4230,6 +4278,7 @@ version = "0.1.0" dependencies = [ "indexmap 2.14.0", "jsonschema", + "litellm-error", "litellm-llms-types", "rstest", "schemars 1.2.2", @@ -4256,6 +4305,7 @@ dependencies = [ "litellm-cache-response", "litellm-callbacks-legacy-python", "litellm-core-utils", + "litellm-error", "litellm-host", "litellm-host-python", "litellm-http", @@ -4286,6 +4336,7 @@ dependencies = [ "serde_with", "sqlx", "strum", + "tempfile", "thiserror 2.0.19", "tokio", "tokio-tungstenite", @@ -4363,6 +4414,7 @@ dependencies = [ "google-cloud-auth", "google-cloud-kms-v1", "litellm-core-utils", + "litellm-error", "litellm-http", "litellm-python-compat", "litellm-secrets-aws", @@ -4421,6 +4473,7 @@ dependencies = [ "sha2 0.10.9", "thiserror 2.0.19", "tokio", + "url", "veil", "wiremock", ] @@ -4444,6 +4497,7 @@ dependencies = [ "tempfile", "thiserror 2.0.19", "tokio", + "url", "veil", "wiremock", ] @@ -4470,6 +4524,7 @@ dependencies = [ "serde_json", "thiserror 2.0.19", "tokio", + "url", "veil", "wiremock", ] @@ -4479,7 +4534,10 @@ name = "litellm-secrets-hashicorp" version = "0.1.0" dependencies = [ "litellm-core-utils", + "litellm-error", + "litellm-http", "litellm-secrets-types", + "reqwest", "rstest", "rustify", "rustify_derive", @@ -4513,6 +4571,7 @@ name = "litellm-spend-clickhouse" version = "0.1.0" dependencies = [ "flate2", + "litellm-error", "litellm-http", "litellm-storage-clickhouse", "rstest", @@ -4533,6 +4592,7 @@ name = "litellm-storage-clickhouse" version = "0.1.0" dependencies = [ "flate2", + "litellm-error", "litellm-http", "rstest", "serde", @@ -4552,6 +4612,7 @@ version = "0.1.0" dependencies = [ "flate2", "futures-util", + "litellm-http", "reqwest", "rstest", "semver", @@ -4575,6 +4636,7 @@ dependencies = [ "criterion", "indexmap 2.14.0", "itoa", + "litellm-error", "litellm-token-counter-fast", "litellm-token-counter-huggingface", "litellm-token-counter-tiktoken", @@ -4591,6 +4653,7 @@ name = "litellm-token-counter-fast" version = "0.1.0" dependencies = [ "base64 0.22.1", + "litellm-error", "rand 0.8.7", "rstest", "rustc-hash", @@ -4616,6 +4679,7 @@ name = "litellm-token-counter-tiktoken" version = "0.1.0" dependencies = [ "base64 0.22.1", + "litellm-error", "once_cell", "rstest", "rustc-hash", diff --git a/litellm-rust/Cargo.toml b/litellm-rust/Cargo.toml index 745b29dd0ff..b7b22cfd32c 100644 --- a/litellm-rust/Cargo.toml +++ b/litellm-rust/Cargo.toml @@ -10,6 +10,7 @@ repository = "https://github.com/BerriAI/litellm" [workspace.dependencies] litellm-config = { path = "crates/config" } +litellm-error = { path = "crates/error" } litellm-router = { path = "crates/router" } litellm-router-types = { path = "crates/router-types" } litellm-tracing = { path = "crates/tracing" } diff --git a/litellm-rust/crates/auth-aws/Cargo.toml b/litellm-rust/crates/auth-aws/Cargo.toml index 1a7e2cd6782..e4578777a08 100644 --- a/litellm-rust/crates/auth-aws/Cargo.toml +++ b/litellm-rust/crates/auth-aws/Cargo.toml @@ -7,6 +7,7 @@ repository.workspace = true [dependencies] litellm-auth-types.workspace = true +litellm-error.workspace = true litellm-http.workspace = true reqwest.workspace = true diff --git a/litellm-rust/crates/auth-aws/src/aws.rs b/litellm-rust/crates/auth-aws/src/aws.rs index d83b01c24ad..81aa4b068bc 100644 --- a/litellm-rust/crates/auth-aws/src/aws.rs +++ b/litellm-rust/crates/auth-aws/src/aws.rs @@ -15,13 +15,13 @@ use reqwest::header::{HeaderMap, HeaderName, HeaderValue}; use litellm_auth_types::AwsParams; -use super::Error; use super::constants::{ AWS_ACCESS_KEY_ID, AWS_EXTERNAL_ID, AWS_PROFILE_NAME, AWS_REGION_NAME, AWS_ROLE_ARN, AWS_ROLE_NAME, AWS_SECRET_ACCESS_KEY, AWS_SESSION_NAME, AWS_SESSION_TOKEN, AWS_SIGNED_HEADER_NAMES, AWS_STS_ENDPOINT, AWS_WEB_IDENTITY_TOKEN, AWS_WEB_IDENTITY_TOKEN_FILE, DEFAULT_BEDROCK_REGION, DEFAULT_SESSION_NAME_PREFIX, SIGV4_COMPUTED_HEADER_NAMES, }; +use crate::error::{CredentialFlow, Error, SigningStage}; const STATIC_CREDENTIALS_TTL: Duration = Duration::from_secs(3600 - 60); const AMBIENT_CREDENTIALS_TTL: Duration = Duration::from_secs(600); @@ -271,7 +271,7 @@ impl AwsAuthService { provider .provide_credentials() .await - .map_err(|error| Error::AwsProfile(error.to_string())) + .map_err(Error::credentials(CredentialFlow::Profile)) } AwsAuthFlow::AssumeRole { role, session_name } => { if is_already_running_as_role(&role, &resolved).await? { @@ -287,7 +287,7 @@ impl AwsAuthService { let credentials = provider .provide_credentials() .await - .map_err(|error| Error::AwsDefaultChain(error.to_string()))?; + .map_err(Error::credentials(CredentialFlow::DefaultChain))?; self.set_cached_credentials( key, credentials.clone(), @@ -327,7 +327,7 @@ impl AwsAuthService { provider .provide_credentials() .await - .map_err(|error| Error::AwsAssumeRole(error.to_string())) + .map_err(Error::credentials(CredentialFlow::AssumeRole)) } AwsAuthFlow::WebIdentity { token, @@ -350,12 +350,14 @@ impl AwsAuthService { .web_identity_token(token) .send() .await - .map_err(|error| Error::AwsWebIdentity(error.to_string()))?; + .map_err(Error::credentials(CredentialFlow::WebIdentity))?; let credentials = response .credentials() - .ok_or(Error::AwsMissingWebIdentityCredentials)?; - let expiration = SystemTime::try_from(*credentials.expiration()) - .map_err(|error| Error::AwsWebIdentityExpiration(error.to_string()))?; + .ok_or(Error::MissingWebIdentityCredentials)?; + let expiration = + SystemTime::try_from(*credentials.expiration()).map_err(|source| { + Error::WebIdentityExpiration(litellm_error::Source::new(source)) + })?; Ok(Credentials::new( credentials.access_key_id(), credentials.secret_access_key(), @@ -376,7 +378,7 @@ impl AwsAuthService { let credentials = provider .provide_credentials() .await - .map_err(|error| Error::AwsDefaultChain(error.to_string()))?; + .map_err(Error::credentials(CredentialFlow::DefaultChain))?; self.set_cached_credentials( key, credentials.clone(), @@ -480,14 +482,16 @@ pub fn sign_headers( .settings(SigningSettings::default()) .build() .map(SigningParams::from) - .map_err(|error| Error::AwsSigningParameters(error.to_string()))?; + .map_err(Error::signing(SigningStage::Parameters))?; let header_refs = headers .iter() .map(|(name, value)| { value .to_str() .map(|value| (name.as_str(), value)) - .map_err(|_| Error::AwsSignableRequest(format!("{name} is not visible ASCII"))) + .map_err(|_| Error::NonAsciiHeader { + name: name.to_string(), + }) }) .collect::, _>>()?; let request = SignableRequest::new( @@ -496,23 +500,20 @@ pub fn sign_headers( header_refs.into_iter(), SignableBody::Bytes(body), ) - .map_err(|error| Error::AwsSignableRequest(error.to_string()))?; + .map_err(Error::signing(SigningStage::Request))?; let (instructions, _) = sign(request, ¶ms) - .map_err(|error| Error::AwsSigning(error.to_string()))? + .map_err(Error::signing(SigningStage::Signature))? .into_parts(); instructions .headers() .map(|(name, value)| { - HeaderName::from_bytes(name.as_bytes()) - .map_err(|error| error.to_string()) - .and_then(|name| { - HeaderValue::from_str(value) - .map(|value| (name, value)) - .map_err(|error| error.to_string()) - }) + let name = HeaderName::from_bytes(name.as_bytes()) + .map_err(Error::signing(SigningStage::Output))?; + let value = + HeaderValue::from_str(value).map_err(Error::signing(SigningStage::Output))?; + Ok((name, value)) }) - .collect::>() - .map_err(Error::AwsSigning) + .collect() } /// Model-id and region parsing shared by every Bedrock route. @@ -1081,6 +1082,55 @@ mod tests { assert!(!signed.contains_key("X-Amz-Security-Token")); } + #[test] + fn a_header_value_outside_visible_ascii_is_named_instead_of_signed() { + let (url, body, mut headers) = parity_inputs(); + headers.insert("x-amz-meta-note", HeaderValue::from_str("café").unwrap()); + let credentials = Credentials::new("AKIDEXAMPLE", "secret", None, None, "test"); + + let error = sign_headers( + "POST", + &url, + &body, + &headers, + SigningScope { + region: "us-east-1", + service: BEDROCK_SERVICE, + credentials: &credentials, + time: UNIX_EPOCH, + }, + ) + .unwrap_err(); + + assert_eq!( + error, + Error::NonAsciiHeader { + name: "x-amz-meta-note".into() + } + ); + } + + #[test] + fn the_shared_auth_error_nests_the_aws_failure_and_its_cause() { + let cause = std::io::Error::new(std::io::ErrorKind::NotFound, "profile not found"); + let error = + litellm_auth_types::Error::from(Error::credentials(CredentialFlow::Profile)(cause)); + + assert!(matches!( + litellm_error::find::(&error), + Some(Error::Credentials { + flow: CredentialFlow::Profile, + .. + }) + )); + let report = litellm_error::Report::new(&error).to_string(); + assert!( + report.contains("could not load AWS profile credentials"), + "{report}" + ); + assert_eq!(report.matches("profile not found").count(), 1, "{report}"); + } + #[ignore] #[tokio::test] async fn live_bedrock_invoke_model_returns_200() -> Result<(), Box> { diff --git a/litellm-rust/crates/auth-aws/src/error.rs b/litellm-rust/crates/auth-aws/src/error.rs index d4c6ae8cf6c..5f88fc8a171 100644 --- a/litellm-rust/crates/auth-aws/src/error.rs +++ b/litellm-rust/crates/auth-aws/src/error.rs @@ -1,46 +1,73 @@ -use thiserror::Error as ThisError; +use litellm_error::Source; -#[derive(Clone, Debug, ThisError, PartialEq, Eq)] +#[derive(Clone, Debug, thiserror::Error, PartialEq, Eq)] pub enum Error { - #[error("AWS profile credentials failed: {0}")] - AwsProfile(String), - #[error("AWS default credentials failed: {0}")] - AwsDefaultChain(String), - #[error("AWS role credentials failed: {0}")] - AwsAssumeRole(String), - #[error("AWS web identity credentials failed: {0}")] - AwsWebIdentity(String), - #[error("AWS web identity expiration was invalid: {0}")] - AwsWebIdentityExpiration(String), - #[error("AWS signing parameters failed: {0}")] - AwsSigningParameters(String), - #[error("AWS signable request failed: {0}")] - AwsSignableRequest(String), - #[error("AWS request signing failed: {0}")] - AwsSigning(String), + #[error("could not load AWS {flow} credentials")] + Credentials { + flow: CredentialFlow, + #[source] + source: Source, + }, #[error("AWS web identity response had no credentials")] - AwsMissingWebIdentityCredentials, + MissingWebIdentityCredentials, + #[error("AWS web identity response had an invalid expiration")] + WebIdentityExpiration(#[source] Source), + #[error("header {name} has a value that is not visible ASCII")] + NonAsciiHeader { name: String }, + #[error("{stage}")] + Signing { + stage: SigningStage, + #[source] + source: Source, + }, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum CredentialFlow { + #[error("profile")] + Profile, + #[error("default chain")] + DefaultChain, + #[error("assumed role")] + AssumeRole, + #[error("web identity")] + WebIdentity, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum SigningStage { + #[error("SigV4 signing parameters are invalid")] + Parameters, + #[error("SigV4 cannot sign this request")] + Request, + #[error("SigV4 could not compute the signature")] + Signature, + #[error("SigV4 produced an invalid header")] + Output, +} + +impl Error { + pub(crate) fn credentials( + flow: CredentialFlow, + ) -> impl FnOnce(E) -> Self { + move |source| Self::Credentials { + flow, + source: Source::new(source), + } + } + + pub(crate) fn signing( + stage: SigningStage, + ) -> impl FnOnce(E) -> Self { + move |source| Self::Signing { + stage, + source: Source::new(source), + } + } } impl From for litellm_auth_types::Error { fn from(error: Error) -> Self { - Self::ProviderAuthentication(error.to_string()) - } -} - -#[cfg(test)] -mod tests { - use super::Error; - - #[test] - fn converts_to_shared_auth_error_without_losing_context() { - let error = litellm_auth_types::Error::from(Error::AwsProfile("profile not found".into())); - - assert_eq!( - error, - litellm_auth_types::Error::ProviderAuthentication( - "AWS profile credentials failed: profile not found".into() - ) - ); + Self::provider(error) } } diff --git a/litellm-rust/crates/auth-aws/src/lib.rs b/litellm-rust/crates/auth-aws/src/lib.rs index 0fe0b390110..bd0f866371f 100644 --- a/litellm-rust/crates/auth-aws/src/lib.rs +++ b/litellm-rust/crates/auth-aws/src/lib.rs @@ -5,5 +5,5 @@ mod signer; pub use aws::*; pub use aws_credential_types::Credentials; -pub use error::Error; +pub use error::{CredentialFlow, Error, SigningStage}; pub use signer::SigV4Signer; diff --git a/litellm-rust/crates/auth-aws/src/signer.rs b/litellm-rust/crates/auth-aws/src/signer.rs index 6c2cb4fd162..dff917d7c13 100644 --- a/litellm-rust/crates/auth-aws/src/signer.rs +++ b/litellm-rust/crates/auth-aws/src/signer.rs @@ -53,13 +53,13 @@ impl RequestSigner for SigV4Signer { .keys() .find(|name| is_sigv4_computed_header(name.as_str())) { - return Err(litellm_http::Error::ComputedHeader(name.to_string())); + return Err(litellm_http::Error::computed_header(name.as_str())); } let body = match request.body() { None => &[], - Some(body) => body.as_bytes().ok_or_else(|| { - litellm_http::Error::Signature("a streaming body cannot be signed".into()) - })?, + Some(body) => body + .as_bytes() + .ok_or_else(litellm_http::Error::unsignable_body)?, }; sign_headers( request.method().as_str(), @@ -73,7 +73,13 @@ impl RequestSigner for SigV4Signer { time: (self.clock)(), }, ) - .map_err(|error| litellm_http::Error::Signature(error.to_string())) + .map_err(|error| match error { + Error::NonAsciiHeader { .. } => litellm_http::Error::unsignable_header(error), + Error::Credentials { .. } + | Error::MissingWebIdentityCredentials + | Error::WebIdentityExpiration(_) + | Error::Signing { .. } => litellm_http::Error::signature(error), + }) } } @@ -183,10 +189,28 @@ mod tests { ) .unwrap_err(); - assert_eq!( - error, - litellm_http::Error::ComputedHeader("authorization".into()) - ); + assert!(error.is_computed_header()); + assert!(error.to_string().contains("authorization"), "{error}"); + } + + #[test] + fn a_forwarded_header_that_cannot_be_signed_is_the_callers_fault() { + let error = signed( + json( + URL, + &[("x-amz-meta-note".into(), "café".into())], + &json_value!({}), + None, + ) + .unwrap(), + &signer("textract"), + ) + .unwrap_err(); + + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("x-amz-meta-note"), "{report}"); + assert!(error.is_request()); + assert!(!error.dispatched()); } #[test] @@ -198,7 +222,7 @@ mod tests { assert!(matches!( signed(request, &signer("textract")), - Err(litellm_http::Error::Signature(_)) + Err(error) if !error.dispatched() && !error.is_request() )); } } diff --git a/litellm-rust/crates/auth-azure/Cargo.toml b/litellm-rust/crates/auth-azure/Cargo.toml index 1fd9d39d113..f99972f813b 100644 --- a/litellm-rust/crates/auth-azure/Cargo.toml +++ b/litellm-rust/crates/auth-azure/Cargo.toml @@ -18,5 +18,6 @@ azure_core = "1.0.0" azure_identity = { version = "1.0.0", features = ["tokio"] } [dev-dependencies] +litellm-error.workspace = true rstest.workspace = true tokio.workspace = true diff --git a/litellm-rust/crates/auth-azure/src/native.rs b/litellm-rust/crates/auth-azure/src/native.rs index 64752162384..9fda67746d1 100644 --- a/litellm-rust/crates/auth-azure/src/native.rs +++ b/litellm-rust/crates/auth-azure/src/native.rs @@ -12,7 +12,7 @@ use azure_identity::{ }; use sha2::{Digest, Sha256}; -use litellm_auth_types::Error; +use litellm_auth_types::{Error, ErrorDetail}; use litellm_auth_types::{InputSource, ResolvedCredential, SecretValue, Sourced}; use super::credential_provider_cache::{ @@ -133,7 +133,9 @@ impl NativeAzureTokenAcquirer { let token = credential .get_token(&[scope.as_str()], None) .await - .map_err(|error| Error::CredentialAcquisition(error.to_string().into()))?; + .map_err(|error| { + Error::CredentialAcquisition(ErrorDetail::failed("Azure token request", error)) + })?; let expires_on = u64::try_from(token.expires_on.unix_timestamp()) .ok() .map(|seconds| UNIX_EPOCH + Duration::from_secs(seconds)); @@ -250,11 +252,8 @@ fn validate_authority(request: &NativeAzureRequest) -> Result<(), Error> { let Some(authority) = authority else { return Ok(()); }; - let url = url::Url::parse(authority.value()).map_err(|_| { - Error::InvalidConfiguration( - "Azure authority must be an HTTPS origin without credentials, query, or fragment" - .into(), - ) + let url = url::Url::parse(authority.value()).map_err(|error| { + Error::InvalidConfiguration(ErrorDetail::failed("Azure authority parsing", error)) })?; if url.scheme() != "https" || url.host_str().is_none() @@ -264,10 +263,10 @@ fn validate_authority(request: &NativeAzureRequest) -> Result<(), Error> { || url.fragment().is_some() || !matches!(url.path(), "" | "/") { - return Err(Error::InvalidConfiguration( - "Azure authority must be an HTTPS origin without credentials, query, or fragment" - .into(), - )); + return Err(Error::InvalidConfiguration(ErrorDetail::InvalidType { + field: "Azure authority".into(), + expected: "an HTTPS origin without credentials, query, or fragment", + })); } Ok(()) } @@ -377,7 +376,7 @@ fn trusted_source(sources: &[InputSource]) -> InputSource { fn mixed_sources() -> Result { Err(Error::InvalidConfiguration( - "request-controlled Azure auth inputs cannot be combined with host credentials".into(), + ErrorDetail::MixedInputSources { provider: "Azure" }, )) } @@ -444,7 +443,7 @@ fn build_credential( .map(|credential| credential as Arc), } .map_err(|error| { - Error::InvalidConfiguration(litellm_auth_types::ErrorDetail::failed( + Error::InvalidConfiguration(ErrorDetail::failed( "Azure credential initialization", error, )) @@ -662,12 +661,11 @@ mod tests { )) .unwrap_err(); - assert_eq!( - error, - litellm_auth_types::Error::InvalidConfiguration( + assert!( + litellm_error::Report::new(&error).to_string().contains( "request-controlled Azure auth inputs cannot be combined with host credentials" - .into() - ) + ), + "{error:?}" ); } @@ -695,12 +693,11 @@ mod tests { authority, )) .unwrap_err(); - assert_eq!( - error, - litellm_auth_types::Error::InvalidConfiguration( + assert!( + litellm_error::Report::new(&error).to_string().contains( "Azure authority must be an HTTPS origin without credentials, query, or fragment" - .into() - ) + ), + "{error:?}" ); } } diff --git a/litellm-rust/crates/auth-azure/src/resolve.rs b/litellm-rust/crates/auth-azure/src/resolve.rs index 2142e22db50..25b71741342 100644 --- a/litellm-rust/crates/auth-azure/src/resolve.rs +++ b/litellm-rust/crates/auth-azure/src/resolve.rs @@ -1,8 +1,8 @@ -use litellm_auth_types::Error; use litellm_auth_types::{ CredentialFileRef, CredentialLookup, CredentialRef, InputSource, ResolvedCredential, SecretValue, Sourced, TokenProviderHandle, }; +use litellm_auth_types::{Error, ErrorDetail}; use std::future::Future; use std::pin::Pin; use std::sync::Arc; @@ -91,9 +91,9 @@ impl AzureAuthService { AzureCredentialPlan::Caller(caller) => { let credential = caller.acquire().await?; if credential.secret().expose().is_empty() { - return Err(Error::EmptyCallerCredential( - "Azure AD token provider returned an empty token", - )); + return Err(Error::EmptyCallerCredential { + caller: "Azure AD token provider", + }); } Ok(Some(Sourced::new(credential, InputSource::Deployment))) } @@ -106,11 +106,9 @@ impl AzureAuthService { } => { let assertion = resolve_reference(inputs, env_lookup, reference.value()) .await? - .ok_or_else(|| { - Error::CredentialAcquisition( - "Azure OIDC reference did not resolve to a value".into(), - ) - })?; + .ok_or(Error::CredentialAcquisition(ErrorDetail::Unresolved { + subject: "Azure OIDC reference", + }))?; let request = ValidatedAzureRequest::new(NativeAzureRequest::ClientAssertion { tenant_id, client_id, @@ -142,7 +140,13 @@ impl AzureAuthService { Err(error) => failures.push(error), } } - Err(Error::CredentialChain(failures)) + match failures.pop() { + Some(last) => Err(Error::CredentialChain { + last: Box::new(last), + earlier: failures, + }), + None => Ok(None), + } } AzureCredentialPlan::Missing => Ok(None), } @@ -173,7 +177,12 @@ pub(crate) fn select_auth_plan( .map(|selector| Sourced::new(selector, value.source())) }) .transpose() - .map_err(|_| Error::InvalidConfiguration("invalid Azure credential selector".into()))?; + .map_err(|error| { + Error::InvalidConfiguration(ErrorDetail::failed( + "Azure credential selector parsing", + error, + )) + })?; let federated_token_file = configured_string( &inputs.federated_token_file, AZURE_FEDERATED_TOKEN_FILE_ENV, @@ -263,9 +272,12 @@ fn select_native_plan( let selection_source = selected.source(); match selected.into_value() { - AzureCredentialType::ClientSecretCredential => Err(Error::InvalidConfiguration( - "ClientSecretCredential requires tenant_id, client_id, and client_secret".into(), - )), + AzureCredentialType::ClientSecretCredential => { + Err(Error::InvalidConfiguration(ErrorDetail::MissingSettings { + credential: "ClientSecretCredential", + settings: "tenant_id, client_id, and client_secret", + })) + } AzureCredentialType::WorkloadIdentityCredential => { Ok(AzureCredentialPlan::Native(ValidatedAzureRequest::new( workload_request(tenant_id, client_id, federated_token_file, scope, authority)?, @@ -349,22 +361,21 @@ fn workload_request( authority: Option>, ) -> Result { Ok(NativeAzureRequest::WorkloadIdentity { - tenant_id: tenant_id.ok_or_else(|| { - Error::InvalidConfiguration("WorkloadIdentityCredential requires tenant_id".into()) - })?, - client_id: client_id.ok_or_else(|| { - Error::InvalidConfiguration("WorkloadIdentityCredential requires client_id".into()) - })?, - token_file_path: token_file_path.ok_or_else(|| { - Error::InvalidConfiguration( - "WorkloadIdentityCredential requires azure_federated_token_file".into(), - ) - })?, + tenant_id: tenant_id.ok_or(workload_requires("tenant_id"))?, + client_id: client_id.ok_or(workload_requires("client_id"))?, + token_file_path: token_file_path.ok_or(workload_requires("azure_federated_token_file"))?, scope, authority, }) } +fn workload_requires(settings: &'static str) -> Error { + Error::InvalidConfiguration(ErrorDetail::MissingSettings { + credential: "WorkloadIdentityCredential", + settings, + }) +} + fn configured_string( configured: &ConfigValue, environment_name: &str, @@ -410,11 +421,10 @@ async fn resolve_reference( .map_or(CredentialLookup::Missing, CredentialLookup::Found), CredentialRef::None => return Ok(None), CredentialRef::File(_) | CredentialRef::Request(_) | CredentialRef::Host(_) => { - let resolver = inputs.credential_resolver.as_ref().ok_or_else(|| { - Error::InvalidConfiguration( - "credential reference requires a host credential resolver".into(), - ) - })?; + let resolver = inputs + .credential_resolver + .as_ref() + .ok_or(Error::InvalidConfiguration(ErrorDetail::MissingResolver))?; resolver.resolve(reference).await? } }; @@ -433,7 +443,7 @@ fn oidc_reference( let value = token.value().expose(); if token.source() == InputSource::Request && value.starts_with("oidc/") { return Err(Error::InvalidConfiguration( - "request-controlled Azure credential references are not allowed".into(), + ErrorDetail::RequestControlledReference { provider: "Azure" }, )); } if let Some(name) = value.strip_prefix("oidc/env/") { @@ -457,19 +467,17 @@ fn oidc_reference( } if value.starts_with("oidc/") { return Err(Error::InvalidConfiguration( - "unsupported OIDC reference".into(), + ErrorDetail::UnsupportedReference, )); } Ok(None) } -fn non_empty_reference(value: &str, kind: &str) -> Result { +fn non_empty_reference(value: &str, kind: &'static str) -> Result { if value.is_empty() { - return Err(Error::InvalidConfiguration( - litellm_auth_types::ErrorDetail::Empty { - subject: kind.into(), - }, - )); + return Err(Error::InvalidConfiguration(ErrorDetail::Empty { + subject: kind, + })); } Ok(value.to_string()) } @@ -518,7 +526,9 @@ mod tests { expires_on: None, }) } else { - Err(Error::CredentialAcquisition(kind.into())) + Err(Error::CredentialAcquisition( + litellm_auth_types::ErrorDetail::Unresolved { subject: kind }, + )) } }) } @@ -624,7 +634,11 @@ mod tests { ))) .expect_err("unsupported backend must fail validation"); - assert!(error.to_string().contains("unsupported OIDC reference")); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("unsupported OIDC reference") + ); } #[rstest::rstest] @@ -649,11 +663,11 @@ mod tests { }) .unwrap_err(); - assert_eq!( - error, - Error::InvalidConfiguration( - "request-controlled Azure credential references are not allowed".into() - ) + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("request-controlled Azure credential references are not allowed"), + "{error:?}" ); } @@ -715,7 +729,16 @@ mod tests { .await .unwrap_err(); - assert!(matches!(error, Error::CredentialChain(errors) if errors.len() == 2)); + assert!(matches!(&error, Error::CredentialChain { earlier, .. } if earlier.len() == 1)); + let report = litellm_error::Report::new(&error).to_string(); + assert!( + report.contains("managed-identity did not resolve"), + "{report}" + ); + assert!( + report.contains("developer-tools did not resolve"), + "{report}" + ); } #[derive(Debug)] @@ -763,7 +786,9 @@ mod tests { assert_eq!( error, - Error::EmptyCallerCredential("Azure AD token provider returned an empty token") + Error::EmptyCallerCredential { + caller: "Azure AD token provider" + } ); } } diff --git a/litellm-rust/crates/auth-gcp/Cargo.toml b/litellm-rust/crates/auth-gcp/Cargo.toml index bb4c022580a..7844b35ecc2 100644 --- a/litellm-rust/crates/auth-gcp/Cargo.toml +++ b/litellm-rust/crates/auth-gcp/Cargo.toml @@ -24,5 +24,6 @@ google-cloud-auth = { workspace = true, optional = true } http = { workspace = true, optional = true } [dev-dependencies] +litellm-error.workspace = true rstest.workspace = true tempfile.workspace = true diff --git a/litellm-rust/crates/auth-gcp/src/auth.rs b/litellm-rust/crates/auth-gcp/src/auth.rs index e09a350c012..8ac1e0f54ba 100644 --- a/litellm-rust/crates/auth-gcp/src/auth.rs +++ b/litellm-rust/crates/auth-gcp/src/auth.rs @@ -73,13 +73,17 @@ impl VertexAuth { .or_else(|| non_empty_env(env_lookup, VERTEXAI_API_KEY_ENV)); let project_id = get_vertex_ai_project(config, env_lookup); - if !has_authorization && static_token.is_none() { - let access = self.get_access_token(config, env_lookup).await?; - return Ok(VertexEnvironment { - headers: apply_credential(headers, &access.token, CredentialPlacement::Bearer)?, - project_id: project_id.unwrap_or(access.project_id), - }); - } + let static_token = match (has_authorization, static_token) { + (true, _) => None, + (false, Some(token)) => Some(token), + (false, None) => { + let access = self.get_access_token(config, env_lookup).await?; + return Ok(VertexEnvironment { + headers: apply_credential(headers, &access.token, CredentialPlacement::Bearer)?, + project_id: project_id.unwrap_or(access.project_id), + }); + } + }; let project_id = match project_id { Some(project_id) => project_id, @@ -90,14 +94,9 @@ impl VertexAuth { .await? } }; - let headers = if has_authorization { - headers - } else { - apply_credential( - headers, - static_token.as_deref().expect("static token was checked"), - CredentialPlacement::Bearer, - )? + let headers = match static_token { + Some(token) => apply_credential(headers, &token, CredentialPlacement::Bearer)?, + None => headers, }; Ok(VertexEnvironment { headers, @@ -234,14 +233,13 @@ enum CredentialCacheKey { Adc, } -const BASE64_CREDENTIALS: &str = "The `vertex_credentials` value is base64-encoded JSON, not the JSON itself. \ - Set it to a readable file path, or to the credentials JSON itself."; - fn load_trusted_service_account(configured: &str) -> Result { if is_base64_encoded_json(configured) { - return Err(Error::InvalidConfiguration(ErrorDetail::Message( - BASE64_CREDENTIALS.into(), - ))); + return Err(Error::InvalidConfiguration( + ErrorDetail::Base64EncodedJson { + field: "vertex_credentials", + }, + )); } if names_a_credentials_file(configured) { CustomServiceAccount::from_file(configured) @@ -299,7 +297,8 @@ mod tests { else { panic!("{configured:?} should not load"); }; - assert!(error.to_string().contains(expected), "{error}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains(expected), "{report}"); } #[rstest::rstest] @@ -321,7 +320,7 @@ mod tests { .await { Ok(_) => panic!("{configured:?} should not load"), - Err(error) => error.to_string(), + Err(error) => litellm_error::Report::new(&error).to_string(), } } diff --git a/litellm-rust/crates/auth-gcp/src/sdk.rs b/litellm-rust/crates/auth-gcp/src/sdk.rs index 566df291b7d..6cfeee64d20 100644 --- a/litellm-rust/crates/auth-gcp/src/sdk.rs +++ b/litellm-rust/crates/auth-gcp/src/sdk.rs @@ -55,7 +55,7 @@ impl CredentialsProvider for GoogleCredentials { entity_tag: EntityTag::new(), data, }) - .map_err(|_| CredentialsError::from_msg(false, "Google authentication failed")) + .map_err(|error| CredentialsError::from_source(false, error)) } async fn universe_domain(&self) -> Option { @@ -102,5 +102,9 @@ mod tests { ); let error = credentials.headers(Extensions::new()).await.unwrap_err(); assert!(!format!("{error:?}").contains("private")); + assert_eq!( + std::error::Error::source(&error).and_then(|cause| cause.downcast_ref::()), + Some(&Error::InvalidHeader) + ); } } diff --git a/litellm-rust/crates/auth-types/Cargo.toml b/litellm-rust/crates/auth-types/Cargo.toml index b54c04f0229..400f2640221 100644 --- a/litellm-rust/crates/auth-types/Cargo.toml +++ b/litellm-rust/crates/auth-types/Cargo.toml @@ -6,6 +6,7 @@ license.workspace = true repository.workspace = true [dependencies] +litellm-error.workspace = true serde.workspace = true serde_json.workspace = true subtle.workspace = true diff --git a/litellm-rust/crates/auth-types/src/credential.rs b/litellm-rust/crates/auth-types/src/credential.rs index 94fb8a58467..4f69c6fffc8 100644 --- a/litellm-rust/crates/auth-types/src/credential.rs +++ b/litellm-rust/crates/auth-types/src/credential.rs @@ -86,9 +86,9 @@ impl CredentialPlan { Self::Caller(caller) => { let credential = caller.acquire().await?; if credential.secret().expose().is_empty() { - return Err(Error::EmptyCallerCredential( - "credential caller returned an empty credential", - )); + return Err(Error::EmptyCallerCredential { + caller: "credential caller", + }); } Ok(CredentialPlanResolution::Resolved(credential)) } @@ -104,8 +104,8 @@ mod tests { CredentialLookup, CredentialLookupFuture, CredentialPlan, CredentialPlanResolution, CredentialRef, CredentialResolver, CredentialResolverHandle, }; - use crate::Error; use crate::SecretValue; + use crate::{Error, ErrorDetail}; #[derive(Debug)] struct HostResolver; @@ -150,9 +150,10 @@ mod tests { impl CredentialResolver for FailingResolver { fn resolve<'a>(&'a self, _reference: &'a CredentialRef) -> CredentialLookupFuture<'a> { Box::pin(async { - Err(Error::CredentialAcquisition( - "host credential lookup failed".into(), - )) + Err(Error::CredentialAcquisition(ErrorDetail::failed( + "host credential lookup", + std::io::Error::from(std::io::ErrorKind::PermissionDenied), + ))) }) } } @@ -168,9 +169,14 @@ mod tests { .await .expect_err("acquisition errors cannot become fallback"); - assert_eq!( - error, - Error::CredentialAcquisition("host credential lookup failed".into()) + assert!( + matches!(error, Error::CredentialAcquisition(_)), + "{error:?}" + ); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("host credential lookup failed") ); } } diff --git a/litellm-rust/crates/auth-types/src/error.rs b/litellm-rust/crates/auth-types/src/error.rs index ff17dcf3810..a8ba7b64e03 100644 --- a/litellm-rust/crates/auth-types/src/error.rs +++ b/litellm-rust/crates/auth-types/src/error.rs @@ -1,17 +1,23 @@ -use thiserror::Error as ThisError; +use litellm_error::{Report, Source}; -#[derive(Clone, Debug, ThisError, PartialEq, Eq)] +use crate::{ParamKind, ParamSpec}; + +#[derive(Clone, Debug, thiserror::Error, PartialEq, Eq)] pub enum Error { - #[error("invalid authentication configuration: {0}")] + #[error("invalid authentication configuration")] InvalidConfiguration(#[source] ErrorDetail), - #[error("credential acquisition failed: {0}")] + #[error("credential acquisition failed")] CredentialAcquisition(#[source] ErrorDetail), - #[error("credential caller failed: {0}")] - EmptyCallerCredential(&'static str), - #[error("{0}")] - ProviderAuthentication(String), - #[error("credential acquisition failed: {}", .0.iter().map(ToString::to_string).collect::>().join("; "))] - CredentialChain(Vec), + #[error("{caller} returned an empty credential")] + EmptyCallerCredential { caller: &'static str }, + #[error("provider authentication failed")] + Provider(#[source] Source), + #[error("no credential in the chain succeeded{}", earlier_attempts(earlier))] + CredentialChain { + #[source] + last: Box, + earlier: Vec, + }, #[error( "Missing {provider} API Key - Set `api_key` or the {environment_variable} environment variable" )] @@ -22,7 +28,7 @@ pub enum Error { #[error("Missing {provider} {} - {}", .spec.setting, .spec.guidance())] MissingParam { provider: &'static str, - spec: &'static crate::ParamSpec, + spec: &'static ParamSpec, }, #[error("Missing {provider} API Base - {guidance}")] MissingApiBase { @@ -33,78 +39,282 @@ pub enum Error { InvalidHeader, } +fn earlier_attempts(earlier: &[Error]) -> String { + if earlier.is_empty() { + return String::new(); + } + let attempts = earlier + .iter() + .map(|failure| Report::new(failure).to_string()) + .collect::>() + .join("; "); + format!(" (earlier attempts: {attempts})") +} + +impl Error { + pub fn provider(cause: impl std::error::Error + Send + Sync + 'static) -> Self { + Self::Provider(Source::new(cause)) + } + + pub fn is_missing_credential(&self) -> bool { + match self { + Self::MissingApiKey { .. } => true, + Self::MissingParam { spec, .. } => match spec.kind { + ParamKind::Credential => true, + ParamKind::Setting => false, + }, + Self::InvalidConfiguration(_) + | Self::CredentialAcquisition(_) + | Self::EmptyCallerCredential { .. } + | Self::Provider(_) + | Self::CredentialChain { .. } + | Self::MissingApiBase { .. } + | Self::InvalidHeader => false, + } + } + + pub fn is_request(&self) -> bool { + match self { + Self::InvalidConfiguration(_) + | Self::EmptyCallerCredential { .. } + | Self::InvalidHeader => true, + Self::CredentialAcquisition(_) | Self::Provider(_) | Self::CredentialChain { .. } => { + false + } + Self::MissingParam { spec, .. } => match spec.kind { + ParamKind::Setting => true, + ParamKind::Credential => false, + }, + Self::MissingApiKey { .. } | Self::MissingApiBase { .. } => false, + } + } +} + #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum ErrorDetail { - #[error("{0}")] - Message(String), #[error("{field} must be {expected}")] InvalidType { field: String, expected: &'static str, }, #[error("{subject} cannot be empty")] - Empty { subject: String }, + Empty { subject: &'static str }, #[error("credential header {0} already exists")] DuplicateHeader(&'static str), - #[error("{operation} failed: {source}")] + #[error("credential plan is not allowed by the provider auth policy")] + PlanNotAllowed, + #[error("{credential} requires {settings}")] + MissingSettings { + credential: &'static str, + settings: &'static str, + }, + #[error("credentials cannot be sent to a request-controlled {provider} endpoint")] + RequestControlledEndpoint { provider: &'static str }, + #[error("request-controlled {provider} credential references are not allowed")] + RequestControlledReference { provider: &'static str }, + #[error("request-controlled {provider} auth inputs cannot be combined with host credentials")] + MixedInputSources { provider: &'static str }, + #[error("unsupported OIDC reference")] + UnsupportedReference, + #[error("credential reference requires a host credential resolver")] + MissingResolver, + #[error("{subject} did not resolve to a value")] + Unresolved { subject: &'static str }, + #[error( + "The `{field}` value is base64-encoded JSON, not the JSON itself. Set it to a readable file path, or to the credentials JSON itself." + )] + Base64EncodedJson { field: &'static str }, + #[error("{operation} failed")] Failed { operation: &'static str, #[source] - source: ErrorSource, + cause: Source, }, } impl ErrorDetail { pub fn failed( operation: &'static str, - source: impl std::error::Error + Send + Sync + 'static, + cause: impl std::error::Error + Send + Sync + 'static, ) -> Self { Self::Failed { operation, - source: ErrorSource::new(source), + cause: Source::new(cause), } } } -impl From for ErrorDetail { - fn from(message: String) -> Self { - Self::Message(message) +#[cfg(test)] +mod tests { + use std::io; + + use litellm_error::{Report, Source}; + use rstest::rstest; + + use super::{Error, ErrorDetail}; + use crate::{AwsParams, VertexParams}; + + #[rstest] + #[case::api_key( + Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" }, + "Missing Example API Key - Set `api_key` or the EXAMPLE_API_KEY environment variable" + )] + #[case::another_api_key( + Error::MissingApiKey { provider: "Custom", environment_variable: "CUSTOM_KEY" }, + "Missing Custom API Key - Set `api_key` or the CUSTOM_KEY environment variable" + )] + #[case::api_base( + Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" }, + "Missing Example API Base - Pass api_base" + )] + #[case::another_api_base( + Error::MissingApiBase { provider: "Custom", guidance: "Set CUSTOM_ENDPOINT" }, + "Missing Custom API Base - Set CUSTOM_ENDPOINT" + )] + fn missing_inputs_keep_the_python_text(#[case] error: Error, #[case] expected: &str) { + assert_eq!(error.to_string(), expected); + } + + #[rstest] + #[case::configuration( + Error::InvalidConfiguration(ErrorDetail::Empty { subject: "credential selector" }), + "invalid authentication configuration: credential selector cannot be empty" + )] + #[case::acquisition( + Error::CredentialAcquisition(ErrorDetail::Unresolved { subject: "OIDC reference" }), + "credential acquisition failed: OIDC reference did not resolve to a value" + )] + #[case::empty_caller( + Error::EmptyCallerCredential { caller: "token provider" }, + "token provider returned an empty credential" + )] + fn the_report_names_every_layer_in_order(#[case] error: Error, #[case] expected: &str) { + assert!( + Report::new(&error).to_string().contains(expected), + "{}", + Report::new(&error) + ); + } + + #[rstest] + #[case::configuration(Error::InvalidConfiguration(ErrorDetail::failed( + "test credential", + denied() + )))] + #[case::acquisition(Error::CredentialAcquisition(ErrorDetail::failed( + "test credential", + denied() + )))] + fn a_failure_keeps_its_cause_as_the_source_and_renders_it_once(#[case] error: Error) { + let cause = + std::iter::successors(std::error::Error::source(&error), |cause| cause.source()) + .find_map(|cause| cause.downcast_ref::()?.downcast_ref::()) + .expect("the io cause is in the source chain"); + assert_eq!(cause.kind(), io::ErrorKind::PermissionDenied); + let report = Report::new(&error).to_string(); + assert_eq!(report.matches("access denied").count(), 1, "{report}"); + } + + #[rstest] + #[case::configuration(Error::InvalidConfiguration(ErrorDetail::failed( + "test credential", + denied() + )))] + #[case::acquisition(Error::CredentialAcquisition(ErrorDetail::failed( + "test credential", + denied() + )))] + fn a_contextual_failure_leaves_its_cause_out_of_its_own_message(#[case] error: Error) { + assert!(!error.to_string().contains("access denied"), "{error}"); + assert!( + Report::new(&error) + .to_string() + .contains("test credential failed: access denied") + ); + } + + #[test] + fn a_provider_failure_keeps_the_provider_error_as_its_source() { + let error = Error::provider(denied()); + assert_eq!( + litellm_error::find::(&error).map(io::Error::kind), + Some(io::ErrorKind::PermissionDenied) + ); + assert!(!error.to_string().contains("access denied"), "{error}"); + let report = Report::new(&error).to_string(); + assert_eq!(report.matches("access denied").count(), 1, "{report}"); + } + + #[test] + fn a_chain_keeps_its_last_failure_as_the_source_and_renders_every_attempt_once() { + let error = Error::CredentialChain { + last: Box::new(Error::provider(denied())), + earlier: vec![ + Error::CredentialAcquisition(ErrorDetail::Unresolved { + subject: "OIDC reference", + }), + Error::EmptyCallerCredential { + caller: "token provider", + }, + ], + }; + + assert_eq!( + litellm_error::find::(&error).map(io::Error::kind), + Some(io::ErrorKind::PermissionDenied) + ); + let report = Report::new(&error).to_string(); + assert_eq!(report.matches("access denied").count(), 1, "{report}"); + assert_eq!( + report.matches("OIDC reference did not resolve").count(), + 1, + "{report}" + ); + assert_eq!( + report + .matches("token provider returned an empty credential") + .count(), + 1, + "{report}" + ); + } + + fn denied() -> io::Error { + io::Error::new(io::ErrorKind::PermissionDenied, "access denied") + } + + #[rstest] + #[case::api_key(Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" }, true)] + #[case::vertex_credentials(VertexParams::CREDENTIALS.missing("Vertex AI"), true)] + #[case::aws_secret_access_key(AwsParams::SECRET_ACCESS_KEY.missing("AWS"), true)] + #[case::vertex_project(VertexParams::PROJECT.missing("Vertex AI"), false)] + #[case::aws_region(AwsParams::REGION.missing("AWS"), false)] + #[case::api_base(Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" }, false)] + #[case::invalid_configuration(Error::InvalidConfiguration(ErrorDetail::PlanNotAllowed), false)] + fn only_an_absent_credential_is_a_missing_credential( + #[case] error: Error, + #[case] expected: bool, + ) { + assert_eq!(error.is_missing_credential(), expected); + } + + #[rstest] + #[case::vertex_project(VertexParams::PROJECT.missing("Vertex AI"), true)] + #[case::aws_region(AwsParams::REGION.missing("AWS"), true)] + #[case::api_base(Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" }, false)] + #[case::vertex_credentials(VertexParams::CREDENTIALS.missing("Vertex AI"), false)] + #[case::api_key(Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" }, false)] + #[case::invalid_configuration(Error::InvalidConfiguration(ErrorDetail::PlanNotAllowed), true)] + #[case::invalid_header(Error::InvalidHeader, true)] + #[case::identity_provider_outage( + Error::CredentialAcquisition(ErrorDetail::MissingResolver), + false + )] + #[case::provider_sdk_failure(Error::provider(denied()), false)] + fn a_setting_the_caller_can_pass_is_theirs_to_fix_but_credentials_and_api_bases_are_not( + #[case] error: Error, + #[case] expected: bool, + ) { + assert_eq!(error.is_request(), expected); } } - -impl From<&str> for ErrorDetail { - fn from(message: &str) -> Self { - Self::Message(message.into()) - } -} - -#[derive(Clone, Debug)] -pub struct ErrorSource(std::sync::Arc); - -impl std::ops::Deref for ErrorSource { - type Target = dyn std::error::Error + Send + Sync; - - fn deref(&self) -> &Self::Target { - self.0.as_ref() - } -} - -impl std::fmt::Display for ErrorSource { - fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - std::fmt::Display::fmt(&self.0, formatter) - } -} - -impl ErrorSource { - pub fn new(error: impl std::error::Error + Send + Sync + 'static) -> Self { - Self(std::sync::Arc::new(error)) - } -} - -impl PartialEq for ErrorSource { - fn eq(&self, other: &Self) -> bool { - std::sync::Arc::ptr_eq(&self.0, &other.0) - } -} - -impl Eq for ErrorSource {} diff --git a/litellm-rust/crates/auth-types/src/http.rs b/litellm-rust/crates/auth-types/src/http.rs index 2de3bad9ecf..2ad07f7d541 100644 --- a/litellm-rust/crates/auth-types/src/http.rs +++ b/litellm-rust/crates/auth-types/src/http.rs @@ -21,9 +21,9 @@ pub fn apply_credential( placement: CredentialPlacement, ) -> Result, Error> { if credential.trim().is_empty() { - return Err(Error::InvalidConfiguration( - "credential cannot be empty".into(), - )); + return Err(Error::InvalidConfiguration(crate::ErrorDetail::Empty { + subject: "credential", + })); } if headers .iter() @@ -71,6 +71,10 @@ mod tests { ) .expect_err("provider policy must handle existing credentials"); - assert!(error.to_string().contains("already exists")); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("already exists") + ); } } diff --git a/litellm-rust/crates/auth-types/src/lib.rs b/litellm-rust/crates/auth-types/src/lib.rs index c875bda7979..99651977867 100644 --- a/litellm-rust/crates/auth-types/src/lib.rs +++ b/litellm-rust/crates/auth-types/src/lib.rs @@ -51,9 +51,9 @@ pub use credential::{ CredentialFileRef, CredentialLookup, CredentialLookupFuture, CredentialPlan, CredentialPlanResolution, CredentialRef, CredentialResolver, CredentialResolverHandle, }; -pub use error::{Error, ErrorDetail, ErrorSource}; +pub use error::{Error, ErrorDetail}; pub use http::CredentialPlacement; -pub use params::{AwsParams, ParamSpec, VertexParams}; +pub use params::{AwsParams, ParamKind, ParamSpec, VertexParams}; pub use policy::{CredentialPlanKind, CredentialRule, ExistingHeaderBehavior, ProviderAuthPolicy}; pub use secret::SecretValue; pub use token::{ResolvedCredential, TokenFuture, TokenProvider, TokenProviderHandle}; diff --git a/litellm-rust/crates/auth-types/src/params/aws.rs b/litellm-rust/crates/auth-types/src/params/aws.rs index 25fe52b2ce1..8075fc15e9f 100644 --- a/litellm-rust/crates/auth-types/src/params/aws.rs +++ b/litellm-rust/crates/auth-types/src/params/aws.rs @@ -4,15 +4,17 @@ use serde::{Deserialize, Serialize}; use serde_json::{Map, Value}; use veil::Redact; -use super::ParamSpec; +use super::{ParamKind, ParamSpec}; const fn spec( setting: &'static str, + kind: ParamKind, wire: &'static [&'static str], env: &'static [&'static str], ) -> ParamSpec { ParamSpec { setting, + kind, wire, module_global: None, env, @@ -53,40 +55,67 @@ pub struct AwsParams { impl AwsParams { pub const ACCESS_KEY_ID: ParamSpec = spec( "access key id", + ParamKind::Credential, &["aws_access_key_id"], &["AWS_ACCESS_KEY_ID"], ); pub const SECRET_ACCESS_KEY: ParamSpec = spec( "secret access key", + ParamKind::Credential, &["aws_secret_access_key"], &["AWS_SECRET_ACCESS_KEY"], ); pub const SESSION_TOKEN: ParamSpec = spec( "session token", + ParamKind::Credential, &["aws_session_token"], &["AWS_SESSION_TOKEN"], ); pub const REGION: ParamSpec = spec( "region", + ParamKind::Setting, &["aws_region_name"], &["AWS_REGION_NAME", "AWS_REGION"], ); - pub const SESSION_NAME: ParamSpec = - spec("session name", &["aws_session_name"], &["AWS_SESSION_NAME"]); - pub const PROFILE_NAME: ParamSpec = - spec("profile name", &["aws_profile_name"], &["AWS_PROFILE_NAME"]); - pub const ROLE_NAME: ParamSpec = spec("role name", &["aws_role_name"], &["AWS_ROLE_NAME"]); + pub const SESSION_NAME: ParamSpec = spec( + "session name", + ParamKind::Setting, + &["aws_session_name"], + &["AWS_SESSION_NAME"], + ); + pub const PROFILE_NAME: ParamSpec = spec( + "profile name", + ParamKind::Setting, + &["aws_profile_name"], + &["AWS_PROFILE_NAME"], + ); + pub const ROLE_NAME: ParamSpec = spec( + "role name", + ParamKind::Setting, + &["aws_role_name"], + &["AWS_ROLE_NAME"], + ); pub const WEB_IDENTITY_TOKEN: ParamSpec = spec( "web identity token", + ParamKind::Credential, &["aws_web_identity_token"], &["AWS_WEB_IDENTITY_TOKEN"], ); - pub const STS_ENDPOINT: ParamSpec = - spec("STS endpoint", &["aws_sts_endpoint"], &["AWS_STS_ENDPOINT"]); - pub const EXTERNAL_ID: ParamSpec = - spec("external id", &["aws_external_id"], &["AWS_EXTERNAL_ID"]); + pub const STS_ENDPOINT: ParamSpec = spec( + "STS endpoint", + ParamKind::Setting, + &["aws_sts_endpoint"], + &["AWS_STS_ENDPOINT"], + ); + pub const EXTERNAL_ID: ParamSpec = spec( + "external id", + ParamKind::Setting, + &["aws_external_id"], + &["AWS_EXTERNAL_ID"], + ); pub const BEDROCK_RUNTIME_ENDPOINT: ParamSpec = spec( "Bedrock runtime endpoint", + ParamKind::Setting, &["aws_bedrock_runtime_endpoint"], &["AWS_BEDROCK_RUNTIME_ENDPOINT"], ); diff --git a/litellm-rust/crates/auth-types/src/params/mod.rs b/litellm-rust/crates/auth-types/src/params/mod.rs index 88bc64cc2fc..daad1aff8a8 100644 --- a/litellm-rust/crates/auth-types/src/params/mod.rs +++ b/litellm-rust/crates/auth-types/src/params/mod.rs @@ -10,11 +10,18 @@ pub use vertex::VertexParams; #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct ParamSpec { pub setting: &'static str, + pub kind: ParamKind, pub wire: &'static [&'static str], pub module_global: Option<&'static str>, pub env: &'static [&'static str], } +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ParamKind { + Credential, + Setting, +} + impl ParamSpec { /// The first non-blank value: each wire name through `params`, then each environment /// name through `env`. diff --git a/litellm-rust/crates/auth-types/src/params/vertex.rs b/litellm-rust/crates/auth-types/src/params/vertex.rs index 06ec3008dfb..2a6b8bc0ca6 100644 --- a/litellm-rust/crates/auth-types/src/params/vertex.rs +++ b/litellm-rust/crates/auth-types/src/params/vertex.rs @@ -2,7 +2,7 @@ use serde::{Deserialize, Deserializer, Serialize, de::Unexpected}; use serde_json::Value; use veil::Redact; -use super::ParamSpec; +use super::{ParamKind, ParamSpec}; /// The Vertex AI fields of Python's `GenericLiteLLMParams`, both the current names and the /// `vertex_ai_*` spellings `VertexBase.safe_get_vertex_ai_*` still read. @@ -27,18 +27,21 @@ pub struct VertexParams { impl VertexParams { pub const CREDENTIALS: ParamSpec = ParamSpec { setting: "credentials", + kind: ParamKind::Credential, wire: &["vertex_credentials", "vertex_ai_credentials"], module_global: None, env: &["VERTEXAI_CREDENTIALS"], }; pub const PROJECT: ParamSpec = ParamSpec { setting: "project", + kind: ParamKind::Setting, wire: &["vertex_project", "vertex_ai_project"], module_global: Some("vertex_project"), env: &["VERTEXAI_PROJECT"], }; pub const LOCATION: ParamSpec = ParamSpec { setting: "location", + kind: ParamKind::Setting, wire: &["vertex_location", "vertex_ai_location"], module_global: Some("vertex_location"), env: &["VERTEXAI_LOCATION", "VERTEX_LOCATION"], diff --git a/litellm-rust/crates/auth-types/src/policy.rs b/litellm-rust/crates/auth-types/src/policy.rs index ee2bf08bede..8fff59e284a 100644 --- a/litellm-rust/crates/auth-types/src/policy.rs +++ b/litellm-rust/crates/auth-types/src/policy.rs @@ -1,5 +1,5 @@ use crate::http::apply_credential; -use crate::{CredentialPlacement, Error, ResolvedCredential}; +use crate::{CredentialPlacement, Error, ErrorDetail, ResolvedCredential}; #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum CredentialPlanKind { @@ -31,11 +31,18 @@ pub struct ProviderAuthPolicy { impl ProviderAuthPolicy { pub fn has_existing_credential(&self, headers: &[(String, String)]) -> bool { - headers.iter().any(|(name, _)| { - self.accepted_existing_headers - .iter() - .any(|accepted| name.eq_ignore_ascii_case(accepted)) - }) + self.existing_credential_header(headers).is_some() + } + + fn existing_credential_header(&self, headers: &[(String, String)]) -> Option<&'static str> { + self.accepted_existing_headers + .iter() + .copied() + .find(|accepted| { + headers + .iter() + .any(|(name, _)| name.eq_ignore_ascii_case(accepted)) + }) } pub fn apply( @@ -44,11 +51,11 @@ impl ProviderAuthPolicy { kind: CredentialPlanKind, credential: &ResolvedCredential, ) -> Result, Error> { - if self.has_existing_credential(&headers) { + if let Some(existing) = self.existing_credential_header(&headers) { return match self.existing_header_behavior { ExistingHeaderBehavior::Preserve => Ok(headers), ExistingHeaderBehavior::Reject => Err(Error::InvalidConfiguration( - "credential header already exists".into(), + ErrorDetail::DuplicateHeader(existing), )), }; } @@ -56,11 +63,7 @@ impl ProviderAuthPolicy { .rules .iter() .find(|rule| rule.kind == kind) - .ok_or_else(|| { - Error::InvalidConfiguration( - "credential plan is not allowed by the provider auth policy".into(), - ) - })?; + .ok_or(Error::InvalidConfiguration(ErrorDetail::PlanNotAllowed))?; apply_credential(headers, credential.secret().expose(), rule.placement) } } @@ -108,6 +111,10 @@ mod tests { ) .unwrap_err(); - assert!(error.to_string().contains("not allowed")); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("not allowed") + ); } } diff --git a/litellm-rust/crates/auth-types/tests/error.rs b/litellm-rust/crates/auth-types/tests/error.rs deleted file mode 100644 index aaedc3814f0..00000000000 --- a/litellm-rust/crates/auth-types/tests/error.rs +++ /dev/null @@ -1,74 +0,0 @@ -use litellm_auth_types::Error; -use rstest::rstest; - -#[rstest] -#[case::api_key( - Error::MissingApiKey { provider: "Example", environment_variable: "EXAMPLE_API_KEY" }, - "Missing Example API Key - Set `api_key` or the EXAMPLE_API_KEY environment variable" -)] -#[case::another_api_key( - Error::MissingApiKey { provider: "Custom", environment_variable: "CUSTOM_KEY" }, - "Missing Custom API Key - Set `api_key` or the CUSTOM_KEY environment variable" -)] -#[case::api_base( - Error::MissingApiBase { provider: "Example", guidance: "Pass api_base" }, - "Missing Example API Base - Pass api_base" -)] -#[case::another_api_base( - Error::MissingApiBase { provider: "Custom", guidance: "Set CUSTOM_ENDPOINT" }, - "Missing Custom API Base - Set CUSTOM_ENDPOINT" -)] -#[case::configuration( - Error::InvalidConfiguration("credential selector is invalid".into()), - "invalid authentication configuration: credential selector is invalid" -)] -#[case::acquisition( - Error::CredentialAcquisition("token expired".into()), - "credential acquisition failed: token expired" -)] -#[case::caller( - Error::EmptyCallerCredential("empty token"), - "credential caller failed: empty token" -)] -#[case::provider( - Error::ProviderAuthentication("provider rejected credentials".into()), - "provider rejected credentials" -)] -#[case::chain( - Error::CredentialChain(vec![ - Error::CredentialAcquisition("token expired".into()), - Error::EmptyCallerCredential("empty token"), - ]), - "credential acquisition failed: credential acquisition failed: token expired; credential caller failed: empty token" -)] -fn display_preserves_failure_phase_and_caller_context( - #[case] error: Error, - #[case] expected: &str, -) { - assert_eq!(error.to_string(), expected); -} - -#[rstest] -#[case::configuration(true)] -#[case::acquisition(false)] -fn contextual_failures_keep_the_original_source(#[case] configuration: bool) { - use litellm_auth_types::ErrorDetail; - - let detail = ErrorDetail::failed( - "test credential", - std::io::Error::from(std::io::ErrorKind::PermissionDenied), - ); - let error = if configuration { - Error::InvalidConfiguration(detail) - } else { - Error::CredentialAcquisition(detail) - }; - let source = std::iter::successors(Some(&error as &dyn std::error::Error), |error| { - error.source() - }) - .find_map(|error| error.downcast_ref::()) - .expect("the original credential error remains available"); - assert_eq!(source.kind(), std::io::ErrorKind::PermissionDenied); - assert!(error.to_string().contains("test credential failed:")); - assert!(error.to_string().ends_with(&source.to_string())); -} diff --git a/litellm-rust/crates/auth-types/tests/params.rs b/litellm-rust/crates/auth-types/tests/params.rs index 9d488b170e1..38890edb39d 100644 --- a/litellm-rust/crates/auth-types/tests/params.rs +++ b/litellm-rust/crates/auth-types/tests/params.rs @@ -1,8 +1,9 @@ -use litellm_auth_types::{AwsParams, ParamSpec}; +use litellm_auth_types::{AwsParams, ParamKind, ParamSpec}; use rstest::rstest; const LOCATION: ParamSpec = ParamSpec { setting: "location", + kind: ParamKind::Setting, wire: &["vertex_location", "vertex_ai_location"], module_global: Some("vertex_location"), env: &["VERTEXAI_LOCATION", "VERTEX_LOCATION"], diff --git a/litellm-rust/crates/auth-types/tests/token.rs b/litellm-rust/crates/auth-types/tests/token.rs index d7b23531596..7f7027942b0 100644 --- a/litellm-rust/crates/auth-types/tests/token.rs +++ b/litellm-rust/crates/auth-types/tests/token.rs @@ -1,5 +1,4 @@ use std::{ - error::Error as StdError, future::{Future, poll_fn}, sync::{ Arc, @@ -53,7 +52,7 @@ async fn callbacks_acquire_fresh_credentials_on_demand(#[case] access_token: boo #[rstest] #[tokio::test] -async fn callback_errors_preserve_the_original_source() { +async fn callback_errors_keep_their_typed_cause() { let provider = TokenProviderHandle::from_callback(|| async { Err(Error::CredentialAcquisition(ErrorDetail::failed( "caller credential", @@ -62,13 +61,15 @@ async fn callback_errors_preserve_the_original_source() { }); let error = provider.acquire().await.unwrap_err(); - assert!(matches!(error, Error::CredentialAcquisition(_))); - let source = std::iter::successors(Some(&error as &(dyn StdError + 'static)), |error| { - (*error).source() - }) - .find_map(|error| error.downcast_ref::()) - .unwrap(); - assert_eq!(source.kind(), std::io::ErrorKind::PermissionDenied); + let Error::CredentialAcquisition(ErrorDetail::Failed { cause, .. }) = &error else { + panic!("the callback's error passes through unchanged: {error:?}"); + }; + assert_eq!( + cause + .downcast_ref::() + .map(std::io::Error::kind), + Some(std::io::ErrorKind::PermissionDenied) + ); } struct Release(Arc); diff --git a/litellm-rust/crates/cache-azure-blob/Cargo.toml b/litellm-rust/crates/cache-azure-blob/Cargo.toml index c28cb90d84a..202300c7ab4 100644 --- a/litellm-rust/crates/cache-azure-blob/Cargo.toml +++ b/litellm-rust/crates/cache-azure-blob/Cargo.toml @@ -16,6 +16,7 @@ azure_core = "1.1.0" azure_storage_blob = "1.1.0" futures-util.workspace = true reqwest.workspace = true +thiserror.workspace = true tokio.workspace = true url.workspace = true diff --git a/litellm-rust/crates/cache-azure-blob/src/cache.rs b/litellm-rust/crates/cache-azure-blob/src/cache.rs index c5c1fdd8ab9..e3b013eba4f 100644 --- a/litellm-rust/crates/cache-azure-blob/src/cache.rs +++ b/litellm-rust/crates/cache-azure-blob/src/cache.rs @@ -18,6 +18,10 @@ use url::Url; use crate::{credential::AzureBlobCredential, transport::ReqwestTransport}; +#[derive(Debug, thiserror::Error)] +#[error("account URL cannot hold a container path")] +struct NotABaseUrl; + pub struct AzureBlobCache { container: BlobContainerClient, codec: C, @@ -57,12 +61,12 @@ impl AzureBlobCache { codec: C, runtime: Handle, ) -> Result { - let parsed = Url::parse(account_url).map_err(|_| Error::Unavailable)?; + let parsed = Url::parse(account_url).map_err(Error::unavailable)?; let account_url = parsed.as_str().trim_end_matches('/').to_string(); let container_url = { let mut url = parsed; url.path_segments_mut() - .map_err(|()| Error::Unavailable)? + .map_err(|()| Error::unavailable(NotABaseUrl))? .pop_if_empty() .push(container); url @@ -75,7 +79,7 @@ impl AzureBlobCache { ..BlobContainerClientOptions::default() }), ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let cache = Self { container: client, codec, @@ -101,7 +105,7 @@ impl AzureBlobCache { Err(error) if is_storage_error(&error, StorageErrorCode::ContainerAlreadyExists) => { Ok(()) } - Err(_) => Err(Error::Unavailable), + Err(error) => Err(Error::unavailable(error)), } } @@ -116,7 +120,7 @@ impl AzureBlobCache { { Ok(_) => Ok(()), Err(error) if !overwrite && is_already_present(&error) => Ok(()), - Err(_) => Err(Error::Unavailable), + Err(error) => Err(Error::unavailable(error)), } } @@ -126,13 +130,9 @@ impl AzureBlobCache { Err(error) if is_storage_error(&error, StorageErrorCode::BlobNotFound) => { return Ok(None); } - Err(_) => return Err(Error::Unavailable), + Err(error) => return Err(Error::unavailable(error)), }; - let bytes = response - .body - .collect() - .await - .map_err(|_| Error::Unavailable)?; + let bytes = response.body.collect().await.map_err(Error::unavailable)?; self.codec.decode(&bytes).map(Some) } @@ -140,16 +140,16 @@ impl AzureBlobCache { let mut pages = self .container .list_blobs(None) - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .into_pages(); - while let Some(page) = pages.try_next().await.map_err(|_| Error::Unavailable)? { - let page = page.into_model().map_err(|_| Error::Unavailable)?; + while let Some(page) = pages.try_next().await.map_err(Error::unavailable)? { + let page = page.into_model().map_err(Error::unavailable)?; for name in page.blob_items.into_iter().filter_map(|item| item.name) { self.container .blob_client(&name) .delete(None) .await - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } } Ok(()) diff --git a/litellm-rust/crates/cache-azure-blob/src/credential.rs b/litellm-rust/crates/cache-azure-blob/src/credential.rs index d1a3d0e44ec..f58a599f8e3 100644 --- a/litellm-rust/crates/cache-azure-blob/src/credential.rs +++ b/litellm-rust/crates/cache-azure-blob/src/credential.rs @@ -17,6 +17,10 @@ const LLM_TOKEN_ENV: &str = "AZURE_AD_TOKEN"; type EnvLookup = Arc Option + Send + Sync>; +#[derive(Debug, thiserror::Error)] +#[error("no Azure credential is available for blob storage")] +struct NoCredential; + pub struct AzureBlobCredential { service: AzureAuthService, env_lookup: EnvLookup, @@ -62,15 +66,8 @@ impl TokenCredential for AzureBlobCredential { &lookup, ) .await - .map_err(|error| { - azure_core::Error::with_message(ErrorKind::Credential, error.to_string()) - })? - .ok_or_else(|| { - azure_core::Error::with_message( - ErrorKind::Credential, - "no Azure credential is available for blob storage", - ) - })?; + .map_err(|error| azure_core::Error::new(ErrorKind::Credential, error))? + .ok_or_else(|| azure_core::Error::new(ErrorKind::Credential, NoCredential))?; let (token, expires_on) = match credential.into_value() { ResolvedCredential::AccessToken { token, expires_on } => (token, expires_on), ResolvedCredential::Static(token) => (token, None), diff --git a/litellm-rust/crates/cache-azure-blob/src/transport.rs b/litellm-rust/crates/cache-azure-blob/src/transport.rs index 3914b92365c..62feee8fae0 100644 --- a/litellm-rust/crates/cache-azure-blob/src/transport.rs +++ b/litellm-rust/crates/cache-azure-blob/src/transport.rs @@ -24,6 +24,7 @@ impl HttpClient for ReqwestTransport { Body::SeekableStream(stream) => outgoing.body(reqwest::Body::wrap_stream(stream)), }; let response = outgoing.send().await.map_err(|error| { + let error = litellm_http::Error::from(error); let kind = if error.is_connect() { ErrorKind::Connection } else { @@ -41,9 +42,9 @@ impl HttpClient for ReqwestTransport { ); } } - let body = response - .bytes_stream() - .map_err(|error| azure_core::Error::new(ErrorKind::Io, error)); + let body = response.bytes_stream().map_err(|error| { + azure_core::Error::new(ErrorKind::Io, litellm_http::Error::from(error)) + }); Ok(AsyncRawResponse::new(status, headers, Box::pin(body))) } } diff --git a/litellm-rust/crates/cache-azure-blob/tests/cache.rs b/litellm-rust/crates/cache-azure-blob/tests/cache.rs index b9f50b824b3..3b014f53e82 100644 --- a/litellm-rust/crates/cache-azure-blob/tests/cache.rs +++ b/litellm-rust/crates/cache-azure-blob/tests/cache.rs @@ -128,7 +128,7 @@ fn connect_surfaces_service_failures() { ResponseCacheCodec, runtime.handle().clone(), )); - assert!(matches!(result, Err(Error::Unavailable))); + assert!(matches!(result, Err(Error::Unavailable(_)))); } #[rstest] @@ -263,7 +263,7 @@ fn malformed_blobs_are_invalid_entries(fixture: Fixture, #[case] key: &str, #[ca fixture.service.seed_blob(key, bytes); assert!(matches!( fixture.cache.get_cache(key, &no_ttl()), - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(_)) )); } @@ -378,15 +378,15 @@ fn service_failures_map_to_unavailable(fixture: Fixture) { fixture.service.set_failing(true); assert!(matches!( fixture.cache.get_cache("key", &no_ttl()), - Err(Error::Unavailable) + Err(Error::Unavailable(_)) )); assert!(matches!( fixture.cache.set_cache("key", entry(json!(1)), &no_ttl()), - Err(Error::Unavailable) + Err(Error::Unavailable(_)) )); assert!(matches!( fixture.cache.flush_cache(), - Err(Error::Unavailable) + Err(Error::Unavailable(_)) )); assert!(matches!( fixture.runtime.block_on( @@ -394,7 +394,7 @@ fn service_failures_map_to_unavailable(fixture: Fixture) { .cache .async_set_cache_pipeline(vec![("k".into(), entry(json!(1)))], no_ttl()) ), - Err(Error::Unavailable) + Err(Error::Unavailable(_)) )); } diff --git a/litellm-rust/crates/cache-disk/Cargo.toml b/litellm-rust/crates/cache-disk/Cargo.toml index 825b23899e7..b2f7004910f 100644 --- a/litellm-rust/crates/cache-disk/Cargo.toml +++ b/litellm-rust/crates/cache-disk/Cargo.toml @@ -12,6 +12,7 @@ rand.workspace = true rusqlite = { version = "0.39", features = ["bundled"] } serde-pickle = "1.2" serde_json.workspace = true +thiserror.workspace = true tokio.workspace = true [dev-dependencies] diff --git a/litellm-rust/crates/cache-disk/src/cache.rs b/litellm-rust/crates/cache-disk/src/cache.rs index b8a2c6ed1e4..880a1f86e71 100644 --- a/litellm-rust/crates/cache-disk/src/cache.rs +++ b/litellm-rust/crates/cache-disk/src/cache.rs @@ -64,7 +64,7 @@ impl DiskCache { { tokio::task::spawn_blocking(move || operation(&store)) .await - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? } } @@ -162,7 +162,7 @@ impl BatchCache for DiskCache Ok(BatchEntry::Hit(value)), Ok(None) => Ok(BatchEntry::Miss), - Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid), + Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid), Err(error) => Err(error), }) .collect() @@ -186,7 +186,7 @@ impl BatchCache for DiskCache match self.decode_stored(value) { Ok(Some(value)) => Ok(BatchEntry::Hit(value)), Ok(None) => Ok(BatchEntry::Miss), - Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid), + Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid), Err(error) => Err(error), }, }) @@ -272,7 +272,7 @@ fn increment( Ok((stored, ttl.map(|ttl| unix_now() + ttl.as_secs_f64()))) }; store.update(key, unix_now(), &mut apply)?; - result.ok_or(Error::InvalidEntry) + result.ok_or(Error::InvalidEntry(None)) } fn unix_now() -> f64 { diff --git a/litellm-rust/crates/cache-disk/src/python/value.rs b/litellm-rust/crates/cache-disk/src/python/value.rs index 645eba7b757..6e7c2ee0473 100644 --- a/litellm-rust/crates/cache-disk/src/python/value.rs +++ b/litellm-rust/crates/cache-disk/src/python/value.rs @@ -3,8 +3,8 @@ use py_literal::Value; use serde_json::{Map, Number}; pub(crate) fn from_pickle(bytes: &[u8]) -> Result { - let value = serde_pickle::value_from_slice(bytes, Default::default()) - .map_err(|_| Error::InvalidEntry)?; + let value = + serde_pickle::value_from_slice(bytes, Default::default()).map_err(Error::invalid_entry)?; from_pickle_value(value) } @@ -70,7 +70,7 @@ fn from_pickle_hashable(value: serde_pickle::HashableValue) -> Result Result { - value.parse().map_err(|_| Error::InvalidEntry) + value.parse().map_err(Error::invalid_entry) } pub(crate) fn from_json(value: serde_json::Value) -> Value { @@ -101,7 +101,7 @@ pub(crate) fn from_json(value: serde_json::Value) -> Value { pub(crate) fn from_json_text(value: &str) -> Result { serde_json::from_str(value) .map(from_json) - .map_err(|_| Error::InvalidEntry) + .map_err(Error::invalid_entry) } pub(crate) fn is_truthy(value: &Value) -> bool { @@ -131,7 +131,7 @@ pub(crate) fn to_f64(value: &Value) -> Option { } pub(crate) fn to_json(value: &Value) -> Result, Error> { - serde_json::to_vec(&to_json_value(value)?).map_err(|_| Error::InvalidEntry) + serde_json::to_vec(&to_json_value(value)?).map_err(Error::invalid_entry) } fn to_json_value(value: &Value) -> Result { @@ -142,12 +142,12 @@ fn to_json_value(value: &Value) -> Result { value .to_string() .parse::() - .map_err(|_| Error::InvalidEntry)?, + .map_err(Error::invalid_entry)?, ), Value::Float(value) => { - serde_json::Value::Number(Number::from_f64(*value).ok_or(Error::InvalidEntry)?) + serde_json::Value::Number(Number::from_f64(*value).ok_or(Error::InvalidEntry(None))?) } - Value::Complex(_) | Value::Bytes(_) => return Err(Error::InvalidEntry), + Value::Complex(_) | Value::Bytes(_) => return Err(Error::InvalidEntry(None)), Value::String(value) => serde_json::Value::String(value.clone()), Value::Tuple(values) | Value::List(values) | Value::Set(values) => { serde_json::Value::Array( @@ -162,7 +162,7 @@ fn to_json_value(value: &Value) -> Result { .iter() .map(|(key, value)| { let Value::String(key) = key else { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); }; Ok((key.clone(), to_json_value(value)?)) }) diff --git a/litellm-rust/crates/cache-disk/src/sqlite.rs b/litellm-rust/crates/cache-disk/src/sqlite.rs index 24c2be2e1c5..14cbf18b2ff 100644 --- a/litellm-rust/crates/cache-disk/src/sqlite.rs +++ b/litellm-rust/crates/cache-disk/src/sqlite.rs @@ -48,13 +48,13 @@ struct Row { impl DiskcacheSqliteStore { pub fn open(directory: impl AsRef) -> Result { let directory = directory.as_ref().to_path_buf(); - fs::create_dir_all(&directory).map_err(|_| Error::Unavailable)?; - let directory = std::path::absolute(&directory).map_err(|_| Error::Unavailable)?; + fs::create_dir_all(&directory).map_err(Error::unavailable)?; + let directory = std::path::absolute(&directory).map_err(Error::unavailable)?; let database = directory.join("cache.db"); - let connection = Connection::open(database).map_err(|_| Error::Unavailable)?; + let connection = Connection::open(database).map_err(Error::unavailable)?; connection .busy_timeout(std::time::Duration::from_secs(60)) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let mut settings = read_settings(&connection)?; for (key, value) in default_settings() { @@ -74,7 +74,7 @@ impl DiskcacheSqliteStore { value )", ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; for (key, value) in &settings { if !matches!(key.as_str(), "count" | "size" | "hits" | "misses") { connection @@ -82,7 +82,7 @@ impl DiskcacheSqliteStore { "INSERT OR REPLACE INTO Settings VALUES (?, ?)", params![key, value], ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } } for (key, value) in [ @@ -96,7 +96,7 @@ impl DiskcacheSqliteStore { "INSERT OR IGNORE INTO Settings VALUES (?, ?)", params![key, value], ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } connection .execute_batch( @@ -117,7 +117,7 @@ impl DiskcacheSqliteStore { CREATE UNIQUE INDEX IF NOT EXISTS Cache_key_raw ON Cache(key, raw); CREATE INDEX IF NOT EXISTS Cache_expire_time ON Cache(expire_time);", ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let eviction_policy = setting_string(&settings, "eviction_policy") .unwrap_or_else(|| "least-recently-stored".to_string()); @@ -128,23 +128,27 @@ impl DiskcacheSqliteStore { .execute_batch( "CREATE INDEX IF NOT EXISTS Cache_store_time ON Cache(store_time)", ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } "least-recently-used" => { connection .execute_batch( "CREATE INDEX IF NOT EXISTS Cache_access_time ON Cache(access_time)", ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } "least-frequently-used" => { connection .execute_batch( "CREATE INDEX IF NOT EXISTS Cache_access_count ON Cache(access_count)", ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; + } + _ => { + return Err(Error::unavailable(SettingFault::EvictionPolicy( + eviction_policy, + ))); } - _ => return Err(Error::Unavailable), } connection .execute_batch( @@ -170,12 +174,12 @@ impl DiskcacheSqliteStore { UPDATE Settings SET value = value - OLD.size WHERE key = \"size\"; END;", ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let min_file_size = setting_i64(&settings, "disk_min_file_size") .unwrap_or(DEFAULT_DISK_MIN_FILE_SIZE) .try_into() - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let size_limit = setting_i64(&settings, "size_limit").unwrap_or(DEFAULT_SIZE_LIMIT); let cull_limit = setting_i64(&settings, "cull_limit").unwrap_or(DEFAULT_CULL_LIMIT); let statistics = setting_i64(&settings, "statistics").unwrap_or_default() != 0; @@ -207,7 +211,7 @@ impl DiskcacheSqliteStore { |row| row.get::<_, Option>(0), ) .optional() - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .flatten() { cleanup.push(old_filename); @@ -221,7 +225,7 @@ impl DiskcacheSqliteStore { |row| row.get::<_, i64>(0), ) .optional() - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; if let Some(rowid) = rowid { connection .execute( @@ -230,7 +234,7 @@ impl DiskcacheSqliteStore { WHERE rowid = ?", params![now, expire_time, now, size, mode, filename, value, rowid], ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } else { connection .execute( @@ -240,7 +244,7 @@ impl DiskcacheSqliteStore { ) VALUES (?, 1, ?, ?, ?, 0, NULL, ?, ?, ?, ?)", params![key, now, expire_time, now, size, mode, filename, value], ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } cleanup.extend(self.cull(connection, now)?); Ok(cleanup) @@ -257,13 +261,13 @@ impl DiskcacheSqliteStore { WHERE expire_time IS NOT NULL AND expire_time < ? ORDER BY expire_time LIMIT ?", ) - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .query_map(params![now, self.cull_limit], |row| { Ok((row.get::<_, i64>(0)?, row.get::<_, Option>(1)?)) }) - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .collect::, _>>() - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; for (_, filename) in &expired { if let Some(filename) = filename { cleanup.push(filename.clone()); @@ -272,7 +276,7 @@ impl DiskcacheSqliteStore { for (rowid, _) in &expired { connection .execute("DELETE FROM Cache WHERE rowid = ?", params![rowid]) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } let remaining = self.cull_limit - i64::try_from(expired.len()).unwrap_or(self.cull_limit); if remaining <= 0 || self.volume(connection)? < self.size_limit { @@ -283,19 +287,23 @@ impl DiskcacheSqliteStore { "least-recently-stored" => "store_time", "least-recently-used" => "access_time", "least-frequently-used" => "access_count", - _ => return Err(Error::Unavailable), + policy => { + return Err(Error::unavailable(SettingFault::EvictionPolicy( + policy.to_owned(), + ))); + } }; let rows = connection .prepare(&format!( "SELECT rowid, filename FROM Cache ORDER BY {order} LIMIT ?" )) - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .query_map(params![remaining], |row| { Ok((row.get::<_, i64>(0)?, row.get::<_, Option>(1)?)) }) - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .collect::, _>>() - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; for (_, filename) in &rows { if let Some(filename) = filename { cleanup.push(filename.clone()); @@ -304,7 +312,7 @@ impl DiskcacheSqliteStore { for (rowid, _) in rows { connection .execute("DELETE FROM Cache WHERE rowid = ?", params![rowid]) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } Ok(cleanup) } @@ -312,15 +320,15 @@ impl DiskcacheSqliteStore { fn volume(&self, connection: &Connection) -> Result { let page_count: i64 = connection .query_row("PRAGMA page_count", [], |row| row.get(0)) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let page_size: i64 = connection .query_row("PRAGMA page_size", [], |row| row.get(0)) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let size: i64 = connection .query_row("SELECT value FROM Settings WHERE key = 'size'", [], |row| { row.get(0) }) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; Ok(page_count.saturating_mul(page_size).saturating_add(size)) } } @@ -331,13 +339,13 @@ impl DiskStore for DiskcacheSqliteStore { } fn get(&self, key: &str, now: f64) -> Result, Error> { - let connection = self.connection.lock().map_err(|_| Error::Unavailable)?; + let connection = self.connection.lock()?; let select = "SELECT rowid, expire_time, mode, filename, value FROM Cache WHERE key = ? AND raw = 1 AND (expire_time IS NULL OR expire_time > ?)"; let row = connection .query_row(select, params![key, now], row_from_query) .optional() - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; if !self.statistics && !has_get_update(&self.eviction_policy) { return row .map(|row| fetch_row(&self.directory, row)) @@ -348,7 +356,7 @@ impl DiskStore for DiskcacheSqliteStore { let row = connection .query_row(select, params![key, now], row_from_query) .optional() - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let Some(row) = row else { if self.statistics { connection @@ -356,7 +364,7 @@ impl DiskStore for DiskcacheSqliteStore { "UPDATE Settings SET value = value + 1 WHERE key = 'misses'", [], ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } return Ok(None); }; @@ -369,14 +377,14 @@ impl DiskStore for DiskcacheSqliteStore { "UPDATE Settings SET value = value + 1 WHERE key = 'hits'", [], ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } else if !hit && self.statistics { connection .execute( "UPDATE Settings SET value = value + 1 WHERE key = 'misses'", [], ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } if has_get_update(&self.eviction_policy) && hit { let update = match self.eviction_policy.as_str() { @@ -384,16 +392,20 @@ impl DiskStore for DiskcacheSqliteStore { "least-frequently-used" => { "UPDATE Cache SET access_count = access_count + 1 WHERE rowid = ?" } - _ => return Err(Error::Unavailable), + policy => { + return Err(Error::unavailable(SettingFault::EvictionPolicy( + policy.to_owned(), + ))); + } }; if self.eviction_policy == "least-recently-used" { connection .execute(update, params![now, rowid]) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } else { connection .execute(update, params![rowid]) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } } value @@ -409,7 +421,7 @@ impl DiskStore for DiskcacheSqliteStore { ) -> Result<(), Error> { let columns = store_value(&self.directory, self.min_file_size, value)?; let new_filename = columns.filename.clone(); - let connection = self.connection.lock().map_err(|_| Error::Unavailable)?; + let connection = self.connection.lock()?; let result = transactional(&connection, |connection| { self.set_locked(connection, key, columns, expire_time, now) }); @@ -428,7 +440,7 @@ impl DiskStore for DiskcacheSqliteStore { } fn pop(&self, key: &str, now: f64) -> Result, Error> { - let connection = self.connection.lock().map_err(|_| Error::Unavailable)?; + let connection = self.connection.lock()?; let selected = transactional(&connection, |connection| { let row = connection .query_row( @@ -439,13 +451,13 @@ impl DiskStore for DiskcacheSqliteStore { row_from_query, ) .optional() - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let Some(row) = row else { return Ok(None); }; connection .execute("DELETE FROM Cache WHERE rowid = ?", params![row.rowid]) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; Ok(Some(row)) })?; let Some(row) = selected else { @@ -460,7 +472,7 @@ impl DiskStore for DiskcacheSqliteStore { } fn clear(&self) -> Result<(), Error> { - let connection = self.connection.lock().map_err(|_| Error::Unavailable)?; + let connection = self.connection.lock()?; let mut last_rowid = 0_i64; loop { let batch = transactional(&connection, |connection| { @@ -469,13 +481,13 @@ impl DiskStore for DiskcacheSqliteStore { "SELECT rowid, filename FROM Cache WHERE rowid > ? ORDER BY rowid LIMIT 100", ) - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .query_map(params![last_rowid], |row| { Ok((row.get::<_, i64>(0)?, row.get::<_, Option>(1)?)) }) - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .collect::, _>>() - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; if rows.is_empty() { return Ok(rows); } @@ -486,7 +498,7 @@ impl DiskStore for DiskcacheSqliteStore { .join(","); connection .execute(&format!("DELETE FROM Cache WHERE rowid IN ({ids})"), []) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; Ok(rows) })?; if batch.is_empty() { @@ -509,7 +521,7 @@ impl DiskStore for DiskcacheSqliteStore { now: f64, apply: &mut dyn FnMut(Option) -> Result<(StoredValue, Option), Error>, ) -> Result<(), Error> { - let connection = self.connection.lock().map_err(|_| Error::Unavailable)?; + let connection = self.connection.lock()?; let mut created_filename = None; let result = transactional(&connection, |connection| { let current = connection @@ -521,7 +533,7 @@ impl DiskStore for DiskcacheSqliteStore { row_from_query, ) .optional() - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .map(|row| fetch_row(&self.directory, row)) .transpose()? .flatten(); @@ -582,21 +594,35 @@ fn read_settings(connection: &Connection) -> Result, Erro }; statement .query_map([], |row| Ok((row.get(0)?, row.get(1)?))) - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .collect::, _>>() - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) +} + +#[derive(Debug, thiserror::Error)] +enum SettingFault { + #[error("unknown eviction policy {0:?}")] + EvictionPolicy(String), + #[error("setting {0:?} is not an SQLite pragma")] + NotPragma(String), + #[error("setting {0:?} has an unsupported value type")] + PragmaValue(String), } fn apply_pragma(connection: &Connection, key: &str, value: &Value) -> Result<(), Error> { - let pragma = key.strip_prefix("sqlite_").ok_or(Error::Unavailable)?; + let pragma = key + .strip_prefix("sqlite_") + .ok_or_else(|| Error::unavailable(SettingFault::NotPragma(key.to_owned())))?; match value { Value::Integer(value) => connection .pragma_update(None, pragma, value) - .map_err(|_| Error::Unavailable), + .map_err(Error::unavailable), Value::Text(value) => connection .pragma_update(None, pragma, value) - .map_err(|_| Error::Unavailable), - _ => Err(Error::Unavailable), + .map_err(Error::unavailable), + _ => Err(Error::unavailable(SettingFault::PragmaValue( + key.to_owned(), + ))), } } @@ -634,7 +660,7 @@ fn fetch_row(directory: &Path, row: Row) -> Result, Error> { Value::Text(value) => Ok(Some(StoredValue::Text(value))), Value::Integer(value) => Ok(Some(StoredValue::Integer(value))), Value::Real(value) => Ok(Some(StoredValue::Float(value))), - Value::Null => Err(Error::InvalidEntry), + Value::Null => Err(Error::InvalidEntry(None)), }, MODE_BINARY | MODE_PICKLE => { let bytes = match row.value { @@ -645,7 +671,7 @@ fn fetch_row(directory: &Path, row: Row) -> Result, Error> { }; value } - _ => return Err(Error::InvalidEntry), + _ => return Err(Error::InvalidEntry(None)), }; Ok(Some(if row.mode == MODE_BINARY { StoredValue::Bytes(bytes) @@ -663,24 +689,24 @@ fn fetch_row(directory: &Path, row: Row) -> Result, Error> { } Value::Blob(value) => value, Value::Text(value) => value.into_bytes(), - _ => return Err(Error::InvalidEntry), + _ => return Err(Error::InvalidEntry(None)), }; Ok(Some(StoredValue::Text( - String::from_utf8(bytes).map_err(|_| Error::InvalidEntry)?, + String::from_utf8(bytes).map_err(Error::invalid_entry)?, ))) } - _ => Err(Error::InvalidEntry), + _ => Err(Error::InvalidEntry(None)), } } fn read_file(directory: &Path, filename: Option<&str>) -> Result>, Error> { let Some(filename) = filename else { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); }; match fs::read(directory.join(filename)) { Ok(value) => Ok(Some(value)), Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None), - Err(_) => Err(Error::Unavailable), + Err(error) => Err(Error::unavailable(error)), } } @@ -712,7 +738,7 @@ fn store_value( let bytes = value.into_bytes(); let filename = write_file(directory, &bytes)?; Ok(StoredColumns { - size: i64::try_from(bytes.len()).map_err(|_| Error::Unavailable)?, + size: i64::try_from(bytes.len()).map_err(Error::unavailable)?, mode: MODE_TEXT, filename: Some(filename), value: None, @@ -727,7 +753,7 @@ fn store_value( StoredValue::Bytes(value) => { let filename = write_file(directory, &value)?; Ok(StoredColumns { - size: i64::try_from(value.len()).map_err(|_| Error::Unavailable)?, + size: i64::try_from(value.len()).map_err(Error::unavailable)?, mode: MODE_BINARY, filename: Some(filename), value: None, @@ -742,7 +768,7 @@ fn store_value( StoredValue::Pickle(value) => { let filename = write_file(directory, &value)?; Ok(StoredColumns { - size: i64::try_from(value.len()).map_err(|_| Error::Unavailable)?, + size: i64::try_from(value.len()).map_err(Error::unavailable)?, mode: MODE_PICKLE, filename: Some(filename), value: None, @@ -761,14 +787,14 @@ fn write_file(directory: &Path, bytes: &[u8]) -> Result { let filename = format!("{}/{}/{}.val", &hex[..2], &hex[2..4], &hex[4..]); let path = directory.join(&filename); if let Some(parent) = path.parent() { - fs::create_dir_all(parent).map_err(|_| Error::Unavailable)?; + fs::create_dir_all(parent).map_err(Error::unavailable)?; } let mut file = OpenOptions::new() .write(true) .create_new(true) .open(path) - .map_err(|_| Error::Unavailable)?; - file.write_all(bytes).map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; + file.write_all(bytes).map_err(Error::unavailable)?; Ok(filename) } @@ -789,12 +815,12 @@ fn transactional( ) -> Result { connection .execute_batch("BEGIN IMMEDIATE") - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; match operation(connection) { Ok(value) => { connection .execute_batch("COMMIT") - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; Ok(value) } Err(error) => { diff --git a/litellm-rust/crates/cache-disk/tests/python_compat.rs b/litellm-rust/crates/cache-disk/tests/python_compat.rs index 9cbef8573bd..e167b0adb37 100644 --- a/litellm-rust/crates/cache-disk/tests/python_compat.rs +++ b/litellm-rust/crates/cache-disk/tests/python_compat.rs @@ -84,7 +84,7 @@ fn python_read_cases(#[case] row: StoredValue, #[case] expected: ReadExpectation match expected { ReadExpectation::Bytes(expected) => assert_eq!(result.unwrap().unwrap(), expected), ReadExpectation::Miss => assert_eq!(result.unwrap(), None), - ReadExpectation::Invalid => assert!(matches!(result, Err(Error::InvalidEntry))), + ReadExpectation::Invalid => assert!(matches!(result, Err(Error::InvalidEntry(_)))), } } diff --git a/litellm-rust/crates/cache-gcs/Cargo.toml b/litellm-rust/crates/cache-gcs/Cargo.toml index 91630879cbe..dbd8c9ac908 100644 --- a/litellm-rust/crates/cache-gcs/Cargo.toml +++ b/litellm-rust/crates/cache-gcs/Cargo.toml @@ -13,9 +13,11 @@ litellm-auth-types.workspace = true litellm-cache.workspace = true percent-encoding.workspace = true reqwest.workspace = true +thiserror.workspace = true tokio.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-http = { workspace = true, features = ["test-support"] } litellm-cache-testing.workspace = true rstest.workspace = true diff --git a/litellm-rust/crates/cache-gcs/src/cache.rs b/litellm-rust/crates/cache-gcs/src/cache.rs index bad81573cb4..a6b08445bc8 100644 --- a/litellm-rust/crates/cache-gcs/src/cache.rs +++ b/litellm-rust/crates/cache-gcs/src/cache.rs @@ -25,6 +25,14 @@ pub fn key_prefix(gcs_path: Option<&str>) -> String { } } +#[derive(Debug, thiserror::Error)] +#[error("GCS answered {0}")] +struct UnexpectedStatus(reqwest::StatusCode); + +#[derive(Debug, thiserror::Error)] +#[error("GCS cache worker thread panicked")] +struct WorkerPanicked; + #[derive(Clone, Debug)] pub struct GcsConfig { pub bucket_name: String, @@ -114,9 +122,9 @@ impl GcsCache { .body(payload) .send() .await - .map_err(|_| Error::Unavailable)?; + .map_err(|error| Error::unavailable(litellm_http::Error::from(error)))?; if !response.status().is_success() { - return Err(Error::Unavailable); + return Err(Error::unavailable(UnexpectedStatus(response.status()))); } Ok(()) } @@ -135,18 +143,18 @@ impl GcsCache { .header(reqwest::header::CONTENT_TYPE, "application/json") .send() .await - .map_err(|_| Error::Unavailable)?; + .map_err(|error| Error::unavailable(litellm_http::Error::from(error)))?; if response.status() == reqwest::StatusCode::NOT_FOUND { return Ok(None); } if !response.status().is_success() { - return Err(Error::Unavailable); + return Err(Error::unavailable(UnexpectedStatus(response.status()))); } - let body = response.bytes().await.map_err(|_| Error::Unavailable)?; - self.codec - .decode(&body) - .map(Some) - .map_err(|_| Error::InvalidEntry) + let body = response + .bytes() + .await + .map_err(|error| Error::unavailable(litellm_http::Error::from(error)))?; + self.codec.decode(&body).map(Some) } fn run_sync(future: F) -> Result @@ -158,7 +166,7 @@ impl GcsCache { tokio::runtime::Builder::new_current_thread() .enable_all() .build() - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) .and_then(|runtime| runtime.block_on(future)) }; match tokio::runtime::Handle::try_current() { @@ -169,7 +177,7 @@ impl GcsCache { scope .spawn(|| run(future)) .join() - .map_err(|_| Error::Unavailable) + .map_err(|_| Error::unavailable(WorkerPanicked)) .and_then(|result| result) }), Err(_) => run(future), @@ -242,7 +250,7 @@ impl BatchCache for GcsCache { match self.async_get_cache(&key, &context).await { Ok(Some(value)) => Ok(BatchEntry::Hit(value)), Ok(None) => Ok(BatchEntry::Miss), - Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid), + Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid), Err(error) => Err(error), } } diff --git a/litellm-rust/crates/cache-gcs/src/token.rs b/litellm-rust/crates/cache-gcs/src/token.rs index adb601c276c..079ec44dbce 100644 --- a/litellm-rust/crates/cache-gcs/src/token.rs +++ b/litellm-rust/crates/cache-gcs/src/token.rs @@ -30,7 +30,7 @@ impl TokenSource for GcpTokenSource { self.auth .access_token(&self.config, &|name| std::env::var(name).ok()) .await - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) } } diff --git a/litellm-rust/crates/cache-gcs/tests/cache.rs b/litellm-rust/crates/cache-gcs/tests/cache.rs index 3e92c3fca19..f8dba94ea8e 100644 --- a/litellm-rust/crates/cache-gcs/tests/cache.rs +++ b/litellm-rust/crates/cache-gcs/tests/cache.rs @@ -55,19 +55,27 @@ async fn set_writes_encoded_object_and_headers(#[future(awt)] server: MockServer Ok(Some(json!({"value": "entry"}))) )] #[case::missing("missing", ResponseTemplate::new(404), Ok(None))] -#[case::server_error("server-error", ResponseTemplate::new(500), Err(Error::Unavailable))] -#[case::unauthorized("unauthorized", ResponseTemplate::new(401), Err(Error::Unavailable))] +#[case::server_error( + "server-error", + ResponseTemplate::new(500), + Err("cache is unavailable: GCS answered 500 Internal Server Error") +)] +#[case::unauthorized( + "unauthorized", + ResponseTemplate::new(401), + Err("cache is unavailable: GCS answered 401 Unauthorized") +)] #[case::invalid( "invalid", ResponseTemplate::new(200).set_body_string("not json"), - Err(Error::InvalidEntry) + Err("invalid cache entry: ") )] #[tokio::test] async fn get_maps_statuses_and_decode_failures( #[future(awt)] server: MockServer, #[case] key: &str, #[case] response: ResponseTemplate, - #[case] expected: Result, Error>, + #[case] expected: Result, &str>, ) { Mock::given(method("GET")) .and(path(format!("/storage/v1/b/bucket/o/{key}"))) @@ -76,8 +84,21 @@ async fn get_maps_statuses_and_decode_failures( .mount(&server) .await; let cache = support::cache(&server, None); - assert_eq!(cache.get_cache(key, &context()), expected); - assert_eq!(cache.async_get_cache(key, &context()).await, expected); + let outcome = |result: Result, Error>| { + result.map_err(|error| litellm_error::Report::new(&error).to_string()) + }; + let sync = outcome(cache.get_cache(key, &context())); + let asynchronous = outcome(cache.async_get_cache(key, &context()).await); + match expected { + Ok(value) => { + assert_eq!(sync, Ok(value.clone())); + assert_eq!(asynchronous, Ok(value)); + } + Err(prefix) => { + assert!(sync.unwrap_err().starts_with(prefix)); + assert!(asynchronous.unwrap_err().starts_with(prefix)); + } + } } #[rstest] @@ -257,7 +278,7 @@ struct FailingTokenSource; impl TokenSource for FailingTokenSource { fn bearer_token(&self) -> Pin> + Send + '_>> { - Box::pin(async { Err(Error::Unavailable) }) + Box::pin(async { Err(Error::Unavailable(None)) }) } } @@ -267,14 +288,14 @@ async fn token_source_failure_skips_http(#[future(awt)] server: MockServer) { let cache = support::cache_with_token(&server, None, Arc::new(FailingTokenSource)); assert_eq!( cache.get_cache("key", &context()).unwrap_err(), - Error::Unavailable + Error::Unavailable(None) ); assert_eq!( cache .async_set_cache("key", json!(1), context()) .await .unwrap_err(), - Error::Unavailable + Error::Unavailable(None) ); assert_eq!(server.received_requests().await.unwrap().len(), 0); } diff --git a/litellm-rust/crates/cache-memory/src/cache.rs b/litellm-rust/crates/cache-memory/src/cache.rs index 74476559cbe..73fbf14b07a 100644 --- a/litellm-rust/crates/cache-memory/src/cache.rs +++ b/litellm-rust/crates/cache-memory/src/cache.rs @@ -2,7 +2,7 @@ use std::{ cmp::Reverse, collections::{BinaryHeap, HashMap, HashSet}, hash::Hash, - sync::{Arc, Mutex}, + sync::{Arc, Mutex, MutexGuard}, time::{Duration, SystemTime, UNIX_EPOCH}, }; @@ -84,6 +84,10 @@ impl InMemoryCache { } } + fn lock(&self) -> Result>, Error> { + Ok(self.state.lock()?) + } + pub fn set_cache( &self, key: impl Into, @@ -94,13 +98,13 @@ impl InMemoryCache { return Ok(CacheWrite::Disabled); } let now = (self.now)(); - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; self.store(&mut state, key.into(), value, ttl, now) } pub fn get_cache(&self, key: &str) -> Result, Error> { let now = (self.now)(); - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; if state .expirations .get(key) @@ -124,7 +128,7 @@ impl InMemoryCache { /// fits. pub fn evict_cache(&self) -> Result<(), Error> { let now = (self.now)(); - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; Self::evict(&mut state, self.max_size_in_memory, now, None); Ok(()) } @@ -132,7 +136,7 @@ impl InMemoryCache { /// `evict_element_if_expired`: `true` when `key` had expired and was removed. pub fn evict_element_if_expired(&self, key: &str) -> Result { let now = (self.now)(); - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; let expired = state .expirations .get(key) @@ -153,12 +157,7 @@ impl InMemoryCache { /// The number of stored entries, expired ones included until they are evicted. pub fn len(&self) -> Result { - Ok(self - .state - .lock() - .map_err(|_| Error::Unavailable)? - .values - .len()) + Ok(self.lock()?.values.len()) } pub fn is_empty(&self) -> Result { @@ -167,12 +166,7 @@ impl InMemoryCache { /// Entries in the expiration heap, stale ones included; bounded by eviction. pub fn expiration_heap_len(&self) -> Result { - Ok(self - .state - .lock() - .map_err(|_| Error::Unavailable)? - .expiration_heap - .len()) + Ok(self.lock()?.expiration_heap.len()) } pub fn max_size_in_memory(&self) -> usize { @@ -184,13 +178,7 @@ impl InMemoryCache { } pub fn expires_at(&self, key: &str) -> Result, Error> { - Ok(self - .state - .lock() - .map_err(|_| Error::Unavailable)? - .expirations - .get(key) - .copied()) + Ok(self.lock()?.expirations.get(key).copied()) } pub async fn async_get_ttl(&self, key: &str) -> Result, Error> { @@ -198,7 +186,7 @@ impl InMemoryCache { } pub async fn async_get_oldest_n_keys(&self, count: usize) -> Result, Error> { - let state = self.state.lock().map_err(|_| Error::Unavailable)?; + let state = self.lock()?; let mut expirations = state .expirations .iter() @@ -213,13 +201,13 @@ impl InMemoryCache { } pub fn delete_cache(&self, key: &str) -> Result<(), Error> { - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; Self::remove(&mut state, key); Ok(()) } pub fn flush_cache(&self) -> Result<(), Error> { - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; state.values.clear(); state.expirations.clear(); state.expiration_heap.clear(); @@ -316,7 +304,7 @@ where return Ok(candidate); } let now = (self.now)(); - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; Self::evict(&mut state, self.max_size_in_memory, now, Some(key)); let existing = state .values @@ -351,7 +339,7 @@ impl CounterCache for InMemoryCache { return Ok(amount); } let now = (self.now)(); - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; let value = Self::live(&mut state, key, now).unwrap_or_default() + amount; self.store(&mut state, key.into(), value, self.get_ttl(&context), now)?; Ok(value) @@ -424,7 +412,7 @@ where return Ok(values); } let now = (self.now)(); - let mut state = self.state.lock().map_err(|_| Error::Unavailable)?; + let mut state = self.lock()?; let mut stored = Self::live(&mut state, key, now).unwrap_or_default(); stored.extend(values.iter().cloned()); self.store(&mut state, key.into(), stored, ttl, now)?; diff --git a/litellm-rust/crates/cache-memory/tests/cache.rs b/litellm-rust/crates/cache-memory/tests/cache.rs index ce7d6ac8a98..e6c74e5ccda 100644 --- a/litellm-rust/crates/cache-memory/tests/cache.rs +++ b/litellm-rust/crates/cache-memory/tests/cache.rs @@ -52,7 +52,7 @@ fn measured(capacity: usize) -> InMemoryCache { Some(4), Some(Arc::new(|value: &String| { if value.is_empty() { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } Ok(value.len()) })), @@ -275,7 +275,7 @@ fn evict_element_if_expired_reports_removal(clock: Clock) { #[case::fits("ok", Ok(true))] #[case::at_limit("four", Ok(true))] #[case::too_large("oversized", Ok(false))] -#[case::measure_error("", Err(Error::InvalidEntry))] +#[case::measure_error("", Err(Error::InvalidEntry(None)))] fn check_value_size_applies_the_entry_limit( #[case] value: &str, #[case] expected: Result, @@ -310,7 +310,7 @@ fn disabled_size_limited_and_validated_writes_are_observable() { assert_eq!(cache.get_cache("small").unwrap(), Some("ok".into())); assert_eq!( cache.set_cache("invalid", String::new(), None), - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); assert_eq!(cache.get_cache("invalid").unwrap(), None); cache.delete_cache("small").unwrap(); diff --git a/litellm-rust/crates/cache-qdrant-semantic/Cargo.toml b/litellm-rust/crates/cache-qdrant-semantic/Cargo.toml index a44bef0a731..8a4caa8def7 100644 --- a/litellm-rust/crates/cache-qdrant-semantic/Cargo.toml +++ b/litellm-rust/crates/cache-qdrant-semantic/Cargo.toml @@ -18,6 +18,7 @@ tokio.workspace = true uuid.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-http = { workspace = true, features = ["test-support"] } futures-executor = "0.3" litellm-cache-testing.workspace = true diff --git a/litellm-rust/crates/cache-qdrant-semantic/src/cache.rs b/litellm-rust/crates/cache-qdrant-semantic/src/cache.rs index a140e0af174..9c93cc45386 100644 --- a/litellm-rust/crates/cache-qdrant-semantic/src/cache.rs +++ b/litellm-rust/crates/cache-qdrant-semantic/src/cache.rs @@ -36,7 +36,7 @@ impl QdrantSemanticCache { let exists = client .collection_exists(config.collection_name.clone()) .await - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; if !exists { client .create_collection( @@ -48,7 +48,7 @@ impl QdrantSemanticCache { .quantization_config(quantization(&config.quantization)), ) .await - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } let _ = client .create_field_index(CreateFieldIndexCollectionBuilder::new( @@ -99,13 +99,13 @@ impl QdrantSemanticCache { .async_embed(&prompt, context.metadata.as_ref()) .await?; let response = - String::from_utf8(self.codec.encode(&value)?).map_err(|_| Error::InvalidEntry)?; + String::from_utf8(self.codec.encode(&value)?).map_err(Error::invalid_entry)?; let payload = Payload::try_from(json!({ "litellm_cache_key": key, "text": prompt, "response": response, })) - .map_err(|_| Error::InvalidEntry)?; + .map_err(Error::invalid_entry)?; self.client .upsert_points( UpsertPointsBuilder::new( @@ -119,7 +119,7 @@ impl QdrantSemanticCache { .wait(true), ) .await - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; Ok(()) } @@ -152,7 +152,7 @@ impl QdrantSemanticCache { ), ) .await - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let Some(point) = result.result.into_iter().next() else { return Ok(SemanticLookup::miss(Some(0.0))); }; @@ -170,7 +170,7 @@ impl QdrantSemanticCache { let response = payload .get("response") .and_then(Value::as_str) - .ok_or(Error::InvalidEntry)?; + .ok_or(Error::InvalidEntry(None))?; Ok(SemanticLookup { value: Some(self.codec.decode(response.as_bytes())?), similarity: Some(similarity), diff --git a/litellm-rust/crates/cache-qdrant-semantic/src/embedder.rs b/litellm-rust/crates/cache-qdrant-semantic/src/embedder.rs index b7fbcd9b02d..bad780bc57f 100644 --- a/litellm-rust/crates/cache-qdrant-semantic/src/embedder.rs +++ b/litellm-rust/crates/cache-qdrant-semantic/src/embedder.rs @@ -55,10 +55,9 @@ impl Embedder for OpenAiEmbedder { } .send() .await - .map_err(|_| Error::Unavailable)? - .error_for_status() - .map_err(|_| Error::Unavailable)?; - let body: Value = response.json().await.map_err(|_| Error::Unavailable)?; + .and_then(reqwest::Response::error_for_status) + .map_err(http_error)?; + let body: Value = response.json().await.map_err(http_error)?; body.get("data") .and_then(Value::as_array) .and_then(|data| data.first()) @@ -70,6 +69,14 @@ impl Embedder for OpenAiEmbedder { .map(|value| value.as_f64().map(|value| value as f32)) .collect::>>() }) - .ok_or(Error::Unavailable) + .ok_or(Error::unavailable(MissingEmbedding)) } } + +fn http_error(error: reqwest::Error) -> Error { + Error::unavailable(litellm_http::Error::from(error)) +} + +#[derive(Debug, thiserror::Error)] +#[error("embeddings response has no embedding")] +struct MissingEmbedding; diff --git a/litellm-rust/crates/cache-qdrant-semantic/tests/embedder.rs b/litellm-rust/crates/cache-qdrant-semantic/tests/embedder.rs index 71fc986616e..42261d21524 100644 --- a/litellm-rust/crates/cache-qdrant-semantic/tests/embedder.rs +++ b/litellm-rust/crates/cache-qdrant-semantic/tests/embedder.rs @@ -134,13 +134,19 @@ async fn posts_embeddings_request_and_parses_vector() { } #[rstest] -#[case::error_status("500 Internal Server Error", "{}", 0, None, Err(Error::Unavailable))] +#[case::error_status( + "500 Internal Server Error", + "{}", + 0, + None, + Err(Error::Unavailable(None)) +)] #[case::timed_out( "200 OK", r#"{"data":[{"embedding":[0.1,0.2]}]}"#, 500, Some(Duration::from_millis(200)), - Err(Error::Unavailable) + Err(Error::Unavailable(None)) )] #[case::within_timeout( "200 OK", @@ -149,7 +155,7 @@ async fn posts_embeddings_request_and_parses_vector() { Some(Duration::from_secs(1)), Ok(vec![0.1, 0.2]) )] -#[case::missing_embedding("200 OK", r#"{"data":[]}"#, 0, None, Err(Error::Unavailable))] +#[case::missing_embedding("200 OK", r#"{"data":[]}"#, 0, None, Err(Error::Unavailable(None)))] #[tokio::test] async fn status_timeout_and_body_errors_are_unavailable( #[case] status: &str, @@ -164,7 +170,14 @@ async fn status_timeout_and_body_errors_are_unavailable( litellm_http::Client::for_test(), config(server.base_url(), timeout), ); - assert_eq!(embedder.async_embed("hello", None).await, expected); + let result = embedder.async_embed("hello", None).await; + if let Err(error) = &result { + let report = litellm_error::Report::new(error).to_string(); + assert!(!report.contains(&server.base_url()), "{report}"); + } + let kind = + |result: Result, Error>| result.map_err(|error| std::mem::discriminant(&error)); + assert_eq!(kind(result), kind(expected)); } #[rstest] diff --git a/litellm-rust/crates/cache-qdrant-semantic/tests/qdrant.rs b/litellm-rust/crates/cache-qdrant-semantic/tests/qdrant.rs index fe25c503a36..b0764b4470b 100644 --- a/litellm-rust/crates/cache-qdrant-semantic/tests/qdrant.rs +++ b/litellm-rust/crates/cache-qdrant-semantic/tests/qdrant.rs @@ -53,7 +53,10 @@ impl Embedder for FixedEmbedder { .lock() .unwrap() .push((input.to_owned(), metadata.cloned())); - self.vectors.get(input).cloned().ok_or(Error::Unavailable) + self.vectors + .get(input) + .cloned() + .ok_or(Error::Unavailable(None)) } } @@ -396,9 +399,9 @@ async fn lookup_reports_python_semantic_similarity( #[rstest] #[case::codec_decodes_the_payload(Some(json!("{\"a\":1}")), Ok(Some(json!({"a": 1}))))] -#[case::undecodable_response(Some(json!("not json")), Err(Error::InvalidEntry))] -#[case::non_string_response(Some(json!(1)), Err(Error::InvalidEntry))] -#[case::missing_response(None, Err(Error::InvalidEntry))] +#[case::undecodable_response(Some(json!("not json")), Err(Error::InvalidEntry(None)))] +#[case::non_string_response(Some(json!(1)), Err(Error::InvalidEntry(None)))] +#[case::missing_response(None, Err(Error::InvalidEntry(None)))] #[tokio::test(flavor = "multi_thread")] async fn stored_responses_go_through_the_codec( #[case] response: Option, @@ -419,9 +422,12 @@ async fn stored_responses_go_through_the_codec( .into(), }); + let kind = |result: Result, Error>| { + result.map_err(|error| std::mem::discriminant(&error)) + }; assert_eq!( - cache.async_get_cache("key", &context("hello")).await, - expected + kind(cache.async_get_cache("key", &context("hello")).await), + kind(expected) ); server.stop(); } @@ -434,7 +440,7 @@ async fn embedding_failures_propagate() { assert_eq!( cache.async_get_cache("key", &context("unknown")).await, - Err(Error::Unavailable) + Err(Error::Unavailable(None)) ); server.stop(); } @@ -493,10 +499,10 @@ async fn stopped_qdrant_server_maps_to_unavailable() { let cache = connect(&server, [("hello", vec![1.0, 0.0])]).await; server.stop(); tokio::time::sleep(Duration::from_millis(50)).await; - assert_eq!( + assert!(matches!( cache.async_get_cache("key", &context("hello")).await, - Err(Error::Unavailable) - ); + Err(Error::Unavailable(Some(_))) + )); } /// `_payload_matches_cache_key` compares `str(cached_key) == str(key)`, so a point whose stored diff --git a/litellm-rust/crates/cache-redis-semantic/src/cache.rs b/litellm-rust/crates/cache-redis-semantic/src/cache.rs index 02feb3cbdf4..42fc6ae585f 100644 --- a/litellm-rust/crates/cache-redis-semantic/src/cache.rs +++ b/litellm-rust/crates/cache-redis-semantic/src/cache.rs @@ -63,13 +63,13 @@ impl Inner { .arg(CACHE_KEY_FIELD) .arg(tag) .query::<()>(connection) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; if let Some(ttl) = ttl { redis::cmd("EXPIRE") .arg(&hash_key) .arg(ttl_seconds(ttl)) .query::<()>(connection) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } Ok(()) } @@ -111,7 +111,7 @@ impl Inner { .arg("vector") .arg(vector_buffer(vector)) .query::(connection) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let Some(fields) = first_document(&result) else { return Ok(SemanticLookup::miss(Some(0.0))); }; diff --git a/litellm-rust/crates/cache-redis-semantic/src/index.rs b/litellm-rust/crates/cache-redis-semantic/src/index.rs index c141e47003a..23401938a0e 100644 --- a/litellm-rust/crates/cache-redis-semantic/src/index.rs +++ b/litellm-rust/crates/cache-redis-semantic/src/index.rs @@ -40,10 +40,10 @@ impl Index { Some(false) => self.isolated(connection, dims)?, None => match create_index(connection, &self.name, dims) { Ok(()) => self.name.clone(), - Err(_) => match index_compatible(connection, &self.name, dims)? { + Err(error) => match index_compatible(connection, &self.name, dims)? { Some(true) => self.name.clone(), Some(false) => self.isolated(connection, dims)?, - None => return Err(Error::Unavailable), + None => return Err(error), }, }, }; @@ -59,7 +59,7 @@ impl Index { redis::cmd("FT.DROPINDEX") .arg(&name) .query::<()>(connection) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; create_index(connection, &name, dims)?; Ok(name) } @@ -109,7 +109,7 @@ fn create_index(connection: &mut ConnectionRef<'_>, name: &str, dims: usize) -> .arg("SEPARATOR") .arg(",") .query::<()>(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) } fn index_compatible( @@ -123,14 +123,18 @@ fn index_compatible( { Ok(info) => info, Err(error) if unknown_index(&error) => return Ok(None), - Err(_) => return Err(Error::Unavailable), + Err(error) => return Err(Error::unavailable(error)), }; Ok(Some(schema_compatible(&info, dims))) } fn unknown_index(error: &redis::RedisError) -> bool { - let message = error.to_string().to_lowercase(); - message.contains("unknown") && message.contains("index") + error + .code() + .is_some_and(|code| code.eq_ignore_ascii_case("unknown")) + && error + .detail() + .is_some_and(|detail| detail.to_ascii_lowercase().starts_with("index")) } struct Attribute { diff --git a/litellm-rust/crates/cache-redis-semantic/tests/cache.rs b/litellm-rust/crates/cache-redis-semantic/tests/cache.rs index fa4ac00e767..cc42f4c3924 100644 --- a/litellm-rust/crates/cache-redis-semantic/tests/cache.rs +++ b/litellm-rust/crates/cache-redis-semantic/tests/cache.rs @@ -79,7 +79,7 @@ fn s(value: &str) -> redis::Value { } fn unknown_index_error() -> redis::RedisError { - redis::RedisError::from((redis::ErrorKind::Extension, "Unknown index name")) + support::server_error("Unknown index name") } fn info_missing(index: &str) -> MockCmd { @@ -320,7 +320,7 @@ fn store_writes_the_redisvl_hash( #[case::below_distance_threshold("key1", "0.05", None, Ok(Some(entry())))] #[case::above_distance_threshold("key1", "0.5", None, Ok(None))] #[case::other_cache_key("other", "0.05", None, Ok(None))] -#[case::malformed_response("key1", "0.05", Some(b"not json!".as_slice()), Err(Error::InvalidEntry))] +#[case::malformed_response("key1", "0.05", Some(b"not json!".as_slice()), Err(Error::InvalidEntry(None)))] fn lookup_applies_threshold_scope_and_codec( #[case] stored_tag: &str, #[case] distance: &str, @@ -343,7 +343,10 @@ fn lookup_applies_threshold_scope_and_codec( FakeEmbedder::new(&[]), ); - assert_eq!(cache.get_cache("key1", &context), expected); + let kind = |result: Result, Error>| { + result.map_err(|error| std::mem::discriminant(&error)) + }; + assert_eq!(kind(cache.get_cache("key1", &context)), kind(expected)); } #[rstest] @@ -451,10 +454,7 @@ fn create_index_race_rechecks_schema_and_stores(entry: Value, context: SemanticC info_missing(INDEX), MockCmd::new( create_index_command(INDEX, 3), - Err::<&str, _>(redis::RedisError::from(( - redis::ErrorKind::Extension, - "Index already exists", - ))), + Err::<&str, _>(support::server_error("Index already exists")), ), info(INDEX, compatible_info(3)), hset(INDEX, PROMPT, "key1", &VECTOR, &entry), @@ -673,10 +673,10 @@ fn shared_base_index_across_dimensions_replaces_the_isolated_index(entry: Value) .set_cache("key1", entry.clone(), &context()) .unwrap(); - assert_eq!( - worker_b.get_cache("key1", &context()).unwrap_err(), - Error::Unavailable - ); + assert!(matches!( + worker_b.get_cache("key1", &context()), + Err(Error::Unavailable(Some(_))) + )); } #[fixture] @@ -732,10 +732,10 @@ fn live_shared_index_is_replaced_across_dimensions(redis_stack_url: Option { let name = text(1); if state.indexes.contains_key(&name) { - return Err(error("Index already exists")); + return Err(server_error("Index already exists")); } let position = |token: &str| args.iter().position(|arg| arg == token.as_bytes()); let prefix = args[position("PREFIX").unwrap() + 2].clone(); @@ -94,7 +94,7 @@ impl FakeSearch { let index = state .indexes .get(&text(1)) - .ok_or_else(|| error("Unknown index name"))?; + .ok_or_else(|| server_error("Unknown index name"))?; Ok(index_info(index)) } "FT.DROPINDEX" => { @@ -118,7 +118,7 @@ impl FakeSearch { let index = state .indexes .get(&text(1)) - .ok_or_else(|| error("no such index"))?; + .ok_or_else(|| server_error("no such index"))?; let query = text(2); let tag = query_tag(&query); let params = args.iter().position(|arg| arg == b"PARAMS").unwrap(); @@ -155,7 +155,7 @@ impl FakeSearch { ])) } "PING" => Ok(redis::Value::SimpleString("PONG".into())), - _ => Err(error("unsupported command")), + _ => Err(server_error("unsupported command")), } } } @@ -192,8 +192,11 @@ impl redis::ConnectionLike for FakeSearch { } } -fn error(message: &'static str) -> redis::RedisError { - redis::RedisError::from((redis::ErrorKind::Extension, message)) +pub fn server_error(reply: &str) -> redis::RedisError { + redis::parse_redis_value(format!("-{reply}\r\n").as_bytes()) + .expect("error reply") + .extract_error() + .expect_err("server error") } fn bulk(bytes: &[u8]) -> redis::Value { diff --git a/litellm-rust/crates/cache-redis/Cargo.toml b/litellm-rust/crates/cache-redis/Cargo.toml index bebfcafd98a..f7621299506 100644 --- a/litellm-rust/crates/cache-redis/Cargo.toml +++ b/litellm-rust/crates/cache-redis/Cargo.toml @@ -7,8 +7,10 @@ repository.workspace = true [dependencies] litellm-cache.workspace = true +litellm-error.workspace = true redis = { workspace = true, features = ["cluster", "tls-rustls"] } r2d2 = "0.8.10" +thiserror.workspace = true tokio.workspace = true [dev-dependencies] diff --git a/litellm-rust/crates/cache-redis/src/cache.rs b/litellm-rust/crates/cache-redis/src/cache.rs index ebcf0b6916b..b873a2c0282 100644 --- a/litellm-rust/crates/cache-redis/src/cache.rs +++ b/litellm-rust/crates/cache-redis/src/cache.rs @@ -141,7 +141,7 @@ where redis::Value::Nil => Ok(None), redis::Value::BulkString(bytes) => self.codec.decode(&bytes).map(Some), redis::Value::SimpleString(text) => self.codec.decode(text.as_bytes()).map(Some), - _ => Err(Error::InvalidEntry), + _ => Err(Error::InvalidEntry(None)), } } @@ -152,7 +152,7 @@ where match self.decode_response(value) { Ok(Some(value)) => Ok(BatchEntry::Hit(value)), Ok(None) => Ok(BatchEntry::Miss), - Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid), + Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid), Err(error) => Err(error), } } diff --git a/litellm-rust/crates/cache-redis/src/claim.rs b/litellm-rust/crates/cache-redis/src/claim.rs index e4cbe8c8d5c..c57a1b03395 100644 --- a/litellm-rust/crates/cache-redis/src/claim.rs +++ b/litellm-rust/crates/cache-redis/src/claim.rs @@ -17,7 +17,7 @@ fn stored_bytes(value: redis::Value) -> Result>, Error> { redis::Value::Nil => Ok(None), redis::Value::BulkString(bytes) => Ok(Some(bytes)), redis::Value::SimpleString(text) => Ok(Some(text.into_bytes())), - _ => Err(Error::InvalidEntry), + _ => Err(Error::InvalidEntry(None)), } } @@ -37,13 +37,13 @@ where { let payload = codec.encode(&candidate)?; if payload.is_empty() { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } for _ in 0..CLAIM_ATTEMPTS { let current = stored_bytes( connection .get::<_, redis::Value>(key) - .map_err(|_| Error::Unavailable)?, + .map_err(Error::unavailable)?, )? .filter(|bytes| !bytes.is_empty()); let existing = current @@ -63,14 +63,18 @@ where .arg(write) .arg(u8::from(refresh)) .query::(connection) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; if applied { return Ok(existing.unwrap_or(candidate)); } } - Err(Error::Unavailable) + Err(Error::unavailable(ClaimContended)) } +#[derive(Debug, thiserror::Error)] +#[error("claim kept losing the race after {CLAIM_ATTEMPTS} attempts")] +struct ClaimContended; + impl ClaimCache for RedisCache where S: CacheCodec + Clone + 'static, diff --git a/litellm-rust/crates/cache-redis/src/connection.rs b/litellm-rust/crates/cache-redis/src/connection.rs index 2f58e2a9b80..99befa16ffa 100644 --- a/litellm-rust/crates/cache-redis/src/connection.rs +++ b/litellm-rust/crates/cache-redis/src/connection.rs @@ -36,19 +36,19 @@ where ) -> Result { match self { Self::Pool(pool) => { - let mut pooled = pool.get().map_err(|_| Error::Unavailable)?; + let mut pooled = pool.get().map_err(Error::unavailable)?; let result = operation(&mut ConnectionRef::Node(&mut pooled.connection)); - pooled.failed = matches!(result, Err(Error::Unavailable)); + pooled.failed = matches!(result, Err(Error::Unavailable(_))); result } Self::Cluster(pool) => { - let mut pooled = pool.get().map_err(|_| Error::Unavailable)?; + let mut pooled = pool.get().map_err(Error::unavailable)?; let result = operation(&mut ConnectionRef::Cluster(&mut pooled.connection)); - pooled.failed = matches!(result, Err(Error::Unavailable)); + pooled.failed = matches!(result, Err(Error::Unavailable(_))); result } Self::Fixed(connection) => { - let mut connection = connection.lock().map_err(|_| Error::Unavailable)?; + let mut connection = connection.lock()?; operation(&mut ConnectionRef::Node(&mut *connection)) } } @@ -61,7 +61,7 @@ where { tokio::task::spawn_blocking(move || connections.execute(operation)) .await - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? } pub fn fixed(connection: C) -> Self { @@ -95,7 +95,7 @@ fn pool(manager: M) -> Result, Error> { .connection_timeout(REDIS_TIMEOUT) .test_on_check_out(false) .build(manager) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) } fn close_idle(pool: &r2d2::Pool) @@ -123,7 +123,7 @@ impl ConnectionManager { fn open(url: &str) -> Result { redis::Client::open(url) .map(Self) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) } } @@ -151,14 +151,26 @@ impl r2d2::ManageConnection for ConnectionManager { } } +#[derive(Debug, thiserror::Error)] +enum TopologyFault { + #[error("Redis Cluster needs at least one startup node")] + NoStartupNodes, + #[error("Redis Cluster nodes need a TCP address")] + NonTcpAddress, + #[error("Redis Cluster answered a fan-out command without a per-node map")] + FanOutReply, + #[error("scan target does not match the connection topology")] + ScanTarget, +} + pub(crate) struct ClusterConnectionManager(ClusterClient); impl ClusterConnectionManager { fn open(url: &str, startup_nodes: &[RedisNode]) -> Result { if startup_nodes.is_empty() { - return Err(Error::Unavailable); + return Err(Error::unavailable(TopologyFault::NoStartupNodes)); } - let info = url.into_connection_info().map_err(|_| Error::Unavailable)?; + let info = url.into_connection_info().map_err(Error::unavailable)?; let nodes = startup_nodes .iter() .map(|node| node_info(&info, node)) @@ -168,7 +180,7 @@ impl ClusterConnectionManager { .response_timeout(REDIS_TIMEOUT) .build() .map(Self) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) } } @@ -185,7 +197,7 @@ fn node_info(info: &ConnectionInfo, node: &RedisNode) -> Result return Err(Error::Unavailable), + _ => return Err(Error::unavailable(TopologyFault::NonTcpAddress)), }; Ok(info.clone().set_addr(addr)) } @@ -298,7 +310,7 @@ impl ConnectionRef<'_> { .and_then(|replies| pipeline.query(&mut Replies(Some(replies)))) } } - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) } pub(crate) fn scan( @@ -311,7 +323,7 @@ impl ConnectionRef<'_> { Self::Node(connection) => { let page = scan_command(0, pattern, count) .query::(*connection) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; vec![(None, page)] } Self::Cluster(connection) => connection @@ -322,7 +334,7 @@ impl ConnectionRef<'_> { Some(ResponsePolicy::Special), )), ) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) .and_then(primary_pages)? .into_iter() .map(|(node, page)| (Some(node), page)) @@ -362,7 +374,7 @@ impl ConnectionRef<'_> { pub(crate) fn node_text(&mut self, command: &redis::Cmd) -> Result { match self { - Self::Node(connection) => command.query(*connection).map_err(|_| Error::Unavailable), + Self::Node(connection) => command.query(*connection).map_err(Error::unavailable), Self::Cluster(connection) => { let value = connection .route_command( @@ -372,18 +384,16 @@ impl ConnectionRef<'_> { Some(ResponsePolicy::Special), )), ) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let redis::Value::Map(entries) = value else { - return Err(Error::Unavailable); + return Err(Error::unavailable(TopologyFault::FanOutReply)); }; let mut replies = entries .into_iter() .map(|(node, reply)| { Ok(( - redis::from_redis_value::(node) - .map_err(|_| Error::Unavailable)?, - redis::from_redis_value::(reply) - .map_err(|_| Error::Unavailable)?, + redis::from_redis_value::(node).map_err(Error::unavailable)?, + redis::from_redis_value::(reply).map_err(Error::unavailable)?, )) }) .collect::, Error>>()?; @@ -400,7 +410,7 @@ impl ConnectionRef<'_> { pub(crate) fn flushall(&mut self) -> Result<(), Error> { let command = redis::cmd("FLUSHALL"); match self { - Self::Node(connection) => command.query(*connection).map_err(|_| Error::Unavailable), + Self::Node(connection) => command.query(*connection).map_err(Error::unavailable), Self::Cluster(connection) => connection .route_command( &command, @@ -410,7 +420,7 @@ impl ConnectionRef<'_> { )), ) .map(|_| ()) - .map_err(|_| Error::Unavailable), + .map_err(Error::unavailable), } } @@ -424,7 +434,7 @@ impl ConnectionRef<'_> { let command = scan_command(cursor, pattern, count); match (self, node) { (Self::Node(connection), None) => { - command.query(*connection).map_err(|_| Error::Unavailable) + command.query(*connection).map_err(Error::unavailable) } (Self::Cluster(connection), Some(node)) => connection .route_command( @@ -434,9 +444,9 @@ impl ConnectionRef<'_> { port: node.port(), }), ) - .map_err(|_| Error::Unavailable) - .and_then(|value| redis::from_redis_value(value).map_err(|_| Error::Unavailable)), - _ => Err(Error::Unavailable), + .map_err(Error::unavailable) + .and_then(|value| redis::from_redis_value(value).map_err(Error::unavailable)), + _ => Err(Error::unavailable(TopologyFault::ScanTarget)), } } } @@ -445,14 +455,14 @@ type ScanPage = (u64, Vec); fn primary_pages(value: redis::Value) -> Result, Error> { let redis::Value::Map(entries) = value else { - return Err(Error::Unavailable); + return Err(Error::unavailable(TopologyFault::FanOutReply)); }; entries .into_iter() .map(|(node, page)| { - let node = redis::from_redis_value::(node).map_err(|_| Error::Unavailable)?; - let node = NodeAddress::try_from(node.as_str()).map_err(|_| Error::Unavailable)?; - let page = redis::from_redis_value::(page).map_err(|_| Error::Unavailable)?; + let node = redis::from_redis_value::(node).map_err(Error::unavailable)?; + let node = NodeAddress::try_from(node.as_str()).map_err(Error::unavailable)?; + let page = redis::from_redis_value::(page).map_err(Error::unavailable)?; Ok((node, page)) }) .collect() diff --git a/litellm-rust/crates/cache-redis/src/counter.rs b/litellm-rust/crates/cache-redis/src/counter.rs index adcd597b2f4..a4beb9f5712 100644 --- a/litellm-rust/crates/cache-redis/src/counter.rs +++ b/litellm-rust/crates/cache-redis/src/counter.rs @@ -100,7 +100,7 @@ fn increment( .arg(amount) .arg(ttl) .query(connection) - .map_err(|_| Error::Unavailable); + .map_err(Error::unavailable); } connection .query_pipeline( @@ -140,7 +140,7 @@ where } fn count(value: redis::Value) -> Result, Error> { - redis::from_redis_value(value).map_err(|_| Error::InvalidEntry) + redis::from_redis_value(value).map_err(Error::invalid_entry) } impl BoundedCounterCache for RedisCache @@ -182,7 +182,7 @@ where .arg(value) .arg(ttl) .query(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) .await } @@ -201,5 +201,5 @@ fn increment_with_floor( .arg(amount) .arg(ttl) .query(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) } diff --git a/litellm-rust/crates/cache-redis/src/keys.rs b/litellm-rust/crates/cache-redis/src/keys.rs index b9d7bb20ff3..2e8a9e60e65 100644 --- a/litellm-rust/crates/cache-redis/src/keys.rs +++ b/litellm-rust/crates/cache-redis/src/keys.rs @@ -16,7 +16,7 @@ where redis::cmd("TTL") .arg(key) .query::(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) .await?; Ok(u64::try_from(ttl).ok().map(Duration::from_secs)) @@ -36,7 +36,7 @@ where .arg(key) .arg(ttl) .query(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) .await } diff --git a/litellm-rust/crates/cache-redis/src/lifecycle.rs b/litellm-rust/crates/cache-redis/src/lifecycle.rs index 6ab8c31509e..54fa8ecc3bf 100644 --- a/litellm-rust/crates/cache-redis/src/lifecycle.rs +++ b/litellm-rust/crates/cache-redis/src/lifecycle.rs @@ -11,11 +11,11 @@ where C: redis::ConnectionLike + Send + 'static, { fn sync_ping(&self) -> Result { - self.execute(|connection| connection.ping().map_err(|_| Error::Unavailable)) + self.execute(|connection| connection.ping().map_err(Error::unavailable)) } async fn ping(&self) -> Result { - self.run(|connection| connection.ping().map_err(|_| Error::Unavailable)) + self.run(|connection| connection.ping().map_err(Error::unavailable)) .await } } @@ -33,9 +33,10 @@ where RedisTopology::Cluster { .. } => "Redis Cluster", }; let ping = self - .run(|connection| Ok(connection.ping().map_err(|error| error.to_string()))) + .run(|connection| Ok(connection.ping().map_err(Error::unavailable))) .await - .unwrap_or_else(|error| Err(error.to_string())); + .and_then(|ping| ping) + .map_err(|error| litellm_error::Report::new(&error).to_string()); Ok(match ping { Ok(true) => CacheConnectionResult { status: CacheConnectionStatus::Success, diff --git a/litellm-rust/crates/cache-redis/src/queue.rs b/litellm-rust/crates/cache-redis/src/queue.rs index 623bce23640..d55e4f49187 100644 --- a/litellm-rust/crates/cache-redis/src/queue.rs +++ b/litellm-rust/crates/cache-redis/src/queue.rs @@ -29,7 +29,7 @@ where ttl: Option, ) -> Result { if values.is_empty() { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } let key = self.namespaced_key(key); let ttl = self.ttl_or_default(ttl); @@ -58,7 +58,7 @@ where async fn async_rpush(&self, key: &str, values: Vec) -> Result { if values.is_empty() { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } let key = self.namespaced_key(key); self.run(move |connection| { @@ -66,7 +66,7 @@ where .arg(key) .arg(values) .query(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) .await } @@ -78,7 +78,7 @@ where max_len: usize, ) -> Result { if values.is_empty() { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } let key = self.namespaced_key(key); let start = i64::try_from(max_len).map_or(i64::MIN, |max_len| -max_len); @@ -108,7 +108,7 @@ where let mut pipeline = redis::pipe(); for operation in operations { if operation.values.is_empty() { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } pipeline .cmd("RPUSH") @@ -130,7 +130,7 @@ where .run(move |connection| { command .query::(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) .await?; lpop_result(value, count.is_some()) @@ -182,7 +182,7 @@ fn redis_bytes(value: redis::Value) -> Result, Error> { match value { redis::Value::BulkString(bytes) => Ok(bytes), redis::Value::SimpleString(text) => Ok(text.into_bytes()), - _ => Err(Error::InvalidEntry), + _ => Err(Error::InvalidEntry(None)), } } @@ -221,6 +221,6 @@ fn lpop_result(value: redis::Value, multiple: bool) -> Result, _>>() .map(RedisLpopResult::Values), value if !multiple => redis_bytes(value).map(RedisLpopResult::Value), - _ => Err(Error::InvalidEntry), + _ => Err(Error::InvalidEntry(None)), } } diff --git a/litellm-rust/crates/cache-redis/src/script.rs b/litellm-rust/crates/cache-redis/src/script.rs index e8a16bcc2b5..09987437ed4 100644 --- a/litellm-rust/crates/cache-redis/src/script.rs +++ b/litellm-rust/crates/cache-redis/src/script.rs @@ -116,7 +116,7 @@ fn eval( .arg(keys) .arg(arguments) .query(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) } impl ScriptCache for RedisCache diff --git a/litellm-rust/crates/cache-redis/src/store.rs b/litellm-rust/crates/cache-redis/src/store.rs index 111d481ad02..daadc940ca9 100644 --- a/litellm-rust/crates/cache-redis/src/store.rs +++ b/litellm-rust/crates/cache-redis/src/store.rs @@ -32,7 +32,7 @@ where self.execute(|connection| { connection .set_ex::<_, _, ()>(key, payload, ttl) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) } @@ -41,7 +41,7 @@ where let value = self.execute(|connection| { connection .get::<_, redis::Value>(key) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) })?; self.decode_response(value) } @@ -58,7 +58,7 @@ where self.run(move |connection| { connection .set_ex::<_, _, ()>(key, payload, ttl) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) .await } @@ -73,7 +73,7 @@ where .run(move |connection| { connection .get::<_, redis::Value>(key) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) }) .await?; self.decode_response(value) @@ -158,7 +158,7 @@ pub(crate) fn mget( redis::cmd("MGET") .arg(keys) .query(connection) - .map_err(|_| Error::Unavailable) + .map_err(Error::unavailable) } impl DeleteCache for RedisCache @@ -168,12 +168,12 @@ where { fn delete_cache(&self, key: &str) -> Result<(), Error> { let key = self.namespaced_key(key); - self.execute(|connection| connection.del::<_, ()>(key).map_err(|_| Error::Unavailable)) + self.execute(|connection| connection.del::<_, ()>(key).map_err(Error::unavailable)) } async fn async_delete_cache(&self, key: &str) -> Result<(), Error> { let key = self.namespaced_key(key); - self.run(move |connection| connection.del::<_, ()>(key).map_err(|_| Error::Unavailable)) + self.run(move |connection| connection.del::<_, ()>(key).map_err(Error::unavailable)) .await } } @@ -188,7 +188,7 @@ where return Ok(0); } let keys = self.namespaced_keys(&keys); - self.run(move |connection| connection.del(keys).map_err(|_| Error::Unavailable)) + self.run(move |connection| connection.del(keys).map_err(Error::unavailable)) .await } } @@ -225,7 +225,7 @@ fn flush_matching(connection: &mut ConnectionRef<'_>, pattern: &str) -> Result<( if !keys.is_empty() { connection .del::<_, usize>(keys) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; } Ok(true) }) diff --git a/litellm-rust/crates/cache-redis/tests/cache.rs b/litellm-rust/crates/cache-redis/tests/cache.rs index 70baeffd572..9ec85251733 100644 --- a/litellm-rust/crates/cache-redis/tests/cache.rs +++ b/litellm-rust/crates/cache-redis/tests/cache.rs @@ -178,11 +178,11 @@ async fn codec_errors_propagate_without_writing_partial_batches(context: ExactCa ]); assert_eq!( cache.set_cache("invalid", 255, &context), - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); assert_eq!( cache.async_set_cache("invalid", 255, context.clone()).await, - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); assert_eq!( cache @@ -191,7 +191,7 @@ async fn codec_errors_propagate_without_writing_partial_batches(context: ExactCa context.clone(), ) .await, - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); assert_eq!( cache @@ -200,15 +200,15 @@ async fn codec_errors_propagate_without_writing_partial_batches(context: ExactCa ("invalid".into(), 255, None), ]) .await, - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); assert_eq!( cache.get_cache("invalid", &context), - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); assert_eq!( cache.async_get_cache("invalid", &context).await, - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); } @@ -516,7 +516,7 @@ async fn sets_add_members_and_arm_the_default_ttl() { cache .async_set_cache_sadd("members", Vec::new(), None) .await, - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); } @@ -547,7 +547,7 @@ async fn queues_push_and_pop_namespaced_lists() { ); assert_eq!( cache.async_rpush("queue", Vec::new()).await, - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); assert_eq!( cache.async_lpop("queue", Some(2)).await.unwrap(), @@ -673,15 +673,15 @@ async fn rpush_and_trim_raises_when_a_queued_command_fails() { )]) .with_namespace(Some("ns".into())); - assert_eq!( + assert!(matches!( cache .async_rpush_and_trim("buf", vec!["c".into(), "d".into()], 3) .await, - Err(Error::Unavailable) - ); + Err(Error::Unavailable(Some(_))) + )); assert_eq!( cache.async_rpush_and_trim("buf", Vec::new(), 3).await, - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); } @@ -1122,24 +1122,30 @@ async fn test_connection_reports_success_with_the_python_message() { } #[rstest] -#[case::unexpected_reply(Ok("NOPE"), "Redis ping returned False", false)] +#[case::unexpected_reply(Ok("NOPE"), "Redis ping returned False", None)] #[case::connection_refused( Err(redis::RedisError::from((redis::ErrorKind::Io, "connection refused"))), "Redis connection failed:", - true + Some("connection refused") )] #[tokio::test] async fn test_connection_failures_use_the_python_result_contract( #[case] reply: redis::RedisResult<&'static str>, #[case] message: &str, - #[case] has_error: bool, + #[case] cause: Option<&str>, ) { let cache = json_cache(vec![MockCmd::new(redis::cmd("PING"), reply)]); let result = cache.test_connection().await.unwrap(); assert_eq!(result.status, CacheConnectionStatus::Failed); assert!(result.message.starts_with(message), "{}", result.message); - assert_eq!(result.error.is_some(), has_error); + assert_eq!(result.error.is_some(), cause.is_some()); + assert!(cause.is_none_or(|cause| { + result + .error + .as_deref() + .is_some_and(|error| error.contains(cause)) + })); } #[rstest] diff --git a/litellm-rust/crates/cache-redis/tests/cluster.rs b/litellm-rust/crates/cache-redis/tests/cluster.rs index c1a9a70a5d2..e1ba01bcb04 100644 --- a/litellm-rust/crates/cache-redis/tests/cluster.rs +++ b/litellm-rust/crates/cache-redis/tests/cluster.rs @@ -66,7 +66,7 @@ fn constructor_rejects_unusable_cluster_configs( JsonCodec::new(), ) .err(); - assert!(matches!(error, Some(Error::Unavailable))); + assert!(matches!(error, Some(Error::Unavailable(_)))); } #[rstest] diff --git a/litellm-rust/crates/cache-redis/tests/support/mod.rs b/litellm-rust/crates/cache-redis/tests/support/mod.rs index bff1e618582..b04e1c29f3b 100644 --- a/litellm-rust/crates/cache-redis/tests/support/mod.rs +++ b/litellm-rust/crates/cache-redis/tests/support/mod.rs @@ -16,7 +16,7 @@ impl CacheCodec for TaggedByteCodec { fn encode(&self, value: &u8) -> Result, Error> { if *value > 127 { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } Ok(vec![self.0, *value]) } @@ -24,7 +24,7 @@ impl CacheCodec for TaggedByteCodec { fn decode(&self, bytes: &[u8]) -> Result { match bytes { [tag, value] if *tag == self.0 => Ok(*value), - _ => Err(Error::InvalidEntry), + _ => Err(Error::InvalidEntry(None)), } } } diff --git a/litellm-rust/crates/cache-response/src/buffer.rs b/litellm-rust/crates/cache-response/src/buffer.rs index 68f2348e28b..deb1a56b503 100644 --- a/litellm-rust/crates/cache-response/src/buffer.rs +++ b/litellm-rust/crates/cache-response/src/buffer.rs @@ -26,7 +26,7 @@ impl WriteBuffer { now: Duration, ) -> Result<(), Error> { let pending = { - let mut entries = self.entries.lock().map_err(|_| Error::Unavailable)?; + let mut entries = self.entries.lock()?; entries.push((request.clone(), response, now)); (entries.len() >= self.flush_size).then(|| std::mem::take(&mut *entries)) }; @@ -39,7 +39,7 @@ impl WriteBuffer { } pub fn clear(&self) -> Result<(), Error> { - self.entries.lock().map_err(|_| Error::Unavailable)?.clear(); + self.entries.lock()?.clear(); Ok(()) } } diff --git a/litellm-rust/crates/cache-response/src/codec.rs b/litellm-rust/crates/cache-response/src/codec.rs index 6b0f29e0a58..1adebe65bc3 100644 --- a/litellm-rust/crates/cache-response/src/codec.rs +++ b/litellm-rust/crates/cache-response/src/codec.rs @@ -14,24 +14,24 @@ impl CacheCodec for ResponseCacheCodec { .timestamp .is_some_and(|timestamp| !timestamp.is_finite()) { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } // Python reads a `response` that is either a dict or a serialized string, so every // other shape is written serialized. A string on the wire is therefore always a // serialized response, which keeps string-valued responses unambiguous. if value.timestamp.is_none() || value.response.is_object() { - return serde_json::to_vec(value).map_err(|_| Error::InvalidEntry); + return serde_json::to_vec(value).map_err(Error::invalid_entry); } - let response = serde_json::to_string(&value.response).map_err(|_| Error::InvalidEntry)?; + let response = serde_json::to_string(&value.response).map_err(Error::invalid_entry)?; serde_json::to_vec(&CacheEntry { timestamp: value.timestamp, response: Value::String(response), }) - .map_err(|_| Error::InvalidEntry) + .map_err(Error::invalid_entry) } fn decode(&self, bytes: &[u8]) -> Result { - let text = std::str::from_utf8(bytes).map_err(|_| Error::InvalidEntry)?; + let text = std::str::from_utf8(bytes).map_err(Error::invalid_entry)?; let value = decode_value(text)?; let Some(timestamp) = value.get("timestamp") else { return Ok(CacheEntry { @@ -40,9 +40,9 @@ impl CacheCodec for ResponseCacheCodec { }); }; let Some(timestamp) = timestamp.as_f64().filter(|timestamp| timestamp.is_finite()) else { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); }; - let response = match value.get("response").ok_or(Error::InvalidEntry)? { + let response = match value.get("response").ok_or(Error::InvalidEntry(None))? { Value::String(text) => decode_value(text)?, response => response.clone(), }; @@ -58,25 +58,25 @@ fn decode_value(text: &str) -> Result { return Ok(value); } check_literal_depth(text)?; - let literal: py_literal::Value = text.parse().map_err(|_| Error::InvalidEntry)?; + let literal: py_literal::Value = text.parse().map_err(Error::invalid_entry)?; literal_value(literal, 0) } fn literal_value(value: py_literal::Value, depth: usize) -> Result { use py_literal::Value as Literal; if depth > 128 { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } match value { Literal::String(text) => Ok(Value::String(text)), Literal::Boolean(value) => Ok(Value::Bool(value)), Literal::None => Ok(Value::Null), Literal::Integer(value) => { - serde_json::from_str(&value.to_string()).map_err(|_| Error::InvalidEntry) + serde_json::from_str(&value.to_string()).map_err(Error::invalid_entry) } Literal::Float(value) => serde_json::Number::from_f64(value) .map(Value::Number) - .ok_or(Error::InvalidEntry), + .ok_or(Error::InvalidEntry(None)), Literal::List(values) | Literal::Tuple(values) => values .into_iter() .map(|value| literal_value(value, depth + 1)) @@ -86,13 +86,13 @@ fn literal_value(value: py_literal::Value, depth: usize) -> Result .into_iter() .map(|(key, value)| { let Literal::String(key) = key else { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); }; Ok((key, literal_value(value, depth + 1)?)) }) .collect::, _>>() .map(Value::Object), - _ => Err(Error::InvalidEntry), + _ => Err(Error::InvalidEntry(None)), } } @@ -118,7 +118,7 @@ fn check_literal_depth(text: &str) -> Result<(), Error> { '[' | '{' | '(' => { depth += 1; if depth > 128 { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } } ']' | '}' | ')' => depth = depth.saturating_sub(1), diff --git a/litellm-rust/crates/cache-response/src/response.rs b/litellm-rust/crates/cache-response/src/response.rs index a5bbef99a3a..1505297315f 100644 --- a/litellm-rust/crates/cache-response/src/response.rs +++ b/litellm-rust/crates/cache-response/src/response.rs @@ -2,7 +2,7 @@ use std::{sync::Arc, time::Duration}; use litellm_cache::{ BaseCache, BatchCache, BatchEntry, CacheConnectionResult, CacheContext, ConnectionCache, Error, - FlushCache, + FlushCache, ShortBatch, semantic::{SemanticCache, SemanticLookup}, }; use serde_json::Value; @@ -114,7 +114,7 @@ where .get_cache(&cache_key(&request.key), &request.context) { Ok(entry) => entry, - Err(Error::InvalidEntry) => None, + Err(Error::InvalidEntry(_)) => None, Err(error) => return Err(error), }; Ok(Self::fresh_or_miss(entry, now, request.max_age)) @@ -134,7 +134,7 @@ where .await { Ok(entry) => entry, - Err(Error::InvalidEntry) => None, + Err(Error::InvalidEntry(_)) => None, Err(error) => return Err(error), }; Ok(Self::fresh_or_miss(entry, now, request.max_age)) @@ -338,7 +338,10 @@ where now: Duration, ) -> Result { if readable.len() != entries.len() { - return Err(Error::Unavailable); + return Err(Error::unavailable(ShortBatch { + requested: readable.len(), + returned: entries.len(), + })); } let mut values = vec![None; requests.len()]; for ((index, request), entry) in readable.into_iter().zip(entries) { @@ -361,7 +364,7 @@ where value: Self::fresh_or_miss(lookup.value, now, max_age), similarity: lookup.similarity, }), - Err(Error::InvalidEntry) => Ok(SemanticLookup::miss(None)), + Err(Error::InvalidEntry(_)) => Ok(SemanticLookup::miss(None)), Err(error) => Err(error), } } diff --git a/litellm-rust/crates/cache-response/tests/codec.rs b/litellm-rust/crates/cache-response/tests/codec.rs index 4fb5a5094cb..289465fcda9 100644 --- a/litellm-rust/crates/cache-response/tests/codec.rs +++ b/litellm-rust/crates/cache-response/tests/codec.rs @@ -44,25 +44,23 @@ fn decode_reads_every_python_envelope(#[case] bytes: &[u8], #[case] response: Va #[case::non_utf8(vec![0xff, 0xfe])] #[case::deep_nesting(format!("{}None{}", "[".repeat(1000), "]".repeat(1000)).into_bytes())] fn decode_rejects_invalid_entries(#[case] bytes: Vec) { - assert_eq!( - ResponseCacheCodec.decode(&bytes).unwrap_err(), - Error::InvalidEntry - ); + assert!(matches!( + ResponseCacheCodec.decode(&bytes), + Err(Error::InvalidEntry(_)) + )); } #[rstest] #[case::nan(f64::NAN)] #[case::infinity(f64::INFINITY)] fn encode_rejects_non_finite_timestamps(#[case] timestamp: f64) { - assert_eq!( - ResponseCacheCodec - .encode(&CacheEntry { - timestamp: Some(timestamp), - response: json!({}), - }) - .unwrap_err(), - Error::InvalidEntry - ); + assert!(matches!( + ResponseCacheCodec.encode(&CacheEntry { + timestamp: Some(timestamp), + response: json!({}), + }), + Err(Error::InvalidEntry(_)) + )); } #[rstest] diff --git a/litellm-rust/crates/cache-response/tests/response.rs b/litellm-rust/crates/cache-response/tests/response.rs index 86ddf4a72cd..70ae259d33a 100644 --- a/litellm-rust/crates/cache-response/tests/response.rs +++ b/litellm-rust/crates/cache-response/tests/response.rs @@ -158,8 +158,16 @@ fn scored(timestamp: f64, similarity: f64) -> Result, )] #[case::no_search(Ok(SemanticLookup::miss(None)), true, Ok(SemanticLookup::miss(None)))] #[case::disabled_reads_skip_the_backend(scored(95.0, 0.95), false, Ok(SemanticLookup::miss(None)))] -#[case::invalid_entry_is_a_miss(Err(Error::InvalidEntry), true, Ok(SemanticLookup::miss(None)))] -#[case::backend_errors_propagate(Err(Error::Unavailable), true, Err(Error::Unavailable))] +#[case::invalid_entry_is_a_miss( + Err(Error::InvalidEntry(None)), + true, + Ok(SemanticLookup::miss(None)) +)] +#[case::backend_errors_propagate( + Err(Error::Unavailable(None)), + true, + Err(Error::Unavailable(None)) +)] #[tokio::test] async fn semantic_lookup_applies_freshness_to_the_value_only( #[case] backend: Result, Error>, diff --git a/litellm-rust/crates/cache-s3/Cargo.toml b/litellm-rust/crates/cache-s3/Cargo.toml index eb3a2fff1ac..d51c282d4d7 100644 --- a/litellm-rust/crates/cache-s3/Cargo.toml +++ b/litellm-rust/crates/cache-s3/Cargo.toml @@ -20,6 +20,7 @@ reqwest.workspace = true tokio.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-http = { workspace = true, features = ["test-support"] } litellm-cache-testing.workspace = true rstest.workspace = true diff --git a/litellm-rust/crates/cache-s3/src/auth.rs b/litellm-rust/crates/cache-s3/src/auth.rs index f4f06e5371d..659390a5659 100644 --- a/litellm-rust/crates/cache-s3/src/auth.rs +++ b/litellm-rust/crates/cache-s3/src/auth.rs @@ -46,7 +46,7 @@ impl ProvideCredentials for S3Credentials { self.auth .resolve_credentials(self.config.clone(), &self.env) .await - .map_err(|_| CredentialsError::provider_error("S3 cache authentication failed")) + .map_err(CredentialsError::provider_error) }) } } diff --git a/litellm-rust/crates/cache-s3/src/cache.rs b/litellm-rust/crates/cache-s3/src/cache.rs index ced948e80c6..5369bca0f18 100644 --- a/litellm-rust/crates/cache-s3/src/cache.rs +++ b/litellm-rust/crates/cache-s3/src/cache.rs @@ -125,7 +125,7 @@ impl S3Cache { } None => request.cache_control("immutable, max-age=31536000, s-maxage=31536000"), }; - request.send().await.map_err(|_| Error::Unavailable)?; + request.send().await.map_err(Error::unavailable)?; Ok(()) } @@ -152,7 +152,7 @@ impl S3Cache { return Ok(None); } } - return Err(Error::Unavailable); + return Err(Error::unavailable(error)); } }; if let Some(expires) = output.expires_string() @@ -165,7 +165,7 @@ impl S3Cache { .body .collect() .await - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? .into_bytes(); self.codec.decode(&bytes).map(Some) } diff --git a/litellm-rust/crates/cache-s3/src/transport.rs b/litellm-rust/crates/cache-s3/src/transport.rs index eabc54ac9e2..bf1711276f2 100644 --- a/litellm-rust/crates/cache-s3/src/transport.rs +++ b/litellm-rust/crates/cache-s3/src/transport.rs @@ -29,9 +29,10 @@ impl HttpConnector for ReqwestHttpClient { .try_into_http1x() .map_err(|error| ConnectorError::other(error.into(), None))? .map(reqwest::Body::wrap); - let request = reqwest::Request::try_from(request) + let request = litellm_http::outbound::from_http(request) .map_err(|error| ConnectorError::other(error.into(), None))?; let response = client.execute(request).await.map_err(|error| { + let error = litellm_http::Error::from(error); if error.is_timeout() { ConnectorError::timeout(error.into()) } else { diff --git a/litellm-rust/crates/cache-s3/tests/cache.rs b/litellm-rust/crates/cache-s3/tests/cache.rs index 53a51f8a289..d0469620493 100644 --- a/litellm-rust/crates/cache-s3/tests/cache.rs +++ b/litellm-rust/crates/cache-s3/tests/cache.rs @@ -147,15 +147,15 @@ async fn async_set_signs_with_the_configured_keys(#[future(awt)] server: MockSer #[case::malformed( "malformed", ResponseTemplate::new(200).set_body_string("not a cache entry"), - Err(Error::InvalidEntry) + Err("invalid cache entry: ") )] -#[case::server_error("broken", ResponseTemplate::new(500), Err(Error::Unavailable))] +#[case::server_error("broken", ResponseTemplate::new(500), Err("cache is unavailable: "))] #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn get_maps_s3_responses( #[future(awt)] server: MockServer, #[case] key: &str, #[case] response: ResponseTemplate, - #[case] expected: Result, Error>, + #[case] expected: Result, &str>, ) { Mock::given(method("GET")) .and(path(format!("/cache-bucket/team/{key}"))) @@ -164,9 +164,21 @@ async fn get_maps_s3_responses( .await; let cache = support::cache(&server.uri()); let context = ExactCacheContext::default(); - - assert_eq!(cache.get_cache(key, &context), expected); - assert_eq!(cache.async_get_cache(key, &context).await, expected); + let outcome = |result: Result, Error>| { + result.map_err(|error| litellm_error::Report::new(&error).to_string()) + }; + let sync = outcome(cache.get_cache(key, &context)); + let asynchronous = outcome(cache.async_get_cache(key, &context).await); + match expected { + Ok(value) => { + assert_eq!(sync, Ok(value.clone())); + assert_eq!(asynchronous, Ok(value)); + } + Err(prefix) => { + assert!(sync.unwrap_err().starts_with(prefix)); + assert!(asynchronous.unwrap_err().starts_with(prefix)); + } + } } #[rstest] diff --git a/litellm-rust/crates/cache-valkey-semantic/Cargo.toml b/litellm-rust/crates/cache-valkey-semantic/Cargo.toml index ec755e61b8c..5b4e99de382 100644 --- a/litellm-rust/crates/cache-valkey-semantic/Cargo.toml +++ b/litellm-rust/crates/cache-valkey-semantic/Cargo.toml @@ -10,9 +10,11 @@ litellm-cache.workspace = true litellm-cache-redis.workspace = true redis = { workspace = true, features = ["tls-rustls"] } sha2.workspace = true +thiserror.workspace = true uuid = { version = "1", features = ["v4"] } [dev-dependencies] +litellm-error.workspace = true litellm-cache-testing.workspace = true redis-test.workspace = true rstest.workspace = true diff --git a/litellm-rust/crates/cache-valkey-semantic/src/index.rs b/litellm-rust/crates/cache-valkey-semantic/src/index.rs index a0b7be4ca7b..3dfff32ff7c 100644 --- a/litellm-rust/crates/cache-valkey-semantic/src/index.rs +++ b/litellm-rust/crates/cache-valkey-semantic/src/index.rs @@ -23,8 +23,7 @@ pub(crate) fn ensure_index( ) -> Result<(), Error> { if index .dimension - .lock() - .map_err(|_| Error::Unavailable)? + .lock()? .is_some_and(|existing| existing == dimension) { return Ok(()); @@ -50,24 +49,42 @@ pub(crate) fn ensure_index( .arg("DISTANCE_METRIC") .arg("COSINE") .query::(connection) - .map(|_| ()) - .map_err(|error| error.to_string()); - if let Err(message) = create { - if !message.to_ascii_lowercase().contains("already exists") { - return Err(Error::Unavailable); + .map(|_| ()); + if let Err(error) = create { + if !index_exists(&error) { + return Err(Error::unavailable(error)); } let info = redis::cmd("FT.INFO") .arg(&index.name) .query::(connection) - .map_err(|_| Error::Unavailable)?; - if index_dimension_from_info(&info).is_some_and(|existing| existing != dimension) { - return Err(Error::Unavailable); + .map_err(Error::unavailable)?; + if let Some(existing) = + index_dimension_from_info(&info).filter(|existing| *existing != dimension) + { + return Err(Error::unavailable(DimensionMismatch { + existing, + dimension, + })); } } - *index.dimension.lock().map_err(|_| Error::Unavailable)? = Some(dimension); + *index.dimension.lock()? = Some(dimension); Ok(()) } +fn index_exists(error: &redis::RedisError) -> bool { + error.code().is_some() + && error + .detail() + .is_some_and(|detail| detail.to_ascii_lowercase().contains("already exists")) +} + +#[derive(Debug, thiserror::Error)] +#[error("index exists with dimension {existing}, not {dimension}")] +struct DimensionMismatch { + existing: usize, + dimension: usize, +} + /// `_extract_index_dim`: flatten each attribute one level and read the value after /// `dimensions`. fn index_dimension_from_info(value: &redis::Value) -> Option { diff --git a/litellm-rust/crates/cache-valkey-semantic/src/search.rs b/litellm-rust/crates/cache-valkey-semantic/src/search.rs index 3b4fae84317..c5e4e748b58 100644 --- a/litellm-rust/crates/cache-valkey-semantic/src/search.rs +++ b/litellm-rust/crates/cache-valkey-semantic/src/search.rs @@ -53,9 +53,7 @@ pub(crate) fn write_document( .arg(ttl.as_secs()) .ignore(); } - pipeline - .query::<()>(connection) - .map_err(|_| Error::Unavailable) + pipeline.query::<()>(connection).map_err(Error::unavailable) } /// The KNN-1 search within `scope`: the closest document's similarity, and its stored response @@ -83,7 +81,7 @@ pub(crate) fn search_document( .arg("DIALECT") .arg(2) .query::(connection) - .map_err(|_| Error::Unavailable)?; + .map_err(Error::unavailable)?; let Some(fields) = search_fields(response)? else { return Ok(SemanticLookup::miss(Some(0.0))); }; @@ -91,7 +89,7 @@ pub(crate) fn search_document( fields .iter() .find_map(|(field, value)| (field == name).then(|| value.clone())) - .ok_or(Error::InvalidEntry) + .ok_or(Error::InvalidEntry(None)) }; let response = field("response")?; let similarity = 1.0 - parse_f64(&field("vector_distance")?)?; @@ -105,24 +103,24 @@ type SearchFields = Vec<(String, Vec)>; fn search_fields(value: redis::Value) -> Result, Error> { let redis::Value::Array(values) = value else { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); }; - let total = parse_i64(values.first().ok_or(Error::InvalidEntry)?)?; + let total = parse_i64(values.first().ok_or(Error::InvalidEntry(None))?)?; if total <= 0 || values.len() < 3 { return Ok(None); } let redis::Value::Array(fields) = &values[2] else { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); }; let (pairs, remainder) = fields.as_chunks::<2>(); if !remainder.is_empty() { - return Err(Error::InvalidEntry); + return Err(Error::InvalidEntry(None)); } pairs .iter() .map(|pair| { Ok(( - value_text(&pair[0]).ok_or(Error::InvalidEntry)?, + value_text(&pair[0]).ok_or(Error::InvalidEntry(None))?, value_bytes(&pair[1])?, )) }) @@ -132,16 +130,16 @@ fn search_fields(value: redis::Value) -> Result, Error> { fn parse_i64(value: &redis::Value) -> Result { value_text(value) - .ok_or(Error::InvalidEntry)? + .ok_or(Error::InvalidEntry(None))? .parse() - .map_err(|_| Error::InvalidEntry) + .map_err(Error::invalid_entry) } fn parse_f64(value: &[u8]) -> Result { std::str::from_utf8(value) - .map_err(|_| Error::InvalidEntry)? + .map_err(Error::invalid_entry)? .parse() - .map_err(|_| Error::InvalidEntry) + .map_err(Error::invalid_entry) } pub(crate) fn value_text(value: &redis::Value) -> Option { @@ -158,6 +156,6 @@ fn value_bytes(value: &redis::Value) -> Result, Error> { redis::Value::BulkString(bytes) => Ok(bytes.clone()), redis::Value::SimpleString(value) => Ok(value.as_bytes().to_vec()), redis::Value::Int(value) => Ok(value.to_string().into_bytes()), - _ => Err(Error::InvalidEntry), + _ => Err(Error::InvalidEntry(None)), } } diff --git a/litellm-rust/crates/cache-valkey-semantic/tests/cache.rs b/litellm-rust/crates/cache-valkey-semantic/tests/cache.rs index 1c693d0875d..10e32c0183b 100644 --- a/litellm-rust/crates/cache-valkey-semantic/tests/cache.rs +++ b/litellm-rust/crates/cache-valkey-semantic/tests/cache.rs @@ -83,10 +83,7 @@ fn ok() -> redis::RedisResult { } fn already_exists() -> redis::RedisResult { - Err(redis::RedisError::from(( - redis::ErrorKind::Io, - "already exists", - ))) + Err(support::server_error("Index already exists")) } fn bulk(value: &[u8]) -> redis::Value { @@ -250,19 +247,29 @@ fn second_set_skips_create_after_dimension_is_cached(entry: Value, context: Sema #[rstest] #[case::nested_matching(nested_dimension_info(3), Ok(()))] -#[case::nested_mismatch(nested_dimension_info(2), Err(Error::Unavailable))] +#[case::nested_mismatch( + nested_dimension_info(2), + Err("cache is unavailable: index exists with dimension 2, not 3") +)] #[case::flat_matching(flat_dimension_info("3"), Ok(()))] -#[case::flat_mismatch(flat_dimension_info("2"), Err(Error::Unavailable))] +#[case::flat_mismatch( + flat_dimension_info("2"), + Err("cache is unavailable: index exists with dimension 2, not 3") +)] #[case::unreported_dimension_is_accepted(redis::Value::Array(vec![]), Ok(()))] fn existing_index_dimension_must_match_embedding( #[case] info: redis::Value, - #[case] expected: Result<(), Error>, + #[case] expected: Result<(), &str>, entry: Value, context: SemanticCacheContext, ) { let recording = recording([already_exists(), Ok(info)]); - assert_eq!(recording.cache.set_cache("key", entry, &context), expected); + let result = recording.cache.set_cache("key", entry, &context); + assert_eq!( + result.map_err(|error| litellm_error::Report::new(&error).to_string()), + expected.map_err(str::to_owned) + ); } #[rstest] @@ -274,10 +281,10 @@ fn index_creation_failures_are_unavailable( ) { let recording = recording([reply]); - assert_eq!( + assert!(matches!( recording.cache.set_cache("key", entry, &context), - Err(Error::Unavailable) - ); + Err(Error::Unavailable(Some(_))) + )); } #[rstest] @@ -291,10 +298,10 @@ fn index_creation_failures_are_unavailable( bulk(b"document"), redis::Value::Array(vec![bulk(b"vector_distance"), bulk(b"0.1")]), ]), - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) )] -#[case::unparsable_distance(search_hit(b"not-json", "abc"), Err(Error::InvalidEntry))] -#[case::undecodable_response(search_hit(b"not-json", "0.1"), Err(Error::InvalidEntry))] +#[case::unparsable_distance(search_hit(b"not-json", "abc"), Err(Error::InvalidEntry(None)))] +#[case::undecodable_response(search_hit(b"not-json", "0.1"), Err(Error::InvalidEntry(None)))] fn get_applies_threshold_and_decodes_entry( #[case] reply: redis::Value, #[case] expected: Result, Error>, @@ -302,7 +309,13 @@ fn get_applies_threshold_and_decodes_entry( ) { let recording = recording([ok(), Ok(reply)]); - assert_eq!(recording.cache.get_cache("key", &context), expected); + let kind = |result: Result, Error>| { + result.map_err(|error| std::mem::discriminant(&error)) + }; + assert_eq!( + kind(recording.cache.get_cache("key", &context)), + kind(expected) + ); } #[rstest] diff --git a/litellm-rust/crates/cache-valkey-semantic/tests/support/mod.rs b/litellm-rust/crates/cache-valkey-semantic/tests/support/mod.rs index 2c6ff06b134..2af05943c98 100644 --- a/litellm-rust/crates/cache-valkey-semantic/tests/support/mod.rs +++ b/litellm-rust/crates/cache-valkey-semantic/tests/support/mod.rs @@ -74,7 +74,7 @@ impl FakeSearch { "FT.CREATE" => { let name = text(1); if state.indexes.contains_key(&name) { - return Err(error("Index already exists")); + return Err(server_error("Index already exists")); } let position = |token: &str| args.iter().position(|arg| arg == token.as_bytes()); let prefix = args[position("PREFIX").unwrap() + 2].clone(); @@ -94,7 +94,7 @@ impl FakeSearch { let index = state .indexes .get(&text(1)) - .ok_or_else(|| error("Unknown index name"))?; + .ok_or_else(|| server_error("Unknown index name"))?; Ok(index_info(index)) } "FT.DROPINDEX" => { @@ -118,7 +118,7 @@ impl FakeSearch { let index = state .indexes .get(&text(1)) - .ok_or_else(|| error("no such index"))?; + .ok_or_else(|| server_error("no such index"))?; let query = text(2); let tag = query_tag(&query); let params = args.iter().position(|arg| arg == b"PARAMS").unwrap(); @@ -155,7 +155,7 @@ impl FakeSearch { ])) } "PING" => Ok(redis::Value::SimpleString("PONG".into())), - _ => Err(error("unsupported command")), + _ => Err(server_error("unsupported command")), } } } @@ -192,8 +192,11 @@ impl redis::ConnectionLike for FakeSearch { } } -fn error(message: &'static str) -> redis::RedisError { - redis::RedisError::from((redis::ErrorKind::Extension, message)) +pub fn server_error(reply: &str) -> redis::RedisError { + redis::parse_redis_value(format!("-{reply}\r\n").as_bytes()) + .expect("error reply") + .extract_error() + .expect_err("server error") } fn bulk(bytes: &[u8]) -> redis::Value { diff --git a/litellm-rust/crates/cache/Cargo.toml b/litellm-rust/crates/cache/Cargo.toml index 51bbe12a85e..282c34e6b02 100644 --- a/litellm-rust/crates/cache/Cargo.toml +++ b/litellm-rust/crates/cache/Cargo.toml @@ -6,6 +6,7 @@ license.workspace = true repository.workspace = true [dependencies] +litellm-error.workspace = true serde.workspace = true serde_json = { workspace = true, features = ["preserve_order"] } strum.workspace = true diff --git a/litellm-rust/crates/cache/src/capabilities.rs b/litellm-rust/crates/cache/src/capabilities.rs index ac9d8fc8764..b30f9e162f2 100644 --- a/litellm-rust/crates/cache/src/capabilities.rs +++ b/litellm-rust/crates/cache/src/capabilities.rs @@ -48,7 +48,7 @@ pub trait BatchCache: BaseCache { .map(|key| match self.get_cache(key, context) { Ok(Some(value)) => Ok(BatchEntry::Hit(value)), Ok(None) => Ok(BatchEntry::Miss), - Err(Error::InvalidEntry) => Ok(BatchEntry::Invalid), + Err(Error::InvalidEntry(_)) => Ok(BatchEntry::Invalid), Err(error) => Err(error), }) .collect() @@ -65,7 +65,7 @@ pub trait BatchCache: BaseCache { entries.push(match self.async_get_cache(&key, &context).await { Ok(Some(value)) => BatchEntry::Hit(value), Ok(None) => BatchEntry::Miss, - Err(Error::InvalidEntry) => BatchEntry::Invalid, + Err(Error::InvalidEntry(_)) => BatchEntry::Invalid, Err(error) => return Err(error), }); } diff --git a/litellm-rust/crates/cache/src/codec.rs b/litellm-rust/crates/cache/src/codec.rs index 6d47c682406..b802e40a8b3 100644 --- a/litellm-rust/crates/cache/src/codec.rs +++ b/litellm-rust/crates/cache/src/codec.rs @@ -41,10 +41,10 @@ where type Value = V; fn encode(&self, value: &Self::Value) -> Result, Error> { - serde_json::to_vec(value).map_err(|_| Error::InvalidEntry) + serde_json::to_vec(value).map_err(Error::invalid_entry) } fn decode(&self, bytes: &[u8]) -> Result { - serde_json::from_slice(bytes).map_err(|_| Error::InvalidEntry) + serde_json::from_slice(bytes).map_err(Error::invalid_entry) } } diff --git a/litellm-rust/crates/cache/src/dual.rs b/litellm-rust/crates/cache/src/dual.rs index c4bc953239d..3deee2560f4 100644 --- a/litellm-rust/crates/cache/src/dual.rs +++ b/litellm-rust/crates/cache/src/dual.rs @@ -2,7 +2,7 @@ use std::{sync::Arc, time::Duration}; use crate::{ BaseCache, BatchCache, BatchEntry, BulkDeleteCache, CacheContext, ClaimCache, CounterCache, - DeleteCache, Error, FlushCache, IncrementOperation, SetCache, TtlCache, + DeleteCache, Error, FlushCache, IncrementOperation, SetCache, ShortBatch, TtlCache, }; #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] @@ -98,7 +98,7 @@ impl DualCache { fn remote(&self, result: Result) -> Result, Error> { match result { Ok(value) => Ok(Some(value)), - Err(Error::Unavailable) + Err(Error::Unavailable(_)) if self.remote_failure_policy == RemoteFailurePolicy::UseLocal => { Ok(None) @@ -136,7 +136,10 @@ where remote: Vec>, ) -> Result>, Error> { if missing.len() != remote.len() { - return Err(Error::Unavailable); + return Err(Error::unavailable(ShortBatch { + requested: missing.len(), + returned: remote.len(), + })); } for (index, entry) in missing.into_iter().zip(remote) { if let BatchEntry::Hit(value) = &entry { @@ -389,7 +392,10 @@ where } let values = self.l2.async_increment_pipeline(operations.clone()).await?; if values.len() != operations.len() { - return Err(Error::Unavailable); + return Err(Error::unavailable(ShortBatch { + requested: operations.len(), + returned: values.len(), + })); } for (operation, value) in operations.iter().zip(&values) { self.l1 diff --git a/litellm-rust/crates/cache/src/error.rs b/litellm-rust/crates/cache/src/error.rs index 79ef9cd18b1..8861651a1c8 100644 --- a/litellm-rust/crates/cache/src/error.rs +++ b/litellm-rust/crates/cache/src/error.rs @@ -1,9 +1,11 @@ +use litellm_error::Source; + #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum Error { #[error("cache is unavailable")] - Unavailable, + Unavailable(#[source] Option), #[error("invalid cache entry")] - InvalidEntry, + InvalidEntry(#[source] Option), #[error("flushing Redis requires an explicit namespace")] UnscopedFlush, #[error("operation is not supported by this cache")] @@ -11,3 +13,59 @@ pub enum Error { #[error("semantic cache requires request messages")] MissingPrompt, } + +#[derive(Debug, thiserror::Error)] +#[error("cache backend returned {returned} results for {requested} keys")] +pub struct ShortBatch { + pub requested: usize, + pub returned: usize, +} + +#[derive(Debug, thiserror::Error)] +#[error("cache lock is poisoned")] +struct Poisoned; + +impl From> for Error { + fn from(_: std::sync::PoisonError) -> Self { + Self::unavailable(Poisoned) + } +} + +impl Error { + pub fn unavailable(cause: impl std::error::Error + Send + Sync + 'static) -> Self { + Self::Unavailable(Some(Source::new(cause))) + } + + pub fn invalid_entry(cause: impl std::error::Error + Send + Sync + 'static) -> Self { + Self::InvalidEntry(Some(Source::new(cause))) + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::Error; + + #[rstest] + #[case::unavailable( + Error::unavailable(std::io::Error::other("connection refused")), + "cache is unavailable" + )] + #[case::invalid_entry( + Error::invalid_entry(std::io::Error::other("connection refused")), + "invalid cache entry" + )] + fn the_backend_cause_is_the_source_and_rendered_once( + #[case] error: Error, + #[case] message: &str, + ) { + let report = litellm_error::Report::new(&error).to_string(); + assert_eq!(report, format!("{message}: connection refused")); + assert!(!error.to_string().contains("connection refused")); + let cause = std::error::Error::source(&error) + .and_then(|cause| cause.downcast_ref::()) + .expect("cause"); + assert!(cause.downcast_ref::().is_some()); + } +} diff --git a/litellm-rust/crates/cache/src/lib.rs b/litellm-rust/crates/cache/src/lib.rs index 55d8a5fa28b..4e07f3b4e8c 100644 --- a/litellm-rust/crates/cache/src/lib.rs +++ b/litellm-rust/crates/cache/src/lib.rs @@ -23,4 +23,4 @@ pub use codec::{CacheCodec, JsonCodec}; pub use dual::{ DEFAULT_DELETE_BATCH_SIZE, DualCache, ReadPolicy, RemoteFailurePolicy, WritePolicy, }; -pub use error::Error; +pub use error::{Error, ShortBatch}; diff --git a/litellm-rust/crates/cache/tests/caching.rs b/litellm-rust/crates/cache/tests/caching.rs index baf968f4659..84a3b8c4d2a 100644 --- a/litellm-rust/crates/cache/tests/caching.rs +++ b/litellm-rust/crates/cache/tests/caching.rs @@ -67,7 +67,7 @@ impl BaseCache for TestCache { } fn set_cache(&self, _: &str, _: Self::Value, _: &ExactCacheContext) -> Result<(), Error> { - Err(Error::Unavailable) + Err(Error::Unavailable(None)) } async fn async_set_cache( @@ -77,7 +77,7 @@ impl BaseCache for TestCache { context: ExactCacheContext, ) -> Result<(), Error> { if key == "unavailable" { - return Err(Error::Unavailable); + return Err(Error::Unavailable(None)); } self.writes .lock() @@ -118,7 +118,7 @@ impl BaseCache for RecordingCounter { impl CounterCache for RecordingCounter { fn increment_cache(&self, key: &str, amount: f64, _: ExactCacheContext) -> Result { if key == "unavailable" { - return Err(Error::Unavailable); + return Err(Error::Unavailable(None)); } let mut total = self.total.lock().unwrap(); *total += amount; @@ -226,7 +226,7 @@ async fn default_batch_operations_use_async_writes_and_stop_on_failure(cache: Te context.clone(), ) .await, - Err(Error::Unavailable) + Err(Error::Unavailable(None)) ); assert_eq!( *cache.writes.lock().unwrap(), @@ -323,7 +323,7 @@ async fn default_increment_pipeline_stops_at_the_first_failure() { operation("skipped", 1.0, None), ]) .await, - Err(Error::Unavailable) + Err(Error::Unavailable(None)) ); let keys = cache .increments diff --git a/litellm-rust/crates/cache/tests/codec.rs b/litellm-rust/crates/cache/tests/codec.rs index 5320545fd49..ff4aeed87e6 100644 --- a/litellm-rust/crates/cache/tests/codec.rs +++ b/litellm-rust/crates/cache/tests/codec.rs @@ -31,12 +31,18 @@ fn json_codec_round_trips_typed_domain_values() { #[case::wrongly_typed(br#"{"deployment":12}"#)] fn json_codec_rejects_malformed_and_wrongly_typed_entries(#[case] bytes: &[u8]) { let codec = JsonCodec::::new(); - assert_eq!(codec.decode(bytes).unwrap_err(), Error::InvalidEntry); + assert!(matches!( + codec.decode(bytes), + Err(Error::InvalidEntry(Some(_))) + )); } #[rstest] fn json_codec_propagates_encoding_errors() { let codec = JsonCodec::>::new(); let value = BTreeMap::from([((1, 2), "invalid JSON object key".into())]); - assert_eq!(codec.encode(&value).unwrap_err(), Error::InvalidEntry); + assert!(matches!( + codec.encode(&value), + Err(Error::InvalidEntry(Some(_))) + )); } diff --git a/litellm-rust/crates/cache/tests/dual.rs b/litellm-rust/crates/cache/tests/dual.rs index 87dd76388ef..86af5e80abb 100644 --- a/litellm-rust/crates/cache/tests/dual.rs +++ b/litellm-rust/crates/cache/tests/dual.rs @@ -71,7 +71,7 @@ where impl CounterCache for TestCache { fn increment_cache(&self, _: &str, amount: f64, _: ExactCacheContext) -> Result { if self.fail { - return Err(Error::Unavailable); + return Err(Error::Unavailable(None)); } let mut value = self.value.lock().unwrap(); let incremented = value.unwrap_or_default() + amount; @@ -92,7 +92,7 @@ where _: ExactCacheContext, ) -> Result { if self.fail { - return Err(Error::Unavailable); + return Err(Error::Unavailable(None)); } let mut value = self.value.lock().unwrap(); let winner = match value.as_ref() { @@ -113,7 +113,7 @@ fn failed_l2_increment_leaves_l1_unchanged() { assert_eq!( cache.increment_cache("counter", 2.0, ExactCacheContext::default()), - Err(Error::Unavailable) + Err(Error::Unavailable(None)) ); assert_eq!( l1.get_cache("counter", &ExactCacheContext::default()) @@ -276,11 +276,11 @@ impl BaseCache for Unavailable { } fn set_cache(&self, _: &str, _: String, _: &ExactCacheContext) -> Result<(), Error> { - Err(Error::Unavailable) + Err(Error::Unavailable(None)) } fn get_cache(&self, _: &str, _: &ExactCacheContext) -> Result, Error> { - Err(Error::Unavailable) + Err(Error::Unavailable(None)) } } @@ -288,13 +288,13 @@ impl BatchCache for Unavailable {} impl DeleteCache for Unavailable { fn delete_cache(&self, _: &str) -> Result<(), Error> { - Err(Error::Unavailable) + Err(Error::Unavailable(None)) } } impl FlushCache for Unavailable { fn flush_cache(&self) -> Result<(), Error> { - Err(Error::Unavailable) + Err(Error::Unavailable(None)) } } @@ -306,7 +306,7 @@ impl ClaimCache for Unavailable { _: &[String], _: ExactCacheContext, ) -> Result { - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) } } @@ -316,9 +316,12 @@ fn remote_failure_policy_selects_propagation_or_the_local_tier() { let strict = DualCache::new(Arc::new(TestCache::new(None, false)), Arc::new(Unavailable)); assert_eq!( strict.set_cache("key", "value".into(), &context), - Err(Error::Unavailable) + Err(Error::Unavailable(None)) + ); + assert_eq!( + strict.get_cache("key", &context), + Err(Error::Unavailable(None)) ); - assert_eq!(strict.get_cache("key", &context), Err(Error::Unavailable)); let l1 = Arc::new(TestCache::new(None, false)); let degraded = DualCache::new(l1.clone(), Arc::new(Unavailable)) @@ -347,7 +350,7 @@ fn claim_fallback_does_not_hide_non_availability_errors() { &[], ExactCacheContext::default() ), - Err(Error::InvalidEntry) + Err(Error::InvalidEntry(None)) ); } @@ -419,7 +422,7 @@ impl Tier { fn check(&self) -> Result<(), Error> { if self.fail { - return Err(Error::Unavailable); + return Err(Error::Unavailable(None)); } Ok(()) } @@ -598,7 +601,7 @@ async fn failed_async_l2_increment_leaves_l1_unchanged(log: Log) { cache .async_increment("counter", 2.0, seconds(30), true) .await, - Err(Error::Unavailable) + Err(Error::Unavailable(None)) ); assert_eq!( l1.counter("counter"), @@ -669,7 +672,7 @@ async fn failed_l2_increment_pipeline_leaves_l1_unchanged(log: Log) { ttl: None, }]) .await, - Err(Error::Unavailable) + Err(Error::Unavailable(None)) ); assert_eq!(events(&log), ["l2 pipeline a"]); assert_eq!(l1.counter("a"), Some((1.0, ExactCacheContext::default()))); @@ -713,7 +716,7 @@ async fn failed_local_set_add_never_reaches_l2(log: Log) { cache .async_set_cache_sadd("members", vec!["a".into()], None) .await, - Err(Error::Unavailable) + Err(Error::Unavailable(None)) ); assert!(events(&log).is_empty()); } @@ -768,7 +771,7 @@ async fn bulk_delete_stops_before_l2_when_the_local_delete_fails(log: Log) { ); assert_eq!( cache.delete_cache_keys(vec!["a".into()]).await, - Err(Error::Unavailable) + Err(Error::Unavailable(None)) ); assert!(events(&log).is_empty()); } diff --git a/litellm-rust/crates/callbacks-legacy-python/Cargo.toml b/litellm-rust/crates/callbacks-legacy-python/Cargo.toml index ed5e0fb9691..1cd05710366 100644 --- a/litellm-rust/crates/callbacks-legacy-python/Cargo.toml +++ b/litellm-rust/crates/callbacks-legacy-python/Cargo.toml @@ -6,12 +6,14 @@ license.workspace = true repository.workspace = true [dependencies] +litellm-error.workspace = true litellm-host.workspace = true litellm-host-python.workspace = true pyo3.workspace = true strum.workspace = true serde_json.workspace = true +thiserror.workspace = true [dev-dependencies] litellm-auth.workspace = true diff --git a/litellm-rust/crates/callbacks-legacy-python/src/adapter.rs b/litellm-rust/crates/callbacks-legacy-python/src/adapter.rs index d891b5a1f0c..493e91bb5ea 100644 --- a/litellm-rust/crates/callbacks-legacy-python/src/adapter.rs +++ b/litellm-rust/crates/callbacks-legacy-python/src/adapter.rs @@ -134,9 +134,10 @@ impl LegacyLogging { } fn logger(&self) -> PyResult<&PythonLogger> { - self.logger.as_ref().ok_or_else(|| { - pyo3::exceptions::PyRuntimeError::new_err("call logging is not initialized") - }) + Ok(self + .logger + .as_ref() + .ok_or(crate::error::Error::LoggingNotInitialized)?) } /// The keyword view the rest of the call reads: a copy, so the deployment hook's own diff --git a/litellm-rust/crates/callbacks-legacy-python/src/error.rs b/litellm-rust/crates/callbacks-legacy-python/src/error.rs new file mode 100644 index 00000000000..97ce62ae99c --- /dev/null +++ b/litellm-rust/crates/callbacks-legacy-python/src/error.rs @@ -0,0 +1,17 @@ +use litellm_error::Report; +use pyo3::{PyErr, exceptions::PyRuntimeError}; + +#[derive(Debug, thiserror::Error)] +pub(crate) enum Error { + #[error("call logging is not initialized")] + LoggingNotInitialized, +} + +impl From for PyErr { + fn from(error: Error) -> Self { + let message = Report::new(&error).to_string(); + match error { + Error::LoggingNotInitialized => PyRuntimeError::new_err(message), + } + } +} diff --git a/litellm-rust/crates/callbacks-legacy-python/src/lib.rs b/litellm-rust/crates/callbacks-legacy-python/src/lib.rs index 38c6b1aedbd..f453237b469 100644 --- a/litellm-rust/crates/callbacks-legacy-python/src/lib.rs +++ b/litellm-rust/crates/callbacks-legacy-python/src/lib.rs @@ -11,6 +11,7 @@ mod adapter; mod call; mod callbacks; mod deferred; +mod error; mod logger; mod mapping; mod python; diff --git a/litellm-rust/crates/config/Cargo.toml b/litellm-rust/crates/config/Cargo.toml index 9746a002bb9..5b0cd22332f 100644 --- a/litellm-rust/crates/config/Cargo.toml +++ b/litellm-rust/crates/config/Cargo.toml @@ -16,6 +16,7 @@ thiserror.workspace = true tracing.workspace = true [dev-dependencies] +litellm-error.workspace = true rstest.workspace = true tempfile.workspace = true tracing-subscriber = { version = "0.3", default-features = false, features = ["registry", "std"] } diff --git a/litellm-rust/crates/config/src/error.rs b/litellm-rust/crates/config/src/error.rs index 95c33967cf0..e5219ac992c 100644 --- a/litellm-rust/crates/config/src/error.rs +++ b/litellm-rust/crates/config/src/error.rs @@ -1,9 +1,16 @@ +use std::{io, path::PathBuf}; + #[derive(Debug, thiserror::Error)] pub enum Error { - #[error("could not read config")] - Read(#[from] std::io::Error), + #[error("could not read config {}", path.display())] + Read { path: PathBuf, source: io::Error }, #[error("invalid YAML config")] Parse(#[from] serde_yaml_ng::Error), + #[error("invalid YAML config {}", path.display())] + ParseFile { + path: PathBuf, + source: serde_yaml_ng::Error, + }, #[error("Redis host does not yet support configured setting {0}")] RedisSetting(String), #[error("invalid Redis setting {setting}: {reason}")] diff --git a/litellm-rust/crates/config/src/includes.rs b/litellm-rust/crates/config/src/includes.rs index 78df8e9a33a..27b908650df 100644 --- a/litellm-rust/crates/config/src/includes.rs +++ b/litellm-rust/crates/config/src/includes.rs @@ -15,11 +15,31 @@ struct Includes { } fn read(path: &Path) -> Result { - Ok(serde_yaml_ng::from_str(&std::fs::read_to_string(path)?)?) + let text = std::fs::read_to_string(path).map_err(|source| Error::Read { + path: path.to_owned(), + source, + })?; + serde_yaml_ng::from_str(&text).map_err(|source| Error::ParseFile { + path: path.to_owned(), + source, + }) +} + +fn canonical(path: &Path) -> Result { + path.canonicalize().map_err(|source| Error::Read { + path: path.to_owned(), + source, + }) } fn entries(config: &Mapping, path: &Path) -> Result, Error> { - let includes: Includes = serde_yaml_ng::from_value(Value::Mapping(config.clone()))?; + let includes: Includes = + serde_yaml_ng::from_value(Value::Mapping(config.clone())).map_err(|source| { + Error::ParseFile { + path: path.to_owned(), + source, + } + })?; Ok(includes .include .into_iter() @@ -28,7 +48,7 @@ fn entries(config: &Mapping, path: &Path) -> Result, Erro } pub(super) fn load(path: &Path) -> Result { - let root = path.canonicalize()?; + let root = canonical(path)?; let mut merged = read(&root)?; let mut pending: VecDeque<_> = entries(&merged, &root)?.into(); let mut loaded = BTreeSet::from([root.clone()]); @@ -36,12 +56,11 @@ pub(super) fn load(path: &Path) -> Result { while let Some((entry, declaring)) = pending.pop_front() { let declared = declaring.parent().unwrap_or(Path::new(".")).join(&entry); let fallback = root.parent().unwrap_or(Path::new(".")).join(&entry); - let location = if declared.exists() { - declared + let location = canonical(if declared.exists() { + &declared } else { - fallback - } - .canonicalize()?; + &fallback + })?; if !loaded.insert(location.clone()) { continue; } diff --git a/litellm-rust/crates/config/tests/config.rs b/litellm-rust/crates/config/tests/config.rs index d425b62efc2..3c7d02203e5 100644 --- a/litellm-rust/crates/config/tests/config.rs +++ b/litellm-rust/crates/config/tests/config.rs @@ -84,9 +84,15 @@ fn rejects_malformed_and_incomplete_config(#[case] yaml: &str) { #[rstest] fn distinguishes_read_errors_from_parse_errors(directory: TempDir) { + let error = Config::load(directory.path().join("missing.yaml")).unwrap_err(); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("missing.yaml") + ); assert!(matches!( - Config::load(directory.path().join("missing.yaml")), - Err(Error::Read(error)) if error.kind() == std::io::ErrorKind::NotFound + error, + Error::Read { source, .. } if source.kind() == std::io::ErrorKind::NotFound )); } diff --git a/litellm-rust/crates/core-utils/Cargo.toml b/litellm-rust/crates/core-utils/Cargo.toml index 1feea5fcc08..2e5857c79f3 100644 --- a/litellm-rust/crates/core-utils/Cargo.toml +++ b/litellm-rust/crates/core-utils/Cargo.toml @@ -7,6 +7,7 @@ repository.workspace = true [dependencies] fancy-regex.workspace = true +litellm-error.workspace = true litellm-tracing.workspace = true litellm-llms-types.workspace = true serde.workspace = true diff --git a/litellm-rust/crates/core-utils/src/call_arguments.rs b/litellm-rust/crates/core-utils/src/call_arguments.rs index 062dad0a92a..3277ca26c7f 100644 --- a/litellm-rust/crates/core-utils/src/call_arguments.rs +++ b/litellm-rust/crates/core-utils/src/call_arguments.rs @@ -1,5 +1,7 @@ use std::ops::Deref; +use litellm_error::Source; + use serde::{Deserialize, Serialize, de::DeserializeOwned}; use serde_json::{Map, Value}; @@ -17,9 +19,11 @@ impl CallArguments { } #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] -#[error("invalid argument: {path}")] +#[error("invalid argument {path}")] pub struct ArgumentError { pub path: String, + #[source] + source: Source, } pub fn parse_options(arguments: &CallArguments) -> Result { @@ -28,6 +32,7 @@ pub fn parse_options(arguments: &CallArguments) -> Result( body: &B, consumed: &[&str], ) -> Result { - let Value::Object(fields) = - serde_json::to_value(body).map_err(|_| crate::params::Error::Body)? + let Value::Object(fields) = serde_json::to_value(body) + .map_err(|error| crate::params::Error::Serialize(Source::new(error)))? else { return Err(crate::params::Error::Body); }; @@ -183,9 +188,9 @@ mod tests { assert_eq!(arguments.get("enabled"), Some(&Value::Null)); assert_eq!(arguments.get("missing"), None); let invalid = serde_json::from_value(json!({"enabled":0})).unwrap(); - assert_eq!( - parse_options::(&invalid).err().unwrap().path, - "enabled" - ); + let error = parse_options::(&invalid).err().unwrap(); + assert_eq!(error.path, "enabled"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.starts_with("invalid argument enabled: invalid type")); } } diff --git a/litellm-rust/crates/core-utils/src/exception_mapping_utils/mod.rs b/litellm-rust/crates/core-utils/src/exception_mapping_utils/mod.rs index 8b63fadf5a6..c99bbf78e02 100644 --- a/litellm-rust/crates/core-utils/src/exception_mapping_utils/mod.rs +++ b/litellm-rust/crates/core-utils/src/exception_mapping_utils/mod.rs @@ -337,15 +337,15 @@ mod tests { #[case::request_timed_out("Request timed out")] #[case::timed_out_generating("Timed out generating response")] #[case::read_operation("The read operation timed out")] - fn timeout_markers_win_over_every_family(#[case] marker: &str) { + fn timeout_markers_win_over_every_family( + #[case] marker: &str, + #[values("mistral", "vertex_ai", "cohere", "reducto")] provider: &str, + ) { let body = format!("rate limit invalid api token {marker}"); - for provider in ["mistral", "vertex_ai", "cohere", "reducto"] { - assert_eq!( - mapped(provider, &http(429, &body)).error, - PublicError::Timeout { status: 408 }, - "{provider}" - ); - } + assert_eq!( + mapped(provider, &http(429, &body)).error, + PublicError::Timeout { status: 408 } + ); } #[test] diff --git a/litellm-rust/crates/core-utils/src/params.rs b/litellm-rust/crates/core-utils/src/params.rs index 9545a3ef17b..f163399b878 100644 --- a/litellm-rust/crates/core-utils/src/params.rs +++ b/litellm-rust/crates/core-utils/src/params.rs @@ -4,6 +4,8 @@ pub enum Error { ExtraBody, #[error("invalid request: body must be a JSON object")] Body, + #[error("invalid request: body could not be serialized")] + Serialize(#[source] litellm_error::Source), } use std::ops::{Deref, DerefMut}; diff --git a/litellm-rust/crates/core-utils/src/url_utils.rs b/litellm-rust/crates/core-utils/src/url_utils.rs index ebbe7fca00b..63e4f041a72 100644 --- a/litellm-rust/crates/core-utils/src/url_utils.rs +++ b/litellm-rust/crates/core-utils/src/url_utils.rs @@ -4,7 +4,7 @@ use url::Url; #[derive(Debug, thiserror::Error)] pub enum ApiUrlError { - #[error("invalid URL: {0}")] + #[error("invalid URL")] Parse(#[from] url::ParseError), #[error("URL cannot be used as a base")] CannotBeBase, diff --git a/litellm-rust/crates/cost/Cargo.toml b/litellm-rust/crates/cost/Cargo.toml index 3ed27e82c9a..6921e6151b5 100644 --- a/litellm-rust/crates/cost/Cargo.toml +++ b/litellm-rust/crates/cost/Cargo.toml @@ -5,6 +5,9 @@ edition.workspace = true license.workspace = true repository.workspace = true +[dependencies] +thiserror.workspace = true + [dev-dependencies] criterion.workspace = true proptest.workspace = true diff --git a/litellm-rust/crates/cost/src/error.rs b/litellm-rust/crates/cost/src/error.rs new file mode 100644 index 00000000000..1753080617f --- /dev/null +++ b/litellm-rust/crates/cost/src/error.rs @@ -0,0 +1,27 @@ +#[derive(Clone, Copy, Debug, Eq, PartialEq, thiserror::Error)] +pub enum Error { + #[error("model has no input rate")] + MissingInputRate, + #[error("model has no output rate")] + MissingOutputRate, + #[error("rate must be finite and nonnegative")] + InvalidRate, + #[error("region multiplier must be finite and positive")] + InvalidRegionMultiplier, + #[error("off-peak pricing needs a billing minute within the day")] + InvalidBillingTime, + #[error("off-peak window must start before it ends within the day")] + InvalidOffPeakWindow, + #[error("cached tokens exceed prompt tokens")] + CacheExceedsPrompt, + #[error("cache write details do not add up to the cache write tokens")] + InvalidCacheWriteDetails, + #[error("token count or cost overflows")] + TokenCountOverflow, + #[error("service tier is repeated or cannot carry its own rates")] + DuplicateTier, + #[error("prompt token threshold is repeated")] + DuplicateThreshold, + #[error("service tier is repeated or cannot carry its own rates within a threshold")] + DuplicateThresholdTier, +} diff --git a/litellm-rust/crates/cost/src/lib.rs b/litellm-rust/crates/cost/src/lib.rs index 32cc755ce15..24c69afd9d4 100644 --- a/litellm-rust/crates/cost/src/lib.rs +++ b/litellm-rust/crates/cost/src/lib.rs @@ -1,3 +1,7 @@ +mod error; + +pub use error::Error; + #[derive(Clone, Copy, Debug, PartialEq)] pub enum Rate { Missing, @@ -156,22 +160,6 @@ pub struct EffectiveRates { pub cache_write_1h: f64, } -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum PricingError { - MissingInputRate, - MissingOutputRate, - InvalidRate, - InvalidRegionMultiplier, - InvalidBillingTime, - InvalidOffPeakWindow, - CacheExceedsPrompt, - InvalidCacheWriteDetails, - TokenCountOverflow, - DuplicateTier, - DuplicateThreshold, - DuplicateThresholdTier, -} - fn selected_tier(tier: ServiceTier) -> ServiceTier { if tier == ServiceTier::Fast { ServiceTier::Priority @@ -210,9 +198,9 @@ fn valid_rates(rates: Rates) -> bool { }) } -fn validate_tiers(tiers: &[TierRates], duplicate: PricingError) -> Result<(), PricingError> { +fn validate_tiers(tiers: &[TierRates], duplicate: Error) -> Result<(), Error> { if tiers.iter().any(|entry| !valid_rates(entry.rates)) { - return Err(PricingError::InvalidRate); + return Err(Error::InvalidRate); } if tiers.iter().enumerate().any(|(index, entry)| { matches!( @@ -227,20 +215,20 @@ fn validate_tiers(tiers: &[TierRates], duplicate: PricingError) -> Result<(), Pr Ok(()) } -pub fn compile(pricing: &Pricing<'_>) -> Result { +pub fn compile(pricing: &Pricing<'_>) -> Result { if !valid_rates(pricing.standard) { - return Err(PricingError::InvalidRate); + return Err(Error::InvalidRate); } - validate_tiers(pricing.tiers, PricingError::DuplicateTier)?; + validate_tiers(pricing.tiers, Error::DuplicateTier)?; if let Some(window) = pricing.off_peak { if window.start_utc_minute >= 1440 || window.end_utc_minute > 1440 || window.start_utc_minute >= window.end_utc_minute { - return Err(PricingError::InvalidOffPeakWindow); + return Err(Error::InvalidOffPeakWindow); } if !valid_rates(window.rates) { - return Err(PricingError::InvalidRate); + return Err(Error::InvalidRate); } } let mut thresholds: Vec<_> = pricing @@ -248,9 +236,9 @@ pub fn compile(pricing: &Pricing<'_>) -> Result { .iter() .map(|entry| { if !valid_rates(entry.standard) { - return Err(PricingError::InvalidRate); + return Err(Error::InvalidRate); } - validate_tiers(entry.tiers, PricingError::DuplicateThresholdTier)?; + validate_tiers(entry.tiers, Error::DuplicateThresholdTier)?; Ok(CompiledThreshold { above_prompt_tokens: entry.above_prompt_tokens, standard: entry.standard, @@ -263,7 +251,7 @@ pub fn compile(pricing: &Pricing<'_>) -> Result { .windows(2) .any(|pair| pair[0].above_prompt_tokens == pair[1].above_prompt_tokens) { - return Err(PricingError::DuplicateThreshold); + return Err(Error::DuplicateThreshold); } Ok(PricingPlan { standard: pricing.standard, @@ -274,11 +262,7 @@ pub fn compile(pricing: &Pricing<'_>) -> Result { } impl PricingPlan { - fn resolve_rates( - &self, - request: &Request, - threshold_tokens: u64, - ) -> Result { + fn resolve_rates(&self, request: &Request, threshold_tokens: u64) -> Result { let tier = selected_tier(request.service_tier); let base = self .tiers @@ -305,13 +289,13 @@ impl PricingPlan { || window.end_utc_minute > 1440 || window.start_utc_minute >= window.end_utc_minute { - return Err(PricingError::InvalidOffPeakWindow); + return Err(Error::InvalidOffPeakWindow); } let minute = request .billed_at_utc_minute - .ok_or(PricingError::InvalidBillingTime)?; + .ok_or(Error::InvalidBillingTime)?; if minute >= 1440 { - return Err(PricingError::InvalidBillingTime); + return Err(Error::InvalidBillingTime); } if (window.start_utc_minute..window.end_utc_minute).contains(&minute) { Ok(window.rates.overlay(selected)) @@ -322,26 +306,26 @@ impl PricingPlan { } } - fn checked_rate(rate: Rate, missing: PricingError) -> Result { + fn checked_rate(rate: Rate, missing: Error) -> Result { let value = rate.value().ok_or(missing)?; if !value.is_finite() || value < 0.0 { - return Err(PricingError::InvalidRate); + return Err(Error::InvalidRate); } Ok(value) } - pub fn calculate(&self, request: &Request) -> Result { + pub fn calculate(&self, request: &Request) -> Result { let usage = request.usage; let cached = usage .cache_read_tokens .checked_add(usage.cache_write_tokens) - .ok_or(PricingError::TokenCountOverflow)?; + .ok_or(Error::TokenCountOverflow)?; let (regular, threshold_tokens) = match usage.prompt_convention { PromptConvention::IncludesCache => ( usage .prompt_tokens .checked_sub(cached) - .ok_or(PricingError::CacheExceedsPrompt)?, + .ok_or(Error::CacheExceedsPrompt)?, usage.prompt_tokens, ), PromptConvention::ExcludesCache => ( @@ -349,7 +333,7 @@ impl PricingPlan { usage .prompt_tokens .checked_add(cached) - .ok_or(PricingError::TokenCountOverflow)?, + .ok_or(Error::TokenCountOverflow)?, ), }; let writes = match (usage.cache_write_5m_tokens, usage.cache_write_1h_tokens) { @@ -357,26 +341,20 @@ impl PricingPlan { (Some(five), Some(one)) if five.checked_add(one) == Some(usage.cache_write_tokens) => { (five, one) } - _ => return Err(PricingError::InvalidCacheWriteDetails), + _ => return Err(Error::InvalidCacheWriteDetails), }; let rates = self.resolve_rates(request, threshold_tokens)?; - let input = Self::checked_rate(rates.input, PricingError::MissingInputRate)?; - let output = Self::checked_rate(rates.output, PricingError::MissingOutputRate)?; - let read = Self::checked_rate( - rates.cache_read.or(rates.input), - PricingError::MissingInputRate, - )?; - let write = Self::checked_rate( - rates.cache_write.or(rates.input), - PricingError::MissingInputRate, - )?; + let input = Self::checked_rate(rates.input, Error::MissingInputRate)?; + let output = Self::checked_rate(rates.output, Error::MissingOutputRate)?; + let read = Self::checked_rate(rates.cache_read.or(rates.input), Error::MissingInputRate)?; + let write = Self::checked_rate(rates.cache_write.or(rates.input), Error::MissingInputRate)?; let write_1h = Self::checked_rate( rates.cache_write_1h.or(rates.cache_write).or(rates.input), - PricingError::MissingInputRate, + Error::MissingInputRate, )?; let multiplier = request.region_multiplier.unwrap_or(1.0); if !multiplier.is_finite() || multiplier <= 0.0 { - return Err(PricingError::InvalidRegionMultiplier); + return Err(Error::InvalidRegionMultiplier); } let cost = Cost { uncached_input: regular as f64 * input, @@ -394,12 +372,12 @@ impl PricingPlan { }, }; if !cost.total().is_finite() { - return Err(PricingError::TokenCountOverflow); + return Err(Error::TokenCountOverflow); } Ok(cost) } } -pub fn calculate(pricing: &Pricing<'_>, request: &Request) -> Result { +pub fn calculate(pricing: &Pricing<'_>, request: &Request) -> Result { compile(pricing)?.calculate(request) } diff --git a/litellm-rust/crates/cost/tests/calculation.rs b/litellm-rust/crates/cost/tests/calculation.rs index f8152483a7c..af1a73d8c61 100644 --- a/litellm-rust/crates/cost/tests/calculation.rs +++ b/litellm-rust/crates/cost/tests/calculation.rs @@ -1,5 +1,5 @@ use litellm_cost::{ - OffPeakRates, Pricing, PricingError, PromptConvention, Rate, Rates, Request, ServiceTier, + Error, OffPeakRates, Pricing, PromptConvention, Rate, Rates, Request, ServiceTier, ThresholdPolicy, ThresholdRates, TierRates, Usage, calculate, compile, }; use rstest::rstest; @@ -132,7 +132,7 @@ fn split_writes_and_invalid_accounting() { }; assert_eq!( calculate(&pricing(standard), &overlapping), - Err(PricingError::CacheExceedsPrompt) + Err(Error::CacheExceedsPrompt) ); let incomplete = Request { usage: Usage { @@ -143,7 +143,7 @@ fn split_writes_and_invalid_accounting() { }; assert_eq!( calculate(&pricing(standard), &incomplete), - Err(PricingError::InvalidCacheWriteDetails) + Err(Error::InvalidCacheWriteDetails) ); } @@ -297,10 +297,10 @@ fn compile_rejects_ambiguous_rates() { }; assert_eq!( compile(&specification).err(), - Some(PricingError::DuplicateThreshold) + Some(Error::DuplicateThreshold) ); let invalid = pricing(rates(Rate::Value(f64::NAN), Rate::Value(1.0))); - assert_eq!(compile(&invalid).err(), Some(PricingError::InvalidRate)); + assert_eq!(compile(&invalid).err(), Some(Error::InvalidRate)); } #[test] @@ -326,7 +326,7 @@ fn off_peak_is_one_non_wrapping_utc_window() { }; assert_eq!( calculate(&specification, &base), - Err(PricingError::InvalidBillingTime) + Err(Error::InvalidBillingTime) ); assert_eq!(calculate(&specification, &start).unwrap().input(), 100.0); assert_eq!(calculate(&specification, &end).unwrap().input(), 200.0); @@ -347,11 +347,11 @@ fn missing_rates_and_free_rates_remain_distinct() { }; assert_eq!( calculate(&pricing(Rates::EMPTY), &empty), - Err(PricingError::MissingInputRate) + Err(Error::MissingInputRate) ); assert_eq!( calculate(&pricing(rates(Rate::Value(0.0), Rate::Missing)), &empty), - Err(PricingError::MissingOutputRate) + Err(Error::MissingOutputRate) ); assert_eq!( calculate(&pricing(rates(Rate::Value(0.0), Rate::Value(0.0))), &empty) @@ -481,7 +481,7 @@ fn regional_multiplier_applies_after_input_components_are_summed() { }; assert_eq!( calculate(&pricing(standard), &invalid), - Err(PricingError::InvalidRegionMultiplier) + Err(Error::InvalidRegionMultiplier) ); } @@ -531,5 +531,5 @@ fn compilation_sorts_thresholds_and_rejects_duplicate_tiers() { thresholds: &[], ..specification }; - assert_eq!(compile(&invalid).err(), Some(PricingError::DuplicateTier)); + assert_eq!(compile(&invalid).err(), Some(Error::DuplicateTier)); } diff --git a/litellm-rust/crates/error/Cargo.toml b/litellm-rust/crates/error/Cargo.toml new file mode 100644 index 00000000000..fa2d26a1b99 --- /dev/null +++ b/litellm-rust/crates/error/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "litellm-error" +version = "0.1.0" +edition.workspace = true +license.workspace = true +repository.workspace = true +description = "Shared error rendering and type-erased causes for litellm-rust boundaries" + +[dependencies] +thiserror.workspace = true + +[dev-dependencies] +rstest.workspace = true diff --git a/litellm-rust/crates/error/src/lib.rs b/litellm-rust/crates/error/src/lib.rs new file mode 100644 index 00000000000..ec8d5c3e1f4 --- /dev/null +++ b/litellm-rust/crates/error/src/lib.rs @@ -0,0 +1,132 @@ +use std::{error::Error, fmt, ops::Deref, sync::Arc}; + +pub struct Report<'a>(&'a (dyn Error + 'static)); + +impl<'a> Report<'a> { + pub fn new(error: &'a (dyn Error + 'static)) -> Self { + Self(error) + } +} + +impl fmt::Display for Report<'_> { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "{}", self.0)?; + std::iter::successors(self.0.source(), |&cause| cause.source()) + .try_for_each(|cause| write!(formatter, ": {cause}")) + } +} + +pub fn find<'a, T: Error + 'static>(error: &'a (dyn Error + 'static)) -> Option<&'a T> { + std::iter::successors(Some(error), |&cause| cause.source()).find_map(|cause| { + cause + .downcast_ref::() + .or_else(|| cause.downcast_ref::()?.downcast_ref::()) + }) +} + +#[derive(Clone, Debug, thiserror::Error)] +#[error(transparent)] +pub struct Source(Arc); + +impl Source { + pub fn new(error: impl Error + Send + Sync + 'static) -> Self { + Self(Arc::new(error)) + } +} + +impl From> for Source { + fn from(error: Box) -> Self { + Self(Arc::from(error)) + } +} + +impl Deref for Source { + type Target = dyn Error + Send + Sync; + + fn deref(&self) -> &Self::Target { + self.0.as_ref() + } +} + +impl PartialEq for Source { + fn eq(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.0, &other.0) + } +} + +impl Eq for Source {} + +#[cfg(test)] +mod tests { + use std::io; + + use rstest::rstest; + + use super::{Report, Source, find}; + + #[derive(Debug, thiserror::Error)] + enum Leaf { + #[error("could not read the config")] + Read(#[source] io::Error), + #[error("could not parse the config")] + Parse(#[source] Source), + #[error("config is empty")] + Empty, + } + + #[derive(Debug, thiserror::Error)] + enum Outer { + #[error("startup failed")] + Startup(#[from] Leaf), + #[error(transparent)] + Nested(Leaf), + } + + fn denied() -> io::Error { + io::Error::new(io::ErrorKind::PermissionDenied, "permission denied") + } + + #[rstest] + #[case::leaf(Outer::Nested(Leaf::Empty), "config is empty")] + #[case::context_over_a_cause( + Outer::Startup(Leaf::Read(denied())), + "startup failed: could not read the config: permission denied" + )] + #[case::transparent_adds_no_text( + Outer::Nested(Leaf::Read(denied())), + "could not read the config: permission denied" + )] + #[case::erased_cause_keeps_its_chain( + Outer::Nested(Leaf::Parse(Source::new(Leaf::Read(denied())))), + "could not parse the config: could not read the config: permission denied" + )] + fn report_renders_every_layer_once(#[case] error: Outer, #[case] expected: &str) { + assert_eq!(Report::new(&error).to_string(), expected); + } + + #[rstest] + #[case::direct_source(Outer::Startup(Leaf::Read(denied())))] + #[case::behind_an_erased_source(Outer::Startup(Leaf::Parse(Source::new(Leaf::Read( + denied() + )))))] + #[case::behind_a_boxed_source(Outer::Startup(Leaf::Parse(Source::from( + Box::::from(denied()) + ))))] + #[case::behind_a_transparent_layer(Outer::Nested(Leaf::Parse(Source::new(denied()))))] + fn find_reaches_the_typed_cause_anywhere_in_the_chain(#[case] error: Outer) { + let cause = find::(&error).expect("io cause"); + assert_eq!(cause.kind(), io::ErrorKind::PermissionDenied); + } + + #[test] + fn find_answers_none_when_the_chain_has_no_such_cause() { + assert!(find::(&Outer::Nested(Leaf::Empty)).is_none()); + } + + #[test] + fn a_clone_shares_its_cause() { + let source = Source::new(denied()); + assert_eq!(source.clone(), source); + assert_ne!(Source::new(denied()), source); + } +} diff --git a/litellm-rust/crates/framer/src/error.rs b/litellm-rust/crates/framer/src/error.rs index 879d7557671..dde421d71af 100644 --- a/litellm-rust/crates/framer/src/error.rs +++ b/litellm-rust/crates/framer/src/error.rs @@ -1,21 +1,21 @@ #[cfg(feature = "sse")] #[derive(Debug, thiserror::Error)] pub enum SseError { - #[error("body stream failed: {0}")] + #[error("body stream failed")] Body(#[from] std::io::Error), - #[error("SSE field is not UTF-8: {0}")] + #[error("SSE field is not UTF-8")] InvalidUtf8(#[from] std::str::Utf8Error), } #[cfg(feature = "aws")] #[derive(Debug, thiserror::Error)] pub enum EventStreamError { - #[error("body stream failed: {0}")] + #[error("body stream failed")] Body(#[from] std::io::Error), #[error("invalid AWS EventStream frame length: {0}")] InvalidLength(usize), #[error("truncated AWS EventStream frame")] Truncated, - #[error("malformed AWS EventStream frame: {0}")] + #[error("malformed AWS EventStream frame")] Malformed(#[from] aws_smithy_eventstream::error::Error), } diff --git a/litellm-rust/crates/framer/src/framer.rs b/litellm-rust/crates/framer/src/framer.rs deleted file mode 100644 index 507aed54700..00000000000 --- a/litellm-rust/crates/framer/src/framer.rs +++ /dev/null @@ -1,13 +0,0 @@ -use futures_util::Stream; - -use crate::Error; - -pub trait Framer: Send { - type Frame: Send; - - fn frame(self, input: S) -> impl Stream> + Send - where - S: Stream> + Send, - B: bytes::Buf + Send, - E: std::error::Error + Send + Sync + 'static; -} diff --git a/litellm-rust/crates/gateway-auth/AGENTS.md b/litellm-rust/crates/gateway-auth/AGENTS.md index 58457a3151b..0a4bc6450ae 100644 --- a/litellm-rust/crates/gateway-auth/AGENTS.md +++ b/litellm-rust/crates/gateway-auth/AGENTS.md @@ -20,7 +20,9 @@ Local UI login continues using axum-login and tower-sessions. Only after session Authentication evidence and principals contain no raw token, password, request body, or mutable accounting state. Session ownership is scoped by principal authority, subject, verifier, and credential ID, so separate credentials do not silently share an MCP session. Credential rotation may retain ownership when the verifier preserves a stable credential ID. Scope and expiry checks still run for each operation -Failures distinguish invalid or expired credentials, forbidden operations, unavailable authentication services, and missing server configuration/context. HTTP adapters map those outcomes to status codes. Inference keeps its API-specific error envelopes and MCP translates policy failures to its own protocol +Failures distinguish invalid or expired credentials, forbidden operations, unavailable authentication services, and missing server configuration/context. Bearer tokens, virtual keys and UI sessions all fail with the one `Error`, classified once by `status()`, and `is_denied()` tells a rejected caller from an auth failure. Inference keeps its API-specific error envelopes, the UI renders `Error` in its own envelope and MCP translates policy failures to its own protocol + +The plain-text rejection body is the top-level message only, never the `Report` chain, because the caller is unauthenticated and the sources carry secret-manager and backend detail. That stays reachable through `source()` for logs Virtual-key storage, JWT verification, OAuth2 introspection, trusted-proxy validation, SSO, and custom Python hook adapters are not implemented here yet. They should supply these contracts rather than bypassing the shared authorization boundary. Request-body-dependent custom hooks will need a bounded endpoint adapter after parsing diff --git a/litellm-rust/crates/gateway-auth/Cargo.toml b/litellm-rust/crates/gateway-auth/Cargo.toml index cf040c6eef3..27f20203843 100644 --- a/litellm-rust/crates/gateway-auth/Cargo.toml +++ b/litellm-rust/crates/gateway-auth/Cargo.toml @@ -10,6 +10,7 @@ axum = { workspace = true, features = ["matched-path"] } axum-login.workspace = true litellm-auth-types.workspace = true litellm-config.workspace = true +litellm-error.workspace = true litellm-secrets.workspace = true sha2.workspace = true subtle.workspace = true diff --git a/litellm-rust/crates/gateway-auth/src/error.rs b/litellm-rust/crates/gateway-auth/src/error.rs index 262a666df80..7bb6292b18c 100644 --- a/litellm-rust/crates/gateway-auth/src/error.rs +++ b/litellm-rust/crates/gateway-auth/src/error.rs @@ -2,44 +2,59 @@ use axum::{ http::StatusCode, response::{IntoResponse, Response}, }; - -#[derive(Debug, thiserror::Error)] -pub enum KeyError { - #[error("missing or invalid virtual key")] - Invalid, - #[error("virtual key has expired")] - Expired, - #[error("virtual key lookup unavailable")] - Lookup(#[source] Box), -} +use litellm_error::Source; #[derive(Debug, thiserror::Error)] pub enum Error { - #[error("operation is not permitted")] - Forbidden, - #[error("credential has expired")] - Expired, - #[error("authenticated request context is missing")] - MissingIdentity, - #[error("authentication service unavailable")] - Unavailable, - #[error("gateway auth not configured")] - Unconfigured, #[error("missing or invalid bearer token")] InvalidToken, + #[error("invalid UI credentials or session")] + InvalidSession, + #[error("credential has expired")] + Expired, + #[error("operation is not permitted")] + Forbidden, + #[error("authentication service unavailable")] + Unavailable(#[source] Source), #[error("gateway authentication unavailable")] Secret(#[from] litellm_secrets::Error), + #[error("UI session unavailable")] + Session(#[from] tower_sessions::session::Error), + #[error("gateway auth not configured")] + Unconfigured, + #[error("UI_USERNAME and UI_PASSWORD must be nonempty")] + UiUnconfigured, + #[error("authenticated request context is missing")] + MissingIdentity, + #[error("UI session layer is not installed")] + MissingSessionLayer, } impl Error { pub fn status(&self) -> StatusCode { match self { - Self::InvalidToken | Self::Expired => StatusCode::UNAUTHORIZED, + Self::InvalidToken | Self::InvalidSession | Self::Expired => StatusCode::UNAUTHORIZED, Self::Forbidden => StatusCode::FORBIDDEN, - Self::Unavailable => StatusCode::SERVICE_UNAVAILABLE, - Self::MissingIdentity | Self::Unconfigured | Self::Secret(_) => { - StatusCode::INTERNAL_SERVER_ERROR - } + Self::Unavailable(_) => StatusCode::SERVICE_UNAVAILABLE, + Self::Secret(_) + | Self::Session(_) + | Self::Unconfigured + | Self::UiUnconfigured + | Self::MissingIdentity + | Self::MissingSessionLayer => StatusCode::INTERNAL_SERVER_ERROR, + } + } + + pub fn is_denied(&self) -> bool { + match self { + Self::InvalidToken | Self::InvalidSession | Self::Expired | Self::Forbidden => true, + Self::Unavailable(_) + | Self::Secret(_) + | Self::Session(_) + | Self::Unconfigured + | Self::UiUnconfigured + | Self::MissingIdentity + | Self::MissingSessionLayer => false, } } } @@ -50,24 +65,31 @@ impl IntoResponse for Error { } } -#[derive(Debug, thiserror::Error)] -pub enum UiAuthError { - #[error("UI_USERNAME and UI_PASSWORD must be nonempty")] - Unconfigured, - #[error("invalid UI credentials or session")] - Unauthorized, - #[error("UI authentication unavailable")] - Unavailable, - #[error("UI session unavailable")] - Session(#[from] tower_sessions::session::Error), -} +#[cfg(test)] +mod tests { + use rstest::rstest; -impl IntoResponse for UiAuthError { - fn into_response(self) -> Response { - let status = match self { - Self::Unauthorized => StatusCode::UNAUTHORIZED, - _ => StatusCode::INTERNAL_SERVER_ERROR, - }; - (status, self.to_string()).into_response() + use super::*; + + #[rstest] + #[tokio::test] + async fn rejection_body_hides_backend_causes() { + let error = + Error::Unavailable(Source::new(std::io::Error::other("private-backend-detail"))); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("private-backend-detail") + ); + let response = error.into_response(); + assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE); + let body = axum::body::to_bytes(response.into_body(), 4096) + .await + .unwrap(); + assert!( + !std::str::from_utf8(&body) + .unwrap() + .contains("private-backend-detail") + ); } } diff --git a/litellm-rust/crates/gateway-auth/src/keys.rs b/litellm-rust/crates/gateway-auth/src/keys.rs index 46d8c02253f..1fe8fa60e0e 100644 --- a/litellm-rust/crates/gateway-auth/src/keys.rs +++ b/litellm-rust/crates/gateway-auth/src/keys.rs @@ -2,7 +2,7 @@ use std::{future::Future, pin::Pin, time::SystemTime}; use litellm_auth_types::SecretValue; -use crate::{KeyError, hash_token}; +use crate::{Error, hash_token}; #[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)] pub struct KeyHash(String); @@ -24,7 +24,7 @@ pub enum KeyStatus { } pub type LookupFuture<'a> = - Pin, KeyError>> + Send + 'a>>; + Pin, Error>> + Send + 'a>>; pub trait KeyLookup: Send + Sync { fn lookup<'a>(&'a self, hash: &'a KeyHash) -> LookupFuture<'a>; @@ -34,15 +34,15 @@ pub async fn verify_key( lookup: &dyn KeyLookup, token: &SecretValue, now: impl FnOnce() -> SystemTime, -) -> Result { +) -> Result { let hash = KeyHash::from_token(token); match lookup.lookup(&hash).await? { Some(KeyStatus::Active { expires_at }) => { if expires_at.is_some_and(|expiry| expiry <= now()) { - return Err(KeyError::Expired); + return Err(Error::Expired); } Ok(hash) } - Some(KeyStatus::Revoked) | None => Err(KeyError::Invalid), + Some(KeyStatus::Revoked) | None => Err(Error::InvalidToken), } } diff --git a/litellm-rust/crates/gateway-auth/src/lib.rs b/litellm-rust/crates/gateway-auth/src/lib.rs index 1b400a94746..31ce17b3aa4 100644 --- a/litellm-rust/crates/gateway-auth/src/lib.rs +++ b/litellm-rust/crates/gateway-auth/src/lib.rs @@ -16,7 +16,7 @@ pub use authorization::{ AccessRequest, AuthenticatedRequest, AuthorizedOperation, Authorizer, McpAction, NoAdditionalPolicy, Permissions, UiAction, }; -pub use error::{Error, KeyError, UiAuthError}; +pub use error::Error; pub use http::{Bearer, CredentialExtractor, RequireMasterKey, authenticate}; pub use identity::{ AuthenticatedCaller, Authentication, AuthenticationMethod, Principal, PrincipalKind, diff --git a/litellm-rust/crates/gateway-auth/src/ui.rs b/litellm-rust/crates/gateway-auth/src/ui.rs index 9e6b77a17d0..3d655bde187 100644 --- a/litellm-rust/crates/gateway-auth/src/ui.rs +++ b/litellm-rust/crates/gateway-auth/src/ui.rs @@ -11,9 +11,9 @@ use tower_sessions::Session; use veil::Redact; use crate::{ - AccessRequest, AuthenticatedRequest, Authentication, AuthenticationMethod, LocalAdministrator, - NoAdditionalPolicy, Permissions, Principal, PrincipalKind, SystemClock, UiAction, UiAuthError, - VerifiedIdentity, + AccessRequest, AuthenticatedRequest, Authentication, AuthenticationMethod, Error, + LocalAdministrator, NoAdditionalPolicy, Permissions, Principal, PrincipalKind, SystemClock, + UiAction, VerifiedIdentity, }; pub const UI_CSRF_KEY: &str = "litellm.ui.csrf"; @@ -50,9 +50,9 @@ pub struct UiBackend { } impl UiBackend { - pub fn new(username: String, password: SecretValue) -> Result { + pub fn new(username: String, password: SecretValue) -> Result { if username.trim().is_empty() || password.expose().trim().is_empty() { - return Err(UiAuthError::Unconfigured); + return Err(Error::UiUnconfigured); } Ok(Self { user: UiUser { @@ -89,28 +89,28 @@ pub struct UiSession { } impl FromRequestParts for UiSession { - type Rejection = UiAuthError; + type Rejection = Error; async fn from_request_parts(parts: &mut Parts, state: &S) -> Result { let auth = UiAuthSession::from_request_parts(parts, state) .await - .map_err(|_| UiAuthError::Unavailable)?; - let user = auth.user.ok_or(UiAuthError::Unauthorized)?; + .map_err(|_| Error::MissingSessionLayer)?; + let user = auth.user.ok_or(Error::InvalidSession)?; let session = Session::from_request_parts(parts, state) .await - .map_err(|_| UiAuthError::Unavailable)?; + .map_err(|_| Error::MissingSessionLayer)?; let expected = session .get::(UI_CSRF_KEY) .await? - .ok_or(UiAuthError::Unauthorized)?; + .ok_or(Error::InvalidSession)?; let provided = parts .headers .get(AUTHORIZATION) .and_then(|value| value.to_str().ok()) .and_then(|value| value.strip_prefix("Bearer ")) - .ok_or(UiAuthError::Unauthorized)?; + .ok_or(Error::InvalidSession)?; if !bool::from(Sha256::digest(expected).ct_eq(&Sha256::digest(provided))) { - return Err(UiAuthError::Unauthorized); + return Err(Error::InvalidSession); } let identity = session_identity(&user).await?; parts.extensions.insert(identity.clone()); @@ -118,7 +118,7 @@ impl FromRequestParts for UiSession { } } -async fn session_identity(user: &UiUser) -> Result { +async fn session_identity(user: &UiUser) -> Result { crate::authentication::resolve( VerifiedIdentity { principal: Principal::new( @@ -142,7 +142,6 @@ async fn session_identity(user: &UiUser) -> Result Ok(Some(SecretValue::new("resolved-key"))), "EMPTY" => Ok(Some(SecretValue::new(""))), - "ERROR" => Err(litellm_secrets::Error::ExternalRead(Box::new( - std::io::Error::other("private-backend-detail"), - ))), + "ERROR" => Err(litellm_secrets::Error::ExternalRead( + litellm_error::Source::new(std::io::Error::other("private-backend-detail")), + )), _ => Ok(None), } }) @@ -132,7 +132,11 @@ impl Authenticator for Verifier { Box::pin(async move { match self.failure { Some(401) => return Err(Error::InvalidToken), - Some(_) => return Err(Error::Unavailable), + Some(_) => { + return Err(Error::Unavailable(litellm_error::Source::new( + std::io::Error::other("verifier offline"), + ))); + } None => (), } let litellm_gateway_auth::Credential::Token(credential) = credential else { diff --git a/litellm-rust/crates/gateway-auth/tests/keys.rs b/litellm-rust/crates/gateway-auth/tests/keys.rs index 51ef23a859f..4d16d3573bc 100644 --- a/litellm-rust/crates/gateway-auth/tests/keys.rs +++ b/litellm-rust/crates/gateway-auth/tests/keys.rs @@ -2,7 +2,7 @@ use std::time::{Duration, SystemTime}; use litellm_auth_types::SecretValue; use litellm_gateway_auth::{ - KeyError, + Error, keys::{KeyHash, KeyLookup, KeyStatus, LookupFuture, verify_key}, }; use rstest::{fixture, rstest}; @@ -59,7 +59,7 @@ async fn checks_expiration( match valid { true => assert_eq!(result.unwrap(), hash), - false => assert!(matches!(result, Err(KeyError::Expired))), + false => assert!(matches!(result, Err(Error::Expired))), } } @@ -79,7 +79,7 @@ async fn rejects_unusable_keys( assert!(matches!( verify_key(&lookup, &token, || now).await, - Err(KeyError::Invalid) + Err(Error::InvalidToken) )); } @@ -93,7 +93,7 @@ async fn rejects_a_different_token(token: SecretValue, now: SystemTime) { assert!(matches!( verify_key(&lookup, &SecretValue::new("sk-other-credential"), || now).await, - Err(KeyError::Invalid) + Err(Error::InvalidToken) )); } @@ -102,9 +102,9 @@ struct Unavailable; impl KeyLookup for Unavailable { fn lookup<'a>(&'a self, _: &'a KeyHash) -> LookupFuture<'a> { Box::pin(async { - Err(KeyError::Lookup(Box::new(std::io::Error::other( - "backend detail", - )))) + Err(Error::Unavailable(litellm_error::Source::new( + std::io::Error::other("backend detail"), + ))) }) } } @@ -114,10 +114,8 @@ impl KeyLookup for Unavailable { async fn lookup_failure_preserves_source_without_exposing_it(token: SecretValue, now: SystemTime) { let error = verify_key(&Unavailable, &token, || now).await.unwrap_err(); - assert!(matches!(&error, KeyError::Lookup(_))); - assert_eq!( - std::error::Error::source(&error).unwrap().to_string(), - "backend detail" - ); + assert_eq!(error.status(), axum::http::StatusCode::SERVICE_UNAVAILABLE); + let cause = litellm_error::find::(&error).expect("lookup cause"); + assert_eq!(cause.to_string(), "backend detail"); assert!(!error.to_string().contains("backend detail")); } diff --git a/litellm-rust/crates/gateway-inference/Cargo.toml b/litellm-rust/crates/gateway-inference/Cargo.toml index 312d44b9a4b..034c3a9826e 100644 --- a/litellm-rust/crates/gateway-inference/Cargo.toml +++ b/litellm-rust/crates/gateway-inference/Cargo.toml @@ -11,6 +11,7 @@ axum = { workspace = true, features = ["json", "multipart", "original-uri"] } base64.workspace = true bytes.workspace = true litellm-auth.workspace = true +litellm-error.workspace = true litellm-gateway-auth.workspace = true litellm-inference.workspace = true litellm-inference-transcription.workspace = true @@ -34,5 +35,6 @@ litellm-cache-memory.workspace = true futures-util.workspace = true tokio = { workspace = true, features = ["io-util"] } rstest.workspace = true +tokio-tungstenite.workspace = true tower = { version = "0.5.3", features = ["util"] } wiremock.workspace = true diff --git a/litellm-rust/crates/gateway-inference/src/audio_transcription.rs b/litellm-rust/crates/gateway-inference/src/audio_transcription.rs index 8d571eccc5b..d4cb467f6fd 100644 --- a/litellm-rust/crates/gateway-inference/src/audio_transcription.rs +++ b/litellm-rust/crates/gateway-inference/src/audio_transcription.rs @@ -27,7 +27,7 @@ async fn handle( upload, }: InferenceBody, ) -> Result { - let deployment = request::resolve_deployment(gateway, &body)?; + let deployment = request::resolve_deployment(gateway, "/audio/transcriptions", &body)?; request::authorize_model(identity, deployment, &body).await?; let audio = match upload { Some(upload) => { diff --git a/litellm-rust/crates/gateway-inference/src/caching.rs b/litellm-rust/crates/gateway-inference/src/caching.rs index 426b8b91182..72f07d5d621 100644 --- a/litellm-rust/crates/gateway-inference/src/caching.rs +++ b/litellm-rust/crates/gateway-inference/src/caching.rs @@ -5,7 +5,7 @@ use litellm_gateway_auth::AuthenticatedRequest; use serde::Deserialize; use serde_json::{Map, Value}; -use crate::Error; +use crate::BodyError; #[derive(Default, Deserialize)] #[serde(default, deny_unknown_fields)] @@ -24,10 +24,9 @@ type Prepared = (Map, CacheOptions); pub(crate) fn prepare( identity: &AuthenticatedRequest, body: Map, -) -> Result { +) -> Result { let controls: Controls = match body.get("cache").filter(|value| !value.is_null()) { - Some(value) => serde_json::from_value(value.clone()) - .map_err(|error| Error::InvalidBody(error.to_string()))?, + Some(value) => serde_json::from_value(value.clone()).map_err(BodyError::CacheControls)?, None => Controls::default(), }; let caching: Option = body @@ -35,7 +34,7 @@ pub(crate) fn prepare( .filter(|value| !value.is_null()) .map(|value| serde_json::from_value(value.clone())) .transpose() - .map_err(|error| Error::InvalidBody(error.to_string()))?; + .map_err(BodyError::Caching)?; let caller = identity.caller(); let options = CacheOptions { policy: CachePolicy { @@ -62,11 +61,12 @@ pub(crate) fn prepare( )) } -fn duration(seconds: f64) -> Result { - Duration::try_from_secs_f64(seconds) - .ok() - .filter(|duration| !duration.is_zero()) - .ok_or_else(|| Error::InvalidBody("cache durations must be finite and positive".into())) +fn duration(seconds: f64) -> Result { + match Duration::try_from_secs_f64(seconds) { + Ok(duration) if !duration.is_zero() => Ok(duration), + Ok(_) => Err(BodyError::CacheDuration(None)), + Err(error) => Err(BodyError::CacheDuration(Some(error))), + } } #[derive(Clone, Default)] diff --git a/litellm-rust/crates/gateway-inference/src/chat_completions.rs b/litellm-rust/crates/gateway-inference/src/chat_completions.rs index 742a373b286..fb8a6f4609d 100644 --- a/litellm-rust/crates/gateway-inference/src/chat_completions.rs +++ b/litellm-rust/crates/gateway-inference/src/chat_completions.rs @@ -3,13 +3,16 @@ use std::sync::Arc; use axum::{ Json, - extract::{Path, State}, + extract::State, response::{IntoResponse, Response}, }; use litellm_inference_chat::types::ChatCompletionsCall; use serde_json::{Map, Value}; -use crate::{Error, Gateway, JsonObject, request}; +use crate::{ + Error, Gateway, JsonObject, + request::{self, Path}, +}; pub(crate) async fn create( State(gateway): State>, @@ -40,7 +43,7 @@ async fn handle( identity: &AuthenticatedRequest, body: Map, ) -> Result { - let deployment = request::resolve_deployment(gateway, &body)?; + let deployment = request::resolve_deployment(gateway, "/chat/completions", &body)?; request::authorize_model(identity, deployment, &body).await?; let (body, cache_options) = crate::caching::prepare(identity, body)?; let route = gateway.chat_completions.clone(); diff --git a/litellm-rust/crates/gateway-inference/src/error.rs b/litellm-rust/crates/gateway-inference/src/error.rs index ec82a6ede12..505483b426c 100644 --- a/litellm-rust/crates/gateway-inference/src/error.rs +++ b/litellm-rust/crates/gateway-inference/src/error.rs @@ -3,7 +3,6 @@ use axum::{ Json, response::{IntoResponse, Response}, }; -use litellm_http::transport::Error as TransportError; use litellm_inference::Error as InferenceError; use litellm_llms::base_llm::ocr::error::Error as OcrError; use serde_json::{Map, Value, json}; @@ -12,12 +11,14 @@ use serde_json::{Map, Value, json}; pub enum Error { #[error(transparent)] Auth(#[from] litellm_gateway_auth::Error), - #[error("invalid request body: {0}")] - InvalidBody(String), + #[error("invalid request body")] + InvalidBody(#[from] BodyError), + #[error("invalid path parameter")] + InvalidPath(#[source] axum::extract::rejection::PathRejection), #[error( - "/v1/messages: Invalid model name passed in model={0}. Call `/v1/models` to view available models for your key." + "{route}: Invalid model name passed in model={model}. Call `/v1/models` to view available models for your key." )] - UnknownModel(String), + UnknownModel { route: &'static str, model: String }, #[error(transparent)] Route(#[from] InferenceError), #[error(transparent)] @@ -26,8 +27,30 @@ pub enum Error { Unsupported(String), #[error("request body exceeds the size limit")] BodyTooLarge, - #[error("{0}")] - Internal(String), + #[error("unexpected HTTP host operation")] + Internal, +} + +#[derive(Debug, thiserror::Error)] +pub enum BodyError { + #[error("could not read the body")] + Read(#[source] axum::extract::rejection::BytesRejection), + #[error("invalid JSON")] + Json(#[source] serde_json::Error), + #[error("expected a JSON object")] + NotAnObject, + #[error("invalid multipart body")] + Multipart(#[source] axum::extract::multipart::MultipartRejection), + #[error("invalid multipart field")] + MultipartField(#[source] axum::extract::multipart::MultipartError), + #[error("model is required")] + MissingModel, + #[error("invalid cache controls")] + CacheControls(#[source] serde_json::Error), + #[error("invalid caching flag")] + Caching(#[source] serde_json::Error), + #[error("cache durations must be finite and positive")] + CacheDuration(#[source] Option), } impl IntoResponse for Error { @@ -40,7 +63,7 @@ impl From> for Error { fn from(error: litellm_host_http::Error) -> Self { match error { litellm_host_http::Error::Call(error) => Self::Route(error), - litellm_host_http::Error::Protocol => Self::Internal(error.to_string()), + litellm_host_http::Error::Protocol => Self::Internal, } } } @@ -49,58 +72,50 @@ impl Error { pub fn status(&self) -> StatusCode { match self { Self::Auth(error) => error.status(), - Self::Unsupported(_) - | Self::Route(InferenceError::Unsupported(_)) - | Self::Ocr(OcrError::Unsupported(_)) => StatusCode::NOT_IMPLEMENTED, + Self::InvalidPath(rejection) => rejection.status(), + Self::Unsupported(_) => StatusCode::NOT_IMPLEMENTED, Self::BodyTooLarge => StatusCode::PAYLOAD_TOO_LARGE, - Self::Ocr( - OcrError::Auth(litellm_auth::Error::MissingApiKey { .. }) - | OcrError::MissingAzureAiCredentials - | OcrError::MissingAzureDocumentIntelligenceCredentials - | OcrError::MissingReductoApiKey, - ) => StatusCode::UNAUTHORIZED, - Self::Ocr(error) => error - .http_status_code() - .and_then(|status| StatusCode::from_u16(status).ok()) - .unwrap_or(StatusCode::INTERNAL_SERVER_ERROR), - Self::InvalidBody(_) | Self::UnknownModel(_) => StatusCode::BAD_REQUEST, - Self::Route(InferenceError::Transport(TransportError::Http { status, .. })) => { - StatusCode::from_u16(*status).unwrap_or(StatusCode::BAD_GATEWAY) - } - Self::Route(InferenceError::Auth(litellm_auth::Error::MissingApiKey { .. })) => { - StatusCode::UNAUTHORIZED - } - Self::Route(error) if error.is_request() => StatusCode::BAD_REQUEST, - Self::Route(_) | Self::Internal(_) => StatusCode::INTERNAL_SERVER_ERROR, + Self::InvalidBody(_) | Self::UnknownModel { .. } => StatusCode::BAD_REQUEST, + Self::Internal => StatusCode::INTERNAL_SERVER_ERROR, + Self::Ocr(error) => ocr_status(error), + Self::Route(error) => route_status(error), } } pub fn openai_response(self) -> Response { + (self.status(), Json(self.openai_body())).into_response() + } + + /// The OpenAI envelope as an `event: error` frame, for a stream that fails after its + /// headers went out. + pub fn responses_sse_frame(&self) -> String { + let event = json!({ + "type": "error", + "code": self.status().as_u16().to_string(), + "message": self.message(), + "param": null, + }); + format!("event: error\ndata: {event}\n\n") + } + + fn openai_body(&self) -> Value { let status = self.status(); - let message = match &self { - Self::UnknownModel(model) => format!("Invalid model name passed in model={model}"), - _ => self.to_string(), - }; - ( - status, - Json(json!({"error": { - "message": message, - "type": error_type(status), - "param": null, - "code": status.as_u16(), - }})), - ) - .into_response() + json!({"error": { + "message": self.message(), + "type": error_type(status), + "param": null, + "code": status.as_u16(), + }}) } /// The Anthropic error envelope Python's `AnthropicExceptionMapping` builds: an upstream /// body already in that shape passes through, any other has its message extracted. pub fn body(&self, request_id: Option<&str>) -> Value { let raw = match self { - Self::Route(InferenceError::Transport(TransportError::Http { body, .. })) => { - body.clone() + Self::Route(error) if let Some((_, body)) = error.upstream_response() => { + body.to_owned() } - other => other.to_string(), + other => other.message(), }; let parsed = serde_json::from_str::(&raw).ok(); let envelope = match parsed { @@ -117,6 +132,52 @@ impl Error { pub fn sse_frame(&self) -> String { format!("event: error\ndata: {}\n\n", self.body(None)) } + + fn message(&self) -> String { + match self { + Self::Auth(error) => error.to_string(), + Self::InvalidBody(_) + | Self::InvalidPath(_) + | Self::UnknownModel { .. } + | Self::Route(_) + | Self::Ocr(_) + | Self::Unsupported(_) + | Self::BodyTooLarge + | Self::Internal => litellm_error::Report::new(self).to_string(), + } + } +} + +fn route_status(error: &InferenceError) -> StatusCode { + if error.is_unsupported() { + return StatusCode::NOT_IMPLEMENTED; + } + if let Some((status, _)) = error.upstream_response() { + return StatusCode::from_u16(status).unwrap_or(StatusCode::BAD_GATEWAY); + } + if error.is_missing_credential() { + return StatusCode::UNAUTHORIZED; + } + if error.is_request() { + return StatusCode::BAD_REQUEST; + } + StatusCode::INTERNAL_SERVER_ERROR +} + +fn ocr_status(error: &OcrError) -> StatusCode { + if error.is_unsupported() { + return StatusCode::NOT_IMPLEMENTED; + } + if let Some((status, ..)) = error.upstream_response() { + return StatusCode::from_u16(status).unwrap_or(StatusCode::BAD_GATEWAY); + } + if error.is_missing_credential() { + return StatusCode::UNAUTHORIZED; + } + error + .http_status_code() + .and_then(|status| StatusCode::from_u16(status).ok()) + .unwrap_or(StatusCode::INTERNAL_SERVER_ERROR) } fn error_type(status: StatusCode) -> &'static str { @@ -133,13 +194,13 @@ fn error_type(status: StatusCode) -> &'static str { } fn envelope(status: StatusCode, message: &str) -> Map { - let Value::Object(envelope) = json!({ - "type": "error", - "error": {"type": error_type(status), "message": message}, - }) else { - unreachable!("a json object literal is an object") - }; - envelope + Map::from_iter([ + ("type".to_owned(), Value::from("error")), + ( + "error".to_owned(), + json!({"type": error_type(status), "message": message}), + ), + ]) } fn is_anthropic_error(object: &Map) -> bool { @@ -177,14 +238,22 @@ mod tests { use super::*; fn upstream(status: u16, body: &str) -> Error { - Error::Route(InferenceError::Transport(TransportError::Http { - status, - body: body.into(), - })) + Error::Route(litellm_http::Error::upstream(status, body.into()).into()) + } + + fn invalid_json() -> Error { + BodyError::Json(serde_json::from_str::("{").unwrap_err()).into() + } + + fn unknown_model() -> Error { + Error::UnknownModel { + route: "/v1/messages", + model: "nope".into(), + } } #[rstest] - #[case::anthropic_body_passes_through( + #[case::anthropic_body( upstream(529, r#"{"type":"error","error":{"type":"overloaded_error","message":"busy","extra":1}}"#), Some("req_1"), json!({"type": "error", "error": {"type": "overloaded_error", "message": "busy", "extra": 1}, "request_id": "req_1"}), @@ -194,30 +263,7 @@ mod tests { Some("caller"), json!({"type": "error", "error": {"type": "x", "message": "m"}, "request_id": "upstream"}), )] - #[case::bedrock_detail( - upstream(403, r#"{"detail":{"message":"denied"}}"#), - None, - json!({"type": "error", "error": {"type": "permission_error", "message": "denied"}}), - )] - #[case::aws_message( - upstream(429, r#"{"Message":"slow down"}"#), - None, - json!({"type": "error", "error": {"type": "rate_limit_error", "message": "slow down"}}), - )] - #[case::plain_text_with_unmapped_status( - upstream(502, "bad gateway"), - None, - json!({"type": "error", "error": {"type": "api_error", "message": "bad gateway"}}), - )] - #[case::unknown_model( - Error::UnknownModel("nope".into()), - None, - json!({"type": "error", "error": { - "type": "invalid_request_error", - "message": "/v1/messages: Invalid model name passed in model=nope. Call `/v1/models` to view available models for your key.", - }}), - )] - fn body_follows_the_anthropic_exception_mapping( + fn an_anthropic_provider_body_passes_through( #[case] error: Error, #[case] request_id: Option<&str>, #[case] expected: Value, @@ -225,9 +271,68 @@ mod tests { assert_eq!(error.body(request_id), expected); } + #[rstest] + #[case::ascii("busy ".repeat(80))] + #[case::unicode("é".repeat(300))] + fn long_provider_errors_preserve_the_messages_envelope(#[case] message: String) { + let body = json!({ + "type": "error", + "error": {"type": "overloaded_error", "message": message}, + "request_id": "provider-request", + }); + let error = upstream(529, &body.to_string()); + let report = litellm_error::Report::new(&error).to_string(); + let diagnostic_body: String = body.to_string().chars().take(256).collect(); + assert_eq!(report.matches(&diagnostic_body).count(), 1, "{report}"); + assert!(report.ends_with("... (truncated)"), "{report}"); + assert!(!report.contains(&message), "{report}"); + assert!(!format!("{error:?}").contains(&message)); + assert_eq!(error.status(), StatusCode::from_u16(529).unwrap()); + assert_eq!(error.body(Some("gateway-request")), body); + } + + #[rstest] + #[case::bedrock_detail( + upstream(403, r#"{"detail":{"message":"denied"}}"#), + "permission_error", + "denied" + )] + #[case::aws_message( + upstream(429, r#"{"Message":"slow down"}"#), + "rate_limit_error", + "slow down" + )] + #[case::plain_text_with_unmapped_status( + upstream(502, "bad gateway"), + "api_error", + "bad gateway" + )] + #[case::unknown_model( + unknown_model(), + "invalid_request_error", + "/v1/messages: Invalid model name passed in model=nope. Call `/v1/models` to view available models for your key." + )] + fn other_failures_get_the_anthropic_envelope_with_their_message( + #[case] error: Error, + #[case] error_type: &str, + #[case] message: &str, + ) { + let body = error.body(Some("req_1")); + assert_eq!(body["type"], "error"); + assert_eq!(body["error"]["type"], error_type); + assert_eq!(body["error"]["message"], message); + assert_eq!(body["request_id"], "req_1"); + } + #[rstest] #[case::upstream_status(upstream(429, ""), StatusCode::TOO_MANY_REQUESTS)] - #[case::rejected_request(Error::Route(InferenceError::InvalidRequest("top_k".into())), StatusCode::BAD_REQUEST)] + #[case::rejected_request( + Error::Route(InferenceError::invalid_request(litellm_llms::ErrorDetail::Expected { + subject: "top_k", + expected: "an integer", + })), + StatusCode::BAD_REQUEST + )] #[case::missing_key( Error::Route(InferenceError::Auth(litellm_auth::Error::MissingApiKey { provider: "Anthropic", @@ -236,10 +341,83 @@ mod tests { StatusCode::UNAUTHORIZED, )] #[case::lost_connection( - Error::Route(InferenceError::Transport(TransportError::Network("reset".into()))), + Error::Route( + litellm_http::Error::from(tokio_tungstenite::tungstenite::Error::ConnectionClosed) + .into() + ), StatusCode::INTERNAL_SERVER_ERROR, )] + #[case::invalid_body(invalid_json(), StatusCode::BAD_REQUEST)] + #[case::missing_model(BodyError::MissingModel.into(), StatusCode::BAD_REQUEST)] + #[case::unknown_model(unknown_model(), StatusCode::BAD_REQUEST)] + #[case::oversized_body(Error::BodyTooLarge, StatusCode::PAYLOAD_TOO_LARGE)] + #[case::unserved_path(Error::Unsupported("/v1/completions".into()), StatusCode::NOT_IMPLEMENTED)] + #[case::host_protocol( + litellm_host_http::Error::::Protocol.into(), + StatusCode::INTERNAL_SERVER_ERROR + )] + #[case::ocr_provider_answer( + Error::Ocr(OcrError::Provider { status: 429, body: "slow".into(), headers: Default::default() }), + StatusCode::TOO_MANY_REQUESTS + )] + #[case::ocr_missing_credential( + Error::Ocr(OcrError::MissingCredential { credential: "Azure AI credentials", guidance: "set a key" }), + StatusCode::UNAUTHORIZED + )] + #[case::ocr_request_fault(Error::Ocr(OcrError::MissingDocumentUrl), StatusCode::BAD_REQUEST)] + #[case::ocr_response_fault( + Error::Ocr(OcrError::EmptyContent), + StatusCode::INTERNAL_SERVER_ERROR + )] fn status_follows_who_is_at_fault(#[case] error: Error, #[case] status: StatusCode) { assert_eq!(error.status(), status); } + + #[test] + fn a_body_fault_names_the_body_and_keeps_its_cause_once() { + let cause = serde_json::from_str::("{").unwrap_err().to_string(); + let body = invalid_json().openai_body(); + let message = body["error"]["message"].as_str().unwrap(); + assert!( + message.starts_with("invalid request body: invalid JSON: "), + "{message}" + ); + assert_eq!(message.matches(&cause).count(), 1, "{message}"); + assert_eq!(body["error"]["type"], "invalid_request_error"); + assert_eq!(body["error"]["code"], 400); + } + + fn auth_outage() -> Error { + Error::Auth(litellm_gateway_auth::Error::Unavailable( + litellm_error::Source::new(std::io::Error::other("private-backend-detail")), + )) + } + + #[rstest] + #[case::openai(auth_outage().openai_body()["error"]["message"].clone())] + #[case::anthropic(auth_outage().body(None)["error"]["message"].clone())] + #[case::responses_sse(Value::from(auth_outage().responses_sse_frame()))] + #[case::anthropic_sse(Value::from(auth_outage().sse_frame()))] + fn an_auth_failure_hides_its_backend_cause(#[case] rendered: Value) { + let top = auth_outage().to_string(); + let rendered = rendered.to_string(); + assert!(rendered.contains(&top), "{rendered}"); + assert!(!rendered.contains("private-backend-detail"), "{rendered}"); + } + + #[rstest] + #[case::provider_answer(upstream(429, "slow down"))] + #[case::caller_fault(invalid_json())] + #[case::host_bug(litellm_host_http::Error::::Protocol.into())] + fn the_responses_sse_frame_is_the_responses_error_event(#[case] error: Error) { + let frame = error.responses_sse_frame(); + let data = frame + .strip_prefix("event: error\ndata: ") + .and_then(|data| data.strip_suffix("\n\n")) + .unwrap_or_else(|| panic!("an event: error frame, got {frame:?}")); + let event = serde_json::from_str::(data).unwrap(); + assert_eq!(event["type"], "error"); + assert_eq!(event["code"], error.status().as_u16().to_string()); + assert_eq!(event["message"], error.openai_body()["error"]["message"]); + } } diff --git a/litellm-rust/crates/gateway-inference/src/lib.rs b/litellm-rust/crates/gateway-inference/src/lib.rs index 4663151f1de..43966f2bee7 100644 --- a/litellm-rust/crates/gateway-inference/src/lib.rs +++ b/litellm-rust/crates/gateway-inference/src/lib.rs @@ -25,7 +25,7 @@ use litellm_inference_transcription::AudioTranscriptionRoute; use litellm_llms::base_llm::ocr::{handler::OcrClient, settings::OcrSettings}; use litellm_secrets::source::SecretSource; -pub use error::Error; +pub use error::{BodyError, Error}; pub use litellm_router::{Deployment, Router as ModelRouter}; pub use request::{JsonObject, RequestId}; diff --git a/litellm-rust/crates/gateway-inference/src/messages.rs b/litellm-rust/crates/gateway-inference/src/messages.rs index d612a042e03..58ecd40204d 100644 --- a/litellm-rust/crates/gateway-inference/src/messages.rs +++ b/litellm-rust/crates/gateway-inference/src/messages.rs @@ -46,7 +46,7 @@ async fn handle( headers: &HeaderMap, body: Map, ) -> Result { - let deployment = request::resolve_deployment(gateway, &body)?; + let deployment = request::resolve_deployment(gateway, "/v1/messages", &body)?; request::authorize_model(identity, deployment, &body).await?; let (body, cache_options) = crate::caching::prepare(identity, body)?; let route = gateway.messages.clone(); diff --git a/litellm-rust/crates/gateway-inference/src/ocr.rs b/litellm-rust/crates/gateway-inference/src/ocr.rs index 03d064b6bae..f200fd7b141 100644 --- a/litellm-rust/crates/gateway-inference/src/ocr.rs +++ b/litellm-rust/crates/gateway-inference/src/ocr.rs @@ -35,7 +35,7 @@ async fn handle( .get("x-req-format") .and_then(|value| value.to_str().ok()) .map(str::to_owned); - let deployment = request::resolve_deployment(gateway, &body)?; + let deployment = request::resolve_deployment(gateway, "/ocr", &body)?; request::authorize_model(identity, deployment, &body).await?; let document = match upload { Some(upload) => OcrDocumentInput::Bytes { diff --git a/litellm-rust/crates/gateway-inference/src/request.rs b/litellm-rust/crates/gateway-inference/src/request.rs index b398da2013b..d41794242ff 100644 --- a/litellm-rust/crates/gateway-inference/src/request.rs +++ b/litellm-rust/crates/gateway-inference/src/request.rs @@ -6,7 +6,7 @@ use axum::{ }; use serde_json::{Map, Value}; -use crate::{Deployment, Error, Gateway}; +use crate::{BodyError, Deployment, Error, Gateway}; pub(crate) const MAX_FILE_BYTES: usize = 50 * 1024 * 1024; pub(crate) const MAX_BODY_BYTES: usize = MAX_FILE_BYTES + 1024 * 1024; @@ -21,6 +21,19 @@ pub struct JsonObject(pub Map); pub struct RequestId(pub Option); +pub(crate) struct Path(pub T); + +impl FromRequestParts for Path { + type Rejection = Error; + + async fn from_request_parts(parts: &mut Parts, state: &S) -> Result { + axum::extract::Path::::from_request_parts(parts, state) + .await + .map(|axum::extract::Path(value)| Self(value)) + .map_err(Error::InvalidPath) + } +} + impl FromRequestParts for RequestId { type Rejection = std::convert::Infallible; @@ -43,10 +56,10 @@ impl FromRequest for JsonObject { if error.status() == axum::http::StatusCode::PAYLOAD_TOO_LARGE { Error::BodyTooLarge } else { - Error::InvalidBody(error.to_string()) + BodyError::Read(error).into() } })?; - object(&body).map(Self) + Ok(Self(object(&body)?)) } } @@ -77,16 +90,16 @@ impl FromRequest for InferenceBody { } let multipart = Multipart::from_request(request, state) .await - .map_err(|error| Error::InvalidBody(error.to_string()))?; + .map_err(BodyError::Multipart)?; parse_multipart(multipart).await } } -fn object(body: &[u8]) -> Result, Error> { +fn object(body: &[u8]) -> Result, BodyError> { match serde_json::from_slice(body) { Ok(Value::Object(body)) => Ok(body), - Ok(_) => Err(Error::InvalidBody("expected a JSON object".into())), - Err(error) => Err(Error::InvalidBody(error.to_string())), + Ok(_) => Err(BodyError::NotAnObject), + Err(error) => Err(BodyError::Json(error)), } } @@ -98,7 +111,7 @@ pub(crate) async fn authorize_model( let name = body .get("model") .and_then(Value::as_str) - .ok_or_else(|| Error::InvalidBody("model is required".into()))?; + .ok_or(BodyError::MissingModel)?; let access = litellm_gateway_auth::AccessRequest::Model { name: name.into(), deployment: deployment.model.clone(), @@ -112,16 +125,20 @@ pub(crate) async fn authorize_model( pub(crate) fn resolve_deployment<'a>( gateway: &'a Gateway, + route: &'static str, body: &Map, ) -> Result<&'a Deployment, Error> { let model = body .get("model") .and_then(Value::as_str) - .ok_or_else(|| Error::InvalidBody("model is required".into()))?; + .ok_or(BodyError::MissingModel)?; gateway .models .get(model) - .ok_or_else(|| Error::UnknownModel(model.to_owned())) + .ok_or_else(|| Error::UnknownModel { + route, + model: model.to_owned(), + }) } async fn parse_multipart(mut multipart: Multipart) -> Result { @@ -159,7 +176,7 @@ fn multipart_error(error: axum::extract::multipart::MultipartError) -> Error { if error.status() == axum::http::StatusCode::PAYLOAD_TOO_LARGE { return Error::BodyTooLarge; } - Error::InvalidBody(error.to_string()) + BodyError::MultipartField(error).into() } pub(crate) async fn unsupported(OriginalUri(uri): OriginalUri) -> impl IntoResponse { diff --git a/litellm-rust/crates/gateway-inference/src/responses.rs b/litellm-rust/crates/gateway-inference/src/responses.rs index 77bebf610e1..df272460d21 100644 --- a/litellm-rust/crates/gateway-inference/src/responses.rs +++ b/litellm-rust/crates/gateway-inference/src/responses.rs @@ -4,7 +4,6 @@ use axum::{Json, body::Bytes, extract::State, response::Response}; use litellm_gateway_auth::AuthenticatedRequest; use litellm_host_http::Sse; use litellm_inference_responses::{route::Responses, types::ResponsesCall}; -use serde_json::json; use crate::{Error, Gateway, JsonObject, request}; @@ -13,7 +12,7 @@ pub(crate) async fn create( identity: AuthenticatedRequest, JsonObject(body): JsonObject, ) -> Result { - let deployment = request::resolve_deployment(&gateway, &body)?; + let deployment = request::resolve_deployment(&gateway, "/responses", &body)?; request::authorize_model(&identity, deployment, &body).await?; let (body, cache_options) = crate::caching::prepare(&identity, body)?; let route = gateway.responses.clone(); @@ -40,11 +39,7 @@ pub(crate) async fn create( }; let machine = route.machine(call, cache_options.policy); let stream = Sse::::new(Json, |error| { - let error = Error::from(error); - Bytes::from(format!( - "event: error\ndata: {}\n\n", - json!({"type": "error", "code": error.status().as_u16().to_string(), "message": error.to_string(), "param": null}) - )) + Bytes::from(Error::from(error).responses_sse_frame()) }); let headers = crate::caching::CacheHeaders::default(); let response = litellm_host_http::serve(machine, (), headers.clone(), stream, None).await?; diff --git a/litellm-rust/crates/gateway-inference/tests/messages.rs b/litellm-rust/crates/gateway-inference/tests/messages.rs index bf11e67e0fc..7d6d1f160d1 100644 --- a/litellm-rust/crates/gateway-inference/tests/messages.rs +++ b/litellm-rust/crates/gateway-inference/tests/messages.rs @@ -6,7 +6,6 @@ use axum::{ }; use rstest::rstest; use serde_json::json; -use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tower::ServiceExt; use wiremock::{ Mock, MockServer, ResponseTemplate, @@ -90,34 +89,11 @@ async fn invalid_messages_stays_an_anthropic_error() { assert_eq!(body["error"]["type"], "invalid_request_error"); } -/// Answers with the SSE head and one event, then drops the connection short of the -/// announced body length. -async fn truncating_upstream() -> String { - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let base = format!("http://{}", listener.local_addr().unwrap()); - tokio::spawn(async move { - let (mut socket, _) = listener.accept().await.unwrap(); - let mut request = vec![0; 4096]; - let _ = socket.read(&mut request).await; - socket - .write_all( - format!( - "HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\ncontent-length: {}\r\n\r\n{FIRST_EVENT}", - FIRST_EVENT.len() * 2 - ) - .as_bytes(), - ) - .await - .unwrap(); - }); - base -} - const FIRST_EVENT: &str = "event: message_start\ndata: {}\n\n"; #[tokio::test] async fn a_stream_that_fails_after_opening_ends_with_an_sse_error_frame() { - let base = truncating_upstream().await; + let base = support::truncating_upstream(FIRST_EVENT).await; let request = Request::post("/v1/messages") .header("content-type", "application/json") .body(Body::from( diff --git a/litellm-rust/crates/gateway-inference/tests/responses.rs b/litellm-rust/crates/gateway-inference/tests/responses.rs index ded7d73c5f3..ed7c8e6fe76 100644 --- a/litellm-rust/crates/gateway-inference/tests/responses.rs +++ b/litellm-rust/crates/gateway-inference/tests/responses.rs @@ -116,3 +116,32 @@ async fn responses_authorize_the_public_model_before_provider_execution() { assert_eq!(response.status(), 403); assert!(upstream.received_requests().await.unwrap().is_empty()); } + +const RESPONSE_CREATED: &str = "event: response.created\ndata: {\"type\":\"response.created\",\"response\":{\"id\":\"response-1\",\"model\":\"test-model\",\"output\":[]}}\n\n"; + +#[rstest] +#[tokio::test] +async fn a_stream_that_fails_after_opening_ends_with_the_responses_error_event() { + let base = support::truncating_upstream(RESPONSE_CREATED).await; + let response = support::post( + support::app("openai/test-model", &base), + "/v1/responses", + json!({"model": "public/model", "input": "hello", "stream": true}), + ) + .await; + assert_eq!(response.status(), 200); + let body = to_bytes(response.into_body(), 4096).await.unwrap(); + let text = std::str::from_utf8(&body).unwrap(); + let frame = text + .strip_prefix(RESPONSE_CREATED) + .and_then(|rest| rest.strip_prefix("event: error\ndata: ")) + .unwrap_or_else(|| panic!("the delivered event then one error frame, got {text:?}")); + let error: serde_json::Value = serde_json::from_str(frame.trim_end()).unwrap(); + assert_eq!(error["type"], "error"); + assert_eq!(error["code"], "500"); + assert!( + error["message"] + .as_str() + .is_some_and(|message| !message.is_empty()) + ); +} diff --git a/litellm-rust/crates/gateway-inference/tests/routes.rs b/litellm-rust/crates/gateway-inference/tests/routes.rs index b0699ff10b5..24b8c5cc694 100644 --- a/litellm-rust/crates/gateway-inference/tests/routes.rs +++ b/litellm-rust/crates/gateway-inference/tests/routes.rs @@ -125,7 +125,10 @@ async fn chat_errors_come_from_core( .await; assert_eq!(response.status(), status); let body = support::json(response).await; - assert_eq!(body["error"]["message"], error.to_string()); + assert_eq!( + body["error"]["message"], + litellm_error::Report::new(&error).to_string() + ); assert_eq!(body["error"]["code"], status); assert!(upstream.received_requests().await.unwrap().is_empty()); } @@ -193,17 +196,20 @@ async fn transcription_aliases_reach_core_validation(#[case] path: &str) { } #[rstest] -#[case::no_extension("test")] -#[case::unsupported_extension("test.invalid")] +#[case::no_extension("test", json!(null))] +#[case::unsupported_extension("test.invalid", json!("invalid"))] #[tokio::test] -async fn upload_audio_format_validation_matches_core(#[case] filename: &str) { +async fn upload_audio_format_validation_matches_core( + #[case] filename: &str, + #[case] format: Value, +) { let upstream = MockServer::start().await; let app = support::app("bedrock/test-model", &upstream.uri()); let path = "/v1/audio/transcriptions"; let expected = support::post( app.clone(), path, - json!({"model": "public/model", "audio": {"data": "YWJj", "format": null}}), + json!({"model": "public/model", "audio": {"data": "YWJj", "format": format}}), ) .await; let payload = format!( diff --git a/litellm-rust/crates/gateway-inference/tests/support/mod.rs b/litellm-rust/crates/gateway-inference/tests/support/mod.rs index 0e37bb0aff9..737dc0c830c 100644 --- a/litellm-rust/crates/gateway-inference/tests/support/mod.rs +++ b/litellm-rust/crates/gateway-inference/tests/support/mod.rs @@ -15,6 +15,7 @@ use litellm_http::{HttpClientPool, HttpSettings, Resolution, media::PublicDnsRes use litellm_inference::resources::CoreResources; use litellm_secrets::{SecretValue, source::SecretSource}; use serde_json::Value; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tower::ServiceExt; pub struct NoSecrets; @@ -166,3 +167,26 @@ impl litellm_gateway_auth::IdentityResolver for TestPermissions { }) } } + +/// Answers with the SSE head and one event, then drops the connection short of the +/// announced body length. +pub async fn truncating_upstream(first_event: &'static str) -> String { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.unwrap(); + let mut request = vec![0; 4096]; + let _ = socket.read(&mut request).await; + socket + .write_all( + format!( + "HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\ncontent-length: {}\r\n\r\n{first_event}", + first_event.len() * 2 + ) + .as_bytes(), + ) + .await + .unwrap(); + }); + base +} diff --git a/litellm-rust/crates/gateway-management/Cargo.toml b/litellm-rust/crates/gateway-management/Cargo.toml index 53ff2ea59aa..502760f8fbe 100644 --- a/litellm-rust/crates/gateway-management/Cargo.toml +++ b/litellm-rust/crates/gateway-management/Cargo.toml @@ -8,6 +8,7 @@ repository.workspace = true [dependencies] base64.workspace = true litellm-auth-types.workspace = true +litellm-error.workspace = true litellm-gateway-auth.workspace = true rand.workspace = true thiserror.workspace = true diff --git a/litellm-rust/crates/gateway-management/src/error.rs b/litellm-rust/crates/gateway-management/src/error.rs index 5da7977c7cb..7194395d668 100644 --- a/litellm-rust/crates/gateway-management/src/error.rs +++ b/litellm-rust/crates/gateway-management/src/error.rs @@ -5,7 +5,7 @@ pub enum Error { #[error("key already exists")] AlreadyExists, #[error("key storage unavailable")] - Storage(#[source] Box), + Storage(#[source] litellm_error::Source), #[error("key generation unavailable")] Entropy(#[source] rand::Error), } diff --git a/litellm-rust/crates/gateway-management/tests/keys.rs b/litellm-rust/crates/gateway-management/tests/keys.rs index dbfedb7ac4d..bf0221cf1be 100644 --- a/litellm-rust/crates/gateway-management/tests/keys.rs +++ b/litellm-rust/crates/gateway-management/tests/keys.rs @@ -9,7 +9,7 @@ use std::{ use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD}; use litellm_auth_types::SecretValue; use litellm_gateway_auth::{ - KeyError, hash_token, + hash_token, keys::{KeyHash, KeyLookup, KeyStatus, verify_key}, }; use litellm_gateway_management::{ @@ -73,7 +73,7 @@ async fn generated_key_can_be_retrieved_verified_and_revoked( ); assert!(matches!( verify_key(lookup.as_ref(), &generated.token, || now).await, - Err(KeyError::Invalid) + Err(litellm_gateway_auth::Error::InvalidToken) )); assert_eq!(keys.revoke(hash).await.unwrap(), Revocation::Revoked); } @@ -137,9 +137,7 @@ async fn storage_failures_are_returned_for_all_operations(now: SystemTime) { assert!(matches!(keys.get(&hash).await, Err(Error::Storage(_)))); let error = keys.revoke(&hash).await.unwrap_err(); assert!(matches!(&error, Error::Storage(_))); - assert_eq!( - std::error::Error::source(&error).unwrap().to_string(), - "backend detail" - ); + let cause = litellm_error::find::(&error).expect("storage cause"); + assert_eq!(cause.to_string(), "backend detail"); assert!(!error.to_string().contains("backend detail")); } diff --git a/litellm-rust/crates/gateway-management/tests/keys/support.rs b/litellm-rust/crates/gateway-management/tests/keys/support.rs index d0efec4c780..71511c7eaba 100644 --- a/litellm-rust/crates/gateway-management/tests/keys/support.rs +++ b/litellm-rust/crates/gateway-management/tests/keys/support.rs @@ -1,10 +1,7 @@ use std::{collections::BTreeMap, sync::Mutex}; use litellm_gateway_auth::keys::KeyStatus; -use litellm_gateway_auth::{ - KeyError, - keys::{KeyHash, KeyLookup, LookupFuture}, -}; +use litellm_gateway_auth::keys::{KeyHash, KeyLookup, LookupFuture}; use litellm_gateway_management::{ Error, keys::{KeyRecord, KeyStore, NewKey, Revocation, StoreFuture}, @@ -56,7 +53,9 @@ impl KeyLookup for MemoryStore { self.get(hash) .await .map(|record| record.map(|record| record.status)) - .map_err(|error| KeyError::Lookup(Box::new(error))) + .map_err(|error| { + litellm_gateway_auth::Error::Unavailable(litellm_error::Source::new(error)) + }) }) } } @@ -64,9 +63,9 @@ impl KeyLookup for MemoryStore { pub struct Unavailable; fn unavailable() -> Result { - Err(Error::Storage(Box::new(std::io::Error::other( - "backend detail", - )))) + Err(Error::Storage(litellm_error::Source::new( + std::io::Error::other("backend detail"), + ))) } impl KeyStore for Unavailable { diff --git a/litellm-rust/crates/gateway-mcp/AGENTS.md b/litellm-rust/crates/gateway-mcp/AGENTS.md index aa608f1e720..c4b00e3acbf 100644 --- a/litellm-rust/crates/gateway-mcp/AGENTS.md +++ b/litellm-rust/crates/gateway-mcp/AGENTS.md @@ -6,6 +6,8 @@ The HTTP router serves `/mcp`, `/{server}/mcp`, `/mcp/{server}`, `/mcp/{server}/ `NativeGateway` aggregates upstream tools, prompts, resources, and resource templates. Tool and prompt calls resolve registered server prefixes, including prefixes containing hyphens. Resource names receive prefixes while their URIs remain unchanged, and reads require exactly one selected server. Tool allowlists apply to discovery and execution. Upstream pagination is drained with cycle detection. Aggregate tool listings retain healthy servers and expose sanitized outcomes under `_meta["litellm.ai/server_outcomes"]`, while scoped REST listings preserve upstream failures +Every HTTP failure, including extractor rejections and the legacy SSE and session transport checks, is an `Error` with its status from `status()` and the `{"detail"}` envelope. The detail is the `litellm_error::Report` of the error, except for upstream and authorization failures, whose causes can carry upstream URLs and backend detail. Startup failures are `ConnectError`, whose `Debug` redacts the SDK's initialization cause for the same reason + ## Integration Construct `Server` entries from connected SDK peers and pass a `Registry` to `NativeGateway::new`. Keep their `RunningService` handles alive in the host and cancel them during shutdown. SDK peers can use stdio, Streamable HTTP, or another SDK transport. Connection establishment, outbound credentials, HTTP client pools, and reconnection remain owned by the host diff --git a/litellm-rust/crates/gateway-mcp/Cargo.toml b/litellm-rust/crates/gateway-mcp/Cargo.toml index 1db5257448f..bed48bad58d 100644 --- a/litellm-rust/crates/gateway-mcp/Cargo.toml +++ b/litellm-rust/crates/gateway-mcp/Cargo.toml @@ -12,6 +12,7 @@ futures-util.workspace = true litellm-config.workspace = true litellm-secrets.workspace = true litellm-auth-types.workspace = true +litellm-error.workspace = true base64.workspace = true http.workspace = true uuid.workspace = true @@ -25,6 +26,7 @@ serde_json.workspace = true thiserror.workspace = true tokio.workspace = true tokio-util = "0.7" +veil.workspace = true [dev-dependencies] rstest.workspace = true diff --git a/litellm-rust/crates/gateway-mcp/src/configured.rs b/litellm-rust/crates/gateway-mcp/src/configured.rs index bb4f5594f1f..cc7a52fc029 100644 --- a/litellm-rust/crates/gateway-mcp/src/configured.rs +++ b/litellm-rust/crates/gateway-mcp/src/configured.rs @@ -22,25 +22,25 @@ use rmcp::{ use sha2::{Digest, Sha256}; use tokio_util::sync::CancellationToken; -use crate::{ConnectError, Limits, NativeGateway, Registry, Server, ServerInfo}; +use crate::{ConfigFault, ConnectError, Limits, NativeGateway, Registry, Server, ServerInfo}; pub struct ConfiguredGateway { pub operations: Arc, services: Vec>, } -fn invalid(server: &str, message: impl Into) -> ConnectError { +fn invalid(server: &str, fault: ConfigFault) -> ConnectError { ConnectError::Configuration { server: server.into(), - message: message.into(), + fault, } } fn timeout(name: &str, config: &McpServer) -> Result { let duration = Duration::try_from_secs_f64(config.timeout.unwrap_or(30.0)) - .map_err(|_| invalid(name, "timeout must be finite and positive"))?; + .map_err(|source| invalid(name, ConfigFault::InvalidTimeout(source)))?; if duration.is_zero() { - return Err(invalid(name, "timeout must be positive")); + return Err(invalid(name, ConfigFault::ZeroTimeout)); } Ok(duration) } @@ -75,27 +75,24 @@ fn validate(name: &str, config: &McpServer) -> Result<(), ConnectError> { .as_ref() .is_some_and(|alias| alias.trim().is_empty()) { - return Err(invalid(name, "server identifiers must not be blank")); + return Err(invalid(name, ConfigFault::BlankIdentifier)); } if let Some(field) = config.unsupported.keys().next() { return Err(invalid( name, - format!("setting '{field}' is not supported by the Rust MCP host"), + ConfigFault::UnsupportedSetting(field.clone()), )); } if config .max_concurrent_requests .is_some_and(|limit| limit == 0 || limit > tokio::sync::Semaphore::MAX_PERMITS) { - return Err(invalid(name, "max_concurrent_requests is out of range")); + return Err(invalid(name, ConfigFault::ConcurrencyOutOfRange)); } if config.mcp_info.contains_key("protocol_version") || config.mcp_info.contains_key("mcp_server_cost_info") { - return Err(invalid( - name, - "mcp_info protocol and cost settings are not supported", - )); + return Err(invalid(name, ConfigFault::UnsupportedMcpInfo)); } if !matches!( config.auth_type, @@ -108,17 +105,11 @@ fn validate(name: &str, config: &McpServer) -> Result<(), ConnectError> { | McpAuth::Token ) ) { - return Err(invalid( - name, - "configured upstream auth mode is not supported by the Rust MCP host", - )); + return Err(invalid(name, ConfigFault::UnsupportedAuth)); } match config.transport { McpTransport::Sse => { - return Err(invalid( - name, - "legacy SSE upstreams are not supported by the pinned SDK; use transport: http", - )); + return Err(invalid(name, ConfigFault::SseTransport)); } McpTransport::Http => { if config.url.is_none() @@ -126,10 +117,7 @@ fn validate(name: &str, config: &McpServer) -> Result<(), ConnectError> { || !config.args.is_empty() || !config.env.is_empty() { - return Err(invalid( - name, - "HTTP transport requires url and does not accept command, args or env", - )); + return Err(invalid(name, ConfigFault::HttpShape)); } } McpTransport::Stdio => { @@ -139,53 +127,49 @@ fn validate(name: &str, config: &McpServer) -> Result<(), ConnectError> { .is_none_or(|command| command.trim().is_empty()) || config.url.is_some() { - return Err(invalid( - name, - "stdio transport requires command and does not accept url", - )); + return Err(invalid(name, ConfigFault::StdioShape)); } if config.authentication_token.is_some() || !config.static_headers.is_empty() || config.upstream_token_header.is_some() || !matches!(config.auth_type, None | Some(McpAuth::None)) { - return Err(invalid( - name, - "stdio credentials must be supplied through env", - )); + return Err(invalid(name, ConfigFault::StdioCredentials)); } } } if matches!(config.auth_type, None | Some(McpAuth::None)) { if config.authentication_token.is_some() || config.upstream_token_header.is_some() { - return Err(invalid( - name, - "authentication_token and upstream_token_header require an auth_type", - )); + return Err(invalid(name, ConfigFault::TokenWithoutAuth)); } } else if config.authentication_token.is_none() { - return Err(invalid( - name, - "configured auth_type requires authentication_token", - )); + return Err(invalid(name, ConfigFault::AuthWithoutToken)); } Ok(()) } -async fn resolve(value: &SecretValue, secrets: &dyn SecretSource) -> Result { +async fn resolve( + name: &str, + value: &SecretValue, + secrets: &dyn SecretSource, +) -> Result { match value.expose().strip_prefix("os.environ/") { Some(key) => secrets .get_secret_str(key) .await? .map(|secret| secret.expose().to_owned()) - .ok_or_else(|| invalid("configuration", "referenced secret is unavailable")), + .ok_or_else(|| invalid(name, ConfigFault::MissingSecret)), None => Ok(value.expose().to_owned()), } } -fn header(name: &str, value: &str) -> Result<(HeaderName, HeaderValue), ConnectError> { +fn header( + server: &str, + name: &str, + value: &str, +) -> Result<(HeaderName, HeaderValue), ConnectError> { let key = HeaderName::try_from(name) - .map_err(|_| invalid("configuration", "invalid upstream header name"))?; + .map_err(|source| invalid(server, ConfigFault::HeaderName(source)))?; if matches!( key.as_str(), "host" @@ -198,13 +182,10 @@ fn header(name: &str, value: &str) -> Result<(HeaderName, HeaderValue), ConnectE | "mcp-protocol-version" | "last-event-id" ) { - return Err(invalid( - "configuration", - "upstream header conflicts with HTTP framing", - )); + return Err(invalid(server, ConfigFault::FramingHeader)); } let mut value = HeaderValue::try_from(value) - .map_err(|_| invalid("configuration", "invalid upstream header value"))?; + .map_err(|source| invalid(server, ConfigFault::HeaderValue(source)))?; value.set_sensitive(true); Ok((key, value)) } @@ -231,21 +212,19 @@ fn basic(value: &str) -> String { } async fn headers( + name: &str, config: &McpServer, secrets: &dyn SecretSource, ) -> Result, ConnectError> { - let entries = try_join_all( - config - .static_headers - .iter() - .map(|(key, value)| async move { header(key, &resolve(value, secrets).await?) }), - ) + let entries = try_join_all(config.static_headers.iter().map(|(key, value)| async move { + header(name, key, &resolve(name, value, secrets).await?) + })) .await?; let auth = match (&config.authentication_token, config.auth_type) { (Some(token), Some(kind)) => { - let token = resolve(token, secrets).await?; + let token = resolve(name, token, secrets).await?; if token.trim().is_empty() { - return Err(invalid("configuration", "upstream credential is empty")); + return Err(invalid(name, ConfigFault::EmptyCredential)); } let (slot, value) = match kind { McpAuth::ApiKey => ("x-api-key", token), @@ -253,9 +232,10 @@ async fn headers( McpAuth::Token => ("authorization", scheme(&token, "token")), McpAuth::Basic => ("authorization", basic(&token)), McpAuth::Authorization => ("authorization", token), - _ => return Err(invalid("configuration", "unsupported credential mode")), + _ => return Err(invalid(name, ConfigFault::UnsupportedAuth)), }; Some(header( + name, config.upstream_token_header.as_deref().unwrap_or(slot), &value, )?) @@ -349,36 +329,34 @@ async fn connect_service( let configured = config .url .as_ref() - .ok_or_else(|| invalid(name, "HTTP transport requires url"))?; - let url = resolve(configured, secrets).await?; - let parsed = url::Url::parse(&url).map_err(|_| invalid(name, "invalid HTTP URL"))?; + .ok_or_else(|| invalid(name, ConfigFault::HttpShape))?; + let url = resolve(name, configured, secrets).await?; + let parsed = url::Url::parse(&url) + .map_err(|source| invalid(name, ConfigFault::InvalidUrl(source)))?; if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() || !parsed.username().is_empty() || parsed.password().is_some() || parsed.fragment().is_some() { - return Err(invalid( - name, - "HTTP URL must use http or https without user credentials or fragments", - )); + return Err(invalid(name, ConfigFault::UrlShape)); } let transport = StreamableHttpClientTransport::with_client( client, StreamableHttpClientTransportConfig::with_uri(url) - .custom_headers(headers(config, secrets).await?), + .custom_headers(headers(name, config, secrets).await?), ); ().serve_with_ct(transport, shutdown).await } McpTransport::Stdio => { let environment = try_join_all(config.env.iter().map(|(key, value)| async move { - Ok::<_, ConnectError>((key, resolve(value, secrets).await?)) + Ok::<_, ConnectError>((key, resolve(name, value, secrets).await?)) })) .await?; let program = config .command .as_deref() - .ok_or_else(|| invalid(name, "stdio requires command"))?; + .ok_or_else(|| invalid(name, ConfigFault::StdioShape))?; let mut command = tokio::process::Command::new(program); command .args(&config.args) @@ -387,7 +365,7 @@ async fn connect_service( ().serve_with_ct(TokioChildProcess::new(command)?, shutdown) .await } - McpTransport::Sse => return Err(invalid(name, "unsupported transport")), + McpTransport::Sse => return Err(invalid(name, ConfigFault::SseTransport)), }; initialized.map_err(|source| ConnectError::Initialize { server: name.into(), @@ -452,7 +430,7 @@ mod tests { "mcp_servers:\n test:\n auth_type: {mode}\n authentication_token: '{token}'\n" )) .unwrap(); - let built = headers(&config.mcp_servers["test"], &Secrets) + let built = headers("test", &config.mcp_servers["test"], &Secrets) .await .unwrap(); let key = HeaderName::try_from(slot).unwrap(); @@ -464,7 +442,7 @@ mod tests { #[tokio::test] async fn explicit_credential_slot_overrides_static_header() { let config = litellm_config::Config::from_yaml("mcp_servers: {test: {auth_type: api_key, authentication_token: os.environ/KEY, upstream_token_header: x-custom, static_headers: {X-Custom: old, x-extra: extra}}}").unwrap(); - let built = headers(&config.mcp_servers["test"], &Secrets) + let built = headers("test", &config.mcp_servers["test"], &Secrets) .await .unwrap(); assert_eq!( diff --git a/litellm-rust/crates/gateway-mcp/src/error.rs b/litellm-rust/crates/gateway-mcp/src/error.rs index 1aa72e34703..9f08abaf26c 100644 --- a/litellm-rust/crates/gateway-mcp/src/error.rs +++ b/litellm-rust/crates/gateway-mcp/src/error.rs @@ -1,58 +1,231 @@ +use std::time::TryFromFloatSecsError; + use axum::{ Json, + extract::rejection::{JsonRejection, PathRejection, QueryRejection}, http::StatusCode, response::{IntoResponse, Response}, }; +use http::header::{InvalidHeaderName, InvalidHeaderValue, ToStrError}; +use litellm_error::{Report, Source}; use rmcp::{ model::{ErrorCode, ErrorData}, - service::ServiceError, + service::{ClientInitializeError, ServiceError}, }; use serde_json::json; #[derive(Debug, thiserror::Error)] pub enum Error { - #[error("{0}")] - InvalidRequest(String), + #[error("{0} is required in request body")] + MissingField(&'static str), + #[error("Gateway listings do not accept an upstream cursor")] + UpstreamCursor, + #[error("read_resource requires exactly one allowed MCP server")] + AmbiguousResourceServer, + #[error("Invalid x-mcp-servers header")] + InvalidServersHeader(#[source] ToStrError), + #[error("invalid JSON body")] + Json(#[source] JsonRejection), + #[error("invalid query string")] + Query(#[source] QueryRejection), + #[error("invalid path")] + Path(#[source] PathRejection), + #[error("request body is too large")] + BodyTooLarge(#[source] axum::Error), #[error("User not allowed to access this MCP server or operation.")] Forbidden, - #[error("{0}")] - Configuration(String), + #[error("request host is not allowed")] + HostNotAllowed, + #[error("request origin is not allowed")] + OriginNotAllowed, + #[error("MCP session belongs to another caller")] + ForeignSession, + #[error("MCP session not found")] + UnknownSession, + #[error("MCP session is busy")] + SessionBusy, #[error("MCP upstream request failed")] Upstream(#[from] ServiceError), + #[error("MCP upstream returned invalid pagination")] + InvalidPagination, #[error("MCP operation returned an unexpected result")] UnexpectedResult, + #[error("MCP authorization unavailable")] + AuthorizationUnavailable(#[source] Source), #[error("MCP request cancelled")] Cancelled, } impl Error { - pub fn into_mcp(self) -> ErrorData { + pub fn status(&self) -> StatusCode { match self { - Self::Upstream(ServiceError::McpError(error)) => error, - Self::InvalidRequest(message) => ErrorData::invalid_params(message, None), - Self::Forbidden => ErrorData::new(ErrorCode(-32003), self.to_string(), None), - _ => ErrorData::internal_error(self.to_string(), None), + Self::MissingField(_) + | Self::UpstreamCursor + | Self::AmbiguousResourceServer + | Self::InvalidServersHeader(_) => StatusCode::BAD_REQUEST, + Self::Json(rejection) => rejection.status(), + Self::Query(rejection) => rejection.status(), + Self::Path(rejection) => rejection.status(), + Self::BodyTooLarge(_) => StatusCode::PAYLOAD_TOO_LARGE, + Self::Forbidden + | Self::HostNotAllowed + | Self::OriginNotAllowed + | Self::ForeignSession => StatusCode::FORBIDDEN, + Self::UnknownSession => StatusCode::NOT_FOUND, + Self::SessionBusy => StatusCode::TOO_MANY_REQUESTS, + Self::Upstream(_) | Self::InvalidPagination => StatusCode::BAD_GATEWAY, + Self::Cancelled => StatusCode::REQUEST_TIMEOUT, + Self::UnexpectedResult | Self::AuthorizationUnavailable(_) => { + StatusCode::INTERNAL_SERVER_ERROR + } + } + } + + fn is_request(&self) -> bool { + match self { + Self::MissingField(_) + | Self::UpstreamCursor + | Self::AmbiguousResourceServer + | Self::InvalidServersHeader(_) + | Self::Json(_) + | Self::Query(_) + | Self::Path(_) + | Self::BodyTooLarge(_) => true, + Self::Forbidden + | Self::HostNotAllowed + | Self::OriginNotAllowed + | Self::ForeignSession + | Self::UnknownSession + | Self::SessionBusy + | Self::Upstream(_) + | Self::InvalidPagination + | Self::UnexpectedResult + | Self::AuthorizationUnavailable(_) + | Self::Cancelled => false, + } + } + + fn detail(&self) -> String { + match self { + Self::Upstream(_) | Self::AuthorizationUnavailable(_) => self.to_string(), + Self::MissingField(_) + | Self::UpstreamCursor + | Self::AmbiguousResourceServer + | Self::InvalidServersHeader(_) + | Self::Json(_) + | Self::Query(_) + | Self::Path(_) + | Self::BodyTooLarge(_) + | Self::Forbidden + | Self::HostNotAllowed + | Self::OriginNotAllowed + | Self::ForeignSession + | Self::UnknownSession + | Self::SessionBusy + | Self::InvalidPagination + | Self::UnexpectedResult + | Self::Cancelled => Report::new(self).to_string(), + } + } + + fn upstream(&self) -> Option<&ServiceError> { + match self { + Self::Upstream(error) => Some(error), + Self::MissingField(_) + | Self::UpstreamCursor + | Self::AmbiguousResourceServer + | Self::InvalidServersHeader(_) + | Self::Json(_) + | Self::Query(_) + | Self::Path(_) + | Self::BodyTooLarge(_) + | Self::Forbidden + | Self::HostNotAllowed + | Self::OriginNotAllowed + | Self::ForeignSession + | Self::UnknownSession + | Self::SessionBusy + | Self::InvalidPagination + | Self::UnexpectedResult + | Self::AuthorizationUnavailable(_) + | Self::Cancelled => None, + } + } + + pub(crate) fn is_upstream(&self) -> bool { + self.upstream().is_some() + } + + pub(crate) fn is_timeout(&self) -> bool { + self.upstream().is_some_and(|error| match error { + ServiceError::Timeout { .. } => true, + ServiceError::McpError(_) + | ServiceError::TransportSend(_) + | ServiceError::TransportClosed + | ServiceError::UnexpectedResponse + | ServiceError::SubscriptionLagged { .. } + | ServiceError::Cancelled { .. } + | ServiceError::InputRequiredRoundsExceeded { .. } => false, + _ => false, + }) + } + + pub(crate) fn is_unreachable(&self) -> bool { + self.upstream().is_some_and(|error| match error { + ServiceError::TransportSend(_) | ServiceError::TransportClosed => true, + ServiceError::McpError(_) + | ServiceError::Timeout { .. } + | ServiceError::UnexpectedResponse + | ServiceError::SubscriptionLagged { .. } + | ServiceError::Cancelled { .. } + | ServiceError::InputRequiredRoundsExceeded { .. } => false, + _ => false, + }) + } + + pub(crate) fn mcp_error(&self) -> Option<&ErrorData> { + self.upstream().and_then(|error| match error { + ServiceError::McpError(error) => Some(error), + ServiceError::TransportSend(_) + | ServiceError::TransportClosed + | ServiceError::Timeout { .. } + | ServiceError::UnexpectedResponse + | ServiceError::SubscriptionLagged { .. } + | ServiceError::Cancelled { .. } + | ServiceError::InputRequiredRoundsExceeded { .. } => None, + _ => None, + }) + } + + pub fn into_mcp(self) -> ErrorData { + if let Some(error) = self.mcp_error() { + return error.clone(); + } + let message = self.detail(); + if self.status() == StatusCode::FORBIDDEN { + ErrorData::new(ErrorCode(-32003), message, None) + } else if self.is_request() { + ErrorData::invalid_params(message, None) + } else { + ErrorData::internal_error(message, None) } } } impl IntoResponse for Error { fn into_response(self) -> Response { - let status = match &self { - Self::InvalidRequest(_) => StatusCode::BAD_REQUEST, - Self::Forbidden => StatusCode::FORBIDDEN, - Self::Upstream(_) => StatusCode::BAD_GATEWAY, - Self::Cancelled => StatusCode::REQUEST_TIMEOUT, - Self::Configuration(_) | Self::UnexpectedResult => StatusCode::INTERNAL_SERVER_ERROR, - }; - (status, Json(json!({"detail": self.to_string()}))).into_response() + (self.status(), Json(json!({"detail": self.detail()}))).into_response() } } -#[derive(thiserror::Error)] +#[derive(thiserror::Error, veil::Redact)] pub enum ConnectError { - #[error("MCP server {server}: {message}")] - Configuration { server: String, message: String }, + #[error("MCP server {server} is misconfigured")] + Configuration { + server: String, + #[source] + fault: ConfigFault, + }, #[error("could not resolve MCP credentials")] Secret(#[from] litellm_secrets::Error), #[error("could not start MCP child process")] @@ -61,38 +234,174 @@ pub enum ConnectError { Initialize { server: String, #[source] - source: Box, + #[redact] + source: Box, }, #[error("MCP upstream {0} initialization timed out")] Timeout(String), #[error("MCP upstream startup cancelled")] Cancelled, - #[error(transparent)] - Registry(#[from] Error), + #[error("MCP server identifiers must not be empty")] + EmptyIdentifier, + #[error("MCP server identifiers and prefixes must be unique")] + DuplicateIdentifier, } -impl std::fmt::Debug for ConnectError { - fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - std::fmt::Display::fmt(self, formatter) - } +#[derive(Debug, thiserror::Error)] +pub enum ConfigFault { + #[error("timeout must be finite and positive")] + InvalidTimeout(#[source] TryFromFloatSecsError), + #[error("timeout must be positive")] + ZeroTimeout, + #[error("server identifiers must not be blank")] + BlankIdentifier, + #[error("setting '{0}' is not supported by the Rust MCP host")] + UnsupportedSetting(String), + #[error("max_concurrent_requests is out of range")] + ConcurrencyOutOfRange, + #[error("mcp_info protocol and cost settings are not supported")] + UnsupportedMcpInfo, + #[error("configured upstream auth mode is not supported by the Rust MCP host")] + UnsupportedAuth, + #[error("legacy SSE upstreams are not supported by the pinned SDK; use transport: http")] + SseTransport, + #[error("HTTP transport requires url and does not accept command, args or env")] + HttpShape, + #[error("stdio transport requires command and does not accept url")] + StdioShape, + #[error("stdio credentials must be supplied through env")] + StdioCredentials, + #[error("authentication_token and upstream_token_header require an auth_type")] + TokenWithoutAuth, + #[error("configured auth_type requires authentication_token")] + AuthWithoutToken, + #[error("referenced secret is unavailable")] + MissingSecret, + #[error("upstream credential is empty")] + EmptyCredential, + #[error("invalid upstream header name")] + HeaderName(#[source] InvalidHeaderName), + #[error("invalid upstream header value")] + HeaderValue(#[source] InvalidHeaderValue), + #[error("upstream header conflicts with HTTP framing")] + FramingHeader, + #[error("invalid HTTP URL")] + InvalidUrl(#[source] url::ParseError), + #[error("HTTP URL must use http or https without user credentials or fragments")] + UrlShape, } #[cfg(test)] mod tests { + use rstest::rstest; + use super::*; - #[rstest::rstest] + const SECRET_URL: &str = "https://example.test/mcp?token=private-secret"; + + #[rstest] fn startup_diagnostics_do_not_expose_upstream_details() { let error = ConnectError::Initialize { server: "docs".into(), - source: Box::new(rmcp::service::ClientInitializeError::ConnectionClosed( - "https://example.test/mcp?token=private-secret".into(), - )), + source: Box::new(ClientInitializeError::ConnectionClosed(SECRET_URL.into())), }; - assert_eq!( - format!("{error:?}"), - "could not initialize MCP upstream docs" - ); + let debug = format!("{error:?}"); + assert!(debug.contains("docs")); + assert!(!debug.contains("private-secret")); assert!(std::error::Error::source(&error).is_some()); } + + #[rstest] + fn configuration_faults_reach_the_startup_diagnostic() { + let error = ConnectError::Configuration { + server: "docs".into(), + fault: ConfigFault::ZeroTimeout, + }; + assert!(format!("{error:?}").contains("ZeroTimeout")); + let report = Report::new(&error).to_string(); + assert!(report.contains("docs"), "{report}"); + assert_eq!( + report.matches("timeout must be positive").count(), + 1, + "{report}" + ); + } + + #[rstest] + #[case::missing_field(Error::MissingField("server_id"), StatusCode::BAD_REQUEST)] + #[case::upstream_cursor(Error::UpstreamCursor, StatusCode::BAD_REQUEST)] + #[case::forbidden(Error::Forbidden, StatusCode::FORBIDDEN)] + #[case::foreign_session(Error::ForeignSession, StatusCode::FORBIDDEN)] + #[case::unknown_session(Error::UnknownSession, StatusCode::NOT_FOUND)] + #[case::busy_session(Error::SessionBusy, StatusCode::TOO_MANY_REQUESTS)] + #[case::invalid_pagination(Error::InvalidPagination, StatusCode::BAD_GATEWAY)] + #[case::upstream( + Error::Upstream(ServiceError::TransportClosed), + StatusCode::BAD_GATEWAY + )] + #[case::cancelled(Error::Cancelled, StatusCode::REQUEST_TIMEOUT)] + fn status_follows_who_is_at_fault(#[case] error: Error, #[case] status: StatusCode) { + assert_eq!(error.status(), status); + } + + fn mcp_failure() -> Error { + Error::Upstream(ServiceError::McpError(ErrorData::invalid_params( + "bad", None, + ))) + } + + #[rstest] + #[case::timeout( + Error::Upstream(ServiceError::Timeout { timeout: std::time::Duration::from_secs(1) }), + true, + false, + false + )] + #[case::closed(Error::Upstream(ServiceError::TransportClosed), false, true, false)] + #[case::mcp_error(mcp_failure(), false, false, true)] + #[case::unexpected_response( + Error::Upstream(ServiceError::UnexpectedResponse), + false, + false, + false + )] + #[case::gateway_cancelled(Error::Cancelled, false, false, false)] + fn upstream_failures_answer_what_went_wrong( + #[case] error: Error, + #[case] timeout: bool, + #[case] unreachable: bool, + #[case] mcp_error: bool, + ) { + assert_eq!(error.is_timeout(), timeout); + assert_eq!(error.is_unreachable(), unreachable); + assert_eq!(error.mcp_error().is_some(), mcp_error); + } + + #[rstest] + fn an_upstream_mcp_error_passes_through_unchanged() { + let mapped = mcp_failure().into_mcp(); + assert_eq!(mapped.code, ErrorCode::INVALID_PARAMS); + assert_eq!(mapped.message, "bad"); + } + + #[rstest] + #[case::request(Error::MissingField("name"), ErrorCode::INVALID_PARAMS)] + #[case::forbidden(Error::Forbidden, ErrorCode(-32003))] + #[case::upstream_fault(Error::InvalidPagination, ErrorCode::INTERNAL_ERROR)] + fn mcp_error_code_follows_the_fault(#[case] error: Error, #[case] code: ErrorCode) { + assert_eq!(error.into_mcp().code, code); + } + + #[rstest] + #[case::upstream(Error::Upstream(ServiceError::Cancelled { + reason: Some(SECRET_URL.into()) + }))] + #[case::authorization(Error::AuthorizationUnavailable(Source::new(std::io::Error::other( + SECRET_URL + ))))] + fn private_causes_stay_out_of_client_messages(#[case] error: Error) { + assert!(Report::new(&error).to_string().contains("private-secret")); + assert!(!error.detail().contains("private-secret")); + assert!(!error.into_mcp().message.contains("private-secret")); + } } diff --git a/litellm-rust/crates/gateway-mcp/src/legacy.rs b/litellm-rust/crates/gateway-mcp/src/legacy.rs index 429fd933135..1673e1b7b19 100644 --- a/litellm-rust/crates/gateway-mcp/src/legacy.rs +++ b/litellm-rust/crates/gateway-mcp/src/legacy.rs @@ -1,8 +1,8 @@ use std::{sync::Arc, time::Duration}; use axum::{ - Json, Router, - extract::{Query, Request, State}, + Router, + extract::{FromRequest, Request, State}, http::{StatusCode, request::Parts}, response::{ IntoResponse, Response, Sse, @@ -18,7 +18,11 @@ use tokio::sync::mpsc; use tokio_util::sync::CancellationToken; use uuid::Uuid; -use crate::{Context, HttpConfig, McpServer, Operations, sessions::fingerprint}; +use crate::{ + Context, Error, HttpConfig, McpServer, Operations, + request::{Json, Query}, + sessions::fingerprint, +}; #[derive(Clone)] struct Session { @@ -63,13 +67,13 @@ pub(crate) fn router( } impl Legacy { - fn validate(&self, parts: &Parts) -> Result<(), StatusCode> { + fn validate(&self, parts: &Parts) -> Result<(), Error> { let host = parts .headers .get("host") .and_then(|value| value.to_str().ok()) .and_then(|value| value.parse::().ok()) - .ok_or(StatusCode::FORBIDDEN)?; + .ok_or(Error::HostNotAllowed)?; let allowed = self.hosts.iter().any(|allowed| { host.as_str().eq_ignore_ascii_case(allowed) || host @@ -78,17 +82,16 @@ impl Legacy { .eq_ignore_ascii_case(allowed) }); if !allowed { - return Err(StatusCode::FORBIDDEN); + return Err(Error::HostNotAllowed); } - if let Some(origin) = parts.headers.get("origin") { - let origin = origin.to_str().map_err(|_| StatusCode::FORBIDDEN)?; - if !self - .origins - .iter() - .any(|allowed| same_origin(origin, allowed)) - { - return Err(StatusCode::FORBIDDEN); - } + if let Some(origin) = parts.headers.get("origin") + && !origin.to_str().is_ok_and(|origin| { + self.origins + .iter() + .any(|allowed| same_origin(origin, allowed)) + }) + { + return Err(Error::OriginNotAllowed); } Ok(()) } @@ -111,27 +114,22 @@ fn same_origin(value: &str, allowed: &str) -> bool { } } -async fn connect(State(state): State, request: Request) -> Response { +async fn connect(State(state): State, request: Request) -> Result { let owner = fingerprint(&request); let (parts, _) = request.into_parts(); - if let Err(status) = state.validate(&parts) { - return status.into_response(); - } - if let Err(error) = state + state.validate(&parts)?; + state .operations .authorize(Context { parts, server: None, mcp: None, }) - .await - { - return error.into_response(); - } + .await?; let connection = start_session(state, owner).await; - Sse::new(connection.events()) + Ok(Sse::new(connection.events()) .keep_alive(KeepAlive::default()) - .into_response() + .into_response()) } struct Connection { @@ -216,29 +214,21 @@ async fn message( State(state): State, Query(query): Query, request: Request, -) -> Response { +) -> Result { let owner = fingerprint(&request); let (parts, body) = request.into_parts(); - if let Err(status) = state.validate(&parts) { - return status.into_response(); - } - let Some(session) = state.sessions.get(&query.session_id).await else { - return StatusCode::NOT_FOUND.into_response(); - }; - if session.owner != owner { - return StatusCode::FORBIDDEN.into_response(); - } - let request = Request::from_parts(parts.clone(), body); - let Json(mut message) = - match as axum::extract::FromRequest<()>>::from_request( - request, - &(), - ) + state.validate(&parts)?; + let session = state + .sessions + .get(&query.session_id) .await - { - Ok(message) => message, - Err(error) => return error.into_response(), - }; + .ok_or(Error::UnknownSession)?; + if session.owner != owner { + return Err(Error::ForeignSession); + } + let Json(mut message) = + Json::::from_request(Request::from_parts(parts.clone(), body), &()) + .await?; match &mut message { ClientJsonRpcMessage::Request(request) => { request.request.extensions_mut().insert(parts); @@ -249,8 +239,8 @@ async fn message( _ => (), } match session.sender.try_send(message) { - Ok(()) => (StatusCode::ACCEPTED, "Accepted").into_response(), - Err(mpsc::error::TrySendError::Full(_)) => StatusCode::TOO_MANY_REQUESTS.into_response(), - Err(mpsc::error::TrySendError::Closed(_)) => StatusCode::NOT_FOUND.into_response(), + Ok(()) => Ok((StatusCode::ACCEPTED, "Accepted").into_response()), + Err(mpsc::error::TrySendError::Full(_)) => Err(Error::SessionBusy), + Err(mpsc::error::TrySendError::Closed(_)) => Err(Error::UnknownSession), } } diff --git a/litellm-rust/crates/gateway-mcp/src/lib.rs b/litellm-rust/crates/gateway-mcp/src/lib.rs index dd72940e24b..fcb7fd98383 100644 --- a/litellm-rust/crates/gateway-mcp/src/lib.rs +++ b/litellm-rust/crates/gateway-mcp/src/lib.rs @@ -3,6 +3,7 @@ mod error; mod legacy; mod native; mod relay; +mod request; mod routes; mod server; mod sessions; @@ -13,7 +14,7 @@ use http::request::Parts; use rmcp::{RoleServer, model::*, service::RequestContext}; pub use configured::ConfiguredGateway; -pub use error::{ConnectError, Error}; +pub use error::{ConfigFault, ConnectError, Error}; pub use native::{Limits, NativeGateway, Registry, Server, ServerResolver}; pub use relay::RelayClient; pub use rmcp; diff --git a/litellm-rust/crates/gateway-mcp/src/native.rs b/litellm-rust/crates/gateway-mcp/src/native.rs index 3a60ce5aca7..7a83f1316e9 100644 --- a/litellm-rust/crates/gateway-mcp/src/native.rs +++ b/litellm-rust/crates/gateway-mcp/src/native.rs @@ -178,14 +178,15 @@ impl NativeGateway { .ok_or(Error::Forbidden)?; let mut upstream = request; upstream.name = tool.tool.name.clone(); - match self + let result = self .send(server, CallToolRequest::new(upstream).into(), &context) - .await - { - Err(Error::Upstream(rmcp::service::ServiceError::McpError(error))) => { - Ok(CallToolResult::error(vec![ContentBlock::text(error.message)]).into()) - } - result => result, + .await; + match result.as_ref().err().and_then(Error::mcp_error) { + Some(error) => Ok(CallToolResult::error(vec![ContentBlock::text( + error.message.clone(), + )]) + .into()), + None => result, } } Operation::ListPrompts(params) => { @@ -321,9 +322,7 @@ impl NativeGateway { return Err(Error::Forbidden); } let [server] = servers.as_ref() else { - return Err(Error::InvalidRequest( - "read_resource requires exactly one allowed MCP server".into(), - )); + return Err(Error::AmbiguousResourceServer); }; self.send(server, ReadResourceRequest::new(request).into(), &context) .await @@ -363,9 +362,7 @@ impl Operations for NativeGateway { fn reject_cursor(params: &Option) -> Result<(), Error> { match params.as_ref().and_then(|params| params.cursor.as_ref()) { - Some(_) => Err(Error::InvalidRequest( - "Gateway listings do not accept an upstream cursor".into(), - )), + Some(_) => Err(Error::UpstreamCursor), None => Ok(()), } } diff --git a/litellm-rust/crates/gateway-mcp/src/native/catalog.rs b/litellm-rust/crates/gateway-mcp/src/native/catalog.rs index 448729ec686..6dd16d1fe63 100644 --- a/litellm-rust/crates/gateway-mcp/src/native/catalog.rs +++ b/litellm-rust/crates/gateway-mcp/src/native/catalog.rs @@ -28,9 +28,7 @@ impl NativeGateway { let state = match next { None => None, Some(cursor) if seen.contains(&cursor) || seen.len() >= 1000 => { - return Err(Error::InvalidRequest( - "MCP upstream returned invalid pagination".into(), - )); + return Err(Error::InvalidPagination); } Some(cursor) => Some(( Some(cursor.clone()), @@ -95,14 +93,9 @@ impl NativeGateway { Ok(tools) => ServerOutcome::Ok { tool_count: tools.len(), }, - Err(Error::Upstream(rmcp::service::ServiceError::Timeout { .. })) => { - ServerOutcome::Timeout - } - Err(Error::Upstream( - rmcp::service::ServiceError::TransportClosed - | rmcp::service::ServiceError::TransportSend(_), - )) => ServerOutcome::Unreachable, - Err(Error::Upstream(_)) => ServerOutcome::UpstreamError, + Err(error) if error.is_timeout() => ServerOutcome::Timeout, + Err(error) if error.is_unreachable() => ServerOutcome::Unreachable, + Err(error) if error.is_upstream() => ServerOutcome::UpstreamError, Err(_) => ServerOutcome::Internal, }; (server.prefix(), outcome) diff --git a/litellm-rust/crates/gateway-mcp/src/native/registry.rs b/litellm-rust/crates/gateway-mcp/src/native/registry.rs index 0aa051d6f32..231737e9772 100644 --- a/litellm-rust/crates/gateway-mcp/src/native/registry.rs +++ b/litellm-rust/crates/gateway-mcp/src/native/registry.rs @@ -1,5 +1,5 @@ use super::NativeGateway; -use crate::{Context, Error, GatewayFuture, ServerInfo}; +use crate::{ConnectError, Context, Error, GatewayFuture, ServerInfo}; use rmcp::{Peer, RoleClient}; use std::sync::Arc; @@ -39,14 +39,12 @@ pub trait ServerResolver: Send + Sync + 'static { pub struct Registry(Arc<[Server]>); impl Registry { - pub fn new(servers: impl Into>) -> Result { + pub fn new(servers: impl Into>) -> Result { let servers = servers.into(); for (index, server) in servers.iter().enumerate() { let identifiers = [&server.info.server_id, &server.info.server_name]; if identifiers.iter().any(|value| value.is_empty()) || server.prefix().is_empty() { - return Err(Error::Configuration( - "MCP server identifiers must not be empty".into(), - )); + return Err(ConnectError::EmptyIdentifier); } if servers[..index].iter().any(|previous| { previous.prefix() == server.prefix() @@ -57,9 +55,7 @@ impl Registry { .as_deref() .is_some_and(|alias| previous.matches(alias)) }) { - return Err(Error::Configuration( - "MCP server identifiers and prefixes must be unique".into(), - )); + return Err(ConnectError::DuplicateIdentifier); } } Ok(Self(servers)) @@ -79,11 +75,7 @@ impl NativeGateway { .parts .headers .get("x-mcp-servers") - .map(|value| { - value - .to_str() - .map_err(|_| Error::InvalidRequest("Invalid x-mcp-servers header".into())) - }) + .map(|value| value.to_str().map_err(Error::InvalidServersHeader)) .transpose()?; let scope = context.server.as_deref().or(header); let Some(scope) = scope else { diff --git a/litellm-rust/crates/gateway-mcp/src/request.rs b/litellm-rust/crates/gateway-mcp/src/request.rs new file mode 100644 index 00000000000..eaa9a6f1840 --- /dev/null +++ b/litellm-rust/crates/gateway-mcp/src/request.rs @@ -0,0 +1,44 @@ +use axum::extract::{FromRequest, FromRequestParts, Request}; +use http::request::Parts; +use serde::de::DeserializeOwned; + +use crate::Error; + +pub(crate) struct Json(pub T); + +impl FromRequest for Json { + type Rejection = Error; + + async fn from_request(request: Request, state: &S) -> Result { + axum::Json::::from_request(request, state) + .await + .map(|axum::Json(value)| Self(value)) + .map_err(Error::Json) + } +} + +pub(crate) struct Query(pub T); + +impl FromRequestParts for Query { + type Rejection = Error; + + async fn from_request_parts(parts: &mut Parts, state: &S) -> Result { + axum::extract::Query::::from_request_parts(parts, state) + .await + .map(|axum::extract::Query(value)| Self(value)) + .map_err(Error::Query) + } +} + +pub(crate) struct Path(pub T); + +impl FromRequestParts for Path { + type Rejection = Error; + + async fn from_request_parts(parts: &mut Parts, state: &S) -> Result { + axum::extract::Path::::from_request_parts(parts, state) + .await + .map(|axum::extract::Path(value)| Self(value)) + .map_err(Error::Path) + } +} diff --git a/litellm-rust/crates/gateway-mcp/src/routes.rs b/litellm-rust/crates/gateway-mcp/src/routes.rs index b5d314e4882..56d67a5bc77 100644 --- a/litellm-rust/crates/gateway-mcp/src/routes.rs +++ b/litellm-rust/crates/gateway-mcp/src/routes.rs @@ -2,7 +2,7 @@ use std::sync::Arc; use axum::{ Json, Router, - extract::{Path, Query, Request, State}, + extract::{Request, State}, http::request::Parts, middleware::{self, Next}, response::{IntoResponse, Response}, @@ -12,7 +12,11 @@ use rmcp::{model::*, transport::streamable_http_server::StreamableHttpServerConf use serde::{Deserialize, Serialize}; use tokio_util::sync::CancellationToken; -use crate::{Context, Error, McpServer, Operation, Operations, ToolCatalog, server::ServerScope}; +use crate::{ + Context, Error, McpServer, Operation, Operations, ToolCatalog, + request::{self, Path, Query}, + server::ServerScope, +}; pub struct HttpConfig { pub allowed_hosts: Vec, @@ -132,16 +136,16 @@ struct ToolCall { async fn rest_call( State(operations): State>, parts: Parts, - Json(call): Json, + request::Json(call): request::Json, ) -> Result { let server = call .server_id .filter(|id| !id.is_empty()) - .ok_or_else(|| Error::InvalidRequest("server_id is required in request body".into()))?; + .ok_or(Error::MissingField("server_id"))?; let name = call .name .filter(|name| !name.is_empty()) - .ok_or_else(|| Error::InvalidRequest("name is required in request body".into()))?; + .ok_or(Error::MissingField("name"))?; let params = CallToolRequestParams::new(name).with_arguments(call.arguments.unwrap_or_default()); let context = Context { diff --git a/litellm-rust/crates/gateway-mcp/src/sessions.rs b/litellm-rust/crates/gateway-mcp/src/sessions.rs index 54d0634bd3d..583a8b86341 100644 --- a/litellm-rust/crates/gateway-mcp/src/sessions.rs +++ b/litellm-rust/crates/gateway-mcp/src/sessions.rs @@ -4,7 +4,7 @@ use axum::{ Router, body::{Body, to_bytes}, extract::{Request, State}, - http::{Method, StatusCode}, + http::Method, response::IntoResponse, routing::any, }; @@ -19,7 +19,7 @@ use rmcp::{ use sha2::{Digest, Sha256}; use tower::ServiceExt; -use crate::{McpServer, server::ServerScope}; +use crate::{Error, McpServer, server::ServerScope}; #[derive(Clone)] pub struct SessionOwner(pub String); @@ -65,7 +65,7 @@ pub(crate) fn transport(server: McpServer, config: StreamableHttpServerConfig) - async fn dispatch( State(transport): State, request: Request, -) -> Result { +) -> Result { let owner = fingerprint(&request); let session = request .headers() @@ -75,8 +75,8 @@ async fn dispatch( if let Some(session) = &session { match transport.owners.get(session).await { Some(expected) if expected == owner => (), - Some(_) => return Err(StatusCode::FORBIDDEN), - None => return Err(StatusCode::NOT_FOUND), + Some(_) => return Err(Error::ForeignSession), + None => return Err(Error::UnknownSession), } } let method = request.method().clone(); @@ -84,7 +84,7 @@ async fn dispatch( let (parts, body) = request.into_parts(); let bytes = to_bytes(body, 4 * 1024 * 1024) .await - .map_err(|_| StatusCode::PAYLOAD_TOO_LARGE)?; + .map_err(Error::BodyTooLarge)?; let initialize = matches!(serde_json::from_slice::(&bytes), Ok(ClientJsonRpcMessage::Request(request)) if matches!(request.request, ClientRequest::InitializeRequest(_))); (Request::from_parts(parts, Body::from(bytes)), initialize) diff --git a/litellm-rust/crates/gateway-mcp/tests/gateway/http.rs b/litellm-rust/crates/gateway-mcp/tests/gateway/http.rs index 5feab0194d3..c5060ebe8f3 100644 --- a/litellm-rust/crates/gateway-mcp/tests/gateway/http.rs +++ b/litellm-rust/crates/gateway-mcp/tests/gateway/http.rs @@ -171,43 +171,44 @@ async fn resolver_receives_each_requests_identity( Duration::from_secs(2), )); let app = router(operations, HttpConfig::default()); - for token in ["allowed", "denied", "allowed"] { - let response = app - .clone() - .oneshot( - Request::builder() - .method(method) - .uri(path) - .header("host", "localhost") - .header("authorization", format!("Bearer {token}")) - .header("content-type", "application/json") - .header("accept", "application/json, text/event-stream") - .header("mcp-protocol-version", "2025-11-25") - .body(Body::from( - json!({"jsonrpc":"2.0","id":1,"method":"tools/list"}).to_string(), - )) - .unwrap(), - ) - .await - .unwrap(); - let status = response.status(); - let body = String::from_utf8( - to_bytes(response.into_body(), 65536) - .await - .unwrap() - .to_vec(), + let (status, body) = list_as(&app, method, path, "allowed").await; + assert_eq!(status, StatusCode::OK); + assert!(body.contains("echo-value"), "{body}"); + let (status, body) = list_as(&app, method, path, "denied").await; + assert_eq!(status == StatusCode::FORBIDDEN, method == "GET"); + assert!(body.contains("not allowed"), "{body}"); + let (status, body) = list_as(&app, method, path, "allowed").await; + assert_eq!(status, StatusCode::OK); + assert!(body.contains("echo-value"), "{body}"); +} + +async fn list_as( + app: &axum::Router, + method: &str, + path: &str, + token: &str, +) -> (StatusCode, String) { + let response = app + .clone() + .oneshot( + Request::builder() + .method(method) + .uri(path) + .header("host", "localhost") + .header("authorization", format!("Bearer {token}")) + .header("content-type", "application/json") + .header("accept", "application/json, text/event-stream") + .header("mcp-protocol-version", "2025-11-25") + .body(Body::from( + json!({"jsonrpc":"2.0","id":1,"method":"tools/list"}).to_string(), + )) + .unwrap(), ) + .await .unwrap(); - if token == "denied" { - if method == "GET" { - assert_eq!(status, StatusCode::FORBIDDEN); - } - assert!(body.contains("not allowed"), "{body}"); - } else { - assert_eq!(status, StatusCode::OK); - assert!(body.contains("echo-value"), "{body}"); - } - } + let status = response.status(); + let body = to_bytes(response.into_body(), 65536).await.unwrap(); + (status, String::from_utf8(body.to_vec()).unwrap()) } fn rpc_request(path: &str, token: &str, session: Option<&str>, body: Value) -> Request { @@ -352,10 +353,14 @@ async fn header_scope_cannot_broaden_path_scope( } #[rstest] -#[case::missing_server(json!({"name":"echo-value"}))] -#[case::missing_name(json!({"server_id":"alpha"}))] +#[case::missing_server(json!({"name":"echo-value"}), "server_id")] +#[case::missing_name(json!({"server_id":"alpha"}), "name")] #[tokio::test] -async fn rest_requires_server_and_tool(#[future] harness: Harness, #[case] body: Value) { +async fn rest_requires_server_and_tool( + #[future] harness: Harness, + #[case] body: Value, + #[case] field: &str, +) { let harness = harness.await; let response = harness .app @@ -368,6 +373,40 @@ async fn rest_requires_server_and_tool(#[future] harness: Harness, #[case] body: .await .unwrap(); assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body: Value = + serde_json::from_slice(&to_bytes(response.into_body(), 65536).await.unwrap()).unwrap(); + assert!(body["detail"].as_str().unwrap().contains(field), "{body}"); +} + +#[rstest] +#[case::malformed_call("/mcp-rest/tools/call", "POST", "{")] +#[case::unknown_list_parameter("/mcp-rest/tools/list?unknown=1", "GET", "")] +#[case::legacy_message_without_session("/mcp/sse/messages", "POST", "{}")] +#[tokio::test] +async fn rejected_requests_use_the_detail_envelope( + #[future] harness: Harness, + #[case] path: &str, + #[case] method: &str, + #[case] body: &'static str, +) { + let harness = harness.await; + let response = harness + .app + .oneshot( + Request::builder() + .method(method) + .uri(path) + .header("host", "localhost") + .header("content-type", "application/json") + .body(Body::from(body)) + .unwrap(), + ) + .await + .unwrap(); + assert!(response.status().is_client_error(), "{}", response.status()); + let body: Value = + serde_json::from_slice(&to_bytes(response.into_body(), 65536).await.unwrap()).unwrap(); + assert!(body["detail"].is_string(), "{body}"); } #[rstest] diff --git a/litellm-rust/crates/gateway-mcp/tests/gateway/native.rs b/litellm-rust/crates/gateway-mcp/tests/gateway/native.rs index aca19b38d6e..54fdabc0da7 100644 --- a/litellm-rust/crates/gateway-mcp/tests/gateway/native.rs +++ b/litellm-rust/crates/gateway-mcp/tests/gateway/native.rs @@ -148,10 +148,12 @@ async fn routes_prompts_and_resources_without_rewriting_uris(#[future] harness: "fixture://{file}" ); let read = Operation::ReadResource(ReadResourceRequestParams::new("fixture://document")); - assert!(matches!( - harness.gateway.execute(read.clone(), context(None)).await, - Err(Error::InvalidRequest(_)) - )); + let ambiguous = harness + .gateway + .execute(read.clone(), context(None)) + .await + .expect_err("read without a server scope"); + assert_eq!(ambiguous.status(), axum::http::StatusCode::BAD_REQUEST); let result = harness .gateway .execute(read, context(Some("alpha"))) diff --git a/litellm-rust/crates/gateway-ui/Cargo.toml b/litellm-rust/crates/gateway-ui/Cargo.toml index 94546ef9118..1c65665be72 100644 --- a/litellm-rust/crates/gateway-ui/Cargo.toml +++ b/litellm-rust/crates/gateway-ui/Cargo.toml @@ -6,6 +6,7 @@ license.workspace = true repository.workspace = true [dependencies] +litellm-error.workspace = true axum = { workspace = true, features = ["json", "original-uri", "query"] } axum-login.workspace = true base64.workspace = true @@ -22,6 +23,7 @@ tower-http = { version = "0.6.11", features = ["fs", "set-header"] } tower-sessions.workspace = true [dev-dependencies] +futures-util.workspace = true serde_json.workspace = true tower = { version = "0.5", features = ["util"] } litellm-auth-types.workspace = true diff --git a/litellm-rust/crates/gateway-ui/src/dashboard.rs b/litellm-rust/crates/gateway-ui/src/dashboard.rs index 2642506fde6..04ec8490411 100644 --- a/litellm-rust/crates/gateway-ui/src/dashboard.rs +++ b/litellm-rust/crates/gateway-ui/src/dashboard.rs @@ -2,7 +2,8 @@ use std::path::Path; use axum::{ Router, - extract::{Query, Request}, + extract::{FromRequestParts, Query, Request}, + http::request::Parts, routing::get, }; use serde::{Deserialize, Serialize}; @@ -34,6 +35,17 @@ struct LogoQuery { variant: Option, } +impl FromRequestParts for LogoQuery { + type Rejection = crate::Error; + + async fn from_request_parts(parts: &mut Parts, state: &S) -> Result { + Query::::from_request_parts(parts, state) + .await + .map(|Query(query)| query) + .map_err(crate::Error::Query) + } +} + fn logo_file(query: &LogoQuery) -> &'static str { match (query.variant, query.theme) { (Some(Variant::Monogram), Some(Theme::Dark)) => "assets/logos/litellm_monogram_dark.svg", @@ -59,7 +71,7 @@ pub fn dashboard_assets(directory: impl AsRef) -> Router { ) .route( "/get_image", - get(move |Query(query): Query, request: Request| { + get(move |query: LogoQuery, request: Request| { ServeFile::new(logos.join(logo_file(&query))).oneshot(request) }), ) diff --git a/litellm-rust/crates/gateway-ui/src/error.rs b/litellm-rust/crates/gateway-ui/src/error.rs index 16e6e205d99..cce01ba76ce 100644 --- a/litellm-rust/crates/gateway-ui/src/error.rs +++ b/litellm-rust/crates/gateway-ui/src/error.rs @@ -1,9 +1,9 @@ use axum::{ Json, + extract::rejection::{JsonRejection, QueryRejection}, http::{StatusCode, header}, response::{IntoResponse, Response}, }; -use litellm_gateway_auth::UiAuthError; #[derive(serde::Serialize)] struct ErrorBody { @@ -21,39 +21,108 @@ pub enum Error { InvalidCredentials, #[error("Too many login attempts, try again in one minute")] RateLimited, - #[error("UI authentication unavailable")] - Auth(#[from] axum_login::Error), + #[error("invalid login request")] + Body(#[source] JsonRejection), + #[error("invalid query string")] + Query(#[source] QueryRejection), + #[error("UI login unavailable")] + Login(#[from] axum_login::Error), #[error("UI session unavailable")] Session(#[from] tower_sessions::session::Error), #[error("UI token unavailable")] Token(#[from] jsonwebtoken::errors::Error), - #[error("Invalid UI session")] - Unauthorized(#[from] UiAuthError), + #[error(transparent)] + Auth(#[from] litellm_gateway_auth::Error), +} + +impl Error { + pub fn status(&self) -> StatusCode { + match self { + Self::InvalidCredentials => StatusCode::UNAUTHORIZED, + Self::RateLimited => StatusCode::TOO_MANY_REQUESTS, + Self::Body(rejection) => rejection.status(), + Self::Query(rejection) => rejection.status(), + Self::Auth(error) => error.status(), + Self::Login(_) | Self::Session(_) | Self::Token(_) => StatusCode::INTERNAL_SERVER_ERROR, + } + } +} + +impl Error { + fn message(&self) -> String { + match self { + Self::Body(_) | Self::Query(_) => litellm_error::Report::new(self).to_string(), + Self::InvalidCredentials + | Self::RateLimited + | Self::Login(_) + | Self::Session(_) + | Self::Token(_) + | Self::Auth(_) => self.to_string(), + } + } } impl IntoResponse for Error { fn into_response(self) -> Response { - let status = match &self { - Self::InvalidCredentials | Self::Unauthorized(UiAuthError::Unauthorized) => { - StatusCode::UNAUTHORIZED + let status = self.status(); + let body = Json(ErrorBody { + error: ErrorMessage { + message: self.message(), + }, + }); + match status { + StatusCode::TOO_MANY_REQUESTS => { + (status, [(header::RETRY_AFTER, "60")], body).into_response() } - Self::RateLimited => StatusCode::TOO_MANY_REQUESTS, - _ => StatusCode::INTERNAL_SERVER_ERROR, - }; - let response = ( - status, - Json(ErrorBody { - error: ErrorMessage { - message: self.to_string(), - }, - }), - ) - .into_response(); - - if matches!(self, Self::RateLimited) { - return ([(header::RETRY_AFTER, "60")], response).into_response(); + _ => (status, body).into_response(), } - - response + } +} + +#[cfg(test)] +mod tests { + use litellm_gateway_auth::Error as AuthError; + use rstest::rstest; + + use super::*; + + #[rstest] + #[case::rejected_session(Error::Auth(AuthError::InvalidSession), StatusCode::UNAUTHORIZED)] + #[case::session_layer_missing( + Error::Auth(AuthError::MissingSessionLayer), + StatusCode::INTERNAL_SERVER_ERROR + )] + #[case::operation_outside_session_scope( + Error::Auth(AuthError::Forbidden), + StatusCode::FORBIDDEN + )] + #[case::rate_limited(Error::RateLimited, StatusCode::TOO_MANY_REQUESTS)] + fn status_follows_who_is_at_fault(#[case] error: Error, #[case] status: StatusCode) { + assert_eq!(error.status(), status); + } + + #[rstest] + #[case::rate_limited(Error::RateLimited, true)] + #[case::invalid_credentials(Error::InvalidCredentials, false)] + fn only_rate_limits_ask_to_retry_later(#[case] error: Error, #[case] retry_after: bool) { + assert_eq!( + error + .into_response() + .headers() + .contains_key(header::RETRY_AFTER), + retry_after + ); + } + + #[tokio::test] + async fn a_body_never_shows_what_the_backend_said() { + let backend = std::io::Error::other("vault token rejected"); + let response = Error::Auth(AuthError::Unavailable(litellm_error::Source::new(backend))) + .into_response(); + let body = axum::body::to_bytes(response.into_body(), 4096) + .await + .unwrap(); + let body = String::from_utf8(body.to_vec()).unwrap(); + assert!(!body.contains("vault token rejected"), "{body}"); } } diff --git a/litellm-rust/crates/gateway-ui/src/session.rs b/litellm-rust/crates/gateway-ui/src/session.rs index b80f40bb376..b5d847ad353 100644 --- a/litellm-rust/crates/gateway-ui/src/session.rs +++ b/litellm-rust/crates/gateway-ui/src/session.rs @@ -4,14 +4,18 @@ use serde::Serialize; use std::sync::Arc; use time::{Duration, OffsetDateTime}; -use axum::{Json, extract::State as ExtractState, response::IntoResponse}; +use axum::{ + Json, + extract::{FromRequest, Request, State as ExtractState}, + response::IntoResponse, +}; use governor::DefaultDirectRateLimiter; use tower_cookies::{Cookie, Cookies}; use tower_sessions::{Expiry, Session, cookie::SameSite}; use crate::Error; use litellm_gateway_auth::{ - AccessRequest, UI_CSRF_KEY, UiAction, UiAuthError, UiAuthSession, UiCredentials, UiSession, + AccessRequest, UI_CSRF_KEY, UiAction, UiAuthSession, UiCredentials, UiSession, }; pub struct State { @@ -60,12 +64,25 @@ struct Discovery { is_control_plane: bool, } +pub struct Credentials(UiCredentials); + +impl FromRequest for Credentials { + type Rejection = Error; + + async fn from_request(request: Request, state: &S) -> Result { + Json::::from_request(request, state) + .await + .map(|Json(credentials)| Self(credentials)) + .map_err(Error::Body) + } +} + pub async fn login( ExtractState(state): ExtractState>, mut auth: UiAuthSession, session: Session, cookies: Cookies, - Json(credentials): Json, + Credentials(credentials): Credentials, ) -> Result { state.login_limit.check().map_err(|_| Error::RateLimited)?; let user = auth @@ -107,13 +124,14 @@ pub async fn login( })) } -pub async fn info(session: Result) -> Result { +pub async fn info( + session: Result, +) -> Result { let session = session?; session .identity .authorize(AccessRequest::Ui(UiAction::SessionInfo)) - .await - .map_err(|_| UiAuthError::Unauthorized)?; + .await?; Ok(Json(SessionInfo { user_id: session.user.username, user_role: "proxy_admin", @@ -121,7 +139,7 @@ pub async fn info(session: Result) -> Result, + validated: Result, mut auth: UiAuthSession, cookies: Cookies, ) -> Result { @@ -129,8 +147,7 @@ pub async fn logout( validated .identity .authorize(AccessRequest::Ui(UiAction::Logout)) - .await - .map_err(|_| UiAuthError::Unauthorized)?; + .await?; auth.logout().await?; cookies.add( Cookie::build(("token", "")) diff --git a/litellm-rust/crates/gateway-ui/tests/assets.rs b/litellm-rust/crates/gateway-ui/tests/assets.rs index ba741fbf2ec..e2819033f95 100644 --- a/litellm-rust/crates/gateway-ui/tests/assets.rs +++ b/litellm-rust/crates/gateway-ui/tests/assets.rs @@ -200,6 +200,24 @@ async fn logo_discovery_points_to_served_image(dashboard: App) { ); } +#[rstest] +#[tokio::test] +async fn unknown_logo_theme_is_rejected_in_the_ui_envelope(dashboard: App) { + let response = dashboard + .router + .oneshot( + Request::get("/get_image?theme=purple") + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body: Value = + serde_json::from_slice(&to_bytes(response.into_body(), 65536).await.unwrap()).unwrap(); + assert!(body["error"]["message"].as_str().unwrap().contains("theme")); +} + #[rstest] #[case::logo("/get_image")] #[case::logo_dark("/get_image?theme=dark")] diff --git a/litellm-rust/crates/gateway-ui/tests/routes.rs b/litellm-rust/crates/gateway-ui/tests/routes.rs index 4a3238f79c2..95246f33a35 100644 --- a/litellm-rust/crates/gateway-ui/tests/routes.rs +++ b/litellm-rust/crates/gateway-ui/tests/routes.rs @@ -330,24 +330,45 @@ async fn reauthentication_rotates_and_revokes_the_previous_session(app: App) { ); } +async fn failed_login(app: &App, username: &str) -> Response { + app.router + .clone() + .oneshot( + Request::post("/v2/login") + .header(header::CONTENT_TYPE, "application/json") + .body(Body::from( + json!({"username": username, "password": "wrong-password"}).to_string(), + )) + .unwrap(), + ) + .await + .unwrap() +} + #[rstest] #[tokio::test] async fn failed_logins_are_generic_and_rate_limited(app: App) { - for attempt in 0..11 { - let response = app.router.clone().oneshot(Request::post("/v2/login") - .header(header::CONTENT_TYPE, "application/json") - .body(Body::from(json!({"username": format!("unknown-{attempt}"), "password": "wrong-password"}).to_string())).unwrap()).await.unwrap(); - if attempt < 10 { - assert_eq!(response.status(), StatusCode::UNAUTHORIZED); - assert_eq!( - body(response).await["error"]["message"], - "Invalid username or password" - ); - } else { - assert_eq!(response.status(), StatusCode::TOO_MANY_REQUESTS); - assert!(response.headers().contains_key(header::RETRY_AFTER)); - } - } + let first = failed_login(&app, "unknown-0").await; + assert_eq!(first.status(), StatusCode::UNAUTHORIZED); + assert_eq!( + body(first).await["error"]["message"], + "Invalid username or password" + ); + let usernames: Vec = (1..10) + .map(|attempt| format!("unknown-{attempt}")) + .collect(); + let burst = futures_util::future::join_all( + usernames + .iter() + .map(|username| failed_login(&app, username)), + ) + .await; + let statuses: Vec = burst.iter().map(Response::status).collect(); + assert_eq!(statuses, vec![StatusCode::UNAUTHORIZED; 9]); + let limited = failed_login(&app, "unknown-10").await; + assert_eq!(limited.status(), StatusCode::TOO_MANY_REQUESTS); + assert!(limited.headers().contains_key(header::RETRY_AFTER)); + assert!(body(limited).await["error"]["message"].is_string()); } #[rstest] @@ -365,6 +386,7 @@ async fn login_rejects_browser_form_posts(app: App) { .unwrap(); assert_eq!(response.status(), StatusCode::UNSUPPORTED_MEDIA_TYPE); assert!(!response.headers().contains_key(header::SET_COOKIE)); + assert!(body(response).await["error"]["message"].is_string()); } #[rstest] @@ -468,6 +490,7 @@ async fn invalid_login_payloads_do_not_create_sessions( .unwrap(); assert_eq!(response.status(), status); assert!(!response.headers().contains_key(header::SET_COOKIE)); + assert!(body(response).await["error"]["message"].is_string()); } #[rstest] diff --git a/litellm-rust/crates/gateway/Cargo.toml b/litellm-rust/crates/gateway/Cargo.toml index 4982c70b18c..4d831aa2314 100644 --- a/litellm-rust/crates/gateway/Cargo.toml +++ b/litellm-rust/crates/gateway/Cargo.toml @@ -19,6 +19,7 @@ litellm-gateway-auth.workspace = true litellm-gateway-ui.workspace = true litellm-auth-types.workspace = true litellm-config.workspace = true +litellm-error.workspace = true litellm-circuit-breaker.workspace = true litellm-redis-client.workspace = true litellm-http.workspace = true diff --git a/litellm-rust/crates/gateway/src/auth.rs b/litellm-rust/crates/gateway/src/auth.rs index 72892be783c..d5d8a229d0d 100644 --- a/litellm-rust/crates/gateway/src/auth.rs +++ b/litellm-rust/crates/gateway/src/auth.rs @@ -64,10 +64,8 @@ impl OperationAuthorizer for McpPolicy { } fn mcp_error(error: Error) -> litellm_gateway_mcp::Error { - match error { - Error::InvalidToken | Error::Expired | Error::Forbidden => { - litellm_gateway_mcp::Error::Forbidden - } - _ => litellm_gateway_mcp::Error::Configuration("MCP authorization unavailable".into()), + if error.is_denied() { + return litellm_gateway_mcp::Error::Forbidden; } + litellm_gateway_mcp::Error::AuthorizationUnavailable(litellm_error::Source::new(error)) } diff --git a/litellm-rust/crates/gateway/src/error.rs b/litellm-rust/crates/gateway/src/error.rs index 170a8284c18..e2f646c5448 100644 --- a/litellm-rust/crates/gateway/src/error.rs +++ b/litellm-rust/crates/gateway/src/error.rs @@ -6,6 +6,8 @@ pub enum Error { Environment, #[error("MCP host does not yet support configured setting {0}")] McpSetting(String), + #[error("MCP host does not yet support guardrails or policies")] + McpGuardrails, #[error(transparent)] Mcp(#[from] litellm_gateway_mcp::ConnectError), #[error(transparent)] @@ -17,6 +19,6 @@ pub enum Error { }, #[error(transparent)] RedisConnect(#[from] litellm_redis_client::ConnectError), - #[error("MCP requires a configured master key")] + #[error("MCP requires working gateway authentication")] Auth(#[from] litellm_gateway_auth::Error), } diff --git a/litellm-rust/crates/gateway/src/lib.rs b/litellm-rust/crates/gateway/src/lib.rs index d7567b83cc1..b8ff8e46033 100644 --- a/litellm-rust/crates/gateway/src/lib.rs +++ b/litellm-rust/crates/gateway/src/lib.rs @@ -109,7 +109,7 @@ pub async fn build_mcp( || !config.policies.is_empty() || !config.policy_attachments.is_empty() { - return Err(Error::McpSetting("guardrails or policies".into())); + return Err(Error::McpGuardrails); } Auth::from_config(config, secrets.clone()) .validate() diff --git a/litellm-rust/crates/gateway/tests/mcp.rs b/litellm-rust/crates/gateway/tests/mcp.rs index ee8a17ae251..3d12efd730e 100644 --- a/litellm-rust/crates/gateway/tests/mcp.rs +++ b/litellm-rust/crates/gateway/tests/mcp.rs @@ -97,11 +97,29 @@ async fn upstream() -> Fixture { } } -#[rstest] -#[tokio::test] -async fn serves_configured_upstream_with_separate_admission_and_outbound_credentials( - #[future] upstream: Fixture, -) { +struct Served { + app: Router, + mcp: litellm_gateway_mcp::ConfiguredGateway, + shutdown: CancellationToken, + upstream: Fixture, +} + +impl Served { + async fn close(self) { + self.shutdown.cancel(); + tokio::time::timeout(Duration::from_secs(5), self.mcp.close()) + .await + .unwrap(); + self.upstream.shutdown.cancel(); + tokio::time::timeout(Duration::from_secs(5), self.upstream.task) + .await + .unwrap() + .unwrap(); + } +} + +#[fixture] +async fn served(#[future] upstream: Fixture) -> Served { let upstream = upstream.await; let config = Config::from_yaml(&format!( r#" @@ -148,30 +166,53 @@ mcp_servers: }, ); let app = litellm_gateway::router(inference, &config, None, Some(mcp_router)); - - for (path, method) in [ - ("/mcp", "POST"), - ("/mcp", "GET"), - ("/mcp", "DELETE"), - ("/mcp/sse", "GET"), - ("/mcp/sse/messages", "POST"), - ("/mcp-rest/tools/list", "GET"), - ("/mcp/enabled", "GET"), - ] { - let response = app - .clone() - .oneshot( - Request::builder() - .method(method) - .uri(path) - .header("host", "localhost") - .body(Body::empty()) - .unwrap(), - ) - .await - .unwrap(); - assert_eq!(response.status(), 401, "{method} {path}"); + Served { + app, + mcp, + shutdown, + upstream, } +} + +#[rstest] +#[case::streamable_post("/mcp", "POST")] +#[case::streamable_get("/mcp", "GET")] +#[case::streamable_delete("/mcp", "DELETE")] +#[case::legacy_sse("/mcp/sse", "GET")] +#[case::legacy_messages("/mcp/sse/messages", "POST")] +#[case::rest_list("/mcp-rest/tools/list", "GET")] +#[case::enabled("/mcp/enabled", "GET")] +#[tokio::test] +async fn every_mcp_route_requires_admission_credentials( + #[future] served: Served, + #[case] path: &str, + #[case] method: &str, +) { + let served = served.await; + let response = served + .app + .clone() + .oneshot( + Request::builder() + .method(method) + .uri(path) + .header("host", "localhost") + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), 401); + served.close().await; +} + +#[rstest] +#[tokio::test] +async fn serves_configured_upstream_with_separate_admission_and_outbound_credentials( + #[future] served: Served, +) { + let served = served.await; + let app = served.app.clone(); let list = app .clone() .oneshot( @@ -213,15 +254,7 @@ mcp_servers: .unwrap(); assert_eq!(blocked.status(), 403); - shutdown.cancel(); - tokio::time::timeout(Duration::from_secs(5), mcp.close()) - .await - .unwrap(); - upstream.shutdown.cancel(); - tokio::time::timeout(Duration::from_secs(5), upstream.task) - .await - .unwrap() - .unwrap(); + served.close().await; } #[rstest] @@ -275,6 +308,7 @@ async fn mcp_tools_are_discovered_through_the_configured_proxy(#[future] upstrea .unwrap(); } +#[rstest] #[rstest] #[tokio::test] async fn mcp_uses_the_configured_ca_bundle() { @@ -299,11 +333,20 @@ async fn mcp_uses_the_configured_ca_bundle() { .await .err() .unwrap(); - assert!(matches!( - error, - litellm_gateway::Error::Http(litellm_http::Error::Read { path, .. }) - if path == missing - )); + let litellm_gateway::Error::Http(error) = error else { + panic!("expected an HTTP configuration error"); + }; + assert_eq!(error.tls_source(), Some(litellm_http::TlsSource::CaBundle)); + assert_eq!( + error.to_string(), + format!("could not read {}", missing.display()) + ); + assert_eq!( + litellm_error::find::(&error) + .unwrap() + .kind(), + std::io::ErrorKind::NotFound, + ); } #[rstest] @@ -336,9 +379,10 @@ async fn rejects_unsupported_or_invalid_settings_before_serving( &inference.http, ) .await; - let error = result.err().expect("startup must fail").to_string(); - assert!(error.contains(expected), "{error}"); - assert!(!error.contains("secret-value")); + let error = result.err().expect("startup must fail"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains(expected), "{report}"); + assert!(!report.contains("secret-value")); } #[rstest] @@ -397,9 +441,10 @@ fn applies_configured_http_environment_before_building_clients() { .unwrap(); let error = litellm_gateway::build_inference(&config).err().unwrap(); assert!(matches!( - error, - litellm_gateway::Error::Http(litellm_http::Error::Read { path, .. }) - if path == missing + &error, + litellm_gateway::Error::Http(http) + if http.tls_source() == Some(litellm_http::TlsSource::CaBundle) + && http.to_string().contains(&missing.display().to_string()) )); } @@ -424,5 +469,8 @@ async fn refuses_unimplemented_global_mcp_policy(#[case] settings: &str) { .await .err() .unwrap(); - assert!(matches!(error, litellm_gateway::Error::McpSetting(_))); + assert!(matches!( + error, + litellm_gateway::Error::McpSetting(_) | litellm_gateway::Error::McpGuardrails + )); } diff --git a/litellm-rust/crates/gateway/tests/server.rs b/litellm-rust/crates/gateway/tests/server.rs index 53a97e7098c..c876c1cbcbb 100644 --- a/litellm-rust/crates/gateway/tests/server.rs +++ b/litellm-rust/crates/gateway/tests/server.rs @@ -94,7 +94,10 @@ async fn authenticates_before_serving_mounted_inference_routes( let response = request.send().await.unwrap(); assert_eq!(response.status().as_u16(), status); if status == 400 { - let expected = Error::UnknownModel("unconfigured-model".into()); + let expected = Error::UnknownModel { + route: "/v1/messages", + model: "unconfigured-model".into(), + }; assert_eq!( response.json::().await.unwrap(), expected.body(Some("gateway-test")) diff --git a/litellm-rust/crates/host-python/Cargo.toml b/litellm-rust/crates/host-python/Cargo.toml index 8d220b19def..0b0107d0548 100644 --- a/litellm-rust/crates/host-python/Cargo.toml +++ b/litellm-rust/crates/host-python/Cargo.toml @@ -7,6 +7,7 @@ repository.workspace = true [dependencies] litellm-coroutine.workspace = true +litellm-error.workspace = true litellm-host.workspace = true bytes.workspace = true diff --git a/litellm-rust/crates/host-python/src/binding.rs b/litellm-rust/crates/host-python/src/binding.rs index 91fbc8c0e52..e36b7d70c28 100644 --- a/litellm-rust/crates/host-python/src/binding.rs +++ b/litellm-rust/crates/host-python/src/binding.rs @@ -5,7 +5,7 @@ use pyo3::types::PyDict; /// Converts requests, responses, stream values and errors at the Python boundary. pub trait PythonBinding: PythonOwned { - type Protocol: Protocol; + type Protocol: Protocol; /// The public exception a native failure maps to, kept as a value until the driver /// raises it. diff --git a/litellm-rust/crates/host-python/src/driver.rs b/litellm-rust/crates/host-python/src/driver.rs index f8e35aa09ba..5dd9a72eb4a 100644 --- a/litellm-rust/crates/host-python/src/driver.rs +++ b/litellm-rust/crates/host-python/src/driver.rs @@ -112,7 +112,7 @@ enum Setup { /// The route's public exception for a native failure. When classification itself fails, /// that failure is raised with the native error's text as its `__context__`. fn classified(py: Python<'_>, binding: &H, error: ErrorOf) -> PyErr { - let native = error.to_string(); + let native = litellm_error::Report::new(&error).to_string(); let classifier_error = match binding.map_error(py, error) { Ok(failure) => return failure.into(), Err(classifier_error) => classifier_error, diff --git a/litellm-rust/crates/host-python/src/error.rs b/litellm-rust/crates/host-python/src/error.rs index b06a4a2fd41..8d47d97df25 100644 --- a/litellm-rust/crates/host-python/src/error.rs +++ b/litellm-rust/crates/host-python/src/error.rs @@ -1,3 +1,4 @@ +use litellm_error::Report; use pyo3::{ PyErr, exceptions::{PyRuntimeError, PyTypeError}, @@ -24,16 +25,19 @@ pub enum Error { SyncInsideTokio, #[error("the context body produced no value")] ContextBodyProducedNothing, + #[error("the context body ran more than once")] + ContextBodyRanTwice, + #[error("the blocking pool dropped a Python call")] + BlockingCallCancelled(#[source] tokio::task::JoinError), #[error("file read must return bytes or str, got {got}")] FileContentType { got: String }, } impl From for PyErr { fn from(error: Error) -> Self { + let message = Report::new(&error).to_string(); match error { - Error::SendBeforeStart | Error::FileContentType { .. } => { - PyTypeError::new_err(error.to_string()) - } + Error::SendBeforeStart | Error::FileContentType { .. } => PyTypeError::new_err(message), Error::MissingState | Error::ExecutionRunning | Error::ExecutionClosed @@ -41,7 +45,9 @@ impl From for PyErr { | Error::ExecutionClosedWhileRunning | Error::NativeExecutionClosed | Error::SyncInsideTokio - | Error::ContextBodyProducedNothing => PyRuntimeError::new_err(error.to_string()), + | Error::ContextBodyProducedNothing + | Error::ContextBodyRanTwice + | Error::BlockingCallCancelled(_) => PyRuntimeError::new_err(message), } } } @@ -51,7 +57,7 @@ impl From for PyErr { /// raised, which reaches the caller as it was raised. #[derive(Debug, thiserror::Error)] pub enum InvokeError { - #[error("{0}")] + #[error(transparent)] Native(E), #[error(transparent)] Python(#[from] PyErr), diff --git a/litellm-rust/crates/host-python/src/fork_gate.rs b/litellm-rust/crates/host-python/src/fork_gate.rs index c4842dd9223..82698931f8f 100644 --- a/litellm-rust/crates/host-python/src/fork_gate.rs +++ b/litellm-rust/crates/host-python/src/fork_gate.rs @@ -12,13 +12,23 @@ pub(crate) struct ForkGate { fork_only_pid: AtomicU32, } -#[derive(Debug, PartialEq, Eq)] +#[derive(Debug, PartialEq, Eq, thiserror::Error)] pub(crate) enum Refused { + #[error( + "this process is reserved for forking workers and cannot run native routes; \ + move the call into a worker, after the fork" + )] ReservedForForking, + #[error( + "this process was forked after the native runtime started, and runtime threads \ + do not survive fork(); start workers with spawn or forkserver, or fork before \ + the first native call" + )] ForkedAfterStart, } -#[derive(Debug, PartialEq, Eq)] +#[derive(Debug, PartialEq, Eq, thiserror::Error)] +#[error("the native runtime already started in this process")] pub struct RuntimeAlreadyStarted; impl ForkGate { diff --git a/litellm-rust/crates/host-python/src/gil.rs b/litellm-rust/crates/host-python/src/gil.rs index aacecb3af81..5f54fae81ed 100644 --- a/litellm-rust/crates/host-python/src/gil.rs +++ b/litellm-rust/crates/host-python/src/gil.rs @@ -1,8 +1,9 @@ use std::sync::{ - Arc, Mutex, + Arc, atomic::{AtomicU64, Ordering}, }; +use parking_lot::Mutex; use pyo3::prelude::*; use crate::Error; @@ -36,12 +37,8 @@ impl PythonContext { let value = Arc::clone(&value); pyo3::types::PyCFunction::new_closure(py, None, None, move |_, _| { Python::attach(|py| -> PyResult<()> { - let body = body - .lock() - .expect("context body slot poisoned") - .take() - .expect("the context body ran more than once"); - *value.lock().expect("context value slot poisoned") = Some(body(py)); + let body = body.lock().take().ok_or(Error::ContextBodyRanTwice)?; + *value.lock() = Some(body(py)); Ok(()) }) })? @@ -49,7 +46,6 @@ impl PythonContext { copy.call_method1("run", (callback,))?; value .lock() - .expect("context value slot poisoned") .take() .ok_or_else(|| Error::ContextBodyProducedNothing.into()) } @@ -84,7 +80,7 @@ where match tokio::task::spawn_blocking(move || Python::attach(|py| context.enter(py, f))).await { Ok(value) => value, Err(error) if error.is_panic() => std::panic::resume_unwind(error.into_panic()), - Err(error) => panic!("the blocking pool dropped a Python call: {error}"), + Err(error) => Err(Error::BlockingCallCancelled(error).into()), } } diff --git a/litellm-rust/crates/host-python/src/handle.rs b/litellm-rust/crates/host-python/src/handle.rs index 2361128bf43..cb4de2e4763 100644 --- a/litellm-rust/crates/host-python/src/handle.rs +++ b/litellm-rust/crates/host-python/src/handle.rs @@ -56,25 +56,27 @@ impl Execution { fn advance(slf: &Bound<'_, Self>, py: Python<'_>, input: Resume) -> PyResult> { let (mut body, started) = { let mut execution = slf.borrow_mut(); - let started = match (&execution.state, &input) { - (ExecutionState::Created(_), Resume::Send(value)) if value.is_none(py) => false, - (ExecutionState::Created(_), Resume::Send(_)) => { + match ( + std::mem::replace(&mut execution.state, ExecutionState::Running), + &input, + ) { + (ExecutionState::Created(body), Resume::Send(value)) if value.is_none(py) => { + (body, false) + } + (ExecutionState::Suspended(body), _) => (body, true), + (ExecutionState::Running, _) => return Err(Error::ExecutionRunning.into()), + (state @ ExecutionState::Created(_), Resume::Send(_)) => { + execution.state = state; return Err(Error::SendBeforeStart.into()); } - (ExecutionState::Suspended(_), _) => true, - (ExecutionState::Running, _) => { - return Err(Error::ExecutionRunning.into()); - } - (ExecutionState::Closed, _) => { - return Err(Error::ExecutionClosed.into()); - } - (ExecutionState::Created(_), Resume::Throw(_)) => { + (state @ ExecutionState::Created(_), Resume::Throw(_)) => { + execution.state = state; return Err(Error::ExecutionOutOfOrder.into()); } - }; - match std::mem::replace(&mut execution.state, ExecutionState::Running) { - ExecutionState::Created(body) | ExecutionState::Suspended(body) => (body, started), - _ => unreachable!(), + (ExecutionState::Closed, _) => { + execution.state = ExecutionState::Closed; + return Err(Error::ExecutionClosed.into()); + } } }; let result = match input { diff --git a/litellm-rust/crates/host-python/src/marshal.rs b/litellm-rust/crates/host-python/src/marshal.rs index ae9066379ae..465521cf9d2 100644 --- a/litellm-rust/crates/host-python/src/marshal.rs +++ b/litellm-rust/crates/host-python/src/marshal.rs @@ -14,7 +14,8 @@ pub fn from_py_argument(value: &Bound<'_, PyAny>) -> PyResult where T: DeserializeOwned, { - pythonize::depythonize(value).map_err(|error| PyValueError::new_err(error.to_string())) + pythonize::depythonize(value) + .map_err(|error| PyValueError::new_err(litellm_error::Report::new(&error).to_string())) } pub fn from_py(value: &Bound<'_, PyAny>) -> PyResult diff --git a/litellm-rust/crates/host-python/src/runtime.rs b/litellm-rust/crates/host-python/src/runtime.rs index c128242ef14..e8872c9340a 100644 --- a/litellm-rust/crates/host-python/src/runtime.rs +++ b/litellm-rust/crates/host-python/src/runtime.rs @@ -7,6 +7,7 @@ use std::time::Duration; use crate::fork_gate::{ForkGate, Refused, RuntimeAlreadyStarted}; use crate::{Error, Pythonized, panic_to_pyerr, release_gil}; use futures_util::FutureExt; +use litellm_error::Report; use pyo3::exceptions::PyRuntimeError; use pyo3::prelude::*; use serde::Serialize; @@ -42,19 +43,23 @@ pub fn reserve_process_for_forking() -> Result<(), RuntimeAlreadyStarted> { /// Claims process-bound native state before runtime startup or tokenizer execution. pub fn enter_native() -> PyResult<()> { - FORK_GATE - .enter(std::process::id()) - .map_err(|refused| match refused { - Refused::ReservedForForking => ProcessReservedForForking::new_err( - "this process is reserved for forking workers and cannot run native routes; \ - move the call into a worker, after the fork", - ), - Refused::ForkedAfterStart => ForkedAfterNativeRuntimeStarted::new_err( - "this process was forked after the native runtime started, and runtime threads \ - do not survive fork(); start workers with spawn or forkserver, or fork before \ - the first native call", - ), - }) + Ok(FORK_GATE.enter(std::process::id())?) +} + +impl From for PyErr { + fn from(refused: Refused) -> Self { + let message = Report::new(&refused).to_string(); + match refused { + Refused::ReservedForForking => ProcessReservedForForking::new_err(message), + Refused::ForkedAfterStart => ForkedAfterNativeRuntimeStarted::new_err(message), + } + } +} + +impl From for PyErr { + fn from(error: RuntimeAlreadyStarted) -> Self { + PyRuntimeError::new_err(Report::new(&error).to_string()) + } } #[expect(clippy::disallowed_methods, reason = "this is the gated door")] diff --git a/litellm-rust/crates/host-python/tests/support/mod.rs b/litellm-rust/crates/host-python/tests/support/mod.rs index ef09135698e..003fce74309 100644 --- a/litellm-rust/crates/host-python/tests/support/mod.rs +++ b/litellm-rust/crates/host-python/tests/support/mod.rs @@ -92,6 +92,8 @@ impl std::fmt::Display for Error { } } +impl std::error::Error for Error {} + impl From for Error { fn from(fault: MachineFault) -> Self { Self(format!("{fault:?}")) diff --git a/litellm-rust/crates/host/Cargo.toml b/litellm-rust/crates/host/Cargo.toml index 51015e99906..d3354fa8622 100644 --- a/litellm-rust/crates/host/Cargo.toml +++ b/litellm-rust/crates/host/Cargo.toml @@ -10,6 +10,7 @@ futures-util.workspace = true litellm-auth.workspace = true litellm-coroutine.workspace = true serde_json.workspace = true +thiserror.workspace = true tokio = { workspace = true, features = ["sync"] } [dev-dependencies] diff --git a/litellm-rust/crates/host/src/machine/coroutine.rs b/litellm-rust/crates/host/src/machine/coroutine.rs index 825536e1d9b..db69f1dceb2 100644 --- a/litellm-rust/crates/host/src/machine/coroutine.rs +++ b/litellm-rust/crates/host/src/machine/coroutine.rs @@ -10,12 +10,14 @@ use super::{ use crate::protocol::{HostRequest, Protocol}; /// The machine's own failures, distinct from anything the provider call reports. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] pub enum MachineFault { /// The host dropped an op's reply unanswered, or went away while the call waited. + #[error("the host abandoned the call")] Abandoned, /// The host resumed the call out of turn. - Protocol(ResumeError), + #[error("the host resumed the call out of turn")] + Protocol(#[source] ResumeError), } pub type ExecuteFuture::Response> = diff --git a/litellm-rust/crates/http/Cargo.toml b/litellm-rust/crates/http/Cargo.toml index 0552f9775fd..6b3e3cc7852 100644 --- a/litellm-rust/crates/http/Cargo.toml +++ b/litellm-rust/crates/http/Cargo.toml @@ -11,6 +11,7 @@ test-support = [] [dependencies] http.workspace = true litellm-core-utils.workspace = true +litellm-error.workspace = true hyper-util.workspace = true reqwest.workspace = true rustls.workspace = true @@ -22,6 +23,7 @@ strum.workspace = true thiserror.workspace = true tokio.workspace = true tracing.workspace = true +url.workspace = true veil.workspace = true webpki-roots.workspace = true diff --git a/litellm-rust/crates/http/src/config.rs b/litellm-rust/crates/http/src/config.rs index b37ea6559b8..5e5f2d66fa2 100644 --- a/litellm-rust/crates/http/src/config.rs +++ b/litellm-rust/crates/http/src/config.rs @@ -136,7 +136,7 @@ mod tests { use rstest::rstest; use super::*; - use crate::TlsSource; + use crate::{TlsSource, error::Repr}; fn settings(ssl_verify: Option, ssl_cert_file: Option<&str>) -> HttpSettings { HttpSettings { @@ -306,11 +306,10 @@ mod tests { }; assert!(matches!( reqwest::ClientBuilder::try_from(&config), - Err(Error::Read { - path: reported, - tls_source: TlsSource::CaBundle, - .. - }) if reported == path + Err(error) if matches!( + error.repr(), + Repr::Read { path: reported, tls_source: TlsSource::CaBundle, .. } if *reported == path + ) )); } @@ -327,11 +326,10 @@ mod tests { std::fs::remove_file(&path).unwrap(); assert!(matches!( result, - Err(Error::InvalidPem { - path: reported, - tls_source: TlsSource::CaBundle, - .. - }) if reported == path + Err(error) if matches!( + error.repr(), + Repr::InvalidPem { path: reported, tls_source: TlsSource::CaBundle, .. } if *reported == path + ) )); } } diff --git a/litellm-rust/crates/http/src/error.rs b/litellm-rust/crates/http/src/error.rs index ae14756baa3..29c8b988c0b 100644 --- a/litellm-rust/crates/http/src/error.rs +++ b/litellm-rust/crates/http/src/error.rs @@ -1,4 +1,10 @@ -use std::path::PathBuf; +use std::{borrow::Cow, fmt, io, path::PathBuf, sync::Arc}; + +use tokio::time::error::Elapsed; +use tokio_tungstenite::tungstenite; + +const UPSTREAM_BODY_MAX_CHARS: usize = 256; +const TRUNCATION_MARKER: &str = "... (truncated)"; #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum TlsSource { @@ -6,34 +12,666 @@ pub enum TlsSource { ClientIdentity, } -#[derive(Clone, Debug, thiserror::Error, PartialEq, Eq)] -pub enum Error { - #[error("could not read {}: {message}", path.display())] +#[derive(Clone, Debug, thiserror::Error)] +#[error(transparent)] +pub struct Error(Arc); + +#[derive(Debug, thiserror::Error)] +pub(crate) enum Repr { + #[error("could not read {}", path.display())] Read { path: PathBuf, - message: String, tls_source: TlsSource, + source: io::Error, }, - #[error("{} is not a PEM file: {message}", path.display())] + #[error("could not load {}", path.display())] InvalidPem { path: PathBuf, - message: String, tls_source: TlsSource, + source: PemFault, }, - #[error("could not build the HTTP client: {0}")] - Client(String), - #[error("request body could not be serialized: {0}")] - RequestBody(String), - #[error("invalid request: {0}")] - Request(String), - #[error("request forwards a header the signer computes: {0}")] + #[error("could not build the HTTP client")] + Client(#[source] reqwest::Error), + #[error("could not build the TLS configuration")] + TlsConfig(#[source] rustls::Error), + + #[error("request body could not be serialized")] + RequestBody(#[source] serde_json::Error), + #[error("invalid request")] + Request(#[source] http::Error), + #[error("invalid request")] + RequestUrl(#[source] reqwest::Error), + #[error("invalid request URL")] + ParseUrl(#[source] url::ParseError), + #[error("request forwards the header {0:?}, which the signer computes")] ComputedHeader(String), - #[error("request signing failed: {0}")] - Signature(String), + #[error("request signing failed: a streaming body cannot be signed")] + UnsignableBody, + #[error("request forwards a header that cannot be signed")] + UnsignableHeader(#[source] litellm_error::Source), + #[error("request signing failed")] + Signature(#[source] litellm_error::Source), + + #[error("could not reach the provider")] + Connect(#[source] reqwest::Error), + #[error("upstream network error")] + Network(#[source] reqwest::Error), + #[error("upstream request timed out")] + Timeout(#[source] reqwest::Error), + #[error("upstream request failed with status {status}: {body}")] + Status { status: u16, body: UpstreamBody }, + + #[error("invalid WebSocket URL")] + WebSocketUrl(#[source] url::ParseError), + #[error("unsupported WebSocket URL scheme {0:?}")] + WebSocketScheme(String), + #[error("could not rewrite the WebSocket URL scheme to {0}")] + WebSocketSchemeRewrite(&'static str), + #[error("WebSocket handshake returned {0}")] + Handshake(HandshakeFault), + #[error("WebSocket connection timed out")] + HandshakeTimeout(#[source] Elapsed), + #[error("upstream WebSocket error")] + WebSocket(#[source] tungstenite::Error), + #[error("the WebSocket connection is closed")] + WebSocketClosed, + + #[error("media URL rejected by network policy")] + BlockedUrl, + #[error("could not resolve the media host")] + Resolve(#[source] io::Error), + #[error("media download is disabled")] + DownloadDisabled, + #[error("media download exceeds the maximum size")] + DownloadTooLarge, + #[error("too many redirects while fetching media")] + TooManyRedirects, + #[error("media redirect is missing a Location header")] + MissingRedirectLocation, + #[error("invalid media redirect")] + InvalidRedirect(#[source] RedirectFault), + #[error("media download failed with status {0}")] + DownloadStatus(u16), + #[error("media download failed")] + Download(#[source] reqwest::Error), + #[error("media download timed out")] + DownloadTimeout(#[source] Elapsed), +} + +#[derive(Debug, thiserror::Error)] +pub(crate) enum HandshakeFault { + #[error("a wrong Sec-WebSocket-Accept")] + Accept, + #[error("an invalid Upgrade header")] + Upgrade, + #[error("an invalid Connection header")] + Connection, + #[error("an invalid subprotocol")] + Subprotocol, +} + +#[derive(Debug, thiserror::Error)] +pub(crate) enum PemFault { + #[error("no certificates found")] + NoCertificates, + #[error("could not parse PEM")] + Parse(#[from] rustls::pki_types::pem::Error), + #[error("rustls rejected the certificate or key")] + Rejected(#[from] rustls::Error), +} + +#[derive(Debug, thiserror::Error)] +pub(crate) enum RedirectFault { + #[error("Location header is not valid text")] + Location(#[from] http::header::ToStrError), + #[error("Location is not a valid URL")] + Url(#[from] url::ParseError), +} + +impl Repr { + pub(crate) fn download(error: reqwest::Error) -> Self { + Self::Download(error.without_url()) + } +} + +impl Error { + pub fn upstream(status: u16, body: String) -> Self { + Repr::Status { + status, + body: UpstreamBody(body), + } + .into() + } + + pub fn computed_header(name: impl Into) -> Self { + Repr::ComputedHeader(name.into()).into() + } + + pub fn unsignable_body() -> Self { + Repr::UnsignableBody.into() + } + + pub fn unsignable_header(cause: impl std::error::Error + Send + Sync + 'static) -> Self { + Repr::UnsignableHeader(litellm_error::Source::new(cause)).into() + } + + pub fn websocket_closed() -> Self { + Repr::WebSocketClosed.into() + } + + pub fn signature(cause: impl std::error::Error + Send + Sync + 'static) -> Self { + Repr::Signature(litellm_error::Source::new(cause)).into() + } + + /// The request may have reached the provider, so retrying elsewhere could + /// repeat a billed call. + pub fn dispatched(&self) -> bool { + match &*self.0 { + Repr::Network(_) + | Repr::Timeout(_) + | Repr::Status { .. } + | Repr::Handshake(_) + | Repr::HandshakeTimeout(_) + | Repr::WebSocket(_) => true, + Repr::WebSocketClosed + | Repr::Read { .. } + | Repr::InvalidPem { .. } + | Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::ComputedHeader(_) + | Repr::UnsignableBody + | Repr::UnsignableHeader(_) + | Repr::Signature(_) + | Repr::Connect(_) + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::BlockedUrl + | Repr::Resolve(_) + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects + | Repr::MissingRedirectLocation + | Repr::InvalidRedirect(_) + | Repr::DownloadStatus(_) + | Repr::Download(_) + | Repr::DownloadTimeout(_) => false, + } + } + + pub fn is_request(&self) -> bool { + match &*self.0 { + Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::ComputedHeader(_) + | Repr::UnsignableHeader(_) + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::BlockedUrl + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects => true, + Repr::Read { .. } + | Repr::InvalidPem { .. } + | Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::UnsignableBody + | Repr::Signature(_) + | Repr::Connect(_) + | Repr::Network(_) + | Repr::Timeout(_) + | Repr::Status { .. } + | Repr::Handshake(_) + | Repr::HandshakeTimeout(_) + | Repr::WebSocket(_) + | Repr::WebSocketClosed + | Repr::Resolve(_) + | Repr::MissingRedirectLocation + | Repr::InvalidRedirect(_) + | Repr::DownloadStatus(_) + | Repr::Download(_) + | Repr::DownloadTimeout(_) => false, + } + } + + pub fn is_response(&self) -> bool { + match &*self.0 { + Repr::MissingRedirectLocation | Repr::InvalidRedirect(_) => true, + Repr::Read { .. } + | Repr::InvalidPem { .. } + | Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::ComputedHeader(_) + | Repr::UnsignableBody + | Repr::UnsignableHeader(_) + | Repr::Signature(_) + | Repr::Connect(_) + | Repr::Network(_) + | Repr::Timeout(_) + | Repr::Status { .. } + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::Handshake(_) + | Repr::HandshakeTimeout(_) + | Repr::WebSocket(_) + | Repr::WebSocketClosed + | Repr::BlockedUrl + | Repr::Resolve(_) + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects + | Repr::DownloadStatus(_) + | Repr::Download(_) + | Repr::DownloadTimeout(_) => false, + } + } + + pub fn is_computed_header(&self) -> bool { + match &*self.0 { + Repr::ComputedHeader(_) => true, + Repr::Read { .. } + | Repr::InvalidPem { .. } + | Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::UnsignableBody + | Repr::UnsignableHeader(_) + | Repr::Signature(_) + | Repr::Connect(_) + | Repr::Network(_) + | Repr::Timeout(_) + | Repr::Status { .. } + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::Handshake(_) + | Repr::HandshakeTimeout(_) + | Repr::WebSocket(_) + | Repr::WebSocketClosed + | Repr::BlockedUrl + | Repr::Resolve(_) + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects + | Repr::MissingRedirectLocation + | Repr::InvalidRedirect(_) + | Repr::DownloadStatus(_) + | Repr::Download(_) + | Repr::DownloadTimeout(_) => false, + } + } + + pub fn is_connect(&self) -> bool { + match &*self.0 { + Repr::Connect(_) => true, + Repr::ComputedHeader(_) + | Repr::Read { .. } + | Repr::InvalidPem { .. } + | Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::UnsignableBody + | Repr::UnsignableHeader(_) + | Repr::Signature(_) + | Repr::Network(_) + | Repr::Timeout(_) + | Repr::Status { .. } + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::Handshake(_) + | Repr::HandshakeTimeout(_) + | Repr::WebSocket(_) + | Repr::WebSocketClosed + | Repr::BlockedUrl + | Repr::Resolve(_) + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects + | Repr::MissingRedirectLocation + | Repr::InvalidRedirect(_) + | Repr::DownloadStatus(_) + | Repr::Download(_) + | Repr::DownloadTimeout(_) => false, + } + } + + pub fn is_timeout(&self) -> bool { + match &*self.0 { + Repr::Timeout(_) | Repr::HandshakeTimeout(_) | Repr::DownloadTimeout(_) => true, + Repr::Read { .. } + | Repr::InvalidPem { .. } + | Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::ComputedHeader(_) + | Repr::UnsignableBody + | Repr::UnsignableHeader(_) + | Repr::Signature(_) + | Repr::Connect(_) + | Repr::Network(_) + | Repr::Status { .. } + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::Handshake(_) + | Repr::WebSocket(_) + | Repr::WebSocketClosed + | Repr::BlockedUrl + | Repr::Resolve(_) + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects + | Repr::MissingRedirectLocation + | Repr::InvalidRedirect(_) + | Repr::DownloadStatus(_) + | Repr::Download(_) => false, + } + } + + pub fn upstream_response(&self) -> Option<(u16, &str)> { + match &*self.0 { + Repr::Status { status, body } => Some((*status, &body.0)), + Repr::Read { .. } + | Repr::InvalidPem { .. } + | Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::ComputedHeader(_) + | Repr::UnsignableBody + | Repr::UnsignableHeader(_) + | Repr::Signature(_) + | Repr::Connect(_) + | Repr::Network(_) + | Repr::Timeout(_) + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::Handshake(_) + | Repr::HandshakeTimeout(_) + | Repr::WebSocket(_) + | Repr::WebSocketClosed + | Repr::BlockedUrl + | Repr::Resolve(_) + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects + | Repr::MissingRedirectLocation + | Repr::InvalidRedirect(_) + | Repr::DownloadStatus(_) + | Repr::Download(_) + | Repr::DownloadTimeout(_) => None, + } + } + + pub fn status(&self) -> Option { + match &*self.0 { + Repr::Status { status, .. } | Repr::DownloadStatus(status) => Some(*status), + Repr::Timeout(_) | Repr::DownloadTimeout(_) => Some(408), + Repr::Read { .. } + | Repr::InvalidPem { .. } + | Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::ComputedHeader(_) + | Repr::UnsignableBody + | Repr::UnsignableHeader(_) + | Repr::Signature(_) + | Repr::Connect(_) + | Repr::Network(_) + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::Handshake(_) + | Repr::HandshakeTimeout(_) + | Repr::WebSocket(_) + | Repr::WebSocketClosed + | Repr::BlockedUrl + | Repr::Resolve(_) + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects + | Repr::MissingRedirectLocation + | Repr::InvalidRedirect(_) + | Repr::Download(_) => None, + } + } + + pub fn tls_source(&self) -> Option { + match &*self.0 { + Repr::Read { tls_source, .. } | Repr::InvalidPem { tls_source, .. } => { + Some(*tls_source) + } + Repr::Client(_) + | Repr::TlsConfig(_) + | Repr::RequestBody(_) + | Repr::Request(_) + | Repr::RequestUrl(_) + | Repr::ParseUrl(_) + | Repr::ComputedHeader(_) + | Repr::UnsignableBody + | Repr::UnsignableHeader(_) + | Repr::Signature(_) + | Repr::Connect(_) + | Repr::Network(_) + | Repr::Timeout(_) + | Repr::Status { .. } + | Repr::WebSocketUrl(_) + | Repr::WebSocketScheme(_) + | Repr::WebSocketSchemeRewrite(_) + | Repr::Handshake(_) + | Repr::HandshakeTimeout(_) + | Repr::WebSocket(_) + | Repr::WebSocketClosed + | Repr::BlockedUrl + | Repr::Resolve(_) + | Repr::DownloadDisabled + | Repr::DownloadTooLarge + | Repr::TooManyRedirects + | Repr::MissingRedirectLocation + | Repr::InvalidRedirect(_) + | Repr::DownloadStatus(_) + | Repr::Download(_) + | Repr::DownloadTimeout(_) => None, + } + } + + #[cfg(test)] + pub(crate) fn repr(&self) -> &Repr { + &self.0 + } +} + +pub(crate) struct UpstreamBody(String); + +impl fmt::Display for UpstreamBody { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Display::fmt(&bounded(&self.0), f) + } +} + +impl fmt::Debug for UpstreamBody { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Debug::fmt(&bounded(&self.0), f) + } +} + +fn bounded(body: &str) -> Cow<'_, str> { + match body.char_indices().nth(UPSTREAM_BODY_MAX_CHARS) { + None => Cow::Borrowed(body), + Some((cut, _)) => Cow::Owned(format!("{}{TRUNCATION_MARKER}", &body[..cut])), + } +} + +impl PartialEq for Error { + fn eq(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.0, &other.0) + } +} + +impl Eq for Error {} + +impl From for Error { + fn from(repr: Repr) -> Self { + Self(Arc::new(repr)) + } } impl From for Error { fn from(error: reqwest::Error) -> Self { - Self::Client(error.without_url().to_string()) + let error = error.without_url(); + if error.is_timeout() { + Repr::Timeout(error).into() + } else if error.is_connect() || error.is_builder() { + Repr::Connect(error).into() + } else { + Repr::Network(error).into() + } + } +} + +impl From for Error { + fn from(error: tungstenite::Error) -> Self { + Repr::WebSocket(error).into() + } +} + +#[cfg(test)] +mod tests { + use litellm_error::Report; + use rstest::rstest; + + use super::{TRUNCATION_MARKER, UPSTREAM_BODY_MAX_CHARS}; + + #[tokio::test] + async fn transport_errors_remove_urls_and_keep_dispatch_context() { + let error = reqwest::Client::builder() + .no_proxy() + .build() + .expect("client") + .get("http://localhost:invalid/private?api_key=secret") + .send() + .await + .expect_err("invalid port"); + let error = crate::Error::from(error); + let report = Report::new(&error).to_string(); + assert!(!error.dispatched()); + assert!(!report.contains("secret"), "{report}"); + assert!(!report.contains("private"), "{report}"); + } + + fn root_cause<'a>( + error: &'a (dyn std::error::Error + 'static), + ) -> &'a (dyn std::error::Error + 'static) { + error.source().map_or(error, root_cause) + } + + #[tokio::test] + async fn connect_failure_reports_its_root_cause_once() { + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind"); + let address = listener.local_addr().expect("address"); + drop(listener); + let error = reqwest::Client::builder() + .no_proxy() + .build() + .expect("client") + .get(format!("http://{address}/private?api_key=secret")) + .send() + .await + .expect_err("nothing listens on the port"); + let error = crate::Error::from(error); + let root_cause = root_cause(&error).to_string(); + let report = Report::new(&error).to_string(); + assert_eq!(report.matches(&root_cause).count(), 1, "{report}"); + assert!(!error.to_string().contains(&root_cause)); + assert!(error.is_connect()); + assert!(!error.is_timeout()); + assert!(!report.contains("secret"), "{report}"); + assert!(!report.contains("private"), "{report}"); + } + + #[tokio::test] + async fn request_timeout_is_not_safe_to_retry_as_a_connect_failure() { + use std::time::Duration; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind"); + let address = listener.local_addr().expect("address"); + let request = reqwest::Client::builder() + .no_proxy() + .build() + .expect("client") + .get(format!("http://{address}")) + .timeout(Duration::from_millis(200)) + .send(); + let (response, accepted) = tokio::join!( + request, + tokio::time::timeout(Duration::from_secs(2), listener.accept()) + ); + let _connection = accepted + .expect("accept deadline") + .expect("accepted connection"); + let error = response.expect_err("server does not respond"); + assert!(error.is_timeout()); + let error = crate::Error::from(error); + assert!(error.dispatched()); + assert!(error.is_timeout()); + assert!(!error.is_connect()); + assert_eq!(error.status(), Some(408)); + } + + #[rstest] + #[case::empty(String::new(), String::new())] + #[case::short("slow down".to_owned(), "slow down".to_owned())] + #[case::ascii_over_the_limit( + format!("{}private-tail", "a".repeat(UPSTREAM_BODY_MAX_CHARS)), + format!("{}{TRUNCATION_MARKER}", "a".repeat(UPSTREAM_BODY_MAX_CHARS)) + )] + #[case::at_the_limit("\u{00e9}".repeat(UPSTREAM_BODY_MAX_CHARS), "\u{00e9}".repeat(UPSTREAM_BODY_MAX_CHARS))] + #[case::over_the_limit( + format!("{}private-tail", "\u{00e9}".repeat(UPSTREAM_BODY_MAX_CHARS)), + format!("{}{TRUNCATION_MARKER}", "\u{00e9}".repeat(UPSTREAM_BODY_MAX_CHARS)) + )] + fn upstream_preserves_the_body_and_bounds_only_diagnostics( + #[case] body: String, + #[case] expected: String, + ) { + let error = crate::Error::upstream(429, body.clone()); + let expected_message = format!("upstream request failed with status 429: {expected}"); + assert_eq!(error.to_string(), expected_message); + assert_eq!(Report::new(&error).to_string(), expected_message); + let debug = format!("{error:?}"); + assert!(debug.contains(&format!("{expected:?}")), "{debug}"); + assert!(!debug.contains("private-tail"), "{debug}"); + assert!(!format!("{error:#?}").contains("private-tail")); + assert_eq!(error.upstream_response(), Some((429, body.as_str()))); + assert_eq!(error.status(), Some(429)); } } diff --git a/litellm-rust/crates/http/src/lib.rs b/litellm-rust/crates/http/src/lib.rs index ef72420af9b..02129c62083 100644 --- a/litellm-rust/crates/http/src/lib.rs +++ b/litellm-rust/crates/http/src/lib.rs @@ -14,7 +14,6 @@ mod proxy; pub mod request; mod settings; mod tls; -pub mod transport; pub mod websocket; pub use client::Client; diff --git a/litellm-rust/crates/http/src/media.rs b/litellm-rust/crates/http/src/media.rs index 960a1d4044a..4acaca5e502 100644 --- a/litellm-rust/crates/http/src/media.rs +++ b/litellm-rust/crates/http/src/media.rs @@ -12,29 +12,7 @@ use reqwest::{ dns::{Addrs, Name, Resolve, Resolving}, }; -use crate::{Client, ClientVariant, HttpClientConfig, HttpClientPool}; - -#[derive(Debug, thiserror::Error)] -pub enum Error { - #[error("media URL rejected by network policy")] - BlockedUrl, - #[error("media download is disabled")] - DownloadDisabled, - #[error("media download exceeds the maximum size")] - DownloadTooLarge, - #[error("too many redirects while fetching media")] - TooManyRedirects, - #[error("media redirect is missing a Location header")] - MissingRedirectLocation, - #[error("invalid media redirect")] - InvalidRedirect, - #[error("media download failed with status {0}")] - Http(u16), - #[error("media download timed out")] - Timeout, - #[error("{0}")] - Transport(#[from] crate::transport::Error), -} +use crate::{Client, ClientVariant, Error, HttpClientConfig, HttpClientPool, error::Repr}; #[derive(Clone, Debug, PartialEq, Eq)] pub struct UrlPolicy { @@ -125,7 +103,7 @@ impl MediaFetcher { pool: &HttpClientPool, config: &HttpClientConfig, url_policy: UrlPolicy, - ) -> Result { + ) -> Result { let uses_proxy: ProxyMatch = Arc::new(config.proxies.matcher()); Self::with_resolution( pool, @@ -142,7 +120,7 @@ impl MediaFetcher { url_policy: UrlPolicy, address_resolver: Arc, uses_proxy: ProxyMatch, - ) -> Result { + ) -> Result { Ok(Self { pinned: pool.client(config, ClientVariant::Media)?, unpinned: pool.client(config, ClientVariant::UnpinnedMedia)?, @@ -167,11 +145,11 @@ impl MediaFetcher { pub async fn fetch(&self, url: Url, policy: DownloadPolicy) -> Result { if policy.max_bytes == 0 { - return Err(Error::DownloadDisabled); + return Err(Repr::DownloadDisabled.into()); } tokio::time::timeout(policy.timeout, self.fetch_before_deadline(url, policy)) .await - .map_err(|_| Error::Timeout)? + .map_err(Repr::DownloadTimeout)? } async fn fetch_before_deadline( @@ -187,22 +165,25 @@ impl MediaFetcher { .get(url.clone()) .send() .await - .map_err(crate::transport::Error::from)?; + .map_err(Repr::download)?; if response.status().is_redirection() { if redirects_followed == policy.max_redirects { - return Err(Error::TooManyRedirects); + return Err(Repr::TooManyRedirects.into()); } let location = response .headers() .get(reqwest::header::LOCATION) - .and_then(|value| value.to_str().ok()) - .ok_or(Error::MissingRedirectLocation)?; - url = url.join(location).map_err(|_| Error::InvalidRedirect)?; + .ok_or(Repr::MissingRedirectLocation)? + .to_str() + .map_err(|error| Repr::InvalidRedirect(error.into()))?; + url = url + .join(location) + .map_err(|error| Repr::InvalidRedirect(error.into()))?; redirects_followed += 1; continue; } if !response.status().is_success() { - return Err(Error::Http(response.status().as_u16())); + return Err(Repr::DownloadStatus(response.status().as_u16()).into()); } enforce_download_size(response.content_length().unwrap_or(0), policy.max_bytes)?; let content_type = response @@ -215,11 +196,7 @@ impl MediaFetcher { .unwrap_or("application/octet-stream") .to_string(); let mut bytes = Vec::new(); - while let Some(chunk) = response - .chunk() - .await - .map_err(crate::transport::Error::from)? - { + while let Some(chunk) = response.chunk().await.map_err(Repr::download)? { enforce_download_size(bytes.len() as u64 + chunk.len() as u64, policy.max_bytes)?; bytes.extend_from_slice(&chunk); } @@ -238,13 +215,13 @@ impl MediaFetcher { || !url.username().is_empty() || url.password().is_some() { - return Err(Error::BlockedUrl); + return Err(Repr::BlockedUrl.into()); } - let host = url.host_str().ok_or(Error::BlockedUrl)?; + let host = url.host_str().ok_or(Repr::BlockedUrl)?; if self.allow_private_network { return Ok(&self.pinned); } - let port = url.port_or_known_default().ok_or(Error::BlockedUrl)?; + let port = url.port_or_known_default().ok_or(Repr::BlockedUrl)?; if self.url_policy.allows(host, port) { return Ok(&self.unpinned); } @@ -262,27 +239,29 @@ impl MediaFetcher { .trim_end_matches(']') .parse::() { - return (!is_blocked_ip(ip)).then_some(()).ok_or(Error::BlockedUrl); + return (!is_blocked_ip(ip)) + .then_some(()) + .ok_or(Repr::BlockedUrl.into()); } let addresses = self .address_resolver .resolve(host, port) .await - .map_err(|error| crate::transport::Error::Network(error.to_string()))?; + .map_err(Repr::Resolve)?; validate_addresses(&addresses) } } fn enforce_download_size(length: u64, max_bytes: u64) -> Result<(), Error> { if length > max_bytes { - return Err(Error::DownloadTooLarge); + return Err(Repr::DownloadTooLarge.into()); } Ok(()) } fn validate_addresses(addresses: &[SocketAddr]) -> Result<(), Error> { if addresses.is_empty() || addresses.iter().any(|address| is_blocked_ip(address.ip())) { - return Err(Error::BlockedUrl); + return Err(Repr::BlockedUrl.into()); } Ok(()) } @@ -356,10 +335,8 @@ impl Resolve for PublicDnsResolver { .await .map_err(|error| Box::new(error) as Box)? .collect::>(); - validate_addresses(&addresses).map_err(|_| { - Box::new(io::Error::other("destination rejected by network policy")) - as Box - })?; + validate_addresses(&addresses) + .map_err(|error| Box::new(error) as Box)?; Ok(Box::new(addresses.into_iter()) as Addrs) }) } @@ -544,7 +521,7 @@ mod tests { .await .expect_err("oversize body is rejected"); server.await.expect("server completes"); - assert!(matches!(error, Error::DownloadTooLarge)); + assert!(matches!(error.repr(), Repr::DownloadTooLarge)); } #[tokio::test] @@ -559,7 +536,7 @@ mod tests { .await .expect_err("stream crossing limit is rejected"); server.await.expect("server completes"); - assert!(matches!(error, Error::DownloadTooLarge)); + assert!(matches!(error.repr(), Repr::DownloadTooLarge)); } #[tokio::test] @@ -595,7 +572,7 @@ mod tests { .expect_err("private redirect is rejected"); let requests = server.await.expect("server completes"); assert_eq!(requests.len(), 1); - assert!(matches!(error, Error::BlockedUrl)); + assert!(matches!(error.repr(), Repr::BlockedUrl)); } #[tokio::test] @@ -622,7 +599,27 @@ mod tests { .await .expect_err("fetch times out"); server.await.expect("server completes"); - assert!(matches!(error, Error::Timeout)); + assert!(matches!(error.repr(), Repr::DownloadTimeout(_))); + assert!(!error.dispatched()); + } + + #[rstest::rstest] + #[case::closed_before_answering(b"")] + #[case::reset_mid_body(b"HTTP/1.1 200 OK\r\nContent-Length: 10\r\n\r\nab")] + #[tokio::test] + async fn a_failed_download_never_reached_the_provider(#[case] response: &'static [u8]) { + let (url, server) = serve(response).await; + let url = url.join("?sig=secret").expect("valid test URL"); + let error = MediaFetcher::for_test(Client::for_test()) + .fetch(url, policy(10, 0)) + .await + .expect_err("download fails"); + server.await.expect("server completes"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(!error.dispatched()); + assert!(!error.is_request()); + assert!(!report.contains("secret"), "{report}"); + assert!(!report.contains("document"), "{report}"); } #[tokio::test] @@ -652,8 +649,13 @@ mod tests { let url = Url::parse("https://user:password@8.8.8.8/document").expect("credentialed URL parses"); assert!(matches!( - fetcher.fetch(url, policy(1, 0)).await, - Err(Error::BlockedUrl) + fetcher + .fetch(url, policy(1, 0)) + .await + .err() + .as_ref() + .map(Error::repr), + Some(Repr::BlockedUrl) )); } @@ -683,7 +685,10 @@ mod tests { let result = fetcher(UNROUTABLE, HashSet::from(["localhost"]), other_port, false) .fetch(url, policy(2, 0)) .await; - assert!(matches!(result, Err(Error::BlockedUrl))); + assert!(matches!( + result.as_ref().err().map(Error::repr), + Some(Repr::BlockedUrl) + )); } #[test] @@ -742,6 +747,9 @@ mod tests { ) .fetch(url, policy(2, 0)) .await; - assert!(matches!(blocked, Err(Error::BlockedUrl))); + assert!(matches!( + blocked.as_ref().err().map(Error::repr), + Some(Repr::BlockedUrl) + )); } } diff --git a/litellm-rust/crates/http/src/outbound.rs b/litellm-rust/crates/http/src/outbound.rs index d51a1994044..20b0b1be972 100644 --- a/litellm-rust/crates/http/src/outbound.rs +++ b/litellm-rust/crates/http/src/outbound.rs @@ -14,7 +14,7 @@ use reqwest::{ }; use serde::Serialize; -use crate::Error; +use crate::{Error, error::Repr}; /// Returns the headers to add to the request; it never sees a mutable request. pub trait RequestSigner: Send + Sync { @@ -27,8 +27,8 @@ pub fn json( body: &impl Serialize, timeout: Option, ) -> Result { - let body = serde_json::to_vec(body).map_err(|error| Error::RequestBody(error.to_string()))?; - let url = Url::parse(url).map_err(|error| Error::Request(error.to_string()))?; + let body = serde_json::to_vec(body).map_err(Repr::RequestBody)?; + let url = Url::parse(url).map_err(Repr::ParseUrl)?; let with_headers = headers.iter().fold( http::Request::post(url.as_str()), |builder, (name, value)| builder.header(name, value), @@ -41,17 +41,15 @@ pub fn json( } else { with_headers.header(CONTENT_TYPE, "application/json") }; - let mut request = builder - .body(body) - .map_err(|error| Error::Request(error.to_string())) - .and_then(|request| { - Request::try_from(request) - .map_err(|error| Error::Request(error.without_url().to_string())) - })?; + let mut request = from_http(builder.body(body).map_err(Repr::Request)?)?; *request.timeout_mut() = timeout; Ok(request) } +pub fn from_http>(request: http::Request) -> Result { + Request::try_from(request).map_err(|error| Repr::RequestUrl(error.without_url()).into()) +} + pub fn signed(mut request: Request, signer: &dyn RequestSigner) -> Result { let signature = signer.sign(&request)?; request.headers_mut().extend(signature); @@ -114,7 +112,7 @@ mod tests { .headers() .contains_key(CONTENT_TYPE) .then(HeaderMap::new) - .ok_or_else(|| Error::Signature("content-type was not signed".into())) + .ok_or_else(|| Error::signature(std::fmt::Error)) } } @@ -148,7 +146,7 @@ mod tests { struct Refuses; impl RequestSigner for Refuses { fn sign(&self, _request: &Request) -> Result { - Err(Error::ComputedHeader("authorization".into())) + Err(Error::computed_header("authorization")) } } @@ -157,7 +155,7 @@ mod tests { &Refuses, ) .unwrap_err(); - assert_eq!(error, Error::ComputedHeader("authorization".into())); + assert!(error.is_computed_header()); } #[rstest] @@ -230,7 +228,21 @@ mod tests { ); assert!(matches!( json(url, &[("x-caller".into(), header.into())], &json!({}), None), - Err(Error::Request(_)) + Err(error) if error.is_request() )); } + + #[test] + fn a_relative_http_request_is_the_callers_fault_and_never_reached_the_provider() { + let request = http::Request::get("/private?api_key=secret") + .body(Vec::::new()) + .unwrap(); + + let error = from_http(request).unwrap_err(); + + assert!(error.is_request()); + assert!(!error.dispatched()); + let report = litellm_error::Report::new(&error).to_string(); + assert!(!report.contains("secret"), "{report}"); + } } diff --git a/litellm-rust/crates/http/src/pool.rs b/litellm-rust/crates/http/src/pool.rs index bf892b09921..3bdc63453ef 100644 --- a/litellm-rust/crates/http/src/pool.rs +++ b/litellm-rust/crates/http/src/pool.rs @@ -9,7 +9,7 @@ use reqwest::dns::Resolve; use crate::{ client::Client, config::{HttpClientConfig, Verify}, - error::Error, + error::{Error, Repr}, proxy::EnvironmentProxies, }; @@ -83,7 +83,8 @@ impl HttpClientPool { } let client = self .apply(variant, reqwest::ClientBuilder::try_from(&key.0)?) - .build()?; + .build() + .map_err(Repr::Client)?; self.lock().insert( key, PooledClient { diff --git a/litellm-rust/crates/http/src/request.rs b/litellm-rust/crates/http/src/request.rs index 5a3dd19ade2..11e19f7fa24 100644 --- a/litellm-rust/crates/http/src/request.rs +++ b/litellm-rust/crates/http/src/request.rs @@ -9,9 +9,7 @@ pub struct HeaderError { use litellm_core_utils::core_helpers::json_type_name; use serde_json::{Map, Value}; -/// Max characters of an upstream error body echoed across the call boundary -/// before truncation, so provider bodies are bounded and data-minimized. -const UPSTREAM_ERROR_BODY_MAX_CHARS: usize = 256; +use crate::Error; pub enum HeaderPolicy<'a> { All, @@ -40,17 +38,15 @@ pub fn with_headers( }) } -pub async fn http_request( - request: reqwest::RequestBuilder, -) -> Result { - request.send().await +pub async fn http_request(request: reqwest::RequestBuilder) -> Result { + Ok(request.send().await?) } pub async fn execute_http_request( client: &reqwest::Client, request: reqwest::Request, -) -> Result { - client.execute(request).await +) -> Result { + Ok(client.execute(request).await?) } pub fn header_pairs(headers: &reqwest::header::HeaderMap) -> Vec<(String, String)> { @@ -65,14 +61,6 @@ pub fn header_pairs(headers: &reqwest::header::HeaderMap) -> Vec<(String, String .collect() } -pub fn truncate_error_body(body: &str) -> String { - if body.chars().count() <= UPSTREAM_ERROR_BODY_MAX_CHARS { - return body.to_string(); - } - let truncated: String = body.chars().take(UPSTREAM_ERROR_BODY_MAX_CHARS).collect(); - format!("{truncated}... (truncated)") -} - pub fn string_headers( context: &'static str, extra_headers: Option>, @@ -215,22 +203,6 @@ mod tests { assert_ne!(request.headers()["content-length"], "0"); } - #[test] - fn truncate_leaves_short_bodies_untouched() { - assert_eq!(truncate_error_body("short"), "short"); - } - - #[test] - fn truncate_bounds_long_bodies_by_characters() { - let body = "\u{00e9}".repeat(UPSTREAM_ERROR_BODY_MAX_CHARS + 10); - let truncated = truncate_error_body(&body); - assert!(truncated.ends_with("... (truncated)")); - assert_eq!( - truncated.chars().count(), - UPSTREAM_ERROR_BODY_MAX_CHARS + "... (truncated)".chars().count() - ); - } - #[test] fn string_headers_rejects_non_string_values() { let headers = Map::from_iter([("x-trace".to_string(), json!(7))]); diff --git a/litellm-rust/crates/http/src/tls.rs b/litellm-rust/crates/http/src/tls.rs index a52ea31af59..a83b0018b2b 100644 --- a/litellm-rust/crates/http/src/tls.rs +++ b/litellm-rust/crates/http/src/tls.rs @@ -1,4 +1,4 @@ -use std::{fmt, path::Path, str::FromStr, sync::Arc}; +use std::{path::Path, str::FromStr, sync::Arc}; use rustls::{ CipherSuite, ClientConfig, DigitallySignedStruct, RootCertStore, SignatureScheme, @@ -9,7 +9,7 @@ use rustls::{ use crate::{ config::{ClientIdentity, HttpClientConfig, Verify}, - error::{Error, TlsSource}, + error::{Error, PemFault, Repr, TlsSource}, }; #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] @@ -187,7 +187,7 @@ impl TryFrom<&HttpClientConfig> for ClientConfig { }); let builder = ClientConfig::builder_with_provider(Arc::clone(&provider)) .with_safe_default_protocol_versions() - .map_err(|error| Error::Client(error.to_string()))?; + .map_err(Repr::TlsConfig)?; let verified = match &config.verify { Verify::Disabled => builder .dangerous() @@ -230,7 +230,7 @@ impl TryFrom<&HttpClientConfig> for ClientConfig { fn bundle_roots(path: &Path, source: TlsSource) -> Result { let certificates = certificates(path, source)?; if certificates.is_empty() { - return Err(invalid_pem(path, source, "no certificates found")); + return Err(invalid_pem(path, source, PemFault::NoCertificates)); } let mut store = RootCertStore::empty(); for certificate in certificates { @@ -248,7 +248,7 @@ fn client_identity( let source = TlsSource::ClientIdentity; let chain = certificates(certificate, source)?; if chain.is_empty() { - return Err(invalid_pem(certificate, source, "no certificates found")); + return Err(invalid_pem(certificate, source, PemFault::NoCertificates)); } let private_key = PrivateKeyDer::from_pem_slice(&read(key, source)?) .map_err(|error| invalid_pem(key, source, error))?; @@ -262,19 +262,23 @@ fn certificates(path: &Path, source: TlsSource) -> Result Result, Error> { - std::fs::read(path).map_err(|error| Error::Read { - path: path.to_path_buf(), - message: error.to_string(), - tls_source: source, + std::fs::read(path).map_err(|error| { + Repr::Read { + path: path.to_path_buf(), + tls_source: source, + source: error, + } + .into() }) } -fn invalid_pem(path: &Path, source: TlsSource, message: impl fmt::Display) -> Error { - Error::InvalidPem { +fn invalid_pem(path: &Path, tls_source: TlsSource, fault: impl Into) -> Error { + Repr::InvalidPem { path: path.to_path_buf(), - message: message.to_string(), - tls_source: source, + tls_source, + source: fault.into(), } + .into() } #[derive(Debug)] @@ -454,11 +458,10 @@ mod tests { std::fs::remove_file(&path).unwrap(); assert!(matches!( result, - Err(Error::InvalidPem { - path: reported, - tls_source: TlsSource::ClientIdentity, - .. - }) if reported == path + Err(error) if matches!( + error.repr(), + Repr::InvalidPem { path: reported, tls_source: TlsSource::ClientIdentity, .. } if *reported == path + ) )); } diff --git a/litellm-rust/crates/http/src/transport.rs b/litellm-rust/crates/http/src/transport.rs deleted file mode 100644 index 8814925bbf2..00000000000 --- a/litellm-rust/crates/http/src/transport.rs +++ /dev/null @@ -1,109 +0,0 @@ -#[derive(Clone, Debug, thiserror::Error, PartialEq, Eq)] -pub enum Error { - #[error("upstream request failed with status {status}: {body}")] - Http { status: u16, body: String }, - #[error("upstream network error: {0}")] - Network(String), - #[error("could not reach the provider: {0}")] - Connect(String), -} - -impl Error { - pub fn from_reqwest_before_dispatch(error: reqwest::Error) -> Self { - let before_dispatch = !error.is_timeout() && (error.is_connect() || error.is_builder()); - let message = describe(error); - if before_dispatch { - Self::Connect(message) - } else { - Self::Network(message) - } - } -} - -impl From for Error { - fn from(error: reqwest::Error) -> Self { - Self::Network(describe(error)) - } -} - -fn describe(error: reqwest::Error) -> String { - let error = error.without_url(); - std::iter::successors(std::error::Error::source(&error), |cause| cause.source()) - .fold(error.to_string(), |message, cause| { - format!("{message}: {cause}") - }) -} - -#[cfg(test)] -mod tests { - #[tokio::test] - async fn transport_errors_remove_urls_and_keep_dispatch_context() { - let error = reqwest::Client::builder() - .no_proxy() - .build() - .expect("client") - .get("http://localhost:invalid/private?api_key=secret") - .send() - .await - .expect_err("invalid port"); - let error = crate::transport::Error::from_reqwest_before_dispatch(error); - assert!(matches!(error, crate::transport::Error::Connect(_))); - assert!(!error.to_string().contains("secret")); - assert!(!error.to_string().contains("private")); - } - - fn root_cause(error: &dyn std::error::Error) -> Option { - match error.source() { - Some(cause) => root_cause(cause).or_else(|| Some(cause.to_string())), - None => None, - } - } - - #[tokio::test] - async fn network_error_message_names_the_underlying_cause() { - let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind"); - let address = listener.local_addr().expect("address"); - drop(listener); - let error = reqwest::Client::builder() - .no_proxy() - .build() - .expect("client") - .get(format!("http://{address}/private?api_key=secret")) - .send() - .await - .expect_err("nothing listens on the port"); - let root_cause = root_cause(&error).expect("reqwest reports a cause"); - let message = crate::transport::Error::from(error).to_string(); - assert!(message.contains(&root_cause), "{message}"); - assert!(!message.contains("secret")); - } - - #[tokio::test] - async fn request_timeout_is_not_safe_to_retry_as_a_connect_failure() { - use std::time::Duration; - let listener = tokio::net::TcpListener::bind("127.0.0.1:0") - .await - .expect("bind"); - let address = listener.local_addr().expect("address"); - let request = reqwest::Client::builder() - .no_proxy() - .build() - .expect("client") - .get(format!("http://{address}")) - .timeout(Duration::from_millis(200)) - .send(); - let (response, accepted) = tokio::join!( - request, - tokio::time::timeout(Duration::from_secs(2), listener.accept()) - ); - let _connection = accepted - .expect("accept deadline") - .expect("accepted connection"); - let error = response.expect_err("server does not respond"); - assert!(error.is_timeout()); - assert!(matches!( - crate::transport::Error::from_reqwest_before_dispatch(error), - crate::transport::Error::Network(_) - )); - } -} diff --git a/litellm-rust/crates/http/src/websocket.rs b/litellm-rust/crates/http/src/websocket.rs index 685c79cf3a1..b2a6571e2b8 100644 --- a/litellm-rust/crates/http/src/websocket.rs +++ b/litellm-rust/crates/http/src/websocket.rs @@ -1,3 +1,5 @@ +use std::time::Duration; + use http::{ HeaderMap, HeaderValue, StatusCode, header::{ @@ -14,7 +16,10 @@ use tokio_tungstenite::{ }, }; -use crate::{client::Client, transport::Error}; +use crate::{ + client::Client, + error::{Error, HandshakeFault, Repr}, +}; pub type UpstreamWebSocket = WebSocketStream; @@ -23,6 +28,20 @@ pub async fn connect( client: &Client, url: &str, headers: HeaderMap, + timeout: Option, +) -> Result { + match timeout { + Some(timeout) => tokio::time::timeout(timeout, handshake(client, url, headers)) + .await + .map_err(Repr::HandshakeTimeout)?, + None => handshake(client, url, headers).await, + } +} + +async fn handshake( + client: &Client, + url: &str, + headers: HeaderMap, ) -> Result { let offered_protocols = headers.get(SEC_WEBSOCKET_PROTOCOL).cloned(); let key = generate_key(); @@ -30,14 +49,13 @@ pub async fn connect( .get(http_url(url)?) .headers(handshake_headers(headers, &key)?) .send() - .await - .map_err(Error::from_reqwest_before_dispatch)?; + .await?; let status = response.status(); if status != StatusCode::SWITCHING_PROTOCOLS { - return Err(Error::Http { - status: status.as_u16(), - body: response.text().await.unwrap_or_default(), - }); + return Err(Error::upstream( + status.as_u16(), + response.text().await.unwrap_or_default(), + )); } let expected = derive_accept_key(key.as_bytes()); if response @@ -46,9 +64,7 @@ pub async fn connect( .map(HeaderValue::as_bytes) != Some(expected.as_bytes()) { - return Err(Error::Network( - "WebSocket handshake returned a wrong Sec-WebSocket-Accept".into(), - )); + return Err(Repr::Handshake(HandshakeFault::Accept).into()); } validate_response_headers(response.headers(), offered_protocols.as_ref())?; let upgraded = response.upgrade().await?; @@ -63,9 +79,7 @@ fn validate_response_headers( .get(UPGRADE) .is_some_and(|value| value.as_bytes().eq_ignore_ascii_case(b"websocket")) { - return Err(Error::Network( - "WebSocket handshake returned an invalid Upgrade header".into(), - )); + return Err(Repr::Handshake(HandshakeFault::Upgrade).into()); } if !headers.get_all(CONNECTION).iter().any(|value| { value.to_str().is_ok_and(|value| { @@ -74,9 +88,7 @@ fn validate_response_headers( .any(|token| token.trim().eq_ignore_ascii_case("upgrade")) }) }) { - return Err(Error::Network( - "WebSocket handshake returned an invalid Connection header".into(), - )); + return Err(Repr::Handshake(HandshakeFault::Connection).into()); } let protocol_matches = match (offered_protocols, headers.get(SEC_WEBSOCKET_PROTOCOL)) { (_, None) => true, @@ -92,31 +104,25 @@ fn validate_response_headers( _ => false, }; if !protocol_matches { - return Err(Error::Network( - "WebSocket handshake returned an invalid subprotocol".into(), - )); + return Err(Repr::Handshake(HandshakeFault::Subprotocol).into()); } Ok(()) } fn http_url(url: &str) -> Result { - let mut url = Url::parse(url).map_err(not_dispatched)?; + let mut url = Url::parse(url).map_err(Repr::WebSocketUrl)?; let scheme = match url.scheme() { "ws" | "http" => "http", "wss" | "https" => "https", - other => { - return Err(not_dispatched(format!( - "unsupported WebSocket URL scheme {other:?}" - ))); - } + other => return Err(Repr::WebSocketScheme(other.to_owned()).into()), }; url.set_scheme(scheme) - .map_err(|()| not_dispatched("WebSocket URL cannot be sent over HTTP"))?; + .map_err(|()| Repr::WebSocketSchemeRewrite(scheme))?; Ok(url) } fn handshake_headers(headers: HeaderMap, key: &str) -> Result { - let key = HeaderValue::from_str(key).map_err(not_dispatched)?; + let key = HeaderValue::from_str(key).map_err(|error| Repr::Request(error.into()))?; Ok([ (CONNECTION, HeaderValue::from_static("Upgrade")), (UPGRADE, HeaderValue::from_static("websocket")), @@ -129,7 +135,3 @@ fn handshake_headers(headers: HeaderMap, key: &str) -> Result headers })) } - -fn not_dispatched(error: impl std::fmt::Display) -> Error { - Error::Connect(error.to_string()) -} diff --git a/litellm-rust/crates/http/tests/websocket.rs b/litellm-rust/crates/http/tests/websocket.rs index 94aba97a84d..a5758d45e56 100644 --- a/litellm-rust/crates/http/tests/websocket.rs +++ b/litellm-rust/crates/http/tests/websocket.rs @@ -7,9 +7,8 @@ use std::{ use futures_util::{SinkExt, StreamExt}; use http::{HeaderMap, HeaderValue}; use litellm_http::{ - ClientIdentity, ClientVariant, EnvironmentProxies, HttpClientConfig, HttpClientPool, - HttpSettings, Resolution, Verify, media::PublicDnsResolver, transport::Error, - websocket::connect, + ClientIdentity, ClientVariant, EnvironmentProxies, Error, HttpClientConfig, HttpClientPool, + HttpSettings, Resolution, Verify, media::PublicDnsResolver, websocket::connect, }; use rcgen::{BasicConstraints, CertificateParams, CertifiedIssuer, IsCa, KeyPair}; use rstest::{Context, rstest}; @@ -122,7 +121,7 @@ async fn open( let client = HttpClientPool::new(Arc::new(PublicDnsResolver)) .client(config, ClientVariant::WebSocket) .unwrap(); - connect(&client, url, headers).await + connect(&client, url, headers, None).await } async fn echo_once( @@ -269,7 +268,7 @@ async fn wss_honors_ssl_verify(#[case] verify: fn(&Pki) -> Verify, #[case] trust match result { Ok(mut socket) if trusted => round_trip(&mut socket).await, - Err(Error::Connect(_)) if !trusted => {} + Err(error) if !trusted && !error.dispatched() => {} other => panic!("unexpected handshake outcome: {:?}", other.map(drop)), } } @@ -290,7 +289,7 @@ async fn websocket_uses_system_roots( let result = open(&config(Verify::BuiltInRoots), &url, HeaderMap::new()).await; match result { Ok(mut socket) if connects => round_trip(&mut socket).await, - Err(Error::Connect(_)) if !connects => {} + Err(error) if !connects && error.is_connect() => {} other => panic!("unexpected handshake outcome: {:?}", other.map(drop)), } return; @@ -407,13 +406,8 @@ async fn refused_upgrade_reports_the_status_and_body() { ) .await; - assert_eq!( - result.map(drop), - Err(Error::Http { - status: 403, - body: "denied".into() - }) - ); + let error = result.map(drop).unwrap_err(); + assert_eq!(error.upstream_response(), Some((403, "denied"))); } #[rstest] @@ -431,7 +425,7 @@ async fn upgrade_with_a_wrong_accept_key_is_rejected() { ) .await; - assert!(matches!(result, Err(Error::Network(_)))); + assert!(matches!(result, Err(error) if error.dispatched() && !error.is_request())); } #[rstest] @@ -441,7 +435,7 @@ async fn upgrade_with_a_wrong_accept_key_is_rejected() { async fn invalid_url_is_rejected_before_dispatch(#[case] url: &str) { let result = open(&config(Verify::BuiltInRoots), url, HeaderMap::new()).await; - assert!(matches!(result, Err(Error::Connect(_)))); + assert!(matches!(result, Err(error) if !error.dispatched() && error.is_request())); } #[rstest] @@ -533,7 +527,34 @@ async fn upgrade_validates_headers_and_negotiated_protocol( match result { Ok(mut socket) if accepted => round_trip(&mut socket).await, - Err(Error::Network(_)) if !accepted => {} + Err(error) if !accepted && error.dispatched() => {} other => panic!("unexpected handshake outcome: {:?}", other.map(drop)), } } + +#[rstest] +#[tokio::test] +async fn a_handshake_that_never_answers_times_out_as_dispatched() { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + tokio::spawn(async move { + let (mut stream, _) = listener.accept().await.unwrap(); + read_head(&mut stream).await; + let _ = stream.read(&mut [0u8; 1]).await; + }); + let client = HttpClientPool::new(Arc::new(PublicDnsResolver)) + .client(&config(Verify::BuiltInRoots), ClientVariant::WebSocket) + .unwrap(); + + let error = connect( + &client, + &format!("ws://{address}/responses"), + HeaderMap::new(), + Some(std::time::Duration::from_millis(100)), + ) + .await + .map(drop) + .unwrap_err(); + + assert!(error.dispatched()); +} diff --git a/litellm-rust/crates/inference-chat/Cargo.toml b/litellm-rust/crates/inference-chat/Cargo.toml index fa3fad067a9..8a779d44976 100644 --- a/litellm-rust/crates/inference-chat/Cargo.toml +++ b/litellm-rust/crates/inference-chat/Cargo.toml @@ -18,11 +18,11 @@ litellm-secrets.workspace = true serde_json = { workspace = true, features = ["preserve_order"] } futures-util.workspace = true -reqwest.workspace = true tracing.workspace = true [dev-dependencies] litellm-cache-memory.workspace = true +litellm-error.workspace = true litellm-host-native.workspace = true litellm-http = { workspace = true, features = ["test-support"] } litellm-inference-testing.workspace = true diff --git a/litellm-rust/crates/inference-chat/src/common_utils.rs b/litellm-rust/crates/inference-chat/src/common_utils.rs index 4063f33432d..b55aba22325 100644 --- a/litellm-rust/crates/inference-chat/src/common_utils.rs +++ b/litellm-rust/crates/inference-chat/src/common_utils.rs @@ -50,14 +50,16 @@ pub(super) fn resolve_request( config: &'static dyn BaseConfig, ) -> Result { let messages = serde_json::from_value::>(call.messages).map_err(|err| { - Error::InvalidRequest(litellm_llms::ErrorDetail::invalid( + Error::invalid_request(litellm_llms::ErrorDetail::invalid( "chat completions messages", err, )) })?; if messages.is_empty() { - return Err(Error::InvalidRequest( - "chat completions requires at least one message".into(), + return Err(Error::invalid_request( + litellm_llms::ErrorDetail::NoMessages { + route: "chat completions", + }, )); } if let Some(reason) = config.unsupported_reason(&messages, &call.optional_params) { diff --git a/litellm-rust/crates/inference-chat/src/handler.rs b/litellm-rust/crates/inference-chat/src/handler.rs index 15a45747997..c34d60e1b8a 100644 --- a/litellm-rust/crates/inference-chat/src/handler.rs +++ b/litellm-rust/crates/inference-chat/src/handler.rs @@ -3,10 +3,7 @@ use std::time::Duration; use litellm_auth::AuthServices; use litellm_host::interceptors::{Interceptors, RawResponse, RequestContext, WireRequest}; -use litellm_http::{ - Client, - request::{truncate_error_body, with_default_headers}, -}; +use litellm_http::{Client, request::with_default_headers}; use litellm_llms::base_llm::{ auth::{Authenticated, ValidatedEnvironment, resolve_auth}, chat::transformation::BaseConfig, @@ -90,39 +87,26 @@ pub(super) async fn execute( interceptors, observers, || async move { - let outbound = outbound_request( + let outbound = litellm_inference::outbound::outbound_request( Authenticated { headers: wire.headers, signer: authenticated.signer, }, &wire.url, &wire.body, - call.timeout, + Some( + call.timeout + .unwrap_or(Duration::from_secs(CHAT_COMPLETIONS_TIMEOUT_SECS)), + ), )?; - let response = litellm_inference::outbound::send(outbound, http) - .await - .map_err(|err| { - // Failing to establish the connection means the request never went out, - // so the host can still serve it. Everything else here, a timeout - // above all, may have reached the provider and been answered. - if err.is_connect() || err.is_builder() { - Error::Transport(litellm_http::transport::Error::Connect(err.to_string())) - } else { - Error::Transport(litellm_http::transport::Error::Network(err.to_string())) - } - })?; + let response = litellm_inference::outbound::send(outbound, http).await?; let status = response.status(); - let text = response.text().await.map_err(|err| { - Error::Transport(litellm_http::transport::Error::Network(err.to_string())) - })?; + let text = response.text().await.map_err(litellm_http::Error::from)?; if !status.is_success() { - return Err(Error::Transport(litellm_http::transport::Error::Http { - status: status.as_u16(), - body: truncate_error_body(&text), - })); + return Err(litellm_http::Error::upstream(status.as_u16(), text).into()); } let raw = RawResponse { body: text.clone() }; if let Some(observers) = observers { @@ -136,7 +120,7 @@ pub(super) async fn execute( .map_err(Error::post_call)?; let body: Value = serde_json::from_str(&text).map_err(|err| { - Error::InvalidResponse(litellm_llms::ErrorDetail::invalid( + Error::invalid_response(litellm_llms::ErrorDetail::invalid( "chat completions response JSON", err, )) @@ -160,33 +144,13 @@ pub(super) async fn execute( /// implementation must not retry those, so collapse them to one variant that /// can only mean the provider was already called. pub(super) fn as_response_error(err: Error) -> Error { - match err { - already @ (Error::InvalidResponse(_) - | Error::Transport(litellm_http::transport::Error::Http { .. })) => already, - other => Error::InvalidResponse(other.to_string().into()), + if err.is_response() || err.upstream_response().is_some() { + return err; } -} - -pub(super) fn outbound_request( - authenticated: Authenticated, - url: &str, - body: &Value, - timeout: Option, -) -> Result { - litellm_inference::outbound::outbound_request( - authenticated, - url, - body, - Some(timeout.unwrap_or(Duration::from_secs(CHAT_COMPLETIONS_TIMEOUT_SECS))), - ) - .map_err(|error| match error { - // Python drops the caller's copy and prefers a forwarded Authorization - // over the signature, so leave the request to it. - litellm_http::Error::ComputedHeader(_) => { - Error::Unsupported("request forwards a header AWS SigV4 computes") - } - other => Error::Http(other), - }) + Error::invalid_response(litellm_llms::ErrorDetail::invalid( + "chat completions response", + err, + )) } #[cfg(test)] @@ -341,31 +305,48 @@ mod tests { .await .expect_err("the upstream failure fails the call"); - assert!(matches!( - error, - Error::Transport(litellm_http::transport::Error::Http { status: 500, .. }) - )); + assert!( + error + .upstream_response() + .is_some_and(|(status, _)| status == 500), + "{error:?}" + ); assert!(interceptors.raw.into_inner().unwrap().is_empty()); } #[rstest::rstest] - fn response_errors_collapse_to_one_variant_that_can_only_mean_already_sent() { - for original in [ - Error::MissingField("usage"), - Error::Unsupported("non-text response content block"), - Error::InvalidRequest("whatever".to_string().into()), - Error::Auth(litellm_auth::Error::InvalidHeader), - ] { - let label = format!("{original:?}"); - assert!( - matches!(as_response_error(original), Error::InvalidResponse(_)), - "{label} must not stay retryable once the provider has answered" - ); - } - let upstream = Error::Transport(litellm_http::transport::Error::Http { - status: 500, - body: "boom".to_string(), + #[case::missing_field(litellm_llms::Error::MissingField("usage").into())] + #[case::unsupported(Error::Unsupported("non-text response content block"))] + #[case::invalid_request(Error::invalid_request(litellm_llms::ErrorDetail::Expected { + subject: "whatever", + expected: "anything else", + }))] + #[case::auth(Error::Auth(litellm_auth::Error::InvalidHeader))] + fn response_errors_collapse_to_a_fault_that_can_only_mean_already_sent( + #[case] original: Error, + ) { + let collapsed = as_response_error(original); + assert!(collapsed.is_response(), "{collapsed:?}"); + assert!(!collapsed.is_request(), "{collapsed:?}"); + assert!(!collapsed.is_unsupported(), "{collapsed:?}"); + } + + #[rstest::rstest] + fn a_response_fault_is_not_wrapped_twice() { + let original = Error::invalid_response(litellm_llms::ErrorDetail::MissingField { + subject: "response", + field: "usage", }); - assert_eq!(as_response_error(upstream.clone()), upstream); + let report = litellm_error::Report::new(&as_response_error(original)).to_string(); + assert!(!report.contains("chat completions response"), "{report}"); + } + + #[test] + fn a_provider_answer_keeps_its_status_and_body() { + let upstream = Error::from(litellm_http::Error::upstream(429, "slow down".into())); + assert_eq!( + as_response_error(upstream).upstream_response(), + Some((429, "slow down")) + ); } } diff --git a/litellm-rust/crates/inference-chat/tests/chat_completions.rs b/litellm-rust/crates/inference-chat/tests/chat_completions.rs index 461d4146a63..6ed513ba32a 100644 --- a/litellm-rust/crates/inference-chat/tests/chat_completions.rs +++ b/litellm-rust/crates/inference-chat/tests/chat_completions.rs @@ -5,7 +5,6 @@ use litellm_host::{ }; use std::time::Duration; -use litellm_http::transport::Error as TransportError; use litellm_inference_chat::{Error, types::ChatCompletionsRequest}; use litellm_llms_types::formats::chat_completions::ChatCompletionsResponse; use rstest::{fixture, rstest}; @@ -180,7 +179,7 @@ async fn a_response_it_cannot_normalize_is_reported_as_already_sent( .await .expect_err("response cannot be normalized"); - assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + assert!(error.is_response(), "{error:?}"); } #[rstest] @@ -201,13 +200,7 @@ async fn an_upstream_error_status_keeps_its_code_and_body( .await .expect_err("upstream rejects"); - assert_eq!( - error, - Error::Transport(TransportError::Http { - status, - body: "slow down".into() - }) - ); + assert_eq!(error.upstream_response(), Some((status, "slow down"))); } #[rstest] @@ -222,10 +215,25 @@ async fn a_connection_that_is_never_established_returns_a_connect_error( .await .expect_err("nothing is listening"); - assert!( - matches!(error, Error::Transport(TransportError::Connect(_))), - "{error:?}" - ); + assert!(!error.dispatched(), "{error:?}"); +} + +#[rstest] +#[tokio::test] +async fn a_transport_error_does_not_echo_the_provider_url( + request: ChatCompletionsRequest<'static>, +) { + let base = format!("{UNREACHABLE_BASE}/tenant-token-1234"); + + let error = complete(ChatCompletionsRequest { + api_base: Some(&base), + ..request + }) + .await + .expect_err("nothing is listening"); + + let report = litellm_error::Report::new(&error).to_string(); + assert!(!report.contains("tenant-token-1234"), "{report}"); } #[rstest] @@ -243,10 +251,8 @@ async fn a_timeout_after_sending_returns_a_network_error(request: ChatCompletion .await .expect_err("the call times out"); - assert!( - matches!(error, Error::Transport(TransportError::Network(_))), - "{error:?}" - ); + assert!(error.dispatched(), "{error:?}"); + assert!(!error.is_request(), "{error:?}"); } #[rstest] @@ -342,7 +348,7 @@ async fn a_post_call_hook_failure_never_looks_safe_to_retry( Ok(wire) } async fn after_provider_response(&self, _: RawResponse) -> Result<(), Error> { - Err(Error::InvalidRequest("callback rejected".into())) + Err(litellm_llms::Error::MissingField("callback rejected").into()) } } let upstream = upstream([anthropic_response(ANTHROPIC_MESSAGE)]).await; @@ -358,10 +364,10 @@ async fn a_post_call_hook_failure_never_looks_safe_to_retry( ) .await .unwrap_err(); - let Error::PostCallHook(source) = error else { - panic!("expected retained callback error") - }; - assert_eq!(*source, Error::InvalidRequest("callback rejected".into())); + assert!(error.dispatched(), "{error:?}"); + assert!(!error.is_request(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("callback rejected"), "{report}"); assert_eq!(received(&upstream).await.len(), 1); } diff --git a/litellm-rust/crates/inference-chat/tests/request.rs b/litellm-rust/crates/inference-chat/tests/request.rs index 5988a7baee5..4025c137cdc 100644 --- a/litellm-rust/crates/inference-chat/tests/request.rs +++ b/litellm-rust/crates/inference-chat/tests/request.rs @@ -211,7 +211,7 @@ async fn rejects_empty_messages_before_resolving_credentials() { }) .await .expect_err("empty messages are rejected"); - assert!(matches!(error, Error::InvalidRequest(_))); + assert!(error.is_request(), "{error:?}"); } #[rstest] @@ -233,26 +233,27 @@ async fn rejects_an_unknown_provider() { #[rstest] #[tokio::test] async fn rejects_a_model_with_no_resolvable_provider() { - assert!(matches!( - complete(request( - "claude-sonnet-4-5", - None, - json!([{"role": "user", "content": "hi"}]), - json!({}), - )) - .await, - Err(Error::InvalidProvider(_)) - )); + let error = complete(request( + "claude-sonnet-4-5", + None, + json!([{"role": "user", "content": "hi"}]), + json!({}), + )) + .await + .unwrap_err(); + assert!(error.is_request(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!( + report.contains("unable to resolve custom_llm_provider"), + "{report}" + ); } #[rstest] -#[case::empty(json!([]), Some("chat completions requires at least one message"))] -#[case::malformed(json!("not a list"), None)] +#[case::empty(json!([]), "chat completions requires at least one message")] +#[case::malformed(json!("not a list"), "invalid chat completions messages")] #[tokio::test] -async fn rejects_an_empty_or_malformed_message_list( - #[case] messages: Value, - #[case] expected: Option<&str>, -) { +async fn rejects_an_empty_or_malformed_message_list(#[case] messages: Value, #[case] reason: &str) { let error = complete(request( "anthropic/claude-sonnet-4-5", None, @@ -261,11 +262,9 @@ async fn rejects_an_empty_or_malformed_message_list( )) .await .unwrap_err(); - if let Some(expected) = expected { - assert_eq!(error, Error::InvalidRequest(expected.to_string().into())); - } else { - assert!(matches!(error, Error::InvalidRequest(_))); - } + assert!(error.is_request(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains(reason), "{report}"); } #[rstest] diff --git a/litellm-rust/crates/inference-messages/Cargo.toml b/litellm-rust/crates/inference-messages/Cargo.toml index e310a0be9b1..694d9b4de31 100644 --- a/litellm-rust/crates/inference-messages/Cargo.toml +++ b/litellm-rust/crates/inference-messages/Cargo.toml @@ -26,6 +26,7 @@ serde_json = { workspace = true, features = ["preserve_order"] } tracing.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-framer = { workspace = true, features = ["sse"] } litellm-cache-memory.workspace = true litellm-host-native.workspace = true diff --git a/litellm-rust/crates/inference-messages/src/common_utils.rs b/litellm-rust/crates/inference-messages/src/common_utils.rs index 9be7ee6ffc3..a57eca1c6a9 100644 --- a/litellm-rust/crates/inference-messages/src/common_utils.rs +++ b/litellm-rust/crates/inference-messages/src/common_utils.rs @@ -1,6 +1,5 @@ use litellm_core_utils::get_llm_provider_logic::LlmProviders; use litellm_http::request::string_headers as shared_string_headers; -pub(super) use litellm_http::request::truncate_error_body; use litellm_llms::tencent::messages::transformation::TENCENT_MESSAGES_CONFIG; use litellm_llms::{ anthropic::messages::transformation::ANTHROPIC_MESSAGES_CONFIG, @@ -77,7 +76,7 @@ mod tests { use rstest::rstest; - use super::{messages_provider, string_headers, truncate_error_body}; + use super::{messages_provider, string_headers}; use crate::Error; use litellm_core_utils::get_llm_provider_logic::LlmProviders; @@ -106,19 +105,6 @@ mod tests { assert!(messages_provider(provider, model).is_none()); } - #[test] - fn truncate_error_body_caps_long_payloads() { - let body = "x".repeat(400); - let truncated = truncate_error_body(&body); - assert!(truncated.ends_with("... (truncated)")); - let prefix_chars = truncated - .strip_suffix("... (truncated)") - .expect("truncated marker present") - .chars() - .count(); - assert_eq!(prefix_chars, 256); - } - #[test] fn string_headers_rejects_non_string_values() { let headers = json!({"x-count": 3}).as_object().unwrap().clone(); diff --git a/litellm-rust/crates/inference-messages/src/handler.rs b/litellm-rust/crates/inference-messages/src/handler.rs index 292995d842c..67c3b8f98b5 100644 --- a/litellm-rust/crates/inference-messages/src/handler.rs +++ b/litellm-rust/crates/inference-messages/src/handler.rs @@ -10,7 +10,7 @@ use litellm_core_utils::{ use litellm_host::interceptors::{ Cost, ExecutionFacts, Interceptors, ProviderIdentity, RequestContext, ResultSource, WireRequest, }; -use litellm_http::{request::with_default_headers, transport::Error as TransportError}; +use litellm_http::request::with_default_headers; use litellm_llms::base_llm::{ auth::{Authenticated, ValidatedEnvironment, resolve_auth}, messages::{ @@ -26,7 +26,7 @@ use serde_json::Value; use super::{ Error, MessagesCall, MessagesCallResponse, MessagesRoute, - common_utils::{MessagesProvider, string_headers, truncate_error_body}, + common_utils::{MessagesProvider, string_headers}, constants::MESSAGES_TIMEOUT_SECS, route::Messages, types::invalid_request, @@ -188,7 +188,7 @@ async fn provider_call( None | Some(Value::Null) => false, Some(Value::Bool(stream)) => *stream, Some(value) => { - return Err(Error::InvalidRequest( + return Err(Error::invalid_request( litellm_llms::ErrorDetail::InvalidValue { field: "stream", expected: "a boolean", @@ -261,7 +261,7 @@ async fn call_provider( provider_name, )); } - let text = response.text().await.map_err(network)?; + let text = response.text().await.map_err(litellm_http::Error::from)?; log_response_body(&text); context.response_received(&text).await?; decode_response(config, &identity.model, &text) @@ -269,16 +269,12 @@ async fn call_provider( } fn serialize_failure(err: serde_json::Error) -> Error { - Error::InvalidRequest(litellm_llms::ErrorDetail::failed( + Error::invalid_request(litellm_llms::ErrorDetail::failed( "Anthropic messages request serialization", err, )) } -fn network(error: reqwest::Error) -> Error { - Error::Transport(TransportError::Network(error.to_string())) -} - async fn send( http: &litellm_http::Client, authenticated: Authenticated, @@ -292,9 +288,7 @@ async fn send( body, Some(timeout.unwrap_or(Duration::from_secs(MESSAGES_TIMEOUT_SECS))), )?; - litellm_inference::outbound::send(request, http) - .await - .map_err(network) + Ok(litellm_inference::outbound::send(request, http).await?) } async fn provider_error(response: reqwest::Response) -> Error { @@ -302,12 +296,9 @@ async fn provider_error(response: reqwest::Response) -> Error { match response.text().await { Ok(text) => { log_error_body(status, &text); - Error::Transport(TransportError::Http { - status, - body: truncate_error_body(&text), - }) + litellm_http::Error::upstream(status, text).into() } - Err(error) => network(error), + Err(error) => litellm_http::Error::from(error).into(), } } @@ -317,7 +308,7 @@ fn decode_response( text: &str, ) -> Result { let response = serde_json::from_str(text).map_err(|err| { - Error::InvalidResponse(litellm_llms::ErrorDetail::invalid( + Error::invalid_response(litellm_llms::ErrorDetail::invalid( "messages response JSON", err, )) @@ -339,7 +330,7 @@ fn streaming_response( .collect(); let chunks = match decoder { None => futures_util::stream::try_unfold(response, move |mut response| async move { - let chunk = response.chunk().await.map_err(network)?; + let chunk = response.chunk().await.map_err(litellm_http::Error::from)?; Ok(chunk.map(|chunk| { log_chunk(provider, "provider_response", &chunk); (chunk, response) @@ -362,7 +353,7 @@ fn decoded_chunks( let bytes: ByteStream = response .bytes_stream() .inspect_ok(move |chunk| log_chunk(provider, "provider_response", chunk)) - .map_err(std::io::Error::other) + .map_err(|error| std::io::Error::other(litellm_http::Error::from(error))) .boxed(); futures_util::stream::try_unfold(decode(bytes), move |mut events| async move { let Some(event) = events.try_next().await? else { @@ -541,14 +532,10 @@ mod tests { shaping, }) .await; - assert!( - matches!( - result, - Err(Error::Auth(litellm_auth::Error::MissingApiKey { .. })) - ), - "{:?}", - result.err() - ); + let Err(error) = result else { + panic!("a call without credentials fails"); + }; + assert!(error.is_missing_credential(), "{error:?}"); } #[rstest] @@ -771,19 +758,18 @@ mod tests { }, ..shaping }; - assert!(matches!( - wire_body( - json!({ - "model": "claude-test", - "messages": [{"role": "user", "content": "hi"}], - "max_tokens": 16, - "metadata": {"user_id": 123} - }), - shaping, - ) - .await, - Err(Error::InvalidRequest(_)) - )); + let error = wire_body( + json!({ + "model": "claude-test", + "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 16, + "metadata": {"user_id": 123} + }), + shaping, + ) + .await + .unwrap_err(); + assert!(error.is_request(), "{error:?}"); } #[rstest] @@ -800,7 +786,7 @@ mod tests { shaping, ) .await, - Err(Error::InvalidRequest( + Err(Error::invalid_request( litellm_llms::ErrorDetail::InvalidValue { field: "metadata.user_id", expected: "a string", @@ -902,7 +888,7 @@ mod tests { let chunk = chunks.next().await.unwrap(); match expected { Some(expected) => assert_eq!(chunk.unwrap().as_ref(), expected.as_bytes()), - None => assert!(matches!(chunk, Err(Error::InvalidResponse(_))), "{chunk:?}"), + None => assert!(chunk.unwrap_err().is_response()), } assert!(chunks.next().await.is_none()); } diff --git a/litellm-rust/crates/inference-messages/src/types.rs b/litellm-rust/crates/inference-messages/src/types.rs index 21eac0574ed..ae346259559 100644 --- a/litellm-rust/crates/inference-messages/src/types.rs +++ b/litellm-rust/crates/inference-messages/src/types.rs @@ -34,11 +34,11 @@ pub fn messages_body(body: Map) -> Result /// Python's `GenericLiteLLMParams(**kwargs)`. pub fn litellm_params(fields: Map) -> Result { serde_json::from_value(Value::Object(fields)) - .map_err(|err| Error::InvalidRequest(ErrorDetail::invalid("litellm params", err))) + .map_err(|err| Error::invalid_request(ErrorDetail::invalid("litellm params", err))) } pub(super) fn invalid_request(err: serde_json::Error) -> Error { - Error::InvalidRequest(ErrorDetail::invalid("Anthropic messages request", err)) + Error::invalid_request(ErrorDetail::invalid("Anthropic messages request", err)) } pub type MessagesCallResponse = diff --git a/litellm-rust/crates/inference-messages/tests/bedrock_native/main.rs b/litellm-rust/crates/inference-messages/tests/bedrock_native/main.rs index 5420477c34a..f91d5c79f7c 100644 --- a/litellm-rust/crates/inference-messages/tests/bedrock_native/main.rs +++ b/litellm-rust/crates/inference-messages/tests/bedrock_native/main.rs @@ -309,7 +309,14 @@ async fn deployment_workspace_is_required_and_sent_as_a_header( }) .await; if !configured { - assert!(matches!(result, Err(Error::Auth(_)))); + let Err(error) = result else { + panic!("the call fails"); + }; + assert!(!error.dispatched(), "{error:?}"); + assert!( + error.is_request() || error.is_missing_credential(), + "{error:?}" + ); assert!(received(&upstream).await.is_empty()); return; } diff --git a/litellm-rust/crates/inference-messages/tests/messages/host.rs b/litellm-rust/crates/inference-messages/tests/messages/host.rs index ba20fb9eec4..48afae99cb5 100644 --- a/litellm-rust/crates/inference-messages/tests/messages/host.rs +++ b/litellm-rust/crates/inference-messages/tests/messages/host.rs @@ -301,7 +301,7 @@ async fn response_mode_follows_the_intercepted_request( let summaries = traces.summaries("litellm.route"); assert_eq!(summaries.len(), 1); let Some(expected_stream) = expected_stream else { - assert!(matches!(result, Err(Error::InvalidRequest(_)))); + assert!(result.unwrap_err().is_request()); assert!(received(&upstream).await.is_empty()); assert_eq!(summaries[0]["outcome"], "failure"); return; @@ -322,14 +322,17 @@ async fn a_before_send_failure_never_sends(call: MessagesCall) { let upstream = upstream([message_response()]).await; let host = RecordingHost::new( authenticated(call, upstream.uri()), - Box::new(|_| Err(Error::InvalidRequest("vetoed by the host".into()))), + Box::new(|_| Err(litellm_llms::Error::MissingField("vetoed by the host").into())), ); let error = run_through(&host) .await .expect_err("the host failure fails the call"); - assert_eq!(error, Error::InvalidRequest("vetoed by the host".into())); + assert_eq!( + error, + Error::from(litellm_llms::Error::MissingField("vetoed by the host")) + ); assert!(received(&upstream).await.is_empty()); assert!(host.raw_responses().is_empty()); } diff --git a/litellm-rust/crates/inference-messages/tests/messages/main.rs b/litellm-rust/crates/inference-messages/tests/messages/main.rs index fcc432acb76..a01e571ec75 100644 --- a/litellm-rust/crates/inference-messages/tests/messages/main.rs +++ b/litellm-rust/crates/inference-messages/tests/messages/main.rs @@ -144,7 +144,7 @@ impl LocalMessagesHost { .lock() .unwrap_or_else(|error| error.into_inner()) .take() - .ok_or_else(|| Error::InvalidRequest("messages request was already projected".into())) + .ok_or(litellm_llms::Error::MissingField("messages request").into()) } pub fn runtime(&self) -> litellm_host_native::in_process::Host<'_, (), Self, ()> { litellm_host_native::in_process::Host { diff --git a/litellm-rust/crates/inference-messages/tests/messages/request.rs b/litellm-rust/crates/inference-messages/tests/messages/request.rs index 93bbbb071ba..e53bdb8d252 100644 --- a/litellm-rust/crates/inference-messages/tests/messages/request.rs +++ b/litellm-rust/crates/inference-messages/tests/messages/request.rs @@ -108,13 +108,7 @@ async fn a_call_without_credentials_fails_before_sending( .await .expect_err("a call without credentials fails"); - assert!( - matches!( - error, - Error::Auth(litellm_auth::Error::MissingApiKey { .. }) - ), - "{error:?}" - ); + assert!(error.is_missing_credential(), "{error:?}"); assert!(received(&upstream).await.is_empty()); } @@ -185,7 +179,9 @@ async fn unsupported_providers_are_rejected_before_sending( .await .expect_err("unsupported provider errors"); - assert_eq!(error, Error::InvalidProvider(reported.into())); + assert!(error.is_request(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains(reported), "{report}"); } #[rstest] @@ -513,10 +509,9 @@ async fn unsupported_params_are_dropped_under_drop_params_and_rejected_without_i let error = run(shaped(false)) .await .expect_err("an unsupported param is rejected without drop_params"); - assert!( - matches!(&error, Error::InvalidRequest(message) if message.to_string().contains(rejected_as)), - "{error:?}" - ); + assert!(error.is_request(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains(rejected_as), "{report}"); assert!(received(&upstream).await.is_empty()); run_message(shaped(true)).await; @@ -957,11 +952,10 @@ async fn provider_validation_runs_before_caller_parameter_removal( )) .await; let error = result.expect_err("metadata is validated before removal"); + let report = litellm_error::Report::new(&error).to_string(); assert!( - error - .to_string() - .contains("metadata.user_id must be a string"), - "{error}" + report.contains("metadata.user_id must be a string"), + "{report}" ); assert!(received(&upstream).await.is_empty()); } diff --git a/litellm-rust/crates/inference-messages/tests/messages/response.rs b/litellm-rust/crates/inference-messages/tests/messages/response.rs index b5d623a5582..7aff1286858 100644 --- a/litellm-rust/crates/inference-messages/tests/messages/response.rs +++ b/litellm-rust/crates/inference-messages/tests/messages/response.rs @@ -2,7 +2,6 @@ use litellm_host::{ interceptors::{ExecutionFacts, ResultSource}, lifecycle::ExecutionEvent, }; -use litellm_http::transport::Error as TransportError; use litellm_inference_messages::{MessagesCallResponse, messages_body}; use litellm_inference_testing::{ RecordingSecrets, http_config, no_secrets, provider_http, resources, @@ -166,17 +165,21 @@ async fn a_json_error_envelope_is_kept_verbatim(call: MessagesCall) { .await .expect_err("upstream error propagates"); - let Error::Transport(TransportError::Http { status, body }) = error else { + let Some((status, body)) = error.upstream_response() else { panic!("{error:?}"); }; assert_eq!(status, 400); - assert_eq!(serde_json::from_str::(&body).unwrap(), envelope); + assert_eq!(serde_json::from_str::(body).unwrap(), envelope); } #[rstest] +#[case::ascii(format!("{}private-tail", "x".repeat(600)))] +#[case::unicode(format!("{}private-tail", "é".repeat(600)))] #[tokio::test] -async fn a_long_error_body_is_truncated_at_the_documented_cap(call: MessagesCall) { - let long = "x".repeat(600); +async fn a_long_error_body_is_preserved_with_bounded_diagnostics( + call: MessagesCall, + #[case] long: String, +) { let upstream = upstream([ResponseTemplate::new(500).set_body_string(long.clone())]).await; let error = run(MessagesCall { @@ -187,13 +190,13 @@ async fn a_long_error_body_is_truncated_at_the_documented_cap(call: MessagesCall .await .expect_err("upstream error propagates"); - assert_eq!( - error, - Error::Transport(TransportError::Http { - status: 500, - body: format!("{}... (truncated)", &long[..256]) - }) - ); + assert_eq!(error.upstream_response(), Some((500, long.as_str()))); + let diagnostic_body: String = long.chars().take(256).collect(); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.ends_with(&format!("{diagnostic_body}... (truncated)"))); + assert_eq!(report.matches(&diagnostic_body).count(), 1, "{report}"); + assert!(!report.contains("private-tail"), "{report}"); + assert!(!format!("{error:?}").contains("private-tail")); } #[rstest] @@ -216,11 +219,8 @@ async fn an_upstream_error_keeps_its_status_and_body(call: MessagesCall, #[case] .expect_err("upstream error propagates"); assert_eq!( - error, - Error::Transport(TransportError::Http { - status, - body: "upstream said no".into() - }) + error.upstream_response(), + Some((status, "upstream said no")) ); } @@ -242,7 +242,7 @@ async fn an_unreadable_success_body_is_an_invalid_response( .await .expect_err("an unreadable body fails"); - assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + assert!(error.is_response(), "{error:?}"); } #[rstest] @@ -259,7 +259,23 @@ async fn a_provider_slower_than_the_timeout_fails_the_call(call: MessagesCall) { .await .expect_err("the call times out"); - assert!(matches!(error, Error::Transport(_)), "{error:?}"); + assert!(error.dispatched(), "{error:?}"); + assert!(!error.is_request(), "{error:?}"); + assert_eq!(error.upstream_response(), None); +} + +#[rstest] +#[tokio::test] +async fn a_transport_error_does_not_echo_the_provider_url(call: MessagesCall) { + let error = run(MessagesCall { + api_key: Some("sk".into()), + api_base: Some(format!("{UNREACHABLE_BASE}/tenant-token-1234")), + ..call + }) + .await + .expect_err("nothing is listening"); + + assert!(!error.to_string().contains("tenant-token-1234"), "{error}"); } #[rstest] @@ -305,8 +321,11 @@ async fn the_facade_sends_through_the_injected_http_pool_configuration(call: Mes fn a_body_that_does_not_parse_is_an_invalid_request(#[case] raw: Value) { let error = messages_body(object(raw)).expect_err("the body is rejected"); + assert!(error.is_request(), "{error:?}"); assert!( - matches!(&error, Error::InvalidRequest(message) if message.to_string().starts_with("invalid Anthropic messages request: ")), + litellm_error::Report::new(&error) + .to_string() + .contains("invalid Anthropic messages request"), "{error:?}" ); } diff --git a/litellm-rust/crates/inference-messages/tests/messages/secrets.rs b/litellm-rust/crates/inference-messages/tests/messages/secrets.rs index d4f7d2bf3a7..b4a3018ab93 100644 --- a/litellm-rust/crates/inference-messages/tests/messages/secrets.rs +++ b/litellm-rust/crates/inference-messages/tests/messages/secrets.rs @@ -109,7 +109,7 @@ async fn a_secret_manager_failure_fails_the_call_before_sending(call: MessagesCa .expect_err("a secret manager failure fails the call"); assert!( - matches!(&error, Error::Secret(source) if matches!(source.source_error(), litellm_secrets::Error::ManagedSecretMissing)), + matches!(&error, Error::Secret(_)) && !error.is_request(), "{error:?}" ); assert!(received(&upstream).await.is_empty()); diff --git a/litellm-rust/crates/inference-messages/tests/messages/stream.rs b/litellm-rust/crates/inference-messages/tests/messages/stream.rs index cda688107b5..4a8a061f764 100644 --- a/litellm-rust/crates/inference-messages/tests/messages/stream.rs +++ b/litellm-rust/crates/inference-messages/tests/messages/stream.rs @@ -260,13 +260,7 @@ async fn an_upstream_error_fails_the_call_without_opening_the_stream( .await .expect_err("upstream error propagates"); - assert_eq!( - error, - Error::Transport(litellm_http::transport::Error::Http { - status: 429, - body: body.into() - }) - ); + assert_eq!(error.upstream_response(), Some((429, body))); assert!(host.seen.into_inner().unwrap().is_empty()); } @@ -332,7 +326,9 @@ async fn the_timeout_covers_a_stalled_stream_body(call: MessagesCall) { .expect("the stalled stream gives up within the timeout") .expect_err("a stalled body fails the call"); - assert!(matches!(error, Error::Transport(_)), "{error:?}"); + assert!(error.dispatched(), "{error:?}"); + assert!(!error.is_request(), "{error:?}"); + assert_eq!(error.upstream_response(), None); let seen = host.seen.into_inner().unwrap(); assert!( matches!(seen.as_slice(), [Seen::Open(_), Seen::Deliver(chunk)] if chunk.as_ref() == b"event: message_start\ndata: {}\n\n"), @@ -390,13 +386,7 @@ async fn the_sdk_returns_http_errors_before_opening_a_stream(call: MessagesCall) .err() .expect("upstream failure is returned by messages()"); - assert_eq!( - error, - Error::Transport(litellm_http::transport::Error::Http { - status: 429, - body: "slow down".into(), - }) - ); + assert_eq!(error.upstream_response(), Some((429, "slow down"))); } #[rstest] @@ -470,7 +460,9 @@ async fn the_sdk_yields_a_body_error_once_after_delivered_chunks(call: MessagesC .expect("the stalled body times out") .unwrap() .unwrap_err(); - assert!(matches!(error, Error::Transport(_)), "{error:?}"); + assert!(error.dispatched(), "{error:?}"); + assert!(!error.is_request(), "{error:?}"); + assert_eq!(error.upstream_response(), None); assert!(chunks.next().await.is_none()); tokio::time::timeout(Duration::from_secs(5), connection) .await diff --git a/litellm-rust/crates/inference-messages/tests/messages/tencent.rs b/litellm-rust/crates/inference-messages/tests/messages/tencent.rs index dce187f4ea2..4bab7de65e1 100644 --- a/litellm-rust/crates/inference-messages/tests/messages/tencent.rs +++ b/litellm-rust/crates/inference-messages/tests/messages/tencent.rs @@ -79,13 +79,9 @@ async fn missing_credentials_fail_before_sending(tencent_call: MessagesCall) { }) .await .unwrap_err(); - assert!(matches!( - error, - Error::Auth(litellm_auth::Error::MissingApiKey { - environment_variable: "TENCENT_API_KEY", - .. - }) - )); + assert!(error.is_missing_credential(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("TENCENT_API_KEY"), "{report}"); assert!(received(&upstream).await.is_empty()); } diff --git a/litellm-rust/crates/inference-messages/tests/openrouter/main.rs b/litellm-rust/crates/inference-messages/tests/openrouter/main.rs index 74444142d37..80243718768 100644 --- a/litellm-rust/crates/inference-messages/tests/openrouter/main.rs +++ b/litellm-rust/crates/inference-messages/tests/openrouter/main.rs @@ -1,7 +1,7 @@ use std::{sync::Arc, time::Duration}; use futures_util::TryStreamExt; -use litellm_inference_messages::{Error, MessagesCall, MessagesCallResponse, MessagesShaping}; +use litellm_inference_messages::{MessagesCall, MessagesCallResponse, MessagesShaping}; use litellm_inference_testing::{RecordingSecrets, no_secrets}; use rstest::{fixture, rstest}; use serde_json::json; @@ -92,13 +92,12 @@ async fn an_anthropic_key_alone_is_not_an_openrouter_credential(call: MessagesCa None, ) .await; - assert!(matches!( - result, - Err(Error::Auth(litellm_auth::Error::MissingApiKey { - provider: "OpenRouter", - .. - })) - )); + let Err(error) = result else { + panic!("the call fails"); + }; + assert!(error.is_missing_credential(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("OpenRouter"), "{report}"); assert!(received(&upstream).await.is_empty()); } diff --git a/litellm-rust/crates/inference-ocr/Cargo.toml b/litellm-rust/crates/inference-ocr/Cargo.toml index 21ec013f873..bbbdaddb050 100644 --- a/litellm-rust/crates/inference-ocr/Cargo.toml +++ b/litellm-rust/crates/inference-ocr/Cargo.toml @@ -25,6 +25,7 @@ tokio.workspace = true tracing.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-auth-aws.workspace = true litellm-auth-gcp.workspace = true litellm-host-native.workspace = true @@ -34,4 +35,5 @@ litellm-llms = { workspace = true, features = ["test-support"] } rand.workspace = true rstest.workspace = true time.workspace = true +url.workspace = true wiremock.workspace = true diff --git a/litellm-rust/crates/inference-ocr/src/document.rs b/litellm-rust/crates/inference-ocr/src/document.rs index 0d9a296d289..44c1049102a 100644 --- a/litellm-rust/crates/inference-ocr/src/document.rs +++ b/litellm-rust/crates/inference-ocr/src/document.rs @@ -46,9 +46,9 @@ pub fn read_path_document(path: &Path, mime_type: Option<&str>) -> Result( model: &'a str, custom_llm_provider: Option<&'a str>, ) -> Result<(ResolvedProvider<'a>, OcrConfigKind), Error> { - let provider = match resolve_llm_provider(model, custom_llm_provider, "ocr") { - Ok(provider) => provider, - Err(_) if get_custom_llm_provider(model, custom_llm_provider).is_none() => { - ResolvedProvider { - model, - provider: LlmProviders::Mistral, - } - } - Err(litellm_inference::Error::InvalidProvider(provider)) => { - return Err(Error::InvalidProvider(provider)); - } - Err(error) => return Err(Error::InvalidProvider(error.to_string())), + let provider = match get_custom_llm_provider(model, custom_llm_provider) { + None => ResolvedProvider { + model, + provider: LlmProviders::Mistral, + }, + Some(CustomLlmProvider { + model, + custom_llm_provider, + }) => ResolvedProvider { + model, + provider: custom_llm_provider + .parse() + .map_err(|_| Error::InvalidProvider(custom_llm_provider.to_owned()))?, + }, }; let config = match provider.provider { LlmProviders::AwsTextract => match TextractOperation::from_model(provider.model)? { diff --git a/litellm-rust/crates/inference-ocr/src/route.rs b/litellm-rust/crates/inference-ocr/src/route.rs index 85125452bda..74ff829e8ad 100644 --- a/litellm-rust/crates/inference-ocr/src/route.rs +++ b/litellm-rust/crates/inference-ocr/src/route.rs @@ -43,7 +43,10 @@ fn caller_token_provider(services: HostServices) -> TokenProviderHandle { .call(OcrOp::AcquireAzureAdToken) .await .map_err(|error| { - litellm_auth::Error::CredentialAcquisition(error.to_string().into()) + litellm_auth::Error::CredentialAcquisition(litellm_auth::ErrorDetail::failed( + "host Azure AD token request", + error, + )) }) } }) diff --git a/litellm-rust/crates/inference-ocr/src/types.rs b/litellm-rust/crates/inference-ocr/src/types.rs index 1bd70c06116..98c609a158f 100644 --- a/litellm-rust/crates/inference-ocr/src/types.rs +++ b/litellm-rust/crates/inference-ocr/src/types.rs @@ -67,6 +67,7 @@ impl OcrConnectionInputs { .map(|value| (name.clone(), value.to_string())) .ok_or_else(|| Error::RequestField { path: format!("extra_headers.{name}"), + cause: None, }) }) .collect() @@ -105,6 +106,7 @@ impl LiteLLMOcrRequest { .filter(|value| *value > 0 && *value <= default_transport.max_response_bytes) .ok_or_else(|| Error::RequestField { path: "max_response_bytes".into(), + cause: None, }) }) .transpose()? @@ -313,7 +315,7 @@ mod tests { }; assert!(matches!( error, - Error::RequestField { ref path } if path == "extra_headers.x-a" + Error::RequestField { ref path, .. } if path == "extra_headers.x-a" )); } } diff --git a/litellm-rust/crates/inference-ocr/src/wire.rs b/litellm-rust/crates/inference-ocr/src/wire.rs index a8f5a77a1cd..56d22be6523 100644 --- a/litellm-rust/crates/inference-ocr/src/wire.rs +++ b/litellm-rust/crates/inference-ocr/src/wire.rs @@ -73,9 +73,8 @@ pub fn decode_request_input>( let timeout = wire .timeout_seconds .map(|seconds| { - Duration::try_from_secs_f64(seconds).map_err(|_| Error::RequestField { - path: "timeout_seconds".into(), - }) + Duration::try_from_secs_f64(seconds) + .map_err(|error| Error::request_field("timeout_seconds", error)) }) .transpose()?; LiteLLMOcrRequest::from_inputs( @@ -123,8 +122,8 @@ mod tests { #[case::non_object(json!([]), "document")] #[case::missing_type(json!({"document_url":"https://example.com/a.pdf"}), "document")] #[case::unsupported_type(json!({"type":"text"}), "type")] - #[case::missing_document_url(json!({"type":"document_url"}), "Document URL")] - #[case::missing_image_url(json!({"type":"image_url"}), "Document URL")] + #[case::missing_document_url(json!({"type":"document_url"}), "document URL")] + #[case::missing_image_url(json!({"type":"image_url"}), "document URL")] fn ocr_contract_malformed_document_is_bad_request( #[case] document: Value, #[case] field: &str, @@ -191,16 +190,13 @@ mod tests { assert!(is_supported_request("deepseek-ocr", Some("vertex_ai"))); } - #[test] - fn missing_document_source_has_a_typed_public_error() { - for document in [ - serde_json::json!({"type": "document_url"}), - serde_json::json!({"type": "image_url"}), - ] { - assert!(matches!( - decode_document(document), - Err(Error::MissingDocumentUrl) - )); - } + #[rstest] + #[case::document_url(serde_json::json!({"type": "document_url"}))] + #[case::image_url(serde_json::json!({"type": "image_url"}))] + fn missing_document_source_has_a_typed_public_error(#[case] document: serde_json::Value) { + assert!(matches!( + decode_document(document), + Err(Error::MissingDocumentUrl) + )); } } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/aws_textract.rs b/litellm-rust/crates/inference-ocr/tests/ocr/aws_textract.rs index c4096767188..f0e72b19fc1 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/aws_textract.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/aws_textract.rs @@ -191,12 +191,15 @@ async fn a_multi_page_rejection_reaches_the_caller_with_the_single_page_limit(#[ .await .unwrap_err(); - let Error::Provider { status, body, .. } = error else { - panic!("expected a provider error, got {error:?}"); - }; - assert_eq!(status, 400); + let message = litellm_error::Report::new(&error).to_string(); + assert!(error.is_request()); + assert_eq!(error.http_status_code(), Some(400)); assert!( - body.contains("multi-page documents are not supported"), - "{body}" + message.contains("Request has unsupported document format"), + "{message}" + ); + assert!( + message.contains("multi-page documents are not supported"), + "{message}" ); } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/azure_ai.rs b/litellm-rust/crates/inference-ocr/tests/ocr/azure_ai.rs index 9bff7490d58..6f591af73e0 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/azure_ai.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/azure_ai.rs @@ -222,24 +222,23 @@ async fn credential_precedence( false, json!({}), numbered_token, - |error: &Error| matches!(error, Error::Auth(litellm_auth::Error::MissingApiBase { - provider: "Azure AI", - guidance: "Set AZURE_AI_API_BASE environment variable or pass api_base parameter", - })), + |error: &Error| litellm_error::Report::new(error).to_string() + == "Missing Azure AI API Base - Set AZURE_AI_API_BASE environment variable or pass api_base parameter", 0 )] #[case::unsupported_oidc_reference( true, json!({"azure_ad_token": "oidc/assertion", "client_id": "client", "tenant_id": "tenant"}), numbered_token, - |error: &Error| matches!(error, Error::Auth(litellm_auth::Error::InvalidConfiguration(detail)) if detail.to_string() == "unsupported OIDC reference"), + |error: &Error| litellm_error::Report::new(error).to_string().contains("unsupported OIDC reference"), 0 )] #[case::empty_provider_token_ignores_static_token( true, json!({"azure_ad_token": "static-token"}), |_| String::new(), - |error: &Error| matches!(error, Error::MissingAzureAiCredentials), + |error: &Error| error.is_missing_credential() + && litellm_error::Report::new(error).to_string().contains("Azure AI credentials"), 1 )] #[tokio::test] diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/azure_document_intelligence.rs b/litellm-rust/crates/inference-ocr/tests/ocr/azure_document_intelligence.rs index cec43e2ed0e..98875cc777c 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/azure_document_intelligence.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/azure_document_intelligence.rs @@ -5,7 +5,7 @@ use std::{ use litellm_host::lifecycle::CallEvent; use litellm_host::lifecycle::ExecutionEvent; -use litellm_llms::base_llm::ocr::settings::OcrSettings; +use litellm_llms::base_llm::ocr::{error::PagesFault, settings::OcrSettings}; use rstest::rstest; @@ -64,10 +64,10 @@ async fn pages_features_and_extra_options_map_to_the_analyze_call() { } #[rstest] -#[case(json!({"pages": [true]}), Error::Pages("expected only integers or only strings".into()))] -#[case(json!({"pages": [1, "2"]}), Error::Pages("expected only integers or only strings".into()))] -#[case(json!({"pages": [-1]}), Error::Pages("negative page index".into()))] -#[case(json!({"pages": "1&&features=bad"}), Error::Pages("invalid native page range".into()))] +#[case(json!({"pages": [true]}), Error::Pages(PagesFault::Mixed))] +#[case(json!({"pages": [1, "2"]}), Error::Pages(PagesFault::Mixed))] +#[case(json!({"pages": [-1]}), Error::Pages(PagesFault::Negative))] +#[case(json!({"pages": "1&&features=bad"}), Error::Pages(PagesFault::Range))] #[case(json!({"features": "languages&pages=1"}), Error::Features)] #[case(json!({"req_format": "azure"}), Error::RequestFormat)] #[tokio::test] @@ -315,7 +315,11 @@ async fn polling_does_not_follow_redirects() { .await .unwrap_err(); - assert!(error.to_string().contains("status 302"), "{error}"); + assert_eq!( + error.upstream_response().map(|(status, ..)| status), + Some(302), + "{error}" + ); assert_eq!(received(&upstream).await.len(), 2); } @@ -389,12 +393,15 @@ async fn malformed_provider_pages_report_the_response_path( } #[rstest] -#[case::missing(None)] -#[case::relative(Some("/relative"))] -#[case::cross_origin(Some("http://example.com/operation"))] -#[case::with_userinfo(Some("http://user:password@127.0.0.1/operation"))] +#[case::missing(None, false)] +#[case::relative(Some("/relative"), true)] +#[case::cross_origin(Some("http://example.com/operation"), false)] +#[case::with_userinfo(Some("http://user:password@127.0.0.1/operation"), false)] #[tokio::test] -async fn an_unusable_operation_location_is_rejected(#[case] location: Option<&str>) { +async fn an_unusable_operation_location_is_rejected( + #[case] location: Option<&str>, + #[case] unparseable: bool, +) { let response = location .into_iter() .fold(ResponseTemplate::new(202), |response, location| { @@ -407,6 +414,11 @@ async fn an_unusable_operation_location_is_rejected(#[case] location: Option<&st .unwrap_err(); assert!(error.to_string().contains("operation-location"), "{error}"); + assert!(error.is_response()); + assert_eq!( + litellm_error::find::(&error).is_some(), + unparseable + ); assert_eq!(received(&upstream).await.len(), 1); } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/cohere.rs b/litellm-rust/crates/inference-ocr/tests/ocr/cohere.rs index 007aa49a2fd..a5076cf0112 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/cohere.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/cohere.rs @@ -37,6 +37,15 @@ async fn a_non_image_document_is_rejected_before_sending( .await .unwrap_err(); - assert!(matches!(error, Error::CohereImageOnly), "{error:?}"); + assert!( + matches!( + error, + Error::UnsupportedDocument { + provider: "Cohere Parse", + .. + } + ), + "{error:?}" + ); assert!(received(&upstream).await.is_empty()); } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/documents.rs b/litellm-rust/crates/inference-ocr/tests/ocr/documents.rs index 338a47c5f57..35de1326386 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/documents.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/documents.rs @@ -176,8 +176,8 @@ async fn a_missing_path_document_fails_before_sending() { assert!( matches!( &error, - Error::FileRead { path: failed, source } - if *failed == path && source.kind() == std::io::ErrorKind::NotFound + Error::FileRead { path: failed, cause } + if *failed == path && cause.kind() == std::io::ErrorKind::NotFound ), "{error:?}" ); @@ -216,7 +216,8 @@ async fn configured_client_preserves_document_url_policy(#[case] allowed: bool) .await; if !allowed { - assert!(matches!(result, Err(Error::BlockedDocumentUrl))); + let error = result.unwrap_err(); + assert!(error.is_request(), "{error:?}"); assert!(received(&documents).await.is_empty()); assert!(received(&upstream).await.is_empty()); return; diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs b/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs index e413781f1a3..75a91092daf 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs @@ -38,7 +38,7 @@ fn recording_host( .with_before_send(move |wire, _| { interceptions.fetch_add(1, Ordering::SeqCst); match block { - true => Err(Error::InvalidRequest("blocked".into())), + true => Err(Error::MissingField("blocked")), false => Ok(wire), } }) @@ -95,10 +95,7 @@ async fn a_blocking_before_send_prevents_the_call_and_emits_one_failure() { .await .unwrap_err(); - assert!( - matches!(&error, Error::InvalidRequest(message) if message == "blocked"), - "{error:?}" - ); + assert!(matches!(error, Error::MissingField("blocked")), "{error:?}"); assert_eq!(interceptions.load(Ordering::SeqCst), 1); assert_eq!(*events.lock().unwrap(), ["started", "failure"]); assert!(received(&upstream).await.is_empty()); diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/machine.rs b/litellm-rust/crates/inference-ocr/tests/ocr/machine.rs index 4f9266c4a8d..9a1c67fab0f 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/machine.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/machine.rs @@ -117,10 +117,14 @@ async fn a_hand_driven_machine_performs_the_same_call() { assert_eq!(outcome.unwrap().pages[0].markdown, "native"); assert_eq!(received(&upstream).await.len(), 1); assert_eq!(ops, ["BeforeSend", "response"]); - assert!(matches!( - machine.resume().await, - Err(Error::InvalidRequest(_)) - )); + let Err(error) = machine.resume().await else { + panic!("resuming a finished call is a host protocol fault"); + }; + assert!(!error.is_request(), "{error:?}"); + assert!( + litellm_error::find::(&error).is_some(), + "{error:?}" + ); } #[rstest::rstest] @@ -153,8 +157,11 @@ async fn a_path_document_is_read_by_core_without_a_host_operation() { } #[rstest] -#[case::failed(HostFailure::Error(Error::InvalidRequest("before_provider_request failed".into())), "before_provider_request failed")] -#[case::cancelled(HostFailure::Cancelled(Error::InvalidRequest("cancelled".into())), "cancelled")] +#[case::failed( + HostFailure::Error(Error::MissingField("before_provider_request failed")), + "before_provider_request failed" +)] +#[case::cancelled(HostFailure::Cancelled(Error::MissingField("cancelled")), "cancelled")] #[tokio::test] async fn a_before_send_failure_ends_the_call_without_reaching_transport( #[case] failure: HostFailure, @@ -174,7 +181,7 @@ async fn a_before_send_failure_ends_the_call_without_reaching_transport( .await; assert!( - matches!(&outcome, Err(Error::InvalidRequest(actual)) if actual == message), + matches!(&outcome, Err(Error::MissingField(actual)) if *actual == message), "{outcome:?}" ); assert_eq!(ops, ["BeforeSend"]); @@ -255,17 +262,17 @@ async fn interrupt_drops_provider_captures_before_returning() { drive_until_notified(&mut machine, &host, &entered).await; assert!(!dropped.load(Ordering::SeqCst)); - let acknowledgement = machine.interrupt(HostFailure::Cancelled(Error::InvalidRequest( - "cancelled".into(), - ))); + let acknowledgement = + machine.interrupt(HostFailure::Cancelled(Error::MissingField("cancelled"))); assert!( dropped.load(Ordering::SeqCst), "interrupt returned while provider captures were still alive" ); - assert!( - matches!(acknowledgement.await, Err(Error::InvalidRequest(message)) if message == "cancelled") - ); + assert!(matches!( + acknowledgement.await, + Err(Error::MissingField("cancelled")) + )); } #[rstest::rstest] @@ -290,7 +297,7 @@ async fn interrupting_an_in_flight_provider_request_closes_its_connection() { let mut machine = ocr_route().machine(host.request().unwrap(), None); drive_until_notified(&mut machine, &host, &received).await; - let cancelled = Error::InvalidRequest("cancelled".into()); + let cancelled = Error::MissingField("cancelled"); assert!( machine diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/main.rs b/litellm-rust/crates/inference-ocr/tests/ocr/main.rs index 5a905f907d2..5a48bba9e1a 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/main.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/main.rs @@ -209,7 +209,7 @@ impl LocalOcrHost { request, caller_token: false, }) - .ok_or_else(|| Error::InvalidRequest("OCR request was already projected".into())) + .ok_or(Error::MissingField("OCR request")) } pub fn runtime(&self) -> litellm_host_native::in_process::Host<'_, Self, Self, ()> { litellm_host_native::in_process::Host { @@ -223,11 +223,11 @@ impl LocalOcrHost { impl litellm_host_native::services::HostCallHandler for LocalOcrHost { async fn handle_host_call(&self, op: OcrOp) -> Result<(), Error> { match op { - OcrOp::AcquireAzureAdToken(_) => { - Err(Error::Auth(litellm_auth::Error::CredentialAcquisition( - "OCR host has no Azure AD token provider".into(), - ))) - } + OcrOp::AcquireAzureAdToken(_) => Err(Error::Auth( + litellm_auth::Error::CredentialAcquisition(litellm_auth::ErrorDetail::Unresolved { + subject: "OCR host Azure AD token provider", + }), + )), } } } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/mistral.rs b/litellm-rust/crates/inference-ocr/tests/ocr/mistral.rs index 528adc7feaf..21cf4ce455d 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/mistral.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/mistral.rs @@ -92,28 +92,13 @@ async fn an_upstream_error_keeps_its_status_whole_body_and_headers( .unwrap_err(); assert_eq!(received(&upstream).await.len(), 1); - let Error::Provider { - status, - body, - headers, - } = error - else { - panic!("expected provider error, got {error:?}"); - }; + let (status, body, headers) = error.upstream_response().expect("the provider answered"); assert_eq!(status, 422); - for (name, value) in [ - ("retry-after", "17"), - ("x-request-id", "request-123"), - ("x-future-header", "retained"), - ] { - assert!( - headers - .iter() - .any(|(key, actual)| key.eq_ignore_ascii_case(name) && actual == value), - "{name} missing from {headers:?}" - ); - } assert_eq!(body, expected_body); + let header = |name: &str| headers.get(name).and_then(|value| value.to_str().ok()); + assert_eq!(header("retry-after"), Some("17")); + assert_eq!(header("x-request-id"), Some("request-123")); + assert_eq!(header("x-future-header"), Some("retained")); } #[rstest] diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/reducto.rs b/litellm-rust/crates/inference-ocr/tests/ocr/reducto.rs index 649dca3d690..f6f458a4916 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/reducto.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/reducto.rs @@ -175,8 +175,11 @@ async fn a_failed_upload_stops_before_parse(#[case] upload: ResponseTemplate) { } #[rstest] -#[case::remote_url("https://example.com/a.pdf", Error::ReductoSource)] -#[case::empty_file_id("reducto://", Error::RequestField { path: "document file id".into() })] +#[case::remote_url( + "https://example.com/a.pdf", + Error::UnsupportedDocument { provider: "Reducto", guidance: "" } +)] +#[case::empty_file_id("reducto://", Error::RequestField { path: "document file id".into(), cause: None })] #[case::data_uri_without_payload("data:application/pdf;base64", Error::InvalidDataUri)] #[case::invalid_base64("data:application/pdf;base64,INVALID!", Error::InvalidDataUri)] #[tokio::test] @@ -202,7 +205,6 @@ async fn invalid_document_sources_are_rejected_before_sending( std::mem::discriminant(&expected) ); assert_eq!(error.http_status_code(), Some(400)); - assert_eq!(error.to_string(), expected.to_string()); } #[tokio::test] diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/vertex_ai.rs b/litellm-rust/crates/inference-ocr/tests/ocr/vertex_ai.rs index 81b52d076a5..dba12de60f6 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/vertex_ai.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/vertex_ai.rs @@ -71,7 +71,8 @@ async fn invalid_credentials_fail_before_sending() { .await .unwrap_err(); - assert!(error.to_string().contains("vertex_credentials"), "{error}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("vertex_credentials"), "{report}"); } #[rstest] @@ -91,11 +92,10 @@ async fn a_request_controlled_api_base_is_rejected_before_vertex_auth( let error = perform(request).await.unwrap_err(); + let report = litellm_error::Report::new(&error).to_string(); assert!( - error - .to_string() - .contains("request-controlled Vertex AI endpoint"), - "{error}" + report.contains("request-controlled Vertex AI endpoint"), + "{report}" ); } diff --git a/litellm-rust/crates/inference-responses/Cargo.toml b/litellm-rust/crates/inference-responses/Cargo.toml index 3dcd9976e0a..083179c50c8 100644 --- a/litellm-rust/crates/inference-responses/Cargo.toml +++ b/litellm-rust/crates/inference-responses/Cargo.toml @@ -28,6 +28,7 @@ tracing.workspace = true litellm-http = { workspace = true, features = ["test-support"] } litellm-cache.workspace = true litellm-cache-memory.workspace = true +litellm-error.workspace = true litellm-host-native.workspace = true litellm-inference-testing.workspace = true litellm-tracing.workspace = true diff --git a/litellm-rust/crates/inference-responses/src/handler.rs b/litellm-rust/crates/inference-responses/src/handler.rs index cbbc8be139d..8d3d1075e8e 100644 --- a/litellm-rust/crates/inference-responses/src/handler.rs +++ b/litellm-rust/crates/inference-responses/src/handler.rs @@ -72,7 +72,14 @@ pub(super) async fn execute( let stream = match wire.body.get("stream") { None => false, Some(serde_json::Value::Bool(value)) => *value, - Some(_) => return Err(Error::InvalidRequest("stream must be a boolean".into())), + Some(_) => { + return Err(Error::invalid_request( + litellm_llms::ErrorDetail::Expected { + subject: "stream", + expected: "a boolean", + }, + )); + } }; let outbound = litellm_inference::outbound::outbound_request( Authenticated { @@ -83,17 +90,11 @@ pub(super) async fn execute( &wire.body, Some(call.timeout.unwrap_or(Duration::from_secs(600))), )?; - let response = litellm_inference::outbound::send(outbound, &route.http) - .await - .map_err(network)?; + let response = litellm_inference::outbound::send(outbound, &route.http).await?; let status = response.status().as_u16(); if !response.status().is_success() { - let body = response.text().await.map_err(network)?; - return Err(litellm_http::transport::Error::Http { - status, - body: litellm_http::request::truncate_error_body(&body), - } - .into()); + let body = response.text().await.map_err(litellm_http::Error::from)?; + return Err(litellm_http::Error::upstream(status, body).into()); } if stream { let headers = response @@ -105,14 +106,14 @@ pub(super) async fn execute( .collect(); let chunks = response .bytes_stream() - .map(|chunk| chunk.map_err(network)) + .map(|chunk| chunk.map_err(|error| litellm_http::Error::from(error).into())) .boxed(); return Ok(ResponsesOutput::Stream { head: ResponsesStreamHead { headers }, chunks, }); } - let body = response.text().await.map_err(network)?; + let body = response.text().await.map_err(litellm_http::Error::from)?; let raw = RawResponse { body: body.clone() }; if let Some(observers) = observers { observers.emit(litellm_host::lifecycle::CallEvent::Execution( @@ -123,8 +124,12 @@ pub(super) async fn execute( .after_provider_response(raw) .await .map_err(Error::post_call)?; - let value = serde_json::from_str(&body) - .map_err(|error| Error::InvalidResponse(error.to_string().into()))?; + let value = serde_json::from_str(&body).map_err(|error| { + Error::invalid_response(litellm_llms::ErrorDetail::invalid( + "Responses API response JSON", + error, + )) + })?; config .transform_response_api_response(value) .map(ResponsesOutput::Complete) @@ -133,7 +138,3 @@ pub(super) async fn execute( ) .await } - -fn network(error: reqwest::Error) -> Error { - litellm_http::transport::Error::Network(error.to_string()).into() -} diff --git a/litellm-rust/crates/inference-responses/src/websocket.rs b/litellm-rust/crates/inference-responses/src/websocket.rs index c2ca89ecbaf..2afcde0a7a4 100644 --- a/litellm-rust/crates/inference-responses/src/websocket.rs +++ b/litellm-rust/crates/inference-responses/src/websocket.rs @@ -11,6 +11,8 @@ use tokio_tungstenite::tungstenite::{ http::{HeaderMap, HeaderName, HeaderValue}, }; +use litellm_llms::ErrorDetail; + use super::Error; #[derive(Clone)] @@ -36,23 +38,22 @@ impl ResponsesWebSocketConnection { .iter() .map(|(name, value)| { Ok(( - name.parse::() - .map_err(|error| Error::InvalidRequest(error.to_string().into()))?, - HeaderValue::from_str(value) - .map_err(|error| Error::InvalidRequest(error.to_string().into()))?, + name.parse::().map_err(|error| { + Error::invalid_request(ErrorDetail::invalid( + "WebSocket header name", + error, + )) + })?, + HeaderValue::from_str(value).map_err(|error| { + Error::invalid_request(ErrorDetail::invalid( + "WebSocket header value", + error, + )) + })?, )) }) .collect::>()?; - let connect = websocket::connect(client, url, headers); - let socket = match timeout { - Some(timeout) => tokio::time::timeout(timeout, connect).await.map_err(|_| { - Error::Transport(litellm_http::transport::Error::Network( - "Responses WebSocket connection timed out".into(), - )) - })?, - None => connect.await, - } - .map_err(Error::Transport)?; + let socket = websocket::connect(client, url, headers, timeout).await?; Ok(Self { socket: Arc::new(Mutex::new(Some(socket))), }) @@ -70,13 +71,12 @@ impl ResponsesWebSocketConnection { litellm_inference::diagnostic::operation("litellm.websocket.send_text", async { let mut socket = self.socket.lock().await; let Some(socket) = socket.as_mut() else { - return Err(Error::Transport(litellm_http::transport::Error::Network( - "Responses WebSocket is closed".into(), - ))); + return Err(litellm_http::Error::websocket_closed().into()); }; - socket.send(Message::Text(text)).await.map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - }) + socket + .send(Message::Text(text)) + .await + .map_err(|error| litellm_http::Error::from(error).into()) }) .await } @@ -97,12 +97,12 @@ impl ResponsesWebSocketConnection { Some(Ok(Message::Text(text))) => Ok(Some(text)), Some(Ok(Message::Binary(bytes))) => String::from_utf8(bytes.to_vec()) .map(Some) - .map_err(|error| Error::InvalidResponse(error.to_string().into())), + .map_err(|error| { + Error::invalid_response(ErrorDetail::invalid("WebSocket text frame", error)) + }), Some(Ok(Message::Close(_))) | None => Ok(None), Some(Ok(_)) => Ok(None), - Some(Err(error)) => Err(Error::Transport(litellm_http::transport::Error::Network( - error.to_string(), - ))), + Some(Err(error)) => Err(litellm_http::Error::from(error).into()), } }) .await @@ -118,9 +118,10 @@ impl ResponsesWebSocketConnection { litellm_inference::diagnostic::operation("litellm.websocket.close", async { let mut socket = self.socket.lock().await; if let Some(socket) = socket.as_mut() { - socket.close(None).await.map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - })?; + socket + .close(None) + .await + .map_err(litellm_http::Error::from)?; } *socket = None; Ok(()) diff --git a/litellm-rust/crates/inference-responses/tests/openrouter.rs b/litellm-rust/crates/inference-responses/tests/openrouter.rs index e52cdba8ce8..8cdb68c376f 100644 --- a/litellm-rust/crates/inference-responses/tests/openrouter.rs +++ b/litellm-rust/crates/inference-responses/tests/openrouter.rs @@ -1,10 +1,7 @@ use std::sync::Arc; use futures_util::TryStreamExt; -use litellm_inference_responses::{ - Error, - types::{ResponsesCall, ResponsesOutput}, -}; +use litellm_inference_responses::types::{ResponsesCall, ResponsesOutput}; use litellm_inference_testing::{RecordingSecrets, no_secrets}; use rstest::{fixture, rstest}; use serde_json::json; @@ -108,13 +105,12 @@ async fn a_missing_key_fails_before_transport(call: ResponsesCall) { None, ) .await; - assert!(matches!( - result, - Err(Error::Auth(litellm_auth::Error::MissingApiKey { - provider: "OpenRouter", - .. - })) - )); + let Err(error) = result else { + panic!("the call fails"); + }; + assert!(error.is_missing_credential(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("OpenRouter"), "{report}"); assert!(received(&upstream).await.is_empty()); } diff --git a/litellm-rust/crates/inference-responses/tests/responses.rs b/litellm-rust/crates/inference-responses/tests/responses.rs index 6d39d55d33c..66402e595eb 100644 --- a/litellm-rust/crates/inference-responses/tests/responses.rs +++ b/litellm-rust/crates/inference-responses/tests/responses.rs @@ -304,6 +304,25 @@ async fn provider_failures_emit_failure_once( ); } +#[rstest] +#[tokio::test] +async fn a_transport_error_does_not_echo_the_provider_url(call: ResponsesCall) { + let host = RecordingCall::::new(ResponsesCall { + api_base: Some(format!("{UNREACHABLE_BASE}/tenant-token-1234")), + ..call + }); + let call = host.request.lock().unwrap().take().unwrap(); + + let Err(error) = responses_route(no_secrets()) + .execute(call, &host, None) + .await + else { + panic!("nothing is listening"); + }; + + assert!(!error.to_string().contains("tenant-token-1234"), "{error}"); +} + #[rstest] #[case::explicit(true)] #[case::from_secrets(false)] @@ -557,12 +576,11 @@ async fn websocket_operations_trace_outcomes_without_capturing_frames_or_credent Some("private-frame-sentinel") ); connection.close().await.unwrap(); - assert!( - connection - .send_text("private-frame-sentinel".into()) - .await - .is_err() - ); + let error = connection + .send_text("private-frame-sentinel".into()) + .await + .expect_err("a closed connection sends nothing"); + assert!(!error.dispatched(), "{error:?}"); }) .await; server.await.unwrap(); diff --git a/litellm-rust/crates/inference-responses/tests/support/mod.rs b/litellm-rust/crates/inference-responses/tests/support/mod.rs index 81f133a0471..984b878eb21 100644 --- a/litellm-rust/crates/inference-responses/tests/support/mod.rs +++ b/litellm-rust/crates/inference-responses/tests/support/mod.rs @@ -13,6 +13,9 @@ use litellm_secrets::source::SecretSource; use serde_json::Value; use wiremock::{Mock, MockServer, Request, ResponseTemplate, matchers::any}; +/// A port nothing listens on, for calls that must fail before any request is sent. +pub const UNREACHABLE_BASE: &str = "http://127.0.0.1:1"; + pub fn responses_route( secrets: Arc, ) -> litellm_inference_responses::ResponsesRoute { diff --git a/litellm-rust/crates/inference-transcription/Cargo.toml b/litellm-rust/crates/inference-transcription/Cargo.toml index c501d329852..cf50b3f84fd 100644 --- a/litellm-rust/crates/inference-transcription/Cargo.toml +++ b/litellm-rust/crates/inference-transcription/Cargo.toml @@ -17,6 +17,7 @@ serde_json.workspace = true tracing.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-inference-testing.workspace = true litellm-tracing.workspace = true rstest.workspace = true diff --git a/litellm-rust/crates/inference-transcription/src/handler.rs b/litellm-rust/crates/inference-transcription/src/handler.rs index 6541ae8263e..ba7eb740e2f 100644 --- a/litellm-rust/crates/inference-transcription/src/handler.rs +++ b/litellm-rust/crates/inference-transcription/src/handler.rs @@ -2,7 +2,7 @@ use std::time::Duration; use litellm_http::{ Client, - request::{string_headers, truncate_error_body, with_default_headers}, + request::{string_headers, with_default_headers}, }; use litellm_llms::base_llm::{ audio_transcription::transformation::BaseAudioTranscriptionConfig, @@ -55,23 +55,14 @@ pub(super) async fn execute( .unwrap_or(Duration::from_secs(AUDIO_TRANSCRIPTION_TIMEOUT_SECS)), ), )?; - let response = litellm_inference::outbound::send(outbound, http) - .await - .map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - })?; + let response = litellm_inference::outbound::send(outbound, http).await?; let status = response.status(); - let text = response.text().await.map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - })?; + let text = response.text().await.map_err(litellm_http::Error::from)?; if !status.is_success() { - return Err(Error::Transport(litellm_http::transport::Error::Http { - status: status.as_u16(), - body: truncate_error_body(&text), - })); + return Err(litellm_http::Error::upstream(status.as_u16(), text).into()); } let response_json = serde_json::from_str(&text).map_err(|error| { - Error::InvalidResponse(litellm_llms::ErrorDetail::invalid( + Error::invalid_response(litellm_llms::ErrorDetail::invalid( "audio response JSON", error, )) diff --git a/litellm-rust/crates/inference-transcription/tests/audio_transcription.rs b/litellm-rust/crates/inference-transcription/tests/audio_transcription.rs index eb252bdff7e..9e48edce0c0 100644 --- a/litellm-rust/crates/inference-transcription/tests/audio_transcription.rs +++ b/litellm-rust/crates/inference-transcription/tests/audio_transcription.rs @@ -152,13 +152,7 @@ async fn invalid_audio_is_rejected_before_sending( .await .expect_err("invalid audio is rejected"); - assert!( - matches!( - error, - Error::InvalidRequest(_) | Error::MissingField(_) | Error::InvalidType { .. } - ), - "{error:?}" - ); + assert!(error.is_request(), "{error:?}"); assert!(received(&upstream).await.is_empty()); } @@ -185,7 +179,9 @@ async fn unsupported_providers_are_rejected_before_sending( .await .expect_err("unsupported provider errors"); - assert_eq!(error, Error::InvalidProvider(reported.into())); + assert!(error.is_request(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains(reported), "{report}"); } #[rstest] @@ -222,14 +218,28 @@ async fn an_upstream_error_keeps_its_status_and_body( .expect_err("upstream error propagates"); assert_eq!( - error, - Error::Transport(litellm_http::transport::Error::Http { - status, - body: "upstream said no".into() - }) + error.upstream_response(), + Some((status, "upstream said no")) ); } +#[rstest] +#[tokio::test] +async fn a_transport_error_does_not_echo_the_provider_url( + request: AudioTranscriptionRequest<'static>, +) { + let base = format!("{UNREACHABLE_BASE}/tenant-token-1234"); + + let error = transcribe(AudioTranscriptionRequest { + api_base: Some(&base), + ..request + }) + .await + .expect_err("nothing is listening"); + + assert!(!error.to_string().contains("tenant-token-1234"), "{error}"); +} + #[rstest] #[case::not_json(ResponseTemplate::new(200).set_body_string("not json"))] #[case::no_output(json_response(json!({"unexpected": true})))] @@ -248,7 +258,7 @@ async fn an_unreadable_success_body_is_an_invalid_response( .await .expect_err("an unreadable body fails"); - assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + assert!(error.is_response(), "{error:?}"); } #[rstest] diff --git a/litellm-rust/crates/inference/AGENTS.md b/litellm-rust/crates/inference/AGENTS.md index 7f2bd0387be..8cc8fc8c26b 100644 --- a/litellm-rust/crates/inference/AGENTS.md +++ b/litellm-rust/crates/inference/AGENTS.md @@ -53,10 +53,10 @@ ## Error placement -- The workspace `Error definitions` rules shape each crate's error. This section decides which crate and module a failure belongs to +- The [rust-error skill](../../.agents/skills/rust-error/SKILL.md) shapes each crate's error. This section decides which crate and module a failure belongs to - A failure is declared once, by the lowest crate that raises it - - every crate above nests it unchanged (`#[error(transparent)] Auth(#[from] litellm_auth::Error)`) or maps it once at its boundary, as `src/error.rs` does for `litellm_llms::Error` - - `Error` collects route failures for every format crate and never re-declares a variant a lower crate raises + - every crate above nests it in one variant and delegates its predicates, as `Error::Llm`, `Error::Auth` and `Error::Http` do, or translates it once in the shared `From` when its meaning changes, as `From` does for `Unsupported` and `From` does for a header the signer computes + - `Error` collects route failures for every format crate and never re-declares a variant a lower crate raises. Routes build transformation failures with `Error::invalid_request` and `Error::invalid_response`, and callers ask `is_unsupported()`, `is_request()`, `is_response()`, `is_missing_credential()`, `dispatched()`, `upstream_response()` and `rejected_request()` instead of matching nested variants - Scope follows the concept, not the first caller - an error type under `litellm-llms`'s `/` is private to that provider: no other provider and nothing in `base_llm` may import it - a failure two providers or two routes can hit (wire framing, stream event decoding, a malformed provider response) belongs to the crate that owns the concept: `litellm-framer` for framing, `litellm_llms::Error` for the transformation layer diff --git a/litellm-rust/crates/inference/Cargo.toml b/litellm-rust/crates/inference/Cargo.toml index acc211f214b..807a3ea85d3 100644 --- a/litellm-rust/crates/inference/Cargo.toml +++ b/litellm-rust/crates/inference/Cargo.toml @@ -29,6 +29,7 @@ time.workspace = true url.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-cache-memory.workspace = true litellm-http = { workspace = true, features = ["test-support"] } litellm-llms = { workspace = true, features = ["test-support"] } diff --git a/litellm-rust/crates/inference/src/error.rs b/litellm-rust/crates/inference/src/error.rs index 895c8c55e01..2b42d2fced1 100644 --- a/litellm-rust/crates/inference/src/error.rs +++ b/litellm-rust/crates/inference/src/error.rs @@ -1,57 +1,47 @@ -//! One error for every route in this crate. OCR still carries its own, richer enum. -//! -//! A variant is declared by the layer that produces it and nested here as is: -//! credentials by `litellm_auth` (AWS folds into it at that crate's boundary), the wire by -//! `litellm_http`, secrets by `litellm_secrets`. The transformation layer's [`LlmError`] -//! maps onto the same-named variants once, here, so no route re-declares them. - use std::sync::Arc; -use litellm_http::transport::Error as TransportError; -use litellm_llms::Error as LlmError; +use litellm_host::machine::MachineFault; +use litellm_llms::{Error as LlmError, ErrorDetail}; #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum Error { - #[error("expected {expected}, got {actual}")] - InvalidType { - expected: &'static str, - actual: &'static str, - }, - #[error("missing required field: {0}")] - MissingField(&'static str), #[error("invalid provider: {0}")] InvalidProvider(String), - #[error("invalid request: {0}")] - InvalidRequest(#[source] litellm_llms::ErrorDetail), - #[error("invalid response: {0}")] - InvalidResponse(#[source] litellm_llms::ErrorDetail), + #[error("invalid provider: unable to resolve custom_llm_provider for {route} request")] + UnresolvedProvider { route: &'static str }, #[error("unsupported by the rust path: {0}")] Unsupported(&'static str), #[error(transparent)] - Auth(#[from] litellm_auth::Error), + Llm(LlmError), #[error(transparent)] - Transport(#[from] TransportError), + Auth(#[from] litellm_auth::Error), #[error(transparent)] Headers(#[from] litellm_http::request::HeaderError), #[error(transparent)] - Http(#[from] litellm_http::Error), + Http(litellm_http::Error), #[error(transparent)] Secret(#[from] SecretError), - #[error("post-call hook failed: {0}")] + #[error("the host driver failed")] + Host(#[source] MachineFault), + #[error("post-call hook failed")] PostCallHook(#[source] Arc), } -impl From for Error { - fn from(fault: litellm_host::machine::MachineFault) -> Self { - use litellm_host::machine::MachineFault; - Self::InvalidRequest(match fault { - MachineFault::Abandoned => "host driver was abandoned".into(), - MachineFault::Protocol(message) => format!("host {message}").into(), - }) +impl From for Error { + fn from(fault: MachineFault) -> Self { + Self::Host(fault) } } impl Error { + pub fn invalid_request(detail: ErrorDetail) -> Self { + Self::Llm(LlmError::InvalidRequest(detail)) + } + + pub fn invalid_response(detail: ErrorDetail) -> Self { + Self::Llm(LlmError::InvalidResponse(detail)) + } + pub fn post_call(error: Self) -> Self { Self::PostCallHook(Arc::new(error)) } @@ -60,31 +50,128 @@ impl Error { /// the provider's answer. pub fn is_request(&self) -> bool { match self { - Self::InvalidType { .. } - | Self::MissingField(_) - | Self::InvalidProvider(_) - | Self::InvalidRequest(_) + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } | Self::Unsupported(_) | Self::Headers(_) => true, - Self::Auth(error) => !matches!(error, litellm_auth::Error::MissingApiKey { .. }), - Self::InvalidResponse(_) - | Self::Transport(_) - | Self::Http(_) + Self::Llm(error) => error.is_request(), + Self::Auth(error) => error.is_request(), + Self::Http(error) => error.is_request(), + Self::Secret(_) | Self::Host(_) | Self::PostCallHook(_) => false, + } + } + + pub fn is_response(&self) -> bool { + match self { + Self::Llm(error) => error.is_response(), + Self::Http(error) => error.is_response(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Unsupported(_) + | Self::Auth(_) + | Self::Headers(_) | Self::Secret(_) + | Self::Host(_) | Self::PostCallHook(_) => false, } } + + pub fn is_missing_credential(&self) -> bool { + match self { + Self::Llm(error) => error.is_missing_credential(), + Self::Auth(error) => error.is_missing_credential(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Unsupported(_) + | Self::Headers(_) + | Self::Http(_) + | Self::Secret(_) + | Self::Host(_) + | Self::PostCallHook(_) => false, + } + } + + pub fn is_unsupported(&self) -> bool { + match self { + Self::Unsupported(_) => true, + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Llm(_) + | Self::Auth(_) + | Self::Headers(_) + | Self::Http(_) + | Self::Secret(_) + | Self::Host(_) + | Self::PostCallHook(_) => false, + } + } + + pub fn dispatched(&self) -> bool { + match self { + Self::Llm(error) => error.dispatched(), + Self::Http(error) => error.dispatched(), + Self::Host(_) | Self::PostCallHook(_) => true, + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Unsupported(_) + | Self::Auth(_) + | Self::Headers(_) + | Self::Secret(_) => false, + } + } + + pub fn rejected_request(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Llm(error) => error.rejected_request(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Unsupported(_) + | Self::Auth(_) + | Self::Headers(_) + | Self::Http(_) + | Self::Secret(_) + | Self::Host(_) + | Self::PostCallHook(_) => None, + } + } + + pub fn upstream_response(&self) -> Option<(u16, &str)> { + match self { + Self::Llm(error) => error.upstream_response(), + Self::Http(error) => error.upstream_response(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Unsupported(_) + | Self::Auth(_) + | Self::Headers(_) + | Self::Secret(_) + | Self::Host(_) + | Self::PostCallHook(_) => None, + } + } +} + +impl From for Error { + fn from(error: litellm_http::Error) -> Self { + // Python drops the caller's copy and prefers a forwarded Authorization + // over the signature, so leave the request to it. + if error.is_computed_header() { + return Self::Unsupported("request forwards a header AWS SigV4 computes"); + } + Self::Http(error) + } } impl From for Error { fn from(error: LlmError) -> Self { match error { - LlmError::InvalidType { expected, actual } => Self::InvalidType { expected, actual }, - LlmError::MissingField(field) => Self::MissingField(field), - LlmError::InvalidRequest(message) => Self::InvalidRequest(message), - LlmError::InvalidResponse(message) => Self::InvalidResponse(message), LlmError::Unsupported(reason) => Self::Unsupported(reason), + LlmError::Http(error) => Self::from(error), LlmError::Auth(error) => Self::Auth(error), + error @ (LlmError::InvalidType { .. } + | LlmError::MissingField(_) + | LlmError::InvalidRequest(_) + | LlmError::InvalidResponse(_)) => Self::Llm(error), } } } @@ -116,26 +203,151 @@ impl Eq for SecretError {} #[cfg(test)] mod tests { use super::Error; + use litellm_host::machine::MachineFault; use litellm_llms::{Error as LlmError, ErrorDetail}; use rstest::rstest; - #[test] - fn a_missing_api_key_is_the_environment_not_the_request() { - assert!( - !Error::Auth(litellm_auth::Error::MissingApiKey { - provider: "Anthropic", - environment_variable: "ANTHROPIC_API_KEY", - }) - .is_request() - ); - assert!(Error::Auth(litellm_auth::Error::InvalidHeader).is_request()); - assert!(Error::InvalidRequest("top_k".into()).is_request()); - assert!(!Error::InvalidResponse("bad json".into()).is_request()); + #[rstest] + #[case::missing_api_key( + litellm_auth::Error::MissingApiKey { + provider: "Anthropic", + environment_variable: "ANTHROPIC_API_KEY", + } + .into(), + false + )] + #[case::invalid_auth_header(litellm_auth::Error::InvalidHeader.into(), true)] + #[case::transformation_missing_api_key( + LlmError::Auth(litellm_auth::Error::MissingApiKey { + provider: "Anthropic", + environment_variable: "ANTHROPIC_API_KEY", + }) + .into(), + false + )] + #[case::invalid_request( + LlmError::InvalidRequest(ErrorDetail::Expected { + subject: "top_k", + expected: "an integer", + }) + .into(), + true + )] + #[case::invalid_response( + LlmError::InvalidResponse(ErrorDetail::MissingField { + subject: "response", + field: "usage", + }) + .into(), + false + )] + #[case::host_driver(MachineFault::Abandoned.into(), false)] + #[case::post_call_hook_over_a_request_fault( + Error::post_call(LlmError::MissingField("usage").into()), + false + )] + fn only_the_callers_input_is_a_request_fault(#[case] error: Error, #[case] is_request: bool) { + assert_eq!(error.is_request(), is_request); } + + #[rstest] + #[case::own_auth(litellm_auth::Error::MissingApiKey { + provider: "Anthropic", + environment_variable: "ANTHROPIC_API_KEY", + } + .into())] + #[case::transformation_auth( + LlmError::Auth(litellm_auth::Error::MissingApiKey { + provider: "Anthropic", + environment_variable: "ANTHROPIC_API_KEY", + }) + .into() + )] + fn a_missing_api_key_is_a_missing_credential_wherever_it_is_raised(#[case] error: Error) { + assert!(error.is_missing_credential()); + } + + #[rstest] + #[case::direct(litellm_http::Error::computed_header("authorization").into())] + #[case::through_a_transformation( + LlmError::Http(litellm_http::Error::computed_header("authorization")).into() + )] + fn a_header_the_signer_computes_is_left_to_the_host_on_every_route(#[case] error: Error) { + assert!(error.is_unsupported(), "{error:?}"); + assert!(error.is_request()); + } + + #[test] + fn a_transformation_unsupported_stays_a_decline() { + assert!(Error::from(LlmError::Unsupported("test capability")).is_unsupported()); + } + + #[test] + fn an_upstream_failure_through_a_transformation_keeps_the_provider_answer() { + let error = Error::from(LlmError::Http(litellm_http::Error::upstream( + 429, + "slow down".into(), + ))); + assert_eq!(error.upstream_response(), Some((429, "slow down"))); + assert!(error.dispatched()); + } + + #[rstest] + #[case::invalid_request( + Error::invalid_request(ErrorDetail::Expected { + subject: "messages", + expected: "non-empty", + }), + Some("messages must be non-empty") + )] + #[case::missing_field( + LlmError::MissingField("max_tokens").into(), + Some("missing required field: max_tokens") + )] + #[case::invalid_response( + Error::invalid_response(ErrorDetail::MissingField { + subject: "response", + field: "usage", + }), + None + )] + #[case::provider_rejection(litellm_http::Error::upstream(400, "bad".into()).into(), None)] + #[case::unresolved_provider(Error::InvalidProvider("unknown".into()), None)] + fn a_rejected_request_names_only_what_the_caller_got_wrong( + #[case] error: Error, + #[case] expected: Option<&str>, + ) { + let rejection = error + .rejected_request() + .map(|rejection| litellm_error::Report::new(rejection).to_string()); + assert_eq!(rejection.as_deref(), expected); + } + + #[test] + fn a_post_call_hook_failure_is_not_the_providers_answer() { + let error = + Error::post_call(litellm_http::Error::upstream(500, "hook backend down".into()).into()); + assert_eq!(error.upstream_response(), None); + assert!(!error.is_request()); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.starts_with("post-call hook failed: "), "{report}"); + assert!(report.contains("hook backend down"), "{report}"); + } + + #[test] + fn a_host_driver_fault_keeps_the_fault_as_its_source() { + let error = Error::from(MachineFault::Abandoned); + assert_eq!( + litellm_error::find::(&error), + Some(&MachineFault::Abandoned) + ); + assert!(error.dispatched()); + } + #[rstest] #[case::request(true)] #[case::response(false)] - fn contextual_errors_preserve_sources_and_route_classification(#[case] request: bool) { + fn contextual_errors_keep_their_cause_and_route_classification(#[case] request: bool) { let source = serde_json::from_str::("{").unwrap_err(); let source_message = source.to_string(); let detail = ErrorDetail::invalid("test payload", source); @@ -145,16 +357,10 @@ mod tests { LlmError::InvalidResponse(detail) }); assert_eq!(error.is_request(), request); - let category = if request { "request" } else { "response" }; - assert_eq!( - error.to_string(), - format!("invalid {category}: invalid test payload: {source_message}") - ); - let source = std::iter::successors(Some(&error as &dyn std::error::Error), |error| { - error.source() - }) - .find_map(|error| error.downcast_ref::()) - .expect("the original JSON error remains available"); - assert_eq!(source.to_string(), source_message); + assert_eq!(error.is_response(), !request); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("test payload"), "{report}"); + assert_eq!(report.matches(&source_message).count(), 1, "{report}"); + assert!(litellm_error::find::(&error).is_some()); } } diff --git a/litellm-rust/crates/inference/src/outbound.rs b/litellm-rust/crates/inference/src/outbound.rs index 9d9b1e606c2..b0cc71d425d 100644 --- a/litellm-rust/crates/inference/src/outbound.rs +++ b/litellm-rust/crates/inference/src/outbound.rs @@ -13,10 +13,10 @@ use serde_json::Value; pub async fn send( request: reqwest::Request, client: &litellm_http::Client, -) -> Result { - client.execute(request).await.inspect(|response| { - tracing::Span::current().record("status", response.status().as_u16()); - }) +) -> Result { + let response = client.execute(request).await?; + tracing::Span::current().record("status", response.status().as_u16()); + Ok(response) } /// Header credentials are already in `headers`; SigV4 is applied here, over the diff --git a/litellm-rust/crates/inference/src/provider.rs b/litellm-rust/crates/inference/src/provider.rs index 305aca85f97..fa226a4ef8d 100644 --- a/litellm-rust/crates/inference/src/provider.rs +++ b/litellm-rust/crates/inference/src/provider.rs @@ -25,11 +25,7 @@ pub fn resolve_llm_provider<'a>( custom_llm_provider: provider, }) }) - .ok_or_else(|| { - Error::InvalidProvider(format!( - "unable to resolve custom_llm_provider for {route} request" - )) - })?; + .ok_or(Error::UnresolvedProvider { route })?; let provider = custom_llm_provider .parse() .map_err(|_| Error::InvalidProvider(custom_llm_provider.to_string()))?; diff --git a/litellm-rust/crates/inference/tests/caching.rs b/litellm-rust/crates/inference/tests/caching.rs index a0a6a757565..02c2edd95dd 100644 --- a/litellm-rust/crates/inference/tests/caching.rs +++ b/litellm-rust/crates/inference/tests/caching.rs @@ -417,7 +417,7 @@ impl litellm_cache::BaseCache for UnavailableCache { _: &str, _: &Self::Context, ) -> Result, litellm_cache::Error> { - Err(litellm_cache::Error::Unavailable) + Err(litellm_cache::Error::Unavailable(None)) } fn set_cache( @@ -426,7 +426,7 @@ impl litellm_cache::BaseCache for UnavailableCache { _: Self::Value, _: &Self::Context, ) -> Result<(), litellm_cache::Error> { - Err(litellm_cache::Error::Unavailable) + Err(litellm_cache::Error::Unavailable(None)) } } diff --git a/litellm-rust/crates/llms/Cargo.toml b/litellm-rust/crates/llms/Cargo.toml index 2d3ea85d1fc..962f61ef4ec 100644 --- a/litellm-rust/crates/llms/Cargo.toml +++ b/litellm-rust/crates/llms/Cargo.toml @@ -11,6 +11,7 @@ test-support = ["litellm-http/test-support"] [dependencies] litellm-llms-types.workspace = true litellm-core-utils.workspace = true +litellm-error.workspace = true litellm-auth = { workspace = true, features = ["aws", "azure", "gcp"] } litellm-auth-aws.workspace = true litellm-auth-azure.workspace = true diff --git a/litellm-rust/crates/llms/src/anthropic/batches/transformation.rs b/litellm-rust/crates/llms/src/anthropic/batches/transformation.rs index b8ea701af99..66658802dd5 100644 --- a/litellm-rust/crates/llms/src/anthropic/batches/transformation.rs +++ b/litellm-rust/crates/llms/src/anthropic/batches/transformation.rs @@ -149,7 +149,12 @@ impl AnthropicBatchesConfig for AnthropicBatchesTransformation { } let mut url = batches_base_url(api_base, env_lookup)?; url.path_segments_mut() - .map_err(|_| Error::InvalidRequest("Anthropic API base cannot be a base URL".into()))? + .map_err(|()| { + Error::InvalidRequest(crate::ErrorDetail::Expected { + subject: "Anthropic API base", + expected: "a base URL", + }) + })? .push(batch_id); Ok(url.into()) } @@ -217,11 +222,11 @@ impl AnthropicBatchesConfig for AnthropicBatchesTransformation { .map(|(index, line)| { let record: BatchResultRecord = serde_json::from_str(line.trim()).map_err(|error| { - Error::InvalidResponse(crate::ErrorDetail::InvalidLine { - subject: "Anthropic batch result", - line: index + 1, - source: crate::ErrorSource::new(error), - }) + Error::InvalidResponse(crate::ErrorDetail::invalid_line( + "Anthropic batch result", + index + 1, + error, + )) })?; match record.result { BatchResult::Succeeded { message } => Ok(*message), @@ -314,18 +319,22 @@ mod tests { assert_eq!(messages[0].id, "msg_1"); } - #[test] - fn reports_malformed_and_unsuccessful_batch_results() { - assert!(matches!( - ANTHROPIC_BATCHES_TRANSFORMATION.transform_batch_results("not-json"), - Err(Error::InvalidResponse(message)) if message.to_string().contains("line 1") - )); - assert!(matches!( - ANTHROPIC_BATCHES_TRANSFORMATION.transform_batch_results( - r#"{"result":{"type":"errored","error":{"type":"invalid_request_error"}}}"# - ), - Err(Error::InvalidResponse(message)) if message.to_string().contains("request failed") - )); + #[rstest::rstest] + #[case::malformed("not-json", "line 1")] + #[case::errored( + r#"{"result":{"type":"errored","error":{"type":"invalid_request_error"}}}"#, + "request failed" + )] + fn reports_malformed_and_unsuccessful_batch_results(#[case] body: &str, #[case] fact: &str) { + let error = ANTHROPIC_BATCHES_TRANSFORMATION + .transform_batch_results(body) + .unwrap_err(); + assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains(fact) + ); } #[test] diff --git a/litellm-rust/crates/llms/src/anthropic/chat/handler.rs b/litellm-rust/crates/llms/src/anthropic/chat/handler.rs deleted file mode 100644 index eaee006f228..00000000000 --- a/litellm-rust/crates/llms/src/anthropic/chat/handler.rs +++ /dev/null @@ -1,163 +0,0 @@ -use std::collections::HashMap; - -use litellm_llms_types::formats::{ - chat_completions::{ - ChatCompletionChunk, ChatCompletionThinkingBlock, ChatCompletionToolCallChunk, - ChatCompletionsUsage, - }, - messages::streaming::{ - MessagesContentBlock, MessagesContentBlockDelta, MessagesStreamEvent, MessagesStreamUsage, - }, -}; -use serde_json::Value; - -use crate::{ - Error, - base_llm::{base_model_iterator::StreamTransformer, chat::streaming::StreamShape}, -}; - -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum AnthropicJsonChunkType { - ValidJson, - AccumulatedJson, -} - -#[derive(Clone, Debug, Eq, PartialEq)] -pub enum AnthropicContentBlockType { - Text, - ToolUse, - ServerToolUse, - Thinking, - RedactedThinking, - Compaction, - ToolResult(String), - Other(String), -} - -#[derive(Clone, Debug, PartialEq)] -pub struct AnthropicContentBlockDeltaEvent { - pub index: u64, - pub delta: MessagesContentBlockDelta, -} - -pub struct ModelResponseIterator { - pub content_blocks: Vec, - pub tool_index: i64, - pub json_mode: bool, - pub speed: Option, - pub tool_name_reverse_map: HashMap, - pub response_id: String, - pub served_model: Option, - pub is_response_format_tool: bool, - pub converted_response_format_tool: bool, - pub accumulated_json: String, - pub chunk_type: AnthropicJsonChunkType, - pub current_content_block_type: Option, - pub web_search_results: Vec, - pub web_search_calls: HashMap, - pub compaction_blocks: Vec, - pub reasoning_content_chunks: Vec, - pub server_tool_inputs: HashMap, - pub tool_results: Vec, - pub current_server_tool_id: Option, - pub container_id: Option, -} - -impl ModelResponseIterator { - pub fn new(_shape: StreamShape) -> Self { - todo!() - } - - pub fn check_empty_tool_call_args(&self) -> bool { - todo!() - } - - pub fn handle_usage(&mut self, _usage: MessagesStreamUsage) -> ChatCompletionsUsage { - todo!() - } - - pub fn handle_content_block_delta( - &mut self, - _index: u64, - _delta: MessagesContentBlockDelta, - ) -> ( - String, - Option, - Vec, - Option, - Option, - ) { - todo!() - } - - pub fn handle_content_block_start( - &mut self, - _index: u64, - _content_block: MessagesContentBlock, - ) -> Result { - todo!() - } - - pub fn handle_json_mode_chunk( - &mut self, - _text: String, - _tool_use: Option, - ) -> (String, Option) { - todo!() - } - - pub fn handle_accumulated_json_chunk( - &mut self, - _data: &str, - _is_final: bool, - ) -> Result, Error> { - todo!() - } - - pub fn handle_redacted_thinking_content( - &mut self, - _content_block: &MessagesContentBlock, - ) -> Vec { - todo!() - } - - pub fn web_search_call_snapshot(&self) -> HashMap { - todo!() - } - - pub fn complete_web_search_call(&mut self, _result: Value) { - todo!() - } - - pub fn build_code_interpreter_results(&self) -> Vec { - todo!() - } - - pub fn handle_message_delta( - &mut self, - _event: MessagesStreamEvent, - ) -> (Option, Option, Option) { - todo!() - } - - pub fn chunk_parser( - &mut self, - _event: MessagesStreamEvent, - ) -> Result { - todo!() - } -} - -impl StreamTransformer for ModelResponseIterator { - type Input = MessagesStreamEvent; - type Output = ChatCompletionChunk; - type Error = Error; - - fn transform(&mut self, _input: Self::Input) -> Result, Self::Error> { - todo!() - } - - fn finish(&mut self) -> Result, Self::Error> { - todo!() - } -} diff --git a/litellm-rust/crates/llms/src/anthropic/chat/mod.rs b/litellm-rust/crates/llms/src/anthropic/chat/mod.rs index f0050b7dc71..f239b6921fa 100644 --- a/litellm-rust/crates/llms/src/anthropic/chat/mod.rs +++ b/litellm-rust/crates/llms/src/anthropic/chat/mod.rs @@ -1,2 +1 @@ -pub mod handler; pub mod transformation; diff --git a/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs b/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs index 91baa247ac8..4d4f110fbe1 100644 --- a/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs @@ -14,23 +14,19 @@ use serde_json::{Map, Value, json}; use crate::{ Error, - anthropic::{ - chat::handler::ModelResponseIterator, - common_utils::{ - API_KEY_PLACEMENT, complete_anthropic_url, forwarded_oauth_bearer, - resolve_anthropic_api_key, - }, + anthropic::common_utils::{ + API_KEY_PLACEMENT, complete_anthropic_url, forwarded_oauth_bearer, + resolve_anthropic_api_key, }, base_llm::{ auth::AuthScheme, chat::{ - streaming::{ChatStream, StreamShape}, + streaming::StreamShape, transformation::{ BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData, Unsupported, ValidatedEnvironment, unsupported_message, unsupported_param, }, }, - messages::streaming::event_stream as messages_event_stream, }, }; @@ -194,13 +190,6 @@ impl BaseConfig for AnthropicConfig { Ok(ValidatedEnvironment { headers, auth }) } - fn model_response_iterator(&self, shape: StreamShape) -> Option { - Some(ChatStream::new( - messages_event_stream, - ModelResponseIterator::new(shape), - )) - } - fn default_headers(&self) -> &'static [(&'static str, &'static str)] { DEFAULT_HEADERS } @@ -676,16 +665,11 @@ mod tests { #[case::missing_model(json!({"content": [], "usage": {"input_tokens": 1, "output_tokens": 1}}))] fn errors_on_a_response_missing_required_fields(#[case] body: Value) { let error = transform_response(body).expect_err("invalid response"); - assert!(matches!(error, Error::InvalidResponse(_))); - let source = std::iter::successors(Some(&error as &dyn std::error::Error), |error| { - error.source() - }) - .find_map(|error| error.downcast_ref::()) - .expect("the JSON decoding source is preserved"); - assert_eq!( - error.to_string(), - format!("invalid response: invalid messages response: {source}") - ); + assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + let cause = litellm_error::find::(&error).expect("JSON cause"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("invalid messages response"), "{report}"); + assert_eq!(report.matches(&cause.to_string()).count(), 1, "{report}"); } #[test] diff --git a/litellm-rust/crates/llms/src/anthropic/messages/handler.rs b/litellm-rust/crates/llms/src/anthropic/messages/handler.rs index 6e8de65d6fa..54f924ddec6 100644 --- a/litellm-rust/crates/llms/src/anthropic/messages/handler.rs +++ b/litellm-rust/crates/llms/src/anthropic/messages/handler.rs @@ -208,24 +208,29 @@ mod tests { #[case::empty(json!({}), Ok(json!({})))] #[case::numeric_user_id( json!({"user_id": 123}), - Err(Error::InvalidRequest("metadata.user_id must be a string, got 123".to_string().into())), + Err(invalid_value("metadata.user_id", "a string", json!(123))), )] #[case::boolean_user_id( json!({"user_id": true}), - Err(Error::InvalidRequest("metadata.user_id must be a string, got true".to_string().into())), + Err(invalid_value("metadata.user_id", "a string", json!(true))), )] #[case::not_an_object( json!(["u-1"]), - Err(Error::InvalidRequest(r#"metadata must be an object, got ["u-1"]"#.to_string().into())), + Err(invalid_value("metadata", "an object", json!(["u-1"]))), )] fn validate_anthropic_api_metadata_passes_only_a_string_user_id( #[case] metadata: Value, #[case] expected: Result, ) { - assert_eq!( - validate_anthropic_api_metadata(&metadata).map_err(|error| error.to_string()), - expected.map_err(|error| error.to_string()), - ); + assert_eq!(validate_anthropic_api_metadata(&metadata), expected); + } + + fn invalid_value(field: &'static str, expected: &'static str, actual: Value) -> Error { + Error::InvalidRequest(crate::ErrorDetail::InvalidValue { + field, + expected, + actual, + }) } #[rstest] diff --git a/litellm-rust/crates/llms/src/aws_textract/ocr/analyze_transformation.rs b/litellm-rust/crates/llms/src/aws_textract/ocr/analyze_transformation.rs index e813d4ac8cc..82119cecfa5 100644 --- a/litellm-rust/crates/llms/src/aws_textract/ocr/analyze_transformation.rs +++ b/litellm-rust/crates/llms/src/aws_textract/ocr/analyze_transformation.rs @@ -123,7 +123,7 @@ impl BaseOcrConfig for TextractAnalyzeDocumentConfig { &self, error_message: String, status_code: u16, - headers: Vec<(String, String)>, + headers: reqwest::header::HeaderMap, ) -> Error { error_class(error_message, status_code, headers) } diff --git a/litellm-rust/crates/llms/src/aws_textract/ocr/common_utils.rs b/litellm-rust/crates/llms/src/aws_textract/ocr/common_utils.rs index bc199f40938..e3a4f947433 100644 --- a/litellm-rust/crates/llms/src/aws_textract/ocr/common_utils.rs +++ b/litellm-rust/crates/llms/src/aws_textract/ocr/common_utils.rs @@ -17,6 +17,7 @@ const AWS_JSON_CONTENT_TYPE: &str = "application/x-amz-json-1.1"; const TARGET_HEADER: &str = "X-Amz-Target"; const CONTENT_TYPE_HEADER: &str = "Content-Type"; const UNSUPPORTED_DOCUMENT: &str = "UnsupportedDocumentException"; +const SYNC_API_GUIDANCE: &str = "aws_textract uses Textract's synchronous API, which reads a JPEG, PNG, or a single-page PDF or TIFF; other formats and multi-page documents are not supported"; const SYNC_DOCUMENT_MAX_BYTES: usize = 10 * 1024 * 1024; const HEALTH_CHECK_IMAGE_DATA_URI: &str = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4//8/AAX+Av4N70a4AAAAAElFTkSuQmCC"; @@ -321,20 +322,18 @@ struct AwsError { /// Textract answers both an unsupported format and a multi-page PDF or TIFF /// with a bare "unsupported document format", which reads like a corrupt file. /// Say what the synchronous API accepts. -pub(super) fn error_class(body: String, status: u16, headers: Vec<(String, String)>) -> Error { - let unsupported = serde_json::from_str::(&body) - .ok() - .filter(|error| error.kind.ends_with(UNSUPPORTED_DOCUMENT)); - Error::Provider { - status, - body: match unsupported { - Some(error) => format!( - "{UNSUPPORTED_DOCUMENT}: {}. aws_textract uses Textract's synchronous API, which reads a JPEG, PNG, or a single-page PDF or TIFF; other formats and multi-page documents are not supported", - error.message - ), - None => body, +pub(super) fn error_class(body: String, status: u16, headers: reqwest::header::HeaderMap) -> Error { + match serde_json::from_str::(&body) { + Ok(error) if error.kind.ends_with(UNSUPPORTED_DOCUMENT) => Error::RejectedDocument { + provider: "AWS Textract", + message: error.message, + guidance: SYNC_API_GUIDANCE, + }, + Ok(_) | Err(_) => Error::Provider { + status, + body, + headers: Box::new(headers), }, - headers, } } @@ -408,11 +407,11 @@ mod tests { fn a_model_outside_the_operations_is_refused_with_the_supported_names(#[case] model: &str) { let error = TextractOperation::from_model(model).unwrap_err(); - assert_eq!( - error.to_string(), - format!( - "invalid model: aws_textract has no model {model:?} - use one of: detect-document-text, analyze-document" - ) + let message = error.to_string(); + assert!(message.contains(&format!("{model:?}")), "{message}"); + assert!( + message.contains("detect-document-text, analyze-document"), + "{message}" ); assert_eq!(error.http_status_code(), Some(400)); } @@ -542,43 +541,46 @@ mod tests { #[rstest] #[case::bare_type( r#"{"__type":"UnsupportedDocumentException","Message":"Request has unsupported document format"}"#, - Some("Request has unsupported document format") + "Request has unsupported document format" )] #[case::namespaced_type( r#"{"__type":"com.amazonaws.textract#UnsupportedDocumentException","Message":"bad"}"#, - Some("bad") + "bad" )] #[case::lowercase_message( r#"{"__type":"UnsupportedDocumentException","message":"bad"}"#, - Some("bad") + "bad" )] - #[case::other_exception(r#"{"__type":"AccessDeniedException","Message":"no"}"#, None)] - #[case::json_without_a_type(r#"{"Message":"no"}"#, None)] - #[case::not_json("bad gateway", None)] - fn only_an_unsupported_document_gains_the_sync_api_hint( + fn an_unsupported_document_is_the_callers_fault_with_the_sync_api_hint( #[case] body: &str, - #[case] hinted_message: Option<&str>, + #[case] provider_message: &str, ) { - let response_headers = vec![("x-amzn-requestid".to_string(), "abc".to_string())]; + let error = error_class(body.into(), 400, reqwest::header::HeaderMap::new()); - let Error::Provider { - status, - body: reported, - headers, - } = error_class(body.into(), 400, response_headers.clone()) - else { - panic!("expected a provider error"); - }; + let message = error.to_string(); + assert!(error.is_request()); + assert_eq!(error.http_status_code(), Some(400)); + assert!(error.upstream_response().is_none()); + assert!(message.contains(provider_message), "{message}"); + assert!(message.contains(HINT), "{message}"); + } - assert_eq!(status, 400); - assert_eq!(headers, response_headers); - match hinted_message { - Some(message) => { - assert!(reported.contains(message), "{reported}"); - assert!(reported.contains(HINT), "{reported}"); - } - None => assert_eq!(reported, body), - } + #[rstest] + #[case::other_exception(r#"{"__type":"AccessDeniedException","Message":"no"}"#)] + #[case::json_without_a_type(r#"{"Message":"no"}"#)] + #[case::not_json("bad gateway")] + fn any_other_rejection_passes_through_untouched(#[case] body: &str) { + let response_headers = reqwest::header::HeaderMap::from_iter([( + reqwest::header::HeaderName::from_static("x-amzn-requestid"), + reqwest::header::HeaderValue::from_static("abc"), + )]); + + let error = error_class(body.into(), 403, response_headers.clone()); + + assert_eq!( + error.upstream_response(), + Some((403, body, &response_headers)) + ); } #[rstest] diff --git a/litellm-rust/crates/llms/src/aws_textract/ocr/transformation.rs b/litellm-rust/crates/llms/src/aws_textract/ocr/transformation.rs index 4b27a191734..34213926942 100644 --- a/litellm-rust/crates/llms/src/aws_textract/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/aws_textract/ocr/transformation.rs @@ -104,7 +104,7 @@ impl BaseOcrConfig for TextractDetectTextConfig { &self, error_message: String, status_code: u16, - headers: Vec<(String, String)>, + headers: reqwest::header::HeaderMap, ) -> Error { error_class(error_message, status_code, headers) } @@ -237,7 +237,7 @@ mod tests { let error = TextractDetectTextConfig.get_error_class( r#"{"__type":"UnsupportedDocumentException","Message":"Request has unsupported document format"}"#.into(), 400, - Vec::new(), + reqwest::header::HeaderMap::new(), ); assert!( diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/cohere_parse_transformation.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/cohere_parse_transformation.rs index 72e321547e3..bec8bee3f34 100644 --- a/litellm-rust/crates/llms/src/azure_ai/ocr/cohere_parse_transformation.rs +++ b/litellm-rust/crates/llms/src/azure_ai/ocr/cohere_parse_transformation.rs @@ -120,7 +120,8 @@ impl BaseOcrConfig for AzureAICohereParseConfig { impl AzureAICohereParseConfig { fn get_complete_url(&self, base: &str) -> Result { - let mut url = reqwest::Url::parse(base).map_err(|_| invalid_api_base())?; + let mut url = + reqwest::Url::parse(base).map_err(|error| Error::request_field("api_base", error))?; if !matches!(url.scheme(), "http" | "https") { return Err(invalid_api_base()); } @@ -133,13 +134,14 @@ impl AzureAICohereParseConfig { ApiUrl::parse(url.as_str()) .and_then(|url| url.complete_path(&AZURE_COHERE_PARSE_PATH)) .map(|url| url.into_string()) - .map_err(|_| invalid_api_base()) + .map_err(|error| Error::request_field("api_base", error)) } } fn invalid_api_base() -> Error { Error::RequestField { path: "api_base".into(), + cause: None, } } diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/common_utils.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/common_utils.rs index 8cbf2455e01..7da685febf9 100644 --- a/litellm-rust/crates/llms/src/azure_ai/ocr/common_utils.rs +++ b/litellm-rust/crates/llms/src/azure_ai/ocr/common_utils.rs @@ -26,7 +26,7 @@ pub(super) async fn resolve_entra( .get_azure_ad_token(config, env_lookup) .await .or_else(|error| match error { - litellm_auth::Error::EmptyCallerCredential(_) => Ok(None), + litellm_auth::Error::EmptyCallerCredential { .. } => Ok(None), other => Err(other), }) .map(|credential| { @@ -48,7 +48,7 @@ pub(super) fn validate_destination( && credential_source != InputSource::Request { return Err(litellm_auth::Error::InvalidConfiguration( - "host credentials cannot be sent to a request-controlled Azure endpoint".into(), + litellm_auth::ErrorDetail::RequestControlledEndpoint { provider: "Azure" }, ) .into()); } diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs index 2a50989f4e4..d7233db0a76 100644 --- a/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs +++ b/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs @@ -13,7 +13,7 @@ use tokio::time::Instant; use crate::base_llm::ocr::{ document::InlineDocument, - error::Error, + error::{Error, PagesFault, PollUrl}, handler::{CallHooks, OcrClient, read_json_response}, settings::OcrSettings, transformation::{ @@ -185,7 +185,10 @@ impl BaseOcrConfig for AzureDocumentIntelligenceOcrConfig { ) -> Result { let endpoint = nonblank(request.connection.api_base.clone()) .or_else(|| nonblank(request.connection.secret(AZURE_DI_ENDPOINT_ENV))) - .ok_or_else(|| Error::Auth(litellm_auth::Error::ProviderAuthentication("Missing Azure Document Intelligence API Base - Set AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT or pass api_base".into())))?; + .ok_or(Error::Auth(litellm_auth::Error::MissingApiBase { + provider: "Azure Document Intelligence", + guidance: "Set AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT or pass api_base", + }))?; self.build_ocr_url( &endpoint, &request.model, @@ -256,14 +259,12 @@ fn normalize_pages_param(pages: Option<&Value>) -> Result, Error> Some(Value::Array(pages)) if pages.iter().all(Value::is_number) => pages .iter() .map(|page| { - let page = page - .as_i64() - .ok_or_else(|| Error::Pages("page index is out of range".into()))?; + let page = page.as_i64().ok_or(Error::Pages(PagesFault::OutOfRange))?; if page < 0 { - return Err(Error::Pages("negative page index".into())); + return Err(Error::Pages(PagesFault::Negative)); } page.checked_add(1) - .ok_or_else(|| Error::Pages("page index is out of range".into())) + .ok_or(Error::Pages(PagesFault::OutOfRange)) }) .collect::, _>>()? .into_iter() @@ -276,7 +277,7 @@ fn normalize_pages_param(pages: Option<&Value>) -> Result, Error> token .as_str() .map(str::trim) - .ok_or_else(|| Error::Pages("expected only integers or only strings".into())) + .ok_or(Error::Pages(PagesFault::Mixed)) }) .collect::, _>>()? .join(","), @@ -286,13 +287,11 @@ fn normalize_pages_param(pages: Option<&Value>) -> Result, Error> .collect::>() .join(","), Some(_) => { - return Err(Error::Pages( - "expected an array of integers or strings, or a native page range".into(), - )); + return Err(Error::Pages(PagesFault::Shape)); } }; if !normalized.split(',').all(valid_page_token) { - return Err(Error::Pages("invalid native page range".into())); + return Err(Error::Pages(PagesFault::Range)); } Ok(Some(normalized)) } @@ -458,8 +457,14 @@ async fn read_operation_response( .and_then(|value| value.to_str().ok()) .ok_or(Error::PollLocation)? .to_string(); - let original = Url::parse(original_url).map_err(|_| Error::PollOrigin)?; - let operation = Url::parse(&location).map_err(|_| Error::PollOrigin)?; + let original = Url::parse(original_url).map_err(|source| Error::InvalidPollUrl { + url: PollUrl::Submission, + source, + })?; + let operation = Url::parse(&location).map_err(|source| Error::InvalidPollUrl { + url: PollUrl::OperationLocation, + source, + })?; if original.origin() != operation.origin() || !operation.username().is_empty() || operation.password().is_some() @@ -504,8 +509,7 @@ async fn poll_operation( let response = tokio::time::timeout_at(deadline, litellm_http::request::http_request(builder)) .await - .map_err(|_| Error::PollTimeout)? - .map_err(litellm_http::transport::Error::from)?; + .map_err(|_| Error::PollTimeout)??; let retry = response .headers() .get(reqwest::header::RETRY_AFTER) @@ -578,9 +582,7 @@ impl AzureDocumentIntelligenceOcrConfig { ) .into_string() }) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } async fn resolve_headers( @@ -623,7 +625,10 @@ impl AzureDocumentIntelligenceOcrConfig { } let token = super::super::common_utils::resolve_entra(auth, config, env_lookup) .await? - .ok_or(Error::MissingAzureDocumentIntelligenceCredentials)?; + .ok_or(Error::MissingCredential { + credential: "Azure Document Intelligence credentials", + guidance: "set AZURE_DOCUMENT_INTELLIGENCE_API_KEY or configure Entra ID", + })?; super::super::common_utils::validate_destination(connection, token.source())?; Ok( std::iter::once(("Authorization".into(), format!("Bearer {}", token.value()))) @@ -842,7 +847,7 @@ mod tests { .unwrap_err(); assert!( - error + litellm_error::Report::new(&error) .to_string() .contains("request-controlled Azure endpoint") ); diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs index 83d88bbb3cb..2f7957fe56e 100644 --- a/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs @@ -169,7 +169,10 @@ impl AzureAiOcrConfig { } let key = super::common_utils::resolve_entra(auth, config, env_lookup) .await? - .ok_or(Error::MissingAzureAiCredentials)?; + .ok_or(Error::MissingCredential { + credential: "Azure AI credentials", + guidance: "set AZURE_AI_API_KEY or configure Entra ID", + })?; super::common_utils::validate_destination(connection, key.source())?; Ok(bearer_headers(connection, key.value())) } @@ -183,9 +186,7 @@ impl AzureAiOcrConfig { ApiUrl::parse(&base) .and_then(|url| url.complete_path(&AZURE_AI_OCR_PATH)) .map(|url| url.into_string()) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } } @@ -302,7 +303,7 @@ mod tests { .unwrap_err(); assert!( - error + litellm_error::Report::new(&error) .to_string() .contains("request-controlled Azure endpoint") ); diff --git a/litellm-rust/crates/llms/src/base_llm/base_model_iterator.rs b/litellm-rust/crates/llms/src/base_llm/base_model_iterator.rs index a283ca84089..6df321b4b16 100644 --- a/litellm-rust/crates/llms/src/base_llm/base_model_iterator.rs +++ b/litellm-rust/crates/llms/src/base_llm/base_model_iterator.rs @@ -5,6 +5,15 @@ use futures_util::{Stream, StreamExt, stream, stream::BoxStream}; pub type ByteStream = BoxStream<'static, Result>; +/// The transport error a body stream carried as an `io::Error`, however many times framing +/// wrapped it on the way. +pub(crate) fn tunnelled_http(error: &io::Error) -> Option<&litellm_http::Error> { + let inner = error.get_ref()?; + inner + .downcast_ref::() + .or_else(|| tunnelled_http(inner.downcast_ref::()?)) +} + pub trait StreamTransformer { type Input; type Output; diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/document.rs b/litellm-rust/crates/llms/src/base_llm/ocr/document.rs index 57976240e1b..898c24369c3 100644 --- a/litellm-rust/crates/llms/src/base_llm/ocr/document.rs +++ b/litellm-rust/crates/llms/src/base_llm/ocr/document.rs @@ -1,9 +1,6 @@ use base64::{Engine, engine::general_purpose::STANDARD}; use data_url::{DataUrl, DataUrlError, forgiving_base64::DecodeError, mime::Mime}; -use litellm_http::{ - media::{DownloadPolicy, Error as MediaError, MediaFetcher}, - transport::Error as TransportError, -}; +use litellm_http::media::{DownloadPolicy, MediaFetcher}; use reqwest::Url; use crate::base_llm::ocr::{ @@ -61,9 +58,7 @@ pub async fn inline_remote_document( validate_inline_document(&document)?; return Ok(document); } - let url = Url::parse(source).map_err(|_| Error::RequestField { - path: "document URL".into(), - })?; + let url = Url::parse(source).map_err(|error| Error::request_field("document URL", error))?; let downloaded = fetcher .fetch( url, @@ -74,7 +69,7 @@ pub async fn inline_remote_document( }, ) .await - .map_err(map_media_error)?; + .map_err(Error::DocumentDownload)?; let result = document.with_source(format!( "data:{};base64,{}", downloaded.content_type, @@ -84,28 +79,6 @@ pub async fn inline_remote_document( Ok(result) } -fn map_media_error(error: MediaError) -> Error { - match error { - MediaError::BlockedUrl => Error::BlockedDocumentUrl, - MediaError::DownloadDisabled => Error::DownloadDisabled, - MediaError::DownloadTooLarge => Error::DownloadTooLarge, - MediaError::TooManyRedirects => Error::TooManyRedirects, - MediaError::MissingRedirectLocation => Error::MissingRedirectLocation, - MediaError::InvalidRedirect => Error::InvalidRedirect, - MediaError::Http(status) => TransportError::Http { - status, - body: "OCR document download failed".into(), - } - .into(), - MediaError::Timeout => TransportError::Http { - status: 408, - body: "OCR document download timed out".into(), - } - .into(), - MediaError::Transport(error) => error.into(), - } -} - #[cfg(test)] mod tests { use std::collections::BTreeMap as Map; @@ -119,20 +92,17 @@ mod tests { } } - #[test] - fn decodes_data_urls_and_limits_decoded_size() { - for (source, expected) in [ - ("data:application/pdf;base64,YWJj", b"abc".as_slice()), - ("DATA:application/pdf;BASE64,YWI", b"ab".as_slice()), - ("data:,a%20b%00%FF", b"a b\0\xff".as_slice()), - ] { - let inline = InlineDocument::parse(source).unwrap().unwrap(); - assert_eq!(inline.decode(expected.len()).unwrap(), expected); - assert!(matches!( - inline.decode(expected.len() - 1), - Err(Error::InlineDocumentTooLarge) - )); - } + #[rstest::rstest] + #[case::base64("data:application/pdf;base64,YWJj", b"abc")] + #[case::uppercase_unpadded("DATA:application/pdf;BASE64,YWI", b"ab")] + #[case::percent_encoded("data:,a%20b%00%FF", b"a b\0\xff")] + fn decodes_data_urls_and_limits_decoded_size(#[case] source: &str, #[case] expected: &[u8]) { + let inline = InlineDocument::parse(source).unwrap().unwrap(); + assert_eq!(inline.decode(expected.len()).unwrap(), expected); + assert!(matches!( + inline.decode(expected.len() - 1), + Err(Error::InlineDocumentTooLarge) + )); } #[test] @@ -150,15 +120,12 @@ mod tests { ); } - #[test] - fn rejects_invalid_inline_documents() { - for source in [ - "https://example.com/document.pdf", - "data:application/pdf;base64", - "data:application/pdf;base64,INVALID!", - ] { - assert!(validate_inline_document(&document(source)).is_err()); - } + #[rstest::rstest] + #[case::remote("https://example.com/document.pdf")] + #[case::no_payload("data:application/pdf;base64")] + #[case::invalid_base64("data:application/pdf;base64,INVALID!")] + fn rejects_invalid_inline_documents(#[case] source: &str) { + assert!(validate_inline_document(&document(source)).is_err()); } #[tokio::test] diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/error.rs b/litellm-rust/crates/llms/src/base_llm/ocr/error.rs index 392dd56412c..9a0e0519f12 100644 --- a/litellm-rust/crates/llms/src/base_llm/ocr/error.rs +++ b/litellm-rust/crates/llms/src/base_llm/ocr/error.rs @@ -1,53 +1,62 @@ +use std::{io, path::PathBuf, sync::Arc}; + +use litellm_core_utils::call_arguments::ArgumentError; +use litellm_error::Source; +use reqwest::header::HeaderMap; + +static NO_HEADERS: std::sync::LazyLock = std::sync::LazyLock::new(HeaderMap::new); + #[derive(Clone, Debug, thiserror::Error)] pub enum Error { #[error("upstream OCR error ({status}): {body}")] Provider { status: u16, body: String, - headers: Vec<(String, String)>, + headers: Box, }, #[error("File is empty or could not be read")] EmptyFile, - #[error("Failed to read OCR file {}: {source}", path.display())] + #[error("could not read OCR file {}", path.display())] FileRead { - path: std::path::PathBuf, + path: PathBuf, #[source] - source: std::sync::Arc, + cause: Arc, }, - #[error("OCR document preparation task failed: {0}")] - DocumentTask(#[source] std::sync::Arc), - #[error("Invalid MIME type: {0}")] + #[error("OCR document preparation task failed")] + DocumentTask(#[source] Arc), + #[error("invalid MIME type {0:?}")] InvalidMimeType(String), - #[error( - "Cohere Parse only accepts `image_url` documents; document_url and PDF inputs are not supported" - )] - CohereImageOnly, + #[error("{provider} cannot read this document - {guidance}")] + UnsupportedDocument { + provider: &'static str, + guidance: &'static str, + }, + #[error("{provider} cannot read this document: {message} - {guidance}")] + RejectedDocument { + provider: &'static str, + message: String, + guidance: &'static str, + }, #[error("Invalid `req_format`. Expected 'native' or 'litellm'.")] RequestFormat, #[error("invalid OCR request field: {path}")] - RequestField { path: String }, + RequestField { + path: String, + #[source] + cause: Option, + }, + #[error("invalid OCR optional params")] + Arguments(#[from] ArgumentError), #[error("missing required field: {0}")] MissingField(&'static str), - #[error("Document URL is required")] + #[error("document URL is required")] MissingDocumentUrl, #[error("invalid OCR document data URI")] InvalidDataUri, - #[error( - "Reducto requires a reducto:// id or a data URI; plain HTTP URLs are not supported, upload the file first" - )] - ReductoSource, #[error("inline OCR document exceeds the size limit")] InlineDocumentTooLarge, - #[error("OCR document URL is blocked by network policy")] - BlockedDocumentUrl, - #[error("OCR document downloads are disabled")] - DownloadDisabled, - #[error("OCR document download exceeds the size limit")] - DownloadTooLarge, - #[error("OCR document download exceeded the redirect limit")] - TooManyRedirects, #[error("invalid OCR pages: {0}")] - Pages(String), + Pages(PagesFault), #[error("invalid OCR features")] Features, #[error("OCR model cannot be a dot segment")] @@ -55,19 +64,25 @@ pub enum Error { #[error("OCR response exceeds the size limit of {limit} bytes")] TooLarge { limit: usize }, #[error("invalid OCR response field: {path}")] - ResponseField { path: String }, + ResponseField { + path: String, + #[source] + cause: Option, + }, #[error("OCR response is missing non-empty content")] EmptyContent, - #[error("OCR document redirect is missing a location")] - MissingRedirectLocation, - #[error("OCR document redirect location is invalid")] - InvalidRedirect, #[error("OCR operation ended with status {0}")] OperationStatus(String), #[error("OCR response numeric value is out of range: {0}")] NumericRange(&'static str), #[error("OCR accepted response is missing a valid operation-location")] PollLocation, + #[error("OCR {url} is not a valid URL")] + InvalidPollUrl { + url: PollUrl, + #[source] + source: url::ParseError, + }, #[error("OCR operation-location must use the submission origin without credentials")] PollOrigin, #[error("OCR polling timed out")] @@ -82,108 +97,313 @@ pub enum Error { model: String, supported: &'static [&'static str], }, - #[error("invalid request: {0}")] - InvalidRequest(String), - #[error("invalid response: {0}")] - InvalidResponse(String), - #[error( - "invalid authentication configuration: Missing Azure AI credentials - set AZURE_AI_API_KEY or configure Entra ID" - )] - MissingAzureAiCredentials, - #[error( - "invalid authentication configuration: Missing Azure Document Intelligence credentials - set AZURE_DOCUMENT_INTELLIGENCE_API_KEY or configure Entra ID" - )] - MissingAzureDocumentIntelligenceCredentials, - #[error( - "Missing REDUCTO_API_KEY - set it in the environment or pass api_key to litellm.ocr()/litellm.aocr()" - )] - MissingReductoApiKey, - #[error("secret resolution failed: {0}")] - Secret(#[source] std::sync::Arc), + #[error("Missing {credential} - {guidance}")] + MissingCredential { + credential: &'static str, + guidance: &'static str, + }, + #[error("secret resolution failed")] + Secret(#[source] Arc), + #[error("the OCR host driver failed")] + Host(#[source] litellm_host::machine::MachineFault), + #[error("the OCR host call raised an exception")] + HostCall, #[error(transparent)] Auth(#[from] litellm_auth::Error), #[error(transparent)] - Transport(#[from] litellm_http::transport::Error), - #[error(transparent)] Params(#[from] litellm_core_utils::params::Error), #[error(transparent)] Headers(#[from] litellm_http::request::HeaderError), #[error(transparent)] Http(#[from] litellm_http::Error), + #[error(transparent)] + DocumentDownload(litellm_http::Error), +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum PagesFault { + #[error("page index is out of range")] + OutOfRange, + #[error("negative page index")] + Negative, + #[error("expected only integers or only strings")] + Mixed, + #[error("expected an array of integers or strings, or a native page range")] + Shape, + #[error("invalid native page range")] + Range, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum PollUrl { + #[error("submission URL")] + Submission, + #[error("operation-location")] + OperationLocation, } impl From for Error { fn from(fault: litellm_host::machine::MachineFault) -> Self { - use litellm_host::machine::MachineFault; - Self::InvalidRequest(match fault { - MachineFault::Abandoned => "OCR host driver was abandoned".into(), - MachineFault::Protocol(message) => format!("OCR {message}"), - }) - } -} - -impl From for Error { - fn from(error: litellm_core_utils::call_arguments::ArgumentError) -> Self { - Self::RequestField { - path: format!("optional_params.{}", error.path), - } + Self::Host(fault) } } impl Error { - pub fn http_status_code(&self) -> Option { - match self { - Self::Provider { status, .. } - | Self::Transport(litellm_http::transport::Error::Http { status, .. }) => Some(*status), - error if error.is_request() => Some(400), - _ => None, + pub fn request_field( + path: impl Into, + cause: impl std::error::Error + Send + Sync + 'static, + ) -> Self { + Self::RequestField { + path: path.into(), + cause: Some(Source::new(cause)), } } - pub fn is_request(&self) -> bool { - matches!( - self, + pub fn response_field( + path: impl Into, + cause: impl std::error::Error + Send + Sync + 'static, + ) -> Self { + Self::ResponseField { + path: path.into(), + cause: Some(Source::new(cause)), + } + } + + pub fn upstream_response(&self) -> Option<(u16, &str, &HeaderMap)> { + match self { + Self::Provider { + status, + body, + headers, + } => Some((*status, body, headers)), + Self::DocumentDownload(error) => error.status().map(|status| { + let body = if error.is_timeout() { + "OCR document download timed out" + } else { + "OCR document download failed" + }; + (status, body, &*NO_HEADERS) + }), + Self::Http(error) => error + .upstream_response() + .map(|(status, body)| (status, body, &*NO_HEADERS)), Self::EmptyFile - | Self::InvalidMimeType(_) - | Self::CohereImageOnly - | Self::RequestFormat - | Self::RequestField { .. } - | Self::MissingField(_) - | Self::MissingDocumentUrl - | Self::InvalidDataUri - | Self::ReductoSource - | Self::InlineDocumentTooLarge - | Self::BlockedDocumentUrl - | Self::DownloadDisabled - | Self::DownloadTooLarge - | Self::TooManyRedirects - | Self::Pages(_) - | Self::Features - | Self::DotModel - | Self::InvalidRequest(_) - | Self::InvalidProvider(_) - | Self::InvalidModel { .. } - | Self::Params(_) - | Self::Headers(_) - | Self::Http(_) - | Self::Auth(litellm_auth::Error::MissingParam { .. }) - ) + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::DotModel + | Self::TooLarge { .. } + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout + | Self::Unsupported(_) + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::MissingCredential { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall + | Self::Auth(_) + | Self::Params(_) + | Self::Headers(_) => None, + } + } + + pub fn is_missing_credential(&self) -> bool { + match self { + Self::MissingCredential { .. } => true, + Self::Auth(error) => error.is_missing_credential(), + Self::Provider { .. } + | Self::EmptyFile + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::DotModel + | Self::TooLarge { .. } + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout + | Self::Unsupported(_) + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall + | Self::Params(_) + | Self::Headers(_) + | Self::Http(_) + | Self::DocumentDownload(_) => false, + } + } + + pub fn is_unsupported(&self) -> bool { + match self { + Self::Unsupported(_) => true, + Self::Provider { .. } + | Self::EmptyFile + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::DotModel + | Self::TooLarge { .. } + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::MissingCredential { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall + | Self::Auth(_) + | Self::Params(_) + | Self::Headers(_) + | Self::Http(_) + | Self::DocumentDownload(_) => false, + } + } + + pub fn http_status_code(&self) -> Option { + if let Some((status, ..)) = self.upstream_response() { + return Some(status); + } + if let Self::Http(error) = self + && let Some(status) = error.status() + { + return Some(status); + } + self.is_request().then_some(400) + } + + pub fn is_request(&self) -> bool { + match self { + Self::EmptyFile + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::DotModel + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::Params(_) + | Self::Headers(_) => true, + Self::Auth(error) => error.is_request(), + Self::Http(error) | Self::DocumentDownload(error) => error.is_request(), + Self::Provider { .. } + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::TooLarge { .. } + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout + | Self::Unsupported(_) + | Self::MissingCredential { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall => false, + } } pub fn is_response(&self) -> bool { - matches!( - self, + match self { Self::TooLarge { .. } - | Self::ResponseField { .. } - | Self::EmptyContent - | Self::MissingRedirectLocation - | Self::InvalidRedirect - | Self::OperationStatus(_) - | Self::NumericRange(_) - | Self::PollLocation - | Self::PollOrigin - | Self::PollTimeout - | Self::InvalidResponse(_) - ) + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout => true, + Self::Http(error) | Self::DocumentDownload(error) => error.is_response(), + Self::Provider { .. } + | Self::EmptyFile + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::DotModel + | Self::Unsupported(_) + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::MissingCredential { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall + | Self::Auth(_) + | Self::Params(_) + | Self::Headers(_) => false, + } } } diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/handler.rs b/litellm-rust/crates/llms/src/base_llm/ocr/handler.rs index a9ddcb87086..7bbc751c614 100644 --- a/litellm-rust/crates/llms/src/base_llm/ocr/handler.rs +++ b/litellm-rust/crates/llms/src/base_llm/ocr/handler.rs @@ -9,7 +9,6 @@ use litellm_http::{ media::{MediaFetcher, UrlPolicy}, outbound::{RequestSigner, json, signed}, request::header_pairs, - transport, }; use serde::{Serialize, de::DeserializeOwned}; use serde_json::Value; @@ -134,16 +133,12 @@ pub async fn ocr( .provider_http() .execute(http) .await - .map_err(transport_error)?; + .map_err(litellm_http::Error::from)?; if !response.status().is_success() { - let headers = header_pairs(response.headers()); - return match read_response_bytes(response, request.connection.max_response_bytes).await { - Err(Error::Transport(transport::Error::Http { status, body })) => { - Err(config.get_error_class(body, status, headers)) - } - Err(error) => Err(error), - Ok(_) => unreachable!("non-success response produces an HTTP error"), - }; + let status = response.status().as_u16(); + let headers = response.headers().clone(); + let body = read_error_body(response, request.connection.max_response_bytes).await?; + return Err(config.get_error_class(body, status, headers)); } let context = OcrResponseContext { client, @@ -171,43 +166,41 @@ pub async fn read_response_bytes( mut response: reqwest::Response, limit: usize, ) -> Result { - let status = response.status(); - if status.is_success() - && response - .content_length() - .is_some_and(|length| length > limit as u64) + if !response.status().is_success() { + let status = response.status().as_u16(); + let headers = response.headers().clone(); + let body = read_error_body(response, limit).await?; + return Err(Error::Provider { + status, + body, + headers: Box::new(headers), + }); + } + if response + .content_length() + .is_some_and(|length| length > limit as u64) { return Err(Error::TooLarge { limit }); } let mut bytes = BytesMut::new(); - while let Some(chunk) = response.chunk().await.map_err(transport_error)? { - let remaining = limit.saturating_sub(bytes.len()); - if status.is_success() && chunk.len() > remaining { + while let Some(chunk) = response.chunk().await.map_err(litellm_http::Error::from)? { + if chunk.len() > limit.saturating_sub(bytes.len()) { return Err(Error::TooLarge { limit }); } - bytes.extend_from_slice(&chunk[..chunk.len().min(remaining)]); - if !status.is_success() && bytes.len() == limit { - break; - } - } - if !status.is_success() { - return Err(transport::Error::Http { - status: status.as_u16(), - body: String::from_utf8_lossy(&bytes).into_owned(), - } - .into()); + bytes.extend_from_slice(&chunk); } Ok(bytes.freeze()) } -pub fn transport_error(error: reqwest::Error) -> Error { - if error.is_timeout() { - return Error::Transport(transport::Error::Http { - status: 408, - body: "OCR request timed out".into(), - }); +async fn read_error_body(mut response: reqwest::Response, limit: usize) -> Result { + let mut bytes = BytesMut::new(); + while bytes.len() < limit + && let Some(chunk) = response.chunk().await.map_err(litellm_http::Error::from)? + { + let remaining = limit - bytes.len(); + bytes.extend_from_slice(&chunk[..chunk.len().min(remaining)]); } - transport::Error::from(error).into() + Ok(String::from_utf8_lossy(&bytes).into_owned()) } pub async fn transform_request_body( @@ -231,6 +224,7 @@ pub async fn transform_request_body( if !changed.body.is_object() { return Err(Error::RequestField { path: "guardrail.body".into(), + cause: None, }); } config.validate_request_body(&changed.body)?; @@ -274,9 +268,8 @@ pub async fn guardrail_document( headers: &[(String, String)], hooks: &dyn CallHooks, ) -> Result<(OcrDocument, Vec<(String, String)>), Error> { - let body = serde_json::to_value(&request.document).map_err(|_| Error::RequestField { - path: "document".into(), - })?; + let body = serde_json::to_value(&request.document) + .map_err(|error| Error::request_field("document", error))?; let changed = hooks .before_provider_request(wire_request(url, headers, body)) .await?; @@ -290,6 +283,7 @@ pub fn body_document(body: &Value) -> Result { .and_then(Value::as_object) .ok_or_else(|| Error::RequestField { path: "body.document".into(), + cause: None, })?; let source = document .iter() @@ -309,8 +303,6 @@ mod tests { net::TcpListener, }; - use super::*; - #[rstest::rstest] #[tokio::test] async fn request_timeout_has_an_http_408_status() { @@ -326,10 +318,10 @@ mod tests { .send() .await .unwrap_err(); - assert!(matches!( - transport_error(error), - Error::Transport(transport::Error::Http { status: 408, .. }) - )); + assert_eq!( + Error::from(litellm_http::Error::from(error)).http_status_code(), + Some(408) + ); server.abort(); } @@ -400,10 +392,7 @@ mod tests { .await .unwrap_err(); - let Error::Transport(litellm_http::transport::Error::Http { status, body }) = error else { - panic!("unexpected error: {error}"); - }; - assert_eq!(status, 429); - assert_eq!(body, prefix); + let (status, body, _) = error.upstream_response().expect("the provider answered"); + assert_eq!((status, body), (429, prefix.as_str())); } } diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs b/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs index 73e7b64638d..19dd8e24226 100644 --- a/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs @@ -194,17 +194,13 @@ pub struct DecodedOcrResponse { pub fn decode_request_value(value: Value, prefix: &str) -> Result { serde_path_to_error::deserialize(value.into_deserializer()).map_err(|error| { - Error::RequestField { - path: format!("{prefix}.{}", error.path()), - } + Error::request_field(format!("{prefix}.{}", error.path()), error.into_inner()) }) } pub fn decode_response_value(value: Value, prefix: &str) -> Result { serde_path_to_error::deserialize(value.into_deserializer()).map_err(|error| { - Error::ResponseField { - path: format!("{prefix}.{}", error.path()), - } + Error::response_field(format!("{prefix}.{}", error.path()), error.into_inner()) }) } @@ -213,19 +209,15 @@ pub fn decode_response( native: bool, ) -> Result, Error> { let mut deserializer = serde_json::Deserializer::from_slice(bytes); - let data = serde_path_to_error::deserialize(&mut deserializer).map_err(|error| { - Error::ResponseField { - path: error.path().to_string(), - } - })?; - deserializer.end().map_err(|_| Error::ResponseField { - path: "response".into(), - })?; + let data = serde_path_to_error::deserialize(&mut deserializer) + .map_err(|error| Error::response_field(error.path().to_string(), error.into_inner()))?; + deserializer + .end() + .map_err(|error| Error::response_field("response", error))?; let native = if native { Some( - serde_json::from_slice(bytes).map_err(|_| Error::ResponseField { - path: "response".into(), - })?, + serde_json::from_slice(bytes) + .map_err(|error| Error::response_field("response", error))?, ) } else { None @@ -369,12 +361,12 @@ pub trait BaseOcrConfig: Send + Sync + Sized + 'static { &self, error_message: String, status_code: u16, - headers: Vec<(String, String)>, + headers: reqwest::header::HeaderMap, ) -> Error { Error::Provider { status: status_code, body: error_message, - headers, + headers: Box::new(headers), } } diff --git a/litellm-rust/crates/llms/src/base_llm/sse.rs b/litellm-rust/crates/llms/src/base_llm/sse.rs index 5ff2e3adddd..23f0c1b9389 100644 --- a/litellm-rust/crates/llms/src/base_llm/sse.rs +++ b/litellm-rust/crates/llms/src/base_llm/sse.rs @@ -15,12 +15,15 @@ use bytes::Bytes; use futures_util::{StreamExt, TryStreamExt, future::ready, stream::BoxStream}; use litellm_framer::{ - frames, + SseError, frames, sse::{SseCodec, SseEvent, decode_all}, }; use serde::{Serialize, de::DeserializeOwned}; -use crate::{Error, ErrorDetail, base_llm::base_model_iterator::ByteStream}; +use crate::{ + Error, ErrorDetail, + base_llm::base_model_iterator::{ByteStream, tunnelled_http}, +}; pub type EventStream = BoxStream<'static, Result>; @@ -70,8 +73,13 @@ pub fn invalid(error: serde_json::Error) -> Error { Error::InvalidResponse(ErrorDetail::invalid("stream event", error)) } -fn framing(error: litellm_framer::SseError) -> Error { - Error::InvalidResponse(ErrorDetail::failed("stream framing", error)) +fn framing(error: SseError) -> Error { + match &error { + SseError::Body(body) if let Some(http) = tunnelled_http(body) => Error::Http(http.clone()), + SseError::Body(_) | SseError::InvalidUtf8(_) => { + Error::InvalidResponse(ErrorDetail::failed("stream framing", error)) + } + } } #[cfg(test)] @@ -94,6 +102,45 @@ mod tests { assert_eq!(is_done(&event), done); } + #[rstest] + #[tokio::test] + async fn a_transport_failure_tunnelled_through_the_body_comes_back_as_itself() { + use futures_util::StreamExt; + + let failure = litellm_http::Error::upstream(502, "gateway went away".into()); + let bytes: ByteStream = futures_util::stream::iter([ + Ok(Bytes::from_static(b"data: {}\n\n")), + Err(std::io::Error::other(failure.clone())), + ]) + .boxed(); + + let mut events = event_stream::(bytes, is_done); + assert!(events.next().await.expect("first event").is_ok()); + let error = events.next().await.expect("failure").unwrap_err(); + assert_eq!(error, Error::Http(failure)); + } + + #[rstest] + #[tokio::test] + async fn a_body_failure_that_is_not_transport_is_an_invalid_response() { + use futures_util::StreamExt; + + let bytes: ByteStream = + futures_util::stream::iter([Err(std::io::Error::other("disk full"))]).boxed(); + + let error = event_stream::(bytes, is_done) + .next() + .await + .expect("failure") + .unwrap_err(); + assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("disk full") + ); + } + #[rstest] fn the_done_frame_we_send_decodes_as_a_control_frame() { let events: Vec = decode(DONE_FRAME, is_done).unwrap(); diff --git a/litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs b/litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs index 61dd264336e..0f8703f223b 100644 --- a/litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs +++ b/litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs @@ -14,7 +14,7 @@ use serde_json::{Map, Value, json}; use strum::IntoStaticStr; use crate::{ - Error, + Error, ErrorDetail, base_llm::{ audio_transcription::transformation::{ AudioTranscriptionRequestData, BaseAudioTranscriptionConfig, Headers, @@ -59,20 +59,12 @@ fn audio_fields(audio: Value) -> Result { .and_then(Value::as_str) .filter(|value| !value.is_empty()) .ok_or(Error::MissingField("audio.data"))?; - let format = object.get("format").cloned().ok_or_else(|| { - Error::InvalidRequest( - "audio.format must be wav, mp3, flac, or ogg" - .to_string() - .into(), - ) - })?; - let format: AudioFormat = serde_json::from_value(format).map_err(|_| { - Error::InvalidRequest( - "audio.format must be wav, mp3, flac, or ogg" - .to_string() - .into(), - ) - })?; + let format = object + .get("format") + .cloned() + .ok_or(Error::MissingField("audio.format"))?; + let format: AudioFormat = serde_json::from_value(format) + .map_err(|error| Error::InvalidRequest(ErrorDetail::invalid("audio.format", error)))?; Ok(AudioInput { data: data.to_string(), format, @@ -135,14 +127,13 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig { ) -> Result { let response: ConverseResponse = serde_json::from_value(response_json).map_err(|error| { - Error::InvalidResponse(format!("invalid Converse response: {error}").into()) + Error::InvalidResponse(ErrorDetail::invalid("Converse response", error)) })?; if response.message_content_is_non_text() { - return Err(Error::InvalidResponse( - "Bedrock response contains non-text transcript content" - .to_string() - .into(), - )); + return Err(Error::InvalidResponse(ErrorDetail::Expected { + subject: "Bedrock transcript content", + expected: "text", + })); } Ok(AudioTranscriptionResponseData { text: response.content_text(), diff --git a/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs b/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs index caa345a7024..acaf7b56f8b 100644 --- a/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs +++ b/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs @@ -19,7 +19,7 @@ use litellm_llms_types::{ use serde_json::{Map, Value, json}; use crate::{ - Error, + Error, ErrorDetail, base_llm::{ auth::AuthScheme, chat::{ @@ -154,21 +154,16 @@ impl BaseConfig for AmazonConverseConfig { response: ProviderChatResponseData, ) -> Result { let body = response.body; - if !body.is_object() { - return Err(Error::InvalidResponse( - "converse response is not an object".into(), - )); - } - for field in ["output", "usage"] { - if body.get(field).is_none() { - return Err(Error::InvalidResponse( - format!("invalid Converse response: missing field `{field}`").into(), - )); - } - } + let has_usage = body.get("usage").is_some(); let response: ConverseResponse = serde_json::from_value(body).map_err(|error| { - Error::InvalidResponse(format!("invalid Converse response: {error}").into()) + Error::InvalidResponse(ErrorDetail::invalid("Converse response", error)) })?; + if !has_usage { + return Err(Error::InvalidResponse(ErrorDetail::MissingField { + subject: "Converse response", + field: "usage", + })); + } // The route declines tool requests, so anything other than a text block // is something this path never asked for. Decline; the host falls back. if response.message_content_is_non_text() { @@ -858,34 +853,24 @@ mod tests { assert_eq!(err, Error::Unsupported("non-text response content block")); } - #[test] - fn errors_on_a_response_missing_required_fields() { - assert_eq!( - transform_response(json!("nope")).expect_err("not an object"), - Error::InvalidResponse("converse response is not an object".to_string().into()) - ); - assert_eq!( - transform_response(json!({"usage": {}})).expect_err("no output"), - Error::InvalidResponse("invalid Converse response: missing field `output`".into()) - ); - assert_eq!( - transform_response(json!({"output": {"message": {"content": []}}})) - .expect_err("no usage"), - Error::InvalidResponse("invalid Converse response: missing field `usage`".into()) - ); - } - - #[test] - fn rejects_malformed_text_and_token_counts() { - for response in [ - json!({"output": {"message": {"content": [{"text": 123}]}}, "usage": {"inputTokens": 1, "outputTokens": 1}}), - json!({"output": {"message": {"content": [{"text": "x"}]}}, "usage": {"inputTokens": "1", "outputTokens": 1}}), - ] { - assert!(matches!( - transform_response(response), - Err(Error::InvalidResponse(message)) if message.to_string().starts_with("invalid Converse response:") - )); - } + #[rstest::rstest] + #[case::not_an_object(json!("nope"), "invalid type")] + #[case::no_output(json!({"usage": {"inputTokens": 1, "outputTokens": 1}}), "missing field `output`")] + #[case::no_usage(json!({"output": {"message": {"content": []}}}), "missing usage")] + #[case::non_string_text( + json!({"output": {"message": {"content": [{"text": 123}]}}, "usage": {"inputTokens": 1, "outputTokens": 1}}), + "invalid type" + )] + #[case::non_integer_tokens( + json!({"output": {"message": {"content": [{"text": "x"}]}}, "usage": {"inputTokens": "1", "outputTokens": 1}}), + "invalid type" + )] + fn rejects_a_malformed_response(#[case] response: Value, #[case] fact: &str) { + let error = transform_response(response).expect_err("malformed response"); + assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("Converse response"), "{report}"); + assert!(report.contains(fact), "{report}"); } #[test] diff --git a/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs b/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs index 878c424ae2d..15a6ff38822 100644 --- a/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs +++ b/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs @@ -2,6 +2,7 @@ use base64::Engine; use bytes::Buf; use futures_util::{Stream, StreamExt}; use litellm_framer::{ + EventStreamError, aws_event_stream::{AwsEventStreamCodec, Message}, frames, }; @@ -11,9 +12,8 @@ use serde_json::Value; use crate::{ Error, - anthropic::chat::handler::ModelResponseIterator, base_llm::{ - chat::streaming::{ChatStream, StreamShape}, + base_model_iterator::tunnelled_http, messages::streaming::{ByteStream, EventStream}, }, }; @@ -47,11 +47,21 @@ where B: Buf + Send, E: std::error::Error + Send + Sync + 'static, { - frames(input, AwsEventStreamCodec).map(|message| { - decode_invoke_chunk(message.map_err(|error| { + frames(input, AwsEventStreamCodec).map(|message| decode_invoke_chunk(message.map_err(framing)?)) +} + +fn framing(error: EventStreamError) -> Error { + match &error { + EventStreamError::Body(body) if let Some(http) = tunnelled_http(body) => { + Error::Http(http.clone()) + } + EventStreamError::Body(_) + | EventStreamError::InvalidLength(_) + | EventStreamError::Truncated + | EventStreamError::Malformed(_) => { Error::InvalidResponse(crate::ErrorDetail::failed("stream framing", error)) - })?) - }) + } + } } pub fn decode_invoke_anthropic_chunk(chunk: Value) -> Result { @@ -64,34 +74,6 @@ pub fn invoke_anthropic_event_stream(bytes: ByteStream) -> EventStream { Box::pin(invoke_chunk_stream(bytes).map(|chunk| decode_invoke_anthropic_chunk(chunk?))) } -#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::EnumString)] -enum InvokeProvider { - #[strum(serialize = "anthropic")] - Anthropic, - #[strum(serialize = "deepseek_r1")] - DeepseekR1, - #[strum(serialize = "moonshot")] - Moonshot, - #[strum(disabled)] - Unsupported, -} - -pub fn invoke_chat_stream(invoke_provider: &str, shape: StreamShape) -> Result { - match invoke_provider - .parse() - .unwrap_or(InvokeProvider::Unsupported) - { - InvokeProvider::Anthropic => Ok(ChatStream::new( - invoke_anthropic_event_stream, - ModelResponseIterator::new(shape), - )), - InvokeProvider::DeepseekR1 | InvokeProvider::Moonshot => Err(Error::Unsupported( - "Bedrock invoke streaming for this model family", - )), - InvokeProvider::Unsupported => Err(Error::Unsupported("Bedrock invoke streaming")), - } -} - #[cfg(test)] mod tests { use aws_smithy_eventstream::frame::write_message_to; @@ -112,22 +94,6 @@ mod tests { const TEXT_DELTA: &str = r#"{"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"hello"}}"#; - #[rstest::rstest] - #[case::anthropic("anthropic", InvokeProvider::Anthropic)] - #[case::deepseek_r1("deepseek_r1", InvokeProvider::DeepseekR1)] - #[case::moonshot("moonshot", InvokeProvider::Moonshot)] - #[case::unknown("qwen", InvokeProvider::Unsupported)] - #[case::case_sensitive("Anthropic", InvokeProvider::Unsupported)] - fn invoke_provider_maps_each_model_family_name( - #[case] name: &str, - #[case] expected: InvokeProvider, - ) { - assert_eq!( - name.parse().unwrap_or(InvokeProvider::Unsupported), - expected - ); - } - fn aws_wire(chunk: &str) -> Vec { let payload = serde_json::json!({"bytes": STANDARD.encode(chunk)}); let message = Message::new(Bytes::from(serde_json::to_vec(&payload).unwrap())).add_header( diff --git a/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs b/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs index 72958fd6cb9..79c30df9b19 100644 --- a/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs @@ -1,6 +1,6 @@ use std::sync::LazyLock; -use litellm_auth::{AwsParams, CredentialPlacement, ParamSpec, SecretValue}; +use litellm_auth::{AwsParams, CredentialPlacement, ParamKind, ParamSpec, SecretValue}; use litellm_auth_aws::{AwsCredentialSource, resolve_aws_region}; use litellm_core_utils::settings::resolve_non_empty; use litellm_http::request::{header_value, with_header, without_headers}; @@ -36,6 +36,7 @@ const WORKSPACE_FIELDS: &[&str] = &[ const WORKSPACE_HEADER: &str = "anthropic-workspace-id"; static WORKSPACE: ParamSpec = ParamSpec { setting: "workspace_id", + kind: ParamKind::Setting, wire: WORKSPACE_FIELDS, module_global: None, env: WORKSPACE_ENVS, diff --git a/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs b/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs index dadab413fe4..76ff27b56fc 100644 --- a/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs @@ -195,11 +195,10 @@ impl CohereParseConfig { .and_then(env_lookup) .filter(|key| !key.trim().is_empty()) }) - .ok_or_else(|| { - Error::Auth(litellm_auth::Error::ProviderAuthentication( - "Missing COHERE_API_KEY - set it in the environment or pass api_key".into(), - )) - })?; + .ok_or(Error::Auth(litellm_auth::Error::MissingApiKey { + provider: "Cohere", + environment_variable: COHERE_API_KEY_ENV, + }))?; Ok( std::iter::once(("Authorization".into(), format!("Bearer {key}"))) .chain(connection.extra_headers.clone()) @@ -208,27 +207,28 @@ impl CohereParseConfig { } fn build_ocr_url(&self, api_base: &str) -> Result { - let parsed = reqwest::Url::parse(api_base).map_err(|_| invalid_api_base())?; + let parsed = reqwest::Url::parse(api_base) + .map_err(|error| Error::request_field("api_base", error))?; if !matches!(parsed.scheme(), "http" | "https") { return Err(invalid_api_base()); } ApiUrl::parse(api_base) .and_then(|url| url.complete_path(&["v2", "parse"])) .map(|url| url.into_string()) - .map_err(|_| invalid_api_base()) + .map_err(|error| Error::request_field("api_base", error)) } } pub fn validate_document(document: &OcrDocument) -> Result<(), Error> { let OcrDocument::ImageUrl { image_url, .. } = document else { - return Err(Error::CohereImageOnly); + return Err(image_only()); }; if image_url.is_empty() { - return Err(Error::CohereImageOnly); + return Err(image_only()); } if let Some(inline) = InlineDocument::parse(image_url)? { if !inline.mime_type().type_.eq_ignore_ascii_case("image") { - return Err(Error::CohereImageOnly); + return Err(image_only()); } inline.decode(OCR_INLINE_MAX_BYTES)?; } @@ -260,7 +260,7 @@ pub fn normalize_response( fn image_url(document: OcrDocument) -> Result { validate_document(&document)?; let OcrDocument::ImageUrl { image_url, .. } = document else { - return Err(Error::CohereImageOnly); + return Err(image_only()); }; Ok(image_url) } @@ -329,9 +329,17 @@ fn billed_pages(response: &CohereResponse) -> Option { response.meta.as_ref()?.billed_units.as_ref()?.pages } +fn image_only() -> Error { + Error::UnsupportedDocument { + provider: "Cohere Parse", + guidance: "it only accepts `image_url` documents, not document_url or PDF inputs", + } +} + fn invalid_api_base() -> Error { Error::RequestField { path: "api_base".into(), + cause: None, } } @@ -368,11 +376,14 @@ mod tests { #[test] fn options_reject_invalid_output_format() { let invalid = serde_json::from_value(json!({"output_format":"html"})).unwrap(); - assert!(matches!( - CohereParseConfig.map_ocr_params(&invalid, "parse"), - Err(Error::RequestField { path }) - if path == "optional_params.output_format" - )); + let Err(error) = CohereParseConfig.map_ocr_params(&invalid, "parse") else { + panic!("html is not a Cohere output format"); + }; + assert!(error.is_request()); + let argument = + litellm_error::find::(&error) + .expect("the argument failure is kept"); + assert_eq!(argument.path, "output_format"); } #[test] @@ -432,7 +443,7 @@ mod tests { .unwrap(); assert!(matches!( normalize_response("parse", response).unwrap_err(), - Error::ResponseField { path } + Error::ResponseField { path, .. } if path == "pages[0].markdown.images[0].image_base64" )); } @@ -657,7 +668,10 @@ mod tests { fn request_requires_image(#[case] value: Value) { assert!(matches!( validate_document(&serde_json::from_value(value).unwrap()), - Err(Error::CohereImageOnly) + Err(Error::UnsupportedDocument { + provider: "Cohere Parse", + .. + }) )); } diff --git a/litellm-rust/crates/llms/src/error.rs b/litellm-rust/crates/llms/src/error.rs index 10a4f696406..eddcb27d1d9 100644 --- a/litellm-rust/crates/llms/src/error.rs +++ b/litellm-rust/crates/llms/src/error.rs @@ -1,3 +1,5 @@ +use litellm_error::Source; + #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum Error { #[error("expected {expected}, got {actual}")] @@ -7,31 +9,117 @@ pub enum Error { }, #[error("missing required field: {0}")] MissingField(&'static str), - #[error("invalid request: {0}")] + #[error("invalid request")] InvalidRequest(#[source] ErrorDetail), - #[error("invalid response: {0}")] + #[error("invalid response")] InvalidResponse(#[source] ErrorDetail), #[error("unsupported: {0}")] Unsupported(&'static str), #[error(transparent)] Auth(#[from] litellm_auth::Error), + #[error(transparent)] + Http(#[from] litellm_http::Error), +} + +impl Error { + pub fn is_request(&self) -> bool { + match self { + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::Unsupported(_) => true, + Self::Auth(error) => error.is_request(), + Self::Http(error) => error.is_request(), + Self::InvalidResponse(_) => false, + } + } + + pub fn is_response(&self) -> bool { + match self { + Self::InvalidResponse(_) => true, + Self::Http(error) => error.is_response(), + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::Unsupported(_) + | Self::Auth(_) => false, + } + } + + pub fn is_missing_credential(&self) -> bool { + match self { + Self::Auth(error) => error.is_missing_credential(), + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::InvalidResponse(_) + | Self::Unsupported(_) + | Self::Http(_) => false, + } + } + + /// A transformation runs on both sides of the send, so only the transport and credential + /// resolution, which always runs before it, can tell whether the request reached the provider. + pub fn dispatched(&self) -> bool { + match self { + Self::Http(error) => error.dispatched(), + Self::Auth(_) => false, + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::InvalidResponse(_) + | Self::Unsupported(_) => true, + } + } + + /// Why the transformation rejected the caller's request, rendered as Python's own + /// validation renders it, without this layer's "invalid request" prefix. + pub fn rejected_request(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::InvalidRequest(detail) => Some(detail), + Self::InvalidType { .. } | Self::MissingField(_) => Some(self), + Self::InvalidResponse(_) | Self::Unsupported(_) | Self::Auth(_) | Self::Http(_) => None, + } + } + + pub fn upstream_response(&self) -> Option<(u16, &str)> { + match self { + Self::Http(error) => error.upstream_response(), + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::InvalidResponse(_) + | Self::Unsupported(_) + | Self::Auth(_) => None, + } + } } #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum ErrorDetail { - #[error("{0}")] - Message(String), - #[error("invalid {subject}: {source}")] + #[error("invalid {subject}")] Invalid { subject: &'static str, #[source] - source: ErrorSource, + cause: Source, }, - #[error("{operation} failed: {source}")] + #[error("{operation} failed")] Failed { operation: &'static str, #[source] - source: ErrorSource, + cause: Source, + }, + #[error("{route} requires at least one message")] + NoMessages { route: &'static str }, + #[error("{subject} must be {expected}")] + Expected { + subject: &'static str, + expected: &'static str, + }, + #[error("{subject} is missing {field}")] + MissingField { + subject: &'static str, + field: &'static str, }, #[error("{field} must be {expected}, got {actual}")] InvalidValue { @@ -60,12 +148,18 @@ pub enum ErrorDetail { value: String, hint: String, }, - #[error("invalid {subject} on line {line}: {source}")] + #[error( + "Missing API Base - A call is being made to LLM Provider but no api base is set either in the environment variables ({{LLM_PROVIDER}}_API_KEY) or via params" + )] + MissingApiBase, + #[error("the host call raised an exception")] + HostCall, + #[error("invalid {subject} on line {line}")] InvalidLine { subject: &'static str, line: usize, #[source] - source: ErrorSource, + cause: Source, }, #[error("{operation} failed: {detail}")] RemoteFailure { @@ -77,64 +171,33 @@ pub enum ErrorDetail { impl ErrorDetail { pub fn invalid( subject: &'static str, - source: impl std::error::Error + Send + Sync + 'static, + cause: impl std::error::Error + Send + Sync + 'static, ) -> Self { Self::Invalid { subject, - source: ErrorSource::new(source), + cause: Source::new(cause), } } pub fn failed( operation: &'static str, - source: impl std::error::Error + Send + Sync + 'static, + cause: impl std::error::Error + Send + Sync + 'static, ) -> Self { Self::Failed { operation, - source: ErrorSource::new(source), + cause: Source::new(cause), + } + } + + pub fn invalid_line( + subject: &'static str, + line: usize, + cause: impl std::error::Error + Send + Sync + 'static, + ) -> Self { + Self::InvalidLine { + subject, + line, + cause: Source::new(cause), } } } - -impl From for ErrorDetail { - fn from(message: String) -> Self { - Self::Message(message) - } -} - -impl From<&str> for ErrorDetail { - fn from(message: &str) -> Self { - Self::Message(message.into()) - } -} - -#[derive(Clone, Debug)] -pub struct ErrorSource(std::sync::Arc); - -impl std::ops::Deref for ErrorSource { - type Target = dyn std::error::Error + Send + Sync; - - fn deref(&self) -> &Self::Target { - self.0.as_ref() - } -} - -impl std::fmt::Display for ErrorSource { - fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - std::fmt::Display::fmt(&self.0, formatter) - } -} - -impl ErrorSource { - pub fn new(error: impl std::error::Error + Send + Sync + 'static) -> Self { - Self(std::sync::Arc::new(error)) - } -} - -impl PartialEq for ErrorSource { - fn eq(&self, other: &Self) -> bool { - std::sync::Arc::ptr_eq(&self.0, &other.0) - } -} - -impl Eq for ErrorSource {} diff --git a/litellm-rust/crates/llms/src/lib.rs b/litellm-rust/crates/llms/src/lib.rs index eb232404d93..68e6e4010c2 100644 --- a/litellm-rust/crates/llms/src/lib.rs +++ b/litellm-rust/crates/llms/src/lib.rs @@ -17,4 +17,4 @@ pub mod reducto; pub mod tencent; pub mod vertex_ai; -pub use error::{Error, ErrorDetail, ErrorSource}; +pub use error::{Error, ErrorDetail}; diff --git a/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs b/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs index 29d4d5c1610..fa4e2254e86 100644 --- a/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs @@ -171,9 +171,7 @@ impl MistralOcrConfig { ApiUrl::parse(base) .and_then(|url| url.complete_path(&["v1", "ocr"])) .map(|url| url.into_string()) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } } @@ -186,6 +184,7 @@ pub fn normalize_response( Some(None) => { return Err(Error::ResponseField { path: "model".into(), + cause: None, }); } None => model.to_string(), @@ -231,7 +230,7 @@ mod tests { let response = serde_json::from_value(json!({"model":null})).unwrap(); assert!(matches!( normalize_response("fallback", response).unwrap_err(), - Error::ResponseField { path } if path == "model" + Error::ResponseField { path, .. } if path == "model" )); } @@ -263,7 +262,7 @@ mod tests { .unwrap_err(); assert!(matches!( error, - Error::ResponseField { path: actual } if actual == path + Error::ResponseField { path: actual, .. } if actual == path )); } diff --git a/litellm-rust/crates/llms/src/openai/responses/transformation.rs b/litellm-rust/crates/llms/src/openai/responses/transformation.rs index 99cc3559514..cf74871894b 100644 --- a/litellm-rust/crates/llms/src/openai/responses/transformation.rs +++ b/litellm-rust/crates/llms/src/openai/responses/transformation.rs @@ -6,7 +6,7 @@ use serde_json::{Map, Value}; use litellm_auth::{CredentialPlacement, SecretValue}; use crate::{ - Error, + Error, ErrorDetail, base_llm::{ auth::{AuthScheme, ValidatedEnvironment}, responses::transformation::{ @@ -180,15 +180,19 @@ pub fn responses_request( params: Map, ) -> Result { if !input.is_string() && !input.is_array() { - return Err(Error::InvalidRequest( - "responses input must be a string or an array".into(), - )); + return Err(Error::InvalidRequest(ErrorDetail::Expected { + subject: "responses input", + expected: "a string or an array", + })); } if params .get("stream") .is_some_and(|stream| !stream.is_boolean()) { - return Err(Error::InvalidRequest("stream must be a boolean".into())); + return Err(Error::InvalidRequest(ErrorDetail::Expected { + subject: "stream", + expected: "a boolean", + })); } Ok(Value::Object( params @@ -202,7 +206,9 @@ pub fn responses_request( } pub fn responses_response(body: Value) -> Result { - serde_json::from_value(body).map_err(|error| Error::InvalidResponse(error.to_string().into())) + serde_json::from_value(body).map_err(|error| { + Error::InvalidResponse(ErrorDetail::invalid("Responses API response", error)) + }) } #[cfg(test)] diff --git a/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs b/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs index 0a89adf90f7..9f208db2471 100644 --- a/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs @@ -123,9 +123,12 @@ impl BaseConfig for OpenAILikeChatConfig { ) -> Result { let mut body = response.body; sanitize_usage(&mut body); - let body = body - .as_object() - .ok_or_else(|| Error::InvalidResponse("chat response is not an object".into()))?; + let body = + body.as_object() + .ok_or(Error::InvalidResponse(crate::ErrorDetail::Expected { + subject: "chat response", + expected: "an object", + }))?; let choices = body .get("choices") @@ -414,9 +417,13 @@ mod tests { #[rstest] fn a_missing_api_base_is_an_error() { assert!(matches!( - OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG - .get_complete_url(None, "my-model", ¶ms(json!({})), &no_env), - Err(Error::InvalidRequest(message)) if message.to_string().starts_with("Missing API Base") + OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG.get_complete_url( + None, + "my-model", + ¶ms(json!({})), + &no_env + ), + Err(Error::InvalidRequest(crate::ErrorDetail::MissingApiBase)) )); } diff --git a/litellm-rust/crates/llms/src/openai_like/common_utils.rs b/litellm-rust/crates/llms/src/openai_like/common_utils.rs index b0f5adafe00..d798f851d79 100644 --- a/litellm-rust/crates/llms/src/openai_like/common_utils.rs +++ b/litellm-rust/crates/llms/src/openai_like/common_utils.rs @@ -1,7 +1,7 @@ //! Shared OpenAI-like credential and endpoint resolution, mirroring //! `litellm/llms/openai_like/common_utils.py`. -use crate::Error; +use crate::{Error, ErrorDetail}; /// `OpenAILikeChatConfig._get_openai_compatible_provider_info`: the deployment's /// `api_base` wins over `OPENAI_LIKE_API_BASE`, and the deployment key over @@ -31,12 +31,7 @@ pub fn complete_openai_like_url( env_lookup: &dyn Fn(&str) -> Option, ) -> Result { let (api_base, _) = openai_compatible_provider_info(api_base, None, env_lookup); - let api_base = api_base.ok_or_else(|| { - Error::InvalidRequest( - "Missing API Base - A call is being made to LLM Provider but no api base is set either in the environment variables ({LLM_PROVIDER}_API_KEY) or via params" - .into(), - ) - })?; + let api_base = api_base.ok_or(Error::InvalidRequest(ErrorDetail::MissingApiBase))?; if custom_endpoint { return Ok(api_base); } diff --git a/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs b/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs index fc73ce7c014..bf0ddc1c7a8 100644 --- a/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs @@ -299,9 +299,16 @@ async fn prepare_upload_request Error { + Error::UnsupportedDocument { + provider: "Reducto", + guidance: "pass a reducto:// id or a data URI, or upload the file first for plain HTTP URLs", + } +} + fn uploaded_file_id(document: OcrDocument) -> Result { if !document.source().starts_with(REDUCTO_ID_PREFIX) { - return Err(Error::ReductoSource); + return Err(unsupported_source()); } if document.source()[REDUCTO_ID_PREFIX.len()..] .trim() @@ -309,6 +316,7 @@ fn uploaded_file_id(document: OcrDocument) -> Result { { return Err(Error::RequestField { path: "document file id".into(), + cause: None, }); } Ok(ReductoFileId(document.source().into())) @@ -392,6 +400,7 @@ fn build_pages_from_reducto(chunks: Vec) -> Result, E Some(Value::String(content)) => Ok(Some(content.as_str())), Some(_) => Err(Error::ResponseField { path: "result.chunks.blocks.content".into(), + cause: None, }), }) .collect::, _>>()?; @@ -436,9 +445,7 @@ fn complete_endpoint_url(api_base: Option<&str>, path: &str) -> Result( response, false, @@ -546,6 +555,7 @@ async fn upload_bytes_async( let Some(file_id) = file_id else { return Err(Error::ResponseField { path: "file_id".into(), + cause: None, }); }; Ok(ReductoFileId(file_id.to_string())) @@ -553,6 +563,8 @@ async fn upload_bytes_async( #[cfg(test)] mod tests { + use rstest::rstest; + use super::*; #[tokio::test] @@ -625,15 +637,16 @@ mod tests { ); } + #[rstest] + #[case::fractional_pages(json!({"num_pages":1.5}))] + #[case::non_numeric_pages(json!({"num_pages":[]}))] + #[case::non_numeric_credits(json!({"credits":{}}))] + fn invalid_usage_is_rejected(#[case] usage: Value) { + assert!(serde_json::from_value::(json!({"usage":usage})).is_err()); + } + #[test] - fn usage_uses_shared_validation_while_block_page_numbers_are_best_effort() { - for usage in [ - json!({"num_pages":1.5}), - json!({"num_pages":[]}), - json!({"credits":{}}), - ] { - assert!(serde_json::from_value::(json!({"usage":usage})).is_err()); - } + fn block_page_numbers_are_best_effort() { let response = serde_json::from_value(json!({"result":{"chunks":[{"blocks":[ {"content":"ignored", "bbox":{"page":"invalid"}}, {"content":"kept", "bbox":{"page":2.5}, "extra":null} diff --git a/litellm-rust/crates/llms/src/vertex_ai/ocr/common_utils.rs b/litellm-rust/crates/llms/src/vertex_ai/ocr/common_utils.rs index 810462c57f8..d760da9073e 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/ocr/common_utils.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/ocr/common_utils.rs @@ -16,7 +16,9 @@ pub(super) fn vertex_config(request: &PreparedOcrRequest) -> Result Result<(), Error> { if connection.api_base.is_some() && connection.api_base_source == InputSource::Request { return Err(litellm_auth::Error::InvalidConfiguration( - "credentials cannot be sent to a request-controlled Vertex AI endpoint".into(), + litellm_auth::ErrorDetail::RequestControlledEndpoint { + provider: "Vertex AI", + }, ) .into()); } diff --git a/litellm-rust/crates/llms/src/vertex_ai/ocr/deepseek_transformation.rs b/litellm-rust/crates/llms/src/vertex_ai/ocr/deepseek_transformation.rs index 67f9122ed67..5d680588612 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/ocr/deepseek_transformation.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/ocr/deepseek_transformation.rs @@ -235,8 +235,9 @@ pub fn normalize_response( &mut output, PythonJsonFormatter, )) - .map_err(|_| response_field("content"))?; - String::from_utf8(output).map_err(|_| response_field("content"))? + .map_err(|error| Error::response_field(content_path("content"), error))?; + String::from_utf8(output) + .map_err(|error| Error::response_field(content_path("content"), error))? }; (data, fallback) } @@ -261,7 +262,12 @@ pub fn normalize_response( }) }) .collect::, Error>>()?, - Some(_) => return Err(response_field("pages")), + Some(_) => { + return Err(Error::ResponseField { + path: content_path("pages"), + cause: None, + }); + } None => Vec::new(), }; let usage = ocr_data @@ -273,7 +279,12 @@ pub fn normalize_response( .transpose()?; let model = match ocr_data.get("model") { Some(Value::String(model)) => model.clone(), - Some(_) => return Err(response_field("model")), + Some(_) => { + return Err(Error::ResponseField { + path: content_path("model"), + cause: None, + }); + } None => model.to_string(), }; Ok(LiteLLMOcrResponse { @@ -369,10 +380,8 @@ impl serde_json::ser::Formatter for PythonJsonFormatter { } } -fn response_field(field: &str) -> Error { - Error::ResponseField { - path: format!("choices[0].message.content.{field}"), - } +fn content_path(field: &str) -> String { + format!("choices[0].message.content.{field}") } pub fn provider_model(model: &str) -> Result { @@ -380,6 +389,7 @@ pub fn provider_model(model: &str) -> Result { if local_model.is_empty() { return Err(Error::RequestField { path: "model".into(), + cause: None, }); } Ok(format!("{MODEL_PREFIX}{local_model}")) @@ -411,9 +421,7 @@ impl VertexAIDeepSeekOCRConfig { ]) }) .map(|url| url.into_string()) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } } @@ -625,20 +633,18 @@ mod tests { assert_eq!(result["future"], "kept"); } - #[test] - fn response_transform_rejects_missing_empty_and_malformed_content() { - for value in [ - json!({"choices":[]}), - json!({"choices":[{"message":{"content":{}}}]}), - json!({"choices":[{"message":{"content":""}}]}), - json!({"choices":[{"message":{"content":"{\"pages\":[{\"markdown\":42}]}"}}]}), - json!({"choices":[{"message":{"content":{"pages":[{"markdown":42}]}}}]}), - ] { - let result = serde_json::from_value::(value) - .map_err(|_| ()) - .and_then(|response| normalize_response("model", response).map_err(|_| ())); - assert!(result.is_err()); - } + #[rstest::rstest] + #[case::no_choices(json!({"choices":[]}))] + #[case::empty_object(json!({"choices":[{"message":{"content":{}}}]}))] + #[case::empty_string(json!({"choices":[{"message":{"content":""}}]}))] + #[case::malformed_string(json!({"choices":[{"message":{"content":"{\"pages\":[{\"markdown\":42}]}"}}]}))] + #[case::malformed_object(json!({"choices":[{"message":{"content":{"pages":[{"markdown":42}]}}}]}))] + fn response_transform_rejects_missing_empty_and_malformed_content(#[case] value: Value) { + let rejected = match serde_json::from_value::(value) { + Ok(response) => normalize_response("model", response).is_err(), + Err(_) => true, + }; + assert!(rejected); } #[test] diff --git a/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs b/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs index ab28ee8bff4..77881cd2e78 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs @@ -174,9 +174,7 @@ impl VertexAiOcrConfig { ]) }) .map(|url| url.into_string()) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } } @@ -198,6 +196,7 @@ fn validate_location(location: &str) -> Result<(), Error> { } Err(Error::RequestField { path: "vertex_location".into(), + cause: None, }) } diff --git a/litellm-rust/crates/model-catalog/Cargo.toml b/litellm-rust/crates/model-catalog/Cargo.toml index 570e68ca4fd..2c2866d7090 100644 --- a/litellm-rust/crates/model-catalog/Cargo.toml +++ b/litellm-rust/crates/model-catalog/Cargo.toml @@ -20,4 +20,5 @@ time.workspace = true [dev-dependencies] jsonschema = { version = "0.55.1", default-features = false } +litellm-error.workspace = true rstest.workspace = true diff --git a/litellm-rust/crates/model-catalog/src/catalog.rs b/litellm-rust/crates/model-catalog/src/catalog.rs index 0b113436a5b..37412ef1e14 100644 --- a/litellm-rust/crates/model-catalog/src/catalog.rs +++ b/litellm-rust/crates/model-catalog/src/catalog.rs @@ -20,11 +20,15 @@ pub struct IntegrityLimits { pub min_reference_ratio: f64, } -#[derive(Clone, Debug, PartialEq, Eq)] +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum AliasIssue { + #[error("model {model:?} has aliases that are not an array")] InvalidList { model: String }, + #[error("model {model:?} has an alias that is not a string")] InvalidName { model: String }, + #[error("alias {alias:?} of model {model:?} is already a model name")] CanonicalCollision { model: String, alias: String }, + #[error("alias {alias:?} of model {model:?} is already an alias of another model")] AliasCollision { model: String, alias: String }, } diff --git a/litellm-rust/crates/model-catalog/src/error.rs b/litellm-rust/crates/model-catalog/src/error.rs index edb6ba1eb11..00902858be4 100644 --- a/litellm-rust/crates/model-catalog/src/error.rs +++ b/litellm-rust/crates/model-catalog/src/error.rs @@ -2,26 +2,20 @@ use thiserror::Error; #[derive(Debug, Error)] pub enum Error { - /// The body is not valid JSON, or a model entry fails typed deserialization. - #[error("invalid JSON: {0}")] + #[error("invalid catalog JSON")] Json(#[from] serde_json::Error), - /// The catalog has no entries at all. #[error("catalog is empty")] Empty, - /// A non-reserved top level value is not a JSON object. #[error("model {model:?} must be an object")] EntryNotObject { model: String }, - /// Canonical entry count is under the configured minimum. #[error("catalog has {actual} models, below minimum {minimum}")] BelowMinimum { actual: usize, minimum: usize }, - /// Canonical entry count is under the configured reference ratio. #[error("catalog has {actual} models, below {ratio} of reference count {reference}")] Shrunk { actual: usize, reference: usize, ratio: f64, }, - /// The configured minimum reference ratio is not finite or outside `[0, 1]`. #[error("minimum reference ratio must be finite and between zero and one")] InvalidRatio, } diff --git a/litellm-rust/crates/model-catalog/src/validation.rs b/litellm-rust/crates/model-catalog/src/validation.rs index 73f08a865c1..6dd4174148c 100644 --- a/litellm-rust/crates/model-catalog/src/validation.rs +++ b/litellm-rust/crates/model-catalog/src/validation.rs @@ -5,24 +5,57 @@ use thiserror::Error; use crate::{AliasIssue, Catalog, ModelInfo, UtcHours, Weekday}; -/// A registry entry violates the checked-in catalog contract. #[derive(Debug, Error)] pub enum RegistryValidationError { - #[error("{reason}")] - Entry { model: String, reason: String }, - #[error("alias issue: {0:?}")] + #[error("invalid registry entry {model:?}")] + Entry { + model: String, + #[source] + reason: EntryViolation, + }, + #[error(transparent)] Alias(AliasIssue), } -/// Validate one registry entry without restricting the tolerant catalog reader. +#[derive(Debug, Error)] +pub enum EntryViolation { + #[error("entry must be an object")] + NotObject, + #[error("aliases must be an array")] + AliasesNotArray, + #[error("aliases must contain strings")] + AliasNotString, + #[error("entry does not match the model schema")] + Schema(#[source] serde_json::Error), + #[error("litellm_provider is required")] + MissingProvider, + #[error("invalid deprecation_date")] + DeprecationDate(#[source] time::error::Parse), + #[error("off_peak_pricing needs hours or windows")] + OffPeakWithoutHours, + #[error("off_peak_pricing.hours_utc is invalid")] + InvalidHours, + #[error("off_peak_pricing.windows is empty")] + EmptyWindows, + #[error("off_peak_pricing.weekdays is invalid")] + InvalidWeekdays, + #[error("typed entry could not be serialized")] + Serialize(#[source] serde_json::Error), + #[error("entry has an unknown field, null, or changed value: {}", fields.join(", "))] + ChangedFields { fields: Vec }, + #[error("{field} must be nonnegative")] + Negative { field: String }, + #[error("{field} must be at least one")] + UpliftBelowOne { field: String }, +} + pub fn validate_model_entry(model: &str, value: &Value) -> Result<(), RegistryValidationError> { - validate_entry_inner(model, value).map_err(|reason| RegistryValidationError::Entry { + validate_entry_inner(value).map_err(|reason| RegistryValidationError::Entry { model: model.to_owned(), reason, }) } -/// Check every model and alias in a parsed catalog against registry rules. pub fn validate_registry(catalog: &Catalog) -> Result<(), RegistryValidationError> { if let Some(issue) = catalog.alias_issues().first() { return Err(RegistryValidationError::Alias(issue.clone())); @@ -53,70 +86,61 @@ fn keys(value: &Map) -> BTreeSet { value.keys().cloned().collect() } -fn symmetric_difference(left: &BTreeSet, right: &BTreeSet) -> BTreeSet { +fn symmetric_difference(left: &BTreeSet, right: &BTreeSet) -> Vec { left.symmetric_difference(right).cloned().collect() } -fn validate_entry_inner(model_name: &str, value: &Value) -> Result<(), String> { - let object = value - .as_object() - .ok_or_else(|| format!("{model_name} must be an object"))?; +fn validate_entry_inner(value: &Value) -> Result<(), EntryViolation> { + let object = value.as_object().ok_or(EntryViolation::NotObject)?; if let Some(aliases) = object.get("aliases") { - let names = aliases - .as_array() - .ok_or_else(|| format!("{model_name}.aliases must be an array"))?; + let names = aliases.as_array().ok_or(EntryViolation::AliasesNotArray)?; if names.iter().any(|name| !name.is_string()) { - return Err(format!("{model_name}.aliases must contain strings")); + return Err(EntryViolation::AliasNotString); } } - let info: ModelInfo = - serde_json::from_value(value.clone()).map_err(|error| format!("{model_name}: {error}"))?; + let info: ModelInfo = serde_json::from_value(value.clone()).map_err(EntryViolation::Schema)?; if info.litellm_provider.is_none() { - return Err(format!("{model_name}.litellm_provider is required")); + return Err(EntryViolation::MissingProvider); } - validate_dates_and_windows(model_name, &info)?; - let serialized = serde_json::to_value(info).map_err(|error| error.to_string())?; + validate_dates_and_windows(&info)?; + let serialized = serde_json::to_value(info).map_err(EntryViolation::Serialize)?; let mut expected = object.clone(); expected.remove("aliases"); if !json_eq(&Value::Object(expected.clone()), &serialized) { let actual = serialized .as_object() .expect("ModelInfo serializes as an object"); - return Err(format!( - "{model_name} has an unknown field, null, or changed value: {:?}", - symmetric_difference(&keys(&expected), &keys(actual)) - )); + return Err(EntryViolation::ChangedFields { + fields: symmetric_difference(&keys(&expected), &keys(actual)), + }); } - check_prices(model_name, value) + check_prices("", value) } -fn validate_dates_and_windows(model_name: &str, info: &ModelInfo) -> Result<(), String> { +fn validate_dates_and_windows(info: &ModelInfo) -> Result<(), EntryViolation> { if let Some(date) = &info.deprecation_date { let format = time::format_description::parse_borrowed::<2>("[year]-[month]-[day]").unwrap(); - time::Date::parse(date, &format) - .map_err(|error| format!("{model_name}.deprecation_date: {error}"))?; + time::Date::parse(date, &format).map_err(EntryViolation::DeprecationDate)?; } let Some(pricing) = &info.off_peak_pricing else { return Ok(()); }; if pricing.hours_utc.is_none() && pricing.windows.is_none() { - return Err(format!( - "{model_name}.off_peak_pricing needs hours or windows" - )); + return Err(EntryViolation::OffPeakWithoutHours); } if let Some(hours) = &pricing.hours_utc { validate_hours(hours)?; } if let Some(windows) = &pricing.windows { if windows.is_empty() { - return Err(format!("{model_name}.off_peak_pricing.windows is empty")); + return Err(EntryViolation::EmptyWindows); } windows.iter().try_for_each(|window| { validate_hours(&window.hours_utc)?; if let Some(days) = &window.weekdays && (days.is_empty() || days.iter().any(|day| !valid_weekday(day))) { - return Err(format!("{model_name}.off_peak_pricing.weekdays is invalid")); + return Err(EntryViolation::InvalidWeekdays); } Ok(()) })?; @@ -124,13 +148,13 @@ fn validate_dates_and_windows(model_name: &str, info: &ModelInfo) -> Result<(), Ok(()) } -fn validate_hours(hours: &UtcHours) -> Result<(), String> { +fn validate_hours(hours: &UtcHours) -> Result<(), EntryViolation> { let values = match hours { UtcHours::Single(value) => std::slice::from_ref(value), UtcHours::Multiple(values) => values.as_slice(), }; if values.is_empty() || values.iter().any(|value| !valid_utc_window(value)) { - return Err("off_peak_pricing.hours_utc is invalid".into()); + return Err(EntryViolation::InvalidHours); } Ok(()) } @@ -176,7 +200,7 @@ fn valid_weekday(day: &Weekday) -> bool { } } -fn check_prices(path: &str, value: &Value) -> Result<(), String> { +fn check_prices(path: &str, value: &Value) -> Result<(), EntryViolation> { let Some(object) = value.as_object() else { return Ok(()); }; @@ -188,13 +212,17 @@ fn check_prices(path: &str, value: &Value) -> Result<(), String> { && let Some(number) = field.as_f64() && number < 0.0 { - return Err(format!("{field_path} must be nonnegative")); + return Err(EntryViolation::Negative { + field: relative(&field_path), + }); } if key.contains("uplift_multiplier") && let Some(number) = field.as_f64() && number < 1.0 { - return Err(format!("{field_path} must be at least one")); + return Err(EntryViolation::UpliftBelowOne { + field: relative(&field_path), + }); } if matches!(key.as_str(), "range" | "max_results_range") && field.as_array().is_some_and(|values| { @@ -203,7 +231,9 @@ fn check_prices(path: &str, value: &Value) -> Result<(), String> { .any(|value| value.as_f64().is_some_and(|n| n < 0.0)) }) { - return Err(format!("{field_path} must be nonnegative")); + return Err(EntryViolation::Negative { + field: relative(&field_path), + }); } if matches!(key.as_str(), "metadata" | "provider_specific_entry") { return Ok(()); @@ -216,3 +246,7 @@ fn check_prices(path: &str, value: &Value) -> Result<(), String> { } }) } + +fn relative(field_path: &str) -> String { + field_path.trim_start_matches('.').to_owned() +} diff --git a/litellm-rust/crates/model-catalog/tests/registry_validation.rs b/litellm-rust/crates/model-catalog/tests/registry_validation.rs index 67589010aba..3b65ef13d78 100644 --- a/litellm-rust/crates/model-catalog/tests/registry_validation.rs +++ b/litellm-rust/crates/model-catalog/tests/registry_validation.rs @@ -52,9 +52,9 @@ fn fallback_generalizations_are_typed(repo_root: PathBuf) { #[case::null_aliases(serde_json::json!({"litellm_provider": "test", "aliases": null}), "aliases must be an array")] #[ignore] fn registry_validation_rejects_malformed_entries(#[case] entry: Value, #[case] expected: &str) { + let error = validate_model_entry("test", &entry).unwrap_err(); assert!( - validate_model_entry("test", &entry) - .unwrap_err() + litellm_error::Report::new(&error) .to_string() .contains(expected) ); diff --git a/litellm-rust/crates/python-bridge/AGENTS.md b/litellm-rust/crates/python-bridge/AGENTS.md index c435ba37e78..d90fe073b20 100644 --- a/litellm-rust/crates/python-bridge/AGENTS.md +++ b/litellm-rust/crates/python-bridge/AGENTS.md @@ -111,8 +111,11 @@ GIL handling to `litellm-host-python`. - Avoid copying large payloads more than needed. The current JSON round-trip is acceptable for the first scaffold, but future performance work should evaluate direct PyO3 conversion before expanding Rust coverage to image-heavy paths. -- Do not expose raw Rust errors that include document contents or upstream - bodies. +- Render a Rust error for Python only in its mapping function, with + `litellm_error::Report`, never `Debug` or a document's contents. An upstream + body reaches Python intact as the `RustUpstreamError` args for provider error + mapping. The 256-character cap applies only to HTTP error diagnostics, not + response bodies. Do not log those args. ## Tests diff --git a/litellm-rust/crates/python-bridge/Cargo.toml b/litellm-rust/crates/python-bridge/Cargo.toml index 8afde8362f6..9d69cd8c14b 100644 --- a/litellm-rust/crates/python-bridge/Cargo.toml +++ b/litellm-rust/crates/python-bridge/Cargo.toml @@ -27,6 +27,7 @@ litellm-host.workspace = true bytes.workspace = true futures-util.workspace = true litellm-cache.workspace = true +litellm-error.workspace = true litellm-cache-memory.workspace = true litellm-cache-redis.workspace = true litellm-cache-response.workspace = true @@ -70,6 +71,7 @@ criterion.workspace = true futures-util.workspace = true rstest.workspace = true sqlx = { workspace = true, features = ["migrate"] } +tempfile.workspace = true tokio-tungstenite.workspace = true wiremock.workspace = true aws-sdk-secretsmanager = "1.117.0" diff --git a/litellm-rust/crates/python-bridge/src/cache/mod.rs b/litellm-rust/crates/python-bridge/src/cache/mod.rs index e190b12e665..87dba3efcb1 100644 --- a/litellm-rust/crates/python-bridge/src/cache/mod.rs +++ b/litellm-rust/crates/python-bridge/src/cache/mod.rs @@ -12,10 +12,93 @@ use pyo3::{ prelude::*, }; -fn cache_error(error: Error) -> PyErr { - match error { - Error::InvalidEntry => PyValueError::new_err(error.to_string()), - Error::UnsupportedOperation => PyNotImplementedError::new_err(error.to_string()), - _ => PyRuntimeError::new_err(error.to_string()), +/// A cache call the bridge rejects itself, before or after the cache ran. +#[derive(Debug, thiserror::Error)] +pub(crate) enum HandleError { + #[error("recreate the v2 cache after fork")] + Forked, + #[error("cache limits must be positive")] + Limits, + #[error("namespace and a positive cache limit are required")] + RedisLimits, + #[error("cache durations must be finite and positive")] + Duration(#[source] Option), + #[error("unsupported v2 cache control: {0}")] + Control(String), + #[error("native inference needs a v2 cache, the catalog keeps Python caches on Python")] + NotNative, + #[error("cache operation without configured cache")] + Unconfigured, + #[error("cache arguments unavailable")] + MissingArguments, + #[error("cache reply without pending operation")] + NothingPending, +} + +impl From for PyErr { + fn from(error: HandleError) -> Self { + let message = litellm_error::Report::new(&error).to_string(); + match error { + HandleError::Limits + | HandleError::RedisLimits + | HandleError::Duration(_) + | HandleError::Control(_) => PyValueError::new_err(message), + HandleError::Forked + | HandleError::NotNative + | HandleError::Unconfigured + | HandleError::MissingArguments + | HandleError::NothingPending => PyRuntimeError::new_err(message), + } + } +} + +fn cache_error(error: Error) -> PyErr { + let message = litellm_error::Report::new(&error).to_string(); + match error { + Error::InvalidEntry(_) => PyValueError::new_err(message), + Error::UnsupportedOperation => PyNotImplementedError::new_err(message), + Error::Unavailable(_) | Error::UnscopedFlush | Error::MissingPrompt => { + PyRuntimeError::new_err(message) + } + } +} + +#[cfg(test)] +mod tests { + use pyo3::{ + exceptions::{PyRuntimeError, PyValueError}, + prelude::*, + }; + use rstest::rstest; + + use super::{Error, cache_error}; + + fn refused() -> std::io::Error { + std::io::Error::other("connection refused") + } + + #[rstest] + #[case::unavailable( + Error::unavailable(refused()), + false, + "cache is unavailable: connection refused" + )] + #[case::invalid_entry( + Error::invalid_entry(refused()), + true, + "invalid cache entry: connection refused" + )] + fn python_sees_the_class_and_the_whole_chain( + #[case] error: Error, + #[case] caller_fault: bool, + #[case] message: &str, + ) { + Python::initialize(); + Python::attach(|py| { + let failure = cache_error(error); + assert_eq!(failure.is_instance_of::(py), caller_fault); + assert_eq!(failure.is_instance_of::(py), !caller_fault); + assert_eq!(failure.value(py).to_string(), message); + }); } } diff --git a/litellm-rust/crates/python-bridge/src/cache/native/v2.rs b/litellm-rust/crates/python-bridge/src/cache/native/v2.rs index eb09b8131c3..c54cb641843 100644 --- a/litellm-rust/crates/python-bridge/src/cache/native/v2.rs +++ b/litellm-rust/crates/python-bridge/src/cache/native/v2.rs @@ -1,4 +1,4 @@ -use crate::cache::cache_error; +use crate::cache::{HandleError, cache_error}; use std::{ sync::Arc, time::{Duration, SystemTime, UNIX_EPOCH}, @@ -14,7 +14,7 @@ use litellm_cache_response::{ CacheEntry, CacheKeyInput, ExactResponseCache, ResponseCache, ResponseCacheCodec, ResponseCacheConfig, ResponseCacheRequest, ResponseCacheService, }; -use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; +use pyo3::{prelude::*, types::PyDict}; #[pyclass( frozen, @@ -37,9 +37,7 @@ enum Storage { impl NativeCacheHandle { fn check_process(&self) -> PyResult<()> { if self.pid != std::process::id() { - return Err(pyo3::exceptions::PyRuntimeError::new_err( - "recreate the v2 cache after fork", - )); + return Err(HandleError::Forked.into()); } Ok(()) } @@ -61,7 +59,7 @@ impl NativeCacheHandle { fn memory(ttl: f64, capacity: usize, max_entry_bytes: usize) -> PyResult { let ttl = duration(ttl)?; if capacity == 0 || max_entry_bytes == 0 { - return Err(PyValueError::new_err("cache limits must be positive")); + return Err(HandleError::Limits.into()); } let storage = Arc::new(InMemoryCache::new(Some(capacity), Some(ttl))); let backend = Arc::new(ResponseCache::new(storage.clone()).with_config( @@ -89,9 +87,7 @@ impl NativeCacheHandle { ) -> PyResult { let ttl = duration(ttl)?; if namespace.is_empty() || max_entry_bytes == 0 { - return Err(PyValueError::new_err( - "namespace and a positive cache limit are required", - )); + return Err(HandleError::RedisLimits.into()); } let storage = Arc::new( release_gil(py, || { @@ -252,11 +248,12 @@ impl NativeCacheHandle { } } -fn duration(seconds: f64) -> PyResult { - Duration::try_from_secs_f64(seconds) - .ok() - .filter(|value| !value.is_zero()) - .ok_or_else(|| PyValueError::new_err("cache durations must be finite and positive")) +fn duration(seconds: f64) -> Result { + match Duration::try_from_secs_f64(seconds) { + Ok(duration) if !duration.is_zero() => Ok(duration), + Ok(_) => Err(HandleError::Duration(None)), + Err(error) => Err(HandleError::Duration(Some(error))), + } } pub(in crate::cache) fn native_handle<'py>( @@ -277,11 +274,7 @@ pub(in crate::cache) fn configured( Option>, litellm_cache_response::CacheOptions, )> { - let handle = native_handle(configured)?.ok_or_else(|| { - pyo3::exceptions::PyRuntimeError::new_err( - "native inference needs a v2 cache, the catalog keeps Python caches on Python", - ) - })?; + let handle = native_handle(configured)?.ok_or(HandleError::NotNative)?; let cache = handle.extract::>()?; cache.check_process()?; let controls = kwargs.get_item("cache")?.filter(|value| !value.is_none()); @@ -296,9 +289,7 @@ pub(in crate::cache) fn configured( name.as_str(), "no-cache" | "no-store" | "ttl" | "s-maxage" | "s-max-age" | "use-cache" ) { - return Err(PyValueError::new_err(format!( - "unsupported v2 cache control: {name}" - ))); + return Err(HandleError::Control(name).into()); } } } @@ -316,7 +307,7 @@ pub(in crate::cache) fn configured( .map(|values| values.get_item(name)) .transpose()? .flatten() - .map(|value| duration(value.extract()?)) + .map(|value| Ok(duration(value.extract()?)?)) .transpose() }; Ok(( diff --git a/litellm-rust/crates/python-bridge/src/cache/python/host.rs b/litellm-rust/crates/python-bridge/src/cache/python/host.rs index d8f8ac7c181..82b7d881df6 100644 --- a/litellm-rust/crates/python-bridge/src/cache/python/host.rs +++ b/litellm-rust/crates/python-bridge/src/cache/python/host.rs @@ -9,6 +9,7 @@ use pyo3::{ use serde_json::Value; use super::service::CacheCall; +use crate::cache::HandleError; enum Pending { Lookup(Reply, Error>>), @@ -43,16 +44,12 @@ impl PythonCache { pub fn begin(&mut self, py: Python<'_>, call: CacheCall) -> PyResult>> { let Some(cache) = self.cache.as_ref() else { - return Err(pyo3::exceptions::PyRuntimeError::new_err( - "cache operation without configured cache", - )); + return Err(HandleError::Unconfigured.into()); }; let arguments = self .arguments .as_ref() - .ok_or_else(|| { - pyo3::exceptions::PyRuntimeError::new_err("cache arguments unavailable") - })? + .ok_or(HandleError::MissingArguments)? .bind(py) .copy()?; let (method, result) = match call { @@ -97,33 +94,30 @@ impl PythonCache { py: Python<'_>, result: PyResult>, ) -> PyResult>> { - if let Err(error) = &result - && !error.is_instance_of::(py) - { - self.pending = None; - return Err(result.err().unwrap()); - } + let result = match result { + Err(error) if !error.is_instance_of::(py) => { + self.pending = None; + return Err(error); + } + result => result, + }; match self.pending.take() { Some(Pending::Lookup(reply)) => { - let value = result.map_err(|_| Error::Unavailable).and_then(|value| { + let value = result.map_err(Error::unavailable).and_then(|value| { if value.bind(py).is_none() { Ok(None) } else { from_py(value.bind(py)) .map(Some) - .map_err(|_| Error::InvalidEntry) + .map_err(Error::invalid_entry) } }); reply.send(value); } Some(Pending::Store(reply)) => { - reply.send(result.map(|_| ()).map_err(|_| Error::Unavailable)); - } - None => { - return Err(pyo3::exceptions::PyRuntimeError::new_err( - "cache reply without pending operation", - )); + reply.send(result.map(|_| ()).map_err(Error::unavailable)); } + None => return Err(HandleError::NothingPending.into()), } Ok(None) } diff --git a/litellm-rust/crates/python-bridge/src/cache/python/service.rs b/litellm-rust/crates/python-bridge/src/cache/python/service.rs index f215e5e97f5..38872b85d30 100644 --- a/litellm-rust/crates/python-bridge/src/cache/python/service.rs +++ b/litellm-rust/crates/python-bridge/src/cache/python/service.rs @@ -17,18 +17,21 @@ fn from_python(value: Value) -> Result { let output = match value.get(STREAM_EVENTS_KEY) { Some(events) => { let events: Vec = - serde_json::from_value(events.clone()).map_err(|_| Error::InvalidEntry)?; + serde_json::from_value(events.clone()).map_err(Error::invalid_entry)?; CachedOutput::Stream(events.concat()) } None => CachedOutput::Response(value), }; - serde_json::to_value(ResponseEnvelope::new("messages", output)).map_err(|_| Error::InvalidEntry) + serde_json::to_value(ResponseEnvelope::new("messages", output)).map_err(Error::invalid_entry) } fn to_python(value: Value) -> Result { let envelope: ResponseEnvelope> = - serde_json::from_value(value).map_err(|_| Error::InvalidEntry)?; - match envelope.decode("messages").ok_or(Error::InvalidEntry)? { + serde_json::from_value(value).map_err(Error::invalid_entry)?; + match envelope + .decode("messages") + .ok_or(Error::InvalidEntry(None))? + { CachedOutput::Response(response) => Ok(response), CachedOutput::Stream(text) => Ok(serde_json::json!({ STREAM_EVENTS_KEY: text.split_inclusive("\n\n").collect::>() @@ -56,7 +59,7 @@ pub(in crate::cache) fn service>( namespace: String, ) -> std::sync::Arc where - P::Error: From, + P::Error: From + std::error::Error, { std::sync::Arc::new(PythonCacheService { services, @@ -69,7 +72,7 @@ where impl> ResponseCacheService for PythonCacheService

where - P::Error: From, + P::Error: From + std::error::Error, { fn config(&self) -> &ResponseCacheConfig { &self.config @@ -84,7 +87,7 @@ where self.services .call(|reply| CacheCall::Lookup { reply }) .await - .map_err(|_| Error::Unavailable)?? + .map_err(Error::unavailable)?? .map(from_python) .transpose() }) @@ -101,7 +104,7 @@ where self.services .call(|reply| CacheCall::Store { value, reply }) .await - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? }) } } diff --git a/litellm-rust/crates/python-bridge/src/cache/selection.rs b/litellm-rust/crates/python-bridge/src/cache/selection.rs index cd19533fc73..00fc7cd822f 100644 --- a/litellm-rust/crates/python-bridge/src/cache/selection.rs +++ b/litellm-rust/crates/python-bridge/src/cache/selection.rs @@ -39,7 +39,7 @@ impl Selection { services: HostServices

, ) -> (Option, CacheOptions) where - P::Error: From, + P::Error: From + std::error::Error, { let service = match self.backend { Backend::Disabled => None, diff --git a/litellm-rust/crates/python-bridge/src/callable.rs b/litellm-rust/crates/python-bridge/src/callable.rs index d0ed76a3e8b..605b649e1f3 100644 --- a/litellm-rust/crates/python-bridge/src/callable.rs +++ b/litellm-rust/crates/python-bridge/src/callable.rs @@ -48,14 +48,8 @@ mod tests { Err(PyErr::from_value(error.clone())) } - #[rstest::rstest] - fn only_ordinary_exceptions_are_reported_under_the_template() { - Python::initialize(); - Python::attach(|py| { - let locals = PyDict::new(py); - py.run( - pyo3::ffi::c_str!( - r#" + const FAILURES: &std::ffi::CStr = pyo3::ffi::c_str!( + r#" class CallerError(Exception): def __format__(self, specification): return 'unavailable' @@ -63,12 +57,19 @@ ordinary = CallerError('must use __format__') type_error = TypeError('signature') abort = KeyboardInterrupt('cancelled') "# - ), - Some(&locals), - Some(&locals), - ) - .unwrap(); + ); + fn failures(py: Python<'_>) -> Bound<'_, PyDict> { + let locals = PyDict::new(py); + py.run(FAILURES, Some(&locals), Some(&locals)).unwrap(); + locals + } + + #[test] + fn ordinary_exceptions_are_reported_under_the_template() { + Python::initialize(); + Python::attach(|py| { + let locals = failures(py); let original = raised(&locals, "ordinary"); let wrapped = wrap_failure(py, TEMPLATE, failure(&original)).unwrap_err(); assert!(wrapped.is_instance_of::(py)); @@ -78,12 +79,19 @@ abort = KeyboardInterrupt('cancelled') wrapped.value(py).str().unwrap().to_str().unwrap(), "Failed to reach the caller: unavailable" ); + }); + } - for name in ["type_error", "abort"] { - let original = raised(&locals, name); - let error = wrap_failure(py, TEMPLATE, failure(&original)).unwrap_err(); - assert!(error.value(py).is(&original)); - } + #[rstest::rstest] + #[case::type_error("type_error")] + #[case::keyboard_interrupt("abort")] + fn other_exceptions_pass_through_unwrapped(#[case] name: &str) { + Python::initialize(); + Python::attach(|py| { + let locals = failures(py); + let original = raised(&locals, name); + let error = wrap_failure(py, TEMPLATE, failure(&original)).unwrap_err(); + assert!(error.value(py).is(&original)); }); } diff --git a/litellm-rust/crates/python-bridge/src/coercion.rs b/litellm-rust/crates/python-bridge/src/coercion.rs index 77c8cabc22b..8cdf1e6c5bc 100644 --- a/litellm-rust/crates/python-bridge/src/coercion.rs +++ b/litellm-rust/crates/python-bridge/src/coercion.rs @@ -1,4 +1,5 @@ use litellm_core_utils::serde_compat::parse_str_bool; +use litellm_error::Report; use pyo3::{ exceptions::{PyAttributeError, PyRuntimeError, PyValueError}, prelude::*, @@ -9,15 +10,20 @@ use pyo3::{ pub(crate) enum ProjectionError { Python(PyErr), InvalidConfiguration(ProjectionDetail), - UnsupportedLiveObject(String), + UnsupportedLiveObject(ProjectionDetail), InternalSchemaFailure(ProjectionDetail), } #[derive(Debug, thiserror::Error)] pub(crate) enum ProjectionDetail { - #[error("{0}")] - Message(String), - #[error("{field}: {source}")] + #[error("{group}.{name}: expected {expected}, got {actual}")] + Expected { + group: &'static str, + name: &'static str, + expected: &'static str, + actual: String, + }, + #[error("{field}")] InvalidField { field: &'static str, #[source] @@ -30,12 +36,6 @@ pub(crate) enum ProjectionDetail { }, } -impl From for ProjectionDetail { - fn from(message: String) -> Self { - Self::Message(message) - } -} - impl From for ProjectionError { fn from(error: PyErr) -> Self { Self::Python(error) @@ -46,12 +46,12 @@ impl From for PyErr { fn from(error: ProjectionError) -> Self { match error { ProjectionError::Python(error) => error, - ProjectionError::InvalidConfiguration(detail) => { - PyValueError::new_err(detail.to_string()) + ProjectionError::InvalidConfiguration(detail) + | ProjectionError::UnsupportedLiveObject(detail) => { + PyValueError::new_err(Report::new(&detail).to_string()) } - ProjectionError::UnsupportedLiveObject(message) => PyValueError::new_err(message), ProjectionError::InternalSchemaFailure(detail) => { - PyRuntimeError::new_err(detail.to_string()) + PyRuntimeError::new_err(Report::new(&detail).to_string()) } } } @@ -124,26 +124,26 @@ impl<'py> Field<'py> { && getter.is(object.getattr("__getattribute__")?)) } - pub(crate) fn path(&self) -> String { - format!("{}.{}", self.group, self.name) - } - /// A member of this field's collection, reported under the same path. pub(crate) fn member(&self, value: Bound<'py, PyAny>) -> Self { Self::new(self.group, self.name, value) } - pub(crate) fn expected(&self, expected: &str) -> Result { - Ok(format!( - "{}: expected {expected}, got {}", - self.path(), - self.value.get_type().name()? - )) + pub(crate) fn expected( + &self, + expected: &'static str, + ) -> Result { + Ok(ProjectionDetail::Expected { + group: self.group, + name: self.name, + expected, + actual: self.value.get_type().name()?.to_string(), + }) } - pub(crate) fn invalid(&self, expected: &str) -> ProjectionError { + pub(crate) fn invalid(&self, expected: &'static str) -> ProjectionError { match self.expected(expected) { - Ok(message) => ProjectionError::InvalidConfiguration(message.into()), + Ok(detail) => ProjectionError::InvalidConfiguration(detail), Err(error) => error, } } @@ -163,24 +163,23 @@ impl<'py> Field<'py> { pub(crate) fn schema_bool(&self) -> Result { if !self.value.is_instance_of::() { return Err(ProjectionError::InternalSchemaFailure( - self.expected("a Boolean")?.into(), + self.expected("a Boolean")?, )); } Ok(self.exact_true()) } pub(crate) fn strict_string(&self) -> Result { - let value = self - .value - .cast::() - .map_err(|_| self.invalid("a string"))?; + let Ok(value) = self.value.cast::() else { + return Err(self.invalid("a string")); + }; Ok(value.to_str()?.to_owned()) } pub(crate) fn schema_string(&self) -> Result { if !self.value.is_instance_of::() { return Err(ProjectionError::InternalSchemaFailure( - self.expected("a string")?.into(), + self.expected("a string")?, )); } self.strict_string() @@ -474,12 +473,7 @@ text = Text(' False ') }); } - #[test] - fn missing_snapshot_fields_and_descriptor_attribute_errors_are_distinct() { - with_python(|py| { - let locals = PyDict::new(py); - py.run( - c" + const RAISING_ATTRIBUTES: &std::ffi::CStr = c" failure = AttributeError('descriptor failed') class Snapshot: @property @@ -491,46 +485,56 @@ class Intercepted: def __getattribute__(self, name): raise failure dynamic = Dynamic() intercepted = Intercepted() -", - Some(&locals), - Some(&locals), - ) - .unwrap(); - let snapshot = locals.get_item("snapshot").unwrap().unwrap(); - let descriptor = PyErr::from(Field::read(&snapshot, "test", "flag").err().unwrap()); +"; + + #[rstest] + #[case::descriptor("snapshot")] + #[case::dynamic("dynamic")] + #[case::intercepted("intercepted")] + fn attribute_errors_raised_by_a_snapshot_surface_as_raised(#[case] name: &str) { + with_python(|py| { + let locals = PyDict::new(py); + py.run(RAISING_ATTRIBUTES, Some(&locals), Some(&locals)) + .unwrap(); + let value = locals.get_item(name).unwrap().unwrap(); + let error = PyErr::from(Field::read(&value, "test", "flag").err().unwrap()); assert!( - descriptor + error .value(py) .is(locals.get_item("failure").unwrap().unwrap()) ); - for name in ["dynamic", "intercepted"] { - let value = locals.get_item(name).unwrap().unwrap(); - let error = PyErr::from(Field::read(&value, "test", "flag").err().unwrap()); - assert!( - error - .value(py) - .is(locals.get_item("failure").unwrap().unwrap()) - ); - } + }); + } + + #[test] + fn missing_snapshot_fields_are_runtime_errors_naming_the_field() { + with_python(|py| { + let locals = PyDict::new(py); + py.run(RAISING_ATTRIBUTES, Some(&locals), Some(&locals)) + .unwrap(); + let snapshot = locals.get_item("snapshot").unwrap().unwrap(); let missing = PyErr::from(Field::read(&snapshot, "test", "missing").err().unwrap()); assert!(missing.is_instance_of::(py)); assert!(missing.to_string().contains("test.missing")); }); } - #[test] - fn configuration_errors_name_fields_without_exposing_values() { + #[rstest] + #[case::mapping("{'secret': 'do-not-print'}")] + #[case::nested("['host.test', {'secret': 'do-not-print'}]")] + fn configuration_errors_name_fields_without_exposing_values(#[case] source: &str) { + with_python(|py| { + let field = Field::new("test", "setting", evaluate(py, source)); + let error = PyErr::from(field.falsy_optional_string().err().unwrap()); + assert!(error.is_instance_of::(py)); + assert!(error.to_string().contains("test.setting")); + assert!(!error.to_string().contains("do-not-print")); + }); + } + + #[test] + fn invalid_collections_and_flags_are_configuration_errors() { with_python(|py| { - for source in [ - "{'secret': 'do-not-print'}", - "['host.test', {'secret': 'do-not-print'}]", - ] { - let field = Field::new("test", "setting", evaluate(py, source)); - let error = PyErr::from(field.falsy_optional_string().err().unwrap()); - assert!(error.is_instance_of::(py)); - assert!(error.to_string().contains("test.setting")); - assert!(!error.to_string().contains("do-not-print")); - } let hosts = Field::new( "url_policy", "user_url_allowed_hosts", diff --git a/litellm-rust/crates/python-bridge/src/credentials.rs b/litellm-rust/crates/python-bridge/src/credentials.rs index 04d8e94c8fe..a4bb064aae0 100644 --- a/litellm-rust/crates/python-bridge/src/credentials.rs +++ b/litellm-rust/crates/python-bridge/src/credentials.rs @@ -10,10 +10,19 @@ use pyo3::{ types::{PyDict, PyString}, }; -const NOT_CALLABLE: &str = "Azure AD token provider must be callable"; const NOT_A_STRING: &str = "Azure AD token must be a string, got {}"; const FAILED: &str = "Failed to get Azure AD token: {}"; +#[derive(Debug, thiserror::Error)] +#[error("Azure AD token provider must be callable")] +struct NotCallable; + +impl From for PyErr { + fn from(error: NotCallable) -> Self { + PyTypeError::new_err(litellm_error::Report::new(&error).to_string()) + } +} + /// The `azure_ad_token_provider` keyword argument, kept alive for the rest of the call. pub(crate) struct CallerTokenProvider { provider: Py, @@ -36,7 +45,7 @@ impl CallerTokenProvider { pub(crate) fn acquire(&self, py: Python<'_>) -> PyResult { let provider = self.provider.bind(py); if !provider.is_callable() { - return Err(PyTypeError::new_err(NOT_CALLABLE)); + return Err(NotCallable.into()); } let token = wrap_failure( py, diff --git a/litellm-rust/crates/python-bridge/src/diagnostics.rs b/litellm-rust/crates/python-bridge/src/diagnostics.rs index 687a090e768..5368da4f930 100644 --- a/litellm-rust/crates/python-bridge/src/diagnostics.rs +++ b/litellm-rust/crates/python-bridge/src/diagnostics.rs @@ -1,5 +1,5 @@ use litellm_host_python::{release_count, runtime_started}; -use pyo3::{exceptions::PyRuntimeError, prelude::*, types::PyDict}; +use pyo3::{prelude::*, types::PyDict}; #[pyfunction] pub(crate) fn gil_stats(py: Python<'_>) -> PyResult> { @@ -18,8 +18,7 @@ pub(crate) fn process_state_started() -> bool { /// so the runtime can never start. Raises if it already has. Forked workers are unaffected. #[pyfunction] pub(crate) fn reserve_process_for_forking() -> PyResult<()> { - litellm_host_python::reserve_process_for_forking() - .map_err(|_| PyRuntimeError::new_err("the native runtime already started in this process")) + Ok(litellm_host_python::reserve_process_for_forking()?) } #[cfg(feature = "panic-test")] diff --git a/litellm-rust/crates/python-bridge/src/errors.rs b/litellm-rust/crates/python-bridge/src/errors.rs index 4dab0297e73..a883ec608e5 100644 --- a/litellm-rust/crates/python-bridge/src/errors.rs +++ b/litellm-rust/crates/python-bridge/src/errors.rs @@ -1,9 +1,12 @@ -use litellm_http::transport::Error as TransportError; +use std::error::Error as StdError; + +use litellm_error::Report; use litellm_inference::Error; use pyo3::{ exceptions::{PyRuntimeError, PyValueError}, prelude::*, }; +use reqwest::header::HeaderMap; pyo3::create_exception!( _native, @@ -16,20 +19,34 @@ pyo3::create_exception!( _native, RustUpstreamError, pyo3::exceptions::PyException, - "The provider call was already issued and failed. Args are (status, message); status is 0 when there was no HTTP response." + "The provider call was already issued and the provider answered with a failure. Args are (status, body); the headers attribute holds the provider's (name, value) response headers." ); -pub(crate) fn route_error_to_pyerr(error: Error) -> PyErr { - match error { - Error::Transport(TransportError::Http { status, body }) => { - RustUpstreamError::new_err((status, body)) - } - other => by_fault(other.is_request(), other.to_string()), - } +const MESSAGES_REQUEST_ERROR: &str = "messages_request_error"; + +pub(crate) fn upstream_error( + py: Python<'_>, + status: u16, + body: &str, + headers: &HeaderMap, +) -> PyResult { + let error = RustUpstreamError::new_err((status, body.to_owned())); + let pairs: Vec<(&str, String)> = headers + .iter() + .map(|(name, value)| { + ( + name.as_str(), + String::from_utf8_lossy(value.as_bytes()).into_owned(), + ) + }) + .collect(); + error.value(py).setattr("headers", pairs)?; + Ok(error) } /// A request the caller got wrong is a `ValueError`; anything else is a `RuntimeError`. -pub(crate) fn by_fault(is_request: bool, message: String) -> PyErr { +pub(crate) fn by_fault(is_request: bool, error: &(dyn StdError + 'static)) -> PyErr { + let message = Report::new(error).to_string(); if is_request { PyValueError::new_err(message) } else { @@ -37,17 +54,82 @@ pub(crate) fn by_fault(is_request: bool, message: String) -> PyErr { } } +/// How a route flags a rejected request for its Python route module, which raises the public +/// exception for it instead of mapping the native one. +#[derive(Clone, Copy)] +pub(crate) enum RequestMarker { + Unmarked, + Messages, +} + +/// Maps a route failure once. A Python exception raised inside the call is returned as it is, +/// everything else is mapped and handed to `public`, the route's public exception mapping. +pub(crate) fn route_failure( + py: Python<'_>, + error: Error, + marker: RequestMarker, + public: impl FnOnce(PyErr) -> PyResult, +) -> PyResult { + if let Some(raised) = crate::secrets::python_error(py, &error) { + return Ok(raised); + } + if let Some((status, body)) = error.upstream_response() { + return public(upstream_error(py, status, body, &HeaderMap::new())?); + } + let native = match (marker, error.rejected_request()) { + (RequestMarker::Messages, Some(rejection)) => { + let native = PyValueError::new_err(Report::new(rejection).to_string()); + native.value(py).setattr(MESSAGES_REQUEST_ERROR, true)?; + native + } + (RequestMarker::Messages, None) | (RequestMarker::Unmarked, _) => { + by_fault(error.is_request(), &error) + } + }; + public(native) +} + +pub(crate) fn route_error_to_pyerr(error: Error) -> PyErr { + Python::attach(|py| { + route_failure(py, error, RequestMarker::Unmarked, Ok).unwrap_or_else(std::convert::identity) + }) +} + #[cfg(test)] mod tests { + use reqwest::header::{HeaderValue, RETRY_AFTER}; + use super::*; + fn upstream_payload(py: Python<'_>, failure: &PyErr) -> ((u16, String), Vec<(String, String)>) { + let value = failure.value(py); + ( + value.getattr("args").unwrap().extract().unwrap(), + value.getattr("headers").unwrap().extract().unwrap(), + ) + } + #[rstest::rstest] #[case::unsupported(Error::Unsupported("test capability"), true)] #[case::invalid_provider(Error::InvalidProvider("unknown".into()), true)] - #[case::invalid_request(Error::InvalidRequest("empty messages".into()), true)] - #[case::connection(TransportError::Connect("unreachable".into()).into(), false)] - #[case::network(TransportError::Network("timed out".into()).into(), false)] - #[case::invalid_response(Error::InvalidResponse("missing usage".into()), false)] + #[case::invalid_request( + Error::invalid_request(litellm_llms::ErrorDetail::Expected { + subject: "messages", + expected: "non-empty", + }), + true + )] + #[case::lost_connection( + litellm_http::Error::from(tokio_tungstenite::tungstenite::Error::ConnectionClosed).into(), + false + )] + #[case::invalid_response( + Error::invalid_response(litellm_llms::ErrorDetail::MissingField { + subject: "response", + field: "usage", + }), + false + )] fn route_failures_are_terminal(#[case] error: Error, #[case] is_request: bool) { Python::initialize(); Python::attach(|py| { @@ -59,29 +141,69 @@ mod tests { } #[rstest::rstest] - fn transport_status_survives_python_mapping() { + #[case::unmarked(RequestMarker::Unmarked)] + #[case::messages(RequestMarker::Messages)] + fn the_provider_answer_passes_through_with_a_headers_attribute(#[case] marker: RequestMarker) { Python::initialize(); Python::attach(|py| { - let upstream = route_error_to_pyerr( - TransportError::Http { - status: 429, - body: "slow down".into(), - } - .into(), - ); + let error = litellm_http::Error::upstream(429, "slow down".into()).into(); + let upstream = route_failure(py, error, marker, Ok).unwrap(); assert!(upstream.is_instance_of::(py)); assert_eq!( + upstream_payload(py, &upstream), + ((429, "slow down".into()), Vec::new()) + ); + assert!( upstream .value(py) - .getattr("args") + .getattr_opt(MESSAGES_REQUEST_ERROR) .unwrap() - .extract::<(u16, String)>() - .unwrap(), - (429, "slow down".into()) + .is_none() ); }); } + #[test] + fn upstream_error_keeps_the_provider_headers() { + Python::initialize(); + Python::attach(|py| { + let headers = HeaderMap::from_iter([(RETRY_AFTER, HeaderValue::from_static("17"))]); + let upstream = upstream_error(py, 503, "busy", &headers).unwrap(); + assert_eq!( + upstream_payload(py, &upstream), + ( + (503, "busy".into()), + vec![("retry-after".into(), "17".into())] + ) + ); + }); + } + + #[rstest::rstest] + #[case::ascii("busy ".repeat(80))] + #[case::unicode("é".repeat(300))] + fn long_provider_bodies_reach_python_intact( + #[case] message: String, + #[values(RequestMarker::Unmarked, RequestMarker::Messages)] marker: RequestMarker, + ) { + let envelope = serde_json::json!({ + "type": "error", + "error": {"type": "overloaded_error", "message": message}, + "request_id": "provider-request", + }); + let body = envelope.to_string(); + let error = litellm_http::Error::upstream(529, body.clone()).into(); + Python::initialize(); + Python::attach(|py| { + let upstream = route_failure(py, error, marker, Ok).unwrap(); + assert!(upstream.is_instance_of::(py)); + let ((status, returned_body), headers) = upstream_payload(py, &upstream); + assert_eq!(status, 529); + assert_eq!(returned_body, body); + assert!(headers.is_empty()); + }); + } + #[test] fn missing_api_key_stays_a_runtime_error_while_other_auth_failures_are_value_errors() { Python::initialize(); @@ -95,4 +217,83 @@ mod tests { assert!(invalid.is_instance_of::(py)); }); } + + #[test] + fn a_wrapped_failure_renders_every_layer() { + Python::initialize(); + Python::attach(|py| { + let error = Error::post_call( + litellm_http::Error::upstream(500, "hook backend down".into()).into(), + ); + let report = Report::new(&error).to_string(); + let failure = route_error_to_pyerr(error); + assert_eq!(failure.value(py).to_string(), report); + assert!(report.contains("hook backend down"), "{report}"); + }); + } + + #[rstest::rstest] + #[case::rejected_request( + Error::invalid_request(litellm_llms::ErrorDetail::UnsupportedParameter { + model: "claude".into(), + param: "top_k".into(), + value: "5".into(), + hint: String::new(), + }), + Some("claude does not support top_k=5. To drop unsupported params, set `litellm.drop_params = True`.") + )] + #[case::missing_field( + litellm_llms::Error::MissingField("max_tokens").into(), + Some("missing required field: max_tokens") + )] + #[case::unresolvable_provider(Error::InvalidProvider("openai".into()), None)] + #[case::invalid_response( + Error::invalid_response(litellm_llms::ErrorDetail::MissingField { + subject: "response", + field: "usage", + }), + None + )] + fn only_messages_request_rejections_carry_the_marker_and_python_text( + #[case] error: Error, + #[case] rejection: Option<&str>, + ) { + Python::initialize(); + Python::attach(|py| { + let unmarked = route_failure(py, error.clone(), RequestMarker::Unmarked, Ok).unwrap(); + assert!( + unmarked + .value(py) + .getattr_opt(MESSAGES_REQUEST_ERROR) + .unwrap() + .is_none() + ); + let native = route_failure(py, error, RequestMarker::Messages, Ok).unwrap(); + let marked = native + .value(py) + .getattr_opt(MESSAGES_REQUEST_ERROR) + .unwrap() + .map(|value| value.extract::().unwrap()); + assert_eq!(marked.unwrap_or(false), rejection.is_some()); + if let Some(rejection) = rejection { + assert!(native.is_instance_of::(py)); + assert_eq!(native.value(py).to_string(), rejection); + } + }); + } + + #[test] + fn a_python_secret_manager_exception_skips_the_public_mapping() { + Python::initialize(); + Python::attach(|py| { + let raised = pyo3::exceptions::PyKeyError::new_err("vault down"); + let identity = raised.value(py).clone().unbind(); + let error = Error::from(crate::secrets::read_error(py, raised)); + let surfaced = route_failure(py, error, RequestMarker::Unmarked, |_| { + panic!("the raised exception must not be remapped") + }) + .unwrap(); + assert!(surfaced.value(py).is(identity.bind(py))); + }); + } } diff --git a/litellm-rust/crates/python-bridge/src/http.rs b/litellm-rust/crates/python-bridge/src/http.rs index 57bf8aafc4e..b4e20a8f750 100644 --- a/litellm-rust/crates/python-bridge/src/http.rs +++ b/litellm-rust/crates/python-bridge/src/http.rs @@ -132,26 +132,14 @@ pub(crate) fn host_client(py: Python<'_>, variant: ClientVariant) -> PyResult PyErr { - match error { - litellm_http::Error::Read { - tls_source: TlsSource::ClientIdentity, - .. - } - | litellm_http::Error::InvalidPem { - tls_source: TlsSource::ClientIdentity, - .. - } => PyValueError::new_err( + match error.tls_source() { + Some(TlsSource::ClientIdentity) => PyValueError::new_err( "http_settings.ssl_certificate: expected a readable PEM certificate and private key", ), - litellm_http::Error::Read { - tls_source: TlsSource::CaBundle, - .. + Some(TlsSource::CaBundle) => { + PyValueError::new_err("http_settings.ssl_verify: expected a readable PEM CA bundle") } - | litellm_http::Error::InvalidPem { - tls_source: TlsSource::CaBundle, - .. - } => PyValueError::new_err("http_settings.ssl_verify: expected a readable PEM CA bundle"), - _ => PyValueError::new_err("http_settings: native HTTP client configuration is invalid"), + None => PyValueError::new_err(litellm_error::Report::new(&error).to_string()), } } @@ -274,21 +262,28 @@ settings = types.SimpleNamespace(**defaults) fn client_error_uses_tls_source_when_paths_match() { Python::initialize(); Python::attach(|py| { - let path = PathBuf::from("/shared.pem"); - let ca_error = client_error(litellm_http::Error::InvalidPem { - path: path.clone(), - message: "invalid".into(), - tls_source: TlsSource::CaBundle, - }); + let file = tempfile::NamedTempFile::new().unwrap(); + std::fs::write(file.path(), b"not a certificate").unwrap(); + let path = file.path().to_path_buf(); + let defaults = Resolution::from(&HttpSettings::default()).config; + let build = |config: HttpClientConfig| { + HttpClientPool::new(Arc::new(PublicDnsResolver)) + .client(&config, ClientVariant::Provider) + .map(drop) + .unwrap_err() + }; + let ca_error = client_error(build(HttpClientConfig { + verify: Verify::CaBundle(path.clone()), + ..defaults.clone() + })); assert_eq!( ca_error.to_string(), "ValueError: http_settings.ssl_verify: expected a readable PEM CA bundle" ); - let client_error = client_error(litellm_http::Error::InvalidPem { - path, - message: "invalid".into(), - tls_source: TlsSource::ClientIdentity, - }); + let client_error = client_error(build(HttpClientConfig { + client_certificate: Some(litellm_http::ClientIdentity::Pem(path)), + ..defaults + })); assert!(client_error.is_instance_of::(py)); assert_eq!( client_error.to_string(), @@ -297,6 +292,34 @@ settings = types.SimpleNamespace(**defaults) }); } + #[test] + fn client_error_without_a_tls_source_reports_the_build_failure() { + Python::initialize(); + Python::attach(|py| { + let error = HttpClientPool::new(Arc::new(PublicDnsResolver)) + .client( + &HttpClientConfig { + user_agent: Some("bad\nagent".into()), + ..Resolution::from(&HttpSettings::default()).config + }, + ClientVariant::Provider, + ) + .map(drop) + .unwrap_err(); + assert_eq!(error.tls_source(), None); + let report = litellm_error::Report::new(&error).to_string(); + + let failure = client_error(error); + + assert!(failure.is_instance_of::(py)); + assert_eq!(failure.value(py).str().unwrap().to_string(), report); + assert!( + report.contains("could not build the HTTP client: "), + "{report}" + ); + }); + } + #[test] fn python_settings_flow_into_the_configured_layer() { Python::initialize(); diff --git a/litellm-rust/crates/python-bridge/src/logger/mod.rs b/litellm-rust/crates/python-bridge/src/logger/mod.rs index bf5c735360b..8f40d94f458 100644 --- a/litellm-rust/crates/python-bridge/src/logger/mod.rs +++ b/litellm-rust/crates/python-bridge/src/logger/mod.rs @@ -81,8 +81,12 @@ impl NativeDiagnosticProcessor { } } -fn processing_error(_: fancy_regex::Error) -> PyErr { - PyRuntimeError::new_err("diagnostic processing failed") +#[derive(Debug, thiserror::Error)] +#[error("diagnostic processing failed")] +struct ProcessingError(#[source] fancy_regex::Error); + +fn processing_error(error: fancy_regex::Error) -> PyErr { + PyRuntimeError::new_err(litellm_error::Report::new(&ProcessingError(error)).to_string()) } struct PythonSink { diff --git a/litellm-rust/crates/python-bridge/src/marshal.rs b/litellm-rust/crates/python-bridge/src/marshal.rs index 038fa8c9630..dec8ff0b0ce 100644 --- a/litellm-rust/crates/python-bridge/src/marshal.rs +++ b/litellm-rust/crates/python-bridge/src/marshal.rs @@ -21,10 +21,27 @@ pub(crate) struct RouteOptions { pub(crate) timeout: Option, } -fn required_object(name: &'static str, value: Value) -> PyResult> { +/// A caller argument the bridge cannot use, raised as a `ValueError`. +#[derive(Debug, thiserror::Error)] +pub(crate) enum ArgumentError { + #[error("{0} is required")] + Required(&'static str), + #[error("{0} must be a dict")] + NotADict(&'static str), + #[error("header values must be strings")] + HeaderValue, +} + +impl From for PyErr { + fn from(error: ArgumentError) -> Self { + PyValueError::new_err(litellm_error::Report::new(&error).to_string()) + } +} + +fn required_object(name: &'static str, value: Value) -> Result, ArgumentError> { match value { Value::Object(values) => Ok(values), - _ => Err(PyValueError::new_err(format!("{name} must be a dict"))), + _ => Err(ArgumentError::NotADict(name)), } } @@ -35,7 +52,7 @@ fn optional_object( if value.is_none() { return Ok(None); } - required_object(name, from_py_argument(value)?).map(Some) + Ok(Some(required_object(name, from_py_argument(value)?)?)) } pub(crate) fn optional_timeout(timeout_seconds: Option) -> Option { @@ -57,11 +74,11 @@ pub(crate) fn python_timeout_seconds(py: Python<'_>, timeout: Py) -> PyRe pub(crate) fn required_field<'py>( fields: &Bound<'py, PyDict>, - name: &str, + name: &'static str, ) -> PyResult> { - fields + Ok(fields .get_item(name)? - .ok_or_else(|| PyValueError::new_err(format!("{name} is required"))) + .ok_or(ArgumentError::Required(name))?) } pub(crate) fn optional_field( @@ -171,21 +188,18 @@ pub(crate) fn request_input_sources<'a>( .collect()) } -pub(crate) fn marshal_headers(headers: Option) -> PyResult> { - let value = match headers { - Some(headers) => headers, - None => Value::Object(Map::new()), - }; - let Value::Object(headers) = value else { - return Err(PyValueError::new_err("headers must be a dict")); +pub(crate) fn marshal_headers( + headers: Option, +) -> Result, ArgumentError> { + let headers = match headers { + Some(headers) => required_object("headers", headers)?, + None => Map::new(), }; headers .into_iter() - .map(|(name, value)| { - value - .as_str() - .map(|value| (name, value.to_string())) - .ok_or_else(|| PyValueError::new_err("header values must be strings")) + .map(|(name, value)| match value { + Value::String(value) => Ok((name, value)), + _ => Err(ArgumentError::HeaderValue), }) .collect() } @@ -395,48 +409,52 @@ module.__getattr__ = fail float: Option, } - #[test] - fn numeric_adapters_agree_across_json_and_python_boundaries() { + #[rstest::rstest] + #[case::empty(json!({}))] + #[case::explicit_nulls(json!({"integers": null, "float": null}))] + #[case::lax_values(json!({ + "integers": [i64::MIN, i64::MAX, "9007199254740993.0", " +1_000.00 ", true, 3.0], + "float": " 1.25 ", + }))] + #[case::integer_overflow(json!({"integers": [u64::MAX]}))] + #[case::imprecise_integer_string(json!({"integers": ["1.0000000000000001"]}))] + #[case::fractional_integer(json!({"integers": [2.5]}))] + #[case::nan_string(json!({"float": "NaN"}))] + #[case::infinity_string(json!({"float": "inf"}))] + #[case::overflowing_float_string(json!({"float": "1e999"}))] + #[case::boolean_float(json!({"float": true}))] + #[case::large_integer_float(json!({"float": u64::MAX}))] + fn numeric_adapters_agree_across_json_and_python_boundaries(#[case] input: Value) { Python::initialize(); Python::attach(|py| { - for input in [ - json!({}), - json!({"integers": null, "float": null}), - json!({"integers": [i64::MIN, i64::MAX, "9007199254740993.0", " +1_000.00 ", true, 3.0], "float": " 1.25 "}), - json!({"integers": [u64::MAX]}), - json!({"integers": ["1.0000000000000001"]}), - json!({"integers": [2.5]}), - json!({"float": "NaN"}), - json!({"float": "inf"}), - json!({"float": "1e999"}), - json!({"float": true}), - json!({"float": u64::MAX}), - ] { - let expected = serde_json::from_value::(input.clone()); - let python = litellm_host_python::to_py(py, &input).unwrap(); - let actual = from_py::(python.bind(py)); - match (expected, actual) { - (Ok(expected), Ok(actual)) => { - assert_eq!(actual, expected); - let serialized = litellm_host_python::to_py(py, &actual).unwrap(); - assert_eq!( - from_py::(serialized.bind(py)).unwrap(), - serde_json::to_value(expected).unwrap() - ); - } - (Err(_), Err(_)) => {} - mismatch => panic!("boundary mismatch for {input}: {mismatch:?}"), + let expected = serde_json::from_value::(input.clone()); + let python = litellm_host_python::to_py(py, &input).unwrap(); + let actual = from_py::(python.bind(py)); + match (expected, actual) { + (Ok(expected), Ok(actual)) => { + assert_eq!(actual, expected); + let serialized = litellm_host_python::to_py(py, &actual).unwrap(); + assert_eq!( + from_py::(serialized.bind(py)).unwrap(), + serde_json::to_value(expected).unwrap() + ); } + (Err(_), Err(_)) => {} + mismatch => panic!("boundary mismatch for {input}: {mismatch:?}"), } - for source in [ - c"{'float': float('nan')}", - c"{'float': float('inf')}", - c"{'integers': [float('inf')]}", - c"{'integers': [2 ** 100]}", - ] { - let value = py.eval(source, None, None).unwrap(); - assert!(from_py::(&value).is_err()); - } + }); + } + + #[rstest::rstest] + #[case::nan(c"{'float': float('nan')}")] + #[case::infinity(c"{'float': float('inf')}")] + #[case::infinite_integer(c"{'integers': [float('inf')]}")] + #[case::integer_overflow(c"{'integers': [2 ** 100]}")] + fn non_finite_and_overflowing_python_numbers_are_rejected(#[case] source: &std::ffi::CStr) { + Python::initialize(); + Python::attach(|py| { + let value = py.eval(source, None, None).unwrap(); + assert!(from_py::(&value).is_err()); }); } diff --git a/litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs b/litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs index e62673d4d4f..21a57123921 100644 --- a/litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs +++ b/litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs @@ -13,7 +13,7 @@ use crate::{ }; async fn execute( - http: Result, + http: litellm_http::Client, secrets: std::sync::Arc, audio: Value, optional_params: Map, @@ -27,7 +27,7 @@ async fn execute( extra_headers, timeout, } = options; - AudioTranscriptionRoute::new(http?, crate::http::resources().auth.clone(), secrets) + AudioTranscriptionRoute::new(http, crate::http::resources().auth.clone(), secrets) .execute(AudioTranscriptionRequest { model: &model, audio, @@ -48,7 +48,8 @@ pub(crate) fn transcription(py: Python<'_>, call: NativeCall<'_>) -> PyResult( litellm_host_python::from_py_argument(&required_field(&arguments, "audio")?)?; let options = value_route_options(&arguments)?; let optional_params = optional_object_field(&arguments, "optional_params")?.unwrap_or_default(); - let http = crate::http::provider_client(py, &call.kwargs, true)?; + let http = + crate::http::provider_client(py, &call.kwargs, true)?.map_err(crate::http::client_error)?; let secrets = crate::secrets::source(py)?; run_async( py, diff --git a/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs b/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs index 5ad9f8df8b8..63ccfc2433f 100644 --- a/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs @@ -79,7 +79,7 @@ impl PythonBinding for ChatCompletionsPythonHost { self.0.error(py, error) } fn host_error(error: &PyErr) -> Error { - Error::InvalidRequest(error.to_string().into()) + crate::routes::inference::host_call_error(error) } } diff --git a/litellm-rust/crates/python-bridge/src/routes/clickhouse_spend.rs b/litellm-rust/crates/python-bridge/src/routes/clickhouse_spend.rs index f8df7a9de23..80d7bd5360e 100644 --- a/litellm-rust/crates/python-bridge/src/routes/clickhouse_spend.rs +++ b/litellm-rust/crates/python-bridge/src/routes/clickhouse_spend.rs @@ -8,24 +8,13 @@ use pyo3::{ }; fn map_error(error: Error) -> PyErr { - use litellm_storage_clickhouse::Error as StorageError; - match error { - Error::InsertTooLarge | Error::Storage(StorageError::InsertTooLarge) => { - PyOverflowError::new_err(error.to_string()) - } - Error::InvalidRow - | Error::InvalidLimit(_) - | Error::InvalidSchema - | Error::Storage( - StorageError::InvalidRow - | StorageError::InvalidLimit(_) - | StorageError::InvalidTable - | StorageError::InvalidSchema - | StorageError::EmptySql - | StorageError::InvalidParameters - | StorageError::InvalidQuery, - ) => PyValueError::new_err(error.to_string()), - Error::Migration(_) | Error::Storage(_) => PyRuntimeError::new_err(error.to_string()), + let message = litellm_error::Report::new(&error).to_string(); + if error.is_insert_too_large() { + PyOverflowError::new_err(message) + } else if error.is_request() { + PyValueError::new_err(message) + } else { + PyRuntimeError::new_err(message) } } @@ -99,26 +88,22 @@ mod tests { use super::*; use rstest::rstest; + use litellm_storage_clickhouse::Error as StorageError; + #[rstest] - #[case::row(Error::InvalidRow, "ValueError")] - #[case::insert_limit(Error::InvalidLimit("CLICKHOUSE_TRACE_MAX_INSERT_BYTES"), "ValueError")] - #[case::insert_timeout( - Error::Storage(litellm_storage_clickhouse::Error::InvalidLimit( - "CLICKHOUSE_INSERT_TIMEOUT_SECONDS" - )), + #[case::timestamp(Error::NonIntegerTimestamp { field: "start_time" }, "ValueError")] + #[case::insert_limit( + StorageError::InvalidLimit("CLICKHOUSE_TRACE_MAX_INSERT_BYTES").into(), "ValueError" )] - #[case::insert_budget(Error::InsertTooLarge, "OverflowError")] - #[case::schema( - Error::Storage(litellm_storage_clickhouse::Error::SchemaFailed(503)), - "RuntimeError" - )] + #[case::insert_budget(StorageError::InsertTooLarge.into(), "OverflowError")] + #[case::schema(StorageError::SchemaFailed(503).into(), "RuntimeError")] #[case::migration( Error::Migration(sqlx::migrate::MigrateError::VersionMismatch(6)), "RuntimeError" )] #[case::storage( - Error::Storage(litellm_storage_clickhouse::Error::InvalidUrl), + Config::new("spend".into(), "ftp://localhost", 1).err().unwrap(), "RuntimeError" )] fn spend_failures_preserve_public_exception_types( @@ -127,7 +112,7 @@ mod tests { ) { Python::initialize(); Python::attach(|py| { - let message = error.to_string(); + let message = litellm_error::Report::new(&error).to_string(); let exception = map_error(error); assert_eq!(exception.get_type(py).name().unwrap(), exception_name); assert_eq!( @@ -136,4 +121,15 @@ mod tests { ); }); } + + #[test] + fn spend_failures_render_their_cause() { + let error = Config::new("spend".into(), "not a url", 1).err().unwrap(); + let cause = std::error::Error::source(&error).unwrap().to_string(); + Python::initialize(); + Python::attach(|py| { + let message = map_error(error).value(py).str().unwrap().to_string(); + assert!(message.ends_with(&cause), "{message}"); + }); + } } diff --git a/litellm-rust/crates/python-bridge/src/routes/embeddings.rs b/litellm-rust/crates/python-bridge/src/routes/embeddings.rs index 0f73eefedd5..0935f42f85a 100644 --- a/litellm-rust/crates/python-bridge/src/routes/embeddings.rs +++ b/litellm-rust/crates/python-bridge/src/routes/embeddings.rs @@ -1,14 +1,23 @@ +use litellm_error::Report; use pyo3::prelude::*; use super::NativeCall; use crate::errors::RustBridgeDeclined; +#[derive(Debug, thiserror::Error)] +#[error("native embeddings route is not implemented")] +struct NotImplemented; + +impl From for PyErr { + fn from(error: NotImplemented) -> Self { + RustBridgeDeclined::new_err(Report::new(&error).to_string()) + } +} + #[pyfunction] pub(crate) fn embedding(call: NativeCall<'_>) -> PyResult> { drop(call); - Err(RustBridgeDeclined::new_err( - "native embeddings route is not implemented", - )) + Err(NotImplemented.into()) } #[pyfunction] diff --git a/litellm-rust/crates/python-bridge/src/routes/inference.rs b/litellm-rust/crates/python-bridge/src/routes/inference.rs index c9eb60e47ed..ca30c94b02f 100644 --- a/litellm-rust/crates/python-bridge/src/routes/inference.rs +++ b/litellm-rust/crates/python-bridge/src/routes/inference.rs @@ -6,11 +6,9 @@ use litellm_callbacks_legacy_python::LoggingOperation; use litellm_core_utils::get_llm_provider_logic::get_custom_llm_provider; use litellm_host::{call::HostedMachine, protocol::Protocol}; use litellm_host_python::{PythonBinding, PythonHostCalls, from_py}; -use litellm_http::transport::Error as TransportError; use litellm_inference::Error; use litellm_secrets::source::SecretSource; use pyo3::{ - exceptions::PyValueError, gc::{PyTraverseError, PyVisit}, prelude::*, types::PyDict, @@ -20,8 +18,10 @@ use serde_json::{Map, Value}; use super::{NativeCall, RequestView}; use crate::{ - errors::{RustUpstreamError, route_error_to_pyerr}, - marshal::{RouteOptions, optional_timeout, public_response, python_timeout_seconds}, + errors::{RequestMarker, route_failure}, + marshal::{ + ArgumentError, RouteOptions, optional_timeout, public_response, python_timeout_seconds, + }, }; pub(super) struct InferenceHost { @@ -44,7 +44,7 @@ impl InferenceHost { &mut self, py: Python<'_>, arguments: &Bound<'_, PyDict>, - input: &str, + input: &'static str, ) -> PyResult { self.view.resolve(py, arguments)?; let argument = |name: &str| self.argument(py, arguments, name); @@ -57,7 +57,7 @@ impl InferenceHost { .map(|value| python_timeout_seconds(py, value.unbind())) .transpose()? .flatten(); - let model = string("model")?.ok_or_else(|| PyValueError::new_err("model is required"))?; + let model = string("model")?.ok_or(ArgumentError::Required("model"))?; let custom_llm_provider = string("custom_llm_provider")?; let provider = get_custom_llm_provider(&model, custom_llm_provider.as_deref()) .map_or("", |resolved| resolved.custom_llm_provider); @@ -82,10 +82,7 @@ impl InferenceHost { .transpose()?, timeout: optional_timeout(timeout), }, - input: from_py( - &argument(input)? - .ok_or_else(|| PyValueError::new_err(format!("{input} is required")))?, - )?, + input: from_py(&argument(input)?.ok_or(ArgumentError::Required(input))?)?, params, }) } @@ -120,29 +117,20 @@ impl InferenceHost { } pub fn error(&self, py: Python<'_>, error: Error) -> PyResult { - if let Error::Secret(source) = &error - && let Some(original) = crate::secrets::python_error(py, source.source_error()) - { - return Ok(original); - } - let native = match error { - Error::Transport(TransportError::Http { status, body }) => { - let error = RustUpstreamError::new_err((status, body)); - error - .value(py) - .setattr("headers", Vec::<(String, String)>::new())?; - error - } - other => route_error_to_pyerr(other), - }; - let mapped = py - .import(self.module)? - .getattr("map_failure")? - .call1((native.value(py), self.view.request(py)?))?; - Ok(PyErr::from_value(mapped)) + route_failure(py, error, RequestMarker::Unmarked, |native| { + let mapped = py + .import(self.module)? + .getattr("map_failure")? + .call1((native.value(py), self.view.request(py)?))?; + Ok(PyErr::from_value(mapped)) + }) } } +pub(super) fn host_call_error(_: &PyErr) -> Error { + Error::invalid_request(litellm_llms::ErrorDetail::HostCall) +} + pub(super) trait InferenceRoute: Sized + 'static { type Protocol: Protocol; const OPERATION: LoggingOperation; diff --git a/litellm-rust/crates/python-bridge/src/routes/messages/host.rs b/litellm-rust/crates/python-bridge/src/routes/messages/host.rs index 5e445fcaa7d..27ce8c943f0 100644 --- a/litellm-rust/crates/python-bridge/src/routes/messages/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/messages/host.rs @@ -3,7 +3,6 @@ use litellm_host_python::{PythonHostCalls, PythonOwned}; use bytes::Bytes; use litellm_host_python::{InvokeError, PythonBinding, from_py, to_py}; -use litellm_http::transport::Error as TransportError; use litellm_inference_messages::{ Error, MessagesCall, MessagesSettings, MessagesShaping, litellm_params, messages_body, route::{Messages, MessagesStreamHead}, @@ -12,7 +11,7 @@ use litellm_llms::base_llm::messages::context::MessagesModelCapabilities; use litellm_llms_types::headers::ProviderSpecificHeaders; use litellm_router_types::LitellmParams; use pyo3::{ - exceptions::{PyException, PyValueError}, + exceptions::PyException, gc::{PyTraverseError, PyVisit}, prelude::*, types::{PyBytes, PyDict}, @@ -20,14 +19,16 @@ use pyo3::{ use serde_json::{Map, Value}; use crate::{ - errors::{RustUpstreamError, route_error_to_pyerr}, - marshal::{optional_timeout, project_optional_fields, public_response, python_timeout_seconds}, + errors::{RequestMarker, route_failure}, + marshal::{ + ArgumentError, optional_timeout, project_optional_fields, public_response, + python_timeout_seconds, + }, python_settings::missing_module, routes::RequestView, }; const ROUTE_HOST_MODULE: &str = "litellm.rust_bridge.host.routes.messages"; -const REQUEST_ERROR_MARKER: &str = "messages_request_error"; fn merge_headers( forwarded: Option>, @@ -68,29 +69,6 @@ fn project_litellm_params<'py>( Ok(litellm_params(fields.into_iter().chain(folded).collect())) } -fn native_error(py: Python<'_>, error: Error) -> PyResult { - match error { - Error::Transport(TransportError::Http { status, body }) => { - let error = RustUpstreamError::new_err((status, body)); - error - .value(py) - .setattr("headers", Vec::<(String, String)>::new())?; - Ok(error) - } - Error::InvalidRequest(message) => { - let error = PyValueError::new_err(message.to_string()); - error.value(py).setattr(REQUEST_ERROR_MARKER, true)?; - Ok(error) - } - Error::MissingField(field) => { - let error = PyValueError::new_err(format!("missing required field: {field}")); - error.value(py).setattr(REQUEST_ERROR_MARKER, true)?; - Ok(error) - } - other => Ok(route_error_to_pyerr(other)), - } -} - /// The Python side of the Messages route: projects the prepared arguments and builds the /// public response, chunks and exceptions. pub(super) struct MessagesPythonHost { @@ -115,9 +93,8 @@ impl MessagesPythonHost { let string = |name: &str| -> PyResult> { argument(name)?.map(|value| value.extract()).transpose() }; - let model = string("model")?.ok_or_else(|| PyValueError::new_err("model is required"))?; - let messages = - argument("messages")?.ok_or_else(|| PyValueError::new_err("messages is required"))?; + let model = string("model")?.ok_or(ArgumentError::Required("model"))?; + let messages = argument("messages")?.ok_or(ArgumentError::Required("messages"))?; let fields = self.view.body_params(py, arguments, ROUTE_HOST_MODULE)?; let body = [ ("model".to_string(), Value::String(model.clone())), @@ -276,16 +253,13 @@ impl PythonBinding for MessagesPythonHost { } fn map_error(&self, py: Python<'_>, error: Error) -> PyResult { - if let Error::Secret(source) = &error - && let Some(original) = crate::secrets::python_error(py, source.source_error()) - { - return Ok(original); - } - Ok(self.map_failure(py, native_error(py, error)?)) + route_failure(py, error, RequestMarker::Messages, |native| { + Ok(self.map_failure(py, native)) + }) } fn host_error(error: &PyErr) -> Error { - Error::InvalidRequest(error.to_string().into()) + crate::routes::inference::host_call_error(error) } } @@ -459,33 +433,9 @@ mod tests { projected, litellm_params(serde_json::from_value(fields).unwrap()).unwrap() ), - (Err(error), Err(())) => assert!(matches!(error, Error::InvalidRequest(_))), + (Err(error), Err(())) => assert!(error.is_request(), "{error:?}"), (projected, expected) => panic!("got {projected:?}, expected {expected:?}"), } }); } - - #[rstest] - #[case::rejected_request(Error::InvalidRequest("does not support top_k=5".into()), true)] - #[case::missing_field(Error::MissingField("max_tokens"), true)] - #[case::unresolvable_provider(Error::InvalidProvider("openai".into()), false)] - #[case::upstream_failure( - Error::Transport(TransportError::Http { status: 400, body: "bad".into() }), - false, - )] - fn only_request_rejections_carry_the_request_error_marker( - #[case] error: Error, - #[case] marked: bool, - ) { - Python::initialize(); - Python::attach(|py| { - let native = native_error(py, error).unwrap(); - let marker = native - .value(py) - .getattr_opt(REQUEST_ERROR_MARKER) - .unwrap() - .map(|value| value.extract::().unwrap()); - assert_eq!(marker.unwrap_or(false), marked); - }); - } } diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/document.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/document.rs index f85d7d4c7cb..71eb743138d 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr/document.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr/document.rs @@ -9,6 +9,29 @@ use pyo3::{ types::{PyBytes, PyString, PyType}, }; +/// A `type='file'` document the bridge cannot read, with the text Python's OCR path raises. +#[derive(Debug, thiserror::Error)] +enum DocumentError { + #[error( + "document with type='file' must include a 'file' field containing a pathlib.Path, file-like object, or bytes" + )] + MissingFile, + #[error( + "OCR file input does not accept bare str values. Pass bytes, a pathlib.Path, or a file-like object." + )] + BareString, + #[error( + "Unsupported file input type: {0}. Expected pathlib.Path, bytes, or a file-like object." + )] + UnsupportedType(String), +} + +impl From for PyErr { + fn from(error: DocumentError) -> Self { + PyValueError::new_err(litellm_error::Report::new(&error).to_string()) + } +} + /// A `type='file'` document as projected: paths and bytes are typed inputs already; a /// file-like object is a reader the projection consumes once every other field is read. pub(super) enum FileDocumentInput { @@ -45,25 +68,18 @@ impl FromPyObject<'_, '_> for FileDocumentInput { Err(error) if error.is_instance_of::(py) => None, Err(error) => return Err(error), }; - let missing = || { - PyValueError::new_err( - "document with type='file' must include a 'file' field containing a pathlib.Path, file-like object, or bytes", - ) - }; - let file = document.get_item("file").map_err(|error| { - if error.is_instance_of::(py) { - missing() - } else { - error + let file = match document.get_item("file") { + Ok(file) => file, + Err(error) if error.is_instance_of::(py) => { + return Err(DocumentError::MissingFile.into()); } - })?; + Err(error) => return Err(error), + }; if file.is_none() { - return Err(missing()); + return Err(DocumentError::MissingFile.into()); } if file.is_instance_of::() { - return Err(PyValueError::new_err( - "OCR file input does not accept bare str values. Pass bytes, a pathlib.Path, or a file-like object.", - )); + return Err(DocumentError::BareString.into()); } static PATH_LIKE: PyOnceLock> = PyOnceLock::new(); if file.is_instance(PATH_LIKE.import(py, "os", "PathLike")?)? { @@ -81,10 +97,7 @@ impl FromPyObject<'_, '_> for FileDocumentInput { } match PythonFileReader::from_file_like(&file)? { Some(reader) => Ok(Self::Deferred { reader, mime_type }), - None => Err(PyValueError::new_err(format!( - "Unsupported file input type: {}. Expected pathlib.Path, bytes, or a file-like object.", - file.get_type(), - ))), + None => Err(DocumentError::UnsupportedType(file.get_type().to_string()).into()), } } } @@ -92,6 +105,7 @@ impl FromPyObject<'_, '_> for FileDocumentInput { #[cfg(test)] mod tests { use pyo3::{exceptions::PyTypeError, types::PyDict}; + use rstest::rstest; use super::*; @@ -108,40 +122,47 @@ mod tests { } } - #[test] - fn extraction_validates_required_file_and_optional_mime_type() { + enum Rejection { + Value(&'static str), + Type, + } + + #[rstest] + #[case::missing_file(c"{}", Rejection::Value("must include a 'file' field"))] + #[case::none_file(c"{'file': None}", Rejection::Value("must include a 'file' field"))] + #[case::none_mime_type(c"{'file': b'abc', 'mime_type': None}", Rejection::Type)] + #[case::numeric_mime_type(c"{'file': b'abc', 'mime_type': 7}", Rejection::Type)] + #[case::bare_str(c"{'file': 'scan.pdf'}", Rejection::Value("bare str"))] + #[case::unsupported_type( + c"{'file': object()}", + Rejection::Value("Unsupported file input type") + )] + fn extraction_rejects_invalid_documents( + #[case] expression: &std::ffi::CStr, + #[case] rejection: Rejection, + ) { Python::initialize(); Python::attach(|py| { - for expression in [c"{}", c"{'file': None}"] { - let document = py.eval(expression, None, None).unwrap(); - let error = document.extract::().err().unwrap(); - assert!(error.is_instance_of::(py)); - assert!(error.to_string().contains("must include a 'file' field")); - } - for expression in [ - c"{'file': b'abc', 'mime_type': None}", - c"{'file': b'abc', 'mime_type': 7}", - ] { - let document = py.eval(expression, None, None).unwrap(); - let error = document.extract::().err().unwrap(); - assert!(error.is_instance_of::(py)); - } let error = py - .eval(c"{'file': 'scan.pdf'}", None, None) + .eval(expression, None, None) .unwrap() .extract::() .err() .unwrap(); - assert!(error.is_instance_of::(py)); - assert!(error.to_string().contains("bare str")); - let error = py - .eval(c"{'file': object()}", None, None) - .unwrap() - .extract::() - .err() - .unwrap(); - assert!(error.is_instance_of::(py)); - assert!(error.to_string().contains("Unsupported file input type")); + match rejection { + Rejection::Value(fragment) => { + assert!(error.is_instance_of::(py)); + assert!(error.to_string().contains(fragment), "{error}"); + } + Rejection::Type => assert!(error.is_instance_of::(py)), + } + }); + } + + #[test] + fn extraction_keeps_bytes_and_their_mime_type() { + Python::initialize(); + Python::attach(|py| { let document = py .eval(c"{'file': b'abc', 'mime_type': 'image/png'}", None, None) .unwrap(); diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs index 2068b6a6e4b..31736c7b654 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs @@ -1,128 +1,203 @@ +use std::io::ErrorKind; + +use litellm_error::Report; use litellm_llms::base_llm::ocr::error::Error; use pyo3::{ exceptions::{PyFileNotFoundError, PyOSError}, prelude::*, }; -use crate::errors::{RustUpstreamError, by_fault}; +use crate::errors::{by_fault, upstream_error}; + +const REQUEST_FORMAT_ERROR: &str = "ocr_request_format_error"; pub(super) fn to_pyerr(error: Error) -> PyErr { - let status = error.http_status_code(); - let mapped = Python::attach(|py| -> PyResult { - Ok(match error { - Error::Provider { - status, - body, - headers, - } => upstream_error(py, status, body, headers)?, - Error::Transport(litellm_http::transport::Error::Http { status, body }) => { - upstream_error(py, status, body, Vec::new())? - } - Error::RequestFormat => { - let error = by_fault(true, Error::RequestFormat.to_string()); - error - .value(py) - .setattr("ocr_request_format_error", true) - .ok(); - error - } - Error::FileRead { path, source } if source.kind() == std::io::ErrorKind::NotFound => { - PyFileNotFoundError::new_err(format!("File not found: {}", path.display())) - } - Error::FileRead { source, .. } => PyOSError::new_err(source.to_string()), - other => by_fault(is_request(&other), other.to_string()), - }) - }) - .unwrap_or_else(|error| error); - attach_status(mapped, status) + Python::attach(|py| native(py, error).unwrap_or_else(std::convert::identity)) } -fn is_request(error: &Error) -> bool { - error.is_request() - || matches!( - error, - Error::Auth(_) - | Error::InvalidProvider(_) - | Error::InvalidRequest(_) - | Error::MissingField(_) - | Error::MissingDocumentUrl - ) -} - -fn upstream_error( +/// Maps an OCR route failure once. A Python exception raised inside the call is returned as it +/// is, everything else is mapped and handed to `public`, the route's public exception mapping. +pub(super) fn route_failure( py: Python<'_>, - status: u16, - body: String, - headers: Vec<(String, String)>, + error: Error, + public: impl FnOnce(PyErr) -> PyResult, ) -> PyResult { - let error = RustUpstreamError::new_err((status, body)); - error.value(py).setattr("headers", headers)?; - Ok(error) + match crate::secrets::python_error(py, &error) { + Some(raised) => Ok(raised), + None => public(native(py, error)?), + } } -fn attach_status(error: PyErr, status: Option) -> PyErr { - if let Some(status) = status { - Python::attach(|py| { - let value = error.value(py); - value.setattr("status_code", status).ok(); - value.setattr("message", value.to_string()).ok(); - }); +fn native(py: Python<'_>, error: Error) -> PyResult { + let mapped = match (&error, error.upstream_response()) { + (_, Some((status, body, headers))) => upstream_error(py, status, body, headers)?, + (Error::RequestFormat, None) => { + let mapped = by_fault(true, &error); + mapped.value(py).setattr(REQUEST_FORMAT_ERROR, true)?; + mapped + } + (Error::FileRead { path, cause }, None) if cause.kind() == ErrorKind::NotFound => { + PyFileNotFoundError::new_err(format!("File not found: {}", path.display())) + } + (Error::FileRead { cause, .. }, None) => { + PyOSError::new_err(Report::new(cause.as_ref()).to_string()) + } + (_, None) => by_fault(error.is_request(), &error), + }; + if let Some(status) = error.http_status_code() { + let value = mapped.value(py); + value.setattr("status_code", status)?; + value.setattr("message", value.str()?)?; } - error + Ok(mapped) } #[cfg(test)] mod tests { - use pyo3::exceptions::PyValueError; + use pyo3::exceptions::{PyRuntimeError, PyValueError}; + use rstest::rstest; use super::*; + use crate::errors::RustUpstreamError; - #[test] - fn preserves_python_validation_and_provider_details() { + fn status_code(py: Python<'_>, mapped: &PyErr) -> Option { + mapped + .value(py) + .getattr_opt("status_code") + .unwrap() + .map(|status| status.extract().unwrap()) + } + + fn file_read(kind: ErrorKind) -> Error { + Error::FileRead { + path: "/missing/scan.pdf".into(), + cause: std::sync::Arc::new(std::io::Error::new(kind, "disk said no")), + } + } + + #[rstest] + #[case::provider( + Error::Provider { + status: 429, + body: r#"{"message":"rate limited"}"#.into(), + headers: Box::new(reqwest::header::HeaderMap::from_iter([( + reqwest::header::RETRY_AFTER, + reqwest::header::HeaderValue::from_static("17"), + )])), + }, + vec![("retry-after".into(), "17".into())] + )] + #[case::transport(litellm_http::Error::upstream(429, r#"{"message":"rate limited"}"#.into()).into(), vec![])] + fn a_provider_answer_keeps_its_status_body_and_headers( + #[case] error: Error, + #[case] headers: Vec<(String, String)>, + ) { Python::initialize(); Python::attach(|py| { - let mapped = to_pyerr(Error::MissingDocumentUrl); - assert!(mapped.is_instance_of::(py)); - assert_eq!(mapped.value(py).to_string(), "Document URL is required"); - assert_eq!( - mapped - .value(py) - .getattr("status_code") - .unwrap() - .extract::() - .unwrap(), - 400 - ); - let mapped = to_pyerr(Error::Provider { - status: 429, - body: r#"{"message":"rate limited"}"#.to_string(), - headers: vec![("Retry-After".to_string(), "17".to_string())], - }); + let mapped = to_pyerr(error); assert!(mapped.is_instance_of::(py)); - let headers: Vec<(String, String)> = mapped - .value(py) - .getattr("headers") - .and_then(|headers| headers.extract()) - .expect("OCR failures retain provider headers"); - assert_eq!(headers, vec![("Retry-After".to_string(), "17".to_string())]); - let args: (u16, String) = mapped - .value(py) - .getattr("args") - .and_then(|args| args.extract()) - .expect("OCR failures retain status and unprefixed provider message"); - assert_eq!(args, (429, r#"{"message":"rate limited"}"#.to_string())); + let value = mapped.value(py); + let args: (u16, String) = value.getattr("args").unwrap().extract().unwrap(); + assert_eq!(args, (429, r#"{"message":"rate limited"}"#.into())); + let kept: Vec<(String, String)> = value.getattr("headers").unwrap().extract().unwrap(); + assert_eq!(kept, headers); + assert_eq!(status_code(py, &mapped), Some(429)); + }); + } - let mapped = to_pyerr(Error::InvalidRequest("invalid format".into())); + #[rstest] + #[case::not_found(Some(404), 404, "OCR document download failed")] + #[case::rate_limited(Some(429), 429, "OCR document download failed")] + #[case::timeout(None, 408, "OCR document download timed out")] + #[tokio::test] + async fn document_download_failures_keep_the_upstream_exception( + #[case] response_status: Option, + #[case] expected_status: u16, + #[case] expected_body: &str, + ) { + use std::time::Duration; + + use litellm_http::{Client, media::MediaFetcher}; + use litellm_llms::base_llm::ocr::{ + document::inline_remote_document, transformation::OcrConnection, + }; + use litellm_llms_types::formats::ocr::OcrDocument; + use tokio::{ + io::{AsyncReadExt, AsyncWriteExt}, + net::TcpListener, + }; + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let server = tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.unwrap(); + let mut request = [0_u8; 1]; + socket.read_exact(&mut request).await.unwrap(); + if let Some(status) = response_status { + socket.write_all(format!("HTTP/1.1 {status} Error\r\nContent-Length: 0\r\nConnection: close\r\n\r\n").as_bytes()).await.unwrap(); + } else { + std::future::pending::<()>().await; + } + }); + let connection = OcrConnection { + timeout: Duration::from_millis(100), + ..OcrConnection::default() + }; + let error = inline_remote_document( + &MediaFetcher::for_test(Client::for_test()), + OcrDocument::DocumentUrl { + document_url: format!("http://{address}/document.pdf"), + extra_fields: Default::default(), + }, + &connection, + ) + .await + .unwrap_err(); + server.abort(); + if response_status.is_none() { + assert!(litellm_error::find::(&error).is_some()); + } + Python::initialize(); + Python::attach(|py| { + let mapped = to_pyerr(error); + assert!(mapped.is_instance_of::(py), "{mapped}"); + let args: (u16, String) = mapped.value(py).getattr("args").unwrap().extract().unwrap(); + assert_eq!(args, (expected_status, expected_body.into())); + assert_eq!(status_code(py, &mapped), Some(expected_status)); + }); + } + + #[rstest] + #[case::missing_document_url(Error::MissingDocumentUrl)] + #[case::invalid_features(Error::Features)] + #[case::invalid_provider(Error::InvalidProvider("unknown".into()))] + fn request_faults_are_value_errors_with_status_400(#[case] error: Error) { + Python::initialize(); + Python::attach(|py| { + let report = Report::new(&error).to_string(); + let mapped = to_pyerr(error); assert!(mapped.is_instance_of::(py)); - assert_eq!( - mapped - .value(py) - .getattr("status_code") - .unwrap() - .extract::() - .unwrap(), - 400 - ); + assert_eq!(mapped.value(py).to_string(), report); + assert_eq!(status_code(py, &mapped), Some(400)); + }); + } + + #[rstest] + #[case::missing_api_key(litellm_auth::Error::MissingApiKey { + provider: "Mistral", + environment_variable: "MISTRAL_API_KEY", + } + .into())] + #[case::missing_credential(Error::MissingCredential { + credential: "Azure AI credentials", + guidance: "set AZURE_AI_API_KEY", + })] + fn a_missing_credential_is_a_statusless_runtime_error(#[case] error: Error) { + Python::initialize(); + Python::attach(|py| { + let mapped = to_pyerr(error); + assert!(mapped.is_instance_of::(py)); + assert_eq!(status_code(py, &mapped), None); }); } @@ -135,19 +210,12 @@ mod tests { assert!(mapped.is_instance_of::(py)); assert!( value - .getattr("ocr_request_format_error") + .getattr(REQUEST_FORMAT_ERROR) .unwrap() .extract::() .unwrap() ); - assert_eq!( - value - .getattr("status_code") - .unwrap() - .extract::() - .unwrap(), - 400 - ); + assert_eq!(status_code(py, &mapped), Some(400)); assert_eq!( value .getattr("message") @@ -159,18 +227,11 @@ mod tests { }); } - fn file_read(kind: std::io::ErrorKind) -> Error { - Error::FileRead { - path: "/missing/scan.pdf".into(), - source: std::sync::Arc::new(std::io::Error::new(kind, "disk said no")), - } - } - #[test] fn missing_files_map_to_file_not_found_naming_the_path() { Python::initialize(); Python::attach(|py| { - let mapped = to_pyerr(file_read(std::io::ErrorKind::NotFound)); + let mapped = to_pyerr(file_read(ErrorKind::NotFound)); assert!(mapped.is_instance_of::(py)); assert_eq!( mapped.value(py).to_string(), @@ -183,28 +244,32 @@ mod tests { fn other_file_read_failures_map_to_os_error_with_the_io_message() { Python::initialize(); Python::attach(|py| { - let mapped = to_pyerr(file_read(std::io::ErrorKind::PermissionDenied)); + let mapped = to_pyerr(file_read(ErrorKind::PermissionDenied)); assert!(mapped.is_instance_of::(py)); assert!(!mapped.is_instance_of::(py)); assert_eq!(mapped.value(py).to_string(), "disk said no"); }); } - #[rstest::rstest] + #[rstest] #[case::oversized(Error::TooLarge { limit: 7 })] - #[case::malformed_field(Error::ResponseField { path: "pages[0].index".into() })] - fn response_failures_are_statusless_runtime_errors(#[case] error: Error) { + #[case::malformed_field(Error::response_field( + "pages[0].index", + "-1".parse::().unwrap_err() + ))] + fn response_failures_are_statusless_runtime_errors( + #[case] error: Error, + #[values("status_code", "ocr_request_format_error", "headers")] attribute: &str, + ) { Python::initialize(); Python::attach(|py| { - let message = error.to_string(); + let report = Report::new(&error).to_string(); let mapped = to_pyerr(error); let value = mapped.value(py); - assert!(mapped.is_instance_of::(py)); + assert!(mapped.is_instance_of::(py)); assert!(!mapped.is_instance_of::(py)); - assert_eq!(value.to_string(), message); - for attribute in ["status_code", "ocr_request_format_error", "headers"] { - assert!(!value.hasattr(attribute).unwrap(), "{attribute}"); - } + assert_eq!(value.to_string(), report); + assert!(!value.hasattr(attribute).unwrap()); }); } } diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/host.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/host.rs index fc1b70392e2..a1a56baef8f 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr/host.rs @@ -12,7 +12,7 @@ use pyo3::{ }; use super::{ - errors::to_pyerr as ocr_error_to_pyerr, + errors::route_failure, project::{OcrHostHandles, SETTINGS_ERROR_MARKER, project_request}, }; use crate::{marshal::public_response, routes::RequestView}; @@ -135,16 +135,11 @@ impl PythonBinding for OcrPythonHost { } fn map_error(&self, py: Python<'_>, error: Error) -> PyResult { - if let Error::Secret(source) = &error - && let Some(original) = crate::secrets::python_error(py, source) - { - return Ok(original); - } - Ok(self.map_failure(py, ocr_error_to_pyerr(error))) + route_failure(py, error, |native| Ok(self.map_failure(py, native))) } - fn host_error(error: &PyErr) -> Error { - Error::InvalidRequest(error.to_string()) + fn host_error(_: &PyErr) -> Error { + Error::HostCall } } diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs index fe2f83bff46..40287c669eb 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs @@ -6,14 +6,16 @@ use litellm_inference_ocr::{ wire::{OcrWireRequest, consumed_optional_params, decode_document, decode_request_input}, }; use litellm_llms::base_llm::ocr::error::Error; -use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; +use pyo3::{prelude::*, types::PyDict}; use serde_json::{Map, Value}; use super::{document::FileDocumentInput, errors::to_pyerr as ocr_error_to_pyerr}; use crate::{ coercion::FieldSpec, credentials::{self, CallerTokenProvider}, - marshal::{project_optional_fields, python_timeout_seconds, request_input_sources}, + marshal::{ + ArgumentError, project_optional_fields, python_timeout_seconds, request_input_sources, + }, python_settings::{PythonSettings, Snapshot}, }; @@ -30,9 +32,9 @@ struct OcrArguments<'a, 'py> { } impl<'py> OcrArguments<'_, 'py> { - fn lookup(&self, name: &str) -> PyResult> { - litellm_host_python::lookup(self.kwargs, self.bound, name)? - .ok_or_else(|| PyValueError::new_err(format!("missing argument: {name}"))) + fn lookup(&self, name: &'static str) -> PyResult> { + Ok(litellm_host_python::lookup(self.kwargs, self.bound, name)? + .ok_or(ArgumentError::Required(name))?) } fn model(&self) -> PyResult { @@ -92,6 +94,7 @@ impl ProjectedDocument { { ocr_error_to_pyerr(Error::RequestField { path: "document.type".into(), + cause: None, }) } else { error @@ -218,8 +221,10 @@ fn fold_module_globals( pub(super) const SETTINGS_ERROR_MARKER: &str = "native_settings_error"; fn settings_error(py: Python<'_>, error: PyErr) -> PyErr { - error.value(py).setattr(SETTINGS_ERROR_MARKER, true).ok(); - error + match error.value(py).setattr(SETTINGS_ERROR_MARKER, true) { + Ok(()) => error, + Err(failure) => failure, + } } #[cfg(test)] diff --git a/litellm-rust/crates/python-bridge/src/routes/responses/host.rs b/litellm-rust/crates/python-bridge/src/routes/responses/host.rs index 9c430ba7868..eb19d299625 100644 --- a/litellm-rust/crates/python-bridge/src/routes/responses/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/responses/host.rs @@ -141,9 +141,9 @@ impl PythonBinding for ResponsesPythonHost { head: ::StreamHead, ) -> PyResult> { let _ = head; - Err(pyo3::exceptions::PyRuntimeError::new_err( - "native Python Responses streaming is not supported", - )) + Err(crate::errors::route_error_to_pyerr(Error::Unsupported( + "native Python responses streaming", + ))) } fn encode_chunk( @@ -152,16 +152,16 @@ impl PythonBinding for ResponsesPythonHost { chunk: ::Chunk, ) -> PyResult> { let _ = chunk; - Err(pyo3::exceptions::PyRuntimeError::new_err( - "native Python Responses streaming is not supported", - )) + Err(crate::errors::route_error_to_pyerr(Error::Unsupported( + "native Python responses streaming", + ))) } fn map_error(&self, py: Python<'_>, error: Error) -> PyResult { self.host.error(py, error) } fn host_error(error: &PyErr) -> Error { - Error::InvalidRequest(error.to_string().into()) + crate::routes::inference::host_call_error(error) } } diff --git a/litellm-rust/crates/python-bridge/src/routes/token_counter.rs b/litellm-rust/crates/python-bridge/src/routes/token_counter.rs index 2c26311231f..4b10eed087f 100644 --- a/litellm-rust/crates/python-bridge/src/routes/token_counter.rs +++ b/litellm-rust/crates/python-bridge/src/routes/token_counter.rs @@ -2,6 +2,7 @@ use crate::execution::run_async; use std::sync::Arc; use std::{num::NonZero, thread::available_parallelism}; +use litellm_error::{Report, Source}; use litellm_host_python::enter_native; use litellm_token_counter::{ CountableRequest, Error, InputTokenCount, TokenCounter as CoreTokenCounter, @@ -50,10 +51,10 @@ impl TokenCounter { let _slot = encode_slots .acquire_owned() .await - .map_err(|error| Error::Task(error.to_string()))?; + .map_err(|error| Error::Task(Source::new(error)))?; tokio::task::spawn_blocking(move || count_body(&counter, &body)) .await - .map_err(|error| Error::Task(error.to_string()))? + .map_err(|error| Error::Task(Source::new(error)))? }, token_count_error_to_pyerr, ) @@ -70,19 +71,12 @@ fn count_body(counter: &CoreTokenCounter, body: &[u8]) -> Result PyErr { - let message = error.to_string(); - match error { - Error::Load(_) - | Error::Ranks(_) - | Error::UnicodeClasses - | Error::UnsupportedTokenizer(_) => PyValueError::new_err(message), - Error::RequestParse(_) - | Error::MissingInput - | Error::FloatText - | Error::ContentBlock - | Error::ArrayItems - | Error::JsonSerialization(_) - | Error::JsonUtf8(_) => RustBridgeDeclined::new_err(message), - Error::Encode(_) | Error::Decode(_) | Error::Task(_) => PyRuntimeError::new_err(message), + let message = Report::new(&error).to_string(); + if error.is_unsupported() { + RustBridgeDeclined::new_err(message) + } else if error.is_load() { + PyValueError::new_err(message) + } else { + PyRuntimeError::new_err(message) } } diff --git a/litellm-rust/crates/python-bridge/src/secrets/config.rs b/litellm-rust/crates/python-bridge/src/secrets/config.rs index d1a1eafd468..7afd06e2373 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/config.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/config.rs @@ -128,9 +128,7 @@ pub(crate) fn read(py: Python<'_>) -> PyResult { .system .is_some_and(|system| system != backend.system()) { - return Err(pyo3::exceptions::PyValueError::new_err( - "native secret manager system does not match configuration", - )); + return Err(super::ManagerError::SystemMismatch.into()); } Ok(SecretManagerSnapshot { client: SecretManagerClient::Native(Box::new(backend)), diff --git a/litellm-rust/crates/python-bridge/src/secrets/error.rs b/litellm-rust/crates/python-bridge/src/secrets/error.rs index 5ccfd0a1554..ca51ee79921 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/error.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/error.rs @@ -1,7 +1,66 @@ use std::fmt; +use litellm_error::Report; use litellm_secrets::Error; -use pyo3::{exceptions::PyBaseException, prelude::*}; +use pyo3::{ + exceptions::{PyBaseException, PyRuntimeError, PyValueError}, + prelude::*, +}; + +/// A secret manager call the bridge rejects, with the text Python's own managers raise. +#[derive(Debug, thiserror::Error)] +pub(crate) enum ManagerError { + #[error("native secret manager must be recreated after fork")] + Forked, + #[error("unknown secret manager system")] + UnknownSystem(#[source] serde_json::Error), + #[error("invalid secret manager settings")] + Settings(#[source] litellm_error::Source), + #[error("native secret manager system does not match configuration")] + SystemMismatch, + #[error("secret manager does not support provider reads")] + ProviderReads, + #[error("invalid timeout")] + Timeout(#[source] std::time::TryFromFloatSecsError), + #[error("No secret found in CyberArk Secret Manager for {0}")] + CyberarkSecretMissing(String), + #[error("Current secret {0} not found")] + CurrentMissing(String), + #[error("Failed to verify new secret {0}")] + ReplacementMissing(String), + #[error("New secret value mismatch")] + ReplacementMismatch, + #[error("native secret manager mutation is unavailable")] + MutationUnsupported, + #[error(transparent)] + Secret(Error), +} + +impl ManagerError { + pub(crate) fn settings(error: impl std::error::Error + Send + Sync + 'static) -> Self { + Self::Settings(litellm_error::Source::new(error)) + } +} + +impl From for PyErr { + fn from(error: ManagerError) -> Self { + let message = Report::new(&error).to_string(); + match error { + ManagerError::Forked => PyRuntimeError::new_err(message), + ManagerError::UnknownSystem(_) + | ManagerError::Settings(_) + | ManagerError::SystemMismatch + | ManagerError::ProviderReads + | ManagerError::Timeout(_) + | ManagerError::CyberarkSecretMissing(_) + | ManagerError::CurrentMissing(_) + | ManagerError::ReplacementMissing(_) + | ManagerError::ReplacementMismatch + | ManagerError::MutationUnsupported + | ManagerError::Secret(_) => PyValueError::new_err(message), + } + } +} #[derive(thiserror::Error)] #[error("Python secret manager failed")] @@ -14,18 +73,23 @@ impl fmt::Debug for PythonSecretError { } pub(super) fn external_error(py: Python<'_>, error: PyErr) -> Error { - Error::ExternalManager(Box::new(PythonSecretError(error.into_value(py)))) + Error::ExternalManager(litellm_error::Source::new(PythonSecretError( + error.into_value(py), + ))) } -pub(super) fn read_error(py: Python<'_>, error: PyErr) -> Error { - Error::ExternalRead(Box::new(PythonSecretError(error.into_value(py)))) +pub(crate) fn read_error(py: Python<'_>, error: PyErr) -> Error { + Error::ExternalRead(litellm_error::Source::new(PythonSecretError( + error.into_value(py), + ))) } -pub(crate) fn python_error(py: Python<'_>, error: &Error) -> Option { - let (Error::ExternalManager(source) | Error::ExternalRead(source)) = error else { - return None; - }; - source - .downcast_ref::() +/// The exception a Python secret manager raised anywhere in the chain, so a route surfaces it +/// unchanged instead of wrapping it. +pub(crate) fn python_error( + py: Python<'_>, + error: &(dyn std::error::Error + 'static), +) -> Option { + litellm_error::find::(error) .map(|error| PyErr::from_value(error.0.clone_ref(py).into_bound(py).into_any())) } diff --git a/litellm-rust/crates/python-bridge/src/secrets/mod.rs b/litellm-rust/crates/python-bridge/src/secrets/mod.rs index cddd47d08db..57042e6a32e 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/mod.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/mod.rs @@ -11,7 +11,9 @@ mod vault; use std::sync::Arc; -pub(crate) use error::python_error; +#[cfg(test)] +pub(crate) use error::read_error; +pub(crate) use error::{ManagerError, python_error}; use litellm_secrets::source::{EnvironmentSecrets, SecretSource}; use pyo3::prelude::*; use python::PythonSecrets; diff --git a/litellm-rust/crates/python-bridge/src/secrets/mutation.rs b/litellm-rust/crates/python-bridge/src/secrets/mutation.rs index eaab06a5b12..b04060651ac 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/mutation.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/mutation.rs @@ -1,7 +1,8 @@ use super::operations::{PythonMutationError, PythonMutationResponse}; +use litellm_error::Report; use litellm_host_python::{json_loads, to_py}; use litellm_secrets::cyberark; -use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; +use pyo3::{prelude::*, types::PyDict}; pub(super) fn mutation_value( result: Result, @@ -23,18 +24,7 @@ pub(super) fn mutation_value( &serde_json::json!({"status": "error", "message": message}), ) } - Err(PythonMutationError::CurrentMissing(name)) => Err(PyValueError::new_err(format!( - "Current secret {name} not found" - ))), - Err(PythonMutationError::ReplacementMissing(name)) => Err(PyValueError::new_err(format!( - "Failed to verify new secret {name}" - ))), - Err(PythonMutationError::ReplacementMismatch) => { - Err(PyValueError::new_err("New secret value mismatch")) - } - Err(PythonMutationError::Unsupported) => Err(PyValueError::new_err( - "native secret manager mutation is unavailable", - )), + Err(PythonMutationError::Manager(error)) => Err(error.into()), }) } @@ -43,18 +33,35 @@ fn cyberark_failure( name: &str, failure: cyberark::WriteFailure, ) -> PyResult { - let message = match failure.source { + let message = if failure.source.is_unsafe_secret_name() { + format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?) + } else { + cyberark_message( + py, + failure.source, + failure.authentication, + failure.request_url.as_ref(), + )? + }; + Ok(if failure.authentication { + format!("Could not authenticate to CyberArk Conjur: {message}") + } else { + message + }) +} + +fn cyberark_message( + py: Python<'_>, + source: cyberark::Error, + authentication: bool, + request_url: Option<&reqwest::Url>, +) -> PyResult { + Ok(match source { cyberark::Error::Status(status) | cyberark::Error::AuthStatus(status) => { - let url = failure - .request_url - .as_ref() - .map_or("", reqwest::Url::as_str); + let url = request_url.map_or("", reqwest::Url::as_str); http_message(py, "POST", url, status)? } - cyberark::Error::Operation(litellm_secrets_types::Error::UnsafeSecretName) => { - format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?) - } - cyberark::Error::Http(source) if failure.authentication => match os_error_code(&source) { + cyberark::Error::Http(source) if authentication => match os_error_code(&source) { Some(code) => { let reason = py.import("os")?.getattr("strerror")?.call1((code,))?; py.import("builtins")? @@ -63,17 +70,12 @@ fn cyberark_failure( .str()? .extract()? } - None => cyberark::Error::Http(source).to_string(), + None => Report::new(&cyberark::Error::Http(source)).to_string(), }, cyberark::Error::Http(source) if source.is_connect() => { "All connection attempts failed".to_owned() } - source => source.to_string(), - }; - Ok(if failure.authentication { - format!("Could not authenticate to CyberArk Conjur: {message}") - } else { - message + source => Report::new(&source).to_string(), }) } diff --git a/litellm-rust/crates/python-bridge/src/secrets/operations.rs b/litellm-rust/crates/python-bridge/src/secrets/operations.rs index 8e338741aff..b458e3f34e8 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/operations.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/operations.rs @@ -4,6 +4,8 @@ use litellm_secrets::cyberark::AuthenticationRetry; use litellm_secrets::{Error, SecretManager}; use litellm_secrets_types::{PythonSecretRead, SecretOperationContext}; +use super::ManagerError; + pub(super) struct PythonReadRequest { pub secret_name: String, pub primary_secret_name: Option, @@ -62,15 +64,18 @@ pub(super) async fn read_python_provider( #[derive(Debug)] pub(super) enum PythonMutationError { - Unsupported, + Manager(ManagerError), Vault(Box), CyberarkWrite { name: String, failure: Box, }, - CurrentMissing(String), - ReplacementMissing(String), - ReplacementMismatch, +} + +impl From for PythonMutationError { + fn from(error: ManagerError) -> Self { + Self::Manager(error) + } } pub(super) async fn write_python_provider( @@ -89,7 +94,7 @@ pub(super) async fn write_python_provider( })?; Ok(write_success(name)) } - _ => Err(PythonMutationError::Unsupported), + _ => Err(ManagerError::MutationUnsupported.into()), } } @@ -115,7 +120,7 @@ pub(super) async fn delete_python_provider( "message": "CyberArk Conjur does not support direct secret deletion. Use policy updates to remove variables.", })) } - _ => Err(PythonMutationError::Unsupported), + _ => Err(ManagerError::MutationUnsupported.into()), } } @@ -138,7 +143,7 @@ pub(super) async fn rotate_python_provider( .flatten() .is_none() { - return Err(PythonMutationError::CurrentMissing(current_name.to_owned())); + return Err(ManagerError::CurrentMissing(current_name.to_owned()).into()); } client .write_with_retry( @@ -157,16 +162,16 @@ pub(super) async fn rotate_python_provider( .await .ok() .flatten() - .ok_or_else(|| PythonMutationError::ReplacementMissing(new_name.to_owned()))?; + .ok_or_else(|| ManagerError::ReplacementMissing(new_name.to_owned()))?; if actual != *value { - return Err(PythonMutationError::ReplacementMismatch); + return Err(ManagerError::ReplacementMismatch.into()); } if current_name != new_name { client.invalidate_cached_secret(current_name).await; } Ok(write_success(new_name)) } - _ => Err(PythonMutationError::Unsupported), + _ => Err(ManagerError::MutationUnsupported.into()), } } fn write_success(name: &str) -> serde_json::Value { diff --git a/litellm-rust/crates/python-bridge/src/secrets/provider.rs b/litellm-rust/crates/python-bridge/src/secrets/provider.rs index 5f4cd193fcf..ff10f596311 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/provider.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/provider.rs @@ -6,7 +6,9 @@ use litellm_secrets_types::{ AwsOperationContext, CyberarkOperationContext, GoogleOperationContext, HashicorpOperationContext, SecretOperationContext, }; -use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; +use pyo3::{prelude::*, types::PyDict}; + +use super::ManagerError; pub(super) fn read_request( system: KeyManagementSystem, @@ -39,9 +41,7 @@ pub(super) fn read_request( SecretOperationContext::Google(GoogleOperationContext::default()) } _ => { - return Err(PyValueError::new_err( - "secret manager does not support provider reads", - )); + return Err(ManagerError::ProviderReads.into()); } }; Ok(PythonReadRequest { @@ -89,9 +89,7 @@ fn aws_context( fn read_timeout(value: Option<&Bound<'_, PyAny>>) -> PyResult> { read_timeout_seconds(value)? - .map(|value| { - Duration::try_from_secs_f64(value).map_err(|_| PyValueError::new_err("invalid timeout")) - }) + .map(|value| Ok(Duration::try_from_secs_f64(value).map_err(ManagerError::Timeout)?)) .transpose() } diff --git a/litellm-rust/crates/python-bridge/src/secrets/runtime.rs b/litellm-rust/crates/python-bridge/src/secrets/runtime.rs index 72ba91ded16..b956ec297b6 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/runtime.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/runtime.rs @@ -7,12 +7,9 @@ use litellm_secrets::{ read_secret_from_python_manager, }; use litellm_secrets_types::PythonSecretRead; -use pyo3::{ - exceptions::{PyAttributeError, PyRuntimeError, PyValueError}, - prelude::*, - types::PyDict, -}; +use pyo3::{exceptions::PyAttributeError, prelude::*, types::PyDict}; +use super::ManagerError; use crate::execution::{run_async_value, run_sync_value}; const SECRET_MANAGER_MODULE: &str = "litellm.rust_bridge.host.secret_manager"; @@ -35,9 +32,7 @@ pub(crate) struct NativeSecretManager { impl NativeSecretManager { pub(super) fn backend(&self) -> PyResult { if self.pid != std::process::id() { - return Err(PyRuntimeError::new_err( - "native secret manager must be recreated after fork", - )); + return Err(ManagerError::Forked.into()); } Ok(self.backend.clone()) } @@ -60,7 +55,7 @@ impl NativeSecretManager { enterprise_enabled, ) .await - .map_err(|error| PyValueError::new_err(error.to_string())) + .map_err(|error| ManagerError::Secret(error).into()) })?; Ok(Self { backend, @@ -82,7 +77,7 @@ impl NativeSecretManager { enterprise_enabled: bool, ) -> PyResult { let system = serde_json::from_value(serde_json::Value::String(system.to_owned())) - .map_err(|_| PyValueError::new_err("unknown secret manager system"))?; + .map_err(ManagerError::UnknownSystem)?; let settings = parse_settings(settings)?; Self::build( py, @@ -146,7 +141,7 @@ impl NativeSecretManager { .collect::>>()?; let settings = from_py::>(&config.getattr("settings")?) - .map_err(|_| PyValueError::new_err("invalid secret manager settings"))?; + .map_err(ManagerError::settings)?; let attributes = config .getattr("settings_attributes")? .extract::>()?; @@ -161,11 +156,11 @@ impl NativeSecretManager { system: serde_json::from_value(serde_json::Value::String( config.getattr("system")?.extract()?, )) - .map_err(|_| PyValueError::new_err("unknown secret manager system"))?, + .map_err(ManagerError::UnknownSystem)?, settings: serde_json::from_value(serde_json::Value::Object( settings.into_iter().chain(setting_overrides).collect(), )) - .map_err(|_| PyValueError::new_err("invalid secret manager settings"))?, + .map_err(ManagerError::settings)?, environment: captured .into_iter() .filter(|(key, _)| !environment_attributes.contains_key(key)) @@ -190,12 +185,8 @@ impl NativeSecretManager { } #[getter] - fn system(&self) -> String { - serde_json::to_value(self.configuration.system) - .expect("serializable system") - .as_str() - .expect("string system") - .to_owned() + fn system(&self) -> &'static str { + self.configuration.system.into() } #[pyo3(signature = (name, settings=None))] @@ -238,7 +229,7 @@ impl NativeSecretManager { run_sync_value(py, async move { super::operations::read_python_provider(&backend, &request, &ProcessEnvironment) .await - .map_err(|error| PyValueError::new_err(error.to_string())) + .map_err(|error| ManagerError::Secret(error).into()) .and_then(|value| python_secret_value(value, &request.secret_name)) }) } @@ -264,7 +255,7 @@ impl NativeSecretManager { run_async_value(py, async move { super::operations::read_python_provider(&backend, &request, &ProcessEnvironment) .await - .map_err(|error| PyValueError::new_err(error.to_string())) + .map_err(|error| ManagerError::Secret(error).into()) .and_then(|value| python_secret_value(value, &request.secret_name)) }) } @@ -438,8 +429,8 @@ fn cached( fn parse_settings(value: Option<&Bound<'_, PyAny>>) -> PyResult { value .map(|value| { - serde_json::from_value(from_py::(value)?) - .map_err(|_| PyValueError::new_err("invalid secret manager settings")) + Ok(serde_json::from_value(from_py::(value)?) + .map_err(ManagerError::settings)?) }) .transpose() .map(Option::unwrap_or_default) @@ -466,11 +457,10 @@ fn python_read_error( name: &str, error: litellm_secrets::Error, ) -> PyErr { - let message = match (system, error) { + match (system, error) { (KeyManagementSystem::Cyberark, litellm_secrets::Error::ManagedSecretMissing) => { - format!("No secret found in CyberArk Secret Manager for {name}") + ManagerError::CyberarkSecretMissing(name.to_owned()).into() } - (_, error) => error.to_string(), - }; - PyValueError::new_err(message) + (_, error) => ManagerError::Secret(error).into(), + } } diff --git a/litellm-rust/crates/python-bridge/src/secrets/vault.rs b/litellm-rust/crates/python-bridge/src/secrets/vault.rs index 79281662848..30e3bd2c413 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/vault.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/vault.rs @@ -2,8 +2,11 @@ mod operation; pub(super) use operation::{Failure, FailureKind, FailureStage, delete, rotate, write}; -use litellm_core_utils::exception_mapping_utils::{TimeoutPhase, timeout_message}; -use litellm_secrets::hashicorp::{Error, RawOperationError}; +use litellm_core_utils::exception_mapping_utils::{ + TimeoutPhase, timeout_message as transport_timeout_message, +}; +use litellm_error::Report; +use litellm_secrets::hashicorp::RawOperationError; use pyo3::prelude::*; use super::mutation::{error_value, http_message, json_value}; @@ -16,40 +19,39 @@ pub(super) fn failure_value( failure: Failure, context: &ErrorContext, ) -> PyResult> { - let message = match *failure.kind { - FailureKind::Native(RawOperationError::Http { - method, - url, - status, - body, - }) => match failure.stage { - FailureStage::Current(name) if status == 404 => { - format!("Current secret {name} not found") + let message = match (&*failure.kind, failure.stage) { + (FailureKind::Native(error), stage) if let Some(response) = error.http_failure() => { + match stage { + FailureStage::Current(name) if response.status == 404 => { + format!("Current secret {name} not found") + } + FailureStage::Replacement(name) if response.status == 404 => { + format!("Failed to verify new secret {name}") + } + FailureStage::Current(_) => format!( + "HTTP error occurred while checking current secret: {}", + response_text(py, &response.body)? + ), + FailureStage::Replacement(_) => format!( + "HTTP error occurred while verifying new secret: {}", + response_text(py, &response.body)? + ), + FailureStage::Mutation => { + http_message(py, &response.method, &response.url, response.status)? + } } - FailureStage::Replacement(name) if status == 404 => { - format!("Failed to verify new secret {name}") - } - FailureStage::Current(_) => format!( - "HTTP error occurred while checking current secret: {}", - response_text(py, &body)? - ), - FailureStage::Replacement(_) => format!( - "HTTP error occurred while verifying new secret: {}", - response_text(py, &body)? - ), - FailureStage::Mutation => http_message(py, &method, &url, status)?, - }, - FailureKind::ValueMismatch { expected, actual } => { - let actual = json_value(py, &actual)?; + } + (FailureKind::ValueMismatch { expected, actual }, _) => { + let actual = json_value(py, actual)?; format!( "New secret value mismatch. Expected: {}, Got: {}", expected.expose(), actual.bind(py).str()? ) } - kind => { + (kind, stage) => { let message = cause_message(py, kind, context)?; - match failure.stage { + match stage { FailureStage::Current(_) => { format!("Error checking current secret: {message}") } @@ -63,86 +65,91 @@ pub(super) fn failure_value( error_value(py, message) } -fn cause_message(py: Python<'_>, kind: FailureKind, context: &ErrorContext) -> PyResult { +fn cause_message(py: Python<'_>, kind: &FailureKind, context: &ErrorContext) -> PyResult { Ok(match kind { - FailureKind::Native(RawOperationError::Local(error)) => error.to_string(), + FailureKind::Native(error) => native_message(py, error, context)?, FailureKind::UnsafeName(name) => { format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?) } - FailureKind::Native(RawOperationError::Timeout { method, elapsed }) => { - if method == "POST" { - let kwargs = pyo3::types::PyDict::new(py); - kwargs.set_item( - "message", - timeout_message( - TimeoutPhase::Read, - context.timeout, - Some(elapsed.as_secs_f64()), - ), - )?; - kwargs.set_item("model", "default-model-name")?; - kwargs.set_item("llm_provider", "litellm-httpx-handler")?; - py.import("litellm.exceptions")? - .getattr("Timeout")? - .call((), Some(&kwargs))? - .str()? - .extract()? - } else if context.aiohttp { - "Timeout on reading data from socket".to_owned() - } else { - String::new() - } - } - FailureKind::Native(RawOperationError::Transport(source)) => { - if let Some(error) = request_error(&source) { - if error.is_timeout() { - String::new() - } else if error.is_connect() { - "All connection attempts failed".to_owned() - } else { - "HashiCorp Vault request failed".to_owned() - } - } else { - "HashiCorp Vault request failed".to_owned() - } - } FailureKind::MissingGet(value) => { - let value = json_value(py, &value)?; + let value = json_value(py, value)?; match value.bind(py).getattr("get") { Err(error) => error.value(py).str()?.extract()?, Ok(_) => "HashiCorp Vault response payload is malformed".to_owned(), } } - FailureKind::Json(body) => match json_value(py, &body) { + FailureKind::Json(body) => match json_value(py, body) { Err(error) => error.value(py).str()?.extract()?, Ok(_) => "HashiCorp Vault response payload is malformed".to_owned(), }, - FailureKind::Native(RawOperationError::Authentication { - source, - url, - certificate, - }) => { - let message = match source { - Error::LoginStatus { status } => http_message(py, "POST", &url, status)?, - error => error.to_string(), - }; - let mechanism = if certificate { "TLS cert" } else { "AppRole" }; - format!("Could not authenticate to Vault via {mechanism}: {message}") - } - FailureKind::Native(RawOperationError::Http { - method, - url, - status, - .. - }) => http_message(py, &method, &url, status)?, FailureKind::ValueMismatch { .. } => "New secret value mismatch".to_owned(), }) } -fn request_error<'a>(error: &'a (dyn std::error::Error + 'static)) -> Option<&'a reqwest::Error> { - error - .downcast_ref::() - .or_else(|| error.source().and_then(request_error)) +fn native_message( + py: Python<'_>, + error: &RawOperationError, + context: &ErrorContext, +) -> PyResult { + if let Some(error) = error.local() { + return Ok(Report::new(error).to_string()); + } + if let Some((method, elapsed)) = error.timeout() { + return timeout_message(py, method, elapsed, context); + } + if let Some(response) = error.http_failure() { + return http_message(py, &response.method, &response.url, response.status); + } + if let Some(login) = error.authentication() { + let message = match login.source.login_status() { + Some(status) => http_message(py, "POST", login.url, status)?, + None => Report::new(login.source).to_string(), + }; + let mechanism = if login.certificate { + "TLS cert" + } else { + "AppRole" + }; + return Ok(format!( + "Could not authenticate to Vault via {mechanism}: {message}" + )); + } + Ok(match error.transport() { + Some(error) if error.status() == Some(408) => String::new(), + Some(error) if error.is_connect() => "All connection attempts failed".to_owned(), + Some(_) | None => "HashiCorp Vault request failed".to_owned(), + }) +} + +fn timeout_message( + py: Python<'_>, + method: &str, + elapsed: std::time::Duration, + context: &ErrorContext, +) -> PyResult { + if method != "POST" { + return Ok(if context.aiohttp { + "Timeout on reading data from socket".to_owned() + } else { + String::new() + }); + } + let kwargs = pyo3::types::PyDict::new(py); + kwargs.set_item( + "message", + transport_timeout_message( + TimeoutPhase::Read, + context.timeout, + Some(elapsed.as_secs_f64()), + ), + )?; + kwargs.set_item("model", "default-model-name")?; + kwargs.set_item("llm_provider", "litellm-httpx-handler")?; + py.import("litellm.exceptions")? + .getattr("Timeout")? + .call((), Some(&kwargs))? + .str()? + .extract() } fn response_text(py: Python<'_>, body: &[u8]) -> PyResult { diff --git a/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs b/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs index cf56beb5efe..41658f0baf8 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs @@ -2,7 +2,7 @@ use std::collections::HashMap; use litellm_secrets::{ SecretValue, - hashicorp::{Error, HashicorpVault, RawOperationError}, + hashicorp::{HashicorpVault, RawOperationError}, }; use litellm_secrets_types::{HashicorpOperationContext, SecretWriteContext}; use serde_json::value::RawValue; @@ -53,12 +53,10 @@ impl Failure { } fn native_failure(name: &str, error: RawOperationError) -> Failure { - match error { - RawOperationError::Local(Error::InvalidSecretName(_)) => { - FailureKind::UnsafeName(name.to_owned()).into() - } - error => FailureKind::Native(error).into(), + if error.is_unsafe_secret_name() { + return FailureKind::UnsafeName(name.to_owned()).into(); } + FailureKind::Native(error).into() } pub(crate) async fn write( diff --git a/litellm-rust/crates/python-bridge/src/tokenizer.rs b/litellm-rust/crates/python-bridge/src/tokenizer.rs index df219d55eb6..2026b2fdc42 100644 --- a/litellm-rust/crates/python-bridge/src/tokenizer.rs +++ b/litellm-rust/crates/python-bridge/src/tokenizer.rs @@ -8,16 +8,18 @@ use std::sync::Arc; #[cfg(feature = "fast")] use std::sync::OnceLock; +use litellm_error::Report; use litellm_host_python::{enter_native, release_gil}; #[cfg(feature = "fast")] use litellm_token_counter::fast::{FastCounter, FastTokenizer}; use litellm_token_counter::{Error, TextCodec}; use pyo3::{exceptions::PyUnicodeEncodeError, prelude::*, types::PyString}; -#[cfg(any(feature = "tiktoken", feature = "huggingface"))] -use pyo3::exceptions::PyValueError; #[cfg(feature = "huggingface")] -use pyo3::{exceptions::PyIOError, types::PyDict}; +use pyo3::{ + exceptions::{PyIOError, PyTypeError, PyValueError}, + types::PyDict, +}; #[cfg(feature = "tiktoken")] use pyo3::{ exceptions::{PyKeyError, PyRuntimeError}, @@ -51,6 +53,70 @@ pub(crate) fn load_tiktoken(py: Python<'_>, encoding: &str) -> PyResult for PyErr { + fn from(error: TokenizerError) -> Self { + let message = Report::new(&error).to_string(); + match error { + #[cfg(not(all(feature = "tiktoken", feature = "huggingface")))] + TokenizerError::MissingFeature(_) => RustBridgeDeclined::new_err(message), + #[cfg(all(feature = "tiktoken", feature = "huggingface"))] + TokenizerError::NotTiktoken | TokenizerError::NotHuggingFace => { + PyValueError::new_err(message) + } + #[cfg(feature = "tiktoken")] + TokenizerError::NoVocabulary | TokenizerError::EncodeSpecial(_) => { + PyRuntimeError::new_err(message) + } + #[cfg(feature = "tiktoken")] + TokenizerError::DecodeBytes(_) => PyKeyError::new_err(message), + #[cfg(feature = "huggingface")] + TokenizerError::Direction { .. } | TokenizerError::EncodingJson(_) => { + PyValueError::new_err(message) + } + #[cfg(feature = "huggingface")] + TokenizerError::PretokenizedMismatch => PyTypeError::new_err(message), + #[cfg(feature = "huggingface")] + TokenizerError::Download(_) => PyIOError::new_err(message), + } + } +} + pub(crate) enum Codec { #[cfg(feature = "tiktoken")] Tiktoken(TiktokenTokenizer), @@ -100,9 +166,7 @@ impl Tokenizer { #[cfg(not(feature = "tiktoken"))] { let _ = (py, encoding); - Err(RustBridgeDeclined::new_err( - "tokenizer backend requires the tiktoken feature", - )) + Err(TokenizerError::MissingFeature("tiktoken").into()) } } @@ -118,9 +182,7 @@ impl Tokenizer { #[cfg(not(feature = "huggingface"))] { let _ = (py, tokenizer_json); - Err(RustBridgeDeclined::new_err( - "tokenizer backend requires the huggingface feature", - )) + Err(TokenizerError::MissingFeature("huggingface").into()) } } @@ -144,16 +206,14 @@ impl Tokenizer { .getattr("hf_hub_download")? .call((), Some(&kwargs))? .extract()?; - let json = - release_gil(py, || std::fs::read_to_string(path)).map_err(PyIOError::new_err)?; + let json = release_gil(py, || std::fs::read_to_string(path)) + .map_err(TokenizerError::Download)?; Self::from_json(py, &json) } #[cfg(not(feature = "huggingface"))] { let _ = (py, identifier, revision, token); - Err(RustBridgeDeclined::new_err( - "tokenizer backend requires the huggingface feature", - )) + Err(TokenizerError::MissingFeature("huggingface").into()) } } @@ -199,7 +259,7 @@ impl Tokenizer { let tokenizer = self.tiktoken()?; let text = self.text(text)?; release_gil(py, || tokenizer.encode_special(&text, &allowed)) - .map_err(PyRuntimeError::new_err) + .map_err(|error| TokenizerError::EncodeSpecial(error.into()).into()) } /// tiktoken's `encode_with_unstable`: `(stable_tokens, completions)`. @@ -258,8 +318,8 @@ impl Tokenizer { fn decode_bytes<'py>(&self, py: Python<'py>, ids: Vec) -> PyResult> { enter_native()?; let tokenizer = self.tiktoken()?; - let bytes = - release_gil(py, || tokenizer.decode_bytes(&ids)).map_err(PyKeyError::new_err)?; + let bytes = release_gil(py, || tokenizer.decode_bytes(&ids)) + .map_err(|error| TokenizerError::DecodeBytes(error.into()))?; Ok(PyBytes::new(py, &bytes)) } @@ -480,15 +540,16 @@ impl Tokenizer { match *self.inner { Codec::Tiktoken(ref tokenizer) => Ok(tokenizer), #[cfg(feature = "huggingface")] - Codec::HuggingFace(_) => Err(PyValueError::new_err("requires a tiktoken encoding")), + Codec::HuggingFace(_) => Err(TokenizerError::NotTiktoken.into()), } } #[cfg(feature = "tiktoken")] fn vocabulary(&self) -> PyResult<&Vocabulary> { - self.tiktoken()?.vocabulary().ok_or_else(|| { - PyRuntimeError::new_err("this encoding was built without its vocabulary") - }) + Ok(self + .tiktoken()? + .vocabulary() + .ok_or(TokenizerError::NoVocabulary)?) } #[cfg(feature = "huggingface")] @@ -496,7 +557,7 @@ impl Tokenizer { match *self.inner { Codec::HuggingFace(ref tokenizer) => Ok(tokenizer), #[cfg(feature = "tiktoken")] - Codec::Tiktoken(_) => Err(PyValueError::new_err("requires a Hugging Face tokenizer")), + Codec::Tiktoken(_) => Err(TokenizerError::NotHuggingFace.into()), } } } @@ -530,21 +591,21 @@ impl Sequence { match (self, is_pretokenized) { (Self::Text(text), false) => Ok(text.into()), (Self::Words(words), true) => Ok(words.into()), - _ => Err(pyo3::exceptions::PyTypeError::new_err( - "input must match is_pretokenized", - )), + _ => Err(TokenizerError::PretokenizedMismatch.into()), } } } #[cfg(feature = "huggingface")] -fn direction(value: &str, left: T, right: T, what: &str) -> PyResult { +fn direction(value: &str, left: T, right: T, what: &'static str) -> PyResult { match value { "left" => Ok(left), "right" => Ok(right), - other => Err(PyValueError::new_err(format!( - "invalid {what} direction {other:?}: expected 'left' or 'right'" - ))), + other => Err(TokenizerError::Direction { + what, + value: other.to_owned(), + } + .into()), } } @@ -564,7 +625,7 @@ impl HuggingFaceEncoding { fn new(json: Option<&str>) -> PyResult { let inner = match json { Some(json) => encoding_from_json(json) - .map_err(|error| PyValueError::new_err(error.to_string()))?, + .map_err(|error| TokenizerError::EncodingJson(error.into()))?, None => Encoding::default(), }; Ok(Self { inner }) @@ -586,7 +647,7 @@ impl HuggingFaceEncoding { py: Python<'py>, ) -> PyResult<(Bound<'py, pyo3::types::PyType>, (String,))> { let json = encoding_to_json(&self.inner) - .map_err(|error| PyValueError::new_err(error.to_string()))?; + .map_err(|error| TokenizerError::EncodingJson(error.into()))?; Ok((py.get_type::(), (json,))) } diff --git a/litellm-rust/crates/python-compat/src/error.rs b/litellm-rust/crates/python-compat/src/error.rs index 43990eb7602..6547907056e 100644 --- a/litellm-rust/crates/python-compat/src/error.rs +++ b/litellm-rust/crates/python-compat/src/error.rs @@ -10,15 +10,17 @@ pub enum Error { Unhashable(&'static str), #[error("value nests deeper than {MAX_DEPTH} levels")] TooDeep, - #[error("invalid pickle: {0}")] - InvalidPickle(String), + #[error("invalid pickle")] + InvalidPickle(#[source] serde_pickle::Error), + #[error("could not write pickle")] + PickleWrite(#[source] serde_pickle::Error), #[error("Object of type {0} is not JSON serializable")] NotJsonSerializable(&'static str), #[error("keys must be str, int, float, bool or None, not {0}")] InvalidJsonKey(&'static str), #[error("Out of range float values are not JSON compliant")] NonFiniteFloat, - #[error("integer does not fit in the JSON number range")] + #[error("integer does not fit in 64 bits")] IntegerOutOfRange, #[error("Object of type {0} cannot be pickled as plain data")] NotPicklable(&'static str), diff --git a/litellm-rust/crates/python-compat/src/literal.rs b/litellm-rust/crates/python-compat/src/literal.rs index 12e9036572f..336c6dfcb2e 100644 --- a/litellm-rust/crates/python-compat/src/literal.rs +++ b/litellm-rust/crates/python-compat/src/literal.rs @@ -58,13 +58,6 @@ impl Term { kind: Kind::Other, } } - - fn is_number(&self) -> bool { - matches!( - self.value, - Value::Int(_) | Value::Float(_) | Value::Complex { .. } - ) - } } struct Parser<'a> { @@ -204,18 +197,25 @@ impl Parser<'_> { self.enter()?; let operand = self.unary()?; self.depth -= 1; - if operand.kind != Kind::Constant || !operand.is_number() { + if operand.kind != Kind::Constant { return Err(Error::InvalidLiteral(self.offset + at)); } - let value = if !negative { - operand.value - } else { - match operand.value { - Value::Int(value) => Value::Int(-value), - Value::Float(value) => Value::Float(-value), - Value::Complex { re, im } => Value::Complex { re: -re, im: -im }, - _ => unreachable!("checked numeric above"), - } + let value = match (operand.value, negative) { + (value @ (Value::Int(_) | Value::Float(_) | Value::Complex { .. }), false) => value, + (Value::Int(value), true) => Value::Int(-value), + (Value::Float(value), true) => Value::Float(-value), + (Value::Complex { re, im }, true) => Value::Complex { re: -re, im: -im }, + ( + Value::None + | Value::Bool(_) + | Value::Str(_) + | Value::Bytes(_) + | Value::List(_) + | Value::Tuple(_) + | Value::Dict(_) + | Value::Set(_), + _, + ) => return Err(Error::InvalidLiteral(self.offset + at)), }; Ok(Term { value, diff --git a/litellm-rust/crates/python-compat/src/pickle.rs b/litellm-rust/crates/python-compat/src/pickle.rs index bff7442dc6b..dd7ac6a0e4f 100644 --- a/litellm-rust/crates/python-compat/src/pickle.rs +++ b/litellm-rust/crates/python-compat/src/pickle.rs @@ -22,10 +22,8 @@ pub fn loads(data: &[u8]) -> Result { let mut deserializer = serde_pickle::Deserializer::new(data, DeOptions::new()); let value = Seed { depth: 0 } .deserialize(&mut deserializer) - .map_err(|error| Error::InvalidPickle(error.to_string()))?; - deserializer - .end() - .map_err(|error| Error::InvalidPickle(error.to_string()))?; + .map_err(Error::InvalidPickle)?; + deserializer.end().map_err(Error::InvalidPickle)?; Ok(value) } @@ -33,8 +31,7 @@ pub fn loads(data: &[u8]) -> Result { /// default. Sets and complex numbers are rejected rather than silently changing type. pub fn dumps(value: &Value) -> Result, Error> { check_picklable(value, 0)?; - serde_pickle::to_vec(&Pickled(value), SerOptions::new()) - .map_err(|error| Error::InvalidPickle(error.to_string())) + serde_pickle::to_vec(&Pickled(value), SerOptions::new()).map_err(Error::PickleWrite) } fn check_picklable(value: &Value, depth: usize) -> Result<(), Error> { @@ -66,7 +63,7 @@ impl Serialize for Pickled<'_> { Value::Bool(value) => serializer.serialize_bool(*value), Value::Int(value) => { let value = i64::try_from(value) - .map_err(|_| serde::ser::Error::custom("integer out of i64 range"))?; + .map_err(|_| serde::ser::Error::custom(Error::IntegerOutOfRange))?; serializer.serialize_i64(value) } Value::Float(value) => serializer.serialize_f64(*value), @@ -94,7 +91,7 @@ impl Serialize for Pickled<'_> { map.end() } value @ (Value::Set(_) | Value::Complex { .. }) => Err(serde::ser::Error::custom( - format!("{} cannot be pickled as plain data", value.type_name()), + Error::NotPicklable(value.type_name()), )), } } diff --git a/litellm-rust/crates/secrets-aws/src/auth.rs b/litellm-rust/crates/secrets-aws/src/auth.rs index 06019cc3c9d..66f4f51bda9 100644 --- a/litellm-rust/crates/secrets-aws/src/auth.rs +++ b/litellm-rust/crates/secrets-aws/src/auth.rs @@ -79,9 +79,7 @@ impl ProvideCredentials for Credentials { self.auth .resolve_credentials(self.config.clone(), &|name| self.environment.get(name)) .await - .map_err(|_| { - CredentialsError::provider_error("secret manager authentication failed") - }) + .map_err(CredentialsError::provider_error) }) } } diff --git a/litellm-rust/crates/secrets-aws/src/error.rs b/litellm-rust/crates/secrets-aws/src/error.rs index c8ca671e9b5..c537da204dd 100644 --- a/litellm-rust/crates/secrets-aws/src/error.rs +++ b/litellm-rust/crates/secrets-aws/src/error.rs @@ -11,7 +11,7 @@ pub enum Error { #[error("KMS response has no plaintext")] MissingPlaintext, #[error("AWS request timed out")] - Timeout, + Timeout(#[source] #[redact] Box>), #[error("AWS KMS decrypt failed")] Decrypt(#[from] #[redact] Box>), #[error("AWS Secrets Manager read failed")] @@ -33,7 +33,7 @@ pub enum Error { #[error("AWS Secrets Manager response has no string payload")] MissingString, #[error("primary secret is not a JSON object")] - PrimarySecret, + PrimarySecret(#[source] #[redact] Option), #[error(transparent)] Operation(#[from] litellm_secrets_types::Error), } diff --git a/litellm-rust/crates/secrets-aws/src/secret_manager/read.rs b/litellm-rust/crates/secrets-aws/src/secret_manager/read.rs index b7583f9785f..d7f10a9847e 100644 --- a/litellm-rust/crates/secrets-aws/src/secret_manager/read.rs +++ b/litellm-rust/crates/secrets-aws/src/secret_manager/read.rs @@ -72,7 +72,7 @@ impl AwsSecretsManagerV2 { } let client = self.client_for_context(context)?; let value = match Self::read_with_client(&client, name, ReadPolicy::Python).await { - Err(Error::Read(_) | Error::MissingString | Error::Timeout) => None, + Err(Error::Read(_) | Error::MissingString | Error::Timeout(_)) => None, result => result?, }; Ok(PythonSecretRead::Value(value.map(Secret::String))) @@ -124,9 +124,10 @@ impl AwsSecretsManagerV2 { Self::read_with_client(&self.client, name, policy).await, policy, ) { - (Err(Error::Read(_) | Error::MissingString | Error::Timeout), ReadPolicy::Python) => { - Ok(None) - } + ( + Err(Error::Read(_) | Error::MissingString | Error::Timeout(_)), + ReadPolicy::Python, + ) => Ok(None), (result, _) => result, } } @@ -173,7 +174,7 @@ impl AwsSecretsManagerV2 { aws_sdk_secretsmanager::error::SdkError::TimeoutError(_) ) || matches!(&error, aws_sdk_secretsmanager::error::SdkError::DispatchFailure(failure) if failure.is_timeout()) => { - Err(Error::Timeout) + Err(Error::Timeout(Box::new(error))) } Err(error) if error @@ -216,9 +217,9 @@ fn resolve_payload(payload: PythonSecretRead, name: &str) -> Result Ok(value), PythonSecretRead::PrimaryJson(document) => { - let object: Value = - serde_json::from_str(document.expose()).map_err(|_| Error::PrimarySecret)?; - let object = object.as_object().ok_or(Error::PrimarySecret)?; + let object: Value = serde_json::from_str(document.expose()) + .map_err(|error| Error::PrimarySecret(Some(error)))?; + let object = object.as_object().ok_or(Error::PrimarySecret(None))?; Ok(object.get(name).cloned().map(Secret::from_json)) } } diff --git a/litellm-rust/crates/secrets-aws/tests/secret_manager/reads.rs b/litellm-rust/crates/secrets-aws/tests/secret_manager/reads.rs index 3c39924f09f..30aec467543 100644 --- a/litellm-rust/crates/secrets-aws/tests/secret_manager/reads.rs +++ b/litellm-rust/crates/secrets-aws/tests/secret_manager/reads.rs @@ -88,7 +88,7 @@ async fn failed_read_returns_none_but_invalid_primary_json_is_an_error( manager .read_secret_for_resolver("KEY", Some("invalid"), &|_: &str| None) .await, - Err(Error::PrimarySecret) + Err(Error::PrimarySecret(_)) )); assert!(matches!( manager.async_read_secret("no-string").await, @@ -143,7 +143,7 @@ async fn trait_read_applies_the_aws_operation_timeout() { }; assert!(matches!( BaseSecretManager::async_read_secret(&loaded_manager(&server), "key", &context).await, - Err(Error::Timeout) + Err(Error::Timeout(_)) )); } @@ -169,7 +169,7 @@ async fn read_timeout_is_an_error_and_cannot_be_mistaken_for_missing() { let manager = AwsSecretsManagerV2::new(Client::from_conf(config), Default::default()); assert!(matches!( manager.async_read_secret("key").await, - Err(Error::Timeout) + Err(Error::Timeout(_)) )); } diff --git a/litellm-rust/crates/secrets-azure/Cargo.toml b/litellm-rust/crates/secrets-azure/Cargo.toml index 7ec03fb98da..2aff6129c37 100644 --- a/litellm-rust/crates/secrets-azure/Cargo.toml +++ b/litellm-rust/crates/secrets-azure/Cargo.toml @@ -15,6 +15,7 @@ litellm-core-utils.workspace = true reqwest.workspace = true serde.workspace = true thiserror.workspace = true +url.workspace = true veil.workspace = true percent-encoding = "2.3" diff --git a/litellm-rust/crates/secrets-azure/src/error.rs b/litellm-rust/crates/secrets-azure/src/error.rs index 2862f08a56c..097f42cb89c 100644 --- a/litellm-rust/crates/secrets-azure/src/error.rs +++ b/litellm-rust/crates/secrets-azure/src/error.rs @@ -7,7 +7,7 @@ pub enum Error { #[error("{0} environment variable is missing")] MissingEnvironment(&'static str), #[error("AZURE_KEY_VAULT_URI is not a valid https vault URL")] - VaultUri, + VaultUri(#[source] Option), #[error("Azure Key Vault credentials are not configured")] MissingCredentials, #[error(transparent)] @@ -20,10 +20,26 @@ pub enum Error { Http( #[source] #[redact] - reqwest::Error, + litellm_http::Error, ), #[error("Azure Key Vault returned HTTP {0}")] Status(u16), #[error("Azure Key Vault response is missing the secret value")] MissingValue, } + +impl Error { + pub fn is_missing_value(&self) -> bool { + match self { + Self::MissingValue => true, + Self::Operation(_) + | Self::Timeout + | Self::MissingEnvironment(_) + | Self::VaultUri(_) + | Self::MissingCredentials + | Self::Auth(_) + | Self::Http(_) + | Self::Status(_) => false, + } + } +} diff --git a/litellm-rust/crates/secrets-azure/src/key_vault.rs b/litellm-rust/crates/secrets-azure/src/key_vault.rs index 095c451927c..d97b7a22cfc 100644 --- a/litellm-rust/crates/secrets-azure/src/key_vault.rs +++ b/litellm-rust/crates/secrets-azure/src/key_vault.rs @@ -38,7 +38,7 @@ impl AzureKeyVault { environment: Arc, ) -> Result { if vault.host_str().is_none() { - return Err(Error::VaultUri); + return Err(Error::VaultUri(None)); } let inputs = AzureAuthInputs { azure_scope: ConfigValue::Value(Sourced::new( @@ -64,9 +64,9 @@ impl AzureKeyVault { let value = environment .get(AZURE_KEY_VAULT_URI) .ok_or(Error::MissingEnvironment(AZURE_KEY_VAULT_URI))?; - let vault = reqwest::Url::parse(&value).map_err(|_| Error::VaultUri)?; + let vault = reqwest::Url::parse(&value).map_err(|error| Error::VaultUri(Some(error)))?; if vault.scheme() != "https" || vault.host_str().is_none() { - return Err(Error::VaultUri); + return Err(Error::VaultUri(None)); } Self::with_client(client, vault, environment) } @@ -95,7 +95,7 @@ impl AzureKeyVault { let url = self .vault .join(&format!("secrets/{encoded_name}?api-version={API_VERSION}")) - .map_err(|_| Error::VaultUri)?; + .map_err(|error| Error::VaultUri(Some(error)))?; let response = self .client .get(url) @@ -103,14 +103,17 @@ impl AzureKeyVault { .header(reqwest::header::ACCEPT, "application/json") .send() .await - .map_err(Error::Http)?; + .map_err(|error| Error::Http(error.into()))?; if response.status() == reqwest::StatusCode::NOT_FOUND { return Ok(None); } if response.status() != reqwest::StatusCode::OK { return Err(Error::Status(response.status().as_u16())); } - let payload: SecretResponse = response.json().await.map_err(Error::Http)?; + let payload: SecretResponse = response + .json() + .await + .map_err(|error| Error::Http(error.into()))?; let value = payload.value.ok_or(Error::MissingValue)?; Ok(Some(SecretValue::new(value))) } diff --git a/litellm-rust/crates/secrets-azure/tests/key_vault.rs b/litellm-rust/crates/secrets-azure/tests/key_vault.rs index 05f62d25dce..a0ba9e6ddf8 100644 --- a/litellm-rust/crates/secrets-azure/tests/key_vault.rs +++ b/litellm-rust/crates/secrets-azure/tests/key_vault.rs @@ -122,13 +122,14 @@ async fn missing_value_is_an_error() { } #[rstest] -#[case::missing(None, true)] -#[case::http(Some("http://vault.example"), false)] -#[case::relative(Some("vault.example"), false)] -#[case::malformed(Some("://"), false)] +#[case::missing(None, true, false)] +#[case::http(Some("http://vault.example"), false, false)] +#[case::relative(Some("vault.example"), false, true)] +#[case::malformed(Some("://"), false, true)] fn new_validates_vault_environment( #[case] uri: Option<&'static str>, #[case] missing_environment: bool, + #[case] unparsable: bool, ) { let result = AzureKeyVault::new( litellm_http::Client::for_test(), @@ -145,7 +146,14 @@ fn new_validates_vault_environment( Err(Error::MissingEnvironment("AZURE_KEY_VAULT_URI")) )); } else { - assert!(matches!(result, Err(Error::VaultUri))); + let error = result.err().expect("invalid vault URI"); + assert!(matches!(error, Error::VaultUri(_))); + assert_eq!( + std::error::Error::source(&error) + .and_then(|cause| cause.downcast_ref::()) + .is_some(), + unparsable + ); } } diff --git a/litellm-rust/crates/secrets-cyberark/Cargo.toml b/litellm-rust/crates/secrets-cyberark/Cargo.toml index f630d5857d8..2ae2c1ea5c3 100644 --- a/litellm-rust/crates/secrets-cyberark/Cargo.toml +++ b/litellm-rust/crates/secrets-cyberark/Cargo.toml @@ -14,6 +14,7 @@ moka.workspace = true reqwest.workspace = true serde_json.workspace = true thiserror.workspace = true +url.workspace = true veil.workspace = true litellm-tracing.workspace = true percent-encoding = "2.3" diff --git a/litellm-rust/crates/secrets-cyberark/src/error.rs b/litellm-rust/crates/secrets-cyberark/src/error.rs index 4f21647225b..b5dbd7342fa 100644 --- a/litellm-rust/crates/secrets-cyberark/src/error.rs +++ b/litellm-rust/crates/secrets-cyberark/src/error.rs @@ -6,7 +6,7 @@ pub enum Error { Http( #[from] #[redact] - reqwest::Error, + litellm_http::Error, ), #[error("CyberArk Conjur authentication returned HTTP {0}")] AuthStatus(u16), @@ -17,15 +17,68 @@ pub enum Error { )] MissingCredentials, #[error("CyberArk client certificate could not be loaded")] - ClientCertificate, + ClientCertificate( + #[source] + #[redact] + litellm_http::Error, + ), #[error("CyberArk Conjur HTTP client could not be built")] - Client(#[redact] Box), + Client( + #[source] + #[redact] + litellm_http::Error, + ), #[error("invalid refresh interval")] - RefreshInterval, + RefreshInterval(#[source] std::num::ParseIntError), #[error("invalid CyberArk Conjur endpoint")] - Endpoint, + Endpoint(#[source] url::ParseError), #[error("CyberArk secret manager requires an enterprise license")] EnterpriseRequired, #[error(transparent)] Operation(#[from] litellm_secrets_types::Error), } + +impl Error { + pub fn is_unsafe_secret_name(&self) -> bool { + match self { + Self::Operation(error) => error.is_unsafe_secret_name(), + Self::Timeout + | Self::Http(_) + | Self::AuthStatus(_) + | Self::Status(_) + | Self::MissingCredentials + | Self::ClientCertificate(_) + | Self::Client(_) + | Self::RefreshInterval(_) + | Self::Endpoint(_) + | Self::EnterpriseRequired => false, + } + } +} + +#[derive(thiserror::Error, veil::Redact)] +#[error("CyberArk Conjur write failed")] +pub struct WriteFailure { + pub source: Error, + #[redact] + pub request_url: Option, + pub authentication: bool, +} + +impl WriteFailure { + pub(crate) fn local(source: Error) -> Self { + Self { + source, + request_url: None, + authentication: false, + } + } + + pub(crate) fn request(source: Error, url: reqwest::Url) -> Self { + Self { + source, + request_url: Some(url), + authentication: false, + } + } +} diff --git a/litellm-rust/crates/secrets-cyberark/src/lib.rs b/litellm-rust/crates/secrets-cyberark/src/lib.rs index 74a8b6febf2..d780fb43921 100644 --- a/litellm-rust/crates/secrets-cyberark/src/lib.rs +++ b/litellm-rust/crates/secrets-cyberark/src/lib.rs @@ -3,5 +3,5 @@ mod error; mod secret_manager; -pub use error::Error; -pub use secret_manager::{AuthenticationRetry, CyberArkSecretManager, DeleteOutcome, WriteFailure}; +pub use error::{Error, WriteFailure}; +pub use secret_manager::{AuthenticationRetry, CyberArkSecretManager, DeleteOutcome}; diff --git a/litellm-rust/crates/secrets-cyberark/src/secret_manager.rs b/litellm-rust/crates/secrets-cyberark/src/secret_manager.rs index 44473c9fe90..c5cb0308b13 100644 --- a/litellm-rust/crates/secrets-cyberark/src/secret_manager.rs +++ b/litellm-rust/crates/secrets-cyberark/src/secret_manager.rs @@ -17,7 +17,7 @@ use litellm_secrets_types::{ use moka::future::Cache; use percent_encoding::{AsciiSet, NON_ALPHANUMERIC, utf8_percent_encode}; -use crate::Error; +use crate::{Error, WriteFailure}; const CYBERARK_API_BASE: &str = "CYBERARK_API_BASE"; const CYBERARK_ACCOUNT: &str = "CYBERARK_ACCOUNT"; @@ -62,38 +62,12 @@ pub enum AuthenticationRetry { Unauthorized, } -#[derive(veil::Redact)] -pub struct WriteFailure { - pub source: Error, - #[redact] - pub request_url: Option, - pub authentication: bool, -} - -impl WriteFailure { - fn local(source: Error) -> Self { - Self { - source, - request_url: None, - authentication: false, - } - } - - fn request(source: Error, url: reqwest::Url) -> Self { - Self { - source, - request_url: Some(url), - authentication: false, - } - } -} - impl CyberArkSecretManager { fn secret_url(&self, name: &str) -> Result { let encoded = utf8_percent_encode(name, SECRET_NAME_SAFE); self.endpoint .join(&format!("secrets/{}/variable/{}", self.account, encoded)) - .map_err(|_| Error::Endpoint) + .map_err(Error::Endpoint) } } diff --git a/litellm-rust/crates/secrets-cyberark/src/secret_manager/client.rs b/litellm-rust/crates/secrets-cyberark/src/secret_manager/client.rs index ea2262392d5..1cda0ae5306 100644 --- a/litellm-rust/crates/secrets-cyberark/src/secret_manager/client.rs +++ b/litellm-rust/crates/secrets-cyberark/src/secret_manager/client.rs @@ -64,25 +64,18 @@ impl CyberArkSecretManager { }), ..config.clone() }; - let client = - pool.client(&config, ClientVariant::Provider) - .map_err(|error| match error { - litellm_http::Error::Read { - tls_source: TlsSource::ClientIdentity, - .. - } - | litellm_http::Error::InvalidPem { - tls_source: TlsSource::ClientIdentity, - .. - } => Error::ClientCertificate, - other => Error::Client(Box::new(other)), - })?; + let client = pool + .client(&config, ClientVariant::Provider) + .map_err(|error| match error.tls_source() { + Some(TlsSource::ClientIdentity) => Error::ClientCertificate(error), + Some(TlsSource::CaBundle) | None => Error::Client(error), + })?; let endpoint = reqwest::Url::parse( &environment .get(CYBERARK_API_BASE) .unwrap_or_else(|| DEFAULT_API_BASE.to_owned()), ) - .map_err(|_| Error::Endpoint)?; + .map_err(Error::Endpoint)?; let account = environment .get(CYBERARK_ACCOUNT) .unwrap_or_else(|| DEFAULT_ACCOUNT.to_owned()); @@ -95,7 +88,7 @@ impl CyberArkSecretManager { value .parse::() .map(Duration::from_secs) - .map_err(|_| Error::RefreshInterval) + .map_err(Error::RefreshInterval) }) .transpose()?; Ok(Self::with_client( @@ -115,7 +108,7 @@ impl CyberArkSecretManager { self.account, utf8_percent_encode(&self.username, SECRET_NAME_SAFE) )) - .map_err(|_| Error::Endpoint) + .map_err(Error::Endpoint) } pub(super) async fn authenticate( @@ -135,11 +128,14 @@ impl CyberArkSecretManager { context, ) .send() - .await?; + .await + .map_err(litellm_http::Error::from)?; if !response.status().is_success() { return Err(Error::AuthStatus(response.status().as_u16())); } - let token = SecretValue::new(STANDARD.encode(response.text().await?)); + let token = SecretValue::new( + STANDARD.encode(response.text().await.map_err(litellm_http::Error::from)?), + ); self.token.insert((), token.clone()).await; Ok(token) } diff --git a/litellm-rust/crates/secrets-cyberark/src/secret_manager/read.rs b/litellm-rust/crates/secrets-cyberark/src/secret_manager/read.rs index daa9ed1114d..2862c6a956d 100644 --- a/litellm-rust/crates/secrets-cyberark/src/secret_manager/read.rs +++ b/litellm-rust/crates/secrets-cyberark/src/secret_manager/read.rs @@ -76,7 +76,8 @@ impl CyberArkSecretManager { context, ) .send() - .await?; + .await + .map_err(litellm_http::Error::from)?; let response = if matches!(retry, AuthenticationRetry::Unauthorized) && had_cached_token && response.status() == reqwest::StatusCode::UNAUTHORIZED @@ -89,7 +90,8 @@ impl CyberArkSecretManager { context, ) .send() - .await? + .await + .map_err(litellm_http::Error::from)? } else { response }; @@ -99,7 +101,7 @@ impl CyberArkSecretManager { if !response.status().is_success() { return Err(Error::Status(response.status().as_u16())); } - let value = SecretValue::new(response.text().await?); + let value = SecretValue::new(response.text().await.map_err(litellm_http::Error::from)?); Ok(Some(value)) } } diff --git a/litellm-rust/crates/secrets-cyberark/src/secret_manager/write.rs b/litellm-rust/crates/secrets-cyberark/src/secret_manager/write.rs index 87f6cea3619..7a2c79f506a 100644 --- a/litellm-rust/crates/secrets-cyberark/src/secret_manager/write.rs +++ b/litellm-rust/crates/secrets-cyberark/src/secret_manager/write.rs @@ -83,7 +83,9 @@ impl CyberArkSecretManager { ) .send() .await - .map_err(|source| WriteFailure::request(source.into(), url.clone())) + .map_err(|source| { + WriteFailure::request(litellm_http::Error::from(source).into(), url.clone()) + }) } pub(super) async fn ensure_variable_exists( @@ -141,8 +143,8 @@ impl CyberArkSecretManager { ); return; } - Err(error) => { - litellm_tracing::warn!("Error ensuring CyberArk variable exists: {error}"); + Err(_) => { + litellm_tracing::warn!("Error ensuring CyberArk variable exists"); return; } } diff --git a/litellm-rust/crates/secrets-cyberark/tests/secret_manager/configuration.rs b/litellm-rust/crates/secrets-cyberark/tests/secret_manager/configuration.rs index a3c20a0db3d..8a0746f8002 100644 --- a/litellm-rust/crates/secrets-cyberark/tests/secret_manager/configuration.rs +++ b/litellm-rust/crates/secrets-cyberark/tests/secret_manager/configuration.rs @@ -201,7 +201,7 @@ async fn trait_read_applies_cyberark_operation_timeout_to_authentication() { match result { Err(Error::Timeout) => {} - Err(Error::Http(error)) => assert!(error.is_timeout()), + Err(Error::Http(error)) => assert_eq!(error.status(), Some(408)), other => panic!("expected timeout, got {other:?}"), } } @@ -237,7 +237,7 @@ fn new_validates_credentials_before_license_and_configuration() { }), true ), - Err(Error::RefreshInterval) + Err(Error::RefreshInterval(_)) )); assert!(matches!( from_environment( @@ -248,7 +248,7 @@ fn new_validates_credentials_before_license_and_configuration() { }), true ), - Err(Error::Endpoint) + Err(Error::Endpoint(_)) )); } @@ -263,7 +263,7 @@ fn certificate_only_credentials_are_validated_as_a_client_identity() { true, ); - assert!(matches!(result, Err(Error::ClientCertificate))); + assert!(matches!(result, Err(Error::ClientCertificate(_)))); } #[rstest] @@ -347,7 +347,7 @@ fn invalid_client_identity_is_not_ignored( true, ); - assert!(matches!(result, Err(Error::ClientCertificate))); + assert!(matches!(result, Err(Error::ClientCertificate(_)))); } #[rstest] @@ -413,7 +413,7 @@ fn new_reports_missing_client_certificate_files() { }), true ), - Err(Error::ClientCertificate) + Err(Error::ClientCertificate(_)) )); } diff --git a/litellm-rust/crates/secrets-google/Cargo.toml b/litellm-rust/crates/secrets-google/Cargo.toml index 3ce14fe7a12..06b3747def5 100644 --- a/litellm-rust/crates/secrets-google/Cargo.toml +++ b/litellm-rust/crates/secrets-google/Cargo.toml @@ -17,6 +17,7 @@ base64.workspace = true crc32c = "0.6.8" serde_json.workspace = true thiserror.workspace = true +url.workspace = true veil.workspace = true google-cloud-kms-v1 = "1.14.0" google-cloud-gax = { version = "1.14.0", default-features = false } diff --git a/litellm-rust/crates/secrets-google/src/error.rs b/litellm-rust/crates/secrets-google/src/error.rs index 159e4983ddc..486c7a3f35e 100644 --- a/litellm-rust/crates/secrets-google/src/error.rs +++ b/litellm-rust/crates/secrets-google/src/error.rs @@ -26,7 +26,7 @@ pub enum Error { Http( #[from] #[redact] - reqwest::Error, + litellm_http::Error, ), #[error("Google Secret Manager returned HTTP {0}")] Status(u16), @@ -35,15 +35,41 @@ pub enum Error { #[error("required environment variable is missing: {0}")] MissingEnvironment(&'static str), #[error("invalid refresh interval")] - RefreshInterval, + RefreshInterval(#[source] std::num::ParseIntError), #[error("payload is not valid base64")] - Base64(#[from] base64::DecodeError), + Base64( + #[from] + #[redact] + base64::DecodeError, + ), #[error("Google Secret Manager payload checksum mismatch")] - Checksum, + Checksum(#[source] Option), #[error("decrypted value is not UTF-8")] - Utf8, + Utf8(#[source] std::str::Utf8Error), #[error("invalid Google Secret Manager endpoint")] - Endpoint, + Endpoint(#[source] url::ParseError), #[error("Google Secret Manager requires an enterprise license")] EnterpriseRequired, } + +impl Error { + pub fn is_not_found(&self) -> bool { + match self { + Self::Status(status) => *status == 404, + Self::Operation(_) + | Self::Timeout + | Self::Client(_) + | Self::Auth(_) + | Self::Kms(_) + | Self::Http(_) + | Self::MissingPayload + | Self::MissingEnvironment(_) + | Self::RefreshInterval(_) + | Self::Base64(_) + | Self::Checksum(_) + | Self::Utf8(_) + | Self::Endpoint(_) + | Self::EnterpriseRequired => false, + } + } +} diff --git a/litellm-rust/crates/secrets-google/src/secret_manager.rs b/litellm-rust/crates/secrets-google/src/secret_manager.rs index 08ba466b799..463d1f6fd73 100644 --- a/litellm-rust/crates/secrets-google/src/secret_manager.rs +++ b/litellm-rust/crates/secrets-google/src/secret_manager.rs @@ -92,12 +92,12 @@ impl GoogleSecretManager { let ttl = environment .get(GOOGLE_SECRET_MANAGER_REFRESH_INTERVAL) .filter(|v| !v.is_empty()) - .map(|v| v.parse::().map_err(|_| Error::RefreshInterval)) + .map(|v| v.parse::().map_err(Error::RefreshInterval)) .transpose()? .unwrap_or( environment .get(SECRET_MANAGER_REFRESH_INTERVAL) - .map(|v| v.parse::().map_err(|_| Error::RefreshInterval)) + .map(|v| v.parse::().map_err(Error::RefreshInterval)) .transpose()? .unwrap_or(DEFAULT_REFRESH_INTERVAL.as_secs() as i64), ); @@ -164,20 +164,21 @@ impl GoogleSecretManager { ), percent_encoding::utf8_percent_encode(name, percent_encoding::NON_ALPHANUMERIC) )) - .map_err(|_| Error::Endpoint)?; + .map_err(Error::Endpoint)?; let response = self .client .get(url) .headers(self.credentials.request_headers().await?) .send() - .await?; + .await + .map_err(litellm_http::Error::from)?; if response.status() == reqwest::StatusCode::NOT_FOUND { return Ok(None); } if response.status() != reqwest::StatusCode::OK { return Err(Error::Status(response.status().as_u16())); } - let response: Response = response.json().await?; + let response: Response = response.json().await.map_err(litellm_http::Error::from)?; let Some(payload) = response.payload else { return Err(Error::MissingPayload); }; @@ -186,12 +187,15 @@ impl GoogleSecretManager { }; let bytes = STANDARD.decode(data)?; if let Some(expected) = payload.data_crc32c { - let expected = expected.parse::().map_err(|_| Error::Checksum)?; + let expected = expected + .parse::() + .map_err(|error| Error::Checksum(Some(error)))?; if crc32c::crc32c(&bytes) != expected { - return Err(Error::Checksum); + return Err(Error::Checksum(None)); } } - let plaintext = String::from_utf8(bytes).map_err(|_| Error::Utf8)?; + let plaintext = + String::from_utf8(bytes).map_err(|error| Error::Utf8(error.utf8_error()))?; let value = SecretValue::new(plaintext); Ok(Some(value)) } diff --git a/litellm-rust/crates/secrets-google/tests/secret_manager.rs b/litellm-rust/crates/secrets-google/tests/secret_manager.rs index f2cb5ff3603..615e331cb96 100644 --- a/litellm-rust/crates/secrets-google/tests/secret_manager.rs +++ b/litellm-rust/crates/secrets-google/tests/secret_manager.rs @@ -152,8 +152,8 @@ async fn failed_or_missing_reads_are_not_cached( assert!(matches!(result, Err(Error::MissingPayload))); } ExpectedReadFailure::Base64 => assert!(matches!(result, Err(Error::Base64(_)))), - ExpectedReadFailure::Utf8 => assert!(matches!(result, Err(Error::Utf8))), - ExpectedReadFailure::Checksum => assert!(matches!(result, Err(Error::Checksum))), + ExpectedReadFailure::Utf8 => assert!(matches!(result, Err(Error::Utf8(_)))), + ExpectedReadFailure::Checksum => assert!(matches!(result, Err(Error::Checksum(None)))), } drop(failing); Mock::given(path( @@ -245,7 +245,7 @@ fn google_manager_rejects_invalid_refresh_intervals(#[case] variable: &'static s assert!(matches!( GoogleSecretManager::new(litellm_http::Client::for_test(), environment, true), - Err(Error::RefreshInterval) + Err(Error::RefreshInterval(_)) )); } diff --git a/litellm-rust/crates/secrets-hashicorp/Cargo.toml b/litellm-rust/crates/secrets-hashicorp/Cargo.toml index 7dd3d3c674f..8731a8deb7a 100644 --- a/litellm-rust/crates/secrets-hashicorp/Cargo.toml +++ b/litellm-rust/crates/secrets-hashicorp/Cargo.toml @@ -6,8 +6,11 @@ license.workspace = true repository.workspace = true [dependencies] +litellm-error.workspace = true litellm-core-utils.workspace = true +litellm-http.workspace = true litellm-secrets-types.workspace = true +reqwest.workspace = true rustify.workspace = true rustify_derive.workspace = true serde.workspace = true diff --git a/litellm-rust/crates/secrets-hashicorp/src/config.rs b/litellm-rust/crates/secrets-hashicorp/src/config.rs index d32491199c4..4232667ae16 100644 --- a/litellm-rust/crates/secrets-hashicorp/src/config.rs +++ b/litellm-rust/crates/secrets-hashicorp/src/config.rs @@ -153,9 +153,11 @@ fn refresh_interval(environment: &dyn Lookup) -> Result { let Some(value) = value else { return Ok(DEFAULT_REFRESH_INTERVAL); }; - let seconds: i64 = value.parse().map_err(|_| Error::RefreshInterval)?; + let seconds: i64 = value + .parse() + .map_err(|error| Error::RefreshInterval(Some(error)))?; if seconds < 0 { - return Err(Error::RefreshInterval); + return Err(Error::RefreshInterval(None)); } Ok(Duration::from_secs(seconds as u64)) } diff --git a/litellm-rust/crates/secrets-hashicorp/src/error.rs b/litellm-rust/crates/secrets-hashicorp/src/error.rs index 1813cb486bc..73399a44eaa 100644 --- a/litellm-rust/crates/secrets-hashicorp/src/error.rs +++ b/litellm-rust/crates/secrets-hashicorp/src/error.rs @@ -1,42 +1,144 @@ +use rustify::errors::ClientError as RustifyClientError; +use vaultrs::error::ClientError; + #[derive(thiserror::Error, veil::Redact)] pub enum Error { #[error("HashiCorp Vault requires an enterprise license")] EnterpriseRequired, #[error("invalid secret name")] - InvalidSecretName(litellm_secrets_types::Error), + InvalidSecretName(#[source] litellm_secrets_types::Error), #[error(transparent)] Operation(#[from] litellm_secrets_types::Error), #[error("HashiCorp Vault client failed")] Client( - #[from] + #[source] #[redact] vaultrs::error::ClientError, ), - #[error("HashiCorp Vault client settings are invalid: {message}")] - ClientSettings { message: String }, - #[error("HashiCorp Vault TLS identity could not be configured for {path}: {message}")] + #[error("HashiCorp Vault request failed")] + Transport( + #[source] + #[redact] + litellm_http::Error, + ), + #[error("HashiCorp Vault client settings are invalid")] + ClientSettings(#[source] vaultrs::client::VaultClientSettingsBuilderError), + #[error("HashiCorp Vault TLS identity could not be configured for {path}")] TlsIdentity { path: std::path::PathBuf, - message: String, + source: litellm_error::Source, }, #[error("HashiCorp Vault login returned HTTP {status}")] LoginStatus { status: u16 }, #[error("HashiCorp Vault login response is malformed")] - MalformedLogin, + MalformedLogin( + #[source] + #[redact] + vaultrs::error::ClientError, + ), #[error("HashiCorp Vault authentication is not configured")] NoAuthConfigured, #[error("HashiCorp Vault returned HTTP {status}")] Status { status: u16 }, #[error("HashiCorp Vault response payload is malformed")] - MalformedPayload, + MalformedPayload( + #[source] + #[redact] + vaultrs::error::ClientError, + ), #[error("HashiCorp Vault secret value is not a string")] NonStringValue, #[error("HashiCorp Vault data key conflicts with description")] DataKeyConflictsWithDescription, #[error("HashiCorp Vault secret version exceeds CAS range")] - CasVersionOverflow, + CasVersionOverflow(#[source] std::num::TryFromIntError), #[error("HashiCorp Vault operation timed out")] Timeout, #[error("invalid HashiCorp Vault refresh interval")] - RefreshInterval, + RefreshInterval(#[source] Option), +} + +impl Error { + pub fn login_status(&self) -> Option { + match self { + Self::LoginStatus { status } => Some(*status), + Self::EnterpriseRequired + | Self::InvalidSecretName(_) + | Self::Operation(_) + | Self::Client(_) + | Self::Transport(_) + | Self::ClientSettings(_) + | Self::TlsIdentity { .. } + | Self::MalformedLogin(_) + | Self::NoAuthConfigured + | Self::Status { .. } + | Self::MalformedPayload(_) + | Self::NonStringValue + | Self::DataKeyConflictsWithDescription + | Self::CasVersionOverflow(_) + | Self::Timeout + | Self::RefreshInterval(_) => None, + } + } + + pub fn is_unsafe_secret_name(&self) -> bool { + match self { + Self::InvalidSecretName(error) | Self::Operation(error) => { + error.is_unsafe_secret_name() + } + Self::EnterpriseRequired + | Self::Client(_) + | Self::Transport(_) + | Self::ClientSettings(_) + | Self::TlsIdentity { .. } + | Self::LoginStatus { .. } + | Self::MalformedLogin(_) + | Self::NoAuthConfigured + | Self::Status { .. } + | Self::MalformedPayload(_) + | Self::NonStringValue + | Self::DataKeyConflictsWithDescription + | Self::CasVersionOverflow(_) + | Self::Timeout + | Self::RefreshInterval(_) => false, + } + } +} + +impl From for Error { + fn from(error: ClientError) -> Self { + match error { + ClientError::RestClientError { source } => match transport(source) { + Ok(error) => Self::Transport(error), + Err(source) => Self::Client(ClientError::RestClientError { source }), + }, + ClientError::RestClientBuildError { source } => Self::Transport(source.into()), + error => Self::Client(error), + } + } +} + +pub(crate) fn transport( + error: RustifyClientError, +) -> Result { + match error { + RustifyClientError::RequestError { + source, + url, + method, + } => source + .downcast::() + .map(litellm_http::Error::from) + .map_err(|source| RustifyClientError::RequestError { + source, + url, + method, + }), + RustifyClientError::ResponseError { source } => source + .downcast::() + .map(litellm_http::Error::from) + .map_err(|source| RustifyClientError::ResponseError { source }), + RustifyClientError::ReqwestBuildError { source } => Ok(source.into()), + error => Err(error), + } } diff --git a/litellm-rust/crates/secrets-hashicorp/src/lib.rs b/litellm-rust/crates/secrets-hashicorp/src/lib.rs index 7aab5a38b01..15c03724f72 100644 --- a/litellm-rust/crates/secrets-hashicorp/src/lib.rs +++ b/litellm-rust/crates/secrets-hashicorp/src/lib.rs @@ -7,4 +7,6 @@ pub mod secret_manager; pub use config::{AppRoleAuth, HashicorpVaultConfig, TlsCertAuth}; pub use error::Error; -pub use secret_manager::{HashicorpVault, RawOperationError, SecretLocation}; +pub use secret_manager::{ + HashicorpVault, HttpFailure, LoginFailure, RawOperationError, SecretLocation, +}; diff --git a/litellm-rust/crates/secrets-hashicorp/src/secret_manager.rs b/litellm-rust/crates/secrets-hashicorp/src/secret_manager.rs index f0d4fe8c0b4..447beacda85 100644 --- a/litellm-rust/crates/secrets-hashicorp/src/secret_manager.rs +++ b/litellm-rust/crates/secrets-hashicorp/src/secret_manager.rs @@ -31,7 +31,7 @@ use vaultrs::{ use crate::{Error, HashicorpVaultConfig, TlsCertAuth, cert_login::CertLoginRequest}; -pub use raw::RawOperationError; +pub use raw::{HttpFailure, LoginFailure, RawOperationError}; const CACHE_CAPACITY: u64 = 200; @@ -178,23 +178,25 @@ fn map_api_error(error: ClientError, context: ErrorContext) -> Error { ErrorContext::Login => Error::LoginStatus { status: code }, ErrorContext::Read | ErrorContext::Secret => Error::Status { status: code }, }, - ClientError::JsonParseError { source } => match context { - ErrorContext::Login => Error::MalformedLogin, - ErrorContext::Read => Error::MalformedPayload, - ErrorContext::Secret => Error::Client(ClientError::JsonParseError { source }), + error @ ClientError::JsonParseError { .. } => match context { + ErrorContext::Login => Error::MalformedLogin(error), + ErrorContext::Read => Error::MalformedPayload(error), + ErrorContext::Secret => Error::Client(error), }, - ClientError::ResponseEmptyError | ClientError::ResponseDataEmptyError => { - malformed_response(context) + error @ (ClientError::ResponseEmptyError | ClientError::ResponseDataEmptyError) => { + malformed_response(context, error) } ClientError::RestClientError { source } => match source { RustifyClientError::ServerResponseError { code, .. } => match context { ErrorContext::Login => Error::LoginStatus { status: code }, ErrorContext::Read | ErrorContext::Secret => Error::Status { status: code }, }, - RustifyClientError::ResponseParseError { .. } => malformed_response(context), - source => Error::Client(ClientError::RestClientError { source }), + source @ RustifyClientError::ResponseParseError { .. } => { + malformed_response(context, ClientError::RestClientError { source }) + } + source => ClientError::RestClientError { source }.into(), }, - error => Error::Client(error), + error => error.into(), } } @@ -208,10 +210,10 @@ fn api_status(error: &ClientError) -> Option { } } -fn malformed_response(context: ErrorContext) -> Error { +fn malformed_response(context: ErrorContext, error: ClientError) -> Error { match context { - ErrorContext::Login => Error::MalformedLogin, - ErrorContext::Read => Error::MalformedPayload, - ErrorContext::Secret => Error::Client(ClientError::ResponseDataEmptyError), + ErrorContext::Login => Error::MalformedLogin(error), + ErrorContext::Read => Error::MalformedPayload(error), + ErrorContext::Secret => Error::Client(error), } } diff --git a/litellm-rust/crates/secrets-hashicorp/src/secret_manager/client.rs b/litellm-rust/crates/secrets-hashicorp/src/secret_manager/client.rs index c1dc073dcf8..01d3807d290 100644 --- a/litellm-rust/crates/secrets-hashicorp/src/secret_manager/client.rs +++ b/litellm-rust/crates/secrets-hashicorp/src/secret_manager/client.rs @@ -83,10 +83,8 @@ impl HashicorpVault { .ca_certs(Vec::new()) .verify(true) .build() - .map_err(|message| Error::ClientSettings { - message: message.to_string(), - })?; - VaultClient::new(settings).map_err(Error::Client) + .map_err(Error::ClientSettings)?; + VaultClient::new(settings).map_err(Error::from) } } @@ -94,16 +92,16 @@ fn identity_for(tls: Option<&TlsCertAuth>) -> Result, Error> { tls.map(|tls| { let cert: Vec = std::fs::read(&tls.cert_path).map_err(|source| Error::TlsIdentity { path: tls.cert_path.clone(), - message: source.to_string(), + source: litellm_error::Source::new(source), })?; let key: Vec = std::fs::read(&tls.key_path).map_err(|source| Error::TlsIdentity { path: tls.key_path.clone(), - message: source.to_string(), + source: litellm_error::Source::new(source), })?; Identity::from_pem(&[cert.as_slice(), key.as_slice()].concat()).map_err(|source| { Error::TlsIdentity { path: tls.cert_path.clone(), - message: source.to_string(), + source: litellm_error::Source::new(source), } }) }) diff --git a/litellm-rust/crates/secrets-hashicorp/src/secret_manager/raw.rs b/litellm-rust/crates/secrets-hashicorp/src/secret_manager/raw.rs index 0325561a7a0..9c722c0b6c8 100644 --- a/litellm-rust/crates/secrets-hashicorp/src/secret_manager/raw.rs +++ b/litellm-rust/crates/secrets-hashicorp/src/secret_manager/raw.rs @@ -4,26 +4,123 @@ use vaultrs::api::kv2::requests::{ DeleteLatestSecretVersionRequest, ReadSecretRequest, SetSecretRequest, }; -#[derive(veil::Redact)] +#[derive(thiserror::Error, veil::Redact)] pub enum RawOperationError { + #[error(transparent)] Local(Error), + #[error("HashiCorp Vault authentication failed")] Authentication { source: Error, url: String, certificate: bool, }, - Http { - method: String, - url: String, - status: u16, + #[error("HashiCorp Vault {} request returned HTTP {}", .0.method, .0.status)] + Http(HttpFailure), + #[error("HashiCorp Vault request failed")] + Transport( + #[source] #[redact] - body: Vec, - }, - Transport(#[redact] RustifyClientError), - Timeout { - method: String, - elapsed: Duration, - }, + litellm_http::Error, + ), + #[error("HashiCorp Vault client failed")] + Client( + #[source] + #[redact] + RustifyClientError, + ), + #[error("HashiCorp Vault {method} request timed out")] + Timeout { method: String, elapsed: Duration }, +} + +#[derive(veil::Redact)] +pub struct HttpFailure { + pub method: String, + pub url: String, + pub status: u16, + #[redact] + pub body: Vec, +} + +pub struct LoginFailure<'a> { + pub source: &'a Error, + pub url: &'a str, + pub certificate: bool, +} + +impl RawOperationError { + pub fn is_unsafe_secret_name(&self) -> bool { + match self { + Self::Local(error) => error.is_unsafe_secret_name(), + Self::Authentication { .. } + | Self::Http(_) + | Self::Transport(_) + | Self::Client(_) + | Self::Timeout { .. } => false, + } + } + + pub fn local(&self) -> Option<&Error> { + match self { + Self::Local(error) => Some(error), + Self::Authentication { .. } + | Self::Http(_) + | Self::Transport(_) + | Self::Client(_) + | Self::Timeout { .. } => None, + } + } + + pub fn authentication(&self) -> Option> { + match self { + Self::Authentication { + source, + url, + certificate, + } => Some(LoginFailure { + source, + url, + certificate: *certificate, + }), + Self::Local(_) + | Self::Http(_) + | Self::Transport(_) + | Self::Client(_) + | Self::Timeout { .. } => None, + } + } + + pub fn http_failure(&self) -> Option<&HttpFailure> { + match self { + Self::Http(failure) => Some(failure), + Self::Local(_) + | Self::Authentication { .. } + | Self::Transport(_) + | Self::Client(_) + | Self::Timeout { .. } => None, + } + } + + pub fn timeout(&self) -> Option<(&str, Duration)> { + match self { + Self::Timeout { method, elapsed } => Some((method, *elapsed)), + Self::Local(_) + | Self::Authentication { .. } + | Self::Http(_) + | Self::Transport(_) + | Self::Client(_) => None, + } + } + + pub fn transport(&self) -> Option<&litellm_http::Error> { + match self { + Self::Transport(error) => Some(error), + Self::Local(_) + | Self::Authentication { .. } + | Self::Http(_) + | Self::Client(_) + | Self::Timeout { .. } => None, + } + } } impl HashicorpVault { @@ -118,7 +215,7 @@ impl HashicorpVault { let request = endpoint .with_middleware(&client.middle) .request(client.http.base()) - .map_err(RawOperationError::Transport)?; + .map_err(RawOperationError::Client)?; let method = request.method().to_string(); let url = format!( "{}/v1/{}{}/data/{}", @@ -141,14 +238,17 @@ impl HashicorpVault { })?, None => client.http.send(request).await, } - .map_err(RawOperationError::Transport)?; + .map_err(|error| match crate::error::transport(error) { + Ok(error) => RawOperationError::Transport(error), + Err(error) => RawOperationError::Client(error), + })?; if !response.status().is_success() { - return Err(RawOperationError::Http { + return Err(RawOperationError::Http(HttpFailure { method, url, status: response.status().as_u16(), body: response.into_body(), - }); + })); } Ok(response.into_body()) } diff --git a/litellm-rust/crates/secrets-hashicorp/src/secret_manager/write.rs b/litellm-rust/crates/secrets-hashicorp/src/secret_manager/write.rs index 0b1acb3a179..15a0e4c7dff 100644 --- a/litellm-rust/crates/secrets-hashicorp/src/secret_manager/write.rs +++ b/litellm-rust/crates/secrets-hashicorp/src/secret_manager/write.rs @@ -37,7 +37,7 @@ impl HashicorpVault { .await { Ok(metadata) => u32::try_from(metadata.current_version) - .map_err(|_| Error::CasVersionOverflow)?, + .map_err(Error::CasVersionOverflow)?, Err(error) if api_status(&error) == Some(404) => 0, Err(_) => return Err(map_api_error(error, ErrorContext::Secret)), }; diff --git a/litellm-rust/crates/secrets-hashicorp/tests/secret_manager/configuration.rs b/litellm-rust/crates/secrets-hashicorp/tests/secret_manager/configuration.rs index be3059ce986..b45b843fde8 100644 --- a/litellm-rust/crates/secrets-hashicorp/tests/secret_manager/configuration.rs +++ b/litellm-rust/crates/secrets-hashicorp/tests/secret_manager/configuration.rs @@ -32,7 +32,7 @@ fn invalid_refresh_intervals_are_rejected(#[case] value: &str) { assert!(matches!( HashicorpVaultConfig::from_environment(environment.as_ref()), - Err(Error::RefreshInterval) + Err(Error::RefreshInterval(_)) )); } @@ -79,6 +79,39 @@ async fn approle_login_uses_namespace_and_reuses_the_token() { assert!(manager.async_read_secret("name-2").await.unwrap().is_none()); } +#[rstest] +#[tokio::test] +async fn an_unreadable_client_certificate_keeps_its_io_cause() { + let server: MockServer = MockServer::start().await; + let directory: tempfile::TempDir = tempfile::tempdir().unwrap(); + let cert_path = directory.path().join("missing.crt"); + let values: HashMap = HashMap::from([ + ("HCP_VAULT_ADDR".to_owned(), server.uri()), + ( + "HCP_VAULT_CLIENT_CERT".to_owned(), + cert_path.to_str().unwrap().to_owned(), + ), + ( + "HCP_VAULT_CLIENT_KEY".to_owned(), + cert_path.to_str().unwrap().to_owned(), + ), + ]); + let environment: Arc = + Arc::new(move |name: &str| values.get(name).cloned()); + let manager: HashicorpVault = HashicorpVault::new(environment, true).unwrap(); + + let error = manager.async_read_secret("name").await.unwrap_err(); + + let cause = std::error::Error::source(&error) + .and_then(|cause| cause.downcast_ref::()) + .and_then(|cause| cause.downcast_ref::()) + .expect("io cause"); + assert_eq!(cause.kind(), std::io::ErrorKind::NotFound); + let report = litellm_error::Report::new(&error).to_string(); + assert_eq!(report.matches(cert_path.to_str().unwrap()).count(), 1); + assert!(report.ends_with(&cause.to_string())); +} + #[rstest] #[tokio::test] async fn tls_login_posts_the_role_and_uses_the_client_identity() { diff --git a/litellm-rust/crates/secrets-hashicorp/tests/secret_manager/reads.rs b/litellm-rust/crates/secrets-hashicorp/tests/secret_manager/reads.rs index 0c662eb5b55..c31b29fb536 100644 --- a/litellm-rust/crates/secrets-hashicorp/tests/secret_manager/reads.rs +++ b/litellm-rust/crates/secrets-hashicorp/tests/secret_manager/reads.rs @@ -138,7 +138,7 @@ async fn read_responses_distinguish_absence_and_malformed_payloads( .await; match expected { ExpectedRead::Missing => assert!(result.unwrap().is_none()), - ExpectedRead::Malformed => assert!(matches!(result, Err(Error::MalformedPayload))), + ExpectedRead::Malformed => assert!(matches!(result, Err(Error::MalformedPayload(_)))), ExpectedRead::NonString => assert!(matches!(result, Err(Error::NonStringValue))), } } @@ -383,3 +383,52 @@ async fn concurrent_reads_share_a_load_but_verification_fetches_fresh( "new" ); } + +fn unreachable_manager() -> HashicorpVault { + let address = std::net::TcpListener::bind("127.0.0.1:0") + .unwrap() + .local_addr() + .unwrap(); + let environment: Arc = Arc::new(move |name: &str| match name { + "HCP_VAULT_ADDR" => Some(format!("http://{address}")), + "HCP_VAULT_TOKEN" => Some("token".to_owned()), + _ => None, + }); + HashicorpVault::new(environment, true).unwrap() +} + +#[tokio::test] +async fn unreachable_vault_reads_are_connect_failures_without_the_url() { + let error = unreachable_manager() + .async_read_secret("do-not-leak") + .await + .unwrap_err(); + + assert!(matches!(&error, Error::Transport(transport) if transport.is_connect())); + let report = litellm_error::Report::new(&error).to_string(); + assert!( + report.starts_with("HashiCorp Vault request failed: "), + "{report}" + ); + assert!(!report.contains("do-not-leak"), "{report}"); +} + +#[tokio::test] +async fn unreachable_vault_raw_reads_are_connect_failures_without_the_url() { + let error = unreachable_manager() + .read_raw("do-not-leak", &HashicorpOperationContext::default()) + .await + .unwrap_err(); + + assert!( + error + .transport() + .is_some_and(litellm_http::Error::is_connect) + ); + let report = litellm_error::Report::new(&error).to_string(); + assert!( + report.starts_with("HashiCorp Vault request failed: "), + "{report}" + ); + assert!(!report.contains("do-not-leak"), "{report}"); +} diff --git a/litellm-rust/crates/secrets-types/src/error.rs b/litellm-rust/crates/secrets-types/src/error.rs index 027dd54d8cc..891d8a0fba9 100644 --- a/litellm-rust/crates/secrets-types/src/error.rs +++ b/litellm-rust/crates/secrets-types/src/error.rs @@ -11,3 +11,15 @@ pub enum Error { #[error("secret manager received an incompatible operation context")] InvalidOperationContext, } + +impl Error { + pub fn is_unsafe_secret_name(&self) -> bool { + match self { + Self::UnsafeSecretName => true, + Self::CurrentSecretMissing + | Self::NewSecretMissing + | Self::NewSecretMismatch + | Self::InvalidOperationContext => false, + } + } +} diff --git a/litellm-rust/crates/secrets-types/tests/cache.rs b/litellm-rust/crates/secrets-types/tests/cache.rs index cec4f5198e3..18efde4289b 100644 --- a/litellm-rust/crates/secrets-types/tests/cache.rs +++ b/litellm-rust/crates/secrets-types/tests/cache.rs @@ -110,7 +110,7 @@ async fn concurrent_misses_share_a_load_and_cancellation_allows_a_retry() { } #[tokio::test] -async fn refresh_bypasses_cached_values_and_errors_and_absence_are_retried() { +async fn refresh_bypasses_cached_values() { let cache = SecretCache::new(10, Duration::from_secs(60)); cache.insert("key", "old").await; assert_eq!( @@ -127,9 +127,21 @@ async fn refresh_bypasses_cached_values_and_errors_and_absence_are_retried() { .unwrap(), Some("fresh") ); - for result in [Err("failure"), Ok(None), Ok(Some("recovered"))] { - assert_eq!(cache.read("retry", async { result }).await, result); - } +} + +#[rstest] +#[case::failure(Err("failure"))] +#[case::absence(Ok(None))] +#[tokio::test] +async fn errors_and_absence_are_retried(#[case] first: Result, &'static str>) { + let cache = SecretCache::new(10, Duration::from_secs(60)); + assert_eq!(cache.read("retry", async { first }).await, first); + assert_eq!( + cache + .read("retry", async { Ok::<_, &str>(Some("recovered")) }) + .await, + Ok(Some("recovered")) + ); } #[tokio::test] diff --git a/litellm-rust/crates/secrets/Cargo.toml b/litellm-rust/crates/secrets/Cargo.toml index 3655ce8bbc2..023d61136e4 100644 --- a/litellm-rust/crates/secrets/Cargo.toml +++ b/litellm-rust/crates/secrets/Cargo.toml @@ -23,6 +23,7 @@ litellm-secrets-hashicorp = { workspace = true, optional = true } litellm-secrets-azure = { workspace = true, optional = true } litellm-secrets-cyberark = { workspace = true, optional = true } litellm-core-utils.workspace = true +litellm-error.workspace = true litellm-http.workspace = true base64.workspace = true serde.workspace = true diff --git a/litellm-rust/crates/secrets/src/compatibility.rs b/litellm-rust/crates/secrets/src/compatibility.rs index 621f7a3f432..368c9931b51 100644 --- a/litellm-rust/crates/secrets/src/compatibility.rs +++ b/litellm-rust/crates/secrets/src/compatibility.rs @@ -40,11 +40,9 @@ pub async fn get_secret_from_python_manager( }; match result { #[cfg(feature = "google")] - Err(Error::Google(litellm_secrets_google::Error::Status(404))) => { - Err(Error::ManagedSecretMissing) - } + Err(Error::Google(error)) if error.is_not_found() => Err(Error::ManagedSecretMissing), #[cfg(feature = "azure")] - Err(Error::Azure(litellm_secrets_azure::Error::MissingValue)) => Ok(None), + Err(Error::Azure(error)) if error.is_missing_value() => Ok(None), Ok(None) if matches!(manager, SecretManager::External(_)) && manager.system() == KeyManagementSystem::AzureKeyVault => diff --git a/litellm-rust/crates/secrets/src/error.rs b/litellm-rust/crates/secrets/src/error.rs index 7fc756c5c7d..ade8555242d 100644 --- a/litellm-rust/crates/secrets/src/error.rs +++ b/litellm-rust/crates/secrets/src/error.rs @@ -7,9 +7,9 @@ pub enum Error { #[error("encrypted environment value is missing")] MissingCiphertext, #[error("ciphertext is not valid base64 for the configured manager")] - InvalidCiphertext, + InvalidCiphertext(#[source] Option), #[error("decrypted value is not UTF-8")] - Utf8, + Utf8(#[source] std::str::Utf8Error), #[error(transparent)] Client(#[from] litellm_http::Error), #[error("unsupported OIDC provider or missing build feature")] @@ -19,21 +19,21 @@ pub enum Error { #[error("OIDC environment variable is missing")] MissingEnvironment, #[error("OIDC request failed")] - OidcHttp, + OidcHttp(#[source] litellm_http::Error), #[error("OIDC provider returned HTTP {0}")] OidcStatus(u16), #[error("OIDC response is invalid")] - OidcResponse, + OidcResponse(#[source] litellm_http::Error), #[error("OIDC file path must be absolute and within the credential allowlist")] UnsafeOidcPath, #[error("OIDC file could not be read")] - OidcFile, + OidcFile(#[source] std::io::Error), #[error("secret cannot be converted to {expected}")] TypeMismatch { expected: &'static str }, #[error("external secret manager failed")] - ExternalManager(#[source] Box), + ExternalManager(#[source] litellm_error::Source), #[error("external secret manager read failed")] - ExternalRead(#[source] Box), + ExternalRead(#[source] litellm_error::Source), #[cfg(feature = "aws")] #[error(transparent)] Aws(#[from] litellm_secrets_aws::Error), diff --git a/litellm-rust/crates/secrets/src/handler.rs b/litellm-rust/crates/secrets/src/handler.rs index db37d793384..f42e09c6bdb 100644 --- a/litellm-rust/crates/secrets/src/handler.rs +++ b/litellm-rust/crates/secrets/src/handler.rs @@ -87,7 +87,8 @@ pub async fn get_secret_from_manager( let plaintext = client .decrypt(decode_ciphertext(&ciphertext, Base64Mode::Permissive)?) .await?; - let value = String::from_utf8(plaintext).map_err(|_| Error::Utf8)?; + let value = + String::from_utf8(plaintext).map_err(|error| Error::Utf8(error.utf8_error()))?; Ok(Some(Secret::String(SecretValue::new(value.trim())))) } #[cfg(feature = "google")] @@ -98,7 +99,8 @@ pub async fn get_secret_from_manager( let plaintext = client .decrypt(decode_ciphertext(&ciphertext, Base64Mode::Canonical)?) .await?; - let value = String::from_utf8(plaintext).map_err(|_| Error::Utf8)?; + let value = + String::from_utf8(plaintext).map_err(|error| Error::Utf8(error.utf8_error()))?; Ok(Some(Secret::String(SecretValue::new(value)))) } #[cfg(feature = "aws")] @@ -149,9 +151,9 @@ fn decode_ciphertext(value: &str, mode: Base64Mode) -> Result, Error> { }; let ciphertext = STANDARD .decode(&encoded) - .map_err(|_| Error::InvalidCiphertext)?; + .map_err(|error| Error::InvalidCiphertext(Some(error)))?; if canonical && STANDARD.encode(&ciphertext) != encoded { - return Err(Error::InvalidCiphertext); + return Err(Error::InvalidCiphertext(None)); } Ok(ciphertext) } diff --git a/litellm-rust/crates/secrets/src/oidc.rs b/litellm-rust/crates/secrets/src/oidc.rs index adeb6c3db00..24e6912c797 100644 --- a/litellm-rust/crates/secrets/src/oidc.rs +++ b/litellm-rust/crates/secrets/src/oidc.rs @@ -191,7 +191,7 @@ impl OidcResolver { .header("Accept", "application/json; api-version=2.0") .send() .await - .map_err(|_| Error::OidcHttp)?; + .map_err(|error| Error::OidcHttp(error.into()))?; if response.status() != reqwest::StatusCode::OK { return Err(Error::OidcStatus(response.status().as_u16())); } @@ -199,7 +199,10 @@ impl OidcResolver { struct Token { value: Option, } - let token: Token = response.json().await.map_err(|_| Error::OidcResponse)?; + let token: Token = response + .json() + .await + .map_err(|error| Error::OidcResponse(error.into()))?; if let Some(value) = &token.value { self.cache .insert( @@ -225,11 +228,14 @@ impl OidcResolver { .header("Metadata-Flavor", "Google") .send() .await - .map_err(|_| Error::OidcHttp)?; + .map_err(|error| Error::OidcHttp(error.into()))?; if response.status() != reqwest::StatusCode::OK { return Err(Error::OidcStatus(response.status().as_u16())); } - let token = response.text().await.map_err(|_| Error::OidcResponse)?; + let token = response + .text() + .await + .map_err(|error| Error::OidcResponse(error.into()))?; let now = (self.clock)(); let ttl = oidc_token_cache_ttl(&token, now, GOOGLE_TOKEN_MAX_TTL); let value = SecretValue::new(token); @@ -259,7 +265,7 @@ async fn read_file(path: &str) -> Result { tokio::fs::read_to_string(path) .await .map(|value| SecretValue::new(value.replace("\r\n", "\n").replace('\r', "\n"))) - .map_err(|_| Error::OidcFile) + .map_err(Error::OidcFile) } async fn read_allowed_file( @@ -271,7 +277,7 @@ async fn read_allowed_file( } let resolved = tokio::fs::canonicalize(path) .await - .map_err(|_| Error::OidcFile)?; + .map_err(Error::OidcFile)?; let allowed = environment .get(OIDC_ALLOWED_CREDENTIAL_DIRS) .filter(|v| !v.is_empty()) @@ -283,7 +289,7 @@ async fn read_allowed_file( return tokio::fs::read_to_string(&resolved) .await .map(|value| SecretValue::new(value.replace("\r\n", "\n").replace('\r', "\n"))) - .map_err(|_| Error::OidcFile); + .map_err(Error::OidcFile); } } Err(Error::UnsafeOidcPath) diff --git a/litellm-rust/crates/secrets/tests/aws.rs b/litellm-rust/crates/secrets/tests/aws.rs index f28f404c3ff..abad04f31c5 100644 --- a/litellm-rust/crates/secrets/tests/aws.rs +++ b/litellm-rust/crates/secrets/tests/aws.rs @@ -218,6 +218,6 @@ async fn aws_handler_reads_ciphertext_decodes_trims_and_redacts() { )); assert!(matches!( get_secret_from_manager(&manager, "KEY", &settings, &|_: &str| Some("abc".into())).await, - Err(Error::InvalidCiphertext) + Err(Error::InvalidCiphertext(_)) )); } diff --git a/litellm-rust/crates/secrets/tests/google.rs b/litellm-rust/crates/secrets/tests/google.rs index fa1db9916e8..5542c864dfd 100644 --- a/litellm-rust/crates/secrets/tests/google.rs +++ b/litellm-rust/crates/secrets/tests/google.rs @@ -108,7 +108,7 @@ async fn google_handler_requires_canonical_base64_and_preserves_plaintext_whites STANDARD.encode("encrypted") ))) .await, - Err(Error::InvalidCiphertext) + Err(Error::InvalidCiphertext(_)) )); assert!(matches!( get_secret_from_manager(&manager, "KEY", &settings, &|_: &str| None).await, diff --git a/litellm-rust/crates/secrets/tests/resolution.rs b/litellm-rust/crates/secrets/tests/resolution.rs index 0858089b94f..8435e051315 100644 --- a/litellm-rust/crates/secrets/tests/resolution.rs +++ b/litellm-rust/crates/secrets/tests/resolution.rs @@ -17,74 +17,81 @@ fn resolver(value: Option<&str>) -> SecretResolver { } #[rstest::rstest] -#[case::environment(false)] -#[case::manager(true)] +#[case::capitalized_true("True", Some(true))] +#[case::padded_false(" FALSE ", Some(false))] +#[case::empty("", None)] +#[case::parenthesized("(True)", None)] +#[case::json("{\"key\":1}", None)] #[tokio::test] -async fn native_reads_preserve_strings_and_report_conversion_errors(#[case] managed: bool) { - for raw in ["True", " FALSE ", "", "(True)", "{\"key\":1}"] { - let state = if managed { - SecretManagerState::new( - SecretManager::External(Arc::new(FixedManager::custom(Ok(Some(Secret::String( - SecretValue::new(raw), - )))))), - KeyManagementSettings::default(), - ) - } else { - SecretManagerState::default() - }; - let resolver = SecretResolver::new( - Arc::new(state), - Arc::new(move |_: &str| Some(raw.to_owned())), - OidcResolver::new(litellm_http::Client::for_test()), - ); - assert_eq!( - resolver - .get_secret_str("key", None) - .await - .unwrap() - .unwrap() - .expose(), - raw - ); - assert_eq!( - resolver.get_secret("key", None).await.unwrap(), - Some(Secret::String(SecretValue::new(raw))) - ); - if raw == "True" || raw == " FALSE " { - assert_eq!( - resolver.get_secret_bool("key", None).await.unwrap(), - Some(raw == "True") - ); - } else { - assert!(matches!( - resolver.get_secret_bool("key", None).await, - Err(Error::TypeMismatch { - expected: "boolean" - }) - )); - } +async fn native_reads_preserve_strings_and_report_conversion_errors( + #[case] raw: &'static str, + #[case] boolean: Option, + #[values(false, true)] managed: bool, +) { + let state = if managed { + SecretManagerState::new( + SecretManager::External(Arc::new(FixedManager::custom(Ok(Some(Secret::String( + SecretValue::new(raw), + )))))), + KeyManagementSettings::default(), + ) + } else { + SecretManagerState::default() + }; + let resolver = SecretResolver::new( + Arc::new(state), + Arc::new(move |_: &str| Some(raw.to_owned())), + OidcResolver::new(litellm_http::Client::for_test()), + ); + assert_eq!( + resolver + .get_secret_str("key", None) + .await + .unwrap() + .unwrap() + .expose(), + raw + ); + assert_eq!( + resolver.get_secret("key", None).await.unwrap(), + Some(Secret::String(SecretValue::new(raw))) + ); + match boolean { + Some(expected) => assert_eq!( + resolver.get_secret_bool("key", None).await.unwrap(), + Some(expected) + ), + None => assert!(matches!( + resolver.get_secret_bool("key", None).await, + Err(Error::TypeMismatch { + expected: "boolean" + }) + )), } } +#[rstest::rstest] +#[case::absent(Ok(None))] +#[case::failed(Err(()))] #[tokio::test] -async fn native_defaults_apply_to_absence_but_never_hide_provider_failures() { - for reply in [Ok(None), Err(())] { - let resolver = SecretResolver::new( - Arc::new(SecretManagerState::new( - SecretManager::External(Arc::new(FixedManager::custom(reply.clone()))), - KeyManagementSettings::default(), - )), - Arc::new(|_: &str| None), - OidcResolver::new(litellm_http::Client::for_test()), - ); - let result = resolver - .get_secret_str("key", Some(SecretValue::new("default"))) - .await; - match reply { - Ok(None) => assert_eq!(result.unwrap().unwrap().expose(), "default"), - Err(()) => assert!(matches!(result, Err(Error::MissingCiphertext))), - Ok(Some(_)) => unreachable!(), - } +async fn native_defaults_apply_to_absence_but_never_hide_provider_failures( + #[case] reply: Result, ()>, +) { + let resolver = SecretResolver::new( + Arc::new(SecretManagerState::new( + SecretManager::External(Arc::new(FixedManager::custom(reply.clone()))), + KeyManagementSettings::default(), + )), + Arc::new(|_: &str| None), + OidcResolver::new(litellm_http::Client::for_test()), + ); + let result = resolver + .get_secret_str("key", Some(SecretValue::new("default"))) + .await; + match reply { + Ok(None) => assert_eq!(result.unwrap().unwrap().expose(), "default"), + Err(()) => assert!(matches!(result, Err(Error::MissingCiphertext))), + Ok(Some(_)) => unreachable!(), } } diff --git a/litellm-rust/crates/spend-clickhouse/Cargo.toml b/litellm-rust/crates/spend-clickhouse/Cargo.toml index 893c5daded3..ef7f4a0efdf 100644 --- a/litellm-rust/crates/spend-clickhouse/Cargo.toml +++ b/litellm-rust/crates/spend-clickhouse/Cargo.toml @@ -7,6 +7,7 @@ repository.workspace = true [dependencies] flate2.workspace = true +litellm-error.workspace = true litellm-http.workspace = true litellm-storage-clickhouse.workspace = true serde.workspace = true diff --git a/litellm-rust/crates/spend-clickhouse/src/error.rs b/litellm-rust/crates/spend-clickhouse/src/error.rs index 0d037ef06f8..c5a9f542ca6 100644 --- a/litellm-rust/crates/spend-clickhouse/src/error.rs +++ b/litellm-rust/crates/spend-clickhouse/src/error.rs @@ -1,15 +1,32 @@ +use litellm_error::Source; + #[derive(Debug, thiserror::Error)] pub enum Error { - #[error("invalid ClickHouse insert row")] - InvalidRow, - #[error("{0} must be a positive integer")] - InvalidLimit(&'static str), - #[error("database must be a nonempty SQL identifier and retention must be positive")] - InvalidSchema, - #[error("ClickHouse insert exceeds the encoded size limit")] - InsertTooLarge, + #[error("{field} must be an integer millisecond timestamp")] + NonIntegerTimestamp { field: &'static str }, + #[error("{field} is outside the supported timestamp range")] + TimestampOutOfRange { field: &'static str, source: Source }, #[error(transparent)] Storage(#[from] litellm_storage_clickhouse::Error), #[error(transparent)] Migration(#[from] sqlx::migrate::MigrateError), } + +impl Error { + pub fn is_request(&self) -> bool { + match self { + Self::NonIntegerTimestamp { .. } | Self::TimestampOutOfRange { .. } => true, + Self::Storage(error) => error.is_request(), + Self::Migration(_) => false, + } + } + + pub fn is_insert_too_large(&self) -> bool { + match self { + Self::Storage(error) => error.is_insert_too_large(), + Self::NonIntegerTimestamp { .. } + | Self::TimestampOutOfRange { .. } + | Self::Migration(_) => false, + } + } +} diff --git a/litellm-rust/crates/spend-clickhouse/src/insert.rs b/litellm-rust/crates/spend-clickhouse/src/insert.rs index 495f0d8ecb2..66d0ea59536 100644 --- a/litellm-rust/crates/spend-clickhouse/src/insert.rs +++ b/litellm-rust/crates/spend-clickhouse/src/insert.rs @@ -13,7 +13,8 @@ use sha2::{Digest, Sha256}; use time::{OffsetDateTime, format_description::well_known::Rfc3339}; use crate::Error; -use litellm_storage_clickhouse::Connection; +use litellm_error::Source; +use litellm_storage_clickhouse::{Connection, Error as StorageError}; fn max_insert_bytes() -> Result { let name = "CLICKHOUSE_TRACE_MAX_INSERT_BYTES"; @@ -22,9 +23,9 @@ fn max_insert_bytes() -> Result { .parse::() .ok() .filter(|value| *value > 0) - .ok_or(Error::InvalidLimit(name)), + .ok_or(StorageError::InvalidLimit(name).into()), Err(std::env::VarError::NotPresent) => Ok(64 * 1024 * 1024), - Err(_) => Err(Error::InvalidLimit(name)), + Err(_) => Err(StorageError::InvalidLimit(name).into()), } } @@ -55,7 +56,11 @@ pub async fn insert_rows( pub fn encode_rows(rows: Vec) -> Result { let body = write_rows(&rows, None, Vec::new(), usize::MAX)?; - String::from_utf8(body).map_err(|_| Error::InvalidRow) + String::from_utf8(body).map_err(invalid_row) +} + +fn invalid_row(error: impl std::error::Error + Send + Sync + 'static) -> Error { + StorageError::InvalidRow(Source::new(error)).into() } fn prepare_insert( @@ -73,9 +78,9 @@ fn prepare_insert( )?; let body = encoder .into_inner() - .map_err(|_| Error::InvalidRow)? + .map_err(|error| invalid_row(error.into_error()))? .finish() - .map_err(|_| Error::InvalidRow)?; + .map_err(invalid_row)?; Ok((token, body)) } @@ -102,7 +107,7 @@ impl Write for LimitedWriter { fn write(&mut self, bytes: &[u8]) -> std::io::Result { if bytes.len() > self.remaining { self.exceeded = true; - return Err(std::io::Error::other(Error::InsertTooLarge)); + return Err(std::io::Error::other(StorageError::InsertTooLarge)); } let written = self.inner.write(bytes)?; self.remaining -= written; @@ -126,17 +131,18 @@ fn write_rows( exceeded: false, }; for (index, row) in rows.iter().enumerate() { + let row = EncodedRow::new(row, received_ms)?; let result = (|| { if index != 0 { writer.write_all(b"\n").map_err(serde_json::Error::io)?; } - serde_json::to_writer(&mut writer, &EncodedRow { row, received_ms }) + serde_json::to_writer(&mut writer, &row) })(); - if result.is_err() { + if let Err(error) = result { return Err(if writer.exceeded { - Error::InsertTooLarge + StorageError::InsertTooLarge.into() } else { - Error::InvalidRow + invalid_row(error) }); } } @@ -144,25 +150,37 @@ fn write_rows( } struct EncodedRow<'a> { - row: &'a InsertRow, + fields: Vec<(&'a str, Cow<'a, Value>)>, received_ms: Option, } +impl<'a> EncodedRow<'a> { + fn new(row: &'a InsertRow, received_ms: Option) -> Result { + let fields = row + .iter() + .map(|(name, value)| Ok((name.as_str(), insert_value(name, value)?))) + .collect::>()?; + Ok(Self { + fields, + received_ms, + }) + } +} + impl Serialize for EncodedRow<'_> { fn serialize(&self, serializer: S) -> Result { let mut map = serializer.serialize_map(None)?; let mut received_ms = self.received_ms; - for (name, value) in self.row { - if name.as_str() >= "EngineReceivedMs" + for (name, value) in &self.fields { + if *name >= "EngineReceivedMs" && let Some(timestamp) = received_ms.take() { map.serialize_entry("EngineReceivedMs", ×tamp)?; } - if name == "EngineReceivedMs" && self.received_ms.is_some() { + if *name == "EngineReceivedMs" && self.received_ms.is_some() { continue; } - let value = insert_value(name, value).map_err(serde::ser::Error::custom)?; - map.serialize_entry(name, &value)?; + map.serialize_entry(name, value)?; } if let Some(timestamp) = received_ms { map.serialize_entry("EngineReceivedMs", ×tamp)?; @@ -171,21 +189,23 @@ impl Serialize for EncodedRow<'_> { } } +const TIMESTAMP_FIELDS: [&str; 3] = ["start_time", "end_time", "completion_start_time"]; + fn insert_value<'a>(name: &str, value: &'a Value) -> Result, Error> { - let multiplier = match name { - "start_time" | "end_time" | "completion_start_time" => 1_000_000, - _ => return Ok(Cow::Borrowed(value)), + let Some(&field) = TIMESTAMP_FIELDS.iter().find(|field| **field == name) else { + return Ok(Cow::Borrowed(value)); }; - if name == "completion_start_time" && value.is_null() { + if field == "completion_start_time" && value.is_null() { return Ok(Cow::Borrowed(value)); } - let timestamp = value.as_i64().ok_or(Error::InvalidRow)?; - let datetime = OffsetDateTime::from_unix_timestamp_nanos(i128::from(timestamp) * multiplier) - .map_err(|_| Error::InvalidRow)?; + let out_of_range = |source| Error::TimestampOutOfRange { field, source }; + let timestamp = value.as_i64().ok_or(Error::NonIntegerTimestamp { field })?; + let datetime = OffsetDateTime::from_unix_timestamp_nanos(i128::from(timestamp) * 1_000_000) + .map_err(|error| out_of_range(Source::new(error)))?; datetime .format(&Rfc3339) .map(|value| Cow::Owned(Value::String(value))) - .map_err(|_| Error::InvalidRow) + .map_err(|error| out_of_range(Source::new(error))) } #[cfg(test)] @@ -204,10 +224,11 @@ mod tests { ]; let encoded = write_rows(&rows, None, Vec::new(), usize::MAX).unwrap(); assert!(write_rows(&rows, None, Vec::new(), encoded.len()).is_ok()); - assert!(matches!( - write_rows(&rows, None, Vec::new(), encoded.len() - 1), - Err(Error::InsertTooLarge) - )); + assert!( + write_rows(&rows, None, Vec::new(), encoded.len() - 1) + .unwrap_err() + .is_insert_too_large() + ); } #[rstest] @@ -264,10 +285,11 @@ mod tests { fn stamped_insert_enforces_the_encoded_limit() { let rows = vec![BTreeMap::new()]; assert!(prepare_insert(&rows, 1, 22).is_ok()); - assert!(matches!( - prepare_insert(&rows, 1, 21), - Err(Error::InsertTooLarge) - )); + assert!( + prepare_insert(&rows, 1, 21) + .unwrap_err() + .is_insert_too_large() + ); } #[rstest] @@ -282,15 +304,36 @@ mod tests { assert_eq!(insert_value(field, &input).unwrap().as_ref(), &expected); } + fn rejected_start_time(input: Value) -> Error { + prepare_insert(&[BTreeMap::from([("start_time".into(), input)])], 456, 4096).unwrap_err() + } + #[rstest] #[case::fractional(json!(1.5))] #[case::string(json!("1234"))] #[case::null(Value::Null)] - #[case::out_of_range(json!(i64::MAX))] - fn invalid_spend_dates_fail_before_transport(#[case] input: Value) { + fn non_integer_spend_dates_fail_before_transport(#[case] input: Value) { + let error = rejected_start_time(input); + assert!(error.is_request()); assert!(matches!( - prepare_insert(&[BTreeMap::from([("start_time".into(), input)])], 456, 4096), - Err(Error::InvalidRow) + error, + Error::NonIntegerTimestamp { + field: "start_time" + } )); } + + #[rstest] + fn out_of_range_spend_dates_keep_their_cause() { + let error = rejected_start_time(json!(i64::MAX)); + assert!(error.is_request()); + let cause = std::error::Error::source(&error) + .and_then(|cause| cause.downcast_ref::()) + .expect("time cause"); + assert!( + cause + .downcast_ref::() + .is_some() + ); + } } diff --git a/litellm-rust/crates/spend-clickhouse/src/lib.rs b/litellm-rust/crates/spend-clickhouse/src/lib.rs index f4c108b1e7f..9da658aa3e8 100644 --- a/litellm-rust/crates/spend-clickhouse/src/lib.rs +++ b/litellm-rust/crates/spend-clickhouse/src/lib.rs @@ -4,7 +4,7 @@ mod schema; pub use error::Error; pub use insert::{encode_rows, insert_rows}; -use litellm_storage_clickhouse::Storage; +use litellm_storage_clickhouse::{Error as StorageError, Storage}; pub use schema::ensure_schema; #[derive(Clone)] @@ -16,7 +16,7 @@ pub struct Config { impl Config { pub fn new(database: String, url: &str, retention_days: u32) -> Result { if retention_days == 0 { - return Err(Error::InvalidSchema); + return Err(StorageError::InvalidSchema.into()); } Ok(Self { storage: Storage::new(database, url)?, diff --git a/litellm-rust/crates/storage-clickhouse/Cargo.toml b/litellm-rust/crates/storage-clickhouse/Cargo.toml index 39d04f2de5d..8b985fc40d8 100644 --- a/litellm-rust/crates/storage-clickhouse/Cargo.toml +++ b/litellm-rust/crates/storage-clickhouse/Cargo.toml @@ -8,6 +8,7 @@ repository.workspace = true [dependencies] flate2.workspace = true +litellm-error.workspace = true litellm-http.workspace = true serde.workspace = true serde_json.workspace = true @@ -17,6 +18,7 @@ thiserror.workspace = true url.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-http = { workspace = true, features = ["test-support"] } rstest.workspace = true testcontainers-modules = { workspace = true, features = ["clickhouse"] } diff --git a/litellm-rust/crates/storage-clickhouse/src/error.rs b/litellm-rust/crates/storage-clickhouse/src/error.rs index fa0c5eabd73..ef7b5046455 100644 --- a/litellm-rust/crates/storage-clickhouse/src/error.rs +++ b/litellm-rust/crates/storage-clickhouse/src/error.rs @@ -1,21 +1,23 @@ +use litellm_error::Source; + #[derive(Clone, Debug, thiserror::Error)] pub enum Error { #[error("invalid ClickHouse insert row")] - InvalidRow, + InvalidRow(#[source] Source), #[error("{0} must be a positive integer")] InvalidLimit(&'static str), #[error("invalid ClickHouse insert table")] InvalidTable, #[error("invalid ClickHouse HTTP URL")] - InvalidUrl, + InvalidUrl(#[source] url::ParseError), + #[error("ClickHouse URL must use http or https and name a host")] + UnsupportedUrl, #[error("database must be a nonempty SQL identifier and retention must be positive")] InvalidSchema, #[error("SQL query must not be empty")] EmptySql, #[error("invalid ClickHouse query parameters")] - InvalidParameters, - #[error("unknown ClickHouse read query")] - InvalidQuery, + InvalidParameters(#[source] Source), #[error("ClickHouse query failed with HTTP status {0}")] QueryFailed(u16), #[error("ClickHouse insert failed with HTTP status {0}")] @@ -24,10 +26,60 @@ pub enum Error { InsertTooLarge, #[error("ClickHouse schema setup failed with HTTP status {0}")] SchemaFailed(u16), + #[error("ClickHouse migrations are forward-only")] + Irreversible, #[error("ClickHouse query exceeded the response size limit")] ResponseTooLarge, - #[error("ClickHouse returned an invalid or failed JSON query response")] - InvalidResponse, - #[error("ClickHouse query transport failed")] - Transport, + #[error("ClickHouse returned an invalid response")] + InvalidResponse(#[source] Source), + #[error("ClickHouse returned a failed or unexpected response")] + UnexpectedResponse, + #[error("ClickHouse transport failed")] + Transport(#[from] litellm_http::Error), +} + +impl Error { + pub fn is_request(&self) -> bool { + match self { + Self::InvalidRow(_) + | Self::InvalidLimit(_) + | Self::InvalidTable + | Self::InvalidSchema + | Self::EmptySql + | Self::InvalidParameters(_) => true, + Self::InvalidUrl(_) + | Self::UnsupportedUrl + | Self::QueryFailed(_) + | Self::InsertFailed(_) + | Self::InsertTooLarge + | Self::SchemaFailed(_) + | Self::Irreversible + | Self::ResponseTooLarge + | Self::InvalidResponse(_) + | Self::UnexpectedResponse + | Self::Transport(_) => false, + } + } + + pub fn is_insert_too_large(&self) -> bool { + match self { + Self::InsertTooLarge => true, + Self::InvalidRow(_) + | Self::InvalidLimit(_) + | Self::InvalidTable + | Self::InvalidUrl(_) + | Self::UnsupportedUrl + | Self::InvalidSchema + | Self::EmptySql + | Self::InvalidParameters(_) + | Self::QueryFailed(_) + | Self::InsertFailed(_) + | Self::SchemaFailed(_) + | Self::Irreversible + | Self::ResponseTooLarge + | Self::InvalidResponse(_) + | Self::UnexpectedResponse + | Self::Transport(_) => false, + } + } } diff --git a/litellm-rust/crates/storage-clickhouse/src/insert.rs b/litellm-rust/crates/storage-clickhouse/src/insert.rs index 1cdad1897c1..29837359f0b 100644 --- a/litellm-rust/crates/storage-clickhouse/src/insert.rs +++ b/litellm-rust/crates/storage-clickhouse/src/insert.rs @@ -1,6 +1,7 @@ use std::{io::Write, time::Duration}; use flate2::{Compression, write::GzEncoder}; +use litellm_error::Source; use litellm_http::Client; use crate::{Connection, Error, valid_identifier}; @@ -34,13 +35,15 @@ pub async fn insert_encoded_rows( return Err(Error::InvalidTable); } let mut encoder = GzEncoder::new(Vec::new(), Compression::default()); - encoder - .write_all(encoded.as_bytes()) - .map_err(|_| Error::InvalidRow)?; - let body = encoder.finish().map_err(|_| Error::InvalidRow)?; + encoder.write_all(encoded.as_bytes()).map_err(invalid_row)?; + let body = encoder.finish().map_err(invalid_row)?; insert_compressed_rows(client, connection, database, table, token, body).await } +fn invalid_row(error: std::io::Error) -> Error { + Error::InvalidRow(Source::new(error)) +} + pub async fn insert_compressed_rows( client: &Client, connection: &Connection, @@ -91,7 +94,7 @@ pub async fn insert_compressed_rows( .body(body) .send() .await - .map_err(|_| Error::Transport)?; + .map_err(litellm_http::Error::from)?; if !response.status().is_success() { return Err(Error::InsertFailed(response.status().as_u16())); } diff --git a/litellm-rust/crates/storage-clickhouse/src/lib.rs b/litellm-rust/crates/storage-clickhouse/src/lib.rs index e2ef62d1bc3..60a4e77a36c 100644 --- a/litellm-rust/crates/storage-clickhouse/src/lib.rs +++ b/litellm-rust/crates/storage-clickhouse/src/lib.rs @@ -16,9 +16,9 @@ pub struct Connection { impl Connection { pub fn parse(value: &str) -> Result { - let url = Url::parse(value).map_err(|_| Error::InvalidUrl)?; + let url = Url::parse(value).map_err(Error::InvalidUrl)?; if !matches!(url.scheme(), "http" | "https") || url.host().is_none() { - return Err(Error::InvalidUrl); + return Err(Error::UnsupportedUrl); } Ok(Self { url }) } @@ -33,11 +33,11 @@ impl Connection { connection .url .set_username(user) - .map_err(|_| Error::InvalidUrl)?; + .map_err(|()| Error::UnsupportedUrl)?; connection .url .set_password(Some(password)) - .map_err(|_| Error::InvalidUrl)?; + .map_err(|()| Error::UnsupportedUrl)?; let pairs: Vec<_> = connection .url .query_pairs() diff --git a/litellm-rust/crates/storage-clickhouse/src/migrate.rs b/litellm-rust/crates/storage-clickhouse/src/migrate.rs index c249020faaf..4da26baa681 100644 --- a/litellm-rust/crates/storage-clickhouse/src/migrate.rs +++ b/litellm-rust/crates/storage-clickhouse/src/migrate.rs @@ -12,7 +12,7 @@ use sqlx::{ migrate::{AppliedMigration, Migrate, MigrateError, Migration}, }; -use crate::{Connection, Error, READ_LIMITS, valid_identifier}; +use crate::{Connection, Error, READ_LIMITS, read::invalid_response, valid_identifier}; pub async fn execute_statement( client: &Client, @@ -22,7 +22,7 @@ pub async fn execute_statement( ) -> Result<(), Error> { let body = execute_sql(client, connection, sql, timeout).await?; if !body.trim().is_empty() { - return Err(Error::InvalidResponse); + return Err(Error::UnexpectedResponse); } Ok(()) } @@ -56,18 +56,18 @@ async fn execute_sql( .body(sql.to_owned()) .send() .await - .map_err(|_| Error::Transport)?; + .map_err(litellm_http::Error::from)?; if !response.status().is_success() { return Err(Error::SchemaFailed(response.status().as_u16())); } let mut body = Vec::new(); - while let Some(chunk) = response.chunk().await.map_err(|_| Error::Transport)? { + while let Some(chunk) = response.chunk().await.map_err(litellm_http::Error::from)? { if body.len() + chunk.len() > READ_LIMITS.response_bytes { return Err(Error::ResponseTooLarge); } body.extend_from_slice(&chunk); } - String::from_utf8(body).map_err(|_| Error::InvalidResponse) + String::from_utf8(body).map_err(invalid_response) } /// The startup runner records success after execution, never marks migrations dirty, and skips locks @@ -131,13 +131,13 @@ fn encode_hex(bytes: &[u8]) -> String { fn decode_hex(value: &str) -> Result, Error> { let (pairs, remainder) = value.as_bytes().as_chunks::<2>(); if !remainder.is_empty() { - return Err(Error::InvalidResponse); + return Err(Error::UnexpectedResponse); } pairs .iter() .map(|pair| { - let high = decode_hex_digit(pair[0]).ok_or(Error::InvalidResponse)?; - let low = decode_hex_digit(pair[1]).ok_or(Error::InvalidResponse)?; + let high = decode_hex_digit(pair[0]).ok_or(Error::UnexpectedResponse)?; + let low = decode_hex_digit(pair[1]).ok_or(Error::UnexpectedResponse)?; Ok((high << 4) | low) }) .collect() @@ -231,7 +231,7 @@ where .filter(|line| !line.trim().is_empty()) .map(|line| { let row = serde_json::from_str::(line) - .map_err(|_| migrate_error(Error::InvalidResponse))?; + .map_err(|error| migrate_error(invalid_response(error)))?; let checksum = decode_hex(&row.checksum).map_err(migrate_error)?; Ok(AppliedMigration { version: row.version, @@ -288,11 +288,7 @@ where _table_name: &'e str, _migration: &'e Migration, ) -> MigrateFuture<'e, Result> { - Box::pin(async { - Err(MigrateError::Execute(SqlxError::AnyDriverError(Box::new( - std::io::Error::other("ClickHouse migrations are forward-only"), - )))) - }) + Box::pin(async { Err(migrate_error(Error::Irreversible)) }) } } diff --git a/litellm-rust/crates/storage-clickhouse/src/read.rs b/litellm-rust/crates/storage-clickhouse/src/read.rs index 03a5522e595..40d4ad0a073 100644 --- a/litellm-rust/crates/storage-clickhouse/src/read.rs +++ b/litellm-rust/crates/storage-clickhouse/src/read.rs @@ -1,5 +1,6 @@ use std::{collections::BTreeMap, time::Duration}; +use litellm_error::Source; use litellm_http::Client; use serde::{Deserialize, Serialize, de::DeserializeOwned}; @@ -115,7 +116,7 @@ async fn execute_read_with_limits( .post(url) .timeout(Duration::from_secs(15)) .body(sql.to_owned()); - let mut response = request.send().await.map_err(|_| Error::Transport)?; + let mut response = request.send().await.map_err(litellm_http::Error::from)?; if !response.status().is_success() { if response .headers() @@ -128,20 +129,19 @@ async fn execute_read_with_limits( } let mut body = Vec::new(); - while let Some(chunk) = response.chunk().await.map_err(|_| Error::Transport)? { + while let Some(chunk) = response.chunk().await.map_err(litellm_http::Error::from)? { if body.len() + chunk.len() > limits.response_bytes { return Err(Error::ResponseTooLarge); } body.extend_from_slice(&chunk); } - let json: serde_json::Value = - serde_json::from_slice(&body).map_err(|_| Error::InvalidResponse)?; + let json: serde_json::Value = serde_json::from_slice(&body).map_err(invalid_response)?; if json.get("exception").is_some() || !json.get("data").is_some_and(serde_json::Value::is_array) { - return Err(Error::InvalidResponse); + return Err(Error::UnexpectedResponse); } - String::from_utf8(body).map_err(|_| Error::InvalidResponse) + String::from_utf8(body).map_err(invalid_response) } pub trait Query { @@ -158,8 +158,8 @@ struct Rows { } fn parameters(params: &T) -> Result, Error> { - let value = serde_json::to_value(params).map_err(|_| Error::InvalidParameters)?; - serde_json::from_value(value).map_err(|_| Error::InvalidParameters) + let value = serde_json::to_value(params).map_err(invalid_parameters)?; + serde_json::from_value(value).map_err(invalid_parameters) } pub async fn fetch( @@ -198,5 +198,13 @@ pub async fn fetch_json( fn decode_rows(body: &str) -> Result, Error> { serde_json::from_str::>(body) .map(|rows| rows.data) - .map_err(|_| Error::InvalidResponse) + .map_err(invalid_response) +} + +pub(crate) fn invalid_response(error: impl std::error::Error + Send + Sync + 'static) -> Error { + Error::InvalidResponse(Source::new(error)) +} + +fn invalid_parameters(error: serde_json::Error) -> Error { + Error::InvalidParameters(Source::new(error)) } diff --git a/litellm-rust/crates/storage-clickhouse/tests/migrations.rs b/litellm-rust/crates/storage-clickhouse/tests/migrations.rs index 49f481df632..09e56766891 100644 --- a/litellm-rust/crates/storage-clickhouse/tests/migrations.rs +++ b/litellm-rust/crates/storage-clickhouse/tests/migrations.rs @@ -346,11 +346,7 @@ async fn revert_reports_forward_only_error() { .revert("_sqlx_migrations", &migration) .await .expect_err("ClickHouse migrations cannot be reverted"); - assert!( - error - .to_string() - .contains("ClickHouse migrations are forward-only") - ); + assert!(matches!(storage_error(&error), Some(Error::Irreversible))); } #[rstest] @@ -391,6 +387,7 @@ async fn applied_int64_versions_accept_numeric_and_quoted_json( #[tokio::test] async fn schema_requests_override_unsafe_connection_settings( #[future(awt)] mock_server: MockServer, + #[values("wait_end_of_query", "send_progress_in_http_headers", "async_insert")] name: &str, ) { Mock::given(method("POST")) .and(query_param("wait_end_of_query", "1")) @@ -417,20 +414,14 @@ async fn schema_requests_override_unsafe_connection_settings( .received_requests() .await .expect("requests recorded"); - for name in [ - "wait_end_of_query", - "send_progress_in_http_headers", - "async_insert", - ] { - assert_eq!( - requests[0] - .url - .query_pairs() - .filter(|(key, _)| key == name) - .count(), - 1 - ); - } + assert_eq!( + requests[0] + .url + .query_pairs() + .filter(|(key, _)| key == name) + .count(), + 1 + ); } #[rstest] diff --git a/litellm-rust/crates/storage-clickhouse/tests/transport.rs b/litellm-rust/crates/storage-clickhouse/tests/transport.rs index 2560c2985f0..6b2c3fb8ef1 100644 --- a/litellm-rust/crates/storage-clickhouse/tests/transport.rs +++ b/litellm-rust/crates/storage-clickhouse/tests/transport.rs @@ -55,18 +55,24 @@ impl litellm_storage_clickhouse::Query for TypedQuery { const SQL: &'static str = "SELECT typed_parameters"; } +enum Expected { + Rows, + Undecodable, + Failed, +} + #[rstest] #[case::valid( r#"{"meta":[],"data":[{"answer":"ok"}],"rows":1,"statistics":{"elapsed":0.1}}"#, - true + Expected::Rows )] -#[case::wrong_type(r#"{"data":[{"answer":1}]}"#, false)] -#[case::missing_column(r#"{"data":[{}]}"#, false)] -#[case::exception(r#"{"data":[],"exception":"failed"}"#, false)] +#[case::wrong_type(r#"{"data":[{"answer":1}]}"#, Expected::Undecodable)] +#[case::missing_column(r#"{"data":[{}]}"#, Expected::Undecodable)] +#[case::exception(r#"{"data":[],"exception":"failed"}"#, Expected::Failed)] #[tokio::test] async fn typed_fetch_encodes_parameters_and_validates_rows( #[case] body: &str, - #[case] valid: bool, + #[case] expected: Expected, ) { use litellm_storage_clickhouse::{fetch, fetch_json}; use wiremock::{ @@ -98,17 +104,24 @@ async fn typed_fetch_encodes_parameters_and_validates_rows( }; let rows = fetch::(&client, &connection, ¶ms).await; let envelope = fetch_json::(&client, &connection, ¶ms).await; - if valid { - assert_eq!( - rows.unwrap(), - vec![QueryRow { - answer: "ok".into() - }] - ); - assert_eq!(envelope.unwrap(), body); - } else { - assert!(matches!(rows, Err(Error::InvalidResponse))); - assert!(matches!(envelope, Err(Error::InvalidResponse))); + match expected { + Expected::Rows => { + assert_eq!( + rows.unwrap(), + vec![QueryRow { + answer: "ok".into() + }] + ); + assert_eq!(envelope.unwrap(), body); + } + Expected::Undecodable => { + assert!(matches!(rows, Err(Error::InvalidResponse(_)))); + assert!(matches!(envelope, Err(Error::InvalidResponse(_)))); + } + Expected::Failed => { + assert!(matches!(rows, Err(Error::UnexpectedResponse))); + assert!(matches!(envelope, Err(Error::UnexpectedResponse))); + } } } @@ -145,21 +158,23 @@ async fn server_result_limits_allow_smaller_pages_without_retrying_other_failure } } -#[test] -fn insert_timeout_environment_controls_transport() { - for value in ["1", "3", "0", "invalid"] { - let result = std::process::Command::new(std::env::current_exe().unwrap()) - .args(["--exact", "insert_timeout_environment_child"]) - .env("LITELLM_TEST_INSERT_TIMEOUT", value) - .env("CLICKHOUSE_INSERT_TIMEOUT_SECONDS", value) - .output() - .unwrap(); - assert!( - result.status.success(), - "{}", - String::from_utf8_lossy(&result.stdout) - ); - } +#[rstest] +#[case::times_out("1")] +#[case::completes("3")] +#[case::zero("0")] +#[case::not_a_number("invalid")] +fn insert_timeout_environment_controls_transport(#[case] value: &str) { + let result = std::process::Command::new(std::env::current_exe().unwrap()) + .args(["--exact", "insert_timeout_environment_child"]) + .env("LITELLM_TEST_INSERT_TIMEOUT", value) + .env("CLICKHOUSE_INSERT_TIMEOUT_SECONDS", value) + .output() + .unwrap(); + assert!( + result.status.success(), + "{}", + String::from_utf8_lossy(&result.stdout) + ); } #[tokio::test] @@ -183,7 +198,7 @@ async fn insert_timeout_environment_child() { ) .await; match value.as_str() { - "1" => assert!(matches!(result, Err(Error::Transport))), + "1" => assert!(matches!(result, Err(Error::Transport(error)) if error.dispatched())), "3" => assert!(result.is_ok()), _ => assert!(matches!( result, @@ -191,3 +206,29 @@ async fn insert_timeout_environment_child() { )), } } + +#[rstest] +#[tokio::test] +async fn unreachable_server_keeps_the_transport_cause() { + let address = std::net::TcpListener::bind("127.0.0.1:0") + .unwrap() + .local_addr() + .unwrap(); + let connection = Connection::parse(&format!("http://{address}")).unwrap(); + let error = execute_read( + &Client::for_test(), + &connection, + "SELECT 1", + &BTreeMap::new(), + ) + .await + .unwrap_err(); + let cause = std::error::Error::source(&error) + .and_then(|cause| cause.downcast_ref::()) + .expect("transport cause"); + assert!(!cause.dispatched()); + assert_ne!( + litellm_error::Report::new(&error).to_string(), + error.to_string() + ); +} diff --git a/litellm-rust/crates/testkit/Cargo.toml b/litellm-rust/crates/testkit/Cargo.toml index 97efb753384..ef3c3e6cd7e 100644 --- a/litellm-rust/crates/testkit/Cargo.toml +++ b/litellm-rust/crates/testkit/Cargo.toml @@ -8,6 +8,7 @@ publish = false [dependencies] flate2.workspace = true +litellm-http.workspace = true reqwest.workspace = true serde.workspace = true semver.workspace = true diff --git a/litellm-rust/crates/testkit/src/agent/configure.rs b/litellm-rust/crates/testkit/src/agent/configure.rs index a41bec9379a..6f1b970e8fb 100644 --- a/litellm-rust/crates/testkit/src/agent/configure.rs +++ b/litellm-rust/crates/testkit/src/agent/configure.rs @@ -5,10 +5,13 @@ use semver::Version; use crate::Error; -#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::VariantArray)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::VariantArray, strum::Display)] pub enum Wire { + #[strum(serialize = "chat completions")] ChatCompletions, + #[strum(serialize = "messages")] Messages, + #[strum(serialize = "responses")] Responses, } diff --git a/litellm-rust/crates/testkit/src/error.rs b/litellm-rust/crates/testkit/src/error.rs index 03520d827f1..8b747c28bda 100644 --- a/litellm-rust/crates/testkit/src/error.rs +++ b/litellm-rust/crates/testkit/src/error.rs @@ -15,7 +15,7 @@ pub enum Error { Request { url: String, #[source] - source: reqwest::Error, + source: litellm_http::Error, }, #[error("{url} answered with status {status}")] Status { url: String, status: u16 }, @@ -47,10 +47,14 @@ pub enum Error { expected: String, reported: String, }, - #[error("{agent} cannot talk to the gateway over {wire:?}")] + #[error("{agent} cannot talk to the gateway over {wire}")] UnsupportedWire { agent: &'static str, wire: Wire }, - #[error("agent did not finish within {0:?}")] + #[error("agent did not finish within {} seconds", .0.as_secs_f64())] Timeout(std::time::Duration), - #[error("io failure")] - Io(#[from] io::Error), + #[error("could not install the binary into {}", dir.display())] + Install { dir: PathBuf, source: io::Error }, + #[error("could not run {}", binary.display())] + Spawn { binary: PathBuf, source: io::Error }, + #[error("could not write agent files under {}", home.display())] + WriteFiles { home: PathBuf, source: io::Error }, } diff --git a/litellm-rust/crates/testkit/src/install/fetch.rs b/litellm-rust/crates/testkit/src/install/fetch.rs index 73008f7a0da..59a61bfaa0a 100644 --- a/litellm-rust/crates/testkit/src/install/fetch.rs +++ b/litellm-rust/crates/testkit/src/install/fetch.rs @@ -38,9 +38,9 @@ impl Fetch for HttpFetch { (Some(token), true) => request.bearer_auth(token), _ => request, }; - let request_error = |source| Error::Request { + let request_error = |source: reqwest::Error| Error::Request { url: url.to_owned(), - source, + source: source.into(), }; let response = request.send().await.map_err(request_error)?; let status = response.status(); diff --git a/litellm-rust/crates/testkit/src/install/mod.rs b/litellm-rust/crates/testkit/src/install/mod.rs index 1104bcec102..73f7deec864 100644 --- a/litellm-rust/crates/testkit/src/install/mod.rs +++ b/litellm-rust/crates/testkit/src/install/mod.rs @@ -52,7 +52,13 @@ impl Installer { version: version.clone(), binary: binary.clone(), }; - if fs::try_exists(&binary).await? && probe_version(&binary, version).await.is_ok() { + let install_error = |source| Error::Install { + dir: dir.clone(), + source, + }; + if fs::try_exists(&binary).await.map_err(install_error)? + && probe_version(&binary, version).await.is_ok() + { return Ok(installed); } @@ -61,21 +67,23 @@ impl Installer { verify_sha256(&release.asset, &release.sha256, &archive)?; let contents = extract_binary(&release.packaging, &archive)?; - fs::create_dir_all(&dir).await?; + fs::create_dir_all(&dir).await.map_err(install_error)?; let staging = dir.join(format!( ".{}.{}.{}.partial", agent.binary(), std::process::id(), STAGING_COUNTER.fetch_add(1, Ordering::Relaxed) )); - fs::write(&staging, contents).await?; - fs::set_permissions(&staging, std::fs::Permissions::from_mode(0o755)).await?; - fs::rename(&staging, &binary).await?; + fs::write(&staging, contents).await.map_err(install_error)?; + fs::set_permissions(&staging, std::fs::Permissions::from_mode(0o755)) + .await + .map_err(install_error)?; + fs::rename(&staging, &binary).await.map_err(install_error)?; match probe_version(&binary, version).await { Ok(()) => Ok(installed), Err(error) => { - fs::remove_file(&binary).await?; + fs::remove_file(&binary).await.map_err(install_error)?; Err(error) } } @@ -98,7 +106,11 @@ async fn probe_version(binary: &Path, expected: &Version) -> Result<(), Error> { .env("DISABLE_AUTOUPDATER", "1") .stdin(Stdio::null()) .output() - .await?; + .await + .map_err(|source| Error::Spawn { + binary: binary.to_owned(), + source, + })?; let stdout = String::from_utf8_lossy(&output.stdout); if stdout .split_whitespace() diff --git a/litellm-rust/crates/testkit/src/session.rs b/litellm-rust/crates/testkit/src/session.rs index 6b06e756cca..cd5eb08ae2b 100644 --- a/litellm-rust/crates/testkit/src/session.rs +++ b/litellm-rust/crates/testkit/src/session.rs @@ -28,7 +28,11 @@ impl Session { ) -> Result { let home = home.into(); let spec = agent.configure(&installed.version, &settings, &home)?; - spec.write_files(&home)?; + spec.write_files(&home) + .map_err(|source| Error::WriteFiles { + home: home.clone(), + source, + })?; Ok(Self { binary: installed.binary.clone(), home, @@ -55,7 +59,11 @@ impl Session { .output(); let output = timeout(limit, child) .await - .map_err(|_| Error::Timeout(limit))??; + .map_err(|_| Error::Timeout(limit))? + .map_err(|source| Error::Spawn { + binary: self.binary.clone(), + source, + })?; let parsed = agent.parse(&self.version, &String::from_utf8_lossy(&output.stdout)); let failed_silently = !output.status.success() && parsed.errors.is_empty(); Ok(Outcome { diff --git a/litellm-rust/crates/token-counter-fast/Cargo.toml b/litellm-rust/crates/token-counter-fast/Cargo.toml index 5127dc17f22..070bec58519 100644 --- a/litellm-rust/crates/token-counter-fast/Cargo.toml +++ b/litellm-rust/crates/token-counter-fast/Cargo.toml @@ -13,6 +13,7 @@ tokenizers.workspace = true unicode-normalization-alignments = "0.1.12" [dev-dependencies] +litellm-error.workspace = true rand.workspace = true rstest.workspace = true serde.workspace = true diff --git a/litellm-rust/crates/token-counter-fast/src/error.rs b/litellm-rust/crates/token-counter-fast/src/error.rs index d2a706191ac..2cd8d5c810c 100644 --- a/litellm-rust/crates/token-counter-fast/src/error.rs +++ b/litellm-rust/crates/token-counter-fast/src/error.rs @@ -2,16 +2,25 @@ use thiserror::Error as ThisError; #[derive(Debug, ThisError)] pub enum Error { - #[error("failed to load tokenizer: {0}")] + #[error("failed to load tokenizer")] Load(#[source] tokenizers::Error), - #[error("failed to load tokenizer: tiktoken rank file: {0}")] + #[error("failed to load tokenizer from a tiktoken rank file")] Ranks(#[source] RankError), #[error("failed to load tokenizer: Unicode character classes are unavailable")] UnicodeClasses, - #[error("tokenization failed: {0}")] + #[error("tokenization failed")] Encode(#[source] tokenizers::Error), } +impl Error { + pub fn is_load(&self) -> bool { + match self { + Self::Load(_) | Self::Ranks(_) | Self::UnicodeClasses => true, + Self::Encode(_) => false, + } + } +} + #[derive(Debug, ThisError)] pub enum RankError { #[error("rank {0} is reserved")] @@ -20,8 +29,8 @@ pub enum RankError { MissingByte(u8), #[error("line without a rank: {0:?}")] MissingRank(String), - #[error("token is not base64: {0}")] + #[error("token is not base64")] InvalidToken(#[source] base64::DecodeError), - #[error("rank is not an integer: {0}")] + #[error("rank is not an integer")] InvalidRank(#[source] std::num::ParseIntError), } diff --git a/litellm-rust/crates/token-counter-fast/src/tiktoken.rs b/litellm-rust/crates/token-counter-fast/src/tiktoken.rs index f2d4de38f8c..11182e84e8d 100644 --- a/litellm-rust/crates/token-counter-fast/src/tiktoken.rs +++ b/litellm-rust/crates/token-counter-fast/src/tiktoken.rs @@ -254,8 +254,12 @@ mod tests { .err() .expect("invalid rank file"); assert!(matches!(error, Error::Ranks(_))); - assert!(error.to_string().starts_with(&format!( - "failed to load tokenizer: tiktoken rank file: {detail}" - ))); + assert!( + litellm_error::Report::new(&error) + .to_string() + .starts_with(&format!( + "failed to load tokenizer from a tiktoken rank file: {detail}" + )) + ); } } diff --git a/litellm-rust/crates/token-counter-huggingface/src/error.rs b/litellm-rust/crates/token-counter-huggingface/src/error.rs index e7f6260321b..b4e09db253f 100644 --- a/litellm-rust/crates/token-counter-huggingface/src/error.rs +++ b/litellm-rust/crates/token-counter-huggingface/src/error.rs @@ -2,10 +2,21 @@ use thiserror::Error as ThisError; #[derive(Debug, ThisError)] pub enum Error { - #[error("failed to load tokenizer: {0}")] + #[error("failed to load tokenizer")] Load(#[source] tokenizers::Error), - #[error("tokenization failed: {0}")] + #[error("invalid encoding JSON")] + EncodingJson(#[source] serde_json::Error), + #[error("tokenization failed")] Encode(#[source] tokenizers::Error), - #[error("token decoding failed: {0}")] + #[error("token decoding failed")] Decode(#[source] tokenizers::Error), } + +impl Error { + pub fn is_load(&self) -> bool { + match self { + Self::Load(_) | Self::EncodingJson(_) => true, + Self::Encode(_) | Self::Decode(_) => false, + } + } +} diff --git a/litellm-rust/crates/token-counter-huggingface/src/lib.rs b/litellm-rust/crates/token-counter-huggingface/src/lib.rs index 0acea0532f8..c3af9b42767 100644 --- a/litellm-rust/crates/token-counter-huggingface/src/lib.rs +++ b/litellm-rust/crates/token-counter-huggingface/src/lib.rs @@ -13,11 +13,11 @@ pub use tokenizers::{ }; pub fn encoding_from_json(json: &str) -> Result { - serde_json::from_str(json).map_err(|error| Error::Load(error.into())) + serde_json::from_str(json).map_err(Error::EncodingJson) } pub fn encoding_to_json(encoding: &Encoding) -> Result { - serde_json::to_string(encoding).map_err(|error| Error::Load(error.into())) + serde_json::to_string(encoding).map_err(Error::EncodingJson) } pub struct HuggingFaceTokenizer { diff --git a/litellm-rust/crates/token-counter-tiktoken/Cargo.toml b/litellm-rust/crates/token-counter-tiktoken/Cargo.toml index d1789a693a3..bca7345dcd4 100644 --- a/litellm-rust/crates/token-counter-tiktoken/Cargo.toml +++ b/litellm-rust/crates/token-counter-tiktoken/Cargo.toml @@ -7,6 +7,7 @@ repository.workspace = true [dependencies] base64.workspace = true +litellm-error.workspace = true once_cell = "1.21.3" rustc-hash = "2.1.3" thiserror.workspace = true diff --git a/litellm-rust/crates/token-counter-tiktoken/src/error.rs b/litellm-rust/crates/token-counter-tiktoken/src/error.rs index e28cbbfb620..024be00f048 100644 --- a/litellm-rust/crates/token-counter-tiktoken/src/error.rs +++ b/litellm-rust/crates/token-counter-tiktoken/src/error.rs @@ -1,5 +1,50 @@ +use std::{io, num::ParseIntError}; + +use litellm_error::Source; use thiserror::Error as ThisError; +use tiktoken_rs::{DecodeKeyError, EncodeError}; #[derive(Debug, ThisError)] -#[error("unsupported tokenizer: {0}")] -pub struct UnsupportedTokenizer(pub String); +pub enum Error { + #[error("unsupported tokenizer: {0}")] + UnsupportedTokenizer(String), + #[error("could not read tiktoken rank file {file}")] + Read { + file: &'static str, + source: io::Error, + }, + #[error("invalid tiktoken rank file")] + Ranks(#[from] RankError), + #[error("could not build the tiktoken encoder")] + Encoder(#[source] Source), + #[error("tokenization failed")] + Encode(#[source] EncodeError), + #[error("token decoding failed")] + Decode(#[source] DecodeKeyError), +} + +impl Error { + pub fn is_load(&self) -> bool { + match self { + Self::UnsupportedTokenizer(_) + | Self::Read { .. } + | Self::Ranks(_) + | Self::Encoder(_) => true, + Self::Encode(_) | Self::Decode(_) => false, + } + } +} + +#[derive(Debug, ThisError)] +pub enum RankError { + #[error("line without a rank: {0:?}")] + MissingRank(String), + #[error("token is not base64")] + InvalidToken(#[source] base64::DecodeError), + #[error("rank is not an integer")] + InvalidRank(#[source] ParseIntError), + #[error("a rank is assigned to more than one token")] + DuplicateRank, + #[error("byte 0x{0:02X} has no token")] + MissingByte(u8), +} diff --git a/litellm-rust/crates/token-counter-tiktoken/src/lib.rs b/litellm-rust/crates/token-counter-tiktoken/src/lib.rs index 30d0f820da3..c4cb786d9e4 100644 --- a/litellm-rust/crates/token-counter-tiktoken/src/lib.rs +++ b/litellm-rust/crates/token-counter-tiktoken/src/lib.rs @@ -5,8 +5,8 @@ mod ranks; use std::collections::HashSet; -pub use error::UnsupportedTokenizer; -pub use ranks::{LoadError, Vocabulary}; +pub use error::{Error, RankError}; +pub use ranks::Vocabulary; pub struct TiktokenTokenizer { encoder: &'static tiktoken_rs::CoreBPE, @@ -22,7 +22,7 @@ impl TiktokenTokenizer { pub fn from_cached_ranks( name: &str, load: impl FnOnce(&str) -> std::io::Result, - ) -> Result { + ) -> Result { let (loaded, name) = ranks::load(name, load)?; Ok(Self { encoder: &loaded.bpe, @@ -32,7 +32,7 @@ impl TiktokenTokenizer { } /// The encodings tiktoken-rs embeds, for hosts without the packaged rank files. - pub fn from_name(name: &str) -> Result { + pub fn from_name(name: &str) -> Result { let (encoder, name) = match name { "cl100k_base" => (tiktoken_rs::cl100k_base_singleton(), "cl100k_base"), "o200k_base" => (tiktoken_rs::o200k_base_singleton(), "o200k_base"), @@ -41,7 +41,7 @@ impl TiktokenTokenizer { "p50k_edit" => (tiktoken_rs::p50k_edit_singleton(), "p50k_edit"), "r50k_base" => (tiktoken_rs::r50k_base_singleton(), "r50k_base"), "gpt2" => (tiktoken_rs::r50k_base_singleton(), "gpt2"), - _ => return Err(UnsupportedTokenizer(name.to_owned())), + _ => return Err(Error::UnsupportedTokenizer(name.to_owned())), }; Ok(Self { encoder, @@ -62,12 +62,12 @@ impl TiktokenTokenizer { self.encoder.encode_ordinary(text) } - pub fn encode_special(&self, text: &str, allowed: &[String]) -> Result, String> { + pub fn encode_special(&self, text: &str, allowed: &[String]) -> Result, Error> { let allowed = allowed.iter().map(String::as_str).collect(); self.encoder .encode(text, &allowed) .map(|(ids, _)| ids) - .map_err(|error| error.to_string()) + .map_err(Error::Encode) } pub fn special_tokens(&self) -> HashSet { @@ -92,17 +92,13 @@ impl TiktokenTokenizer { (stable, completions) } - pub fn decode_bytes(&self, ids: &[u32]) -> Result, String> { - self.encoder - .decode_bytes(ids) - .map_err(|error| error.to_string()) + pub fn decode_bytes(&self, ids: &[u32]) -> Result, Error> { + self.encoder.decode_bytes(ids).map_err(Error::Decode) } - pub fn decode(&self, ids: &[u32]) -> Result { - self.encoder - .decode_bytes(ids) + pub fn decode(&self, ids: &[u32]) -> Result { + self.decode_bytes(ids) .map(|bytes| String::from_utf8_lossy(&bytes).into_owned()) - .map_err(|error| error.to_string()) } pub fn name(&self) -> &str { @@ -161,7 +157,8 @@ mod tests { #[test] fn unsupported_encoding_preserves_its_name() { - let Err(UnsupportedTokenizer(name)) = TiktokenTokenizer::from_name("unknown-encoding") + let Err(Error::UnsupportedTokenizer(name)) = + TiktokenTokenizer::from_name("unknown-encoding") else { panic!("unknown encoding must be rejected"); }; diff --git a/litellm-rust/crates/token-counter-tiktoken/src/ranks.rs b/litellm-rust/crates/token-counter-tiktoken/src/ranks.rs index 192e6f65a0b..36c30e711c1 100644 --- a/litellm-rust/crates/token-counter-tiktoken/src/ranks.rs +++ b/litellm-rust/crates/token-counter-tiktoken/src/ranks.rs @@ -1,10 +1,10 @@ use base64::{Engine, engine::general_purpose::STANDARD}; +use litellm_error::Source; use once_cell::sync::OnceCell; use rustc_hash::FxHashMap; -use thiserror::Error; use tiktoken_rs::{CoreBPE, O200K_BASE_PAT_STR, Rank}; -use crate::UnsupportedTokenizer; +use crate::error::{Error, RankError}; const CL100K: &str = "9b5ad71b2ce5302211f9c61530b329a4922fc6a4"; const O200K: &str = "fb374d419588a4632f3f557e76b4b70aebbca790"; @@ -76,22 +76,30 @@ impl Vocabulary { } } -#[derive(Debug, Error)] -pub enum LoadError { - #[error(transparent)] - Unsupported(#[from] UnsupportedTokenizer), - #[error("failed to load tiktoken ranks: {0}")] - Ranks(String), -} - /// Loads `name` once per process. The returned name is the one requested (`gpt2` stays /// `gpt2`, as `tiktoken.get_encoding("gpt2").name` does), while `gpt2` and `r50k_base` share /// one cached encoder. pub(super) fn load( name: &str, load_file: impl FnOnce(&str) -> std::io::Result, -) -> Result<(&'static Loaded, &'static str), LoadError> { - let (requested, canonical, file, cache) = match name { +) -> Result<(&'static Loaded, &'static str), Error> { + let (requested, canonical, file, cache) = encoding(name)?; + let loaded = cache.get_or_try_init(|| { + let ranks = load_file(file).map_err(|source| Error::Read { file, source })?; + build(canonical, &ranks) + })?; + Ok((loaded, requested)) +} + +type Encoding = ( + &'static str, + &'static str, + &'static str, + &'static OnceCell, +); + +fn encoding(name: &str) -> Result { + Ok(match name { "cl100k_base" => ("cl100k_base", "cl100k_base", CL100K, &CL100K_ENCODER), "o200k_base" => ("o200k_base", "o200k_base", O200K, &O200K_ENCODER), "o200k_harmony" => ("o200k_harmony", "o200k_harmony", O200K, &HARMONY_ENCODER), @@ -99,16 +107,11 @@ pub(super) fn load( "p50k_edit" => ("p50k_edit", "p50k_edit", P50K, &EDIT_ENCODER), "r50k_base" => ("r50k_base", "r50k_base", P50K, &R50K_ENCODER), "gpt2" => ("gpt2", "r50k_base", P50K, &R50K_ENCODER), - _ => return Err(UnsupportedTokenizer(name.to_owned()).into()), - }; - let loaded = cache.get_or_try_init(|| { - let ranks = load_file(file).map_err(|error| LoadError::Ranks(error.to_string()))?; - build(canonical, &ranks) - })?; - Ok((loaded, requested)) + _ => return Err(Error::UnsupportedTokenizer(name.to_owned())), + }) } -fn build(name: &str, ranks: &str) -> Result { +fn build(name: &str, ranks: &str) -> Result { let parsed = ranks .lines() .map(parse_rank) @@ -122,9 +125,11 @@ fn build(name: &str, ranks: &str) -> Result { .collect::>() .len() != encoder.len() - || (0..=u8::MAX).any(|byte| !encoder.contains_key(&[byte][..])) { - return Err(LoadError::Ranks("invalid vocabulary ranks".into())); + return Err(RankError::DuplicateRank.into()); + } + if let Some(byte) = (0..=u8::MAX).find(|byte| !encoder.contains_key(&[*byte][..])) { + return Err(RankError::MissingByte(byte).into()); } let (pattern, specials): (&str, &[(&str, Rank)]) = match name { "cl100k_base" => ( @@ -185,9 +190,12 @@ fn build(name: &str, ranks: &str) -> Result { .chain(special_tokens.values()) .copied() .max() - .ok_or_else(|| LoadError::Ranks("empty vocabulary".into()))?; - let bpe = CoreBPE::new(encoder.clone(), special_tokens.clone(), pattern) - .map_err(|error| LoadError::Ranks(error.to_string()))?; + .unwrap_or_default(); + let bpe = CoreBPE::new(encoder.clone(), special_tokens.clone(), pattern).map_err(|error| { + Error::Encoder(Source::from( + Box::::from(error), + )) + })?; Ok(Loaded { bpe, vocabulary: Vocabulary { @@ -198,16 +206,12 @@ fn build(name: &str, ranks: &str) -> Result { }) } -fn parse_rank(line: &str) -> Result<(Vec, Rank), LoadError> { +fn parse_rank(line: &str) -> Result<(Vec, Rank), RankError> { let (token, rank) = line .split_once(' ') - .ok_or_else(|| LoadError::Ranks("missing rank".into()))?; - let bytes = STANDARD - .decode(token) - .map_err(|error| LoadError::Ranks(error.to_string()))?; - let rank = rank - .parse() - .map_err(|error: std::num::ParseIntError| LoadError::Ranks(error.to_string()))?; + .ok_or_else(|| RankError::MissingRank(line.to_owned()))?; + let bytes = STANDARD.decode(token).map_err(RankError::InvalidToken)?; + let rank = rank.parse().map_err(RankError::InvalidRank)?; Ok((bytes, rank)) } @@ -226,79 +230,117 @@ mod tests { ) } - #[test] - fn packaged_encodings_match_embedded_encodings_and_reuse_successful_loads() { - for name in [ + #[rstest] + #[case::cl100k_base("cl100k_base", None)] + #[case::o200k_base("o200k_base", None)] + #[case::o200k_harmony("o200k_harmony", None)] + #[case::p50k_base("p50k_base", None)] + #[case::p50k_edit("p50k_edit", None)] + #[case::r50k_base_shared_with_gpt2("r50k_base", Some("gpt2"))] + fn packaged_encodings_load_once_and_match_embedded_encodings( + #[case] name: &str, + #[case] alias: Option<&str>, + ) { + assert!( + TiktokenTokenizer::from_cached_ranks(name, |_| { + Err(std::io::Error::other("unreadable vocabulary")) + }) + .is_err() + ); + let loads = std::sync::atomic::AtomicUsize::new(0); + let barrier = std::sync::Barrier::new(4); + let encoders = std::thread::scope(|scope| { + let tasks: Vec<_> = (0..4) + .map(|_| { + scope.spawn(|| { + barrier.wait(); + TiktokenTokenizer::from_cached_ranks(name, |file| { + loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + read_packaged_ranks(file) + }) + .unwrap() + }) + }) + .collect(); + tasks + .into_iter() + .map(|task| task.join().unwrap()) + .collect::>() + }); + assert_eq!(loads.into_inner(), 1); + let actual = &encoders[0]; + let expected = TiktokenTokenizer::from_name(name).unwrap(); + assert_eq!(actual.special_tokens(), expected.special_tokens()); + let specials: Vec<_> = expected.special_tokens().into_iter().collect(); + let special_text = specials.join(" "); + assert_eq!( + actual.encode_special(&special_text, &specials).unwrap(), + expected.encode_special(&special_text, &specials).unwrap() + ); + assert_eq!(actual.encode(&special_text), expected.encode(&special_text)); + let cached = + TiktokenTokenizer::from_cached_ranks(name, |_| panic!("reloaded cached vocabulary")) + .unwrap(); + assert_eq!(cached.encode("cached"), expected.encode("cached")); + assert_eq!(cached.name(), name); + assert!(expected.vocabulary().is_none()); + assert_vocabulary_lookups(name, actual); + if let Some(alias) = alias { + let shared = TiktokenTokenizer::from_cached_ranks(alias, |_| { + panic!("reloaded the shared vocabulary") + }) + .unwrap(); + assert_eq!(shared.name(), alias); + assert_vocabulary_lookups(alias, &shared); + } + } + + fn uncached(name: &str) -> TiktokenTokenizer { + static BUILT: std::sync::Mutex> = + std::sync::Mutex::new(Vec::new()); + let (requested, canonical, file, _) = encoding(name).unwrap(); + let mut built = BUILT.lock().unwrap(); + let loaded = match built.iter().find(|(built, _)| *built == requested) { + Some((_, loaded)) => *loaded, + None => { + let ranks = read_packaged_ranks(file).unwrap(); + let loaded: &'static Loaded = + Box::leak(Box::new(build(canonical, &ranks).unwrap())); + built.push((requested, loaded)); + loaded + } + }; + TiktokenTokenizer { + encoder: &loaded.bpe, + vocabulary: Some(&loaded.vocabulary), + name: requested, + } + } + + #[rstest] + fn packaged_ranks_encode_like_embedded_encodings( + #[values( "cl100k_base", "o200k_base", "o200k_harmony", "p50k_base", "p50k_edit", "r50k_base", - "gpt2", - ] { - if name != "gpt2" { - assert!( - TiktokenTokenizer::from_cached_ranks(name, |_| { - Err(std::io::Error::other("unreadable vocabulary")) - }) - .is_err() - ); - } - let loads = std::sync::atomic::AtomicUsize::new(0); - let barrier = std::sync::Barrier::new(4); - let encoders = std::thread::scope(|scope| { - let tasks: Vec<_> = (0..4) - .map(|_| { - scope.spawn(|| { - barrier.wait(); - TiktokenTokenizer::from_cached_ranks(name, |file| { - loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed); - read_packaged_ranks(file) - }) - .unwrap() - }) - }) - .collect(); - tasks - .into_iter() - .map(|task| task.join().unwrap()) - .collect::>() - }); - assert_eq!(loads.into_inner(), usize::from(name != "gpt2")); - let actual = &encoders[0]; - let expected = TiktokenTokenizer::from_name(name).unwrap(); - assert_eq!(actual.special_tokens(), expected.special_tokens()); - let specials: Vec<_> = expected.special_tokens().into_iter().collect(); - let special_text = specials.join(" "); - assert_eq!( - actual.encode_special(&special_text, &specials).unwrap(), - expected.encode_special(&special_text, &specials).unwrap() - ); - for text in [ - "", - "café 漢字 ع 🙂", - "a\r\nb\t ", - " hello 123456789", - &special_text, - ] { - let ids = expected.encode(text); - assert_eq!(actual.encode(text), ids, "{name}: {text:?}"); - assert_eq!(actual.count_tokens(text), ids.len(), "{name}: {text:?}"); - assert_eq!( - actual.decode_bytes(&ids).unwrap(), - expected.decode_bytes(&ids).unwrap() - ); - } - let cached = TiktokenTokenizer::from_cached_ranks(name, |_| { - panic!("reloaded cached vocabulary") - }) - .unwrap(); - assert_eq!(cached.encode("cached"), expected.encode("cached")); - assert_eq!(cached.name(), name); - assert!(expected.vocabulary().is_none()); - assert_vocabulary_lookups(name, actual); - } + "gpt2" + )] + name: &str, + #[values("", "café 漢字 ع 🙂", "a\r\nb\t ", " hello 123456789")] text: &str, + ) { + let actual = uncached(name); + let expected = TiktokenTokenizer::from_name(name).unwrap(); + + let ids = expected.encode(text); + assert_eq!(actual.encode(text), ids); + assert_eq!(actual.count_tokens(text), ids.len()); + assert_eq!( + actual.decode_bytes(&ids).unwrap(), + expected.decode_bytes(&ids).unwrap() + ); } /// The token-level lookups tiktoken's Python `Encoding` exposes, checked against the diff --git a/litellm-rust/crates/token-counter/Cargo.toml b/litellm-rust/crates/token-counter/Cargo.toml index 67e5bd60537..01ce4b6bac8 100644 --- a/litellm-rust/crates/token-counter/Cargo.toml +++ b/litellm-rust/crates/token-counter/Cargo.toml @@ -14,6 +14,7 @@ tiktoken = ["dep:litellm-token-counter-tiktoken"] [dependencies] indexmap = { version = "2.14.0", features = ["serde"] } itoa = "1.0" +litellm-error.workspace = true litellm-token-counter-fast = { workspace = true, optional = true } litellm-token-counter-huggingface = { workspace = true, optional = true } litellm-token-counter-tiktoken = { workspace = true, optional = true } diff --git a/litellm-rust/crates/token-counter/src/counter.rs b/litellm-rust/crates/token-counter/src/counter.rs index 6370d9d74f8..f3c16f93b84 100644 --- a/litellm-rust/crates/token-counter/src/counter.rs +++ b/litellm-rust/crates/token-counter/src/counter.rs @@ -1,12 +1,12 @@ use serde::Serialize; -use crate::Error; use crate::python_json; use crate::tools::format_function_definitions; use crate::types::{ ContentItem, CountableContentBlock, CountableRequest, Message, MessageContent, TextValue, ToolChoice, ToolDefinition, }; +use crate::{Error, Unsupported}; const TOKENS_PER_MESSAGE: usize = 3; const TOKENS_PER_NAME: usize = 1; @@ -53,7 +53,7 @@ impl TokenCounter { self.count_optional_text_value(request.query.as_ref())? + self.count_optional_text_value(request.documents.as_ref())? } else { - return Err(Error::MissingInput); + return Err(Unsupported::MissingInput.into()); }; Ok(InputTokenCount { model: request.model.clone(), @@ -96,7 +96,7 @@ impl TokenCounter { TextValue::Number(number) => match (number.as_i64(), number.as_u64()) { (Some(number), _) => self.count_text(itoa::Buffer::new().format(number)), (_, Some(number)) => self.count_text(itoa::Buffer::new().format(number)), - _ => Err(Error::FloatText), + _ => Err(Unsupported::FloatText.into()), }, TextValue::Text(text) => self.count_text(text), TextValue::List(items) => items @@ -143,7 +143,9 @@ impl TokenCounter { None => Ok(0), } } - ContentItem::Block(CountableContentBlock::Unsupported) => Err(Error::ContentBlock), + ContentItem::Block(CountableContentBlock::Unsupported) => { + Err(Unsupported::ContentBlock.into()) + } } } diff --git a/litellm-rust/crates/token-counter/src/error.rs b/litellm-rust/crates/token-counter/src/error.rs index 7a74138c074..89a97423c48 100644 --- a/litellm-rust/crates/token-counter/src/error.rs +++ b/litellm-rust/crates/token-counter/src/error.rs @@ -1,39 +1,66 @@ use std::string::FromUtf8Error; +use litellm_error::Source; use thiserror::Error as ThisError; #[derive(Debug, ThisError)] pub enum Error { - #[error("unsupported tokenizer: {0}")] - UnsupportedTokenizer(String), - #[error("failed to load tokenizer: {0}")] - Load(#[source] Box), - #[error("failed to load tokenizer: tiktoken rank file: {0}")] - Ranks(#[source] Box), - #[error("failed to load tokenizer: Unicode character classes are unavailable")] - UnicodeClasses, - #[error("unsupported by the rust token counter: request body could not be parsed: {0}")] - RequestParse(#[source] serde_json::Error), - #[error("unsupported by the rust token counter: request has no countable input")] - MissingInput, - #[error( - "unsupported by the rust token counter: float text values are counted by the python path" - )] - FloatText, - #[error( - "unsupported by the rust token counter: content block type is counted by the python path" - )] - ContentBlock, - #[error("unsupported by the rust token counter: array parameter without items")] - ArrayItems, - #[error("unsupported by the rust token counter: text value could not be serialized: {0}")] - JsonSerialization(#[source] serde_json::Error), - #[error("unsupported by the rust token counter: serialized text value is not UTF-8: {0}")] - JsonUtf8(#[source] FromUtf8Error), - #[error("tokenization failed: {0}")] - Encode(#[source] Box), - #[error("token decoding failed: {0}")] - Decode(String), - #[error("token counting task failed: {0}")] - Task(String), + #[cfg(feature = "fast")] + #[error(transparent)] + Fast(#[from] litellm_token_counter_fast::Error), + #[cfg(feature = "huggingface")] + #[error(transparent)] + HuggingFace(#[from] litellm_token_counter_huggingface::Error), + #[cfg(feature = "tiktoken")] + #[error(transparent)] + Tiktoken(#[from] litellm_token_counter_tiktoken::Error), + #[error("unsupported by the rust token counter")] + Unsupported(#[from] Unsupported), + #[error("token counting task failed")] + Task(#[source] Source), +} + +impl Error { + pub fn is_unsupported(&self) -> bool { + match self { + Self::Unsupported(_) => true, + #[cfg(feature = "fast")] + Self::Fast(_) => false, + #[cfg(feature = "huggingface")] + Self::HuggingFace(_) => false, + #[cfg(feature = "tiktoken")] + Self::Tiktoken(_) => false, + Self::Task(_) => false, + } + } + + pub fn is_load(&self) -> bool { + match self { + #[cfg(feature = "fast")] + Self::Fast(error) => error.is_load(), + #[cfg(feature = "huggingface")] + Self::HuggingFace(error) => error.is_load(), + #[cfg(feature = "tiktoken")] + Self::Tiktoken(error) => error.is_load(), + Self::Unsupported(_) | Self::Task(_) => false, + } + } +} + +#[derive(Debug, ThisError)] +pub enum Unsupported { + #[error("request body could not be parsed")] + RequestParse(#[source] serde_json::Error), + #[error("request has no countable input")] + MissingInput, + #[error("float text values are counted by the python path")] + FloatText, + #[error("content block type is counted by the python path")] + ContentBlock, + #[error("array parameter without items")] + ArrayItems, + #[error("text value could not be serialized")] + JsonSerialization(#[source] serde_json::Error), + #[error("serialized text value is not UTF-8")] + JsonUtf8(#[source] FromUtf8Error), } diff --git a/litellm-rust/crates/token-counter/src/fast.rs b/litellm-rust/crates/token-counter/src/fast.rs index d1f270dcdc9..3ef0021e355 100644 --- a/litellm-rust/crates/token-counter/src/fast.rs +++ b/litellm-rust/crates/token-counter/src/fast.rs @@ -1,4 +1,3 @@ -use litellm_token_counter_fast::Error as BackendError; pub use litellm_token_counter_fast::FastTokenizer; use crate::{Error, TextCodec, TokenCounter, Tokenizer}; @@ -53,17 +52,6 @@ impl Tokenizer for FastTokenizer { } } -impl From for Error { - fn from(error: BackendError) -> Self { - match error { - BackendError::Load(source) => Self::Load(source), - BackendError::Ranks(source) => Self::Ranks(source.into()), - BackendError::UnicodeClasses => Self::UnicodeClasses, - BackendError::Encode(source) => Self::Encode(source), - } - } -} - #[cfg(all(test, feature = "huggingface", feature = "tiktoken"))] mod tests { use super::*; diff --git a/litellm-rust/crates/token-counter/src/huggingface.rs b/litellm-rust/crates/token-counter/src/huggingface.rs index 43613fac8ea..5730e9ce873 100644 --- a/litellm-rust/crates/token-counter/src/huggingface.rs +++ b/litellm-rust/crates/token-counter/src/huggingface.rs @@ -1,4 +1,3 @@ -use litellm_token_counter_huggingface::Error as BackendError; pub use litellm_token_counter_huggingface::{ AddedToken, EncodeInput, Encoding, HuggingFaceTokenizer, InputSequence, PaddingDirection, PaddingParams, PaddingStrategy, TruncationDirection, TruncationParams, encoding_from_json, @@ -34,13 +33,3 @@ impl TextCodec for HuggingFaceTokenizer { HuggingFaceTokenizer::name(self) } } - -impl From for Error { - fn from(error: BackendError) -> Self { - match error { - BackendError::Load(source) => Self::Load(source), - BackendError::Encode(source) => Self::Encode(source), - BackendError::Decode(source) => Self::Decode(source.to_string()), - } - } -} diff --git a/litellm-rust/crates/token-counter/src/lib.rs b/litellm-rust/crates/token-counter/src/lib.rs index 84bf35ca2ba..c58b8cf372f 100644 --- a/litellm-rust/crates/token-counter/src/lib.rs +++ b/litellm-rust/crates/token-counter/src/lib.rs @@ -19,6 +19,6 @@ pub mod huggingface; pub mod tiktoken; pub use counter::{InputTokenCount, TokenCounter}; -pub use error::Error; +pub use error::{Error, Unsupported}; pub use tokenizer::{TextCodec, Tokenizer}; pub use types::CountableRequest; diff --git a/litellm-rust/crates/token-counter/src/python_json.rs b/litellm-rust/crates/token-counter/src/python_json.rs index 00a55ad300c..b5c074a24c7 100644 --- a/litellm-rust/crates/token-counter/src/python_json.rs +++ b/litellm-rust/crates/token-counter/src/python_json.rs @@ -6,8 +6,8 @@ use std::io::{self, Write}; use serde::Serialize; use serde_json::ser::{Formatter, Serializer}; -use super::Error; use super::types::TextValue; +use super::{Error, Unsupported}; pub(super) fn dumps(value: &TextValue) -> Result { let mut output = Vec::with_capacity(serialized_len(value)?); @@ -16,9 +16,9 @@ pub(super) fn dumps(value: &TextValue) -> Result { &mut output, PythonFormatter, )) - .map_err(Error::JsonSerialization)?; + .map_err(Unsupported::JsonSerialization)?; debug_assert_eq!(output.len(), output.capacity()); - String::from_utf8(output).map_err(Error::JsonUtf8) + String::from_utf8(output).map_err(|error| Unsupported::JsonUtf8(error).into()) } fn serialized_len(value: &TextValue) -> Result { @@ -29,7 +29,7 @@ fn serialized_len(value: &TextValue) -> Result { TextValue::Number(number) => match (number.as_i64(), number.as_u64()) { (Some(number), _) => Ok(unsigned_len(number.unsigned_abs()) + usize::from(number < 0)), (_, Some(number)) => Ok(unsigned_len(number)), - _ => Err(Error::FloatText), + _ => Err(Unsupported::FloatText.into()), }, TextValue::Text(text) => Ok(quoted_len(text)), TextValue::List(items) => items @@ -147,9 +147,9 @@ mod tests { let value = serde_json::from_str(input).expect("fixture parses"); let error = dumps(&value).expect_err("floats are declined"); - assert_eq!( - error.to_string(), - "unsupported by the rust token counter: float text values are counted by the python path" - ); + assert!(matches!( + error, + crate::Error::Unsupported(crate::Unsupported::FloatText) + )); } } diff --git a/litellm-rust/crates/token-counter/src/tiktoken.rs b/litellm-rust/crates/token-counter/src/tiktoken.rs index b95621f3ccc..136280b6115 100644 --- a/litellm-rust/crates/token-counter/src/tiktoken.rs +++ b/litellm-rust/crates/token-counter/src/tiktoken.rs @@ -1,4 +1,3 @@ -use litellm_token_counter_tiktoken::{LoadError, UnsupportedTokenizer}; pub use litellm_token_counter_tiktoken::{TiktokenTokenizer, Vocabulary, encoding_for_model}; use crate::{Error, TextCodec, TokenCounter, Tokenizer}; @@ -23,25 +22,10 @@ impl TextCodec for TiktokenTokenizer { } fn decode(&self, ids: &[u32], _skip_special_tokens: bool) -> Result { - TiktokenTokenizer::decode(self, ids).map_err(|error| Error::Decode(error.to_string())) + TiktokenTokenizer::decode(self, ids).map_err(Error::from) } fn name(&self) -> &str { TiktokenTokenizer::name(self) } } - -impl From for Error { - fn from(error: UnsupportedTokenizer) -> Self { - Self::UnsupportedTokenizer(error.0) - } -} - -impl From for Error { - fn from(error: LoadError) -> Self { - match error { - LoadError::Unsupported(error) => error.into(), - LoadError::Ranks(source) => Self::Ranks(source.into()), - } - } -} diff --git a/litellm-rust/crates/token-counter/src/tools.rs b/litellm-rust/crates/token-counter/src/tools.rs index 22150a4e4b1..a5a3fdbfabf 100644 --- a/litellm-rust/crates/token-counter/src/tools.rs +++ b/litellm-rust/crates/token-counter/src/tools.rs @@ -3,8 +3,8 @@ use std::fmt::Write; -use super::Error; use super::types::{EnumValue, Schema, SchemaType, ToolDefinition}; +use super::{Error, Unsupported}; pub(super) fn format_function_definitions(tools: &[ToolDefinition]) -> Result { ToolFormatter::new(tools.len()).format(tools) @@ -117,7 +117,7 @@ impl ToolFormatter { None => self.output.push_str("number"), }, "array" => { - let items = props.items.as_deref().ok_or(Error::ArrayItems)?; + let items = props.items.as_deref().ok_or(Unsupported::ArrayItems)?; self.write_type(items, indent)?; self.output.push_str("[]"); } @@ -298,7 +298,7 @@ mod tests { assert!(matches!( format_function_definitions(&tools), - Err(Error::ArrayItems) + Err(Error::Unsupported(Unsupported::ArrayItems)) )); } } diff --git a/litellm-rust/crates/token-counter/src/types.rs b/litellm-rust/crates/token-counter/src/types.rs index c1236f94d9a..7e1513d7706 100644 --- a/litellm-rust/crates/token-counter/src/types.rs +++ b/litellm-rust/crates/token-counter/src/types.rs @@ -5,7 +5,7 @@ use serde::de::{MapAccess, SeqAccess, Visitor}; use serde::{Deserialize, Deserializer, Serialize}; use serde_json::Number; -use super::Error; +use super::{Error, Unsupported}; /// The parts of a request body the host's budget counter reads. Chat and /// Anthropic Messages bodies carry `messages`; completions carry `prompt`; @@ -33,7 +33,7 @@ pub struct CountableRequest { impl CountableRequest { pub fn parse(body: &[u8]) -> Result { - serde_json::from_slice(body).map_err(Error::RequestParse) + serde_json::from_slice(body).map_err(|error| Unsupported::RequestParse(error).into()) } } diff --git a/litellm-rust/crates/token-counter/tests/token_counter.rs b/litellm-rust/crates/token-counter/tests/token_counter.rs index 542bd4a1fc4..cc18bad5402 100644 --- a/litellm-rust/crates/token-counter/tests/token_counter.rs +++ b/litellm-rust/crates/token-counter/tests/token_counter.rs @@ -2,7 +2,7 @@ use rstest::rstest; #[cfg(any(feature = "fast", feature = "huggingface", feature = "tiktoken"))] use litellm_token_counter::TokenCounter; -use litellm_token_counter::{CountableRequest, Error}; +use litellm_token_counter::{CountableRequest, Error, Unsupported}; #[cfg(any(feature = "fast", feature = "huggingface"))] mod json { @@ -100,7 +100,12 @@ mod json { let request = CountableRequest::parse(body).expect("shape parses"); assert!(matches!( counter(load).count_request(&request), - Err(Error::MissingInput | Error::FloatText | Error::ContentBlock | Error::ArrayItems) + Err(Error::Unsupported( + Unsupported::MissingInput + | Unsupported::FloatText + | Unsupported::ContentBlock + | Unsupported::ArrayItems + )) )); } @@ -139,7 +144,7 @@ mod json { } fn assert_loading_a_bad_tokenizer_is_a_load_error(load: JsonLoader) { - assert!(matches!(load("{}"), Err(Error::Load(_)))); + assert!(load("{}").is_err_and(|error| error.is_load())); } macro_rules! json_backend_tests { @@ -206,7 +211,7 @@ mod json { let counter = $loader(&tokenizer.to_string(false).unwrap()).unwrap(); let request = CountableRequest::parse(br#"{"prompt":"hello"}"#).unwrap(); let error = counter.count_request(&request).unwrap_err(); - assert!(matches!(error, Error::Encode(_))); + assert!(!error.is_load() && !error.is_unsupported()); assert_eq!(error.source().unwrap().to_string(), expected.to_string()); } @@ -250,7 +255,7 @@ mod json { fn shapes_outside_the_mirror_are_declined_at_parse(#[case] body: &[u8]) { assert!(matches!( CountableRequest::parse(body), - Err(Error::RequestParse(_)) + Err(Error::Unsupported(Unsupported::RequestParse(_))) )); } @@ -443,7 +448,7 @@ mod tiktoken { #[case] rank_file: &str, #[values(CL100K, O200K)] encoding: TiktokenEncoding, ) { - assert!(matches!((encoding.load)(rank_file), Err(Error::Ranks(_)))); + assert!((encoding.load)(rank_file).is_err_and(|error| error.is_load())); } } @@ -452,6 +457,7 @@ mod tiktoken { fn unsupported_encoding_reaches_the_counter_caller() { assert!(matches!( TokenCounter::from_tiktoken("unknown-encoding"), - Err(Error::UnsupportedTokenizer(name)) if name == "unknown-encoding" + Err(error) if error.is_load() + && litellm_error::Report::new(&error).to_string().contains("unknown-encoding") )); }