diff --git a/litellm-rust/docs/exception-mapping-plan.md b/litellm-rust/docs/exception-mapping-plan.md new file mode 100644 index 00000000000..b2864fd2ccf --- /dev/null +++ b/litellm-rust/docs/exception-mapping-plan.md @@ -0,0 +1,200 @@ +# Exception mapping: plan to align the Rust path with the five-stage design + +Scope: branch `litellm_ocr_rust_parity_gaps`. The types and the trait are generic, but only the OCR route is moved in this work. Chat, messages, responses and audio transcription are a later change (section 4.5) and nothing here depends on them. Nothing in this plan has been built or run. The target design is the five-stage chain (capture, normalize, classify, construct, finalize) where each stage has one owner and the pyo3 layer owns none of them + +## 1. What this branch changed + +- `fd18a6c9d70`, `75cbb689b8a`: tests only + - Failing contract tests for the OCR Rust path gaps + - A differential parity sweep in `tests/test_litellm_rust/ocr/test_parity.py` with `support/parity.py` (`Outcome`, `Divergence`, `assert_parity`) +- `9a0c45f6e3e`: `core/src/litellm_core_utils/secret_redaction.rs`, a port of the Python redaction patterns +- `0eb60785854`: the mapping itself + - `core/src/litellm_core_utils/exception_mapping_utils.rs` (1021 lines, one file): `ProviderException`, `ExceptionContext`, `PublicFailure`, `exception_type`, with `map_openai_exception`, `map_vertex_exception`, `map_cohere_exception` and `map_exception_by_status` as hand-written functions, plus 13 rstest groups + - `python-bridge/src/failures.rs`: `provider_exception` (parses a `RustUpstreamError` back into a `ProviderException`), `exception_context` (reads `litellm.*` globals and kwargs), `public_exception` (serde dict to `litellm.rust_bridge.failures.public_exception`), `map_upstream_failure` + - `python-bridge/src/routes/ocr/host.rs`: `map_failure` tries `map_upstream_failure` first, then falls back to the Python `route_host.map_failure` + - `litellm/rust_bridge/failures.py`: Pydantic `PublicFailure` with `extra="forbid"` and the constructor `public_exception`. The old `map_failure` that calls `litellm.exception_type` is still there + - `litellm/rust_bridge/dispatch.py`: `finalize_failure`, `_finalized`, `_afinalized` wrap the native call in `run` and `arun` + +## 2. Where the branch stands against each stage + +| Stage | State on the branch | Gap | +|---|---|---| +| 1. Capture | `ocr::Error` carries `Provider{status, body, headers}`, `Transport`, validation, auth and file variants | None in core. The bridge turns it into a `PyErr` too early (`routes/ocr/errors.rs::to_pyerr`) | +| 2. Normalize | `OcrConfigKind::get_error_class(message, status, headers)` exists but returns `ocr::Error`. The real normalization happens in the bridge: `provider_exception` rebuilds `ProviderException` from `RustUpstreamError.args` and a `headers` attribute set with `setattr` | Wrong owner and a lossy round trip through a Python exception | +| 3. Classify | `exception_type(&ExceptionContext, &ProviderException)` in core | Only `Provider` and `Transport::Http` errors reach it. Everything else (`ValueError` with a patched `status_code`, `FileNotFoundError`, `OSError`, `RuntimeError`) goes to the Python `litellm.exception_type`, which is Python-path-only code. `RequestFormat` is classified in Python through the `ocr_request_format_error` attribute. Provider dispatch is on strings (`"mistral" \| "azure_ai"`, `"vertex_ai"`, `"cohere"`) | +| 4. Construct | `failures.public_exception` | Matches the design. The payload shape does not: flat `response` and `request` fields that both have `Omitted`, an `upstream_content` override applied with `setattr` after construction, `stdout` carried as a string, no `provider_specific_fields` | +| 5. Finalize | `dispatch.finalize_failure`, tested for sync and async | Done. Only `str(error)` coverage is missing | +| Driver | `machine_failed` keeps the interrupted host `PyErr` apart from the native error, then calls `H::native_error` and `map_failure(&PyErr)` | `map_failure(py, &error).unwrap_or(error)` at `driver.rs:390` swallows a mapper failure | + +Two things the design text does not say but the code shows: + +- Only OCR runs on the `RouteHost` driver. Chat, messages, responses and audio transcription use `run_sync` / `run_async` with `*_error_to_pyerr`. Step 5 of the design is therefore larger than moving a call site. See 4.5 +- `runtime._raise_upstream` and the `map_failure` functions in `chat_completions/route_host.py`, `messages/route_host.py` and `responses/route_host.py` are dead stand-ins + - No Rust code calls those three `map_failure` functions. Only `routes/ocr/host.rs` imports a `route_host.map_failure` + - `catalog.RULES` enables only OCR and bedrock transcription. Transcription errors go through `core_error_to_pyerr` and never raise `RustUpstreamError`. OCR's `RustUpstreamError` is mapped inside the driver. The only way one reaches `_raise_upstream` today is the `unwrap_or(error)` swallow in the driver +- OCR invents an HTTP response for timeouts: `ocr/client.rs::transport_error` returns `Transport::Http{status: 408, body: "OCR request timed out"}` and `ocr/document.rs:207` does the same for document downloads. The Python path has no response at all there, it maps an httpx exception. The sweep does not cover transport failures yet +- `RequestFormat` maps to `litellm.UnsupportedParamsError`, which is not in the design's `PublicFailure` list. It fits as one more `Status` class (it subclasses `BadRequestError`), so no new kind is needed + +## 3. Target shapes + +### Core (`core/src/litellm_core_utils/exception_mapping_utils/`) + +- `original.rs` + - `enum OriginalException { Http{status, body, headers}, Connection{message}, Timeout{message}, Response{message}, Local{class: LocalClass, message, status: Option} }` + - `enum LocalClass { ValueError, FileNotFound, OsError }` + - `ProviderException`, `status_code_is_synthesized` and the status 0 sentinel are deleted +- `public.rs` + - `struct PublicFailure { kind, message, model, llm_provider, litellm_debug_info, litellm_response_headers, provider_specific_fields, print_banner }` + - `enum PublicKind { Status{class: StatusClass, response: Option}, Timeout{status: Option, response: Option}, ApiConnection{request: RequestStub}, Api{status: u16, request: RequestStub}, Passthrough{class: LocalClass, message} }` + - `StatusClass` is today's simple kinds minus `ApiConnection`, plus `UnsupportedParams` + - `ResponseArg` stays `Upstream | Stub`, without `Omitted`. `Option` says whether the constructor gets one + - `upstream_content` goes away: the vertex case becomes `Status{response: Some(Upstream{status: , body, headers})}`, decided in the rule, so Python never patches `error.response` after construction + - `stdout: String` becomes `print_banner: bool`. The banner text moves to `litellm/rust_bridge/failures.py`, because printing is part of stage 4 and the text today lives in Python-path-only code +- `mod.rs`: `ExceptionContext`, the message prefixes (`exception_provider`, `python_capitalize`, `extra_information`), redaction of `error_str`, the timeout marker check, and the dispatch `match provider: Provider`. `ExceptionContext.custom_llm_provider` becomes core's provider enum plus the raw string for messages, so dispatch never compares strings +- `status.rs`: the fallback table, today's `map_exception_by_status` +- `rules.rs`: `struct Rule { when: When, kind, message: MessageTemplate, response: ResponseChoice, debug: bool }` and the one interpreter `fn apply(rules: &[Rule], mapping: &Mapping) -> Option`, first match wins +- `openai.rs`, `vertex_ai.rs`, `cohere.rs`: one ordered `&[Rule]` each +- Dead branches: drop every branch that only matches a string the Python SDKs produce, for example the `BadRequestError.__init__() missing 1 required positional argument: 'param'` check in `map`. Each removal is a `Divergence` row in the sweep if the sweep can reach it, and a line in the PR body either way + +### Provider config (stage 2) + +- `OcrConfigKind::error_class(&ocr::Error) -> OriginalException` replaces `get_error_class` + - `Provider{..}` -> `Http` with headers + - `Transport::Http` -> `Http` without headers + - `Transport::Network` -> `Timeout` when the reqwest error was a timeout, else `Connection`. `transport::Error::Network(String)` loses that bit today, so add `transport::Error::Timeout(String)` and set it in `from_reqwest_before_dispatch` and `From` + - Delete the invented 408 in `ocr/client.rs::transport_error` and `ocr/document.rs`. Both return `transport::Error::Timeout`. `Transport::Connect` (refused, DNS) -> `Connection` + - The message text cannot match: Python embeds httpx's exception text and Rust has reqwest's. Parity is on class, `status_code`, the absence of a response, and the litellm-owned prefix of the message. The httpx part of the message is one recorded `Divergence` on `message` for the transport rows only + - Invalid and oversized responses (`InvalidResponse`, `ResponseField`, `EmptyContent`, `TooLarge`, `NumericRange`) -> a new variant `OriginalException::Response{message}`. The Python path raises inside response transformation and `exception_type` turns that into whatever the sweep shows (expected `APIConnectionError`). This replaces open question 1 + - `FileRead` with `NotFound` -> `Local{FileNotFound, "File not found: ", None}`. Other `FileRead` -> `Local{OsError, , None}` + - `is_request()` errors, auth errors and the other value-error cases from `core_error_to_pyerr` -> `Local{ValueError, message, Some(400)}` + - `RequestFormat` needs its own arm, because its public class is `UnsupportedParams` and its message includes the rejected `req_format` value. Carry the value in the error (`RequestFormat{value: String}`) so the message is built in core + - Response errors (`TooLarge`, `ResponseField`, ...) are `RuntimeError` today. `LocalClass` has no `RuntimeError`. Open question 1 below + +### Bridge (`python-bridge`) and driver (`host-python`) + +- `RouteHost` loses `native_error` and `map_failure(&PyErr)` and gains `fn classify(&self, py, error: ::Error) -> PyResult` + - The OCR impl builds `ExceptionContext` with `failures::exception_context`, calls `config.error_class(&error)`, then `exception_type` +- `struct BridgeFailure(PublicFailure)` with `impl From for PyErr` in `python-bridge/src/failures.rs`. It is the only caller of `public_exception`. If the Python constructor itself fails (import error, Pydantic validation), that error is what the `From` impl returns + - `host-python` must not depend on `python-bridge`, so the driver cannot name `BridgeFailure`. The trait method returns `PyResult` built through the `From` impl inside the route's `classify`, or the trait gets an associated `type Failure: Into`. Recommendation: the associated type, because it keeps the `PublicFailure` value visible to the driver's fake route in tests +- Driver: `machine_failed` calls `classify` exactly once, only when `self.interrupted` is `None`. `failure()` loses the `FailureOrigin::Call => map_failure(...).unwrap_or(error)` arm. A `classify` error is raised with the classifier error as the exception and the native error's message attached through `__context__` (a `RuntimeError` holding `error.to_string()`), so nothing is swallowed +- Deleted: `RustUpstreamError` (class, `.pyi` entry, `bindings.native_exception_types` second slot), `provider_exception`, `map_upstream_failure`, `routes/ocr/errors.rs` (`to_pyerr`, `upstream_error`, `attach_status`), `litellm/rust_bridge/ocr/route_host.map_failure`, and at the end `failures.map_failure` with its `ExceptionMapper` protocol +- Kept: `create_exception!` for `RustBridgeDeclined` only +- Not used, per the design: `import_exception!`, lazy `PyErrArguments`, `#[pyclass(extends=PyException)]`, a `#[pyclass]` mirror of `PublicFailure` + +### Shared Python (`litellm/rust_bridge/failures.py`) + +- The Pydantic models mirror the new `PublicKind`. `extra="forbid"` stays +- `_simple`'s eleven near-identical arms collapse into a `Mapping[StatusClass, type[Exception]]` lookup plus one call, because every `Status` class now takes the same keyword arguments +- `Passthrough` builds the builtin (`ValueError`, `FileNotFoundError`, `OSError`) and sets `status_code` when present +- `public_exception` prints the banner when `print_banner` is true, sets `litellm_response_headers` and `provider_specific_fields` + +## 4. Steps, each with its unit tests + +The parity sweep must be green after every step except for recorded divergences. Every step lands with the tests listed under it. Rust tests use rstest tables and assert the whole `PublicFailure` value with `assert_eq!`, never a single field, so a mutation in any field fails a test + +### 4.1 `OriginalException` and `error_class` in core + +- Sweep first: add a transport axis to `test_parity.py` before touching the code, so the 408 shows up as a red row. Cases: read timeout, connection refused, body that is not JSON, JSON missing required fields, body over the size limit, each for sync and async and per provider. The local server in the sweep's support code needs a "sleep past the timeout" handler, a closed port, and a large-body handler +- Code: add `original.rs`, `transport::Error::Timeout`, remove both invented 408s, `OcrConfigKind::error_class`. `exception_type` takes `&OriginalException`. Keep a private `From` shim only inside this step if needed to keep the diff reviewable, and delete it before the step's commit +- Tests (Rust, permanent) + - `error_class`: one case per `ocr::Error` variant group listed in section 3, per provider config where the config overrides anything. The `FileRead` cases assert the exact message, including the path + - A test that iterates every `ocr::Error` variant that `is_request()` and asserts `Local{ValueError, _, Some(400)}`, so a new request variant cannot be forgotten + - `transport`: a timeout reqwest error becomes `Timeout`, a connect error before dispatch stays `Connect` + - `ocr/client.rs`: replace `request_timeout_has_an_http_408_status` with a test that a timed-out request and a timed-out document download both yield `transport::Error::Timeout` and that no `Http` variant is ever produced without a real response + - `exception_type` on `Response{..}`: whole-value case per provider family + - `exception_type` on `Connection` and `Timeout` inputs: replaces today's `synthesized_status_is_a_connection_error` + +### 4.2 `classify` on the trait, one `From` impl, OCR moved over + +- Code: trait change in `host-python/src/adapter.rs`, driver change in `driver.rs`, `BridgeFailure`, OCR `classify`, delete `routes/ocr/errors.rs` and the `RustUpstreamError` re-parsing +- Dead stand-ins go in this step, because fixing the `unwrap_or(error)` swallow removes the last way a `RustUpstreamError` can escape + - Delete `runtime._raise_upstream`, the `except upstream` arms in `attempt` and `aattempt`, and the second slot of `bindings.native_exception_types` + - Delete `map_failure` from `chat_completions/route_host.py`, `messages/route_host.py` and `responses/route_host.py`, with their tests under `tests/test_litellm/rust_bridge//` + - `chat_completions_error_to_pyerr` still names `RustUpstreamError`. Chat has no rule in `catalog.RULES`, so map those two arms to `PyRuntimeError` for now and let 4.5 replace the function + - `test_runtime.py`: drop the `_raise_upstream` cases, keep and extend the declined-falls-back cases +- Tests (`host-python` driver, existing fake route with its `log`) + - A native error reaches `classify` exactly once and the `Failed` event carries the classified error + - A host-raised error during an op skips `classify` and is emitted unchanged (`self.interrupted` path) + - A cancellation is returned as is, with no `classify` call and no `Failed` event + - A `classify` that returns `Err` surfaces that error, and the native error's text is reachable from `__context__`. This is the regression test for `unwrap_or(error)` + - Failures in `begin` and `after_success` stay unmapped (already covered, keep) +- Tests (`python-bridge`) + - `exception_context`: reads `suppress_debug_info` and `redact_messages_in_exceptions` from `litellm`, reads `vertex_project`, `vertex_location` and the four metadata keys, tolerates `metadata=None` and a non-mapping `metadata`, stringifies non-string values the way Python's `str()` does + - `BridgeFailure -> PyErr`: a valid payload yields the litellm class. A payload the Python side rejects yields the Pydantic error, not a panic and not a silent fallback + +### 4.3 Reshape `PublicFailure`, the constructor, and add the golden fixtures + +- Code: `public.rs`, the Pydantic models, the collapsed constructor, banner text moved to Python, `provider_specific_fields` +- Tests + - Rust: a test builds one `PublicFailure` per kind (`Status` once per `StatusClass`, `Timeout` with and without status, `ApiConnection`, `Api`, `Passthrough` once per `LocalClass`), serializes them and compares against `tests/test_litellm/rust_bridge/fixtures/public_failures/*.json`. An env switch regenerates the files. The comparison is what runs in CI + - Python, permanent, in `tests/test_litellm/rust_bridge/test_failures.py`: parametrized over the fixture files. For each one assert the exact class, `isinstance` of the matching `openai.*` type, `status_code`, `message`, `llm_provider`, `model`, response body and headers, `litellm_response_headers`, `provider_specific_fields`, and that the banner is written to stdout only when `print_banner` is true + - Python: an unknown field and an unknown `StatusClass` are both rejected. A test asserts every `StatusClass` literal has a constructor entry and a fixture, so adding a class on one side fails on the other + - Delete the three `map_failure` tests at the top of `test_failures.py` in step 4.4, not here + +### 4.4 Cover `Local`, then delete the Python fallback + +- Code: classifier handles `Local`. `ValueError` with status 400 follows what the Python `exception_type` does with it today for OCR (the sweep is the oracle for whether that is `BadRequestError`, `APIConnectionError` or a passthrough, per provider). `FileNotFound` and `OsError` become `Passthrough`. `RequestFormat` becomes `Status{UnsupportedParams}`. Then delete `ocr/route_host.map_failure`, the Python call in `host.rs` and `failures.map_failure` with its `ExceptionMapper` protocol. After this step nothing under `litellm/rust_bridge/` imports `litellm.exception_type`. Add a test in `tests/test_litellm/rust_bridge/` that fails if any module there references it, since that is the invariant the step exists for +- Tests + - Rust: one table row per `LocalClass` x status x provider family, whole-value assertions + - Rust: `RequestFormat{value}` message equals the Python text, including `repr` quoting of the value. Cover a string with a quote in it + - Sweep: extend `test_parity.py` with a validation axis, written before the port so the rows decide the table above. Cases: bad `req_format`, empty file, missing file, unreadable file, missing document URL, invalid MIME type, invalid data URI, Cohere with a PDF, Reducto with a plain URL, invalid `pages`, invalid provider, missing Azure AI credentials, missing Document Intelligence credentials, missing Reducto key. Each for sync and async. Every row asserts no request reached the local server + - `tests/test_litellm/rust_bridge/ocr/test_route_host.py`: drop the `map_failure` cases with the function + +### 4.5 Chat, messages, responses, audio transcription (later, not part of the OCR work) + +- These routes are not on the driver. They also have decline semantics that OCR does not: an error before dispatch must stay `RustBridgeDeclined` so `runtime.attempt` can fall back to Python +- Code + - Per route in core: `fn disposition(&Error) -> Disposition` with `enum Disposition { Declined, Failed(OriginalException) }`. This moves the pre-dispatch list out of `chat_completions_error_to_pyerr` into core, where it can be tested without Python + - Per route provider config: `error_class` + - Bridge: build `ExceptionContext` at entry while attached, before `run_sync` / `run_async` detach. The `map_err` closure becomes `Declined -> RustBridgeDeclined`, `Failed(original) -> BridgeFailure(exception_type(&context, &original)).into()` + - `RustUpstreamError` itself is deleted here, with its `.pyi` entry +- Tests + - Rust core: `disposition` table per route, one row per error variant. Assert that every variant is covered by matching exhaustively in the test helper, so a new variant fails to compile until it has a row + - Rust core: `error_class` per provider config + - Python `test_runtime.py`: a declined call still falls back, a classified litellm exception passes through `attempt` untouched (no wrapping, no prefix), sync and async + - Sweep: add `tests/test_litellm_rust/chat/` and `messages/` parity files using the same `support/parity.py` + - Proxy and Router: one smoke sweep per route, same shape as 4.7 + +### 4.6 Split the classifier into rule tables + +- Done last on purpose: by then the sweep and the whole-value tests pin the behavior, so the split is a pure refactor +- Code: `rules.rs`, `status.rs`, `openai.rs`, `vertex_ai.rs`, `cohere.rs`. Existing test cases move next to their rule file unchanged +- Tests + - One case per rule, named after the rule + - One case per pair of rules that can both match, asserting the earlier one wins (for example a 400 whose body has both a context-window marker and a content-policy marker) + - Separate test modules for `status.rs` (every mapped status plus one unmapped), `extra_information` (each optional context field present and absent), redaction (message redacted, markers still matched on the redacted string), and `print_banner` against `suppress_debug_info` + - Run `cargo mutants -p litellm-core --file 'src/litellm_core_utils/exception_mapping_utils/**'` and hold the 90% kill rate. Survivors become table rows + +### 4.7 Router and proxy smoke sweep + +- `tests/test_litellm_rust/ocr/test_router_and_proxy.py` has five hand-written tests today and they pass. Shrink it to one parametrized sweep: `(upstream status, body, headers) -> (proxy HTTP status, error JSON type and code, retried or not, deployment cooled down or not)`, run against both backends through `support/parity.py` +- Keep one success row. Add one row each from the transport and validation axes, so the Router's retry and cooldown decisions are checked for a `Timeout`, an `APIConnectionError` and a `Passthrough` +- Router and proxy only consume the exception, so nothing beyond this smoke sweep belongs here. A failure in it is fixed in the classifier or the constructor, never in the Router +- Can be done at any point. Doing it right after 4.1 gives the later steps a cheaper end-to-end signal + +### 4.8 Known gaps in the `exception_type` port + +All three are unreachable for today's OCR models, so they cannot be `Divergence` rows (`assert_parity` fails a registered divergence that does not diverge). They are recorded as a `KNOWN_GAPS` doc comment at the top of `exception_mapping_utils/mod.rs` and in the PR body. Each has a trigger that says when it stops being acceptable + +- The Vertex partner URL for "claude" models in `extra_information`. Trigger: a Vertex route whose models include Anthropic partner models. Then `api_base` gets the branch and a table row +- A stripped model name that happens to resolve in `get_llm_provider`, which changes the provider Python reports. Trigger: a route whose model names overlap the model cost map. Needs the provider resolution port, not a classifier change +- Python tracebacks embedded in fallback messages (`traceback.format_exc()` in the `APIConnectionError` fallback). Rust cannot reproduce this and should not. Permanent, deliberate divergence. If a sweep row ever reaches it, register a `Divergence` on `message` and compare the prefix before the traceback + +## 5. Test ownership summary + +| Contract | Owner | Location | Lifetime | +|---|---|---|---| +| `Error -> OriginalException` | core provider config | `core/src//provider_config.rs` tests | permanent | +| `Error -> Declined \| Failed` | core route | `core/src//error.rs` tests | permanent | +| `(OriginalException, ExceptionContext) -> PublicFailure` | core classifier | next to each rule file | permanent | +| `PublicFailure` JSON shape | core + shared Python | golden fixtures, written by Rust, read by `test_failures.py` | permanent | +| classify once, host errors skip, cancellation, classifier failure surfaces | `host-python` driver | `driver.rs` tests with the fake route | permanent | +| `litellm.*` globals and kwargs -> `ExceptionContext` | `python-bridge` | `failures.rs` tests | permanent | +| `num_retries`, `timeout`, `str(error)` | Python dispatch | `test_dispatch.py` | permanent | +| Python path vs Rust path | nobody, it is an oracle | `tests/test_litellm_rust/` | deleted with the Python path | + +## 6. Open questions + +1. Polling errors (`PollTimeout`, `PollLocation`, `PollOrigin`, `OperationStatus`) are `RuntimeError` today. They likely fit `Timeout` and `Response`, but the Python polling code decides. Add sweep rows for Document Intelligence polling before choosing +2. The mapped error's `__context__` is the `RustUpstreamError` today. After 4.2 there is no native `PyErr`. If the sweep compares `__context__` or `__cause__`, record one divergence. Otherwise nothing to do +3. `ExceptionContext` needs core's provider enum. OCR has its own `OcrProvider`, and chat has another. If there is no shared enum yet, 4.1 uses a small `enum ExceptionFamily { OpenAiCompatible, VertexAi, Cohere, Other }` derived by each route, and the shared provider enum is a separate change diff --git a/tests/test_litellm_rust/conftest.py b/tests/test_litellm_rust/conftest.py index 4387ea2e2fd..6979162c361 100644 --- a/tests/test_litellm_rust/conftest.py +++ b/tests/test_litellm_rust/conftest.py @@ -84,6 +84,13 @@ async def isolate_ocr_test_state() -> AsyncIterator[None]: await GLOBAL_LOGGING_WORKER.stop() +@pytest.fixture(params=[False, True], ids=["python", "rust"]) +def ocr_backend(request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch) -> bool: + enabled: Final = bool(request.param) + monkeypatch.setenv("LITELLM_RUST", "1" if enabled else "0") + return enabled + + @pytest.fixture def recording_server() -> Generator[RecordingServer]: with recording_service() as server: diff --git a/tests/test_litellm_rust/ocr/conftest.py b/tests/test_litellm_rust/ocr/conftest.py new file mode 100644 index 00000000000..5f06364837f --- /dev/null +++ b/tests/test_litellm_rust/ocr/conftest.py @@ -0,0 +1,12 @@ +from typing import Final + +import pytest + +from tests.test_litellm_rust.ocr.parity_todo import TODO + + +@pytest.fixture(autouse=True) +def parity_todo(request: pytest.FixtureRequest) -> None: + reason: Final = TODO.get(request.node.name) + if reason is not None: + request.applymarker(pytest.mark.xfail(strict=True, reason=reason)) diff --git a/tests/test_litellm_rust/ocr/parity_todo.py b/tests/test_litellm_rust/ocr/parity_todo.py new file mode 100644 index 00000000000..7f3e9fe34ca --- /dev/null +++ b/tests/test_litellm_rust/ocr/parity_todo.py @@ -0,0 +1,229 @@ +from types import MappingProxyType +from typing import Final + +TODO: Final = MappingProxyType( + { + "test_transport_parity[azure-cohere-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[azure-cohere-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[azure-cohere-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[azure-cohere-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[azure-cohere-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[azure-cohere-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[azure-cohere-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[azure-cohere-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[azure-cohere-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[azure-document-intelligence-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[azure-document-intelligence-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[azure-document-intelligence-missing-fields-async]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[azure-document-intelligence-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[azure-document-intelligence-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[azure-document-intelligence-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[azure-document-intelligence-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[azure-document-intelligence-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[azure-document-intelligence-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[azure-document-intelligence-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[azure-mistral-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[azure-mistral-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[azure-mistral-missing-fields-async]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[azure-mistral-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[azure-mistral-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[azure-mistral-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[azure-mistral-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[azure-mistral-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[azure-mistral-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[azure-mistral-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[cohere-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[cohere-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[cohere-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[cohere-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[cohere-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[cohere-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[cohere-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[cohere-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[cohere-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[mistral-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[mistral-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[mistral-missing-fields-async]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[mistral-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[mistral-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[mistral-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[mistral-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[mistral-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[mistral-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[mistral-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[reducto-legacy-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[reducto-legacy-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[reducto-legacy-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[reducto-legacy-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[reducto-legacy-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[reducto-legacy-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[reducto-legacy-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[reducto-legacy-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[reducto-legacy-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[reducto-v3-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[reducto-v3-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[reducto-v3-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[reducto-v3-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[reducto-v3-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[reducto-v3-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[reducto-v3-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[reducto-v3-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[reducto-v3-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[vertex-deepseek-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[vertex-deepseek-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[vertex-deepseek-missing-fields-async]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[vertex-deepseek-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[vertex-deepseek-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[vertex-deepseek-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[vertex-deepseek-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[vertex-deepseek-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[vertex-deepseek-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[vertex-deepseek-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[vertex-mistral-connection-refused-async]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[vertex-mistral-connection-refused-sync]": "4.2: Python reports a refused connection as a 500 from the provider", + "test_transport_parity[vertex-mistral-missing-fields-async]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[vertex-mistral-missing-fields-sync]": "4.2: responses missing fields do not map like the Python transformation", + "test_transport_parity[vertex-mistral-not-json-async]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[vertex-mistral-not-json-sync]": "4.2: unparseable responses carry the Rust parser's message", + "test_transport_parity[vertex-mistral-oversized-async]": "4.2: Python has no response size limit", + "test_transport_parity[vertex-mistral-oversized-sync]": "4.2: Python has no response size limit", + "test_transport_parity[vertex-mistral-read-timeout-async]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_transport_parity[vertex-mistral-read-timeout-sync]": "4.2: timeouts carry a Rust message instead of the Python handler's", + "test_validation_parity[azure-cohere-pdf-async]": "4.4: Cohere image-only text differs", + "test_validation_parity[azure-cohere-pdf-sync]": "4.4: Cohere image-only text differs", + "test_validation_parity[bad-req-format-azure-cohere-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-azure-cohere-sync]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-azure-document-intelligence-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-azure-document-intelligence-sync]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-azure-mistral-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-azure-mistral-sync]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-cohere-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-cohere-sync]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-mistral-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-mistral-sync]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-reducto-legacy-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-reducto-legacy-sync]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-reducto-v3-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-reducto-v3-sync]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-vertex-deepseek-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-vertex-deepseek-sync]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-vertex-mistral-async]": "4.4: req_format text differs", + "test_validation_parity[bad-req-format-vertex-mistral-sync]": "4.4: req_format text differs", + "test_validation_parity[cohere-pdf-async]": "4.4: Cohere image-only text differs", + "test_validation_parity[cohere-pdf-sync]": "4.4: Cohere image-only text differs", + "test_validation_parity[empty-file-azure-cohere-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-azure-cohere-sync]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-azure-document-intelligence-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-azure-document-intelligence-sync]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-azure-mistral-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-azure-mistral-sync]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-cohere-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-cohere-sync]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-mistral-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-mistral-sync]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-reducto-legacy-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-reducto-legacy-sync]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-reducto-v3-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-reducto-v3-sync]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-vertex-deepseek-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-vertex-deepseek-sync]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-vertex-mistral-async]": "4.4: local ValueError classification", + "test_validation_parity[empty-file-vertex-mistral-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-data-uri-azure-cohere-async]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-azure-cohere-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-azure-document-intelligence-async]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-azure-document-intelligence-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-azure-mistral-async]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-azure-mistral-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-cohere-async]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-cohere-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-mistral-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-reducto-legacy-async]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-reducto-legacy-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-reducto-v3-async]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-reducto-v3-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-vertex-deepseek-async]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-vertex-deepseek-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-vertex-mistral-async]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-data-uri-vertex-mistral-sync]": "4.4: Rust rejects data URIs Python forwards", + "test_validation_parity[invalid-mime-type-azure-cohere-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-azure-cohere-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-azure-document-intelligence-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-azure-document-intelligence-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-azure-mistral-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-azure-mistral-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-cohere-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-cohere-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-mistral-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-mistral-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-reducto-legacy-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-reducto-legacy-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-reducto-v3-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-reducto-v3-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-vertex-deepseek-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-vertex-deepseek-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-vertex-mistral-async]": "4.4: local ValueError classification", + "test_validation_parity[invalid-mime-type-vertex-mistral-sync]": "4.4: local ValueError classification", + "test_validation_parity[invalid-pages-async]": "4.4: pages validation text differs", + "test_validation_parity[invalid-pages-sync]": "4.4: pages validation text differs", + "test_validation_parity[invalid-provider-async]": "4.4: invalid provider text differs", + "test_validation_parity[invalid-provider-sync]": "4.4: invalid provider text differs", + "test_validation_parity[missing-azure-ai-credentials-async]": "4.4: credential text differs", + "test_validation_parity[missing-azure-ai-credentials-sync]": "4.4: credential text differs", + "test_validation_parity[missing-document-intelligence-credentials-async]": "4.4: credential text differs", + "test_validation_parity[missing-document-intelligence-credentials-sync]": "4.4: credential text differs", + "test_validation_parity[missing-document-url-azure-cohere-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-azure-cohere-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-azure-document-intelligence-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-azure-document-intelligence-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-azure-mistral-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-azure-mistral-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-cohere-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-cohere-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-mistral-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-mistral-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-reducto-legacy-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-reducto-legacy-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-reducto-v3-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-reducto-v3-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-vertex-deepseek-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-vertex-deepseek-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-vertex-mistral-async]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-document-url-vertex-mistral-sync]": "4.4: Python rejects an empty document URL before provider resolution", + "test_validation_parity[missing-file-cohere-async]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-cohere-sync]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-reducto-legacy-async]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-reducto-legacy-sync]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-reducto-v3-async]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-reducto-v3-sync]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-vertex-deepseek-async]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-vertex-deepseek-sync]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-vertex-mistral-async]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-file-vertex-mistral-sync]": "4.4: local FileNotFoundError classification", + "test_validation_parity[missing-reducto-key-async]": "4.4: credential error classification", + "test_validation_parity[missing-reducto-key-sync]": "4.4: credential error classification", + "test_validation_parity[reducto-legacy-plain-url-async]": "4.4: Reducto source text differs", + "test_validation_parity[reducto-legacy-plain-url-sync]": "4.4: Reducto source text differs", + "test_validation_parity[reducto-plain-url-async]": "4.4: Reducto source text differs", + "test_validation_parity[reducto-plain-url-sync]": "4.4: Reducto source text differs", + "test_validation_parity[unreadable-file-azure-cohere-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-azure-cohere-sync]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-azure-document-intelligence-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-azure-document-intelligence-sync]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-azure-mistral-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-azure-mistral-sync]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-cohere-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-cohere-sync]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-mistral-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-mistral-sync]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-reducto-legacy-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-reducto-legacy-sync]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-reducto-v3-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-reducto-v3-sync]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-vertex-deepseek-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-vertex-deepseek-sync]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-vertex-mistral-async]": "4.4: local OSError classification", + "test_validation_parity[unreadable-file-vertex-mistral-sync]": "4.4: local OSError classification", + "test_router_and_proxy_parity[missing-document-url]": "4.4: Python rejects an empty document URL before provider resolution", + } +) diff --git a/tests/test_litellm_rust/ocr/test_callbacks.py b/tests/test_litellm_rust/ocr/test_callbacks.py index 27cdcc4d997..53c92a210e3 100644 --- a/tests/test_litellm_rust/ocr/test_callbacks.py +++ b/tests/test_litellm_rust/ocr/test_callbacks.py @@ -13,6 +13,7 @@ from tests.test_litellm_rust.support.callback_recorder import RecordingLogger from tests.test_litellm_rust.support.requests import ( OCR_DOCUMENT, OCR_RESPONSE, + call_native, call_native_aocr, call_native_ocr, request_body, @@ -37,6 +38,24 @@ async def call_native_aocr_with_callbacks(server: RecordingServer, callbacks: li return await call_native_aocr(server, callbacks=callbacks, **kwargs) +@pytest.mark.asyncio +@pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"]) +async def test_ocr_contract_post_call_event_carries_the_api_key_and_only_the_request_body( + ocr_server: RecordingServer, ocr_backend: bool, asynchronous: bool +) -> None: + observations: Final = [] + + class Observe(CustomLogger): + def log_post_api_call(self, kwargs, response_obj, start_time, end_time): + observations.append((kwargs["api_key"], copy.deepcopy(kwargs["additional_args"]))) + + await call_native(ocr_server, asynchronous, callbacks=[Observe()]) + + assert observations == [ + ("test-key", {"complete_input_dict": {"model": "mistral-ocr-latest", "document": OCR_DOCUMENT}}) + ] + + def test_native_ocr_pre_call_callback_receives_transformed_provider_request(ocr_server: RecordingServer) -> None: observations: Final = [] @@ -123,9 +142,7 @@ async def test_native_ocr_pre_call_nested_document_edit_updates_caller_callback_ "callbacks": [Retain(), Edit()], } response: Final = ( - await call_native_aocr(ocr_server, **arguments) - if asynchronous - else call_native_ocr(ocr_server, **arguments) + await call_native_aocr(ocr_server, **arguments) if asynchronous else call_native_ocr(ocr_server, **arguments) ) assert aliases == [True] diff --git a/tests/test_litellm_rust/ocr/test_parity.py b/tests/test_litellm_rust/ocr/test_parity.py new file mode 100644 index 00000000000..74cafff37b3 --- /dev/null +++ b/tests/test_litellm_rust/ocr/test_parity.py @@ -0,0 +1,357 @@ +import re +import socket +from collections.abc import Awaitable, Callable, Mapping +from dataclasses import dataclass +from pathlib import Path +from types import MappingProxyType +from typing import Final + +import pytest + +import litellm +from litellm.llms.base_llm.ocr.transformation import OCRResponse +from tests.test_litellm_rust.support.parity import Divergence, Masked, Outcome, assert_parity, run_on +from tests.test_litellm_rust.support.recording_server import RecordingServer, ResponseSpec +from tests.test_litellm_rust.support.requests import OCR_DOCUMENT + +pytestmark = pytest.mark.requires_rust_extension + +IMAGE_DOCUMENT: Final = {"type": "image_url", "image_url": "data:image/png;base64,YWJj"} +REDUCTO_DOCUMENT: Final = {"type": "document_url", "document_url": "reducto://ready.pdf"} +VERTEX: Final = {"vertex_project": "test-project", "vertex_location": "us-central1"} + + +@dataclass(frozen=True, slots=True) +class Provider: + model: str + document: Mapping[str, object] + arguments: Mapping[str, object] + + +PROVIDERS: Final = { + "mistral": Provider("mistral/mistral-ocr-latest", OCR_DOCUMENT, {}), + "azure-mistral": Provider("azure_ai/mistral-ocr-latest", OCR_DOCUMENT, {}), + "azure-document-intelligence": Provider("azure_ai/doc-intelligence/prebuilt-layout", OCR_DOCUMENT, {}), + "azure-cohere": Provider("azure_ai/Cohere-parse-v5.0", IMAGE_DOCUMENT, {}), + "cohere": Provider("cohere/parse-v5.0", IMAGE_DOCUMENT, {}), + "vertex-mistral": Provider("vertex_ai/mistral-ocr-latest", OCR_DOCUMENT, VERTEX), + "vertex-deepseek": Provider("vertex_ai/deepseek-ocr", OCR_DOCUMENT, VERTEX), + "reducto-v3": Provider("reducto/parse-v3", REDUCTO_DOCUMENT, {}), + "reducto-legacy": Provider("reducto/parse-legacy", REDUCTO_DOCUMENT, {}), +} + +STATUSES: Final = (400, 401, 403, 404, 408, 409, 413, 422, 429, 498, 500, 502, 503, 504, 529) + +BODIES: Final = { + "json-object": {"message": "rejected"}, + "json-string": "rejected", + "empty-object": {}, + "rate-limit-text": {"message": "rate limit reached for requests"}, + "context-window-text": {"message": "This model's maximum context length is 10 tokens"}, + "content-policy-text": {"message": "content_policy_violation"}, + "invalid-request-text": {"error": {"type": "invalid_request_error", "message": "bad field"}}, + "model-not-found-text": {"error": {"type": "invalid_request_error", "code": "model_not_found"}}, + "timeout-text": {"message": "Request timed out"}, + "vertex-quota-text": {"message": "429 Quota exceeded for aiplatform"}, + "vertex-api-key-text": {"message": "API key not valid. Please pass a valid API key."}, + "cohere-api-token-text": {"message": "invalid api token"}, + "cohere-internal-text": {"message": "Internal Server Error"}, +} + +REPRESENTATIVE_STATUSES: Final = (400, 429, 500) + +NO_RETRIES: Final = MappingProxyType({"num_retries": 0}) +CALL_OPTIONS: Final = { + "no-retries": NO_RETRIES, + "retries-and-timeout": MappingProxyType({"num_retries": 0, "timeout": 7.5}), + "defaults": MappingProxyType({}), +} + +VERTEX_UPSTREAM_CONTENT: Final = tuple( + Divergence(field, "Rust keeps the upstream body and headers that _map_vertex_exception replaces with a stub") + for field in ("response_text", "response_headers") +) + + +def upstream_divergences( + asynchronous: bool, provider: Provider, options: Mapping[str, object], python: Outcome +) -> tuple[Divergence, ...]: + sync: Final = tuple( + Divergence(field, "Rust sets the request retry count and timeout on sync failures; Python only on async") + for field in ("num_retries", "timeout") + if not asynchronous and options.get(field) is not None + ) + vertex: Final = provider.model.startswith("vertex_ai/") and python.response_text == "" + return sync + (VERTEX_UPSTREAM_CONTENT if vertex else ()) + + +def arguments(server: RecordingServer, provider: Provider, **extra: object) -> dict[str, object]: + return { + "model": provider.model, + "document": dict(provider.document), + "api_key": "test-key", + "api_base": server.base_url, + **provider.arguments, + **extra, + } + + +async def call( + server: RecordingServer, asynchronous: bool, provider: Provider, options: Mapping[str, object] +) -> OCRResponse: + if asynchronous: + return await litellm.aocr(**arguments(server, provider, **options)) + return litellm.ocr(**arguments(server, provider, **options)) + + +async def compare_upstream( + server: RecordingServer, + monkeypatch: pytest.MonkeyPatch, + provider: Provider, + response: ResponseSpec, + asynchronous: bool, + options: Mapping[str, object] = NO_RETRIES, + masks: tuple[Masked, ...] = (), +) -> None: + server.expected_requests = None + server.enqueue(response) + python: Final = await run_on(False, monkeypatch, lambda: call(server, asynchronous, provider, options)) + server.enqueue(response) + rust: Final = await run_on(True, monkeypatch, lambda: call(server, asynchronous, provider, options)) + assert_parity(python, rust, upstream_divergences(asynchronous, provider, options, python), masks) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"]) +@pytest.mark.parametrize("status", STATUSES) +@pytest.mark.parametrize("provider", PROVIDERS.values(), ids=PROVIDERS.keys()) +async def test_upstream_status_parity( + recording_server: RecordingServer, + monkeypatch: pytest.MonkeyPatch, + provider: Provider, + status: int, + asynchronous: bool, +) -> None: + await compare_upstream( + recording_server, + monkeypatch, + provider, + ResponseSpec(body=BODIES["json-object"], status=status, headers={"retry-after": "7"}), + asynchronous, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"]) +@pytest.mark.parametrize("status", REPRESENTATIVE_STATUSES) +@pytest.mark.parametrize("body", BODIES.values(), ids=BODIES.keys()) +@pytest.mark.parametrize("provider", PROVIDERS.values(), ids=PROVIDERS.keys()) +async def test_upstream_body_parity( + recording_server: RecordingServer, + monkeypatch: pytest.MonkeyPatch, + provider: Provider, + body: object, + status: int, + asynchronous: bool, +) -> None: + await compare_upstream( + recording_server, monkeypatch, provider, ResponseSpec(body=body, status=status), asynchronous + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"]) +@pytest.mark.parametrize("options", CALL_OPTIONS.values(), ids=CALL_OPTIONS.keys()) +@pytest.mark.parametrize("status", (401, 500)) +@pytest.mark.parametrize("provider", PROVIDERS.values(), ids=PROVIDERS.keys()) +async def test_call_option_parity( + recording_server: RecordingServer, + monkeypatch: pytest.MonkeyPatch, + provider: Provider, + status: int, + options: Mapping[str, object], + asynchronous: bool, +) -> None: + await compare_upstream( + recording_server, + monkeypatch, + provider, + ResponseSpec(body=BODIES["json-object"], status=status), + asynchronous, + options, + ) + + +def closed_port_base_url() -> str: + with socket.socket() as probe: + probe.bind(("127.0.0.1", 0)) + host, port = probe.getsockname() + return f"http://{host}:{port}" + + +@dataclass(frozen=True, slots=True) +class Transport: + response: ResponseSpec | None + options: Mapping[str, object] + + +TRANSPORTS: Final = { + "read-timeout": Transport(ResponseSpec(body={}, delay=1.5), {**NO_RETRIES, "timeout": 0.5}), + "connection-refused": Transport(None, NO_RETRIES), + "not-json": Transport(ResponseSpec(body=None, raw=b"bad gateway"), NO_RETRIES), + "missing-fields": Transport(ResponseSpec(body={"unexpected": True}), NO_RETRIES), + "oversized": Transport(ResponseSpec(body={"padding": "x" * 64}), {**NO_RETRIES, "max_response_bytes": 16}), +} + + +def replaced(pattern: re.Pattern[str], placeholder: str) -> Callable[[object], object]: + return lambda value: pattern.sub(placeholder, value) if isinstance(value, str) else value + + +ELAPSED: Final = tuple( + Masked( + field, + "both handlers report how long the timed-out call ran", + replaced(re.compile(r"time taken=[0-9.]+ seconds"), "time taken= seconds"), + ) + for field in ("text", "message") +) + +REFUSED_CONNECTION: Final = re.compile(r"\[Errno \d+\] Connection refused|error sending request") + + +def refused_connection_divergences(provider: Provider) -> tuple[Divergence, ...]: + reason: Final = "httpx and reqwest word a refused connection differently" + text: Final = tuple( + Divergence(field, reason, replaced(REFUSED_CONNECTION, "")) for field in ("text", "message") + ) + vertex: Final = (Divergence("response_text", reason),) if provider.model.startswith("vertex_ai/") else () + return text + vertex + + +@pytest.mark.asyncio +@pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"]) +@pytest.mark.parametrize("transport", TRANSPORTS.values(), ids=TRANSPORTS.keys()) +@pytest.mark.parametrize("provider", PROVIDERS.values(), ids=PROVIDERS.keys()) +async def test_transport_parity( + recording_server: RecordingServer, + monkeypatch: pytest.MonkeyPatch, + provider: Provider, + transport: Transport, + asynchronous: bool, +) -> None: + if transport.response is not None: + await compare_upstream( + recording_server, monkeypatch, provider, transport.response, asynchronous, transport.options, ELAPSED + ) + return + recording_server.expected_requests = 0 + unreachable: Final = {**transport.options, "api_base": closed_port_base_url()} + python: Final = await run_on( + False, monkeypatch, lambda: call(recording_server, asynchronous, provider, unreachable) + ) + rust: Final = await run_on(True, monkeypatch, lambda: call(recording_server, asynchronous, provider, unreachable)) + assert_parity( + python, + rust, + upstream_divergences(asynchronous, provider, unreachable, python) + refused_connection_divergences(provider), + ) + + +@dataclass(frozen=True, slots=True) +class Validation: + document: Callable[[Path], Mapping[str, object]] | None = None + options: Mapping[str, object] = NO_RETRIES + api_key: str | None = "test-key" + + +def file_document(file: object, **extra: object) -> Mapping[str, object]: + return {"type": "file", "file": file, **extra} + + +def unreadable_file(directory: Path) -> Mapping[str, object]: + path: Final = directory / "unreadable.pdf" + if not path.exists(): + path.write_bytes(b"%PDF-1.4") + path.chmod(0) + return file_document(path) + + +class RaisingReader: + def read(self) -> bytes: + raise ValueError("reader exploded") + + +def raising_token_provider() -> str: + raise ValueError("token unavailable") + + +def url_document(url: str) -> Callable[[Path], Mapping[str, object]]: + return lambda _: {"type": "document_url", "document_url": url} + + +GENERIC_VALIDATIONS: Final = { + "bad-req-format": Validation(options={**NO_RETRIES, "req_format": "it's-bogus"}), + "empty-file": Validation(lambda _: file_document(b"")), + "missing-file": Validation(lambda directory: file_document(directory / "missing.pdf")), + "unreadable-file": Validation(unreadable_file), + "missing-document-url": Validation(url_document("")), + "invalid-mime-type": Validation(lambda _: file_document(b"abc", mime_type="not a mime type")), + "invalid-data-uri": Validation(url_document("data:application/pdf;base64")), +} + +SPECIFIC_VALIDATIONS: Final = { + "cohere-pdf": ("cohere", Validation(url_document(str(OCR_DOCUMENT["document_url"])))), + "azure-cohere-pdf": ("azure-cohere", Validation(url_document(str(OCR_DOCUMENT["document_url"])))), + "reducto-plain-url": ("reducto-v3", Validation(url_document("https://example.com/scan.pdf"))), + "reducto-legacy-plain-url": ("reducto-legacy", Validation(url_document("https://example.com/scan.pdf"))), + "invalid-pages": ("azure-document-intelligence", Validation(options={**NO_RETRIES, "pages": [-1]})), + "missing-azure-ai-credentials": ("azure-mistral", Validation(api_key=None)), + "missing-document-intelligence-credentials": ("azure-document-intelligence", Validation(api_key=None)), + "missing-reducto-key": ("reducto-v3", Validation(api_key=None)), + "raising-file-reader": ("mistral", Validation(lambda _: file_document(RaisingReader()))), + "raising-token-provider": ( + "azure-mistral", + Validation(options={**NO_RETRIES, "azure_ad_token_provider": raising_token_provider}, api_key=None), + ), +} + +VALIDATION_ROWS: Final = { + **{ + f"{name}-{provider}": (PROVIDERS[provider], validation) + for name, validation in GENERIC_VALIDATIONS.items() + for provider in PROVIDERS + }, + **{name: (PROVIDERS[provider], validation) for name, (provider, validation) in SPECIFIC_VALIDATIONS.items()}, + "invalid-provider": (Provider("not_a_provider/ocr-model", OCR_DOCUMENT, {}), Validation()), +} + +CREDENTIAL_VARIABLES: Final = ("AZURE_DOCUMENT_INTELLIGENCE_API_KEY", "REDUCTO_API_KEY", "MISTRAL_API_KEY") + + +@pytest.mark.asyncio +@pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"]) +@pytest.mark.parametrize("row", VALIDATION_ROWS.values(), ids=VALIDATION_ROWS.keys()) +async def test_validation_parity( + recording_server: RecordingServer, + monkeypatch: pytest.MonkeyPatch, + isolated_azure_auth: None, + tmp_path: Path, + row: tuple[Provider, Validation], + asynchronous: bool, +) -> None: + provider, validation = row + for variable in CREDENTIAL_VARIABLES: + monkeypatch.delenv(variable, raising=False) + recording_server.expected_requests = None + + def invoke() -> Awaitable[OCRResponse]: + document: Final = provider.document if validation.document is None else validation.document(tmp_path) + options: Final = {**validation.options, "document": dict(document), "api_key": validation.api_key} + return call(recording_server, asynchronous, provider, options) + + python: Final = await run_on(False, monkeypatch, invoke) + python_requests: Final = len(recording_server.requests) + rust: Final = await run_on(True, monkeypatch, invoke) + assert len(recording_server.requests) - python_requests == python_requests + assert_parity(python, rust, upstream_divergences(asynchronous, provider, validation.options, python)) diff --git a/tests/test_litellm_rust/ocr/test_requests.py b/tests/test_litellm_rust/ocr/test_requests.py index 5e9d2c78808..4c4e595dfb4 100644 --- a/tests/test_litellm_rust/ocr/test_requests.py +++ b/tests/test_litellm_rust/ocr/test_requests.py @@ -24,13 +24,6 @@ from tests.test_litellm_rust.support.requests import ( pytestmark = pytest.mark.requires_rust_extension -@pytest.fixture(params=[False, True], ids=["python", "rust"]) -def ocr_backend(request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch) -> bool: - enabled: Final = bool(request.param) - monkeypatch.setenv("LITELLM_RUST", "1" if enabled else "0") - return enabled - - @pytest.mark.asyncio @pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"]) async def test_ocr_contract_upstream_status( diff --git a/tests/test_litellm_rust/ocr/test_router_and_proxy.py b/tests/test_litellm_rust/ocr/test_router_and_proxy.py new file mode 100644 index 00000000000..e99d5f438d4 --- /dev/null +++ b/tests/test_litellm_rust/ocr/test_router_and_proxy.py @@ -0,0 +1,98 @@ +from collections.abc import Mapping +from dataclasses import dataclass +from types import MappingProxyType +from typing import Final + +import pytest +from fastapi.testclient import TestClient + +from litellm import Router +from litellm.proxy import proxy_server +from tests.test_litellm_rust.support.parity import assert_parity +from tests.test_litellm_rust.support.recording_server import RecordingServer, ResponseSpec +from tests.test_litellm_rust.support.requests import OCR_DOCUMENT, OCR_MODEL, OCR_RESPONSE + +pytestmark = pytest.mark.requires_rust_extension + +MODEL_GROUP: Final = "ocr" +ROUTER_RETRIES: Final = 1 + + +@dataclass(frozen=True, slots=True) +class Row: + upstream: ResponseSpec + document: Mapping[str, object] = MappingProxyType(dict(OCR_DOCUMENT)) + deployment: Mapping[str, object] = MappingProxyType({}) + + +@dataclass(frozen=True, slots=True) +class ProxyOutcome: + status: int + error_type: object + error_code: object + upstream_requests: int + cooled_down: int + timeout_header: bool + markdown: object + + +ROWS: Final = { + "success": Row(ResponseSpec(body=OCR_RESPONSE)), + "unauthorized": Row(ResponseSpec(body={"message": "rejected"}, status=401)), + "not-found": Row(ResponseSpec(body={"message": "no such model"}, status=404)), + "rate-limited": Row(ResponseSpec(body={"message": "slow down"}, status=429, headers={"retry-after": "0"})), + "provider-failure": Row(ResponseSpec(body={"message": "provider unavailable"}, status=500)), + "read-timeout": Row(ResponseSpec(body=OCR_RESPONSE, delay=1.5), deployment=MappingProxyType({"timeout": 0.5})), + "not-json": Row(ResponseSpec(body=None, raw=b"bad gateway")), + "missing-document-url": Row( + ResponseSpec(body=OCR_RESPONSE), document=MappingProxyType({"type": "document_url", "document_url": ""}) + ), +} + + +def observe(server: RecordingServer, row: Row, monkeypatch: pytest.MonkeyPatch, backend: bool) -> ProxyOutcome: + monkeypatch.setenv("LITELLM_RUST", "1" if backend else "0") + server.default_response = row.upstream + before: Final = len(server.requests) + router: Final = Router( + model_list=[ + { + "model_name": MODEL_GROUP, + "litellm_params": { + "model": OCR_MODEL, + "api_key": "test-key", + "api_base": server.base_url, + **row.deployment, + }, + } + ], + num_retries=ROUTER_RETRIES, + retry_after=0, + ) + monkeypatch.setattr(proxy_server, "llm_router", router) + monkeypatch.setattr(proxy_server, "master_key", None) + monkeypatch.setattr(proxy_server, "general_settings", {}) + with TestClient(proxy_server.app, raise_server_exceptions=False) as client: + response: Final = client.post("/v1/ocr", json={"model": MODEL_GROUP, "document": dict(row.document)}) + body: Final = response.json() + error: Final = body.get("error") if isinstance(body, dict) else None + pages: Final = body.get("pages") if isinstance(body, dict) else None + if response.status_code == 200: + assert response.headers.get("x-litellm-rust") == ("true" if backend else None) + return ProxyOutcome( + status=response.status_code, + error_type=None if not isinstance(error, dict) else error.get("type"), + error_code=None if not isinstance(error, dict) else error.get("code"), + upstream_requests=len(server.requests) - before, + cooled_down=len(router.cooldown_cache.get_active_cooldowns(router.get_model_ids(), parent_otel_span=None)), + timeout_header="x-litellm-timeout" in response.headers, + markdown=pages[0].get("markdown") if isinstance(pages, list) and pages else None, + ) + + +@pytest.mark.parametrize("row", ROWS.values(), ids=ROWS.keys()) +def test_router_and_proxy_parity(recording_server: RecordingServer, monkeypatch: pytest.MonkeyPatch, row: Row) -> None: + recording_server.expected_requests = None + python: Final = observe(recording_server, row, monkeypatch, backend=False) + rust: Final = observe(recording_server, row, monkeypatch, backend=True) + assert_parity(python, rust) diff --git a/tests/test_litellm_rust/support/parity.py b/tests/test_litellm_rust/support/parity.py new file mode 100644 index 00000000000..187ae61cf0b --- /dev/null +++ b/tests/test_litellm_rust/support/parity.py @@ -0,0 +1,142 @@ +from collections.abc import Awaitable, Callable, Mapping +from dataclasses import dataclass, fields, replace +from types import MappingProxyType +from typing import TYPE_CHECKING, Final, TypeVar + +import httpx +import pytest + +from litellm.llms.base_llm.ocr.transformation import OCRResponse + +if TYPE_CHECKING: + from _typeshed import DataclassInstance + +VOLATILE_HEADERS: Final = frozenset({"date", "server"}) + + +@dataclass(frozen=True, slots=True) +class Outcome: + kind: str + error_class: str | None = None + text: str | None = None + message: str | None = None + status_code: object = None + llm_provider: object = None + model: object = None + litellm_debug_info: object = None + num_retries: object = None + max_retries: object = None + timeout: object = None + response_status: int | None = None + response_text: str | None = None + response_headers: Mapping[str, str] | None = None + litellm_response_headers: Mapping[str, str] | None = None + result: object = None + + +def _headers(value: object) -> Mapping[str, str] | None: + if not isinstance(value, httpx.Headers | Mapping): + return None + return MappingProxyType( + {str(name).lower(): str(item) for name, item in value.items() if str(name).lower() not in VOLATILE_HEADERS} + ) + + +def _response(error: BaseException) -> httpx.Response | None: + try: + response: Final = getattr(error, "response", None) + except RuntimeError: + return None + return response if isinstance(response, httpx.Response) else None + + +def failure_outcome(error: BaseException) -> Outcome: + response: Final = _response(error) + return Outcome( + kind="error", + error_class=f"{type(error).__module__}.{type(error).__qualname__}", + text=str(error), + message=getattr(error, "message", None), + status_code=getattr(error, "status_code", None), + llm_provider=getattr(error, "llm_provider", None), + model=getattr(error, "model", None), + litellm_debug_info=getattr(error, "litellm_debug_info", None), + num_retries=getattr(error, "num_retries", None), + max_retries=getattr(error, "max_retries", None), + timeout=getattr(error, "timeout", None), + response_status=None if response is None else response.status_code, + response_text=None if response is None else response.text, + response_headers=None if response is None else _headers(response.headers), + litellm_response_headers=_headers(getattr(error, "litellm_response_headers", None)), + ) + + +def success_outcome(response: OCRResponse) -> Outcome: + return Outcome(kind="ok", result=response.model_dump()) + + +@dataclass(frozen=True, slots=True) +class Divergence: + """A field that differs by design. With `normalize`, both sides are compared after it + instead of the field being dropped.""" + + field: str + reason: str + normalize: Callable[[object], object] | None = None + + +@dataclass(frozen=True, slots=True) +class Masked: + """A field that holds a run-dependent value on both sides, compared after `mask`.""" + + field: str + reason: str + mask: Callable[[object], object] + + +async def observe(call: Callable[[], Awaitable[OCRResponse]]) -> Outcome: + try: + return success_outcome(await call()) + except Exception as error: + return failure_outcome(error) + + +async def run_on(backend: bool, monkeypatch: pytest.MonkeyPatch, call: Callable[[], Awaitable[OCRResponse]]) -> Outcome: + monkeypatch.setenv("LITELLM_RUST", "1" if backend else "0") + return await observe(call) + + +ObservedT = TypeVar("ObservedT", bound="DataclassInstance") + + +def without(outcome: ObservedT, divergences: tuple[Divergence, ...], masks: tuple[Masked, ...] = ()) -> ObservedT: + compared: Final = { + **{ + divergence.field: None + if divergence.normalize is None + else divergence.normalize(getattr(outcome, divergence.field)) + for divergence in divergences + }, + **{masked.field: masked.mask(getattr(outcome, masked.field)) for masked in masks}, + } + return replace(outcome, **compared) + + +def assert_parity( + python: ObservedT, rust: ObservedT, divergences: tuple[Divergence, ...] = (), masks: tuple[Masked, ...] = () +) -> None: + known: Final = frozenset(divergence.field for divergence in divergences) | {masked.field for masked in masks} + unknown: Final = known - {field.name for field in fields(python)} + assert not unknown, f"divergence registry names unknown fields: {sorted(unknown)}" + for divergence in divergences: + assert getattr(python, divergence.field) != getattr(rust, divergence.field), ( + f"registered divergence on {divergence.field!r} no longer diverges; remove it: {divergence.reason}" + ) + expected: Final = without(python, divergences, masks) + actual: Final = without(rust, divergences, masks) + differences: Final = { + field.name: {"python": getattr(expected, field.name), "rust": getattr(actual, field.name)} + for field in fields(python) + if getattr(expected, field.name) != getattr(actual, field.name) + } + assert not differences, differences diff --git a/tests/test_litellm_rust/support/recording_server.py b/tests/test_litellm_rust/support/recording_server.py index 228ed2cc454..abb5346f93c 100644 --- a/tests/test_litellm_rust/support/recording_server.py +++ b/tests/test_litellm_rust/support/recording_server.py @@ -25,6 +25,7 @@ class ResponseSpec: status: int = 200 headers: dict[str, str] = field(default_factory=dict) delay: float = 0 + raw: bytes | None = None @dataclass @@ -73,7 +74,7 @@ def recording_service() -> Iterator[RecordingServer]: response: Final = responses.pop(0) if responses else copy.deepcopy(recording_server.default_response) if response.delay: time.sleep(response.delay) - payload: Final = json.dumps(response.body).encode() + payload: Final = json.dumps(response.body).encode() if response.raw is None else response.raw self.send_response(response.status) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(payload)))