litellm/litellm-rust

litellm-rust

A Rust port of LiteLLM's Mistral OCR transform logic. One pure, shared core (src/llms/..., src/pipeline.rs) is exposed two ways:

1. A Python extension module litellm_rust (via PyO3) — the SDK/proxy call this.
2. A standalone axum HTTP server binary litellm-rust-server serving POST /v1/ocr.

The transform modules are pure (no PyO3, no I/O), mirroring the Python litellm/llms/ tree:

• src/llms/base_llm/ocr/transformation.rs — BaseOcrConfig trait + serde response types (OcrResponse, OcrPage, OcrPageImage, OcrUsageInfo, OcrPageDimensions) and OcrRequest.
• src/llms/mistral/ocr/transformation.rs — MistralOcrConfig.

Build the Python extension

PyO3 is an optional dependency behind the python feature, so the server binary never links libpython.

# from inside litellm-rust/
maturin develop --features python

Then from Python:

import litellm_rust

resp = litellm_rust.ocr(
    "mistral-ocr-latest",
    {"type": "document_url", "document_url": "https://example.com/doc.pdf"},
    None,            # api_key (falls back to MISTRAL_API_KEY)
    None,            # api_base (defaults to https://api.mistral.ai/v1)
    {"include_image_base64": True},
)
# -> {"pages": [...], "model": ..., "document_annotation": ..., "usage_info": {...}, "object": "ocr"}

Run the standalone server

The binary builds without the python feature:

# from inside litellm-rust/
PORT=8088 cargo run --bin litellm-rust-server

Example request:

curl -s http://localhost:8088/v1/ocr \
  -H 'Content-Type: application/json' \
  -d '{
        "model": "mistral-ocr-latest",
        "document": {"type": "document_url", "document_url": "https://example.com/doc.pdf"},
        "include_image_base64": true
      }'

api_key may be supplied in the body; if omitted, the server reads MISTRAL_API_KEY from the environment. GET /health returns {"status":"ok"}.

Dev

cargo fmt
cargo clippy --all-targets -- -D warnings
cargo clippy --lib --features python -- -D warnings
cargo check --bin litellm-rust-server