mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
73 lines
2.1 KiB
Markdown
73 lines
2.1 KiB
Markdown
# litellm-rust
|
|
|
|
A Rust port of LiteLLM's Mistral OCR transform logic. One pure, shared core
|
|
(`src/llms/...`, `src/pipeline.rs`) is exposed two ways:
|
|
|
|
1. A Python extension module `litellm_rust` (via PyO3) — the SDK/proxy call this.
|
|
2. A standalone axum HTTP server binary `litellm-rust-server` serving `POST /v1/ocr`.
|
|
|
|
The transform modules are pure (no PyO3, no I/O), mirroring the Python
|
|
`litellm/llms/` tree:
|
|
|
|
- `src/llms/base_llm/ocr/transformation.rs` — `BaseOcrConfig` trait + serde
|
|
response types (`OcrResponse`, `OcrPage`, `OcrPageImage`, `OcrUsageInfo`,
|
|
`OcrPageDimensions`) and `OcrRequest`.
|
|
- `src/llms/mistral/ocr/transformation.rs` — `MistralOcrConfig`.
|
|
|
|
## Build the Python extension
|
|
|
|
PyO3 is an optional dependency behind the `python` feature, so the server binary
|
|
never links libpython.
|
|
|
|
```bash
|
|
# from inside litellm-rust/
|
|
maturin develop --features python
|
|
```
|
|
|
|
Then from Python:
|
|
|
|
```python
|
|
import litellm_rust
|
|
|
|
resp = litellm_rust.ocr(
|
|
"mistral-ocr-latest",
|
|
{"type": "document_url", "document_url": "https://example.com/doc.pdf"},
|
|
None, # api_key (falls back to MISTRAL_API_KEY)
|
|
None, # api_base (defaults to https://api.mistral.ai/v1)
|
|
{"include_image_base64": True},
|
|
)
|
|
# -> {"pages": [...], "model": ..., "document_annotation": ..., "usage_info": {...}, "object": "ocr"}
|
|
```
|
|
|
|
## Run the standalone server
|
|
|
|
The binary builds without the `python` feature:
|
|
|
|
```bash
|
|
# from inside litellm-rust/
|
|
PORT=8088 cargo run --bin litellm-rust-server
|
|
```
|
|
|
|
Example request:
|
|
|
|
```bash
|
|
curl -s http://localhost:8088/v1/ocr \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
"model": "mistral-ocr-latest",
|
|
"document": {"type": "document_url", "document_url": "https://example.com/doc.pdf"},
|
|
"include_image_base64": true
|
|
}'
|
|
```
|
|
|
|
`api_key` may be supplied in the body; if omitted, the server reads
|
|
`MISTRAL_API_KEY` from the environment. `GET /health` returns `{"status":"ok"}`.
|
|
|
|
## Dev
|
|
|
|
```bash
|
|
cargo fmt
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo clippy --lib --features python -- -D warnings
|
|
cargo check --bin litellm-rust-server
|
|
```
|