scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-21 00:21:49 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 117
litellm/litellm-rust
History
Akhil Sanjay Potdar 5ad7f8dcd2 fix(bedrock): trim comment verbosity, drop dispatch-boundary test 
Shorten the two comments added for the system-message fix to one line
each, per this repo's comment policy in litellm-rust/CLAUDE.md.

Remove test_bedrock_transcription_dispatch_adds_no_system_message: it
mocks out the entire Rust bridge call, so it never exercises the
request body construction where the bug and the fix both live. The
Rust test request_omits_system_message_alongside_audio_content already
covers this by calling transform_transcription_request directly and
asserting on its output.
2026-09-04 23:03:19 -04:00
..
crates fix(bedrock): trim comment verbosity, drop dispatch-boundary test 2026-09-04 23:03:19 -04:00
.gitignore feat: add LiteLLM Rust workspace with Mistral OCR bridge (#31033) 2026-06-23 13:16:47 -07:00
ADDING_A_PROVIDER.md docs(litellm-rust): fix the gateway run commands and point ADDING_A_PROVIDER at the one checks runbook 2026-09-03 00:07:19 -07:00
AGENTS.md refactor(rust): extract domain-neutral Python interop 2026-09-02 12:16:26 -07:00
Cargo.lock test(ocr): complete Rust unit test parity (#39689) 2026-09-03 21:15:01 -07:00
Cargo.toml refactor(tests): restructure rust python harness around strategy definitions (#39628) 2026-09-03 21:15:01 -07:00
CLAUDE.md ci(rust): lint every gateway feature and keep one checks runbook 2026-09-02 22:16:40 -07:00
README.md ci(rust): lint every gateway feature and keep one checks runbook 2026-09-02 22:16:40 -07:00

README.md

LiteLLM Rust

This workspace contains the staged Rust implementation for LiteLLM.

litellm-core is the LiteLLM SDK in Rust: one entrypoint per top-level call that makes the LLM call and hands back a typed response, the same shape as litellm.messages() in Python.

let response = litellm_core::messages::messages(MessagesRequest {
    model: "claude-sonnet-4-5",
    body,
    api_key: Some(key),
    ..
})
.await?;

Python continues to own configuration, retries, routing policy, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.

Crates

Crate Role
litellm-core The SDK. Per-route entrypoints (messages::messages()), types, provider transforms (modules under providers/), provider resolution, auth, the provider HTTP call, and the router.
litellm-ai-gateway The axum server (behind the server feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers.
litellm-python-interop Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion.
litellm-python-bridge PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping.

Dependency direction is acyclic: litellm-python-bridge depends on the domain layers and litellm-python-interop; the interop foundation depends on no LiteLLM domain crate.

Layout

crates/
  core/           The SDK: route modules + provider transforms.
    src/messages/   mod.rs (entrypoint), types, transformation, prepare, handler, client
    src/providers/anthropic/messages/transformation.rs
  ai-gateway/     Axum server + WebSocket hosts; calls core entrypoints.
  python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
  python-bridge/  PyO3 API adapter for Python LiteLLM.

The folder shape follows the Python provider tree: core/src/providers/<provider>/<route>/transformation.rs. The bridge exposes one function per top-level route, mirroring the core entrypoints.

Checks

Run the commands under "Checks" in CLAUDE.md before pushing Rust changes. That list is the single source of truth and matches what GitHub Actions runs for changes under litellm-rust/.