mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-21 00:21:49 +00:00
63d994ade4
Extracted from #41733 without the router loop, the cache machine layer, streaming, or the error, timeout and route-pruning work that moved to #41745 litellm-callbacks holds the contract a native call and its host share: Machine, HostOp, CallEvent, the in-process run loop, and Passthrough, which is built only by comparing the caller's inputs with the body the route sends, so a route can never mark a key it rewrote. litellm-host-python (formerly python-interop) owns the CPython driver and the Execution handle, and litellm-callbacks-legacy is the @client wrapper as the native call sees it: function_setup, the deployment hooks, pre_call and post_call, the success and failure fan-out and the deferred proxy release. OCR is the one route on it, and the old core and bridge lifecycles are gone The passthrough rule is the structural fix for the bug #41719 patched in core and #41716 reworks: an inlined remote document no longer counts as the caller's value, so the legacy adapter never hands the caller's URL back into the body. core/tests/ocr/passthrough.rs pins it for every route and document source, including that unchanged values stay passthrough, and callbacks-legacy/tests/payload.rs pins the adapter side with a real pre_call callback Python OCR integration tests that only exercised core behavior now live as Rust tests, so tests/test_litellm_rust keeps the cases that need the full Python stack
1.9 KiB
1.9 KiB
CLAUDE.md
Rules for litellm-rust/crates/python-bridge.
Responsibility
python-bridge is the PyO3 boundary between Python LiteLLM and Rust transforms.
Keep this crate thin. It exposes LiteLLM Rust APIs, assembles domain requests,
maps domain errors to Python exceptions, and delegates generic conversion and
GIL handling to litellm-host-python.
Bridge Shape
- Prefer one stable method per top-level LiteLLM route, for example
messages(...), calling the matchinglitellm-coreentrypoint. - Do not add one exported PyO3 function per provider helper unless there is a measured reason.
- Provider dispatch belongs in the
litellm-coreroute module (e.g.litellm_core::messages), not in this PyO3 crate. - Python owns rollout state and fallback. Rust should return errors; Python decides whether to raise or fall back. For a rust-only provider/route (no Python reference), the Python side is a thin dispatch that calls Rust and raises when the bridge is unavailable, with no fallback.
- Keep the Python interface minimal (well under 100 lines per route): it only
marshals inputs and calls Rust. Do not add per-route feature flags, and do
not put provider dispatch in
litellm/main.py; it lives in a thin dispatch class underlitellm/llms/<provider>/<route>/.
Data Handling
- OCR payloads can contain personal data and large base64 images. Do not log payloads or provider responses.
- Avoid copying large payloads more than needed. The current JSON round-trip is acceptable for the first scaffold, but future performance work should evaluate direct PyO3 conversion before expanding Rust coverage to image-heavy paths.
- Do not expose raw Rust errors that include document contents or upstream bodies.
Tests
cargo test --workspacemust compile this crate.- Python tests must cover bridge disabled, bridge enabled, and module-missing fallback behavior for every exposed route.