scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-21 00:21:49 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 74
litellm/litellm-rust/crates/host-python/AGENTS.md
Yujong Lee 63d994ade4 refactor(rust): run OCR through a route-neutral callback contract and a legacy Logging adapter 
Extracted from #41733 without the router loop, the cache machine layer, streaming, or the
error, timeout and route-pruning work that moved to #41745

litellm-callbacks holds the contract a native call and its host share: Machine, HostOp,
CallEvent, the in-process run loop, and Passthrough, which is built only by comparing the
caller's inputs with the body the route sends, so a route can never mark a key it rewrote.
litellm-host-python (formerly python-interop) owns the CPython driver and the Execution
handle, and litellm-callbacks-legacy is the @client wrapper as the native call sees it:
function_setup, the deployment hooks, pre_call and post_call, the success and failure fan-out
and the deferred proxy release. OCR is the one route on it, and the old core and bridge
lifecycles are gone

The passthrough rule is the structural fix for the bug #41719 patched in core and #41716
reworks: an inlined remote document no longer counts as the caller's value, so the legacy
adapter never hands the caller's URL back into the body. core/tests/ocr/passthrough.rs pins
it for every route and document source, including that unchanged values stay passthrough,
and callbacks-legacy/tests/payload.rs pins the adapter side with a real pre_call callback

Python OCR integration tests that only exercised core behavior now live as Rust tests, so
tests/test_litellm_rust keeps the cases that need the full Python stack
2026-09-17 21:13:16 -07:00

2.8 KiB
Raw Blame History

  • Target invariants; implementation and runtime validation may lag these rules
  • Keep this crate the CPython runtime adapter and nothing more: Serde marshalling, interpreter detachment, tokio/asyncio glue, the Execution handle, the call driver and the CallbackAdapter/RouteHost traits
    • No LiteLLM domain dependencies beyond litellm-callbacks: no route types, no Logging policy, no public API registration, no cdylib build features
    • The driver emits Succeeded or Failed exactly once and never dispatches after a cancellation; which Python objects consume those events is the adapter's business
    • RouteHost::invoke receives the keyword view the adapter's begin returned, not the caller's dict; a route host that projects from it inherits that adapter's rewrites (for the legacy adapter: setup, deployment hooks, credential inheritance)
    • A failure that surfaces inside the call, including a host op the call asked for, is mapped through the route's map_failure; a failure in begin or after_success is raised as is
  • Use standard PyO3 ownership and conversion APIs
    • Prefer Bound<'py, T> for attached operations/results, Py<T> for retention; binding/unbinding does not copy payloads
    • Use pythonize for selected Serde data, never a JSON-text round trip; share conversion with Pythonized<T>
    • Preserve PythonizeError's standard conversion into PyErr; do not stringify original Python exceptions into new ValueErrors
    • Keep serializer-panic containment in Pythonized<T>: async output conversion can run in an unjoined blocking task and otherwise strand delivery
  • Use Python::detach for Rust-only work; Python operations require attachment
    • Keep diagnostic counters in the consumer; wrapper invocations do not measure every interpreter release
    • Release exclusive class borrows/locks before Python calls or decrements that can invoke finalizers; expose retained Python edges to GC without calling Python during traversal
  • Keep coroutine driving in the shared Python driver and the native handle
    • Driver: litellm/rust_bridge/lifecycle.py; handle: src/handle.rs; call driver: src/driver.rs; native-backed behavior tests: tests/lifecycle.py
    • Every adapter suspension is awaited inline in the caller's task; into_future creates a separate task and cannot satisfy this contract
  • References: ownership, conversions, pythonize errors