scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-26 01:12:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 99
litellm/litellm-rust/crates/host-python/AGENTS.md
devin-ai-integration[bot] 1f8997398e
refactor(rust): extract the host coroutine into its own crate (#43129) 
Move the generic async coroutine out of the host crate into litellm-coroutine,
with its requirements documented in the crate's AGENTS.md. RouteMachine becomes
CallMachine on top of it, host protocol ops move to host/src/protocol.rs, and
the messages, OCR, host-python, python-bridge and legacy callbacks crates adopt
the new types. Adds error definition rules to litellm-rust/AGENTS.md.

Co-authored-by: Yujong Lee <yujong@berri.ai>
2026-09-25 04:20:06 +00:00

3 KiB
Raw Permalink Blame History

  • Target invariants; implementation and runtime validation may lag these rules
  • Keep this crate the CPython runtime adapter and nothing more: Serde marshalling, interpreter detachment, tokio/asyncio glue, the Execution handle, the call driver and the PythonLifecycle/ProtocolHost traits
    • No LiteLLM domain dependencies beyond litellm-host: no route types, no Logging policy, no public API registration, no cdylib build features
    • The driver emits Succeeded or Failed exactly once and never dispatches after a cancellation; which Python objects consume those events is the adapter's business
    • ProtocolHost::project receives the keyword view the adapter's begin returned, not the caller's dict; a protocol host that projects from it inherits that adapter's rewrites (for the legacy adapter: setup, deployment hooks, credential inheritance)
    • A native failure, including one a host op returns as InvokeError::Native, is classified exactly once through the route's classify; a Python exception raised inside the call, and a failure in begin or after_success, is raised as is
    • A failing classify is raised with the native error's text as its __context__, never swallowed
  • Use standard PyO3 ownership and conversion APIs
    • Prefer Bound<'py, T> for attached operations/results, Py<T> for retention; binding/unbinding does not copy payloads
    • Use pythonize for selected Serde data, never a JSON-text round trip; share conversion with Pythonized<T>
    • Preserve PythonizeError's standard conversion into PyErr; do not stringify original Python exceptions into new ValueErrors
    • Keep serializer-panic containment in Pythonized<T>: async output conversion can run in an unjoined blocking task and otherwise strand delivery
  • Use Python::detach for Rust-only work; Python operations require attachment
    • Keep diagnostic counters in the consumer; wrapper invocations do not measure every interpreter release
    • Release exclusive class borrows/locks before Python calls or decrements that can invoke finalizers; expose retained Python edges to GC without calling Python during traversal
  • Keep coroutine driving in the shared Python driver and the native handle
    • Driver: litellm/rust_bridge/lifecycle.py; handle: src/handle.rs; call driver: src/driver.rs; native-backed behavior tests: tests/lifecycle.py
    • Every adapter suspension is awaited inline in the caller's task; into_future creates a separate task and cannot satisfy this contract
  • References: ownership, conversions, pythonize errors