scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-21 00:21:49 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 74
litellm/litellm-rust/crates/host-python/AGENTS.md
Yujong Lee b4bfd92a2a refactor(rust): route-neutral callback contract 
Every legacy callback call from callbacks-legacy now goes through one typed
Python shim, litellm.rust_bridge.legacy_callbacks, the only Python module
the crate reaches. Before, the crate called Logging methods, litellm.utils
hooks, the logging worker, the executor and several litellm globals
directly, and its tests retyped those signatures by hand, so an outdated
fake could accept a call the real code rejects. python_contract.json lists
each shim function's parameters: a Python test pins it to the real
signatures and a Rust test pins it to the Rust enum.

The lifecycle contract changes to match the Python @client wrapper:
- the driver emits CallEvent::Started before begin, so every host sees one
  start time
- RequestContext carries the route-resolved api_key, so legacy pre_call and
  post_call receive it, and post_call's additional_args match the Python OCR
  path
- Passthrough and its re-aliasing are gone
- async deployment hooks always run, and the "no callbacks" shortcut that
  skipped the logging payload is removed, as in the Python path

The OCR api_key is a SecretValue from the wire request onward, so Debug
output upstream of the callback contract cannot leak it.

host-python's RouteHost now classifies native failures once through
classify, and host ops return HostOpError. The OCR route host keeps main's
public errors by sending both through the existing Python map_failure.
2026-09-18 15:43:08 -07:00

3 KiB
Raw Blame History

  • Target invariants; implementation and runtime validation may lag these rules
  • Keep this crate the CPython runtime adapter and nothing more: Serde marshalling, interpreter detachment, tokio/asyncio glue, the Execution handle, the call driver and the PythonLifecycle/RouteHost traits
    • No LiteLLM domain dependencies beyond litellm-callbacks: no route types, no Logging policy, no public API registration, no cdylib build features
    • The driver emits Succeeded or Failed exactly once and never dispatches after a cancellation; which Python objects consume those events is the adapter's business
    • RouteHost::invoke receives the keyword view the adapter's begin returned, not the caller's dict; a route host that projects from it inherits that adapter's rewrites (for the legacy adapter: setup, deployment hooks, credential inheritance)
    • A native failure, including one a host op returns as HostOpError::Native, is classified exactly once through the route's classify; a Python exception raised inside the call, and a failure in begin or after_success, is raised as is
    • A failing classify is raised with the native error's text as its __context__, never swallowed
  • Use standard PyO3 ownership and conversion APIs
    • Prefer Bound<'py, T> for attached operations/results, Py<T> for retention; binding/unbinding does not copy payloads
    • Use pythonize for selected Serde data, never a JSON-text round trip; share conversion with Pythonized<T>
    • Preserve PythonizeError's standard conversion into PyErr; do not stringify original Python exceptions into new ValueErrors
    • Keep serializer-panic containment in Pythonized<T>: async output conversion can run in an unjoined blocking task and otherwise strand delivery
  • Use Python::detach for Rust-only work; Python operations require attachment
    • Keep diagnostic counters in the consumer; wrapper invocations do not measure every interpreter release
    • Release exclusive class borrows/locks before Python calls or decrements that can invoke finalizers; expose retained Python edges to GC without calling Python during traversal
  • Keep coroutine driving in the shared Python driver and the native handle
    • Driver: litellm/rust_bridge/lifecycle.py; handle: src/handle.rs; call driver: src/driver.rs; native-backed behavior tests: tests/lifecycle.py
    • Every adapter suspension is awaited inline in the caller's task; into_future creates a separate task and cannot satisfy this contract
  • References: ownership, conversions, pythonize errors