mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-17 09:34:48 +00:00
Compose the legacy and v1 lifecycles behind one PythonLifecycle through LifecycleChain, built once in python-bridge, and simplify the boundary: - drop LifecycleEvent::Started; begin carries the start time and is the first thing a lifecycle hears of a call - give PythonLifecycle pass-through defaults for the threading steps and the stream notices, and document which steps thread and which observe - export one is_cancellation from litellm-host-python - callbacks-v1: private RequestFacts fields, a golden file per EventKind by construction, Python EVENTS and SUPPORTED_SCHEMAS pinned to the goldens - callbacks-v1-python: the snapshot only parses (register is the only place that rejects), Handler owns call/is_async, Pending keeps a subscriber index - callbacks-legacy-python: LegacyCallbacks becomes inherent PythonLogger methods, deployment hooks become free functions, retained headers are traversed and cleared - define the shared terms in host-python/AGENTS.md and each contract's own terms in its crate - remove test_loader.py, whose fork hook was never committed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5.7 KiB
5.7 KiB
- Target invariants; implementation and runtime validation may lag these rules
- Keep this crate the CPython runtime adapter and nothing more: Serde marshalling, interpreter detachment, tokio/asyncio glue, the
Executionhandle, the call driver and thePythonLifecycle/RouteHosttraits- No LiteLLM domain dependencies beyond
litellm-host: no route types, noLoggingpolicy, no public API registration, no cdylib build features - The driver emits
SucceededorFailedexactly once and never dispatches after a cancellation; which Python objects consume those events is the adapter's business RouteHost::invokereceives the keyword view the adapter'sbeginreturned, not the caller's dict; a route host that projects from it inherits that adapter's rewrites (for the legacy adapter: setup, deployment hooks, credential inheritance)- A native failure, including one a host op returns as
InvokeError::Native, is classified exactly once through the route'sclassify; a Python exception raised inside the call, and a failure inbeginorafter_success, is raised as is - A failing
classifyis raised with the native error's text as its__context__, never swallowed
- No LiteLLM domain dependencies beyond
- Terms; the callback crates build on these and never redefine them
- Machine: the native call; it never calls out, it yields a
HostOp(Route,BeforeSend,Emit,Open,Deliver) and waits for theHostResult - Host: whatever answers those ops; the Python host is two capabilities the driver composes:
RouteHost(route ops, argument and response projection) andPythonLifecycle(callbacks) - Host language = callback language: a lifecycle speaks
Py<...>andPyErrdirectly; another host language gets its own crate of this shape and reuses onlylitellm-hostandlitellm-callbacks-v1 - Driver: runs one call; owns the machine, one route host and exactly one lifecycle
- Lifecycle: one implementation of
PythonLifecycle, one callback contract's view of a call; chain: several lifecycles behind one, in a fixed order - Step: one driver-to-lifecycle method call, answered with a
LifecycleStep; a threading step (begin,before_send,after_success) returns the value the call continues with and defaults to returning what it was given; an observing step (emit,opened,delivered) returns nothing to the call - Suspension: a step answered with
Await; the driver awaits it in the caller's task and continues the same step throughresume - Terminal event:
SucceededorFailed, exactly once, last - Cancellation: a raised
BaseExceptionthat is not anException;is_cancellationis the only definition; never reported, never swallowed, never followed by dispatch - Keyword view: the kwargs dict as rewritten so far;
beginthreads it and the route projects from the result - Wire: the
WireRequest(URL, headers, body) about to be sent;before_sendthreads it - Surface: the static facts a route declares about itself to a contract (
call_type, ...) - Python contract: a crate's
python.rsenum, itspython_contract.jsonand the onelitellm/rust_bridge/*.pymodule behind them, tested from both sides
- Machine: the native call; it never calls out, it yields a
PythonLifecycleis the whole callback boundary: the driver holds exactly one and never learns how many contracts are behind itLifecycleChaincomposes several in a fixed order:begin,before_sendandafter_successthread their value forward, events reach each in turn, any error ends the step with no further dispatchbegincarries the call's start time and is the first thing a lifecycle hears of a call; there is no separate started event- A lifecycle whose
beginwas never reached receives nothing of that call; the terminal event goes to the onesbeginreached - A suspension resumes the lifecycle that suspended, then continues the same step with the ones after it; a retained failure is the raised exception object itself
- Ordering policy (which contract runs first) belongs to whoever builds the chain, never to this crate
- Use standard PyO3 ownership and conversion APIs
- Prefer
Bound<'py, T>for attached operations/results,Py<T>for retention; binding/unbinding does not copy payloads - Use
pythonizefor selected Serde data, never a JSON-text round trip; share conversion withPythonized<T> - Preserve
PythonizeError's standard conversion intoPyErr; do not stringify original Python exceptions into newValueErrors - Keep serializer-panic containment in
Pythonized<T>: async output conversion can run in an unjoined blocking task and otherwise strand delivery
- Prefer
- Use
Python::detachfor Rust-only work; Python operations require attachment- Keep diagnostic counters in the consumer; wrapper invocations do not measure every interpreter release
- Release exclusive class borrows/locks before Python calls or decrements that can invoke finalizers; expose retained Python edges to GC without calling Python during traversal
- Keep coroutine driving in the shared Python driver and the native handle
- Driver:
litellm/rust_bridge/lifecycle.py; handle:src/handle.rs; call driver:src/driver.rs; native-backed behavior tests:tests/lifecycle.py - Every adapter suspension is awaited inline in the caller's task;
into_futurecreates a separate task and cannot satisfy this contract
- Driver:
- References: ownership, conversions, pythonize errors