scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-21 00:21:49 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 73
litellm/litellm-rust/crates/python-bridge/AGENTS.md
Yujong Lee b4bfd92a2a refactor(rust): route-neutral callback contract 
Every legacy callback call from callbacks-legacy now goes through one typed
Python shim, litellm.rust_bridge.legacy_callbacks, the only Python module
the crate reaches. Before, the crate called Logging methods, litellm.utils
hooks, the logging worker, the executor and several litellm globals
directly, and its tests retyped those signatures by hand, so an outdated
fake could accept a call the real code rejects. python_contract.json lists
each shim function's parameters: a Python test pins it to the real
signatures and a Rust test pins it to the Rust enum.

The lifecycle contract changes to match the Python @client wrapper:
- the driver emits CallEvent::Started before begin, so every host sees one
  start time
- RequestContext carries the route-resolved api_key, so legacy pre_call and
  post_call receive it, and post_call's additional_args match the Python OCR
  path
- Passthrough and its re-aliasing are gone
- async deployment hooks always run, and the "no callbacks" shortcut that
  skipped the logging payload is removed, as in the Python path

The OCR api_key is a SecretValue from the wire request onward, so Debug
output upstream of the callback contract cannot leak it.

host-python's RouteHost now classifies native failures once through
classify, and host ops return HostOpError. The OCR route host keeps main's
public errors by sending both through the existing Python map_failure.
2026-09-18 15:43:08 -07:00

5.5 KiB
Raw Blame History

  • Target invariants, not completion claims; these supersede older conflicting bridge guidance
  • Keep this crate the product-specific PyO3 consumer of litellm-host-python
    • Own registration, input projection, the route host and the caller callables it answers operations with (file readers, token providers), public response/error construction and the per-call composition of machine, route host and callback contract
    • Legacy callback sharing (the caller's args, kwargs and request object, body/header roots, re-aliasing unchanged body keys) lives in litellm-callbacks-legacy behind PublicCall and run_legacy_call; the bridge hands the public call over and keeps no copy
    • Value-oriented execution, sync waiting, nested-runtime checks, signal polling and panic containment live in litellm-host-python; native async work uses pyo3-async-runtimes, Serde output uses Pythonized<T>
    • Core owns typed native state, the route machine, provider preparation/I/O and normalization; the host driver owns terminal events; the legacy adapter in litellm-callbacks-legacy owns Logging dispatch policy
    • Python, Rust SDK and gateway use one lifecycle-bearing core route entrypoint; provider helpers stay private, never bridge-accessible transport drivers
    • Built-in provider/config/secret/auth/document preparation stays in Rust; caller-authored callbacks and focused Python-file reads run only at core-selected points
  • Target GIL-enabled CPython explicitly with #[pymodule(gil_used = true)]; detach Rust-only work
    • Free-threading requires separate runtime/concurrency validation; omitting the attribute does not opt out on PyO3 0.28+
  • Preserve public argument binding and Python object provenance
    • Project only consumed fields at reference read points; no eager whole-graph serialization or equality-based alias reconstruction
    • Preserve provider-specific upload/submission/poll observation and encoding boundaries; signed/build-captured bytes must not be silently reserialized
  • Conversion errors and every failure after the call starts are terminal
    • Disabled/unavailable native execution may select legacy once; callback exceptions never authorize fallback or replay
  • Use one ordinary inline async def driver in litellm/rust_bridge/lifecycle.py, with the native handle and call driver in litellm-host-python
    • Contract: start, resume_value, resume_error, idempotent close; explicitly tagged Await/Complete preserve awaitable final values
    • Validate Created/Running/Suspended/Closed protocol states; the machine yields ops, the driver emits one terminal event, the adapter chooses dispatch policy
    • Defer effectful setup/context reads/timestamps until start; unstarted-handle destruction releases inputs independently of Python finally
    • Catch only the selected await's errors; start/resume errors propagate, GeneratorExit closes without further awaits
    • Inline hooks preserve caller task/thread/loop and context writes; into_future creates a separate task and cannot satisfy this contract
  • Finalize fallible public response/error construction, replacements and metadata before terminal dispatch
  • Make ownership safe across suspension, re-entry, cancellation and GC
    • Keep native provider state typed in core; do not shuttle it through opaque Python transport/response classes
    • Prefer one retained Py<PyBaseException> via PyErr::into_value(py); reconstruct transient PyErrs, preserving identity, traceback, cause and context
    • Traverse every owned Python edge, including duplicate references; traversal cannot call Python
    • Take state out and mark Running under a short borrow, release borrows/locks before Python invocation, publish terminal state before finalizer-capable drops
    • Close/GC/deferred release are idempotent and re-entry-safe, including during Rust unwinding; release only owned references, never clear caller containers or mask the selected error
    • The machine owns its in-flight provider future; interrupt drops it synchronously, so provider captures are released before the driver returns and no task outlives the call
  • Verify behavior through a fresh, provenance-checked installed extension and positive native execution evidence before replacing the custom coroutine
    • Cover admitted provider workflows, binding/read-point/identity behavior, failure continuation, finalization, no replay, deferred gates, re-entry, GC and cancellation termination
    • Measure real conversion/copy costs before optimizing; preserve input contracts and capture lifetimes with PyBackedBytes, and lookup timing when interning names
    • Ship accurate _native.pyi declarations and typing markers; distinguish Future-returning bindings from coroutine-returning bindings
  • References: ownership, GC, exception transfer, re-entry