diff --git a/litellm-rust/crates/python-interop/benches/README.md b/litellm-rust/crates/python-interop/benches/README.md deleted file mode 100644 index 355643ced66..00000000000 --- a/litellm-rust/crates/python-interop/benches/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Byte-stream handoff benchmarks - -Run from `litellm-rust`, using an installed Python interpreter with development libraries. No provider, HTTP server, API credentials, or installed LiteLLM extension is needed - -```sh -cargo bench -p litellm-python-interop \ - --features pyo3/abi3-py310 --bench streaming -``` - -Set `PYO3_PYTHON` to choose the interpreter. The executable embeds Python using the production stable ABI target without `extension-module`. It reuses the interpreter, Python event loop, and Tokio runtime - -Criterion filters work normally: - -```sh -cargo bench -p litellm-python-interop \ - --features pyo3/abi3-py310 --bench streaming \ - -- python_async_for/ready/256 -``` - -Each stream contains 256 chunks of 64, 256, 4096, or 65536 bytes. `ready` yields immediately; `pending` returns `Pending` once before each chunk and wakes its caller. Completion is a no-op future. Payload preparation and stream construction happen outside timing; polling, EOF, completion, and consumed-count validation happen inside - -`conversion` compares the production `to_py_bytes` function against the legacy temporary `to_vec()` followed by that same function, with Python already attached. `reader` measures production polling, synchronization, and completion without Python payload objects. `python_async_for` runs the production Python iterator and counts chunks and bytes without retaining payloads, including Python/Tokio scheduling and conversion - -The console reports time and byte throughput. The generated `target/criterion/streaming-summary.csv` also reports amortized time per chunk and chunks per second for stream workloads. Conversion rows represent one chunk. Metadata alongside the results records Python version, GIL configuration, Rust compiler and target architecture, ABI selection, arguments, and workload parameters. `CARGO_TARGET_DIR` relocates these outputs - -Metadata files have unique timestamped names so baseline and comparison settings remain available. The summary contains only cases measured in the current run; save it before another measurement if needed - -Save and compare a baseline using the same interpreter and machine settings: - -```sh -cargo bench -p litellm-python-interop --features pyo3/abi3-py310 \ - --bench streaming -- --save-baseline vec -cargo bench -p litellm-python-interop --features pyo3/abi3-py310 \ - --bench streaming -- --baseline vec -``` - -For the original migration, capture `vec` before replacing the stream's per-chunk `to_vec()` with a moved `Bytes`. The final harness only moves `Bytes`; naming a new baseline `vec` does not restore the old implementation. Only `conversion/legacy_vec` retains that copy for ongoing comparisons - -Smoke tests compile and execute all cases, including totals validation, without timing thresholds: - -```sh -cargo bench --profile dev -p litellm-python-interop \ - --features pyo3/abi3-py310 --bench streaming -- --test -``` - -These measurements describe handoff overhead. They do not measure token throughput, provider latency, or allocation counts. Compare stream results directly rather than subtracting isolated layers: scheduling, caching, and allocation behavior interact - -Reader byte throughput counts the bytes represented by each handle; the reader does not scan payload contents. It is not a memory-bandwidth measurement diff --git a/litellm-rust/crates/python-interop/benches/RESULTS.md b/litellm-rust/crates/python-interop/benches/RESULTS.md deleted file mode 100644 index 2961fedd4ae..00000000000 --- a/litellm-rust/crates/python-interop/benches/RESULTS.md +++ /dev/null @@ -1,43 +0,0 @@ -# Initial handoff comparison - -Sequential `Vec` baseline and `Bytes` comparison on the same machine, using CPython 3.12.13 with the GIL enabled, Rust 1.98.0, and target `aarch64-apple-darwin`. Both used `pyo3/abi3-py310` without `extension-module`, 20 samples, 1 second warmup, and 3 seconds requested measurement time per case - -The baseline was captured after extracting the production reader and Python adapter, while the synthetic source still copied each `Bytes` into a `Vec` during polling. The comparison moved the original `Bytes` instead. Stream construction and payload setup were excluded from both measurements - -All numbers below use Criterion mean estimates. Each stream contains 256 chunks. These are handoff overhead measurements from one local experiment, with no allocation instrumentation or provider traffic - -Reader byte throughput counts bytes represented by handles without scanning their contents, so it does not measure memory bandwidth - -| Workload | Bytes/chunk | Vec µs/stream | Bytes µs/stream | Change | Bytes ns/chunk | Chunks/s | MiB/s | -|---|---:|---:|---:|---:|---:|---:|---:| -| reader/ready | 64 | 14.79 | 12.31 | -16.8% | 48.1 | 20,799,470 | 1269.50 | -| reader/ready | 256 | 16.89 | 12.25 | -27.5% | 47.9 | 20,895,638 | 5101.47 | -| reader/ready | 4096 | 32.77 | 12.20 | -62.8% | 47.6 | 20,989,739 | 81991.17 | -| reader/ready | 65536 | 188.07 | 12.18 | -93.5% | 47.6 | 21,019,860 | 1313741.26 | -| reader/pending | 64 | 17.43 | 14.79 | -15.1% | 57.8 | 17,305,782 | 1056.26 | -| reader/pending | 256 | 19.75 | 14.73 | -25.4% | 57.5 | 17,383,462 | 4244.01 | -| reader/pending | 4096 | 35.44 | 14.74 | -58.4% | 57.6 | 17,369,188 | 67848.39 | -| reader/pending | 65536 | 189.28 | 14.73 | -92.2% | 57.5 | 17,378,513 | 1086157.06 | -| python_async_for/ready | 64 | 9337.35 | 8975.12 | -3.9% | 35059.1 | 28,523 | 1.74 | -| python_async_for/ready | 256 | 9343.62 | 9131.62 | -2.3% | 35670.4 | 28,034 | 6.84 | -| python_async_for/ready | 4096 | 9520.66 | 9111.25 | -4.3% | 35590.8 | 28,097 | 109.75 | -| python_async_for/ready | 65536 | 10154.28 | 9731.90 | -4.2% | 38015.2 | 26,305 | 1644.08 | -| python_async_for/pending | 64 | 9241.88 | 9464.63 | +2.4% | 36971.2 | 27,048 | 1.65 | -| python_async_for/pending | 256 | 8980.96 | 9156.51 | +2.0% | 35767.6 | 27,958 | 6.83 | -| python_async_for/pending | 4096 | 9353.44 | 9150.96 | -2.2% | 35746.0 | 27,975 | 109.28 | -| python_async_for/pending | 65536 | 10143.89 | 9811.42 | -3.3% | 38325.9 | 26,092 | 1630.75 | - -Direct conversion and the benchmark-only legacy copy, measured within the final run with Python already attached: - -| Bytes/chunk | Direct ns/chunk | Legacy Vec + PyBytes ns/chunk | -|---:|---:|---:| -| 64 | 7.5 | 22.4 | -| 256 | 8.5 | 24.3 | -| 4096 | 59.0 | 108.6 | -| 65536 | 670.9 | 1779.7 | - -Reader time decreased across all cases. Full Python iteration changed between approximately −4% and +2%, including small regressions for pending sources at 64 and 256 bytes. Scheduling and run-to-run variation matter at this boundary; these results do not establish a broad Python streaming speedup - -The synthetic source shares a prepared payload allocation across chunks. Cache locality and a single sequential before/after run limit how broadly these numbers apply. The production path now avoids the intermediate Vec allocation and payload copy by construction, but timing alone does not measure allocation counts - -The saved `vec` Criterion baseline, confidence intervals, samples, timestamped metadata, and current summary remain under `litellm-rust/target/criterion` on the measurement machine. See [the benchmark guide](README.md) to rerun or save a new comparison diff --git a/litellm-rust/docs/chat-completions-callback-baseline.md b/litellm-rust/docs/chat-completions-callback-baseline.md deleted file mode 100644 index b128ffc2d36..00000000000 --- a/litellm-rust/docs/chat-completions-callback-baseline.md +++ /dev/null @@ -1,57 +0,0 @@ -# Chat completions callback baseline - -Source revision: `67d24ff9d0` - -The checkout was clean before implementation. Python behavior in `litellm/utils.py`, `litellm/litellm_core_utils/litellm_logging.py`, and the Anthropic and Bedrock provider handlers is the compatibility source - -## Callback contract - -| Provider | Operation | Placement | Invocation | Arguments and return | Exception | Owner | -| --- | --- | --- | --- | --- | --- | --- | -| Anthropic | Async deployment pre-call | Before provider selection and request construction | Awaited inline in the caller task on the async API | Shallow wrapper argument dictionary; each non-`None` result replaces the dictionary read by the next hook | Ordinary exceptions escape | Call invocation | -| Anthropic | `Logging.pre_call` | After transformation and header construction, before body encoding and send | Direct synchronous call on sync and async APIs | Original input list, logging API key, and `additional_args`; callback return is ignored and in-place mutation remains visible | Per-callback `Exception` handling stays inside the existing dispatcher; an escaping exception prevents send | Logging object and call invocation | -| Anthropic | Deployment success | After response normalization, before public completion | Awaited inline in the caller task | Current hook argument dictionary and public response; non-`None` results replace the public response | Ordinary exceptions escape through the existing success path | Call invocation | -| Anthropic | Success and failure logging | After the deployment result according to the sync, async, internal-call, and fallback branches | Existing inline, executor, or background schedule | Existing `Logging` object and callback-visible request, response, or original exception; logging-only replacement does not replace the public response | Existing dispatcher policy | Logging task or call invocation according to schedule | -| Bedrock Converse | Async deployment pre-call | Before provider selection and request construction | Awaited inline in the caller task on the async API | Same replacement contract as Anthropic | Ordinary exceptions escape | Call invocation | -| Bedrock Converse | `Logging.pre_call` | After JSON serialization and SigV4 authorization, before provider send | Direct synchronous call on sync and async APIs | Original input list, logging API key, serialized `complete_input_dict`, and a `HeadersDict`; return is ignored | Same dispatcher policy as Anthropic | Logging object and call invocation | -| Bedrock Converse | Deployment success and terminal logging | Same lifecycle branches as Anthropic | Existing awaited, direct, executor, or background schedule | Same response replacement and logging-only replacement rules as Anthropic | Existing dispatcher policy | Call invocation or background logging owner | - -## Object and read map - -| Object | Anthropic | Bedrock Converse | -| --- | --- | --- | -| Boundary argument dictionary | Retained in full before deployment hooks; a returned deployment dictionary becomes the current hook dictionary without clearing the original | Same | -| Input messages | The `Logging.pre_call` `input` aliases the wrapper-selected messages object | Same | -| Callback body | A new provider-shaped dictionary whose selected optional-parameter values preserve the wrapper-created aliases; it is encoded after `pre_call` | A serialized string created from the provider body before `pre_call` | -| Execution body | Read from the retained structured callback body at send, so nested mutation reaches the wire; replacing `additional_args["complete_input_dict"]` does not redirect the retained body | Captured bytes are authorized before `pre_call`; body mutation and field replacement cannot redirect those bytes | -| Execution headers | The independently retained header object is read at send; in-place mutation reaches transport and replacing `additional_args["headers"]` does not redirect it | The independently retained `HeadersDict` is read at send while authorization remains coupled to the captured body | -| Logging object | The wrapper-selected instance is restored after deployment pre-call replacement and reused for terminal dispatch | Same | -| Response | Core normalization produces the response used to construct the existing `ModelResponse`; deployment replacement changes the public response | Same | -| Logging-only response replacement | Retained by later logging operations but does not replace the public response | Same | -| Exception | The original Python exception remains the escaping object where Python propagates it; core classification must not replace it with a string | Same | -| Release point | Request roots release after the call and any scheduled logging owner finish; cleanup drops references without clearing shared containers | Same | - -Admission is the only replay boundary. A decline must happen before opening a call session, invoking a callback, reading credentials, consuming deferred input, or performing provider I/O. Every error after admission remains on the native lifecycle and cannot select legacy Python execution - -## Core capability boundaries - -Callback is the umbrella term. Core models each callback contract as a focused capability instead of reproducing Python's `CustomGuardrail(CustomLogger)` inheritance - -| Capability | Existing Python contract | Core responsibility | -| --- | --- | --- | -| `PreCallHooks` | `async_pre_call_hook` | Apply request-level allow, replacement, or rejection before provider request preparation | -| Route preparation | No callback contract | Resolve the provider request, URL, headers, body, credentials, and authorization at the route-defined phase | -| `ModerationHooks` | `async_moderation_hook` | Apply allow, replacement, or rejection to the prepared provider request | -| Prepared-call logging | `Logging.pre_call` | Invoke direct logging with callback-visible request roots at the route-defined phase | -| Deployment hooks | `async_pre_call_deployment_hook` and post-call deployment hooks | Run once per deployment attempt and adopt replacements where Python does | -| `TerminalDispatcher` | success and failure logging handlers | Deliver the authoritative terminal record without changing the public response or original error | - -Guardrails implement pre-call or moderation hook capabilities. They do not prepare provider requests, call providers, or independently dispatch terminal failures. The shared lifecycle converts their rejection into the same terminal failure path used by provider and transformation errors - -`CallLifecycle` owns both typed native execution and the operation plan used by Python-backed routes. The Python loop only schedules direct operations on the caller's Python task so synchronous callbacks retain `contextvars`, event-loop identity, and nested-call behavior. Core selects every transition and whether an operation is awaited; the bridge only maps an abstract operation to its Python adapter method. - -Long-lived streaming calls own a completion guard. HTTP streams, Responses WebSockets, and Realtime WebSockets dispatch a cancelled terminal record from that guard when the consumer drops the call before normal settlement. - -Destination-specific callback transport belongs outside core. The LiteLLM Python proxy HTTP logger and its batching worker live in `litellm-ai-gateway`; core exposes only the destination-agnostic `CustomLogger` and terminal dispatcher contracts. - -The Rust bridge exposes Bedrock callback headers through `requests.structures.CaseInsensitiveDict` instead of botocore’s concrete `HeadersDict` type. Lookup, mutation, deletion, and readback are case-insensitive; iteration uses the most recently assigned spelling of a key. Core selects header behavior independently of body serialization policy diff --git a/litellm-rust/docs/lifecycle-acceptance.md b/litellm-rust/docs/lifecycle-acceptance.md deleted file mode 100644 index a8d1bafcd23..00000000000 --- a/litellm-rust/docs/lifecycle-acceptance.md +++ /dev/null @@ -1,32 +0,0 @@ -# Native lifecycle acceptance - -This is an implementation checkpoint, not sign-off on the complete lifecycle migration - -The behavioral references are the current Python implementations in `litellm/rust_bridge`, `litellm/litellm_core_utils/litellm_logging.py`, `litellm/proxy/common_request_processing.py`, and the Anthropic Messages streaming iterator. The callback design notes under `Rust Native SDK/callback` supply the ownership and compatibility requirements - -| Area | Implemented foundation | Remaining acceptance | -| --- | --- | --- | -| Messages buffered | Route-specific preparation/send permits; single-use operation tickets; private raw executor | Carry one completion/accounting owner through Python setup, preparation, provider execution, and terminal operations | -| Messages streaming | Registered completion handle; synchronous core terminal recording; cancellation/drop cleanup; incomplete drain-capacity outcome; legacy Python drain logging | Core deferred completion must follow the real proxy decision; prove every handoff and disconnect boundary | -| Chat Completions | Route-specific permits; prepared execution no longer creates a nested lifecycle; native usage captured before response replacement/rejection | Remove remaining public signing helpers; bind preparation and provider commitment to the same accounting owner | -| OCR | Route-specific permits; prepared requests cannot enter the public executor without a send permit; no nested prepared lifecycle | Carry completion ownership across credential resolution and Python policy phases; integrate deferred proxy decision | -| Audio Transcription | Shared native preparation/runner and cancellation guard | Complete the same admitted/resumable owner migration and route-specific boundary tests | -| Realtime | Shared WebSocket completion established before dialing; cancellation recorded independently; observed usage retained | Separate logical identity from provider session identity; deterministic dial cancellation and warmed-transfer accounting gates | -| Responses WebSocket | Shared WebSocket completion established before dialing; handshake errors return terminal records; usage provenance | Deterministic dial cancellation, incomplete protocol outcomes, and logical/attempt accounting gates | -| Deferred result | Single-use accept/reject/drop capability; `into_result` explicitly accepts | Wire the capability through production Python and the proxy acceptance/rejection path; acceptance must release delivery as well as accounting | - -`Operation::contract` now determines scheduling, replacement handling, and failure disposition. Python continues invoking direct and inline-awaited callbacks in its caller context. The bridge retains Python objects and exceptions; the core receives outcomes and owned native values - -`CallLifecycle::issue` permits one outstanding operation. A ticket identifies its execution, generation, and operation. Completion rejects foreign or stale tickets. Preparation and provider permits have private constructors, are route-specific, and can only be issued through the sealed route lifecycle API. Compile-fail doctests cover the raw executor and permit construction boundaries - -`BeforeProvider` describes transport commitment, not permission to replay through Python. Existing Python admission/fallback guards remain authoritative. Commitment still needs transport-level response receipt and normalization observations; a successful decoded response is not an adequate substitute for receipt of HTTP headers - -Buffered cancellation records a core terminal without invoking ordinary Python failure callbacks. HTTP streaming records synchronously when the observed stream terminates, independently of callback-task scheduling. Runtime teardown drops native work and records cancellation even if the runtime cannot deliver callbacks. Repeated polling, close, drop, and completion-handle release cannot record the same observed stream twice - -The native usage snapshot precedes deployment-success replacement. Post-response rejection and cancellation retain that snapshot. WebSocket snapshots distinguish unavailable usage from observed partial usage and promote it on successful session completion. Decoder failures before usage extraction still need route-specific accounting observations - -The installed-wheel differential suite passed 291 selected tests at this checkpoint, including real integration implementations with controlled dependencies. The cancellation regression now runs on both Python and Rust: once the provider finishes a detached drain successfully, Python's source requires success billing even though the consumer was cancelled. The test uses a first-chunk signal and releases the provider explicitly instead of establishing ordering with a sleep - -The transition tests use independently written expected traces. The focused mutation campaign covers operation identity, single provider issuance, completion transfer, cancellation recording, terminal deduplication, trusted usage, captured body bytes, drain-capacity classification, and callback sequencing. All nine mutations were killed after strengthening the same-phase foreign-ticket regression. This 100% result is a bounded regression score, not a claim of repository-wide mutation coverage - -Completion of the original plan still requires every item in the remaining-acceptance column, complete host-accessible preparation/settlement/signing encapsulation, production deferred-capability wiring, and the requested exhaustive transport/cancellation and integration schedules. No new provider, endpoint, retry engine, or cache engine is introduced by this checkpoint