diff --git a/litellm-rust/ADDING_A_PROVIDER.md b/litellm-rust/ADDING_A_PROVIDER.md deleted file mode 100644 index ae8ae5a6870..00000000000 --- a/litellm-rust/ADDING_A_PROVIDER.md +++ /dev/null @@ -1,29 +0,0 @@ -# Adding a provider / route to litellm-rust - -Everything for a route lives in `crates/core/src//`; `crates/core/src/messages` is the reference. A host (the axum gateway, the Python bridge) only calls the route's entrypoint. - -1. **Entrypoint** — `mod.rs`: `pub async fn (request) -> CoreResult`, the Rust equivalent of `litellm.()`, plus a `_stream` variant when the route streams. It is the only thing a host touches. -2. **Transform contract** — `transformation.rs`: a `…ProviderConfig` trait (URL build + request/response transforms) with types in `types.rs`. -3. **Provider config** — `crates/core/src/providers///transformation.rs`: implement that trait as a `const __CONFIG`, mirroring the Python provider tree. Add parity unit tests. -4. **Prepare + handler** — `prepare.rs` resolves provider/model, credentials, auth headers, and URL, then transforms the request; `handler.rs` performs the provider call through the shared client in `client.rs` and transforms the response. - -## Coding standards - -Before writing new logic, look for an existing base to extend. When a change is -“the same behavior for one more provider/endpoint/integration”, the codebase -almost always already has a shared abstraction for it (for example, provider -`BaseConfig` transformation classes in `litellm/llms/base_llm/`, shared -helpers in `litellm_core_utils/`, typed request/response models, or factory -functions). Find it first with a search, then add the new variant by inheriting -from or composing that base, overriding only what genuinely differs (model -name, parameter mapping, or auth). - -Never copy an existing implementation and edit it in place, and never hand-roll -a parallel version of logic a base already provides. If you catch yourself -writing a second copy of a pattern that exists twice already, stop and extract a -base instead: put the shared shape in one place and make both call sites thin -variants of it. The test for a good abstraction is that adding the next provider -is a few declarative lines, not a new file of duplicated flow. Only diverge from -the base when behavior is genuinely different, and say so explicitly in the PR. - -**Calling:** hosts invoke the core entrypoint — the Python bridge and the `ai-gateway` route service both call `litellm_core::messages::messages`. Never add a provider handler to `ai-gateway`. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md). diff --git a/litellm-rust/AGENTS.md b/litellm-rust/AGENTS.md deleted file mode 100644 index b8d1f2db4d7..00000000000 --- a/litellm-rust/AGENTS.md +++ /dev/null @@ -1,44 +0,0 @@ -# AGENTS.md - -litellm-rust has five crates. A crate is a layer or shared foundation, not a route. Routes (ocr, realtime, chat) and providers (mistral, openai) are modules inside the layers. - -## Crates - -| Crate | Role | -|-------|------| -| litellm-core | The LiteLLM SDK in Rust. One public entrypoint per top-level call (`messages::messages()`), owning types, transforms, provider resolution, auth, and the provider HTTP call. Call it, get a typed response. | -| litellm-config | Config-loading boundary. Returns resolved core deployment data and optionally delegates loading to Python. | -| litellm-ai-gateway | The axum server (behind the `server` feature) plus the WebSocket hosts. Translates HTTP/WS to core entrypoints; owns no provider logic and no handlers. | -| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | -| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. | - -Dependency direction is acyclic: `litellm-config` depends on `litellm-core`, the gateway depends on both, and `litellm-python-bridge` depends on the domain layers and `litellm-python-interop`. The interop foundation depends on no LiteLLM domain crate. - -## Where a route lives - -A top-level LiteLLM call is a module under `crates/core/src//`, shaped like `messages`: - -``` -core/src/messages/ - mod.rs # pub async fn messages(..) -> CoreResult<..> (+ messages_stream for SSE) - types.rs # request/response types, MessagesRequest - transformation.rs # the provider template trait - prepare.rs # provider resolution, auth headers, URL - handler.rs # the provider call - client.rs # the shared reqwest client -``` - -Handlers never live in `ai-gateway`. `ocr`, `audio_transcription`, and `realtime` are still hosted there from before this rule; they move to `core` as they are touched. - -Adding a crate: default to a module. A new crate requires a real trigger: separate artifact (binary/cdylib), proc-macro, shared foundation, or publishable standalone. A new provider or route is none of these. - -Adding a crate fails crates/core/tests/workspace_crate_allowlist.rs until you update its allowlist and this file — intentional. - -## Style - -All Rust in `litellm-rust/` follows the official Rust Style Guide: -https://doc.rust-lang.org/style-guide/ - -`rustfmt` implements its formatting by default, so run `cargo fmt` before committing; CI gates every PR on `cargo fmt --check`. Do not hand-format against rustfmt or add a `rustfmt.toml` that diverges from the default style. - -Beyond formatting, follow the guide's naming and idiom conventions rustfmt cannot auto-apply: `snake_case` items/functions/modules, `UpperCamelCase` types/traits/variants, `SCREAMING_SNAKE_CASE` constants/statics (acronyms as one word, e.g. `HttpClient`), and the import grouping and item ordering it prescribes. See CLAUDE.md for the detailed version. diff --git a/litellm-rust/CLAUDE.md b/litellm-rust/CLAUDE.md deleted file mode 100644 index 3b1e84dd91c..00000000000 --- a/litellm-rust/CLAUDE.md +++ /dev/null @@ -1,197 +0,0 @@ -# CLAUDE.md - -This file defines the rules for Rust work in LiteLLM. - -## Provider Coding Standards - -Before writing new logic, look for an existing base to extend. When a change is -“the same behavior for one more provider/endpoint/integration”, the codebase -almost always already has a shared abstraction for it (for example, provider -`BaseConfig` transformation classes in `litellm/llms/base_llm/`, shared -helpers in `litellm_core_utils/`, typed request/response models, or factory -functions). Find it first with a search, then add the new variant by inheriting -from or composing that base, overriding only what genuinely differs (model -name, parameter mapping, or auth). - -Never copy an existing implementation and edit it in place, and never hand-roll -a parallel version of logic a base already provides. If you catch yourself -writing a second copy of a pattern that exists twice already, stop and extract a -base instead: put the shared shape in one place and make both call sites thin -variants of it. The test for a good abstraction is that adding the next provider -is a few declarative lines, not a new file of duplicated flow. Only diverge from -the base when behavior is genuinely different, and say so explicitly in the PR. - -## Crates (see AGENTS.md) - -`litellm-core` **is** the LiteLLM SDK in Rust: it makes the LLM call. -`litellm-config` is the config-loading boundary and returns resolved core types. -`litellm-ai-gateway` is an HTTP/WebSocket server in front of it, and -`litellm-python-bridge` exposes it to the Python SDK. `litellm-python-interop` -holds domain-neutral PyO3 primitives shared by Python-facing Rust code. A crate -is a layer or shared foundation, not a route; add modules, not crates. - -## Core Boundary - -`litellm-core` owns the whole call. The Rust equivalent of `litellm.messages()` -is `litellm_core::messages::messages(request).await`: you call it, it does the -provider call, and you get a typed non-streaming response back. - -Route-level Rust structure mirrors LiteLLM's Python responsibilities: -- `core/src//` owns the route end to end: the public entrypoint fn named - after the route in `mod.rs`, the request/response types (`types.rs`), the - provider template trait (`transformation.rs`), the provider/auth/URL - resolution (`prepare.rs`), the HTTP client (`client.rs`), and the handler that - performs the call (`handler.rs`). `core/src/messages` is the reference. -- `core/src/providers///transformation.rs` owns the - provider-specific transform. For Anthropic Messages, this means - `core/src/providers/anthropic/messages/transformation.rs`. -- Handlers live in `core`, never in a host. `ai-gateway` must not contain a - route handler that talks to a provider; its axum route reads the HTTP request, - picks a deployment, and calls the `core` entrypoint. `python-bridge` marshals - Python objects and calls the same entrypoint. - -Streaming keeps the same shape: the route entrypoint has a `_stream` -variant in `core` that returns the upstream response so a host can splice it to -its own caller; the host still owns no provider logic. - -Call-hook and lifecycle instrumentation, including phase timing, usage -accumulation, and callback payload construction, always lives in `core`. -Hosts feed observed events into core and dispatch the completed payloads through -their I/O logger; hosts must not own callback orchestration. - -Allowed in `core`: -- The public entrypoint for a top-level LiteLLM call -- Request/response transforms and stream chunk normalization -- Provider resolution, auth header construction, and URL building -- The provider HTTP call itself, through a shared reused client with connect and - request timeouts -- Shared data types and validation errors -- Deterministic token/cost helper logic - -Not allowed in `core`: -- Serving HTTP: axum routes, extractors, and transport concerns stay in the host -- Filesystem access -- Database access -- Config file reading and rollout state -- Logging callbacks, spend writes, or custom callbacks -- Global mutable runtime state - -Env reads in `core` are limited to credential fallback inside a route's -`prepare.rs` (the `env_lookup` closure), mirroring what the Python SDK does when -no key is passed. Everything else config-shaped is resolved by the host and -passed in. - -Routes still hosted in `ai-gateway` (`ocr`, `audio_transcription`, `realtime`) -predate this rule and are being moved into `core` route modules; do not add new -ones there, and prefer moving one when you touch it. - -Python owns rollout state and fallback while Rust is being introduced. Rust -paths must be off by default until parity tests prove equivalence with Python. -A new provider/route may instead be implemented rust-only with no Python -reference; then the Python interface is a thin dispatch that calls Rust with no -fallback, and you state the rust-only choice explicitly in the PR. Either way -the Python side stays minimal (it only marshals inputs and calls the Rust -interface), never add a per-route feature flag, and never push provider -dispatch into `litellm/main.py`; put it in a thin dispatch class under -`litellm/llms///`. - -## Production Bar - -Rust code in this workspace is held to a strict parity and robustness bar from -the first PR: - -- Correctness parity is proven with tests. Do not rely on README claims or - manual inspection for a port that mirrors Python behavior. -- Every provider transform must have unit tests for supported-parameter - filtering, request body shape, response normalization, missing/null fields, - and bad-input errors. -- When Rust is exposed through Python, add Python tests that prove disabled, - enabled, and unavailable-bridge fallback behavior. -- Avoid panics on user/provider input. Return typed errors and let the host map - them to Python exceptions or HTTP responses. -- OCR handles documents that often contain personal data. Do not log document - contents, base64 payloads, provider response bodies, or secrets. -- Error messages must be useful but data-minimized. Truncate or sanitize any - upstream body before it crosses a host boundary. -- Treat empty or whitespace-only credentials, URLs, and config values as absent - at the host/config resolution layer. -- Preserve Python output shape intentionally. If a field is always serialized as - `null` for Python parity, leave a short comment explaining that parity choice. - -## Network I/O Rules - -These rules apply to every module that executes network I/O, whether it is a -`core` route handler or a host such as `ai-gateway`: - -- Set connect and full-request timeouts. No unbounded waits. -- Reuse HTTP clients; do not construct clients per request. -- Prefer rustls TLS for portable Python wheels and Linux images unless there is - a documented reason not to. -- Add request IDs and structured tracing at the host layer, without logging OCR - document contents or secrets. -- Do not echo raw upstream response bodies to callers. Sanitize and bound them. -- Avoid `expect`/`unwrap` in server startup and request paths unless the panic is - impossible by construction and documented. - -## Rust Style Guide - -All Rust in `litellm-rust/` follows the official Rust Style Guide: -https://doc.rust-lang.org/style-guide/ - -`rustfmt` implements the guide's formatting rules by default, so the mechanical -side is enforced for you: run `cargo fmt` before committing and CI gates every -PR on `cargo fmt --check` (see Checks). Do not hand-format against rustfmt or add -a `rustfmt.toml` that diverges from the default style; the default style *is* the -guide. - -The guide also covers conventions rustfmt cannot auto-apply; follow these too: -- Naming: `snake_case` for items, functions, and modules; `UpperCamelCase` for - types, traits, and enum variants; `SCREAMING_SNAKE_CASE` for constants and - statics; acronyms count as one word (`HttpClient`, not `HTTPClient`). -- Ordering and grouping the guide prescribes: imports grouped std / external / - crate-local, derives before other attributes, and consistent item order. -- Idioms the guide recommends over the formatter fighting you (e.g. prefer - restructuring an over-long expression rather than forcing an awkward wrap). - -## Constants - -Magic numbers and fixed strings go in a crate-level `constants.rs`, never -hardcoded inline — the Rust mirror of Python's `litellm/constants.py`. - -- Each crate that needs them has `src/constants.rs` (declared `mod constants;`); - import from it (`use crate::constants::...`). Don't scatter `const` values at - the top of feature modules. -- An env-overridable tunable still lives in `constants.rs` as its `DEFAULT_*` - value; the env read (with fallback to that default) happens at the host/config - resolution layer, not in `core`/`providers`. -- Exception: a value that is purely local to one function and has no meaning - elsewhere may stay inline, but prefer `constants.rs` when in doubt. - -## Checks - -Run these before pushing Rust changes. The same checks run in GitHub Actions -for changes under `litellm-rust/`. - -```bash -cd litellm-rust -cargo fmt --check -cargo clippy --workspace --all-targets -- -D warnings -cargo clippy -p litellm-core --all-targets --features bedrock-auth -- -D warnings -# the ai-gateway binary + server code is behind the `server` feature -cargo clippy -p litellm-ai-gateway --all-targets --all-features -- -D warnings -cargo test --workspace -cargo test -p litellm-core --features bedrock-auth -# the `auth`, `routes`, `state` and `realtime` tests only exist under `server` -cargo test -p litellm-ai-gateway --features server -``` - -When a Rust path is exposed through Python, add Python parity tests that compare -the existing Python output with the Rust-backed output. - -For Python-integrated tests that need the repository's Python environment, run -from the repository root: - -```bash -make test-rust-python -make lint-rust-python-fixtures -``` diff --git a/litellm-rust/README.md b/litellm-rust/README.md deleted file mode 100644 index eb2f459c2b3..00000000000 --- a/litellm-rust/README.md +++ /dev/null @@ -1,186 +0,0 @@ -# LiteLLM Rust - -This workspace contains the staged Rust implementation for LiteLLM. - -`litellm-core` is the LiteLLM SDK in Rust: one entrypoint per top-level call -that makes the LLM call and hands back a typed response, the same shape as -`litellm.messages()` in Python. - -```rust -let response = litellm_core::messages::messages(MessagesRequest { - model: "claude-sonnet-4-5", - body, - api_key: Some(key), - .. -}) -.await?; -``` - -Python continues to own configuration, retries, routing policy, logging, -callbacks, spend tracking, and customer plugins until each Rust path has parity -coverage and production evidence. - -## Crates - -| Crate | Role | -|-------|------| -| litellm-core | The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router. | -| litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. | -| litellm-ai-gateway | The axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. | -| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | -| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. | - -Dependency direction is acyclic: config depends on core, the gateway depends on config and core, and the Python bridge depends on the domain layers and Python interop. - -## Layout - -```text -crates/ - core/ The SDK: route modules + provider transforms. - src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client - src/providers/anthropic/messages/transformation.rs - config/ Config loading and resolved deployments. - ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints. - python-interop/ Domain-neutral PyO3 conversion and GIL primitives. - python-bridge/ PyO3 API adapter for Python LiteLLM. -``` - -The folder shape follows the Python provider tree: -`core/src/providers///transformation.rs`. The bridge exposes one -function per top-level route, mirroring the core entrypoints. - -## Checks - -### Private Native OCR Proof - -Public `litellm.ocr` and `litellm.aocr` always use the existing Python lifecycle, -including when `litellm.rust(True)` or `LITELLM_RUST=1` enables other Rust paths. -Native OCR remains a private proof until full lifecycle parity is established. -Only tests requesting the private `native_ocr` fixture replace those public -functions with test-only route selection: Rust enabled calls the native bridge, -and Rust disabled calls the captured production Python functions - -The bridge retains the complete call argument dictionary as a Python object, -including opaque callback and metadata objects. It creates callback-visible -request dictionaries with shared parameter references, then reads the execution -roots after pre-call dispatch. Mistral retains the original document; Azure and -Vertex Mistral use a shallow document copy, and Vertex DeepSeek projects it into -chat messages using the existing Rust transform. Rust performs provider preparation, -encoding, HTTP and response normalization. Python continues to dispatch existing -logging operations and construct the public response object - -This is a private implementation scaffold, not full OCR parity. Azure Mistral and -Vertex Mistral accept inline data URIs with supplied keys/tokens, native environment -keys or auth headers. Azure also accepts a supplied `azure_ad_token`. Vertex -DeepSeek uses its existing chat request and OCR response transforms. Cloud -credential acquisition fails explicitly only when no native credential is available. -HTTP document URL conversion fails only for configs requiring data URIs. Azure -Document Intelligence selects its own config but fails at the polling capability -check before sending a billable analyze request. Cohere transforms, file inputs, -streaming, native response format and compression remain unsupported. -Direct private bridge calls with a missing native extension also fail; -neither case falls back to Python execution within the private route. Transport -failures currently use a generic error rather than the SDK's timeout-specific exception - -Run the commands under "Checks" in [CLAUDE.md](CLAUDE.md) before pushing Rust -changes. That list is the single source of truth and matches what GitHub Actions -runs for changes under `litellm-rust/`. - -### Python-Integrated Tests - -From the repository root, run the ignored Cargo tests that need the repository's -Python dependencies and the pinned Ruff checks over the interop crate's Python -test fixtures: - -```bash -make test-rust-python -make lint-rust-python-fixtures -``` - -`test-rust-python` installs the locked SDK dependencies plus the `proxy` extra -(the integration fixtures import `litellm.proxy.*` guardrails, which need -`fastapi`) with uv, points `PYO3_PYTHON` at the project interpreter, and runs -`cargo test -p litellm-python-interop --tests --locked -- --include-ignored`. -`lint-rust-python-fixtures` runs pinned Ruff lint and formatting checks without -syncing the project environment - -Run the private native OCR proof gate from the repository root: - -```bash -make test-rust-ocr -``` - -This builds the current release wheel, installs locked SDK dependencies, the -`dev` test group, and the `proxy` extra in a temporary Python 3.12 environment, -then installs the wheel without resolving dependencies again. The proxy extra -is needed by the shared pytest fixtures. Python isolated mode and pytest's -importlib mode keep the checkout from shadowing the installed wheel - -The gate checks that native `ocr` and `aocr` are importable, then runs -`tests/test_litellm/ocr/test_rust_bridge.py` with -`LITELLM_REQUIRE_NATIVE_OCR=1`, so unavailable native OCR fails instead of -skipping. CI uses `make test-rust-ocr RUST_OCR_WHEEL=/absolute/path/to/current.whl` -to test the release wheel it just built. The stdlib-only -`native_route_wheel_test.py` also exercises sync/async OCR through the retained -argument dictionary, including native request preparation and public 429 error -mapping, alongside the other native routes - -The Python-integrated Cargo tests validate retained callback identity, mutation, -invocation context and ownership against Python behavior, including existing -LiteLLM components. Short synthetic pre-call contracts use Rust-owned table-driven -cases with inline Python callbacks; larger component scenarios share Python -fixtures. These generic proofs complement, rather than replace, native OCR -private proof tests - -The standard-library-only tests in -`crates/python-interop/tests/synthetic/patterns.rs` define small inline Python -callbacks, with Rust controlling invocation, ownership and assertions. They -compare Python-reference and Rust-retained calls using both direct and awaited -invocation. They model the behavior -groups in the callback use-case inventory: live versus serialized queues, -mutation before an error, ignored returns, identity-based redaction, block-state -stash, background writes after return, parallel live data versus snapshots, and -shallow/deep copies with uncopyable-value fallback. Copy controls deliberately -produce different observations; event gates establish ordering without sleeps. -Existing synthetic lifecycle cases also cover streams, context and cancellation - -Run this matrix without LiteLLM, vendor SDKs, credentials or services: - -```bash -cargo test --manifest-path litellm-rust/Cargo.toml -p litellm-python-interop --test synthetic -``` - -These are behavioral models, not tests of vendor authentication, delivery or -production dispatcher policy. The optional component and integration fixtures -exercise existing LiteLLM implementations with fake transports and credentials -as supplementary coverage; run them with `make test-rust-python` - -The interop tests have two explicit Cargo targets, each rooted in its directory's -`mod.rs`: `tests/synthetic/` for custom, minimal Python implementations and -`tests/integration/` for real LiteLLM components with fake transports. The latter -are component-level compatibility tests, not complete SDK or proxy route tests. -Shared Rust fixtures live in `tests/support/`, and Python scenarios live in -`tests/fixtures/` - -Use `#[fixture]` composition for setup: initialize Python once, but create a fresh -scenario scope and owner factory for each case. Use named `#[case::behavior]` -entries for scenarios and `#[values(Backend::Python, Backend::PreparedCall)]` for -the invocation matrix. Keep copying and ownership controls alongside the behavior -they distinguish. Register new Rust modules in the appropriate `mod.rs`; Cargo -test autodiscovery is disabled so new files cannot silently become a third group - -Run `--test synthetic` for the standard-library-only group. Integration cases -remain explicitly ignored without the repository Python environment; -`make test-rust-python` configures that environment and runs both groups with -`--include-ignored`. To inspect the groups without running them, use -`cargo test --manifest-path litellm-rust/Cargo.toml -p litellm-python-interop --tests -- --list` - -The callback lifecycle scenarios use -`#[serial(python_interpreter)]` to isolate CPython GC and interpreter-wide -LiteLLM settings under `cargo test`. Compatible tests in the same binary use -`#[parallel(python_interpreter)]`: they may overlap each other, but not an -exclusive scenario. Unannotated tests do not participate in this isolation. -Keep the attribute below `#[rstest]` so generated cases acquire it before -fixture setup and Python attachment. Tasks and threads inside each scenario -still run concurrently. Separate test processes have separate interpreters, -so these attributes need no cross-process lock when using nextest diff --git a/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md b/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md deleted file mode 100644 index 952bbc38b43..00000000000 --- a/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md +++ /dev/null @@ -1,53 +0,0 @@ -# Provider coding standards (litellm-rust) - -Rules for adding or changing an LLM provider/route in `litellm-rust`. `messages` (`core/src/messages`, `ANTHROPIC_MESSAGES_CONFIG`) is the reference: a route is a `core` module with a public entrypoint that makes the call and returns a typed response. - -## Provider resolution - -1. Always resolve the provider/model first with `get_custom_llm_provider` (`core/src/routing_utils/provider.rs`). Nothing downstream may branch on a raw model string. -2. Model/provider is resolved once, in `prepare.rs`, and passed down as typed fields. Don't re-resolve or re-parse it in transforms or handlers. - -## Transforms and the base config - -3. Every route defines a base config trait with `transform_request` + `transform_response` (+ `complete_url`, `supported_params`), living in `core/src//transformation.rs` (e.g. `AnthropicMessagesProviderConfig`, mirroring `OcrProviderConfig`). -4. Each provider implements that trait as a `const __CONFIG` in `core/src/providers///transformation.rs`, mirroring the Python provider tree. -5. Individual configs implement only the request/response transforms. Shared behavior (param filtering, defaults) stays as trait default methods so future providers inherit existing logic instead of reimplementing it. -6. Prefer composition: a provider that extends another reuses the base trait's defaults or wraps another config; don't copy transform bodies between providers. - -## Boundaries - -7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `ai-gateway` = serving HTTP/WS (routing, extractors, auth of *our* callers, streaming to the client); `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint; they never build a provider request. -8. Generic/route files contain zero provider-specific branches. A provider is one module under `core/src/providers///`; a route is a module, never a new crate. -9. Route entry point stays thin: `core::::()` -> `prepare_*` -> handler (or `CallLifecycle::run_request`, which owns the pre_call -> during_call -> provider call -> success/failure order and phase timing). Axum handlers validate and delegate to a service that calls the entrypoint; no business logic in them. -10. Constants (URLs, env-var names, API versions, error messages) live in a crate `constants.rs`, never inline. Config-shaped env reads happen at the host/config layer with the `DEFAULT_*` fallback defined in `constants.rs`; the only env read in `core` is the credential fallback in a route's `prepare.rs`. - -## Types and errors - -11. Typed contracts only: no bare `serde_json::Value` / `String` / `Vec` as a transform input or output. Parse wire bytes into typed structs/enums at the host edge; a `type` discriminator is a typed field, not a raw string. -12. Model failures as values: return typed `CoreError`, don't panic. No `unwrap`/`expect`/`panic!` on user or provider input. -13. No mutation: build values in one shot (comprehensions/iterators, `collect`), prefer immutable bindings and owned typed structs over seeding-and-mutating. -14. Early returns over deep nesting; small focused files over god modules. -15. Preserve Python output shape intentionally. If a field is always serialized as `null` for parity, keep it and pin it with a test. - -## Safety and data minimization - -16. Never log request/response bodies, base64 payloads, document contents, or secrets. Truncate and bound any upstream body before it crosses a host boundary. -17. Treat empty/whitespace credentials, URLs, and config values as absent at the host resolution layer. -18. Network I/O sets connect + request timeouts (no unbounded waits), reuses a shared HTTP client, and prefers rustls TLS. - -## Tests and rollout - -19. Every provider transform ships tests for: supported-param filtering, request body shape, response normalization, missing/null fields, bad input, and `*_match_python` fixture parity. -20. Lifecycle/hook tests cover hook order, success + failure callback payloads, pre-call guardrail blocking before any provider I/O, during-call body mutation, and provider-error mapping. -21. When a route has a Python reference implementation, the Rust path stays off by default and behind Python parity tests (disabled / enabled-equals-Python / bridge-unavailable fallback) until parity is proven. A new provider/route may instead be implemented rust-only with no Python reference; then the Python interface is a thin dispatch to Rust with no fallback, and tests cover the rust-backed path plus the unavailable-bridge error. State the rust-only choice explicitly in the PR. - -## Python bridge (SDK side) - -22. A Python -> Rust bridge keeps the Python side minimal: the Python interface only marshals inputs and calls the Rust interface, with no transform, handler, or business logic. Aim for well under 100 lines of interface code per route; if the Python grows past that, the logic belongs in Rust. -23. Do not bloat `litellm/main.py`. A route's provider dispatch lives in a thin dispatch class under `litellm/llms///` that calls the Rust bridge; `main.py` only instantiates it and calls its sync/async method. -24. Do not add new feature flags unless explicitly requested. Reuse the existing LiteLLM Rust rollout mechanism (`litellm.rust`); never introduce a per-route env flag such as `LITELLM_USE_RUST_`. - -## Checks before push - -25. Run, and keep green, the commands under "Checks" in `litellm-rust/CLAUDE.md`. - That list is the single source of truth and matches what GitHub Actions runs. diff --git a/litellm-rust/crates/ai-gateway/AGENTS.md b/litellm-rust/crates/ai-gateway/AGENTS.md index b2fd583316b..9a970008ae0 100644 --- a/litellm-rust/crates/ai-gateway/AGENTS.md +++ b/litellm-rust/crates/ai-gateway/AGENTS.md @@ -1,54 +1,20 @@ -# ai-gateway — folder architecture +# ai-gateway -The Axum server that fronts the Rust gateway. It owns transport + config + auth -only; deployment selection lives in `core::router`, and the LLM call itself -(transforms, auth headers, provider HTTP) lives behind a `core` route entrypoint -such as `litellm_core::messages::messages`. No provider handler lives here. +The Axum server fronting the Rust gateway. Owns transport + config + auth only; deployment selection is `core::router`, and the LLM call (transforms, auth headers, provider HTTP) is a `core` route entrypoint. No provider handler lives here. -``` -src/ - main.rs # entrypoint: build AppState (router + master key), bind, serve - state.rs # AppState — shared Arc + master_key - auth/ # authentication as an axum extractor — added to handler args - mod.rs # RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY) - routes/ # one module per route, all matching the same template - AGENTS.md # ← the route template (read this before adding a route) - mod.rs # app(): merges every module's router() - health.rs # simple route (one file): router() + liveness/readiness - realtime/ # route with logic → axum surface + a no-axum service: - mod.rs # router() + handler + WS<->events adapter (the axum surface) - service.rs # business logic (select deployment, call provider) — no axum, testable -``` - -## Rules - -- **Routes follow one template.** Each route module exposes - `pub fn router() -> Router`; `routes/mod.rs` only merges them. Simple - routes are one file; non-trivial routes are a folder (`handler`/`service`/ - `transport`). See `routes/AGENTS.md`. -- **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's - args; it runs during extraction. Never re-implement the check per route. -- **Handlers are thin.** A handler validates and delegates to its `service`. No - business logic, no provider calls, no transforms in handlers. -- **Services call `core`, they don't reimplement it.** A `service` picks the - deployment and calls the `core` route entrypoint. Provider resolution, auth - headers, URL building, and the HTTP call are `core`'s job; a service that - builds a provider request itself is a bug (`routes/messages/service.rs` is - the reference). -- **State is shared and cheap to clone.** Long-lived handles live behind `Arc` in - `state.rs`; read env/config only in `main.rs` when building state. +- Routes follow one template: each module exposes `pub fn router() -> Router`; `routes/mod.rs` only merges them. Simple routes are one file, non-trivial routes a folder (`handler`/`service`/`transport`). See `routes/AGENTS.md` +- Auth is an extractor: add `crate::auth::RequireMasterKey` to handler args; never re-implement the check per route +- Handlers are thin: validate and delegate to `service`; no business logic, no provider calls, no transforms +- Services call `core`, they never reimplement it: pick the deployment, call the entrypoint. Provider resolution, auth headers, URL, and the HTTP call are `core`'s job +- State is shared and cheap to clone: long-lived handles behind `Arc` in `state.rs`; read env/config only in `main.rs` ## Auth (interim) -A single **master key** (`LITELLM_MASTER_KEY`), enforced by the -`auth::RequireMasterKey` extractor: any caller presenting it as -`Authorization: Bearer ` may invoke the gateway. Fails closed (500) when -unset; constant-time compare. The server binds `127.0.0.1` by default (`HOST` to -override). Full per-key auth + budgets/rate-limits are delegated to the Python -proxy in a later phase. Health routes don't add the extractor (unauthenticated). +- Single master key (`LITELLM_MASTER_KEY`) enforced by `auth::RequireMasterKey`; `Authorization: Bearer ` +- Fails closed (500) when unset; constant-time compare; binds `127.0.0.1` by default (`HOST` to override) +- Per-key auth, budgets, rate limits delegated to the Python proxy later; health routes unauthenticated ## Python interop -Python-backed loading lives in `litellm-config` and is **load-time only**. The -gateway's `python-config` feature forwards to that crate. The realtime data path -never takes the GIL. +- Python-backed loading lives in `litellm-config`, load-time only; `python-config` feature forwards to it +- The realtime data path never takes the GIL diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/hooks.rs b/litellm-rust/crates/ai-gateway/src/audio_transcription/hooks.rs index dbe2d3a325b..35243a6dcf0 100644 --- a/litellm-rust/crates/ai-gateway/src/audio_transcription/hooks.rs +++ b/litellm-rust/crates/ai-gateway/src/audio_transcription/hooks.rs @@ -9,13 +9,13 @@ use std::future::Future; use std::pin::Pin; use super::types::PreparedAudioTranscriptionRequest; -use crate::integrations::custom_guardrail::{ +use litellm_core::integrations::custom_guardrail::{ CustomGuardrailRunner, GuardrailContext, GuardrailError, GuardrailRequest, }; -use crate::integrations::custom_logger::{ +use litellm_core::integrations::custom_logger::{ CallType, CallbackTiming, CallbackValue, CustomLoggerRunner, LoggingError, ModelCallDetails, }; -use crate::integrations::types::{ +use litellm_core::integrations::types::{ RequestMetadata, StandardLoggingMetadata, StandardLoggingPayload, }; diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/prepare.rs b/litellm-rust/crates/ai-gateway/src/audio_transcription/prepare.rs index a475d58635f..93113808f98 100644 --- a/litellm-rust/crates/ai-gateway/src/audio_transcription/prepare.rs +++ b/litellm-rust/crates/ai-gateway/src/audio_transcription/prepare.rs @@ -5,8 +5,8 @@ use litellm_core::routing_utils::provider::{CustomLlmProvider, get_custom_llm_pr use super::hooks::AudioTranscriptionLifecycleHooks; use super::types::{AudioTranscriptionRequest, PreparedAudioTranscriptionRequest}; -use crate::integrations::custom_guardrail::CustomGuardrailRunner; -use crate::integrations::custom_logger::CustomLoggerRunner; +use litellm_core::integrations::custom_guardrail::CustomGuardrailRunner; +use litellm_core::integrations::custom_logger::CustomLoggerRunner; pub(crate) struct PreparedAudioTranscriptionCall { pub(crate) request: PreparedAudioTranscriptionRequest, diff --git a/litellm-rust/crates/ai-gateway/src/audio_transcription/types.rs b/litellm-rust/crates/ai-gateway/src/audio_transcription/types.rs index b470638264e..72f9ef7a47e 100644 --- a/litellm-rust/crates/ai-gateway/src/audio_transcription/types.rs +++ b/litellm-rust/crates/ai-gateway/src/audio_transcription/types.rs @@ -4,9 +4,9 @@ use std::time::Duration; use litellm_core::call_lifecycle::{CallLifecycleContext, CallLifecycleRequest}; use serde_json::{Map, Value}; -use crate::integrations::custom_guardrail::CustomGuardrail; -use crate::integrations::custom_logger::CustomLogger; -use crate::integrations::types::RequestMetadata; +use litellm_core::integrations::custom_guardrail::CustomGuardrail; +use litellm_core::integrations::custom_logger::CustomLogger; +use litellm_core::integrations::types::RequestMetadata; pub struct AudioTranscriptionRequest<'a> { pub model: &'a str, diff --git a/litellm-rust/crates/ai-gateway/src/client.rs b/litellm-rust/crates/ai-gateway/src/client.rs deleted file mode 100644 index ff2606f0229..00000000000 --- a/litellm-rust/crates/ai-gateway/src/client.rs +++ /dev/null @@ -1,14 +0,0 @@ -use std::sync::OnceLock; -use std::time::Duration; - -const HTTP_CLIENT_TIMEOUT_SECS: u64 = 600; - -pub(crate) fn http_client() -> &'static reqwest::Client { - static CLIENT: OnceLock = OnceLock::new(); - CLIENT.get_or_init(|| { - reqwest::Client::builder() - .timeout(Duration::from_secs(HTTP_CLIENT_TIMEOUT_SECS)) - .build() - .expect("failed to build reqwest client") - }) -} diff --git a/litellm-rust/crates/ai-gateway/src/constants.rs b/litellm-rust/crates/ai-gateway/src/constants.rs index 78af374bf70..b7e317237b8 100644 --- a/litellm-rust/crates/ai-gateway/src/constants.rs +++ b/litellm-rust/crates/ai-gateway/src/constants.rs @@ -5,26 +5,6 @@ //! modules. Env-overridable tunables keep their `DEFAULT_*` value here; the env //! read + fallback happens at the host/config layer. -/// Default LiteLLM control-plane base URL for request-log egress when -/// `LITELLM_PROXY_BASE_URL` is unset. -pub(crate) const DEFAULT_PROXY_BASE_URL: &str = "http://localhost:4000"; - -/// The logs ingest path appended to the proxy base. Not a tunable; it is the -/// proxy's API contract (the rust-control-plane router on the Python proxy). -pub(crate) const RUST_CONTROL_PLANE_LOGS_PATH: &str = "/v1/rust_control_plane/logs"; - -/// Default bounded channel depth for the log-egress worker. -/// Override: `LITELLM_LOG_CHANNEL_CAPACITY`. -pub(crate) const DEFAULT_CHANNEL_CAPACITY: usize = 4096; - -/// Default max records POSTed per request to the control plane. -/// Override: `LITELLM_LOG_BATCH_SIZE`. -pub(crate) const DEFAULT_MAX_BATCH_SIZE: usize = 256; - -/// Default partial-batch flush cadence, in ms. -/// Override: `LITELLM_LOG_FLUSH_INTERVAL_MS`. -pub(crate) const DEFAULT_FLUSH_INTERVAL_MS: u64 = 500; - /// Provider attributed to realtime sessions in the logging payload. #[cfg(feature = "server")] pub(crate) const DEFAULT_PROVIDER: &str = "openai"; diff --git a/litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/types.rs b/litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/types.rs deleted file mode 100644 index 481a437747f..00000000000 --- a/litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/types.rs +++ /dev/null @@ -1,72 +0,0 @@ -use std::time::Duration; - -use serde::Serialize; - -use crate::constants::{ - DEFAULT_CHANNEL_CAPACITY, DEFAULT_FLUSH_INTERVAL_MS, DEFAULT_MAX_BATCH_SIZE, -}; -use crate::integrations::types::StandardLoggingPayload; - -#[derive(Serialize)] -pub struct CallbackLogsRequest { - pub records: Vec, -} - -#[derive(Serialize)] -pub struct CallbackLogRecord { - pub status: String, - pub standard_logging_payload: StandardLoggingPayload, - #[serde(skip_serializing_if = "Option::is_none")] - pub error: Option, -} - -#[derive(Clone, Debug)] -pub struct LogRecord { - pub status: String, - pub payload: StandardLoggingPayload, - pub error: Option, -} - -impl LogRecord { - pub fn into_callback_record(self) -> CallbackLogRecord { - CallbackLogRecord { - status: self.status, - standard_logging_payload: self.payload, - error: self.error, - } - } -} - -pub(super) struct EgressTunables { - pub channel_capacity: usize, - pub max_batch_size: usize, - pub flush_interval: Duration, -} - -impl EgressTunables { - pub fn from_env() -> Self { - Self { - channel_capacity: env_positive( - "LITELLM_LOG_CHANNEL_CAPACITY", - DEFAULT_CHANNEL_CAPACITY, - ), - max_batch_size: env_positive("LITELLM_LOG_BATCH_SIZE", DEFAULT_MAX_BATCH_SIZE), - flush_interval: Duration::from_millis(env_positive( - "LITELLM_LOG_FLUSH_INTERVAL_MS", - DEFAULT_FLUSH_INTERVAL_MS, - )), - } - } -} - -fn env_positive(name: &str, default: T) -> T -where - T: std::str::FromStr + PartialOrd + From, -{ - let zero = T::from(0u8); - std::env::var(name) - .ok() - .and_then(|value| value.trim().parse::().ok()) - .filter(|n| *n > zero) - .unwrap_or(default) -} diff --git a/litellm-rust/crates/ai-gateway/src/integrations/mod.rs b/litellm-rust/crates/ai-gateway/src/integrations/mod.rs deleted file mode 100644 index c62f1821ef8..00000000000 --- a/litellm-rust/crates/ai-gateway/src/integrations/mod.rs +++ /dev/null @@ -1,12 +0,0 @@ -//! Pure-Rust logging integrations. Names map 1:1 to Python -//! `litellm/integrations/`: -//! - [`custom_guardrail::CustomGuardrail`] — the guardrail callback trait -//! - [`custom_logger::CustomLogger`] — the callback trait -//! - [`litellm_python_proxy_api::LiteLLMPythonProxyAPILogger`] — ships events -//! to the Python proxy's `/v1/rust_control_plane/logs` endpoint -//! - [`types`] — the typed `StandardLoggingPayload` wire contract - -pub mod custom_guardrail; -pub mod custom_logger; -pub mod litellm_python_proxy_api; -pub mod types; diff --git a/litellm-rust/crates/ai-gateway/src/io/mod.rs b/litellm-rust/crates/ai-gateway/src/io/mod.rs index 7098d67993f..fc51e45c8ca 100644 --- a/litellm-rust/crates/ai-gateway/src/io/mod.rs +++ b/litellm-rust/crates/ai-gateway/src/io/mod.rs @@ -1,5 +1,4 @@ pub mod audio_transcription; -pub mod ocr; pub mod realtime; pub mod realtime_pool; pub mod responses_ws; diff --git a/litellm-rust/crates/ai-gateway/src/io/ocr.rs b/litellm-rust/crates/ai-gateway/src/io/ocr.rs deleted file mode 100644 index 2fc82f0b61f..00000000000 --- a/litellm-rust/crates/ai-gateway/src/io/ocr.rs +++ /dev/null @@ -1 +0,0 @@ -pub use crate::ocr::{OcrRequest, ocr}; diff --git a/litellm-rust/crates/ai-gateway/src/lib.rs b/litellm-rust/crates/ai-gateway/src/lib.rs index 08fbde564ed..b3f3028f361 100644 --- a/litellm-rust/crates/ai-gateway/src/lib.rs +++ b/litellm-rust/crates/ai-gateway/src/lib.rs @@ -13,9 +13,7 @@ //! binary turns on. pub mod audio_transcription; -mod client; pub mod io; -pub mod ocr; #[cfg(feature = "server")] pub mod auth; @@ -27,6 +25,5 @@ pub mod state; pub mod trace_parity; mod constants; -pub mod integrations; #[cfg(feature = "server")] mod realtime; diff --git a/litellm-rust/crates/ai-gateway/src/main.rs b/litellm-rust/crates/ai-gateway/src/main.rs index 88d7b1dbcf8..4cfd784dda4 100644 --- a/litellm-rust/crates/ai-gateway/src/main.rs +++ b/litellm-rust/crates/ai-gateway/src/main.rs @@ -10,6 +10,7 @@ //! wires startup. use std::sync::Arc; +use std::time::Duration; use litellm_ai_gateway::io::realtime_pool::{PoolConfig, RealtimePool, upstream_key}; use litellm_ai_gateway::routes; @@ -18,13 +19,19 @@ use litellm_ai_gateway::state::AppState; use litellm_config::load_model_list; use litellm_core::router::{Deployment, LiteLLMParams, Router}; -use litellm_ai_gateway::integrations::custom_logger::CustomLogger; -use litellm_ai_gateway::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger; +use litellm_core::integrations::custom_logger::CustomLogger; +use litellm_core::integrations::litellm_python_proxy_api::{ + LiteLLMPythonProxyAPILogger, LogEgressConfig, +}; /// Bind to localhost by default so the gateway is not a public, unauthenticated /// provider proxy out of the box. Override with `HOST` (e.g. `0.0.0.0`). const DEFAULT_HOST: &str = "127.0.0.1"; const DEFAULT_PORT: u16 = 4001; +const DEFAULT_PROXY_BASE_URL: &str = "http://localhost:4000"; +const DEFAULT_CHANNEL_CAPACITY: usize = 4096; +const DEFAULT_MAX_BATCH_SIZE: usize = 256; +const DEFAULT_FLUSH_INTERVAL_MS: u64 = 500; #[tokio::main] async fn main() { @@ -44,7 +51,7 @@ async fn main() { // Spawn the realtime-logging worker (drains a channel → POSTs batches to the // Python proxy's /v1/callbacks/logs). Built here so the spawn lands on the // tokio runtime. `from_env` reads LITELLM_PROXY_BASE_URL + LITELLM_MASTER_KEY. - let proxy_logger = LiteLLMPythonProxyAPILogger::from_env(); + let proxy_logger = configured_proxy_logger(); let loggers: Vec> = vec![proxy_logger]; let router = Arc::new(build_router()); @@ -86,6 +93,41 @@ async fn main() { .expect("server error"); } +fn configured_proxy_logger() -> Arc { + let base = std::env::var("LITELLM_PROXY_BASE_URL") + .ok() + .filter(|value| !value.trim().is_empty()) + .unwrap_or_else(|| DEFAULT_PROXY_BASE_URL.to_string()); + let key = std::env::var("LITELLM_MASTER_KEY").unwrap_or_default(); + LiteLLMPythonProxyAPILogger::start( + base, + key, + LogEgressConfig { + channel_capacity: positive_env( + "LITELLM_LOG_CHANNEL_CAPACITY", + DEFAULT_CHANNEL_CAPACITY, + ), + max_batch_size: positive_env("LITELLM_LOG_BATCH_SIZE", DEFAULT_MAX_BATCH_SIZE), + flush_interval: Duration::from_millis(positive_env( + "LITELLM_LOG_FLUSH_INTERVAL_MS", + DEFAULT_FLUSH_INTERVAL_MS, + )), + }, + ) +} + +fn positive_env(name: &str, default: T) -> T +where + T: std::str::FromStr + PartialOrd + From, +{ + let zero = T::from(0u8); + std::env::var(name) + .ok() + .and_then(|value| value.trim().parse::().ok()) + .filter(|value| *value > zero) + .unwrap_or(default) +} + /// Register every deployment's upstream key with the pool so the replenisher /// pre-warms it. Mirrors `service::run`'s key derivation (strip `openai/`, resolve /// api_key); deployments whose key can't be resolved are skipped (they fresh-dial diff --git a/litellm-rust/crates/ai-gateway/src/ocr/mod.rs b/litellm-rust/crates/ai-gateway/src/ocr/mod.rs deleted file mode 100644 index 2acdd232c80..00000000000 --- a/litellm-rust/crates/ai-gateway/src/ocr/mod.rs +++ /dev/null @@ -1,174 +0,0 @@ -use litellm_core::Error; -use litellm_core::call_lifecycle::CallLifecycle; -use serde_json::Value; - -mod common_utils; -mod handler; -mod hooks; -mod prepare; -mod types; - -pub use types::OcrRequest; - -use handler::execute_ocr_provider_call; -use prepare::{PreparedOcrCall, prepare_ocr_call}; - -#[tracing::instrument(target = "litellm::function_trace", level = "trace", skip_all)] -pub async fn ocr(request: OcrRequest<'_>) -> Result { - let PreparedOcrCall { request, hooks } = prepare_ocr_call(request); - CallLifecycle::default() - .run_request(request, &hooks, |request| { - execute_ocr_provider_call(request, &hooks) - }) - .await -} - -#[cfg(test)] -mod tests { - use serde_json::{Map, json}; - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - use tokio::net::{TcpListener, TcpStream}; - - use super::{OcrRequest, ocr}; - use crate::integrations::types::RequestMetadata; - - async fn read_http_request(socket: &mut TcpStream) -> String { - let mut request = Vec::new(); - let mut buffer = [0_u8; 1024]; - let header_end = loop { - let n = socket.read(&mut buffer).await.expect("reads request"); - if n == 0 { - break request.len(); - } - request.extend_from_slice(&buffer[..n]); - if let Some(position) = request.windows(4).position(|window| window == b"\r\n\r\n") { - break position + 4; - } - }; - let headers = String::from_utf8_lossy(&request[..header_end]); - let content_length = headers - .lines() - .find_map(|line| { - let (name, value) = line.split_once(':')?; - name.eq_ignore_ascii_case("content-length") - .then(|| value.trim().parse::().ok()) - .flatten() - }) - .unwrap_or(0); - while request.len().saturating_sub(header_end) < content_length { - let n = socket.read(&mut buffer).await.expect("reads body"); - if n == 0 { - break; - } - request.extend_from_slice(&buffer[..n]); - } - String::from_utf8(request).expect("request is utf8") - } - - fn base_ocr_request(model: &str) -> OcrRequest<'_> { - OcrRequest { - model, - document: json!({ - "type": "document_url", - "document_url": "https://example.com/doc.pdf" - }), - api_key: Some("sk-test"), - api_base: None, - custom_llm_provider: None, - extra_headers: None, - optional_params: Map::new(), - timeout: None, - callbacks: Vec::new(), - guardrails: Vec::new(), - request_metadata: RequestMetadata::default(), - litellm_call_id: None, - } - } - - #[tokio::test] - async fn reducto_file_upload_then_parse_maps_response() { - let listener = TcpListener::bind("127.0.0.1:0") - .await - .expect("test listener binds"); - let address = listener.local_addr().expect("listener has local address"); - let server = tokio::spawn(async move { - let (mut upload_socket, _) = listener.accept().await.expect("accepts upload request"); - let upload_request = read_http_request(&mut upload_socket).await; - let upload_body = r#"{"file_id":"reducto://uploaded.pdf"}"#; - let upload_response = format!( - "HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}", - upload_body.len(), - upload_body - ); - upload_socket - .write_all(upload_response.as_bytes()) - .await - .expect("writes upload response"); - - let (mut parse_socket, _) = listener.accept().await.expect("accepts parse request"); - let parse_request = read_http_request(&mut parse_socket).await; - let parse_body = r#"{"job_id":"job_123","usage":{"num_pages":3,"credits":3},"result":{"chunks":[{"content":"Page 1 block A","blocks":[{"content":"Page 1 block A","bbox":{"page":1},"kind":"text"}]},{"content":"Page 2 block A","blocks":[{"content":"Page 2 block A","bbox":{"page":2},"kind":"table"}]},{"content":"Page 1 block B","blocks":[{"content":"Page 1 block B","bbox":{"page":1},"kind":"text"}]},{"content":"Page 3 block A","blocks":[{"content":"Page 3 block A","bbox":{"page":3},"kind":"figure"}]}]}}"#; - let parse_response = format!( - "HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}", - parse_body.len(), - parse_body - ); - parse_socket - .write_all(parse_response.as_bytes()) - .await - .expect("writes parse response"); - (upload_request, parse_request) - }); - let api_base = format!("http://{address}"); - let mut request = base_ocr_request("reducto/parse-v3"); - request.api_base = Some(&api_base); - request.api_key = None; - request.extra_headers = Some(Map::from_iter([ - ("Authorization".to_string(), json!("Bearer test-key")), - ("x-trace-id".to_string(), json!("trace-1")), - ])); - request.document = json!({ - "type": "document_url", - "document_url": "data:application/pdf;base64,JVBERi0xLjQ=" - }); - request.optional_params = Map::from_iter([ - ( - "formatting".to_string(), - json!({"table_output_format": "html"}), - ), - ("retrieval".to_string(), json!({"chunk_mode": "section"})), - ("settings".to_string(), json!({"ocr_system": "standard"})), - ]); - - let response = ocr(request).await.expect("Reducto OCR succeeds"); - - assert_eq!(response["pages"].as_array().map(Vec::len), Some(3)); - assert_eq!( - response["pages"][0]["markdown"], - "Page 1 block A\n\nPage 1 block B" - ); - assert_eq!(response["pages"][1]["markdown"], "Page 2 block A"); - assert_eq!(response["pages"][2]["markdown"], "Page 3 block A"); - assert_eq!(response["usage_info"]["pages_processed"], 3); - assert_eq!(response["usage_info"]["credits"], 3); - assert_eq!(response["provider_native_response"]["job_id"], "job_123"); - let (upload_request, parse_request) = server.await.expect("server task completes"); - assert!( - upload_request - .to_ascii_lowercase() - .contains("authorization: bearer test-key") - ); - assert!(upload_request.contains("application/pdf")); - assert!(upload_request.contains("%PDF-1.4")); - assert!(upload_request.contains("x-trace-id: trace-1")); - assert!( - parse_request - .to_ascii_lowercase() - .contains("authorization: bearer test-key") - ); - assert!(parse_request.contains(r#""input":"reducto://uploaded.pdf""#)); - assert!(parse_request.contains(r#""table_output_format":"html""#)); - assert!(parse_request.contains(r#""chunk_mode":"section""#)); - assert!(parse_request.contains(r#""ocr_system":"standard""#)); - } -} diff --git a/litellm-rust/crates/ai-gateway/src/ocr/prepare.rs b/litellm-rust/crates/ai-gateway/src/ocr/prepare.rs deleted file mode 100644 index fa9ca1a193e..00000000000 --- a/litellm-rust/crates/ai-gateway/src/ocr/prepare.rs +++ /dev/null @@ -1,163 +0,0 @@ -use std::sync::atomic::{AtomicU64, Ordering}; -use std::time::{SystemTime, UNIX_EPOCH}; - -use litellm_core::routing_utils::provider::{CustomLlmProvider, get_custom_llm_provider}; -use serde_json::{Map, Value}; - -use super::common_utils::ocr_provider_config; -use super::hooks::OcrLifecycleHooks; -use super::types::{OcrRequest, PreparedOcrRequest}; -use crate::integrations::custom_guardrail::CustomGuardrailRunner; -use crate::integrations::custom_logger::CustomLoggerRunner; - -pub(crate) struct PreparedOcrCall { - pub(crate) request: PreparedOcrRequest, - pub(crate) hooks: OcrLifecycleHooks, -} - -#[tracing::instrument(target = "litellm::function_trace", level = "trace", skip_all)] -pub(crate) fn prepare_ocr_call(request: OcrRequest<'_>) -> PreparedOcrCall { - let call_id = request - .litellm_call_id - .map(str::to_string) - .unwrap_or_else(new_ocr_call_id); - let provider_info = get_custom_llm_provider(request.model, request.custom_llm_provider) - .unwrap_or(CustomLlmProvider { - model: request.model, - custom_llm_provider: "mistral", - }); - let model = provider_info.model.to_string(); - let custom_llm_provider = provider_info.custom_llm_provider.to_string(); - let config = ocr_provider_config(&custom_llm_provider, &model) - .ok_or_else(|| litellm_core::Error::InvalidProvider(custom_llm_provider.clone())) - .and_then(|config| { - validate_request_format(config, &request.optional_params, &custom_llm_provider)?; - Ok(config) - }); - let optional_params = match &config { - Ok(config) => { - let supported = config.supported_ocr_params(); - let mut mapped = config.map_ocr_params( - &request - .optional_params - .iter() - .filter(|(name, _)| supported.contains(&name.as_str())) - .map(|(name, value)| (name.clone(), value.clone())) - .collect(), - ); - for name in [ - "vertex_project", - "vertex_ai_project", - "vertex_location", - "vertex_ai_location", - ] { - if let Some(value) = request.optional_params.get(name) { - mapped.insert(name.to_string(), value.clone()); - } - } - mapped - } - Err(_) => request.optional_params, - }; - - PreparedOcrCall { - request: PreparedOcrRequest { - config, - model, - custom_llm_provider, - litellm_call_id: call_id, - document: request.document, - api_key: request.api_key.map(str::to_string), - api_base: request.api_base.map(str::to_string), - extra_headers: request.extra_headers, - optional_params, - timeout: request.timeout, - }, - hooks: OcrLifecycleHooks::new( - CustomLoggerRunner::new(request.callbacks), - CustomGuardrailRunner::new(request.guardrails), - request.request_metadata, - ), - } -} - -fn validate_request_format( - config: &'static dyn litellm_core::ocr::transformation::OcrProviderConfig, - optional_params: &Map, - provider: &str, -) -> Result<(), litellm_core::Error> { - let Some(format) = optional_params.get("req_format") else { - return Ok(()); - }; - match format.as_str() { - Some("litellm") => Ok(()), - Some("native") if config.supported_ocr_params().contains(&"req_format") => Ok(()), - Some("native") => Err(litellm_core::Error::InvalidRequest(format!( - "`req_format=native` is not supported for provider {provider}" - ))), - _ => Err(litellm_core::Error::InvalidRequest(format!( - "Invalid `req_format`: {format}. Expected `litellm` or `native`" - ))), - } -} - -fn new_ocr_call_id() -> String { - static COUNTER: AtomicU64 = AtomicU64::new(1); - let sequence = COUNTER.fetch_add(1, Ordering::Relaxed); - let timestamp = SystemTime::now() - .duration_since(UNIX_EPOCH) - .map(|duration| duration.as_nanos()) - .unwrap_or(0); - format!("ocr-{timestamp}-{sequence}") -} - -#[cfg(test)] -mod tests { - use litellm_core::error::Error; - use serde_json::{Map, json}; - - use super::{OcrRequest, prepare_ocr_call}; - use crate::integrations::types::RequestMetadata; - - fn base_ocr_request(model: &str) -> OcrRequest<'_> { - OcrRequest { - model, - document: json!({ - "type": "document_url", - "document_url": "https://example.com/doc.pdf" - }), - api_key: Some("sk-test"), - api_base: None, - custom_llm_provider: None, - extra_headers: None, - optional_params: Map::new(), - timeout: None, - callbacks: Vec::new(), - guardrails: Vec::new(), - request_metadata: RequestMetadata::default(), - litellm_call_id: None, - } - } - - fn request_with_format(format: &str) -> OcrRequest<'_> { - let mut request = base_ocr_request("mistral/mistral-ocr-latest"); - request.optional_params = Map::from_iter([("req_format".to_string(), json!(format))]); - request - } - - #[test] - fn native_format_rejected_for_provider_without_support_as_bad_request() { - let prepared = prepare_ocr_call(request_with_format("native")); - assert!( - matches!(prepared.request.config, Err(Error::InvalidRequest(message)) if message.contains("not supported for provider")) - ); - } - - #[test] - fn unknown_format_rejected_for_provider_without_support_as_bad_request() { - let prepared = prepare_ocr_call(request_with_format("raw")); - assert!( - matches!(prepared.request.config, Err(Error::InvalidRequest(message)) if message.contains("Invalid `req_format`")) - ); - } -} diff --git a/litellm-rust/crates/ai-gateway/src/realtime/streaming.rs b/litellm-rust/crates/ai-gateway/src/realtime/streaming.rs index c0d72e90b77..8ce3ea5a123 100644 --- a/litellm-rust/crates/ai-gateway/src/realtime/streaming.rs +++ b/litellm-rust/crates/ai-gateway/src/realtime/streaming.rs @@ -13,10 +13,10 @@ use litellm_core::realtime::types::RealtimeEvent; use serde_json::Value; use crate::constants::DEFAULT_PROVIDER; -use crate::integrations::custom_logger::{ +use litellm_core::integrations::custom_logger::{ CallbackTiming, CallbackValue, CustomLogger, CustomLoggerRunner, LoggingError, ModelCallDetails, }; -use crate::integrations::types::{ +use litellm_core::integrations::types::{ RequestMetadata, StandardLoggingMetadata, StandardLoggingPayload, Usage, }; @@ -232,8 +232,8 @@ impl RealTimeStreaming { #[cfg(test)] mod tests { use super::*; - use crate::integrations::custom_logger::LogError; - use crate::integrations::custom_logger::LogFuture; + use litellm_core::integrations::custom_logger::LogError; + use litellm_core::integrations::custom_logger::LogFuture; use std::sync::atomic::{AtomicU64, Ordering}; fn event(raw: &str) -> RealtimeEvent { diff --git a/litellm-rust/crates/ai-gateway/src/routes/AGENTS.md b/litellm-rust/crates/ai-gateway/src/routes/AGENTS.md index c675916f71a..74d7f03ac6b 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/AGENTS.md +++ b/litellm-rust/crates/ai-gateway/src/routes/AGENTS.md @@ -1,43 +1,23 @@ # routes/ — the route template -Every route follows the **same shape** so the layout is predictable. The rule: - -> **Each route module exposes `pub fn router() -> Router`.** -> `routes/mod.rs::app` merges them all and applies state once. Adding a route is: -> create the module, then add one `.merge(::router())` line. +Every route follows one shape: each module exposes `pub fn router() -> Router`; `routes/mod.rs::app` merges them all and applies state once. Adding a route means creating the module and adding one `.merge(::router())` line. ## Default: one file -A route is a single file containing `router()` + its handler(s) (handlers stay -private). This is the norm — don't split until it hurts. -``` -pub fn router() -> Router { Router::new().route(PATH, get(handle)) } -async fn handle(...) -> impl IntoResponse { ... } -``` -`health.rs` is the example. + +- A route is one file with `router()` + its handler(s) (handlers stay private); `health.rs` is the example +- Don't split until it hurts ## Split out `service` when there's real logic -When a route has business logic worth testing without axum, put it in a sibling -`service` (a file, or a folder if the route grows). The route file stays the -**axum surface** (router + handler + any socket/SSE adapter); `service` is plain -Rust with **no axum types**, and its job is to pick the deployment and call the -`core` route entrypoint (see `messages/service.rs` calling -`litellm_core::messages::messages`). Never build a provider request, resolve a -key, or perform the provider call here. `realtime/` is the older example: -``` -realtime/ - mod.rs # axum surface: router() + handler + the WS<->events adapter - service.rs # pure logic: select deployment + call provider (no axum) — testable -``` -Split `service` further (or add `transport`, `repo`, …) only once a single file -genuinely gets hard to read. + +- Put testable business logic in a sibling `service` (file, or folder if it grows) +- The route file stays the axum surface (router + handler + any socket/SSE adapter); `service` is plain Rust with no axum types +- `service` picks the deployment and calls the `core` entrypoint (see `messages/service.rs`), never builds a provider request, resolves a key, or performs the provider call +- Split further (`transport`, `repo`, ...) only when one file genuinely gets hard to read ## Invariants -- **Auth is an extractor, not a manual call.** A handler requires auth by adding - `crate::auth::RequireMasterKey` to its arguments; it runs during extraction. - Never re-implement the check per route. -- **Handlers contain no business logic; `service` contains no axum types.** -- **No provider handlers in this crate.** Transforms, auth headers, and the - provider HTTP call live in `core/src//`. -- A route owns its paths in its own `router()`; `mod.rs` only merges. -- Cross-cutting concerns (logging, CORS, timeouts) → Tower layers in `mod.rs`, - not duplicated in handlers. + +- Auth is an extractor, not a manual call +- Handlers contain no business logic; `service` contains no axum types +- No provider handlers here: transforms, auth headers, and the provider HTTP call live in `core/src//` +- A route owns its paths in its own `router()`; `mod.rs` only merges +- Cross-cutting concerns go in Tower layers in `mod.rs`, not duplicated in handlers diff --git a/litellm-rust/crates/ai-gateway/src/routes/realtime/mod.rs b/litellm-rust/crates/ai-gateway/src/routes/realtime/mod.rs index f9144ad1fdb..96c5d429e96 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/realtime/mod.rs +++ b/litellm-rust/crates/ai-gateway/src/routes/realtime/mod.rs @@ -23,10 +23,10 @@ use litellm_core::router::Router as ModelRouter; use serde::Deserialize; use crate::auth::RequireMasterKey; -use crate::integrations::custom_logger::CustomLogger; -use crate::integrations::types::RequestMetadata; use crate::realtime::streaming::{RealTimeStreaming, SessionStatus}; use crate::state::AppState; +use litellm_core::integrations::custom_logger::CustomLogger; +use litellm_core::integrations::types::RequestMetadata; /// Process-local monotonic counter, mixed into the per-session call id so two /// sessions opened in the same nanosecond still get distinct ids. diff --git a/litellm-rust/crates/ai-gateway/src/routes/responses/mod.rs b/litellm-rust/crates/ai-gateway/src/routes/responses/mod.rs index a94853e106d..dcaacbba668 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/responses/mod.rs +++ b/litellm-rust/crates/ai-gateway/src/routes/responses/mod.rs @@ -16,9 +16,9 @@ use litellm_core::router::Router as ModelRouter; use serde::Deserialize; use crate::auth::RequireMasterKey; -use crate::integrations::custom_logger::CustomLogger; -use crate::integrations::types::RequestMetadata; use crate::state::AppState; +use litellm_core::integrations::custom_logger::CustomLogger; +use litellm_core::integrations::types::RequestMetadata; static CALL_SEQ: AtomicU64 = AtomicU64::new(0); diff --git a/litellm-rust/crates/ai-gateway/src/routes/responses/service.rs b/litellm-rust/crates/ai-gateway/src/routes/responses/service.rs index e8f840c0c8e..e525e80cbf8 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/responses/service.rs +++ b/litellm-rust/crates/ai-gateway/src/routes/responses/service.rs @@ -10,10 +10,10 @@ use litellm_core::responses::instrumentation::{ }; use litellm_core::responses::types::ResponsesWsEvent; -use crate::integrations::custom_logger::{ +use litellm_core::integrations::custom_logger::{ CallbackTiming, CallbackValue, CustomLogger, CustomLoggerRunner, LoggingError, ModelCallDetails, }; -use crate::integrations::types::RequestMetadata; +use litellm_core::integrations::types::RequestMetadata; #[allow(clippy::too_many_arguments)] pub async fn run( @@ -126,7 +126,7 @@ fn logging_values( let end_time = payload.end_time; let callback = CallbackValue::new(callback.object, callback.value); let details = ModelCallDetails::from_standard_logging_payload( - crate::integrations::types::StandardLoggingPayload { + litellm_core::integrations::types::StandardLoggingPayload { id: payload.id, litellm_call_id: payload.litellm_call_id, call_type: payload.call_type, @@ -139,7 +139,7 @@ fn logging_values( start_time: payload.start_time, end_time: payload.end_time, stream: payload.stream, - metadata: crate::integrations::types::StandardLoggingMetadata { + metadata: litellm_core::integrations::types::StandardLoggingMetadata { user_api_key_hash: payload.metadata.user_api_key_hash, user_api_key_user_id: payload.metadata.user_api_key_user_id, user_api_key_team_id: payload.metadata.user_api_key_team_id, diff --git a/litellm-rust/crates/ai-gateway/src/state.rs b/litellm-rust/crates/ai-gateway/src/state.rs index 3b61d8309ea..34b4534fc74 100644 --- a/litellm-rust/crates/ai-gateway/src/state.rs +++ b/litellm-rust/crates/ai-gateway/src/state.rs @@ -3,7 +3,7 @@ use std::sync::Arc; use crate::io::realtime_pool::RealtimePool; use litellm_core::router::Router; -use crate::integrations::custom_logger::CustomLogger; +use litellm_core::integrations::custom_logger::CustomLogger; /// Shared application state handed to every route handler. #[derive(Clone)] diff --git a/litellm-rust/crates/config/AGENTS.md b/litellm-rust/crates/config/AGENTS.md new file mode 100644 index 00000000000..08c1250485b --- /dev/null +++ b/litellm-rust/crates/config/AGENTS.md @@ -0,0 +1,6 @@ +litellm-config is the config-loading boundary. It returns resolved core deployment data and optionally delegates loading to Python. + +- Depends on `litellm-core`; may use PyO3 behind the `python` feature +- `load_model_list(path)` calls Python's `litellm.proxy.read_model_list` and parses the JSON into `core::router::Deployment` +- Load-time/startup only, never on a request or stream path +- Returns typed `Error` variants (`PythonLoading`, `Serialization`, `ModelListParsing`) diff --git a/litellm-rust/crates/core/AGENTS.md b/litellm-rust/crates/core/AGENTS.md index aee8b4937ef..25691348e42 100644 --- a/litellm-rust/crates/core/AGENTS.md +++ b/litellm-rust/crates/core/AGENTS.md @@ -1,7 +1,126 @@ -litellm-core is the LiteLLM SDK in Rust — it makes the LLM call. Each top-level call is a module under `src//` exposing a public entrypoint named after the route (`messages::messages()`, the Rust equivalent of `litellm.messages()`): you call it and get a typed non-streaming response back. +# core — the runtime and the single core route -A route module owns everything the call needs: types, the provider template trait, provider transforms (under `providers/`), provider/auth/URL resolution, and the handler that performs the HTTP call. Handlers belong here, never in a host crate. +`litellm-core` is the LiteLLM SDK in Rust: one language-neutral route per +endpoint (`messages::messages`, `ocr`, `audio_transcription`, `realtime`, +`responses`). Every host (the Rust SDK, the PyO3 bridge, the gateway) calls the +same route program; they differ only in the service implementations they +supply. Core owns route sequencing, provider policy, transformation, +authentication, protocol operations, provider I/O and callback phase placement. +Hosts provide capabilities; they never orchestrate the route. -Not here: serving HTTP (axum routes, extractors), config file reading, rollout state, databases, or callback dispatch. Env reads are limited to credential fallback in a route's `prepare.rs`. +## What core owns -Routes (messages, ocr, realtime) and providers (anthropic, mistral, openai) are modules, not crates. +Core owns all decisions that must stay identical across hosts. A host adapter +that performs any of the following around a core entrypoint is a bug: + +- private admission, and the point after which execution cannot be replayed; +- deployment and provider interpretation supplied in the typed route inputs; +- callback and guardrail phase placement and result interpretation; +- provider transformation, credential acquisition, final authorization and + encoding; +- explicit multi-operation protocols such as OCR upload, submission and + polling; +- provider I/O, response normalization and terminal success or failure; +- transfer of stream, connection and deferred-completion ownership. + +A host adapter performs only work specific to its public boundary: construct +the runtime composition, convert public inputs into `request` + `options` + +`CallContext`, invoke exactly one core route entrypoint, and map the outcome to +the host's response or error contract. Never a second provider pipeline. + +## Route shape + +A top-level call is a module under `src//`, shaped like `messages`: + +``` +core/src/messages/ + mod.rs # pub async fn messages(..) -> Result<.., Error> (+ _stream for SSE) + types.rs # request/response types + transformation.rs # the provider template trait + prepare.rs # provider resolution, auth headers, URL + handler.rs # the provider call + client.rs # the shared reqwest client +``` + +`ocr` is in flight: today it holds only `transformation` and `types`; the rest +of its lifecycle still lives in the gateway and moves here as it migrates. +`audio_transcription` and `realtime` are the same. Bringing a route to full +core shape means giving it a `mod.rs` entrypoint that owns the sequence above. + +The invariant is one function body owns the route lifecycle. The conceptual +shape is: + +```rust +pub async fn ocr( + services: &S, + request: OcrRequest, + options: OcrOptions, + context: CallContext, +) -> NativeResult +where + S: OcrServices, +``` + +The exact spelling may be a method on `LiteLlm`. Public adapters may wrap +that function but can never reimplement admission, callbacks, provider +preparation or transport around it. + +## Services, not a context + +Capabilities are supplied through focused trait implementations. Route-specific +requirements traits (an `OcrServices` bundle) declare exactly what a route needs +(transport, calls, clock, ...); they are not host contexts, and core is never +passed a catch-all gateway environment. The ordinary Rust client supplies native +defaults; callers override implementations at construction. + +`CallServices` opens one request-scoped sessions per call, owning per-call state +(timing, logging state, retained host objects, deferred completion, correlation +state). No-op call services are the default, and their presence must not move +provider behavior into a host or force callback payload materialization on an +unaffected fast path. + +Your `Callback`/`CallServices` operations must stay distinct where their +contracts differ (argument identity, replacement adoption, exception policy, +scheduling, direct vs awaited execution). A universal `emit(Event, Json)` is +not enough. core still decides which operation runs next and how its result +affects execution; the adapter only dispatches it (Python callback, native +logger call, or nothing). + +## Language neutrality + +Core types and service contracts must not contain `Py`, `Py`, +Axum requests, gateway state or Python logging objects. Host implementations may +retain those values privately; core sees only the trait operations and the typed +values they return. `CallContext`, `*Request`, `*Options`, `*Response` and core +errors are language-neutral. + +## Dependency rules + +``` +core must not depend on PyO3, Axum or gateway integration types +Tower/Axum types stop at the gateway adapter boundary +provider transformation, auth and I/O remain in core +request-scoped host state belongs to a call session +service construction dependencies do not leak into service interfaces +``` + +## Not the target + +The current `CallLifecycleHooks` shape is not the final public service API: it +folds lifecycle sequencing into a stateless generic transformation interface, +needs `Send` futures, cannot express the full replacement and error contracts, +and does not model request-scoped retained ownership or Python caller-task +driving. It is a stepping stone, not the contract to build new routes against. + +Do not introduce a dynamic `TypeId` service map, a shared gateway callback +environment reused from core or the bridge, per-callback JSON serialization, or +a full Effect layer API. Services traits plus constructors and a scoped call +owner are the minimum design; add more machinery only when concrete consumers +require it. + +## Gateway migration + +Existing gateway-hosted OCR, transcription and WebSocket provider execution +predates this boundary and must move here as those routes migrate. Keeping a +gateway callback as a `CallServices` implementation does not justify keeping +provider orchestration beside it in the gateway. diff --git a/litellm-rust/crates/core/CLAUDE.md b/litellm-rust/crates/core/CLAUDE.md index 5d36305ded5..3a9bfcd1ff0 100644 --- a/litellm-rust/crates/core/CLAUDE.md +++ b/litellm-rust/crates/core/CLAUDE.md @@ -1,66 +1 @@ -# CLAUDE.md - -Rules for `litellm-rust/crates/core`. - -## Responsibility - -`core` is the LiteLLM SDK in Rust: it makes the LLM call. Every top-level -LiteLLM call has a public entrypoint here, named after the route -(`messages::messages()` is the Rust equivalent of `litellm.messages()`), and -calling it returns a typed non-streaming response. - -Allowed: -- The public entrypoint for a route, plus its `_stream` variant when the - route supports streaming. -- Provider resolution, auth header construction, URL building, and the provider - HTTP call (shared reused client, connect + request timeouts). -- Shared request/response structs. -- Typed errors with stable, non-sensitive messages. -- Deterministic validation helpers. -- Serialization helpers that intentionally mirror Python output shape. -- Route templates that match Python base config responsibilities, such as - `messages::transformation::AnthropicMessagesProviderConfig`. - -Not allowed: -- Serving HTTP: axum routers, extractors, and other transport concerns. -- Filesystem, database, or cache access. -- Config file reading or rollout state; the host resolves those and passes them - in. Env reads are limited to credential fallback in a route's `prepare.rs`. -- Logging callbacks, tracing spans, spend writes, or customer callbacks. -- Provider-specific branching that belongs in `providers`. -- Panics for user/provider-controlled input. - -## Typed Contracts (core rule) - -Trait and function boundaries MUST be strongly typed. No stringly-typed JSON -(`&str` / `String` / `Vec` / bare `serde_json::Value`) as a transform -input or output. Parse wire bytes into typed structs/enums at the host edge; -`core` and `providers` operate only on those types (e.g. `RealtimeEvent`, -`RealtimeTransformResult`, `OcrRequestData`). A `type`-style discriminator is a -typed field on a struct, not a raw string threaded through the API. - -## Structure - -Use route names directly under `src/`: `messages`, `ocr`, future -`chat_completions`, `embeddings`, and similar top-level LiteLLM calls. Do not -invent broad names like `engine` for route contracts. - -`src/messages` is the reference shape for a route module: - -``` -mod.rs pub async fn messages(..) (+ messages_stream) -types.rs request/response types -transformation.rs the provider template trait -prepare.rs provider resolution, auth headers, URL -handler.rs the provider call -client.rs the shared reqwest client -``` - -## Parity Rules - -- Every shared type used by a provider transform needs unit tests for - serialization shape. -- If Python parity requires always emitting a `null` field instead of omitting - it, document that in code and pin it with a test. -- Error enums should preserve enough detail for Python/HTTP hosts to map errors - consistently without exposing document contents or upstream bodies. +See ./AGENTS.md diff --git a/litellm-rust/crates/core/Cargo.toml b/litellm-rust/crates/core/Cargo.toml index c0de7ff3977..65f190db63b 100644 --- a/litellm-rust/crates/core/Cargo.toml +++ b/litellm-rust/crates/core/Cargo.toml @@ -13,6 +13,7 @@ serde.workspace = true serde_json.workspace = true thiserror.workspace = true tracing.workspace = true +tokio = { workspace = true, features = ["rt", "sync", "time"] } tracing-subscriber = { workspace = true, optional = true } sha2.workspace = true aws-config = { version = "1.9.0", default-features = false, features = ["rustls", "rt-tokio"], optional = true } @@ -36,5 +37,5 @@ observability = ["dep:tracing-subscriber"] [dev-dependencies] rstest.workspace = true -tokio = { workspace = true, features = ["macros", "rt-multi-thread"] } +tokio = { workspace = true, features = ["io-util", "macros", "net", "rt-multi-thread"] } tracing-subscriber.workspace = true diff --git a/litellm-rust/crates/core/src/constants.rs b/litellm-rust/crates/core/src/constants.rs index ec1a2988e75..2240de45237 100644 --- a/litellm-rust/crates/core/src/constants.rs +++ b/litellm-rust/crates/core/src/constants.rs @@ -45,3 +45,5 @@ pub const EMPTY_TEXT_PLACEHOLDER: &str = "[System: Empty message content sanitised to satisfy protocol]"; pub const FUNCTION_TRACE_TARGET: &str = "litellm::function_trace"; + +pub(crate) const RUST_CONTROL_PLANE_LOGS_PATH: &str = "/v1/rust_control_plane/logs"; diff --git a/litellm-rust/crates/ai-gateway/src/integrations/custom_guardrail/mod.rs b/litellm-rust/crates/core/src/integrations/custom_guardrail/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/custom_guardrail/mod.rs rename to litellm-rust/crates/core/src/integrations/custom_guardrail/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/custom_guardrail/types.rs b/litellm-rust/crates/core/src/integrations/custom_guardrail/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/custom_guardrail/types.rs rename to litellm-rust/crates/core/src/integrations/custom_guardrail/types.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/custom_logger/mod.rs b/litellm-rust/crates/core/src/integrations/custom_logger/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/custom_logger/mod.rs rename to litellm-rust/crates/core/src/integrations/custom_logger/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/custom_logger/types.rs b/litellm-rust/crates/core/src/integrations/custom_logger/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/custom_logger/types.rs rename to litellm-rust/crates/core/src/integrations/custom_logger/types.rs diff --git a/litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/mod.rs b/litellm-rust/crates/core/src/integrations/litellm_python_proxy_api/mod.rs similarity index 81% rename from litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/mod.rs rename to litellm-rust/crates/core/src/integrations/litellm_python_proxy_api/mod.rs index 3dad18cb7a3..5d91f7b75fa 100644 --- a/litellm-rust/crates/ai-gateway/src/integrations/litellm_python_proxy_api/mod.rs +++ b/litellm-rust/crates/core/src/integrations/litellm_python_proxy_api/mod.rs @@ -16,12 +16,12 @@ use reqwest::Client; use tokio::sync::mpsc::{self, Receiver, Sender}; use tokio::time::interval; -use crate::constants::{DEFAULT_PROXY_BASE_URL, RUST_CONTROL_PLANE_LOGS_PATH}; +use crate::constants::RUST_CONTROL_PLANE_LOGS_PATH; use crate::integrations::custom_logger::{ CallbackTiming, CallbackValue, CustomLogger, LogError, LogFuture, LoggingError, ModelCallDetails, }; -use types::{CallbackLogsRequest, EgressTunables, LogRecord}; +use types::{CallbackLogsRequest, LogRecord}; pub mod types; @@ -33,9 +33,8 @@ pub struct LiteLLMPythonProxyAPILogger { impl LiteLLMPythonProxyAPILogger { /// Spawn the background worker and return a logger handle. `base` is the /// proxy base URL (no trailing path); `master_key` is sent as a bearer token. - pub fn start(base: String, master_key: String) -> Arc { - let tunables = EgressTunables::from_env(); - let (sink, receiver) = mpsc::channel::(tunables.channel_capacity); + pub fn start(base: String, master_key: String, config: LogEgressConfig) -> Arc { + let (sink, receiver) = mpsc::channel::(config.channel_capacity); let url = format!( "{}{}", base.trim_end_matches('/'), @@ -47,29 +46,12 @@ impl LiteLLMPythonProxyAPILogger { client, url, master_key, - tunables.max_batch_size, - tunables.flush_interval, + config.max_batch_size, + config.flush_interval, )); Arc::new(Self { sink }) } - /// Build a logger from the environment: `LITELLM_PROXY_BASE_URL` (default - /// `http://localhost:4000`) and `LITELLM_MASTER_KEY`. - /// - /// `LITELLM_PROXY_BASE_URL` is treated as the full base and the route is - /// appended verbatim, so if the proxy runs under a `SERVER_ROOT_PATH` - /// (e.g. served at `https://host/litellm`), include it in the base - /// (`LITELLM_PROXY_BASE_URL=https://host/litellm`) and the POST lands at - /// `https://host/litellm/v1/rust_control_plane/logs`. - pub fn from_env() -> Arc { - let base = std::env::var("LITELLM_PROXY_BASE_URL") - .ok() - .filter(|value| !value.trim().is_empty()) - .unwrap_or_else(|| DEFAULT_PROXY_BASE_URL.to_string()); - let key = std::env::var("LITELLM_MASTER_KEY").unwrap_or_default(); - Self::start(base, key) - } - fn enqueue(&self, record: LogRecord) -> Result<(), LogError> { self.sink.try_send(record).map_err(|err| match err { mpsc::error::TrySendError::Full(_) => LogError::channel_full(), @@ -78,6 +60,13 @@ impl LiteLLMPythonProxyAPILogger { } } +#[derive(Clone, Copy, Debug)] +pub struct LogEgressConfig { + pub channel_capacity: usize, + pub max_batch_size: usize, + pub flush_interval: Duration, +} + impl CustomLogger for LiteLLMPythonProxyAPILogger { fn async_log_success_event<'a>( &'a self, diff --git a/litellm-rust/crates/core/src/integrations/litellm_python_proxy_api/types.rs b/litellm-rust/crates/core/src/integrations/litellm_python_proxy_api/types.rs new file mode 100644 index 00000000000..767fd718e3a --- /dev/null +++ b/litellm-rust/crates/core/src/integrations/litellm_python_proxy_api/types.rs @@ -0,0 +1,33 @@ +use serde::Serialize; + +use crate::integrations::types::StandardLoggingPayload; + +#[derive(Serialize)] +pub struct CallbackLogsRequest { + pub records: Vec, +} + +#[derive(Serialize)] +pub struct CallbackLogRecord { + pub status: String, + pub standard_logging_payload: StandardLoggingPayload, + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, +} + +#[derive(Clone, Debug)] +pub struct LogRecord { + pub status: String, + pub payload: StandardLoggingPayload, + pub error: Option, +} + +impl LogRecord { + pub fn into_callback_record(self) -> CallbackLogRecord { + CallbackLogRecord { + status: self.status, + standard_logging_payload: self.payload, + error: self.error, + } + } +} diff --git a/litellm-rust/crates/core/src/integrations/mod.rs b/litellm-rust/crates/core/src/integrations/mod.rs new file mode 100644 index 00000000000..382c90979fb --- /dev/null +++ b/litellm-rust/crates/core/src/integrations/mod.rs @@ -0,0 +1,4 @@ +pub mod custom_guardrail; +pub mod custom_logger; +pub mod litellm_python_proxy_api; +pub mod types; diff --git a/litellm-rust/crates/ai-gateway/src/integrations/types.rs b/litellm-rust/crates/core/src/integrations/types.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/integrations/types.rs rename to litellm-rust/crates/core/src/integrations/types.rs diff --git a/litellm-rust/crates/core/src/lib.rs b/litellm-rust/crates/core/src/lib.rs index b93e084f57e..25f4fce6f43 100644 --- a/litellm-rust/crates/core/src/lib.rs +++ b/litellm-rust/crates/core/src/lib.rs @@ -5,6 +5,7 @@ pub mod chat_completions; pub mod constants; pub mod error; pub mod http_utils; +pub mod integrations; pub mod messages; #[cfg(any(feature = "observability", test))] pub mod observability; diff --git a/litellm-rust/crates/core/src/ocr/client.rs b/litellm-rust/crates/core/src/ocr/client.rs new file mode 100644 index 00000000000..73f5bb00800 --- /dev/null +++ b/litellm-rust/crates/core/src/ocr/client.rs @@ -0,0 +1,13 @@ +use std::sync::OnceLock; +use std::time::Duration; + +pub(super) fn http_client() -> &'static reqwest::Client { + static CLIENT: OnceLock = OnceLock::new(); + CLIENT.get_or_init(|| { + reqwest::Client::builder() + .timeout(Duration::from_secs(600)) + .connect_timeout(Duration::from_secs(10)) + .build() + .unwrap_or_else(|_| reqwest::Client::new()) + }) +} diff --git a/litellm-rust/crates/ai-gateway/src/ocr/common_utils.rs b/litellm-rust/crates/core/src/ocr/common_utils.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/ocr/common_utils.rs rename to litellm-rust/crates/core/src/ocr/common_utils.rs diff --git a/litellm-rust/crates/ai-gateway/src/ocr/handler.rs b/litellm-rust/crates/core/src/ocr/handler.rs similarity index 92% rename from litellm-rust/crates/ai-gateway/src/ocr/handler.rs rename to litellm-rust/crates/core/src/ocr/handler.rs index 6c6e12724cd..4c8401cd1fc 100644 --- a/litellm-rust/crates/ai-gateway/src/ocr/handler.rs +++ b/litellm-rust/crates/core/src/ocr/handler.rs @@ -1,12 +1,12 @@ -use litellm_core::error::Error; -use litellm_core::http_utils::http_request; -use litellm_core::ocr::transformation::OcrResponseHandling; +use crate::error::Error; +use crate::http_utils::http_request; +use crate::ocr::transformation::OcrResponseHandling; use serde_json::Value; +use super::client::http_client; use super::common_utils::{poll_document_intelligence, truncate_error_body}; use super::hooks::OcrLifecycleHooks; -use super::types::PreparedOcrRequest; -use crate::client::http_client; +use super::runtime_types::PreparedOcrRequest; #[tracing::instrument(target = "litellm::function_trace", level = "trace", skip_all)] pub(crate) async fn execute_ocr_provider_call( diff --git a/litellm-rust/crates/ai-gateway/src/ocr/hooks.rs b/litellm-rust/crates/core/src/ocr/hooks.rs similarity index 98% rename from litellm-rust/crates/ai-gateway/src/ocr/hooks.rs rename to litellm-rust/crates/core/src/ocr/hooks.rs index ed41f1ff9e7..e385a0de775 100644 --- a/litellm-rust/crates/ai-gateway/src/ocr/hooks.rs +++ b/litellm-rust/crates/core/src/ocr/hooks.rs @@ -1,15 +1,15 @@ -use litellm_core::call_lifecycle::{CallLifecycleContext, CallLifecycleHooks, CallLifecycleTiming}; -use litellm_core::error::Error; -use litellm_core::providers::reducto::ocr::transformation::{ +use crate::call_lifecycle::{CallLifecycleContext, CallLifecycleHooks, CallLifecycleTiming}; +use crate::error::Error; +use crate::providers::reducto::ocr::transformation::{ build_upload_request, extract_document_source, extract_upload_file_id, }; use serde_json::{Map, Value, json}; use std::future::Future; use std::pin::Pin; +use super::client::http_client; use super::common_utils::{convert_document_url_to_data_uri, string_headers, truncate_error_body}; -use super::types::{PreparedOcrRequest, ProviderOcrRequest}; -use crate::client::http_client; +use super::runtime_types::{PreparedOcrRequest, ProviderOcrRequest}; use crate::integrations::custom_guardrail::{ CustomGuardrailRunner, GuardrailContext, GuardrailError, GuardrailRequest, }; diff --git a/litellm-rust/crates/ai-gateway/src/ocr/types.rs b/litellm-rust/crates/core/src/ocr/runtime_types.rs similarity index 89% rename from litellm-rust/crates/ai-gateway/src/ocr/types.rs rename to litellm-rust/crates/core/src/ocr/runtime_types.rs index 75a8e61ddbf..5e9e531a0c3 100644 --- a/litellm-rust/crates/ai-gateway/src/ocr/types.rs +++ b/litellm-rust/crates/core/src/ocr/runtime_types.rs @@ -1,8 +1,8 @@ use std::sync::Arc; use std::time::Duration; -use litellm_core::call_lifecycle::{CallLifecycleContext, CallLifecycleRequest}; -use litellm_core::ocr::transformation::OcrProviderConfig; +use crate::call_lifecycle::{CallLifecycleContext, CallLifecycleRequest}; +use crate::ocr::transformation::OcrProviderConfig; use serde_json::{Map, Value}; use crate::integrations::custom_guardrail::CustomGuardrail; @@ -25,7 +25,7 @@ pub struct OcrRequest<'a> { } pub(crate) struct PreparedOcrRequest { - pub(crate) config: Result<&'static dyn OcrProviderConfig, litellm_core::Error>, + pub(crate) config: Result<&'static dyn OcrProviderConfig, crate::Error>, pub(crate) model: String, pub(crate) custom_llm_provider: String, pub(crate) litellm_call_id: String, diff --git a/litellm-rust/crates/ai-gateway/tests/ocr_lifecycle.rs b/litellm-rust/crates/core/tests/ocr_lifecycle.rs similarity index 97% rename from litellm-rust/crates/ai-gateway/tests/ocr_lifecycle.rs rename to litellm-rust/crates/core/tests/ocr_lifecycle.rs index 60e90ed2a7c..eb5b167200a 100644 --- a/litellm-rust/crates/ai-gateway/tests/ocr_lifecycle.rs +++ b/litellm-rust/crates/core/tests/ocr_lifecycle.rs @@ -1,22 +1,22 @@ use std::sync::{Arc, Mutex}; use std::time::Duration; -use litellm_ai_gateway::integrations::custom_guardrail::{ +use litellm_core::error::Error; +use litellm_core::integrations::custom_guardrail::{ CustomGuardrail, GuardrailContext, GuardrailDecision, GuardrailError, GuardrailEventHook, GuardrailFuture, GuardrailRequest, }; -use litellm_ai_gateway::integrations::custom_logger::{ +use litellm_core::integrations::custom_logger::{ CallbackTiming, CallbackValue, CustomLogger, LogFuture, ModelCallDetails, }; -use litellm_ai_gateway::integrations::types::RequestMetadata; -use litellm_ai_gateway::ocr::{OcrRequest, ocr}; -use litellm_core::error::Error; -#[cfg(feature = "trace-parity")] +use litellm_core::integrations::types::RequestMetadata; +#[cfg(feature = "observability")] use litellm_core::observability::FunctionTrace; +use litellm_core::ocr::{OcrRequest, ocr}; use serde_json::{Map, Value, json}; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; -#[cfg(feature = "trace-parity")] +#[cfg(feature = "observability")] use tracing::instrument::WithSubscriber; async fn read_http_headers(socket: &mut TcpStream) -> String { @@ -324,7 +324,7 @@ async fn ocr_lifecycle_runs_pre_during_and_success_hooks() { GuardrailEventHook::PreCall, GuardrailEventHook::DuringCall, ])); - #[cfg(feature = "trace-parity")] + #[cfg(feature = "observability")] let trace = FunctionTrace::default(); let api_base = format!("http://{addr}"); let call = ocr(OcrRequest { @@ -347,7 +347,7 @@ async fn ocr_lifecycle_runs_pre_during_and_success_hooks() { }, litellm_call_id: Some("ocr-call-1"), }); - #[cfg(feature = "trace-parity")] + #[cfg(feature = "observability")] let call = call.with_subscriber(trace.dispatcher()); let response = call.await.expect("ocr request succeeds"); @@ -367,7 +367,7 @@ async fn ocr_lifecycle_runs_pre_during_and_success_hooks() { error_kind: None, }] ); - #[cfg(feature = "trace-parity")] + #[cfg(feature = "observability")] assert_eq!( trace .events() @@ -406,7 +406,7 @@ async fn ocr_lifecycle_runs_failure_hook_on_provider_error() { }); let logger = Arc::new(RecordingOcrLogger::default()); - #[cfg(feature = "trace-parity")] + #[cfg(feature = "observability")] let trace = FunctionTrace::default(); let api_base = format!("http://{addr}"); let call = ocr(OcrRequest { @@ -426,7 +426,7 @@ async fn ocr_lifecycle_runs_failure_hook_on_provider_error() { request_metadata: RequestMetadata::default(), litellm_call_id: Some("ocr-call-2"), }); - #[cfg(feature = "trace-parity")] + #[cfg(feature = "observability")] let call = call.with_subscriber(trace.dispatcher()); let err = call.await.expect_err("provider error propagates"); @@ -443,7 +443,7 @@ async fn ocr_lifecycle_runs_failure_hook_on_provider_error() { error_kind: Some("HttpError".to_string()), }] ); - #[cfg(feature = "trace-parity")] + #[cfg(feature = "observability")] assert_eq!( trace .events() diff --git a/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs b/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs deleted file mode 100644 index fc0ab2b62a3..00000000000 --- a/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs +++ /dev/null @@ -1,112 +0,0 @@ -//! Enforcement: the litellm-rust workspace has exactly five crates. -//! -//! `core` (the Rust SDK), `config` (the config-loading boundary), -//! `ai-gateway` (the HTTP/WebSocket host), -//! `python-interop` (domain-neutral PyO3 primitives), and `python-bridge` (the -//! PyO3 cdylib). Adding or removing a crate must be a -//! deliberate act: this test fails until the allowlist here is updated, forcing -//! whoever changes the crate set to justify the new crate per the rule that a -//! crate is a layer needing independent compilation / its own deps / a separate -//! artifact — and to keep `litellm-rust/AGENTS.md` in sync. -//! -//! Std-only (no toml crate): we scan the workspace manifest's `members = [...]` -//! block and the `crates/` directory directly. - -use std::collections::BTreeSet; -use std::fs; -use std::path::{Path, PathBuf}; - -/// The one true crate set. Update BOTH this and `litellm-rust/AGENTS.md` when the -/// workspace legitimately gains or loses a crate. -const EXPECTED_MEMBERS: &[&str] = &[ - "crates/core", - "crates/config", - "crates/ai-gateway", - "crates/python-interop", - "crates/python-bridge", -]; - -/// The crate subdirectory names that must exist under `crates/`. -const EXPECTED_CRATE_DIRS: &[&str] = &[ - "core", - "config", - "ai-gateway", - "python-interop", - "python-bridge", -]; - -const MISMATCH: &str = "litellm-rust crate set changed — update this allowlist AND litellm-rust/AGENTS.md, and justify the crate per the rule (crate = layer needing independent compilation / its own deps / a separate artifact)."; - -/// Absolute path to the workspace root (`litellm-rust/`). -fn workspace_root() -> PathBuf { - // CARGO_MANIFEST_DIR is `.../litellm-rust/crates/core`; the workspace root is - // two levels up. - Path::new(concat!(env!("CARGO_MANIFEST_DIR"), "/../..")) - .canonicalize() - .expect("workspace root should resolve") -} - -/// Parse the `members = [ ... ]` array out of the workspace `[workspace]` table. -/// -/// Minimal hand-rolled scan: find `members`, then collect every double-quoted -/// string up to the closing `]`. Good enough for our fixed manifest shape and -/// keeps this test dependency-free. -fn parse_members(manifest: &str) -> BTreeSet { - let after_members = manifest - .split_once("members") - .map(|(_, rest)| rest) - .expect("workspace manifest should declare members"); - let open = after_members.find('[').expect("members should be an array"); - let close = after_members[open..] - .find(']') - .map(|offset| open + offset) - .expect("members array should be closed"); - let body = &after_members[open + 1..close]; - - let mut members = BTreeSet::new(); - let mut rest = body; - while let Some(start) = rest.find('"') { - let after_quote = &rest[start + 1..]; - let end = after_quote - .find('"') - .expect("opening quote should be matched"); - members.insert(after_quote[..end].to_string()); - rest = &after_quote[end + 1..]; - } - members -} - -/// The crate subdirectory names under `crates/`. -/// -/// A directory counts as a crate only when it holds a `Cargo.toml`; non-crate -/// directories (e.g. docs like `CODING_STANDARDS/`) are ignored so they can live -/// under `crates/` without tripping the crate-proliferation guard. -fn crate_dirs(root: &Path) -> BTreeSet { - fs::read_dir(root.join("crates")) - .expect("crates/ directory should exist") - .filter_map(Result::ok) - .filter(|entry| entry.file_type().map(|ty| ty.is_dir()).unwrap_or(false)) - .filter(|entry| entry.path().join("Cargo.toml").is_file()) - .map(|entry| entry.file_name().to_string_lossy().into_owned()) - .collect() -} - -#[test] -fn workspace_members_match_allowlist() { - let root = workspace_root(); - let manifest = fs::read_to_string(root.join("Cargo.toml")) - .expect("workspace Cargo.toml should be readable"); - - let actual = parse_members(&manifest); - let expected: BTreeSet = EXPECTED_MEMBERS.iter().map(|s| s.to_string()).collect(); - assert_eq!(actual, expected, "{MISMATCH}"); -} - -#[test] -fn crates_directory_matches_allowlist() { - let root = workspace_root(); - - let actual = crate_dirs(&root); - let expected: BTreeSet = EXPECTED_CRATE_DIRS.iter().map(|s| s.to_string()).collect(); - assert_eq!(actual, expected, "{MISMATCH}"); -} diff --git a/litellm-rust/crates/python-bridge/AGENTS.md b/litellm-rust/crates/python-bridge/AGENTS.md index 42282ca4da4..6ab7bf39bb5 100644 --- a/litellm-rust/crates/python-bridge/AGENTS.md +++ b/litellm-rust/crates/python-bridge/AGENTS.md @@ -1,3 +1,10 @@ -litellm-python-bridge is the PyO3 cdylib that exposes LiteLLM Rust APIs to the Python SDK. Keep API registration, domain dependency wiring, request assembly, and Python exception mapping here. Put domain-neutral Python/Serde conversion and GIL primitives in litellm-python-interop. +litellm-python-bridge is the PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. -Keep it thin: no business logic, no transforms, no I/O orchestration — just marshal in/out and call the core entrypoint. +- Owns API registration, domain dependency wiring, request assembly, and exception mapping +- Keep it thin: no business logic, no transforms, no I/O orchestration; just marshal in/out and call the core entrypoint +- One stable method per top-level route (`ocr`/`aocr`, `messages`/`amessages`, ...); do not add per-provider helpers +- Provider dispatch lives in `litellm-core`, never here +- Put domain-neutral Python/Serde conversion and GIL primitives in `litellm-python-interop` +- Two logical parts, kept as modules: the domain adapter (`src/routes/*`, `marshal`, `errors`) and the binding artifact (`#[pymodule]`, `#[pyfunction]`, registration in `lib.rs`) +- Data handling: do not log OCR payloads or provider responses; avoid copying large payloads; sanitize errors before they cross the boundary +- Tests: `cargo test --workspace` compiles here; Python tests cover disabled, enabled, and module-missing fallback for every exposed route diff --git a/litellm-rust/crates/python-bridge/CLAUDE.md b/litellm-rust/crates/python-bridge/CLAUDE.md index d25ae5a8130..3a9bfcd1ff0 100644 --- a/litellm-rust/crates/python-bridge/CLAUDE.md +++ b/litellm-rust/crates/python-bridge/CLAUDE.md @@ -1,43 +1 @@ -# CLAUDE.md - -Rules for `litellm-rust/crates/python-bridge`. - -## Responsibility - -`python-bridge` is the PyO3 boundary between Python LiteLLM and Rust transforms. -Keep this crate thin. It exposes LiteLLM Rust APIs, assembles domain requests, -maps domain errors to Python exceptions, and delegates generic conversion and -GIL handling to `litellm-python-interop`. - -## Bridge Shape - -- Prefer one stable method per top-level LiteLLM route, for example - `messages(...)`, calling the matching `litellm-core` entrypoint. -- Do not add one exported PyO3 function per provider helper unless there is a - measured reason. -- Provider dispatch belongs in the `litellm-core` route module (e.g. - `litellm_core::messages`), not in this PyO3 crate. -- Python owns rollout state and fallback. Rust should return errors; Python - decides whether to raise or fall back. For a rust-only provider/route (no - Python reference), the Python side is a thin dispatch that calls Rust and - raises when the bridge is unavailable, with no fallback. -- Keep the Python interface minimal (well under 100 lines per route): it only - marshals inputs and calls Rust. Do not add per-route feature flags, and do - not put provider dispatch in `litellm/main.py`; it lives in a thin dispatch - class under `litellm/llms///`. - -## Data Handling - -- OCR payloads can contain personal data and large base64 images. Do not log - payloads or provider responses. -- Avoid copying large payloads more than needed. The current JSON round-trip is - acceptable for the first scaffold, but future performance work should evaluate - direct PyO3 conversion before expanding Rust coverage to image-heavy paths. -- Do not expose raw Rust errors that include document contents or upstream - bodies. - -## Tests - -- `cargo test --workspace` must compile this crate. -- Python tests must cover bridge disabled, bridge enabled, and module-missing - fallback behavior for every exposed route. +See ./AGENTS.md diff --git a/litellm-rust/crates/python-interop/AGENTS.md b/litellm-rust/crates/python-interop/AGENTS.md index d1d61e5dfa0..07c7cffff46 100644 --- a/litellm-rust/crates/python-interop/AGENTS.md +++ b/litellm-rust/crates/python-interop/AGENTS.md @@ -1 +1,5 @@ -litellm-python-interop is the domain-neutral PyO3 foundation. Keep generic Python/Serde conversion and interpreter primitives here. Do not add LiteLLM domain crates, route types, API registration, or cdylib build features. +litellm-python-interop is the domain-neutral PyO3 foundation. + +- Owns generic Python/Serde conversion and interpreter primitives (`gil`, `marshal`) +- Depends on PyO3 but no LiteLLM domain crate; no route types, no API registration, no cdylib +- Keep it free of `litellm-core`, `OcrRequest`, `CallServices`, LiteLLM exceptions or `_native` surface