mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
remove bunch of wrong code
This commit is contained in:
Yujong Lee
parent
f250e3dded
commit
e1f009df86
49 changed files with 332 additions and 1353 deletions
|
|
@ -1,29 +0,0 @@
|
|||
# Adding a provider / route to litellm-rust
|
||||
|
||||
Everything for a route lives in `crates/core/src/<route>/`; `crates/core/src/messages` is the reference. A host (the axum gateway, the Python bridge) only calls the route's entrypoint.
|
||||
|
||||
1. **Entrypoint** — `mod.rs`: `pub async fn <route>(request) -> CoreResult<Response>`, the Rust equivalent of `litellm.<route>()`, plus a `<route>_stream` variant when the route streams. It is the only thing a host touches.
|
||||
2. **Transform contract** — `transformation.rs`: a `…ProviderConfig` trait (URL build + request/response transforms) with types in `types.rs`.
|
||||
3. **Provider config** — `crates/core/src/providers/<provider>/<route>/transformation.rs`: implement that trait as a `const <PROVIDER>_<ROUTE>_CONFIG`, mirroring the Python provider tree. Add parity unit tests.
|
||||
4. **Prepare + handler** — `prepare.rs` resolves provider/model, credentials, auth headers, and URL, then transforms the request; `handler.rs` performs the provider call through the shared client in `client.rs` and transforms the response.
|
||||
|
||||
## Coding standards
|
||||
|
||||
Before writing new logic, look for an existing base to extend. When a change is
|
||||
“the same behavior for one more provider/endpoint/integration”, the codebase
|
||||
almost always already has a shared abstraction for it (for example, provider
|
||||
`BaseConfig` transformation classes in `litellm/llms/base_llm/`, shared
|
||||
helpers in `litellm_core_utils/`, typed request/response models, or factory
|
||||
functions). Find it first with a search, then add the new variant by inheriting
|
||||
from or composing that base, overriding only what genuinely differs (model
|
||||
name, parameter mapping, or auth).
|
||||
|
||||
Never copy an existing implementation and edit it in place, and never hand-roll
|
||||
a parallel version of logic a base already provides. If you catch yourself
|
||||
writing a second copy of a pattern that exists twice already, stop and extract a
|
||||
base instead: put the shared shape in one place and make both call sites thin
|
||||
variants of it. The test for a good abstraction is that adding the next provider
|
||||
is a few declarative lines, not a new file of duplicated flow. Only diverge from
|
||||
the base when behavior is genuinely different, and say so explicitly in the PR.
|
||||
|
||||
**Calling:** hosts invoke the core entrypoint — the Python bridge and the `ai-gateway` route service both call `litellm_core::messages::messages`. Never add a provider handler to `ai-gateway`. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md).
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
# AGENTS.md
|
||||
|
||||
litellm-rust has five crates. A crate is a layer or shared foundation, not a route. Routes (ocr, realtime, chat) and providers (mistral, openai) are modules inside the layers.
|
||||
|
||||
## Crates
|
||||
|
||||
| Crate | Role |
|
||||
|-------|------|
|
||||
| litellm-core | The LiteLLM SDK in Rust. One public entrypoint per top-level call (`messages::messages()`), owning types, transforms, provider resolution, auth, and the provider HTTP call. Call it, get a typed response. |
|
||||
| litellm-config | Config-loading boundary. Returns resolved core deployment data and optionally delegates loading to Python. |
|
||||
| litellm-ai-gateway | The axum server (behind the `server` feature) plus the WebSocket hosts. Translates HTTP/WS to core entrypoints; owns no provider logic and no handlers. |
|
||||
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
|
||||
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
|
||||
|
||||
Dependency direction is acyclic: `litellm-config` depends on `litellm-core`, the gateway depends on both, and `litellm-python-bridge` depends on the domain layers and `litellm-python-interop`. The interop foundation depends on no LiteLLM domain crate.
|
||||
|
||||
## Where a route lives
|
||||
|
||||
A top-level LiteLLM call is a module under `crates/core/src/<route>/`, shaped like `messages`:
|
||||
|
||||
```
|
||||
core/src/messages/
|
||||
mod.rs # pub async fn messages(..) -> CoreResult<..> (+ messages_stream for SSE)
|
||||
types.rs # request/response types, MessagesRequest
|
||||
transformation.rs # the provider template trait
|
||||
prepare.rs # provider resolution, auth headers, URL
|
||||
handler.rs # the provider call
|
||||
client.rs # the shared reqwest client
|
||||
```
|
||||
|
||||
Handlers never live in `ai-gateway`. `ocr`, `audio_transcription`, and `realtime` are still hosted there from before this rule; they move to `core` as they are touched.
|
||||
|
||||
Adding a crate: default to a module. A new crate requires a real trigger: separate artifact (binary/cdylib), proc-macro, shared foundation, or publishable standalone. A new provider or route is none of these.
|
||||
|
||||
Adding a crate fails crates/core/tests/workspace_crate_allowlist.rs until you update its allowlist and this file — intentional.
|
||||
|
||||
## Style
|
||||
|
||||
All Rust in `litellm-rust/` follows the official Rust Style Guide:
|
||||
https://doc.rust-lang.org/style-guide/
|
||||
|
||||
`rustfmt` implements its formatting by default, so run `cargo fmt` before committing; CI gates every PR on `cargo fmt --check`. Do not hand-format against rustfmt or add a `rustfmt.toml` that diverges from the default style.
|
||||
|
||||
Beyond formatting, follow the guide's naming and idiom conventions rustfmt cannot auto-apply: `snake_case` items/functions/modules, `UpperCamelCase` types/traits/variants, `SCREAMING_SNAKE_CASE` constants/statics (acronyms as one word, e.g. `HttpClient`), and the import grouping and item ordering it prescribes. See CLAUDE.md for the detailed version.
|
||||
|
|
@ -1,197 +0,0 @@
|
|||
# CLAUDE.md
|
||||
|
||||
This file defines the rules for Rust work in LiteLLM.
|
||||
|
||||
## Provider Coding Standards
|
||||
|
||||
Before writing new logic, look for an existing base to extend. When a change is
|
||||
“the same behavior for one more provider/endpoint/integration”, the codebase
|
||||
almost always already has a shared abstraction for it (for example, provider
|
||||
`BaseConfig` transformation classes in `litellm/llms/base_llm/`, shared
|
||||
helpers in `litellm_core_utils/`, typed request/response models, or factory
|
||||
functions). Find it first with a search, then add the new variant by inheriting
|
||||
from or composing that base, overriding only what genuinely differs (model
|
||||
name, parameter mapping, or auth).
|
||||
|
||||
Never copy an existing implementation and edit it in place, and never hand-roll
|
||||
a parallel version of logic a base already provides. If you catch yourself
|
||||
writing a second copy of a pattern that exists twice already, stop and extract a
|
||||
base instead: put the shared shape in one place and make both call sites thin
|
||||
variants of it. The test for a good abstraction is that adding the next provider
|
||||
is a few declarative lines, not a new file of duplicated flow. Only diverge from
|
||||
the base when behavior is genuinely different, and say so explicitly in the PR.
|
||||
|
||||
## Crates (see AGENTS.md)
|
||||
|
||||
`litellm-core` **is** the LiteLLM SDK in Rust: it makes the LLM call.
|
||||
`litellm-config` is the config-loading boundary and returns resolved core types.
|
||||
`litellm-ai-gateway` is an HTTP/WebSocket server in front of it, and
|
||||
`litellm-python-bridge` exposes it to the Python SDK. `litellm-python-interop`
|
||||
holds domain-neutral PyO3 primitives shared by Python-facing Rust code. A crate
|
||||
is a layer or shared foundation, not a route; add modules, not crates.
|
||||
|
||||
## Core Boundary
|
||||
|
||||
`litellm-core` owns the whole call. The Rust equivalent of `litellm.messages()`
|
||||
is `litellm_core::messages::messages(request).await`: you call it, it does the
|
||||
provider call, and you get a typed non-streaming response back.
|
||||
|
||||
Route-level Rust structure mirrors LiteLLM's Python responsibilities:
|
||||
- `core/src/<route>/` owns the route end to end: the public entrypoint fn named
|
||||
after the route in `mod.rs`, the request/response types (`types.rs`), the
|
||||
provider template trait (`transformation.rs`), the provider/auth/URL
|
||||
resolution (`prepare.rs`), the HTTP client (`client.rs`), and the handler that
|
||||
performs the call (`handler.rs`). `core/src/messages` is the reference.
|
||||
- `core/src/providers/<provider>/<route>/transformation.rs` owns the
|
||||
provider-specific transform. For Anthropic Messages, this means
|
||||
`core/src/providers/anthropic/messages/transformation.rs`.
|
||||
- Handlers live in `core`, never in a host. `ai-gateway` must not contain a
|
||||
route handler that talks to a provider; its axum route reads the HTTP request,
|
||||
picks a deployment, and calls the `core` entrypoint. `python-bridge` marshals
|
||||
Python objects and calls the same entrypoint.
|
||||
|
||||
Streaming keeps the same shape: the route entrypoint has a `<route>_stream`
|
||||
variant in `core` that returns the upstream response so a host can splice it to
|
||||
its own caller; the host still owns no provider logic.
|
||||
|
||||
Call-hook and lifecycle instrumentation, including phase timing, usage
|
||||
accumulation, and callback payload construction, always lives in `core`.
|
||||
Hosts feed observed events into core and dispatch the completed payloads through
|
||||
their I/O logger; hosts must not own callback orchestration.
|
||||
|
||||
Allowed in `core`:
|
||||
- The public entrypoint for a top-level LiteLLM call
|
||||
- Request/response transforms and stream chunk normalization
|
||||
- Provider resolution, auth header construction, and URL building
|
||||
- The provider HTTP call itself, through a shared reused client with connect and
|
||||
request timeouts
|
||||
- Shared data types and validation errors
|
||||
- Deterministic token/cost helper logic
|
||||
|
||||
Not allowed in `core`:
|
||||
- Serving HTTP: axum routes, extractors, and transport concerns stay in the host
|
||||
- Filesystem access
|
||||
- Database access
|
||||
- Config file reading and rollout state
|
||||
- Logging callbacks, spend writes, or custom callbacks
|
||||
- Global mutable runtime state
|
||||
|
||||
Env reads in `core` are limited to credential fallback inside a route's
|
||||
`prepare.rs` (the `env_lookup` closure), mirroring what the Python SDK does when
|
||||
no key is passed. Everything else config-shaped is resolved by the host and
|
||||
passed in.
|
||||
|
||||
Routes still hosted in `ai-gateway` (`ocr`, `audio_transcription`, `realtime`)
|
||||
predate this rule and are being moved into `core` route modules; do not add new
|
||||
ones there, and prefer moving one when you touch it.
|
||||
|
||||
Python owns rollout state and fallback while Rust is being introduced. Rust
|
||||
paths must be off by default until parity tests prove equivalence with Python.
|
||||
A new provider/route may instead be implemented rust-only with no Python
|
||||
reference; then the Python interface is a thin dispatch that calls Rust with no
|
||||
fallback, and you state the rust-only choice explicitly in the PR. Either way
|
||||
the Python side stays minimal (it only marshals inputs and calls the Rust
|
||||
interface), never add a per-route feature flag, and never push provider
|
||||
dispatch into `litellm/main.py`; put it in a thin dispatch class under
|
||||
`litellm/llms/<provider>/<route>/`.
|
||||
|
||||
## Production Bar
|
||||
|
||||
Rust code in this workspace is held to a strict parity and robustness bar from
|
||||
the first PR:
|
||||
|
||||
- Correctness parity is proven with tests. Do not rely on README claims or
|
||||
manual inspection for a port that mirrors Python behavior.
|
||||
- Every provider transform must have unit tests for supported-parameter
|
||||
filtering, request body shape, response normalization, missing/null fields,
|
||||
and bad-input errors.
|
||||
- When Rust is exposed through Python, add Python tests that prove disabled,
|
||||
enabled, and unavailable-bridge fallback behavior.
|
||||
- Avoid panics on user/provider input. Return typed errors and let the host map
|
||||
them to Python exceptions or HTTP responses.
|
||||
- OCR handles documents that often contain personal data. Do not log document
|
||||
contents, base64 payloads, provider response bodies, or secrets.
|
||||
- Error messages must be useful but data-minimized. Truncate or sanitize any
|
||||
upstream body before it crosses a host boundary.
|
||||
- Treat empty or whitespace-only credentials, URLs, and config values as absent
|
||||
at the host/config resolution layer.
|
||||
- Preserve Python output shape intentionally. If a field is always serialized as
|
||||
`null` for Python parity, leave a short comment explaining that parity choice.
|
||||
|
||||
## Network I/O Rules
|
||||
|
||||
These rules apply to every module that executes network I/O, whether it is a
|
||||
`core` route handler or a host such as `ai-gateway`:
|
||||
|
||||
- Set connect and full-request timeouts. No unbounded waits.
|
||||
- Reuse HTTP clients; do not construct clients per request.
|
||||
- Prefer rustls TLS for portable Python wheels and Linux images unless there is
|
||||
a documented reason not to.
|
||||
- Add request IDs and structured tracing at the host layer, without logging OCR
|
||||
document contents or secrets.
|
||||
- Do not echo raw upstream response bodies to callers. Sanitize and bound them.
|
||||
- Avoid `expect`/`unwrap` in server startup and request paths unless the panic is
|
||||
impossible by construction and documented.
|
||||
|
||||
## Rust Style Guide
|
||||
|
||||
All Rust in `litellm-rust/` follows the official Rust Style Guide:
|
||||
https://doc.rust-lang.org/style-guide/
|
||||
|
||||
`rustfmt` implements the guide's formatting rules by default, so the mechanical
|
||||
side is enforced for you: run `cargo fmt` before committing and CI gates every
|
||||
PR on `cargo fmt --check` (see Checks). Do not hand-format against rustfmt or add
|
||||
a `rustfmt.toml` that diverges from the default style; the default style *is* the
|
||||
guide.
|
||||
|
||||
The guide also covers conventions rustfmt cannot auto-apply; follow these too:
|
||||
- Naming: `snake_case` for items, functions, and modules; `UpperCamelCase` for
|
||||
types, traits, and enum variants; `SCREAMING_SNAKE_CASE` for constants and
|
||||
statics; acronyms count as one word (`HttpClient`, not `HTTPClient`).
|
||||
- Ordering and grouping the guide prescribes: imports grouped std / external /
|
||||
crate-local, derives before other attributes, and consistent item order.
|
||||
- Idioms the guide recommends over the formatter fighting you (e.g. prefer
|
||||
restructuring an over-long expression rather than forcing an awkward wrap).
|
||||
|
||||
## Constants
|
||||
|
||||
Magic numbers and fixed strings go in a crate-level `constants.rs`, never
|
||||
hardcoded inline — the Rust mirror of Python's `litellm/constants.py`.
|
||||
|
||||
- Each crate that needs them has `src/constants.rs` (declared `mod constants;`);
|
||||
import from it (`use crate::constants::...`). Don't scatter `const` values at
|
||||
the top of feature modules.
|
||||
- An env-overridable tunable still lives in `constants.rs` as its `DEFAULT_*`
|
||||
value; the env read (with fallback to that default) happens at the host/config
|
||||
resolution layer, not in `core`/`providers`.
|
||||
- Exception: a value that is purely local to one function and has no meaning
|
||||
elsewhere may stay inline, but prefer `constants.rs` when in doubt.
|
||||
|
||||
## Checks
|
||||
|
||||
Run these before pushing Rust changes. The same checks run in GitHub Actions
|
||||
for changes under `litellm-rust/`.
|
||||
|
||||
```bash
|
||||
cd litellm-rust
|
||||
cargo fmt --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
cargo clippy -p litellm-core --all-targets --features bedrock-auth -- -D warnings
|
||||
# the ai-gateway binary + server code is behind the `server` feature
|
||||
cargo clippy -p litellm-ai-gateway --all-targets --all-features -- -D warnings
|
||||
cargo test --workspace
|
||||
cargo test -p litellm-core --features bedrock-auth
|
||||
# the `auth`, `routes`, `state` and `realtime` tests only exist under `server`
|
||||
cargo test -p litellm-ai-gateway --features server
|
||||
```
|
||||
|
||||
When a Rust path is exposed through Python, add Python parity tests that compare
|
||||
the existing Python output with the Rust-backed output.
|
||||
|
||||
For Python-integrated tests that need the repository's Python environment, run
|
||||
from the repository root:
|
||||
|
||||
```bash
|
||||
make test-rust-python
|
||||
make lint-rust-python-fixtures
|
||||
```
|
||||
|
|
@ -1,186 +0,0 @@
|
|||
# LiteLLM Rust
|
||||
|
||||
This workspace contains the staged Rust implementation for LiteLLM.
|
||||
|
||||
`litellm-core` is the LiteLLM SDK in Rust: one entrypoint per top-level call
|
||||
that makes the LLM call and hands back a typed response, the same shape as
|
||||
`litellm.messages()` in Python.
|
||||
|
||||
```rust
|
||||
let response = litellm_core::messages::messages(MessagesRequest {
|
||||
model: "claude-sonnet-4-5",
|
||||
body,
|
||||
api_key: Some(key),
|
||||
..
|
||||
})
|
||||
.await?;
|
||||
```
|
||||
|
||||
Python continues to own configuration, retries, routing policy, logging,
|
||||
callbacks, spend tracking, and customer plugins until each Rust path has parity
|
||||
coverage and production evidence.
|
||||
|
||||
## Crates
|
||||
|
||||
| Crate | Role |
|
||||
|-------|------|
|
||||
| litellm-core | The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router. |
|
||||
| litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. |
|
||||
| litellm-ai-gateway | The axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
|
||||
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
|
||||
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
|
||||
|
||||
Dependency direction is acyclic: config depends on core, the gateway depends on config and core, and the Python bridge depends on the domain layers and Python interop.
|
||||
|
||||
## Layout
|
||||
|
||||
```text
|
||||
crates/
|
||||
core/ The SDK: route modules + provider transforms.
|
||||
src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client
|
||||
src/providers/anthropic/messages/transformation.rs
|
||||
config/ Config loading and resolved deployments.
|
||||
ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints.
|
||||
python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
|
||||
python-bridge/ PyO3 API adapter for Python LiteLLM.
|
||||
```
|
||||
|
||||
The folder shape follows the Python provider tree:
|
||||
`core/src/providers/<provider>/<route>/transformation.rs`. The bridge exposes one
|
||||
function per top-level route, mirroring the core entrypoints.
|
||||
|
||||
## Checks
|
||||
|
||||
### Private Native OCR Proof
|
||||
|
||||
Public `litellm.ocr` and `litellm.aocr` always use the existing Python lifecycle,
|
||||
including when `litellm.rust(True)` or `LITELLM_RUST=1` enables other Rust paths.
|
||||
Native OCR remains a private proof until full lifecycle parity is established.
|
||||
Only tests requesting the private `native_ocr` fixture replace those public
|
||||
functions with test-only route selection: Rust enabled calls the native bridge,
|
||||
and Rust disabled calls the captured production Python functions
|
||||
|
||||
The bridge retains the complete call argument dictionary as a Python object,
|
||||
including opaque callback and metadata objects. It creates callback-visible
|
||||
request dictionaries with shared parameter references, then reads the execution
|
||||
roots after pre-call dispatch. Mistral retains the original document; Azure and
|
||||
Vertex Mistral use a shallow document copy, and Vertex DeepSeek projects it into
|
||||
chat messages using the existing Rust transform. Rust performs provider preparation,
|
||||
encoding, HTTP and response normalization. Python continues to dispatch existing
|
||||
logging operations and construct the public response object
|
||||
|
||||
This is a private implementation scaffold, not full OCR parity. Azure Mistral and
|
||||
Vertex Mistral accept inline data URIs with supplied keys/tokens, native environment
|
||||
keys or auth headers. Azure also accepts a supplied `azure_ad_token`. Vertex
|
||||
DeepSeek uses its existing chat request and OCR response transforms. Cloud
|
||||
credential acquisition fails explicitly only when no native credential is available.
|
||||
HTTP document URL conversion fails only for configs requiring data URIs. Azure
|
||||
Document Intelligence selects its own config but fails at the polling capability
|
||||
check before sending a billable analyze request. Cohere transforms, file inputs,
|
||||
streaming, native response format and compression remain unsupported.
|
||||
Direct private bridge calls with a missing native extension also fail;
|
||||
neither case falls back to Python execution within the private route. Transport
|
||||
failures currently use a generic error rather than the SDK's timeout-specific exception
|
||||
|
||||
Run the commands under "Checks" in [CLAUDE.md](CLAUDE.md) before pushing Rust
|
||||
changes. That list is the single source of truth and matches what GitHub Actions
|
||||
runs for changes under `litellm-rust/`.
|
||||
|
||||
### Python-Integrated Tests
|
||||
|
||||
From the repository root, run the ignored Cargo tests that need the repository's
|
||||
Python dependencies and the pinned Ruff checks over the interop crate's Python
|
||||
test fixtures:
|
||||
|
||||
```bash
|
||||
make test-rust-python
|
||||
make lint-rust-python-fixtures
|
||||
```
|
||||
|
||||
`test-rust-python` installs the locked SDK dependencies plus the `proxy` extra
|
||||
(the integration fixtures import `litellm.proxy.*` guardrails, which need
|
||||
`fastapi`) with uv, points `PYO3_PYTHON` at the project interpreter, and runs
|
||||
`cargo test -p litellm-python-interop --tests --locked -- --include-ignored`.
|
||||
`lint-rust-python-fixtures` runs pinned Ruff lint and formatting checks without
|
||||
syncing the project environment
|
||||
|
||||
Run the private native OCR proof gate from the repository root:
|
||||
|
||||
```bash
|
||||
make test-rust-ocr
|
||||
```
|
||||
|
||||
This builds the current release wheel, installs locked SDK dependencies, the
|
||||
`dev` test group, and the `proxy` extra in a temporary Python 3.12 environment,
|
||||
then installs the wheel without resolving dependencies again. The proxy extra
|
||||
is needed by the shared pytest fixtures. Python isolated mode and pytest's
|
||||
importlib mode keep the checkout from shadowing the installed wheel
|
||||
|
||||
The gate checks that native `ocr` and `aocr` are importable, then runs
|
||||
`tests/test_litellm/ocr/test_rust_bridge.py` with
|
||||
`LITELLM_REQUIRE_NATIVE_OCR=1`, so unavailable native OCR fails instead of
|
||||
skipping. CI uses `make test-rust-ocr RUST_OCR_WHEEL=/absolute/path/to/current.whl`
|
||||
to test the release wheel it just built. The stdlib-only
|
||||
`native_route_wheel_test.py` also exercises sync/async OCR through the retained
|
||||
argument dictionary, including native request preparation and public 429 error
|
||||
mapping, alongside the other native routes
|
||||
|
||||
The Python-integrated Cargo tests validate retained callback identity, mutation,
|
||||
invocation context and ownership against Python behavior, including existing
|
||||
LiteLLM components. Short synthetic pre-call contracts use Rust-owned table-driven
|
||||
cases with inline Python callbacks; larger component scenarios share Python
|
||||
fixtures. These generic proofs complement, rather than replace, native OCR
|
||||
private proof tests
|
||||
|
||||
The standard-library-only tests in
|
||||
`crates/python-interop/tests/synthetic/patterns.rs` define small inline Python
|
||||
callbacks, with Rust controlling invocation, ownership and assertions. They
|
||||
compare Python-reference and Rust-retained calls using both direct and awaited
|
||||
invocation. They model the behavior
|
||||
groups in the callback use-case inventory: live versus serialized queues,
|
||||
mutation before an error, ignored returns, identity-based redaction, block-state
|
||||
stash, background writes after return, parallel live data versus snapshots, and
|
||||
shallow/deep copies with uncopyable-value fallback. Copy controls deliberately
|
||||
produce different observations; event gates establish ordering without sleeps.
|
||||
Existing synthetic lifecycle cases also cover streams, context and cancellation
|
||||
|
||||
Run this matrix without LiteLLM, vendor SDKs, credentials or services:
|
||||
|
||||
```bash
|
||||
cargo test --manifest-path litellm-rust/Cargo.toml -p litellm-python-interop --test synthetic
|
||||
```
|
||||
|
||||
These are behavioral models, not tests of vendor authentication, delivery or
|
||||
production dispatcher policy. The optional component and integration fixtures
|
||||
exercise existing LiteLLM implementations with fake transports and credentials
|
||||
as supplementary coverage; run them with `make test-rust-python`
|
||||
|
||||
The interop tests have two explicit Cargo targets, each rooted in its directory's
|
||||
`mod.rs`: `tests/synthetic/` for custom, minimal Python implementations and
|
||||
`tests/integration/` for real LiteLLM components with fake transports. The latter
|
||||
are component-level compatibility tests, not complete SDK or proxy route tests.
|
||||
Shared Rust fixtures live in `tests/support/`, and Python scenarios live in
|
||||
`tests/fixtures/`
|
||||
|
||||
Use `#[fixture]` composition for setup: initialize Python once, but create a fresh
|
||||
scenario scope and owner factory for each case. Use named `#[case::behavior]`
|
||||
entries for scenarios and `#[values(Backend::Python, Backend::PreparedCall)]` for
|
||||
the invocation matrix. Keep copying and ownership controls alongside the behavior
|
||||
they distinguish. Register new Rust modules in the appropriate `mod.rs`; Cargo
|
||||
test autodiscovery is disabled so new files cannot silently become a third group
|
||||
|
||||
Run `--test synthetic` for the standard-library-only group. Integration cases
|
||||
remain explicitly ignored without the repository Python environment;
|
||||
`make test-rust-python` configures that environment and runs both groups with
|
||||
`--include-ignored`. To inspect the groups without running them, use
|
||||
`cargo test --manifest-path litellm-rust/Cargo.toml -p litellm-python-interop --tests -- --list`
|
||||
|
||||
The callback lifecycle scenarios use
|
||||
`#[serial(python_interpreter)]` to isolate CPython GC and interpreter-wide
|
||||
LiteLLM settings under `cargo test`. Compatible tests in the same binary use
|
||||
`#[parallel(python_interpreter)]`: they may overlap each other, but not an
|
||||
exclusive scenario. Unannotated tests do not participate in this isolation.
|
||||
Keep the attribute below `#[rstest]` so generated cases acquire it before
|
||||
fixture setup and Python attachment. Tasks and threads inside each scenario
|
||||
still run concurrently. Separate test processes have separate interpreters,
|
||||
so these attributes need no cross-process lock when using nextest
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
# Provider coding standards (litellm-rust)
|
||||
|
||||
Rules for adding or changing an LLM provider/route in `litellm-rust`. `messages` (`core/src/messages`, `ANTHROPIC_MESSAGES_CONFIG`) is the reference: a route is a `core` module with a public entrypoint that makes the call and returns a typed response.
|
||||
|
||||
## Provider resolution
|
||||
|
||||
1. Always resolve the provider/model first with `get_custom_llm_provider` (`core/src/routing_utils/provider.rs`). Nothing downstream may branch on a raw model string.
|
||||
2. Model/provider is resolved once, in `prepare.rs`, and passed down as typed fields. Don't re-resolve or re-parse it in transforms or handlers.
|
||||
|
||||
## Transforms and the base config
|
||||
|
||||
3. Every route defines a base config trait with `transform_request` + `transform_response` (+ `complete_url`, `supported_params`), living in `core/src/<route>/transformation.rs` (e.g. `AnthropicMessagesProviderConfig`, mirroring `OcrProviderConfig`).
|
||||
4. Each provider implements that trait as a `const <PROVIDER>_<ROUTE>_CONFIG` in `core/src/providers/<provider>/<route>/transformation.rs`, mirroring the Python provider tree.
|
||||
5. Individual configs implement only the request/response transforms. Shared behavior (param filtering, defaults) stays as trait default methods so future providers inherit existing logic instead of reimplementing it.
|
||||
6. Prefer composition: a provider that extends another reuses the base trait's defaults or wraps another config; don't copy transform bodies between providers.
|
||||
|
||||
## Boundaries
|
||||
|
||||
7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `ai-gateway` = serving HTTP/WS (routing, extractors, auth of *our* callers, streaming to the client); `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint; they never build a provider request.
|
||||
8. Generic/route files contain zero provider-specific branches. A provider is one module under `core/src/providers/<provider>/<route>/`; a route is a module, never a new crate.
|
||||
9. Route entry point stays thin: `core::<route>::<route>()` -> `prepare_*` -> handler (or `CallLifecycle::run_request`, which owns the pre_call -> during_call -> provider call -> success/failure order and phase timing). Axum handlers validate and delegate to a service that calls the entrypoint; no business logic in them.
|
||||
10. Constants (URLs, env-var names, API versions, error messages) live in a crate `constants.rs`, never inline. Config-shaped env reads happen at the host/config layer with the `DEFAULT_*` fallback defined in `constants.rs`; the only env read in `core` is the credential fallback in a route's `prepare.rs`.
|
||||
|
||||
## Types and errors
|
||||
|
||||
11. Typed contracts only: no bare `serde_json::Value` / `String` / `Vec<String>` as a transform input or output. Parse wire bytes into typed structs/enums at the host edge; a `type` discriminator is a typed field, not a raw string.
|
||||
12. Model failures as values: return typed `CoreError`, don't panic. No `unwrap`/`expect`/`panic!` on user or provider input.
|
||||
13. No mutation: build values in one shot (comprehensions/iterators, `collect`), prefer immutable bindings and owned typed structs over seeding-and-mutating.
|
||||
14. Early returns over deep nesting; small focused files over god modules.
|
||||
15. Preserve Python output shape intentionally. If a field is always serialized as `null` for parity, keep it and pin it with a test.
|
||||
|
||||
## Safety and data minimization
|
||||
|
||||
16. Never log request/response bodies, base64 payloads, document contents, or secrets. Truncate and bound any upstream body before it crosses a host boundary.
|
||||
17. Treat empty/whitespace credentials, URLs, and config values as absent at the host resolution layer.
|
||||
18. Network I/O sets connect + request timeouts (no unbounded waits), reuses a shared HTTP client, and prefers rustls TLS.
|
||||
|
||||
## Tests and rollout
|
||||
|
||||
19. Every provider transform ships tests for: supported-param filtering, request body shape, response normalization, missing/null fields, bad input, and `*_match_python` fixture parity.
|
||||
20. Lifecycle/hook tests cover hook order, success + failure callback payloads, pre-call guardrail blocking before any provider I/O, during-call body mutation, and provider-error mapping.
|
||||
21. When a route has a Python reference implementation, the Rust path stays off by default and behind Python parity tests (disabled / enabled-equals-Python / bridge-unavailable fallback) until parity is proven. A new provider/route may instead be implemented rust-only with no Python reference; then the Python interface is a thin dispatch to Rust with no fallback, and tests cover the rust-backed path plus the unavailable-bridge error. State the rust-only choice explicitly in the PR.
|
||||
|
||||
## Python bridge (SDK side)
|
||||
|
||||
22. A Python -> Rust bridge keeps the Python side minimal: the Python interface only marshals inputs and calls the Rust interface, with no transform, handler, or business logic. Aim for well under 100 lines of interface code per route; if the Python grows past that, the logic belongs in Rust.
|
||||
23. Do not bloat `litellm/main.py`. A route's provider dispatch lives in a thin dispatch class under `litellm/llms/<provider>/<route>/` that calls the Rust bridge; `main.py` only instantiates it and calls its sync/async method.
|
||||
24. Do not add new feature flags unless explicitly requested. Reuse the existing LiteLLM Rust rollout mechanism (`litellm.rust`); never introduce a per-route env flag such as `LITELLM_USE_RUST_<ROUTE>`.
|
||||
|
||||
## Checks before push
|
||||
|
||||
25. Run, and keep green, the commands under "Checks" in `litellm-rust/CLAUDE.md`.
|
||||
That list is the single source of truth and matches what GitHub Actions runs.
|
||||
|
|
@ -1,54 +1,20 @@
|
|||
# ai-gateway — folder architecture
|
||||
# ai-gateway
|
||||
|
||||
The Axum server that fronts the Rust gateway. It owns transport + config + auth
|
||||
only; deployment selection lives in `core::router`, and the LLM call itself
|
||||
(transforms, auth headers, provider HTTP) lives behind a `core` route entrypoint
|
||||
such as `litellm_core::messages::messages`. No provider handler lives here.
|
||||
The Axum server fronting the Rust gateway. Owns transport + config + auth only; deployment selection is `core::router`, and the LLM call (transforms, auth headers, provider HTTP) is a `core` route entrypoint. No provider handler lives here.
|
||||
|
||||
```
|
||||
src/
|
||||
main.rs # entrypoint: build AppState (router + master key), bind, serve
|
||||
state.rs # AppState — shared Arc<Router> + master_key
|
||||
auth/ # authentication as an axum extractor — added to handler args
|
||||
mod.rs # RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY)
|
||||
routes/ # one module per route, all matching the same template
|
||||
AGENTS.md # ← the route template (read this before adding a route)
|
||||
mod.rs # app(): merges every module's router()
|
||||
health.rs # simple route (one file): router() + liveness/readiness
|
||||
realtime/ # route with logic → axum surface + a no-axum service:
|
||||
mod.rs # router() + handler + WS<->events adapter (the axum surface)
|
||||
service.rs # business logic (select deployment, call provider) — no axum, testable
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **Routes follow one template.** Each route module exposes
|
||||
`pub fn router() -> Router<AppState>`; `routes/mod.rs` only merges them. Simple
|
||||
routes are one file; non-trivial routes are a folder (`handler`/`service`/
|
||||
`transport`). See `routes/AGENTS.md`.
|
||||
- **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's
|
||||
args; it runs during extraction. Never re-implement the check per route.
|
||||
- **Handlers are thin.** A handler validates and delegates to its `service`. No
|
||||
business logic, no provider calls, no transforms in handlers.
|
||||
- **Services call `core`, they don't reimplement it.** A `service` picks the
|
||||
deployment and calls the `core` route entrypoint. Provider resolution, auth
|
||||
headers, URL building, and the HTTP call are `core`'s job; a service that
|
||||
builds a provider request itself is a bug (`routes/messages/service.rs` is
|
||||
the reference).
|
||||
- **State is shared and cheap to clone.** Long-lived handles live behind `Arc` in
|
||||
`state.rs`; read env/config only in `main.rs` when building state.
|
||||
- Routes follow one template: each module exposes `pub fn router() -> Router<AppState>`; `routes/mod.rs` only merges them. Simple routes are one file, non-trivial routes a folder (`handler`/`service`/`transport`). See `routes/AGENTS.md`
|
||||
- Auth is an extractor: add `crate::auth::RequireMasterKey` to handler args; never re-implement the check per route
|
||||
- Handlers are thin: validate and delegate to `service`; no business logic, no provider calls, no transforms
|
||||
- Services call `core`, they never reimplement it: pick the deployment, call the entrypoint. Provider resolution, auth headers, URL, and the HTTP call are `core`'s job
|
||||
- State is shared and cheap to clone: long-lived handles behind `Arc` in `state.rs`; read env/config only in `main.rs`
|
||||
|
||||
## Auth (interim)
|
||||
|
||||
A single **master key** (`LITELLM_MASTER_KEY`), enforced by the
|
||||
`auth::RequireMasterKey` extractor: any caller presenting it as
|
||||
`Authorization: Bearer <key>` may invoke the gateway. Fails closed (500) when
|
||||
unset; constant-time compare. The server binds `127.0.0.1` by default (`HOST` to
|
||||
override). Full per-key auth + budgets/rate-limits are delegated to the Python
|
||||
proxy in a later phase. Health routes don't add the extractor (unauthenticated).
|
||||
- Single master key (`LITELLM_MASTER_KEY`) enforced by `auth::RequireMasterKey`; `Authorization: Bearer <key>`
|
||||
- Fails closed (500) when unset; constant-time compare; binds `127.0.0.1` by default (`HOST` to override)
|
||||
- Per-key auth, budgets, rate limits delegated to the Python proxy later; health routes unauthenticated
|
||||
|
||||
## Python interop
|
||||
|
||||
Python-backed loading lives in `litellm-config` and is **load-time only**. The
|
||||
gateway's `python-config` feature forwards to that crate. The realtime data path
|
||||
never takes the GIL.
|
||||
- Python-backed loading lives in `litellm-config`, load-time only; `python-config` feature forwards to it
|
||||
- The realtime data path never takes the GIL
|
||||
|
|
|
|||
|
|
@ -9,13 +9,13 @@ use std::future::Future;
|
|||
use std::pin::Pin;
|
||||
|
||||
use super::types::PreparedAudioTranscriptionRequest;
|
||||
use crate::integrations::custom_guardrail::{
|
||||
use litellm_core::integrations::custom_guardrail::{
|
||||
CustomGuardrailRunner, GuardrailContext, GuardrailError, GuardrailRequest,
|
||||
};
|
||||
use crate::integrations::custom_logger::{
|
||||
use litellm_core::integrations::custom_logger::{
|
||||
CallType, CallbackTiming, CallbackValue, CustomLoggerRunner, LoggingError, ModelCallDetails,
|
||||
};
|
||||
use crate::integrations::types::{
|
||||
use litellm_core::integrations::types::{
|
||||
RequestMetadata, StandardLoggingMetadata, StandardLoggingPayload,
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -5,8 +5,8 @@ use litellm_core::routing_utils::provider::{CustomLlmProvider, get_custom_llm_pr
|
|||
|
||||
use super::hooks::AudioTranscriptionLifecycleHooks;
|
||||
use super::types::{AudioTranscriptionRequest, PreparedAudioTranscriptionRequest};
|
||||
use crate::integrations::custom_guardrail::CustomGuardrailRunner;
|
||||
use crate::integrations::custom_logger::CustomLoggerRunner;
|
||||
use litellm_core::integrations::custom_guardrail::CustomGuardrailRunner;
|
||||
use litellm_core::integrations::custom_logger::CustomLoggerRunner;
|
||||
|
||||
pub(crate) struct PreparedAudioTranscriptionCall {
|
||||
pub(crate) request: PreparedAudioTranscriptionRequest,
|
||||
|
|
|
|||
|
|
@ -4,9 +4,9 @@ use std::time::Duration;
|
|||
use litellm_core::call_lifecycle::{CallLifecycleContext, CallLifecycleRequest};
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use crate::integrations::custom_guardrail::CustomGuardrail;
|
||||
use crate::integrations::custom_logger::CustomLogger;
|
||||
use crate::integrations::types::RequestMetadata;
|
||||
use litellm_core::integrations::custom_guardrail::CustomGuardrail;
|
||||
use litellm_core::integrations::custom_logger::CustomLogger;
|
||||
use litellm_core::integrations::types::RequestMetadata;
|
||||
|
||||
pub struct AudioTranscriptionRequest<'a> {
|
||||
pub model: &'a str,
|
||||
|
|
|
|||
|
|
@ -1,14 +0,0 @@
|
|||
use std::sync::OnceLock;
|
||||
use std::time::Duration;
|
||||
|
||||
const HTTP_CLIENT_TIMEOUT_SECS: u64 = 600;
|
||||
|
||||
pub(crate) fn http_client() -> &'static reqwest::Client {
|
||||
static CLIENT: OnceLock<reqwest::Client> = OnceLock::new();
|
||||
CLIENT.get_or_init(|| {
|
||||
reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(HTTP_CLIENT_TIMEOUT_SECS))
|
||||
.build()
|
||||
.expect("failed to build reqwest client")
|
||||
})
|
||||
}
|
||||
|
|
@ -5,26 +5,6 @@
|
|||
//! modules. Env-overridable tunables keep their `DEFAULT_*` value here; the env
|
||||
//! read + fallback happens at the host/config layer.
|
||||
|
||||
/// Default LiteLLM control-plane base URL for request-log egress when
|
||||
/// `LITELLM_PROXY_BASE_URL` is unset.
|
||||
pub(crate) const DEFAULT_PROXY_BASE_URL: &str = "http://localhost:4000";
|
||||
|
||||
/// The logs ingest path appended to the proxy base. Not a tunable; it is the
|
||||
/// proxy's API contract (the rust-control-plane router on the Python proxy).
|
||||
pub(crate) const RUST_CONTROL_PLANE_LOGS_PATH: &str = "/v1/rust_control_plane/logs";
|
||||
|
||||
/// Default bounded channel depth for the log-egress worker.
|
||||
/// Override: `LITELLM_LOG_CHANNEL_CAPACITY`.
|
||||
pub(crate) const DEFAULT_CHANNEL_CAPACITY: usize = 4096;
|
||||
|
||||
/// Default max records POSTed per request to the control plane.
|
||||
/// Override: `LITELLM_LOG_BATCH_SIZE`.
|
||||
pub(crate) const DEFAULT_MAX_BATCH_SIZE: usize = 256;
|
||||
|
||||
/// Default partial-batch flush cadence, in ms.
|
||||
/// Override: `LITELLM_LOG_FLUSH_INTERVAL_MS`.
|
||||
pub(crate) const DEFAULT_FLUSH_INTERVAL_MS: u64 = 500;
|
||||
|
||||
/// Provider attributed to realtime sessions in the logging payload.
|
||||
#[cfg(feature = "server")]
|
||||
pub(crate) const DEFAULT_PROVIDER: &str = "openai";
|
||||
|
|
|
|||
|
|
@ -1,72 +0,0 @@
|
|||
use std::time::Duration;
|
||||
|
||||
use serde::Serialize;
|
||||
|
||||
use crate::constants::{
|
||||
DEFAULT_CHANNEL_CAPACITY, DEFAULT_FLUSH_INTERVAL_MS, DEFAULT_MAX_BATCH_SIZE,
|
||||
};
|
||||
use crate::integrations::types::StandardLoggingPayload;
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct CallbackLogsRequest {
|
||||
pub records: Vec<CallbackLogRecord>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct CallbackLogRecord {
|
||||
pub status: String,
|
||||
pub standard_logging_payload: StandardLoggingPayload,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub error: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct LogRecord {
|
||||
pub status: String,
|
||||
pub payload: StandardLoggingPayload,
|
||||
pub error: Option<String>,
|
||||
}
|
||||
|
||||
impl LogRecord {
|
||||
pub fn into_callback_record(self) -> CallbackLogRecord {
|
||||
CallbackLogRecord {
|
||||
status: self.status,
|
||||
standard_logging_payload: self.payload,
|
||||
error: self.error,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) struct EgressTunables {
|
||||
pub channel_capacity: usize,
|
||||
pub max_batch_size: usize,
|
||||
pub flush_interval: Duration,
|
||||
}
|
||||
|
||||
impl EgressTunables {
|
||||
pub fn from_env() -> Self {
|
||||
Self {
|
||||
channel_capacity: env_positive(
|
||||
"LITELLM_LOG_CHANNEL_CAPACITY",
|
||||
DEFAULT_CHANNEL_CAPACITY,
|
||||
),
|
||||
max_batch_size: env_positive("LITELLM_LOG_BATCH_SIZE", DEFAULT_MAX_BATCH_SIZE),
|
||||
flush_interval: Duration::from_millis(env_positive(
|
||||
"LITELLM_LOG_FLUSH_INTERVAL_MS",
|
||||
DEFAULT_FLUSH_INTERVAL_MS,
|
||||
)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn env_positive<T>(name: &str, default: T) -> T
|
||||
where
|
||||
T: std::str::FromStr + PartialOrd + From<u8>,
|
||||
{
|
||||
let zero = T::from(0u8);
|
||||
std::env::var(name)
|
||||
.ok()
|
||||
.and_then(|value| value.trim().parse::<T>().ok())
|
||||
.filter(|n| *n > zero)
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
//! Pure-Rust logging integrations. Names map 1:1 to Python
|
||||
//! `litellm/integrations/`:
|
||||
//! - [`custom_guardrail::CustomGuardrail`] — the guardrail callback trait
|
||||
//! - [`custom_logger::CustomLogger`] — the callback trait
|
||||
//! - [`litellm_python_proxy_api::LiteLLMPythonProxyAPILogger`] — ships events
|
||||
//! to the Python proxy's `/v1/rust_control_plane/logs` endpoint
|
||||
//! - [`types`] — the typed `StandardLoggingPayload` wire contract
|
||||
|
||||
pub mod custom_guardrail;
|
||||
pub mod custom_logger;
|
||||
pub mod litellm_python_proxy_api;
|
||||
pub mod types;
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
pub mod audio_transcription;
|
||||
pub mod ocr;
|
||||
pub mod realtime;
|
||||
pub mod realtime_pool;
|
||||
pub mod responses_ws;
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
pub use crate::ocr::{OcrRequest, ocr};
|
||||
|
|
@ -13,9 +13,7 @@
|
|||
//! binary turns on.
|
||||
|
||||
pub mod audio_transcription;
|
||||
mod client;
|
||||
pub mod io;
|
||||
pub mod ocr;
|
||||
|
||||
#[cfg(feature = "server")]
|
||||
pub mod auth;
|
||||
|
|
@ -27,6 +25,5 @@ pub mod state;
|
|||
pub mod trace_parity;
|
||||
|
||||
mod constants;
|
||||
pub mod integrations;
|
||||
#[cfg(feature = "server")]
|
||||
mod realtime;
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@
|
|||
//! wires startup.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use litellm_ai_gateway::io::realtime_pool::{PoolConfig, RealtimePool, upstream_key};
|
||||
use litellm_ai_gateway::routes;
|
||||
|
|
@ -18,13 +19,19 @@ use litellm_ai_gateway::state::AppState;
|
|||
use litellm_config::load_model_list;
|
||||
use litellm_core::router::{Deployment, LiteLLMParams, Router};
|
||||
|
||||
use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
|
||||
use litellm_ai_gateway::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger;
|
||||
use litellm_core::integrations::custom_logger::CustomLogger;
|
||||
use litellm_core::integrations::litellm_python_proxy_api::{
|
||||
LiteLLMPythonProxyAPILogger, LogEgressConfig,
|
||||
};
|
||||
|
||||
/// Bind to localhost by default so the gateway is not a public, unauthenticated
|
||||
/// provider proxy out of the box. Override with `HOST` (e.g. `0.0.0.0`).
|
||||
const DEFAULT_HOST: &str = "127.0.0.1";
|
||||
const DEFAULT_PORT: u16 = 4001;
|
||||
const DEFAULT_PROXY_BASE_URL: &str = "http://localhost:4000";
|
||||
const DEFAULT_CHANNEL_CAPACITY: usize = 4096;
|
||||
const DEFAULT_MAX_BATCH_SIZE: usize = 256;
|
||||
const DEFAULT_FLUSH_INTERVAL_MS: u64 = 500;
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
|
|
@ -44,7 +51,7 @@ async fn main() {
|
|||
// Spawn the realtime-logging worker (drains a channel → POSTs batches to the
|
||||
// Python proxy's /v1/callbacks/logs). Built here so the spawn lands on the
|
||||
// tokio runtime. `from_env` reads LITELLM_PROXY_BASE_URL + LITELLM_MASTER_KEY.
|
||||
let proxy_logger = LiteLLMPythonProxyAPILogger::from_env();
|
||||
let proxy_logger = configured_proxy_logger();
|
||||
let loggers: Vec<Arc<dyn CustomLogger>> = vec![proxy_logger];
|
||||
|
||||
let router = Arc::new(build_router());
|
||||
|
|
@ -86,6 +93,41 @@ async fn main() {
|
|||
.expect("server error");
|
||||
}
|
||||
|
||||
fn configured_proxy_logger() -> Arc<LiteLLMPythonProxyAPILogger> {
|
||||
let base = std::env::var("LITELLM_PROXY_BASE_URL")
|
||||
.ok()
|
||||
.filter(|value| !value.trim().is_empty())
|
||||
.unwrap_or_else(|| DEFAULT_PROXY_BASE_URL.to_string());
|
||||
let key = std::env::var("LITELLM_MASTER_KEY").unwrap_or_default();
|
||||
LiteLLMPythonProxyAPILogger::start(
|
||||
base,
|
||||
key,
|
||||
LogEgressConfig {
|
||||
channel_capacity: positive_env(
|
||||
"LITELLM_LOG_CHANNEL_CAPACITY",
|
||||
DEFAULT_CHANNEL_CAPACITY,
|
||||
),
|
||||
max_batch_size: positive_env("LITELLM_LOG_BATCH_SIZE", DEFAULT_MAX_BATCH_SIZE),
|
||||
flush_interval: Duration::from_millis(positive_env(
|
||||
"LITELLM_LOG_FLUSH_INTERVAL_MS",
|
||||
DEFAULT_FLUSH_INTERVAL_MS,
|
||||
)),
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
fn positive_env<T>(name: &str, default: T) -> T
|
||||
where
|
||||
T: std::str::FromStr + PartialOrd + From<u8>,
|
||||
{
|
||||
let zero = T::from(0u8);
|
||||
std::env::var(name)
|
||||
.ok()
|
||||
.and_then(|value| value.trim().parse::<T>().ok())
|
||||
.filter(|value| *value > zero)
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Register every deployment's upstream key with the pool so the replenisher
|
||||
/// pre-warms it. Mirrors `service::run`'s key derivation (strip `openai/`, resolve
|
||||
/// api_key); deployments whose key can't be resolved are skipped (they fresh-dial
|
||||
|
|
|
|||
|
|
@ -1,174 +0,0 @@
|
|||
use litellm_core::Error;
|
||||
use litellm_core::call_lifecycle::CallLifecycle;
|
||||
use serde_json::Value;
|
||||
|
||||
mod common_utils;
|
||||
mod handler;
|
||||
mod hooks;
|
||||
mod prepare;
|
||||
mod types;
|
||||
|
||||
pub use types::OcrRequest;
|
||||
|
||||
use handler::execute_ocr_provider_call;
|
||||
use prepare::{PreparedOcrCall, prepare_ocr_call};
|
||||
|
||||
#[tracing::instrument(target = "litellm::function_trace", level = "trace", skip_all)]
|
||||
pub async fn ocr(request: OcrRequest<'_>) -> Result<Value, Error> {
|
||||
let PreparedOcrCall { request, hooks } = prepare_ocr_call(request);
|
||||
CallLifecycle::default()
|
||||
.run_request(request, &hooks, |request| {
|
||||
execute_ocr_provider_call(request, &hooks)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use serde_json::{Map, json};
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::net::{TcpListener, TcpStream};
|
||||
|
||||
use super::{OcrRequest, ocr};
|
||||
use crate::integrations::types::RequestMetadata;
|
||||
|
||||
async fn read_http_request(socket: &mut TcpStream) -> String {
|
||||
let mut request = Vec::new();
|
||||
let mut buffer = [0_u8; 1024];
|
||||
let header_end = loop {
|
||||
let n = socket.read(&mut buffer).await.expect("reads request");
|
||||
if n == 0 {
|
||||
break request.len();
|
||||
}
|
||||
request.extend_from_slice(&buffer[..n]);
|
||||
if let Some(position) = request.windows(4).position(|window| window == b"\r\n\r\n") {
|
||||
break position + 4;
|
||||
}
|
||||
};
|
||||
let headers = String::from_utf8_lossy(&request[..header_end]);
|
||||
let content_length = headers
|
||||
.lines()
|
||||
.find_map(|line| {
|
||||
let (name, value) = line.split_once(':')?;
|
||||
name.eq_ignore_ascii_case("content-length")
|
||||
.then(|| value.trim().parse::<usize>().ok())
|
||||
.flatten()
|
||||
})
|
||||
.unwrap_or(0);
|
||||
while request.len().saturating_sub(header_end) < content_length {
|
||||
let n = socket.read(&mut buffer).await.expect("reads body");
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
request.extend_from_slice(&buffer[..n]);
|
||||
}
|
||||
String::from_utf8(request).expect("request is utf8")
|
||||
}
|
||||
|
||||
fn base_ocr_request(model: &str) -> OcrRequest<'_> {
|
||||
OcrRequest {
|
||||
model,
|
||||
document: json!({
|
||||
"type": "document_url",
|
||||
"document_url": "https://example.com/doc.pdf"
|
||||
}),
|
||||
api_key: Some("sk-test"),
|
||||
api_base: None,
|
||||
custom_llm_provider: None,
|
||||
extra_headers: None,
|
||||
optional_params: Map::new(),
|
||||
timeout: None,
|
||||
callbacks: Vec::new(),
|
||||
guardrails: Vec::new(),
|
||||
request_metadata: RequestMetadata::default(),
|
||||
litellm_call_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn reducto_file_upload_then_parse_maps_response() {
|
||||
let listener = TcpListener::bind("127.0.0.1:0")
|
||||
.await
|
||||
.expect("test listener binds");
|
||||
let address = listener.local_addr().expect("listener has local address");
|
||||
let server = tokio::spawn(async move {
|
||||
let (mut upload_socket, _) = listener.accept().await.expect("accepts upload request");
|
||||
let upload_request = read_http_request(&mut upload_socket).await;
|
||||
let upload_body = r#"{"file_id":"reducto://uploaded.pdf"}"#;
|
||||
let upload_response = format!(
|
||||
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
|
||||
upload_body.len(),
|
||||
upload_body
|
||||
);
|
||||
upload_socket
|
||||
.write_all(upload_response.as_bytes())
|
||||
.await
|
||||
.expect("writes upload response");
|
||||
|
||||
let (mut parse_socket, _) = listener.accept().await.expect("accepts parse request");
|
||||
let parse_request = read_http_request(&mut parse_socket).await;
|
||||
let parse_body = r#"{"job_id":"job_123","usage":{"num_pages":3,"credits":3},"result":{"chunks":[{"content":"Page 1 block A","blocks":[{"content":"Page 1 block A","bbox":{"page":1},"kind":"text"}]},{"content":"Page 2 block A","blocks":[{"content":"Page 2 block A","bbox":{"page":2},"kind":"table"}]},{"content":"Page 1 block B","blocks":[{"content":"Page 1 block B","bbox":{"page":1},"kind":"text"}]},{"content":"Page 3 block A","blocks":[{"content":"Page 3 block A","bbox":{"page":3},"kind":"figure"}]}]}}"#;
|
||||
let parse_response = format!(
|
||||
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
|
||||
parse_body.len(),
|
||||
parse_body
|
||||
);
|
||||
parse_socket
|
||||
.write_all(parse_response.as_bytes())
|
||||
.await
|
||||
.expect("writes parse response");
|
||||
(upload_request, parse_request)
|
||||
});
|
||||
let api_base = format!("http://{address}");
|
||||
let mut request = base_ocr_request("reducto/parse-v3");
|
||||
request.api_base = Some(&api_base);
|
||||
request.api_key = None;
|
||||
request.extra_headers = Some(Map::from_iter([
|
||||
("Authorization".to_string(), json!("Bearer test-key")),
|
||||
("x-trace-id".to_string(), json!("trace-1")),
|
||||
]));
|
||||
request.document = json!({
|
||||
"type": "document_url",
|
||||
"document_url": "data:application/pdf;base64,JVBERi0xLjQ="
|
||||
});
|
||||
request.optional_params = Map::from_iter([
|
||||
(
|
||||
"formatting".to_string(),
|
||||
json!({"table_output_format": "html"}),
|
||||
),
|
||||
("retrieval".to_string(), json!({"chunk_mode": "section"})),
|
||||
("settings".to_string(), json!({"ocr_system": "standard"})),
|
||||
]);
|
||||
|
||||
let response = ocr(request).await.expect("Reducto OCR succeeds");
|
||||
|
||||
assert_eq!(response["pages"].as_array().map(Vec::len), Some(3));
|
||||
assert_eq!(
|
||||
response["pages"][0]["markdown"],
|
||||
"Page 1 block A\n\nPage 1 block B"
|
||||
);
|
||||
assert_eq!(response["pages"][1]["markdown"], "Page 2 block A");
|
||||
assert_eq!(response["pages"][2]["markdown"], "Page 3 block A");
|
||||
assert_eq!(response["usage_info"]["pages_processed"], 3);
|
||||
assert_eq!(response["usage_info"]["credits"], 3);
|
||||
assert_eq!(response["provider_native_response"]["job_id"], "job_123");
|
||||
let (upload_request, parse_request) = server.await.expect("server task completes");
|
||||
assert!(
|
||||
upload_request
|
||||
.to_ascii_lowercase()
|
||||
.contains("authorization: bearer test-key")
|
||||
);
|
||||
assert!(upload_request.contains("application/pdf"));
|
||||
assert!(upload_request.contains("%PDF-1.4"));
|
||||
assert!(upload_request.contains("x-trace-id: trace-1"));
|
||||
assert!(
|
||||
parse_request
|
||||
.to_ascii_lowercase()
|
||||
.contains("authorization: bearer test-key")
|
||||
);
|
||||
assert!(parse_request.contains(r#""input":"reducto://uploaded.pdf""#));
|
||||
assert!(parse_request.contains(r#""table_output_format":"html""#));
|
||||
assert!(parse_request.contains(r#""chunk_mode":"section""#));
|
||||
assert!(parse_request.contains(r#""ocr_system":"standard""#));
|
||||
}
|
||||
}
|
||||
|
|
@ -1,163 +0,0 @@
|
|||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
use litellm_core::routing_utils::provider::{CustomLlmProvider, get_custom_llm_provider};
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use super::common_utils::ocr_provider_config;
|
||||
use super::hooks::OcrLifecycleHooks;
|
||||
use super::types::{OcrRequest, PreparedOcrRequest};
|
||||
use crate::integrations::custom_guardrail::CustomGuardrailRunner;
|
||||
use crate::integrations::custom_logger::CustomLoggerRunner;
|
||||
|
||||
pub(crate) struct PreparedOcrCall {
|
||||
pub(crate) request: PreparedOcrRequest,
|
||||
pub(crate) hooks: OcrLifecycleHooks,
|
||||
}
|
||||
|
||||
#[tracing::instrument(target = "litellm::function_trace", level = "trace", skip_all)]
|
||||
pub(crate) fn prepare_ocr_call(request: OcrRequest<'_>) -> PreparedOcrCall {
|
||||
let call_id = request
|
||||
.litellm_call_id
|
||||
.map(str::to_string)
|
||||
.unwrap_or_else(new_ocr_call_id);
|
||||
let provider_info = get_custom_llm_provider(request.model, request.custom_llm_provider)
|
||||
.unwrap_or(CustomLlmProvider {
|
||||
model: request.model,
|
||||
custom_llm_provider: "mistral",
|
||||
});
|
||||
let model = provider_info.model.to_string();
|
||||
let custom_llm_provider = provider_info.custom_llm_provider.to_string();
|
||||
let config = ocr_provider_config(&custom_llm_provider, &model)
|
||||
.ok_or_else(|| litellm_core::Error::InvalidProvider(custom_llm_provider.clone()))
|
||||
.and_then(|config| {
|
||||
validate_request_format(config, &request.optional_params, &custom_llm_provider)?;
|
||||
Ok(config)
|
||||
});
|
||||
let optional_params = match &config {
|
||||
Ok(config) => {
|
||||
let supported = config.supported_ocr_params();
|
||||
let mut mapped = config.map_ocr_params(
|
||||
&request
|
||||
.optional_params
|
||||
.iter()
|
||||
.filter(|(name, _)| supported.contains(&name.as_str()))
|
||||
.map(|(name, value)| (name.clone(), value.clone()))
|
||||
.collect(),
|
||||
);
|
||||
for name in [
|
||||
"vertex_project",
|
||||
"vertex_ai_project",
|
||||
"vertex_location",
|
||||
"vertex_ai_location",
|
||||
] {
|
||||
if let Some(value) = request.optional_params.get(name) {
|
||||
mapped.insert(name.to_string(), value.clone());
|
||||
}
|
||||
}
|
||||
mapped
|
||||
}
|
||||
Err(_) => request.optional_params,
|
||||
};
|
||||
|
||||
PreparedOcrCall {
|
||||
request: PreparedOcrRequest {
|
||||
config,
|
||||
model,
|
||||
custom_llm_provider,
|
||||
litellm_call_id: call_id,
|
||||
document: request.document,
|
||||
api_key: request.api_key.map(str::to_string),
|
||||
api_base: request.api_base.map(str::to_string),
|
||||
extra_headers: request.extra_headers,
|
||||
optional_params,
|
||||
timeout: request.timeout,
|
||||
},
|
||||
hooks: OcrLifecycleHooks::new(
|
||||
CustomLoggerRunner::new(request.callbacks),
|
||||
CustomGuardrailRunner::new(request.guardrails),
|
||||
request.request_metadata,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_request_format(
|
||||
config: &'static dyn litellm_core::ocr::transformation::OcrProviderConfig,
|
||||
optional_params: &Map<String, Value>,
|
||||
provider: &str,
|
||||
) -> Result<(), litellm_core::Error> {
|
||||
let Some(format) = optional_params.get("req_format") else {
|
||||
return Ok(());
|
||||
};
|
||||
match format.as_str() {
|
||||
Some("litellm") => Ok(()),
|
||||
Some("native") if config.supported_ocr_params().contains(&"req_format") => Ok(()),
|
||||
Some("native") => Err(litellm_core::Error::InvalidRequest(format!(
|
||||
"`req_format=native` is not supported for provider {provider}"
|
||||
))),
|
||||
_ => Err(litellm_core::Error::InvalidRequest(format!(
|
||||
"Invalid `req_format`: {format}. Expected `litellm` or `native`"
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
fn new_ocr_call_id() -> String {
|
||||
static COUNTER: AtomicU64 = AtomicU64::new(1);
|
||||
let sequence = COUNTER.fetch_add(1, Ordering::Relaxed);
|
||||
let timestamp = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|duration| duration.as_nanos())
|
||||
.unwrap_or(0);
|
||||
format!("ocr-{timestamp}-{sequence}")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use litellm_core::error::Error;
|
||||
use serde_json::{Map, json};
|
||||
|
||||
use super::{OcrRequest, prepare_ocr_call};
|
||||
use crate::integrations::types::RequestMetadata;
|
||||
|
||||
fn base_ocr_request(model: &str) -> OcrRequest<'_> {
|
||||
OcrRequest {
|
||||
model,
|
||||
document: json!({
|
||||
"type": "document_url",
|
||||
"document_url": "https://example.com/doc.pdf"
|
||||
}),
|
||||
api_key: Some("sk-test"),
|
||||
api_base: None,
|
||||
custom_llm_provider: None,
|
||||
extra_headers: None,
|
||||
optional_params: Map::new(),
|
||||
timeout: None,
|
||||
callbacks: Vec::new(),
|
||||
guardrails: Vec::new(),
|
||||
request_metadata: RequestMetadata::default(),
|
||||
litellm_call_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn request_with_format(format: &str) -> OcrRequest<'_> {
|
||||
let mut request = base_ocr_request("mistral/mistral-ocr-latest");
|
||||
request.optional_params = Map::from_iter([("req_format".to_string(), json!(format))]);
|
||||
request
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_format_rejected_for_provider_without_support_as_bad_request() {
|
||||
let prepared = prepare_ocr_call(request_with_format("native"));
|
||||
assert!(
|
||||
matches!(prepared.request.config, Err(Error::InvalidRequest(message)) if message.contains("not supported for provider"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_format_rejected_for_provider_without_support_as_bad_request() {
|
||||
let prepared = prepare_ocr_call(request_with_format("raw"));
|
||||
assert!(
|
||||
matches!(prepared.request.config, Err(Error::InvalidRequest(message)) if message.contains("Invalid `req_format`"))
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -13,10 +13,10 @@ use litellm_core::realtime::types::RealtimeEvent;
|
|||
use serde_json::Value;
|
||||
|
||||
use crate::constants::DEFAULT_PROVIDER;
|
||||
use crate::integrations::custom_logger::{
|
||||
use litellm_core::integrations::custom_logger::{
|
||||
CallbackTiming, CallbackValue, CustomLogger, CustomLoggerRunner, LoggingError, ModelCallDetails,
|
||||
};
|
||||
use crate::integrations::types::{
|
||||
use litellm_core::integrations::types::{
|
||||
RequestMetadata, StandardLoggingMetadata, StandardLoggingPayload, Usage,
|
||||
};
|
||||
|
||||
|
|
@ -232,8 +232,8 @@ impl RealTimeStreaming {
|
|||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::integrations::custom_logger::LogError;
|
||||
use crate::integrations::custom_logger::LogFuture;
|
||||
use litellm_core::integrations::custom_logger::LogError;
|
||||
use litellm_core::integrations::custom_logger::LogFuture;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
fn event(raw: &str) -> RealtimeEvent {
|
||||
|
|
|
|||
|
|
@ -1,43 +1,23 @@
|
|||
# routes/ — the route template
|
||||
|
||||
Every route follows the **same shape** so the layout is predictable. The rule:
|
||||
|
||||
> **Each route module exposes `pub fn router() -> Router<AppState>`.**
|
||||
> `routes/mod.rs::app` merges them all and applies state once. Adding a route is:
|
||||
> create the module, then add one `.merge(<name>::router())` line.
|
||||
Every route follows one shape: each module exposes `pub fn router() -> Router<AppState>`; `routes/mod.rs::app` merges them all and applies state once. Adding a route means creating the module and adding one `.merge(<name>::router())` line.
|
||||
|
||||
## Default: one file
|
||||
A route is a single file containing `router()` + its handler(s) (handlers stay
|
||||
private). This is the norm — don't split until it hurts.
|
||||
```
|
||||
pub fn router() -> Router<AppState> { Router::new().route(PATH, get(handle)) }
|
||||
async fn handle(...) -> impl IntoResponse { ... }
|
||||
```
|
||||
`health.rs` is the example.
|
||||
|
||||
- A route is one file with `router()` + its handler(s) (handlers stay private); `health.rs` is the example
|
||||
- Don't split until it hurts
|
||||
|
||||
## Split out `service` when there's real logic
|
||||
When a route has business logic worth testing without axum, put it in a sibling
|
||||
`service` (a file, or a folder if the route grows). The route file stays the
|
||||
**axum surface** (router + handler + any socket/SSE adapter); `service` is plain
|
||||
Rust with **no axum types**, and its job is to pick the deployment and call the
|
||||
`core` route entrypoint (see `messages/service.rs` calling
|
||||
`litellm_core::messages::messages`). Never build a provider request, resolve a
|
||||
key, or perform the provider call here. `realtime/` is the older example:
|
||||
```
|
||||
realtime/
|
||||
mod.rs # axum surface: router() + handler + the WS<->events adapter
|
||||
service.rs # pure logic: select deployment + call provider (no axum) — testable
|
||||
```
|
||||
Split `service` further (or add `transport`, `repo`, …) only once a single file
|
||||
genuinely gets hard to read.
|
||||
|
||||
- Put testable business logic in a sibling `service` (file, or folder if it grows)
|
||||
- The route file stays the axum surface (router + handler + any socket/SSE adapter); `service` is plain Rust with no axum types
|
||||
- `service` picks the deployment and calls the `core` entrypoint (see `messages/service.rs`), never builds a provider request, resolves a key, or performs the provider call
|
||||
- Split further (`transport`, `repo`, ...) only when one file genuinely gets hard to read
|
||||
|
||||
## Invariants
|
||||
- **Auth is an extractor, not a manual call.** A handler requires auth by adding
|
||||
`crate::auth::RequireMasterKey` to its arguments; it runs during extraction.
|
||||
Never re-implement the check per route.
|
||||
- **Handlers contain no business logic; `service` contains no axum types.**
|
||||
- **No provider handlers in this crate.** Transforms, auth headers, and the
|
||||
provider HTTP call live in `core/src/<route>/`.
|
||||
- A route owns its paths in its own `router()`; `mod.rs` only merges.
|
||||
- Cross-cutting concerns (logging, CORS, timeouts) → Tower layers in `mod.rs`,
|
||||
not duplicated in handlers.
|
||||
|
||||
- Auth is an extractor, not a manual call
|
||||
- Handlers contain no business logic; `service` contains no axum types
|
||||
- No provider handlers here: transforms, auth headers, and the provider HTTP call live in `core/src/<route>/`
|
||||
- A route owns its paths in its own `router()`; `mod.rs` only merges
|
||||
- Cross-cutting concerns go in Tower layers in `mod.rs`, not duplicated in handlers
|
||||
|
|
|
|||
|
|
@ -23,10 +23,10 @@ use litellm_core::router::Router as ModelRouter;
|
|||
use serde::Deserialize;
|
||||
|
||||
use crate::auth::RequireMasterKey;
|
||||
use crate::integrations::custom_logger::CustomLogger;
|
||||
use crate::integrations::types::RequestMetadata;
|
||||
use crate::realtime::streaming::{RealTimeStreaming, SessionStatus};
|
||||
use crate::state::AppState;
|
||||
use litellm_core::integrations::custom_logger::CustomLogger;
|
||||
use litellm_core::integrations::types::RequestMetadata;
|
||||
|
||||
/// Process-local monotonic counter, mixed into the per-session call id so two
|
||||
/// sessions opened in the same nanosecond still get distinct ids.
|
||||
|
|
|
|||
|
|
@ -16,9 +16,9 @@ use litellm_core::router::Router as ModelRouter;
|
|||
use serde::Deserialize;
|
||||
|
||||
use crate::auth::RequireMasterKey;
|
||||
use crate::integrations::custom_logger::CustomLogger;
|
||||
use crate::integrations::types::RequestMetadata;
|
||||
use crate::state::AppState;
|
||||
use litellm_core::integrations::custom_logger::CustomLogger;
|
||||
use litellm_core::integrations::types::RequestMetadata;
|
||||
|
||||
static CALL_SEQ: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
|
|
|
|||
|
|
@ -10,10 +10,10 @@ use litellm_core::responses::instrumentation::{
|
|||
};
|
||||
use litellm_core::responses::types::ResponsesWsEvent;
|
||||
|
||||
use crate::integrations::custom_logger::{
|
||||
use litellm_core::integrations::custom_logger::{
|
||||
CallbackTiming, CallbackValue, CustomLogger, CustomLoggerRunner, LoggingError, ModelCallDetails,
|
||||
};
|
||||
use crate::integrations::types::RequestMetadata;
|
||||
use litellm_core::integrations::types::RequestMetadata;
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub async fn run<In, Out>(
|
||||
|
|
@ -126,7 +126,7 @@ fn logging_values(
|
|||
let end_time = payload.end_time;
|
||||
let callback = CallbackValue::new(callback.object, callback.value);
|
||||
let details = ModelCallDetails::from_standard_logging_payload(
|
||||
crate::integrations::types::StandardLoggingPayload {
|
||||
litellm_core::integrations::types::StandardLoggingPayload {
|
||||
id: payload.id,
|
||||
litellm_call_id: payload.litellm_call_id,
|
||||
call_type: payload.call_type,
|
||||
|
|
@ -139,7 +139,7 @@ fn logging_values(
|
|||
start_time: payload.start_time,
|
||||
end_time: payload.end_time,
|
||||
stream: payload.stream,
|
||||
metadata: crate::integrations::types::StandardLoggingMetadata {
|
||||
metadata: litellm_core::integrations::types::StandardLoggingMetadata {
|
||||
user_api_key_hash: payload.metadata.user_api_key_hash,
|
||||
user_api_key_user_id: payload.metadata.user_api_key_user_id,
|
||||
user_api_key_team_id: payload.metadata.user_api_key_team_id,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ use std::sync::Arc;
|
|||
use crate::io::realtime_pool::RealtimePool;
|
||||
use litellm_core::router::Router;
|
||||
|
||||
use crate::integrations::custom_logger::CustomLogger;
|
||||
use litellm_core::integrations::custom_logger::CustomLogger;
|
||||
|
||||
/// Shared application state handed to every route handler.
|
||||
#[derive(Clone)]
|
||||
|
|
|
|||
6
litellm-rust/crates/config/AGENTS.md
Normal file
6
litellm-rust/crates/config/AGENTS.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
litellm-config is the config-loading boundary. It returns resolved core deployment data and optionally delegates loading to Python.
|
||||
|
||||
- Depends on `litellm-core`; may use PyO3 behind the `python` feature
|
||||
- `load_model_list(path)` calls Python's `litellm.proxy.read_model_list` and parses the JSON into `core::router::Deployment`
|
||||
- Load-time/startup only, never on a request or stream path
|
||||
- Returns typed `Error` variants (`PythonLoading`, `Serialization`, `ModelListParsing`)
|
||||
|
|
@ -1,7 +1,126 @@
|
|||
litellm-core is the LiteLLM SDK in Rust — it makes the LLM call. Each top-level call is a module under `src/<route>/` exposing a public entrypoint named after the route (`messages::messages()`, the Rust equivalent of `litellm.messages()`): you call it and get a typed non-streaming response back.
|
||||
# core — the runtime and the single core route
|
||||
|
||||
A route module owns everything the call needs: types, the provider template trait, provider transforms (under `providers/`), provider/auth/URL resolution, and the handler that performs the HTTP call. Handlers belong here, never in a host crate.
|
||||
`litellm-core` is the LiteLLM SDK in Rust: one language-neutral route per
|
||||
endpoint (`messages::messages`, `ocr`, `audio_transcription`, `realtime`,
|
||||
`responses`). Every host (the Rust SDK, the PyO3 bridge, the gateway) calls the
|
||||
same route program; they differ only in the service implementations they
|
||||
supply. Core owns route sequencing, provider policy, transformation,
|
||||
authentication, protocol operations, provider I/O and callback phase placement.
|
||||
Hosts provide capabilities; they never orchestrate the route.
|
||||
|
||||
Not here: serving HTTP (axum routes, extractors), config file reading, rollout state, databases, or callback dispatch. Env reads are limited to credential fallback in a route's `prepare.rs`.
|
||||
## What core owns
|
||||
|
||||
Routes (messages, ocr, realtime) and providers (anthropic, mistral, openai) are modules, not crates.
|
||||
Core owns all decisions that must stay identical across hosts. A host adapter
|
||||
that performs any of the following around a core entrypoint is a bug:
|
||||
|
||||
- private admission, and the point after which execution cannot be replayed;
|
||||
- deployment and provider interpretation supplied in the typed route inputs;
|
||||
- callback and guardrail phase placement and result interpretation;
|
||||
- provider transformation, credential acquisition, final authorization and
|
||||
encoding;
|
||||
- explicit multi-operation protocols such as OCR upload, submission and
|
||||
polling;
|
||||
- provider I/O, response normalization and terminal success or failure;
|
||||
- transfer of stream, connection and deferred-completion ownership.
|
||||
|
||||
A host adapter performs only work specific to its public boundary: construct
|
||||
the runtime composition, convert public inputs into `request` + `options` +
|
||||
`CallContext`, invoke exactly one core route entrypoint, and map the outcome to
|
||||
the host's response or error contract. Never a second provider pipeline.
|
||||
|
||||
## Route shape
|
||||
|
||||
A top-level call is a module under `src/<route>/`, shaped like `messages`:
|
||||
|
||||
```
|
||||
core/src/messages/
|
||||
mod.rs # pub async fn messages(..) -> Result<.., Error> (+ _stream for SSE)
|
||||
types.rs # request/response types
|
||||
transformation.rs # the provider template trait
|
||||
prepare.rs # provider resolution, auth headers, URL
|
||||
handler.rs # the provider call
|
||||
client.rs # the shared reqwest client
|
||||
```
|
||||
|
||||
`ocr` is in flight: today it holds only `transformation` and `types`; the rest
|
||||
of its lifecycle still lives in the gateway and moves here as it migrates.
|
||||
`audio_transcription` and `realtime` are the same. Bringing a route to full
|
||||
core shape means giving it a `mod.rs` entrypoint that owns the sequence above.
|
||||
|
||||
The invariant is one function body owns the route lifecycle. The conceptual
|
||||
shape is:
|
||||
|
||||
```rust
|
||||
pub async fn ocr<S>(
|
||||
services: &S,
|
||||
request: OcrRequest,
|
||||
options: OcrOptions,
|
||||
context: CallContext,
|
||||
) -> NativeResult<OcrResponse>
|
||||
where
|
||||
S: OcrServices,
|
||||
```
|
||||
|
||||
The exact spelling may be a method on `LiteLlm<S>`. Public adapters may wrap
|
||||
that function but can never reimplement admission, callbacks, provider
|
||||
preparation or transport around it.
|
||||
|
||||
## Services, not a context
|
||||
|
||||
Capabilities are supplied through focused trait implementations. Route-specific
|
||||
requirements traits (an `OcrServices` bundle) declare exactly what a route needs
|
||||
(transport, calls, clock, ...); they are not host contexts, and core is never
|
||||
passed a catch-all gateway environment. The ordinary Rust client supplies native
|
||||
defaults; callers override implementations at construction.
|
||||
|
||||
`CallServices` opens one request-scoped sessions per call, owning per-call state
|
||||
(timing, logging state, retained host objects, deferred completion, correlation
|
||||
state). No-op call services are the default, and their presence must not move
|
||||
provider behavior into a host or force callback payload materialization on an
|
||||
unaffected fast path.
|
||||
|
||||
Your `Callback`/`CallServices` operations must stay distinct where their
|
||||
contracts differ (argument identity, replacement adoption, exception policy,
|
||||
scheduling, direct vs awaited execution). A universal `emit(Event, Json)` is
|
||||
not enough. core still decides which operation runs next and how its result
|
||||
affects execution; the adapter only dispatches it (Python callback, native
|
||||
logger call, or nothing).
|
||||
|
||||
## Language neutrality
|
||||
|
||||
Core types and service contracts must not contain `Py<PyAny>`, `Py<PyDict>`,
|
||||
Axum requests, gateway state or Python logging objects. Host implementations may
|
||||
retain those values privately; core sees only the trait operations and the typed
|
||||
values they return. `CallContext`, `*Request`, `*Options`, `*Response` and core
|
||||
errors are language-neutral.
|
||||
|
||||
## Dependency rules
|
||||
|
||||
```
|
||||
core must not depend on PyO3, Axum or gateway integration types
|
||||
Tower/Axum types stop at the gateway adapter boundary
|
||||
provider transformation, auth and I/O remain in core
|
||||
request-scoped host state belongs to a call session
|
||||
service construction dependencies do not leak into service interfaces
|
||||
```
|
||||
|
||||
## Not the target
|
||||
|
||||
The current `CallLifecycleHooks` shape is not the final public service API: it
|
||||
folds lifecycle sequencing into a stateless generic transformation interface,
|
||||
needs `Send` futures, cannot express the full replacement and error contracts,
|
||||
and does not model request-scoped retained ownership or Python caller-task
|
||||
driving. It is a stepping stone, not the contract to build new routes against.
|
||||
|
||||
Do not introduce a dynamic `TypeId` service map, a shared gateway callback
|
||||
environment reused from core or the bridge, per-callback JSON serialization, or
|
||||
a full Effect layer API. Services traits plus constructors and a scoped call
|
||||
owner are the minimum design; add more machinery only when concrete consumers
|
||||
require it.
|
||||
|
||||
## Gateway migration
|
||||
|
||||
Existing gateway-hosted OCR, transcription and WebSocket provider execution
|
||||
predates this boundary and must move here as those routes migrate. Keeping a
|
||||
gateway callback as a `CallServices` implementation does not justify keeping
|
||||
provider orchestration beside it in the gateway.
|
||||
|
|
|
|||
|
|
@ -1,66 +1 @@
|
|||
# CLAUDE.md
|
||||
|
||||
Rules for `litellm-rust/crates/core`.
|
||||
|
||||
## Responsibility
|
||||
|
||||
`core` is the LiteLLM SDK in Rust: it makes the LLM call. Every top-level
|
||||
LiteLLM call has a public entrypoint here, named after the route
|
||||
(`messages::messages()` is the Rust equivalent of `litellm.messages()`), and
|
||||
calling it returns a typed non-streaming response.
|
||||
|
||||
Allowed:
|
||||
- The public entrypoint for a route, plus its `<route>_stream` variant when the
|
||||
route supports streaming.
|
||||
- Provider resolution, auth header construction, URL building, and the provider
|
||||
HTTP call (shared reused client, connect + request timeouts).
|
||||
- Shared request/response structs.
|
||||
- Typed errors with stable, non-sensitive messages.
|
||||
- Deterministic validation helpers.
|
||||
- Serialization helpers that intentionally mirror Python output shape.
|
||||
- Route templates that match Python base config responsibilities, such as
|
||||
`messages::transformation::AnthropicMessagesProviderConfig`.
|
||||
|
||||
Not allowed:
|
||||
- Serving HTTP: axum routers, extractors, and other transport concerns.
|
||||
- Filesystem, database, or cache access.
|
||||
- Config file reading or rollout state; the host resolves those and passes them
|
||||
in. Env reads are limited to credential fallback in a route's `prepare.rs`.
|
||||
- Logging callbacks, tracing spans, spend writes, or customer callbacks.
|
||||
- Provider-specific branching that belongs in `providers`.
|
||||
- Panics for user/provider-controlled input.
|
||||
|
||||
## Typed Contracts (core rule)
|
||||
|
||||
Trait and function boundaries MUST be strongly typed. No stringly-typed JSON
|
||||
(`&str` / `String` / `Vec<String>` / bare `serde_json::Value`) as a transform
|
||||
input or output. Parse wire bytes into typed structs/enums at the host edge;
|
||||
`core` and `providers` operate only on those types (e.g. `RealtimeEvent`,
|
||||
`RealtimeTransformResult`, `OcrRequestData`). A `type`-style discriminator is a
|
||||
typed field on a struct, not a raw string threaded through the API.
|
||||
|
||||
## Structure
|
||||
|
||||
Use route names directly under `src/`: `messages`, `ocr`, future
|
||||
`chat_completions`, `embeddings`, and similar top-level LiteLLM calls. Do not
|
||||
invent broad names like `engine` for route contracts.
|
||||
|
||||
`src/messages` is the reference shape for a route module:
|
||||
|
||||
```
|
||||
mod.rs pub async fn messages(..) (+ messages_stream)
|
||||
types.rs request/response types
|
||||
transformation.rs the provider template trait
|
||||
prepare.rs provider resolution, auth headers, URL
|
||||
handler.rs the provider call
|
||||
client.rs the shared reqwest client
|
||||
```
|
||||
|
||||
## Parity Rules
|
||||
|
||||
- Every shared type used by a provider transform needs unit tests for
|
||||
serialization shape.
|
||||
- If Python parity requires always emitting a `null` field instead of omitting
|
||||
it, document that in code and pin it with a test.
|
||||
- Error enums should preserve enough detail for Python/HTTP hosts to map errors
|
||||
consistently without exposing document contents or upstream bodies.
|
||||
See ./AGENTS.md
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ serde.workspace = true
|
|||
serde_json.workspace = true
|
||||
thiserror.workspace = true
|
||||
tracing.workspace = true
|
||||
tokio = { workspace = true, features = ["rt", "sync", "time"] }
|
||||
tracing-subscriber = { workspace = true, optional = true }
|
||||
sha2.workspace = true
|
||||
aws-config = { version = "1.9.0", default-features = false, features = ["rustls", "rt-tokio"], optional = true }
|
||||
|
|
@ -36,5 +37,5 @@ observability = ["dep:tracing-subscriber"]
|
|||
|
||||
[dev-dependencies]
|
||||
rstest.workspace = true
|
||||
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
|
||||
tokio = { workspace = true, features = ["io-util", "macros", "net", "rt-multi-thread"] }
|
||||
tracing-subscriber.workspace = true
|
||||
|
|
|
|||
|
|
@ -45,3 +45,5 @@ pub const EMPTY_TEXT_PLACEHOLDER: &str =
|
|||
"[System: Empty message content sanitised to satisfy protocol]";
|
||||
|
||||
pub const FUNCTION_TRACE_TARGET: &str = "litellm::function_trace";
|
||||
|
||||
pub(crate) const RUST_CONTROL_PLANE_LOGS_PATH: &str = "/v1/rust_control_plane/logs";
|
||||
|
|
|
|||
|
|
@ -16,12 +16,12 @@ use reqwest::Client;
|
|||
use tokio::sync::mpsc::{self, Receiver, Sender};
|
||||
use tokio::time::interval;
|
||||
|
||||
use crate::constants::{DEFAULT_PROXY_BASE_URL, RUST_CONTROL_PLANE_LOGS_PATH};
|
||||
use crate::constants::RUST_CONTROL_PLANE_LOGS_PATH;
|
||||
use crate::integrations::custom_logger::{
|
||||
CallbackTiming, CallbackValue, CustomLogger, LogError, LogFuture, LoggingError,
|
||||
ModelCallDetails,
|
||||
};
|
||||
use types::{CallbackLogsRequest, EgressTunables, LogRecord};
|
||||
use types::{CallbackLogsRequest, LogRecord};
|
||||
|
||||
pub mod types;
|
||||
|
||||
|
|
@ -33,9 +33,8 @@ pub struct LiteLLMPythonProxyAPILogger {
|
|||
impl LiteLLMPythonProxyAPILogger {
|
||||
/// Spawn the background worker and return a logger handle. `base` is the
|
||||
/// proxy base URL (no trailing path); `master_key` is sent as a bearer token.
|
||||
pub fn start(base: String, master_key: String) -> Arc<Self> {
|
||||
let tunables = EgressTunables::from_env();
|
||||
let (sink, receiver) = mpsc::channel::<LogRecord>(tunables.channel_capacity);
|
||||
pub fn start(base: String, master_key: String, config: LogEgressConfig) -> Arc<Self> {
|
||||
let (sink, receiver) = mpsc::channel::<LogRecord>(config.channel_capacity);
|
||||
let url = format!(
|
||||
"{}{}",
|
||||
base.trim_end_matches('/'),
|
||||
|
|
@ -47,29 +46,12 @@ impl LiteLLMPythonProxyAPILogger {
|
|||
client,
|
||||
url,
|
||||
master_key,
|
||||
tunables.max_batch_size,
|
||||
tunables.flush_interval,
|
||||
config.max_batch_size,
|
||||
config.flush_interval,
|
||||
));
|
||||
Arc::new(Self { sink })
|
||||
}
|
||||
|
||||
/// Build a logger from the environment: `LITELLM_PROXY_BASE_URL` (default
|
||||
/// `http://localhost:4000`) and `LITELLM_MASTER_KEY`.
|
||||
///
|
||||
/// `LITELLM_PROXY_BASE_URL` is treated as the full base and the route is
|
||||
/// appended verbatim, so if the proxy runs under a `SERVER_ROOT_PATH`
|
||||
/// (e.g. served at `https://host/litellm`), include it in the base
|
||||
/// (`LITELLM_PROXY_BASE_URL=https://host/litellm`) and the POST lands at
|
||||
/// `https://host/litellm/v1/rust_control_plane/logs`.
|
||||
pub fn from_env() -> Arc<Self> {
|
||||
let base = std::env::var("LITELLM_PROXY_BASE_URL")
|
||||
.ok()
|
||||
.filter(|value| !value.trim().is_empty())
|
||||
.unwrap_or_else(|| DEFAULT_PROXY_BASE_URL.to_string());
|
||||
let key = std::env::var("LITELLM_MASTER_KEY").unwrap_or_default();
|
||||
Self::start(base, key)
|
||||
}
|
||||
|
||||
fn enqueue(&self, record: LogRecord) -> Result<(), LogError> {
|
||||
self.sink.try_send(record).map_err(|err| match err {
|
||||
mpsc::error::TrySendError::Full(_) => LogError::channel_full(),
|
||||
|
|
@ -78,6 +60,13 @@ impl LiteLLMPythonProxyAPILogger {
|
|||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub struct LogEgressConfig {
|
||||
pub channel_capacity: usize,
|
||||
pub max_batch_size: usize,
|
||||
pub flush_interval: Duration,
|
||||
}
|
||||
|
||||
impl CustomLogger for LiteLLMPythonProxyAPILogger {
|
||||
fn async_log_success_event<'a>(
|
||||
&'a self,
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
use serde::Serialize;
|
||||
|
||||
use crate::integrations::types::StandardLoggingPayload;
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct CallbackLogsRequest {
|
||||
pub records: Vec<CallbackLogRecord>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct CallbackLogRecord {
|
||||
pub status: String,
|
||||
pub standard_logging_payload: StandardLoggingPayload,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub error: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct LogRecord {
|
||||
pub status: String,
|
||||
pub payload: StandardLoggingPayload,
|
||||
pub error: Option<String>,
|
||||
}
|
||||
|
||||
impl LogRecord {
|
||||
pub fn into_callback_record(self) -> CallbackLogRecord {
|
||||
CallbackLogRecord {
|
||||
status: self.status,
|
||||
standard_logging_payload: self.payload,
|
||||
error: self.error,
|
||||
}
|
||||
}
|
||||
}
|
||||
4
litellm-rust/crates/core/src/integrations/mod.rs
Normal file
4
litellm-rust/crates/core/src/integrations/mod.rs
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
pub mod custom_guardrail;
|
||||
pub mod custom_logger;
|
||||
pub mod litellm_python_proxy_api;
|
||||
pub mod types;
|
||||
|
|
@ -5,6 +5,7 @@ pub mod chat_completions;
|
|||
pub mod constants;
|
||||
pub mod error;
|
||||
pub mod http_utils;
|
||||
pub mod integrations;
|
||||
pub mod messages;
|
||||
#[cfg(any(feature = "observability", test))]
|
||||
pub mod observability;
|
||||
|
|
|
|||
13
litellm-rust/crates/core/src/ocr/client.rs
Normal file
13
litellm-rust/crates/core/src/ocr/client.rs
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
use std::sync::OnceLock;
|
||||
use std::time::Duration;
|
||||
|
||||
pub(super) fn http_client() -> &'static reqwest::Client {
|
||||
static CLIENT: OnceLock<reqwest::Client> = OnceLock::new();
|
||||
CLIENT.get_or_init(|| {
|
||||
reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(600))
|
||||
.connect_timeout(Duration::from_secs(10))
|
||||
.build()
|
||||
.unwrap_or_else(|_| reqwest::Client::new())
|
||||
})
|
||||
}
|
||||
|
|
@ -1,12 +1,12 @@
|
|||
use litellm_core::error::Error;
|
||||
use litellm_core::http_utils::http_request;
|
||||
use litellm_core::ocr::transformation::OcrResponseHandling;
|
||||
use crate::error::Error;
|
||||
use crate::http_utils::http_request;
|
||||
use crate::ocr::transformation::OcrResponseHandling;
|
||||
use serde_json::Value;
|
||||
|
||||
use super::client::http_client;
|
||||
use super::common_utils::{poll_document_intelligence, truncate_error_body};
|
||||
use super::hooks::OcrLifecycleHooks;
|
||||
use super::types::PreparedOcrRequest;
|
||||
use crate::client::http_client;
|
||||
use super::runtime_types::PreparedOcrRequest;
|
||||
|
||||
#[tracing::instrument(target = "litellm::function_trace", level = "trace", skip_all)]
|
||||
pub(crate) async fn execute_ocr_provider_call(
|
||||
|
|
@ -1,15 +1,15 @@
|
|||
use litellm_core::call_lifecycle::{CallLifecycleContext, CallLifecycleHooks, CallLifecycleTiming};
|
||||
use litellm_core::error::Error;
|
||||
use litellm_core::providers::reducto::ocr::transformation::{
|
||||
use crate::call_lifecycle::{CallLifecycleContext, CallLifecycleHooks, CallLifecycleTiming};
|
||||
use crate::error::Error;
|
||||
use crate::providers::reducto::ocr::transformation::{
|
||||
build_upload_request, extract_document_source, extract_upload_file_id,
|
||||
};
|
||||
use serde_json::{Map, Value, json};
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
|
||||
use super::client::http_client;
|
||||
use super::common_utils::{convert_document_url_to_data_uri, string_headers, truncate_error_body};
|
||||
use super::types::{PreparedOcrRequest, ProviderOcrRequest};
|
||||
use crate::client::http_client;
|
||||
use super::runtime_types::{PreparedOcrRequest, ProviderOcrRequest};
|
||||
use crate::integrations::custom_guardrail::{
|
||||
CustomGuardrailRunner, GuardrailContext, GuardrailError, GuardrailRequest,
|
||||
};
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use litellm_core::call_lifecycle::{CallLifecycleContext, CallLifecycleRequest};
|
||||
use litellm_core::ocr::transformation::OcrProviderConfig;
|
||||
use crate::call_lifecycle::{CallLifecycleContext, CallLifecycleRequest};
|
||||
use crate::ocr::transformation::OcrProviderConfig;
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use crate::integrations::custom_guardrail::CustomGuardrail;
|
||||
|
|
@ -25,7 +25,7 @@ pub struct OcrRequest<'a> {
|
|||
}
|
||||
|
||||
pub(crate) struct PreparedOcrRequest {
|
||||
pub(crate) config: Result<&'static dyn OcrProviderConfig, litellm_core::Error>,
|
||||
pub(crate) config: Result<&'static dyn OcrProviderConfig, crate::Error>,
|
||||
pub(crate) model: String,
|
||||
pub(crate) custom_llm_provider: String,
|
||||
pub(crate) litellm_call_id: String,
|
||||
|
|
@ -1,22 +1,22 @@
|
|||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use litellm_ai_gateway::integrations::custom_guardrail::{
|
||||
use litellm_core::error::Error;
|
||||
use litellm_core::integrations::custom_guardrail::{
|
||||
CustomGuardrail, GuardrailContext, GuardrailDecision, GuardrailError, GuardrailEventHook,
|
||||
GuardrailFuture, GuardrailRequest,
|
||||
};
|
||||
use litellm_ai_gateway::integrations::custom_logger::{
|
||||
use litellm_core::integrations::custom_logger::{
|
||||
CallbackTiming, CallbackValue, CustomLogger, LogFuture, ModelCallDetails,
|
||||
};
|
||||
use litellm_ai_gateway::integrations::types::RequestMetadata;
|
||||
use litellm_ai_gateway::ocr::{OcrRequest, ocr};
|
||||
use litellm_core::error::Error;
|
||||
#[cfg(feature = "trace-parity")]
|
||||
use litellm_core::integrations::types::RequestMetadata;
|
||||
#[cfg(feature = "observability")]
|
||||
use litellm_core::observability::FunctionTrace;
|
||||
use litellm_core::ocr::{OcrRequest, ocr};
|
||||
use serde_json::{Map, Value, json};
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::net::{TcpListener, TcpStream};
|
||||
#[cfg(feature = "trace-parity")]
|
||||
#[cfg(feature = "observability")]
|
||||
use tracing::instrument::WithSubscriber;
|
||||
|
||||
async fn read_http_headers(socket: &mut TcpStream) -> String {
|
||||
|
|
@ -324,7 +324,7 @@ async fn ocr_lifecycle_runs_pre_during_and_success_hooks() {
|
|||
GuardrailEventHook::PreCall,
|
||||
GuardrailEventHook::DuringCall,
|
||||
]));
|
||||
#[cfg(feature = "trace-parity")]
|
||||
#[cfg(feature = "observability")]
|
||||
let trace = FunctionTrace::default();
|
||||
let api_base = format!("http://{addr}");
|
||||
let call = ocr(OcrRequest {
|
||||
|
|
@ -347,7 +347,7 @@ async fn ocr_lifecycle_runs_pre_during_and_success_hooks() {
|
|||
},
|
||||
litellm_call_id: Some("ocr-call-1"),
|
||||
});
|
||||
#[cfg(feature = "trace-parity")]
|
||||
#[cfg(feature = "observability")]
|
||||
let call = call.with_subscriber(trace.dispatcher());
|
||||
let response = call.await.expect("ocr request succeeds");
|
||||
|
||||
|
|
@ -367,7 +367,7 @@ async fn ocr_lifecycle_runs_pre_during_and_success_hooks() {
|
|||
error_kind: None,
|
||||
}]
|
||||
);
|
||||
#[cfg(feature = "trace-parity")]
|
||||
#[cfg(feature = "observability")]
|
||||
assert_eq!(
|
||||
trace
|
||||
.events()
|
||||
|
|
@ -406,7 +406,7 @@ async fn ocr_lifecycle_runs_failure_hook_on_provider_error() {
|
|||
});
|
||||
|
||||
let logger = Arc::new(RecordingOcrLogger::default());
|
||||
#[cfg(feature = "trace-parity")]
|
||||
#[cfg(feature = "observability")]
|
||||
let trace = FunctionTrace::default();
|
||||
let api_base = format!("http://{addr}");
|
||||
let call = ocr(OcrRequest {
|
||||
|
|
@ -426,7 +426,7 @@ async fn ocr_lifecycle_runs_failure_hook_on_provider_error() {
|
|||
request_metadata: RequestMetadata::default(),
|
||||
litellm_call_id: Some("ocr-call-2"),
|
||||
});
|
||||
#[cfg(feature = "trace-parity")]
|
||||
#[cfg(feature = "observability")]
|
||||
let call = call.with_subscriber(trace.dispatcher());
|
||||
let err = call.await.expect_err("provider error propagates");
|
||||
|
||||
|
|
@ -443,7 +443,7 @@ async fn ocr_lifecycle_runs_failure_hook_on_provider_error() {
|
|||
error_kind: Some("HttpError".to_string()),
|
||||
}]
|
||||
);
|
||||
#[cfg(feature = "trace-parity")]
|
||||
#[cfg(feature = "observability")]
|
||||
assert_eq!(
|
||||
trace
|
||||
.events()
|
||||
|
|
@ -1,112 +0,0 @@
|
|||
//! Enforcement: the litellm-rust workspace has exactly five crates.
|
||||
//!
|
||||
//! `core` (the Rust SDK), `config` (the config-loading boundary),
|
||||
//! `ai-gateway` (the HTTP/WebSocket host),
|
||||
//! `python-interop` (domain-neutral PyO3 primitives), and `python-bridge` (the
|
||||
//! PyO3 cdylib). Adding or removing a crate must be a
|
||||
//! deliberate act: this test fails until the allowlist here is updated, forcing
|
||||
//! whoever changes the crate set to justify the new crate per the rule that a
|
||||
//! crate is a layer needing independent compilation / its own deps / a separate
|
||||
//! artifact — and to keep `litellm-rust/AGENTS.md` in sync.
|
||||
//!
|
||||
//! Std-only (no toml crate): we scan the workspace manifest's `members = [...]`
|
||||
//! block and the `crates/` directory directly.
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
/// The one true crate set. Update BOTH this and `litellm-rust/AGENTS.md` when the
|
||||
/// workspace legitimately gains or loses a crate.
|
||||
const EXPECTED_MEMBERS: &[&str] = &[
|
||||
"crates/core",
|
||||
"crates/config",
|
||||
"crates/ai-gateway",
|
||||
"crates/python-interop",
|
||||
"crates/python-bridge",
|
||||
];
|
||||
|
||||
/// The crate subdirectory names that must exist under `crates/`.
|
||||
const EXPECTED_CRATE_DIRS: &[&str] = &[
|
||||
"core",
|
||||
"config",
|
||||
"ai-gateway",
|
||||
"python-interop",
|
||||
"python-bridge",
|
||||
];
|
||||
|
||||
const MISMATCH: &str = "litellm-rust crate set changed — update this allowlist AND litellm-rust/AGENTS.md, and justify the crate per the rule (crate = layer needing independent compilation / its own deps / a separate artifact).";
|
||||
|
||||
/// Absolute path to the workspace root (`litellm-rust/`).
|
||||
fn workspace_root() -> PathBuf {
|
||||
// CARGO_MANIFEST_DIR is `.../litellm-rust/crates/core`; the workspace root is
|
||||
// two levels up.
|
||||
Path::new(concat!(env!("CARGO_MANIFEST_DIR"), "/../.."))
|
||||
.canonicalize()
|
||||
.expect("workspace root should resolve")
|
||||
}
|
||||
|
||||
/// Parse the `members = [ ... ]` array out of the workspace `[workspace]` table.
|
||||
///
|
||||
/// Minimal hand-rolled scan: find `members`, then collect every double-quoted
|
||||
/// string up to the closing `]`. Good enough for our fixed manifest shape and
|
||||
/// keeps this test dependency-free.
|
||||
fn parse_members(manifest: &str) -> BTreeSet<String> {
|
||||
let after_members = manifest
|
||||
.split_once("members")
|
||||
.map(|(_, rest)| rest)
|
||||
.expect("workspace manifest should declare members");
|
||||
let open = after_members.find('[').expect("members should be an array");
|
||||
let close = after_members[open..]
|
||||
.find(']')
|
||||
.map(|offset| open + offset)
|
||||
.expect("members array should be closed");
|
||||
let body = &after_members[open + 1..close];
|
||||
|
||||
let mut members = BTreeSet::new();
|
||||
let mut rest = body;
|
||||
while let Some(start) = rest.find('"') {
|
||||
let after_quote = &rest[start + 1..];
|
||||
let end = after_quote
|
||||
.find('"')
|
||||
.expect("opening quote should be matched");
|
||||
members.insert(after_quote[..end].to_string());
|
||||
rest = &after_quote[end + 1..];
|
||||
}
|
||||
members
|
||||
}
|
||||
|
||||
/// The crate subdirectory names under `crates/`.
|
||||
///
|
||||
/// A directory counts as a crate only when it holds a `Cargo.toml`; non-crate
|
||||
/// directories (e.g. docs like `CODING_STANDARDS/`) are ignored so they can live
|
||||
/// under `crates/` without tripping the crate-proliferation guard.
|
||||
fn crate_dirs(root: &Path) -> BTreeSet<String> {
|
||||
fs::read_dir(root.join("crates"))
|
||||
.expect("crates/ directory should exist")
|
||||
.filter_map(Result::ok)
|
||||
.filter(|entry| entry.file_type().map(|ty| ty.is_dir()).unwrap_or(false))
|
||||
.filter(|entry| entry.path().join("Cargo.toml").is_file())
|
||||
.map(|entry| entry.file_name().to_string_lossy().into_owned())
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn workspace_members_match_allowlist() {
|
||||
let root = workspace_root();
|
||||
let manifest = fs::read_to_string(root.join("Cargo.toml"))
|
||||
.expect("workspace Cargo.toml should be readable");
|
||||
|
||||
let actual = parse_members(&manifest);
|
||||
let expected: BTreeSet<String> = EXPECTED_MEMBERS.iter().map(|s| s.to_string()).collect();
|
||||
assert_eq!(actual, expected, "{MISMATCH}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn crates_directory_matches_allowlist() {
|
||||
let root = workspace_root();
|
||||
|
||||
let actual = crate_dirs(&root);
|
||||
let expected: BTreeSet<String> = EXPECTED_CRATE_DIRS.iter().map(|s| s.to_string()).collect();
|
||||
assert_eq!(actual, expected, "{MISMATCH}");
|
||||
}
|
||||
|
|
@ -1,3 +1,10 @@
|
|||
litellm-python-bridge is the PyO3 cdylib that exposes LiteLLM Rust APIs to the Python SDK. Keep API registration, domain dependency wiring, request assembly, and Python exception mapping here. Put domain-neutral Python/Serde conversion and GIL primitives in litellm-python-interop.
|
||||
litellm-python-bridge is the PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK.
|
||||
|
||||
Keep it thin: no business logic, no transforms, no I/O orchestration — just marshal in/out and call the core entrypoint.
|
||||
- Owns API registration, domain dependency wiring, request assembly, and exception mapping
|
||||
- Keep it thin: no business logic, no transforms, no I/O orchestration; just marshal in/out and call the core entrypoint
|
||||
- One stable method per top-level route (`ocr`/`aocr`, `messages`/`amessages`, ...); do not add per-provider helpers
|
||||
- Provider dispatch lives in `litellm-core`, never here
|
||||
- Put domain-neutral Python/Serde conversion and GIL primitives in `litellm-python-interop`
|
||||
- Two logical parts, kept as modules: the domain adapter (`src/routes/*`, `marshal`, `errors`) and the binding artifact (`#[pymodule]`, `#[pyfunction]`, registration in `lib.rs`)
|
||||
- Data handling: do not log OCR payloads or provider responses; avoid copying large payloads; sanitize errors before they cross the boundary
|
||||
- Tests: `cargo test --workspace` compiles here; Python tests cover disabled, enabled, and module-missing fallback for every exposed route
|
||||
|
|
|
|||
|
|
@ -1,43 +1 @@
|
|||
# CLAUDE.md
|
||||
|
||||
Rules for `litellm-rust/crates/python-bridge`.
|
||||
|
||||
## Responsibility
|
||||
|
||||
`python-bridge` is the PyO3 boundary between Python LiteLLM and Rust transforms.
|
||||
Keep this crate thin. It exposes LiteLLM Rust APIs, assembles domain requests,
|
||||
maps domain errors to Python exceptions, and delegates generic conversion and
|
||||
GIL handling to `litellm-python-interop`.
|
||||
|
||||
## Bridge Shape
|
||||
|
||||
- Prefer one stable method per top-level LiteLLM route, for example
|
||||
`messages(...)`, calling the matching `litellm-core` entrypoint.
|
||||
- Do not add one exported PyO3 function per provider helper unless there is a
|
||||
measured reason.
|
||||
- Provider dispatch belongs in the `litellm-core` route module (e.g.
|
||||
`litellm_core::messages`), not in this PyO3 crate.
|
||||
- Python owns rollout state and fallback. Rust should return errors; Python
|
||||
decides whether to raise or fall back. For a rust-only provider/route (no
|
||||
Python reference), the Python side is a thin dispatch that calls Rust and
|
||||
raises when the bridge is unavailable, with no fallback.
|
||||
- Keep the Python interface minimal (well under 100 lines per route): it only
|
||||
marshals inputs and calls Rust. Do not add per-route feature flags, and do
|
||||
not put provider dispatch in `litellm/main.py`; it lives in a thin dispatch
|
||||
class under `litellm/llms/<provider>/<route>/`.
|
||||
|
||||
## Data Handling
|
||||
|
||||
- OCR payloads can contain personal data and large base64 images. Do not log
|
||||
payloads or provider responses.
|
||||
- Avoid copying large payloads more than needed. The current JSON round-trip is
|
||||
acceptable for the first scaffold, but future performance work should evaluate
|
||||
direct PyO3 conversion before expanding Rust coverage to image-heavy paths.
|
||||
- Do not expose raw Rust errors that include document contents or upstream
|
||||
bodies.
|
||||
|
||||
## Tests
|
||||
|
||||
- `cargo test --workspace` must compile this crate.
|
||||
- Python tests must cover bridge disabled, bridge enabled, and module-missing
|
||||
fallback behavior for every exposed route.
|
||||
See ./AGENTS.md
|
||||
|
|
|
|||
|
|
@ -1 +1,5 @@
|
|||
litellm-python-interop is the domain-neutral PyO3 foundation. Keep generic Python/Serde conversion and interpreter primitives here. Do not add LiteLLM domain crates, route types, API registration, or cdylib build features.
|
||||
litellm-python-interop is the domain-neutral PyO3 foundation.
|
||||
|
||||
- Owns generic Python/Serde conversion and interpreter primitives (`gil`, `marshal`)
|
||||
- Depends on PyO3 but no LiteLLM domain crate; no route types, no API registration, no cdylib
|
||||
- Keep it free of `litellm-core`, `OcrRequest`, `CallServices`, LiteLLM exceptions or `_native` surface
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue