Provider coding standards (litellm-rust)

Rules for adding or changing an LLM provider/route in litellm-rust. messages (core/src/messages, ANTHROPIC_MESSAGES_CONFIG) is the reference: a route is a core module with a public entrypoint that makes the call and returns a typed response.

Provider resolution

Always resolve the provider/model first with get_custom_llm_provider (core/src/routing_utils/provider.rs). Nothing downstream may branch on a raw model string.
Model/provider is resolved once, in prepare.rs, and passed down as typed fields. Don't re-resolve or re-parse it in transforms or handlers.

Transforms and the base config

Every route defines a base config trait with transform_request + transform_response (+ complete_url, supported_params), living in core/src/<route>/transformation.rs (e.g. AnthropicMessagesProviderConfig, mirroring OcrProviderConfig).
Each provider implements that trait as a const <PROVIDER>_<ROUTE>_CONFIG in core/src/providers/<provider>/<route>/transformation.rs, mirroring the Python provider tree.
Individual configs implement only the request/response transforms. Shared behavior (param filtering, defaults) stays as trait default methods so future providers inherit existing logic instead of reimplementing it.
Prefer composition: a provider that extends another reuses the base trait's defaults or wraps another config; don't copy transform bodies between providers.

Boundaries

Layers never cross: core = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); ai-gateway = serving HTTP/WS (routing, extractors, auth of our callers, streaming to the client); python-bridge = thin PyO3 adapter. Hosts call the core entrypoint; they never build a provider request.
Generic/route files contain zero provider-specific branches. A provider is one module under core/src/providers/<provider>/<route>/; a route is a module, never a new crate.
Route entry point stays thin: core::<route>::<route>() -> prepare_* -> handler (or CallLifecycle::run_request, which owns the pre_call -> during_call -> provider call -> success/failure order and phase timing). Axum handlers validate and delegate to a service that calls the entrypoint; no business logic in them.
Constants (URLs, env-var names, API versions, error messages) live in a crate constants.rs, never inline. Config-shaped env reads happen at the host/config layer with the DEFAULT_* fallback defined in constants.rs; the only env read in core is the credential fallback in a route's prepare.rs.

Types and errors

Typed contracts only: no bare serde_json::Value / String / Vec<String> as a transform input or output. Parse wire bytes into typed structs/enums at the host edge; a type discriminator is a typed field, not a raw string.
Model failures as values: return typed CoreError, don't panic. No unwrap/expect/panic! on user or provider input.
No mutation: build values in one shot (comprehensions/iterators, collect), prefer immutable bindings and owned typed structs over seeding-and-mutating.
Early returns over deep nesting; small focused files over god modules.
Preserve Python output shape intentionally. If a field is always serialized as null for parity, keep it and pin it with a test.

Safety and data minimization

Never log request/response bodies, base64 payloads, document contents, or secrets. Truncate and bound any upstream body before it crosses a host boundary.
Treat empty/whitespace credentials, URLs, and config values as absent at the host resolution layer.
Network I/O sets connect + request timeouts (no unbounded waits), reuses a shared HTTP client, and prefers rustls TLS.

Tests and rollout

Every provider transform ships tests for: supported-param filtering, request body shape, response normalization, missing/null fields, bad input, and *_match_python fixture parity.
Lifecycle/hook tests cover hook order, success + failure callback payloads, pre-call guardrail blocking before any provider I/O, during-call body mutation, and provider-error mapping.
When a route has a Python reference implementation, the Rust path stays off by default and behind Python parity tests (disabled / enabled-equals-Python / bridge-unavailable fallback) until parity is proven. A new provider/route may instead be implemented rust-only with no Python reference; then the Python interface is a thin dispatch to Rust with no fallback, and tests cover the rust-backed path plus the unavailable-bridge error. State the rust-only choice explicitly in the PR.

Python bridge (SDK side)

A Python -> Rust bridge keeps the Python side minimal: the Python interface only marshals inputs and calls the Rust interface, with no transform, handler, or business logic. Aim for well under 100 lines of interface code per route; if the Python grows past that, the logic belongs in Rust.
Do not bloat litellm/main.py. A route's provider dispatch lives in a thin dispatch class under litellm/llms/<provider>/<route>/ that calls the Rust bridge; main.py only instantiates it and calls its sync/async method.
Do not add new feature flags unless explicitly requested. Reuse the existing litellm rust rollout mechanism (use_litellm_rust); never introduce a per-route env flag such as LITELLM_USE_RUST_<ROUTE>.

Checks before push

Run, and keep green:

cd litellm-rust
cargo fmt --check
cargo clippy -p litellm-ai-gateway --all-targets --features server -- -D warnings
cargo clippy -p litellm-core -p litellm-python-interop -p litellm-python-bridge --all-targets -- -D warnings
cargo test --workspace

5.7 KiB Raw Permalink Blame History