mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-11 08:38:24 +00:00
c0d7264e8b
The Responses route now resolves openrouter alongside openai. The OpenRouter config reuses the OpenAI request and response transforms and owns only its bearer credential (OPENROUTER_API_KEY, then OR_API_KEY) and base URL (OPENROUTER_API_BASE). Wire model ids follow Python's get_llm_provider, including the openrouter/auto native id with an explicit provider. The Python route host no longer hands litellm.openai_key to OpenRouter. The OpenRouter live test lives in tests/live/openrouter.rs and returns early when the provider key is not exported, so cargo test stays offline by default instead of hiding it behind --ignored. Co-authored-by: nate-berri <nate@berri.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
79 lines
6.4 KiB
Markdown
79 lines
6.4 KiB
Markdown
# rules
|
|
|
|
## Scope
|
|
|
|
- This crate mirrors `litellm/llms/`: base config traits in `src/base_llm/<format>/`, provider transformations in `src/<provider>/<format>/transformation.rs`
|
|
- Transport (clients, media fetching, header helpers, transport errors) lives in `litellm-http`. See `../inference/AGENTS.md` for crate layering
|
|
|
|
## Layering
|
|
|
|
- `litellm-llms-types` owns API data contracts
|
|
- `src/base_llm/<format>/` owns the adapter contract and provider-independent machinery. It never imports a provider or embeds provider policy in trait defaults, normalization or context defaults
|
|
- `src/<provider>/<format>/` owns that provider's implementation and policy
|
|
- `inference-<format>` owns call orchestration
|
|
- These are the intended boundaries, not a claim that all code already satisfies them
|
|
|
|
## Provider folders
|
|
|
|
- `src/<provider>/` holds provider-wide policy: credentials, auth, endpoints, model capabilities. `common_utils.rs` means shared across that provider's formats, not across providers
|
|
- Shared constants and wire types used by several of a provider's formats go in `litellm-llms-types/src/providers/<provider>/`
|
|
- A provider may explicitly reuse another provider's helper when its policy applies to the backend (Bedrock, Vertex and Azure reuse `anthropic/messages` shaping for Claude). That does not make the policy format-wide
|
|
- Pure payload rewrites belong with transformations, not transport handlers, even in a file named `handler.rs`
|
|
|
|
## AGENTS.md convention
|
|
|
|
- Every file is `# rules` then `# references`, both concise bullets. Split a long `# rules` into `##` topic sections
|
|
- A folder gets an AGENTS.md only when it adds something: a deviation, an explicit reuse of another provider, or upstream docs no other file lists. A folder without one follows its nearest parent
|
|
- Every `src/<provider>/<format>/` folder gets one, since it talks to its own upstream endpoint
|
|
- A rule lives in the highest file where it holds. Never restate a parent's rule
|
|
- Each upstream URL appears in exactly one AGENTS.md, the one owning that contract. Others point to that file by path
|
|
- Format specs: `litellm-llms-types/src/formats/<format>/`
|
|
- Provider wire-type docs: `litellm-llms-types/src/providers/<provider>/`
|
|
- Provider-wide docs (auth, errors, regions): `src/<provider>/`
|
|
- One host's endpoint docs: `src/<provider>/<format>/`
|
|
|
|
## Streaming
|
|
|
|
- Each format's `src/base_llm/<format>/streaming.rs` exposes the same three names: `event_stream` (bytes to typed events), `decode` (a whole stream to typed events) and `encode` (one typed event to the SSE frame LiteLLM's proxy sends clients). Callers use the module path, never a vendor prefix
|
|
- SSE framing, `[DONE]` and the generic helpers live in `src/base_llm/sse.rs`. Messages frames are `event:` plus `data:`; Chat Completions and Responses frames are bare `data:`, as Python's proxy writes them
|
|
- Decoding is strict: a malformed event, or `[DONE]` on a Messages stream, is `InvalidResponse` and ends the stream. Python never fails a stream this way; the divergence is deliberate and `src/base_llm/sse.rs` lists what Python does per format. A stream cut inside an event drops that event, as the SSE spec requires
|
|
- What ends or fails a stream belongs on the format's event type (`is_terminal`, `is_failure`). The cache's success check in `inference/src/caching.rs` still matches type strings; move it onto the typed events rather than adding new string matches
|
|
|
|
## Python pairs
|
|
|
|
- Python paths identify counterparts but do not dictate Rust module names or class hierarchy. Preserve behavior and concepts, not structure
|
|
- Keep operation and parameter names when responsibilities match. Rust types keep the Python name with Rust acronym casing (`BaseOCRConfig` -> `BaseOcrConfig`). Private Python helpers drop the leading underscore
|
|
- Give Rust-only helpers distinct responsibility names instead of reusing trait method names
|
|
- Use trait defaults for unchanged inherited behavior and explicit delegation for shared provider behavior. Config traits represent real provider contracts, so do not recreate inheritance with extra traits
|
|
- Every provider format writes out its own `impl Base<Format>Config for <Provider><Format>Config`. Do not route several providers' methods through one cross-provider wrapper
|
|
- The base OCR and Mistral OCR pairs are the reference when aligning transformations
|
|
|
|
## Method order
|
|
|
|
- OCR configs: supported params, credential metadata and connection resolution, health-check input, param mapping, env validation, URL, request, async request, response, async response, error conversion
|
|
- Other formats: param support and mapping, env validation, URL, request, response, then Rust-only runtime hooks
|
|
- Constants and data types before the config, private helpers after it in operation order, tests last
|
|
|
|
## Known partial ports
|
|
|
|
- `AnthropicConfig`, `AmazonConverseConfig` and the non-OCR base traits are partial. Preserve their acceptance gates, passthrough and host fallback contracts
|
|
- `OpenAiResponsesApiConfig` implements WebSocket transformations and direct HTTP Responses, without Python's model-specific param rewriting or Responses-to-Chat emulation
|
|
- OpenAI-compatible Responses hosts (`openrouter`) reuse the OpenAI request and response transforms as free functions and own only their credentials and base URL
|
|
- Chat `supported_openai_param_mappings` lists OpenAI/provider name pairs, unlike Python's name-only list. Audio `map_transcription_params` is a Rust filtering helper
|
|
- Auth resolution is split between configs and route preparation in `inference-<format>`
|
|
|
|
## OCR specifics
|
|
|
|
- `BaseOcrConfig::prepare_request` matches Python's HTTP-handler preparation. `validate_request_body` is Rust-only
|
|
- `src/base_llm/ocr/error.rs` and `document.rs` are Rust-only: the OCR error taxonomy shared with the route and inline-document helpers
|
|
- Mistral auth key handling and top-level response-extra preservation differ from Python. Layout refactors must keep those behaviors and their tests
|
|
|
|
## Code
|
|
|
|
- Providers choose auth and header policy, shared auth and HTTP infrastructure apply it. Generic config lookup uses the existing settings utilities
|
|
- Pick traversal by operation (per-block map, filter, or whole-message when blocks depend on each other). Add an abstraction only for a repeated responsibility
|
|
|
|
## Tests
|
|
|
|
- Named `#[rstest]` cases instead of loops, `#[fixture]` for setup, `#[with(...)]` for overrides
|
|
- Verify observable auth precedence, headers, serialization, passthrough and transformations, never code structure
|