scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-24 00:52:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 90
litellm/litellm-rust/crates/core/AGENTS.md
Yujong Lee d77c144c6c refactor(rust): split custom_httpx into litellm-http and the OCR handler 
custom_httpx mirrored a Python module that mixes transport plumbing with
OCR orchestration. The transport half (media fetcher, transport errors,
request and header helpers) now lives in litellm-http next to the pool,
TLS, proxies and settings, and the OCR request handler moves to
base_llm/ocr/handler.rs. Drops the unused deserialize_optional_param and
stale dead_code allows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 20:46:36 -07:00

2.3 KiB
Raw Blame History

litellm-core is the LiteLLM SDK in Rust — it makes the LLM call. Each top-level call is a module under src/<route>/ exposing a public entrypoint named after the route (messages::messages(), the Rust equivalent of litellm.messages()): you call it and get a typed non-streaming response back.

Crate layering

Each crate mirrors one top-level Python package, so a Rust path reads as its Python path with the crate name in place of the package directory. Dependencies only point down:

  • litellm-types mirrors litellm/types/: pure serde data, no I/O
  • litellm-core-utils mirrors litellm/litellm_core_utils/: pure helpers (provider resolution, prompt factory, call arguments, settings lookup and layer merge), no network I/O
  • litellm-http is Rust-only and route-neutral: settings resolution, the pooled reqwest clients, TLS, proxies, the SSRF-safe media fetcher, request and header helpers, and transport errors. Python's litellm/llms/custom_httpx/ is split by responsibility instead of mirrored: its transport half lives here, its OCR handler in litellm-llms
  • litellm-llms mirrors litellm/llms/: base_llm/<api>/transformation.rs, <provider>/<api>/transformation.rs, and base_llm/ocr/handler.rs (the OCR request handler)
  • litellm-core mirrors the route packages (litellm/ocr/, litellm/messages/, ...): entrypoints, route request types, provider dispatch, the route machine, and hooks

A route module owns the call entrypoint, route request types (*Request<'a>), credential fallback, provider dispatch, and the handler glue that runs a provider config. Provider code never imports from core; when it needs the caller's hooks mid-call it goes through litellm_llms::base_llm::ocr::handler::CallHooks, which each route implements over its host. Import every item from its canonical path. Never re-export another crate's items or give an item a second public path; the only re-export allowed is a private submodule surfacing its item at its module root (mod error; pub use error::Error;). Handlers belong in core or llms, never in a host crate

Not here: serving HTTP (axum routes, extractors), config file reading, rollout state, databases, or callback execution of any kind. Core runs each route as a machine that yields host operations and call events; which integrations consume those events is the host's business.