scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-11 22:51:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 154
litellm/litellm-rust/crates/ai-gateway/ARCHITECTURE.md
yujonglee 7276caecd4
refactor(rust): extract config crate (#39706) 
* refactor(rust): extract config crate

* refactor(config): split crate modules

* refactor(gateway): remove gil health counter
2026-09-04 08:17:07 -07:00

462 B
Raw Permalink Blame History

ai-gateway architecture

The Rust ai-gateway does LLM inference (realtime WebSocket). Spend tracking is an API callback: it POSTs each finished session to the LiteLLM proxy, which records spend and runs the usual callbacks.

flowchart LR
  C[client] <--> G[Rust ai-gateway<br/>LLM inference]
  G <--> O[OpenAI realtime]
  G -. spend tracking callback .-> P[litellm proxy]
  F[litellm-config<br/>load-time only] --> G
  F -. Python backend .-> P