ishaan-berri
|
1d5ab42e14
|
feat: add minimal rust router + axum ai-gateway calling router.realtime (2/2) (#31135)
* add CoreError::Routing variant for deployment selection failures
* add minimal Rust Router (simple-shuffle) mirroring router.py spec
* add litellm-router crate manifest
* add ai-gateway POST /v1/realtime handler calling router.realtime
* add ai-gateway health routes
* wire ai-gateway routes into the axum app
* add ai-gateway AppState holding the shared router
* add ai-gateway axum server entrypoint
* add litellm-ai-gateway binary crate manifest
* docs: add ai-gateway folder-architecture AGENTS.md
* register router + ai-gateway crates and axum/rand deps in workspace
* update Cargo.lock for router + ai-gateway crates
* split router: extract model_list types into deployment module
* split router: extract routing policy into strategy module
* split router: move Router orchestration into router module
* router lib: wire submodules and re-export public API
* add read_model_list helper reusing ProxyConfig env/secret resolution
* add GIL-activity tracker (records acquisitions, 30s window)
* add GET /health/gil endpoint for polling GIL activity
* add pyo3 load_router_from_config bridge (feature-gated, load-time only)
* register /health/gil route in ai-gateway
* wire build_router: load from python config when feature enabled
* add optional pyo3 dep + python-config feature to ai-gateway
* update Cargo.lock for optional pyo3 dependency
* fix: satisfy strict ruff budget (FA100) in read_model_list
* test: cover read_model_list env resolution + empty config
* ai-gateway: bind localhost by default, warn on bad PORT/missing keys, wire gateway key
* ai-gateway: add gateway_key to AppState for realtime auth
* ai-gateway: require bearer auth + map unknown model to 404 on /v1/realtime
* ai-gateway: move python interop into python/ with load-time-only AGENTS.md
* ai-gateway: document auth, gil, and python folder in AGENTS.md
* core: add router module (model_list types + simple-shuffle selection)
* ai-gateway: dispatch realtime via core router + providers (drop router crate dep)
* update Cargo.lock: fold router into core
* workspace: drop crates/router member and litellm-router dep
* read_model_list: reuse ProxyConfig.get_config (includes + os.environ + DB) instead of thin yaml read
* ai-gateway: constant-time bearer compare + 500 (not 503) for unconfigured key
* ai-gateway: trim stored gateway key to match trimmed bearer token
* ai-gateway: add subtle dep for constant-time comparison
* workspace: add subtle dependency
* update Cargo.lock for subtle
* core router: make strategy a folder (one module per strategy, simple_shuffle)
* providers: make realtime() a streaming splice (client stream <-> OpenAI) instead of collect
* providers: add futures-channel dev-dep for the streaming live test
* ai-gateway: make /v1/realtime a WebSocket (auth before upgrade, splice typed events)
* ai-gateway: dispatch realtime as a stream splice
* ai-gateway: route /v1/realtime via GET (WebSocket), drop POST
* ai-gateway: enable axum ws feature + futures-util
* update Cargo.lock for ws feature + futures-channel
* core router: add has_deployment() for pre-flight model checks
* ai-gateway: extract auth into auth/ module (single master key, LITELLM_MASTER_KEY)
* ai-gateway routes: adopt router()-per-module template + merge in app()
* ai-gateway: document auth/ + routes template in AGENTS.md
* ai-gateway: realtime route as thin handler + service + transport
* ai-gateway: auth as a RequireMasterKey extractor (idiomatic axum FromRequestParts)
* ai-gateway: docs for auth extractor + simplified route template
* ai-gateway: collapse realtime route to mod.rs + service.rs; docs for extractor/template
* providers realtime: enforce idle timeout around the splice (reap stalled sessions)
* ai-gateway: rename realtime service timeout param to idle_timeout
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
|
2026-06-23 19:16:34 -07:00 |
|