Deployments sharing a model name now form a model group instead of
overwriting each other. router_settings drive selection (simple-shuffle
weighted by weight, rpm or tpm, or least-busy), num_retries on a
different deployment for connect errors, 408, 429 and 5xx received
before any byte reaches the client, and allowed_fails / cooldown_time
cooldown with fallback to cooling deployments when no other is left.
Per-deployment health is lock-free atomics. /v1/messages (Anthropic and
Bedrock deployments) and /v1/chat/completions (native OpenAI passthrough
and the core chat route) route through it, and the metrics label records
the deployment each attempt used. Other endpoints pick one deployment
per request without retries.
Implements the Bedrock InvokeModel request shaping that returned 501: anthropic_version, the top-level allowlist, model/stream moved to the URL, header and feature betas in the body's anthropic_beta, clear_thinking, cache_control, structured output and tool fixes in Python's order. Streams decode the AWS event stream into Anthropic SSE. genconfig and Terraform now default to half Bedrock deployments signed with SigV4.
Add the openai provider to /v1/chat/completions. The caller's body is passed
through with only the model renamed and LiteLLM-owned keys removed, and the
provider's JSON or SSE bytes are relayed as received. Streams have no total
timeout, only a per-read idle limit, so 30+ minute generations are not cut off.
openai_like and the translated providers keep their existing path.
Adds /health/liveliness, /health/liveness and /health/readiness with the Python proxy's bodies,
a /metrics endpoint (request count, latency and time to first byte histograms by route, model
group and status, in-flight gauge, upstream failures by deployment) and a drain on SIGTERM or
SIGINT: readiness turns 503, the listener closes after LITELLM_UNREADY_DELAY_SECONDS and open
requests and streams get up to LITELLM_DRAIN_SECONDS (default 900) to finish
Adds litellm-rust/loadtest: a mock LLM (OpenAI chat, Anthropic messages, Bedrock invoke with
event-stream framing) that takes its timing from the prompt, an open-loop load generator with
workloads covering 10k to 1M token contexts, 6 minute non-streaming calls and 30 minute streams,
a config and virtual key generator, a summarizer and a /proc sampler, plus Terraform for an AWS
rig with load generators, gateway and mocks on separate machines
The gateway sets TCP_NODELAY on accepted sockets, removing a ~40 ms stall per streamed response,
and its Docker build takes LTO and CODEGEN_UNITS build args for faster iteration builds
* docs(rust): add ADR folder with ADR 000 and ADR 001 scaffold
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): add ADR 000 and ADR 001 on typing requests inside Rust
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): list four ADR sections in ADR 000
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): move ADR process doc from adr_000 to AGENTS.md
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
