The gateway now authenticates LiteLLM virtual keys as well as the master key. Keys are looked up by
sha256 hash in the proxy's LiteLLM_VerificationToken table (general_settings.database_url or
DATABASE_URL), honoring expires, blocked, models and allowed_routes, and are read from
Authorization: Bearer, x-api-key or api-key
The same query LEFT JOINs the key's team. A blocked or missing team refuses the key and the team's
models narrow the key's, as in the proxy's can_key_call_model and can_team_access_model. The key's
team also stays in its grant: a deployment whose model_info.team_id names a team serves only that
team's keys, so routing and retries skip it for any other virtual key, as the proxy's
filter_team_based_models does. The master key routes as before
Keys with budgets pass without spend tracking: the gateway does not record spend for its own
requests, so a key or team gets the proxy's budget-exceeded error only once the spend the Python
proxy has stored reaches its max_budget
Every other restriction the proxy enforces and this gateway does not yet fails closed. The query also
joins the key's team membership, user and budget tables and reduces each restriction column to
whether it is set, and a key carrying one (rate limits and max_parallel_requests, including those in
a budget table or per model in model_max_budget, a membership's allowed_models, projects,
organizations, agents, guardrails, tags, tool allowlists, enforced params and the like, on the key,
its team, its membership or its user) gets a 401 saying that kind of key is not supported by this
gateway yet
Lookups sit behind a moka cache (60 s for found keys, 5 s for unknown ones) with single-flight
misses. When the database fails, recently seen keys keep working from a stale copy and store reads
pause briefly; the master key never needs the database. Auth failures return the proxy's
ProxyException JSON, including 403 key_model_access_denied for a model outside the key's list
The load-test README documents running with virtual keys and x-api-key, and key-churn asks for the
4 model groups genconfig creates by default
Deployments sharing a model name now form a model group instead of
overwriting each other. router_settings drive selection (simple-shuffle
weighted by weight, rpm or tpm, or least-busy), num_retries on a
different deployment for connect errors, 408, 429 and 5xx received
before any byte reaches the client, and allowed_fails / cooldown_time
cooldown with fallback to cooling deployments when no other is left.
Per-deployment health is lock-free atomics. /v1/messages (Anthropic and
Bedrock deployments) and /v1/chat/completions (native OpenAI passthrough
and the core chat route) route through it, and the metrics label records
the deployment each attempt used. Other endpoints pick one deployment
per request without retries.
Implements the Bedrock InvokeModel request shaping that returned 501: anthropic_version, the top-level allowlist, model/stream moved to the URL, header and feature betas in the body's anthropic_beta, clear_thinking, cache_control, structured output and tool fixes in Python's order. Streams decode the AWS event stream into Anthropic SSE. genconfig and Terraform now default to half Bedrock deployments signed with SigV4.
Add the openai provider to /v1/chat/completions. The caller's body is passed
through with only the model renamed and LiteLLM-owned keys removed, and the
provider's JSON or SSE bytes are relayed as received. Streams have no total
timeout, only a per-read idle limit, so 30+ minute generations are not cut off.
openai_like and the translated providers keep their existing path.
Adds /health/liveliness, /health/liveness and /health/readiness with the Python proxy's bodies,
a /metrics endpoint (request count, latency and time to first byte histograms by route, model
group and status, in-flight gauge, upstream failures by deployment) and a drain on SIGTERM or
SIGINT: readiness turns 503, the listener closes after LITELLM_UNREADY_DELAY_SECONDS and open
requests and streams get up to LITELLM_DRAIN_SECONDS (default 900) to finish
Adds litellm-rust/loadtest: a mock LLM (OpenAI chat, Anthropic messages, Bedrock invoke with
event-stream framing) that takes its timing from the prompt, an open-loop load generator with
workloads covering 10k to 1M token contexts, 6 minute non-streaming calls and 30 minute streams,
a config and virtual key generator, a summarizer and a /proc sampler, plus Terraform for an AWS
rig with load generators, gateway and mocks on separate machines
The gateway sets TCP_NODELAY on accepted sockets, removing a ~40 ms stall per streamed response,
and its Docker build takes LTO and CODEGEN_UNITS build args for faster iteration builds
