Adds a spend writer observer that records every gateway request in the
proxy's own Postgres tables with the same statements the Python proxy
uses: spend logs, key/user/team/team member/end user spend, and the
daily user, team and end-user activity tables. Writes are batched on the
proxy's timers and are additive, so replicas and Python proxies can
write side by side. The database pool is now shared between virtual
keys and the writer.
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(gateway): hooks and observers around inference requests, with usage and cost
Adds the litellm-gateway-hooks crate: request facts built only when something
reads them, on-path hooks that declare what they read and what they may
change and run in config order, off-path observers on bounded queues, a
usage tap for SSE and JSON responses, and per-deployment prices from the
model cost map overlaid by config prices. The messages and chat handlers run
through it when anything is installed; with nothing installed requests take
the old path.
The key store now also returns the key alias, user, team alias and key and
team metadata, and router attempts carry the deployment id.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* fix(model-catalog): read token counts the cost map writes as whole floats
The published cost map now has prompt_cache_min_tokens: 4096.0, which the
typed u64 fields rejected, so the gateway dropped the whole map at startup
and priced every request at zero. Counts accept any whole non-negative
number, as Python reads any JSON number, and a test keeps the checked-in
map parsing.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gateway replicas now share state through the coordination Redis the
Python proxy uses (general_settings.coordination_redis, a redis
response cache, or REDIS_*):
- A deployment one replica cools down is written under Python's
deployment:<id>:cooldown key and applied by every other replica's
background poll (one MGET a second), so the fleet stops sending to a
failing deployment together. allowed_fails counts failures across
replicas, as Python's v1 rule does; the v2 failure-rate rule stays
per replica, as in Python. Deployments are named by model_info.id, or
an id derived from the config that is the same on every replica.
- The virtual key cache follows the proxy's
litellm_proxy.auth_cache_invalidation channel, so a key blocked,
deleted or edited through the admin API (or its team or user) is
re-read on every replica at once instead of after up to 60 s.
general_settings.user_api_key_cache_ttl sets the key TTL.
Nothing is added to the path of a successful request. Without Redis,
or while it is down, each replica keeps its own state as before.
Redis Cluster and Sentinel are refused at startup for now.
The gateway now authenticates LiteLLM virtual keys as well as the master key. Keys are looked up by
sha256 hash in the proxy's LiteLLM_VerificationToken table (general_settings.database_url or
DATABASE_URL), honoring expires, blocked, models and allowed_routes, and are read from
Authorization: Bearer, x-api-key or api-key
The same query LEFT JOINs the key's team. A blocked or missing team refuses the key and the team's
models narrow the key's, as in the proxy's can_key_call_model and can_team_access_model. The key's
team also stays in its grant: a deployment whose model_info.team_id names a team serves only that
team's keys, so routing and retries skip it for any other virtual key, as the proxy's
filter_team_based_models does. The master key routes as before
Keys with budgets pass without spend tracking: the gateway does not record spend for its own
requests, so a key or team gets the proxy's budget-exceeded error only once the spend the Python
proxy has stored reaches its max_budget
Every other restriction the proxy enforces and this gateway does not yet fails closed. The query also
joins the key's team membership, user and budget tables and reduces each restriction column to
whether it is set, and a key carrying one (rate limits and max_parallel_requests, including those in
a budget table or per model in model_max_budget, a membership's allowed_models, projects,
organizations, agents, guardrails, tags, tool allowlists, enforced params and the like, on the key,
its team, its membership or its user) gets a 401 saying that kind of key is not supported by this
gateway yet
Lookups sit behind a moka cache (60 s for found keys, 5 s for unknown ones) with single-flight
misses. When the database fails, recently seen keys keep working from a stale copy and store reads
pause briefly; the master key never needs the database. Auth failures return the proxy's
ProxyException JSON, including 403 key_model_access_denied for a model outside the key's list
The load-test README documents running with virtual keys and x-api-key, and key-churn asks for the
4 model groups genconfig creates by default
Deployments sharing a model name now form a model group instead of
overwriting each other. router_settings drive selection (simple-shuffle
weighted by weight, rpm or tpm, or least-busy), num_retries on a
different deployment for connect errors, 408, 429 and 5xx received
before any byte reaches the client, and allowed_fails / cooldown_time
cooldown with fallback to cooling deployments when no other is left.
Per-deployment health is lock-free atomics. /v1/messages (Anthropic and
Bedrock deployments) and /v1/chat/completions (native OpenAI passthrough
and the core chat route) route through it, and the metrics label records
the deployment each attempt used. Other endpoints pick one deployment
per request without retries.
Implements the Bedrock InvokeModel request shaping that returned 501: anthropic_version, the top-level allowlist, model/stream moved to the URL, header and feature betas in the body's anthropic_beta, clear_thinking, cache_control, structured output and tool fixes in Python's order. Streams decode the AWS event stream into Anthropic SSE. genconfig and Terraform now default to half Bedrock deployments signed with SigV4.
Add the openai provider to /v1/chat/completions. The caller's body is passed
through with only the model renamed and LiteLLM-owned keys removed, and the
provider's JSON or SSE bytes are relayed as received. Streams have no total
timeout, only a per-read idle limit, so 30+ minute generations are not cut off.
openai_like and the translated providers keep their existing path.
Adds /health/liveliness, /health/liveness and /health/readiness with the Python proxy's bodies,
a /metrics endpoint (request count, latency and time to first byte histograms by route, model
group and status, in-flight gauge, upstream failures by deployment) and a drain on SIGTERM or
SIGINT: readiness turns 503, the listener closes after LITELLM_UNREADY_DELAY_SECONDS and open
requests and streams get up to LITELLM_DRAIN_SECONDS (default 900) to finish
Adds litellm-rust/loadtest: a mock LLM (OpenAI chat, Anthropic messages, Bedrock invoke with
event-stream framing) that takes its timing from the prompt, an open-loop load generator with
workloads covering 10k to 1M token contexts, 6 minute non-streaming calls and 30 minute streams,
a config and virtual key generator, a summarizer and a /proc sampler, plus Terraform for an AWS
rig with load generators, gateway and mocks on separate machines
The gateway sets TCP_NODELAY on accepted sockets, removing a ~40 ms stall per streamed response,
and its Docker build takes LTO and CODEGEN_UNITS build args for faster iteration builds
* docs(rust): add ADR folder with ADR 000 and ADR 001 scaffold
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): add ADR 000 and ADR 001 on typing requests inside Rust
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): list four ADR sections in ADR 000
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): move ADR process doc from adr_000 to AGENTS.md
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
