mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-27 06:13:07 +00:00
3d976a5791
* feat(gateway): share router cooldowns and follow proxy key invalidations through Redis Gateway replicas now share state through the coordination Redis the Python proxy uses (general_settings.coordination_redis, a redis response cache, or REDIS_*): - A deployment one replica cools down is written under Python's deployment:<id>:cooldown key and applied by every other replica's background poll (one MGET a second), so the fleet stops sending to a failing deployment together. allowed_fails counts failures across replicas, as Python's v1 rule does; the v2 failure-rate rule stays per replica, as in Python. Deployments are named by model_info.id, or an id derived from the config that is the same on every replica. - The virtual key cache follows the proxy's litellm_proxy.auth_cache_invalidation channel, so a key blocked, deleted or edited through the admin API (or its team or user) is re-read on every replica at once instead of after up to 60 s. general_settings.user_api_key_cache_ttl sets the key TTL. Nothing is added to the path of a successful request. Without Redis, or while it is down, each replica keeps its own state as before. Redis Cluster and Sentinel are refused at startup for now. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): hooks and observers around inference requests, with usage and cost Adds the litellm-gateway-hooks crate: request facts built only when something reads them, on-path hooks that declare what they read and what they may change and run in config order, off-path observers on bounded queues, a usage tap for SSE and JSON responses, and per-deployment prices from the model cost map overlaid by config prices. The messages and chat handlers run through it when anything is installed; with nothing installed requests take the old path. The key store now also returns the key alias, user, team alias and key and team metadata, and router attempts carry the deployment id. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): write spend to the proxy's database from a native observer Adds a spend writer observer that records every gateway request in the proxy's own Postgres tables with the same statements the Python proxy uses: spend logs, key/user/team/team member/end user spend, and the daily user, team and end-user activity tables. Writes are batched on the proxy's timers and are additive, so replicas and Python proxies can write side by side. The database pool is now shared between virtual keys and the writer. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): enforce key, team, user, member and end-user budgets on shared spend counters Adds a built-in budgets hook that runs the proxy's budget checks, in its order and with its messages, on the proxy's Redis spend:* counters. Each request reads and seeds the counters, checks them and reserves its maximum cost in one Redis script, then settles the reservation to its real cost off the request path, so replicas and Python proxies sharing the Redis enforce one budget. Without Redis the counters are kept per replica. The key store now reads user, team member and end-user budgets, and auth no longer checks budgets against stored spend. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters Adds a built-in rate limits hook with the proxy's v3 limiter semantics: requests per window, tokens per window and in-flight requests per key, on the proxy's own {scope:id}:window|requests|tokens|max_parallel_requests Redis keys and with its scripts, so replicas and Python proxies sharing the Redis enforce one limit. Tokens are reserved from the proxy's estimate and settled to usage when the response ends. Refusals carry the proxy's message, body and headers on each endpoint, and complete responses carry its x-ratelimit-* headers. Without Redis the counters are kept per replica. The key store now reads key, user, team and member rpm/tpm limits and the key's max_parallel_requests, and end users' budget limits are read with their budgets; keys with plain rate limits are no longer refused. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): configurable header, spend metadata and provider metadata hooks Adds four built-in hooks a config lists under litellm_settings.hooks and runs in that order: forward_headers copies named client headers to the provider, strip_headers removes headers from the provider request (including the gateway's default pass-through), spend_metadata adds header and key-metadata values to the spend log's spend_logs_metadata, and provider_metadata sets Messages body fields and headers from templates over request facts. Hooks' upstream header edits now reach the provider on both endpoints, and body edits can set nested fields. Credentials and signer headers are never forwarded or set, and a misconfigured hook stops the gateway at startup. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): model capacity hook, the proxy's dynamic rate limiter Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3 (or model_capacity) in litellm_settings.callbacks. It caps each model group at the sum of its deployments' rpm and tpm across every replica, and with priority_reservation holds each priority's pool to its share once the group's saturation reaches the threshold, counting it below. It uses the proxy's model_saturation_check and priority_model keys and scripts, its refusal messages and headers, and adds a successful response's tokens to the group and the pool admission chose. The rate limit counters gain a general check-and-add over request and token pairs, and the throttling refusal shape is shared. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * test(gateway-auth): expect the narrowed unsupported rate limit message Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): prometheus observer with the proxy's metric and label names Adds the prometheus observer, turned on by listing prometheus in litellm_settings.callbacks, success_callback or failure_callback. Each finished request updates the proxy's request, spend, token, latency and deployment metrics under its names and labels, and /metrics serves them after the gateway's own. prometheus_metrics_config, the exclude lists and prometheus_latency_buckets are read as the proxy reads them. The metrics the gateway already served under the same names come from the observer alone when it is on. Outcomes now carry when the attempt was sent upstream, for the LLM API latency and time to first token. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): s3_v2 observer writing the proxy's logging payload to S3 Adds the s3_v2 observer, turned on by listing s3_v2 (or s3_log) in litellm_settings.callbacks, success_callback or failure_callback. Each finished request becomes the proxy's StandardLoggingPayload, with the messages from the request body and the response as the client got it, streams put back together. It is queued and written every flush interval under the proxy's key layout, with its Content-MD5, Content-Disposition and encryption headers, one PUT per request or one NDJSON object per folder with s3_batch_file_upload. s3_callback_params is read as the proxy reads it, os.environ references included. Uploads are retried, dropped and kept for the next flush by the proxy's rules, and what is queued is written at shutdown. auth-aws gains an S3 signer that signs the path as sent and every header, as botocore does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough Adds azure/<deployment> chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/<base_model> when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client * feat(gateway): Anthropic and Bedrock chat completions streaming Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/ deployments, which were refused with 501 before. The request is the non-streaming one with "stream": true added for Anthropic and sent to /converse-stream for Bedrock, and the provider's events are decoded the way the proxy's chunk parsers decode them: text, tool call deltas with empty arguments sent as "{}", reasoning_content and thinking_blocks with their index and signature, citations and the mapped finish_reason. A wrapper in the shape of the proxy's stream wrapper gives every chunk one id and the client's model name, sends role on the first chunk, adds the empty finish chunk, strips usage from the chunks and, when stream_options.include_usage is set, ends with the usage chunk before [DONE]. The gateway waits for the first chunk before answering, so an upstream error up to there is retried or falls back to another deployment, and an error after it ends the stream with the OpenAI error frame. Usage, cache reads and writes included, is handed to the meter through the response whether or not the client asked for it, and dropping the client stream drops the upstream request * feat(gateway): full OpenAI chat request translation for Anthropic and Bedrock Anthropic and Bedrock Converse deployments now take the whole OpenAI chat request the Python proxy accepts instead of a narrow allowlist. Messages translate as Python builds them: system as a string, parts or several messages, text, image_url (URL or data URL), file and document parts, assistant tool_calls, the tool and function roles, consecutive role merging, empty content, cache_control on Anthropic and cachePoint on Bedrock. Params map per provider: max_tokens and max_completion_tokens with Anthropic's model default, temperature, top_p, stop, tools and tool_choice, parallel_tool_calls, response_format as a json tool or native output format, reasoning_effort and thinking, user as metadata, with n and any other unsupported param refused with a 400 unless drop_params is set globally, on the deployment or on the request, and allowed_openai_params and additional_drop_params honoured. Responses carry tool calls, thinking blocks and response_format answers back in OpenAI shape, streamed and not. The route downloads media URLs a provider cannot take by reference, merges the beta headers a request needs with forwarded ones, and the gateway hands the cost map to both providers for the capability rules that read it. * fix(rust-bridge): read Python's model_cost for chat translation capabilities Chat translation for Anthropic and Bedrock reads capability flags from the cost map. The Python bridge passed none, so every SDK chat call through the Rust path failed. It now reads litellm.model_cost live, so lookups match Python's, runtime-registered models included. ModelInfo::entry returns a Cow so a map read from Python can hand over a copy. Two tests sent temperature != 1 to a model that removed sampling params, which Python itself refuses; they now send temperature=1. * fix(model-catalog): read token counts the cost map writes as whole floats The published cost map now has prompt_cache_min_tokens: 4096.0, which the typed u64 fields rejected, so the gateway dropped the whole map at startup and priced every request at zero. Counts accept any whole non-negative number, as Python reads any JSON number, and a test keeps the checked-in map parsing. * feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough (#47074) Adds azure/<deployment> chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/<base_model> when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client * feat(gateway): Anthropic and Bedrock chat completions streaming (#47077) Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/ deployments, which were refused with 501 before. The request is the non-streaming one with "stream": true added for Anthropic and sent to /converse-stream for Bedrock, and the provider's events are decoded the way the proxy's chunk parsers decode them: text, tool call deltas with empty arguments sent as "{}", reasoning_content and thinking_blocks with their index and signature, citations and the mapped finish_reason. A wrapper in the shape of the proxy's stream wrapper gives every chunk one id and the client's model name, sends role on the first chunk, adds the empty finish chunk, strips usage from the chunks and, when stream_options.include_usage is set, ends with the usage chunk before [DONE]. The gateway waits for the first chunk before answering, so an upstream error up to there is retried or falls back to another deployment, and an error after it ends the stream with the OpenAI error frame. Usage, cache reads and writes included, is handed to the meter through the response whether or not the client asked for it, and dropping the client stream drops the upstream request --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| _support | ||
| base_sdk_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| e2e_harness | ||
| fixtures | ||
| harness_e2e | ||
| integration | ||
| rust-python-harness | ||
| test_litellm_rust | ||
| typing | ||
| unit | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _master_key.py | ||
| _openai_record_replay_proxy.py | ||
| _process_helpers.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| AGENTS.md | ||
| capturing_transport.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_logging.conf | ||
| test_rust_python_harness.py | ||
| VCR_TESTING.md | ||
| white_100x100.png | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/unit
This folder can only run mock tests.