mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-04 07:27:51 +00:00
54735 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
narmstrong-berry
|
8b5dcc8d81
|
feat(loadtest): sustained-250 workload | ||
|
Nate Armstrong
|
967026feef
|
feat(loadtest): sustained-rate workloads at 100, 500 and 1,000 rps (#47719)
Some checks failed
Rust gateway load-test kit / vet, build, image (push) Has been cancelled
Rust gateway load-test kit / terraform fmt, validate (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Evenly spaced arrivals ("arrivals": "constant"), per-request lag_ms so a
run shows when the generator falls behind, and an s3stub that stores
objects compressed and lists from a sorted index, so a 15-minute run at
1,000 rps per gateway fits in memory and verify can list it.
|
||
|
Nate Armstrong
|
334e7119c3
|
feat(gateway): serve /v1/models (#47640)
List the model groups a caller may use at GET /v1/models and /models, in the
proxy's OpenAI list shape, and answer GET /v1/models/{model_id} and
/models/{model_id} with one model or the proxy's 404
A group is listed when the caller is authorized for one of its deployments the
same way an inference request naming it would be: key and team model lists,
all-proxy-models, team-owned deployments. Groups whose deployments all set
model_info.discoverable: false are listed for the master key only, and wildcard
groups are left out. openai_routes, llm_api_routes and info_routes now allow
the listing routes
|
||
|
Nate Armstrong
|
2ea33a5496
|
fix(gateway): reject malformed messages with 400 (#47637)
Chat Completions and /v1/messages now check the body's messages where the handler already parsed it, before hooks admit the request and before any provider call. A missing, non-array or empty messages, a message that is not an object, or a role that is not a string answers 400 invalid_request_error in the route's error shape, and /v1/messages also needs a string role and a string or array content. Missing params use the proxy's ProxyMissingRequiredParamError wording. /v1/messages refuses a missing max_tokens after hooks run, as the proxy does, since the rate limits hook may set one, and a deployment that sets max_tokens stands in for the caller's |
||
|
Nate Armstrong
|
59f0026f9a
|
feat(gateway): serve /v1/messages for chat-completions providers (#47642)
* feat(gateway): serve /v1/messages for chat-completions providers A /v1/messages request for a deployment with no Messages API of its own (openai, azure, hosted_vllm and the other OpenAI-compatible providers) failed with "invalid provider". The messages route now translates the request to a Chat Completions body, sends it through the chat route's OpenAI passthrough and translates the JSON or SSE answer back into Anthropic Messages, as the Python proxy's pass-through adapter does * fix(gateway): account translated messages by upstream usage and keep chat code out of the wheel Translated /v1/messages calls now record what the chat provider reported (usage and response model) apart from the Anthropic wire events. A stream cut off before its usage chunk records no usage, so budgets and rate limits keep their reservations, and Azure deployments priced by response model price translated Messages the same way as chat completions. ChatCompletions is now its own machine that the gateway picks per deployment instead of a branch inside MessagesRoute, so the Python wheel no longer links the OpenAI-compatible providers and their tokenizer data. |
||
|
Nate Armstrong
|
1a5f0c9445
|
perf(gateway): store each prometheus label set once and share it across metrics (#47427)
Some checks are pending
Rust gateway load-test kit / vet, build, image (push) Waiting to run
Rust gateway load-test kit / terraform fmt, validate (push) Waiting to run
LiteLLM Rust / rust-lint (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
LiteLLM Rust / rust-wheel (push) Waiting to run
Every metric family kept its own copy of a series' label strings, so a key x deployment combination cost about 9 KB across the ~20 families. Families now hold a shared, reference-counted label set. /metrics output is unchanged. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Nate Armstrong
|
4d7802f169
|
fix(gateway): stop serving routes outside the hooks and return reservations of failed requests (#47355)
* fix(gateway): refuse spent keys on every route and return reservations of requests that fail before routing Auth refuses a key or team whose stored spend has reached its budget again, so /v1/responses, audio and OCR, which skip the hook pipeline, stop exhausted keys as the proxy does. The budgets hook still enforces the live counters on messages and chat. A request whose body is rejected after the hooks ran, such as an invalid `cache` option, now records its failure so the budget reservation and rate limit leases are returned. A request dropped mid-route, as when the client goes away, is recorded as interrupted with status 499 for the same reason. * fix(gateway): stop serving responses, audio and ocr instead of refusing spent keys in auth /v1/responses, /v1/audio/transcriptions and /v1/ocr never ran the hook pipeline, so budgets, rate limits and spend logging did not apply to them. They now answer 501 like the other routes the gateway does not serve, and the stored-spend check in auth is dropped again: the budgets hook covers every route that remains. * chore(gateway): update Cargo.lock for the dropped route crates * fix(gateway): keep the reservation of a request cut off after it reached the provider |
||
|
Nate Armstrong
|
8fb0643e35
|
perf(gateway): admit rate limits in one redis round trip and send scripts by hash (#47197)
* feat(gateway): hooks and observers around inference requests, with usage and cost Adds the litellm-gateway-hooks crate: request facts built only when something reads them, on-path hooks that declare what they read and what they may change and run in config order, off-path observers on bounded queues, a usage tap for SSE and JSON responses, and per-deployment prices from the model cost map overlaid by config prices. The messages and chat handlers run through it when anything is installed; with nothing installed requests take the old path. The key store now also returns the key alias, user, team alias and key and team metadata, and router attempts carry the deployment id. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * fix(model-catalog): read token counts the cost map writes as whole floats The published cost map now has prompt_cache_min_tokens: 4096.0, which the typed u64 fields rejected, so the gateway dropped the whole map at startup and priced every request at zero. Counts accept any whole non-negative number, as Python reads any JSON number, and a test keeps the checked-in map parsing. * feat(gateway): write spend to the proxy's database from a native observer Adds a spend writer observer that records every gateway request in the proxy's own Postgres tables with the same statements the Python proxy uses: spend logs, key/user/team/team member/end user spend, and the daily user, team and end-user activity tables. Writes are batched on the proxy's timers and are additive, so replicas and Python proxies can write side by side. The database pool is now shared between virtual keys and the writer. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): enforce key, team, user, member and end-user budgets on shared spend counters Adds a built-in budgets hook that runs the proxy's budget checks, in its order and with its messages, on the proxy's Redis spend:* counters. Each request reads and seeds the counters, checks them and reserves its maximum cost in one Redis script, then settles the reservation to its real cost off the request path, so replicas and Python proxies sharing the Redis enforce one budget. Without Redis the counters are kept per replica. The key store now reads user, team member and end-user budgets, and auth no longer checks budgets against stored spend. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters Adds a built-in rate limits hook with the proxy's v3 limiter semantics: requests per window, tokens per window and in-flight requests per key, on the proxy's own {scope:id}:window|requests|tokens|max_parallel_requests Redis keys and with its scripts, so replicas and Python proxies sharing the Redis enforce one limit. Tokens are reserved from the proxy's estimate and settled to usage when the response ends. Refusals carry the proxy's message, body and headers on each endpoint, and complete responses carry its x-ratelimit-* headers. Without Redis the counters are kept per replica. The key store now reads key, user, team and member rpm/tpm limits and the key's max_parallel_requests, and end users' budget limits are read with their budgets; keys with plain rate limits are no longer refused. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * test(gateway-auth): expect the narrowed unsupported rate limit message Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * perf(gateway): admit rate limits in one redis round trip and send scripts by hash A key with requests, in-flight and tokens limits made three sequential EVALs before its request was routed, each carrying the whole Lua script. One script now counts the request, takes the slot and reserves the tokens, stopping at the first limit it is over (giving the slot back when tokens are over), with the same keys and semantics as before. Every rate limit and budget script is sent with EVALSHA and loaded on NOSCRIPT. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Nate Armstrong
|
ac33b81be5
|
perf(gateway): run admission hooks concurrently (#47403)
Hooks read only the request and write only their own edit, so admit now runs them at once and merges their edits in config order. When several refuse, the first in config order answers; the hooks that let the request through settle with that refusal, as before. |
||
|
Nate Armstrong
|
af12504e30
|
fix(gateway): keep drop_params and its siblings out of the openai request (#47368)
drop_params, additional_drop_params and allowed_openai_params are completion() arguments Python consumes before the provider call, but the OpenAI passthrough forwarded them, so OpenAI got an argument it rejects. They join the LiteLLM-owned keys the passthrough strips. |
||
|
Nate Armstrong
|
0099dad675
|
fix(gateway): estimate stream usage an openai-compatible provider does not send (#47367)
* fix(gateway): estimate stream usage an openai-compatible provider does not send A client asking for stream_options.include_usage from mistral, cerebras, perplexity and the other providers whose configs drop stream_options got no usage chunk, since the provider streamed none. Python's stream wrapper adds one estimated with token_counter; the relay now does the same, with cl100k: prompt tokens from the messages alone, completion tokens from the content plus each tool call as model_dump_json() writes it, plus the reasoning counted apart. The chunk is shaped like Python's, comes before [DONE] and is recorded as the call's usage. * fix(gateway): bound the tokenizing a stream usage estimate does Encoding runs inline and grows faster than the text on a long run of one character, so a large prompt could hold a worker for seconds and allocate gigabytes. Each estimate now encodes at most 64 KiB and extrapolates the rest at the rate of the text encoded. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Nate Armstrong
|
afebcb3455
|
fix(gateway): translated chat answers carry a chatcmpl id and object (#47366)
The provider configs leave id out because Python mints it on its own ModelResponse, so non-streaming Anthropic, Bedrock, Vertex and Gemini answers reached gateway clients with no id and no object. The chat handler now names each translated answer as Python does: a minted chatcmpl- id, or Gemini's responseId, and object chat.completion. It does so before the answer is cached, so a cache hit replays the same id. |
||
|
Nate Armstrong
|
28feeab36c
|
fix(gateway): bedrock chat uses the aws settings in the deployment config (#47365)
The deployment's aws_* litellm_params (keys, region, session, role, aws_bedrock_runtime_endpoint) sit in their own group, not in extra, and only the Vertex group was copied into the call's params, so Bedrock chat signed with the environment's credentials and called the default regional endpoint. Copy the AWS group the same way, resolving os.environ/ references. |
||
|
Nate Armstrong
|
0ac7a383db
|
feat(loadtest): chat completions parity check between the python proxy and the gateway (#47362)
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
* feat(gateway): full openai chat request translation for anthropic and bedrock (#47088)
* feat(gateway): share router cooldowns and follow proxy key invalidations through Redis
Gateway replicas now share state through the coordination Redis the
Python proxy uses (general_settings.coordination_redis, a redis
response cache, or REDIS_*):
- A deployment one replica cools down is written under Python's
deployment:<id>:cooldown key and applied by every other replica's
background poll (one MGET a second), so the fleet stops sending to a
failing deployment together. allowed_fails counts failures across
replicas, as Python's v1 rule does; the v2 failure-rate rule stays
per replica, as in Python. Deployments are named by model_info.id, or
an id derived from the config that is the same on every replica.
- The virtual key cache follows the proxy's
litellm_proxy.auth_cache_invalidation channel, so a key blocked,
deleted or edited through the admin API (or its team or user) is
re-read on every replica at once instead of after up to 60 s.
general_settings.user_api_key_cache_ttl sets the key TTL.
Nothing is added to the path of a successful request. Without Redis,
or while it is down, each replica keeps its own state as before.
Redis Cluster and Sentinel are refused at startup for now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): hooks and observers around inference requests, with usage and cost
Adds the litellm-gateway-hooks crate: request facts built only when something
reads them, on-path hooks that declare what they read and what they may
change and run in config order, off-path observers on bounded queues, a
usage tap for SSE and JSON responses, and per-deployment prices from the
model cost map overlaid by config prices. The messages and chat handlers run
through it when anything is installed; with nothing installed requests take
the old path.
The key store now also returns the key alias, user, team alias and key and
team metadata, and router attempts carry the deployment id.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): write spend to the proxy's database from a native observer
Adds a spend writer observer that records every gateway request in the
proxy's own Postgres tables with the same statements the Python proxy
uses: spend logs, key/user/team/team member/end user spend, and the
daily user, team and end-user activity tables. Writes are batched on the
proxy's timers and are additive, so replicas and Python proxies can
write side by side. The database pool is now shared between virtual
keys and the writer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): enforce key, team, user, member and end-user budgets on shared spend counters
Adds a built-in budgets hook that runs the proxy's budget checks, in its
order and with its messages, on the proxy's Redis spend:* counters. Each
request reads and seeds the counters, checks them and reserves its
maximum cost in one Redis script, then settles the reservation to its
real cost off the request path, so replicas and Python proxies sharing
the Redis enforce one budget. Without Redis the counters are kept per
replica. The key store now reads user, team member and end-user budgets,
and auth no longer checks budgets against stored spend.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters
Adds a built-in rate limits hook with the proxy's v3 limiter semantics:
requests per window, tokens per window and in-flight requests per key,
on the proxy's own {scope:id}:window|requests|tokens|max_parallel_requests
Redis keys and with its scripts, so replicas and Python proxies sharing
the Redis enforce one limit. Tokens are reserved from the proxy's
estimate and settled to usage when the response ends. Refusals carry the
proxy's message, body and headers on each endpoint, and complete
responses carry its x-ratelimit-* headers. Without Redis the counters are
kept per replica. The key store now reads key, user, team and member
rpm/tpm limits and the key's max_parallel_requests, and end users' budget
limits are read with their budgets; keys with plain rate limits are no
longer refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): configurable header, spend metadata and provider metadata hooks
Adds four built-in hooks a config lists under litellm_settings.hooks and
runs in that order: forward_headers copies named client headers to the
provider, strip_headers removes headers from the provider request
(including the gateway's default pass-through), spend_metadata adds
header and key-metadata values to the spend log's spend_logs_metadata,
and provider_metadata sets Messages body fields and headers from
templates over request facts. Hooks' upstream header edits now reach the
provider on both endpoints, and body edits can set nested fields.
Credentials and signer headers are never forwarded or set, and a
misconfigured hook stops the gateway at startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): model capacity hook, the proxy's dynamic rate limiter
Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3
(or model_capacity) in litellm_settings.callbacks. It caps each model
group at the sum of its deployments' rpm and tpm across every replica,
and with priority_reservation holds each priority's pool to its share
once the group's saturation reaches the threshold, counting it below.
It uses the proxy's model_saturation_check and priority_model keys and
scripts, its refusal messages and headers, and adds a successful
response's tokens to the group and the pool admission chose. The rate
limit counters gain a general check-and-add over request and token
pairs, and the throttling refusal shape is shared.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* test(gateway-auth): expect the narrowed unsupported rate limit message
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): prometheus observer with the proxy's metric and label names
Adds the prometheus observer, turned on by listing prometheus in
litellm_settings.callbacks, success_callback or failure_callback. Each
finished request updates the proxy's request, spend, token, latency and
deployment metrics under its names and labels, and /metrics serves them
after the gateway's own. prometheus_metrics_config, the exclude lists
and prometheus_latency_buckets are read as the proxy reads them. The
metrics the gateway already served under the same names come from the
observer alone when it is on. Outcomes now carry when the attempt was
sent upstream, for the LLM API latency and time to first token.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): s3_v2 observer writing the proxy's logging payload to S3
Adds the s3_v2 observer, turned on by listing s3_v2 (or s3_log) in
litellm_settings.callbacks, success_callback or failure_callback. Each
finished request becomes the proxy's StandardLoggingPayload, with the
messages from the request body and the response as the client got it,
streams put back together. It is queued and written every flush interval
under the proxy's key layout, with its Content-MD5, Content-Disposition
and encryption headers, one PUT per request or one NDJSON object per
folder with s3_batch_file_upload. s3_callback_params is read as the
proxy reads it, os.environ references included. Uploads are retried,
dropped and kept for the next flush by the proxy's rules, and what is
queued is written at shutdown. auth-aws gains an S3 signer that signs
the path as sent and every header, as botocore does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
* feat(gateway): full OpenAI chat request translation for Anthropic and Bedrock
Anthropic and Bedrock Converse deployments now take the whole OpenAI chat request the
Python proxy accepts instead of a narrow allowlist. Messages translate as Python builds
them: system as a string, parts or several messages, text, image_url (URL or data URL),
file and document parts, assistant tool_calls, the tool and function roles, consecutive
role merging, empty content, cache_control on Anthropic and cachePoint on Bedrock.
Params map per provider: max_tokens and max_completion_tokens with Anthropic's model
default, temperature, top_p, stop, tools and tool_choice, parallel_tool_calls,
response_format as a json tool or native output format, reasoning_effort and thinking,
user as metadata, with n and any other unsupported param refused with a 400 unless
drop_params is set globally, on the deployment or on the request, and
allowed_openai_params and additional_drop_params honoured. Responses carry tool calls,
thinking blocks and response_format answers back in OpenAI shape, streamed and not. The
route downloads media URLs a provider cannot take by reference, merges the beta headers
a request needs with forwarded ones, and the gateway hands the cost map to both
providers for the capability rules that read it.
* fix(rust-bridge): read Python's model_cost for chat translation capabilities
Chat translation for Anthropic and Bedrock reads capability flags from the
cost map. The Python bridge passed none, so every SDK chat call through the
Rust path failed. It now reads litellm.model_cost live, so lookups match
Python's, runtime-registered models included. ModelInfo::entry returns a Cow
so a map read from Python can hand over a copy.
Two tests sent temperature != 1 to a model that removed sampling params,
which Python itself refuses; they now send temperature=1.
* fix(model-catalog): read token counts the cost map writes as whole floats
The published cost map now has prompt_cache_min_tokens: 4096.0, which the
typed u64 fields rejected, so the gateway dropped the whole map at startup
and priced every request at zero. Counts accept any whole non-negative
number, as Python reads any JSON number, and a test keeps the checked-in
map parsing.
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough (#47074)
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming (#47077)
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(gateway): gemini chat completions on google ai studio and vertex ai (#47098)
* feat(gateway): Gemini chat completions on Google AI Studio and Vertex AI
gemini/ deployments and vertex_ai/ deployments of Gemini models now serve
/v1/chat/completions, streamed and not, instead of being refused. Both surfaces share
one translation ported from the Python VertexGeminiConfig. Params map as Python maps
them: temperature, top_p, max_tokens and max_completion_tokens, n, stop, penalties
where the model takes them, seed, logprobs, tools with function declarations and the
built in search, code execution, url context, maps and computer use tools,
tool_choice, response_format as a response schema, reasoning_effort and thinking as a
thinking config, modalities, audio and labels. Messages become contents with a system
instruction when the cost map allows it, images, files, audio, function calls and
responses, and thought signatures carried back on tool call ids
AI Studio sends the key in x-goog-api-key from the deployment or GOOGLE_API_KEY,
GEMINI_API_KEY or PALM_API_KEY, and calls v1alpha for Gemini 3 and v1beta otherwise.
Vertex AI calls the project and location URL with a forwarded bearer or a Google
access token from vertex_credentials, the environment or ADC, and grafts the path onto
a custom api_base. Responses carry text, reasoning, tool calls, grounding annotations,
logprobs and the vertex_ai metadata, a blocked prompt or flagged answer comes back as
content_filter, and usage counts thoughts and cached tokens as Python does so cost is
recorded. The gateway resolves os.environ references in the deployment key, base and
Vertex params, passes the deployment Vertex params to the call, and loads the cost map
for gemini and vertex_ai deployments. Claude and partner models on Vertex AI stay
refused
* test(gateway): gemini cost map test double returns a Cow entry
* feat(gateway): native chat for openai-compatible providers (#47099)
Serve chat completions for hosted_vllm, nvidia_nim, groq, together_ai,
fireworks_ai, xai, mistral, deepseek, openrouter, cerebras and perplexity
through the OpenAI passthrough, with each provider's Python config ported
as a request transform in llms and its response and stream fixups applied
on the way back.
Streams ask for stream_options.include_usage on the client's behalf where
the provider takes it, record the usage through ReportedUsage for the spend
hooks, and keep it from the client unless the client asked for it.
* feat(gateway): claude chat completions on vertex ai (#47101)
vertex_ai/ deployments of Claude models now serve /v1/chat/completions,
streamed and not, instead of being refused. The request is the Anthropic
chat translation built for the vertex_ai host, as Python's
VertexAIAnthropicConfig builds it on top of AnthropicConfig: capabilities
read the vertex_ai/ cost map entry first and native structured output
reads only that entry, service_tier and speed are refused or dropped with
drop_params, and the Anthropic API betas are not added. The body drops
model, carries anthropic_version vertex-2023-10-16, folds output_format
into output_config and keeps effort only for models that take it, and
names its context management and caller betas in anthropic_beta and the
anthropic-beta header, filtered to what Vertex accepts, with the web
search beta alone when there are none.
The call goes to the publisher rawPredict URL for the project and
location, regional or global, or a custom api_base or VERTEXAI_API_BASE,
with a Google access token from vertex_credentials, the environment or
ADC. Streams call streamRawPredict with stream true, alt=sse on a custom
base, decode the Anthropic events as the anthropic/ path does and hand
usage to the hooks through ReportedUsage. Every remote image is
downloaded first, since Vertex takes none by URL, and the response names
the requested model.
* fix(gateway): find anthropic stream usage by event name (#47206)
The stream meter kept a data line only when "message_start" or
"message_delta" appeared in its first 96 bytes. An upstream that puts
"type" after the message object (JSON key order is not fixed) had every
message_start skipped, so input tokens were recorded as 0 and spend,
budgets, rate limits, metrics and S3 logs undercounted.
Anthropic streams name every event, so the event: line now decides; the
byte-window check remains for streams without event lines.
* fix(gateway): stop hooks pinning connection read buffers for a stream's life (#47220)
The s3 logger asks for the response text, and the tap kept every
response chunk as a Bytes slice of the upstream connection's read
buffer, so each open stream held a buffer per chunk until it ended
(about 460 KB per open 10k-token stream locally). Chunks are now copied
into one owned buffer.
The same applies on the request side: the facts kept the raw request
slice and a clone of the client headers, both slices of the client
connection's read buffer. The body is now released once hooks have run
(an owned copy is kept only when an observer reads it) and the headers
are copied.
* feat(loadtest): full-feature mode with recorded-data verification (#47207)
* feat(loadtest): full-feature mode with recorded-data verification
genconfig -features full turns on what both gateways do per request:
owners with rpm/tpm/budgets/max_parallel, deployment capacity, the
dynamic_rate_limiter_v3, prometheus and s3_v2 callbacks, and the four
request hooks with Python stand-ins (custom_callbacks.py). -key-sets
gives each gateway its own keys, users and teams in a shared database.
verify checks spend logs, owner spend, daily tables, Prometheus counters,
S3 logs and the headers the provider received against what loadgen sent,
and compares per-deployment prices across sides. s3stub serves S3 for
the logs. mockllm takes in= for exact input tokens, returns unique
response ids, and counts chosen request headers; loadgen takes -keys
and -header.
* feat(loadtest): s3stub -max-string keeps long runs' logs in memory
* feat(loadtest): bifrost as a third side of the full-feature run (#47236)
genconfig writes a "bifrost" key set as a Bifrost config: a custom provider
per deployment, a routing rule per model group, virtual keys, and with
-features full key/team/customer budgets and rate limits, model configs for
deployment limits, the logs store with S3 payloads, the telemetry plugin and
header forwarding/stripping/logging. A pricing file copies the cost map's
prices per deployment.
mockllm answers Bedrock Converse (streamed and not), serves a self-signed
TLS listener for Bifrost's HTTPS-only Bedrock provider, and echoes the
request's model, which Bifrost prices by. s3stub answers HeadBucket.
loadgen takes -chat-path and -messages-path for Bifrost's
/anthropic/v1/messages.
verify reads a side named bifrost in Bifrost's schema and reports what it
has no counterpart for (users, members, daily tables, provider_metadata,
cross-replica state) as n/a; with three sides it compares every pair.
Claude-Session: https://claude.ai/code/session_01XF7rGf49TWYH5Hqkb14eSL
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(loadtest): chat completions parity check between the python proxy and the gateway
chatparity sends the same /v1/chat/completions requests through the Python
proxy and the Rust gateway, both loading one config pointed at its mock
provider, and diffs what each provider was sent (path, headers, body) and
what the client got back (status, body, the assembled stream, and the
stream chunk by chunk as information).
The mock answers OpenAI-compatible, Azure, Anthropic, Bedrock Converse
(event-stream framing), Gemini, Vertex Gemini and Vertex Claude the same
way for both proxies, and issues Vertex access tokens for a throwaway
service account. Cases cover 19 deployments and 15 request scenarios:
text, multi-turn, tools, tool results, response_format, images,
reasoning_effort, cache_control, unsupported params refused and dropped,
max_completion_tokens, and streams with and without include_usage.
scripts/chat-parity.sh builds and starts everything locally and writes
report.md plus one capture per case. Reviewed differences go in
cmd/chatparity/known.json with a reason; the run fails on the rest.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
Nate Armstrong
|
6f345e3eb9
|
feat(gateway): land the chat provider series and full-feature load test on the poc branch (#47388)
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
* feat(gateway): full openai chat request translation for anthropic and bedrock (#47088)
* feat(gateway): share router cooldowns and follow proxy key invalidations through Redis
Gateway replicas now share state through the coordination Redis the
Python proxy uses (general_settings.coordination_redis, a redis
response cache, or REDIS_*):
- A deployment one replica cools down is written under Python's
deployment:<id>:cooldown key and applied by every other replica's
background poll (one MGET a second), so the fleet stops sending to a
failing deployment together. allowed_fails counts failures across
replicas, as Python's v1 rule does; the v2 failure-rate rule stays
per replica, as in Python. Deployments are named by model_info.id, or
an id derived from the config that is the same on every replica.
- The virtual key cache follows the proxy's
litellm_proxy.auth_cache_invalidation channel, so a key blocked,
deleted or edited through the admin API (or its team or user) is
re-read on every replica at once instead of after up to 60 s.
general_settings.user_api_key_cache_ttl sets the key TTL.
Nothing is added to the path of a successful request. Without Redis,
or while it is down, each replica keeps its own state as before.
Redis Cluster and Sentinel are refused at startup for now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): hooks and observers around inference requests, with usage and cost
Adds the litellm-gateway-hooks crate: request facts built only when something
reads them, on-path hooks that declare what they read and what they may
change and run in config order, off-path observers on bounded queues, a
usage tap for SSE and JSON responses, and per-deployment prices from the
model cost map overlaid by config prices. The messages and chat handlers run
through it when anything is installed; with nothing installed requests take
the old path.
The key store now also returns the key alias, user, team alias and key and
team metadata, and router attempts carry the deployment id.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): write spend to the proxy's database from a native observer
Adds a spend writer observer that records every gateway request in the
proxy's own Postgres tables with the same statements the Python proxy
uses: spend logs, key/user/team/team member/end user spend, and the
daily user, team and end-user activity tables. Writes are batched on the
proxy's timers and are additive, so replicas and Python proxies can
write side by side. The database pool is now shared between virtual
keys and the writer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): enforce key, team, user, member and end-user budgets on shared spend counters
Adds a built-in budgets hook that runs the proxy's budget checks, in its
order and with its messages, on the proxy's Redis spend:* counters. Each
request reads and seeds the counters, checks them and reserves its
maximum cost in one Redis script, then settles the reservation to its
real cost off the request path, so replicas and Python proxies sharing
the Redis enforce one budget. Without Redis the counters are kept per
replica. The key store now reads user, team member and end-user budgets,
and auth no longer checks budgets against stored spend.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters
Adds a built-in rate limits hook with the proxy's v3 limiter semantics:
requests per window, tokens per window and in-flight requests per key,
on the proxy's own {scope:id}:window|requests|tokens|max_parallel_requests
Redis keys and with its scripts, so replicas and Python proxies sharing
the Redis enforce one limit. Tokens are reserved from the proxy's
estimate and settled to usage when the response ends. Refusals carry the
proxy's message, body and headers on each endpoint, and complete
responses carry its x-ratelimit-* headers. Without Redis the counters are
kept per replica. The key store now reads key, user, team and member
rpm/tpm limits and the key's max_parallel_requests, and end users' budget
limits are read with their budgets; keys with plain rate limits are no
longer refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): configurable header, spend metadata and provider metadata hooks
Adds four built-in hooks a config lists under litellm_settings.hooks and
runs in that order: forward_headers copies named client headers to the
provider, strip_headers removes headers from the provider request
(including the gateway's default pass-through), spend_metadata adds
header and key-metadata values to the spend log's spend_logs_metadata,
and provider_metadata sets Messages body fields and headers from
templates over request facts. Hooks' upstream header edits now reach the
provider on both endpoints, and body edits can set nested fields.
Credentials and signer headers are never forwarded or set, and a
misconfigured hook stops the gateway at startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): model capacity hook, the proxy's dynamic rate limiter
Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3
(or model_capacity) in litellm_settings.callbacks. It caps each model
group at the sum of its deployments' rpm and tpm across every replica,
and with priority_reservation holds each priority's pool to its share
once the group's saturation reaches the threshold, counting it below.
It uses the proxy's model_saturation_check and priority_model keys and
scripts, its refusal messages and headers, and adds a successful
response's tokens to the group and the pool admission chose. The rate
limit counters gain a general check-and-add over request and token
pairs, and the throttling refusal shape is shared.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* test(gateway-auth): expect the narrowed unsupported rate limit message
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): prometheus observer with the proxy's metric and label names
Adds the prometheus observer, turned on by listing prometheus in
litellm_settings.callbacks, success_callback or failure_callback. Each
finished request updates the proxy's request, spend, token, latency and
deployment metrics under its names and labels, and /metrics serves them
after the gateway's own. prometheus_metrics_config, the exclude lists
and prometheus_latency_buckets are read as the proxy reads them. The
metrics the gateway already served under the same names come from the
observer alone when it is on. Outcomes now carry when the attempt was
sent upstream, for the LLM API latency and time to first token.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): s3_v2 observer writing the proxy's logging payload to S3
Adds the s3_v2 observer, turned on by listing s3_v2 (or s3_log) in
litellm_settings.callbacks, success_callback or failure_callback. Each
finished request becomes the proxy's StandardLoggingPayload, with the
messages from the request body and the response as the client got it,
streams put back together. It is queued and written every flush interval
under the proxy's key layout, with its Content-MD5, Content-Disposition
and encryption headers, one PUT per request or one NDJSON object per
folder with s3_batch_file_upload. s3_callback_params is read as the
proxy reads it, os.environ references included. Uploads are retried,
dropped and kept for the next flush by the proxy's rules, and what is
queued is written at shutdown. auth-aws gains an S3 signer that signs
the path as sent and every header, as botocore does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
* feat(gateway): full OpenAI chat request translation for Anthropic and Bedrock
Anthropic and Bedrock Converse deployments now take the whole OpenAI chat request the
Python proxy accepts instead of a narrow allowlist. Messages translate as Python builds
them: system as a string, parts or several messages, text, image_url (URL or data URL),
file and document parts, assistant tool_calls, the tool and function roles, consecutive
role merging, empty content, cache_control on Anthropic and cachePoint on Bedrock.
Params map per provider: max_tokens and max_completion_tokens with Anthropic's model
default, temperature, top_p, stop, tools and tool_choice, parallel_tool_calls,
response_format as a json tool or native output format, reasoning_effort and thinking,
user as metadata, with n and any other unsupported param refused with a 400 unless
drop_params is set globally, on the deployment or on the request, and
allowed_openai_params and additional_drop_params honoured. Responses carry tool calls,
thinking blocks and response_format answers back in OpenAI shape, streamed and not. The
route downloads media URLs a provider cannot take by reference, merges the beta headers
a request needs with forwarded ones, and the gateway hands the cost map to both
providers for the capability rules that read it.
* fix(rust-bridge): read Python's model_cost for chat translation capabilities
Chat translation for Anthropic and Bedrock reads capability flags from the
cost map. The Python bridge passed none, so every SDK chat call through the
Rust path failed. It now reads litellm.model_cost live, so lookups match
Python's, runtime-registered models included. ModelInfo::entry returns a Cow
so a map read from Python can hand over a copy.
Two tests sent temperature != 1 to a model that removed sampling params,
which Python itself refuses; they now send temperature=1.
* fix(model-catalog): read token counts the cost map writes as whole floats
The published cost map now has prompt_cache_min_tokens: 4096.0, which the
typed u64 fields rejected, so the gateway dropped the whole map at startup
and priced every request at zero. Counts accept any whole non-negative
number, as Python reads any JSON number, and a test keeps the checked-in
map parsing.
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough (#47074)
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming (#47077)
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(gateway): gemini chat completions on google ai studio and vertex ai (#47098)
* feat(gateway): Gemini chat completions on Google AI Studio and Vertex AI
gemini/ deployments and vertex_ai/ deployments of Gemini models now serve
/v1/chat/completions, streamed and not, instead of being refused. Both surfaces share
one translation ported from the Python VertexGeminiConfig. Params map as Python maps
them: temperature, top_p, max_tokens and max_completion_tokens, n, stop, penalties
where the model takes them, seed, logprobs, tools with function declarations and the
built in search, code execution, url context, maps and computer use tools,
tool_choice, response_format as a response schema, reasoning_effort and thinking as a
thinking config, modalities, audio and labels. Messages become contents with a system
instruction when the cost map allows it, images, files, audio, function calls and
responses, and thought signatures carried back on tool call ids
AI Studio sends the key in x-goog-api-key from the deployment or GOOGLE_API_KEY,
GEMINI_API_KEY or PALM_API_KEY, and calls v1alpha for Gemini 3 and v1beta otherwise.
Vertex AI calls the project and location URL with a forwarded bearer or a Google
access token from vertex_credentials, the environment or ADC, and grafts the path onto
a custom api_base. Responses carry text, reasoning, tool calls, grounding annotations,
logprobs and the vertex_ai metadata, a blocked prompt or flagged answer comes back as
content_filter, and usage counts thoughts and cached tokens as Python does so cost is
recorded. The gateway resolves os.environ references in the deployment key, base and
Vertex params, passes the deployment Vertex params to the call, and loads the cost map
for gemini and vertex_ai deployments. Claude and partner models on Vertex AI stay
refused
* test(gateway): gemini cost map test double returns a Cow entry
* feat(gateway): native chat for openai-compatible providers (#47099)
Serve chat completions for hosted_vllm, nvidia_nim, groq, together_ai,
fireworks_ai, xai, mistral, deepseek, openrouter, cerebras and perplexity
through the OpenAI passthrough, with each provider's Python config ported
as a request transform in llms and its response and stream fixups applied
on the way back.
Streams ask for stream_options.include_usage on the client's behalf where
the provider takes it, record the usage through ReportedUsage for the spend
hooks, and keep it from the client unless the client asked for it.
* feat(gateway): claude chat completions on vertex ai (#47101)
vertex_ai/ deployments of Claude models now serve /v1/chat/completions,
streamed and not, instead of being refused. The request is the Anthropic
chat translation built for the vertex_ai host, as Python's
VertexAIAnthropicConfig builds it on top of AnthropicConfig: capabilities
read the vertex_ai/ cost map entry first and native structured output
reads only that entry, service_tier and speed are refused or dropped with
drop_params, and the Anthropic API betas are not added. The body drops
model, carries anthropic_version vertex-2023-10-16, folds output_format
into output_config and keeps effort only for models that take it, and
names its context management and caller betas in anthropic_beta and the
anthropic-beta header, filtered to what Vertex accepts, with the web
search beta alone when there are none.
The call goes to the publisher rawPredict URL for the project and
location, regional or global, or a custom api_base or VERTEXAI_API_BASE,
with a Google access token from vertex_credentials, the environment or
ADC. Streams call streamRawPredict with stream true, alt=sse on a custom
base, decode the Anthropic events as the anthropic/ path does and hand
usage to the hooks through ReportedUsage. Every remote image is
downloaded first, since Vertex takes none by URL, and the response names
the requested model.
* fix(gateway): find anthropic stream usage by event name (#47206)
The stream meter kept a data line only when "message_start" or
"message_delta" appeared in its first 96 bytes. An upstream that puts
"type" after the message object (JSON key order is not fixed) had every
message_start skipped, so input tokens were recorded as 0 and spend,
budgets, rate limits, metrics and S3 logs undercounted.
Anthropic streams name every event, so the event: line now decides; the
byte-window check remains for streams without event lines.
* fix(gateway): stop hooks pinning connection read buffers for a stream's life (#47220)
The s3 logger asks for the response text, and the tap kept every
response chunk as a Bytes slice of the upstream connection's read
buffer, so each open stream held a buffer per chunk until it ended
(about 460 KB per open 10k-token stream locally). Chunks are now copied
into one owned buffer.
The same applies on the request side: the facts kept the raw request
slice and a clone of the client headers, both slices of the client
connection's read buffer. The body is now released once hooks have run
(an owned copy is kept only when an observer reads it) and the headers
are copied.
* feat(loadtest): full-feature mode with recorded-data verification (#47207)
* feat(loadtest): full-feature mode with recorded-data verification
genconfig -features full turns on what both gateways do per request:
owners with rpm/tpm/budgets/max_parallel, deployment capacity, the
dynamic_rate_limiter_v3, prometheus and s3_v2 callbacks, and the four
request hooks with Python stand-ins (custom_callbacks.py). -key-sets
gives each gateway its own keys, users and teams in a shared database.
verify checks spend logs, owner spend, daily tables, Prometheus counters,
S3 logs and the headers the provider received against what loadgen sent,
and compares per-deployment prices across sides. s3stub serves S3 for
the logs. mockllm takes in= for exact input tokens, returns unique
response ids, and counts chosen request headers; loadgen takes -keys
and -header.
* feat(loadtest): s3stub -max-string keeps long runs' logs in memory
* feat(loadtest): bifrost as a third side of the full-feature run (#47236)
genconfig writes a "bifrost" key set as a Bifrost config: a custom provider
per deployment, a routing rule per model group, virtual keys, and with
-features full key/team/customer budgets and rate limits, model configs for
deployment limits, the logs store with S3 payloads, the telemetry plugin and
header forwarding/stripping/logging. A pricing file copies the cost map's
prices per deployment.
mockllm answers Bedrock Converse (streamed and not), serves a self-signed
TLS listener for Bifrost's HTTPS-only Bedrock provider, and echoes the
request's model, which Bifrost prices by. s3stub answers HeadBucket.
loadgen takes -chat-path and -messages-path for Bifrost's
/anthropic/v1/messages.
verify reads a side named bifrost in Bifrost's schema and reports what it
has no counterpart for (users, members, daily tables, provider_metadata,
cross-replica state) as n/a; with three sides it compares every pair.
Claude-Session: https://claude.ai/code/session_01XF7rGf49TWYH5Hqkb14eSL
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* build(deps): bump aiohttp to 3.14.5 to clear osv-scan (#47369)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Nate Armstrong
|
6dc59535e3
|
feat(gateway): Anthropic and Bedrock chat completions streaming (#47077)
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
|
||
|
Nate Armstrong
|
3bc3ce9536
|
feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough (#47074)
Adds azure/<deployment> chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/<base_model> when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client |
||
|
Nate Armstrong
|
2fa1d8e52b
|
feat(gateway): s3_v2 observer writing the proxy's logging payload to S3 (#47063)
Adds the s3_v2 observer, turned on by listing s3_v2 (or s3_log) in litellm_settings.callbacks, success_callback or failure_callback. Each finished request becomes the proxy's StandardLoggingPayload, with the messages from the request body and the response as the client got it, streams put back together. It is queued and written every flush interval under the proxy's key layout, with its Content-MD5, Content-Disposition and encryption headers, one PUT per request or one NDJSON object per folder with s3_batch_file_upload. s3_callback_params is read as the proxy reads it, os.environ references included. Uploads are retried, dropped and kept for the next flush by the proxy's rules, and what is queued is written at shutdown. auth-aws gains an S3 signer that signs the path as sent and every header, as botocore does. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Nate Armstrong
|
9688179335
|
feat(gateway): prometheus observer with the proxy's metric and label names (#47060)
Adds the prometheus observer, turned on by listing prometheus in litellm_settings.callbacks, success_callback or failure_callback. Each finished request updates the proxy's request, spend, token, latency and deployment metrics under its names and labels, and /metrics serves them after the gateway's own. prometheus_metrics_config, the exclude lists and prometheus_latency_buckets are read as the proxy reads them. The metrics the gateway already served under the same names come from the observer alone when it is on. Outcomes now carry when the attempt was sent upstream, for the LLM API latency and time to first token. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Nate Armstrong
|
16dacb02b7
|
feat(gateway): model capacity hook, the proxy's dynamic rate limiter (#47055)
Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3 (or model_capacity) in litellm_settings.callbacks. It caps each model group at the sum of its deployments' rpm and tpm across every replica, and with priority_reservation holds each priority's pool to its share once the group's saturation reaches the threshold, counting it below. It uses the proxy's model_saturation_check and priority_model keys and scripts, its refusal messages and headers, and adds a successful response's tokens to the group and the pool admission chose. The rate limit counters gain a general check-and-add over request and token pairs, and the throttling refusal shape is shared. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Nate Armstrong
|
40d4c8b2d8
|
feat(gateway): configurable header, spend metadata and provider metadata hooks (#47054)
Adds four built-in hooks a config lists under litellm_settings.hooks and runs in that order: forward_headers copies named client headers to the provider, strip_headers removes headers from the provider request (including the gateway's default pass-through), spend_metadata adds header and key-metadata values to the spend log's spend_logs_metadata, and provider_metadata sets Messages body fields and headers from templates over request facts. Hooks' upstream header edits now reach the provider on both endpoints, and body edits can set nested fields. Credentials and signer headers are never forwarded or set, and a misconfigured hook stops the gateway at startup. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Nate Armstrong
|
0919823385
|
feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters (#47050)
* feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters
Adds a built-in rate limits hook with the proxy's v3 limiter semantics:
requests per window, tokens per window and in-flight requests per key,
on the proxy's own {scope:id}:window|requests|tokens|max_parallel_requests
Redis keys and with its scripts, so replicas and Python proxies sharing
the Redis enforce one limit. Tokens are reserved from the proxy's
estimate and settled to usage when the response ends. Refusals carry the
proxy's message, body and headers on each endpoint, and complete
responses carry its x-ratelimit-* headers. Without Redis the counters are
kept per replica. The key store now reads key, user, team and member
rpm/tpm limits and the key's max_parallel_requests, and end users' budget
limits are read with their budgets; keys with plain rate limits are no
longer refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* test(gateway-auth): expect the narrowed unsupported rate limit message
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
Nate Armstrong
|
b83353906b
|
feat(gateway): enforce key, team, user, member and end-user budgets on shared spend counters (#47044)
Adds a built-in budgets hook that runs the proxy's budget checks, in its order and with its messages, on the proxy's Redis spend:* counters. Each request reads and seeds the counters, checks them and reserves its maximum cost in one Redis script, then settles the reservation to its real cost off the request path, so replicas and Python proxies sharing the Redis enforce one budget. Without Redis the counters are kept per replica. The key store now reads user, team member and end-user budgets, and auth no longer checks budgets against stored spend. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Nate Armstrong
|
2d2ca95481
|
feat(gateway): write spend to the proxy's database from a native observer (#47041)
Adds a spend writer observer that records every gateway request in the proxy's own Postgres tables with the same statements the Python proxy uses: spend logs, key/user/team/team member/end user spend, and the daily user, team and end-user activity tables. Writes are batched on the proxy's timers and are additive, so replicas and Python proxies can write side by side. The database pool is now shared between virtual keys and the writer. Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Nate Armstrong
|
e2691a7031
|
feat(gateway): hooks and observers around inference requests, with usage and cost (#47032)
* feat(gateway): hooks and observers around inference requests, with usage and cost Adds the litellm-gateway-hooks crate: request facts built only when something reads them, on-path hooks that declare what they read and what they may change and run in config order, off-path observers on bounded queues, a usage tap for SSE and JSON responses, and per-deployment prices from the model cost map overlaid by config prices. The messages and chat handlers run through it when anything is installed; with nothing installed requests take the old path. The key store now also returns the key alias, user, team alias and key and team metadata, and router attempts carry the deployment id. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * fix(model-catalog): read token counts the cost map writes as whole floats The published cost map now has prompt_cache_min_tokens: 4096.0, which the typed u64 fields rejected, so the gateway dropped the whole map at startup and priced every request at zero. Counts accept any whole non-negative number, as Python reads any JSON number, and a test keeps the checked-in map parsing. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Nate Armstrong
|
ee03f153bd
|
feat(gateway): share router cooldowns and follow proxy key invalidations through Redis (#47017)
Gateway replicas now share state through the coordination Redis the Python proxy uses (general_settings.coordination_redis, a redis response cache, or REDIS_*): - A deployment one replica cools down is written under Python's deployment:<id>:cooldown key and applied by every other replica's background poll (one MGET a second), so the fleet stops sending to a failing deployment together. allowed_fails counts failures across replicas, as Python's v1 rule does; the v2 failure-rate rule stays per replica, as in Python. Deployments are named by model_info.id, or an id derived from the config that is the same on every replica. - The virtual key cache follows the proxy's litellm_proxy.auth_cache_invalidation channel, so a key blocked, deleted or edited through the admin API (or its team or user) is re-read on every replica at once instead of after up to 60 s. general_settings.user_api_key_cache_ttl sets the key TTL. Nothing is added to the path of a successful request. Without Redis, or while it is down, each replica keeps its own state as before. Redis Cluster and Sentinel are refused at startup for now. |
||
|
Nate Armstrong
|
bf1e8cabe1
|
Merge main into litellm_customer_proxy
Some checks failed
LiteLLM Rust / rust-lint (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
LiteLLM Rust / rust-wheel (push) Waiting to run
Rust gateway load-test kit / vet, build, image (push) Has been cancelled
Rust gateway load-test kit / terraform fmt, validate (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
|
||
|
ryan-crabbe-berri
|
9aab6ec47b
|
fix(azure_ai): count Grok reasoning tokens in completion_tokens (#46969)
* fix(azure_ai): count Grok reasoning tokens in completion_tokens Grok on Azure AI Foundry reports reasoning_tokens outside completion_tokens, like xAI does, so azure_ai/grok-* calls billed only max(visible, reasoning) output tokens. Fold them with XAIChatConfig.fold_reasoning_tokens_into_completion in transform_response and in a new streaming iterator, matching the xai provider. Split out of #46377. * fix(azure_ai): fold reasoning tokens for every deployment name, streaming or not Non-streaming checked the deployment name for grok while streaming checked the chunk model field, so a Grok deployment named without grok was only folded when streaming. The fold already only fires when total_tokens equals prompt plus completion plus reasoning, so it now runs on every azure_ai response. |
||
|
jesus-berri
|
8efb99b4ac
|
feat(ui): shareable URL state for usage views and users-to-usage deeplinks (#43425)
* feat(ui): shareable URL state for usage views and users-to-usage deeplinks The Usage page now keeps the view, global view tab, date range and entity filters in the URL, so links can be bookmarked and shared. Relative presets are stored as range=7d and keep rolling, custom ranges as UTC from/to. Invalid params are dropped, and a link to a view or user the viewer cannot access shows a toast and falls back to the default view. Internal Users rows get a "View usage" action that opens usage filtered to that user, and the per-user usage views link back to the user's record. A "Copy Share Link" button sits next to "Ask AI" * fix(ui): clean usage links for sessions without the organizations query The organizations query only runs for premium sessions, and a disabled query stays pending forever, so the URL cleanup never ran for other sessions. Gate on isLoading instead, and give the access denied toast an id so a re-run effect shows it once * fix(ui): type-check usage entity filters and address review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): keep team usage available to internal users Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): format users table usage action Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): sync date picker preset with the current usage range Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): restore dashboard AGENTS.md from main and drop unused usage state Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): share custom usage date ranges as calendar days Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): keep per-user agent usage page across rerenders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
40efc6e07e
|
feat(cost-map): support s3:// URIs for LITELLM_MODEL_COST_MAP_URL (#46972)
* feat(cost-map): support s3:// URIs for LITELLM_MODEL_COST_MAP_URL * fix(cost-map): handle incomplete S3 ClientError responses * refactor(cost-map): drop S3 retry classification and s3_client argument * test(cost-map): cover non-object remote map fallback --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
53ef63145b
|
test(ui): wait for the logout navigation before checking the session is gone (#47008) | ||
|
yuneng-jiang
|
e9a8581a02
|
test(e2e): fix MCP tool search hit matching and the retired Bedrock image model (#46998)
* test(e2e): compare parsed MCP tool search hit names instead of raw text * test(e2e): move Bedrock image generation off the retired Nova Canvas model |
||
|
yuneng-jiang
|
910c30381b
|
test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001)
* test(integration): expect the tool search call replayed beside the function call for Anthropic targets * test(integration): wait for the team spend flush before killing a worker * test(integration): poll for the S3 attempt the batch sinks write concurrently * test(integration): queue the terminal object and its sibling into the same s3_v2 flush * test(integration): time the long Bedrock version against an equally long unversioned name * test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it * test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries |
||
|
berriai-litellm-provider-info-sync[bot]
|
b6ca57b36d
|
fix(bedrock): set supports_none_reasoning_effort on gpt-6-sol and gpt-6-luna (#47000)
Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
cc24e7cfa4
|
refactor(ui): validate model edit prices in the zod schema (#46978)
Each price field now parses its per 1M text into a per token number or null inside the schema, so the save handler gets clean values and the hand-rolled conversion goes away. Negative or non-numeric prices now show a field error instead of being saved |
||
|
Mateo Wang
|
b39eadcb82
|
feat(openai)!: honor per-deployment workload identity on non-chat routes (#39966)
* feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * feat(openai): honor per-deployment workload identity on non-chat routes Image generation, audio speech and transcription, moderation, text completions, files, batches, and fine-tuning built their OpenAI client from the env vars alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file failed with the api_key error on every route other than chat, Responses, and embeddings. Those handlers now build their client through the shared workload identity resolver, and the managed batch file-access credential extraction keeps the trio * fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys Non-chat handlers built a fresh SDK client per call, so files, batches, fine-tuning, text completion and sync moderation exchanged the OIDC token on every request. The builders now reuse the cached workload identity client per identity, base, timeout, retries and organization; static-key clients are still built fresh. A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no longer silently overrides a deployment's openai_identity_* params. An explicit distinct api_key on the deployment or call still wins. * fix(openai): keep shared workload identity clients on the process http session The workload identity client is cached and shared across requests, so it must never borrow the transport a single caller passed in. Static-key clients keep using the caller's http client; federated clients always ride the process-wide session. Also makes the http client helpers on BaseOpenAILLM public since the builder now needs them from outside the class. * test(openai): pin partial deployment identity beating the process key * test(openai): type the managed-object invoke callback * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals The httpx-based OpenAI handlers built their Authorization header from the deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file sent "Bearer None". They now resolve the bearer through resolve_openai_bearer_token, which exchanges the deployment's identity token the same way the SDK-based routes do. A deployment that carries only part of the identity trio now raises a 500 naming the missing field instead of silently falling back to the process key, and the SDK client builders take a factory for the static-key http client so the federated branch never touches litellm.client_session. * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * feat(openai): resolve per-deployment workload identity for the assistants routes OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * feat(openai): resolve per-deployment workload identity for image variations Build the image variations client through the shared workload identity builders instead of scraping locals(), and give the async path the async http session it was missing. Let a federation trio serve the model-less batch fallback, and hold the standard LRU size of exchanged tokens. * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * style(tests): keep two blank lines before the batch_cost_is_final block * test(batches): type the workload identity fallback test fixtures * fix(moderation): stop validating unrelated kwargs on the sync workload identity path * fix(auth): refuse server-owned workload identity keys in array form and inside a client router config * fix(openai): keep a deployment's own key ahead of workload identity on every route Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation, image generation, and assistants so a deployment key equal to the process key still wins. A partial identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and /v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments * fix(containers): pass deployment params to container file auth headers Container file routes built their auth headers without the deployment's litellm_params, so a keyless deployment with an OpenAI workload identity sent Bearer None on upload, list, retrieve, delete and content calls * fix(openai): keep process-wide key precedence and move token exchange off the event loop Restore the merge base order where an explicit or process-wide OpenAI key wins over federation, so exporting OPENAI_API_KEY keeps working the way it did before. Run the header-building validate_environment calls (image edit, videos, containers, vector stores, vector store files) and the decisions and System One request preparation in a worker thread, so a slow token exchange no longer stalls the loop. Federate realtime websocket sessions, live sessions, realtime client secrets and transcription sessions, and the OpenAI moderation guardrail when no key is set. * fix(openai): answer 401 for a partial identity trio and federate realtime health checks A deployment with part of the workload identity trio and no static key now fails as an auth error naming the missing params, so the router does not retry it as a server error. The realtime health check sends the exchanged bearer for a keyless federated OpenAI deployment, and API token counting reads the same static keys as every other route before minting. * fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config * fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting * test(integration): refuse caller-chosen identity params in user_config and bracketed form keys * fix(openai): send the honored secret manager key from SDK client builders A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key on image, assistants, files, batches, and other SDK-built routes instead of failing with no api_key, matching what the workload identity resolver already treats as a static key * test(openai): check the token exchange runs off the event loop instead of timing the loop --------- Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
9cf09c18fe
|
fix(providers): route Azure Responses past dated api versions, count Grok reasoning tokens on azure_ai, refresh provider sets after register_model, fix gpt-6.1-sol Azure rows (#46377)
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version * fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider * fix(utils): refresh provider model sets after register_model * fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row * test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version * fix(utils): keep router deployment ids out of provider model sets register_model now refreshes the provider sets, so the per-request and router registrations keyed by a deployment id put those opaque ids into wildcard model listings. Those callers now pass add_to_provider_model_sets=False. * fix(azure): carry responses_api_version through the chat to Responses bridge completion() dropped responses_api_version from the litellm_params it hands the Responses bridge, so azure/responses/<model> on a dated chat api_version still built the 404ing /openai/responses URL. Also cover the auto-router marker never forwarding the new param. * fix(azure): stop listing a wrong set of reasoning efforts when none is rejected The message said only low, medium and high are supported, which is wrong for models that also take xhigh such as gpt-6.1-sol. * test(azure): use the latest grok and gpt-5.4 rows in the new azure tests * fix(proxy): list each wildcard-expanded model once A provider set can hold a model both bare and provider-prefixed, which now happens after any priced request registers its openai/<model> key, so the openai/* listing showed that model twice. * fix(azure): route Responses past dated GA api_versions by default A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before * chore(azure_ai): mark the grok chunk_parser dict as an override contract * fix(azure): rewrite only the route's /openai segment when moving a call to v1 The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses. Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too, so the rewrite now only touches the last /openai path segment. The Responses version resolver no longer duplicates the builder's default for an unset api_version. Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite now has one test for the dated GA default and one where only responses_api_version can make the call work * test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params * fix(utils): list ai21 chat and vertex code models under their provider wildcard register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too. * fix(main): keep caller-chosen model names out of provider wildcard listings --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
yuneng-jiang
|
0403f8953e
|
fix(cost): bill bridged /v1/messages Responses replies like native /v1/responses for reasoning tokens and service tier (#46975)
* fix(cost): price reasoning tokens of /v1/messages bridged onto the Responses API in the cost header and breakdown * fix(cost): record the bridged Responses API result only after the served body is built * fix(cost): keep the served service tier when a bridged Responses reply has no message item |
||
|
litellm-stampbot
|
1c317e3e61
|
feat(cost-map): add baseten and nebius rows for Reflection Beam 1.0 (#46991)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
berriai-litellm-provider-info-sync[bot]
|
87f959798d
|
fix(bedrock): backfill Priority and Batch prices from the AWS offer index (#46987)
Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
29cd11bec6
|
fix(sso): concurrent first sign-ins land on the winner's account, worker-kill test stops waiting on the reap (#46855)
* fix(sso): concurrent first sign-ins land on the account the winner created, worker-kill test pins half its streams to each worker * test(integration): a worker kill returns once the worker is dead, the reap waits at block exit uvicorn can take a full worker healthcheck to reap a worker that died mid-ping, so kill() held the caller that long and peers held on the surviving worker hit their client timeout * fix(sso): a sign-in that loses the first-insert race joins the default teams The losing sign-in could read the winner's row before new_user added the default teams, so its CLI key was minted with no team. The race tests now run the real new_user against a fake user table |
||
|
Mateo Wang
|
d76ec3a47e
|
fix(proxy)!: answer 400 for malformed tools on count_tokens instead of crashing (#46583)
* fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing A non-list tools value crashed the vector store lookup during auth and the built-in tool cost lookup in the failure logging path, so the proxy answered an unhandled 500 and closed the keep-alive connection. count_tokens now answers a 400 invalid_request_error for a malformed body, drops malformed tools when drop_params is on, and both lookups skip values of the wrong shape. * fix(proxy): skip non-list tools in the tool allowlist check Also validate the count_tokens body without recursion, which the recursion detector rejects. * test(integration): restore the int tools case on count_tokens The proxy now answers a non-list tools value with a 400, so the case is deterministic again. * fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids * fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids Check tools after request setup so the Claude Code default and the logged call type apply, read the request, global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off * fix(proxy): keep the request's call type when logging a gate rejection after request setup The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses rejection as aembedding, so it now only applies when the logging object carries no known call type * fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry * fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket * refactor(proxy): leave the responses websocket out of the request-level tool check * fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections * fix(proxy): read router default drop lists and deployment ids in the malformed tools check * fix(proxy): leave malformed tools to the router when it can pick another deployment Fallbacks, auto-routers, model strings, comma batches and team model names let the router pick a deployment after the proxy checks the request, so the proxy now forwards a malformed tools field there as main does, and only rejects it when the deployment it checks is the one that serves. Token counting never reroutes, so count_tokens and input_tokens keep reading the group they count with. additional_drop_params names are read the way core reads them, so a list with a non-string entry still names its strings. * test(integration): cover malformed tools on reroutable models, router defaults and deployment ids * fix(proxy): resolve malformed tools at the deployment the router picks * fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells. * test(integration): expect a 400 for an empty-string tools field on /v1/messages Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test * fix(router): keep web search routing for tools passed as a tuple * fix(router): keep a deployment's own tools when drop_params drops a malformed request tools * test(router): list _merged_tools as covered through the deployment tools merge tests * fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them * fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError * fix(prometheus): keep deployment health metrics clean on a malformed tools rejection * test: type the new malformed tools tests * test: mark new test locals Final and type kwargs dicts |
||
|
Mateo Wang
|
57e69ce713
|
docs(agents): keep directory rules in nested AGENTS.md and pin integration tests to CircleCI (#46796)
* docs(agents): keep directory rules in nested AGENTS.md and pin integration tests to CircleCI * docs(agents): scope the CircleCI rule to the files it covers and ask agents to read nested AGENTS.md * ci(coverage): reject GITHUB_FILES integration contracts run from a test-unit.yml shard * ci(coverage): report integration ownership findings under their own header * ci(coverage): flag GITHUB_FILES contracts run from any non-dedicated workflow * docs(integration): say a pre-merge run is no reason to route an integration file to GitHub * ci(coverage): tie each dedicated integration workflow to the directory it owns |
||
|
ryan-crabbe-berri
|
0dac61a555
|
fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338)
* fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open GuardrailToolParam required a type key, so Gemini native tools (function_declarations, googleSearch) failed validation inside the hook and either returned a 500 or skipped the scan when fail_on_error was false. The type key is now optional and left out of the forwarded tool when absent, so an echoing guardrail server cannot write type: null into the provider request. structured_messages was re-validated against the chat completion message types, which silently emptied tool_result, tool_use, image and document content blocks to content: [] and dropped undeclared keys before the guardrail saw them. The request model now forwards rows as opaque mappings, which also retires the write-back shim that restored the original rows for echoed-back entries. The fail-open critical log line now carries error_type so a timeout, a refused connection and an upstream HTTP status can be told apart without the traceback. * fix(httpx): retry aiohttp server disconnects the way httpx does aiohttp raises ServerDisconnectedError when a kept-alive connection is closed by the peer before a response arrives, which is what a load balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces. The transport mapped it to httpx.ReadError, which the handler's connection-error retry does not cover, so the request failed instead of being resent on a fresh connection. It now maps to httpx.RemoteProtocolError, the error httpx itself raises for the same condition, so the existing retry fires. * test(httpx): build the stub server reply in one bytes literal * fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable The connection-error retry closed its fresh client before a streamed body was read, so an aiohttp disconnect now billed a second upstream request and still failed. It also skipped masking on a status error from the resend, resent PATCH and DELETE as POST, dropped multipart files and ignored the handler's ssl_verify. The retry now resends the request the first attempt built, on a client with the handler's TLS settings, closes that client when a streamed body is closed, and masks a status error the same way the first attempt does. * fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type * fix(guardrails): resend a generic guardrail check cut before any response A load balancer that closes an idle keepalive connection made the guardrail POST fail with "Server disconnected", because #46336 only resends POST on ConnectError. AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to False, that gives a RemoteProtocolError the same single fresh-client resend PUT and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a guardrail check is a pure verdict over the request. Every other POST keeps the #46336 rule The fail-open log line now puts error_type at the end so the pre-PR text stays contiguous for existing alert matchers test_rows_reach_the_guardrail_verbatim now reads the body off a real local guardrail server instead of mock call args * fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration A ServerDisconnectedError raised while reading a response body now stays httpx.ReadError, as it was before this PR. Only a disconnect before any response maps to RemoteProtocolError, which is the case the resend rule is about. Stream readers that end on ReadError, like the SAP stream iterator, keep their old behavior The loopback-server disconnect, TLS and guardrail resend tests move to tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is in-process only. Unit keeps in-process MockTransport checks for the idempotent POST resend, masking on the resend, streamed and buffered client closing, the guardrail resend, and the guardrail receiving rows verbatim * fix(guardrails): resend a reset guardrail check and keep typed echoes on texts An idempotent POST now resends once on any network error after connect (ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load balancer that answers a stale keepalive with a TCP reset rides it out too. That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes back to ReadError. This stops PUT and DELETE from stacking litellm's resend on top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil Guard from replaying its analyze POST on a disconnect Every resend now runs inside the method's own error mapping, so a timeout on the resend is a litellm.Timeout like the first attempt and a status error is masked once by the caller A guardrail server that parses rows with its own message types and echoes them back without the keys those types lack (annotations, tool_calls[].index) is read as an echo again, so its texts redaction reaches the LLM. A row only counts as an echo when no value changed and every key the chat completion types declare is kept The wire tests for the PUT/DELETE resend, its masking and its TLS settings now run on the httpx transport, where litellm's resend is the only retry, and the POST and guardrail tests cover both transports and both cut styles * test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head * fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement * refactor(guardrails): move the dropped-connection guardrail resend out of this PR The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line * test(guardrails): pick the closed guardrail port from the integration port allocator Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port * feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
3637dd711f
|
feat(ui): add Authorization URL, Token URL and Scopes fields for OAuth Delegate and True Passthrough MCP servers (#46941)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
6530b329f4
|
fix(proxy): give /team/delete transactions a 30s timeout (#46979)
Both interactive transactions in /team/delete used Prisma's 5s default. A lock wait or a short database stall past 5s expired the transaction after the keys were already deleted, so the call returned 503 and left the team row, memberships and team models behind |
||
|
yuneng-jiang
|
ac6300f380
|
fix(mcp): stop proxy-started legacy listeners from consuming server rpm (#46968)
* fix(mcp): stop proxy-started legacy listeners from consuming server rpm The listener the proxy opens on notifications/initialized for legacy MCP sessions ran the server rate-limit admission once per target, so every session spent an rpm slot before the client sent anything. Admission is now injected into the listener and applied only to client-initiated subscriptions (subscriptions/listen and resources/subscribe). Authorization and its periodic recheck still run on every path. * test(mcp): key the fake admission window by caller and server * fix(mcp): stop subscription catalog refreshes from consuming server rpm A subscription listener lists the caller's catalog to diff list_changed events, at setup and on every upstream event. Those listings went through the normal per-server admission, so against an upstream that declares listChanged each legacy session spent several rpm slots before the client asked for anything. The listener now marks its targets as already admitted for its own lifetime, so its refreshes reuse the subscription's admission. Client requests on the session are still metered. * test(mcp): exercise subscription admission through the proxy rate limiter |
||
|
devin-ai-integration[bot]
|
8bb8ac7f18
|
fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure (#45136)
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first. * fix(proxy): log a provider error frame through an attempt-scoped logging copy The error-frame failure stashed the superseded attempt's usage on the logging object the retry shares, so the retry's success log carried the failed attempt's token counts. The failure now logs through a copy of the logging object with its own model_call_details, and the helper moves into the Anthropic passthrough logging handler so the proxy streaming handler only routes by endpoint type. * fix(proxy): scope the error-frame failure log to streams closed at the frame A stream that ran to its end at the provider's error frame kept logging through the success route before this PR, and the attempt-scoped failure log is now taken only for a stream the router closed at that frame, with the copy made at the close so the retry's deployment never reaches it * chore: drop duplicate litellm import and mark the stream_ended rebind * fix(proxy): log only router-superseded attempts as error-frame failures A client that hangs up at a forwarded error frame closed the stream the same way the router closes a superseded attempt, so a non-retried stream lost its spend. The router now closes a superseded attempt through aclose_superseded_attempt, and only that close routes the attempt to the failure log * test: type the _settle helper's recorded events --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
ee6ca049b3
|
build(deps): bump claude-agent-sdk to 0.2.165 to unblock osv-scan (#46971)
OSV started flagging claude-agent-sdk 0.1.44 today, which turns OSV Scan red on every PR. 0.2.165 is the newest release outside the 3-day uv cooldown. The package sits only in the ci dependency group and is only imported by the cookbook |
||
|
yuneng-jiang
|
5bcced0187
|
feat(ui): redesign the playground chat around a single composer and inspector (#46934)
* feat(ui): add clean-slate playground chat prototype
* feat(ui): replace the playground chat with the redesigned composer and inspector
* fix(ui): close the playground chat gaps found in self-review
Resend only complete text exchanges, so empty replies, MCP direct calls and
transcriptions no longer leak into chat history. A stopped request can no longer
append output, attach its gateway trace to a newer reply or clear the newer
request's loading state. MCP selections are only pruned once both the server and
toolset lists load. The MCP tool picker lists toolsets again and calls the server
that backs the chosen tool, and a saved model the current key cannot use is
cleared when the model list loads, matching the old chat
Also vendors the scroll-fade-x utility the attachment primitive uses
* fix(ui): resolve playground toolset tools per server and drop late session ids
A toolset now lists only its own tools, each resolved on the server the toolset
names, so the args form and the call always target the same tool. A saved
toolset no longer asks for tools with its own id while the toolset list loads,
and a failed tool list says so instead of claiming the server has no tools
A stopped request can no longer overwrite the Responses session id, starting a
new chat drops a request that is still running, embeddings replies are no longer
resent as chat history, and a saved model missing from an already cached model
list is cleared too. A failing toolset list falls back right away instead of
retrying while the MCP pickers show loading
* fix(ui): guard playground sends and label inspector stats
Pressing Enter twice no longer sends the same message twice, an attachment the
new endpoint cannot take is dropped when the endpoint changes, a stopped
Responses request can no longer show its code output on a newer reply, and a
reply that broke mid-stream is no longer resent as history
The inspector's speed, cost and routing stats get the same icons as the reply
footer, and the footer's metrics now line up with its copy and Inspect buttons
* fix(ui): write playground key settings only through the encoded storage path
Key source, virtual key and custom headers now go through their own write loop, separate from the plain settings, so a key value can never take the plain sessionStorage branch
* fix(ui): restore playground BYOK connect and reconnect
Picking a BYOK server for a direct tool call now shows a key button next to the
server picker, so the key can be connected without leaving the MCP endpoint. A
connected BYOK server gets a Replace key action in the MCP dialog, also shown
when its tools fail to load, and saving a key reloads that server's tool list
* revert: restore the single playground settings storage loop
Reverts
|