The Lua ARGV layout for `CHECK_AND_INCREMENT_BY_N_SCRIPT` documents two
distinct slots — `ttl_seconds` (counter-key TTL) and `window_size_seconds`
(sliding-window length) — but the Python call site collapsed both to
`int(window_size)`. They happen to be equal today, so the bug was latent,
but a future descriptor carrying a custom counter TTL would have been
silently mis-applied (counter key expiring at the wrong time relative to
the window).
Make the two values separate variables (`ttl_seconds`, `window_size_seconds`)
before the `script_args.extend([...])` call so the ARGV layout matches the
documented contract by construction. The values still flow through to
`per_counter_meta` so the in-memory fallback uses the same source of truth.
Also:
- Document the COUNTER-level (not descriptor-level) parallel-indexing
invariant on `_build_atomic_response` and `per_counter_meta`. A descriptor
with both RPM and TPM emits two counters and two meta entries; the Lua
script's `i` enumerates counters and the over-limit return tuple's index
maps directly to `per_counter_meta[i - 1]`. Comment surfaces this so a
future contributor doesn't refactor the arrays apart.
- Annotate the `increments` literal in the batch limiter directly instead
of routing through `cast(...)`. Drops an unneeded `cast` import.
- Comment on the dynamic limiter's tracking-only branch explaining why
`should_rate_limit` (non-atomic) is intentional there: we want to bump
the priority counter so future enforced windows have accurate usage,
and the OVER_LIMIT response is intentionally ignored.
Tests: 65 passed (1 skipped), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
If atomic_check_and_increment_by_n returns overall_code=OVER_LIMIT but no
status entry matches a descriptor key the dynamic limiter dispatcher knows
how to translate into a 429 (`model_saturation_check` or `priority_model`),
the for-loop previously exited cleanly and execution fell through to the
priority-tracking increment + the data["litellm_proxy_rate_limit_response"]
write — silently admitting an over-limit request.
This is the fail-open path a future contributor would hit by wiring a new
descriptor type into enforced_descriptors without updating the dispatcher.
Refuse the request with a generic 429 carrying the offending descriptor
metadata so the operator can see what slipped past, and emit an error log
to surface the wiring gap.
Adds a regression test (test_dynamic_rate_limiter_v3_fails_closed_on_unknown_descriptor)
that drives the limiter with a synthetic OVER_LIMIT response carrying an
unrecognized descriptor_key and asserts a 429 is raised.
Tests: 65 passed (1 skipped), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Lua script now reads time via redis.call('TIME') instead of a client-supplied
timestamp. Prevents window-reset divergence across replicas with skewed
wall-clocks, which could otherwise reopen the cross-replica TOCTOU window.
- Per-descriptor window_size is now plumbed through both the Lua ARGV layout
and the in-memory fallback. Previously the in-memory path used the global
self.window_size while Lua honored the per-descriptor override, so a
descriptor with a custom window would be enforced inconsistently between
Redis-available and Redis-unavailable code paths.
- Lua-failure fallback path now logs at error severity and explicitly
documents the in-memory ↔ Redis state divergence risk so operators can
alert on it. Prior `warning` log understated the impact.
- Coarse-granularity lock is now documented inline with the conditions under
which a per-descriptor sharded lock would be worth introducing.
- New regression test: zero-token batch consumes RPM only and is properly
capped by the RPM ceiling (validates the asymmetric quota path that arises
from `inc_amount <= 0: continue`).
Tests: 64 passed (1 skipped), 0 regressions. Multi-instance Redis loadtest
re-verified: chat 20/80 success @ RPM=20, batches 3/20 @ TPM=200.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix for the TOCTOU bypass relied on a per-instance asyncio.Lock,
which closed the window only within a single proxy worker. Multi-replica
deployments still raced across processes — A and B both read counter=99,
both passed validation, both incremented to 100/100 → effective limit doubled.
Add `CHECK_AND_INCREMENT_BY_N_SCRIPT` Lua script that processes any number of
(window_key, counter_key, limit, increment, ttl) descriptors atomically with
all-or-nothing semantics: if any descriptor would exceed its limit, no counter
is modified and the script returns OVER_LIMIT with the offending descriptor's
state. When Redis isn't configured, the in-memory fallback uses the existing
asyncio.Lock for single-process atomicity.
Expose this as `_PROXY_MaxParallelRequestsHandler_v3.atomic_check_and_increment_by_n`
and rewire both call sites:
- batch_rate_limiter._check_and_increment_batch_counters: replace the
read_only=True check + separate async_increment_tokens_with_ttl_preservation
with a single atomic call passing the batch's (request_count, total_tokens)
as the increment.
- dynamic_rate_limiter_v3._check_rate_limits: bundle model_saturation_check
(always enforced) and priority_model (enforced only when saturated) into
one atomic call. When priority is unenforced, increment its counter via
the existing should_rate_limit(read_only=False) path for tracking only.
Update structural regression tests to assert the new atomic path is used
rather than the legacy two-phase pattern.
Tests: 4/4 TOCTOU tests pass, 59 existing rate-limiter tests pass, no
regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The batch rate limiter (`_check_and_increment_batch_counters`) and the
dynamic rate limiter (`_check_rate_limits`) implemented rate limiting in
two disjoint awaits: a `should_rate_limit(read_only=True)` check followed
by a separate increment. Concurrent requests could all observe the same
pre-increment state, all pass enforcement, and all then increment —
multiplying the effective quota by the concurrency level.
Demonstrated bypass (see new test):
- Batch: 5 concurrent batches of 40 tokens each against TPM=100 consumed
200 tokens (100% over).
- Dynamic: 5 concurrent priority="high" requests against RPM=2 all
passed Phase 1 + Phase 3.
Wrap both critical sections in a per-instance asyncio.Lock so the read
and increment execute atomically within a process. Multi-replica
deployments still rely on Redis Lua atomicity for cross-process safety;
that is a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Unify cost calc in success_handler dict and typed branches
* Trim verbose comments and docstrings
---------
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local>
Claude 3.5 Sonnet v2 reached EOL on Bedrock 2026-03-01, returning the same
404 EOL error as 3.7 Sonnet. Sonnet 4.5 supports both InvokeModel and
Converse APIs on Bedrock, so use the same model for both routes.
AWS Bedrock has reached end-of-life for `claude-3-7-sonnet-20250219-v1:0`,
returning 404s with "This model version has reached the end of its life."
Update test references to `claude-sonnet-4-5-20250929-v1:0` (same capability
surface: thinking, tools, prompt caching, PDF input, vision, computer use).
The bedrock/invoke pass-through tests stay on Sonnet 3.5 since Sonnet 4.5
is converse-only on Bedrock.
Companion to the prior commit. process_items only converted empty
`items: {}` to `{"type": "object"}`. But anyOf branches like
`{"type": "array"}` (no items field at all) were untouched, so after
convert_anyof_null_to_nullable stripped the null branch and added
nullable, the array branch was sent to Vertex as
`{"type": "array", "nullable": true}` — which Vertex rejects with
INVALID_ARGUMENT (`any_of[0].items: missing field`).
Make process_items synthesize `items: {"type": "object"}` for any
`type == "array"` schema where items is missing or empty.
Also:
- Convert test_gemini_tool_calling_working_demo to a hermetic mock
test asserting items is present on the array branch in the sent
body. Was previously a real-network call to Vertex and was the
test the user reported still failing in CI.
- Add unit test test_build_vertex_schema_array_branch_missing_items_in_anyof
covering the missing-items shape directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cache provider config lookups for Vertex Anthropic messages so repeated requests reuse the same config object and preserve credential cache state. Add a regression test to catch any future loss of config reuse.
Made-with: Cursor
* fix(caching): preserve prompt_tokens_details through embedding cache round-trip
The embedding caching layer was dropping prompt_tokens_details (including
image_count) because CachedEmbedding had no field for usage metadata and
the cache retrieval code reconstructed Usage without it. This caused
inconsistent responses where the first call returned image_count but
cached responses did not, breaking cost tracking for multimodal embeddings.
Add prompt_tokens_details to CachedEmbedding, persist per-item details
during cache storage, aggregate them on retrieval, and merge them in
combine_usage() for partial cache hits.
* style: apply Black formatting to caching files
* fix(caching): address Greptile review — cyclic import, guarded construction, nested dict merge
Move PromptTokensDetailsWrapper to inline import to resolve CodeQL cyclic
import warning. Guard PromptTokensDetailsWrapper construction with
try/except to handle unexpected cached keys. Add recursive dict merging
in _merge_prompt_tokens_details for nested fields like
cache_creation_token_details.
convert_anyof_null_to_nullable was stripping the items field from array
branches inside anyOf when a sibling null branch was present, leaving
{"type": "array"} without items. Vertex requires items whenever
type == "array" (even inside anyOf) and rejects the call with
INVALID_ARGUMENT.
Leave the (possibly empty) items in place so the downstream process_items
step can convert {} to {"type": "object"}, which is what Vertex wants.
Also:
- Update test_build_vertex_schema expected output, which was codifying
the broken shape.
- Convert test_gemini_tool_calling_not_working to a hermetic mock test
that asserts the request body sent to Vertex includes items inside
the callbacks anyOf array branch. The previous form made a real
network call and was flaky in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vi.clearAllMocks does not reset mockImplementation, so the error-notification
test was inadvertently relying on a deleteField stub set up in earlier tests
and would time out when run in isolation.
Previously, useStoreRequestInSpendLogs and useDeleteProxyConfigField
did not refresh the proxyConfig cache on success, so the Logging
Settings form continued to render the pre-save values until React
Query refetched on its own. Wire both hooks to invalidate
proxyConfigKeys on success so any active observer (currently the
Logging Settings page) repulls fresh data.
Export proxyConfigKeys for cross-hook reuse.
Switch the spend-logs save flow from mutateAsync + try/catch to
mutate + callbacks. Errors now surface through a single onError path
(no more double toast on failure), and the delete-then-update sequencing
runs through onSettled instead of awaited promises. handleFormSubmit is
no longer async.
Tighten the corresponding test to assert exactly one error toast fires.
Adds a CLI flag (`--timeout_worker_healthcheck`, env `TIMEOUT_WORKER_HEALTHCHECK`)
that forwards to uvicorn's `timeout_worker_healthcheck` Config kwarg (added in
uvicorn 0.37.0). Lets operators raise the supervisor's worker-ping timeout above
the default 5s when triaging workers being killed and respawned under load.
The helper introspects `uvicorn.Config.__init__` and only sets the kwarg if
supported, otherwise prints a warning - so the existing uvicorn>=0.32.1,<1.0.0
floor pin is unaffected. Gunicorn and Hypercorn paths are unchanged (the uvicorn
supervisor isn't running there); the value is also not passed to the helper at
all on those paths so the "uvicorn too old" warning never fires spuriously.
* Use auth key name if there are no app id in in headers or in extra_data
* use key alias instead of key name
* Fix
* last priority key alias
* Fix
* Add tests
* [Feat] Day-0 support for GPT-5.5 and GPT-5.5 Pro (#26449)
* feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro
Add pricing + capability entries for the new GPT-5.5 family launched by
OpenAI on 2026-04-24:
- gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M
input/output/cached input
- gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6
per 1M input/output/cached input
Other fees (long-context >272k, flex, batches, priority, cache
discounts) follow the same ratios as GPT-5.4, with context window
retained at 1.05M input / 128K output.
No transformation / classifier code changes are required:
OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via
numeric version parsing, and model registration is driven from the
JSON. The existing responses-API bridge for tools + reasoning_effort
(litellm/main.py:970) already covers gpt-5.5-pro.
Tests:
- GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants
- New test_generic_cost_per_token_gpt55_pro cost-calc test
- Updated test_generic_cost_per_token_gpt55 for long-context fields
* fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants
gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the
supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and
supports_minimal_reasoning_effort flags that their non-dated
counterparts define. Reasoning-effort routing in OpenAIGPT5Config is
fully capability-driven from these JSON flags — since an absent flag
is treated as False for opt-in levels (xhigh), users pinning to a
dated snapshot would silently lose xhigh support and diverge from the
base alias on logprobs + flexible temperature handling.
Copy the flags onto both dated variants so every dated snapshot
inherits the base model's reasoning-effort capability profile.
Adds a parametrized regression test that asserts
supports_{none,minimal,xhigh}_reasoning_effort parity between each
dated variant and its non-dated counterpart, preventing future drift
when new snapshots are added.
* [Feat] Add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants) (#26361)
* feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants)
Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet
shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page
is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the
established precedent for azure/gpt-5.4* (which were in the cost map
before the Azure rollout) so cost tracking and capability flags work
the moment customers deploy.
Schema follows the existing azure/gpt-5.4* shape:
- Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat,
$60/$360 pro per 1M, with priority tier 2x base
- Azure variants drop the flex/batches keys (Azure has no flex tier)
but keep priority pricing, matching gpt-5.4* precedent
- mode=chat for the thinking model, mode=responses for pro
reasoning_effort capability flags mirror the OpenAI variants exactly
since Azure proxies the same API contract: minimal rejection on both
chat and pro, low/none rejection on pro. Once #26456 (which sets
supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*)
lands, OpenAI and Azure flag profiles align.
Tests pin entry presence + pricing for all four Azure variants and
verify the live-API-derived reasoning_effort flags.
* test: register supports_low_reasoning_effort in cost-map JSON schema
azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch
carry supports_low_reasoning_effort=false. The strict
'additionalProperties: false' schema in
test_aaamodel_prices_and_context_window_json_is_valid rejected the new
key. Register it alongside the other supports_*_reasoning_effort
entries.
Note: the runtime side of this flag (code that reads it) lands in
#26456. Until that PR merges the flag is inert for both Azure and
OpenAI pro entries, but having the schema accept it lets cost-map
tests pass on either merge order.
* Use sanitize deep copy style to replace deepcopy usage
* Added test checking error is not happening anymore
* Added warning log when json copy failed
* Reduce to one change
* Fix spaces
---------
Co-authored-by: Ido Lavi <ido@noma.security>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: TomAlon <tom@noma.security>
tool_calls on assistant messages were translated to OllamaToolCall format
but never copied into the outgoing OllamaChatCompletionMessage, so Ollama
received {role: assistant, content: ''} with no tool_calls. The model
then had no record of having made a tool call, causing it to re-issue
the identical call on every turn (infinite loop).
Similarly, tool_call_id on role:tool messages was silently dropped.
Ollama uses this field to resolve the tool name from conversation history.
Also add tool_call_id to OllamaChatCompletionMessage TypedDict.
Fixes#26094
* fix(proxy): invoke post-call guardrails on pass-through endpoint responses (#20270)
Wire post_call_success_hook into non-streaming pass-through response path,
gated on explicit guardrail config (opt-in only, no backwards-compat break).
- Call post_call_success_hook after reading non-streaming response body
- Build enriched hook_data with guardrails metadata and litellm_logging_obj
at call site (avoids mutation of _parsed_body which is shared by logging)
- Handle ModifyResponseException with provider-agnostic error envelope,
post_call_failure_hook, and defensive try/except
- Strip stale content-length when guardrail modifies response body
- Move ModifyResponseException to litellm.exceptions to break cyclic import;
re-export from custom_guardrail for backwards compat
- Add call_type fallback in UnifiedLLMGuardrails for pass-through endpoints
using CallTypes.pass_through.value enum
* test: add unit tests for pass-through post-call guardrails
5 tests covering the post-call guardrail invocation on pass-through endpoints:
- post_call_success_hook fires when guardrails configured
- post_call_success_hook skipped when no guardrails (backwards compat)
- ModifyResponseException returns 200 with provider-agnostic error
- UnifiedLLMGuardrails resolves call_type from logging_obj for pass-through
- ModifyResponseException re-export from custom_guardrail stays in sync
Bedrock enforces non-increasing TTL ordering across cache_control blocks
(tools → system → messages). The tool cache_control TTL was being
unconditionally dropped to the default 5m, while system blocks preserved
the user-specified TTL for Claude 4.5+ models. This mismatch caused
"a ttl='1h' block must not come after a ttl='5m' block" errors when
users set ttl='1h' on both tools and system.
Converse path: add_cache_point_tool_block() now accepts a model param
and preserves TTL for Claude 4.5+, matching _get_cache_point_block().
Invoke path: _remove_ttl_from_cache_control() now also processes tools
(was only processing system and messages).
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
