* fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits
A native Responses WebSocket turn logs its raw events instead of a usage-carrying
response object, so the v3 rate limiter read no usage and never charged the key
or tag token window. Read the turn's usage from its events the way the cost
calculator does, and charge the TPM-limited tags the socket was opened under.
* fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage
A later response.create turn on an open Responses WebSocket now gets a read-only
check of every TPM window it counts toward (key, team, user, org, end user, team
member, model-scoped limits and the socket's tags). A turn with no tokens left gets
the 429 rate_limit_exceeded frame and the socket stays open. The check never
reserves or increments a counter and leaves RPM, budget, PTU and project windows
to their own paths. It runs on native and managed sockets.
Managed turns carry the socket's call id in their metadata, so they charge the
socket's TPM-limited tags. A failed native turn charges the partial usage its
failure row already carries. A socket refused at open by a rate limit gets the
same refusal frame and closes 1008, logged once without a traceback.
/v1/realtime sessions now count toward TPM through the realtime usage combiner,
and the v1 limiter reads socket usage through the same helper
* fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn
A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with.
* fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero
* fix(proxy): stop charging a realtime session's whole usage into its close minute
Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case.
* test(proxy): check each per-turn TPM window on its own
* fix(proxy): send a fail-closed per-turn socket refusal with its 503 status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The shared disconnect release dispatched to the legacy limiter, which decrements unconditionally, so a rejected realtime attempt freed a slot it never took. The generic release is v3-only again and the Responses WebSocket close releases through its own method
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn
* test(responses): mock the websocket slot release and mark the per-turn start time naive
* style(responses): keep the per-turn start time line within ruff format
* refactor(responses): type websocket turn events as read-only mappings
* fix(responses): settle the websocket budget reservation on the first turn only
Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly
* fix(proxy): keep the legacy limiter's websocket slot until disconnect
A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): let admins make per-model key/team rate limits a hard 429 instead of falling back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): sync per-model rate limit API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type per-model fallback setting access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the per-model hard cap inside the fallback loop and test it through the real limiter
* fix(proxy): list the per-model hard cap setting on the settings page and cover every per-model descriptor
* fix(proxy): read disable_fallbacks_on_per_model_rate_limits as a boolean value and cover the team-wide per-model cap
* test(proxy): cover per-model rate limit 429s skipping fallbacks on a live proxy
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(proxy): expose public names for private proxy helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep original class names behind public aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve internal callback filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep _PROXY_ class names for managed files hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names in package exports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve recursive auth helper name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align MCP limiter tests with server enforcement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): match main's MCP limiter tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names bound in importing modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve compatibility imports through strict lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use exact pyright suppression in password helper test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): add reasons to compatibility import noqa comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update IN-list baseline for renamed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): stub HIBP through respx by disabling the aiohttp transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): share the httpx transport fixture across proxy unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): restore proxy globals without a missing-value sentinel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): package moved dirs and stub the login breach check at the HTTP boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): isolate the mcp server manager per test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-01 10:11:45 -07:00
Renamed from tests/test_litellm/proxy/hooks/test_parallel_request_limiter.py (Browse further)
