* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns.
* fix(responses): bill the usage a failed native WebSocket turn reports
* fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect
The v3 limiter stashed whichever litellm_call_id the request body carried as
the slot owner, so a client-supplied id on a pass-through body left the
proxy's own success and failure events unable to release the slot. The owner
is now the proxy's logging call id. A WebSocket connection whose turns log
under their own ids never settles the TPM, ITPM, or OTPM estimate the
connection reserved, so the disconnect release now refunds any reservation
no callback settled, once.
* fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect
A failed turn's reported usage now counts toward the key's TPM and the
project ITPM/OTPM scopes the same way a success does, and closing the
socket hands back only the estimate still unused in the window that
holds it, so a window that already rolled is left alone.
An id-less error event is booked against the newest pending turn, the
one the backend rejected at parse time, and a turn that nests its input
under response logs its user message. A connection's one budget
reservation goes to the first turn that settles it, every later turn
charging its cost directly.
* test(proxy): expect the settling call id on a pass-through budget reservation
* fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM
A native Responses WebSocket turn reaches the rate limiter's success
callback with its raw event list as the response and no
combined_usage_object, so the turn's tokens never reached the key's TPM
window or the project's input and output token limits. Read the usage
from the turn's events when the call type is a Responses WebSocket turn.
* fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas
The limiter's failure hook looked up the parallel slot by the request
body's litellm_call_id, while the slot belongs to the proxy's logging
call id, so a pass-through request a pre-call hook rejected held its
slot for an hour when its body carried an id. Both now resolve the id
the same way.
Each response.create frame already charges its estimate to project
ITPM/OTPM, so the turn's own usage no longer adds a second count.
The cost callback's failure hook read the body's litellm_call_id
through a strict model and raised on a non-string value, which dropped
the failure row. It now reads a string id or none.
* fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right
Free a rejected request's parallel slot through the request it runs in, and skip
native Responses WebSocket connections, whose disconnect frees the slot. Count
WebSocket turns toward the connection's tag TPM. In collector mode, claim the
shared budget reservation before the spend event is built, so only the first
turn settles it. Store each turn's own frame as its request body, and send an
id-less error to the oldest waiting turn unless an earlier turn is streaming.
* fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag
* fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls
* fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes
* fix(router): count native Responses WebSocket turns toward deployment limits and budgets
* fix(router): re-check deployment budget and RPM on each Responses WebSocket turn
* test(router): list the WebSocket turn admission method as covered through its partial
* fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn
* fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure
* fix(proxy): log the routed model group on a WebSocket turn's failure row
* fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn
* test(integration): cover per-turn admission on Responses WebSockets
Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the
project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP
controls for TPM counting and the failure row model_group
* fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails
A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success
callback such as the response cache no longer runs each CustomLogger's sync success hook on top of
the async one. That second run counted every turn's tokens twice toward deployment TPM.
A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket
closes with 1011, and the error becomes the socket's failure, on both the native and the managed
relay, instead of the socket dropping with no frame and no failure row.
A refused turn's failure row now carries the request's tags from the router's metadata, the same
way it already carries the model group.
* test(integration): cover a crashed WebSocket turn check and failure-row tags
A turn whose admission check raises gets a 500 server_error frame, the
socket closes with 1011 and its one failure row carries the error, on
both relays. The refused turn's failure row keeps the caller's tags, and
HTTP failure rows on chat, responses and messages keep them too.
* fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows
A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket
now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which
a client that sends its next response.create early would lose.
A refused turn's failure row now takes the router's tags only when every tag is a string, so the
fallback never copies a caller's unvalidated tags onto the row.
* test(integration): cover a turn still streaming when the next WebSocket turn's check crashes
A native turn still streaming when the next turn's admission check raises now has its own cell. It
must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for
one more turn, so a relay that stopped checking turns after a crash goes red.
* fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets
A turn after the first now runs the project quota check, then the proxy's
pre_call guardrails, then the deployment's turn admission, on both the
native and the managed relay, the order HTTP calls use. A turn a guardrail
blocks gets the guardrail's error even past the budget and no longer counts
toward the deployment's rpm or runs its pre-call checks.
* fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it
A turn the per-turn guardrail passes and the deployment then refuses now writes
its own failed spend row with that turn's guardrail information, on both relays,
the way a guardrail-blocked frame and an HTTP refusal already do
* fix(responses): bill a refused WebSocket turn on one row only
A refused turn whose guardrails recorded a cost is already billed on its own row,
so it no longer also becomes the connection's close-time failure row
* fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row
The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them
* fix(responses): bill a WebSocket's refused turns when the session limit closes it
* fix(responses): bill refused WebSocket turns when the session limit cancels the relay
* fix(responses): free the WebSocket slot at the session limit without waiting on the failure row
* fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot
* fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown
* fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after
* fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it
* fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close
* fix(proxy): always commit spend again after the shutdown wait for pending logging tasks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
c80f1c2ec8