* feat(spend-logs): opt-in latest_turn mode that stores only the current turn of stateless requests
* fix(spend-logs): keep bridged Responses rows and classifier audit in latest_turn mode, reject unknown values at boot
* fix(spend-logs): drop the recursive call in the latest_turn trim and type the stateless test cases
* test(spend-logs): format the stateless latest_turn cases
* test(spend-logs): wrap the classifier audit fixture
* fix(spend-logs): type the latest_turn trim inputs
* fix(spend-logs): leave the raw request body unvalidated before the latest_turn trim
* fix(spend-logs): trim classifier rows and ignore continuation keys off /v1/responses in latest_turn mode
* fix(spend-logs): take the classifier's originating request from the logging payload and redact a copy of vector store results
* fix(spend-logs): redact the body selected for storage, after the latest_turn trim
* test(e2e): retire the quiet_stack lock
quiet_stack held an exclusive host-wide lock for six tests, draining every
other xdist worker while they ran. On the full e2e lane that left 35-38% of
worker time idle. Each of those tests now lives where it can run without
pausing the suite
- the two memory tests move to perf/ behind the perf marker, so the perf lane
runs them on its own fresh stack. Their refusing deployments use a literal
key because the perf stack carries no provider keys
- the stored-request size check from the memory test becomes an integration
test on an owned proxy that stores prompts
- the discount and margin spend-row tests are dropped from e2e. Their
breakdown assertions join the existing integration pricing test, which
already owns the discount and margin repricing flow
- the Bedrock guardrail cost test stays in e2e without the pricing reset
fixture, since nothing else on the stack writes pricing config now
- the MCP semantic tool filter test turns on a proxy-wide setting, so it gets
a new proxy_wide_config opt-in marker and runs after the main run
* test(e2e): register the claude_code deployments as one batch
The claude_code session fixture registered its 24 deployments one at a time on every xdist worker, and each one waited out the 15 s propagation window on its own, about 6 minutes per worker. ProxyClient.register_models posts the whole batch, polls each model until every replica serves it, then waits out propagation once from the last write. A model that fails to post or never becomes servable comes back as ModelNotRegistered, and an unservable one is deleted, so the fixture keeps reporting per-deployment failures the way it did
* fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API
* fix(logging): log bridged chat completions with the chat response they returned
* fix(logging): log bridged text completions and /v1/messages as served, with the request's options
Bridged rows were translated without the request's options, so structured
output and replay fields could differ from what the client got. Text
completions and non-streamed /v1/messages through the chat bridge also kept
a Responses API object or the encoded id.
* fix(logging): redact reasoning summaries on bridged rows and log bridged text completions as text completions
Redaction now drops reasoning_items on chat messages and deltas, so a bridged row keeps no reasoning summary when message logging is off. A non-streamed bridged text completion logs the text completion the client got, priced like the chat response it came from. The bridge handler's input TypedDict fields are ReadOnly.
* fix(logging): redact bridged text completions for opted-out callbacks and pin the sync logged shape and usage ack
* test(integration): look up bridged chat spend rows by the id the client got
Bridged chat spend rows are keyed by the client-facing id and logged with the caller's call type, so the helpers that decoded the old resp_<base64> key or pinned call_type responses now assert the new contract
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: Return Anthropic error envelopes on Messages auth failures
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Answer Anthropic envelopes on Messages auth failures without relabelling unmapped statuses
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): answer the Anthropic error envelope on the /anthropic pass-through
The pass-through relays upstream errors in Anthropic's shape, but the proxy's
own refusals there (bad key, model access, budget) still answered the OpenAI
body, so an Anthropic SDK pointed at /anthropic could not read them
* fix(proxy): answer the Anthropic envelope for 404, 405 and unhandled 500 errors on Messages routes
A disabled Claude Code gateway 404, a wrong-method 405 and an unhandled 500 on the
Messages and /anthropic/ routes kept FastAPI's detail body or the OpenAI shape. The
central HTTP and unhandled-exception handlers now pick the Anthropic envelope by route.
The dashboard playground toast titles Anthropic error types when the body has no code.
* fix(proxy): keep LiteLLM error types and Anthropic envelopes on Messages routes
Error bodies on the Messages routes now carry error.litellm_error_type when the
LiteLLM type differs from the Anthropic type, so clients and the Admin UI keep
causes like budget_exceeded. Unlisted 4xx statuses map to invalid_request_error
per Anthropic's errors page, model-list requests that send anthropic-version get
the Anthropic envelope, and the admission, decompression and request-size
middleware errors use it on these routes too.
* test(proxy): expect litellm_error_type in Messages integration tests and type middleware tests
* fix(proxy): keep handler error bodies unchanged and timeouts as api_error
Errors the Messages and count_tokens handlers already returned in the Anthropic
shape keep their original bodies, so litellm_error_type now appears only on
errors the proxy raises before the handler. A 408 stays api_error as before,
and the playground toast titles the new Anthropic types and ignores long
numbers in front of an embedded error
* fix(proxy): keep the base error types inside the Messages handler and scope toast titles to Anthropic bodies
The wider status table (402 billing_error, 409 conflict_error, 504 timeout_error,
other unlisted 4xx invalid_request_error) now applies only to errors the proxy
raises before the handler. Errors inside the Messages handler, streamed error
frames and the SDK streaming adapters keep the merge base's table.
The dashboard toast reads a status before embedded JSON and titles by Anthropic
type only when the body is an Anthropic error envelope, so other pages keep
their titles.
* test(anthropic): mark the new test variables Final
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): log the encrypted response id on streamed /v1/responses spend rows
With response id security on, the stream hook encrypts the resp_ id on every
frame, but logging had already received the SDK's own copy of the terminal
response, so the spend row, the standard logging payload and callbacks got the
managed id (or the chatcmpl id on a bridged stream) the client never saw. The
hook now records the exact id it advertises on the request's logging object,
and the logging layer stamps that id on the logged streamed response
* fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id
Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the
id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new
session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes
to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response,
and a raw provider id still gets a managed id
Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or
<id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's
request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The
background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id
* fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner
A GET or cancel of a stored response now logs a new encrypted id of its own,
so callbacks and spend rows never reuse the create's id, while the owner of
the id still gets back the exact id they sent. An admin reading someone
else's response gets an id bound to the admin, as before. The spend row id
suffix for reads and cancels is gone, so with response id security off the
rows and session history match the default branch. A streamed GET follows
the same rules.
* fix(responses): echo the sent id to every reader that passes the response access check
A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed
the access check, not only when the caller is the user or team the id was issued to. An admin, the master key
or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not
reach the first turn from it, so a bridged follow-up through that id lost the conversation history
Every read and cancel still logs a fresh id bound to the reader, never the create's id
* fix(responses): keep a read or cancel row in session history only when it brings new output
A GET, streamed GET or cancel sent with a session header joins the
create's session, and its stored response repeats the create's output.
The replay then sent the same assistant turn once per read.
Session history now keeps an aget_responses or acancel_responses row
only when its stored Responses output has an item id that no earlier
kept row carried. A polled background response keeps the first poll
that returns the finished output, since its queued create row has none,
and later polls of the same output are dropped. Every other row keeps
today's rule
* fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart
A create row now owns its output items in session history, and a read or cancel row adds only items no
create and no earlier read carried, in one linear pass, so a background response polled while it grows,
or a read that ends before its streamed create, replays the answer once and after its prompt
DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id
like a read does
Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row
id, so a streamed background create and its cost event, which now share the client's id, keep separate
files
The integration tests that decoded the managed id now match the exact id the client received
* fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events
A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh
encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET
Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage
file and unbatched GCS object, so batch retrieves and events with no id keep main's names
An item a session read replays takes the newest read's copy at the first read's place, so an item a
poll caught half written replays finished
Tests pin the client's id over the router's continued response id, and the storage test no longer reads
the clock after the upload
* fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items
A streamed background create and its later cost event share the client's
id, so OpenMeter dropped the second event and DynamoDB overwrote the
create's item. Both now key the cost event as <id>_background_cost, like
its spend row and storage file.
Session history again replays a read's output items that carry no id,
as main did.
* fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras
* fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id
A streamed background create and its later cost event now share the id the
client received, so ClickHouse and DeepEval saw both events under one id.
Route both through logged_event_key, the way the Azure storage, GCS,
OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse
request_id matches its spend row's `<id>_background_cost` and its DeepEval
span gets its own uuid. ClickHouse response_id stays the raw provider id.
* test(deepeval): annotate the background cost span test's locals with Final
* test(responses): run the DELETE wire test that pins the echoed client id
DELETE /v1/responses/{id} now answers with the id the caller sent, so the
skip that marked it as a known bug no longer applies. The test passes on
the /v1, /openai/v1 and bare routes.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(a2a): bill completion-bridge agents at their own prices on /a2a send and stream
message/send on an agent served through the completion bridge (custom_llm_provider
agents such as LangGraph or databricks_agent) logged a $0, 0-token row, and
message/stream logged no row at all, so cost_per_query and per-token prices never
reached spend.
The bridge send path now records counted usage, the agent's pricing keys, and the
agent id on its logging object, the same way URL agents do. The bridge stream is
wrapped in the A2A streaming logger, which now reads the bridge's artifact event
for the reply text and gets the agent name from the proxy. Streamed calls also
count prompt tokens again, since the message text reader now reads pydantic
messages as dicts.
* fix(a2a): count chunked and Task replies once and skip agent prices on error replies
Bridge streams now follow the A2A append flag per artifact, so chunked
pydantic-ai, watsonx, and AgentCore replies count the whole reply once.
Send replies that are a Task count their artifact text. A JSON-RPC error
reply is logged at cost 0 instead of the agent's price.
* fix(a2a): keep one reply text per stream and bill every artifact and status reply
* fix(a2a): ignore a null error field on streams and run the stream per-token billing test
A stream event whose error field is null no longer zeroes the call's cost. The integration test for per-token stream billing now runs instead of skipping, and the URL agent SDK case skips when the optional a2a-sdk is not installed.
* fix(a2a): bill streams the client drops and skip the bridge's working placeholder
A stream the client closes or cancels mid-reply now logs its spend row at the agent's prices on both the bridge and URL paths, and a stream dropped before the agent sends anything still logs nothing. The bridge's "Processing request..." status no longer counts as reply text, so an empty bridge reply bills no completion tokens, as send does. Adds proxy-level tests that a bridge stream bills under the agent's card name.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(batches): cost an unmanaged gs:// batch through the deployment that created it
The create now stores the router deployment id as model_id inside the
managed object row's file_object JSON and CheckBatchCost reads it before
falling back to the first deployment matching the base model in the
gs:// path. Rows without the field keep the old lookup
* fix(batches): tolerate a non-string or malformed file_object when reading the stored deployment
* fix(batches): read the stored creating deployment from the writer, not a lagging read replica
* fix(batches): resolve the writer database once so the type gate count does not grow
* fix(batches): decode a dict-shaped file_object when reading the stored deployment, accumulate wire samples, annotate test locals
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): raise live budget counters for failure spend rows
A stream that breaks mid-flight writes its recovered cost to the spend log, but the failure hook
had already released the reservation and never touched the live counters, so an over-budget key
kept passing until the counter was next repaired from the database
Every charged failure row with a cost now raises the same counters the success path raises. The
error waits at most a second for them, so a hung Redis cannot hold it back
Fixes#35568
* test(proxy): cover failure spend counters across budgets, endpoints and outages
Integration cells for team, user, end-user, tag and window budgets,
concurrent streams, billed retries, native /v1/messages, Bedrock guardrail
cost, a Redis stall on two workers and disable_spend_updates
* test(proxy): check stalled failure counters and use the real spend writer
- The Redis stall cell now reads the burst key's live counter and requires at least the full failure spend
- Unit tests drop the charging writer double, bare dicts, list mutation and wall-clock timestamps
- Partial-stream test models are frozen
* test(proxy): move the stalled counter bound reason into the assert message
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(spend): keep a NaN or infinite cost from wedging daily spend writes
A non-finite provider-reported cost now counts as not reported, so the token price bills the request. Any other non-finite cost is logged and recorded as 0, and spend log JSON writes NaN and Infinity as null so Postgres accepts the row.
* fix(spend): price non-finite provider costs on every route and drain parked NaN batches
Perplexity, OpenRouter pass-through, and Responses streaming now treat a NaN or infinite reported cost as unreported, so the token price bills the request. A daily spend batch parked in the Redis buffer by an older version with a NaN spend now drains as 0 instead of failing every flush. The cost header never carries NaN or Infinity, and a set holding a non-finite float no longer breaks spend log JSON
* fix(spend): record non-finite savings as 0 and warn when a parked NaN batch drains
An overflowing price on a compressed request made compression savings infinite, which still wedged the daily spend batch. Every savings column now records a non-finite value as 0 with a warning. The Redis drain warns when it reads a NaN an older version parked, since the spend summed into it is lost from daily spend. The writer test now runs the real payload builder and spend log insert
* fix(spend): keep non-finite costs out of headers, responses, and admin JSON
Writing NaN as null is now opt-in for spend log columns, the Redis buffer, and
streamed frames, so admin writes and logging integrations keep their output.
The cost calculator records an overflowing total as 0 with a warning, cost
headers leave out non-finite values, a non-streaming NaN usage.cost is dropped
instead of failing the response, and a parked NaN spend log row reads back as 0.
* fix(spend): keep caller-set call ids from splitting the non-finite cost warning
The overflow warning strips newlines from litellm_call_id, and neither
non-finite cost warning prints the raw value, which said nothing beyond
"non-finite"
* fix(spend): drop non-finite Responses and OpenRouter costs and the breakdown of an unpriceable request
* fix(types): drop a usage cost breakdown that holds a non-finite number
* fix(streaming): keep a dropped non-finite final usage cost from falling back to an earlier chunk's cost
* fix(spend): keep the dropped-cost flag out of reach of provider usage keys
A provider usage key named dropped_non_finite_cost shadowed the Usage method on the instance, so the stream builder raised and dropped the request's spend. The builder now reads the flag through the class, and Usage sets the flag after copying provider keys, so a _dropped_non_finite_cost key cannot set it either.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(usage): show subscription-covered providers and account fees separately
* fix(usage): write only the named fee columns and sync the root schema
The fee update now writes just the columns the request names, so two admins changing
different fields at once both land, and the dashboard edit form sends only what changed.
The root schema.prisma carries the new table like the two copies, and a billing period
anchored in the calendar's last month ends on December 31, 9999 instead of failing.
* fix(usage): register the subscription table span and keep provider logic under llms
* fix(subscription_accounts): register the fee routes as management routes and reject an empty account id
* fix(usage): run the new tests in CI and close subscription card review gaps
Adds the subscription test file to its unit shard, sets the migration lock timeout, rejects infinite fees, keeps the card from rewriting login files, lists each deployment once without aliases, shows the provider badge on entity views, and asks before removing a fee
* fix(usage): resolve named-credential ChatGPT accounts and keep admin viewers read-only on the fee card
* fix(usage): count a subscription fee only for accounts behind a current deployment
Usage listed a fee row for an account no deployment uses while another ChatGPT
deployment existed and dropped it when none did. It now lists only accounts behind a
current deployment in both cases. Adds a real proxy and Postgres integration test for
the fee API and for ChatGPT Responses rows on a deployment's own login, and names the
card's multi-component test with the integration suffix.
* fix(usage): keep a stored subscription fee listed after its account's deployments are removed
The usage route lists every account behind a subscription deployment plus every account carrying a stored fee, and reads the fees on every proxy, so a fee stays visible, counted, and editable for past ranges after its last deployment goes. The integration test now polls every day the calls spanned, so a UTC midnight crossing no longer fails it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* fix(proxy): stop answering with the literal string "None" on the spend and management routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param
* fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param
A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None".
* fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400
* fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap
* fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param
* fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does
Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error.
* test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor
* fix(proxy): bill streamed /v1/completions without stream_options like chat
* test(proxy): mark the text completion stream usage test variable Final
* fix(completions): drop Azure's empty prompt-filter frame when the proxy asked for stream usage
* test(spend): drop the module docstring from the text completion stream usage test
* fix(proxy): keep the internal stream usage strip flag out of the logged request body
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: drain shutdown callbacks and persist pending tag spend
* fix: catch shutdown timeouts on Python 3.10
* fix: share shutdown callback budget with HTTP draining
* fix: type shutdown cancellation tasks and document its clock
* test: retain callbacks after bounded worker cancellation
* test(proxy): inspect the lifespan body after the SIGTERM wrapper split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): keep the success callback enqueue on its own task
The synchronous enqueue spawned the logging worker loop inside the awaited
call and ran callbacks that per-call loops used to drop, which hung the SDK
callback tests on the autouse worker flush. The shutdown worker flush covers
the queued callback on its own, so the one-tick task is restored
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback
The aggressive clear ran the rejected callback and marked it done against
the queue it was never put in, so flush() could return while a real
callback was still running and the shutdown drain cancelled it early. The
rejected callback is now re-enqueued once the clear made room, and only
runs uncounted when the queue is still full
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(logging_worker): build the aggressive-clear run set immutably
Final tuple instead of an appended list, with the re-enqueue in a
_requeue helper, and Final annotations on the overflow regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): track overflow callbacks through shutdown
* chore(logging): explain shared callback ownership
* test(logging): isolate deferred callback completion
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): resume background response streams with stream and starting_after
Retrieving a response with stream=true and starting_after=N now returns the
replayed event stream from OpenAI and Azure instead of one JSON object, on the
SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id}
Providers that cannot replay a stored stream drop the params under drop_params
and raise UnsupportedParamsError without it. Malformed values answer 400 with
the param named. A replay that reaches its terminal event settles the
background response's bill once and passes the provider's usage through
without estimating one. An empty replay (cursor at the end) ends normally
* fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams
A replayed background stream now gets the same response id encryption as a
created one, so the key that made the response can reuse the id. Azure AI
Foundry deployments resume streams like Azure ones, and the chat-bridged
Responses iterator closes its wrapped stream instead of raising.
* fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve
A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query.
* fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params
* fix(responses): keep deployment stream off retrieves and reject oversized cursors
* fix(responses): keep router default stream off plain retrieves
* test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing
* test(responses): replay a real stream in the billing test and type the fake wrapper chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): audit #46997
Add request-path cells for the global proxy spend counter: a key with no user spends toward litellm max_budget, and two proxies sharing Redis add every request to one counter that matches the proxy budget row
* test(proxy): wait for the recorded budget spend and shared counter before checking them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds integration cells for a different key reading the response, message redaction, a rotated, deleted, or team-deleted creator, and a creator whose Langfuse keys resolve to nothing
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(daily-activity): read guardrail spend per key from the daily guardrail usage table
Adds DailyActivityRepository.guardrail_spend, which sums LiteLLM_DailyGuardrailUsageUnits.cost by date, guardrail and key. A guardrail row counts only when its key has a spend row on that day under the same scope filters as the rest of the daily activity response, so user, key and date filters apply to guardrail cost too
* feat(daily-activity): add a per-guardrail spend breakdown to /user/daily/activity/aggregated
Each day now has breakdown.guardrails[<guardrail_id>] with metrics.spend and an api_key_breakdown carrying each key's spend and metadata. Guardrail entries only report spend, since that is the one metric stored per key. The breakdown is on the user endpoint only, is empty when a model filter is set because guardrail cost is not stored per model, and lists only keys within api_key_limit while the guardrail total keeps every key. spend and budgets are unchanged
Resolves LIT-9785
* feat(daily-activity): add the guardrail breakdown to team and organization activity, with guardrail names
/team/daily/activity/aggregated and /organization/daily/activity/aggregated now return breakdown.guardrails the same way the user endpoint does, since each key belongs to one team and one organization. Each guardrail entry also carries metadata.guardrail_name, read from LiteLLM_GuardrailsTable for guardrails created through the UI or API and falling back to the breakdown key for config.yaml guardrails, whose key is already their name
* fix(daily-activity): keep daily activity up when the guardrail breakdown query fails, and treat an empty model filter as none
A database error from the guardrail spend query now logs a warning and returns the response without guardrails, where it used to fail the whole aggregated response with a 500. An empty model= no longer drops the guardrail breakdown, matching how the rest of the endpoint already ignores it. The daily activity golden files and fixtures gain the empty guardrails key every breakdown now carries
* refactor(daily-activity): fold guardrail breakdown in a worker thread and drop docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(daily-activity): report untracked guardrail units next to spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): cover the batch cost poller gaps left after #47190
* fix(batches): keep unbillable batches from blocking file deletes and new-batch polling
The file delete check now ignores batches past MANAGED_OBJECT_STALENESS_CUTOFF_DAYS, a provider 404 for the output object retires the batch, each poll page takes the arrival lane first, a batch expires only after the poller visits it past the cutoff, and the compare-and-swap matches batch_processed instead of updated_at
* test(batches): cover the full arrival page and a failed expiry write in the batch cost poller
Old batches still get polled every other cycle while new batches fill the page, old batches that fill a page never push a new batch out after boot, and a failed stale_expired write no longer stops the rest of the page from being polled
* fix(batches): share the poll page between old and new batches and hold output files until billed
Each lane now gets at least half of the poll page, a lane with nothing left gives its half to the other, and a lane read that resumed mid-lane wraps to the start, so neither old nor new batches can starve the other.
The delete guard applies the 7 day cutoff to input files only. Output and error files stay locked until the batch is billed or retired, and stale_expired rows no longer lock anything.
Only a lasting provider failure expires a batch past the cutoff, so a timeout or a 5xx no longer gives up on a batch that may still bill. Past the cutoff a lasting failure, a lasting cost-tracking failure, or an output file id that never appeared retires a finished row instead of polling it forever.
* fix(batches): retry refused finished batches past 7 days, expire unfinished ones on a failed retrieve
A finished batch the provider refuses after 7 days (rotated key, bucket 403, a 404 that does not name the batch) now keeps retrying so its spend is written once the config is fixed, instead of retiring unbilled. A batch the provider still reports running keeps polling. A batch that is not finished in the database becomes stale_expired once a retrieve fails after 7 days, so an unreachable provider is not polled forever. An S3 NoSuchKey for the output object now counts as gone.
* fix(batches): expire an unfinished old batch only after 3 failed retrieves in a row
* fix(batches): keep retrying an unfinished old batch through timeouts, 429 and 5xx
* fix(batches): wait 3 polls before expiring an old batch whose output id lags
* fix(batches): match a resource id in a 404 message followed by punctuation
* fix(batches): keep refusal and pending-output expiry streaks apart, and void a pending-output retirement on a client write
* fix(batches): give up on unsettled batches past twice the staleness cutoff
Replace the per-reason refusal and pending-output streaks with one give-up age, and inject a clock into the poller and the delete check so tests pin exact age boundaries
* fix(batches): keep polling finished batches past the give-up age so their spend stays recoverable
* test(batches): expect a restarted poller's first page to split between old and new batches
* test(batches): pin the delete-guard tests to a fixed clock and type a new journal list
* test(batches): type the provider helper and the storage 404 cases
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47358
Add integration cells for pass-through bodies and settings-endpoint mappings across workers
* test(proxy): wait for every worker to hold the saved mapping before probing
* test(proxy): compare passthrough bodies to the client body and probe each worker on its own connection
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A DeepSeek V4 stream that ends without a usage block now has a spend-row cell proving its prompt_tokens are counted with DeepSeek's tokenizer, the one path #47353's own cells did not reach
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds an integration cell proving a failed /v1/responses request counts its input from the newest compaction item on its failure spend row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(passthrough): let upstreams report the model name via x-litellm-upstream-model
A pass-through upstream that prices its own requests already reports cost and
tokens in x-litellm-response-cost / x-litellm-total-tokens. Product endpoints
that take no `model` in the body still landed under `unknown`, so their spend
could not be broken down.
When the upstream sends x-litellm-upstream-model, record its value as the
request's model on success and failure rows, overriding any body `model`. The
name is a label only: it never marks the request as upstream-priced and is not
used to look up a price. A blank or missing header changes nothing.
Refs LIT-9910
* fix(passthrough): keep the upstream-reported model name bounded
The upstream model name becomes a spend row, a daily rollup key and a metric
label. Use only the first element of a repeated header (RFC 9110 field
combining, which is how httpx joins it) and cut the name to
MAX_SPEND_LOG_MODEL_NAME_LENGTH, the limit the spend log already applies to
model names.
Refs LIT-9910
* test(passthrough): cover x-litellm-upstream-model through spend rows and rollups
Drives a config-defined pass-through against a scripted upstream and reads
back the spend row, the daily team rollup and /team/daily/activity. Covers the
header winning over a body model on buffered and streamed responses, failure
rows, a model-only header keeping cost_per_request, the no-header and blank
header cases, and repeated or long header values.
Refs LIT-9910
* test(passthrough): annotate new test locals with Final
Refs LIT-9910
* fix(team): keep the team reset schedule when a temp budget increase is set on a member inheriting the team default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect the temp-only member row to carry the team default cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reconcile team member total_spend with current cycle spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read /team/list as a list and drain the scripted provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(repositories): compose the membership reset writer and drop SQL text asserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: type the cascade repair test and flatten memberships without a stacked comprehension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: keep the reset job test file's existing typing untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): consolidate gateway and usage assistants as Aria Agent
* refactor(ui): give Aria an original blue sprite mascot
* feat(ui): prompt shared Aria from usage and retire duplicate usage agent
* fix(ui): keep Aria naming consistent and reachable across dashboard
* feat(ui): open Aria in a floating popup with suggested questions
* fix(ui): simplify the shared Aria Agent title
* fix: Send the include_usage chunk with an empty choices array
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Skip chunks with no choices when reading streams
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Strip usage from plain-iterator async stream chunks
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Expect empty choices on the usage chunk in the gemma tracing test
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage
The usage chunk now has choices=[] only when the caller set
stream_options.include_usage itself. When the OpenAI handler, the proxy's
always_include_stream_usage setting, or the harness added include_usage on the
caller's behalf, the chunk keeps its legacy single empty-delta choice, so
clients that read choices[0] on every chunk keep working.
The proxy and harness mark their injection with the private
_litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy.
Plain-iterator streams keep usage on their chunk when usage was not requested,
and the LLM Shield trailing flush ignores chunks with no choices.
* fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape
The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them.
* fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route
* refactor(router): move the stream usage injection check into router_utils
* fix(router): read stream_options from pydantic deployment params when marking injected usage
Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices.
* fix(proxy): keep the stream usage injection marker out of stored request bodies
* test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths
* test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): the billed background response cost reaches the creating key's and team's loggers
* fix(responses): send the billed background response cost to the creating key's and team's loggers
* fix(responses): bill an unreadable or rejected creator through proxy loggers, redact and honor disabled callbacks on the billed event
* fix(responses): follow the creator's message logging setting on the billed event and read team logging for non-key creators
The billed event no longer forces redaction. It follows the creating key's or team's turn_off_message_logging, else the proxy's, and redacts when the creator cannot be read or its settings are rejected. A creator without a virtual key hash bills through its team's settings. function_setup keeps its original callback split, and the billed path uses a separate non-mutating helper.
* fix(responses): bill rotated, deleted, and custom auth creators through their own logging and retry failed creator lookups
* fix(responses): follow a key's whole grace-period rotation chain to the live key for its billed charge's logging
* fix(responses): bill a background response whose creator cannot be read, without the creator's settings
* docs(enterprise): say the background response row is claimed before its creator is read
* fix(responses): keep the loaded creator rows' budgets and aliases when one creator lookup fails
The Gemini TTS billing and text completion guardrail tests read spend rows from the owned proxy's own database, which #47213 made private. The cursor wire test derives its effort levels from the gpt-5 cost map row that #43935 now enforces, and the Azure effort check accepts the unquoted level. The pin TTL test asserts the one hour pin #46411 writes for a tool-only mark, the video callback test decodes ids with the product decoder after #44614, and the two Gemini 3.8 flash stream cost cases bill audio at the text rate their rows imply
* fix(proxy): stop update_cache from resurrecting evicted auth cache entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update update_cache pins for synchronous write-back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): wrap long assert in update_cache tag test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover refill race and pin TTL with injected clock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count global proxy spend for every request with a user, cached on this worker or not
* fix(proxy): add global proxy spend with an atomic Redis increment that never re-creates a reset counter
The global proxy spend bump read the total, awaited, and wrote old + cost back
from a background task, so a budget reset that deleted the key in between got
the pre-reset total written back, and workers overwrote each other's totals.
RedisCache.async_increment_if_present runs INCRBYFLOAT only when the key
exists, keeping its TTL, and this worker's copy is then set from the shared
total with no await between its read and write. The bump now runs whenever a
proxy max_budget is set, matching the DB writer, and the armed post-call read
and the batch-read declaration it used are removed.
* fix(proxy): report the global spend Redis increment to service metrics and request traces
The global proxy spend increment now emits Redis service success and
failure events under the spend_counters target and passes the request
span through, so it shows as redis.incr spend_counters like the write
it replaced.
The unit tests run the real RedisCache increment against fakeredis
instead of mocks, and a new test checks the span lands under the
request span on success and on a Redis failure.
* test(proxy): keep the spend counter Redis fake off the network
* test(proxy): count increment_if_present as a shared-cache call in the key-family guard
* fix(proxy): keep the newer global spend total when Redis replies arrive out of order
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tests): read the stream stall cooldown spend rows from the owned proxy's database
The cooldown cells boot an owned proxy, which writes its spend logs to its
own scratch database since owned proxies went private, while the cells polled
the worker's database and never saw a row. Read each row through the gateway
that served the request, so the cells check the proxy they called.
* fix(tests): read passthrough spend rows from the owned proxy's database
The vertex regional uplift and passthrough stream pricing spend files boot
private owned proxies, then polled the worker database for spend rows that
only reach the proxy's own database. Read them through the owned proxy's
gateway instead.
Also hold the stream stall file's hold_headers peer past the longest row
wait, so a slow spend row cannot fail the peer teardown.
* fix(tests): check agent deletion on the owned proxy's database
The agent least privilege test deletes its agent through an owned proxy,
then checked the worker database for the row, which is always empty
there. Read it through the owned proxy's gateway so a delete that leaves
the row fails the test.
* fix(tests): read turn admission spend rows from the owned proxy's database
* test(responses): billed background responses count toward the creator's project and model budgets
* fix(responses): carry the creator's project and per-model budgets on the billed background response
* fix(responses): count a key's budget tier toward background per-model budgets
* fix(responses): bill a background response's project only while the key stays on its team
* revert(responses): keep billing the creating key's project after a team move
A key can only keep or detach its project, so the billed project always pairs with the team it was created under. This reverts commit ded4f084c5.
* fix(responses): bill a background response under the model name the client sent, with its project alias
* fix(responses): name the team's public model as a billed background response's model group
The model group lookup now passes the creating key's team, so a team model or team wildcard bills under the public name a normal call logs and per-model budgets on that name move
* feat(proxy): map baggage values to budget tags
* fix(proxy): preserve mapped tags in the parsed request cache
* fix(proxy): preserve mapper accounting and tag policy
* fix(proxy): preserve baggage tags across native transports
* fix(proxy): keep mapper rejection and passthrough logging safe
* fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing
Mapped tags now apply only to billed LLM API routes, skipping token counting,
model listing, MCP and A2A. They classify spend and budgets but never select
deployments unless a key, team or client set the same tag. WebSocket inference
returns 403 only while a required rule exists. A tag created with /tag/new resets
its spend counter so its budget counts from creation. The General settings editor
keeps a new mapping unsaved until every required field is filled in.
* fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews
* fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403
* fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values
* test(proxy): cover baggage tag mappings end to end
Adds integration cells for request_tag_mappings across chat, messages,
responses, A2A and custom pass-through spend, required and budgeted
rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags,
tag routing and policy isolation, a dropped tag read with and without
allow_requests_on_db_unavailable, and the /tag/new spend counter reset
* fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags
Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags.
* fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies
Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering.
* feat(proxy): parse baggage headers into request tag mappings
* fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes
* fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset
Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request
* fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage
Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag.
* test(spend): cover string metadata.tags under mapped baggage
* fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings
Mapped baggage tags are applied again after the header and key tag merges, so
a header or key tag that replaces a string metadata.tags no longer hides the
mapped tag from the tag budget check. An empty or false metadata.tags is read
like a missing one, and a tags list with non-string items is kept as sent with
the mapped tags added after it, instead of failing auth with a 401
Local token counts for DeepSeek V3 and V4 model names now use DeepSeek's own bundled tokenizer instead of cl100k_base, and tool definitions on V4 names count with V4's chat-template tool framing
test_router_cooldown_failure_row.py boots an owned proxy, which writes to a private database since #47213. The file was written on a base without #47213 and read its rows from the worker database through read_rows, so all 13 cases timed out with State did not converge: []
Every row read now goes through Gateway.rows on the owned proxy's gateway, the same fix#47410 made for the stream stall cooldown cases
* fix(speech): correctly propagate spend logging and usage for Gemini TTS bridge
* fix(speech): call Gemini once per TTS request and log the bridge's token usage
aspeech re-ran speech whenever the first result was not a coroutine, so the
speech-to-completion bridge sent every Gemini TTS request upstream twice. It
now returns the sync result.
The bridge's inner completion runs as an internal sub call, so only the
speech call fires success callbacks, and the completion's usage rides on the
audio response so the spend log and usage_object carry real token counts.
Replaces the separate inner Logging object and the cost_calculator fallback,
which never reached the spend row.
* fix(speech): flag a cached Gemini TTS reply as a cache hit so it logs at no charge
* fix(speech): price a cached Gemini TTS reply's savings from its token usage and keep its cache key
* test(integration): cover Gemini TTS speech billing, logging, cache hits and deployment TPM
* test(integration): start the speech TPM cell in a minute's first 10 s so the two requests never cross a minute
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): record cooled deployments on the all-cooled failure row
A RouterRateLimitError now carries the cooled deployments of the route the
request resolves to for the caller's team (model group, alias, wildcard or
team route), each with its masked api_base. get_error_information copies them
into error_information.error_cooldown_deployments so the SpendLogs failure row
and every logging callback can tell which deployments were cooling
* fix(router): name cooled deployments from the router's own resolved route
Use the deployments the router resolved for the request, before health and
cooldown filtering, instead of re-deriving the route. Alias chains, the
provider-model fallback, wildcards and team filtering now match routing by
construction. A routing-strategy refusal records null.
* fix(router): name only the cooled deployments tag and region routing allow
The refusal's cooled-deployment list now leaves out deployments the
request's tags or the end user's allowed region keep it from, so a
tenant's refusal row no longer shows another tenant's api_base
* fix(router): keep failure logging working for refusal subclasses that skip init
A RouterRateLimitError subclass that never calls its parent init now logs a null cooled-deployment list instead of breaking the failure payload
* test(router): call the refusal routing helpers directly
The router coverage check needs each router function called by name from a unit test
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns.
* fix(responses): bill the usage a failed native WebSocket turn reports
* fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect
The v3 limiter stashed whichever litellm_call_id the request body carried as
the slot owner, so a client-supplied id on a pass-through body left the
proxy's own success and failure events unable to release the slot. The owner
is now the proxy's logging call id. A WebSocket connection whose turns log
under their own ids never settles the TPM, ITPM, or OTPM estimate the
connection reserved, so the disconnect release now refunds any reservation
no callback settled, once.
* fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect
A failed turn's reported usage now counts toward the key's TPM and the
project ITPM/OTPM scopes the same way a success does, and closing the
socket hands back only the estimate still unused in the window that
holds it, so a window that already rolled is left alone.
An id-less error event is booked against the newest pending turn, the
one the backend rejected at parse time, and a turn that nests its input
under response logs its user message. A connection's one budget
reservation goes to the first turn that settles it, every later turn
charging its cost directly.
* test(proxy): expect the settling call id on a pass-through budget reservation
* fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM
A native Responses WebSocket turn reaches the rate limiter's success
callback with its raw event list as the response and no
combined_usage_object, so the turn's tokens never reached the key's TPM
window or the project's input and output token limits. Read the usage
from the turn's events when the call type is a Responses WebSocket turn.
* fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas
The limiter's failure hook looked up the parallel slot by the request
body's litellm_call_id, while the slot belongs to the proxy's logging
call id, so a pass-through request a pre-call hook rejected held its
slot for an hour when its body carried an id. Both now resolve the id
the same way.
Each response.create frame already charges its estimate to project
ITPM/OTPM, so the turn's own usage no longer adds a second count.
The cost callback's failure hook read the body's litellm_call_id
through a strict model and raised on a non-string value, which dropped
the failure row. It now reads a string id or none.
* fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right
Free a rejected request's parallel slot through the request it runs in, and skip
native Responses WebSocket connections, whose disconnect frees the slot. Count
WebSocket turns toward the connection's tag TPM. In collector mode, claim the
shared budget reservation before the spend event is built, so only the first
turn settles it. Store each turn's own frame as its request body, and send an
id-less error to the oldest waiting turn unless an earlier turn is streaming.
* fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag
* fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls
* fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes
* fix(router): count native Responses WebSocket turns toward deployment limits and budgets
* fix(router): re-check deployment budget and RPM on each Responses WebSocket turn
* test(router): list the WebSocket turn admission method as covered through its partial
* fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn
* fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure
* fix(proxy): log the routed model group on a WebSocket turn's failure row
* fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn
* test(integration): cover per-turn admission on Responses WebSockets
Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the
project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP
controls for TPM counting and the failure row model_group
* fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails
A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success
callback such as the response cache no longer runs each CustomLogger's sync success hook on top of
the async one. That second run counted every turn's tokens twice toward deployment TPM.
A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket
closes with 1011, and the error becomes the socket's failure, on both the native and the managed
relay, instead of the socket dropping with no frame and no failure row.
A refused turn's failure row now carries the request's tags from the router's metadata, the same
way it already carries the model group.
* test(integration): cover a crashed WebSocket turn check and failure-row tags
A turn whose admission check raises gets a 500 server_error frame, the
socket closes with 1011 and its one failure row carries the error, on
both relays. The refused turn's failure row keeps the caller's tags, and
HTTP failure rows on chat, responses and messages keep them too.
* fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows
A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket
now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which
a client that sends its next response.create early would lose.
A refused turn's failure row now takes the router's tags only when every tag is a string, so the
fallback never copies a caller's unvalidated tags onto the row.
* test(integration): cover a turn still streaming when the next WebSocket turn's check crashes
A native turn still streaming when the next turn's admission check raises now has its own cell. It
must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for
one more turn, so a relay that stopped checking turns after a crash goes red.
* fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets
A turn after the first now runs the project quota check, then the proxy's
pre_call guardrails, then the deployment's turn admission, on both the
native and the managed relay, the order HTTP calls use. A turn a guardrail
blocks gets the guardrail's error even past the budget and no longer counts
toward the deployment's rpm or runs its pre-call checks.
* fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it
A turn the per-turn guardrail passes and the deployment then refuses now writes
its own failed spend row with that turn's guardrail information, on both relays,
the way a guardrail-blocked frame and an HTTP refusal already do
* fix(responses): bill a refused WebSocket turn on one row only
A refused turn whose guardrails recorded a cost is already billed on its own row,
so it no longer also becomes the connection's close-time failure row
* fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row
The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them
* fix(responses): bill a WebSocket's refused turns when the session limit closes it
* fix(responses): bill refused WebSocket turns when the session limit cancels the relay
* fix(responses): free the WebSocket slot at the session limit without waiting on the failure row
* fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot
* fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown
* fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after
* fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it
* fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close
* fix(proxy): always commit spend again after the shutdown wait for pending logging tasks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
