* fix(a2a): bill completion-bridge agents at their own prices on /a2a send and stream
message/send on an agent served through the completion bridge (custom_llm_provider
agents such as LangGraph or databricks_agent) logged a $0, 0-token row, and
message/stream logged no row at all, so cost_per_query and per-token prices never
reached spend.
The bridge send path now records counted usage, the agent's pricing keys, and the
agent id on its logging object, the same way URL agents do. The bridge stream is
wrapped in the A2A streaming logger, which now reads the bridge's artifact event
for the reply text and gets the agent name from the proxy. Streamed calls also
count prompt tokens again, since the message text reader now reads pydantic
messages as dicts.
* fix(a2a): count chunked and Task replies once and skip agent prices on error replies
Bridge streams now follow the A2A append flag per artifact, so chunked
pydantic-ai, watsonx, and AgentCore replies count the whole reply once.
Send replies that are a Task count their artifact text. A JSON-RPC error
reply is logged at cost 0 instead of the agent's price.
* fix(a2a): keep one reply text per stream and bill every artifact and status reply
* fix(a2a): ignore a null error field on streams and run the stream per-token billing test
A stream event whose error field is null no longer zeroes the call's cost. The integration test for per-token stream billing now runs instead of skipping, and the URL agent SDK case skips when the optional a2a-sdk is not installed.
* fix(a2a): bill streams the client drops and skip the bridge's working placeholder
A stream the client closes or cancels mid-reply now logs its spend row at the agent's prices on both the bridge and URL paths, and a stream dropped before the agent sends anything still logs nothing. The bridge's "Processing request..." status no longer counts as reply text, so an empty bridge reply bills no completion tokens, as send does. Adds proxy-level tests that a bridge stream bills under the agent's card name.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(batches): cost an unmanaged gs:// batch through the deployment that created it
The create now stores the router deployment id as model_id inside the
managed object row's file_object JSON and CheckBatchCost reads it before
falling back to the first deployment matching the base model in the
gs:// path. Rows without the field keep the old lookup
* fix(batches): tolerate a non-string or malformed file_object when reading the stored deployment
* fix(batches): read the stored creating deployment from the writer, not a lagging read replica
* fix(batches): resolve the writer database once so the type gate count does not grow
* fix(batches): decode a dict-shaped file_object when reading the stored deployment, accumulate wire samples, annotate test locals
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): raise live budget counters for failure spend rows
A stream that breaks mid-flight writes its recovered cost to the spend log, but the failure hook
had already released the reservation and never touched the live counters, so an over-budget key
kept passing until the counter was next repaired from the database
Every charged failure row with a cost now raises the same counters the success path raises. The
error waits at most a second for them, so a hung Redis cannot hold it back
Fixes#35568
* test(proxy): cover failure spend counters across budgets, endpoints and outages
Integration cells for team, user, end-user, tag and window budgets,
concurrent streams, billed retries, native /v1/messages, Bedrock guardrail
cost, a Redis stall on two workers and disable_spend_updates
* test(proxy): check stalled failure counters and use the real spend writer
- The Redis stall cell now reads the burst key's live counter and requires at least the full failure spend
- Unit tests drop the charging writer double, bare dicts, list mutation and wall-clock timestamps
- Partial-stream test models are frozen
* test(proxy): move the stalled counter bound reason into the assert message
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(spend): keep a NaN or infinite cost from wedging daily spend writes
A non-finite provider-reported cost now counts as not reported, so the token price bills the request. Any other non-finite cost is logged and recorded as 0, and spend log JSON writes NaN and Infinity as null so Postgres accepts the row.
* fix(spend): price non-finite provider costs on every route and drain parked NaN batches
Perplexity, OpenRouter pass-through, and Responses streaming now treat a NaN or infinite reported cost as unreported, so the token price bills the request. A daily spend batch parked in the Redis buffer by an older version with a NaN spend now drains as 0 instead of failing every flush. The cost header never carries NaN or Infinity, and a set holding a non-finite float no longer breaks spend log JSON
* fix(spend): record non-finite savings as 0 and warn when a parked NaN batch drains
An overflowing price on a compressed request made compression savings infinite, which still wedged the daily spend batch. Every savings column now records a non-finite value as 0 with a warning. The Redis drain warns when it reads a NaN an older version parked, since the spend summed into it is lost from daily spend. The writer test now runs the real payload builder and spend log insert
* fix(spend): keep non-finite costs out of headers, responses, and admin JSON
Writing NaN as null is now opt-in for spend log columns, the Redis buffer, and
streamed frames, so admin writes and logging integrations keep their output.
The cost calculator records an overflowing total as 0 with a warning, cost
headers leave out non-finite values, a non-streaming NaN usage.cost is dropped
instead of failing the response, and a parked NaN spend log row reads back as 0.
* fix(spend): keep caller-set call ids from splitting the non-finite cost warning
The overflow warning strips newlines from litellm_call_id, and neither
non-finite cost warning prints the raw value, which said nothing beyond
"non-finite"
* fix(spend): drop non-finite Responses and OpenRouter costs and the breakdown of an unpriceable request
* fix(types): drop a usage cost breakdown that holds a non-finite number
* fix(streaming): keep a dropped non-finite final usage cost from falling back to an earlier chunk's cost
* fix(spend): keep the dropped-cost flag out of reach of provider usage keys
A provider usage key named dropped_non_finite_cost shadowed the Usage method on the instance, so the stream builder raised and dropped the request's spend. The builder now reads the flag through the class, and Usage sets the flag after copying provider keys, so a _dropped_non_finite_cost key cannot set it either.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(usage): show subscription-covered providers and account fees separately
* fix(usage): write only the named fee columns and sync the root schema
The fee update now writes just the columns the request names, so two admins changing
different fields at once both land, and the dashboard edit form sends only what changed.
The root schema.prisma carries the new table like the two copies, and a billing period
anchored in the calendar's last month ends on December 31, 9999 instead of failing.
* fix(usage): register the subscription table span and keep provider logic under llms
* fix(subscription_accounts): register the fee routes as management routes and reject an empty account id
* fix(usage): run the new tests in CI and close subscription card review gaps
Adds the subscription test file to its unit shard, sets the migration lock timeout, rejects infinite fees, keeps the card from rewriting login files, lists each deployment once without aliases, shows the provider badge on entity views, and asks before removing a fee
* fix(usage): resolve named-credential ChatGPT accounts and keep admin viewers read-only on the fee card
* fix(usage): count a subscription fee only for accounts behind a current deployment
Usage listed a fee row for an account no deployment uses while another ChatGPT
deployment existed and dropped it when none did. It now lists only accounts behind a
current deployment in both cases. Adds a real proxy and Postgres integration test for
the fee API and for ChatGPT Responses rows on a deployment's own login, and names the
card's multi-component test with the integration suffix.
* fix(usage): keep a stored subscription fee listed after its account's deployments are removed
The usage route lists every account behind a subscription deployment plus every account carrying a stored fee, and reads the fees on every proxy, so a fee stays visible, counted, and editable for past ranges after its last deployment goes. The integration test now polls every day the calls spanned, so a UTC midnight crossing no longer fails it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* fix(proxy): stop answering with the literal string "None" on the spend and management routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param
* fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param
A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None".
* fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400
* fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap
* fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param
* fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does
Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error.
* test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor
* fix(proxy): bill streamed /v1/completions without stream_options like chat
* test(proxy): mark the text completion stream usage test variable Final
* fix(completions): drop Azure's empty prompt-filter frame when the proxy asked for stream usage
* test(spend): drop the module docstring from the text completion stream usage test
* fix(proxy): keep the internal stream usage strip flag out of the logged request body
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: drain shutdown callbacks and persist pending tag spend
* fix: catch shutdown timeouts on Python 3.10
* fix: share shutdown callback budget with HTTP draining
* fix: type shutdown cancellation tasks and document its clock
* test: retain callbacks after bounded worker cancellation
* test(proxy): inspect the lifespan body after the SIGTERM wrapper split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): keep the success callback enqueue on its own task
The synchronous enqueue spawned the logging worker loop inside the awaited
call and ran callbacks that per-call loops used to drop, which hung the SDK
callback tests on the autouse worker flush. The shutdown worker flush covers
the queued callback on its own, so the one-tick task is restored
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback
The aggressive clear ran the rejected callback and marked it done against
the queue it was never put in, so flush() could return while a real
callback was still running and the shutdown drain cancelled it early. The
rejected callback is now re-enqueued once the clear made room, and only
runs uncounted when the queue is still full
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(logging_worker): build the aggressive-clear run set immutably
Final tuple instead of an appended list, with the re-enqueue in a
_requeue helper, and Final annotations on the overflow regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): track overflow callbacks through shutdown
* chore(logging): explain shared callback ownership
* test(logging): isolate deferred callback completion
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): resume background response streams with stream and starting_after
Retrieving a response with stream=true and starting_after=N now returns the
replayed event stream from OpenAI and Azure instead of one JSON object, on the
SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id}
Providers that cannot replay a stored stream drop the params under drop_params
and raise UnsupportedParamsError without it. Malformed values answer 400 with
the param named. A replay that reaches its terminal event settles the
background response's bill once and passes the provider's usage through
without estimating one. An empty replay (cursor at the end) ends normally
* fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams
A replayed background stream now gets the same response id encryption as a
created one, so the key that made the response can reuse the id. Azure AI
Foundry deployments resume streams like Azure ones, and the chat-bridged
Responses iterator closes its wrapped stream instead of raising.
* fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve
A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query.
* fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params
* fix(responses): keep deployment stream off retrieves and reject oversized cursors
* fix(responses): keep router default stream off plain retrieves
* test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing
* test(responses): replay a real stream in the billing test and type the fake wrapper chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): audit #46997
Add request-path cells for the global proxy spend counter: a key with no user spends toward litellm max_budget, and two proxies sharing Redis add every request to one counter that matches the proxy budget row
* test(proxy): wait for the recorded budget spend and shared counter before checking them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds integration cells for a different key reading the response, message redaction, a rotated, deleted, or team-deleted creator, and a creator whose Langfuse keys resolve to nothing
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(daily-activity): read guardrail spend per key from the daily guardrail usage table
Adds DailyActivityRepository.guardrail_spend, which sums LiteLLM_DailyGuardrailUsageUnits.cost by date, guardrail and key. A guardrail row counts only when its key has a spend row on that day under the same scope filters as the rest of the daily activity response, so user, key and date filters apply to guardrail cost too
* feat(daily-activity): add a per-guardrail spend breakdown to /user/daily/activity/aggregated
Each day now has breakdown.guardrails[<guardrail_id>] with metrics.spend and an api_key_breakdown carrying each key's spend and metadata. Guardrail entries only report spend, since that is the one metric stored per key. The breakdown is on the user endpoint only, is empty when a model filter is set because guardrail cost is not stored per model, and lists only keys within api_key_limit while the guardrail total keeps every key. spend and budgets are unchanged
Resolves LIT-9785
* feat(daily-activity): add the guardrail breakdown to team and organization activity, with guardrail names
/team/daily/activity/aggregated and /organization/daily/activity/aggregated now return breakdown.guardrails the same way the user endpoint does, since each key belongs to one team and one organization. Each guardrail entry also carries metadata.guardrail_name, read from LiteLLM_GuardrailsTable for guardrails created through the UI or API and falling back to the breakdown key for config.yaml guardrails, whose key is already their name
* fix(daily-activity): keep daily activity up when the guardrail breakdown query fails, and treat an empty model filter as none
A database error from the guardrail spend query now logs a warning and returns the response without guardrails, where it used to fail the whole aggregated response with a 500. An empty model= no longer drops the guardrail breakdown, matching how the rest of the endpoint already ignores it. The daily activity golden files and fixtures gain the empty guardrails key every breakdown now carries
* refactor(daily-activity): fold guardrail breakdown in a worker thread and drop docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(daily-activity): report untracked guardrail units next to spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): cover the batch cost poller gaps left after #47190
* fix(batches): keep unbillable batches from blocking file deletes and new-batch polling
The file delete check now ignores batches past MANAGED_OBJECT_STALENESS_CUTOFF_DAYS, a provider 404 for the output object retires the batch, each poll page takes the arrival lane first, a batch expires only after the poller visits it past the cutoff, and the compare-and-swap matches batch_processed instead of updated_at
* test(batches): cover the full arrival page and a failed expiry write in the batch cost poller
Old batches still get polled every other cycle while new batches fill the page, old batches that fill a page never push a new batch out after boot, and a failed stale_expired write no longer stops the rest of the page from being polled
* fix(batches): share the poll page between old and new batches and hold output files until billed
Each lane now gets at least half of the poll page, a lane with nothing left gives its half to the other, and a lane read that resumed mid-lane wraps to the start, so neither old nor new batches can starve the other.
The delete guard applies the 7 day cutoff to input files only. Output and error files stay locked until the batch is billed or retired, and stale_expired rows no longer lock anything.
Only a lasting provider failure expires a batch past the cutoff, so a timeout or a 5xx no longer gives up on a batch that may still bill. Past the cutoff a lasting failure, a lasting cost-tracking failure, or an output file id that never appeared retires a finished row instead of polling it forever.
* fix(batches): retry refused finished batches past 7 days, expire unfinished ones on a failed retrieve
A finished batch the provider refuses after 7 days (rotated key, bucket 403, a 404 that does not name the batch) now keeps retrying so its spend is written once the config is fixed, instead of retiring unbilled. A batch the provider still reports running keeps polling. A batch that is not finished in the database becomes stale_expired once a retrieve fails after 7 days, so an unreachable provider is not polled forever. An S3 NoSuchKey for the output object now counts as gone.
* fix(batches): expire an unfinished old batch only after 3 failed retrieves in a row
* fix(batches): keep retrying an unfinished old batch through timeouts, 429 and 5xx
* fix(batches): wait 3 polls before expiring an old batch whose output id lags
* fix(batches): match a resource id in a 404 message followed by punctuation
* fix(batches): keep refusal and pending-output expiry streaks apart, and void a pending-output retirement on a client write
* fix(batches): give up on unsettled batches past twice the staleness cutoff
Replace the per-reason refusal and pending-output streaks with one give-up age, and inject a clock into the poller and the delete check so tests pin exact age boundaries
* fix(batches): keep polling finished batches past the give-up age so their spend stays recoverable
* test(batches): expect a restarted poller's first page to split between old and new batches
* test(batches): pin the delete-guard tests to a fixed clock and type a new journal list
* test(batches): type the provider helper and the storage 404 cases
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47358
Add integration cells for pass-through bodies and settings-endpoint mappings across workers
* test(proxy): wait for every worker to hold the saved mapping before probing
* test(proxy): compare passthrough bodies to the client body and probe each worker on its own connection
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A DeepSeek V4 stream that ends without a usage block now has a spend-row cell proving its prompt_tokens are counted with DeepSeek's tokenizer, the one path #47353's own cells did not reach
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds an integration cell proving a failed /v1/responses request counts its input from the newest compaction item on its failure spend row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(passthrough): let upstreams report the model name via x-litellm-upstream-model
A pass-through upstream that prices its own requests already reports cost and
tokens in x-litellm-response-cost / x-litellm-total-tokens. Product endpoints
that take no `model` in the body still landed under `unknown`, so their spend
could not be broken down.
When the upstream sends x-litellm-upstream-model, record its value as the
request's model on success and failure rows, overriding any body `model`. The
name is a label only: it never marks the request as upstream-priced and is not
used to look up a price. A blank or missing header changes nothing.
Refs LIT-9910
* fix(passthrough): keep the upstream-reported model name bounded
The upstream model name becomes a spend row, a daily rollup key and a metric
label. Use only the first element of a repeated header (RFC 9110 field
combining, which is how httpx joins it) and cut the name to
MAX_SPEND_LOG_MODEL_NAME_LENGTH, the limit the spend log already applies to
model names.
Refs LIT-9910
* test(passthrough): cover x-litellm-upstream-model through spend rows and rollups
Drives a config-defined pass-through against a scripted upstream and reads
back the spend row, the daily team rollup and /team/daily/activity. Covers the
header winning over a body model on buffered and streamed responses, failure
rows, a model-only header keeping cost_per_request, the no-header and blank
header cases, and repeated or long header values.
Refs LIT-9910
* test(passthrough): annotate new test locals with Final
Refs LIT-9910
* fix(team): keep the team reset schedule when a temp budget increase is set on a member inheriting the team default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect the temp-only member row to carry the team default cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reconcile team member total_spend with current cycle spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read /team/list as a list and drain the scripted provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(repositories): compose the membership reset writer and drop SQL text asserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: type the cascade repair test and flatten memberships without a stacked comprehension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: keep the reset job test file's existing typing untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): consolidate gateway and usage assistants as Aria Agent
* refactor(ui): give Aria an original blue sprite mascot
* feat(ui): prompt shared Aria from usage and retire duplicate usage agent
* fix(ui): keep Aria naming consistent and reachable across dashboard
* feat(ui): open Aria in a floating popup with suggested questions
* fix(ui): simplify the shared Aria Agent title
* fix: Send the include_usage chunk with an empty choices array
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Skip chunks with no choices when reading streams
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Strip usage from plain-iterator async stream chunks
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Expect empty choices on the usage chunk in the gemma tracing test
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage
The usage chunk now has choices=[] only when the caller set
stream_options.include_usage itself. When the OpenAI handler, the proxy's
always_include_stream_usage setting, or the harness added include_usage on the
caller's behalf, the chunk keeps its legacy single empty-delta choice, so
clients that read choices[0] on every chunk keep working.
The proxy and harness mark their injection with the private
_litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy.
Plain-iterator streams keep usage on their chunk when usage was not requested,
and the LLM Shield trailing flush ignores chunks with no choices.
* fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape
The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them.
* fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route
* refactor(router): move the stream usage injection check into router_utils
* fix(router): read stream_options from pydantic deployment params when marking injected usage
Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices.
* fix(proxy): keep the stream usage injection marker out of stored request bodies
* test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths
* test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): the billed background response cost reaches the creating key's and team's loggers
* fix(responses): send the billed background response cost to the creating key's and team's loggers
* fix(responses): bill an unreadable or rejected creator through proxy loggers, redact and honor disabled callbacks on the billed event
* fix(responses): follow the creator's message logging setting on the billed event and read team logging for non-key creators
The billed event no longer forces redaction. It follows the creating key's or team's turn_off_message_logging, else the proxy's, and redacts when the creator cannot be read or its settings are rejected. A creator without a virtual key hash bills through its team's settings. function_setup keeps its original callback split, and the billed path uses a separate non-mutating helper.
* fix(responses): bill rotated, deleted, and custom auth creators through their own logging and retry failed creator lookups
* fix(responses): follow a key's whole grace-period rotation chain to the live key for its billed charge's logging
* fix(responses): bill a background response whose creator cannot be read, without the creator's settings
* docs(enterprise): say the background response row is claimed before its creator is read
* fix(responses): keep the loaded creator rows' budgets and aliases when one creator lookup fails
The Gemini TTS billing and text completion guardrail tests read spend rows from the owned proxy's own database, which #47213 made private. The cursor wire test derives its effort levels from the gpt-5 cost map row that #43935 now enforces, and the Azure effort check accepts the unquoted level. The pin TTL test asserts the one hour pin #46411 writes for a tool-only mark, the video callback test decodes ids with the product decoder after #44614, and the two Gemini 3.8 flash stream cost cases bill audio at the text rate their rows imply
* fix(proxy): stop update_cache from resurrecting evicted auth cache entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update update_cache pins for synchronous write-back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): wrap long assert in update_cache tag test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover refill race and pin TTL with injected clock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count global proxy spend for every request with a user, cached on this worker or not
* fix(proxy): add global proxy spend with an atomic Redis increment that never re-creates a reset counter
The global proxy spend bump read the total, awaited, and wrote old + cost back
from a background task, so a budget reset that deleted the key in between got
the pre-reset total written back, and workers overwrote each other's totals.
RedisCache.async_increment_if_present runs INCRBYFLOAT only when the key
exists, keeping its TTL, and this worker's copy is then set from the shared
total with no await between its read and write. The bump now runs whenever a
proxy max_budget is set, matching the DB writer, and the armed post-call read
and the batch-read declaration it used are removed.
* fix(proxy): report the global spend Redis increment to service metrics and request traces
The global proxy spend increment now emits Redis service success and
failure events under the spend_counters target and passes the request
span through, so it shows as redis.incr spend_counters like the write
it replaced.
The unit tests run the real RedisCache increment against fakeredis
instead of mocks, and a new test checks the span lands under the
request span on success and on a Redis failure.
* test(proxy): keep the spend counter Redis fake off the network
* test(proxy): count increment_if_present as a shared-cache call in the key-family guard
* fix(proxy): keep the newer global spend total when Redis replies arrive out of order
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tests): read the stream stall cooldown spend rows from the owned proxy's database
The cooldown cells boot an owned proxy, which writes its spend logs to its
own scratch database since owned proxies went private, while the cells polled
the worker's database and never saw a row. Read each row through the gateway
that served the request, so the cells check the proxy they called.
* fix(tests): read passthrough spend rows from the owned proxy's database
The vertex regional uplift and passthrough stream pricing spend files boot
private owned proxies, then polled the worker database for spend rows that
only reach the proxy's own database. Read them through the owned proxy's
gateway instead.
Also hold the stream stall file's hold_headers peer past the longest row
wait, so a slow spend row cannot fail the peer teardown.
* fix(tests): check agent deletion on the owned proxy's database
The agent least privilege test deletes its agent through an owned proxy,
then checked the worker database for the row, which is always empty
there. Read it through the owned proxy's gateway so a delete that leaves
the row fails the test.
* fix(tests): read turn admission spend rows from the owned proxy's database
* test(responses): billed background responses count toward the creator's project and model budgets
* fix(responses): carry the creator's project and per-model budgets on the billed background response
* fix(responses): count a key's budget tier toward background per-model budgets
* fix(responses): bill a background response's project only while the key stays on its team
* revert(responses): keep billing the creating key's project after a team move
A key can only keep or detach its project, so the billed project always pairs with the team it was created under. This reverts commit ded4f084c5.
* fix(responses): bill a background response under the model name the client sent, with its project alias
* fix(responses): name the team's public model as a billed background response's model group
The model group lookup now passes the creating key's team, so a team model or team wildcard bills under the public name a normal call logs and per-model budgets on that name move
* feat(proxy): map baggage values to budget tags
* fix(proxy): preserve mapped tags in the parsed request cache
* fix(proxy): preserve mapper accounting and tag policy
* fix(proxy): preserve baggage tags across native transports
* fix(proxy): keep mapper rejection and passthrough logging safe
* fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing
Mapped tags now apply only to billed LLM API routes, skipping token counting,
model listing, MCP and A2A. They classify spend and budgets but never select
deployments unless a key, team or client set the same tag. WebSocket inference
returns 403 only while a required rule exists. A tag created with /tag/new resets
its spend counter so its budget counts from creation. The General settings editor
keeps a new mapping unsaved until every required field is filled in.
* fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews
* fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403
* fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values
* test(proxy): cover baggage tag mappings end to end
Adds integration cells for request_tag_mappings across chat, messages,
responses, A2A and custom pass-through spend, required and budgeted
rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags,
tag routing and policy isolation, a dropped tag read with and without
allow_requests_on_db_unavailable, and the /tag/new spend counter reset
* fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags
Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags.
* fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies
Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering.
* feat(proxy): parse baggage headers into request tag mappings
* fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes
* fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset
Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request
* fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage
Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag.
* test(spend): cover string metadata.tags under mapped baggage
* fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings
Mapped baggage tags are applied again after the header and key tag merges, so
a header or key tag that replaces a string metadata.tags no longer hides the
mapped tag from the tag budget check. An empty or false metadata.tags is read
like a missing one, and a tags list with non-string items is kept as sent with
the mapped tags added after it, instead of failing auth with a 401
Local token counts for DeepSeek V3 and V4 model names now use DeepSeek's own bundled tokenizer instead of cl100k_base, and tool definitions on V4 names count with V4's chat-template tool framing
test_router_cooldown_failure_row.py boots an owned proxy, which writes to a private database since #47213. The file was written on a base without #47213 and read its rows from the worker database through read_rows, so all 13 cases timed out with State did not converge: []
Every row read now goes through Gateway.rows on the owned proxy's gateway, the same fix#47410 made for the stream stall cooldown cases
* fix(speech): correctly propagate spend logging and usage for Gemini TTS bridge
* fix(speech): call Gemini once per TTS request and log the bridge's token usage
aspeech re-ran speech whenever the first result was not a coroutine, so the
speech-to-completion bridge sent every Gemini TTS request upstream twice. It
now returns the sync result.
The bridge's inner completion runs as an internal sub call, so only the
speech call fires success callbacks, and the completion's usage rides on the
audio response so the spend log and usage_object carry real token counts.
Replaces the separate inner Logging object and the cost_calculator fallback,
which never reached the spend row.
* fix(speech): flag a cached Gemini TTS reply as a cache hit so it logs at no charge
* fix(speech): price a cached Gemini TTS reply's savings from its token usage and keep its cache key
* test(integration): cover Gemini TTS speech billing, logging, cache hits and deployment TPM
* test(integration): start the speech TPM cell in a minute's first 10 s so the two requests never cross a minute
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): record cooled deployments on the all-cooled failure row
A RouterRateLimitError now carries the cooled deployments of the route the
request resolves to for the caller's team (model group, alias, wildcard or
team route), each with its masked api_base. get_error_information copies them
into error_information.error_cooldown_deployments so the SpendLogs failure row
and every logging callback can tell which deployments were cooling
* fix(router): name cooled deployments from the router's own resolved route
Use the deployments the router resolved for the request, before health and
cooldown filtering, instead of re-deriving the route. Alias chains, the
provider-model fallback, wildcards and team filtering now match routing by
construction. A routing-strategy refusal records null.
* fix(router): name only the cooled deployments tag and region routing allow
The refusal's cooled-deployment list now leaves out deployments the
request's tags or the end user's allowed region keep it from, so a
tenant's refusal row no longer shows another tenant's api_base
* fix(router): keep failure logging working for refusal subclasses that skip init
A RouterRateLimitError subclass that never calls its parent init now logs a null cooled-deployment list instead of breaking the failure payload
* test(router): call the refusal routing helpers directly
The router coverage check needs each router function called by name from a unit test
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns.
* fix(responses): bill the usage a failed native WebSocket turn reports
* fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect
The v3 limiter stashed whichever litellm_call_id the request body carried as
the slot owner, so a client-supplied id on a pass-through body left the
proxy's own success and failure events unable to release the slot. The owner
is now the proxy's logging call id. A WebSocket connection whose turns log
under their own ids never settles the TPM, ITPM, or OTPM estimate the
connection reserved, so the disconnect release now refunds any reservation
no callback settled, once.
* fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect
A failed turn's reported usage now counts toward the key's TPM and the
project ITPM/OTPM scopes the same way a success does, and closing the
socket hands back only the estimate still unused in the window that
holds it, so a window that already rolled is left alone.
An id-less error event is booked against the newest pending turn, the
one the backend rejected at parse time, and a turn that nests its input
under response logs its user message. A connection's one budget
reservation goes to the first turn that settles it, every later turn
charging its cost directly.
* test(proxy): expect the settling call id on a pass-through budget reservation
* fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM
A native Responses WebSocket turn reaches the rate limiter's success
callback with its raw event list as the response and no
combined_usage_object, so the turn's tokens never reached the key's TPM
window or the project's input and output token limits. Read the usage
from the turn's events when the call type is a Responses WebSocket turn.
* fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas
The limiter's failure hook looked up the parallel slot by the request
body's litellm_call_id, while the slot belongs to the proxy's logging
call id, so a pass-through request a pre-call hook rejected held its
slot for an hour when its body carried an id. Both now resolve the id
the same way.
Each response.create frame already charges its estimate to project
ITPM/OTPM, so the turn's own usage no longer adds a second count.
The cost callback's failure hook read the body's litellm_call_id
through a strict model and raised on a non-string value, which dropped
the failure row. It now reads a string id or none.
* fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right
Free a rejected request's parallel slot through the request it runs in, and skip
native Responses WebSocket connections, whose disconnect frees the slot. Count
WebSocket turns toward the connection's tag TPM. In collector mode, claim the
shared budget reservation before the spend event is built, so only the first
turn settles it. Store each turn's own frame as its request body, and send an
id-less error to the oldest waiting turn unless an earlier turn is streaming.
* fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag
* fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls
* fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes
* fix(router): count native Responses WebSocket turns toward deployment limits and budgets
* fix(router): re-check deployment budget and RPM on each Responses WebSocket turn
* test(router): list the WebSocket turn admission method as covered through its partial
* fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn
* fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure
* fix(proxy): log the routed model group on a WebSocket turn's failure row
* fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn
* test(integration): cover per-turn admission on Responses WebSockets
Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the
project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP
controls for TPM counting and the failure row model_group
* fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails
A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success
callback such as the response cache no longer runs each CustomLogger's sync success hook on top of
the async one. That second run counted every turn's tokens twice toward deployment TPM.
A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket
closes with 1011, and the error becomes the socket's failure, on both the native and the managed
relay, instead of the socket dropping with no frame and no failure row.
A refused turn's failure row now carries the request's tags from the router's metadata, the same
way it already carries the model group.
* test(integration): cover a crashed WebSocket turn check and failure-row tags
A turn whose admission check raises gets a 500 server_error frame, the
socket closes with 1011 and its one failure row carries the error, on
both relays. The refused turn's failure row keeps the caller's tags, and
HTTP failure rows on chat, responses and messages keep them too.
* fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows
A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket
now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which
a client that sends its next response.create early would lose.
A refused turn's failure row now takes the router's tags only when every tag is a string, so the
fallback never copies a caller's unvalidated tags onto the row.
* test(integration): cover a turn still streaming when the next WebSocket turn's check crashes
A native turn still streaming when the next turn's admission check raises now has its own cell. It
must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for
one more turn, so a relay that stopped checking turns after a crash goes red.
* fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets
A turn after the first now runs the project quota check, then the proxy's
pre_call guardrails, then the deployment's turn admission, on both the
native and the managed relay, the order HTTP calls use. A turn a guardrail
blocks gets the guardrail's error even past the budget and no longer counts
toward the deployment's rpm or runs its pre-call checks.
* fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it
A turn the per-turn guardrail passes and the deployment then refuses now writes
its own failed spend row with that turn's guardrail information, on both relays,
the way a guardrail-blocked frame and an HTTP refusal already do
* fix(responses): bill a refused WebSocket turn on one row only
A refused turn whose guardrails recorded a cost is already billed on its own row,
so it no longer also becomes the connection's close-time failure row
* fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row
The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them
* fix(responses): bill a WebSocket's refused turns when the session limit closes it
* fix(responses): bill refused WebSocket turns when the session limit cancels the relay
* fix(responses): free the WebSocket slot at the session limit without waiting on the failure row
* fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot
* fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown
* fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after
* fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it
* fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close
* fix(proxy): always commit spend again after the shutdown wait for pending logging tasks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): price streamed pass-through calls with the regional uplift
The streamed Vertex pass-through handler builds the response with
stream_chunk_builder, which stamps a cost into _hidden_params without the
vertex_location. The handler then prices the response again with the
location from the URL, but only into kwargs, and the logger prefers the
stamped cost. Streamed calls on a regional endpoint were billed at the
global rate.
Stamp the location-aware cost on the response in the shared
generateContent payload builder so the logger records it.
Fixes#44755
* style(vertex_ai): keep the response_cost assignment within the 120-char limit
* refactor(vertex_ai): set the passthrough response cost with set_hidden_param
Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this.
* test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed
* fix(proxy): price streamed Vertex pass-through spend like the non-streamed call
Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls
* fix(proxy): price custom-priced Vertex pass-through deployments at their own rates
The stamped response_cost from the generateContent logging helper is returned before the
logger checks custom pricing, so a deployment with custom input/output rates was billed at
the cost-map price. The helper now passes custom_pricing and the deployment id into
completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls,
and deployments without custom pricing keep the regional uplift.
* fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price
* fix(proxy): give the new pyright ignore a reason for the LIT004 gate
* test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included
* fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same
* fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini
A top-level base_model in a pass-through body now reaches the logging
params only for a key or team with allow_client_pricing_override, so a
plain key cannot pick a cheaper model's price once the logger prices
these calls. The test that pinned every litellm-owned body name into the
logging params now leaves base_model to its own key-gated test.
The Vertex handler now classifies a non-Google upstream host as Vertex
only for Vertex resource paths, so a /gemini stream through a
GEMINI_API_BASE gateway is priced as Gemini without the regional uplift.
Adds the audit's integration spend test for Claude on Vertex, /v1beta,
the named custom-priced /gemini deployment, and a cut OpenAI stream.
* test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing
The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path.
* fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model
The Vertex pass-through handler picked Gemini pricing from the URL path,
so a configured pass-through to a Vertex gateway with a short path (no
/projects/ segment) was billed as Gemini, and its Claude streams at $0.
The provider now comes from the route: a call through /gemini or to
generativelanguage.googleapis.com prices as Gemini, and everything else
as Vertex, the same rule main used. Matching the GEMINI_API_BASE host
was rejected because one gateway can serve both APIs, and the path rule
was rejected because short gateway paths look like Gemini paths.
A non-string base_model (a number, list or object) made the cost
calculation fail and bill $0 once the stamped stream costs were gone.
get_base_model_from_metadata now ignores it and falls back to the
deployment's base_model, so the call bills the list price. A 400 was the
other option, but the logger runs after the upstream answered.
* test(integration): bill a non-string body base_model at the list price on every pass-through route
* fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info
A key allowed client pricing can send metadata.model_info in a pass-through
body. A non-string model_info.base_model made the cost lookup fail, and
once the stamped stream and Gemini costs were gone those calls billed $0
on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores
a base_model that is not a string, the same rule the root base_model got,
so the call bills the list price.
A model_info that is not an object (a string or a list) broke several
logging steps that read it as a mapping, so a non-streamed /gemini call
fired no success callback and no pass-through call wrote a spend row. The
pass-through route now leaves a non-object model_info out of the logging
metadata, so the call bills the list price. A 400 was the other option,
but it would turn calls that answer 200 today into errors, and guarding
every reader of model_info would spread the same check over many files.
* fix: reject a non-object model_info before the provider call again
The previous commit made the base_model lookup return None for a
model_info that is not an object. That lookup also runs during call
setup on every router route and SDK entrypoint, where such a value used
to fail before any provider request. With the early return the call went
upstream, the provider billed it, and the request then failed in a later
reader that still expects an object. On /v1/messages and /v1/responses
the key was also charged for a response the client never got.
The lookup again reads base_model from any truthy model_info, so a
non-object value fails during setup the way it did before this PR. A
non-string base_model inside an object still bills the list price, and
the pass-through route still drops a non-object client model_info at
intake, so pass-through calls keep billing the list price.
* fix: match the Gemini API host exactly when picking the pass-through cost provider
The provider check accepted any host ending in
generativelanguage.googleapis.com, so a Vertex host such as
vertex-generativelanguage.googleapis.com was priced as Gemini and lost
the regional uplift. It now accepts that host or a subdomain of it.
* test: pin that a subdomain of the Gemini API host keeps the Gemini price
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): stop GET /__observations from draining the upstream log
GET /__observations used to empty the queue, so a second read in a test
saw nothing and a "never reached upstream" check could pass whether or not
the request was forwarded. GET now returns every request since the last
reset and DELETE /__observations is the reset. An autouse fixture resets
the worker upstream before each test, which also covers tests that read
without the gateway fixture.
Discarded reads used as resets become DELETEs, helpers that accumulated
drained reads now read one snapshot, and tests with several phases reset
between them. The three vacuous negative checks (rag allowlist, guardrail
opt-out chaos, xAI batch output) now assert the absence on the same read
that shows a request which did reach the upstream.
* style(integration): space the observations unit test like its neighbours
* test(integration): give each owned proxy a private database and Redis by default
owned_proxy and owned_proxy_process now boot on a scratch database and a
per-proxy redis-server unless the caller opts into the worker's state with
state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips
DATABASE_URL. Under the replica profile the private database gets the
writer and reader roles so the reader path stays covered.
Gateway carries the database_url its proxy writes to, and the Scenario
cleanup checks and delete_key_if_present read from it, so they keep
checking something on a private database.
The 189 test files with owned proxy sites that depend on the worker's
database or Redis opt in with a shares_worker_state module marker, which a
module-scoped conftest fixture turns into the shared state. The 7 helper
modules with such sites opt in per call, since conftest fixtures and other
test files reach them.
* test(integration): reach session fixtures with the shared marker and keep the worker retry default
The module-scoped fixture that applied shares_worker_state ran after
session-scoped fixtures, so proxies those booted went private in marked
files. The marker is now applied around each marked item's setup, call
and teardown, which covers fixtures of every scope.
Owned proxies on the worker database inherited the worker gateway's
router_settings.num_retries of 0 from it. A private proxy now gets the same
default in its config copy unless the config sets num_retries itself.
Five providers files read the owned proxy's spend rows or response cache
through the worker's database or Redis and opt into sharing.
* feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response
Count endpoints now say where each number came from in litellm_count_source
(provider, fallback_model, local_estimate) and name the fallback model in
litellm_count_fallback_model. When the provider answers 400 or 404 for a model
it cannot count, the proxy counts the same request with token_count_fallback_model.
token_count_framing adds a per-model constant to local estimates. Both settings
ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and
Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info.
* test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests
* test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing
* fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown
Moves the cost map lookup off the event loop
* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty
* fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing
A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback
failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries
no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty
token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and
rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as
their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too.
* test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates
* fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment
A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock
calls and about 200 ms on every count, since Opus 5.5 cannot count there either.
The Kimi K3 e2e also runs the large system prompt shape now.
* test(count_tokens): use the shared integration port allocator in the fallback wire test
* fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today
* test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings
Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it
* fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body
A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases.
* fix(count_tokens): answer the provider's error whenever a framed estimate raises
A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error.
* fix(count_tokens): load the custom tokenizer inside the guarded framed estimate
A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors.
* fix(count_tokens): look up framing under the deployment's custom_llm_provider
* fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error
A framed deployment whose custom_tokenizer is not cached locally made every
count wait out the Hugging Face hub's whole retry cycle when the hub could not
be reached, and a burst of those loads held the default thread limiter that
every count's cost map lookup also needs. Counts on a worker now share one
download per minute, a failed download is reused for the rest of that minute,
and after a provider error a count waits at most 5 s before answering the
provider's error. The swallowed-estimate warning logs only the error type.
* fix(count_tokens): keep one tokenizer download per key running and load the local route per request
* test(count_tokens): type the scripted counter and wait on events instead of wall time where possible
* fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error
Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500
* feat(proxy): count tokens with token_count_model in its pinned region
token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable
* fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys
* fix(proxy): keep the upstream error envelope when adding the region hint
* test(proxy): annotate the token count setting echo test
* ci(cost map): require non-blank token_count_model strings and test the region rules
* fix(count_tokens): answer the token_count_model's error even when framing is set
* fix(count_tokens): keep the provider error for Google contents instead of a framed estimate
* fix(count_tokens): estimate Google contents the route translated to messages
* fix(count_tokens): keep the provider error for Google contents with non-text parts
* test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock
* fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags
* feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer
* test(bedrock): keep the Kimi K3 image check in-process
* fix(count_tokens): join the region hint to an upstream message that already ends with a period
Also pins that a blank token_count_model in config falls through to the cost map default
* fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts
The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts
The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool
A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning
* fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them
A flat Responses API function or custom tool becomes the chat tool the
Responses route sends to inference, and every other tool goes to the
Converse body as is, so hosted tools Bedrock drops are left out of the
count instead of being counted or turning it into an estimate.
* test(count_tokens): annotate new token counter test locals with Final
* fix(count_tokens): keep shipped count regions inside the deployment's geography
A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added
* fix(count_tokens): resolve count settings off the event loop
* chore: Add failing test case
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Restamp nested response.model in streamed Responses events
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* chore: Add tests
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* chore: Run format and lint, and fix
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Type nested Responses restamp to satisfy basedpyright gate
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): restamp the nested model on streamed response.queued events
Background streams send response.queued as a GenericEvent, so its nested
response.model still carried the backend name. Also drop the new comments
* fix(proxy): restamp a nested response kept as a dict on streamed lifecycle events
* test(proxy): drive typed events through the nested restamp no-op test
* fix(proxy): restamp the nested model on guardrail block frames in streamed responses
* test(proxy): assert a cache-polled background response reports the requested model
* fix(proxy): restamp nested model on dict events and on every SSE data line
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(test-harness): preserve sibling auth modules in isolated worker configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect servers in MCP connect-flow descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect ptu_hours in daily activity metrics after #43043
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): kill every serving worker once the SCIM push blocks on the row lock
* fix(cost): price exact cost-map names before price-free capability rule matches
* test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price an explicit base_model under its own provider after candidate reordering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): retain priced generalized candidates and skip explicit-pricing lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): assert custom pricing makes no remote model-info requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): keep the per-candidate pricing provider final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments
* fix(cost): sort cost candidates from the static cost map and rules, without provider lookups
* test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* test(integration): expect the tool search call replayed beside the function call for Anthropic targets
* test(integration): wait for the team spend flush before killing a worker
* test(integration): poll for the S3 attempt the batch sinks write concurrently
* test(integration): queue the terminal object and its sibling into the same s3_v2 flush
* test(integration): time the long Bedrock version against an equally long unversioned name
* test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it
* test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries
* fix(responses): scope session history lookup to the referenced key
* Implement SQLite-backed DB for session tests
Added a SQLite-backed database class for testing and a new test to ensure session isolation between different API keys.
* fix(proxy): honor caller litellm_session_id in spend log session_id
Updated docstring to clarify session ID resolution logic for spend log rows.
* test(proxy): caller litellm_session_id wins over the trace id
* style(proxy): fix blank lines around new tests
* fix(proxy): read the caller session from the logging payload for spend logs
* fix(responses): match user and team too when replaying session history
* fix(proxy): keep bridged Responses chains whole and scope session history by owner
* fix(proxy): keep a metadata session next to a trace header and let session headers win
* fix(proxy): send the trace-header session only to spend logs and let session headers win beside a trace header
* fix(proxy): drop a client-sent caller session and keep it out of requester metadata
* fix(proxy): keep the trace id as the spend log session when the caller session contains a NUL byte
* fix(responses): keep a continued bridged chain's session under request_correlation_in_logs
With the flag on, the caller session outranked the chain session the bridge sets for a continued Responses chain, so a chain whose session changed or first appeared on a later turn split. The continuation now logs under the chain's session and drops the turn's caller session key, as on main. Adds the /audit integration cells for the spend log caller session.
* fix(responses): log a continued bridged chain under its session through the spend-log key only
The previous commit set the chain session as the turn's litellm_session_id, which also moved the OTel conversation id of a continued turn away from its trace header. The bridge now writes the chain session into the internal spend-log session key of the metadata the spend log reads, so only the spend log session changes, as on main. Adds an OTel cell for a continued chain.
* test(spend): cover bridged chains that share one caller session
* test(spend): cover the caller session on configured pass-through routes
* test(spend): type the SQLite fake and split stacked comprehensions in the caller session tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(interactions): keep background interaction bills through crashes, late restarts, and deployment keys
A settlement claim is now a lease, so a row whose settler died between claim and
outcome is claimable again after the lease and boot resumes it. A resumed poll
gets the full poll window and fetches with the serving deployment's credentials
from the live router. The stored create context keeps the request tags and the
request metadata that the creating worker's own spend row used
* fix(interactions): bill a lapsed claim's takeover only when no spend row exists
A settler that billed and then died, or failed to record the outcome, left the row unsettled, so the replica that took over its lapsed claim billed the interaction a second time. A takeover now looks up the spend row keyed by the interaction id and records the row as billed when it finds one.
* fix(interactions): settle Vertex AI and edited deployments, never double bill with spend logs off
A takeover with spend logs off records the interaction unsettled instead of
billing it again, and a release records its outcome before freeing the budget
reservation. A resumed or live fetch carries the deployment's Vertex AI project,
location, and credentials, and a stale deployment id falls back to the model
group only when one same-provider deployment serves it. A poll stops at the
first 400, 401, 403, or 404, string request tags no longer fail the create,
and settled_at gets an index built concurrently.
* fix(interactions): leave a live replica's interaction to it when a resumed poll cannot read it
A replica that boots with credentials unable to read an interaction got a 404 on its resumed poll and settled the row as unsettled at once, so the replica that created it, still polling, found it settled and never billed it. A resumed poll that gets a 400, 401, 403, or 404 now stops fetching and keeps checking the row until its poll window ends, and settles it as unsettled only when no other replica did.
* fix(interactions): keep the creator's metadata on a resumed bill and retry outcome writes inside the claim lease
A bill another replica writes now carries the metadata the creating worker's own
spend row is built from, so a client's metadata.model_group no longer relabels it.
A failed outcome write is retried for half the claim lease before the budget
reservation is left to a takeover, and the creating replica retries every fetch
error for its whole window again, as main does. The resume integration case now
checks that a resumed fetch uses the deployment's own key.
* fix(interactions): resume background interaction polls only after database models load
Boot resume ran before the proxy loaded the models stored in the database,
so a resumed poll for a model added through /model/new or the Admin UI
fetched with env credentials. A 4xx there stops a resumed poll, so the bill
was lost. The settlement now installs after the database models load.
The integration test resumes a database model with an eager first fetch
and fails with the old startup order.
* fix(interactions): renew a settling claim so a slow settle is never taken over and billed twice
A settle runs every success callback in turn, and one slow callback could
hold it past the 5 minute claim lease. A booting replica or a DELETE then
took the live claim over, found no spend row yet, and billed the
interaction a second time. The settler now renews its claim every third
of the lease until it records the outcome, so only a settler that stopped
renewing is taken over.
A takeover also read the spend log after writing its claim, so a failed
read left the row under the taker's own fresh claim, which its poll then
took for another settler's and left unbilled until a reboot. The spend
row is now read before the claim is written, so a failed read takes
nothing and the claim is retried on the next poll.
* test(spend): cover lapsed claim takeovers, rotated deployment keys, late restarts, and resumed bill tags
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn
* test(responses): mock the websocket slot release and mark the per-turn start time naive
* style(responses): keep the per-turn start time line within ruff format
* refactor(responses): type websocket turn events as read-only mappings
* fix(responses): settle the websocket budget reservation on the first turn only
Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly
* fix(proxy): keep the legacy limiter's websocket slot until disconnect
A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* test(count_tokens): expect the converted Responses input in the budgeted-key count integration test
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): narrow the provider response id with isinstance instead of cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): validate callback fields for current type gates
* test(gcs_pubsub): assert provider response id in spend metadata
* fix(ollama): normalize reused response ids automatically
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
