Commit graph

22149 commits

Author SHA1 Message Date
mateo-berri
45a3c12de1 test(integration): expect a deployment's own drop list to beat a router default drop list
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
2026-11-13 21:34:12 -08:00
mateo-berri
7e181ec485 fix(router): keep max_retries router-owned so a deployment value cannot stack SDK retries on router retries 2026-11-13 21:04:06 -08:00
mateo-berri
20566c0eed fix(router): keep a deployment's own litellm_params over router default_litellm_params
Router defaults now fill only the keys a deployment leaves unset, so a deployment's own value (vertex_location, api_base, drop_params) beats default_litellm_params. The proxy token counters, drop-params checks, and the Entra cooldown check use the same order
2026-11-13 20:36:58 -08:00
moe-berri
752eb7c5b7
fix(opencode): stream Go Messages-protocol models through chat completions (#51715)
* test(opencode): cover Go Messages chat streaming

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): delegate Go Messages chat streaming to Anthropic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): preserve Messages stream usage and cleanup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 22:21:52 -05:00
Hoya
8ef451a187
fix: raise on failed OpenRouter non-streaming completions (#50880)
* fix: propagate OpenRouter errors and preserve failure usage

* test: cover OpenRouter errors and failure accounting

* fix: bill a recovered OpenRouter failure on the success log and parse the body once

A retry or fallback that recovers from an OpenRouter finish_reason error now
carries the failed attempt's usage and cost into the success log, so the key
pays for both calls. The response body is parsed once and handed to the error
check, and the failure-log test no longer waits on a 10-second deadline

* fix: carry a failed attempt's usage before each SDK retry

* test: cover OpenRouter's documented failure body without usage

Pins the recovered-failure test's start time to a fixed date and adds a regression test for the documented non-streaming error body, which carries no usage

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:02:00 +00:00
Nate Armstrong
cd2e4b6b60
feat(custom-logger-v2): add the contract for rust callbacks and bind the proxy request scope (#51526)
* feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0)

* feat(custom-logger-v2): add the rust contract for native callbacks and bind the proxy request scope

* fix(custom-logger-v2): add field projections and keep the request scope across copies

* fix(custom-logger-v2): pass the request scope to the async failure attempt_ended line

* fix(custom-logger-v2): add v2 callbacks from litellm.callbacks to the success and failure lists

_safe_add_callback_to_list dropped CustomLoggerV2 instances, so a V2 callback registered through litellm.callbacks got provider events but never log_prepare or attempt_ended. The manager now adds each V2 instance once by identity, and proxy startup fans V2 callbacks out like CustomLogger ones.

* fix(custom-logger-v2): contain a failing v2 callback on logging hooks

A V2 log_prepare writer that raised, or returned Abort or Respond, ended the line and escaped async_success_handler, so attempt_ended delivery, sync-for-async scheduling and the Redis flush were skipped. On hooks off the request path a V2 entry's exception or outcome is now counted and logged like a legacy logger's exception, and the line goes on. ContractViolation and BaseException still propagate, and legacy entries are unchanged.

* fix(custom-logger-v2): run blocking and awaited v2 callbacks on synchronous logging lines

run_plan_sync raised ContractViolation for a V2 log_prepare writer or any awaited V2 subscriber, the logging plane swallowed it, and every legacy logger at that hook was skipped. Legacy entries now always run in order as before. A V2 writer or awaited subscriber runs inline in list order through asyncio.run when the thread has no running event loop, so sync SDK calls apply its patch and finish it before the handler returns. On a thread with a running loop it cannot block, so it is delivered off-path and its patch is not applied. Lines without such entries take the unchanged legacy path.

* fix(custom-logger-v2): map a v2 abort to its http error at every proxy hook point

Only pre_call turned a V2 Abort into an HTTP error. during_call, post_call_success, post_mcp_call, list_models and the deferred stream hook raised V2Outcome, which reached the client as a 500, and post_call_failure, which never raised before, raised it too. Abort now raises the exception it carries or an HTTPException with its status, message and headers at those points, and at post_call_failure it becomes the transformed error through keep_first. Respond is still not wired and keeps raising V2Outcome. The pre_call routing test is updated to the HTTPException raise_v2_abort already makes.

* fix(custom-logger-v2): apply no-log to v2 logging subscribers without a request scope

The proxy request scope already carries no-log and caller opt-outs, so a V2 callback that honors no-log is skipped on proxy requests. SDK calls have no scope, and the logging plane ran V2 subscribers with an empty selection, so a no-log call was still delivered to them. Without a bound scope the logging lines now open one with no_log set when the call's litellm_params carry no-log, unless global_disable_no_log_param is on. Callbacks that do not honor no-log, and so every writer or abortable, are unaffected.

* fix(custom-logger-v2): deliver a failed attempt to v2 subscribers once per logging object

Callers such as MCP tool errors and Responses streaming run both failure handlers without for_async_request, so a V2 attempt_ended subscriber, now present in both failure lists, got the failure twice. The first failure handler to include V2 subscribers claims the delivery on the Logging object and the other skips them. Legacy callbacks are unchanged.

* refactor(custom-logger-v2): settle a synchronous v2 run through the shared gather and use aware times in the new tests

Drops the blind except the sync line used to capture a V2 callback's exception, reusing _gather_started, and gives the new logging tests timezone-aware times, so the strict ruff counts do not grow.

* fix(custom_logger_v2): await a single-entry v2 stage in the caller's context

A native handler in its own stage ran inside an asyncio.gather child task, so a ContextVar it set was lost when the stage ended, while a legacy callback's stays visible to later callbacks and the rest of the request. A stage with one v2 entry is now awaited directly, with gather's exception handling kept. Multi-entry parallel stages still gather.

* fix(custom-logger-v2): raise a native callback panic as a runtime error

* fix(custom-logger-v2): run a synchronous v2 line on a private event loop

* fix(custom-logger-v2): deliver an mcp tool failure to v2 subscribers from the async failure handler

* fix(custom-logger-v2): raise an abort's carried exception at the deployment hooks and keep the original failure at post_deployment_failure

* fix(custom-logger-v2): apply caller opt-outs to v2 logging lines without a bound request scope

* fix(custom-logger-v2): dedupe v2 callbacks in the event lists by logger key so a native built twice is delivered once

* fix(custom-logger-v2): log an unregistered abort error kind and keep the carried traceback at post_call_failure

* fix(callbacks): reject a selection name or alias that is not lowercase

* fix(custom-logger-v2): run v2 deployment callbacks on rust bridge deployment hooks

* fix(router): type the fallback event kwargs handed to the v2 dispatch

* feat(custom_logger_v2): type the contract fields as records and mark derived-only fields

Fields that the host derives from a hook's arguments now carry pydantic records on the Python side and matching serde records in litellm-callbacks, with records.json as the shared sample both sides check. Twelve new fields take bits 30 to 41, a write to a derived-only field is a manifest error, and pre_call can write budget_reservation and rate_limit_statuses in place. The payload reference field and the PayloadHost trait give Rust a handle to the logging payload instead of a copy

* feat(custom_logger_v2): derive typed fields from hook arguments and always build call facts

A slot table maps each typed field to a function over the hook's arguments. A derived slot runs only when a subscriber reads it, once per hook invocation, and off-path readers settle their reads before the hook returns. Auth fields come from the auth object only, budget_plan is claimed once, and budget_reservation and rate_limit_statuses write through to their carriers so the next pre_call subscriber sees them. Call facts are now built on unscoped logging lines and at post_call_success, post_call_failure and response_headers

* feat(callbacks-python): hand the logging payload to native callbacks by reference

NativeCallback stashes the shallow payload copy it is given and passes Rust a PayloadRef. The Python host renders a handle with a safe_dumps compatible encoder, on the caller's event loop while the hook runs and on the blocking pool after it, and dropping the last reference releases the stash entry. The encoder and its JSON helpers move here from the S3 logging bridge so every port shares one copy

* feat(custom_logger_v2): accept every value python accepts for typed members and serve logged key identity

Typed members keep the bool, int, float or str python holds through a Scalar
type on both sides, so ids, limits and priorities no longer lose values that
lax pydantic refused. A member the contract cannot carry is named in rejected
so a port can reproduce what python does with it.

Logging hooks get logged_key_identity, built from the payload metadata the
python loggers read on every route, while key_identity stays on request hooks
and keeps reading the auth object.

* fix(custom_logger_v2): read the live budget reservation dict and carry non-finite floats both ways

budget_reservation read the dict budget_reservation.py builds with a KeyError
on callback_bound. Undeclared keys are now ignored, callback_bound is a member,
entries keep a non-mapping item as null so capacity's model access group check
stays exact, and costs read as the float budgets computes with float(x or 0.0).

Float members cross as the raw json marker when they are not finite, and a
native verdict returns through json_to_py, so an infinite or NaN cost survives
a read and a write instead of turning into null.

* feat(callbacks): carry a python value as its str() text and truthiness in scalar members

Scalar gains a Stringified variant with the str() text and bool() of a value
that is not a bool, int within i128, float or str, so a port can reproduce
the label, path or log text python formats and the falsiness of an empty
container without confusing it with a real string. Budget reservations say
whether python held the entries as a list.

* feat(custom_logger_v2): carry non-scalar values and any iterable tags or entries across the boundary

A Scalar member now carries a list, dict, set, bytes, object or an int past
i128 as its str() text and truthiness instead of naming it in rejected, which
stays for values whose str() or bool() raises. request_tags carries any
iterable as the items prometheus iterates, and budget reservation entries
carry what budgets iterates with entries_is_list for capacity's list check,
never written back to the shared dict.

* feat(custom_logger_v2): say whether a stringified value hashes and whether it is an int

A Stringified scalar now carries hashable, from trying hash() at the
boundary, so a port can raise the TypeError capacity's priority lookup and
the limiter's descriptor identities raise for a list, dict or set and not
for a tuple or object. It also carries integer, so an int past i128 is told
apart from any other value whose text is digits.

* fix(custom_logger_v2): wrap a non-list guardrail mode as one mode like prometheus does

Prometheus treats any guardrail_mode that is not a list as a single mode, so a
tuple mode holds no str mode and a missing mode is one None mode. The
derivation unpacked tuples and rejected other non-str modes instead.

* feat(callbacks): let a native handler await typed host calls the python caller serves

* test(callbacks-python): drive a native handler's host calls through the python dispatcher

* test(custom_logger_v2): route a native logger with a host through the execution driver

* fix(custom_logger_v2): type the wrapped guardrail modes for pyright

* feat(custom_logger_v2): carry the python type name and hash error text of a stringified value

* feat(custom_logger_v2): serve the model group the logging handlers compute and key remaining limits by it

* feat(custom_logger_v2): tell a budget reservation that is not a mapping apart from none

* feat(custom_logger_v2): name malformed team model preferences in rejected with the text capacity logs

* feat(custom_logger_v2): serve a logging payload that is not a mapping apart from a missing one

* fix(callbacks-python): render null keys like a python dict merge, adding a missing key last

* feat(custom_logger_v2): carry an explicit output cap past i64 as a scalar

* feat(custom_logger_v2): serve the custom metadata labels prometheus derives from the logged payload

* feat(custom_logger_v2): serve the input sequence length bucket prometheus picks at attempt end

* feat(custom_logger_v2): carry the start and end times prometheus reads from the call details

* feat(custom_logger_v2): serve the limiter's body tags, session id, resolved agent and agent policy limits at pre_call

* fix(callbacks): export the limiter request record

* test(custom_logger_v2): give the type name cases stable ids and sort the record exports

* fix(custom_logger_v2): type the new derivations and tests for pyright

* fix(custom_logger_v2): drop team model preferences and serve the fallback_on_saturation handoff main's capacity callback reads

* style(callbacks): rustfmt the record exports

* refactor(custom_logger_v2): rename native callbacks to rust callbacks

native.py becomes rust.py, with native_logger, NativeLogger, NativeHandle,
NativeState and HostedNative renamed to their rust forms. The pyo3 classes
NativeCallback and NativeRequestState become RustCallback and
RustRequestState, and the tests, fixtures and messages follow.

* refactor(custom_logger_v2): rename slots to field sources

slots.py becomes field_sources.py: Slot is FieldSource, SlotFields is
SourcedFields, and with_slots, NO_SLOTS and slot_table_problem follow. A
table that only derives is named DERIVED_*_FIELDS, and a table that can
write back keeps SOURCES (PRE_CALL_AUTH_SOURCES, PRE_CALL_SOURCES). The
include_v2 comment moves down to the functions that take it.

* refactor(litellm_logging): place the v2 members of Logging beside what they serve

v2_scope, v2_auth and v2_failure_delivered join the other per-request
class defaults, v2_logging_scope and its opt-out helper follow
should_run_callback, v2_call_facts sits by the standard logging payload
builder, and claim_v2_failure sits above failure_handler.
bind_v2_request_scope checks for a Logging object directly instead of two
runtime protocols.

* refactor(proxy): rename v2_fields to v2_proxy_fields and say why it is separate

* refactor(custom_logger_v2): drop the logged_payload_not_a_mapping field

A logged payload that is present but not a mapping now looks the same to a
rust callback as a missing one, a parity gap accepted for a case that does
not occur. metadata_labels, input_sequence_length and limiter_request move
down to bits 44-46, on both sides of the contract.

* feat(callbacks): type a rust callback's request state

Callback gains an associated State type: request_state returns it and
handle and handle_hosted receive Option<&State> beside the input, which no
longer carries the state. ErasedCallback is the object-safe form the python
wrapper stores; it carries the state type-erased and downcasts it once,
returning CallbackError::StateType rather than panicking on a mismatch.
handle_hosted's default names its unused host _host. The record sample
count follows the dropped logged_payload_not_a_mapping field.

* docs(callbacks): say what PayloadRef, PayloadHost and RenderOptions.null_keys are

* refactor(callbacks-python): render payloads with python-compat's json writers

pyjson.rs reimplemented py_encode_basestring_ascii and json's floatstr.
python-compat's json module now exposes write_string, generic over a String
or byte sink, and float_text, and the payload renderer uses them.

* docs(callbacks): note why Respond, Reroute, call_ended and flush are not wired yet

* docs(custom_logger_v2): add a README for the package

* fix(custom_logger_v2): import the field sources tests from field_sources

* test(callbacks-python): stub the litellm package in each payload test

* fix(custom_logger_v2): type the disabled callbacks lookup and the v2 dedupe check
2026-11-14 02:33:27 +00:00
norkelbyte
5849906ddb
fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079)
* fix(proxy): count thinking, tool_choice, and effort the way inference sends them, and count DeepSeek on its own count route

/v1/messages/count_tokens dropped the body's thinking, tool_choice, and output_config, so a Claude count with
tool_choice any came out lower than the bill. The route now passes those options to the provider counter, which
runs them through the deployment's own /v1/messages transform first, so the count sees what inference sends.
A body inference would reject now answers the same 400 before any provider call.

DeepSeek had no provider counter and fell back to a local estimate that ignores thinking. It now counts on
DeepSeek's Anthropic-compatible count_tokens route with the same base, key, tool sanitizer, and reasoning
mapping inference uses, and falls back to the estimate only when that route is missing.

* perf(proxy): delete count drop paths from one copy of the options

additional_drop_params on the count route now removes every path from a single deep copy of the request options, so a long client path list over a large schema stays linear

* fix(count_tokens): count deployment thinking defaults and match inference on betas, drops, timeouts, and DeepSeek listing

Count routes now merge the deployment's and router's default thinking and output_config under the request's own, the way inference does, so a deployment with thinking disabled counts what it bills

Claude counters send the thinking betas inference adds for thinking.display and block_binding, so those counts no longer answer 400

additional_drop_params that is not a list is skipped and non-string entries are ignored, the way inference skips them, instead of turning the count into a 500

The DeepSeek counter passes the deployment timeout to its count call, and DeepSeek model listing reads the bundled model list instead of recursing through get_valid_models

* fix(count_tokens): count deployment thinking on Responses and let a request drop list replace the deployment's

A Responses count now merges deployment and router defaults under the bridged chat body, so deployment thinking and reasoning_effort reach the count the way they reach chat inference

A request's additional_drop_params now replaces the deployment's list for both count drop passes, as inference does

The Anthropic, Azure AI, and DeepSeek counters read litellm_params through one typed helper instead of Any

* fix(count_tokens): estimate DeepSeek counts when the count route is down, and count deployment thinking on /utils/token_counter and Vertex Claude

A timeout, 429, 5xx, or connect error from DeepSeek's count endpoint now falls back to the local estimate instead of failing the count. A 401 or 422 still answers DeepSeek's own status

/utils/token_counter now counts the thinking and effort a DeepSeek deployment sets, and Vertex Claude counts run the Anthropic messages transform on the count options instead of sending none

* fix(count_tokens): send the Vertex count's thinking betas and count a Responses request's own thinking

A Vertex Claude count with thinking display or block binding now sends the beta headers Vertex inference adds, so Vertex no longer rejects a count for a request inference accepts

A Responses count now takes the request's own top-level thinking over the deployment's, the order inference uses, since the Responses-to-chat body never carried it

* fix(count_tokens): cap DeepSeek count waits, remember missing count routes, and count /utils request thinking

A DeepSeek count now waits at most 10 s even when the deployment sets no timeout, a 422 from the count route falls back to the estimate, and a base whose count route answered 404 or 405 is not called again for an hour. /utils/token_counter now passes the request's thinking, reasoning_effort, and tool_choice to the count

* fix(count_tokens): count on DeepSeek's endpoint only for api.deepseek.com and send Messages count options to Bedrock Mantle

* fix(bedrock): send the thinking beta headers on Mantle counts

A Mantle count with thinking display updates or block_binding now sends the
same anthropic-beta header Mantle inference sends

* fix(count_tokens): drop the region hint on a Mantle count 400 for forwarded count options and unskip the count options wire test

* fix(deepseek): count only the tools DeepSeek chat inference sends

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:29:16 +00:00
mika-lt
b77a1295a5
fix(gemini): stream TTS audio in delta.audio on chat completions (#51750)
* fix(gemini): stream TTS audio in delta.audio on chat completions

* fix(logging): drop audio from logged responses when message logging is off

turn_off_message_logging blanked the text of a logged response object
but left message.audio and delta.audio in place, so callbacks reading
response_obj or the assembled stream still got the whole clip. The dict
path already cleared audio. Streamed Gemini TTS now carries audio, which
made this reachable on a second path.

Also pin the TTS stream test to gemini-3.8-flash-tts.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:27:55 +00:00
quietjules
f5ef14308b
fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved

Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.

* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows

* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs

* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params

* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params

* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase

* fix(bedrock): mark the service tier block field read-only

* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params

* test(prometheus): sort the service tier label test imports

* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400

* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova

* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk

The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one

* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name

The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it

* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added

The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:33:57 +00:00
devin-ai-integration[bot]
03f14051b9
test(otel): keep proxy startup out of the per-test timeout in the excluded services matrix (#51759)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 01:05:05 +00:00
Ankit Jha
5f381ad472
fix(proxy): keep carried end-user spend when a budget reset is retried (#51143)
* fix(proxy): keep carried end-user spend when a budget reset is retried

End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.

Fixes #50898

* fix(proxy): leave never-reset tiers unguarded and test each tier's own window

A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.

* test(proxy): give the end-user ledger its own fake prisma client

* fix(proxy): pin a reset time on tiers without one before end-user pages commit

* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced

* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset

* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration

* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window

* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time

A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users

* fix(proxy): name each end-user reset window by the tier's last write

Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due

* fix(proxy): name end-user reset windows by the tier and its advance count

Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.

* fix(proxy): keep carried spend for customer teams and members on a retried reset

Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.

The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.

* test(integration): list every budget column and relation in the budget update field check

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:03:19 +00:00
shrey
68d6db45ef
fix(mcp): give the MCP client exactly the servers left on in the connect page (#50973) 2026-11-13 16:36:49 -08:00
Mateo Wang
c4a72b8fac
refactor(mcp): let the semantic filter take its MCP tool loader as a parameter (#50483)
* test(mcp): check the tool dict conversion without patching the MCP handler

* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter

* refactor(mcp): type the semantic filter's default tool loader

Annotate the chosen loader as _OpenAIToolLoader and narrow its return to
FunctionToolParam, so a default that returns raw MCP tools fails the type
check. Drop the unused patch import.
2026-11-13 16:34:00 -08:00
ryan-crabbe-berri
0f97850c50
fix(logging): keep a content-block system prompt in logged messages (#51699)
* fix(logging): keep a content-block system prompt in logged messages

A /v1/messages request whose top-level system prompt is a list of content
blocks never reached the logged messages, because only a plain-text system
prompt was prepended. Accept a non-empty list too, so every logger that reads
messages sees it. Claude on Vertex also normalizes a plain-text system prompt
into blocks, so it was dropped there as well.

Fixes #36402

* fix(logging): type the logged system message

* fix(logging): type the system message without a mutable annotation, close the test client
2026-11-13 16:23:29 -08:00
joshua-berri
d11917e89c
fix(proxy): preserve caller metadata on auth failures (#51745)
* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs

Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.

* fix(proxy): type the spend logs metadata header parser honestly

The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.

* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket

A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.

* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler

Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.

* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer

The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.

* test(proxy): read auth failure spend rows from a real unconnected PrismaClient

Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 16:13:50 -08:00
joshua-berri
2df4f9a78f
fix: preserve routed prompt generation settings (#51702)
* fix: preserve async prompt generation settings

* fix: honor ignored settings on routed prompts

* fix: type prompt option handoff boundaries

* fix: bound prompt hook types without duplicate inference

* fix: preserve legacy prompt initializer parameters

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 16:00:18 -08:00
yucheng-berri
8cf7376df6
test(s3_v2): give the slow-sink ramp test a callback limit that outlasts its burst (#51744) 2026-11-13 23:23:55 +00:00
devin-ai-integration[bot]
d2bae98d97
fix(model-prices): drop Flex prices from Vertex AI Gemini 4 Argon and Hydrogen (#51735)
* fix(model-prices): drop Flex prices from Vertex AI Gemini 4 Argon and Hydrogen

* test(cost-calc): stop comparing Flex prices between Vertex AI and Gemini API Gemini 4 Argon

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
2026-11-13 23:18:03 +00:00
yujonglee
6523ab920a
fix(cost-map): restore backward compatibility after the Rust loader move (#51545)
A cost map with a top-level value that is not an object made a reload answer 500
after the new table was already published, so one worker billed new prices while
the others kept the old ones. The document is now refused before anything changes,
which also lets boot fall back to the bundled map instead of failing the import.

Also restores behavior the Rust cost map move changed for SDK callers:

- a document that starts with a UTF-8 or UTF-16 byte order mark loads again
- an alias_of row that carries other fields is served as its target
- register_model(url) fetches nothing when LITELLM_OFFLINE_MODE is set
- register_model(None) is a no-op again
- the public names RESERVED_TOP_LEVEL_KEYS, FALLBACK_GENERALIZATIONS_KEY,
  RETRYABLE_FETCH_STATUS_CODES and git_blob_id import again, and so does
  utils._runtime_registered_model_cost, which downstream test code replaces

litellm_core_utils/AGENTS.md now says which removed names come back.
2026-11-14 08:12:17 +09:00
mrinal-berri
cd6ef91115
fix(helm): route every gateway allowlist path to the gateway in the componentized ingress (#50760)
* fix(helm): route every gateway allowlist path to the gateway in the componentized ingress

The ingress gateway prefix list had drifted from gateway/routes/allowlist.py, so requests for decisions, systemone, extraction, traces, logs, live sessions and several provider passthroughs fell into the backend catch-all, and /eu_jev/v1/systemone returned 404. The missing paths are added, the exact gateway paths move into a list, and a unit test now fails whenever the chart and the allowlist disagree in either direction

* fix(helm): keep extraPaths that repeat a built-in gateway path rendering on upgrade

Values written for an older chart add the newly built-in gateway prefixes through ingress.extraPaths with service gateway, which the duplicate check now rejected, so helm upgrade failed to render. Such an entry is skipped, and a built-in path pointed at another Service or port is still rejected. Also asserts that the new exact path and a new prefix render to the gateway Service

* revert(helm): drop the extraPaths skip for built-in gateway paths

Keeps this PR to the missing paths and the drift test. An extraPaths entry that repeats a now built-in gateway path is rejected at render time as before, and operators remove it on upgrade. The checks that the new paths render to the gateway Service stay

* fix(helm): skip an ingress.extraPaths entry identical to a built-in route

Values that added a path through extraPaths before the chart routed it stopped rendering once the path became built in, since the duplicate check rejected every repeat. A repeat with the same path, pathType, Service and port (by number or the http port name) now renders nothing, and a repeat that routes elsewhere is still rejected

* refactor(helm): limit the ingress fix to gateway-only routes and the drift test

Drops the extraPaths identical-route skip, so the duplicate check is the same as on main. /v1/traces and /v1/logs leave the gateway list and keep going to the backend, since the backend also serves them and their gateway entries date from the retired OTLP ingest. The drift test now requires only paths that the gateway alone keeps

* test(helm): require every gateway exact path the backend does not list exactly

A backend prefix such as /debug/ made /debug/memory/summary count as shared, so the drift test stopped requiring its Exact gateway rule. Exact paths now count as shared only when the backend lists the same exact path

* chore(helm): keep the gateway prefix comment as it is on main

* fix(helm): fit the componentized ingress under the default ALB rule quota

The AWS Load Balancer Controller makes one listener rule per Ingress path and an ALB allows 100 rules by default, so the 113 single-host paths would fail with TooManyRules. Gateway prefixes are now listed in pairs, and on alb each pair is one rule, its first prefix as the Ingress path with a use-annotation forward to the gateway and the second added through a conditions annotation. The default alb render drops to 62 rules for the same path patterns, and nginx, monolith and component-host renders keep one path per prefix

* fix(helm): order grouped ALB rules by their shorter prefix and keep gateway conditions

Each gateway prefix pair now lists its shorter prefix first and uses it as the Ingress path, so a longer extraPaths child of either prefix still sorts ahead of the grouped rule. Conditions an operator set for the gateway Service are carried onto every grouped rule, which uses a generated backend name. The grouped annotations and paths are written inline from the pair list
2026-11-13 15:06:44 -08:00
Mateo Wang
ac3277dce4
fix(dynamic_rate_limiter_v3): publish a team's share gauges for requests served while the model is idle (#51709) 2026-11-13 14:58:54 -08:00
moe-berri
62cf355d9f
fix(spend-logs): make token redaction opt-in (#51729) 2026-11-13 14:41:12 -08:00
Kerry Lu
db75214bdc
fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry (#51694)
* fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry

The reset job now evicts the customer_org_alias entries of the organizations it resets, and a
hierarchy load that began before an eviction can no longer cache what it read. The cache hands
a lease out on a miss and refuses the write when the key was evicted since, on this worker and,
with enable_redis_auth_cache, on any worker through a generation key checked in Redis.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop auth cache memory before the generation write and skip an empty alias eviction

delete cleared memory only after its new Redis round trip, so a deleted entry stayed readable on
the handling worker while Redis was slow. The reset job also broadcast an empty alias eviction for
a cascade with no customer organizations.

* fix(proxy): delete an auth cache entry and void its leases in one script on the bulk path too

The bulk delete wrote the generation and sent the DEL as two round trips, so a failed generation
write still let the DEL run and an in-flight load could re-cache what it read.

* fix(proxy): delete the auth cache entry before the generation write inside the script

A Redis at its memory limit with noeviction refused the script at its leading SET, so the DEL
never ran and a deleted entry stayed in Redis. With the DEL first the script still deletes it.

* test(proxy): mark the new reset cache tests' bindings Final and use rstest for the new Rust test

* fix(proxy): refuse a leased auth cache write for a key no generation key can guard

* test(proxy): undo the unrelated reformat of the spend ledger parity tests

* test(redis-store): mark the updated memory-only auth cache test with rstest

* chore(proxy): drop two pointer comments from the cache lease code

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 14:21:20 -08:00
moe-berri
0d422a1b6c
fix(auth): let dashboard sessions use the user's team models (#51663)
* fix: inherit team models for dashboard sessions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: authorize dashboard models from owner grants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve dashboard model listing filters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): attribute dashboard team model spend to granting teams

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Keep dashboard fallbacks on the original billing team

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Preserve admitted dashboard billing through primary model rewrites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reject dashboard batches that need team model grants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Respect wildcard exclusions in dashboard model listings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 17:13:06 -05:00
Nate Armstrong
20b6e855bd
feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) (#51214) 2026-11-13 22:08:05 +00:00
tin-berri
7b0a37ba74
feat(ui): add overall customer usage and sortable budgets (#51713)
* feat(ui): add overall customer usage and sortable budgets

* test(ui): complete customer usage metric fixtures

* fix(ui): preserve exact customer organization IDs in overview
2026-11-13 22:03:22 +00:00
Mateo Wang
c7747002e5
test(integration): measure dynamic limiter shares after saturation (#51714)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:46:06 +00:00
kenneth-berri
31067d0f9f
fix(bedrock): log and bill Titan embedding replies on the invoke passthrough (#51701)
* fix(bedrock): log Titan embedding replies on the invoke passthrough as embeddings

A native /bedrock/model/<titan-embed>/invoke call returned its vector, but success logging read the reply as a chat completion and crashed, so the call was never priced or written to spend logs. Fixes #51455

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): log binary Titan embeddings and bill input images on /invoke passthrough

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): charge text tokens alongside the image on Titan multimodal /invoke passthrough

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:41:53 +00:00
devin-ai-integration[bot]
f4a17048c2
chore(cost-map): add the 4K output rate to runwayml/seedance2_5 (#51697)
* chore(cost-map): add the 4K output rate to runwayml/seedance2_5

* test(cost-map): cap per-second, per-image and per-request prices at 1000 and drop the seedance exceptions

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:40:54 +00:00
joshua-berri
88873a70d4
fix(proxy): retain unwritten rollups after database disconnects (#51706)
* fix(proxy): retain unwritten rollups after database disconnects

* fix(proxy): initialize auto-router turns per client

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 13:36:12 -08:00
devin-ai-integration[bot]
49ea54ee7a
test(streaming): make unbuffered Noma secret exposure deterministic (#51707)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:25:53 +00:00
moe-berri
9cc808c90b
feat(ui): let admins control Aria access for internal users (#51641)
* feat: configure Aria access for internal users

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor: keep Aria session access within complexity budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: match Aria role helper to authorized role type

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: type Aria settings and allow permitted team selection

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 16:17:12 -05:00
Kerry Lu
a920b5be6e
fix(proxy): delete a customer team's override budget when the team has no overall budget (#51703)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 13:05:19 -08:00
kenneth-berri
316f9ef905
feat(fal_ai): bill fal video clips by duration and resolution (#51679)
* feat: add per-clip video billing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve Pixverse resolution and explicit clip duration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(fal): allow clip pricing with a null billable-unit rate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 12:55:55 -08:00
yucheng-berri
9939e62b62
fix(types): build Usage schemas eagerly so parallel first requests cannot race (#51591)
* fix: eagerly build usage schemas before parallel requests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: tolerate upstream schema rebuild changes in race probe

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Move usage schema race regression to SDK integration tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(types): move the Usage schema race regression into the unit suite

The probe needs no proxy or upstream, so it belongs with the other Usage tests
and runs on every pull request. It now builds each model with real token counts,
asserts them in the dump, and falls back to a plain start barrier when pydantic's
model_rebuild cannot be inspected.

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 12:52:55 -08:00
moe-berri
945ee1ca43
fix(customer-portal): keep existing memberships and scope free tier capacity locks (#51698)
* fix(customer-portal): preserve memberships and isolate capacity locks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(customer-portal): map model write lock timeouts to retryable failures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 15:51:54 -05:00
mrinal-berri
57c6bbfa12
fix(mcp): route aggregate resources/read to the server that lists the URI (#51462)
* fix(mcp): resolve aggregate resource reads by unique listed owner

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): discover resource owners concurrently

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): remove obsolete multi-server resource read guard assertion

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): skip blocked stdio, scope bearer and admit candidates in resource owner lookup

* fix(mcp): return resource owner outcomes as values and skip blocked stdio candidates

* fix(mcp): withhold the caller bearer from every consuming server in aggregate resource reads

* fix(mcp): report aggregate resource read failures as MCPError

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 12:50:43 -08:00
devin-ai-integration[bot]
b1e07d7a17
chore(cost-map): add Soniox v6 and date Soniox, Bedrock Claude and Azure Whisper retirements (#51620)
* chore(cost-map): add Soniox v6 rows and date the v5, v4, v3 and preview-v2 retirements

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(cost-map): date Bedrock Claude 3.5/3.7 Sonnet EOL, fix Azure Whisper retirement, default Soniox placeholder to v6

Co-authored-by: llmintel <social@llmintel.ai>
Co-authored-by: granbellllc-png <granbell.llc@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: absorb verified DeepInfra registry updates

Absorb #51649 and #51652 without unverified output limits.

Co-authored-by: xinglan163 <250816997+xinglan163@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: correct Gemini TTS caching and Vertex cyber capabilities

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: correct Bedrock limits and Azure Flex pricing metadata

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: declare verified Fireworks prompt caching capabilities

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: add Perplexity Hydrogen and Mistral embedding cache rates

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: derive Azure Flex expectations from registered tier rates

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: llmintel <social@llmintel.ai>
Co-authored-by: granbellllc-png <granbell.llc@gmail.com>
Co-authored-by: xinglan163 <250816997+xinglan163@users.noreply.github.com>
2026-11-13 12:37:12 -08:00
tin-berri
b798cf30ab
feat(customer-portal): configure budget groups atomically (#51661)
* feat(ui): add customer organization budget groups tab

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Add cap policy controls to customer budget groups

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Clarify per-group cap policy help text

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Fix customer budget groups formatting and lint budgets

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(customer-portal): save budget group configuration atomically

* refactor(customer-portal): name unchanged budget group state

* fix(customer-portal): chunk removed budget reference checks

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 20:29:36 +00:00
kenneth-berri
534ef43526
fix(mistral): declare low, medium and high reasoning effort for mistral-large-4 (#51693)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 12:20:18 -08:00
kenneth-berri
853773e75a
fix(vertex_ai): send document_ocr, audio_track_extraction and auto_truncate on Gemini embeddings (#51686)
The three params were dropped without an error on the online path and in Vertex batch rows,
and a PDF embedding reported 0 tokens because its DOCUMENT token count failed usage validation.
They now go in embedContentConfig online and in embed_content_config inside each batch row's
request, and DOCUMENT tokens are counted at the text rate.

Resolves LIT-10384

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 12:10:13 -08:00
tin-berri
b34478b960
fix(customers): align budget displays with recorded reset windows (#51658) 2026-11-13 20:02:16 +00:00
devin-ai-integration[bot]
45b726fdab
test(security): resolve budget request route identifiers from their own fixtures (#51659)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 19:49:35 +00:00
joshua-berri
a166574ba4
fix(sdk): stop requiring redis to import the Messages streaming path (#51685)
* fix(sdk): stop requiring redis to import the Messages streaming path

redis_update_buffer.py and dynamic_rate_limiter_v3.py imported RedisError at module
level. Both sit on the import chain that the SDK /v1/messages stream loads, so an SDK
install without redis lost every streaming success callback. Import RedisError only
where a Redis call can actually raise it.

* style(tests): space the optional redis import fixture per ruff format

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 11:43:53 -08:00
moe-berri
1a682f9dbf
feat(customer-portal): add free tier limits with Enterprise upgrade path (#51687)
* feat: add customer portal free tier limits

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: narrow customer portal free tier enforcement

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Show customer portal free tier limits in dashboard

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: check customer key scope before organization registration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(customer-portal): preserve overflow requests and refresh limits after deletion

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 14:43:15 -05:00
joshua-berri
3dcc550aae
fix(router): prevent Responses fallback from replaying delivered tool calls (#51683)
* fix(router): stop a Responses fallback from replaying a delivered tool call

A tool-only Responses stream carries no output text, so a retriable error or
response.failed after a completed function call was treated as a failure before
output and the fallback replayed the whole request, running the tool twice.
Check for a started tool call before the before-output shortcut so the primary
error surfaces instead. Pre-output fallback and text continuation are unchanged.

* test(router): read the delivered tool call events one at a time

Consume the created and output_item.done events with anext and expect the
primary error on the next read, so the raises block holds a single statement.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 11:43:03 -08:00
Kerry Lu
bad8b1dae8
fix(proxy): drop a customer override expiry sent without an override (#51667)
* fix(proxy): drop a customer override expiry sent without an override

* test(proxy): clear the override before deleting the row so the integration test leaves no budget behind

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 11:24:05 -08:00
kenneth-berri
2680bd0417
fix(vertex_ai): report PDF tokens and cost on Gemini embedContent responses (#51673)
* fix(vertex_ai): report PDF tokens and cost on Gemini embedContent responses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(vertex_ai): mark PromptTokensDetails fields ReadOnly and type the new usage tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 19:22:39 +00:00
Kerry Lu
6549e114cc
fix(proxy): /customer/new budgets reset on the same schedule as other budgets (#51689)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 19:22:14 +00:00
yucheng-berri
02d6fce6f7
Use strict supervision for worker replacement tests (#51590)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 11:18:52 -08:00