* fix(count_tokens): answer 400, not 500, for a message without role or content
Anthropic, Azure AI Anthropic and Bedrock Claude counters now reject a
message missing role or content before calling the provider, with the
same 400 invalid_request_error the provider answers, instead of a 500
from the count transform.
* fix(count_tokens): count a chat message without content the way chat inference sends it, and check every Mantle count
* refactor(bedrock): leave message validation to the Bedrock count handlers
Main's Bedrock runtime and Mantle count handlers already answer 400 for a message without role or content before any AWS call, so the counter's own check was a second copy
* fix(count_tokens): name the chat message chat translation cannot send
A contentless system, developer, or tool message, or a message without a role, now answers "Message N must have a 'content' field" or "'role' field" instead of the translator's raw error text, which echoed the whole conversation back
* fix(bedrock): keep converse Claude deployments answering 400 for a chat message without content
Converse inference drops a non-assistant message without content, and for a last turn Claude then refuses the assistant prefill, so the count has no inference request to match. Invoke and Mantle deployments use Anthropic's chat translation and count the message the way their inference sends it
* fix(count_tokens): keep base counting for null-content tool and named turns on Bedrock converse
The converse check now covers only messages with just role and content, the
ones the merge base rejected. A content null message is never blamed for a
translation error, and a body whose messages all drop in translation answers
400 instead of a local estimate
* fix(count_tokens): reject only contentless user turns before counting on Bedrock converse
A system turn with content null counted on the merge base and converse
inference serves it, so it goes through chat translation again
* fix(count_tokens): keep the merge base's converse handling and leave null turns chat translation cannot send as given
* fix(count_tokens): keep a non-string message role from crashing the Anthropic count checks
* test(integration): expect the 400 for a count whose leading system message has no content
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cli): let Claude Code renew a --pkce login's key through lite's apiKeyHelper
* fix(cli): keep an explicit --api-key over lite's apiKeyHelper and persist its absolute path
An explicit --api-key or LITELLM_PROXY_API_KEY is now exported to Claude Code
even when lite's renewing helper is configured, since the helper prints the
stored login's key instead. The helper command also uses the absolute path of
the lite found on PATH, so a relative PATH entry cannot make Claude Code run
another repository's bin/lite
* fix(cli): keep an explicit key and --pkce when lite's renewing helper is set
A --settings that blanks apiKeyHelper now keeps the exported key, and a key passed with --api-key turns lite's helper off for that launch, so Claude Code no longer warns about two credentials or sends the stored login's key beside it
The helper names the lite installed beside the Python that ran the login before one found on PATH, so a pyenv shim no longer fails in projects pinned to another Python
lite claude signs in again with --pkce when the stored login used it, and print-token's hints name the proxy and --pkce
* fix(cli): keep a shell token from beating the renewing helper and reuse --pkce only for the same proxy
The --pkce --config-claude login now writes an empty env.ANTHROPIC_AUTH_TOKEN beside the helper, since a token exported in the shell wins over apiKeyHelper
lite claude sends any other key through a private settings file that blanks the helper and merges the user's own --settings
A re-login keeps --pkce only when the stored login is for the same proxy, else the saved default decides
* fix(cli): let any lite install's helper renew for its proxy and name the renewing login in print-token hints
lite claude now treats a lite auth print-token helper for this proxy as renewing whichever lite path it names, since every lite reads the same token store
Every print-token hint to sign in again, including an expired key and an unreadable keychain, names lite login --pkce with the proxy
Shorter docstrings
* fix(cli): send a launch key to the proxy it was issued for
The private settings file lite claude writes for a key that lite's helper would not renew now also sets ANTHROPIC_BASE_URL. Without it, lite --base-url B --api-key <B key> claude after a --pkce --config-claude login to A sent the B key to A, since the login's settings point Claude Code at A
* fix(proxy): preserve caller metadata on auth failures (#51745)
* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs
Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.
* fix(proxy): type the spend logs metadata header parser honestly
The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.
* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket
A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.
* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler
Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.
* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer
The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.
* test(proxy): read auth failure spend rows from a real unconnected PrismaClient
Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(logging): keep a content-block system prompt in logged messages (#51699)
* fix(logging): keep a content-block system prompt in logged messages
A /v1/messages request whose top-level system prompt is a list of content
blocks never reached the logged messages, because only a plain-text system
prompt was prepended. Accept a non-empty list too, so every logger that reads
messages sees it. Claude on Vertex also normalizes a plain-text system prompt
into blocks, so it was dropped there as well.
Fixes#36402
* fix(logging): type the logged system message
* fix(logging): type the system message without a mutable annotation, close the test client
* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter (#50483)
* test(mcp): check the tool dict conversion without patching the MCP handler
* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter
* refactor(mcp): type the semantic filter's default tool loader
Annotate the chosen loader as _OpenAIToolLoader and narrow its return to
FunctionToolParam, so a default that returns raw MCP tools fails the type
check. Drop the unused patch import.
* fix(mcp): give the MCP client exactly the servers left on in the connect page (#50973)
* fix(cli): send the launch key under a project apiKeyHelper and give each key its own settings file
A `lite login --pkce --config-claude` leaves an empty ANTHROPIC_AUTH_TOKEN in Claude Code's user settings. When a repo's project settings set their own apiKeyHelper, `lite claude` skipped its private settings file, so the empty token hid the exported key and Claude Code sent the project helper's key. `lite claude` now also writes that file whenever the settings files leave ANTHROPIC_AUTH_TOKEN empty
Every key launch also wrote the one file `--subscription` uses, so two launches started together could read each other's key and proxy. A key launch's file is now named after a hash of its contents, and every overlay is written to a temp file and renamed so a starting Claude Code never reads half of one
* fix(proxy): keep carried end-user spend when a budget reset is retried (#51143)
* fix(proxy): keep carried end-user spend when a budget reset is retried
End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.
Fixes#50898
* fix(proxy): leave never-reset tiers unguarded and test each tier's own window
A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.
* test(proxy): give the end-user ledger its own fake prisma client
* fix(proxy): pin a reset time on tiers without one before end-user pages commit
* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced
* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset
* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration
* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window
* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time
A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users
* fix(proxy): name each end-user reset window by the tier's last write
Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due
* fix(proxy): name end-user reset windows by the tier and its advance count
Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.
* fix(proxy): keep carried spend for customer teams and members on a retried reset
Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.
The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.
* test(integration): list every budget column and relation in the budget update field check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(otel): keep proxy startup out of the per-test timeout in the excluded services matrix (#51759)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): skip the working directory when the login looks up lite for its helper
On Windows, shutil.which searches the current directory before the directories it is given, so a lite.exe planted in the repo where someone ran `lite login --pkce --config-claude` could become their user-level apiKeyHelper
The login now checks each directory by full path, this Python's scripts directory first and then each PATH entry, skipping empty entries
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved
Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.
* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows
* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs
* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params
* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params
* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase
* fix(bedrock): mark the service tier block field read-only
* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params
* test(prometheus): sort the service tier label test imports
* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400
* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova
* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk
The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one
* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name
The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it
* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added
The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cli): delete the key files lite claude leaves on logout and unconfigure
lite claude hands its process to Claude Code, so the --settings files holding
its key stayed in the Claude config dir. lite logout and lite unconfigure claude
now delete them, and Claude Code reads one only at startup, so open sessions
keep working
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: joshua-berri <joshua@berri.ai>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: shrey <shrey@berri.ai>
Co-authored-by: Ankit Jha <45779665+PyExtreme@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: quietjules <349048734+quietjules@users.noreply.github.com>
* feat(opensearch): map vector store search filters to k-NN filters and add hybrid search
* test(code-quality): allow the bounded OpenSearch filter recursion
* fix(opensearch): retry an unfilterable engine without filters under drop_params, match strings on keyword subfields, keep per-request field keys
Under drop_params, a 400 'does not support filters' from an nmslib or older
engine re-runs the search once without filters, as filtered searches behaved
before. String eq, ne, in and nin match the key's .keyword subfield, falling
back to the key where no subfield is mapped. Only the index name, search
pipeline and hybrid flag are registration-only, so opensearch_text_field and
opensearch_embedding_field pass through per request again
* fix(opensearch): match string filters on the exact keyword field and embed once on the filter retry
String filters now read the index mapping and compare the exact keyword field, refusing a key the index cannot match exactly. The drop_params retry reuses the first embedding and also covers OpenSearch before 2.4. An index that refuses filters without drop_params answers litellm's own 400, and a filters value that is not an object stays a 400 under drop_params. The Admin UI store form gains the search pipeline and hybrid search fields
* test(ui): expect the OpenSearch search pipeline and hybrid search fields in the provider field list
* fix(opensearch): drop string filters under drop_params when the mapping read fails, and skip normalized keyword fields
* fix(opensearch): name the permissions a refused index mapping read needs instead of returning OpenSearch's body
* fix(opensearch): check the index mapping for every filter, resolve aliases, and keep failed mapping reads from dropping filters
* fix(opensearch): keep an alias filter's own name and refuse rewrites that reach an index without the alias
* fix(vector_stores): sign the caller's headers as given and reject filters on keys only some OpenSearch indexes map
A null or non-str extra_headers value no longer fails Bedrock Knowledge Base, S3 Vectors, or other provider searches before signing. The mapping read gets only the string-valued headers.
A filter on a key that some of a store's indexes map and others do not answers 400 (drops under drop_params), since OpenSearch returns its k-NN hits on some calls and none on others.
* fix(opensearch): send the caller's headers on the index mapping read as given and name the fix for keys only some indexes map
A header httpx cannot send now fails the filtered search on the mapping read, the same way it fails the search, before the query is embedded. Bytes headers now reach the mapping read too.
* test(opensearch): pin that a timeout on the index mapping read answers 408 without embedding the query
* fix(opensearch): answer 400 for a filter key inside a nested field or naming an object
A root-level term query on a field under a nested mapping, or on an object field, matches nothing, so the search answered 200 with no hits. These keys now answer 400 naming the mapping, or drop the filters under drop_params
* fix(opensearch): filter on a nested field whose values OpenSearch copies to the root
A nested field mapped with include_in_root, or include_in_parent on it and
every nested field above it, has its values in the root document, so a root
term query matches it. Those filters now search instead of answering 400.
A key that names an object or nested field answers that it holds fields
rather than values before the nested check, so its remedy no longer leads
into a second 400
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: propagate OpenRouter errors and preserve failure usage
* test: cover OpenRouter errors and failure accounting
* fix: bill a recovered OpenRouter failure on the success log and parse the body once
A retry or fallback that recovers from an OpenRouter finish_reason error now
carries the failed attempt's usage and cost into the success log, so the key
pays for both calls. The response body is parsed once and handed to the error
check, and the failure-log test no longer waits on a 10-second deadline
* fix: carry a failed attempt's usage before each SDK retry
* test: cover OpenRouter's documented failure body without usage
Pins the recovered-failure test's start time to a fixed date and adds a regression test for the documented non-streaming error body, which carries no usage
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0)
* feat(custom-logger-v2): add the rust contract for native callbacks and bind the proxy request scope
* fix(custom-logger-v2): add field projections and keep the request scope across copies
* fix(custom-logger-v2): pass the request scope to the async failure attempt_ended line
* fix(custom-logger-v2): add v2 callbacks from litellm.callbacks to the success and failure lists
_safe_add_callback_to_list dropped CustomLoggerV2 instances, so a V2 callback registered through litellm.callbacks got provider events but never log_prepare or attempt_ended. The manager now adds each V2 instance once by identity, and proxy startup fans V2 callbacks out like CustomLogger ones.
* fix(custom-logger-v2): contain a failing v2 callback on logging hooks
A V2 log_prepare writer that raised, or returned Abort or Respond, ended the line and escaped async_success_handler, so attempt_ended delivery, sync-for-async scheduling and the Redis flush were skipped. On hooks off the request path a V2 entry's exception or outcome is now counted and logged like a legacy logger's exception, and the line goes on. ContractViolation and BaseException still propagate, and legacy entries are unchanged.
* fix(custom-logger-v2): run blocking and awaited v2 callbacks on synchronous logging lines
run_plan_sync raised ContractViolation for a V2 log_prepare writer or any awaited V2 subscriber, the logging plane swallowed it, and every legacy logger at that hook was skipped. Legacy entries now always run in order as before. A V2 writer or awaited subscriber runs inline in list order through asyncio.run when the thread has no running event loop, so sync SDK calls apply its patch and finish it before the handler returns. On a thread with a running loop it cannot block, so it is delivered off-path and its patch is not applied. Lines without such entries take the unchanged legacy path.
* fix(custom-logger-v2): map a v2 abort to its http error at every proxy hook point
Only pre_call turned a V2 Abort into an HTTP error. during_call, post_call_success, post_mcp_call, list_models and the deferred stream hook raised V2Outcome, which reached the client as a 500, and post_call_failure, which never raised before, raised it too. Abort now raises the exception it carries or an HTTPException with its status, message and headers at those points, and at post_call_failure it becomes the transformed error through keep_first. Respond is still not wired and keeps raising V2Outcome. The pre_call routing test is updated to the HTTPException raise_v2_abort already makes.
* fix(custom-logger-v2): apply no-log to v2 logging subscribers without a request scope
The proxy request scope already carries no-log and caller opt-outs, so a V2 callback that honors no-log is skipped on proxy requests. SDK calls have no scope, and the logging plane ran V2 subscribers with an empty selection, so a no-log call was still delivered to them. Without a bound scope the logging lines now open one with no_log set when the call's litellm_params carry no-log, unless global_disable_no_log_param is on. Callbacks that do not honor no-log, and so every writer or abortable, are unaffected.
* fix(custom-logger-v2): deliver a failed attempt to v2 subscribers once per logging object
Callers such as MCP tool errors and Responses streaming run both failure handlers without for_async_request, so a V2 attempt_ended subscriber, now present in both failure lists, got the failure twice. The first failure handler to include V2 subscribers claims the delivery on the Logging object and the other skips them. Legacy callbacks are unchanged.
* refactor(custom-logger-v2): settle a synchronous v2 run through the shared gather and use aware times in the new tests
Drops the blind except the sync line used to capture a V2 callback's exception, reusing _gather_started, and gives the new logging tests timezone-aware times, so the strict ruff counts do not grow.
* fix(custom_logger_v2): await a single-entry v2 stage in the caller's context
A native handler in its own stage ran inside an asyncio.gather child task, so a ContextVar it set was lost when the stage ended, while a legacy callback's stays visible to later callbacks and the rest of the request. A stage with one v2 entry is now awaited directly, with gather's exception handling kept. Multi-entry parallel stages still gather.
* fix(custom-logger-v2): raise a native callback panic as a runtime error
* fix(custom-logger-v2): run a synchronous v2 line on a private event loop
* fix(custom-logger-v2): deliver an mcp tool failure to v2 subscribers from the async failure handler
* fix(custom-logger-v2): raise an abort's carried exception at the deployment hooks and keep the original failure at post_deployment_failure
* fix(custom-logger-v2): apply caller opt-outs to v2 logging lines without a bound request scope
* fix(custom-logger-v2): dedupe v2 callbacks in the event lists by logger key so a native built twice is delivered once
* fix(custom-logger-v2): log an unregistered abort error kind and keep the carried traceback at post_call_failure
* fix(callbacks): reject a selection name or alias that is not lowercase
* fix(custom-logger-v2): run v2 deployment callbacks on rust bridge deployment hooks
* fix(router): type the fallback event kwargs handed to the v2 dispatch
* feat(custom_logger_v2): type the contract fields as records and mark derived-only fields
Fields that the host derives from a hook's arguments now carry pydantic records on the Python side and matching serde records in litellm-callbacks, with records.json as the shared sample both sides check. Twelve new fields take bits 30 to 41, a write to a derived-only field is a manifest error, and pre_call can write budget_reservation and rate_limit_statuses in place. The payload reference field and the PayloadHost trait give Rust a handle to the logging payload instead of a copy
* feat(custom_logger_v2): derive typed fields from hook arguments and always build call facts
A slot table maps each typed field to a function over the hook's arguments. A derived slot runs only when a subscriber reads it, once per hook invocation, and off-path readers settle their reads before the hook returns. Auth fields come from the auth object only, budget_plan is claimed once, and budget_reservation and rate_limit_statuses write through to their carriers so the next pre_call subscriber sees them. Call facts are now built on unscoped logging lines and at post_call_success, post_call_failure and response_headers
* feat(callbacks-python): hand the logging payload to native callbacks by reference
NativeCallback stashes the shallow payload copy it is given and passes Rust a PayloadRef. The Python host renders a handle with a safe_dumps compatible encoder, on the caller's event loop while the hook runs and on the blocking pool after it, and dropping the last reference releases the stash entry. The encoder and its JSON helpers move here from the S3 logging bridge so every port shares one copy
* feat(custom_logger_v2): accept every value python accepts for typed members and serve logged key identity
Typed members keep the bool, int, float or str python holds through a Scalar
type on both sides, so ids, limits and priorities no longer lose values that
lax pydantic refused. A member the contract cannot carry is named in rejected
so a port can reproduce what python does with it.
Logging hooks get logged_key_identity, built from the payload metadata the
python loggers read on every route, while key_identity stays on request hooks
and keeps reading the auth object.
* fix(custom_logger_v2): read the live budget reservation dict and carry non-finite floats both ways
budget_reservation read the dict budget_reservation.py builds with a KeyError
on callback_bound. Undeclared keys are now ignored, callback_bound is a member,
entries keep a non-mapping item as null so capacity's model access group check
stays exact, and costs read as the float budgets computes with float(x or 0.0).
Float members cross as the raw json marker when they are not finite, and a
native verdict returns through json_to_py, so an infinite or NaN cost survives
a read and a write instead of turning into null.
* feat(callbacks): carry a python value as its str() text and truthiness in scalar members
Scalar gains a Stringified variant with the str() text and bool() of a value
that is not a bool, int within i128, float or str, so a port can reproduce
the label, path or log text python formats and the falsiness of an empty
container without confusing it with a real string. Budget reservations say
whether python held the entries as a list.
* feat(custom_logger_v2): carry non-scalar values and any iterable tags or entries across the boundary
A Scalar member now carries a list, dict, set, bytes, object or an int past
i128 as its str() text and truthiness instead of naming it in rejected, which
stays for values whose str() or bool() raises. request_tags carries any
iterable as the items prometheus iterates, and budget reservation entries
carry what budgets iterates with entries_is_list for capacity's list check,
never written back to the shared dict.
* feat(custom_logger_v2): say whether a stringified value hashes and whether it is an int
A Stringified scalar now carries hashable, from trying hash() at the
boundary, so a port can raise the TypeError capacity's priority lookup and
the limiter's descriptor identities raise for a list, dict or set and not
for a tuple or object. It also carries integer, so an int past i128 is told
apart from any other value whose text is digits.
* fix(custom_logger_v2): wrap a non-list guardrail mode as one mode like prometheus does
Prometheus treats any guardrail_mode that is not a list as a single mode, so a
tuple mode holds no str mode and a missing mode is one None mode. The
derivation unpacked tuples and rejected other non-str modes instead.
* feat(callbacks): let a native handler await typed host calls the python caller serves
* test(callbacks-python): drive a native handler's host calls through the python dispatcher
* test(custom_logger_v2): route a native logger with a host through the execution driver
* fix(custom_logger_v2): type the wrapped guardrail modes for pyright
* feat(custom_logger_v2): carry the python type name and hash error text of a stringified value
* feat(custom_logger_v2): serve the model group the logging handlers compute and key remaining limits by it
* feat(custom_logger_v2): tell a budget reservation that is not a mapping apart from none
* feat(custom_logger_v2): name malformed team model preferences in rejected with the text capacity logs
* feat(custom_logger_v2): serve a logging payload that is not a mapping apart from a missing one
* fix(callbacks-python): render null keys like a python dict merge, adding a missing key last
* feat(custom_logger_v2): carry an explicit output cap past i64 as a scalar
* feat(custom_logger_v2): serve the custom metadata labels prometheus derives from the logged payload
* feat(custom_logger_v2): serve the input sequence length bucket prometheus picks at attempt end
* feat(custom_logger_v2): carry the start and end times prometheus reads from the call details
* feat(custom_logger_v2): serve the limiter's body tags, session id, resolved agent and agent policy limits at pre_call
* fix(callbacks): export the limiter request record
* test(custom_logger_v2): give the type name cases stable ids and sort the record exports
* fix(custom_logger_v2): type the new derivations and tests for pyright
* fix(custom_logger_v2): drop team model preferences and serve the fallback_on_saturation handoff main's capacity callback reads
* style(callbacks): rustfmt the record exports
* refactor(custom_logger_v2): rename native callbacks to rust callbacks
native.py becomes rust.py, with native_logger, NativeLogger, NativeHandle,
NativeState and HostedNative renamed to their rust forms. The pyo3 classes
NativeCallback and NativeRequestState become RustCallback and
RustRequestState, and the tests, fixtures and messages follow.
* refactor(custom_logger_v2): rename slots to field sources
slots.py becomes field_sources.py: Slot is FieldSource, SlotFields is
SourcedFields, and with_slots, NO_SLOTS and slot_table_problem follow. A
table that only derives is named DERIVED_*_FIELDS, and a table that can
write back keeps SOURCES (PRE_CALL_AUTH_SOURCES, PRE_CALL_SOURCES). The
include_v2 comment moves down to the functions that take it.
* refactor(litellm_logging): place the v2 members of Logging beside what they serve
v2_scope, v2_auth and v2_failure_delivered join the other per-request
class defaults, v2_logging_scope and its opt-out helper follow
should_run_callback, v2_call_facts sits by the standard logging payload
builder, and claim_v2_failure sits above failure_handler.
bind_v2_request_scope checks for a Logging object directly instead of two
runtime protocols.
* refactor(proxy): rename v2_fields to v2_proxy_fields and say why it is separate
* refactor(custom_logger_v2): drop the logged_payload_not_a_mapping field
A logged payload that is present but not a mapping now looks the same to a
rust callback as a missing one, a parity gap accepted for a case that does
not occur. metadata_labels, input_sequence_length and limiter_request move
down to bits 44-46, on both sides of the contract.
* feat(callbacks): type a rust callback's request state
Callback gains an associated State type: request_state returns it and
handle and handle_hosted receive Option<&State> beside the input, which no
longer carries the state. ErasedCallback is the object-safe form the python
wrapper stores; it carries the state type-erased and downcasts it once,
returning CallbackError::StateType rather than panicking on a mismatch.
handle_hosted's default names its unused host _host. The record sample
count follows the dropped logged_payload_not_a_mapping field.
* docs(callbacks): say what PayloadRef, PayloadHost and RenderOptions.null_keys are
* refactor(callbacks-python): render payloads with python-compat's json writers
pyjson.rs reimplemented py_encode_basestring_ascii and json's floatstr.
python-compat's json module now exposes write_string, generic over a String
or byte sink, and float_text, and the payload renderer uses them.
* docs(callbacks): note why Respond, Reroute, call_ended and flush are not wired yet
* docs(custom_logger_v2): add a README for the package
* fix(custom_logger_v2): import the field sources tests from field_sources
* test(callbacks-python): stub the litellm package in each payload test
* fix(custom_logger_v2): type the disabled callbacks lookup and the v2 dedupe check
* fix(proxy): count thinking, tool_choice, and effort the way inference sends them, and count DeepSeek on its own count route
/v1/messages/count_tokens dropped the body's thinking, tool_choice, and output_config, so a Claude count with
tool_choice any came out lower than the bill. The route now passes those options to the provider counter, which
runs them through the deployment's own /v1/messages transform first, so the count sees what inference sends.
A body inference would reject now answers the same 400 before any provider call.
DeepSeek had no provider counter and fell back to a local estimate that ignores thinking. It now counts on
DeepSeek's Anthropic-compatible count_tokens route with the same base, key, tool sanitizer, and reasoning
mapping inference uses, and falls back to the estimate only when that route is missing.
* perf(proxy): delete count drop paths from one copy of the options
additional_drop_params on the count route now removes every path from a single deep copy of the request options, so a long client path list over a large schema stays linear
* fix(count_tokens): count deployment thinking defaults and match inference on betas, drops, timeouts, and DeepSeek listing
Count routes now merge the deployment's and router's default thinking and output_config under the request's own, the way inference does, so a deployment with thinking disabled counts what it bills
Claude counters send the thinking betas inference adds for thinking.display and block_binding, so those counts no longer answer 400
additional_drop_params that is not a list is skipped and non-string entries are ignored, the way inference skips them, instead of turning the count into a 500
The DeepSeek counter passes the deployment timeout to its count call, and DeepSeek model listing reads the bundled model list instead of recursing through get_valid_models
* fix(count_tokens): count deployment thinking on Responses and let a request drop list replace the deployment's
A Responses count now merges deployment and router defaults under the bridged chat body, so deployment thinking and reasoning_effort reach the count the way they reach chat inference
A request's additional_drop_params now replaces the deployment's list for both count drop passes, as inference does
The Anthropic, Azure AI, and DeepSeek counters read litellm_params through one typed helper instead of Any
* fix(count_tokens): estimate DeepSeek counts when the count route is down, and count deployment thinking on /utils/token_counter and Vertex Claude
A timeout, 429, 5xx, or connect error from DeepSeek's count endpoint now falls back to the local estimate instead of failing the count. A 401 or 422 still answers DeepSeek's own status
/utils/token_counter now counts the thinking and effort a DeepSeek deployment sets, and Vertex Claude counts run the Anthropic messages transform on the count options instead of sending none
* fix(count_tokens): send the Vertex count's thinking betas and count a Responses request's own thinking
A Vertex Claude count with thinking display or block binding now sends the beta headers Vertex inference adds, so Vertex no longer rejects a count for a request inference accepts
A Responses count now takes the request's own top-level thinking over the deployment's, the order inference uses, since the Responses-to-chat body never carried it
* fix(count_tokens): cap DeepSeek count waits, remember missing count routes, and count /utils request thinking
A DeepSeek count now waits at most 10 s even when the deployment sets no timeout, a 422 from the count route falls back to the estimate, and a base whose count route answered 404 or 405 is not called again for an hour. /utils/token_counter now passes the request's thinking, reasoning_effort, and tool_choice to the count
* fix(count_tokens): count on DeepSeek's endpoint only for api.deepseek.com and send Messages count options to Bedrock Mantle
* fix(bedrock): send the thinking beta headers on Mantle counts
A Mantle count with thinking display updates or block_binding now sends the
same anthropic-beta header Mantle inference sends
* fix(count_tokens): drop the region hint on a Mantle count 400 for forwarded count options and unskip the count options wire test
* fix(deepseek): count only the tools DeepSeek chat inference sends
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): stream TTS audio in delta.audio on chat completions
* fix(logging): drop audio from logged responses when message logging is off
turn_off_message_logging blanked the text of a logged response object
but left message.audio and delta.audio in place, so callbacks reading
response_obj or the assembled stream still got the whole clip. The dict
path already cleared audio. Streamed Gemini TTS now carries audio, which
made this reachable on a second path.
Also pin the TTS stream test to gemini-3.8-flash-tts.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved
Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.
* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows
* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs
* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params
* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params
* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase
* fix(bedrock): mark the service tier block field read-only
* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params
* test(prometheus): sort the service tier label test imports
* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400
* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova
* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk
The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one
* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name
The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it
* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added
The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): keep carried end-user spend when a budget reset is retried
End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.
Fixes#50898
* fix(proxy): leave never-reset tiers unguarded and test each tier's own window
A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.
* test(proxy): give the end-user ledger its own fake prisma client
* fix(proxy): pin a reset time on tiers without one before end-user pages commit
* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced
* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset
* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration
* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window
* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time
A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users
* fix(proxy): name each end-user reset window by the tier's last write
Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due
* fix(proxy): name end-user reset windows by the tier and its advance count
Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.
* fix(proxy): keep carried spend for customer teams and members on a retried reset
Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.
The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.
* test(integration): list every budget column and relation in the budget update field check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(mcp): check the tool dict conversion without patching the MCP handler
* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter
* refactor(mcp): type the semantic filter's default tool loader
Annotate the chosen loader as _OpenAIToolLoader and narrow its return to
FunctionToolParam, so a default that returns raw MCP tools fails the type
check. Drop the unused patch import.
* fix(logging): keep a content-block system prompt in logged messages
A /v1/messages request whose top-level system prompt is a list of content
blocks never reached the logged messages, because only a plain-text system
prompt was prepended. Accept a non-empty list too, so every logger that reads
messages sees it. Claude on Vertex also normalizes a plain-text system prompt
into blocks, so it was dropped there as well.
Fixes#36402
* fix(logging): type the logged system message
* fix(logging): type the system message without a mutable annotation, close the test client
* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs
Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.
* fix(proxy): type the spend logs metadata header parser honestly
The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.
* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket
A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.
* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler
Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.
* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer
The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.
* test(proxy): read auth failure spend rows from a real unconnected PrismaClient
Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(model-prices): drop Flex prices from Vertex AI Gemini 4 Argon and Hydrogen
* test(cost-calc): stop comparing Flex prices between Vertex AI and Gemini API Gemini 4 Argon
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
A cost map with a top-level value that is not an object made a reload answer 500
after the new table was already published, so one worker billed new prices while
the others kept the old ones. The document is now refused before anything changes,
which also lets boot fall back to the bundled map instead of failing the import.
Also restores behavior the Rust cost map move changed for SDK callers:
- a document that starts with a UTF-8 or UTF-16 byte order mark loads again
- an alias_of row that carries other fields is served as its target
- register_model(url) fetches nothing when LITELLM_OFFLINE_MODE is set
- register_model(None) is a no-op again
- the public names RESERVED_TOP_LEVEL_KEYS, FALLBACK_GENERALIZATIONS_KEY,
RETRYABLE_FETCH_STATUS_CODES and git_blob_id import again, and so does
utils._runtime_registered_model_cost, which downstream test code replaces
litellm_core_utils/AGENTS.md now says which removed names come back.
* fix(helm): route every gateway allowlist path to the gateway in the componentized ingress
The ingress gateway prefix list had drifted from gateway/routes/allowlist.py, so requests for decisions, systemone, extraction, traces, logs, live sessions and several provider passthroughs fell into the backend catch-all, and /eu_jev/v1/systemone returned 404. The missing paths are added, the exact gateway paths move into a list, and a unit test now fails whenever the chart and the allowlist disagree in either direction
* fix(helm): keep extraPaths that repeat a built-in gateway path rendering on upgrade
Values written for an older chart add the newly built-in gateway prefixes through ingress.extraPaths with service gateway, which the duplicate check now rejected, so helm upgrade failed to render. Such an entry is skipped, and a built-in path pointed at another Service or port is still rejected. Also asserts that the new exact path and a new prefix render to the gateway Service
* revert(helm): drop the extraPaths skip for built-in gateway paths
Keeps this PR to the missing paths and the drift test. An extraPaths entry that repeats a now built-in gateway path is rejected at render time as before, and operators remove it on upgrade. The checks that the new paths render to the gateway Service stay
* fix(helm): skip an ingress.extraPaths entry identical to a built-in route
Values that added a path through extraPaths before the chart routed it stopped rendering once the path became built in, since the duplicate check rejected every repeat. A repeat with the same path, pathType, Service and port (by number or the http port name) now renders nothing, and a repeat that routes elsewhere is still rejected
* refactor(helm): limit the ingress fix to gateway-only routes and the drift test
Drops the extraPaths identical-route skip, so the duplicate check is the same as on main. /v1/traces and /v1/logs leave the gateway list and keep going to the backend, since the backend also serves them and their gateway entries date from the retired OTLP ingest. The drift test now requires only paths that the gateway alone keeps
* test(helm): require every gateway exact path the backend does not list exactly
A backend prefix such as /debug/ made /debug/memory/summary count as shared, so the drift test stopped requiring its Exact gateway rule. Exact paths now count as shared only when the backend lists the same exact path
* chore(helm): keep the gateway prefix comment as it is on main
* fix(helm): fit the componentized ingress under the default ALB rule quota
The AWS Load Balancer Controller makes one listener rule per Ingress path and an ALB allows 100 rules by default, so the 113 single-host paths would fail with TooManyRules. Gateway prefixes are now listed in pairs, and on alb each pair is one rule, its first prefix as the Ingress path with a use-annotation forward to the gateway and the second added through a conditions annotation. The default alb render drops to 62 rules for the same path patterns, and nginx, monolith and component-host renders keep one path per prefix
* fix(helm): order grouped ALB rules by their shorter prefix and keep gateway conditions
Each gateway prefix pair now lists its shorter prefix first and uses it as the Ingress path, so a longer extraPaths child of either prefix still sorts ahead of the grouped rule. Conditions an operator set for the gateway Service are carried onto every grouped rule, which uses a generated backend name. The grouped annotations and paths are written inline from the pair list
* fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry
The reset job now evicts the customer_org_alias entries of the organizations it resets, and a
hierarchy load that began before an eviction can no longer cache what it read. The cache hands
a lease out on a miss and refuses the write when the key was evicted since, on this worker and,
with enable_redis_auth_cache, on any worker through a generation key checked in Redis.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop auth cache memory before the generation write and skip an empty alias eviction
delete cleared memory only after its new Redis round trip, so a deleted entry stayed readable on
the handling worker while Redis was slow. The reset job also broadcast an empty alias eviction for
a cascade with no customer organizations.
* fix(proxy): delete an auth cache entry and void its leases in one script on the bulk path too
The bulk delete wrote the generation and sent the DEL as two round trips, so a failed generation
write still let the DEL run and an in-flight load could re-cache what it read.
* fix(proxy): delete the auth cache entry before the generation write inside the script
A Redis at its memory limit with noeviction refused the script at its leading SET, so the DEL
never ran and a deleted entry stayed in Redis. With the DEL first the script still deletes it.
* test(proxy): mark the new reset cache tests' bindings Final and use rstest for the new Rust test
* fix(proxy): refuse a leased auth cache write for a key no generation key can guard
* test(proxy): undo the unrelated reformat of the spend ledger parity tests
* test(redis-store): mark the updated memory-only auth cache test with rstest
* chore(proxy): drop two pointer comments from the cache lease code
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: inherit team models for dashboard sessions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: authorize dashboard models from owner grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve dashboard model listing filters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): attribute dashboard team model spend to granting teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Keep dashboard fallbacks on the original billing team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve admitted dashboard billing through primary model rewrites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reject dashboard batches that need team model grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Respect wildcard exclusions in dashboard model listings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): log Titan embedding replies on the invoke passthrough as embeddings
A native /bedrock/model/<titan-embed>/invoke call returned its vector, but success logging read the reply as a chat completion and crashed, so the call was never priced or written to spend logs. Fixes#51455
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): log binary Titan embeddings and bill input images on /invoke passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): charge text tokens alongside the image on Titan multimodal /invoke passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): add the 4K output rate to runwayml/seedance2_5
* test(cost-map): cap per-second, per-image and per-request prices at 1000 and drop the seedance exceptions
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: configure Aria access for internal users
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: keep Aria session access within complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: match Aria role helper to authorized role type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: type Aria settings and allow permitted team selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: eagerly build usage schemas before parallel requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: tolerate upstream schema rebuild changes in race probe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Move usage schema race regression to SDK integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(types): move the Usage schema race regression into the unit suite
The probe needs no proxy or upstream, so it belongs with the other Usage tests
and runs on every pull request. It now builds each model with real token counts,
asserts them in the dump, and falls back to a plain start barrier when pydantic's
model_rebuild cannot be inspected.
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add customer organization budget groups tab
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add cap policy controls to customer budget groups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Clarify per-group cap policy help text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Fix customer budget groups formatting and lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(customer-portal): save budget group configuration atomically
* refactor(customer-portal): name unchanged budget group state
* fix(customer-portal): chunk removed budget reference checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The three params were dropped without an error on the online path and in Vertex batch rows,
and a PDF embedding reported 0 tokens because its DOCUMENT token count failed usage validation.
They now go in embedContentConfig online and in embed_content_config inside each batch row's
request, and DOCUMENT tokens are counted at the text rate.
Resolves LIT-10384
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sdk): stop requiring redis to import the Messages streaming path
redis_update_buffer.py and dynamic_rate_limiter_v3.py imported RedisError at module
level. Both sit on the import chain that the SDK /v1/messages stream loads, so an SDK
install without redis lost every streaming success callback. Import RedisError only
where a Redis call can actually raise it.
* style(tests): space the optional redis import fixture per ruff format
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(router): stop a Responses fallback from replaying a delivered tool call
A tool-only Responses stream carries no output text, so a retriable error or
response.failed after a completed function call was treated as a failure before
output and the fallback replayed the whole request, running the tool twice.
Check for a started tool call before the before-output shortcut so the primary
error surfaces instead. Pre-output fallback and text continuation are unchanged.
* test(router): read the delivered tool call events one at a time
Consume the created and output_item.done events with anext and expect the
primary error on the next read, so the raises block holds a single statement.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): drop a customer override expiry sent without an override
* test(proxy): clear the override before deleting the row so the integration test leaves no budget behind
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): report PDF tokens and cost on Gemini embedContent responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): mark PromptTokensDetails fields ReadOnly and type the new usage tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>