* fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing
A non-list tools value crashed the vector store lookup during auth and the
built-in tool cost lookup in the failure logging path, so the proxy answered
an unhandled 500 and closed the keep-alive connection. count_tokens now
answers a 400 invalid_request_error for a malformed body, drops malformed
tools when drop_params is on, and both lookups skip values of the wrong shape.
* fix(proxy): skip non-list tools in the tool allowlist check
Also validate the count_tokens body without recursion, which the recursion detector rejects.
* test(integration): restore the int tools case on count_tokens
The proxy now answers a non-list tools value with a 400, so the case is deterministic again.
* fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids
* fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids
Check tools after request setup so the Claude Code default and the logged call type apply, read the request,
global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini
generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off
* fix(proxy): keep the request's call type when logging a gate rejection after request setup
The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses
rejection as aembedding, so it now only applies when the logging object carries no known call type
* fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry
* fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket
* refactor(proxy): leave the responses websocket out of the request-level tool check
* fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections
* fix(proxy): read router default drop lists and deployment ids in the malformed tools check
* fix(proxy): leave malformed tools to the router when it can pick another deployment
Fallbacks, auto-routers, model strings, comma batches and team model names
let the router pick a deployment after the proxy checks the request, so the
proxy now forwards a malformed tools field there as main does, and only
rejects it when the deployment it checks is the one that serves. Token
counting never reroutes, so count_tokens and input_tokens keep reading the
group they count with. additional_drop_params names are read the way core
reads them, so a list with a non-string entry still names its strings.
* test(integration): cover malformed tools on reroutable models, router defaults and deployment ids
* fix(proxy): resolve malformed tools at the deployment the router picks
* fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools
Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells.
* test(integration): expect a 400 for an empty-string tools field on /v1/messages
Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test
* fix(router): keep web search routing for tools passed as a tuple
* fix(router): keep a deployment's own tools when drop_params drops a malformed request tools
* test(router): list _merged_tools as covered through the deployment tools merge tests
* fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them
* fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError
* fix(prometheus): keep deployment health metrics clean on a malformed tools rejection
* test: type the new malformed tools tests
* test: mark new test locals Final and type kwargs dicts
* fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open
GuardrailToolParam required a type key, so Gemini native tools
(function_declarations, googleSearch) failed validation inside the hook
and either returned a 500 or skipped the scan when fail_on_error was
false. The type key is now optional and left out of the forwarded tool
when absent, so an echoing guardrail server cannot write type: null into
the provider request.
structured_messages was re-validated against the chat completion message
types, which silently emptied tool_result, tool_use, image and document
content blocks to content: [] and dropped undeclared keys before the
guardrail saw them. The request model now forwards rows as opaque
mappings, which also retires the write-back shim that restored the
original rows for echoed-back entries.
The fail-open critical log line now carries error_type so a timeout, a
refused connection and an upstream HTTP status can be told apart without
the traceback.
* fix(httpx): retry aiohttp server disconnects the way httpx does
aiohttp raises ServerDisconnectedError when a kept-alive connection is
closed by the peer before a response arrives, which is what a load
balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces.
The transport mapped it to httpx.ReadError, which the handler's
connection-error retry does not cover, so the request failed instead of
being resent on a fresh connection. It now maps to
httpx.RemoteProtocolError, the error httpx itself raises for the same
condition, so the existing retry fires.
* test(httpx): build the stub server reply in one bytes literal
* fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable
The connection-error retry closed its fresh client before a streamed body was read,
so an aiohttp disconnect now billed a second upstream request and still failed. It
also skipped masking on a status error from the resend, resent PATCH and DELETE as
POST, dropped multipart files and ignored the handler's ssl_verify.
The retry now resends the request the first attempt built, on a client with the
handler's TLS settings, closes that client when a streamed body is closed, and masks
a status error the same way the first attempt does.
* fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type
* fix(guardrails): resend a generic guardrail check cut before any response
A load balancer that closes an idle keepalive connection made the guardrail POST
fail with "Server disconnected", because #46336 only resends POST on ConnectError.
AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to
False, that gives a RemoteProtocolError the same single fresh-client resend PUT
and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a
guardrail check is a pure verdict over the request. Every other POST keeps the
#46336 rule
The fail-open log line now puts error_type at the end so the pre-PR text stays
contiguous for existing alert matchers
test_rows_reach_the_guardrail_verbatim now reads the body off a real local
guardrail server instead of mock call args
* fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration
A ServerDisconnectedError raised while reading a response body now stays
httpx.ReadError, as it was before this PR. Only a disconnect before any
response maps to RemoteProtocolError, which is the case the resend rule is
about. Stream readers that end on ReadError, like the SAP stream iterator,
keep their old behavior
The loopback-server disconnect, TLS and guardrail resend tests move to
tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is
in-process only. Unit keeps in-process MockTransport checks for the idempotent
POST resend, masking on the resend, streamed and buffered client closing, the
guardrail resend, and the guardrail receiving rows verbatim
* fix(guardrails): resend a reset guardrail check and keep typed echoes on texts
An idempotent POST now resends once on any network error after connect
(ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load
balancer that answers a stale keepalive with a TCP reset rides it out too.
That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes
back to ReadError. This stops PUT and DELETE from stacking litellm's resend on
top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil
Guard from replaying its analyze POST on a disconnect
Every resend now runs inside the method's own error mapping, so a timeout on
the resend is a litellm.Timeout like the first attempt and a status error is
masked once by the caller
A guardrail server that parses rows with its own message types and echoes them
back without the keys those types lack (annotations, tool_calls[].index) is
read as an echo again, so its texts redaction reaches the LLM. A row only counts
as an echo when no value changed and every key the chat completion types
declare is kept
The wire tests for the PUT/DELETE resend, its masking and its TLS settings now
run on the httpx transport, where litellm's resend is the only retry, and the
POST and guardrail tests cover both transports and both cut styles
* test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend
Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head
* fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement
* refactor(guardrails): move the dropped-connection guardrail resend out of this PR
The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line
* test(guardrails): pick the closed guardrail port from the integration port allocator
Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port
* feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent
generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): stop proxy-started legacy listeners from consuming server rpm
The listener the proxy opens on notifications/initialized for legacy MCP
sessions ran the server rate-limit admission once per target, so every
session spent an rpm slot before the client sent anything. Admission is
now injected into the listener and applied only to client-initiated
subscriptions (subscriptions/listen and resources/subscribe). Authorization
and its periodic recheck still run on every path.
* test(mcp): key the fake admission window by caller and server
* fix(mcp): stop subscription catalog refreshes from consuming server rpm
A subscription listener lists the caller's catalog to diff list_changed
events, at setup and on every upstream event. Those listings went through
the normal per-server admission, so against an upstream that declares
listChanged each legacy session spent several rpm slots before the client
asked for anything. The listener now marks its targets as already admitted
for its own lifetime, so its refreshes reuse the subscription's admission.
Client requests on the session are still metered.
* test(mcp): exercise subscription admission through the proxy rate limiter
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure
A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.
* fix(proxy): log a provider error frame through an attempt-scoped logging copy
The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.
* fix(proxy): scope the error-frame failure log to streams closed at the frame
A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it
* chore: drop duplicate litellm import and mark the stream_ended rebind
* fix(proxy): log only router-superseded attempts as error-frame failures
A client that hangs up at a forwarded error frame closed the stream the same way the router closes a superseded attempt, so a non-retried stream lost its spend. The router now closes a superseded attempt through aclose_superseded_attempt, and only that close routes the attempt to the failure log
* test: type the _settle helper's recorded events
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): add the 1 hour cache write cost to the add model and model edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover both cache write tiers in the add model and edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): hoist the expected pricing payload out of the inline assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): share the per-token conversion between both cache write tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): derive the touched pricing field type from the pricing field list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): collapse the duplicated pricing source keys into one map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): convert add model pricing once for all selected models and describe the 1 hour fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): type the cache write tier deployment read back and fill the 1 hour field
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(e2e): keep the cache write tier spec on the dashboard prettier config
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): streaming responses hand the event loop back every 50 ms instead of holding it for the whole buffered stream
* fix(proxy): yield to the event loop on every streamed chunk, no hold budget or clock
* docs(proxy): say why the streaming loop yields on every chunk
* fix(proxy): streaming loop yields once it has held the event loop 50 ms, clock read inline
* docs(proxy): streaming loop comment names the 50 ms hold
* Revert "docs(proxy): streaming loop comment names the 50 ms hold"
This reverts commit cbf1371873.
* refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50
* Revert "refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50"
This reverts commit 356454def6.
* docs(proxy): streaming loop comment notes the yield is a floor, not a count of releases
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count
/v1/messages runs clear_tool_uses context edits before it sends an Anthropic
body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted
the body unedited, so a request with older tool results cleared counted about
2.8x what inference bills.
The count now runs the same history and context-edit step inference runs,
honors additional_drop_params on the deployment, and skips compact edits so a
count never makes a paid summary call.
* fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs
Run compact edits in the count without the summary call so trigger checks and
history slicing match /v1/messages, honor a request-level additional_drop_params,
keep the full body when any deployment in the model group drops context_management,
and answer 400 naming a block Gemini cannot count when an edit trigger fails on it
* fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check
* fix(count_tokens): log a failed edit in the count at debug, since the send already logs it
* fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does
* fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check
The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits.
* fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name
* fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to
* fix(messages): drop an invalid context_management spec under drop_params so the send matches the count
* fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them
The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes.
* test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code
* fix(responses): log and bill every round of non-stream MCP auto-execute
Both model rounds run as internal sub-calls on the request's logging object, and the outer call logs the response the client received with usage, cost, and cost breakdown summed across the rounds. The internal-call gate now also covers the sync success callbacks, so an inner round no longer takes their first-wins slot
* fix(caching): let the parent call log and bill an internal sub-call's cache hit
A round of non-stream MCP auto-execute that the response cache served fired the
cache-hit callbacks on the shared logging object, so the spend log kept round 1's
function_call row and dropped the billed row for the answer. Internal sub-call cache
hits now fire no callbacks and carry a zero cost, and the summed cost breakdown leaves
out rounds served from the cache
* fix(logging): keep a parent row unique when its response came from the response cache
A RAG query or an MCP turn whose final round is served from the cache hands back the cached response's id, which already belongs to the row that cached it, so SpendLogs dropped the new row as a duplicate. Such a row now gets the cache-hit id suffix, and it is logged as a cache hit when it cost nothing.
* fix(logging): skip the zero-cost warning for a response served from the cache at no charge
A non-stream MCP auto-execute request whose rounds the response cache served is priced
at $0 before the parent marks it a cache hit, so it logged a false 'priced at $0'
warning and counted in litellm_zero_cost_requests_total. The diagnostic now treats a
response the cache served at no charge the same as a marked cache hit
* fix(responses): bill the paid first MCP round when a later step of the request fails
* fix(responses): log each MCP tool call under its own id so a failed follow-up keeps its spend row
The non-stream auto-execute path passed the request's call id to the tool call, so the tool's spend row and the request's failure row shared one request_id and the failure row was dropped on insert, leaving the paid first round unbilled.
* fix(responses): bill round 1 on a client disconnect, keep MCP tool rows in the request's session, and charge a fixed margin once per request
* fix(responses): keep bridged MCP auto-execute in one session and trace and continue its conversation
The bridged follow-up no longer chains on round 1, so a thinking model keeps its reasoning and the request stays in one session. The tool call log continues the request trace without renaming it, the tools are fetched once, and a later turn finds the MCP turn under the id the client got, skipping the tool rows.
* fix(responses): log the non-stream MCP tool listing in the request's session and trace
* fix(responses): find a bridged MCP turn by the id the client holds and log its listing in the bridged trace
* fix(responses): derive the MCP tool row id from the request's call id
The tool call's spend row now uses <call id>_<tool call id>, so it stays
linked to the request without colliding with the request's failure row.
Also tighten the internal cache-hit test to a fixed clock and typed params.
* fix(responses): bill a retried MCP attempt for its own rounds
A router retry or fallback reuses the request's logging object, so a failed attempt's stashed round 1 usage made the next attempt's success row log stale tokens. Each non-stream MCP attempt now starts without that stash.
* fix(responses): bill every failed MCP attempt's paid first round on the request failure
A retry or fallback now sets a failed attempt's stashed round 1 aside instead of dropping it, and the proxy's failure row adds it to the last attempt's own, so a request whose later attempt fails on its first call still bills the round the provider charged. A cached final round also keeps the paid first round's cost breakdown.
* fix(responses): keep MCP tool catalog and results out of the bridged next-turn replay
* fix(responses): keep MCP tool spend, client body keys, and Langfuse traces working on auto-execute
Count the MCP tools an auto-executed Responses API call ran in the tool spend rollup, accept a client body key named previous_response_client_ids without a 500, and point MCP listing and tool spans at the trace Langfuse gives the parent request, including x-litellm-session-id requests
* fix(responses): join MCP spans to the Langfuse trace when the request sends body metadata
* test(integration): cover non-stream MCP auto-execute billing, failure rows, and bridged follow-ups
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): delete callbacks configured under callbacks or failure_callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert on configured callback rows and use a callback absent from CI configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): follow logging suite conventions for callback delete tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): flatten remaining callbacks with a helper generator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type callback delete helpers to satisfy basedpyright budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): save callback config without mutating the loaded dict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): delete callbacks from every litellm_settings list GET shows
POST /config/callback/delete only searched success_callback while
GET /get/config/callbacks also lists failure_callback and callbacks, so
the Admin UI Delete returned 404 for those rows
Remove the name from every list that holds it, matching entries
case-insensitively, and skip lists the config file owns when the
database owns another copy so the save is not refused
Replace the e2e test with an integration test: the shared e2e stack
config owns litellm_settings.callbacks, so it cannot exercise that list
* test(proxy): cover config-owned success_callback on callback delete and wait on runtime state
Add the mirror mixed-ownership case where the config file sets
success_callback and the database owns failure_callback, which main
refused with a 400, plus a string-valued list case
Poll GET listings and /active/callbacks instead of reading them once, so
a request served by a worker that has not synced yet cannot fail the
test, and reuse the existing callback list key map in the route
* test(proxy): annotate callback delete test helper locals with Final
* fix(proxy): save callback delete through the config it read so concurrent settings edits are kept
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): stop queueing budget increments for Redis when no Redis is configured
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert Redis-less model budget limiter tracks spend locally without queueing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): scope session history lookup to the referenced key
* Implement SQLite-backed DB for session tests
Added a SQLite-backed database class for testing and a new test to ensure session isolation between different API keys.
* fix(proxy): honor caller litellm_session_id in spend log session_id
Updated docstring to clarify session ID resolution logic for spend log rows.
* test(proxy): caller litellm_session_id wins over the trace id
* style(proxy): fix blank lines around new tests
* fix(proxy): read the caller session from the logging payload for spend logs
* fix(responses): match user and team too when replaying session history
* fix(proxy): keep bridged Responses chains whole and scope session history by owner
* fix(proxy): keep a metadata session next to a trace header and let session headers win
* fix(proxy): send the trace-header session only to spend logs and let session headers win beside a trace header
* fix(proxy): drop a client-sent caller session and keep it out of requester metadata
* fix(proxy): keep the trace id as the spend log session when the caller session contains a NUL byte
* fix(responses): keep a continued bridged chain's session under request_correlation_in_logs
With the flag on, the caller session outranked the chain session the bridge sets for a continued Responses chain, so a chain whose session changed or first appeared on a later turn split. The continuation now logs under the chain's session and drops the turn's caller session key, as on main. Adds the /audit integration cells for the spend log caller session.
* fix(responses): log a continued bridged chain under its session through the spend-log key only
The previous commit set the chain session as the turn's litellm_session_id, which also moved the OTel conversation id of a continued turn away from its trace header. The bridge now writes the chain session into the internal spend-log session key of the metadata the spend log reads, so only the spend log session changes, as on main. Adds an OTel cell for a continued chain.
* test(spend): cover bridged chains that share one caller session
* test(spend): cover the caller session on configured pass-through routes
* test(spend): type the SQLite fake and split stacked comprehensions in the caller session tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(interactions): keep background interaction bills through crashes, late restarts, and deployment keys
A settlement claim is now a lease, so a row whose settler died between claim and
outcome is claimable again after the lease and boot resumes it. A resumed poll
gets the full poll window and fetches with the serving deployment's credentials
from the live router. The stored create context keeps the request tags and the
request metadata that the creating worker's own spend row used
* fix(interactions): bill a lapsed claim's takeover only when no spend row exists
A settler that billed and then died, or failed to record the outcome, left the row unsettled, so the replica that took over its lapsed claim billed the interaction a second time. A takeover now looks up the spend row keyed by the interaction id and records the row as billed when it finds one.
* fix(interactions): settle Vertex AI and edited deployments, never double bill with spend logs off
A takeover with spend logs off records the interaction unsettled instead of
billing it again, and a release records its outcome before freeing the budget
reservation. A resumed or live fetch carries the deployment's Vertex AI project,
location, and credentials, and a stale deployment id falls back to the model
group only when one same-provider deployment serves it. A poll stops at the
first 400, 401, 403, or 404, string request tags no longer fail the create,
and settled_at gets an index built concurrently.
* fix(interactions): leave a live replica's interaction to it when a resumed poll cannot read it
A replica that boots with credentials unable to read an interaction got a 404 on its resumed poll and settled the row as unsettled at once, so the replica that created it, still polling, found it settled and never billed it. A resumed poll that gets a 400, 401, 403, or 404 now stops fetching and keeps checking the row until its poll window ends, and settles it as unsettled only when no other replica did.
* fix(interactions): keep the creator's metadata on a resumed bill and retry outcome writes inside the claim lease
A bill another replica writes now carries the metadata the creating worker's own
spend row is built from, so a client's metadata.model_group no longer relabels it.
A failed outcome write is retried for half the claim lease before the budget
reservation is left to a takeover, and the creating replica retries every fetch
error for its whole window again, as main does. The resume integration case now
checks that a resumed fetch uses the deployment's own key.
* fix(interactions): resume background interaction polls only after database models load
Boot resume ran before the proxy loaded the models stored in the database,
so a resumed poll for a model added through /model/new or the Admin UI
fetched with env credentials. A 4xx there stops a resumed poll, so the bill
was lost. The settlement now installs after the database models load.
The integration test resumes a database model with an eager first fetch
and fails with the old startup order.
* fix(interactions): renew a settling claim so a slow settle is never taken over and billed twice
A settle runs every success callback in turn, and one slow callback could
hold it past the 5 minute claim lease. A booting replica or a DELETE then
took the live claim over, found no spend row yet, and billed the
interaction a second time. The settler now renews its claim every third
of the lease until it records the outcome, so only a settler that stopped
renewing is taken over.
A takeover also read the spend log after writing its claim, so a failed
read left the row under the taker's own fresh claim, which its poll then
took for another settler's and left unbilled until a reboot. The spend
row is now read before the claim is written, so a failed read takes
nothing and the claim is retried on the next poll.
* test(spend): cover lapsed claim takeovers, rotated deployment keys, late restarts, and resumed bill tags
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count prompt-only token_counter requests with the provider when call_endpoint is set
* fix(proxy): keep the local count for a prompt-only token_counter request when the provider count fails
* fix(proxy): log a prompt-only token count fallback as a one-line warning
* fix(proxy): send a prompt-only token_counter request to the provider only on Gemini and Vertex AI
* fix(proxy): count a token_counter prompt sent next to an empty messages or contents list
* fix(proxy): count a prompt next to messages on the local token count instead of answering 500
* fix(proxy): answer 400 for a token count prompt next to a conversation unless drop_params is set
With drop_params the conversation is counted without the prompt. On the local count, a prompt sent with system or tools is counted as a user message with them, like the messages form.
* test(proxy): count tools with a prompt in the token counter fallback test
* test(proxy): cover prompt counting on /utils/token_counter in the Gemini count wire tests
* test(proxy): name the token counter fallback test by what it checks
* fix(proxy): track and drain management audit log writes on shutdown
* fix(proxy): track hook spawns and defer registration in audit log drain
* fix: satisfy LIT006 on audit log task registry by documenting the required cast
* fix(proxy): satisfy LIT006 and LIT010 in audit task registry
* test(proxy): drive the audit drain tests with events instead of wall-clock waits
* fix(proxy): drain audit writes beside the shutdown flushes and track only premium writes
Run the audit drain concurrently with the gateway and request-error flushes so
the two waits overlap inside the teardown budget, register the running audit
task only after the premium check, drop the four duplicate task-failure
loggers, and cover the shutdown order and a real hook spawn with tests
* fix(proxy): start hook audit writes before their email and secret manager calls
Key and user hooks now spawn their tracked audit write first, so a slow
invite email or secret manager call no longer spends the drain budget
before the write starts. The endpoints stop tracking the hook tasks, so
shutdown never waits on hook I/O when audit logging is off.
Adds regression tests for the hook order, the failed block or unblock
update, and the shutdown drain without a database.
* fix(proxy): track only the audit insert, not the code that awaits it
The shutdown drain now waits on each audit row insert itself, so it never waits on a caller's later work and covers every caller without per-endpoint wrappers. The user/new hook still tracks its own task because it reads the user row before the insert
* test(proxy): drain the audit registry in the config audit tests
The audit insert now runs in its own tracked task, one hop after the caller's task, so a single yield no longer reaches it
* fix(proxy): name dropped audit rows in the drain timeout warning
Each tracked audit task is named after its action, table, object id and destination, so a shutdown that drops a row says which one. Adds tests that the drain waits for slow audit callbacks and that the warning names the dropped row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(guardrails): pin content filter logging_only matches and UI modes in an integration test
* fix(guardrails): record every content filter match in logging_only and offer the mode in the UI
* style(tests): format the content filter logging_only integration test
* refactor(guardrails): keep the logging_only predicate and scan override within the type gates
* fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning
* test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test
* test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects
* refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach
* test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter
* test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up
* test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): reject unconvertible blocks nested inside a tool_result
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
* fix(proxy): count endpoints return missing-key and request errors in their source API shape
* fix(proxy): drop imports the count error change left unused
* fix(proxy): let Entra-authenticated Azure AI Claude count tokens and keep Google count errors in the Google envelope
* fix(proxy): send Azure environment keys as x-api-key on Azure AI Claude counts and log unexpected Google count failures
* fix(proxy): count with the global OpenAI and Anthropic keys chat already uses
* fix(proxy): resolve the OpenAI count base with the key so a global key stays on its gateway
* fix(proxy): keep unexpected Google count failures generic so internal details stay out of the response
* fix(proxy): resolve the Azure AI Claude count base from the same settings chat reads
* fix(proxy): count OpenAI tokens with the workload identity bearer chat already uses
* fix(proxy): mint the OpenAI workload identity bearer inside the count error handling
* fix(proxy): report a failed OpenAI workload identity exchange on the count endpoints as a 401
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(bedrock-passthrough): return AWS-shaped errors so AWS SDKs read the code and message
A guardrail block on a /bedrock route now answers with x-amzn-errortype ValidationException and a top-level message, and an error Bedrock returned (such as a 429 ThrottlingException) is passed through with its status, x-amzn-errortype and body
* fix(bedrock-passthrough): keep 5xx guardrail failures and failure-hook replacements unchanged
* test(bedrock-passthrough): guardrail vendor failures and unknown models keep the litellm error
* fix(bedrock-passthrough): only answer guardrail exceptions that blocked content as ValidationException
* fix(bedrock-passthrough): pass Bedrock's content-type through with its error body
* refactor(bedrock-passthrough): wrap the guardrail ProxyException instead of copying its fields
Unit tests drive the error response with the real request processor and ProxyLogging
* test(bedrock-passthrough): cover the direct model route, invoke, and auth, budget and rate-limit errors
* fix(bedrock-passthrough): pass through only errors a Bedrock model operation returned
A guardrail service's HTTP error that carries x-amzn-errortype keeps the litellm error
* refactor(bedrock-passthrough): mark Bedrock's errors where the passthrough raises them instead of matching the URL
A guardrail service's HTTP error never goes through the passthrough call, so it keeps the litellm error whatever its URL
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
The shared disconnect release dispatched to the legacy limiter, which decrements unconditionally, so a rejected realtime attempt freed a slot it never took. The generic release is v3-only again and the Responses WebSocket close releases through its own method
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them
* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): reject unconvertible blocks nested inside a tool_result
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): fetch Gemini remote media without blocking the event loop
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* chore(count_tokens): give the Gemini count helpers' list returns a reason
* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does
* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400
* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(proxy): reset end-user spend in committed pages so the budget cascade survives large LiteLLM_EndUserTable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): collapse _enduser_ids_by_tier return to one line for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the end-user page IN list as bounded for the unbounded-IN check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn
* test(responses): mock the websocket slot release and mark the per-turn start time naive
* style(responses): keep the per-turn start time line within ruff format
* refactor(responses): type websocket turn events as read-only mappings
* fix(responses): settle the websocket budget reservation on the first turn only
Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly
* fix(proxy): keep the legacy limiter's websocket slot until disconnect
A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): share SAML sign-in state over Redis without enable_redis_auth_cache
SAML login tickets, the single-use assertion mark and cached IdP metadata
lived in user_api_key_cache, which only gets Redis when
enable_redis_auth_cache is on. With several replicas or workers the IdP
callback often landed on a process that never saw the ticket and got a
401, and IdP-initiated sign-in was refused even with Redis configured.
SAML state now has its own DualCache that _attach_redis_usage_cache
always wires to Redis, like the CLI SSO session cache, and the SAML cache
calls are async.
* test(sso): drop docstring from the SAML proxy pair fixture
* test(sso): cover the SAML login and metadata endpoints
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does
* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends
* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types
* test(count_tokens): compare tools by value in the public api test
* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI
Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.
* test(count_tokens): expect the image detail the adapter now sends after merging main
* fix(count_tokens): type normalize_native_messages without the unimported Any
On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError
* test(count_tokens): expect the converted Responses input in the budgeted-key count integration test
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
* fix(guardrails): honor streaming settings on the Noma v2 guardrail
The Noma v2 initializer now passes streaming_buffer_until_moderated,
streaming_end_of_stream_only, streaming_sampling_rate and
streaming_buffer_release_on_scan from the guardrail config to
NomaV2Guardrail, so the unified streaming hook applies them. The
constructor validates them, so a custom guardrail class built on
NomaV2Guardrail gets the same checks. Unset keys keep the hook defaults,
and an invalid value is ignored with a warning naming the guardrail, so
the guardrail still loads.
Resolves LIT-9633
* test(guardrails): cover Noma v2 streaming settings across clients, failures and workers
Adds integration cells for the custom-subclass loader, string and invalid
values, unset and false keys, a Noma outage with and without block_failures,
an upstream rejection, n=2 choices, pre_call and non-streaming guardrails,
the OpenAI and Anthropic SDKs (sync and async), stored-guardrail info, a
PATCH while traffic flows, a burst while Noma is down, a slow Noma, and
replaced workers.
Resolves LIT-9633
* test(guardrails): pin the PATCH probes to one worker at a time
Each probe suspends the other worker and opens a fresh connection, so the
test checks that every worker applies the patched setting.
Resolves LIT-9633
* test(guardrails): check sampled scans by count, not window position
/v1/messages hands the streaming hook raw network reads, so where the
default sampling windows fall depends on timing; the base leg caught one
shifted window. Check the scan count and the final full-text scan instead.
Resolves LIT-9633
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): carry a 429 status on cooldown and no-deployment errors
RouterRateLimitError and RouterRateLimitErrorBasic had no status_code, so the
proxy answered an all-deployments-cooled-down request as a generic 500 class
error and logged the bug report notice for every cooldown. Both now carry
status_code 429, which the proxy maps to a rate_limit_error and which the
retry policy already treats as retryable
* fix(proxy): start Sentry at config load and never attach request bodies
Sentry only came up inside set_callbacks on the first LLM call, so boot and
background errors were never reported, and general_settings.alerting: [sentry]
always raised Missing SENTRY_DSN because the alerting path read a dead
litellm.utils pin instead of the live SDK. The proxy now initializes Sentry
once while loading the config whenever sentry is a configured callback or
alerting target, the alerting and failure handlers read the live instance, and
max_request_body_size is pinned to never so prompts and completions stay out
of Sentry events
* fix(tracing): log Lens failures at ERROR and export row counters to Prometheus
A tracing store that fails to start, or a Lens service that rejects a batch,
was only logged at WARNING, so operators running at the default ERROR level
never saw tracing silently go dark. Both now log at ERROR and name the lens
store. The exporter's rows_written and rows_dropped tallies were never
exported, so the two counts are now Prometheus counters
litellm_lens_export_rows_written_total and
litellm_lens_export_rows_dropped_total, registered lazily on the default
registry like the other proxy-level counters and charted on the all-metrics
Grafana dashboard
* test(proxy): drop the litellm import the alerting tests no longer use
* fix(proxy): start Sentry once a DSN is set, from any config source
Boot-time init used to lock Sentry in even with no SENTRY_DSN, so a DSN loaded later from DB environment_variables was never picked up. initialize_sentry now skips when no DSN is set and re-inits when the DSN changes. alerting_handler calls it too, so alerting: [sentry] set through the DB works. Drops the litellm_settings.callbacks check since that config fails to load with sentry in it
* fix(sentry): keep stack frame locals out of events unless PII is opted in
Frame locals carried prompts (messages, request bodies, provider options) into every error event. include_local_variables now follows SENTRY_SEND_DEFAULT_PII
* test(router): drop the test that only pinned the 429 class attribute
* fix(proxy): start Sentry at boot when it is only an input callback
* fix(proxy): keep Sentry init and alerting within the strict ruff and basedpyright ceilings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align A2A model visibility assertions with supported routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep A2A model entries on their existing routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): point the listing model at the harness upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): coerce require_key_agent_access_defined like the settings validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject agent access lookups instead of patching the handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GET /v1/mcp/server takes an optional user_id (admin, or the user themselves) and returns the union of the servers each of that user's teams reaches plus allow_all_keys servers. A user in no team gets the unscoped list, a deleted team still listed on the user is skipped, an unknown user is a 404 and a restricted virtual key is refused like for team_id. team_id keeps precedence when both are passed.
The dashboard MCPServerSelector passes the team and user it edits (key edit sends keyData.team_id and user_id, key create sends the picked team and owner, user edit sends the user) and narrows the server options to members of the access groups selected in the same control, keeping allow_all_keys servers and servers already granted so they can still be removed.
Resolves LIT-7784
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fastapi_sso opens a bare httpx.AsyncClient for the non-PKCE token exchange and
userinfo call, so litellm_settings.ssl_verify and SSL_VERIFY never applied and
an IdP on an internal CA failed with an opaque 503. A LiteLLMHttpClientSSO mixin
replays that exchange on litellm's shared SSO client for the generic, Google and
Microsoft providers and surfaces IdP connection and TLS failures as a 401
ProxyException naming the endpoint.
Resolves LIT-9445
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): name the successor model for deprecating deployments
* chore(ui): add successor_model to the generated deprecation api types
* fix(proxy): keep the cost map successor when a deployment overrides only the date
* fix(proxy): read the cost map successor from metadata.successor
The cost map schema rejects a top-level successor_model key, and curated
entries already carry the successor under metadata.successor. Read that key,
and look it up even when the cost map entry has no deprecation date, so a
deployment that sets only its own date still names the successor.
* fix(proxy): keep the first deployment when deduping, let a blank successor hide the cost map one, and name a served successor by its model group
* fix(proxy): name only the deployment's own successor_model, since cost map successors are LiteLLM ids rather than served model names
* fix(proxy): leave out a successor_model the deployment's callers cannot call, with a warning
* fix(proxy): report successor_model as the admin set it and only warn when no model matches it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs
* fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off
* fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off
* test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(agents): support Databricks Agent Platform agents in the agent gateway
Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire
(Responses-shaped input, output_text items, output_text.delta SSE) to Model
Serving endpoints (/invocations or the unified /serving-endpoints/responses)
and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials
Register it through the agents config block or the Admin UI agent catalog and
call it through /a2a/<agent> (message/send, message/stream) or model:
a2a/<agent> chat completions. The chat route now runs every completion-bridge
agent through its own provider, and attaches the minted Databricks OAuth token
for both bridge and URL agents. Consumed OAuth settings never reach the
provider call
* fix(agents): harden databricks_agent input, output, and OAuth handling
Non-text content parts answer 400 unless drop_params is set, a reply without
an output list answers 502, flat OAuth fields resolve an os.environ api_base,
the bridge debug log no longer echoes the request model, routing tests stub
only the HTTP boundary, and the catalog description fits the dropdown
* fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block
* fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App
* refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name
* fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty
* test(agents): drive the flat OAuth a2a send through the real bridge
* fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change
* fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents
* fix(databricks_agent): address review caveats on chat bridging, params, and auth
Map max_tokens and max_completion_tokens to max_output_tokens and pass
temperature, top_p, and user, so /v1/messages works without drop_params.
Send A2A message metadata only with string values, answering 400 or
dropping the rest under drop_params. Join message items with a blank line
in both stream and non-stream replies. Cache the env service principal
token until shortly before it expires. Keep databricks_agent replies out
of the response cache.
On the chat bridge, let request fields win over the agent's stored params
while pinning the agent's api_base and api_key, and apply the agent's
static_headers between client and agent extra_headers, as /a2a does.
Resolve os.environ/ only for databricks_agent api_base and api_key instead
of every bridge provider. Ignore an unusable databricks_oauth block on URL
agents on chat, as at the merge base, instead of answering 400.
* test(internal_context): list the databricks agent token cache as process-local
* fix(databricks_agent): keep the agent's endpoint, input, and target out of client control
extra_body model and input answer 400, or are dropped under drop_params.
A client api_base or base_url never replaces the agent's card URL on the
bridge. Chat to a URL agent keeps its merge-base call, and an incomplete
databricks_oauth block on a URL agent keeps its merge-base ValueError.
* test(databricks): expect the merge-base 400 for a client api_base on agent chat
* fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach
* fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(decisions): read a deployment's own supports_box_questions flag before the cost map
The box-question check only read the shared cost-map row and required a decisions mode there, so an OpenAI-format deployment whose model_info sets supports_box_questions was refused with a 400, which kept the wire test from #46633 red on main
* fix(decisions): keep the box flag on its deployment and out of request bodies
A model_info that is not a plain dict no longer fails the call, the proxy
decisions endpoint drops a caller-sent model_info, and the router stops
copying supports_box_questions onto the shared cost-map key.
* fix(decisions): keep the box flag shared with same-model deployments like main
Scoping supports_box_questions to one deployment turned box questions on an unflagged sibling of the same backend model from reaching the provider into a 400. Siblings inherit the flag through the shared cost-map row again, as they do on main. Also formats two new tests.
* test(decisions): pin box flag inheritance by an unflagged same-model deployment
* test(decisions): audit box-question deployment flag across providers, routes, and chaos cases
* fix(cost): price bridged /v1/messages reasoning tokens from the chat completion, not the served Anthropic body
* fix(cost): keep the type gate flat for the bridged /v1/messages billing path
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
