* fix(bedrock): block realtime replay after input starts the session and reject non-string instructions and voice
* test(bedrock): name the malformed realtime setting test for its string cases
* fix(bedrock): replay the opening realtime message until Bedrock accepts the stream and refuse any fallback after commit
Also rejects a malformed modalities, input_sample_rate_hertz or tools setting with invalid_type, dropped under drop_params.
* fix(bedrock): replay a refused opening message on any fallback and reject OpenAI voices and non-string audio formats
* fix(realtime): read the client scope only when the websocket has one and reject any-case OpenAI and empty Bedrock voices
* test(bedrock): cover realtime replay, refusal and committed failure through a real proxy
Fix the scripted Bedrock peer's port import and the nested tool call cell's stale expected events, so the Nova Sonic integration file collects again
* fix(openai_like): surface provider response headers on non-streaming chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(openai_like): mark the header test locals Final
* fix(proxy): leave out response header values HTTP cannot carry
Non-streaming OpenAI-like chat now forwards every provider header, so on the
httpx transport a provider header value outside latin-1 made the proxy answer
500 after the provider had served the call. get_custom_headers now leaves such
a value out and keeps the other headers, which also ends the same 500 on
streaming
Adds integration cells for the provider request id on non-streaming chat and
the Responses bridge, LiteLLM headers an upstream cannot overwrite, spend
parity under an upstream cost header, the latin-1 case, and the logging
callback payload
* fix(cost): ignore an upstream response cost header that is negative or not a number
* test(integration): cover invalid upstream cost headers and non-latin-1 headers on every endpoint
* fix(prometheus): keep deployment metrics when a provider rate-limit header is not a number
* test(prometheus): type the rate-limit header test and drop its clock reads
* test(integration): cover deployment metrics under an unparseable provider rate-limit header
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): preserve key settings drafts during background refreshes
* fix(ui): refresh untouched key settings while preserving edits
* fix(ui): keep team-change MCP clears across key refreshes
clearMcpGrants now marks the MCP fields dirty, so a background refresh with
keepDirtyValues no longer restores the grants a team change just cleared.
Also adapts the draft tests to the onSubmit(values, dirtyFields) signature
on main and asserts a refresh does not mark untouched fields dirty, since the
team admin member-key path rejects saves with non-budget dirty fields.
KeyEditView becomes a one-line arrow to stay under the max-lines limit.
* fix(ui): keep MCP tool defaults out of a key's edited fields
MCPToolPermissions writes non-delete tool defaults for unrestricted servers
on mount. Marking that write dirty made a team admin's budget-only save on a
member key get rejected for editing mcp_tool_permissions. onChange now says
whether a write is a default or a user edit, and the key editor dirties only
edits.
The edit session is now keyed by token or token_id, matching the form's own
fallback, so switching between keys that only carry token_id starts fresh.
* fix(ui): keep a new MCP server's default tool limits across refreshes
Default tool writes for a server the user just added in this edit session
are part of that edit, so they are marked dirty and survive a background
refresh. Defaults written for an already saved server stay clean, so a team
admin's budget-only save on a member key is still accepted.
* fix(azure_ai): count Grok reasoning tokens in completion_tokens
Grok on Azure AI Foundry reports reasoning_tokens outside completion_tokens,
like xAI does, so azure_ai/grok-* calls billed only max(visible, reasoning)
output tokens. Fold them with XAIChatConfig.fold_reasoning_tokens_into_completion
in transform_response and in a new streaming iterator, matching the xai provider.
Split out of #46377.
* fix(azure_ai): fold reasoning tokens for every deployment name, streaming or not
Non-streaming checked the deployment name for grok while streaming checked the
chunk model field, so a Grok deployment named without grok was only folded when
streaming. The fold already only fires when total_tokens equals prompt plus
completion plus reasoning, so it now runs on every azure_ai response.
* feat(ui): shareable URL state for usage views and users-to-usage deeplinks
The Usage page now keeps the view, global view tab, date range and entity filters in the
URL, so links can be bookmarked and shared. Relative presets are stored as range=7d and
keep rolling, custom ranges as UTC from/to. Invalid params are dropped, and a link to a
view or user the viewer cannot access shows a toast and falls back to the default view.
Internal Users rows get a "View usage" action that opens usage filtered to that user, and
the per-user usage views link back to the user's record. A "Copy Share Link" button sits
next to "Ask AI"
* fix(ui): clean usage links for sessions without the organizations query
The organizations query only runs for premium sessions, and a disabled query stays
pending forever, so the URL cleanup never ran for other sessions. Gate on isLoading
instead, and give the access denied toast an id so a re-run effect shows it once
* fix(ui): type-check usage entity filters and address review findings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep team usage available to internal users
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format users table usage action
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): sync date picker preset with the current usage range
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): restore dashboard AGENTS.md from main and drop unused usage state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): share custom usage date ranges as calendar days
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep per-user agent usage page across rerenders
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect the tool search call replayed beside the function call for Anthropic targets
* test(integration): wait for the team spend flush before killing a worker
* test(integration): poll for the S3 attempt the batch sinks write concurrently
* test(integration): queue the terminal object and its sibling into the same s3_v2 flush
* test(integration): time the long Bedrock version against an equally long unversioned name
* test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it
* test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries
Each price field now parses its per 1M text into a per token number or null inside the
schema, so the save handler gets clean values and the hand-rolled conversion goes away.
Negative or non-numeric prices now show a field error instead of being saved
* feat(anthropic): workload identity federation and pluggable identity sources
Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one
commit on top of litellm_internal_staging without the dashboard changes.
Deployments on anthropic/ without a static api_key can exchange an OIDC
workload assertion for a short-lived sk-ant-oat01 token through a shared
RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file,
an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment,
per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation
fields are server-owned: refused inline in request bodies and on
POST /model/new, proxy-admin only on credentials, and the token exchange
is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS
adds a host. GET /credentials/{name}/jwks exports the public key set of a
LiteLLM-signed credential for the Claude Console.
The OpenAI federation trio from #39613 rides along on the backend side with
the same server-owned handling.
Fixes#28607
Resolves LIT-6107
Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>
* fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries
The files handler enabled workload identity on batch-result downloads but never received the
deployment's litellm_params, so a deployment authenticating through a named credential could only
mint from process-wide env vars. It now threads litellm_params through to the auth header the way
the batch retrieve path already does.
LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme,
so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the
gateway. Entries are now parsed as network locations whether or not they carry a scheme.
* fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle
* test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries
* fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly
* fix(proxy): decrypt stored litellm_params before the WIF write gate
* fix(proxy): hide WIF secret references from /health output
* fix(proxy): keep the proxy error shape on credential endpoint refusals
* fix(proxy): hide identity token file paths from /health output
* feat(openai): honor per-deployment workload identity on non-chat routes
Image generation, audio speech and transcription, moderation, text
completions, files, batches, and fine-tuning built their OpenAI client from
the env vars alone, so a keyless deployment carrying
openai_identity_provider_id, openai_service_account_id, and
openai_identity_token_file failed with the api_key error on every route
other than chat, Responses, and embeddings. Those handlers now build their
client through the shared workload identity resolver, and the managed batch
file-access credential extraction keeps the trio
* fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys
Non-chat handlers built a fresh SDK client per call, so files, batches,
fine-tuning, text completion and sync moderation exchanged the OIDC token on
every request. The builders now reuse the cached workload identity client per
identity, base, timeout, retries and organization; static-key clients are
still built fresh.
A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no
longer silently overrides a deployment's openai_identity_* params. An explicit
distinct api_key on the deployment or call still wins.
* fix(openai): keep shared workload identity clients on the process http session
The workload identity client is cached and shared across requests, so it must
never borrow the transport a single caller passed in. Static-key clients keep
using the caller's http client; federated clients always ride the process-wide
session. Also makes the http client helpers on BaseOpenAILLM public since the
builder now needs them from outside the class.
* test(openai): pin partial deployment identity beating the process key
* test(openai): type the managed-object invoke callback
* fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working
The Bedrock Claude Platform route already reads anthropic_workspace_id from
optional_params, so banning that spelling as a server-owned federation
parameter broke a pre-existing client capability. The federation field is now
anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID),
which restores the base branch's behavior for Bedrock callers, drops the
Bedrock-specific hint from the refusal message, and deletes the unconditional
ban constant that no longer had a reader
* fix(auth): share one exchanged token across workers reading the same assertion
Anthropic accepts each identity assertion exactly once, so two uvicorn
workers reading the same token file both minting from it means the second
exchange is denied with jti_reused. Minted tokens now land in a per-user
0700 cache directory guarded by a file lock, so workers on the same host
reuse one exchange until the token expires or the assertion rotates. A 401
is only retried when the re-read assertion actually differs, and the denial
hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache
and an empty value disables it
* feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals
The httpx-based OpenAI handlers built their Authorization header from the
deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying
openai_identity_provider_id, openai_service_account_id, and
openai_identity_token_file sent "Bearer None". They now resolve the bearer
through resolve_openai_bearer_token, which exchanges the deployment's
identity token the same way the SDK-based routes do.
A deployment that carries only part of the identity trio now raises a 500
naming the missing field instead of silently falling back to the process
key, and the SDK client builders take a factory for the static-key http
client so the federated branch never touches litellm.client_session.
* fix: keep anthropic federation from being shadowed or leaked
An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated
deployment sent an empty x-api-key on every call instead of minting a token.
Blank values now read as unset, and a real static key on a federated deployment
logs once that it outranks federation and nothing is being federated.
The exchange-host allowlist matched hostnames only, so a second process on
another port of an allowed host was trusted with the workload's identity token.
An entry that names a port now trusts that port alone, while a bare host still
trusts every port.
The shared token store exists so the workers reading one projected token file do
not each spend its single-use jti. A source that mints its own assertion per
exchange shares nothing with another worker, so it no longer writes a live token
to disk for a lookup that can never hit.
* feat(openai): resolve per-deployment workload identity for the assistants routes
OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders.
* fix: unlink a staged token file a failed write leaves behind
The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads.
* refactor: move anthropic jwks derivation behind a provider-owned tagged union
* feat(openai): resolve per-deployment workload identity for image variations
Build the image variations client through the shared workload identity
builders instead of scraping locals(), and give the async path the async
http session it was missing. Let a federation trio serve the model-less
batch fallback, and hold the standard LRU size of exchanged tokens.
* fix: unlink the staged token file when its write fails at close
A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token.
* fix(anthropic): close the staging descriptor before writing the shared token file
* fix(wif): judge federation writes by what they set, not what is stored
The admin gate read the stored deployment, so a team admin lost edit, delete
and Test Connection on any deployment carrying federation params. It now
returns early unless the submitted fields touch the federation surface, and a
Test Connection probe that points the deployment at its own api_base is still
refused, with the 403 no longer wrapped into a 500
The rest of the same review pass: POST /model/new refuses only a blocking
value of `blocked`, so a client that always sends `blocked: false` is not
turned away; a request body can no longer pick which federated identity to
mint as by naming a stored credential; an advisory refresh the executor
refuses disarms the entry instead of wedging the identity until the follower
timeout; the static-key shadow warning resolves its env fallback inside the
cache instead of once per request; credential writes drop nulls before
storing them; the token exchange validates the endpoint URL before reading an
assertion and keeps refusing redirects across a client heal; /health hides
every server-owned federation field from non-admins; and the async create_file
and create_batch paths say which setting is missing when the provider resolves
no URL
* style(tests): keep two blank lines before the batch_cost_is_final block
* test(batches): type the workload identity fallback test fixtures
* fix(moderation): stop validating unrelated kwargs on the sync workload identity path
* fix(auth): refuse server-owned workload identity keys in array form and inside a client router config
* fix(openai): keep a deployment's own key ahead of workload identity on every route
Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation,
image generation, and assistants so a deployment key equal to the process key still wins. A partial
identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and
/v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments
* fix(containers): pass deployment params to container file auth headers
Container file routes built their auth headers without the deployment's
litellm_params, so a keyless deployment with an OpenAI workload identity
sent Bearer None on upload, list, retrieve, delete and content calls
* fix(openai): keep process-wide key precedence and move token exchange off the event loop
Restore the merge base order where an explicit or process-wide OpenAI key wins over
federation, so exporting OPENAI_API_KEY keeps working the way it did before.
Run the header-building validate_environment calls (image edit, videos, containers,
vector stores, vector store files) and the decisions and System One request
preparation in a worker thread, so a slow token exchange no longer stalls the loop.
Federate realtime websocket sessions, live sessions, realtime client secrets and
transcription sessions, and the OpenAI moderation guardrail when no key is set.
* fix(openai): answer 401 for a partial identity trio and federate realtime health checks
A deployment with part of the workload identity trio and no static key now
fails as an auth error naming the missing params, so the router does not retry
it as a server error. The realtime health check sends the exchanged bearer for
a keyless federated OpenAI deployment, and API token counting reads the same
static keys as every other route before minting.
* fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config
* fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting
* test(integration): refuse caller-chosen identity params in user_config and bracketed form keys
* fix(openai): send the honored secret manager key from SDK client builders
A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key
on image, assistants, files, batches, and other SDK-built routes instead of failing with
no api_key, matching what the workload identity resolver already treats as a static key
* test(openai): check the token exchange runs off the event loop instead of timing the loop
---------
Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version
* fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider
* fix(utils): refresh provider model sets after register_model
* fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row
* test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version
* fix(utils): keep router deployment ids out of provider model sets
register_model now refreshes the provider sets, so the per-request and
router registrations keyed by a deployment id put those opaque ids into
wildcard model listings. Those callers now pass
add_to_provider_model_sets=False.
* fix(azure): carry responses_api_version through the chat to Responses bridge
completion() dropped responses_api_version from the litellm_params it hands
the Responses bridge, so azure/responses/<model> on a dated chat api_version
still built the 404ing /openai/responses URL. Also cover the auto-router
marker never forwarding the new param.
* fix(azure): stop listing a wrong set of reasoning efforts when none is rejected
The message said only low, medium and high are supported, which is wrong for
models that also take xhigh such as gpt-6.1-sol.
* test(azure): use the latest grok and gpt-5.4 rows in the new azure tests
* fix(proxy): list each wildcard-expanded model once
A provider set can hold a model both bare and provider-prefixed, which now
happens after any priced request registers its openai/<model> key, so the
openai/* listing showed that model twice.
* fix(azure): route Responses past dated GA api_versions by default
A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before
* chore(azure_ai): mark the grok chunk_parser dict as an override contract
* fix(azure): rewrite only the route's /openai segment when moving a call to v1
The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway
api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses.
Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too,
so the rewrite now only touches the last /openai path segment.
The Responses version resolver no longer duplicates the builder's default for an unset api_version.
Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite
now has one test for the dated GA default and one where only responses_api_version can make the call work
* test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none
Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params
* fix(utils): list ai21 chat and vertex code models under their provider wildcard
register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too.
* fix(main): keep caller-chosen model names out of provider wildcard listings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(cost): price reasoning tokens of /v1/messages bridged onto the Responses API in the cost header and breakdown
* fix(cost): record the bridged Responses API result only after the served body is built
* fix(cost): keep the served service tier when a bridged Responses reply has no message item
* fix(sso): concurrent first sign-ins land on the account the winner created, worker-kill test pins half its streams to each worker
* test(integration): a worker kill returns once the worker is dead, the reap waits at block exit
uvicorn can take a full worker healthcheck to reap a worker that died mid-ping, so kill() held
the caller that long and peers held on the surviving worker hit their client timeout
* fix(sso): a sign-in that loses the first-insert race joins the default teams
The losing sign-in could read the winner's row before new_user added the default teams, so its
CLI key was minted with no team. The race tests now run the real new_user against a fake user table
* fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing
A non-list tools value crashed the vector store lookup during auth and the
built-in tool cost lookup in the failure logging path, so the proxy answered
an unhandled 500 and closed the keep-alive connection. count_tokens now
answers a 400 invalid_request_error for a malformed body, drops malformed
tools when drop_params is on, and both lookups skip values of the wrong shape.
* fix(proxy): skip non-list tools in the tool allowlist check
Also validate the count_tokens body without recursion, which the recursion detector rejects.
* test(integration): restore the int tools case on count_tokens
The proxy now answers a non-list tools value with a 400, so the case is deterministic again.
* fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids
* fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids
Check tools after request setup so the Claude Code default and the logged call type apply, read the request,
global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini
generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off
* fix(proxy): keep the request's call type when logging a gate rejection after request setup
The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses
rejection as aembedding, so it now only applies when the logging object carries no known call type
* fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry
* fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket
* refactor(proxy): leave the responses websocket out of the request-level tool check
* fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections
* fix(proxy): read router default drop lists and deployment ids in the malformed tools check
* fix(proxy): leave malformed tools to the router when it can pick another deployment
Fallbacks, auto-routers, model strings, comma batches and team model names
let the router pick a deployment after the proxy checks the request, so the
proxy now forwards a malformed tools field there as main does, and only
rejects it when the deployment it checks is the one that serves. Token
counting never reroutes, so count_tokens and input_tokens keep reading the
group they count with. additional_drop_params names are read the way core
reads them, so a list with a non-string entry still names its strings.
* test(integration): cover malformed tools on reroutable models, router defaults and deployment ids
* fix(proxy): resolve malformed tools at the deployment the router picks
* fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools
Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells.
* test(integration): expect a 400 for an empty-string tools field on /v1/messages
Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test
* fix(router): keep web search routing for tools passed as a tuple
* fix(router): keep a deployment's own tools when drop_params drops a malformed request tools
* test(router): list _merged_tools as covered through the deployment tools merge tests
* fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them
* fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError
* fix(prometheus): keep deployment health metrics clean on a malformed tools rejection
* test: type the new malformed tools tests
* test: mark new test locals Final and type kwargs dicts
* docs(agents): keep directory rules in nested AGENTS.md and pin integration tests to CircleCI
* docs(agents): scope the CircleCI rule to the files it covers and ask agents to read nested AGENTS.md
* ci(coverage): reject GITHUB_FILES integration contracts run from a test-unit.yml shard
* ci(coverage): report integration ownership findings under their own header
* ci(coverage): flag GITHUB_FILES contracts run from any non-dedicated workflow
* docs(integration): say a pre-merge run is no reason to route an integration file to GitHub
* ci(coverage): tie each dedicated integration workflow to the directory it owns
* fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open
GuardrailToolParam required a type key, so Gemini native tools
(function_declarations, googleSearch) failed validation inside the hook
and either returned a 500 or skipped the scan when fail_on_error was
false. The type key is now optional and left out of the forwarded tool
when absent, so an echoing guardrail server cannot write type: null into
the provider request.
structured_messages was re-validated against the chat completion message
types, which silently emptied tool_result, tool_use, image and document
content blocks to content: [] and dropped undeclared keys before the
guardrail saw them. The request model now forwards rows as opaque
mappings, which also retires the write-back shim that restored the
original rows for echoed-back entries.
The fail-open critical log line now carries error_type so a timeout, a
refused connection and an upstream HTTP status can be told apart without
the traceback.
* fix(httpx): retry aiohttp server disconnects the way httpx does
aiohttp raises ServerDisconnectedError when a kept-alive connection is
closed by the peer before a response arrives, which is what a load
balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces.
The transport mapped it to httpx.ReadError, which the handler's
connection-error retry does not cover, so the request failed instead of
being resent on a fresh connection. It now maps to
httpx.RemoteProtocolError, the error httpx itself raises for the same
condition, so the existing retry fires.
* test(httpx): build the stub server reply in one bytes literal
* fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable
The connection-error retry closed its fresh client before a streamed body was read,
so an aiohttp disconnect now billed a second upstream request and still failed. It
also skipped masking on a status error from the resend, resent PATCH and DELETE as
POST, dropped multipart files and ignored the handler's ssl_verify.
The retry now resends the request the first attempt built, on a client with the
handler's TLS settings, closes that client when a streamed body is closed, and masks
a status error the same way the first attempt does.
* fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type
* fix(guardrails): resend a generic guardrail check cut before any response
A load balancer that closes an idle keepalive connection made the guardrail POST
fail with "Server disconnected", because #46336 only resends POST on ConnectError.
AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to
False, that gives a RemoteProtocolError the same single fresh-client resend PUT
and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a
guardrail check is a pure verdict over the request. Every other POST keeps the
#46336 rule
The fail-open log line now puts error_type at the end so the pre-PR text stays
contiguous for existing alert matchers
test_rows_reach_the_guardrail_verbatim now reads the body off a real local
guardrail server instead of mock call args
* fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration
A ServerDisconnectedError raised while reading a response body now stays
httpx.ReadError, as it was before this PR. Only a disconnect before any
response maps to RemoteProtocolError, which is the case the resend rule is
about. Stream readers that end on ReadError, like the SAP stream iterator,
keep their old behavior
The loopback-server disconnect, TLS and guardrail resend tests move to
tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is
in-process only. Unit keeps in-process MockTransport checks for the idempotent
POST resend, masking on the resend, streamed and buffered client closing, the
guardrail resend, and the guardrail receiving rows verbatim
* fix(guardrails): resend a reset guardrail check and keep typed echoes on texts
An idempotent POST now resends once on any network error after connect
(ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load
balancer that answers a stale keepalive with a TCP reset rides it out too.
That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes
back to ReadError. This stops PUT and DELETE from stacking litellm's resend on
top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil
Guard from replaying its analyze POST on a disconnect
Every resend now runs inside the method's own error mapping, so a timeout on
the resend is a litellm.Timeout like the first attempt and a status error is
masked once by the caller
A guardrail server that parses rows with its own message types and echoes them
back without the keys those types lack (annotations, tool_calls[].index) is
read as an echo again, so its texts redaction reaches the LLM. A row only counts
as an echo when no value changed and every key the chat completion types
declare is kept
The wire tests for the PUT/DELETE resend, its masking and its TLS settings now
run on the httpx transport, where litellm's resend is the only retry, and the
POST and guardrail tests cover both transports and both cut styles
* test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend
Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head
* fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement
* refactor(guardrails): move the dropped-connection guardrail resend out of this PR
The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line
* test(guardrails): pick the closed guardrail port from the integration port allocator
Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port
* feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent
generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Both interactive transactions in /team/delete used Prisma's 5s default.
A lock wait or a short database stall past 5s expired the transaction
after the keys were already deleted, so the call returned 503 and left
the team row, memberships and team models behind
* fix(mcp): stop proxy-started legacy listeners from consuming server rpm
The listener the proxy opens on notifications/initialized for legacy MCP
sessions ran the server rate-limit admission once per target, so every
session spent an rpm slot before the client sent anything. Admission is
now injected into the listener and applied only to client-initiated
subscriptions (subscriptions/listen and resources/subscribe). Authorization
and its periodic recheck still run on every path.
* test(mcp): key the fake admission window by caller and server
* fix(mcp): stop subscription catalog refreshes from consuming server rpm
A subscription listener lists the caller's catalog to diff list_changed
events, at setup and on every upstream event. Those listings went through
the normal per-server admission, so against an upstream that declares
listChanged each legacy session spent several rpm slots before the client
asked for anything. The listener now marks its targets as already admitted
for its own lifetime, so its refreshes reuse the subscription's admission.
Client requests on the session are still metered.
* test(mcp): exercise subscription admission through the proxy rate limiter
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure
A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.
* fix(proxy): log a provider error frame through an attempt-scoped logging copy
The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.
* fix(proxy): scope the error-frame failure log to streams closed at the frame
A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it
* chore: drop duplicate litellm import and mark the stream_ended rebind
* fix(proxy): log only router-superseded attempts as error-frame failures
A client that hangs up at a forwarded error frame closed the stream the same way the router closes a superseded attempt, so a non-retried stream lost its spend. The router now closes a superseded attempt through aclose_superseded_attempt, and only that close routes the attempt to the failure log
* test: type the _settle helper's recorded events
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
OSV started flagging claude-agent-sdk 0.1.44 today, which turns OSV Scan red on every PR. 0.2.165 is the newest release outside the 3-day uv cooldown. The package sits only in the ci dependency group and is only imported by the cookbook
* feat(ui): add clean-slate playground chat prototype
* feat(ui): replace the playground chat with the redesigned composer and inspector
* fix(ui): close the playground chat gaps found in self-review
Resend only complete text exchanges, so empty replies, MCP direct calls and
transcriptions no longer leak into chat history. A stopped request can no longer
append output, attach its gateway trace to a newer reply or clear the newer
request's loading state. MCP selections are only pruned once both the server and
toolset lists load. The MCP tool picker lists toolsets again and calls the server
that backs the chosen tool, and a saved model the current key cannot use is
cleared when the model list loads, matching the old chat
Also vendors the scroll-fade-x utility the attachment primitive uses
* fix(ui): resolve playground toolset tools per server and drop late session ids
A toolset now lists only its own tools, each resolved on the server the toolset
names, so the args form and the call always target the same tool. A saved
toolset no longer asks for tools with its own id while the toolset list loads,
and a failed tool list says so instead of claiming the server has no tools
A stopped request can no longer overwrite the Responses session id, starting a
new chat drops a request that is still running, embeddings replies are no longer
resent as chat history, and a saved model missing from an already cached model
list is cleared too. A failing toolset list falls back right away instead of
retrying while the MCP pickers show loading
* fix(ui): guard playground sends and label inspector stats
Pressing Enter twice no longer sends the same message twice, an attachment the
new endpoint cannot take is dropped when the endpoint changes, a stopped
Responses request can no longer show its code output on a newer reply, and a
reply that broke mid-stream is no longer resent as history
The inspector's speed, cost and routing stats get the same icons as the reply
footer, and the footer's metrics now line up with its copy and Inspect buttons
* fix(ui): write playground key settings only through the encoded storage path
Key source, virtual key and custom headers now go through their own write loop, separate from the plain settings, so a key value can never take the plain sessionStorage branch
* fix(ui): restore playground BYOK connect and reconnect
Picking a BYOK server for a direct tool call now shows a key button next to the
server picker, so the key can be connected without leaving the MCP endpoint. A
connected BYOK server gets a Replace key action in the MCP dialog, also shown
when its tools fail to load, and saving a key reloads that server's tool list
* revert: restore the single playground settings storage loop
Reverts ca4e742115. The CodeQL finding covers the same encoded sessionStorage handling the old chat used, so it is left as is
* test(ui): type the BYOK credential mock without deep instantiation
* fix(ui): let users back out of replacing a BYOK key
The replace-key form in the MCP dialog now has Cancel, and it resets when another server is picked or the dialog closes, so a working key is never forced to be overwritten
* refactor(ui): name the MCP dialog row focus handler
* fix(ui): accept exact custom models, tags and decimal temperatures in the playground
A typed model or tag that is a prefix of a listed one could not be used,
because Enter picked the partial match and the custom action only showed
when nothing matched. The custom value is now a list option whenever no
exact match exists. Temperature gets an editable input beside the slider
so values finer than the 0.1 step can be sent again.
* fix(ui): stop claiming no guardrails ran when the gateway sent no headers
Direct MCP calls, A2A agents and any request that fails before headers
arrive built their trace from an empty header set, so the inspector said
no guardrails or policies ran. The gateway fields are now left unset in
that case and the inspector says the endpoint does not report them.
* fix(ui): leave policies unknown when the gateway headers are hidden
The proxy exposes x-litellm-applied-guardrails to other origins but not
x-litellm-applied-policies, so a dashboard on another origin read every
request as having run no policies. Policies are now unknown unless the
call id header is readable, and the inspector says so.
* fix(bedrock): drop dangerous beta without safeguards
* fix(bedrock): drop the dangerous-tool-use beta only next to structured outputs, keep it on Mantle
Bedrock rejects the pair with 'invalid beta flag' on Claude models before Opus 4.7, while requests without structured outputs and Bedrock Mantle accept the beta, so the drop now only applies to Invoke requests with output_config.format or a strict tool and no safeguards.
* fix(anthropic): keep the AnthropicThinkingParam import that main now uses
* fix(bedrock): drop the client's dangerous-tool-use beta whenever safeguards is absent
Bedrock rejects the beta with 'invalid beta flag' on Sonnet 4.5, Opus 4.5, and Opus 4.6 even without structured outputs, so dropping it only next to structured outputs left those beta-only requests failing. The beta does nothing without safeguards, so Invoke now drops it in every such request. Mantle keeps it. The thinking display helper goes back to main's version.
* fix(bedrock): drop the client's dangerous-tool-use beta on Invoke chat without safeguards
* test(bedrock): wire cells for the dangerous-tool-use beta on Invoke messages, chat and responses
* test(bedrock): type the new dangerous-tool-use beta tests
* test(bedrock_mantle): type the dangerous-tool-use beta Mantle test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): add the 1 hour cache write cost to the add model and model edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover both cache write tiers in the add model and edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): hoist the expected pricing payload out of the inline assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): share the per-token conversion between both cache write tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): derive the touched pricing field type from the pricing field list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): collapse the duplicated pricing source keys into one map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): convert add model pricing once for all selected models and describe the 1 hour fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): type the cache write tier deployment read back and fill the 1 hour field
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(e2e): keep the cache write tier spec on the dashboard prettier config
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openai): route gpt-transcribe to GPT audio transcription config
* test(openai): move gpt-transcribe tests under tests/unit
* fix(openai): bill gpt-transcribe from provider usage seconds when duration is unknown
Fall back to the provider-reported usage.seconds when neither a measured
file duration nor a response duration is available, so json responses
for undecodable uploads are no longer billed at zero. Move the routing
test next to the other provider config tests and replace the structure
test with a behavior test.
* fix(openai): bill provider usage seconds only for gpt transcription models
* fix(cost): decide provider-billed transcription seconds from the deployment model, not its pricing id
* test(integration): cover gpt-transcribe default request, provider-billed spend, and health probe
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(cost): remove price fields the real gemini and azure rows do not carry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): key vertex gemini cases on the vertex_ai prefixed rows and add gemini-4-argon
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add current gemini and vertex usage keys to stored responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add bedrock mantle claude-opus-5-5 cost tracking cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add bedrock mantle gpt cost tracking cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_two_identical_cache_hit_rejects_write_matching_spend_rows passed every run with the skip removed, so the dropped spend row it was guarding against no longer reproduces
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): streaming responses hand the event loop back every 50 ms instead of holding it for the whole buffered stream
* fix(proxy): yield to the event loop on every streamed chunk, no hold budget or clock
* docs(proxy): say why the streaming loop yields on every chunk
* fix(proxy): streaming loop yields once it has held the event loop 50 ms, clock read inline
* docs(proxy): streaming loop comment names the 50 ms hold
* Revert "docs(proxy): streaming loop comment names the 50 ms hold"
This reverts commit cbf1371873.
* refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50
* Revert "refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50"
This reverts commit 356454def6.
* docs(proxy): streaming loop comment notes the yield is a floor, not a count of releases
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count
/v1/messages runs clear_tool_uses context edits before it sends an Anthropic
body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted
the body unedited, so a request with older tool results cleared counted about
2.8x what inference bills.
The count now runs the same history and context-edit step inference runs,
honors additional_drop_params on the deployment, and skips compact edits so a
count never makes a paid summary call.
* fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs
Run compact edits in the count without the summary call so trigger checks and
history slicing match /v1/messages, honor a request-level additional_drop_params,
keep the full body when any deployment in the model group drops context_management,
and answer 400 naming a block Gemini cannot count when an edit trigger fails on it
* fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check
* fix(count_tokens): log a failed edit in the count at debug, since the send already logs it
* fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does
* fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check
The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits.
* fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name
* fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to
* fix(messages): drop an invalid context_management spec under drop_params so the send matches the count
* fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them
The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes.
* test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code
* fix(responses): log and bill every round of non-stream MCP auto-execute
Both model rounds run as internal sub-calls on the request's logging object, and the outer call logs the response the client received with usage, cost, and cost breakdown summed across the rounds. The internal-call gate now also covers the sync success callbacks, so an inner round no longer takes their first-wins slot
* fix(caching): let the parent call log and bill an internal sub-call's cache hit
A round of non-stream MCP auto-execute that the response cache served fired the
cache-hit callbacks on the shared logging object, so the spend log kept round 1's
function_call row and dropped the billed row for the answer. Internal sub-call cache
hits now fire no callbacks and carry a zero cost, and the summed cost breakdown leaves
out rounds served from the cache
* fix(logging): keep a parent row unique when its response came from the response cache
A RAG query or an MCP turn whose final round is served from the cache hands back the cached response's id, which already belongs to the row that cached it, so SpendLogs dropped the new row as a duplicate. Such a row now gets the cache-hit id suffix, and it is logged as a cache hit when it cost nothing.
* fix(logging): skip the zero-cost warning for a response served from the cache at no charge
A non-stream MCP auto-execute request whose rounds the response cache served is priced
at $0 before the parent marks it a cache hit, so it logged a false 'priced at $0'
warning and counted in litellm_zero_cost_requests_total. The diagnostic now treats a
response the cache served at no charge the same as a marked cache hit
* fix(responses): bill the paid first MCP round when a later step of the request fails
* fix(responses): log each MCP tool call under its own id so a failed follow-up keeps its spend row
The non-stream auto-execute path passed the request's call id to the tool call, so the tool's spend row and the request's failure row shared one request_id and the failure row was dropped on insert, leaving the paid first round unbilled.
* fix(responses): bill round 1 on a client disconnect, keep MCP tool rows in the request's session, and charge a fixed margin once per request
* fix(responses): keep bridged MCP auto-execute in one session and trace and continue its conversation
The bridged follow-up no longer chains on round 1, so a thinking model keeps its reasoning and the request stays in one session. The tool call log continues the request trace without renaming it, the tools are fetched once, and a later turn finds the MCP turn under the id the client got, skipping the tool rows.
* fix(responses): log the non-stream MCP tool listing in the request's session and trace
* fix(responses): find a bridged MCP turn by the id the client holds and log its listing in the bridged trace
* fix(responses): derive the MCP tool row id from the request's call id
The tool call's spend row now uses <call id>_<tool call id>, so it stays
linked to the request without colliding with the request's failure row.
Also tighten the internal cache-hit test to a fixed clock and typed params.
* fix(responses): bill a retried MCP attempt for its own rounds
A router retry or fallback reuses the request's logging object, so a failed attempt's stashed round 1 usage made the next attempt's success row log stale tokens. Each non-stream MCP attempt now starts without that stash.
* fix(responses): bill every failed MCP attempt's paid first round on the request failure
A retry or fallback now sets a failed attempt's stashed round 1 aside instead of dropping it, and the proxy's failure row adds it to the last attempt's own, so a request whose later attempt fails on its first call still bills the round the provider charged. A cached final round also keeps the paid first round's cost breakdown.
* fix(responses): keep MCP tool catalog and results out of the bridged next-turn replay
* fix(responses): keep MCP tool spend, client body keys, and Langfuse traces working on auto-execute
Count the MCP tools an auto-executed Responses API call ran in the tool spend rollup, accept a client body key named previous_response_client_ids without a 500, and point MCP listing and tool spans at the trace Langfuse gives the parent request, including x-litellm-session-id requests
* fix(responses): join MCP spans to the Langfuse trace when the request sends body metadata
* test(integration): cover non-stream MCP auto-execute billing, failure rows, and bridged follow-ups
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(cost): add tier 1 cost tracking cases for gpt-5.6-terra, gemini-4-argon and gemini-3.1-pro-preview
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add audit 3 shape cases for vertex_ai and azure_ai claude
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): mark azure gpt-5.6-terra priority case as known bug
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep reasoning, thinking blocks and the json_mode finish_reason on fake streams
A Bedrock Converse model without native structured output fakes the stream
when response_format carries a json_schema. The fake stream transformed the
response without json_mode and rebuilt a chunk from text, tool_use and token
counts only, so it dropped reasoning_content, the signed thinking_blocks and
the reasoning token count, and ended on finish_reason tool_calls.
The three fake-stream sites now pass json_mode to _transform_response and
stream the resulting ModelResponse through the shared
convert_model_response_to_streaming converter, which now numbers each
streamed thinking block with its index the way real streams do
* fix(bedrock): parse text tool calls on Converse fake streams like the non-stream answer
* fix(responses): stream the answer text when one chat chunk carries reasoning and the answer
* fix(streaming): keep every field on the OpenAI and openai_like fake streams and split multi-block thinking
Azure o-series and openai_like fake streams now use the shared MockResponseIterator, so extra tool calls,
reasoning and cached and reasoning tokens survive; the Bedrock-local iterator is deleted.
The /v1/responses bridge also splits a chunk whose thinking block is signed or redacted with no reasoning
text, and the sync iterator closes the reasoning item before the answer opens, like the async one.
The /v1/messages bridge gives each thinking block in a chunk its own content block, starts a new block when
the thinking block index changes, and streams redacted thinking as redacted_thinking.
Snowflake replays drop the stream index from thinking blocks.
* fix(anthropic): stream each thinking block's text once on /v1/messages
An unsigned thinking chunk now opens its content block with empty text, since its thinking_delta carries the text. A multi-block chunk keeps each piece's thinking block, so an unsigned block followed by a signed one keeps its own text and index.
* docs(anthropic): describe which thinking block start carries its text
* fix(streaming): close merged reasoning before the answer when one chunk carries both
* test(streaming): cover fake-stream reasoning across the proxy bridges and type the Snowflake replay test
