A Responses API stream that fails partway now bills the usage it already consumed, ends with an error instead of a clean stop, and honors stream_timeout between chunks. Bridged /v1/messages calls report those failures to the proxy's failure hooks, so spend, budgets and logs see them
* fix(aiohttp_openai): forward service_tier in transform_response
* chore(aiohttp_openai): mark service_tier assignment rebind-ok for the type gate
* fix(aiohttp_openai): set service_tier with setattr like other providers
ModelResponse has no declared service_tier field, so attribute assignment
added a reportAttributeAccessIssue error to the type gate.
* test(aiohttp_openai): verify serialized served service tier
* fix(aiohttp_openai): copy response when preserving served tier
---------
Co-authored-by: tin-berri <tin@berri.ai>
* feat(providers): add ZeroGPU as an OpenAI-compatible provider
Registers zerogpu through the JSON provider registry so zerogpu/<model> routes to
https://api.zerogpu.ai/v1/chat/completions with ZEROGPU_API_KEY and an optional ZEROGPU_API_BASE.
Adds LlmProviders.ZEROGPU, the constants.py provider and endpoint lists, cost map entries for the
10 ZeroGPU text generation models, the endpoint matrix, the Add Model form fields, the README row
and the dashboard provider entry
* feat(zerogpu): add capability flags verified against the live API
Live calls to https://api.zerogpu.ai/v1/chat/completions confirmed tools, tool_choice and json_schema
response_format on 8 of the 10 catalog text models, cached token reporting on deepseek-v4.1-flash,
gpt-4.1-mini, gpt-5.4-nano and gpt-5.6-luna, and reasoning_effort on gpt-5.6-luna. LFM2.5-1.2B-Instruct
answers stream requests with plain JSON instead of SSE and LFM2.5-1.2B-Thinking returns 500 on every
request, so both stay out of the cost map until the API serves them correctly
* feat(zerogpu): price LFM2.5-1.2B-Instruct
Adds zerogpu/LFM2.5-1.2B-Instruct to the cost map with the catalog prices and context window. It
gets no capability flags because the live API ignores tools, response_format and
max_completion_tokens for this model, so LiteLLM drops tool params for it instead of sending them.
LFM2.5-1.2B-Thinking stays out of the cost map
* fix(zerogpu): reject transcription requests before calling the upstream API
ZeroGPU only serves chat completions, so drop it from the OpenAI transcription transport set. A zerogpu transcription request now fails locally instead of hitting an /audio/transcriptions endpoint that does not exist
* test(zerogpu): cover the responses and messages bridges
Both endpoints reach ZeroGPU through chat completions, so assert the request each one sends upstream and the reply it returns
* refactor(zerogpu): move zerogpu out of the provider list tails
New upstream providers append to the end of openai_compatible_endpoints, openai_compatible_providers and providers.json, so an entry at the tail conflicts with every one of them. Placing zerogpu next to long-stable entries keeps those merges clean. No behavior change
* fix(zerogpu): price cached input for qwen3-30b-a3b-fp8
The ZeroGPU model catalog (https://docs.zerogpu.ai/docs/model-catalog, checked 2026-10-09) lists cached input for qwen3-30b-a3b-fp8 at $0.05 per 1M tokens. Without cache_read_input_token_cost, cached tokens were billed at the full $0.10 input rate
* revert(zerogpu): leave the audio transcription provider set unchanged
Per review, keep OPENAI_AUDIO_TRANSCRIPTION_PROVIDERS as upstream defines it. This PR only covers chat completions, and ZeroGPU now serves /audio/transcriptions anyway, so the local refusal and its test are removed
* fix(zerogpu): add max_tokens so trim_messages can size ZeroGPU history
trim_messages reads max_tokens as the fallback for max_input_tokens and evaluates it eagerly, so without it the lookup raised, got swallowed, and the full history came back untrimmed. max_tokens matches each model's maxTokens in ZeroGPU's models API (https://api-dashboard.zerogpu.ai/api/models, checked 2026-10-10)
* test(zerogpu): record mocked requests on a recorder object
The mock handler appended to a list the caller passed in, which AGENTS.md
forbids. A recorder now owns the captured requests as a tuple
* fix(zerogpu): cap max_tokens at each model's output limit
Each cap is the max_output_tokens ZeroGPU stores for the model in its production models table (read 2026-10-20), the same column its /v1/models serves as max_length. That gives 65536 for deepseek-v4.1-flash, 32768 for gpt-4.1-mini and gpt-oss-120b, 128000 for gpt-5.4-nano and gpt-5.6-luna, 8192 for LFM2.5-1.2B-Instruct and llama-3.1-8b-instruct-fast, and 16384 for qwen3-30b-a3b-fp8. glm-5.3-flash is stored as 32786, which reads as a typo for 32768, so that row uses 32768
trim_messages sizes history from max_input_tokens, so lowering max_tokens leaves it unaffected
* fix(zerogpu): flag prompt caching on glm-5.3-flash
Repeating the same ~4.3K-token prompt against ZeroGPU on 2026-10-20 returned cached_tokens of 4320 on some glm-5.3-flash calls. gpt-oss-120b and llama-3.1-8b-instruct-fast returned 0 on all nine calls, and qwen3-30b-a3b-fp8 never returned cached_tokens, so they stay unflagged. Their cached prices stay as ZeroGPU publishes them, so any cache hit is still billed at the right rate
* fix(zerogpu): set gpt-5.4-nano context window to 272000
ZeroGPU accepts a 271001-token gpt-5.4-nano prompt and rejects 273000 tokens as over the context window (checked 2026-10-21), so max_input_tokens is 272000 rather than the 400000 its catalog lists
* fix(zerogpu): list zerogpu models for zerogpu/* wildcards
zerogpu had no models_by_provider entry, so get_provider_models("zerogpu") returned None and a zerogpu/* wildcard listed no models. zerogpu_models is now filled from the cost map rows like openinfer_models and registered in models_by_provider
* fix(zerogpu): flag gpt-5.4-nano as a reasoning model
ZeroGPU honors reasoning_effort on gpt-5.4-nano (checked 2026-10-21). Without it the model spends 0 reasoning tokens, and with low or high it spends 32 or more. LiteLLM only treated it as a reasoning model by falling back to OpenAI's bare gpt-5.4-nano row, so get_model_info and /model/info reported supports_reasoning as null. The row now carries the flag itself
* fix(zerogpu): flag deepseek, glm, gpt-oss and qwen3 as reasoning models
ZeroGPU's /v1/models marks these four as reasoning models and accepts reasoning_effort on all of them (checked 2026-10-21), but LiteLLM rejected the param with UnsupportedParamsError because their rows had no supports_reasoning flag. glm reports reasoning_tokens under reasoning_effort, and gpt-oss and qwen3 spend hidden reasoning tokens on every reply. The gpt-oss test case now expects reasoning_effort to reach ZeroGPU
* fix(zerogpu): set gpt-oss-120b context window to 128000
ZeroGPU routes gpt-oss-120b across several upstream backends, and at least one of them returns 413 past a 128000-token window, so max_input_tokens uses that as the safe floor. The backends reached on 2026-10-21 accepted up to about 131K prompt tokens, so this only gives up a little headroom on the larger ones
* feat(zerogpu): call ZeroGPU's Responses API natively
ZeroGPU serves /v1/responses for all nine catalog models, so LiteLLM no longer bridges Responses requests through chat completions. Native calls return ZeroGPU's reasoning output items for glm, gpt-oss and qwen3, which the bridge dropped. Checked live on 2026-10-22 through litellm.responses for every model, non-streaming and streaming, plus a gpt-oss tool call. LFM2.5-1.2B-Instruct streams correctly now that ZeroGPU's text_generation lane emits Responses SSE
* fix(zerogpu): stop inferring the zerogpu provider from api_base alone
Listing https://api.zerogpu.ai/v1 in openai_compatible_endpoints rerouted bare model names on that api_base from openai to zerogpu. A bare gpt-5.6-luna call with verbosity used to return 200 and started raising UnsupportedParamsError. Drop the entry and its api_base-alone test, so zerogpu is only selected through the zerogpu/ prefix
* fix(zerogpu): price gpt-6-luna, llama-3.2-3b, LFM2.5-1.2B-Thinking and qwen3.6
A zerogpu/* wildcard served these four chat models with no cost-map row, so their calls logged $0 spend. Add rows at ZeroGPU's catalog prices from https://api-dashboard.zerogpu.ai/api/models, with context windows and output caps from ZeroGPU's models table. LFM2.5-1.2B-Thinking has no cached price, like LFM2.5-1.2B-Instruct. gpt-6-luna is flagged supports_reasoning because LiteLLM already routes gpt-6 names as reasoning models. Checked live on 2026-10-28, all four now report nonzero response_cost
* fix(zerogpu): drop the retired max_tokens field from the zerogpu rows
Upstream #48471 retired max_tokens from the cost map in favor of max_input_tokens and max_output_tokens, and test_no_row_carries_the_retired_max_tokens_field failed on all 13 zerogpu rows. Every row already carries the same value in max_output_tokens, so nothing is lost. The zerogpu limits test now checks that each row's output limit fits its context window
* chore(zerogpu): regenerate provider endpoint support for zerogpu
Upstream #48619 generates provider_endpoints_support.json from the cost map and checks it in CI, and the hand-written zerogpu entry no longer matched. Regenerated with scripts/generate_provider_endpoints_support.py. Only the zerogpu entry changed, and it now matches what the generator emits for other JSON-registry providers such as openinfer
* fix(zerogpu): flag tools, json_schema and prompt caching on gpt-6-luna
ZeroGPU serves function calling, tool_choice, json_schema response_format and prompt caching on gpt-6-luna, so set supports_function_calling, supports_tool_choice, supports_response_schema and supports_prompt_caching on its row in both cost maps. Checked live on 2026-10-28, a weather tool call came back with the right arguments, json_schema output validated, and repeated prompts reported 3260 of 3263 tokens cached
* fix(zerogpu): set qwen3.6-35b-a3b context to 262144 and flag its tools and reasoning
ZeroGPU accepts up to 262,144 input tokens on qwen3.6-35b-a3b and serves tools, tool_choice, json_schema response_format and reasoning_effort on it. Raise max_input_tokens from 16384 and set supports_function_calling, supports_tool_choice, supports_response_schema and supports_reasoning in both cost maps. Before this, LiteLLM rejected tools and reasoning_effort for the model. Checked live on 2026-10-28, a 209,905-token prompt succeeded, an oversized one was refused with a 262144-token limit, and tool calls, json_schema output and reasoning_effort all worked
* fix(zerogpu): flag vision on gpt-6-luna and qwen3.6-35b-a3b
ZeroGPU reads image inputs on gpt-6-luna and qwen3.6-35b-a3b, so set supports_vision on both rows in both cost maps. Checked live on 2026-10-28, both models named the animal in an image_url photo, and the image showed up in their prompt token counts
* test(azure): audit #47348
Add proxy integration tests for #47348's cached Entra token providers and
off-loop token refresh, and fix the token herd the audit found: once the
fetch moved to a worker thread, concurrent cold requests each fetched their
own token. Every cached bearer token provider now serializes its calls so
the first fetch fills the cache and the rest reuse it
* fix(azure): build each cached Entra token provider once under concurrent first calls
lru_cache does not hold back concurrent misses, so worker threads that miss together each built their own credential and token provider, and each fetched its own token. A lock around each provider cache makes the first caller build it and the rest reuse it
* fix(azure): share Entra token fetches only while no valid token is held
Concurrent callers now share one result (or one error) from the cold
fetch, while callers inside the refresh window keep the still-valid
token. The oidc/azure provider build and the client-assertion exchange
are single-flighted per audience, and the shared helpers no longer call
subscripted generic aliases, which crashed on Python 3.10.
* fix(azure): refresh Entra tokens once while other requests keep the valid token
MSAL treats a token with less than five minutes left as expired, so every
request in that window fetched a new one. Now one request refreshes while
the others keep using the still-valid token.
Requests that waited on a fetch that failed retry once through a fresh
fetch instead of all failing with the first error.
Credentials that only implement get_token keep working, and OIDC token
exchanges are cached per scope.
* fix(azure): bound Entra token waits during an outage
Requests waiting on a failed token fetch now retry only when the fetch
before it succeeded, so during an Entra outage later bursts fail after one
fetch instead of two.
Requests during a refresh only get the held token while it has more than
60 seconds left, and a failed refresh falls back to that token instead of
failing its request.
* fix(azure): share one retry per failed Entra token fetch and raise on a failed refresh
Waiters of a failed fetch now retry the flight they joined, so a fast-failing Entra makes one retry instead of one per waiter. A failed refresh raises again instead of returning the held token, which Postgres token auth read as a fresh one. The DefaultAzureCredential fallback reuses one provider per scope.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): audit #47363
Adds integration cells for the input token count paths #47363 changed: the provider id of a managed file and the 403 for another team's file on OpenAI, file_search and additional_tools parity plus cache-mark stripping and untranslatable tools on Claude, and the Bedrock runtime count leaving out the options it refuses.
* fix(count-tokens): reject a lone tool with no chat equivalent before the Claude count call
A Claude input token count whose only tool was mcp or computer_use_preview skipped the pre-call check and went to the provider, so the caller got the provider error instead of the 400 naming the tool. The check now runs whenever a tool would get a placeholder name.
* fix(responses): count a lone Anthropic toolset and name the uncountable tool
The lone-tool 400 now fires only for OpenAI Responses built-in tool types, so a nameless Anthropic toolset still counts. The error says LiteLLM cannot count that tool type for the model. Adds a Bedrock unit cell for the lone-tool 400 and pins the runtime wire test's tool shape per request type.
* fix(responses): take the uncountable tool types from the OpenAI SDK and keep the region hint off LiteLLM's 400
The Responses built-in tool types now come from openai's ToolParam union, so a type the SDK adds is refused before the count call too. The AWS region hint no longer lands on an error LiteLLM raised before calling the provider. Tests cover two more lone types and the hint.
* fix(responses): keep LiteLLM's own list of uncountable tool types and pass a non-string tool type through
* fix(responses): count tool_search and namespace tools the way Claude chat inference sends them
The count refused a lone tool_search with LiteLLM's 400 and sent a lone
namespace raw or squashed it into an empty tool, while Claude chat
inference maps tool_search to Anthropic's tool search tool and unpacks
a namespace into its function tools. Both now go through the
deployment's own map_openai_params, so the count sends what inference
sends, and tool_search leaves the list of uncountable types.
* fix(bedrock): count a Claude request with its deployment's own chat translation
Bedrock invoke and Mantle inference turn a chat tool_search tool into Bedrock's
tool search tool, but the Bedrock counter always translated with the Anthropic
config, so the count answered 400 for a request inference serves.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mistral): replay assistant reasoning as a Mistral thinking chunk
When a client sends back an earlier Mistral assistant turn, its reasoning
(reasoning_content, thinking_blocks, or Mistral's own thinking chunk in
content) now reaches Mistral as a ThinkChunk inside content. The top-level
LiteLLM fields stay stripped because Mistral rejects them with 422.
* fix(mistral): replay reasoning only to Mistral models that accept it
Mistral answers 400 to a thinking chunk on models without reasoning support,
so a mid-chat model switch, a fallback, or Vertex AI Mistral broke once the
replay was added. Gate the replay on the same check reasoning_effort uses and
keep dropping the reasoning for the other models, as before.
* fix(mistral): drop native thinking chunks for models without reasoning and mark the devstral, code agent, small and leanstral aliases as reasoning
* fix(mistral): keep the leanstral rows as on main, since their reasoning support cannot be checked live
* fix(mistral): send only the text items of a native thinking chunk, so a reference or image item no longer gets Mistral's 422
* fix(mistral): keep closed false only on the last chunk of a final prefix turn, so an earlier turn no longer gets Mistral's 400
* fix(mistral): drop closed false when merged text follows the thinking chunk, so a prefix turn with text first no longer gets Mistral's 400
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(chatgpt): derive Responses session id from prompt_cache_key
ChatGPT partitions the Codex prompt cache by the Responses session-id
header (the Codex CLI sends a stable per-conversation key derived from
prompt_cache_key). The provider only consulted explicit session ids
and otherwise stamped a per-request uuid4, so no turn ever reused the
prompt cache -- 0% cache hits on repeat turns, an order-of-magnitude
quota/cost multiplier on subscription accounts.
Resolution order after this change:
explicit session config (session_id / litellm_session_id /
metadata.session_id) > prompt_cache_key (sanitized with
_safe_header_value) > litellm-internal ids > uuid4.
Two guards, mirroring get_fireworks_session_id:
- proxy-generated session ids (missing_session_id: "generate") are
per-request and are skipped, checked in both "metadata" and
"litellm_metadata" (LITELLM_METADATA_ROUTES, responses included)
- internal per-request ids (litellm_trace_id / litellm_call_id) lose
to a stable conversation key
Supersedes the body-key forwarding of #37630: a random session header
overrides the body key on the provider side, so the header must carry
the caller's key. Related: #29978, #29993.
Verified against a real ChatGPT Codex account: stable session id ->
97-100% cached on repeat turns; random per-request -> 0%.
* style: ruff format
* fix(chatgpt): address review - hash unsafe cache keys, move tests, drop comments
- hash prompt_cache_keys containing header-unsafe chars instead of
mapping them through _safe_header_value, whose replacement char could
collapse distinct keys onto one cache shard and serialize unrelated
requests (greptile review)
- strip the explanatory comment blocks per AGENTS.md comment policy
- move the tests into the existing mapped responses transformation test
file per AGENTS.md test-placement rule
* fix(chatgpt): keep session typing and test diff scoped
* fix(chatgpt): hash surrogate cache keys, type session params, flatten tests
* test(chatgpt): record upstream requests with a mock instead of appending to a list
* fix(chatgpt): keep proxy-mapped session headers ahead of body session_id and ignore non-string cache keys
* fix(chatgpt): hash cache keys with leading or trailing whitespace so the session header stays valid
* test(chatgpt): type the session id test parameters
* fix(chatgpt): keep trace ids copied into litellm_session_id from outranking prompt_cache_key
* fix(chatgpt): mark session ids copied from trace_id so explicit ones keep precedence
* fix(chatgpt): keep the trace-id session marker out of provider params
The marker now uses the _litellm_ internal prefix, so bridges that pass litellm_params back into completion() drop it instead of sending it as a provider param
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(huggingface): cover TEI rerank truncate passthrough and server default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(huggingface): stop forcing truncate=false on TEI rerank and pass truncate params through
Co-authored-by: Luke Schaefer <225965040+luke-e-schaefer@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(huggingface): keep TEI truncate options provider-specific
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(huggingface): use MockTransport for rerank truncation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Luke Schaefer <225965040+luke-e-schaefer@users.noreply.github.com>
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* test(streaming): cover reasoning text and error frames that mention [DONE]
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): avoid assistant prefill in Responses fallbacks
* fix(router): keep prefill for Claude fallbacks that take it and skip constrained continuations
A Responses fallback to a Claude model that takes assistant prefill now gets
the partial response as prefill again, so it continues mid-word without the
prompt-shape drift. Every other fallback keeps the request to continue.
A stream cut after text no longer continues when the request constrains its
output (a json_schema or json_object text format, or a forced tool_choice),
since a continued JSON document or tool call would not parse. The client gets
the primary's error, as on the chat path.
The fallback's re-sent leading space at the seam is dropped, and the prefill
shaping runs per deployment after the encrypted reasoning strip, so stripped
reasoning is never replayed.
* fix(router): guard SDK text_format, narrow forced tool_choice, read Claude base_model, and cap Responses continuation output
Cut Responses streams that asked for structured output through the SDK's text_format now surface the primary's
error instead of stitching two models' JSON. Only a required or named tool_choice counts as forced, so allowed_tools
in auto mode and hosted tool choices still continue. A Claude deployment named only through base_model keeps its
prefill continuation, and the continuation's max_output_tokens is lowered by what the cut stream already emitted.
* fix(router): quote the partial's ending in the continue request and surface the primary's error after a cut tool call
* fix(router): stop Responses fallbacks from continuing content-filter cuts and client-run tool calls
A content filter that cuts a /v1/responses stream mid-answer now returns its error, as on chat, instead of letting a content_policy_fallbacks backup finish the filtered text. Shell, apply_patch, computer, local_shell, and MCP approval items count as a started tool call. Chat continuation goes back to treating any tool_choice object as constraining, as on main
* fix(responses): count a stream of tool arguments as a started tool call
* fix(router): let Responses fallbacks continue under allowed_tools auto and after server-run tool calls
A Responses tool_choice of allowed_tools in auto mode no longer blocks the continuation, since the model may still answer in text. Hosted MCP calls and the bridge's server-run tool search no longer count as a started client tool call.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): let drop_params drop speed on claude-opus-4-7 after Anthropic removed its fast mode
* test(anthropic): cite the Claude Code fast-mode page for the Opus 4.7 removal date
* test(anthropic): cover the drop_params speed drop for claude-opus-4-7
* test(anthropic): type the model parameter of the speed drop test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params
* test(bedrock): cover the strict tools 400 retry under drop_params
* fix(bedrock): anchor the strict tools rejection match and strip toolSpec strict
* test(bedrock): return captured requests from the strict tools test helper
* fix(bedrock): retry without strict only when a tool sent strict to Bedrock
A strict-limit 400 on a request whose tools carry strict: false, or on a model litellm never sends strict to, was sent again with an identical body. The fallback now takes the model and returns nothing unless a tool puts strict on the wire. Also types the fallback test parameter and records the mock transport's requests on the recorder instead of a mutated list
* fix(bedrock): decide the strict retry on the profile's base model and retry grammar timeouts
An application inference profile with base_model sends strict on the wire, so the retry check now resolves the
capability model the way the request transformation does. Bedrock's "Grammar compilation timed out" 400 is a
strict-grammar rejection too, so it takes the same single retry under drop_params.
* test(bedrock): annotate the strict-tools retry test helper's values as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #46815
Add integration tests for every count route #46815 changed, and count a Claude
subscription caller's tokens with their forwarded token when the deployment has
no configured key, which #46815 turned into a 401
* test(proxy): accept the forwarded subscription token in the count endpoint mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add aiand provider
* feat: expose aiand provider in admin UI
* feat(aiand): add ai& model pricing to the cost map
* feat(aiand): list ai& models for aiand/* wildcards and mark /v1/messages supported
* feat(aiand): send Anthropic /v1/messages to ai& natively
* fix(aiand): drop the retired max_tokens field and regenerate endpoint support
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Fixes#46977: Cohere chat API reference expects stop_sequences to be a list of strings. Convert string stop parameter to a single-element list for CohereChatConfig and CohereV2ChatConfig while preserving lists.
Co-authored-by: Praising Harris Ratnam <praisingharris@gmail.com>
* fix(tavily): return raw_content, score, favicon and the response envelope from Tavily search
Tavily search dropped every field except title, url and content, so
include_raw_content, include_answer, include_images and include_favicon
reached Tavily but their output never reached the caller. Each result now
carries raw_content, score and favicon when Tavily returned them, and the
top-level fields (answer, images, request_id, usage, ...) come back as
provider_metadata, same as the Exa fix in #47423
* test(tavily): move the Tavily search wire test's sample data to module constants, like the Exa test
* feat(morph): support open-weight chat models and cached input pricing
* chore(morph): preserve existing catalog number formatting
* test(morph): include provider regressions in CI shard
* fix(morph): satisfy catalog and test package checks
* fix(morph): declare abliterated model output limit
* fix(chatgpt): send system input items to Codex as developer
The Codex backend rejects {"type": "message", "role": "system"} input
items with 400 "System messages are not allowed", while it accepts the
same item with role=developer. The chat completions bridge emits that
shape for every non-leading system message, and typed Responses API
clients send it directly, so both paths failed. Rewrite system items to
developer in place, which keeps their position and content.
Fixes#48562
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(chatgpt): annotate system-item rewrite with read-only types
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(chatgpt): rewrite system items on the typed input before the base transform
Keeps the rewrite on known types so basedpyright sees no new unknowns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
An OpenAI or Azure SDK client litellm built and evicted from the client
cache is usually garbage collected inside the closer's 900 second grace
window, so the closer's weak reference dies and the client's aiohttp
session is never closed. LiteLLMAiohttpTransport.__del__ now hands an
owned, still-open ClientSession to _close_recycled_session, which already
covers the current-loop, foreign-loop and closed-loop cases.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: BangerSoul <bangersoul@users.noreply.github.com>
* fix(proxy): build proxy-only failure logs from an allowlist and ignore request-time GCS credentials
* fix(proxy): drop caller metadata from proxy-only failure logs
metadata and litellm_metadata leave the proxy-only failure-log allowlist, so a rejected request can no longer point its failure log at its own Langsmith or Langfuse host through nested metadata.
* fix(proxy): keep allowlisted metadata in proxy-only failure logs
* test(proxy): isolate proxy-only failure log metadata tests
* fix(logging): ignore non-list request tags when building the logging payload
A guardrail-rejected request with metadata tags set to a non-list value crashed
the standard logging payload, so its spend row lost model attribution.
* fix(proxy): keep router attribution and drop unauthenticated tags in proxy-only failure logs
A request rejected before routing now logs the configured deployment and model group even when its metadata or litellm_metadata carries its own values. Requests rejected at auth no longer write caller tags into failure logs and tag spend.
* fix(proxy): always strip client guardrail info and mirror failure attribution into litellm_metadata
allow_client_pricing_override no longer keeps a caller-supplied standard_logging_guardrail_information on the chat or pass-through paths, since its guardrail_cost is billed. Proxy-only failure logs now also carry key and team attribution in litellm_metadata.
* refactor(proxy): keep the proxy-only failure-log fixes inside the type gates
Types the picked helpers so basedpyright and the type-discipline gate stay at main's counts. No runtime change.
* fix(proxy): keep the WS frame guardrail-info strip and string interaction tags as sent
The guardrail-info key left the pricing strip list, so the Responses WS frame guard now drops it by name. get_request_tags no longer raises on string tags, so background interaction polling checks for them before keeping them as sent.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bill native Bedrock guardrail trace usage on chat, responses and pass-through
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): record the Bedrock guardrail trace of a streamed metadata chunk on the logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): apply guardrailConfig on /v1/messages and price its trace, record the stream trace on both builder paths, redact the native guardrail_response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): validate the guardrail trace with one pydantic model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): record the native guardrail entry when litellm_params metadata is None
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): import the renamed messages invoke transformation module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): skip native guardrail billing on cache hits and redact the trace with message logging off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): walk the guardrail trace iteratively when redacting it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): record the native guardrail trace where the Bedrock response is parsed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type the native guardrail trace fixtures and locals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): drop the experimental label from the MCP server and client paths
Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims
pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages
Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): keep Final variables out of loops in the rename shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): match the warning text in the rename shim test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): mark the alias loader spec restore for the type discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): format the renamed package shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): keep the rebind-ok marker on the spec restore line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): carry the new package search path on alias specs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): declare the shim test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure),
the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py,
anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py,
and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol
names are unchanged.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(refuseless): add provider and model pricing
* feat(refuseless): expose provider in Add Model catalog
* test(refuseless): verify model input trimming after upstream fix
* fix(refuseless): exclude unavailable free model from catalogs
* feat(abliteration): add model pricing and capabilities
* feat(abliteration): expose provider in Add Model catalog
* chore(abliteration): keep provider entries independent of other additions
* fix(abliteration): correct reasoning capabilities and verify input trimming
* refactor(cost-map): drop the legacy max_tokens field in favor of max_input_tokens and max_output_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop assertions that pinned the retired cost-map max_tokens field
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): name the timeout phase and the limit that fired instead of 'after None seconds'
Every HTTP verb on both handlers raised litellm.Timeout with the per-call
timeout argument interpolated into the message, which is None whenever the
caller relied on the client default, and always said "Connection" even for a
read timeout. One helper now reads the limit httpx actually enforced from the
failed request and names the phase that fired
Fixes#14635
* fix(rust_bridge): name the timeout phase in the vault timeout message to match the httpx handler
* fix(http_handler): drop the unreachable timeout fallbacks httpx already rules out
* refactor(rust_bridge): drop a doc comment that restated the timeout helper
