Resolve litellm/types/google_genai/main.py and litellm/types/utils.py by
keeping this branch's modernized annotations on top of staging's removal
of inert type: ignore comments. Rebuild ruff-strict-budget.json from
measured merged-tree counts where de-excluding litellm/types adds
violations, keeping the stricter of the two sides' limits everywhere
else so no rule gains headroom. Fix the four type-discipline additions
the merge surfaced: freeze GEMINI_1_5_ACCEPTED_FILE_TYPES, drop a
callback_args parameter rebind in guardrails, and give the two remaining
mutations reasoned suppressions
ruff.toml has excluded litellm/types/* since 2024, so no lint rule ever ran
on the types tree. Remove the exclusion, apply ruff --fix and ruff format
across litellm/types, and hand-fix what autofix cannot reach so the
pyupgrade budgets stay at zero: implicit type aliases converted to PEP 604
unions, RootModel[Union[...]] bases, duplicate imports, and a stray print.
Load-bearing import X as X re-exports deleted by preview-mode F401 are
restored, and the six star-imported hub modules keep their re-export
surface via per-file F401 ignores. Star-import consumers that silently
relied on typing names leaking from those hubs are modernized to builtin
generics and PEP 604 unions.
Runtime annotation introspection that only recognized typing.Union is
taught types.UnionType (guardrail UI field schemas, volcengine response
fill), with regression tests for both. Strict budget limits for the rules
the types tree now trips are raised to exact measured totals, so any
net-new violation still fails the gate
The re-landed closer test asserted a reaped handler's client stays
closed; with #35862 the handler heals on next access, so the test now
pins the inner client up front and asserts the heal as the contract.
Also adds an end-to-end regression test that evicts an init-held
handler through LLMClientCache, waits out the grace close, and proves
the next request succeeds.
Since #35492 the evicted-client closer really closes litellm-owned
httpx clients once their cache entry is evicted and the grace window
passes. Objects that fetch get_async_httpx_client once in __init__ and
hold the handler forever (40 guardrail classes, pagerduty and email
callbacks, and more) then fail every request with 'RuntimeError: Cannot
send a request, as the client has been closed.'
AsyncHTTPHandler.client and HTTPHandler.client are now properties that
rebuild the inner client from the constructor's stored config when the
handler owns it and finds it closed. Caller-supplied or assigned
clients are never rebuilt, and close paths use the backing field so
closing a handler does not resurrect it.
This reverts commit 66bc70365f and the
follow-up 2-line type fix a6d4654261 (#35706), which only retyped a
signature #35492 introduced.
Closing evicted litellm-owned clients breaks every object that fetches
get_async_httpx_client once in __init__ and holds the handler for the
life of the process: 40 guardrail classes plus the pagerduty and email
callbacks. Once the cache entry is evicted (TTL 3600s or the 200-entry
size cap) and the 900s grace passes, the held client is closed and every
subsequent request through it fails with RuntimeError: Cannot send a
request, as the client has been closed. On a production deployment with
a default-on guardrail this surfaced as every request 500ing roughly 75
minutes after boot.
The connection-reclaim goal of #35492 can re-land once handlers survive
their inner client being closed.
The azure_storage logging callback and the azure blob files backend built every
storage URL against the hardcoded commercial host, so an Azure Government account
was unreachable with no way to override it.
Read AZURE_STORAGE_ENDPOINT_SUFFIX (default core.windows.net) once in
AzureBlobStorageLogger and derive the Data Lake and Blob hosts from it, so all
seven previously hardcoded sites follow the configured cloud. Parse stored blob
URLs with urlparse instead of matching the commercial host, so URLs persisted
before the suffix was configured still resolve, and pin the resulting
host-validation boundary with tests.
Converse rejects a request that carries both toolConfig.toolChoice and an
additionalModelRequestFields.tool_choice.type, so any request that pairs
parallel_tool_calls with an explicit tool_choice 400s with "The additional field
tool_choice/type conflicts with the existing field toolConfig.toolChoice.auto".
That pairing is what agentic clients send by default; Codex CLI sends
tool_choice "auto" and parallel_tool_calls false on every turn, so tool calling
was broken outright on Bedrock models that advertise
supports_parallel_tool_use_config.
Drop the type from the Anthropic passthrough once toolChoice carries it, and keep
disable_parallel_tool_use, which has no toolConfig equivalent and is accepted
alongside toolChoice. Measured against Bedrock directly: toolChoice plus
{disable_parallel_tool_use} succeeds for auto, any and tool, while an empty
tool_choice with no toolChoice is rejected for a missing type, so the type still
has to be emitted when the caller sends no tool_choice.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An evicted client was left for the garbage collector, but every OpenAI/Azure
SDK client is a reference cycle, so nothing freed the client or its pooled TCP
connections until a generational sweep ran. Driving 2000 azure calls through
the official image with no forced collection, live clients and open sockets
climbed from 202 to 1361 while the cache stayed at its 200-entry bound, and RSS
grew 279 MB to 456 MB against a TLS upstream.
Closing on eviction is what caused the earlier 'Cannot send a request, as the
client has been closed' regression, so an evicted client litellm created is now
closed only once a grace window has passed, by which point any request that was
already holding it has finished. A client the caller supplied is never closed,
since litellm does not own its lifecycle.
Resolves LIT-4883
The explicit AssumeRole branch of BaseAWSLLM.get_credentials returned without
touching the process-wide IAM cache, so every model request issued a fresh
sts:AssumeRole, and on ECS/EC2 an uncached sts:GetCallerIdentity ahead of it.
Route the whole role branch through _get_or_set_cached_credentials with the TTL
_auth_with_aws_role already computed and discarded. The cache key is the same
aws_* argument snapshot the other flows use, taken before the session-name
default is filled in, so each aws_session_name keeps its own STS session and no
attributed identity can be served another's credentials.
Credential fetches now single-flight behind striped locks. Without that, a burst
of concurrent misses on one key each issued their own STS call, which is the
same thundering herd the cache exists to prevent, moved to the miss window.
* fix(aiohttp): dispose recycled client sessions deterministically
LiteLLMAiohttpTransport replaced its cached aiohttp.ClientSession on
loop-mismatch, loop-inspection failure, and "Session is closed" retry
without reliably closing the previous session:
- the close task from asyncio.create_task() was never referenced, so
it could be garbage-collected before running;
- the (RuntimeError, AttributeError) fallback branch replaced the
session without closing it at all;
- sessions bound to a closed event loop were abandoned to the GC
("rely on GC"), and sessions bound to a loop running in another
thread were closed from the wrong loop.
Replaced sessions surfaced as intermittent "Unclosed client session" /
"Unclosed connector" errors from the event-loop exception handler at
GC time.
_close_recycled_session() now covers the three lifecycles a recycled
session can be in: same-loop closes keep a strong task reference until
completion; sessions owned by a loop running elsewhere are closed on
their own loop via run_coroutine_threadsafe; sessions whose loop is
gone are disposed synchronously through the connector teardown that
aiohttp's own finalizer uses, which releases pooled connections and
silences the finalizer warnings.
Fixes#24230
* fix(aiohttp): guard threadsafe close callback against cancelled futures
---------
Co-authored-by: Anmol Jaiswal <68013660+anmolg1997@users.noreply.github.com>
* fix(anthropic): split mixed reasoning stream chunks
* style: use builtin generic annotation
* fix(anthropic): split mixed stream chunks by payload kind
The mixed-chunk split cleared only the fields it knew about on each
deep-copied piece, so any other payload riding the chunk survived on
both pieces: tool_calls were emitted as two tool_use blocks with the
same id, thinking_blocks on the text piece emitted duplicated thinking
into a text block while dropping the answer text, and chunks whose
reasoning arrived only as thinking_blocks never split at all
Rebuild each piece's delta from scratch with exactly one payload kind
(reasoning, text, tool calls), ordered to match native Anthropic block
order. Fresh Delta construction keeps unset attributes deleted, which
matters because the translators branch on hasattr, and prevents future
Delta fields from riding along on every piece
* fix(anthropic): keep continuation and multi-choice chunks unsplit, emit signature-less thinking once
Adversarial verification against the merge-base found three shapes where
the payload-kind split changed behavior beyond its target: a mixed chunk
carrying a tool argument continuation was torn into a truncated block
plus a fabricated one, a multi-choice chunk lost its secondary choices'
payload, and a signature-less thinking_blocks piece inherited the
non-empty block start body so accumulators collected the thinking twice
Continuation and multi-choice chunks now pass through the splitter
untouched, matching the merge-base byte for byte, and signature-less
thinking_blocks pieces are normalized to reasoning_content so the block
start opens empty and the thinking text is emitted exactly once
---------
Co-authored-by: Napuh <naamanynadiemas@gmail.com>
AWS rolled out the 2026-07-30 GPT-5.6 price cut the same day, but the
bedrock_mantle entries still carried values derived from the pre-cut OpenAI
base, so Terra billed 1.25x and Luna 5x over the published rate.
Re-derive both from the AWS Bedrock pricing page, which prices in-region
inference at parity with OpenAI's data residency tier (1.1x base). Sol was
not cut and is unchanged.
Also drop tests/test_litellm/test_gpt_5_6_model_metadata.py; its Azure and
openai pricing assertions are covered by test_llm_cost_calc_utils.py.
Fireworks publishes a 262144-token context window for the Kimi K2.5, K2.6
and K2.7 models but caps generation well below that. Every fireworks_ai
Kimi K2.5/K2.6/K2.7 alias had max_output_tokens/max_tokens flattened to
262144 (equal to the context window), so the pre-call context-window check
admitted requests asking for a full 262144-token completion that Fireworks
rejects. Correct max_output_tokens/max_tokens to 32768 while keeping
max_input_tokens at 262144, and add a regression test pinning the limits
for all ten aliases.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Gemini returns each thoughtSignature on exactly one part. LiteLLM
stores a function-call signature both message-level (thought_signatures)
and on the tool call itself, then re-attached it to BOTH the text part
and the function-call part when serializing history. gemini-3 and newer
models bill every replayed copy as the previous turn's full reasoning
token count, so long agentic sessions doubled their context growth and
hit the 1,048,576-token limit
Only attach a message-level signature to the text part when the same
signature is not already carried by a tool-call part:
- compare signature values instead of boolean presence so a distinct
text-part signature is never dropped
- ignore the gemini-3 dummy-signature fallback during detection so
replaying gemini-2.5 history to a newer model keeps the real text
signature
- count signatures carried by server-side tool invocations so they are
not re-attached to the text part
gemini-2.5 responses (signature on the text part, function call
unsigned) are unaffected: the text signature is preserved as before
Exercises the streaming field-fill heuristics, model_construct fallbacks,
and the get/cancel/delete/list request and response transforms that had no
tests.
Cache web identity STS credentials in the shared IAM cache (restores the
pre-v1.85.0 behavior removed by #27125) and cap the Google OIDC token cache
TTL at the token's own exp claim minus a 60s margin, never caching an
already-expired token
* fix(router): don't cool down parent deployment on advisor sub-call failure
Advisor orchestration issues a sub-call to a different provider/credentials than the selected deployment. When that sub-call fails (e.g. a 401 because no advisor API key is configured), the exception propagates up and the router's deployment_callback_on_failure attributes it to the healthy parent deployment's model_info.id, cooling it down and rejecting unrelated callers to the same model group.
Tag advisor sub-call failures on the exception and skip cooldown for them in deployment_callback_on_failure. The exception is tagged rather than wrapped so its type is preserved and retry/fallback classification and the client-facing error are unchanged. Genuine executor/deployment failures are untagged and still cool down as before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): tag advisor orchestration failures via provider-neutral util
Address review on LIT-4565: move the cooldown-exemption marker into
litellm/router_utils/cooldown_handlers.py so the router imports it at
module top instead of an in-function anthropic import, and extend the
exemption to AdvisorMaxIterationsError so a max-iterations orchestration
failure no longer cools down the healthy executor deployment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Vertex AI now accepts and returns `id` on functionCall and functionResponse parts for Gemini 3+ on the v1 endpoint, so the provider check added in #28324 is stale. It silently drops the id for every Vertex caller, which breaks strict tool-call matching
Gate the id on model version alone, which is what the code did before #28324 and what Google AI Studio already does. `_forward_gemini_function_call_id` no longer takes `custom_llm_provider`, and the decision is resolved once in `_gemini_convert_messages_with_history` and passed to both converters as a bool rather than re-derived independently in each. The context caching path is covered by the same change, since it already passes `model` and the gate needs nothing else
The `id` comments on `FunctionCall`, `FunctionResponse` and `HttpxFunctionCall` were also written by #28324 and asserted the opposite of current behaviour, so they are corrected here
translate_thinking_for_model duplicated the same summary/auto_summary
wrapping logic as _translate_thinking_to_openai without the
disabled-thinking guard, so it could still wrap "none" into an
{effort, summary} dict when reasoning_auto_summary is enabled (caught
by Cursor Bugbot). Extract the wrapping rule into one shared
_apply_reasoning_summary_wrapping helper used by both call sites so
this invariant can't drift apart again.
Codecov flagged the empty-list early-return in
_translate_stop_sequences_to_openai as an uncovered line in the diff —
add a regression test asserting stop_sequences=[] does not set
new_kwargs["stop"].
Guard against reasoning_auto_summary wrapping "none" into a dict when
thinking is disabled — there's no reasoning trace to summarize, and
non-Claude providers (e.g. Fireworks) expect reasoning_effort as a
plain string.
