aiohttp shields its DNS resolution task; when the connector closes it cancels
that child, so the request task sees CancelledError without ever being
cancelled itself. map_aiohttp_exceptions() only caught Exception, so the
BaseException skipped transport mapping, router retries and proxy error
handling, and /v1/responses answered 500 "No response returned".
Catch CancelledError in the mapper, re-raise when the current task is really
being cancelled (Task.cancelling() > 0), and otherwise map it to
httpx.ConnectError so the usual retry, fallback and error mapping apply.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Extract the Soniox SRT/VTT cue grouping and rendering into a shared
litellm_core_utils/audio_utils/subtitle_utils module, have Gemini
transcription request word timestamps whenever response_format is srt or
vtt, and let the http handler rewrite the response text into the
synthesized subtitle document (dropping the internally requested words
array) for any provider config that opts in via
supports_subtitle_synthesis
The fixture was matched by host alone, so it answered any method and path and
the tests would have stayed green if the request went somewhere else. It now
matches POST on the Foundry route, and asserts the route was called.
Worth pinning on its own: the real path carries a /models prefix, which the
first attempt at this got wrong, so the match now also holds the routing in
place rather than only the retry.
The source drops the rejected field in place, so a payload shared across
tests could in principle be consumed by whichever case ran first. It does
not happen today, because the request is copied before the transform runs,
and the cases pass in reverse and async-first order alike. Building the
payload per call costs nothing and keeps that true if the copy ever goes.
Azure AI is the only provider that retries a 422 inside the translation
layer: when the endpoint rejects a field, litellm drops that field and sends
the request again, up to twice. That is the difference between a customer's
tool call working and coming back as a hard 400, and none of it was covered.
The retry loop in llm_http_handler.py is 13,419 lines of source against a
0.20 test-to-source ratio, and nothing exercised this path at all.
Drives real litellm.completion and litellm.acompletion calls against a
recorded Azure AI endpoint, so the assertions read the bytes that actually
went over the wire rather than a mock's call list. Nothing internal is
patched: respx fakes the HTTP boundary and the provider config, retry loop
and serialization are all the real ones.
Pins:
- a tool field the endpoint rejects is dropped and the call retried, and the
caller gets a normal completion
- the retry changes only the field the provider named
- a provider that keeps rejecting stops after exactly two attempts
- a rejection the provider cannot fix is not retried at all
- an extra input outside a tool is retried only when drop_params was asked for
Mutating the source confirms these bite: raising the retry cap from 2 to 3,
and making the tool-level field check always return False, each turn the
suite red.
The async cases pin the transport to httpx, because the aiohttp default
carries its own transport that an httpx-level fake cannot intercept. Without
that the two async tests reached the real Azure endpoint and failed on a 401.
* fix(rerank): emit latency and cost headers on /rerank
Thread the logging object into the rerank httpx calls and pass hidden_params through to get_custom_headers, so x-litellm-overhead-duration-ms, x-litellm-response-duration-ms, x-litellm-response-cost, x-litellm-call-id and the LITELLM_DETAILED_TIMING x-litellm-timing-* headers show up on rerank like they do on chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rerank): keep zero response cost in the /rerank cost header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: assign the new rerank endpoint tests to the proxy-endpoints shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: suppress TQ008 on the rerank header tests with reasons
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
* fix(http_handler): dispose aiohttp session when finalized without a running loop
AsyncHTTPHandler.__del__ can only schedule an async close when a running
event loop exists at finalization time; in any other context (worker
threads whose loop has closed, sync contexts, interpreter shutdown) the
RuntimeError from get_running_loop() is swallowed and the underlying
aiohttp ClientSession is abandoned to GC, emitting 'Unclosed client
session' / 'Unclosed connector' warnings.
This is the disposal gap left after the recycle-time fix: clients created
for short-lived event loops (the loop-id-keyed LLM client cache mints one
handler per loop) are never recycled - they live and die with their loop,
and their finalization is precisely the loop-less case.
Fix:
- no running loop: fall back to the connector's synchronous teardown via
LiteLLMAiohttpTransport._mark_connector_closed - the same finalizer-safe
path used for dead-loop recycles - honoring _owns_session so a shared
session is never closed.
- running loop: keep the async close, but hold a strong reference to the
scheduled task until it completes (a bare create_task() result may be
collected before running), mirroring _background_close_tasks.
Tests: loop-less finalization closes a dead-loop session; running-loop
finalization registers and drains the close task; the sync fallback
respects session ownership. All three fail without the fix.
* lint: conform new finalizer code to the type-discipline budget
Final on the five never-rebound locals (LIT010); the class-level task
registry keeps its mutable set with the sanctioned mutable-ok reason,
mirroring the aiohttp transport's registry (LIT001).
* lint: reasoned pyright ignore on the cross-class teardown call
The handler deliberately reuses the transport's finalizer-safe connector
teardown; no public seam exists and an async close can never run at
loop-less finalization. Clears the net-new reportPrivateUsage the
basedpyright budget gate flagged once the LIT stage passed.
* fix(http_handler): retrieve exceptions from finalizer close tasks
A bare discard done-callback dropped the task without consuming its
exception, so a failing aclose() emitted "Task exception was never
retrieved" at GC, the same noise class this path exists to remove.
Mirror the transport's _on_close_task_done: discard, early-return on
cancellation, retrieve and debug-log the exception.
* fix(http_handler): dispose foreign-loop sessions instead of scheduling aclose on the live loop
GC on a live loop (e.g. the app's) of a handler whose session belongs to
another, possibly dead, loop scheduled aclose() on the current loop, the
cross-loop path the transport refuses. Route both that case and the
loop-less case through the transport's lifecycle-aware
_close_recycled_session, which picks async close on the session's own
loop, threadsafe handoff, or the synchronous connector teardown.
Regression test: a dead-loop session collected while another loop runs
is disposed without scheduling anything on that loop.
* chore: retrigger CI (test_mcp_logging payload-order flake, also failed on litellm_spendlogs_fallback_metadata minutes earlier)
* test(mcp): select the MCP tool-call payload instead of the last-delivered one
TestMCPLogger kept a single last-writer slot; an async success event from
another call (a mocked acompletion whose log task lands late) races the
MCP event for it, so the cost assertions intermittently read the wrong
payload. This PR's finalizer change shifts task interleaving on the loop
and tips that latent race over (also seen on an unrelated PR minutes
earlier). Collect call_type=call_mcp_tool payloads in their own list and
assert on those.
* test(mcp): MCPLoggerHook inherits the order-independent payload capture
It duplicated TestMCPLogger's init and success handler verbatim; the
hook test reads the same MCP payload selection, so subclass instead.
AzureVideoConfig subclasses OpenAIVideoConfig and so inherits the new
use_multipart_form_data() -> True. Azure's /openai/v1/videos surface is
OpenAI-SDK-compatible, so the JSON->multipart flip is intentional; assert it
through the real handler so the inherited behavior can't silently regress.
POST /v1/videos without an input_reference file now goes out as
multipart/form-data the way the OpenAI SDK always sends it, instead of a
JSON body that OpenAI-compatible backends (SGLang Diffusion, vLLM-Omni)
reject; gemini, vertex, and runwayml keep their JSON bodies
/v1/images/edits on the openai/azure/openai-compatible path now forwards
unknown provider params (e.g. seed) and honors extra_body, matching
/v1/images/generations, and aimage_edit forwards
extra_headers/extra_query/extra_body instead of dropping them
Generic pass-through no longer downgrades a file-less multipart form to
application/x-www-form-urlencoded
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
Nine tests in test_http_handler.py captured litellm.disable_aiohttp_transport,
force_ipv4, ssl_ecdh_curve or the request_timeout pair, wrapped their whole body
in a try, and put the value back in a finally. monkeypatch.setattr does all of
that, so the captures, the try and the finally go away and the bodies lose a
level of indentation. The class-scoped restore_request_timeout fixture existed
only for that same bookkeeping and goes with them.
litellm.in_memory_llm_clients_cache is left alone on purpose: the eviction tests
assert a handler is garbage collected, and monkeypatch holds the replaced value
alive until teardown, which keeps the weakref they check from clearing.
The Responses WebSocket path, the pre-call deployment hook and the
per-frame project quota hook are all selected by small predicates that
nothing asserted directly. Mutating those four decisions left 4 of 6
mutants alive against the mapped test file.
Cover them at the boundary: the rust WebSocket path needs both the
openai provider and the rust flag, a plain CustomLogger must not
advertise a pre-call deployment hook while an overriding or inheriting
one must, and only callbacks that actually expose a callable
enforce_project_io_token_quota_for_frame reach the WebSocket loop.
Kill rate on those four decisions goes 2/6 -> 6/6; the file goes 66 -> 75
passing.
Add sync and async regression tests for the BaseLLMHTTPHandler streaming
path, which forwards provider response headers for the ~30 providers that
ride the generic handler and had no coverage. Also drop redundant setup
prose from the moonshot invoke test docstring.
* test: enforce PT012 so a pytest.raises block cannot hide dead assertions
`with pytest.raises(...)` stops at the first statement that raises. Anything
sequenced after it inside the block never runs, so an assertion written there is
never checked and the test still reports green.
Two sites were doing exactly that, and both assertions turned out to be wrong
once they started running. tests/llm_translation/test_prompt_factory.py asserted
the bedrock rejection names "requires at least one non-system message", which
holds. tests/proxy_unit_tests/test_proxy_server.py asserted the prisma startup
failure mentions "httpx.ConnectError", which never appears: the failure is an
httpx.ConnectError whose message is "All connection attempts failed", so that
test now asserts the type. Its DATABASE_URL override moves to monkeypatch, since
the old restore sat below the assertion and leaked the invalid URL into every
later DB test the moment the assertion started being able to fail.
The remaining 72 sites are rewritten without changing what they exercise: setup
that cannot raise moves above the block, a nested `patch` moves outside it, and
bodies with real control flow (a stream drain, an if/else on sync_mode, a
retry loop) move into a local closure the block calls.
Fixing PT012 unmasked two B017s, since ruff only reports a blind
pytest.raises(Exception) once the block holds a single statement.
tests/proxy_unit_tests/test_auth_checks.py narrows to the ProxyException
can_key_call_model actually raises. tests/local_testing/test_completion_cost.py
was asserting vertex_ai/medlm-medium has no cost entry, which stopped being true
at some point; that dead first half is gone and the rest of the test, which
checks medlm pricing resolves above zero, now runs instead of being skipped.
* chore(ci): ratchet TQ004 to 768 after the prisma test moved to monkeypatch
* test(lint): ban blind pytest.raises(Exception) with ruff B017
A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.
All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.
Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.
* fix(test): widen the openai batch-dispatch assertion to OpenAIError
The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
The generic container handler returned response.content for endpoints marked
returns_binary before it ran any status or error check, so a non-2xx answer
from the provider was handed back to the caller as raw bytes. Asking for the
content of a container file that does not exist returned the provider's 404
error body as an opaque payload instead of raising.
Move the check ahead of the binary short-circuit and apply it to every
container file endpoint, falling back to the response text when the error body
is not JSON.
A /v1/realtime connection to a Vertex AI Live model accepted the WebSocket
upgrade and then went silent: a stalled Google OAuth token fetch blocked the
handler before any session event, and the eventual failure closed the socket
with a bare 1011 and no error event, so callers saw an open socket, no frames,
and no reason.
Bound the pre-session token fetch with
REALTIME_CREDENTIAL_RESOLUTION_TIMEOUT_SECONDS (20s default) and, on any
realtime failure, send an OpenAI-style error event before closing with a reason
that names the failure. Close reasons are truncated by bytes, not characters,
since an over-long reason makes the close frame itself fail.
The proxy pre-creates the logging object before the router picks a deployment,
and the native /v1/messages handler never copied the deployment's
vertex_location into the logging params it updates, so cost resolution fell
back to the environment or the default region and priced every call on this
surface with the regional endpoint uplift. Copy the explicitly configured
location from the request's litellm params, the same source dispatch builds
the request URL from, and register the new regional_endpoint_uplift_multiplier
field in the cost map schema test.
* fix(vector_stores): stop leaking stored credentials in direct search debug logs
Direct vector store providers (RESP datastores like Valkey) have no HTTP
request to echo, so both search handlers called `logging_obj.pre_call` with
no `api_base`. The logging helper treats an empty `api_base` as "nothing to
render" and falls back to `str(self.model_call_details)`, which carries the
resolved `litellm_params`: the stored `valkey_password` and the embedding
config's `api_key` among them.
The stdout logger's regex redaction hid this, but `pre_call` also writes the
same string to `litellm_params["metadata"]["raw_request"]`, which ships
unredacted to every logging callback (Langfuse, OTel, etc.).
Pass a synthetic `<provider>://<vector_store_id>` endpoint plus an explicit
`request_str` so the debug output describes the call instead of dumping call
details, and fold the duplicated sync/async blocks into one helper so the
sanitized descriptor cannot drift between them.
* fix(vector_stores): type direct search query as Sequence[str]
The new helper's list[str] annotation pushed LIT001 over its
type-discipline ceiling. Sequence is the read-only shape the helper
actually needs, and list[str] still satisfies it at both call sites.
* feat(vector_stores): add Valkey as a managed vector store provider
Adds a valkey provider for managed vector stores, searchable via the
valkey-search module over RESP. Introduces BaseDirectVectorStoreConfig
for datastores that execute searches directly instead of building an
HTTP request, and refactors the valkey semantic cache to share the new
connection URL helper. Registered in the provider enum, router params,
proxy config registry, Admin UI Add Vector Store modal, and provider
endpoint support matrix.
* fix(vector_stores): join list queries and bound valkey socket timeouts
Review feedback: multi-string queries are now space-joined like every
other embedding-based provider instead of dropping all but the first,
and the request timeout is threaded through the direct vector store
interface into bounded socket_connect_timeout / socket_timeout values
on both redis clients so an unreachable Valkey host cannot pin proxy
workers until the OS TCP timeout.
* chore(ui): regenerate schema.d.ts for valkey vector store fields
* docs(ui): make the Valkey vector store setup note and field tooltips explicit
* feat(ui): pick the Valkey embedding model from the proxy's models like Milvus
* fix(ui): number the setup steps in the vector store provider alerts
#35978 stopped the pooled A2A client replaying one upstream's Set-Cookie to
another by installing a blocking policy on that client's httpx cookie jar. That
covers only one of the two jars on the request path. AiohttpTransport is the
default transport unless it is explicitly disabled, and the aiohttp ClientSession
behind it keeps its own cookie jar which no httpx-level assertion can observe, so
the leak is still live on the default path: a live proxy on that commit still
delivers agent-alpha's session cookie to agent-beta's card fetch and JSON-RPC
call.
The reason it looked fixed is that aiohttp's default CookieJar is built with
unsafe=False and refuses to store cookies for IP hosts, so a proof addressed to
127.0.0.1 comes back clean whether or not that jar is blocked.
Cookie persistence is now blocked where the clients are built rather than at one
call site: blocked_cookie_jar() gives every httpx client, async and sync, a jar
whose DefaultCookiePolicy(allowed_domains=()) rejects every domain in both
directions, and both ClientSession constructions litellm owns, the transport's
session factory and the proxy's shared startup session, get a DummyCookieJar.
LiteLLM reads a response cookie nowhere, and an explicitly supplied Cookie header
still goes out, so passthrough forwarding and an agent's extra_headers are
unaffected. The A2A-scoped policy #35978 added is removed, since it is now dead.
The two suites that drive the aiohttp session factory synchronously mock
ClientSession because a real one needs a running event loop; DummyCookieJar has
the same requirement, so they mock it for the same reason.
A cached HTTPHandler hands its raw httpx.Client out to consumers that keep it
for the process lifetime. When the shared client cache expires the entry on its
TTL or evicts it under the 200-entry cap, nothing references the handler, so it
is collected and its finalizer closed the client those consumers still hold.
Langfuse ingestion then failed silently on the SDK's background flush thread
until the process restarted.
A finalizer running proves only that nothing references the handler; it proves
nothing about the client. Both handlers now close the client during finalization
only when they built it and are still its sole referrer, so an unshared client is
still released promptly and a handed-out one is left alone. That keeps the
pooled-socket reclamation the finalizer was providing, which measures identical
to base over 2000 handler create-and-drop cycles.
Explicit close() stays, now gated on _owns_client so the wrapper never closes a
caller-injected client, and __aexit__ routes through it.
LangFuseLogger also keeps a reference to the handler whose client it hands the
SDK. Previously that handler was a local that went out of scope immediately,
leaving the client reachable only from the SDK. It still shares the cached
client, so no extra clients are created per logger.
The re-landed closer test asserted a reaped handler's client stays
closed; with #35862 the handler heals on next access, so the test now
pins the inner client up front and asserts the heal as the contract.
Also adds an end-to-end regression test that evicts an init-held
handler through LLMClientCache, waits out the grace close, and proves
the next request succeeds.
Since #35492 the evicted-client closer really closes litellm-owned
httpx clients once their cache entry is evicted and the grace window
passes. Objects that fetch get_async_httpx_client once in __init__ and
hold the handler forever (40 guardrail classes, pagerduty and email
callbacks, and more) then fail every request with 'RuntimeError: Cannot
send a request, as the client has been closed.'
AsyncHTTPHandler.client and HTTPHandler.client are now properties that
rebuild the inner client from the constructor's stored config when the
handler owns it and finds it closed. Caller-supplied or assigned
clients are never rebuilt, and close paths use the backing field so
closing a handler does not resurrect it.
* fix(aiohttp): dispose recycled client sessions deterministically
LiteLLMAiohttpTransport replaced its cached aiohttp.ClientSession on
loop-mismatch, loop-inspection failure, and "Session is closed" retry
without reliably closing the previous session:
- the close task from asyncio.create_task() was never referenced, so
it could be garbage-collected before running;
- the (RuntimeError, AttributeError) fallback branch replaced the
session without closing it at all;
- sessions bound to a closed event loop were abandoned to the GC
("rely on GC"), and sessions bound to a loop running in another
thread were closed from the wrong loop.
Replaced sessions surfaced as intermittent "Unclosed client session" /
"Unclosed connector" errors from the event-loop exception handler at
GC time.
_close_recycled_session() now covers the three lifecycles a recycled
session can be in: same-loop closes keep a strong task reference until
completion; sessions owned by a loop running elsewhere are closed on
their own loop via run_coroutine_threadsafe; sessions whose loop is
gone are disposed synchronously through the connector teardown that
aiohttp's own finalizer uses, which releases pooled connections and
silences the finalizer warnings.
Fixes#24230
* fix(aiohttp): guard threadsafe close callback against cancelled futures
---------
Co-authored-by: Anmol Jaiswal <68013660+anmolg1997@users.noreply.github.com>
The http_handler pair only received its full mutation verdict after the
first removal batch landed; these five tests pass unchanged when every
function they execute is mutated and the owning file killed none of
their scored mutants. The ssl tests excluded from mutation scoring are
untouched.
* fix(bedrock): stop stale SigV4 headers clobbering fresh signature on re-sign
When the Anthropic /v1/messages strip-thinking-and-retry path re-signs a
Bedrock request, _sign_request received attempt 1's already-signed headers
and copied the old Authorization and X-Amz-Date back over the freshly
computed SigV4 signature, so the retry POSTed the stripped body with a
signature for the original body and AWS returned 403 SignatureDoesNotMatch.
Skip SigV4-computed headers (authorization, x-amz-date,
x-amz-security-token, date) when restoring caller headers after signing,
and only preserve a caller-supplied Authorization that is not itself a
SigV4 header so bearer-token setups keep working.
* fix(bedrock): apply the same stale-header guard to get_request_headers
* fix(batches): price anthropic passthrough message batches correctly in batch cost job
Anthropic message batches created via the /anthropic passthrough were never
cost tracked. The CheckBatchCost job fetched batch results from the Files API
(POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id
must have file_ prefix"; the error response was silently wrapped as file
content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend
row, and the job was marked batch_processed=true so the $0 was permanent.
Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the
anthropic files transformation, raise on HTTP error status in
retrieve_file_content instead of returning the error body as content, parse
Anthropic's results JSONL shape (result.type == "succeeded",
result.message.usage with cache creation/read tokens) in batch_utils, price
cache creation tokens at cache_creation_input_token_cost in the batch cost
fallback (50% batch discount preserved for base input, cache reads, cache
writes, and output), and leave the managed object row unprocessed when cost
tracking fails so a later poll retries instead of permanently recording $0.
* fix(batches): carry cache token details into aggregated anthropic batch usage
* fix(llm_http_handler): send dict transcription request data as a JSON body
httpx form-encodes dicts passed via data= and silently ignores json=, so the
generic audio transcription path never actually sent a JSON body. No provider
hit this before; JSON-body speech APIs need it.
* feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support
Adds a VertexAIAudioTranscriptionConfig wired through ProviderConfigManager so
vertex_ai/chirp_3 works on /v1/audio/transcriptions (sync and async) via the
Speech-to-Text v2 recognize API. Auth reuses the standard Vertex credential
resolution (vertex_project/vertex_location/vertex_credentials or ADC); the
location defaults to the us multi-region since chirp_3 is only served from the
us and eu multi-regions, and non-global locations use the regional
<location>-speech.googleapis.com host. Maps language to languageCodes (auto
language detection by default), joins all result alternatives into the
transcript, and tracks cost from totalBilledDuration with a
vertex_ai/chirp_3 price entry at Google's published $0.016/min.
* fix(vertex_ai): map bare ISO-639-1 language codes to BCP-47 for Speech-to-Text
OpenAI clients send language codes like "en", which Google rejects with 400
("not supported by the model chirp_3 in the location us"); Speech-to-Text
wants region-qualified BCP-47 like "en-US". Adds a shared
normalize_transcription_language_to_bcp47 helper in audio_utils (NVIDIA Riva's
transcription config already hand-rolled the same table privately) that maps
common bare codes and passes region-qualified ones through, and applies it in
the Vertex transcription request. Also narrows the response JSON parse guard
to ValueError.
* fix(vertex_ai): drop zero output_cost_per_second so chirp_3 cost tracking works
cost_per_second prefers output_cost_per_second whenever it is not None, so the
0.0 in the chirp_3 entry priced every transcription at $0.00 instead of using
input_cost_per_second. Remove it from both cost maps and pin the behavior with
a regression test computing 18s of chirp_3 audio to ~$0.0048.
* fix(vertex_ai): validate client-controllable location to prevent SSRF in Speech-to-Text
get_complete_url interpolated vertex_location straight into the request host,
and vertex_location is client-controllable on the proxy (it flows from the
request body and is not on the request-body blocklist). An authenticated caller
could send vertex_location="attacker.example/" to point the host at their own
server, so the proxy would POST the audio plus its admin-minted Google bearer
token and x-goog-user-project header to the attacker, exfiltrating a
cloud-platform-scoped OAuth token minted from the admin's credentials.
Factor the location validation the rest of vertex_ai already applied in
get_vertex_base_url (^[a-z][a-z0-9-]*$ plus the global allowance) into a shared
validate_vertex_location helper in common_utils and call it from both the chat
host builder and the new speech host builder. Invalid locations now raise a 400
VertexAIError instead of building a host. Also reject vertex_project values that
carry URL-structural characters, since it lands in the URL path.
Regression tests assert on the parsed netloc so the security property is pinned:
valid locations always resolve to a *speech.googleapis.com host and injection
inputs are rejected.
* fix(vertex_ai): reject unsupported transcription response_format values instead of silently ignoring
* fix(anthropic_messages): forward provider response headers on streaming /v1/messages responses
* fix(anthropic_messages): forward aclose to inner streaming iterator
* fix(anthropic_messages): forward aclose through the streaming response wrapper
The proxy's streaming cleanup closes the handler's return value via
hasattr(response, "aclose"); the new wrapper hid the upstream
generator's aclose, so provider connections could linger on client
disconnect. The wrapper now delegates aclose to the wrapped stream and
AgenticAnthropicStreamingIterator closes its inner and follow-up
streams. Also adds test coverage for the agentic streaming branch
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(websearch): wire chat completion agentic loop to correct hooks
maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller.
Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path.
Regression test added.
* fix(websearch): strip tool_choice from follow-up request
When the original request forces tool_choice to litellm_web_search,
the follow-up request after search execution inherited that tool_choice,
causing the model to call the search tool again instead of synthesizing
an answer from the results.
* fix(websearch): inject api_key into agentic hook kwargs for anthropic messages
Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's
synthesis call after executing Exa/Perplexity searches) were missing api_key
because the named api_key param in async_anthropic_messages_handler was never
merged into the kwargs dict forwarded downstream. Result: every /v1/messages
websearch follow-up failed with "x-api-key header is required" and the caller
received the raw tool_use response instead of the synthesized answer.
* ci: trigger CI run
* fix(websearch): support unified agentic hooks alongside chat-completion-specific hooks
CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with
_agentic_loop_api_surface to handle both surfaces from one hook. The chat
completion loop must also check _gate_overridden so callbacks using the
unified hook pattern still fire for chat completions.
* fix(websearch): strip tool_choice from legacy chat completion follow-up call
The _execute_chat_completion_agentic_loop path merged original optional_params
(which includes forced tool_choice) into follow-up params without explicit
removal. _build_chat_completion_request_patch already excluded tool_choice from
its optional_params output, but dict.update() with a missing key leaves the
original value intact. Explicit pop after the merge removes it.
* fix(websearch): always strip tool_choice from plan-path follow-up params
The tool_choice removal was gated on patch.tools is not None. WebSearch sets
tools via patch.optional_params not patch.tools, so the gate was False and
forced tool_choice from the original request survived into the synthesis call.
Move the pop outside the patch.tools branch so it applies unconditionally.
* fix(passthrough): drop top-level additional_drop_params on /v1/messages
On the Anthropic Messages pass-through path, additional_drop_params only
stripped nested dotted paths, so plain top-level keys like `thinking` and
`context_management` were forwarded to the provider. Bedrock rejects these
with "Extra inputs are not permitted", returning a 400 to Claude App/CLI
even when the user configured `additional_drop_params: ["thinking"]`.
delete_nested_value already handles plain top-level fields, so route every
drop param through it and remove the nested-only filter. Fixes#25931.
* fix(passthrough): drop thinking for bedrock inference-profile ARNs on /v1/messages
Opaque Bedrock Application Inference Profile ARNs contain neither "anthropic"
nor "claude", so is_anthropic_claude_model returned False and the thinking
param was rewritten to reasoning_effort before additional_drop_params ran.
That made additional_drop_params: ["thinking"] a no-op for the converse-ARN
form, and the Bedrock Converse transform re-expanded reasoning_effort back into
additionalModelRequestFields.thinking, so the request 400'd.
Extend the thinking-translation gates to also accept bedrock ARNs via the
existing is_bedrock_arn_model helper, mirroring the cache_control path, so
thinking is preserved as thinking and additional_drop_params can drop it.
* fix(realtime): stop sending a second Gemini Live setup on follow-up session.update
Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client
message; a second setup closes the socket with 1007 Request contains an invalid
argument. The AI Studio Gemini path forwarded every client session.update after
the first as a follow-up setup, and GA clients (pipecat) send several while
configuring the session, so the second one tore the session down before the
first turn. Callers saw silence after the first response, exponential per-turn
latency from reconnect/retry churn, and intermittent 1011 errors.
Drop subsequent session.updates instead of resending setup, matching what the
Vertex subclass already does. Tools and instructions must ride on the first
session.update before any conversation content.
Adds regression tests covering the plain follow-up, a follow-up that adds tools
(the case the previous identical-only dedup still forwarded), and the guardrail
create_response=False warning path.
* fix(realtime): retry the backend open handshake instead of failing with 1011
The upstream Live API open handshake (e.g. Gemini Live) intermittently hangs;
waiting longer never recovers a hung attempt, but a fresh attempt almost always
connects in ~1s. The proxy opened the backend websocket once with the default
open_timeout and no retry, so a single slow handshake surfaced to the caller as
a fatal 1011 internal error and dropped the call.
Bound each open attempt with a short open_timeout and retry; a bounded attempt
that already timed out spaces out the next try, so no backoff is needed.
Deterministic handshake-status rejections (auth/4xx) are not retried, and the
retry only ever wraps the open, never a live session.
Adds tests for retry-then-succeed, raise-after-max-attempts, and
no-retry-on-auth-failure.
* fix(realtime): close guardrail bypass + surface handshake status; drop obsolete tests
Three review fixes on the Gemini Live realtime path.
Transcription-guardrail bypass: Gemini Live rejects a second setup (1007), so once
the initial setup is sent the guardrail's automaticActivityDetection.disabled=true
can no longer be delivered as a follow-up session.update. With that follow-up now
dropped, the model's auto-response stayed enabled and a realtime_input_transcription
guardrail was bypassed (the model answered before the proxy could gate the turn).
Fold the disable into the one-and-only setup instead: the handler injects it into
the auto-sent setup (gemini_live_defer_setup false) and _send_to_backend injects it
into the deferred first setup. OpenAI sessions accept follow-up updates and are left
untouched.
Backend handshake status: the open-retry treated only InvalidStatusCode as
deterministic; websockets>=15 raises InvalidStatus for a rejected client handshake,
so a 401/403 fell into the broad WebSocketException branch and was retried before
the caller closed the client with 1011 instead of the upstream status. Treat both as
non-retryable.
Obsolete tests: the four tests asserting a follow-up session.update is merged and
re-sent as a second setup asserted behavior that crashes Gemini Live with 1007
(verified directly against the API). Removed; the drop is covered by new regression
tests.
* style(realtime): reformat changed files to ruff line-length 120
Post-merge with litellm_internal_staging, which unified ruff format width to 120
(#31518). The realtime change set was formatted at 88, so the changed lines
tripped the whole-file ruff format check. Reformat with ruff 0.15.3 at the repo's
120 width; no logic changes.
