mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
docs: correct which surfaces the loop ceiling covers
This commit is contained in:
parent
7c02c089f6
commit
0d1e2a5b11
2 changed files with 21 additions and 15 deletions
|
|
@ -235,18 +235,20 @@ model_list:
|
|||
Clients cannot set it. `max_agentic_loops` is on the proxy's untrusted-field list, so a request body that
|
||||
carries it is ignored and one request can never drive an unbounded number of upstream model calls.
|
||||
|
||||
When the ceiling is reached on a non-streaming `/v1/messages` request, the turn ends there and the client gets
|
||||
the last response back with the internal `litellm_web_search` tool call removed and `stop_reason: end_turn`.
|
||||
A streaming request the interceptor converted to non-streaming counts as one of these, since the client is
|
||||
still waiting on a single response. The client never declared that tool, so leaving the block in would hand it
|
||||
a tool call it has no way to answer. The answer can be less complete than it would have been with more loops,
|
||||
When the ceiling is reached on a `/v1/messages` request, the turn ends there and the client gets the last
|
||||
response back with the internal `litellm_web_search` tool call removed and `stop_reason: end_turn`. The client
|
||||
never declared that tool, so leaving the block in would hand it a tool call it has no way to answer. The answer can be less complete than it would have been with more loops,
|
||||
which is the tradeoff the ceiling buys, and where the refused call was the only block left the turn can come
|
||||
back with no text in it at all.
|
||||
|
||||
Two paths do not get that treatment yet. A request that streams all the way through, meaning one the
|
||||
interceptor did not convert, has already put its message on the wire before the ceiling is checked. And
|
||||
`/v1/responses` returns its own shape that the finalizer does not rewrite, so it still hands back the internal
|
||||
call. Both are tracked separately
|
||||
Streaming is covered by the same path rather than a separate one, because interception always converts an
|
||||
intercepted `stream=True` request to non-streaming before the loop runs, then rebuilds the SSE stream from the
|
||||
finalized turn. So the ceiling is reached on a response the client has not seen yet either way.
|
||||
|
||||
Two other surfaces do not get that treatment yet. `/v1/responses` returns its own shape that the finalizer does
|
||||
not rewrite, so it still hands back the internal call. And `/v1/chat/completions` runs its own copy of these
|
||||
rails in `litellm_core_utils/chat_completion_agentic_loop.py`, which still raises rather than ending the turn.
|
||||
Both are tracked separately
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -5186,12 +5186,16 @@ class BaseLLMHTTPHandler:
|
|||
"""
|
||||
Whether a refused rerun can still be answered with a finalized turn.
|
||||
|
||||
Only the non-streaming anthropic messages path can. A streaming caller
|
||||
has already sent the original message to the client, so a finalized one
|
||||
would arrive as a second message rather than as a replacement, and the
|
||||
responses surface carries a pydantic model that the finalizer does not
|
||||
rewrite. Both keep raising, which is what every surface did before this
|
||||
path learned to end the turn.
|
||||
Only the anthropic messages surface can. The responses surface carries a
|
||||
pydantic model the finalizer does not rewrite, so it keeps raising, which
|
||||
is what every surface did before this path learned to end the turn.
|
||||
|
||||
Every call site passes ``stream=False`` today, because interception
|
||||
converts an intercepted stream to non-streaming before the loop runs and
|
||||
rebuilds the SSE stream from the finalized turn afterwards. The flag is
|
||||
still checked so a streaming call site added later cannot replace a turn
|
||||
already on the wire, which would reach the client as a second message
|
||||
rather than as a replacement.
|
||||
"""
|
||||
return not stream and api_surface == "anthropic_messages"
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue