* fix(websearch): wire chat completion agentic loop to correct hooks
maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller.
Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path.
Regression test added.
* fix(websearch): strip tool_choice from follow-up request
When the original request forces tool_choice to litellm_web_search,
the follow-up request after search execution inherited that tool_choice,
causing the model to call the search tool again instead of synthesizing
an answer from the results.
* fix(websearch): inject api_key into agentic hook kwargs for anthropic messages
Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's
synthesis call after executing Exa/Perplexity searches) were missing api_key
because the named api_key param in async_anthropic_messages_handler was never
merged into the kwargs dict forwarded downstream. Result: every /v1/messages
websearch follow-up failed with "x-api-key header is required" and the caller
received the raw tool_use response instead of the synthesized answer.
* ci: trigger CI run
* fix(websearch): support unified agentic hooks alongside chat-completion-specific hooks
CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with
_agentic_loop_api_surface to handle both surfaces from one hook. The chat
completion loop must also check _gate_overridden so callbacks using the
unified hook pattern still fire for chat completions.
* fix(websearch): strip tool_choice from legacy chat completion follow-up call
The _execute_chat_completion_agentic_loop path merged original optional_params
(which includes forced tool_choice) into follow-up params without explicit
removal. _build_chat_completion_request_patch already excluded tool_choice from
its optional_params output, but dict.update() with a missing key leaves the
original value intact. Explicit pop after the merge removes it.
* fix(websearch): always strip tool_choice from plan-path follow-up params
The tool_choice removal was gated on patch.tools is not None. WebSearch sets
tools via patch.optional_params not patch.tools, so the gate was False and
forced tool_choice from the original request survived into the synthesis call.
Move the pop outside the patch.tools branch so it applies unconditionally.
failing test is not related to the pr
* fix(websearch): sync tool_choice when converting web_search tools
Claude Code forces native web search via tool_choice pointing at web_search
while websearch_interception renames the tool to litellm_web_search, causing
Anthropic 400s. Forward tool_choice into pre-request hooks and rewrite forced
tool_choice to match the converted tool name.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(websearch): re-wrap agentic loop responses as SSE for streaming clients
When websearch interception converts stream=true to false for the agentic
loop, dict responses from the loop were returned as application/json even
though the client requested SSE. Wrap those responses in
FakeAnthropicMessagesStreamIterator so /v1/messages streaming callers
(e.g. Claude Code) receive text/event-stream after search completes.
Fixes#27721
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(websearch): cover tool_choice sync and post-loop SSE wrap; fix UP006
Add regression tests for both websearch interception fixes: _sync_forced_tool_choice
repointing a forced web_search tool_choice to litellm_web_search (the 400 fix) and
_maybe_websearch_fake_stream_wrap re-wrapping agentic loop dict responses as SSE for
streaming clients (#27721). Switch the new helper annotations to builtin dict/list so
the ruff UP006 strict-rule ceiling stays within budget.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(websearch): resolve merge conflict and unify fake stream wrapping
Remove the duplicate _maybe_websearch_fake_stream_wrap helper left by a bad merge that caused a SyntaxError in CI, and route all call sites through _maybe_wrap_in_fake_stream instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MBP.localdomain>
* fix(websearch): wrap agentic loop response in fake stream for streaming requests
When websearch_interception converts stream=True to stream=False internally,
the agentic loop returns a plain dict. Previously this dict was returned
directly to the client expecting SSE events, resulting in empty streams.
Added _maybe_wrap_in_fake_stream() which checks the
websearch_interception_converted_stream flag and wraps dict responses in
FakeAnthropicMessagesStreamIterator. Applied to all return paths in
_call_agentic_completion_hooks:
- async_run_agentic_loop (legacy path)
- _execute_anthropic_agentic_plan (plan-based path)
- plan.response_override
- plan.terminate
Includes unit tests for _maybe_wrap_in_fake_stream().
* test(websearch): cover agentic-loop wrap paths; gate fake-stream on anthropic_messages surface
Guard _maybe_wrap_in_fake_stream on api_surface == anthropic_messages so the
responses API surface is never wrapped in an Anthropic SSE iterator, and type
logging_obj as Optional to match the None call sites. Adds regression tests
that drive the legacy, response_override, and terminate return paths of
_call_agentic_completion_hooks end to end.
* test(websearch): cover _execute_anthropic_agentic_plan and tail wrap paths
Drives the remaining two fake-stream return paths of
_call_agentic_completion_hooks (the _execute_anthropic_agentic_plan branch via
a stubbed handler, and the tail path when no agentic loop runs) so every
converted-stream return path is regression-tested.
---------
Co-authored-by: Clawd <fffff.c@gmail.com>
* fix(datadog): pass callback_specific_params so DatadogCostManagementLogger receives cost_tag_keys (#29590)
* fix(datadog): pass callback_specific_params so DatadogCostManagementLogger receives cost_tag_keys
* test(proxy): regression test that load_config forwards callback_specific_params
* fix(proxy): guard lakera_prompt_injection callback_specific_params against non-dict
Addresses review feedback: forwarding callback_settings as callback_specific_params
(so DatadogCostManagementLogger receives cost_tag_keys) exposed the
lakera_prompt_injection branch, which did lakeraAI_Moderation(**callback_specific_params
["lakera_prompt_injection"]) with no type guard. A config like
`callback_settings: {lakera_prompt_injection: "any-string"}` then hit `**"any-string"`
-> TypeError: argument after ** must be a mapping, not str.
Guard the lakera branch with isinstance(dict), matching the existing presidio and
datadog_cost_management branches (non-dict values fall back to {}). Add a regression
test asserting initialize_callbacks_on_proxy ignores a non-dict value instead of crashing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: inject fake lakera_ai module to avoid importing the real one
CI fix for the lakera regression test: it stubbed litellm.proxy.proxy_server with
a SimpleNamespace and then monkeypatch.setattr'd the real lakera_ai module, which
forces importing it — and lakera_ai does `from litellm.proxy.proxy_server import
LiteLLM_TeamTable`, absent on the stub -> ImportError under proxy-infra tests.
Inject a fake lakera_ai module into sys.modules instead, so the callbacks branch's
`from ...lakera_ai import lakeraAI_Moderation` resolves to the stub without loading
the real module. The guard under test (isinstance(dict) in the lakera branch) is
unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(callbacks): guard compression/websearch interceptors against non-dict callback_settings (#30153)
#29590 forwards the full callback_settings dict into initialize_callbacks_on_proxy, which activates the compression_interception and websearch_interception consumers. Their initialize_from_proxy_config read the callback_settings subkey without an isinstance(dict) guard, so a non-dict value such as `compression_interception: true` reached from_config_yaml(...).get(...) and aborted proxy startup with AttributeError. #29590 added that guard for lakera_prompt_injection but not for these two
Mirror the isinstance(dict) guard already used by the lakera, presidio, and datadog branches so a non-dict value is ignored and the callback initializes with defaults. A parametrized test feeds every callback_settings consumer a non-dict value through initialize_callbacks_on_proxy to catch a future consumer that forgets the guard
* fix(callbacks): normalize non-dict callback_specific_params to empty dict
A blank callback_settings: key in YAML loads as None, and
config.get('callback_settings', {}) returns None because dict.get only
falls back to the default when the key is absent. Forwarding that value
verbatim to initialize_callbacks_on_proxy made the first
'<name>' in callback_specific_params membership test raise
TypeError: argument of type 'NoneType' is not iterable, aborting proxy
startup. Same failure for any non-dict root such as callback_settings: true.
Normalize the value at the function boundary so both callsites (and any
future ones) initialize callbacks with their defaults instead of crashing.
---------
Co-authored-by: Hedi Daoud <150018939+hdaoud23@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(custom_logger): add async_post_agentic_loop_response_hook
Lets a CustomLogger shape the response returned by the agentic-loop
follow-up call without bypassing the loop's safety / observability
machinery (depth tracking, fingerprinting, etc.). Default returns the
response unchanged.
Used by websearch_interception to inject Anthropic-native
web_search_tool_result blocks when the originating client requested a
native web_search_* tool.
* feat(llm_http_handler): call post-agentic-loop hook on the originating callback
In _execute_anthropic_agentic_plan, after anthropic_messages.acreate
returns, call the originating callback's
async_post_agentic_loop_response_hook so it can mutate the final
response (e.g. inject native tool_result blocks). Pass the callback
through from _call_agentic_completion_hooks.
Exceptions in the post-hook are caught and logged so a buggy callback
can't kill the request.
* feat(websearch_interception): add is_anthropic_native_web_search_tool
Identifies tools the Anthropic-native clients (Claude Desktop, the
Anthropic SDK, the Anthropic Console) use to request native search:
type starts with "web_search_" (e.g. web_search_20250305). Rejects the
LiteLLM standard tool, the OpenAI-function variant, the bare
"WebSearch" legacy name, and the bare "web_search" Claude Code shape.
This lets us decide per-request whether the client expects
web_search_tool_result content blocks in the response, without
renaming any existing constants or touching native-provider skip
logic.
* feat(websearch_interception): add build_web_search_tool_result_block
Produces the Anthropic-native web_search_tool_result content block
from a structured SearchResponse. Anthropic-native clients use this
block to populate citations / source links — the existing text-blob
flatten path only feeds readable evidence to the model and discards
the structure, so this builder gives us the missing piece.
Shape matches https://docs.anthropic.com/en/api/web-search-tool —
web_search_result items carry url, title, page_age, encrypted_content
(empty string when the search provider doesn't supply one).
* feat(websearch_interception): emit native web_search_tool_result blocks
When the originating client request carried a native Anthropic
web_search_* tool, the final response now also carries
web_search_tool_result content blocks alongside the model's text
answer — so Claude Desktop / Anthropic SDK clients can populate the
citations panel and replay conversation history with structured search
evidence.
Wiring:
- Pre-request hooks (both deployment + Anthropic path) set a flag on
kwargs when they see a native web_search_* tool, so the signal
survives the conversion-to-litellm_web_search step regardless of
which hook fires first.
- _execute_search now returns (text, SearchResponse) so the structured
results aren't lost when the text is flattened for the follow-up
model call.
- _build_anthropic_request_patch returns the parallel list of
SearchResponse objects.
- async_build_agentic_loop_plan pre-builds the web_search_tool_result
blocks (one per tool_use_id) and stashes them on plan.metadata when
the flag is set.
- async_post_agentic_loop_response_hook reads the metadata and
prepends the blocks to response.content.
- _execute_agentic_loop mirrors the injection for the legacy path so
both paths behave identically.
Clients that send the LiteLLM standard tool keep the existing
text-only behavior — no regression.
* test(websearch_interception): cover native web_search_tool_result emission
18 tests across:
- detector branches (native vs litellm-standard, OpenAI-function shape,
Claude Desktop builtin WebSearch, bare web_search, missing type)
- block-builder shape (results, none, empty)
- pre-request hook flag-setting (native sets, standard does not)
- async_build_agentic_loop_plan attaches blocks to plan.metadata when
the flag is present, leaves metadata untouched when absent
- post-hook injection into dict and object responses
- legacy _execute_agentic_loop mirrors the injection so both paths
return the same shape
* test(websearch_short_circuit): keep _execute_search mocks in sync with new tuple return
* test(websearch_thinking_constraint): keep _execute_search mocks in sync with new tuple return
* feat(websearch_interception): emit native blocks from try_short_circuit_search
The agentic-loop post-hook only fires when the model returns a tool_use
block. Cowork / Claude Desktop on Bedrock actually make TWO requests
per user turn: the main /v1/messages with their builtin tool, and a
separate standalone /v1/messages whose only tool is
web_search_20250305. That second request hits try_short_circuit_search
— no agentic loop, no post-hook — and was returning text-only, leaving
the citations panel empty.
When the short-circuit input carries a native web_search_* tool, build
a synthetic server_tool_use + web_search_tool_result pair (using the
structured SearchResponse already returned by _execute_search) so the
client gets the native shape it expects. The legacy text block is
preserved so non-native short-circuit callers (Claude Code,
github_copilot, etc.) see the same payload as before.
Failure path still emits the native block pair (with empty results)
plus the text-error block, so the client gets a well-formed response
rather than a malformed half-shape.
* test(websearch_native_blocks): cover short-circuit native-block emission
Three new cases on top of the existing 18:
- native web_search_20250305 short-circuit → [server_tool_use,
web_search_tool_result, text], ids paired, urls/titles carried.
- litellm_web_search short-circuit → text-only (no regression).
- native short-circuit on search failure → still emits the native
block pair (empty results) plus the text-error block, so the client
never sees a malformed half-shape.
* test(websearch_short_circuit): index assertions by block type, not by position
Native short-circuit responses now have [server_tool_use,
web_search_tool_result, text] when the input carries
web_search_20250305 — find the text block by type rather than relying
on content[0].
* fix(websearch_interception): gate legacy WebSearch name on schema absence
Clients like Cowork / Claude Desktop ship a client-side tool named
"WebSearch" with a full input_schema — they handle it themselves and
expect to make a separate native web_search_20250305 sub-request for
the actual search.
Today is_web_search_tool matches the bare name regardless of other
fields, which hijacks the client's tool server-side. The agentic loop
fires on the main request, the model never gets to emit the
client-side tool_use, and the separate native sub-request (where
citation data flows) is never made. Net: citations panel empty.
Real Anthropic client tools always carry input_schema (the API rejects
them otherwise), so a bare {name: "WebSearch"} with no schema is the
only thing that could be a legacy interception marker. Gate the match
on schema absence: legacy callers (if any) keep working, real
client-side WebSearch tools pass through untouched.
* fix(websearch_interception): drop "WebSearch" from response-detection lists
Post-conversion the model always sees ``litellm_web_search``, so the
"WebSearch" entry in the response-side tool_use detection lists was
dead at best. If a model ever did return ``tool_use(name="WebSearch")``
it would now (incorrectly) hijack the client's own ``WebSearch`` tool
again — same Cowork problem we just fixed on the input side. Drop it.
* test(websearch_native_blocks): cover the WebSearch legacy-name schema gate
Three new cases:
- {name: "WebSearch"} (bare interception marker) → still matched
- {name: "WebSearch", input_schema: {...}} (Cowork client tool) →
passes through untouched
- {name: "WebSearch", description: "..."} (no schema) → still matched
on the assumption it's a legacy marker rather than a malformed real
client tool.
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The deployment hook now converts stream=True→False in wrapper_async's
scope so the streaming early-return path is skipped and logging executes.
logging_obj.stream is synced after the hook, and the original stream
intent is recovered for the short-circuit path.
Made-with: Cursor
* update bedrock models in tests
* updated more tests and model_prices_and_context_window
* fix model id and pricing
* replace more sonnet models
* update tests
* git push
* update pricing
* flaky total cost
* monkey patch
* relax the cost change
* fix and revert some changes
* revert the pricing
* chore: move cost/pricing changes to bedrock-cost-fixes branch
* chore: split Bedrock file-api beta stripping to separate branch
Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch;
see litellm_bedrock_invoke_strip_file_api_betas for that fix.
Made-with: Cursor
- Skip short-circuit for providers that have a BaseAnthropicMessagesConfig
(bedrock, vertex_ai, azure_ai, anthropic) — they use the agentic loop
which includes a follow-up LLM synthesis step. Short-circuiting would
return raw search text instead of an LLM-synthesized answer.
- Add fallback to litellm.get_llm_provider() for custom_llm_provider
derivation when litellm_params is overwritten by kwargs.
- Add test for bedrock guard.
Addresses Greptile review comments #3 and #4.
- Replace hand-rolled _extract_search_query with existing
get_last_user_message from common_utils
- Use full UUID (str(uuid.uuid4())) to match codebase convention
- Move uuid import to module level per CLAUDE.md
Addresses Greptile review feedback:
- Save original stream flag before pre-request hooks convert it, so
streaming callers get SSE events instead of a plain dict
- Propagate custom_llm_provider derived inside _execute_pre_request_hooks
when it was not explicitly passed by the caller
- Add tests covering both scenarios
For providers like github_copilot that don't natively support web search,
Claude Code's search sub-conversations were falling through to the adapter
path which strips the web_search tool and has no stream reconversion.
Instead of routing search requests through the full LLM pipeline, detect
web-search-only requests early (all tools are web_search, simple prompt)
and execute the search directly via Tavily/Perplexity, returning a
synthetic Anthropic response. No adapter, no backend LLM call needed.
Fixes#21733
When extended thinking is enabled, the websearch interception agentic loop
builds a follow-up assistant message with only tool_use blocks. Anthropic's
API requires assistant messages to start with thinking/redacted_thinking
blocks when thinking is enabled, causing a 400 Bad Request.
Extract thinking blocks from the model's initial response, thread them
through the agentic loop, and prepend them to the follow-up assistant
message — matching the pattern used by anthropic_messages_pt in factory.py.
Fixes the error: "Expected 'thinking' or 'redacted_thinking', but found
'tool_use'"
The websearch interception handler was passing internal flags like
`_websearch_interception_converted_stream` to the follow-up LLM request.
This caused "Extra inputs are not permitted" errors from providers like
Bedrock that use strict Pydantic validation.
Fix: Filter out all kwargs starting with `_websearch_interception` prefix
before making the follow-up anthropic_messages.acreate() call.
