litellm/tests
zoveki 901f914ba2
fix(responses): replay Claude tool search on follow-up turns (#50821)
* fix(responses): replay Claude tool search on follow-up turns

Claude tool search on /v1/responses now returns a tool_search_output after each tool_search_call, streamed and not, listing the tools the search loaded as the caller sent them

A stateless follow-up sends the search back to Anthropic as the server_tool_use and tool_search_tool_result Claude produced, and the Anthropic request drops a replayed search whose loaded tools are no longer sent, so a changed tool list no longer answers 400

The guardrail write-back for SDK output items reads part text through a typed helper, so the new output item type adds no attribute errors

* fix(responses): emit Claude's tool search output in stream order

The streamed tool_search_output item was announced at the end of the stream with an output index reserved earlier, so output_item.added events arrived as 0, 2, 1 and the OpenAI SDK's stream helper raised IndexError. The item is now emitted when Claude's search result arrives.

* fix(anthropic): keep the caller's tool names in tool search results

A tool name Anthropic cannot take gets rewritten on the request, so Claude's
tool_search_tool_result named the rewritten tool. The response now maps those
tool references back to the caller's names, streamed and not, and a replayed
search result maps them forward again on the next turn.

* fix(responses): replay Claude's tool search only on routes that run it

A follow-up sent to a route that cannot run tool search, a Claude model without it included, now leaves out the replayed search the way it did before, instead of sending Claude search blocks that model cannot take. The stateless gate and the stored-turn filter now use the same check that already decides whether the tool_search tool itself is kept

A streamed tool_search_call now gets its output_item.done when Claude's search result arrives, so it finishes before its tool_search_output. Clients that build history from done events, like Codex, kept the output ahead of its call

* fix(responses): keep a replayed search when the caller sent Anthropic's own search tool

A Claude model the cost map does not flag for tool search still runs it when the request carries Anthropic's own tool_search_tool_regex or bm25 tool, which is how turn 1 already decides to keep it. Both replay gates now follow that rule, so a stored follow-up on such a model keeps Claude's search as it did before, and a stateless one replays it

* fix(responses): replay a BM25 tool search under its own name

* fix(responses): keep a replayed tool search visible to guardrails and prune only missing tools

A replayed Claude tool search now reaches pre-call guardrails and the Purview DLP scan with its
search arguments, a stale search drops only the tool references the follow-up no longer sends,
and the stored previous_response_id path keeps the base gate for routes the cost map does not flag

* fix(responses): replay Claude's search on a stateless follow-up only when it still sends a search tool

A store false follow-up that sent no search tool still got the search replayed, and Claude then said it called a tool it no longer had

* fix(responses): replay a stored Claude search only when the follow-up still sends a search tool

A previous_response_id follow-up that sent no search tool still got the stored search, the same gap a store false follow-up had. The stale search pruning now logs the tools it leaves out at debug level.

handler.py reads the reported speed from the usage chunk, and the guardrail write-back skips tool search outputs, so the branch adds no type errors.

* fix(responses): leave out a stored Claude search kept in server_tool_calls

Main stores Claude's server-side tool search on the assistant message in
provider_specific_fields.server_tool_calls and tool_results, not in
tool_calls. The previous_response_id path only looked at tool_calls, so a
follow-up that sent no search tool still replayed the stored search.

The session handler now finds the search in either place and drops it from
both server_tool_calls and tool_results when the follow-up cannot run a
search.

* test(responses): give the replayed search guardrail fixture its own call id

The guardrail test reused toolu_01Weather, an id that streaming and bridge
tests already put in the process-wide TOOL_CALLS_CACHE. When one of those ran
first in the same worker, the guardrail write-back took its rebuild path and
the test failed for a reason unrelated to the replayed search.

* fix(anthropic): keep Claude's replayed tool search on renamed tools and stored follow-ups

A search-only assistant turn now gets its search result renamed to the sanitized
tool names too, so the stale-tool trim no longer drops it. Stored follow-ups on
Claude routes replay the search as base did, trimmed to the tools still sent, and
Bedrock invoke sends the tool search beta whenever the history holds a search

* test(responses): pin that guardrails scan a replayed Claude search on a follow-up without a search tool

* fix(anthropic): keep Anthropic's tool names in chat tool search results

A chat history whose search result used the request's tool names (weather.get) got
Anthropic's 400 when replayed on a proxy still on the old code, during a rolling deploy or a
rollback. Chat completions now returns Claude's search result as main does, and the Responses
bridge maps a rewritten name back to the request's tool when it builds tool_search_output.
That also fills tool_search_output on Bedrock invoke streams, which carry no name map

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 17:13:40 +00:00
..
_support feat(mcp): add an opt-in per-server setting to skip MCP tool output validation (#49961) 2026-11-03 20:42:10 -08:00
audio_tests feat(vertex-ai)!: support gemini 3.1 flash tts (#31915) 2026-10-21 04:18:42 +00:00
base_sdk_tests feat(packaging): build litellm-proxy gateway wheels (#49605) 2026-11-05 16:37:12 -08:00
benchmarks fix(realtime)!: send the Gemini Live setup at the first content so session.update settings apply (#39864) 2026-10-30 01:31:39 -07:00
code_coverage_tests refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
documentation_tests fix(proxy): let configured pass-through endpoints outrank built-in provider prefix routes (#44133) 2026-10-10 12:26:17 -07:00
e2e fix(ui): preserve credential actions across dashboard renders (#50845) 2026-11-10 06:17:41 +00:00
e2e_harness test(e2e): add a Codex row where a sub-agent must read the task its parent hands it (#50494) 2026-11-07 21:16:39 +00:00
fixtures test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
harness_e2e fix(harness): set PWD to the working directory of local sandbox commands (#47530) 2026-10-20 20:06:11 +00:00
integration fix(responses): replay Claude tool search on follow-up turns (#50821) 2026-11-10 17:13:40 +00:00
rust-python-harness fix(messages): make rust messages copy python messages behaviour (#49611) 2026-11-02 21:02:52 -08:00
test_litellm_rust fix(messages): write the native unary cache entry after the router stamps it like python (#50426) 2026-11-10 00:24:21 +00:00
typing fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
unit fix(responses): replay Claude tool search on follow-up turns (#50821) 2026-11-10 17:13:40 +00:00
windows_tests refactor(rust-bridge): require the native extension (#48852) 2026-10-28 13:16:56 -07:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_logging.conf
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
VCR_TESTING.md test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.