scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 35
litellm/tests/e2e/claude_code
History
Yassin Kortam d5b91b94d3
test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 
The tool_search x bedrock_invoke cell only ever probed the first turn, so
nothing in the suite has sent a server_tool_use block back to a provider.
Every turn of a real Claude Code session after the first carries the
server_tool_use and tool_search_tool_result blocks the previous turn
produced, and that path was uncovered.

Adds probe_tool_search_multiturn, which takes the real assistant turn
back, answers any client-side tool_use with the id the model actually
emitted, and replays the whole thing as history with the tools still
declared. The assertion refuses to go green unless both server-tool
blocks made it into the replayed history, so a first turn truncated at
max_tokens reads as a failure instead of a vacuous pass.

The replay assertion's red paths never run in a green cell, so they get
markerless harness tests of their own alongside the existing
_builder_unit_tests tree.

No production code.
2026-08-17 11:59:26 -07:00
..
_builder_unit_tests chore(e2e): port the compat-matrix cron publisher to tests/e2e/claude_code 2026-08-10 21:46:56 +00:00
_probe_unit_tests test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
basic_messaging_non_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
basic_messaging_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
count_tokens refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods (#33760) 2026-07-18 19:03:01 +00:00
cron_vm fix(compat-matrix): keep publish token out of the job-wide process env 2026-08-11 00:57:25 +00:00
long_context_1m test(e2e): harness fixes for stage job green (skips + router/UI/budget) (#33634) 2026-07-16 20:30:30 -07:00
passthrough test(e2e): harness fixes for stage job green (skips + router/UI/budget) (#33634) 2026-07-16 20:30:30 -07:00
pdf_input fix(bedrock): add text block to converse user messages carrying documents 2026-08-11 02:35:49 +00:00
prompt_caching_1h fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
prompt_caching_5m fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
structured_outputs fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
thinking fix(bedrock): preserve adaptive thinking effort through the /v1/messages bridge 2026-08-11 03:23:16 +00:00
thinking_with_tool_use fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
tool_search test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
tool_use test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
tool_use_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
vision fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
web_search fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
__init__.py test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
_basic_messaging.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
_compat_models.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
_env.py refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods (#33760) 2026-07-18 19:03:01 +00:00
_gpt_cells.py test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
_passthrough.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
cli_driver.py test(e2e/claude_code): retry rate-limit-shaped CLI failures with backoff 2026-07-15 18:06:28 -07:00
conftest.py test(e2e): rename Gateway to ProxyClient and expose it as a session-scoped fixture (#33750) 2026-07-18 18:41:18 +00:00
http_probe.py test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
manifest.yaml Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_claude_code_e2e_gpt_big3 2026-07-16 15:04:08 -07:00
matrix_builder.py chore(e2e): port the compat-matrix cron publisher to tests/e2e/claude_code 2026-08-10 21:46:56 +00:00
pr_gate_version_resolver.py test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
rate_limiter.py test(e2e/claude_code): add GPT-5.6 Sol/Terra/Luna provider columns for OpenAI, Azure OpenAI, and Bedrock Mantle 2026-07-15 16:25:32 -07:00
run_compat.sh test(e2e/claude_code): update run_compat.sh flag docs to COMPAT_MANTLE_CELLS 2026-07-16 17:26:46 -07:00
sample_compatibility-matrix.json test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
test_config.yaml test(e2e/claude_code): reuse AZURE_API_BASE/KEY for the azure_openai GPT column 2026-07-16 16:45:18 -07:00