litellm/tests/e2e/claude_code
Yuneng Jiang c63d0e6922
fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process
The cache edge keyed every recording on its own process's PYTEST_CURRENT_TEST.
Under xdist that names whatever test the serving worker is in, which is
unrelated to the caller: the proxy is a separate pod, and the Claude Code compat
matrix registered its shared aliases from every worker, each pointing at that
worker's edge, so the router spread one worker's calls across all eight edges.
Builds 234 and 235 of litellm-e2e, same commit, credited the same Bedrock
request to unrelated tests 92% of the time, and Bedrock never converged past a
~20% hit rate while OpenAI, whose deployments are per test, sat at 90%.

A deployment registered from inside a test now carries its test's slug in the
edge URL it is pointed at, `{edge}/{mount}/t/{slug}`, and the edge reads that
segment off every request before forwarding. A request without one is forwarded
live and never cached, and the edge no longer falls back to process state. The
compat aliases are registered with provider_live=True and stay on their real
provider path: no single test owns them, and the matrix exists to prove the real
CLI against real providers.
2026-09-16 17:08:17 -07:00
..
_builder_unit_tests chore(e2e): port the compat-matrix cron publisher to tests/e2e/claude_code 2026-08-10 21:46:56 +00:00
_driver_unit_tests fix(e2e): keep the CLI determinism test out of the in-cluster suite 2026-09-16 16:01:32 -07:00
_probe_unit_tests test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
basic_messaging_non_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
basic_messaging_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
count_tokens refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods (#33760) 2026-07-18 19:03:01 +00:00
cron_vm fix(compat-matrix): keep publish token out of the job-wide process env 2026-08-11 00:57:25 +00:00
long_context_1m test(e2e): harness fixes for stage job green (skips + router/UI/budget) (#33634) 2026-07-16 20:30:30 -07:00
passthrough test(e2e): harness fixes for stage job green (skips + router/UI/budget) (#33634) 2026-07-16 20:30:30 -07:00
pdf_input fix(bedrock): add text block to converse user messages carrying documents 2026-08-11 02:35:49 +00:00
prompt_caching_1h fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
prompt_caching_5m fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
structured_outputs fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
thinking fix(bedrock): preserve adaptive thinking effort through the /v1/messages bridge 2026-08-11 03:23:16 +00:00
thinking_with_tool_use fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
tool_search test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
tool_use test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
tool_use_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
vision fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
web_search fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
__init__.py test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
_basic_messaging.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
_compat_models.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
_env.py fix(e2e): wait for every gateway before using a new model and keep the network rerun 2026-09-05 16:10:40 -07:00
_gpt_cells.py test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
_passthrough.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
cli_driver.py fix(e2e): stage the seeded device id per thread, not per process 2026-09-16 13:47:07 -07:00
conftest.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
http_probe.py test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
manifest.yaml Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_claude_code_e2e_gpt_big3 2026-07-16 15:04:08 -07:00
matrix_builder.py chore(e2e): port the compat-matrix cron publisher to tests/e2e/claude_code 2026-08-10 21:46:56 +00:00
pr_gate_version_resolver.py test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
rate_limiter.py test(e2e/claude_code): add GPT-5.6 Sol/Terra/Luna provider columns for OpenAI, Azure OpenAI, and Bedrock Mantle 2026-07-15 16:25:32 -07:00
run_compat.sh test(e2e/claude_code): update run_compat.sh flag docs to COMPAT_MANTLE_CELLS 2026-07-16 17:26:46 -07:00
sample_compatibility-matrix.json test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
test_config.yaml test(e2e/claude_code): reuse AZURE_API_BASE/KEY for the azure_openai GPT column 2026-07-16 16:45:18 -07:00