litellm/tests/e2e/claude_code
devin-ai-integration[bot] 1bc7d34a84
chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job (#42608)
* chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job

The daily Claude Code compatibility-matrix job ran as a systemd timer on
the litellm-compatibility-matrix-populator VM in the vertex-check GCP
project. Replace that with a Render Docker cron job built from a new
Dockerfile in tests/e2e/claude_code/cron_vm: pinned and checksummed
debian base, gh, uv, and Claude Code CLI, a non-root populator user, and
run_daily.sh as the entrypoint. run_daily.sh now clones a fresh blobless
checkout per run (Render cron disks are ephemeral), reads the publish PAT
from the github-token secret file under CREDENTIALS_DIRECTORY, and its
comments no longer describe systemd. The .service and .timer units are
gone; README.md and the env example describe the Render service, its
secret files, and the local docker build instead.

* docs(e2e): name the plan and trigger route Render's cron-job API accepts

Render answers a bare 404 for the legacy pro_max plan name on a cron job
(4c-16g is the same 4 CPU / 16 GB size) and the manual trigger route is
/v1/cron-jobs, not /v1/cronjobs.

* fix(e2e): install the published litellm wheel instead of building the tag from source

The tag builds a Rust extension through maturin, which needs a C and Rust
toolchain the cron image does not carry, so the first Render run failed at
uv sync with "linker cc not found". Sync the locked dependencies with
--no-install-project, install the PyPI wheel (what users run) with
--no-build, and pass --no-sync to every uv run so uv never puts the source
build back.

* fix(e2e): keep the SKIP_PUBLISH matrix where a Render run can read it

The validation run wrote the matrix into the image checkout, which nobody
can read once the container exits. Save it under HOME and print it at the
end of the log instead.

* fix(e2e): let the stale compat-matrix PR sweep see past the newest 100 docs PRs

The docs repo has a few hundred open PRs, so a 100-item list never
reached the week-old compat-matrix PR and the sweep left it open on
every run.

* docs(e2e): say the Render cron needs a manual deploy after each merge

Pushes never started a deploy during setup because Render only hears
about them through its GitHub app, which the org does not have, so the
README now carries the deploy command and the wait-for-live rule

* ci: build the compat-matrix cron image on pull requests

The CI coverage gate requires every Dockerfile to be built by a job, and
building this one on each PR that touches it also catches a broken pin
or checksum before Render does

* fix(e2e): shim the whole tests/e2e tree into the compat-matrix worktree

The five-file helper allowlist missed fixture_mode, which e2e_config now
imports, so the first Render run died at conftest load with
ModuleNotFoundError. Copy the image's whole tests/e2e tree instead and
keep pytest from loading the EKS-harness conftest with --confcutdir

* fix(e2e): scope the compat-matrix sweep to the publishing account's own PRs

The stale-PR sweep selected every open docs PR whose head branch starts
with compat-matrix/, so a contributor's fork PR under that name would
have been closed once a newer matrix PR existed. The sweep now resolves
the publishing login from the token and only closes same-repo PRs that
account opened

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-22 19:43:31 -07:00
..
_builder_unit_tests chore(e2e): port the compat-matrix cron publisher to tests/e2e/claude_code 2026-08-10 21:46:56 +00:00
_driver_unit_tests fix(e2e): keep the CLI determinism test out of the in-cluster suite 2026-09-16 16:01:32 -07:00
_probe_unit_tests test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
basic_messaging_non_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
basic_messaging_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
count_tokens refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods (#33760) 2026-07-18 19:03:01 +00:00
cron_vm chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job (#42608) 2026-09-22 19:43:31 -07:00
long_context_1m test(e2e): harness fixes for stage job green (skips + router/UI/budget) (#33634) 2026-07-16 20:30:30 -07:00
passthrough test(e2e): harness fixes for stage job green (skips + router/UI/budget) (#33634) 2026-07-16 20:30:30 -07:00
pdf_input fix(bedrock): add text block to converse user messages carrying documents 2026-08-11 02:35:49 +00:00
prompt_caching_1h fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
prompt_caching_5m fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
structured_outputs fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
thinking fix(bedrock): preserve adaptive thinking effort through the /v1/messages bridge 2026-08-11 03:23:16 +00:00
thinking_with_tool_use fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
tool_search test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
tool_use test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
tool_use_streaming test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
vision fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
web_search fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
__init__.py test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
_basic_messaging.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
_compat_models.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
_env.py fix(e2e): wait for every gateway before using a new model and keep the network rerun 2026-09-05 16:10:40 -07:00
_gpt_cells.py test(e2e): skip flaky OpenAI GPT cells; raise multi-window max_tokens (#33799) 2026-07-17 19:07:18 -07:00
_passthrough.py fix(e2e/claude_code): unblock stage collection, align proxy env names, register compat models (#33433) 2026-07-16 11:05:31 -07:00
cli_driver.py fix(e2e): stage the seeded device id per thread, not per process 2026-09-16 13:47:07 -07:00
conftest.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
http_probe.py test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
manifest.yaml Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_claude_code_e2e_gpt_big3 2026-07-16 15:04:08 -07:00
matrix_builder.py chore(e2e): port the compat-matrix cron publisher to tests/e2e/claude_code 2026-08-10 21:46:56 +00:00
pr_gate_version_resolver.py test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
rate_limiter.py test(e2e/claude_code): add GPT-5.6 Sol/Terra/Luna provider columns for OpenAI, Azure OpenAI, and Bedrock Mantle 2026-07-15 16:25:32 -07:00
run_compat.sh test(e2e/claude_code): update run_compat.sh flag docs to COMPAT_MANTLE_CELLS 2026-07-16 17:26:46 -07:00
sample_compatibility-matrix.json test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
test_config.yaml test(e2e/claude_code): reuse AZURE_API_BASE/KEY for the azure_openai GPT column 2026-07-16 16:45:18 -07:00