litellm/tests/e2e
devin-ai-integration[bot] 1bc7d34a84
chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job (#42608)
* chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job

The daily Claude Code compatibility-matrix job ran as a systemd timer on
the litellm-compatibility-matrix-populator VM in the vertex-check GCP
project. Replace that with a Render Docker cron job built from a new
Dockerfile in tests/e2e/claude_code/cron_vm: pinned and checksummed
debian base, gh, uv, and Claude Code CLI, a non-root populator user, and
run_daily.sh as the entrypoint. run_daily.sh now clones a fresh blobless
checkout per run (Render cron disks are ephemeral), reads the publish PAT
from the github-token secret file under CREDENTIALS_DIRECTORY, and its
comments no longer describe systemd. The .service and .timer units are
gone; README.md and the env example describe the Render service, its
secret files, and the local docker build instead.

* docs(e2e): name the plan and trigger route Render's cron-job API accepts

Render answers a bare 404 for the legacy pro_max plan name on a cron job
(4c-16g is the same 4 CPU / 16 GB size) and the manual trigger route is
/v1/cron-jobs, not /v1/cronjobs.

* fix(e2e): install the published litellm wheel instead of building the tag from source

The tag builds a Rust extension through maturin, which needs a C and Rust
toolchain the cron image does not carry, so the first Render run failed at
uv sync with "linker cc not found". Sync the locked dependencies with
--no-install-project, install the PyPI wheel (what users run) with
--no-build, and pass --no-sync to every uv run so uv never puts the source
build back.

* fix(e2e): keep the SKIP_PUBLISH matrix where a Render run can read it

The validation run wrote the matrix into the image checkout, which nobody
can read once the container exits. Save it under HOME and print it at the
end of the log instead.

* fix(e2e): let the stale compat-matrix PR sweep see past the newest 100 docs PRs

The docs repo has a few hundred open PRs, so a 100-item list never
reached the week-old compat-matrix PR and the sweep left it open on
every run.

* docs(e2e): say the Render cron needs a manual deploy after each merge

Pushes never started a deploy during setup because Render only hears
about them through its GitHub app, which the org does not have, so the
README now carries the deploy command and the wait-for-live rule

* ci: build the compat-matrix cron image on pull requests

The CI coverage gate requires every Dockerfile to be built by a job, and
building this one on each PR that touches it also catches a broken pin
or checksum before Render does

* fix(e2e): shim the whole tests/e2e tree into the compat-matrix worktree

The five-file helper allowlist missed fixture_mode, which e2e_config now
imports, so the first Render run died at conftest load with
ModuleNotFoundError. Copy the image's whole tests/e2e tree instead and
keep pytest from loading the EKS-harness conftest with --confcutdir

* fix(e2e): scope the compat-matrix sweep to the publishing account's own PRs

The stale-PR sweep selected every open docs PR whose head branch starts
with compat-matrix/, so a contributor's fork PR under that name would
have been closed once a newer matrix PR existed. The sweep now resolves
the publishing login from the token and only closes same-repo PRs that
account opened

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-22 19:43:31 -07:00
..
a2a test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
access_control Merge remote-tracking branch 'github/main' into litellm_rust_bridge_declarative_route_catalog 2026-09-17 11:08:08 -07:00
batches fix(bedrock): treat blank AWS_S3_* env vars as unset for batch jobs (#42528) 2026-09-22 17:55:22 -07:00
claude_code chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job (#42608) 2026-09-22 19:43:31 -07:00
coverage_registry test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
gateway test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
guardrails feat(proxy): opt-in include_guardrail_response returns guardrail_information in the response (#42327) 2026-09-22 12:43:30 -07:00
llm_translation fix(e2e): route credential, cost map, and UI login calls to the control plane (#42506) 2026-09-22 12:59:56 -07:00
load chore: merge litellm_internal_staging into litellm_e2e_reliability_module_cells 2026-09-12 13:05:02 -07:00
logging fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters (#42106) 2026-09-22 20:50:53 +00:00
management fix(proxy): write key deleted audit logs for cascade and alias key deletions (#42446) 2026-09-22 11:52:08 -07:00
mcp test(e2e): restore LIT-3467 implementation for rework 2026-09-19 16:21:53 -07:00
migrations test(migrations): close the gaps the upgrade assertions left open 2026-09-21 13:15:46 -07:00
other fix(jwt): accept a team alias in x-litellm-team-id (#42445) 2026-09-22 02:22:00 -07:00
quota_management feat(logging): add normalized_error cluster key to error_information (#41715) 2026-09-22 15:56:50 -07:00
router test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
secret_manager test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
ui test(e2e-ui): check the MCP Tools tab against the upstream's own tools/list (#42397) 2026-09-21 20:35:30 -07:00
AGENTS.md test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
conftest.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
CONTRIBUTING.md test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
e2e_config.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
fixture_bundle.py test: add strict stateless provider replay identity 2026-09-14 16:55:43 -07:00
fixture_canonical.py feat(e2e): key the provider cache per test and mount Bedrock behind it 2026-09-16 02:15:46 -07:00
fixture_mode.py fix(e2e): own a shared fixture's deployment by the fixture's node, not the first test 2026-09-16 17:35:05 -07:00
fixture_profile.py test: preserve strict replay numeric spelling 2026-09-14 17:11:56 -07:00
idp.py test(e2e): restore LIT-3467 implementation for rework 2026-09-19 16:21:53 -07:00
idp_realm.json test(e2e): harden JWT fixtures and cover management lifecycles 2026-09-11 16:36:12 -07:00
junit_properties.py test: bind management E2E callers and isolate JWT actors 2026-09-12 13:29:04 -07:00
lifecycle.py fix(e2e): clean up batch files reliably and expire Azure inputs 2026-09-07 12:15:06 -07:00
memory_readings.py test(e2e): hold every worker under an idle RSS budget before any traffic (#42552) 2026-09-22 14:47:14 -07:00
models.py feat(logging): add normalized_error cluster key to error_information (#41715) 2026-09-22 15:56:50 -07:00
otel_client.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00
PROVIDER_CACHE.md fix(e2e): own a shared fixture's deployment by the fixture's node, not the first test 2026-09-16 17:35:05 -07:00
provider_cache.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
provider_cache_redis.py chore(e2e): report the key components behind a mount that never converges 2026-09-16 15:01:08 -07:00
provider_cache_routing.py revert(e2e): unmount Gemini, its api_base means two things 2026-09-16 08:37:25 -07:00
provider_edge.py fix(responses): forward safety_identifier through the chat completion bridge 2026-09-21 04:09:59 +00:00
provider_edge_bedrock.py feat(e2e): cache the responses and embeddings endpoints behind the edge 2026-09-16 02:34:09 -07:00
proxy_client.py test(e2e): hold every worker under an idle RSS budget before any traffic (#42552) 2026-09-22 14:47:14 -07:00
pytest.ini test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
stack_lock.py test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
test_e2e_http.py test: bind management E2E callers and isolate JWT actors 2026-09-12 13:29:04 -07:00
test_fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
test_fixture_canonical.py test(e2e): pin query params and multipart form fields as replay match-key identity 2026-08-20 15:59:34 -04:00
test_fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
test_idp.py test: enforce isolated actors and stop OIDC process groups 2026-09-12 13:49:49 -07:00
test_junit_properties.py test(e2e): read JUnit properties off the real collected pytest Item 2026-09-01 19:12:07 -07:00
test_provider_edge.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
test_proxy_client.py test(e2e): assert a cooldown reaches a sibling replica within the 1s Redis read interval (#42422) 2026-09-22 12:42:41 -07:00
test_stack_lock.py test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
transport.py fix(e2e): route credential, cost map, and UI login calls to the control plane (#42506) 2026-09-22 12:59:56 -07:00