litellm/tests/e2e/claude_code/cron_vm

Cron VM setup for the Claude Code compatibility-matrix populator

The populator runs daily on a dedicated GCP VM (litellm-compatibility-matrix-populator) rather than as a GitHub Action. Trade-offs:

• ✅ Real VM means we can gh auth login against an account that's already a collaborator on BerriAI/litellm-docs, instead of provisioning a GitHub App with pull-requests: write.
• ✅ Persistent state (a single ~/litellm-cron-worktree/ and its .venv) is reused across runs, so each daily run does a fast git checkout + incremental uv sync rather than a fresh clone + cold sync.
• ✅ No Docker dependency — the proxy runs directly via uv run litellm.
• ⚠️ The VM has to actually be on. systemd's Persistent=true recovers from short outages, but a multi-day outage means the matrix goes stale until the VM is back.
• ⚠️ Provider credentials live on the VM filesystem (/etc/litellm-compat-matrix.env) instead of GitHub secrets. Treat the VM as an environment with comparable blast radius to a CI runner.

This directory used to live at tests/claude_code/cron_vm/ (paired with the standalone tests/claude_code/ suite); it now runs the maintained tests/e2e/claude_code/ suite instead. The pytest env interface changed accordingly: the runner exports LITELLM_PROXY_URL / LITELLM_MASTER_KEY (previously LITELLM_PROXY_BASE_URL / LITELLM_PROXY_API_KEY), the azure column reads AZURE_AI_API_KEY / AZURE_AI_API_BASE (previously AZURE_FOUNDRY_*), and the GPT columns need OPENAI_API_KEY and AZURE_API_BASE / AZURE_API_KEY — see litellm-compat-matrix.env.example.

Layout

File	Purpose
run_daily.sh	The actual cron job. Resolves versions, updates the worktree, boots the proxy, runs pytest, builds the JSON, opens (or updates) a docs PR, sweeps stale compat-matrix PRs.
build_matrix.py	Tiny Python CLI that wraps claude_code.matrix_builder.build_from_paths. Exists only because the bash script needs some way to render the per-cell aggregation, and the builder is already Python.
check_regressions.py	Tiny Python CLI that wraps claude_code.matrix_builder.find_regressions. Diffs the freshly built matrix against the currently-published one and exits 3 if any cell flipped green→red, which gates auto-merge.
litellm-compat-matrix.service	systemd oneshot that invokes run_daily.sh.
litellm-compat-matrix.timer	OnCalendar=*-*-* 06:00:00 UTC, Persistent=true.
litellm-compat-matrix.env.example	Template for /etc/litellm-compat-matrix.env.

What run_daily.sh does

1. Resolves the latest LiteLLM final release tag (newest bare vX.Y.Z, skipping -rc.N/-dev.N pre-releases) by paging the GitHub Releases API (curl | jq).
2. Reads the local Claude Code CLI version via claude --version. The cron does not auto-upgrade the CLI — operators do that out-of-band by running npm install -g @anthropic-ai/claude-code@latest.
3. Updates the persistent worktree at ~/litellm-cron-worktree/: git fetch --tags --force, git reset --hard, git clean -fdx -e .venv -e .uv-bin, git checkout --force <tag>. The .venv is preserved across runs so uv sync --frozen is incremental. Then shims the test suite: tests/e2e/ in the worktree is rebuilt from the dev checkout — the claude_code/ suite plus the five shared transport helpers it imports (proxy_client.py, e2e_http.py, models.py, e2e_config.py, transport.py) — so the cron always runs today's tests against the latest stable proxy. The tag's own tests/e2e/ tree (including the EKS-harness conftest.py, whose imports the stable venv doesn't install) is deliberately not used.
4. Boots the proxy as a setsid background process on port 4100 (so it can't collide with a developer's :4000), then polls /health/liveliness until it's up.
5. Runs pytest on tests/e2e/claude_code/ with LITELLM_PROXY_URL pointed at the proxy and COMPAT_RESULTS_PATH set so the conftest hook writes the per-test results artifact. Test failures become fail cells in the JSON, not script errors.
6. Builds compatibility-matrix.json by handing the artifact + manifest to build_matrix.py.
7. Opens or updates a docs PR: gh repo clone of litellm-docs into a tempdir, deterministic head branch (compat-matrix/<litellm-version>-<claude-code-version>-<UTC-date>), --force push directly to BerriAI/litellm-docs (the mateo-berri token has write access, so this is a same-repo branch, not a fork), gh pr create. A re-run on the same day fast-forwards the existing branch and gh pr create no-ops ("a pull request for branch ... already exists" is treated as success). These PRs are no longer gated on a second human review.
8. Gates auto-merge on a regression check: before enabling auto-merge, check_regressions.py diffs the new matrix against the one currently on main. Auto-merge (gh pr merge --auto --squash) is only enabled when no cell flipped green→red — i.e. every transition is red→green, green→green, or red→red. A pre-existing red cell (e.g. a provider that's out of API credits) is red→red and does not block; only a pass→fail flip does. When a regression is detected the PR is still opened/updated (with a warning banner naming the offending cells) but auto-merge is left off — and any auto-merge a prior same-day run enabled is explicitly disabled — so a human reviews before it lands on the public table. The check fails closed: if it errors, auto-merge is withheld.
9. Sweeps stale compat-matrix PRs: once today's PR exists, every other open compat-matrix/* PR on the docs repo is closed (and its bot-owned branch deleted), so at most one compat-matrix PR is ever open — the newest.

One-time VM setup

Run as mateo on the cron VM:

# 1. Toolchain
sudo apt-get update
sudo apt-get install -y git nodejs npm jq curl
curl -LsSf https://astral.sh/uv/install.sh | sh
sudo apt-get install -y gh   # or follow https://cli.github.com/

# 2. Claude Code CLI (the cron does NOT auto-upgrade this; rerun this
#    line out-of-band when you want a fresh CLI to be tested)
sudo npm install -g @anthropic-ai/claude-code@latest

# 3. Litellm checkout. Used by systemd's WorkingDirectory and as the
#    source of the .service / .timer files. The cron itself runs out
#    of the separate worktree at ~/litellm-cron-worktree/.
mkdir -p ~/litellm
git clone https://github.com/BerriAI/litellm.git ~/litellm/litellm
git -C ~/litellm/litellm checkout litellm_internal_staging

# 4. gh auth — must be a collaborator on BerriAI/litellm-docs.
gh auth login   # follow prompts; pick HTTPS + token paste flow

# 5. Provider credentials + the publish token.
sudo cp ~/litellm/litellm/tests/e2e/claude_code/cron_vm/litellm-compat-matrix.env.example \
        /etc/litellm-compat-matrix.env
sudoedit /etc/litellm-compat-matrix.env   # fill in real values
sudo chmod 0600 /etc/litellm-compat-matrix.env
# The mateo-berri PAT lives in its own file, mapped into the service via
# systemd LoadCredential so it stays out of the test processes' env
# (see the env.example comment for why).
sudo install -m 0600 /dev/null /etc/litellm-compat-matrix-github-token
sudoedit /etc/litellm-compat-matrix-github-token   # single line: the PAT

# 6. systemd units.
sudo cp ~/litellm/litellm/tests/e2e/claude_code/cron_vm/litellm-compat-matrix.service /etc/systemd/system/
sudo cp ~/litellm/litellm/tests/e2e/claude_code/cron_vm/litellm-compat-matrix.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now litellm-compat-matrix.timer

Operating it

# When does it run next?
systemctl list-timers litellm-compat-matrix.timer

# Trigger a real run right now (PRs to litellm-docs).
sudo systemctl start litellm-compat-matrix.service

# Trigger a run that does NOT open a PR (good for first-time validation).
SKIP_PUBLISH=1 ~/litellm/litellm/tests/e2e/claude_code/cron_vm/run_daily.sh

# Narrow to one cell while debugging.
SKIP_PUBLISH=1 PYTEST_K='basic_messaging_non_streaming and anthropic' \
  ~/litellm/litellm/tests/e2e/claude_code/cron_vm/run_daily.sh

# Watch the most recent run.
journalctl -u litellm-compat-matrix.service -f

# Read older runs.
journalctl -u litellm-compat-matrix.service --since '2 days ago'

# Disable until further notice (e.g. while debugging).
sudo systemctl disable --now litellm-compat-matrix.timer

Gotchas

• The venv is pinned to Python 3.12 (CRON_PYTHON_VERSION). The e2e suite uses PEP 695 type aliases, which the VM's system Python (3.11) can't parse; run_daily.sh has uv fetch a managed CPython into ~/litellm-cron-worktree/.uv-python/ and syncs the venv against it. The first run after a version bump is a cold venv rebuild.
• The proxy port is 4100, not 4000. This is so a developer SSH'd into the same VM with their own :4000 proxy doesn't collide with a cron run. Override with PROXY_PORT=... in /etc/litellm-compat-matrix.env if you need to.
• uv sync --frozen requires the resolved tag to be tagged on GitHub. If the latest stable release was made but not pushed as a git tag, the git checkout step fails. Push the tag, then rerun.
• Publish-token rotation is your problem. The cron does not refresh the token; if mateo-berri's PAT in /etc/litellm-compat-matrix-github-token expires, the run fails at the git push/gh pr create step with a 401 ("Bad credentials" / "Authentication failed"). Mint a fresh PAT and update that file. The token needs write access to BerriAI/litellm-docs (classic repo scope, or fine-grained Contents:RW + Pull requests:RW). It is delivered via systemd LoadCredential, not the env file, so pytest, the proxy, and the claude CLI never inherit it; manual runs export GITHUB_TOKEN instead.
• First run after upgrading the Claude Code CLI is the riskiest one. If the new CLI changes its wire format the matrix run can produce systematic failures. Always run with SKIP_PUBLISH=1 after a CLI upgrade before letting the next scheduled fire happen.
• Disk: the worktree's .venv is ~1.3 GB and the .git directory is ~1 GB. Plan for at least 5 GB free on the VM, otherwise uv sync will fail mid-run and leave you with a half-installed venv.