# Render cron job for the Claude Code compatibility-matrix populator

The populator runs daily as the Render cron job `litellm-compat-matrix`
(Docker runtime, built from the `Dockerfile` in this directory) rather
than as a GitHub Action or on a dedicated VM. Trade-offs:

- ✅ No machine to keep on or patch. Render builds the image from this
  directory on every push to `main` that touches `tests/e2e/**` and
  runs it on the schedule.
- ✅ Credentials live in Render env vars and secret files, scoped to
  this one service, instead of on a VM filesystem.
- ✅ The publish token still uses the `mateo-berri` account, which is a
  collaborator on `BerriAI/litellm-docs`, so no GitHub App with
  `pull-requests: write` has to be provisioned.
- ⚠️ The disk is ephemeral, so every run starts from a fresh (blobless)
  clone of litellm plus a cold `uv sync`. That adds a few minutes on
  top of the ~10 minute test run; the job's 12 hour ceiling is nowhere
  near.
- ✅ The Claude Code CLI under test is chosen on every run (the newest
  npm release published at least 3 days ago) and downloaded
  checksum-verified, so the matrix follows CLI releases without a PR;
  see the gotchas for pinning a run.

## Layout

| File | Purpose |
| --- | --- |
| `Dockerfile` | The image Render builds: Debian bookworm-slim plus pinned, checksum-verified `gh` and `uv`, with this `tests/e2e/` tree copied to `/opt/litellm/tests/e2e/`. Runs as the non-root user `populator` (uid/gid 1000, which is what Render's secret files are readable by). |
| `run_daily.sh` | The actual cron job. Resolves versions, clones the worktree, installs the Claude Code CLI under test, boots the proxy, runs pytest, builds the JSON, opens (or updates) a docs PR, sweeps stale compat-matrix PRs. |
| `install_claude_code.sh` | Downloads one Claude Code release (`<version> <dest-dir>`) from the vendor's native release channel, verifies it against the sha256 in that release's `manifest.json`, and refuses a binary whose `--version` disagrees. Run by the cron and by the `compat-matrix-image` GitHub workflow. |
| `build_matrix.py` | Tiny Python CLI that wraps `claude_code.matrix_builder.build_from_paths`. Exists only because the bash script needs *some* way to render the per-cell aggregation, and the builder is already Python. |
| `check_regressions.py` | Tiny Python CLI that wraps `claude_code.matrix_builder.find_regressions`. Diffs the freshly built matrix against the currently-published one and exits `3` if any cell flipped green→red, which gates auto-merge. |
| `litellm-compat-matrix.env.example` | The service's env vars, one per line, with what each is for. |

## What `run_daily.sh` does

1. **Resolves the latest LiteLLM final release tag** (newest bare
   `vX.Y.Z`, skipping `-rc.N`/`-dev.N` pre-releases) by paging the
   GitHub Releases API (`curl | jq`).
2. **Clones the worktree** at `~/litellm-cron-worktree/` (a
   `--filter=blob:none` clone, so only the checked-out tag's blobs are
   fetched), `git checkout --force <tag>`, then `uv sync --frozen
   --no-install-project` against a uv-managed CPython 3.12 followed by
   `uv pip install --no-build litellm==<version>`, so the proxy under
   test is the published PyPI wheel (what users install) rather than a
   source build: the tag builds a Rust extension through maturin, and
   the image ships no C or Rust toolchain. Then **shims the test suite**:
   `tests/e2e/` in the worktree is replaced by the image's copy of this
   whole tree, so the cron always runs *today's* tests against the
   latest stable proxy, and pytest runs with `--confcutdir` pointed at
   `claude_code/` so the tree's EKS-harness `conftest.py` (whose imports
   the stable venv doesn't install) is never loaded. The tag's own
   `tests/e2e/` is deliberately not used.
3. **Resolves and installs the Claude Code CLI under test**:
   `pr_gate_version_resolver.py` (run on the venv, the image has no
   Python of its own) picks the newest `@anthropic-ai/claude-code` npm
   release published at least 3 days ago, the same buffer the PR gate
   uses, unless `CLAUDE_CODE_VERSION` pins one, and
   `install_claude_code.sh` downloads that release's `linux-x64` binary
   into the run's scratch dir, verified against the release manifest.
4. **Boots the proxy** as a `setsid` background process on port `4100`
   bound to loopback, then polls `/health/liveliness` until it's up.
5. **Runs pytest** on `tests/e2e/claude_code/` with that CLI first on
   `PATH`, `LITELLM_PROXY_URL` pointed at the proxy, and
   `COMPAT_RESULTS_PATH` set so the conftest hook writes the per-test
   results artifact. Test failures become `fail` cells in the JSON, not
   script errors.
6. **Builds `compatibility-matrix.json`** by handing the artifact +
   manifest to `build_matrix.py`.
7. **Opens or updates a docs PR**: `gh repo clone` of `litellm-docs`
   into a tempdir, deterministic head branch
   (`compat-matrix/<litellm-version>-<claude-code-version>-<UTC-date>`),
   `--force` push **directly to `BerriAI/litellm-docs`** (the
   `mateo-berri` token has write access, so this is a same-repo branch,
   not a fork), `gh pr create`. A re-run on the same day fast-forwards
   the existing branch and `gh pr create` no-ops ("a pull request for
   branch ... already exists" is treated as success). If the JSON is
   byte-identical to what `main` already publishes, the push is skipped
   entirely. These PRs are not gated on a second human review.

   **Auto-merge is gated on a regression check**: before enabling
   auto-merge, `check_regressions.py` diffs the new matrix against the
   one currently on `main`. Auto-merge (`gh pr merge --auto --squash`)
   is only enabled when **no cell flipped green→red** — i.e. every
   transition is red→green, green→green, or red→red. A pre-existing red
   cell (e.g. a provider that's out of API credits) is `red→red` and
   does **not** block; only a `pass`→`fail` flip does. When a regression
   is detected the PR is still opened/updated (with a warning banner
   naming the offending cells) but auto-merge is left **off** — and any
   auto-merge a prior same-day run enabled is explicitly disabled — so a
   human reviews before it lands on the public table. The check fails
   *closed*: if it errors, auto-merge is withheld.
8. **Sweeps stale compat-matrix PRs**: once today's PR exists, every
   other open `compat-matrix/*` PR that the publishing account opened
   from a branch on the docs repo itself is closed (and its bot-owned
   branch deleted), so at most one compat-matrix PR is ever open — the
   newest. A contributor's PR under that prefix is never touched.

## The Render service

Everything below is what the live service is set to; recreate it with
the same values if it ever has to be rebuilt.

| Setting | Value |
| --- | --- |
| Workspace | Litellm (the one that already builds the other litellm services) |
| Type | Cron job, Docker runtime |
| Repo / branch | `BerriAI/litellm` @ `main` |
| Dockerfile path | `tests/e2e/claude_code/cron_vm/Dockerfile` |
| Docker build context | `tests/e2e` (the repo root `.dockerignore` excludes `tests`, so the context has to start below it) |
| Build filter | included paths `tests/e2e/**` |
| Schedule | `0 6 * * *` (06:00 UTC daily) |
| Plan / region | `4c-16g` (4 CPU, 16 GB, what the dashboard calls Pro Max; the suite fans out to ~75 concurrent CLI calls) / Oregon |
| Env vars | every key in `litellm-compat-matrix.env.example` |
| Secret files | `github-token` (the publish PAT, one line) and `vertex-service-account.json` (the Vertex service-account key) |

Render mounts secret files at `/etc/secrets/<name>`, which is where
`CREDENTIALS_DIRECTORY` and `GOOGLE_APPLICATION_CREDENTIALS` in the env
example point. Render also passes env vars to `docker build` as build
args, which is why the `Dockerfile` declares no `ARG` that could ever
be given a secret's name.

Creating it through the API looks like this (fill `envVars` and
`secretFiles` from the env example and the two secrets; `ownerId` is
the workspace id from `GET /v1/owners`):

```bash
curl -fsS https://api.render.com/v1/services \
  -H "Authorization: Bearer ${RENDER_API_KEY}" \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "cron_job",
    "name": "litellm-compat-matrix",
    "ownerId": "<workspace id>",
    "repo": "https://github.com/BerriAI/litellm",
    "branch": "main",
    "autoDeploy": "yes",
    "buildFilter": {"paths": ["tests/e2e/**"], "ignoredPaths": []},
    "envVars": [{"key": "ANTHROPIC_API_KEY", "value": "..."}],
    "secretFiles": [{"name": "github-token", "content": "..."},
                    {"name": "vertex-service-account.json", "content": "..."}],
    "serviceDetails": {
      "runtime": "docker",
      "schedule": "0 6 * * *",
      "plan": "4c-16g",
      "region": "oregon",
      "envSpecificDetails": {
        "dockerfilePath": "tests/e2e/claude_code/cron_vm/Dockerfile",
        "dockerContext": "tests/e2e"
      }
    }
  }'
```

## Operating it

```bash
# Trigger a real run right now (PRs to litellm-docs). The id is the
# service id (`crn-...`) from the dashboard URL or `GET /v1/services`.
curl -fsS -X POST "https://api.render.com/v1/cron-jobs/${CRON_ID}/runs" \
  -H "Authorization: Bearer ${RENDER_API_KEY}"

# Follow a run: the Logs tab on the service, or the API.
curl -fsS "https://api.render.com/v1/logs?ownerId=${OWNER_ID}&resource=${CRON_ID}&limit=100" \
  -H "Authorization: Bearer ${RENDER_API_KEY}"

# Rebuild the image after a merge that touches tests/e2e/** (see the
# auto-deploy gotcha below). The deploy is done once its status is
# `live`; a run triggered before that still uses the previous image.
curl -fsS -X POST "https://api.render.com/v1/services/${CRON_ID}/deploys" \
  -H "Authorization: Bearer ${RENDER_API_KEY}" \
  -H 'Content-Type: application/json' -d '{"clearCache": "do_not_clear"}'
curl -fsS "https://api.render.com/v1/services/${CRON_ID}/deploys?limit=1" \
  -H "Authorization: Bearer ${RENDER_API_KEY}"

# A run that does NOT open a PR (first-time validation, a CLI pinned
# with CLAUDE_CODE_VERSION):
# set SKIP_PUBLISH=1 on the service, trigger a run, then remove it.
# The matrix JSON is printed at the end of the run's log (nothing on
# the container's disk outlives the run) and saved to
# ~/compatibility-matrix.json for a local docker run.
# PYTEST_K='basic_messaging_non_streaming and anthropic' narrows the
# run to one cell the same way.

# Build and run the image locally (docker on Apple silicon needs the
# platform flag; the context is tests/e2e, see the table above).
docker build --platform linux/amd64 \
  -f tests/e2e/claude_code/cron_vm/Dockerfile -t compat-matrix tests/e2e
docker run --rm --platform linux/amd64 \
  --env-file litellm-compat-matrix.env -e SKIP_PUBLISH=1 \
  -v "$PWD/secrets:/etc/secrets:ro" compat-matrix
```

## Gotchas

- **The venv is pinned to Python 3.12 (`CRON_PYTHON_VERSION`).** The
  e2e suite uses PEP 695 `type` aliases, which the image's Debian
  Python can't parse; `run_daily.sh` has uv fetch a managed CPython
  into `~/litellm-cron-worktree/.uv-python/` and syncs the venv against
  it.
- **The proxy port is `4100`, not `4000`.** Kept from the VM days so a
  developer running the script locally next to their own `:4000` proxy
  doesn't collide. Override with `PROXY_PORT=...`.
- **`uv sync --frozen` requires the resolved tag to be tagged on
  GitHub, and the wheel install requires it on PyPI.** If the latest
  stable release was made but not pushed as a git tag, the `git
  checkout` step fails; push the tag, then rerun. PyPI has had every
  stable version days before its GitHub release so far (1.102.0 was
  uploaded 2026-09-20, released on GitHub 2026-09-22), so the
  `--no-build` install failing means the wheel is genuinely missing,
  not late.
- **Pushes do not redeploy the service; deploy by hand.** `autoDeploy`
  is `yes` on the service, but Render only hears about pushes through
  its GitHub app, which is not installed on the `BerriAI` org (an org
  admin step), so no push to the branch has ever started a deploy.
  After a merge that changes anything under `tests/e2e/**`, run the
  deploy command from the operating section (or "Manual Deploy" on the
  dashboard) and wait for `live` before triggering a run, otherwise
  the next scheduled run still executes the old image.
- **Publish-token rotation is your problem.** The cron does not
  refresh the token; if `mateo-berri`'s PAT in the `github-token`
  secret file expires, the run fails at the `git push`/`gh pr create`
  step with a 401 ("Bad credentials" / "Authentication failed"). Mint
  a fresh PAT and replace the secret file on the service. The token
  needs write access to `BerriAI/litellm-docs` (classic `repo` scope,
  or fine-grained Contents:RW + Pull requests:RW). It is delivered as
  a file, not an env var, so pytest, the proxy, and the claude CLI
  never inherit it; manual runs export `GITHUB_TOKEN` instead.
- **The Claude Code CLI is chosen per run, not pinned.** Each run
  tests the newest `@anthropic-ai/claude-code` npm release published
  at least 3 days ago, downloaded from
  `https://downloads.claude.ai/claude-code-releases/<version>/linux-x64/claude`
  and verified against the sha256 in that release's `manifest.json`.
  A CLI release that breaks a cell shows up as a green→red flip, which
  withholds auto-merge on that day's docs PR for review. To rerun the
  matrix on one specific CLI, set `CLAUDE_CODE_VERSION` on the run.
  `gh` and `uv` stay pinned in the `Dockerfile`; bump them in a PR with
  the checksum from the release's `gh_<version>_checksums.txt` and the
  tarball's `.sha256` sidecar respectively.
- **A local build on Apple silicon only proves the image assembles.**
  Under QEMU the Claude Code binary (a Bun executable) dies with
  `CPU lacks AVX support` and `gh` panics in the Go runtime, so the CLI
  download and `claude --version` are verified by the
  `compat-matrix-image` GitHub workflow (an x86 runner that builds the
  image and runs `install_claude_code.sh` in it on every PR touching
  this directory) and a full run with a `SKIP_PUBLISH=1` run on Render,
  not locally.
- **Nothing persists between runs.** A failed run leaves no
  half-installed venv behind, but also no cache: don't expect a rerun
  to be faster than the first one.