Cursor security review flagged that the cron VM downloads the uv
release tarball and pipes it straight through `tar -xzO ... > file ;
chmod +x`, with no integrity check. A tampered release artifact would
execute in a credentialed cron context with access to the docs-repo
push token.
Download the tarball + the official .sha256 sidecar Astral publishes
alongside every uv release to a temp dir, run `sha256sum -c` against
the sidecar, and only extract+install on success. On mismatch we wipe
the tempdir and `die` with a clear refusal.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Bugbot flagged that the resolver fetches only the first page (default
30 entries) of the releases endpoint. LiteLLM ships multiple non-stable
releases per day, so 30+ non-stable releases between consecutive
v*-stable tags is routinely the case — when it happens, the jq filter
matches nothing, LITELLM_VERSION is empty, and the daily matrix update
silently dies.
Walk pages 1..5 (100 per page = 500 releases max) and short-circuit as
soon as a page contains at least one v*-stable tag. Same final jq
filter, same sort order.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Add pytest_sessionstart hook in tests/claude_code/conftest.py that
clears the module-level _COLLECTOR so results don't leak across
pytest.main() invocations within the same process.
- Export LITELLM_MASTER_KEY=${PROXY_API_KEY} when launching the cron
VM proxy so the auth token the tests send actually matches what
the proxy expects.
The resolver picks the latest `v*-stable` tag of BerriAI/litellm. Until
the compat matrix work itself lands in a stable release, that tag's
tree won't contain `tests/claude_code/` at all — the proxy config and
test files only exist on the work-in-progress stack. Without a shim,
the cron dies with 'proxy config not found at .../test_config.yaml'
on every run.
Fix: after `git checkout <tag>`, if
`<worktree>/tests/claude_code/test_config.yaml` is missing, copy the
directory from ${LITELLM_REPO} (the dev checkout, which has the
in-flight matrix work). The `git clean -e tests/claude_code` line
preserves the shim across runs.
Once the matrix work is in the resolved tag, the if-branch is a no-op
and the shim is never used. No code path needs to be removed later;
the bridge self-disables.
Tier 1 (single anthropic cell) and tier 2 (full
basic_messaging_non_streaming row across 5 providers) now run cleanly
from `run_daily.sh` on the GCP VM. Five issues showed up during
validation; each is fixed in this commit.
1. uv version pin
----------------
The litellm worktree pins an exact uv version in
`pyproject.toml`'s `[tool.uv] required-version` field. The cron
VM's system uv (currently 0.11.8) refused to sync against the
v1.83.10-stable lockfile (which pins ==0.10.9). Fix: parse the
pinned version out of the worktree's pyproject.toml, download the
matching standalone binary into `<worktree>/.uv-bin/uv-<version>`,
and use it for sync/run/proxy. Cached across runs.
2. Missing proxy + dev extras
---------------------------
`uv sync --frozen` only installed the base dependency set, so
`uv run litellm` died at startup with
`ModuleNotFoundError: No module named 'websockets'`. Per the
repo's AGENTS.md the canonical incantation is
`uv sync --frozen --group proxy-dev --extra proxy`.
3. Wrong env vars for the test driver
-----------------------------------
The script was setting `ANTHROPIC_BASE_URL` and
`ANTHROPIC_AUTH_TOKEN`, which is what Claude Code itself reads,
but the test files read `LITELLM_PROXY_BASE_URL` and
`LITELLM_PROXY_API_KEY` (search the test_config-driven test files
for `PROXY_BASE_URL_ENV = "LITELLM_PROXY_BASE_URL"`). Tests were
marking themselves `fail` with
"missing required env: set LITELLM_PROXY_BASE_URL...". Fix: rename
the two env vars in the pytest invocation. The driver still
propagates them onward as ANTHROPIC_BASE_URL/AUTH_TOKEN to Claude.
4. Cleanup couldn't find the proxy
--------------------------------
The previous setup did
`( ... && setsid uv run litellm ... ) &; PROXY_PID=$!`. `setsid`
detaches the inner uv into its own session, but `$!` records the
PID of the outer subshell, not the long-lived python proxy. So
`kill -TERM "-${PROXY_PID}"` in the EXIT trap targeted the wrong
pgid and the proxy survived as an orphan whenever the script was
killed externally. Fix: replace the subshell with
`setsid bash -c '...'` that writes $$ to a known pid file before
exec'ing the proxy. The cleanup trap reads that file and uses it
as the pgid. Belt-and-braces: the trap also `pgrep -f`s by port
number and SIGKILLs survivors. Trap now fires on `INT TERM` too,
not just `EXIT`.
5. .uv-bin cache survives git clean
---------------------------------
The original `git clean -fdx -e .venv` wiped `.uv-bin/` between
runs, forcing re-download of the pinned uv binary on every
invocation. Now excluded.
Things that worked first try
----------------------------
* Worktree clone + checkout to the resolved tag.
* gh auth on this VM (mateo-berri account, collaborator on
BerriAI/litellm-docs).
* The matrix builder produced a well-formed
`compatibility-matrix.json` with the right per-cell aggregation
even when 4 of 5 cells failed (tier 2 was: anthropic=pass,
bedrock_invoke=fail, bedrock_converse=fail, vertex_ai=fail with a
real 403 from GCP for insufficient scopes, azure=fail with
timeout).
The previous iteration of this PR ported the populator to a Python
module (`publisher.py`) with 8 unit-tested pure helpers for branch
naming and PR-body rendering. After the docker code came out, the
GitHub App auth came out, and the worktree-vs-tempdir decision was
made, what was left was: 'git fetch + git checkout + uv sync + start
a subprocess + run pytest + clone docs + git commit + gh pr create'.
That's a bash script.
This commit replaces 800 lines of Python (publisher.py + resolver.py +
their unit tests) with a 297-line run_daily.sh and a 48-line
build_matrix.py whose only job is to be importable Python that can
call into the matrix_builder we already have. Net deletion: -822 lines.
Removed
-------
* tests/claude_code/publisher.py — the full Python orchestrator.
Every code path it had is now in run_daily.sh.
* tests/claude_code/resolver.py — the GitHub Releases v*-stable
resolver. Replaced by ~10 lines of jq inside run_daily.sh.
* tests/claude_code/_publisher_unit_tests/ — both test files. The
pure helpers they covered (commit message, file allowlist, branch
name, PR title/body) only existed because publisher.py was Python.
The bash equivalents are short heredoc strings.
Added
-----
* tests/claude_code/cron_vm/run_daily.sh — the actual cron job,
structured as numbered phases (resolve / worktree / proxy /
pytest / build / publish) so journalctl output is readable.
* tests/claude_code/cron_vm/build_matrix.py — a 48-line CLI that
calls the existing matrix_builder.build_from_paths. Kept in
Python because the builder itself is Python and well-tested.
Modified
--------
* tests/claude_code/cron_vm/litellm-compat-matrix.service —
ExecStart now invokes run_daily.sh instead of
'python -m tests.claude_code.publisher'.
* tests/claude_code/cron_vm/README.md — updated layout table,
file roles, and operating commands to match.
Why this is the right shape
---------------------------
* The failure mode at 06:00 UTC is 'read journalctl, see the literal
failing command with its + prefix, copy-paste it into a shell to
reproduce'. Bash makes that immediate; Python's subprocess.run
output looks similar but the surrounding orchestration is harder
to step through interactively.
* Every operation the script does is already a shell command (git,
uv, gh, jq, curl, pytest). The Python wrapper was translating
between argv arrays and back.
* The two pieces that genuinely benefit from being in a typed
language are matrix_builder (already Python) and the resolver's
semver sort (now done in jq, with the version_key tuple sort
inline). 'Already Python' wins, 'tiny jq pipeline' wins.
What's preserved
----------------
* Idempotency: same (litellm, claude, UTC date) -> same branch ->
same PR. force-with-lease push, gh-pr-create no-op-on-exists.
* Byte-identical-JSON early return (git diff --cached --quiet).
* Per-feature status table in the PR body (jq pipeline mirroring
the Python pr_body_for_matrix logic).
* Persistent worktree approach so disk doesn't grow unboundedly.
* Proxy bound to :4100 to avoid colliding with a developer's :4000.
* SKIP_PUBLISH=1 and PYTEST_K=... operator escape hatches.
The daily compat-matrix runs from the dedicated GCP VM
`litellm-compatibility-matrix-populator` rather than from a GitHub
Actions runner. The VM has no docker daemon, has `gh` already
authenticated against an account with `pull-requests: write` on
`BerriAI/litellm-docs`, and runs a long-lived litellm checkout we can
reuse across runs. That makes Docker, the GitHub App auth flow, and the
GHA workflow itself dead code.
Removed
-------
* `.github/workflows/claude_code_compat_matrix.yml` — no longer
triggers anything; the systemd timer in this PR owns the daily fire.
* `docker_image_for_tag` + `DOCKER_IMAGE_BASE` constants and the two
unit tests that covered them.
* `_start_proxy(image, port)` / `_stop_proxy(container_id)` /
`docker run` flow, replaced by direct `uv run litellm` subprocess
management with a sigterm-the-process-group teardown.
* `--skip-proxy` CLI flag (was only useful when the GHA workflow
split docker-bringup from publish into separate jobs).
* `docs_token` parameter and `DOCS_REPO_TOKEN` env var; `gh` on
the VM is already authenticated, so we don't pass an explicit
token through the publisher.
Added
-----
* Persistent worktree flow in `publisher.py`. First run clones
`BerriAI/litellm` into `~/litellm-cron-worktree/`; subsequent runs
do `git fetch --tags && git checkout --force <stable-tag> &&
uv sync --frozen`. Disk footprint is bounded because uv sync
removes packages no longer pinned and `git clean -fdx -e .venv`
wipes per-run cruft while keeping the venv around.
* `tests/claude_code/cron_vm/` containing systemd units and a
setup README:
- `litellm-compat-matrix.service` (`Type=oneshot`, runs as the
`mateo` user, sources `/etc/litellm-compat-matrix.env` for
provider creds, hardened with `NoNewPrivileges` /
`ProtectSystem=strict` / `PrivateTmp`);
- `litellm-compat-matrix.timer` (`OnCalendar=*-*-* 06:00:00 UTC`,
`Persistent=true` so a missed run fires when the VM is back up,
`RandomizedDelaySec=10min`);
- `.env.example` documenting the provider-credential surface;
- `README.md` covering one-time install, daily operation,
`journalctl` debugging, and the gotchas (proxy port `4100` to
avoid colliding with a developer's `:4000`, `gh` token
rotation, what to do after a Claude Code CLI upgrade).
Operator notes
--------------
* The proxy now binds `:4100` by default so a developer SSH'd into
the VM with their own `:4000` proxy isn't preempted by the cron.
* The Claude Code CLI is exercised as-is from the system install;
the populator does NOT `npm install` it. Operators upgrade the
CLI by running `npm install -g @anthropic-ai/claude-code@latest`
out of band, typically after watching a `--skip-publish` run to
verify the matrix doesn't suddenly turn red.
* 20 publisher unit tests pass (`pytest
tests/claude_code/_publisher_unit_tests/`).
* End-to-end validation on the VM happens after this PR lands as
follow-up commits on the same branch — the systemd unit is
`Type=oneshot` so a manual `systemctl start` reproduces the cron.
