mirror of https://github.com/BerriAI/litellm.git synced 2026-09-23 00:41:40 +00:00

mateo-berri bb60422e7d compat-matrix: replace publisher.py + resolver.py with a bash run_daily.sh

The previous iteration of this PR ported the populator to a Python
module (`publisher.py`) with 8 unit-tested pure helpers for branch
naming and PR-body rendering. After the docker code came out, the
GitHub App auth came out, and the worktree-vs-tempdir decision was
made, what was left was: 'git fetch + git checkout + uv sync + start
a subprocess + run pytest + clone docs + git commit + gh pr create'.
That's a bash script.

This commit replaces 800 lines of Python (publisher.py + resolver.py +
their unit tests) with a 297-line run_daily.sh and a 48-line
build_matrix.py whose only job is to be importable Python that can
call into the matrix_builder we already have. Net deletion: -822 lines.

Removed
-------

  * tests/claude_code/publisher.py — the full Python orchestrator.
    Every code path it had is now in run_daily.sh.
  * tests/claude_code/resolver.py — the GitHub Releases v*-stable
    resolver. Replaced by ~10 lines of jq inside run_daily.sh.
  * tests/claude_code/_publisher_unit_tests/ — both test files. The
    pure helpers they covered (commit message, file allowlist, branch
    name, PR title/body) only existed because publisher.py was Python.
    The bash equivalents are short heredoc strings.

Added
-----

  * tests/claude_code/cron_vm/run_daily.sh — the actual cron job,
    structured as numbered phases (resolve / worktree / proxy /
    pytest / build / publish) so journalctl output is readable.
  * tests/claude_code/cron_vm/build_matrix.py — a 48-line CLI that
    calls the existing matrix_builder.build_from_paths. Kept in
    Python because the builder itself is Python and well-tested.

Modified
--------

  * tests/claude_code/cron_vm/litellm-compat-matrix.service —
    ExecStart now invokes run_daily.sh instead of
    'python -m tests.claude_code.publisher'.
  * tests/claude_code/cron_vm/README.md — updated layout table,
    file roles, and operating commands to match.

Why this is the right shape
---------------------------

  * The failure mode at 06:00 UTC is 'read journalctl, see the literal
    failing command with its + prefix, copy-paste it into a shell to
    reproduce'. Bash makes that immediate; Python's subprocess.run
    output looks similar but the surrounding orchestration is harder
    to step through interactively.
  * Every operation the script does is already a shell command (git,
    uv, gh, jq, curl, pytest). The Python wrapper was translating
    between argv arrays and back.
  * The two pieces that genuinely benefit from being in a typed
    language are matrix_builder (already Python) and the resolver's
    semver sort (now done in jq, with the version_key tuple sort
    inline). 'Already Python' wins, 'tiny jq pipeline' wins.

What's preserved
----------------

  * Idempotency: same (litellm, claude, UTC date) -> same branch ->
    same PR. force-with-lease push, gh-pr-create no-op-on-exists.
  * Byte-identical-JSON early return (git diff --cached --quiet).
  * Per-feature status table in the PR body (jq pipeline mirroring
    the Python pr_body_for_matrix logic).
  * Persistent worktree approach so disk doesn't grow unboundedly.
  * Proxy bound to :4100 to avoid colliding with a developer's :4000.
  * SKIP_PUBLISH=1 and PYTEST_K=... operator escape hatches.

2026-05-06 23:27:14 +00:00

6.5 KiB

Raw Blame History

Cron VM setup for the Claude Code compatibility-matrix populator

The populator runs daily on a dedicated GCP VM (litellm-compatibility-matrix-populator) rather than as a GitHub Action. Trade-offs:

✅ Real VM means we can gh auth login against an account that's already a collaborator on BerriAI/litellm-docs, instead of provisioning a GitHub App with pull-requests: write.
✅ Persistent state (a single ~/litellm-cron-worktree/ and its .venv) is reused across runs, so each daily run does a fast git checkout + incremental uv sync rather than a fresh clone + cold sync.
✅ No Docker dependency — the proxy runs directly via uv run litellm.
⚠️ The VM has to actually be on. systemd's Persistent=true recovers from short outages, but a multi-day outage means the matrix goes stale until the VM is back.
⚠️ Provider credentials live on the VM filesystem (/etc/litellm-compat-matrix.env) instead of GitHub secrets. Treat the VM as an environment with comparable blast radius to a CI runner.

Layout

File	Purpose
`run_daily.sh`	The actual cron job. Resolves versions, updates the worktree, boots the proxy, runs pytest, builds the JSON, opens (or updates) a docs PR.
`build_matrix.py`	Tiny Python CLI that wraps `tests.claude_code.matrix_builder.build_from_paths`. Exists only because the bash script needs some way to render the per-cell aggregation, and the builder is already Python.
`litellm-compat-matrix.service`	systemd oneshot that invokes `run_daily.sh`.
`litellm-compat-matrix.timer`	`OnCalendar=--* 06:00:00 UTC`, `Persistent=true`.
`litellm-compat-matrix.env.example`	Template for `/etc/litellm-compat-matrix.env`.

What `run_daily.sh` does

Resolves the latest LiteLLM v*-stable tag by hitting the GitHub Releases API (curl | jq).
Reads the local Claude Code CLI version via claude --version. The cron does not auto-upgrade the CLI — operators do that out-of-band by running npm install -g @anthropic-ai/claude-code@latest.
Updates the persistent worktree at ~/litellm-cron-worktree/: git fetch --tags --force, git reset --hard, git clean -fdx -e .venv, git checkout --force <tag>. The .venv is preserved across runs so uv sync --frozen is incremental.
Boots the proxy as a setsid background process on port 4100 (so it can't collide with a developer's :4000), then polls /health/liveliness until it's up.
Runs pytest with ANTHROPIC_BASE_URL pointed at the proxy and COMPAT_RESULTS_PATH set so the conftest hook writes the per-test results artifact. Test failures become fail cells in the JSON, not script errors.
Builds compatibility-matrix.json by handing the artifact + manifest to build_matrix.py.
Opens or updates a docs PR: gh repo clone of litellm-docs into a tempdir, deterministic head branch (compat-matrix/<litellm-version>-<claude-code-version>-<UTC-date>), --force-with-lease push, gh pr create. A re-run on the same day fast-forwards the existing branch and gh pr create no-ops ("a pull request for branch ... already exists" is treated as success).

One-time VM setup

Run as mateo on the cron VM:

# 1. Toolchain
sudo apt-get update
sudo apt-get install -y git nodejs npm jq curl
curl -LsSf https://astral.sh/uv/install.sh | sh
sudo apt-get install -y gh   # or follow https://cli.github.com/

# 2. Claude Code CLI (the cron does NOT auto-upgrade this; rerun this
#    line out-of-band when you want a fresh CLI to be tested)
sudo npm install -g @anthropic-ai/claude-code@latest

# 3. Litellm checkout. Used by systemd's WorkingDirectory and as the
#    source of the .service / .timer files. The cron itself runs out
#    of the separate worktree at ~/litellm-cron-worktree/.
mkdir -p ~/litellm
git clone https://github.com/BerriAI/litellm.git ~/litellm/litellm

# 4. gh auth — must be a collaborator on BerriAI/litellm-docs.
gh auth login   # follow prompts; pick HTTPS + token paste flow

# 5. Provider credentials.
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.env.example \
        /etc/litellm-compat-matrix.env
sudoedit /etc/litellm-compat-matrix.env   # fill in real values
sudo chmod 0600 /etc/litellm-compat-matrix.env

# 6. systemd units.
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.service /etc/systemd/system/
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now litellm-compat-matrix.timer

Operating it

# When does it run next?
systemctl list-timers litellm-compat-matrix.timer

# Trigger a real run right now (PRs to litellm-docs).
sudo systemctl start litellm-compat-matrix.service

# Trigger a run that does NOT open a PR (good for first-time validation).
SKIP_PUBLISH=1 ~/litellm/litellm/tests/claude_code/cron_vm/run_daily.sh

# Narrow to one cell while debugging.
SKIP_PUBLISH=1 PYTEST_K='basic_messaging_non_streaming and anthropic' \
  ~/litellm/litellm/tests/claude_code/cron_vm/run_daily.sh

# Watch the most recent run.
journalctl -u litellm-compat-matrix.service -f

# Read older runs.
journalctl -u litellm-compat-matrix.service --since '2 days ago'

# Disable until further notice (e.g. while debugging).
sudo systemctl disable --now litellm-compat-matrix.timer

Gotchas

The proxy port is 4100, not 4000. This is so a developer SSH'd into the same VM with their own :4000 proxy doesn't collide with a cron run. Override with PROXY_PORT=... in /etc/litellm-compat-matrix.env if you need to.
uv sync --frozen requires the resolved tag to be tagged on GitHub. If the latest stable release was made but not pushed as a git tag, the git checkout step fails. Push the tag, then rerun.
gh auth token rotation is your problem. The cron does not refresh the token; if the bot account's PAT expires the run will fail at gh repo clone with a 401. Re-run gh auth login.
First run after upgrading the Claude Code CLI is the riskiest one. If the new CLI changes its wire format the matrix run can produce systematic failures. Always run with SKIP_PUBLISH=1 after a CLI upgrade before letting the next scheduled fire happen.
Disk: the worktree's .venv is ~1.3 GB and the .git directory is ~1 GB. Plan for at least 5 GB free on the VM, otherwise uv sync will fail mid-run and leave you with a half-installed venv.

6.5 KiB Raw Blame History