scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-09 03:18:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 109
litellm/tests/claude_code/cron_vm/README.md
mateo-berri bb60422e7d compat-matrix: replace publisher.py + resolver.py with a bash run_daily.sh 
The previous iteration of this PR ported the populator to a Python
module (`publisher.py`) with 8 unit-tested pure helpers for branch
naming and PR-body rendering. After the docker code came out, the
GitHub App auth came out, and the worktree-vs-tempdir decision was
made, what was left was: 'git fetch + git checkout + uv sync + start
a subprocess + run pytest + clone docs + git commit + gh pr create'.
That's a bash script.

This commit replaces 800 lines of Python (publisher.py + resolver.py +
their unit tests) with a 297-line run_daily.sh and a 48-line
build_matrix.py whose only job is to be importable Python that can
call into the matrix_builder we already have. Net deletion: -822 lines.

Removed
-------

  * tests/claude_code/publisher.py — the full Python orchestrator.
    Every code path it had is now in run_daily.sh.
  * tests/claude_code/resolver.py — the GitHub Releases v*-stable
    resolver. Replaced by ~10 lines of jq inside run_daily.sh.
  * tests/claude_code/_publisher_unit_tests/ — both test files. The
    pure helpers they covered (commit message, file allowlist, branch
    name, PR title/body) only existed because publisher.py was Python.
    The bash equivalents are short heredoc strings.

Added
-----

  * tests/claude_code/cron_vm/run_daily.sh — the actual cron job,
    structured as numbered phases (resolve / worktree / proxy /
    pytest / build / publish) so journalctl output is readable.
  * tests/claude_code/cron_vm/build_matrix.py — a 48-line CLI that
    calls the existing matrix_builder.build_from_paths. Kept in
    Python because the builder itself is Python and well-tested.

Modified
--------

  * tests/claude_code/cron_vm/litellm-compat-matrix.service —
    ExecStart now invokes run_daily.sh instead of
    'python -m tests.claude_code.publisher'.
  * tests/claude_code/cron_vm/README.md — updated layout table,
    file roles, and operating commands to match.

Why this is the right shape
---------------------------

  * The failure mode at 06:00 UTC is 'read journalctl, see the literal
    failing command with its + prefix, copy-paste it into a shell to
    reproduce'. Bash makes that immediate; Python's subprocess.run
    output looks similar but the surrounding orchestration is harder
    to step through interactively.
  * Every operation the script does is already a shell command (git,
    uv, gh, jq, curl, pytest). The Python wrapper was translating
    between argv arrays and back.
  * The two pieces that genuinely benefit from being in a typed
    language are matrix_builder (already Python) and the resolver's
    semver sort (now done in jq, with the version_key tuple sort
    inline). 'Already Python' wins, 'tiny jq pipeline' wins.

What's preserved
----------------

  * Idempotency: same (litellm, claude, UTC date) -> same branch ->
    same PR. force-with-lease push, gh-pr-create no-op-on-exists.
  * Byte-identical-JSON early return (git diff --cached --quiet).
  * Per-feature status table in the PR body (jq pipeline mirroring
    the Python pr_body_for_matrix logic).
  * Persistent worktree approach so disk doesn't grow unboundedly.
  * Proxy bound to :4100 to avoid colliding with a developer's :4000.
  * SKIP_PUBLISH=1 and PYTEST_K=... operator escape hatches.
2026-05-06 23:27:14 +00:00

141 lines
6.5 KiB
Markdown
Raw Blame History

# Cron VM setup for the Claude Code compatibility-matrix populator
The populator runs daily on a dedicated GCP VM
(`litellm-compatibility-matrix-populator`) rather than as a GitHub
Action. Trade-offs:
- ✅ Real VM means we can `gh auth login` against an account that's
already a collaborator on `BerriAI/litellm-docs`, instead of
provisioning a GitHub App with `pull-requests: write`.
- ✅ Persistent state (a single `~/litellm-cron-worktree/` and its `.venv`)
is reused across runs, so each daily run does a fast `git checkout` +
incremental `uv sync` rather than a fresh clone + cold sync.
- ✅ No Docker dependency — the proxy runs directly via `uv run litellm`.
- ⚠️ The VM has to actually be on. systemd's `Persistent=true` recovers
from short outages, but a multi-day outage means the matrix goes
stale until the VM is back.
- ⚠️ Provider credentials live on the VM filesystem
(`/etc/litellm-compat-matrix.env`) instead of GitHub secrets. Treat
the VM as an environment with comparable blast radius to a CI runner.
## Layout
| File | Purpose |
| --- | --- |
| `run_daily.sh` | The actual cron job. Resolves versions, updates the worktree, boots the proxy, runs pytest, builds the JSON, opens (or updates) a docs PR. |
| `build_matrix.py` | Tiny Python CLI that wraps `tests.claude_code.matrix_builder.build_from_paths`. Exists only because the bash script needs *some* way to render the per-cell aggregation, and the builder is already Python. |
| `litellm-compat-matrix.service` | systemd oneshot that invokes `run_daily.sh`. |
| `litellm-compat-matrix.timer` | `OnCalendar=*-*-* 06:00:00 UTC`, `Persistent=true`. |
| `litellm-compat-matrix.env.example` | Template for `/etc/litellm-compat-matrix.env`. |
## What `run_daily.sh` does
1. **Resolves the latest LiteLLM `v*-stable` tag** by hitting the
GitHub Releases API (`curl | jq`).
2. **Reads the local Claude Code CLI version** via `claude --version`.
The cron does not auto-upgrade the CLI — operators do that
out-of-band by running `npm install -g @anthropic-ai/claude-code@latest`.
3. **Updates the persistent worktree** at `~/litellm-cron-worktree/`:
`git fetch --tags --force`, `git reset --hard`,
`git clean -fdx -e .venv`, `git checkout --force <tag>`. The
`.venv` is preserved across runs so `uv sync --frozen` is
incremental.
4. **Boots the proxy** as a `setsid` background process on port `4100`
(so it can't collide with a developer's `:4000`), then polls
`/health/liveliness` until it's up.
5. **Runs pytest** with `ANTHROPIC_BASE_URL` pointed at the proxy and
`COMPAT_RESULTS_PATH` set so the conftest hook writes the per-test
results artifact. Test failures become `fail` cells in the JSON,
not script errors.
6. **Builds `compatibility-matrix.json`** by handing the artifact +
manifest to `build_matrix.py`.
7. **Opens or updates a docs PR**: `gh repo clone` of `litellm-docs`
into a tempdir, deterministic head branch
(`compat-matrix/<litellm-version>-<claude-code-version>-<UTC-date>`),
`--force-with-lease` push, `gh pr create`. A re-run on the same
day fast-forwards the existing branch and `gh pr create` no-ops
("a pull request for branch ... already exists" is treated as
success).
## One-time VM setup
Run as `mateo` on the cron VM:
```bash
# 1. Toolchain
sudo apt-get update
sudo apt-get install -y git nodejs npm jq curl
curl -LsSf https://astral.sh/uv/install.sh | sh
sudo apt-get install -y gh # or follow https://cli.github.com/
# 2. Claude Code CLI (the cron does NOT auto-upgrade this; rerun this
# line out-of-band when you want a fresh CLI to be tested)
sudo npm install -g @anthropic-ai/claude-code@latest
# 3. Litellm checkout. Used by systemd's WorkingDirectory and as the
# source of the .service / .timer files. The cron itself runs out
# of the separate worktree at ~/litellm-cron-worktree/.
mkdir -p ~/litellm
git clone https://github.com/BerriAI/litellm.git ~/litellm/litellm
# 4. gh auth — must be a collaborator on BerriAI/litellm-docs.
gh auth login # follow prompts; pick HTTPS + token paste flow
# 5. Provider credentials.
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.env.example \
/etc/litellm-compat-matrix.env
sudoedit /etc/litellm-compat-matrix.env # fill in real values
sudo chmod 0600 /etc/litellm-compat-matrix.env
# 6. systemd units.
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.service /etc/systemd/system/
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now litellm-compat-matrix.timer
```
## Operating it
```bash
# When does it run next?
systemctl list-timers litellm-compat-matrix.timer
# Trigger a real run right now (PRs to litellm-docs).
sudo systemctl start litellm-compat-matrix.service
# Trigger a run that does NOT open a PR (good for first-time validation).
SKIP_PUBLISH=1 ~/litellm/litellm/tests/claude_code/cron_vm/run_daily.sh
# Narrow to one cell while debugging.
SKIP_PUBLISH=1 PYTEST_K='basic_messaging_non_streaming and anthropic' \
~/litellm/litellm/tests/claude_code/cron_vm/run_daily.sh
# Watch the most recent run.
journalctl -u litellm-compat-matrix.service -f
# Read older runs.
journalctl -u litellm-compat-matrix.service --since '2 days ago'
# Disable until further notice (e.g. while debugging).
sudo systemctl disable --now litellm-compat-matrix.timer
```
## Gotchas
- **The proxy port is `4100`, not `4000`.** This is so a developer SSH'd
into the same VM with their own `:4000` proxy doesn't collide with a
cron run. Override with `PROXY_PORT=...` in `/etc/litellm-compat-matrix.env`
if you need to.
- **`uv sync --frozen` requires the resolved tag to be tagged on
GitHub.** If the latest stable release was made but not pushed as a
git tag, the `git checkout` step fails. Push the tag, then rerun.
- **`gh auth` token rotation is your problem.** The cron does not
refresh the token; if the bot account's PAT expires the run will
fail at `gh repo clone` with a 401. Re-run `gh auth login`.
- **First run after upgrading the Claude Code CLI is the riskiest one.**
If the new CLI changes its wire format the matrix run can produce
systematic failures. Always run with `SKIP_PUBLISH=1` after a CLI
upgrade before letting the next scheduled fire happen.
- **Disk:** the worktree's `.venv` is ~1.3 GB and the `.git` directory
is ~1 GB. Plan for at least 5 GB free on the VM, otherwise
`uv sync` will fail mid-run and leave you with a half-installed venv.
Reference in a new issue View git blame Copy permalink