mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-22 00:31:44 +00:00
fix(pr-gate): greptile — exclude web_search from gate and remove cron_vm README
This commit is contained in:
mateo-berri
parent
16f2eba313
commit
de75079b99
3 changed files with 11 additions and 142 deletions
|
|
@ -2474,6 +2474,16 @@ jobs:
|
|||
# row alone. The daily cron in
|
||||
# tests/claude_code/cron_vm/run_daily.sh still exercises
|
||||
# it and publishes the cell.
|
||||
#
|
||||
# `web_search` is excluded from the gate too: the
|
||||
# assertion ("a `tool_use` block with name=WebSearch
|
||||
# appeared") depends on the model deciding to call the
|
||||
# tool. `--allowed-tools WebSearch` permits the call but
|
||||
# cannot force it, and Haiku 4.5 in particular routinely
|
||||
# answers from training data instead. Running it inside
|
||||
# the merge-blocking gate would flake on non-deterministic
|
||||
# model behaviour; the daily cron still exercises the row
|
||||
# and publishes the cell to the matrix.
|
||||
env -i \
|
||||
PATH="$PATH" \
|
||||
HOME="$HOME" \
|
||||
|
|
@ -2492,6 +2502,7 @@ jobs:
|
|||
--ignore=tests/claude_code/_publisher_unit_tests \
|
||||
--ignore=tests/claude_code/_pr_gate_unit_tests \
|
||||
--ignore=tests/claude_code/long_context_1m \
|
||||
--ignore=tests/claude_code/web_search \
|
||||
--junitxml=test-results/junit.xml \
|
||||
--durations=10
|
||||
no_output_timeout: 30m
|
||||
|
|
|
|||
|
|
@ -1,141 +0,0 @@
|
|||
# Cron VM setup for the Claude Code compatibility-matrix populator
|
||||
|
||||
The populator runs daily on a dedicated GCP VM
|
||||
(`litellm-compatibility-matrix-populator`) rather than as a GitHub
|
||||
Action. Trade-offs:
|
||||
|
||||
- ✅ Real VM means we can `gh auth login` against an account that's
|
||||
already a collaborator on `BerriAI/litellm-docs`, instead of
|
||||
provisioning a GitHub App with `pull-requests: write`.
|
||||
- ✅ Persistent state (a single `~/litellm-cron-worktree/` and its `.venv`)
|
||||
is reused across runs, so each daily run does a fast `git checkout` +
|
||||
incremental `uv sync` rather than a fresh clone + cold sync.
|
||||
- ✅ No Docker dependency — the proxy runs directly via `uv run litellm`.
|
||||
- ⚠️ The VM has to actually be on. systemd's `Persistent=true` recovers
|
||||
from short outages, but a multi-day outage means the matrix goes
|
||||
stale until the VM is back.
|
||||
- ⚠️ Provider credentials live on the VM filesystem
|
||||
(`/etc/litellm-compat-matrix.env`) instead of GitHub secrets. Treat
|
||||
the VM as an environment with comparable blast radius to a CI runner.
|
||||
|
||||
## Layout
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `run_daily.sh` | The actual cron job. Resolves versions, updates the worktree, boots the proxy, runs pytest, builds the JSON, opens (or updates) a docs PR. |
|
||||
| `build_matrix.py` | Tiny Python CLI that wraps `tests.claude_code.matrix_builder.build_from_paths`. Exists only because the bash script needs *some* way to render the per-cell aggregation, and the builder is already Python. |
|
||||
| `litellm-compat-matrix.service` | systemd oneshot that invokes `run_daily.sh`. |
|
||||
| `litellm-compat-matrix.timer` | `OnCalendar=*-*-* 06:00:00 UTC`, `Persistent=true`. |
|
||||
| `litellm-compat-matrix.env.example` | Template for `/etc/litellm-compat-matrix.env`. |
|
||||
|
||||
## What `run_daily.sh` does
|
||||
|
||||
1. **Resolves the latest LiteLLM `v*-stable` tag** by hitting the
|
||||
GitHub Releases API (`curl | jq`).
|
||||
2. **Reads the local Claude Code CLI version** via `claude --version`.
|
||||
The cron does not auto-upgrade the CLI — operators do that
|
||||
out-of-band by running `npm install -g @anthropic-ai/claude-code@latest`.
|
||||
3. **Updates the persistent worktree** at `~/litellm-cron-worktree/`:
|
||||
`git fetch --tags --force`, `git reset --hard`,
|
||||
`git clean -fdx -e .venv`, `git checkout --force <tag>`. The
|
||||
`.venv` is preserved across runs so `uv sync --frozen` is
|
||||
incremental.
|
||||
4. **Boots the proxy** as a `setsid` background process on port `4100`
|
||||
(so it can't collide with a developer's `:4000`), then polls
|
||||
`/health/liveliness` until it's up.
|
||||
5. **Runs pytest** with `ANTHROPIC_BASE_URL` pointed at the proxy and
|
||||
`COMPAT_RESULTS_PATH` set so the conftest hook writes the per-test
|
||||
results artifact. Test failures become `fail` cells in the JSON,
|
||||
not script errors.
|
||||
6. **Builds `compatibility-matrix.json`** by handing the artifact +
|
||||
manifest to `build_matrix.py`.
|
||||
7. **Opens or updates a docs PR**: `gh repo clone` of `litellm-docs`
|
||||
into a tempdir, deterministic head branch
|
||||
(`compat-matrix/<litellm-version>-<claude-code-version>-<UTC-date>`),
|
||||
`--force-with-lease` push, `gh pr create`. A re-run on the same
|
||||
day fast-forwards the existing branch and `gh pr create` no-ops
|
||||
("a pull request for branch ... already exists" is treated as
|
||||
success).
|
||||
|
||||
## One-time VM setup
|
||||
|
||||
Run as `mateo` on the cron VM:
|
||||
|
||||
```bash
|
||||
# 1. Toolchain
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y git nodejs npm jq curl
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
sudo apt-get install -y gh # or follow https://cli.github.com/
|
||||
|
||||
# 2. Claude Code CLI (the cron does NOT auto-upgrade this; rerun this
|
||||
# line out-of-band when you want a fresh CLI to be tested)
|
||||
sudo npm install -g @anthropic-ai/claude-code@latest
|
||||
|
||||
# 3. Litellm checkout. Used by systemd's WorkingDirectory and as the
|
||||
# source of the .service / .timer files. The cron itself runs out
|
||||
# of the separate worktree at ~/litellm-cron-worktree/.
|
||||
mkdir -p ~/litellm
|
||||
git clone https://github.com/BerriAI/litellm.git ~/litellm/litellm
|
||||
|
||||
# 4. gh auth — must be a collaborator on BerriAI/litellm-docs.
|
||||
gh auth login # follow prompts; pick HTTPS + token paste flow
|
||||
|
||||
# 5. Provider credentials.
|
||||
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.env.example \
|
||||
/etc/litellm-compat-matrix.env
|
||||
sudoedit /etc/litellm-compat-matrix.env # fill in real values
|
||||
sudo chmod 0600 /etc/litellm-compat-matrix.env
|
||||
|
||||
# 6. systemd units.
|
||||
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.service /etc/systemd/system/
|
||||
sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.timer /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now litellm-compat-matrix.timer
|
||||
```
|
||||
|
||||
## Operating it
|
||||
|
||||
```bash
|
||||
# When does it run next?
|
||||
systemctl list-timers litellm-compat-matrix.timer
|
||||
|
||||
# Trigger a real run right now (PRs to litellm-docs).
|
||||
sudo systemctl start litellm-compat-matrix.service
|
||||
|
||||
# Trigger a run that does NOT open a PR (good for first-time validation).
|
||||
SKIP_PUBLISH=1 ~/litellm/litellm/tests/claude_code/cron_vm/run_daily.sh
|
||||
|
||||
# Narrow to one cell while debugging.
|
||||
SKIP_PUBLISH=1 PYTEST_K='basic_messaging_non_streaming and anthropic' \
|
||||
~/litellm/litellm/tests/claude_code/cron_vm/run_daily.sh
|
||||
|
||||
# Watch the most recent run.
|
||||
journalctl -u litellm-compat-matrix.service -f
|
||||
|
||||
# Read older runs.
|
||||
journalctl -u litellm-compat-matrix.service --since '2 days ago'
|
||||
|
||||
# Disable until further notice (e.g. while debugging).
|
||||
sudo systemctl disable --now litellm-compat-matrix.timer
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **The proxy port is `4100`, not `4000`.** This is so a developer SSH'd
|
||||
into the same VM with their own `:4000` proxy doesn't collide with a
|
||||
cron run. Override with `PROXY_PORT=...` in `/etc/litellm-compat-matrix.env`
|
||||
if you need to.
|
||||
- **`uv sync --frozen` requires the resolved tag to be tagged on
|
||||
GitHub.** If the latest stable release was made but not pushed as a
|
||||
git tag, the `git checkout` step fails. Push the tag, then rerun.
|
||||
- **`gh auth` token rotation is your problem.** The cron does not
|
||||
refresh the token; if the bot account's PAT expires the run will
|
||||
fail at `gh repo clone` with a 401. Re-run `gh auth login`.
|
||||
- **First run after upgrading the Claude Code CLI is the riskiest one.**
|
||||
If the new CLI changes its wire format the matrix run can produce
|
||||
systematic failures. Always run with `SKIP_PUBLISH=1` after a CLI
|
||||
upgrade before letting the next scheduled fire happen.
|
||||
- **Disk:** the worktree's `.venv` is ~1.3 GB and the `.git` directory
|
||||
is ~1 GB. Plan for at least 5 GB free on the VM, otherwise
|
||||
`uv sync` will fail mid-run and leave you with a half-installed venv.
|
||||
|
|
@ -32,7 +32,6 @@
|
|||
|
||||
[Unit]
|
||||
Description=Claude Code compatibility-matrix populator (oneshot)
|
||||
Documentation=file:///home/mateo/litellm/litellm/tests/claude_code/cron_vm/README.md
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue