From de75079b99404089c02dcb2def15f0f895582f86 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 19 May 2026 03:26:56 +0000 Subject: [PATCH] =?UTF-8?q?fix(pr-gate):=20greptile=20=E2=80=94=20exclude?= =?UTF-8?q?=20web=5Fsearch=20from=20gate=20and=20remove=20cron=5Fvm=20READ?= =?UTF-8?q?ME?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .circleci/config.yml | 11 ++ tests/claude_code/cron_vm/README.md | 141 ------------------ .../cron_vm/litellm-compat-matrix.service | 1 - 3 files changed, 11 insertions(+), 142 deletions(-) delete mode 100644 tests/claude_code/cron_vm/README.md diff --git a/.circleci/config.yml b/.circleci/config.yml index 6492794bef4..6ab1db9b4a8 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -2474,6 +2474,16 @@ jobs: # row alone. The daily cron in # tests/claude_code/cron_vm/run_daily.sh still exercises # it and publishes the cell. + # + # `web_search` is excluded from the gate too: the + # assertion ("a `tool_use` block with name=WebSearch + # appeared") depends on the model deciding to call the + # tool. `--allowed-tools WebSearch` permits the call but + # cannot force it, and Haiku 4.5 in particular routinely + # answers from training data instead. Running it inside + # the merge-blocking gate would flake on non-deterministic + # model behaviour; the daily cron still exercises the row + # and publishes the cell to the matrix. env -i \ PATH="$PATH" \ HOME="$HOME" \ @@ -2492,6 +2502,7 @@ jobs: --ignore=tests/claude_code/_publisher_unit_tests \ --ignore=tests/claude_code/_pr_gate_unit_tests \ --ignore=tests/claude_code/long_context_1m \ + --ignore=tests/claude_code/web_search \ --junitxml=test-results/junit.xml \ --durations=10 no_output_timeout: 30m diff --git a/tests/claude_code/cron_vm/README.md b/tests/claude_code/cron_vm/README.md deleted file mode 100644 index 7c05be00cf2..00000000000 --- a/tests/claude_code/cron_vm/README.md +++ /dev/null @@ -1,141 +0,0 @@ -# Cron VM setup for the Claude Code compatibility-matrix populator - -The populator runs daily on a dedicated GCP VM -(`litellm-compatibility-matrix-populator`) rather than as a GitHub -Action. Trade-offs: - -- ✅ Real VM means we can `gh auth login` against an account that's - already a collaborator on `BerriAI/litellm-docs`, instead of - provisioning a GitHub App with `pull-requests: write`. -- ✅ Persistent state (a single `~/litellm-cron-worktree/` and its `.venv`) - is reused across runs, so each daily run does a fast `git checkout` + - incremental `uv sync` rather than a fresh clone + cold sync. -- ✅ No Docker dependency — the proxy runs directly via `uv run litellm`. -- ⚠️ The VM has to actually be on. systemd's `Persistent=true` recovers - from short outages, but a multi-day outage means the matrix goes - stale until the VM is back. -- ⚠️ Provider credentials live on the VM filesystem - (`/etc/litellm-compat-matrix.env`) instead of GitHub secrets. Treat - the VM as an environment with comparable blast radius to a CI runner. - -## Layout - -| File | Purpose | -| --- | --- | -| `run_daily.sh` | The actual cron job. Resolves versions, updates the worktree, boots the proxy, runs pytest, builds the JSON, opens (or updates) a docs PR. | -| `build_matrix.py` | Tiny Python CLI that wraps `tests.claude_code.matrix_builder.build_from_paths`. Exists only because the bash script needs *some* way to render the per-cell aggregation, and the builder is already Python. | -| `litellm-compat-matrix.service` | systemd oneshot that invokes `run_daily.sh`. | -| `litellm-compat-matrix.timer` | `OnCalendar=*-*-* 06:00:00 UTC`, `Persistent=true`. | -| `litellm-compat-matrix.env.example` | Template for `/etc/litellm-compat-matrix.env`. | - -## What `run_daily.sh` does - -1. **Resolves the latest LiteLLM `v*-stable` tag** by hitting the - GitHub Releases API (`curl | jq`). -2. **Reads the local Claude Code CLI version** via `claude --version`. - The cron does not auto-upgrade the CLI — operators do that - out-of-band by running `npm install -g @anthropic-ai/claude-code@latest`. -3. **Updates the persistent worktree** at `~/litellm-cron-worktree/`: - `git fetch --tags --force`, `git reset --hard`, - `git clean -fdx -e .venv`, `git checkout --force `. The - `.venv` is preserved across runs so `uv sync --frozen` is - incremental. -4. **Boots the proxy** as a `setsid` background process on port `4100` - (so it can't collide with a developer's `:4000`), then polls - `/health/liveliness` until it's up. -5. **Runs pytest** with `ANTHROPIC_BASE_URL` pointed at the proxy and - `COMPAT_RESULTS_PATH` set so the conftest hook writes the per-test - results artifact. Test failures become `fail` cells in the JSON, - not script errors. -6. **Builds `compatibility-matrix.json`** by handing the artifact + - manifest to `build_matrix.py`. -7. **Opens or updates a docs PR**: `gh repo clone` of `litellm-docs` - into a tempdir, deterministic head branch - (`compat-matrix/--`), - `--force-with-lease` push, `gh pr create`. A re-run on the same - day fast-forwards the existing branch and `gh pr create` no-ops - ("a pull request for branch ... already exists" is treated as - success). - -## One-time VM setup - -Run as `mateo` on the cron VM: - -```bash -# 1. Toolchain -sudo apt-get update -sudo apt-get install -y git nodejs npm jq curl -curl -LsSf https://astral.sh/uv/install.sh | sh -sudo apt-get install -y gh # or follow https://cli.github.com/ - -# 2. Claude Code CLI (the cron does NOT auto-upgrade this; rerun this -# line out-of-band when you want a fresh CLI to be tested) -sudo npm install -g @anthropic-ai/claude-code@latest - -# 3. Litellm checkout. Used by systemd's WorkingDirectory and as the -# source of the .service / .timer files. The cron itself runs out -# of the separate worktree at ~/litellm-cron-worktree/. -mkdir -p ~/litellm -git clone https://github.com/BerriAI/litellm.git ~/litellm/litellm - -# 4. gh auth — must be a collaborator on BerriAI/litellm-docs. -gh auth login # follow prompts; pick HTTPS + token paste flow - -# 5. Provider credentials. -sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.env.example \ - /etc/litellm-compat-matrix.env -sudoedit /etc/litellm-compat-matrix.env # fill in real values -sudo chmod 0600 /etc/litellm-compat-matrix.env - -# 6. systemd units. -sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.service /etc/systemd/system/ -sudo cp ~/litellm/litellm/tests/claude_code/cron_vm/litellm-compat-matrix.timer /etc/systemd/system/ -sudo systemctl daemon-reload -sudo systemctl enable --now litellm-compat-matrix.timer -``` - -## Operating it - -```bash -# When does it run next? -systemctl list-timers litellm-compat-matrix.timer - -# Trigger a real run right now (PRs to litellm-docs). -sudo systemctl start litellm-compat-matrix.service - -# Trigger a run that does NOT open a PR (good for first-time validation). -SKIP_PUBLISH=1 ~/litellm/litellm/tests/claude_code/cron_vm/run_daily.sh - -# Narrow to one cell while debugging. -SKIP_PUBLISH=1 PYTEST_K='basic_messaging_non_streaming and anthropic' \ - ~/litellm/litellm/tests/claude_code/cron_vm/run_daily.sh - -# Watch the most recent run. -journalctl -u litellm-compat-matrix.service -f - -# Read older runs. -journalctl -u litellm-compat-matrix.service --since '2 days ago' - -# Disable until further notice (e.g. while debugging). -sudo systemctl disable --now litellm-compat-matrix.timer -``` - -## Gotchas - -- **The proxy port is `4100`, not `4000`.** This is so a developer SSH'd - into the same VM with their own `:4000` proxy doesn't collide with a - cron run. Override with `PROXY_PORT=...` in `/etc/litellm-compat-matrix.env` - if you need to. -- **`uv sync --frozen` requires the resolved tag to be tagged on - GitHub.** If the latest stable release was made but not pushed as a - git tag, the `git checkout` step fails. Push the tag, then rerun. -- **`gh auth` token rotation is your problem.** The cron does not - refresh the token; if the bot account's PAT expires the run will - fail at `gh repo clone` with a 401. Re-run `gh auth login`. -- **First run after upgrading the Claude Code CLI is the riskiest one.** - If the new CLI changes its wire format the matrix run can produce - systematic failures. Always run with `SKIP_PUBLISH=1` after a CLI - upgrade before letting the next scheduled fire happen. -- **Disk:** the worktree's `.venv` is ~1.3 GB and the `.git` directory - is ~1 GB. Plan for at least 5 GB free on the VM, otherwise - `uv sync` will fail mid-run and leave you with a half-installed venv. diff --git a/tests/claude_code/cron_vm/litellm-compat-matrix.service b/tests/claude_code/cron_vm/litellm-compat-matrix.service index 5a93df66cc9..01f0de301de 100644 --- a/tests/claude_code/cron_vm/litellm-compat-matrix.service +++ b/tests/claude_code/cron_vm/litellm-compat-matrix.service @@ -32,7 +32,6 @@ [Unit] Description=Claude Code compatibility-matrix populator (oneshot) -Documentation=file:///home/mateo/litellm/litellm/tests/claude_code/cron_vm/README.md Wants=network-online.target After=network-online.target