- run_daily.sh: extract semver via grep -oE instead of awk '{print $1}' so
the parsed version survives a 'claude --version' output that ever prepends
a label (e.g. 'Claude Code vX.Y.Z'). The previous awk pattern would silently
publish the wrong string in that case.
- _driver_unit_tests/test_basic_messaging.py: narrow pytest.raises(BaseException)
to pytest.raises(pytest.fail.Exception). pytest.fail() raises Failed, which
inherits from BaseException; the new bound matches what the helper raises
without also swallowing KeyboardInterrupt/SystemExit.
The release pagination loop in run_daily.sh used to break the moment a
page contained any v*-stable tag. GitHub's /releases endpoint orders by
created_at, not semver, so a freshly-cut backport on an older series
(e.g. v1.80.1-stable published today) can appear on an earlier page than
a higher-versioned release (v1.83.0-stable published two weeks ago).
The early-break would silently pin the cron to the stale tag because
the higher-versioned release on a later page never made it into the
merged set the final sort_by consumed — and the cron would publish a
compatibility matrix against a stale LiteLLM version with no visible
signal that anything was wrong.
Keep the empty-page guard (so a quiet release feed still doesn't burn
through the full 5-page cap) but drop the broken early-break.
Tests live in tests/claude_code/_publisher_unit_tests/ (mirroring the
existing _driver_unit_tests / _builder_unit_tests / _pr_gate_unit_tests
naming convention already excluded from the PR-gate pytest run). They:
- Statically assert the buggy length>0 + break combo is not in the
pagination loop body.
- Statically assert the empty-page guard is still in place.
- Drive the actual run_daily.sh resolution snippet with a fake curl
whose page 1 contains a low-version backport stable and page 2
contains the high-version stable, then assert that the high-version
tag is the one resolved. This is the end-to-end regression test.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The cron host has no write access to BerriAI/litellm-docs by design. PRs
now open from a long-lived fork at agent-shin/litellm-docs:
- run_daily.sh validates AGENT_SHIN_GITHUB_TOKEN up front (failing 30 min
into a run because the env file is missing one line is wasted spend).
- The pre-commit shim adds a transient `fork` remote with the token
embedded in the URL, force-pushes the branch, then removes the remote
so the token never lives on disk.
- `gh pr create --head agent-shin:<branch>` opens the cross-repo PR
with GH_TOKEN scoped to AGENT_SHIN_GITHUB_TOKEN. A second
`gh pr edit --add-reviewer` runs under GITHUB_TOKEN (mateo-berri's
PAT) because agent-shin's PAT lacks RequestReviewsByLogin permission
on the upstream repo.
- PR_REVIEWERS env var (default `mateo-berri`) controls who gets
auto-tagged; empty disables.
Also bring litellm-compat-matrix.service to working state:
- Hardcode `/home/mateo` paths everywhere %h was used. systemd expands
%h against the *manager's* home (/root for PID 1) in *system* units,
not against the User= directive. The mismatch made ReadWritePaths
point at /root/.cache and the namespace setup failed with
status=226/NAMESPACE before run_daily.sh ever started.
- Explicit Environment=PATH so `uv` and `claude` under
~/.local/bin are visible to the up-front command-presence check;
systemd's default PATH excludes them.
- Expand ReadWritePaths to include ~/.claude (CLI per-session state)
and ~/.config/gh (gh host config fallback); both are written under
ProtectHome=read-only.
env.example refreshed: drop AWS_ACCESS_KEY_ID/SECRET +
GOOGLE_APPLICATION_CREDENTIALS in favor of AWS_BEARER_TOKEN_BEDROCK and
ADC via the VM's metadata server; document AGENT_SHIN_GITHUB_TOKEN,
FORK_OWNER/FORK_REPO overrides, and VERTEXAI_LOCATION=global.
The shim was previously gated on `tests/claude_code/test_config.yaml`
not existing in the worktree, so the moment a v*-stable tag landed
with its own copy of `tests/claude_code/` the cron would happily run
whatever frozen tests that release happened to ship — even though the
populator's whole purpose is to exercise *today's* tests against
*today's* stable proxy.
Drop the if-guard and make the shim unconditional: `rm -rf` the
worktree's `tests/claude_code/` after the tag checkout, then
`cp -r` the dev checkout's copy back in. This way fixes that land
on the dev checkout (like the stream-json vision rewrite, the
`--effort` thinking knob, the WebSearch tool_use assertion change)
take effect on the next cron run rather than waiting for a stable
release. The tag's own version of the tree, if any, is discarded each
run, so the worktree is byte-identical to
`${LITELLM_REPO}/tests/claude_code` every time.
Also drops `-e tests/claude_code` from the post-checkout `git
clean` invocation. We used to keep the prior run's shim across
`git clean -fdx` so subsequent runs didn't have to re-shim;
that's no longer necessary now that we re-shim unconditionally,
and keeping it would only delay garbage-collecting deleted files
across runs.
Bugbot flagged that the awk extracting PINNED_UV_VERSION required the
value to start with a specifier (`==`, `>=`, etc.) and silently
fell through to the system uv if pyproject.toml later switched to a
bare `required-version = "0.10.9"` form. Strip any leading
specifier prefix from the quoted value rather than requiring one to
be present, so both prefixed and bare forms resolve to the same
download URL.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Cursor security review flagged that the cron VM proxy binds to
0.0.0.0 (litellm's default when --host is omitted) while running
under the predictable default `LITELLM_MASTER_KEY=sk-cron-matrix`.
On a VM where :PROXY_PORT is reachable, anything that can hit the
port can authenticate with the predictable fallback secret and burn
upstream provider credentials.
The populator proxy is only ever talked to by the same-host pytest
run (the health check and tests both use http://127.0.0.1:${PORT}),
so there's no reason for it to listen on external interfaces. Pass
`--host 127.0.0.1` explicitly.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Cursor security review flagged that the cron VM downloads the uv
release tarball and pipes it straight through `tar -xzO ... > file ;
chmod +x`, with no integrity check. A tampered release artifact would
execute in a credentialed cron context with access to the docs-repo
push token.
Download the tarball + the official .sha256 sidecar Astral publishes
alongside every uv release to a temp dir, run `sha256sum -c` against
the sidecar, and only extract+install on success. On mismatch we wipe
the tempdir and `die` with a clear refusal.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Bugbot flagged that the resolver fetches only the first page (default
30 entries) of the releases endpoint. LiteLLM ships multiple non-stable
releases per day, so 30+ non-stable releases between consecutive
v*-stable tags is routinely the case — when it happens, the jq filter
matches nothing, LITELLM_VERSION is empty, and the daily matrix update
silently dies.
Walk pages 1..5 (100 per page = 500 releases max) and short-circuit as
soon as a page contains at least one v*-stable tag. Same final jq
filter, same sort order.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Add pytest_sessionstart hook in tests/claude_code/conftest.py that
clears the module-level _COLLECTOR so results don't leak across
pytest.main() invocations within the same process.
- Export LITELLM_MASTER_KEY=${PROXY_API_KEY} when launching the cron
VM proxy so the auth token the tests send actually matches what
the proxy expects.
The resolver picks the latest `v*-stable` tag of BerriAI/litellm. Until
the compat matrix work itself lands in a stable release, that tag's
tree won't contain `tests/claude_code/` at all — the proxy config and
test files only exist on the work-in-progress stack. Without a shim,
the cron dies with 'proxy config not found at .../test_config.yaml'
on every run.
Fix: after `git checkout <tag>`, if
`<worktree>/tests/claude_code/test_config.yaml` is missing, copy the
directory from ${LITELLM_REPO} (the dev checkout, which has the
in-flight matrix work). The `git clean -e tests/claude_code` line
preserves the shim across runs.
Once the matrix work is in the resolved tag, the if-branch is a no-op
and the shim is never used. No code path needs to be removed later;
the bridge self-disables.
Tier 1 (single anthropic cell) and tier 2 (full
basic_messaging_non_streaming row across 5 providers) now run cleanly
from `run_daily.sh` on the GCP VM. Five issues showed up during
validation; each is fixed in this commit.
1. uv version pin
----------------
The litellm worktree pins an exact uv version in
`pyproject.toml`'s `[tool.uv] required-version` field. The cron
VM's system uv (currently 0.11.8) refused to sync against the
v1.83.10-stable lockfile (which pins ==0.10.9). Fix: parse the
pinned version out of the worktree's pyproject.toml, download the
matching standalone binary into `<worktree>/.uv-bin/uv-<version>`,
and use it for sync/run/proxy. Cached across runs.
2. Missing proxy + dev extras
---------------------------
`uv sync --frozen` only installed the base dependency set, so
`uv run litellm` died at startup with
`ModuleNotFoundError: No module named 'websockets'`. Per the
repo's AGENTS.md the canonical incantation is
`uv sync --frozen --group proxy-dev --extra proxy`.
3. Wrong env vars for the test driver
-----------------------------------
The script was setting `ANTHROPIC_BASE_URL` and
`ANTHROPIC_AUTH_TOKEN`, which is what Claude Code itself reads,
but the test files read `LITELLM_PROXY_BASE_URL` and
`LITELLM_PROXY_API_KEY` (search the test_config-driven test files
for `PROXY_BASE_URL_ENV = "LITELLM_PROXY_BASE_URL"`). Tests were
marking themselves `fail` with
"missing required env: set LITELLM_PROXY_BASE_URL...". Fix: rename
the two env vars in the pytest invocation. The driver still
propagates them onward as ANTHROPIC_BASE_URL/AUTH_TOKEN to Claude.
4. Cleanup couldn't find the proxy
--------------------------------
The previous setup did
`( ... && setsid uv run litellm ... ) &; PROXY_PID=$!`. `setsid`
detaches the inner uv into its own session, but `$!` records the
PID of the outer subshell, not the long-lived python proxy. So
`kill -TERM "-${PROXY_PID}"` in the EXIT trap targeted the wrong
pgid and the proxy survived as an orphan whenever the script was
killed externally. Fix: replace the subshell with
`setsid bash -c '...'` that writes $$ to a known pid file before
exec'ing the proxy. The cleanup trap reads that file and uses it
as the pgid. Belt-and-braces: the trap also `pgrep -f`s by port
number and SIGKILLs survivors. Trap now fires on `INT TERM` too,
not just `EXIT`.
5. .uv-bin cache survives git clean
---------------------------------
The original `git clean -fdx -e .venv` wiped `.uv-bin/` between
runs, forcing re-download of the pinned uv binary on every
invocation. Now excluded.
Things that worked first try
----------------------------
* Worktree clone + checkout to the resolved tag.
* gh auth on this VM (mateo-berri account, collaborator on
BerriAI/litellm-docs).
* The matrix builder produced a well-formed
`compatibility-matrix.json` with the right per-cell aggregation
even when 4 of 5 cells failed (tier 2 was: anthropic=pass,
bedrock_invoke=fail, bedrock_converse=fail, vertex_ai=fail with a
real 403 from GCP for insufficient scopes, azure=fail with
timeout).
The previous iteration of this PR ported the populator to a Python
module (`publisher.py`) with 8 unit-tested pure helpers for branch
naming and PR-body rendering. After the docker code came out, the
GitHub App auth came out, and the worktree-vs-tempdir decision was
made, what was left was: 'git fetch + git checkout + uv sync + start
a subprocess + run pytest + clone docs + git commit + gh pr create'.
That's a bash script.
This commit replaces 800 lines of Python (publisher.py + resolver.py +
their unit tests) with a 297-line run_daily.sh and a 48-line
build_matrix.py whose only job is to be importable Python that can
call into the matrix_builder we already have. Net deletion: -822 lines.
Removed
-------
* tests/claude_code/publisher.py — the full Python orchestrator.
Every code path it had is now in run_daily.sh.
* tests/claude_code/resolver.py — the GitHub Releases v*-stable
resolver. Replaced by ~10 lines of jq inside run_daily.sh.
* tests/claude_code/_publisher_unit_tests/ — both test files. The
pure helpers they covered (commit message, file allowlist, branch
name, PR title/body) only existed because publisher.py was Python.
The bash equivalents are short heredoc strings.
Added
-----
* tests/claude_code/cron_vm/run_daily.sh — the actual cron job,
structured as numbered phases (resolve / worktree / proxy /
pytest / build / publish) so journalctl output is readable.
* tests/claude_code/cron_vm/build_matrix.py — a 48-line CLI that
calls the existing matrix_builder.build_from_paths. Kept in
Python because the builder itself is Python and well-tested.
Modified
--------
* tests/claude_code/cron_vm/litellm-compat-matrix.service —
ExecStart now invokes run_daily.sh instead of
'python -m tests.claude_code.publisher'.
* tests/claude_code/cron_vm/README.md — updated layout table,
file roles, and operating commands to match.
Why this is the right shape
---------------------------
* The failure mode at 06:00 UTC is 'read journalctl, see the literal
failing command with its + prefix, copy-paste it into a shell to
reproduce'. Bash makes that immediate; Python's subprocess.run
output looks similar but the surrounding orchestration is harder
to step through interactively.
* Every operation the script does is already a shell command (git,
uv, gh, jq, curl, pytest). The Python wrapper was translating
between argv arrays and back.
* The two pieces that genuinely benefit from being in a typed
language are matrix_builder (already Python) and the resolver's
semver sort (now done in jq, with the version_key tuple sort
inline). 'Already Python' wins, 'tiny jq pipeline' wins.
What's preserved
----------------
* Idempotency: same (litellm, claude, UTC date) -> same branch ->
same PR. force-with-lease push, gh-pr-create no-op-on-exists.
* Byte-identical-JSON early return (git diff --cached --quiet).
* Per-feature status table in the PR body (jq pipeline mirroring
the Python pr_body_for_matrix logic).
* Persistent worktree approach so disk doesn't grow unboundedly.
* Proxy bound to :4100 to avoid colliding with a developer's :4000.
* SKIP_PUBLISH=1 and PYTEST_K=... operator escape hatches.
The daily compat-matrix runs from the dedicated GCP VM
`litellm-compatibility-matrix-populator` rather than from a GitHub
Actions runner. The VM has no docker daemon, has `gh` already
authenticated against an account with `pull-requests: write` on
`BerriAI/litellm-docs`, and runs a long-lived litellm checkout we can
reuse across runs. That makes Docker, the GitHub App auth flow, and the
GHA workflow itself dead code.
Removed
-------
* `.github/workflows/claude_code_compat_matrix.yml` — no longer
triggers anything; the systemd timer in this PR owns the daily fire.
* `docker_image_for_tag` + `DOCKER_IMAGE_BASE` constants and the two
unit tests that covered them.
* `_start_proxy(image, port)` / `_stop_proxy(container_id)` /
`docker run` flow, replaced by direct `uv run litellm` subprocess
management with a sigterm-the-process-group teardown.
* `--skip-proxy` CLI flag (was only useful when the GHA workflow
split docker-bringup from publish into separate jobs).
* `docs_token` parameter and `DOCS_REPO_TOKEN` env var; `gh` on
the VM is already authenticated, so we don't pass an explicit
token through the publisher.
Added
-----
* Persistent worktree flow in `publisher.py`. First run clones
`BerriAI/litellm` into `~/litellm-cron-worktree/`; subsequent runs
do `git fetch --tags && git checkout --force <stable-tag> &&
uv sync --frozen`. Disk footprint is bounded because uv sync
removes packages no longer pinned and `git clean -fdx -e .venv`
wipes per-run cruft while keeping the venv around.
* `tests/claude_code/cron_vm/` containing systemd units and a
setup README:
- `litellm-compat-matrix.service` (`Type=oneshot`, runs as the
`mateo` user, sources `/etc/litellm-compat-matrix.env` for
provider creds, hardened with `NoNewPrivileges` /
`ProtectSystem=strict` / `PrivateTmp`);
- `litellm-compat-matrix.timer` (`OnCalendar=*-*-* 06:00:00 UTC`,
`Persistent=true` so a missed run fires when the VM is back up,
`RandomizedDelaySec=10min`);
- `.env.example` documenting the provider-credential surface;
- `README.md` covering one-time install, daily operation,
`journalctl` debugging, and the gotchas (proxy port `4100` to
avoid colliding with a developer's `:4000`, `gh` token
rotation, what to do after a Claude Code CLI upgrade).
Operator notes
--------------
* The proxy now binds `:4100` by default so a developer SSH'd into
the VM with their own `:4000` proxy isn't preempted by the cron.
* The Claude Code CLI is exercised as-is from the system install;
the populator does NOT `npm install` it. Operators upgrade the
CLI by running `npm install -g @anthropic-ai/claude-code@latest`
out of band, typically after watching a `--skip-publish` run to
verify the matrix doesn't suddenly turn red.
* 20 publisher unit tests pass (`pytest
tests/claude_code/_publisher_unit_tests/`).
* End-to-end validation on the VM happens after this PR lands as
follow-up commits on the same branch — the systemd unit is
`Type=oneshot` so a manual `systemctl start` reproduces the cron.
